Age | Commit message (Collapse) | Author | Files | Lines |
|
If there are more than 32 sockets "1 << s" doesn't work. Use "1L << s"
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
A misconfigured system appeared to have a huge number of sockets. The
code here did not handle this gracefully as the bitmask of sockets is
only 64-bits wide, so the extra sockets were not counted.
It doesn't seem worth changiing the code to support more sockets as such
systems do not exist.
Just check, warn, and exit if a socket > 63 is found.
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
On Intel there is a race between a memory controller reporting it
saw an error with CMCI and the consumption of an uncorrected error
reporting with machine check. If the CMCI wins the race, Linux
takes the page offline before any consumption can occur. Thus
there may be no machine check.
Some users want to explicity test the #MC recovery case. They
disable CMCI in the kernel with the boot flag "mce=no_cmci".
In this case there will always be a machine check. But the test
reports "fail" because it was expecting to se a CMCI.
Add a check to see if CMCI is disabled. If it is, mask out the
F_CMCI expectation.
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
These made sense in the early days of this tool when only a
few CPU models supported recovery from poisoned memory consumption.
But more new models support recovery than do not.
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
AMD EPYC CPUs also support APEI EINJ error injection. Tested on AMD Milan
and Genoa.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Support PCIe error injection, e.g. fatal error, through APEI EINJ
interface. Tested on ARM platform (Alibaba Yitian 710) and X86 platform
(Intel Sapphire Rapids).
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Lots of files declare the same EINJ related macros like EINJ_ETYPE and
functions like wfile(), include the same header files.
To simplify the code and make it easier to maintain, move all common EINJ
related operations to a header file.
[Tony: Move the code out of einj.h and into new file einj.c]
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
To support Guest Error injection, add two extra arguments:
- '-j': skip error injection, this step should do with host physical
address on host which creates GPA->HPA mappings for the guest.
- '-k': kick off trigger by writing a file from remote (host).
The steps to inject guest error are:
STEP 1: start a VM with a stdio monitor which allows giving complex
commands to the QEMU emulator.
qemu-system-aarch64 -enable-kvm \
-cpu host \
-M virt,gic-version=3 \
-m 8G \
-d guest_errors \
-rtc base=localtime,clock=host \
-smp cores=2,threads=2,sockets=2 \
-object memory-backend-ram,id=mem0,size=4G \
-object memory-backend-ram,id=mem1,size=4G \
-numa node,memdev=mem0,cpus=0-3,nodeid=0 \
-numa node,memdev=mem1,cpus=4-7,nodeid=1 \
-bios /usr/share/AAVMF/AAVMF_CODE.fd \
-drive driver=qcow2,media=disk,cache=writeback,if=virtio,id=alinu1_rootfs,file=/path/to/image.qcow2 \
-netdev user,id=n1,hostfwd=tcp::5555-:22 \
-serial telnet:localhost:4321,server,nowait \
-device virtio-net-pci,netdev=n1 \
-monitor stdio
QEMU 7.2.0 monitor - type 'help' for more information
(qemu) VNC server running on 127.0.0.1:5900
STEP 2: login guest and install ras-tools, then run `einj_mem_uc` to
allocate a page in userspace, dumps the virtual and physical address of the
page. The `-j` is to skip error injection and `-k` is to wait for a kick.
$ ./einj_mem_uc single -j -k
0: single vaddr = 0xffffbd88c400 paddr = 151f21400
STEP 3: run command `gpa2hpa` in QEMU monitor and it will print the host
physical address at which the guest's physical address addr is mapped.
(qemu) gpa2hpa 0x151f21400
Host physical address for 0x151f21400 (mem1) is 0x935757400
STEP 4: inject an uncorrected error via the APEI interface to the finally
translated host physical address on host.
echo 0x949a84400 > /sys/kernel/debug/apei/einj/param1
echo 0xfffffffffffff000 > /sys/kernel/debug/apei/einj/param2
echo 0x0 > /sys/kernel/debug/apei/einj/flags
echo 0x10 > /sys/kernel/debug/apei/einj/error_type
echo 1 > /sys/kernel/debug/apei/einj/notrigger
echo 1 > /sys/kernel/debug/apei/einj/error_inject
STEP 5: then kick `einj_mem_uc` to trigger the error by writing
"trigger_start". In this example, the kick is done on host.
ssh -p 5555 root@localhost "echo trigger > ~/trigger_start"
STEP 6: We will observe that the QEMU process exit.
(qemu) qemu-system-aarch64: Hardware memory error!
Signed-off-by: zhangyangzeyu.zyzy <xiaoque@linux.alibaba.com>
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Run einj_mem_uc in 3.10 kernel:
./einj_mem_uc: cannot open '/sys/kernel/debug/apei/einj/flags'
The 'flags' is added by 3482fb5e0c1c (ACPI, APEI, EINJ: Changes to the
ACPI/APEI/EINJ debugfs interface) on 3.14 kernel. Add kernel version
check.
Signed-off-by: Bixuan Cui <cuibixuan@linux.alibaba.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Fixes: 65d692c5ce8e (einj_mem_uc: Count Ice Lake Xeon as "advanced RAS")
Signed-off-by: Bixuan Cui <cuibixuan@linux.alibaba.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Trigger two UCE by reading from two target addresses at the same time.
The OVER(Error overflow) bit will be set and probably fatal.
Signed-off-by: Bixuan Cui <cuibixuan@linux.alibaba.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Some compilers (GCC 9.2.1) complain:
einj_mem_uc.c: In function ‘trigger_share’:
einj_mem_uc.c:656:3: error: a label can only be part of a statement and a declaration is not a statement
656 | char *p = mmap(NULL, pagesize, PROT_READ, MAP_SHARED, fileno(pcfile), 0);
| ^~~~
make: *** [<builtin>: einj_mem_uc.o] Error 1
Make all declarations precede all statements within the block.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Share memory is read by two tasks to target address.
[Tony: Fix some indenting. Rename page_cache_alloc() to
map_file_alloc() now it is used for another test]
Signed-off-by: Bixuan Cui <cuibixuan@linux.alibaba.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
There are multiple implementations of vtop() function, remove extra copies
of the vtop() function and use the one from proc_pagemap.c
Suggested-by: Luck, Tony <tony.luck@intel.com>
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Victim workes under user context, which provides target memory chunk for
error injection. It can be used for all kinds of error types,
including Corrected error and Uncorrected error(IFU/DCU).
Here is an simple example for DCU:
Mmap one page memory and returns starting address, and then translate
virtual address to physical address. Caller like shell script can
inject UC error (error type 0x10 in EINJ table) on returned physical
address. Meanwhile, victim continues to read/write on returned memory
space to trigger DCU happening ASAP.
NOTE: this workload is borrowed from mce-test. Thanks to the origial
authors, Tony Luck, Gong Chen, Wen Jin.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Move the memattr test case out of driver directory and rename it
with a generic name.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Some firmwares support advanced RAS but does not fill in the vendor id,
e.g. Kunpeng BIOS v1.91. Users complain that they can not use ras-tools
directly.
Therefore, relax vendor id check and just print an warning.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Some compilers complain:
mca-recover.c:135:9: warning: ignoring return value of ‘fgets’ \
declared with attribute ‘warn_unused_result’ [-Wunused-result]
Check the return value (even though it doesn't really matter).
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
If the copy-on-write test passes, then the child process gets a
SIGBUS and longjmp's back to the main loop. If "-c" had specified
to repeat the test, both parent and child will go back around.
Later loops will also include the grand-children and cousins!
Set a flag to break out of the loop for the child of a copy-on-write
test.
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
hornet use ptrace(2) with PTRACE_GETREGS request to read the tracee's
general-purpose registers, but it does not work on arm64 platform.
To extend hornet on both X86 and arm64 platform, retrieve rip or PC in an
architecture-dependent way when PTRACE_GETREGS is not available.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Someday I'd like to fix this case in the kernel. For now just
create a test case to generate the issue.
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Check for MAP_FAILED instead.
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Compiler complains (correctly) that the format string specifies
two additional arguments, but only one is present. Fix it.
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
These tools are all under GPLv2. Add the missing LICENSE.
Tony: cherry-picked from https://gitee.com/anolis/ras-tools.git
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Tony: Cherry-picked from https://gitee.com/anolis/ras-tools.git with
the last paragraph about being a clone of this repo dropped.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Lots of bugs fixes & cleanups. Plus ARM support!
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
When a hardware error occurs for a non corrected ras event the kernel can
take different actions. If the severity is fatal, the kernel panic
immediately.
This driver allows to overwrite error severity to a lower level at runtime,
recoverable by default. It is useful for test.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
|
|
This patch add
- pgprot_drv: a driver that allows a user-space program to mmap a buffer of
contiguous physical memory with specific memory attribute.
- test.c: a test case to poison the remaped memory.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
|
|
These tools are all under GPLv2. Add the missing tags.
Reported-by: Jiaqi Yan <jiaqiyan@google.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Single read by two threads to target address at the same time.
Signed-off-by: Bixuan Cui <cuibixuan@linux.alibaba.com>
|
|
The current sig action only prints fault address and restores the
environment saved before and , we can not tell the SIGBUS reason.
Therefore, explictly print si_code, 4 for BUS_MCEERR_AR, and 5 for
BUS_MCEERR_AO.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
|
|
The Advanced ECC X4 employs symbol-based Reed-Solomon encoding. One symbol
is 8 bits, message (data) length is 32 symbols (256 bits), ECC parity
length is 4 symbols (32 bits). When we inject a UC error, the platfrom may
only poison 32 symbols, in other words, only half cacheline is poisoned.
Therefore, add a parameter to trigger with offset, e.g:
./einj_mem_uc single #equals to ./einj_mem_uc -z 0 single
./einj_mem_uc -z 32 single
In such scenario, the former will signal a exception while the latter will
not.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
|
|
Einj interrupt may be a SPI on arm64 and could be dispatched to any core,
so the current process could run trigger action before the injection takes
effect.
Add a sleep to wait for injection completion, and then trigger.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
|
|
Add cases for platform specific, including CMN, GIC, SMMU, etc.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
|
|
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
|
|
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
|
|
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
|
|
Add cases to explicitly trigger write with STR, STRB, and STRH
instruction on Arm64 platform.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
|
|
On some platform, write to different offset within the poison cacheline
performs differently. Add a z flag so that we could trigger write with
an offset.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
|
|
The error injection mechanism is a two-step process. First inject the
error, then perform some actions to trigger it. When the system is in
early kill mode, trigger step is not needed. Explicitly print step
which are runnig on, so we can tell the how the error occurs.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
|
|
There is only X86 assembly version of memcpy, add Arm64 version for
memcpy case.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
|
|
The vendor interface of EINJ provides vendor_id, device_id, rev_id, etc.
Check advanced RAS support by vendor id.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
|
|
It is not necessary to check configuration when use madvise(2) to
simulate poison on a page. Add Sflag as condition.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
|
|
eing_mem_uc injects Memory Uncorrectable non-fatal errors through APEI
Error INJection (EINJ) interface, which is arch independent. However,
einj_mem_uc fails to compile due to arch dependent configuration checks.
Simply surround target arch macros to avoid compile error so we could debug
and test with this tool in both X86 and arm platforms.
Signed-off-by: Shuai Xue <xueshuai@linux.alibaba.com>
|
|
AMD systems don't use CMCI in the same way as Intel systems.
Add a flag to skip reporting of CMCI counts.
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Optional argument to rep_ce_page for how many times to inject
a corrected error to the target page.
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
This test injects and consumes corrected errors from a single page
until either the page is taken offline (and replaced) by the OS, or
a limit of 30 tries is reached.
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
It may take up to 20 seconds for the patrol scrubber to restart
and scan to the specific location where the error was injected.
Add a flag to indicate that the test should wait for much longer
to check for patrol scrub CMCI.
Also change the message to print the actual delay in units of
seconds, not microseconds (since the values are large enough
that this is a more human readable format).
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
When we detached the pftraced process, we miss out isolating the poisoned
page. Adding a goto statement to make sure this.
Signed-off-by: 启瑞 <qirui.001@bytedance.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Commit 38f47153c2c1 ("Check the injected error type available before
write error type") didn't skip the extra text on the end of each line
when checking whether a specific error type is supported.
Reported-by: Liu Xinpeng <liuxp11@chinatelecom.cn>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Complier is grousing about missing prototypes for strncmp()
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
before:
0: llc vaddr = 0x7fee6b865400 paddr = f90e0eb400
./einj_mem_uc: write error on '/sys/kernel/debug/apei/einj/error_type'
after:
0: llc vaddr = 0x7f86e6bac400 paddr = f915477400
./einj_mem_uc: no support for error type: 0x2
[Tony: re-word error message]
Signed-off-by: Liu Xinpeng <liuxp11@chinatelecom.cn>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
The number of arguments must be three.
Signed-off-by: Yizhan Xu <yizhan.xu@intel.com>
|
|
An error occurs during the execution of the einj_mem_uc "instr" test
and exits. The error message shows that the instr memory page is not
found, and the einj_mem_uc exits. The detailed log is as follows:
$ ./einj_mem_uc -c 10 -f instr
page not present
0: instr vaddr = 0x403000 paddr = ffffffffffffffff
~/einj_mem_uc : write error on '/sys/kernel/debug/apei/einj/error_inject'
Test Environment:
OS Version:
CentOS Linux release 7.9.2009 (Core)
Kernel Version:
3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64
Solution: Call the dosum function in instr_alloc to load the dosum
memory page in advance to prevent vtop conversion failure.
Signed-off-by: Zhongyu Gao <gzy@sangfor.com.cn>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Starting with Icelake Xeon patrol scrub errors are signalled using
CMCI with a UCNA signature instead of a machine check with an SRAO
signature.
Add a new flag "F_EITHER" to indicate that CMCI or MCE (but not both)
is an acceptable response for the patrol scrub test.
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
All SKUs of Ice Lake Xeon support the memory recovery advanced
RAS feature.
Just check for the model number instead of looking for Platinum/Gold
in the model name.
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
[Also changed flags for the copyin case to remove F_SIGBUS]
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
1.In if check, ret should be compared to memcpy_size;
2.In else branch, correct the fprintf parameter order.
Signed-off-by: Aili Yao <yaoaili@kingsoft.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Instead of using ACPI/EINJ to inject a real error, use madvise(2)
to simulate poison on a page. Note that this only has any effect for
tests that use inject_uc() for injection ... and doesn't really match
the behavior with a real injection.
[Idea to use MADV_HWPOISON from Aili Yao <yaoaili@kingsoft.com>]
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
For ".o" files and executables
|
|
It seems that modern compilers have become smart enough to not just
blindly read from "buf" at the point where the C code says "i = buf[0];"
Throw in a function call, and a volatile cast, to make sure the consumption
happens at the right point in code flow.
[Also fix one duplicated "0x" on output, and one missing "0x"]
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
When convering a page frame number to an address there is a hard-coded
shift by 12 but the mask for the low order bits is computed based on
the "pagesize" variable.
Fix this inconsistency by swapping out the shift for a multiply.
Reported-by: 葛士建 <geshijian@bytedance.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
The copyin test fails at just over 1000 iterations because it can no
longer open new file descriptors.
The problem is that the test opens a file, and closes at the end of
the function. But since successful recover sends a SIGBUS, the close
is never executed.
Make the file descriptor global and close it in the main() function
after the normal or SIGBUS return paths.
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
In order to force different code paths in the kernel, allow user to
adjust start/size of kernel copy so that the poison is consumed at
different points in the copy.
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Bug in the code. The page_cache_alloc() function uses a local "FILE *pcfile"
instead of the global one. Result is that the trigger step gets a NULL
dereference accessing the global pcfile.
Reported-by: Youquan Song <youquan.song@intel.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
If the write fails then print out the system error code. If it
partially fails, print how many bytes were written.
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Create a file and write a page of data to it. Map that file and
use an address in the mapped range to inject an error.
Trigger the error by issing a read(2) system call which will
make the kernel copy from the page cache copy of the data (which
has been injected with a UC error).
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Validation teams using this test would find it easier to build into
a test script if it reported success/fail both with a message, and
by exit code.
Reported-by: Jun J Li <jun.j.li@intel.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Design different cases to validate LMCE feature:
1. multi thread run on same or different cores;
2. inject memory error into one same address or two different addresses;
3. trigger IFU or DCU error individually.
Note that injecting errors on the same core will likely result
in undefined behavior as logical processors sharing a core also
share machine check banks that log recoverable machine checks.
Signed-off-by: Jin Wen <wenx.jin@intel.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
1) Fix check_ptrace macro to stringify the "req" argument in error message
2) Set lo & hi so that pickaddr() will print reasonable range with "-v"
3) Replace NULL with empty string in verbose print
4) Remove unused "pagesize" variable.
Signed-off-by: Jin Wen <wenx.jin@intel.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Picking instruction addresses inside running processes is rather
hit or miss. We may pick an address for injection that is never
executed.
Using ptrace(2) to stop the process we can find the precise address
of the next instruction to be executed and thus guarantee that we
will immediately hit the injected address when we resume running the
process.
Signed-off-by: Jin Wen <wenx.jin@intel.com>
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
recovery
Both Platinum and Gold Skylake SKUs support advanced RAS recovery features
Reported-by: Youquan Song
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
Skylake (Xeon-SP) doesn't use the "E7-" model name convention.
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
|
|
I'd messed with this to do something with repeated recoveries. But
that doesn't match with the use case that we've been explaining to
people.
Go back to just one recovery.
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
|
|
writeback
|
|
Instead of hard coding a fixed large time to sleep before checking how
many CMCIs were logged, just sleep in 100us increments until we see at least
the expected number. Give up waiting after 1000 such rechecks. If we blocked
for any more than 100us, then report the actual time.
Original patch by Carl Sapp. Modified to use gettimeofday() to report actual delay.
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
for all cpus to wake from C6. Bump to 10ms.
Reported-by: Carl Sapp
Signed-off-by: Tony Luck <tony.luck@intel.com>
|
|
cmcistorm - inject a bunch of corrected errors, then trigger them all quickly
hornet - inject a UC memory error into some other process
einj_mem_uc - inject a UC error and then trigger it in one of a variety of ways.
|
|
|
|
This is pretty trivial - just shows how to setup a SIGBUS handler
for recoverable machine checks. Injection of the actual error is
handled externally (e.g. using EINJ).
Signed-off-by: Tony Luck <tony.luck@intel.com>
|