- Description
- In the Linux kernel, the following vulnerability has been resolved:
ceph: fix i_nlink underrun during async unlink
During async unlink, we drop the `i_nlink` counter before we receive
the completion (that will eventually update the `i_nlink`) because "we
assume that the unlink will succeed". That is not a bad idea, but it
races against deletions by other clients (or against the completion of
our own unlink) and can lead to an underrun which emits a WARNING like
this one:
WARNING: CPU: 85 PID: 25093 at fs/inode.c:407 drop_nlink+0x50/0x68
Modules linked in:
CPU: 85 UID: 3221252029 PID: 25093 Comm: php-cgi8.1 Not tainted 6.14.11-cm4all1-ampere #655
Hardware name: Supermicro ARS-110M-NR/R12SPD-A, BIOS 1.1b 10/17/2023
pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : drop_nlink+0x50/0x68
lr : ceph_unlink+0x6c4/0x720
sp : ffff80012173bc90
x29: ffff80012173bc90 x28: ffff086d0a45aaf8 x27: ffff0871d0eb5680
x26: ffff087f2a64a718 x25: 0000020000000180 x24: 0000000061c88647
x23: 0000000000000002 x22: ffff07ff9236d800 x21: 0000000000001203
x20: ffff07ff9237b000 x19: ffff088b8296afc0 x18: 00000000f3c93365
x17: 0000000000070000 x16: ffff08faffcbdfe8 x15: ffff08faffcbdfec
x14: 0000000000000000 x13: 45445f65645f3037 x12: 34385f6369706f74
x11: 0000a2653104bb20 x10: ffffd85f26d73290 x9 : ffffd85f25664f94
x8 : 00000000000000c0 x7 : 0000000000000000 x6 : 0000000000000002
x5 : 0000000000000081 x4 : 0000000000000481 x3 : 0000000000000000
x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff08727d3f91e8
Call trace:
drop_nlink+0x50/0x68 (P)
vfs_unlink+0xb0/0x2e8
do_unlinkat+0x204/0x288
__arm64_sys_unlinkat+0x3c/0x80
invoke_syscall.constprop.0+0x54/0xe8
do_el0_svc+0xa4/0xc8
el0_svc+0x18/0x58
el0t_64_sync_handler+0x104/0x130
el0t_64_sync+0x154/0x158
In ceph_unlink(), a call to ceph_mdsc_submit_request() submits the
CEPH_MDS_OP_UNLINK to the MDS, but does not wait for completion.
Meanwhile, between this call and the following drop_nlink() call, a
worker thread may process a CEPH_CAP_OP_IMPORT, CEPH_CAP_OP_GRANT or
just a CEPH_MSG_CLIENT_REPLY (the latter of which could be our own
completion). These will lead to a set_nlink() call, updating the
`i_nlink` counter to the value received from the MDS. If that new
`i_nlink` value happens to be zero, it is illegal to decrement it
further. But that is exactly what ceph_unlink() will do then.
The WARNING can be reproduced this way:
1. Force async unlink; only the async code path is affected. Having
no real clue about Ceph internals, I was unable to find out why the
MDS wouldn't give me the "Fxr" capabilities, so I patched
get_caps_for_async_unlink() to always succeed.
(Note that the WARNING dump above was found on an unpatched kernel,
without this kludge - this is not a theoretical bug.)
2. Add a sleep call after ceph_mdsc_submit_request() so the unlink
completion gets handled by a worker thread before drop_nlink() is
called. This guarantees that the `i_nlink` is already zero before
drop_nlink() runs.
The solution is to skip the counter decrement when it is already zero,
but doing so without a lock is still racy (TOCTOU). Since
ceph_fill_inode() and handle_cap_grant() both hold the
`ceph_inode_info.i_ceph_lock` spinlock while set_nlink() runs, this
seems like the proper lock to protect the `i_nlink` updates.
I found prior art in NFS and SMB (using `inode.i_lock`) and AFS (using
`afs_vnode.cb_lock`). All three have the zero check as well.
- Source
- 416baaa9-dc9f-4396-8d5f-8c081fb06d67
- NVD status
- Analyzed
- Products
- linux_kernel
[
{
"nodes": [
{
"cpeMatch": [
{
"criteria": "cpe:2.3:o:linux:linux_kernel:*:*:*:*:*:*:*:*",
"matchCriteriaId": "0B1A046E-0C62-426C-8D6D-9BCC6EB56D6E",
"versionEndExcluding": "5.10.253",
"versionStartIncluding": "5.7",
"vulnerable": true
},
{
"criteria": "cpe:2.3:o:linux:linux_kernel:*:*:*:*:*:*:*:*",
"matchCriteriaId": "20DDB3E9-AABF-4107-ADB0-5362AA067045",
"versionEndExcluding": "5.15.203",
"versionStartIncluding": "5.11",
"vulnerable": true
},
{
"criteria": "cpe:2.3:o:linux:linux_kernel:*:*:*:*:*:*:*:*",
"matchCriteriaId": "2EDC6BAF-B710-4E26-B6AA-D68922EE7B43",
"versionEndExcluding": "6.1.167",
"versionStartIncluding": "5.16",
"vulnerable": true
},
{
"criteria": "cpe:2.3:o:linux:linux_kernel:*:*:*:*:*:*:*:*",
"matchCriteriaId": "C57BB918-DF28-46B3-94F7-144176841267",
"versionEndExcluding": "6.6.130",
"versionStartIncluding": "6.2",
"vulnerable": true
},
{
"criteria": "cpe:2.3:o:linux:linux_kernel:*:*:*:*:*:*:*:*",
"matchCriteriaId": "28D591F5-B196-4CC9-905C-DC80F116E7A8",
"versionEndExcluding": "6.12.78",
"versionStartIncluding": "6.7",
"vulnerable": true
},
{
"criteria": "cpe:2.3:o:linux:linux_kernel:*:*:*:*:*:*:*:*",
"matchCriteriaId": "D394AC60-6F28-435F-872A-CCDF384B8331",
"versionEndExcluding": "6.18.19",
"versionStartIncluding": "6.13",
"vulnerable": true
},
{
"criteria": "cpe:2.3:o:linux:linux_kernel:*:*:*:*:*:*:*:*",
"matchCriteriaId": "E825E7C3-FEAC-4FD3-8A81-78D7387948C9",
"versionEndExcluding": "6.19.9",
"versionStartIncluding": "6.19",
"vulnerable": true
},
{
"criteria": "cpe:2.3:o:linux:linux_kernel:7.0:rc1:*:*:*:*:*:*",
"matchCriteriaId": "F253B622-8837-4245-BCE5-A7BF8FC76A16",
"vulnerable": true
},
{
"criteria": "cpe:2.3:o:linux:linux_kernel:7.0:rc2:*:*:*:*:*:*",
"matchCriteriaId": "4AE85AD8-4641-4E7C-A2F4-305E2CD9EE64",
"vulnerable": true
},
{
"criteria": "cpe:2.3:o:linux:linux_kernel:7.0:rc3:*:*:*:*:*:*",
"matchCriteriaId": "F666C8D8-6538-46D4-B318-87610DE64C34",
"vulnerable": true
}
],
"negate": false,
"operator": "OR"
}
]
}
]