CA-392674: nbd_client_manager retry connect on nbd device busy #6021
Conversation
This would be better fixed by adding a rule to udev to ensure that system-udevd leaves the devices alone and thus doesn't result in them being found busy. |
liulinC
force-pushed
the
private/linl/dev
branch
from
5b46ab3
to
802818f
Compare
XS8 stream udev handle the nbd device as well. (and has this race issue as well). BTW, udev rules is only run and executed with the device been opened and lock been hold by systemd-udevd(workers), So I prefer to resolve the races issues other than just totally ignore nbd devices in the udev rules. |
liulinC
force-pushed
the
private/linl/dev
branch
from
802818f
to
e230323
Compare
to connect to nbd devices, nbd_client_manager will 1. protect the operation with /var/run/nonpersistent/nbd_client_manager file lock 2. check whether nbd is being used by `nbd-client -check` 3. load nbd kernel module by `modprobe nbd` 4. call `nbd-client` to connect to nbd device However, step 3 will trigger systemd-udevd run asyncly, which would open and lock the same nbd devices, run udev rules, etc. This introduce races with step 4, e.g. both process want to open and lock the nbd device. Note: the file lock in step 1 does NOT resovle the issue here, as it only coordinate multiple nbd_client_manager processes. To fix the issue, - we patch nbd-client to report the device busy from kernel to nbd_client_manager - nbd_client_manager should check nbd-client exit code, and retry on device busy - nbd_client_manager call `udevadm settle` to wait for udevd parsing udev rules Note: checking nbd-client exit code is still necessary in case of racing with others Signed-off-by: Lin Liu <[email protected]>
liulinC
force-pushed
the
private/linl/dev
branch
from
e230323
to
1e55415
Compare
|
job: 4126316~4126325 looks good. |
|
Can we get reviews from @MarkSymsCtx and @robhoes ? |
to connect to nbd devices, nbd_client_manager will
nbd-client -checkmodprobe nbdnbd-clientto connect to nbd deviceHowever, step 3 will trigger systemd-udevd run asyncly, which would open and lock the same nbd devices, run udev rules, etc. This introduce races with step 4, e.g. both process want to open and lock the nbd device.
Note: the file lock in step 1 does NOT resovle the issue here, as it only coordinate multiple nbd_client_manager processes.
To fix the issue,