nvme TLS key insert fails with ENOKEY when modules is missing

[igaw]

When trying to insert a TLS key to the kernel keyring and the module is not loaded the error reported is

nvme check-tls-key -I 1 -c nqn.io -d NVMeTLSkey-1:01:QQ+JhVDGETH9daL0dqcOGfMDo71Xn4Ig2aCWcwqk1pamtw/5: -i
Failed to insert key, error 126

which maps to

ENOKEY 126 Required key not available

Extend error message to something meaningful.

[wangyugui-e16]

same error?

# nvme-2.11 connect-all -t tcp -a 192.168.2.76
Failed to set keyring
failed to add controller, error Required key not available
# nvme-2.11 --version
nvme version 2.11 (git 2.11-1-g64b2a25)
libnvme version 1.11 (git 1.11-2-g89ea2b7)

[igaw]

Is the nvme-tcp module loaded?

[wangyugui-e16]

Is the nvme-tcp module loaded?

nvme-tcp module is loaded.
but this issue happen only when build on rhel7, not happen when build on rhel9.

there seems some releationship to openssl version.

[igaw]

Does the rhel7 kernel announce support for tls? Can you post the output of cat /dev/nvme-fabrics for both kernels and the output from the meson setup steps? Is libkeyutils also enabled in the build?

But to answer your question, it's likely something else.

[bluca]

same error?

# nvme-2.11 connect-all -t tcp -a 192.168.2.76
Failed to set keyring
failed to add controller, error Required key not available
# nvme-2.11 --version
nvme version 2.11 (git 2.11-1-g64b2a25)
libnvme version 1.11 (git 1.11-2-g89ea2b7)

I am seeing the same issue in the systemd CI where nvme-cli is used. nvme-ctp is loaded:

[    4.741966] TEST-84-STORAGETM.sh[577]: + modprobe -v nvmet-tcp
[    4.750298] TEST-84-STORAGETM.sh[580]: insmod /lib/modules/6.11.5-cloud-amd64/kernel/drivers/nvme/target/nvmet.ko.xz
[    4.750298] TEST-84-STORAGETM.sh[580]: insmod /lib/modules/6.11.5-cloud-amd64/kernel/drivers/nvme/target/nvmet-tcp.ko.xz
[    4.750828] TEST-84-STORAGETM.sh[577]: + modprobe -v nvme-tcp
[    4.757204] TEST-84-STORAGETM.sh[585]: insmod /lib/modules/6.11.5-cloud-amd64/kernel/drivers/nvme/host/nvme-fabrics.ko.xz
[    4.757204] TEST-84-STORAGETM.sh[585]: insmod /lib/modules/6.11.5-cloud-amd64/kernel/drivers/nvme/host/nvme-tcp.ko.xz

The command, which used to work with nvme-cli 2.10, started failing since Debian upgraded to 2.11:

6234s [   25.416521] TEST-84-STORAGETM.sh[548]: + nvme connect-all -t tcp -a 127.0.0.1 -s 16858 --hostid=158f4d7d-6759-499e-b260-b152e49fa9a2
6234s [   25.446043] TEST-84-STORAGETM.sh[567]: Failed to set keyring
6234s [   25.446699] TEST-84-STORAGETM.sh[567]: failed to add controller, error Required key not available

Full strace output at systemd/systemd#35130 (comment)

Is setting up TLS keys required now to use nvme connect-all -t tcp?

This is on kernel 6.11.5. Note that Debian does not enable CONFIG_NVME_TCP_TLS and CONFIG_NVME_HOST_AUTH

[bluca]

Looks like a slightly different issue from the original one though, so opened a new one: #2573

[igaw]

@wangyugui-e16 should be fixed with libnvme v1.11.1. Was a stupid mistake on my side. Sorry about it.

[wangyugui-e16]

@wangyugui-e16 should be fixed with libnvme v1.11.1. Was a stupid mistake on my side. Sorry about it.

the problem here is fixed by libnvme v1.11.1. thanks a lot.

[wangyugui-e16]

Does the rhel7 kernel announce support for tls? Can you post the output of cat /dev/nvme-fabrics for both kernels and the output from the meson setup steps? Is libkeyutils also enabled in the build?

But to answer your question, it's likely something else.

rhel7 kernel does NOT announce support for tls.

the output of cat /dev/nvme-fabrics:

# cat /dev/nvme-fabrics
instance=-1,cntlid=-1,transport=%s,traddr=%s,trsvcid=%s,nqn=%s,queue_size=%d,nr_io_queues=%d,reconnect_delay=%d,ctrl_loss_tmo=%d,keep_alive_tmo=%d,hostnqn=%s,host_traddr=%s,host_iface=%s,hostid=%s,duplicate_connect,disable_sqflow,hdr_digest,data_digest,nr_write_queues=%d,nr_poll_queues=%d,tos=%d,fast_io_fail_tmo=%d,discovery

this problem is fixed in libnvme v1.11.1

[igaw]

Thanks @bluca 's report I understood what the issue was. Glad it is fixed now.