Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ioctl: get_log_page by nvme uring cmd #927

Merged
merged 2 commits into from
Dec 20, 2024

Conversation

996refuse
Copy link
Contributor

@996refuse 996refuse commented Dec 7, 2024

hello libnvme master,

io_uring on linux is pretty mature now, for the system with liburing, do the batch nvme_get_log by nvme uring cmd.
This change could gain about 10% performance improvement for some large log pages on my lab.

i believe the performance can be better if implement producer and consumer in two separated threads, if it's worth and acceptable, i can spend some time on this later.

thanks

src/nvme/ioctl.c Show resolved Hide resolved
src/nvme/ioctl.c Outdated
@@ -344,14 +431,30 @@ int nvme_get_log_page(int fd, __u32 xfer_len, struct nvme_get_log_args *args)
args->len = xfer;
args->log = ptr;
args->rae = offset + xfer < data_len || retain;
#ifdef CONFIG_LIBURING
if (n >= NVME_URING_ENTRIES) {
nvme_uring_cmd_wait_complete(&ring, n);
Copy link
Collaborator

@igaw igaw Dec 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am fine with using uring but would it be possible to have it fallback during runtime to the ioctl interface when uring is not available, like on older kernels?

src/nvme/ioctl.c Outdated Show resolved Hide resolved
src/nvme/ioctl.c Outdated Show resolved Hide resolved
@igaw
Copy link
Collaborator

igaw commented Dec 16, 2024

This should be just one commit. I'll squash the changes into one when I am reviewing and testing it. We need also to update our CI build accordingly.

@996refuse
Copy link
Contributor Author

thanks for your help !

there're 2 failed testcase. it requires mock liburing api

 8/34 ana                                              FAIL            0.01s   killed by signal 6 SIGABRT
10/34 discovery                                        FAIL            0.01s   killed by signal 6 SIGABRT

besides, it doesn't work on SPDK NVME CUSE devices. FYI

@igaw
Copy link
Collaborator

igaw commented Dec 16, 2024

there're 2 failed testcase. it requires mock liburing api

Ah, I've missed this. Unfortunately, this is important, I really don't want the tests cases to break when liburing is used.

besides, it doesn't work on SPDK NVME CUSE devices. FYI

That is likely due the missing support of the CUSE devices.

@igaw
Copy link
Collaborator

igaw commented Dec 16, 2024

I suppose, for an initial attempt the io_uring_opcode_supported(probe, IORING_OP_URING_CMD) should just fail.

io_uring_opcode_supported calls __sys_io_uring_register, this means you need to mock the io_uring_register:

https://github.com/axboe/liburing/blob/6aeac2041987649c32a8f15de104ec4339725c49/src/arch/syscall-defs.h#L64C1-L69C2

static inline int __sys_io_uring_register(unsigned int fd, unsigned int opcode,
					  const void *arg, unsigned int nr_args)
{
	return (int) __do_syscall4(__NR_io_uring_register, fd, opcode, arg,
				   nr_args);
}

I haven't looked into liburing at all, maybe it supports some sort of mocking for testing?

@996refuse
Copy link
Contributor Author

996refuse commented Dec 17, 2024

io_uring_get_probe returning NULL could avoid breaking the cases perfectly. after a quick study, think there are no official mocking support of liburing.

Workaround avoid breaking ana and discovery testcase

Signed-off-by: letli <[email protected]>
Use io_uring for fetching log pages.

This showed about a 10% performance improvement for some large log pages.

Signed-off-by: letli <[email protected]>
@igaw igaw force-pushed the nvme_uring_cmd_get_log_page branch from be29f8b to ddfbc51 Compare December 20, 2024 09:49
@igaw
Copy link
Collaborator

igaw commented Dec 20, 2024

Just squashed the changes into one patch. Thanks!

@igaw igaw merged commit adee4ed into linux-nvme:master Dec 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants