Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

u3: specify madv_random for file-backed snapshot mapping #423

Draft
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

joemfb
Copy link
Member

@joemfb joemfb commented May 31, 2023

Since #402 was released, there have been isolated reports of unusually high disk i/o, specifically reads. This is unexpected behavior -- our current, best guess is the kernel's readahead logic for file-backed mappings. This PR implements the standard solution to overeager readahead (MADV_RANDOM), as a draft since the diagnosis is not confirmed, and the advice may need to be behind a flag.

@mopfel-winrux
Copy link
Contributor

I was able to replicate the issue using v2.7 on Digital Ocean running 14 comets with 4GB of ram and 20GB of swap.

Switching over to this PR I tried the same thing and still saw ~40MB/s reads.

image

@mopfel-winrux
Copy link
Contributor

I ran
trace -fp [$PID] -Trfe trace=read,lseek 2> system.trace

with all the urbits running and got the below trace
https://nyc3.digitaloceanspaces.com/mopfel-winrux-s3/mopfel-winrux/2023.6.01..12.58.45-system.trace

@ashelkovnykov
Copy link
Contributor

For posterity: the actual issue was #427

@joemfb
Copy link
Member Author

joemfb commented Jun 14, 2023

Thanks @mopfel-winrux, @ashelkovnykov.

I think this advice is the right thing for north.bin regardless. And I'd rather not add yet-another flag to control it. But it's not pressing, and should be measured first. Leaving as a draft for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Discussion Needed
Development

Successfully merging this pull request may close these issues.

3 participants