Decrease load on DB #2291
Merged
Decrease load on DB #2291
+150
−83
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We have seen that in some cases, when storage is slow or the node is way behind on its event stream, it can trigger many fetches of old proposals, which other nodes have all garbage collected. These fetches accumulate over time, leading to a worsening problem. This change limits the number of parallelism in proposal fetching. Instead of spawning a new task dynamically each time we need to fetch a proposal, we spawn a fixed number of worker tasks, each of which will only fetch one proposal at a time. A scanner task follows the event stream and detects when a proposal needs to be fetched, then broadcasts the request to fetch it to the worker tasks. It will be picked up when a worker is free.
jbearer
requested review from
philippecamacho,
ImJeremyHe,
sveitser,
tbro and
imabdulbasit
as code owners
jbearer
changed the title
sveitser
reviewed
Nov 18, 2024
|
|
||
| // If we fail fetching the proposal, don't let it clog up the fetching task. Just push | ||
| // it back onto the queue and move onto the next proposal. | ||
| sender.broadcast_direct((view, leaf)).await.ok(); |
There was a problem hiding this comment.
Does this create a busy loop if the task is failing consistently?
There was a problem hiding this comment.
I guess it kind of does. I was thinking we would be rate limited by the finite number of workers and time spent waiting on I/O for each failure. But I guess that's not guaranteed, depending on what the failure is. I'll add a sleep here
rob-maron
approved these changes
Nov 18, 2024
|
Backport failed for Please cherry-pick the changes locally and resolve any conflicts. git fetch origin release-thehounds
git worktree add -d .worktree/backport-2291-to-release-thehounds origin/release-thehounds
cd .worktree/backport-2291-to-release-thehounds
git switch --create backport-2291-to-release-thehounds
git cherry-pick -x 4f4efbaee341f679fa33bf676e3b0f90904ce295 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We have seen that in some cases, when storage is slow or the node is way behind on its event stream, it can trigger many fetches of old proposals, which other nodes have all garbage collected. These fetches accumulate over time, leading to a worsening problem.
While debugging this, I also noticed that we don't "unfill" the leaf payload when storing anchor leaves, so DB performance gets worse when blocks are very large, because we are storing/loading these large payloads when we think we are only dealing with leaves.
This PR:
Limits the number of parallelism in proposal fetching. Instead of spawning a new task dynamically each time we need to fetch a proposal, we spawn a fixed number of worker tasks, each of which will only fetch one proposal at a time. A scanner task follows the event stream and detects when a proposal needs to be fetched, then broadcasts the request to fetch it to the worker tasks. It will be picked up when a worker is free.
Unfills leaf payloads before storing decided leaves.