Split storage ranges to parallelize execution #7733
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
16 tasks
LukaszRozmej
approved these changes
Nov 8, 2024
src/Nethermind/Nethermind.Synchronization/SnapSync/ProgressTracker.cs
Outdated
Show resolved
Hide resolved
| @@ -29,6 +29,7 @@ public class ProgressTracker : IDisposable | |||
| public const int HIGH_STORAGE_QUEUE_SIZE = STORAGE_BATCH_SIZE * 100; | |||
| private const int CODES_BATCH_SIZE = 1_000; | |||
| public const int HIGH_CODES_QUEUE_SIZE = CODES_BATCH_SIZE * 5; | |||
| private const uint StorageRangeSplitFactor = 2; | |||
There was a problem hiding this comment.
have you experimented with more radical splitting factors? 4? 8? 16? 32?
asdacap
reviewed
Nov 8, 2024
src/Nethermind/Nethermind.Synchronization/SnapSync/ProgressTracker.cs
Outdated
Show resolved
Hide resolved
|
Can add unit test? |
damian-orzechowski
force-pushed
the
feature/snap-sync-parallel-storage-2
branch
from
525c8aa
to
f734316
Compare
Added |
marcindsobczak
approved these changes
Nov 12, 2024
asdacap
approved these changes
Nov 12, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
At the end of state ranges sync, when only large storage trie is left to be synced, snap batches will be processed sequentially. This change splits the range left to be requested for a given storage trie, if the processed data is less than 50% of requested range (maybe this shouldn't be 50%). The aim is to speed up processing of large storage tries, especially towards the end of snap ranges phase, when progress gets "stuck" at 100%. This is to aid the same problem as #7688, but with a slightly different approach.
Tested on OP-Mainnet - time between logging 100% progress for State Ranges and finishing:
In both test runs in the last phase, there are messages with incorrect snap ranges response - missing hash in proofs. The #7741 might improve this.
UPDATE:
Looks really good with #7741
Unfortunately healing failed - can't merge
Types of changes
What types of changes does your code introduce?
Testing
Requires testing
If yes, did you write tests?
Notes on testing
Optional. Remove if not applicable.
Documentation
Requires documentation update
If yes, link the PR to the docs update or the issue with the details labeled
docs. Remove if not applicable.Requires explanation in Release Notes