Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(trie): use rayon for parallel storage proofs #13360

Closed
wants to merge 2 commits into from

Conversation

shekhirin
Copy link
Collaborator

@shekhirin shekhirin commented Dec 12, 2024

This removes the mpsc channels from the ParallelProof::multiproof implementation to solve issues with rayon threads getting blocked on rx.recv() when ParallelProof::multiproof is spawned inside a rayon task.

Instead, we calculate storage proofs in parallel using rayon iterators, collect all results, and then query them when calculating the account trie leaf nodes.

@shekhirin shekhirin force-pushed the alexey/parallel-proof-rayon-iter branch from 8cf9855 to 9daaa9f Compare December 12, 2024 17:24
@shekhirin shekhirin force-pushed the alexey/parallel-proof-rayon-iter branch from 9daaa9f to c75b951 Compare December 12, 2024 17:25
@shekhirin shekhirin marked this pull request as ready for review December 13, 2024 13:29
@shekhirin shekhirin marked this pull request as draft December 13, 2024 13:35
@shekhirin
Copy link
Collaborator Author

shekhirin commented Dec 13, 2024

The difference with the previous impl is that we now first wait until all storage proofs are calculated, and only then query them when building the account leaf nodes. In the previous impl, we started all storage proof calculations at once, and then queried in the same order, waiting on the channel if they're not ready yet.

I think we should be able to transform it further using rayon iterators, so that we do not block until all storage proofs are calculated.

@gakonst
Copy link
Member

gakonst commented Dec 13, 2024

Yeah seems like we don't want to block on the parallel storage root and ideally you can background spawn them? This should be diff from the previous implementation given we are no longer traversing left to right and we can do updates unsorted?

@shekhirin
Copy link
Collaborator Author

Me and @fgimenez were running nodes with #13260 for a while now, and it doesn't seem to have any problems. Closing for now.

@shekhirin shekhirin closed this Dec 13, 2024
@rkrasiuk rkrasiuk deleted the alexey/parallel-proof-rayon-iter branch December 13, 2024 20:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants