Add blocks in batches #19003
Add blocks in batches #19003
Conversation
arvidn
added
Changed
|
this is the same as |
github-actions
bot
added
the
merge_conflict
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
…ch (avoiding the pipeline stall every 64 blocks)
arvidn
force-pushed
the
add-blocks-in-batches
branch
from
63abf4e to
20d7681
Compare
|
yes, there was overlap with that PR. I've rebased ths now |
github-actions
bot
removed
the
merge_conflict
|
Conflicts have been resolved. A maintainer will review the pull request shortly. |
| ) | ||
| await full_node.peak_post_processing_2(peak_fb, None, state_change_summary, ppp_result) | ||
| # vs is updated by the call to add_block_batch() | ||
| success, state_change_summary = await full_node.add_block_batch(blocks, PeerInfo("0.0.0.0", 0), fork_info, vs) |
There was a problem hiding this comment.
we call add_blocks_in_batches sometimes with huge lists of of blocks since we expect it to batch them into validation, this seems to diverge more then what we do in the full node
also this doesn't add blocks in batches anymore, it just adds all the blocks so the name is misleading
There was a problem hiding this comment.
the number of blocks to be validated concurrently is limited by the size of the ThreadPoolExecutor and blocks are streamed into its job queue. So I don't think the size of the list is a problem, we'll do our best to maximize the available threads at all times, without stalling the pipeline every 64 blocks.
yes, I agree that this makes the name misleading. Do you have a suggestion for a better name? I think the name add_blocks() may be a bit too generic
These commits can be reviewed one at a time.
Purpose:
Simplify
add_blocks_in_batches()to add all blocks in a single batch (this is a tiny performance boost). This is a test function.Simplify
FullNode.add_block_batch()by removing an unused (left over) parameter.Current Behavior:
No change in behavior
New Behavior:
No change in behavior
Test
running
Before and after this change gives a small improvement from 42.86s -> 40.62s