Implement PeerDAS #14129
Draft
Implement PeerDAS #14129
+17,716
−1,422
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nisdas
force-pushed
the
peerDAS
branch
3 times, most recently
from
dc39693 to
3fb1fa4
Compare
nalepae
force-pushed
the
peerDAS
branch
2 times, most recently
from
f2dea1c to
bd7ec3f
Compare
nalepae
force-pushed
the
peerDAS
branch
2 times, most recently
from
1488044 to
ddea2d7
Compare
3 tasks
nalepae
force-pushed
the
peerDAS
branch
3 times, most recently
from
e92cde5 to
2dd4302
Compare
* Add Support For Discovery Of Column Subnets * Lint for SubnetsPerNode * Manu's Review * Change to a better name
* Add Data Column Subscriber * Add Data Column Vaidator * Wire all Handlers In * Fix Build * Fix Test * Fix IP in Test * Fix IP in Test
* Add RPC Handler * Add Column Requests * Update beacon-chain/db/filesystem/blob.go Co-authored-by: Manu NALEPA <[email protected]> * Update beacon-chain/p2p/rpc_topic_mappings.go Co-authored-by: Manu NALEPA <[email protected]> * Manu's Review * Manu's Review * Interface Fixes * mock manager --------- Co-authored-by: Manu NALEPA <[email protected]>
* Bump `c-kzg-4844` lib to the `das` branch. * Implement `MerkleProofKZGCommitments`. * Implement `das-core.md`. * Use `peerdas.CustodyColumnSubnets` and `peerdas.CustodyColumns`. * `CustodyColumnSubnets`: Include `i` in the for loop. * Remove `computeSubscribedColumnSubnet`. * Remove `peerdas.CustodyColumns` out of the for loop.
* Remove capital letter from error messages. * `[4]byte` => `[fieldparams.VersionLength]byte`. * Prometheus: Remove extra `committee`. They are probably due to a bad copy/paste. Note: The name of the probe itself is remaining, to ensure backward compatibility. * Implement Proposer RPC for data columns. * Fix TestProposer_ProposeBlock_OK test. * Remove default peerDAS activation. * `validateDataColumn`: Workaround to return a `VerifiedRODataColumn`
* Add new DA check * Exit early in the event no commitments exist. * Gazelle * Fix Mock Broadcaster * Fix Test Setup * Update beacon-chain/blockchain/process_block.go Co-authored-by: Manu NALEPA <[email protected]> * Manu's Review * Fix Build --------- Co-authored-by: Manu NALEPA <[email protected]>
* Update `consensus_spec_version` to `v1.5.0-alpha.1`. * `CustodyColumns`: Fix and implement spec tests. * Make deepsource happy. * `^uint64(0)` => `math.MaxUint64`. * Fix `TestLoadConfigFile` test.
* `CustodyCountFromRemotePeer`: Set happy path in the outer scope. * `FindPeersWithSubnet`: Improve logging. * `listenForNewNodes`: Avoid infinite loop in a small subnet. * Address Nishant's comment. * FIx Nishant's comment.
* `sendBatchRootRequest`: Refactor and add comments. * `sendBatchRootRequest`: Do send requests to peers that custodies a superset of our columns. Before this commit, we sent "data columns by root requests" for data columns peers do not custody. * Data columns: Use subnet sampling only. (Instead of peer sampling.) aaa * `areDataColumnsAvailable`: Improve logs. * `GetBeaconBlock`: Improve logs. Rationale: A `begin` log should always be followed by a `success` log or a `failure` log.
* `validateDataColumn`: Refactor logging. * `dataColumnSidecarByRootRPCHandler`: Improve logging. * `isDataAvailable`: Improve logging. * Add hidden debug flag: `--data-columns-reject-slot-multiple`. * Add more logs about peer disconnection. * `validPeersExist` --> `enoughPeersAreConnected` * `beaconBlocksByRangeRPCHandler`: Add remote Peer ID in logs. * Stop calling twice `writeErrorResponseToStream` in case of rate limit.
* `scheduleReconstructedDataColumnsBroadcast`: Really minor refactor. * `receivedDataColumnsFromRootLock` -> `dataColumnsFromRootLock` * `reconstructDataColumns`: Stop looking into the DB to know if we have some columns. Before this commit: Each time we receive a column, we look into the filesystem for all columns we store. ==> For 128 columns, it looks for 1 + 2 + 3 + ... + 128 = 128(128+1)/2 = 8256 files look. Also, as soon as a column is saved into the file system, then if, right after, we look at the filesystem again, we assume the column will be available (strict consistency). It happens not to be always true. ==> Sometimes, we can reconstruct and reseed columns more than once, because of this lack of filesystem strict consistency. After this commit: We use a (strictly consistent) cache to determine if we received a column or not. ==> No more consistency issue, and less stress for the filesystem. * `dataColumnSidecarByRootRPCHandler`: Improve logging. Before this commit, logged values assumed that all requested columns correspond to the same block root, which is not always the case. After this commit, we know which columns are requested for which root. * Add a log when broadcasting a data column. This is useful to debug "lost data columns" in devnet. * Address Nishant's comment
* `columnErrBuilder`: Uses `Wrap` instead of `Join`.
Reason: `Join` makes a carriage return. The log is quite unreadable.
* `validateDataColumn`: Improve log.
* `areDataColumnsAvailable`: Improve log.
* `SendDataColumnSidecarByRoot` ==> `SendDataColumnSidecarsByRootRequest`.
* `handleDA`: Refactor error message.
* `sendRecentBeaconBlocksRequest` ==> `sendBeaconBlocksRequest`.
Reason: There is no notion at all of "recent" in the function.
If the caller decides to call this function only with "recent" blocks, that's fine.
However, the function itself will know nothing about the "recentness" of these blocks.
* `sendBatchRootRequest`: Improve comments.
* `sendBeaconBlocksRequest`: Avoid `else` usage and use map of bool instead of `struct{}`.
* `wrapAndReportValidation`: Remove `agent` from log.
Reason: This prevent the log to hold on one line, and it is not really useful to debug.
* `validateAggregateAndProof`: Add comments.
* `GetValidCustodyPeers`: Fix typo.
* `GetValidCustodyPeers` ==> `DataColumnsAdmissibleCustodyPeers`.
* `CustodyHandler` ==> `DataColumnsHandler`.
* `CustodyCountFromRemotePeer` ==> `DataColumnsCustodyCountFromRemotePeer`.
* Implement `DataColumnsAdmissibleSubnetSamplingPeers`.
* Use `SubnetSamplingSize` instead of `CustodySubnetCount` where needed.
* Revert "`wrapAndReportValidation`: Remove `agent` from log."
This reverts commit 55db351.
* `retrieveMissingDataColumnsFromPeers`: Improve logging. * `dataColumnSidecarByRootRPCHandler`: Stop decreasing peer's score if asking for a column we do not custody. * `dataColumnSidecarByRootRPCHandler`: If a data column is unavailable, stop waiting for it. This behaviour was useful for peer sampling. Now, just return the data column if we store it. If we don't, skip. * Dirty code comment. * `retrieveMissingDataColumnsFromPeers`: Improve logs. * `SendDataColumnsByRangeRequest`: Improve logs. * `dataColumnSidecarsByRangeRPCHandler`: Improve logs.
* `BestFinalized`: Refactor (no functional change). * `BestNonFinalized`: Refactor (no functional change). * `beaconBlocksByRangeRPCHandler`: Remove useless log. The same is already printed at the start of the function. * `calculateHeadAndTargetEpochs`: Avoid `else`. * `ConvertPeerIDToNodeID`: Improve error. * Stop printing noisy "peer should be banned" logs. * Initial sync: Request data columns from peers which: - custody a superset of columns we need, and - have a head slot >= our target slot. * `requestDataColumnsFromPeers`: Shuffle peers before requesting. Before this commit, we always requests peers in the same order, until one responds something. Without shuffling, we always requests data columns from the same peer. * `requestDataColumnsFromPeers`: If error from a peer, just log the error and skip the peer. * Improve logging. * Fix tests.
* Improve logging. * `retrieveMissingDataColumnsFromPeers`: Limit to `512` items per request. * `retrieveMissingDataColumnsFromPeers`: Allow `nil` peers. Before this commit: If, when this funcion is called, we are not yet connected to enough peers, then `peers` will be possibly not be satisfaying, and, if new peers are connected, we will never see them. After this commit: If `peers` is `nil`, then we regularly check for all connected peers. If `peers` is not `nil`, then we use them.
* `retrieveMissingDataColumnsFromPeers`: Search only for needed peers. * Improve logging.
* Fix Commitments Check * `highestFinalizedEpoch`: Refactor (no functional change). * `retrieveMissingDataColumnsFromPeers`: Fix logs. * `VerifyDataColumnSidecarKZGProofs`: Optimise with capacity. * Save data columns when initial syncing. * `dataColumnSidecarsByRangeRPCHandler`: Add logs when a request enters. * Improve logging. * Improve logging. * `peersWithDataColumns: Do not filter any more on peer head slot. * Fix Nishant's comment. --------- Co-authored-by: Manu NALEPA <[email protected]>
…llow syncing from full nodes. (#14532) * `validateDataColumnsByRange`: `current` ==> `currentSlot`. * `validateRequest`: Extract `remotePeer` variable. * `dataColumnSidecarsByRangeRPCHandler`: Small non functional refactor. * `streamDataColumnBatch`: Fix major bug. Before this commit, the node was unable to respond with a data column index higher than the count of stored data columns. For example, if there is 8 data columns stored for a given block, the node was able to respond for data columns indices 1, 3, and 5, but not for 10, 16 or 127. The issue was visible only for full nodes, since super nodes always store 128 data columns. * Initial sync: Fetch data columns from all peers. (Not only from supernodes.) * Nishant's comment: Fix `lastSlot` and `endSlot` duplication. * Address Nishant's comment.
https://github.com/ethereum/consensus-specs/blob/dev/specs/_features/eip7594/p2p-interface.md#datacolumnsidecarsbyrange-v1 The following data column sidecars, where they exist, MUST be sent in (slot, column_index) order.
* `ColumnAlignsWithBlock`: Split lines. * Data columns verifications: Batch * Remove completely `DataColumnBatchVerifier`. Only `DataColumnsVerifier` (with `s`) on columns remains. It is the responsability of the function which receive the data column (either by gossip, by range request or by root request) to verify the data column wrt. corresponding checks. * Fix Nishant's comment.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Remaining tasks:
[1, 31]. (Starting a node since genesis works, starting a node from epoch 1 and later works.) Note: This issue may be already here with blobs (without data columns.)cKZGlibrary by thegoKZGone (when available) - In progress.internalBroadcastDataColumn)--minimum-peers-per-subnet=<n>GetValidCustodyPeersis used. ==> We need to fetch data from peers that are not super nodes.prysmctl: Implement byRoot and byRange data columns requestsiterator.Nextis soooo slow.subscribeWithParameters: Should we subscribe to subnets / find peers 1 epoch in advance forgetSubnetsToSubscribeandgetSubnetsToFindPeersOnly?