From 35a09a864e5afbfe14ced13001d6d525e852b811 Mon Sep 17 00:00:00 2001 From: evan-forbes Date: Sun, 11 Jun 2023 21:31:05 -0500 Subject: [PATCH 01/16] docs: minor update to the blob module --- x/blob/README.md | 50 +++++++++++++++++++++++++++++++++++------------- 1 file changed, 37 insertions(+), 13 deletions(-) diff --git a/x/blob/README.md b/x/blob/README.md index 08c9f4bae7..3a2fc45b9d 100644 --- a/x/blob/README.md +++ b/x/blob/README.md @@ -13,25 +13,45 @@ The `x/blob` module enables users to pay for arbitrary data to be published to t 1. `NamespaceIds []byte`: the namespaces they wish to publish each blob to. The namespaces here must match the namespaces in the `Blob`s. 1. `ShareCommitment []byte`: a share commitment that is the root of a Merkle tree where the leaves are share commitments to each blob associated with this BlobTx. -After the `BlobTx` is submitted to the network, a block producer separates the transaction from the blob. Both components get included in the data square in different namespaces: the BlobTx gets included in the PayForBlobNamespace and the associated blob gets included in the namespace the user specified in the original `BlobTx`. Further reading: [Message Block Layout](https://github.com/celestiaorg/celestia-specs/blob/master/src/rationale/message_block_layout.md) +After the `BlobTx` is submitted to the network, a block producer separates the transaction from the blob. Both components get included in the data square in different namespaces: the BlobTx gets included in the PayForBlobNamespace (one of the [reserved namespaces](../../specs/src/specs/consensus.md#reserved-namespaces)) and the associated blob gets included in the namespace the user specified in the original `BlobTx`. Further reading: [Message Block Layout](https://github.com/celestiaorg/celestia-specs/blob/master/src/rationale/message_block_layout.md) After a block has been created, the user can verify that their data was included in a block via a blob inclusion proof. A blob inclusion proof uses the `ShareCommitment` in the original transaction and subtree roots of the block's data square to prove to the user that the shares that compose their original data do in fact exist in a particular block. ## State -The blob module doesn't maintain it's own state. +The blob module doesn't maintain it's own state outside of a single param. When a `MsgPayForBlob` is processed, it consumes gas based on the blob size. ## Messages -- [`MsgPayForBlob`](https://github.com/celestiaorg/celestia-app/blob/8b9c4c9d13fe0ccb6ea936cc26dee3f52b6f6129/proto/blob/tx.proto#L39-L44) pays for the blob to be included in the block. +- [`MsgPayForBlobs`](https://github.com/celestiaorg/celestia-app/blob/v1.0.0-rc2/proto/celestia/blob/v1/tx.proto#L16-L31) + pays for a set of blobs to be included in the block. + +```proto +// MsgPayForBlobs pays for the inclusion of a blob in the block. +message MsgPayForBlobs { + string signer = 1; + // namespaces is a list of namespaces that the blobs are associated with. A + // namespace is a byte slice of length 33 where the first byte is the + // namespaceVersion and the subsequent 32 bytes are the namespaceId. + repeated bytes namespaces = 2; + repeated uint32 blob_sizes = 3; + // share_commitments is a list of share commitments (one per blob). + repeated bytes share_commitments = 4; + // share_versions are the versions of the share format that the blobs + // associated with this message should use when included in a block. The + // share_versions specified must match the share_versions used to generate the + // share_commitment in this message. + repeated uint32 share_versions = 8; +} +``` -## PrepareProposal +## `PrepareProposal` -When a block producer is preparing a block, they must perform an extra step for `BlobTx`s so that end-users can find the blob shares relevant to their submitted `BlobTx`. In particular, block proposers wrap the `BlobTx` in the PayForBlobs namespace with the index of the first share of the blob in the data square. See [Non-interactive Default Rules](https://github.com/celestiaorg/celestia-specs/blob/master/src/rationale/message_block_layout.md#non-interactive-default-rules) for more details. +When a block producer is preparing a block, they must perform an extra step for `BlobTx`s so that end-users can find the blob shares relevant to their submitted `BlobTx`. In particular, block proposers wrap the `BlobTx` in the PFB namespace with the index of the first share of the blob in the data square. See [Non-interactive Default Rules](https://github.com/celestiaorg/celestia-specs/blob/master/src/rationale/message_block_layout.md#non-interactive-default-rules) for more details. -Since `BlobTx`s can contain multiple blobs, the `BlobTx` is wrapped with one share index per blob in the transaction. The index wrapped transaction is called an [IndexWrapper](https://github.com/celestiaorg/celestia-core/blob/2d2a65f59eabf1993804168414b86d758f30c383/proto/tendermint/types/types.proto#L192-L198) and this is the type that gets marshalled and written to the PayForBlobNamespace. +Since `BlobTx`s can contain multiple blobs, the `sdk.Tx` portion of the `BlobTx` is wrapped with one share index per blob in the transaction. The index wrapped transaction is called an [IndexWrapper](https://github.com/celestiaorg/celestia-core/blob/2d2a65f59eabf1993804168414b86d758f30c383/proto/tendermint/types/types.proto#L192-L198) and this is the type that gets marshalled and written to the PayForBlobNamespace. ## Events @@ -39,7 +59,7 @@ The blob module emits the following events: ### Blob Events -#### EventPayForBlob +#### `EventPayForBlob` | Attribute Key | Attribute Value | |---------------|-----------------------------------------------| @@ -65,12 +85,16 @@ While not directly supported, the steps in the [`SubmitPayForBlob`](https://gith -### How is the `ShareCommitment` of a blob generated? +### Generating the `ShareCommitment` -See [`CreateCommitment`](https://github.com/celestiaorg/celestia-app/blob/ead76d2bb607ac8a2deaba552de86d4df74a116b/x/blob/types/payforblob.go#L133). +See [`CreateCommitment`](https://github.com/celestiaorg/celestia-app/blob/ead76d2bb607ac8a2deaba552de86d4df74a116b/x/blob/types/payforblob.go#L133), [data square layout](../../specs/src/specs/data_square_layout.md), and [ADR013](../../docs/architecture/adr-013-non-interactive-default-rules-for-zero-padding.md) for rational on what the share commitment is created this way. 1. Split the blob into shares of size `appconsts.ShareSize` -1. Determine the `BlobMinSquareSize` (the minimum square size the blob can fit into). This is done by taking the number of shares from the previous step and rounding up to the next perfect square that is a power of two. -1. Arrange the shares into a Merkle mountain range where each tree in the mountain range has a maximum size of the `BlobMinSquareSize`. Note: each share is prefixed with a duplicate copy of the namespace so that the roots in the next step match up with the subtree roots of the NMT of the data square. -1. Take the roots of the trees in the Merkle mountain range and create a new Merkle tree. -1. The share commitment of this blob is the Merkle root of the Merkle tree from the previous step. +1. Determine the `SubtreeWidth` by dividing the length in shares by the + `SubtreeRootThreshold`. +1. Generate each subtree root by diving the blob shares into `SubtreeWidth` + sized sets, then take the binary [namespaced merkle + tree (NMT)](https://github.com/celestiaorg/nmt/blob/v0.16.0/docs/spec/nmt.md) root + of each set of shares. +1. Calculate the final share commitment by taking the merkle root (note: not an + NMT, just a normal binary merkle root) of the subtree roots from the previous step. From 6a5251568352fae9a29313d562e09df529f0e9e4 Mon Sep 17 00:00:00 2001 From: evan-forbes Date: Sun, 11 Jun 2023 21:59:34 -0500 Subject: [PATCH 02/16] docs: add validity rules --- x/blob/README.md | 126 ++++++++++++++++++++++++++++++++++++----------- 1 file changed, 96 insertions(+), 30 deletions(-) diff --git a/x/blob/README.md b/x/blob/README.md index 3a2fc45b9d..3e76803ed0 100644 --- a/x/blob/README.md +++ b/x/blob/README.md @@ -2,20 +2,37 @@ ## Abstract -The `x/blob` module enables users to pay for arbitrary data to be published to the Celestia blockchain. Users create a single `BlobTx` that is composed of: +The `x/blob` module enables users to pay for arbitrary data to be published to +the Celestia blockchain. Users create a single `BlobTx` that is composed of: -1. Multiple `Blob`s (Binary Large OBjects): the data they wish to publish. A single `Blob` is composed of: +1. Multiple `Blob`s (Binary Large OBjects): the data they wish to publish. A + single `Blob` is composed of: 1. `NamespaceId []byte`: the namespace this blob should be published to. 1. `Data []byte`: the data to be published. - 1. `ShareVersion uint32`: the version of the share format used to encode this blob into a share. + 1. `ShareVersion uint32`: the version of the share format used to encode + this blob into a share. 1. A single `sdk.Tx` which is composed of: 1. `Signer string`: the transaction signer - 1. `NamespaceIds []byte`: the namespaces they wish to publish each blob to. The namespaces here must match the namespaces in the `Blob`s. - 1. `ShareCommitment []byte`: a share commitment that is the root of a Merkle tree where the leaves are share commitments to each blob associated with this BlobTx. - -After the `BlobTx` is submitted to the network, a block producer separates the transaction from the blob. Both components get included in the data square in different namespaces: the BlobTx gets included in the PayForBlobNamespace (one of the [reserved namespaces](../../specs/src/specs/consensus.md#reserved-namespaces)) and the associated blob gets included in the namespace the user specified in the original `BlobTx`. Further reading: [Message Block Layout](https://github.com/celestiaorg/celestia-specs/blob/master/src/rationale/message_block_layout.md) - -After a block has been created, the user can verify that their data was included in a block via a blob inclusion proof. A blob inclusion proof uses the `ShareCommitment` in the original transaction and subtree roots of the block's data square to prove to the user that the shares that compose their original data do in fact exist in a particular block. + 1. `NamespaceIds []byte`: the namespaces they wish to publish each blob to. + The namespaces here must match the namespaces in the `Blob`s. + 1. `ShareCommitment []byte`: a share commitment that is the root of a Merkle + tree where the leaves are share commitments to each blob associated with + this BlobTx. + +After the `BlobTx` is submitted to the network, a block producer separates the +transaction from the blob. Both components get included in the data square in +different namespaces: the BlobTx gets included in the PayForBlobNamespace (one +of the [reserved +namespaces](../../specs/src/specs/consensus.md#reserved-namespaces)) and the +associated blob gets included in the namespace the user specified in the +original `BlobTx`. Further reading: [Message Block +Layout](https://github.com/celestiaorg/celestia-specs/blob/master/src/rationale/message_block_layout.md) + +After a block has been created, the user can verify that their data was included +in a block via a blob inclusion proof. A blob inclusion proof uses the +`ShareCommitment` in the original transaction and subtree roots of the block's +data square to prove to the user that the shares that compose their original +data do in fact exist in a particular block. ## State @@ -26,7 +43,8 @@ When a `MsgPayForBlob` is processed, it consumes gas based on the blob size. ## Messages - [`MsgPayForBlobs`](https://github.com/celestiaorg/celestia-app/blob/v1.0.0-rc2/proto/celestia/blob/v1/tx.proto#L16-L31) - pays for a set of blobs to be included in the block. + pays for a set of blobs to be included in the block. Transactions containing + this `sdk.Msg` are better known as "PFBs". ```proto // MsgPayForBlobs pays for the inclusion of a blob in the block. @@ -47,11 +65,70 @@ message MsgPayForBlobs { } ``` -## `PrepareProposal` - -When a block producer is preparing a block, they must perform an extra step for `BlobTx`s so that end-users can find the blob shares relevant to their submitted `BlobTx`. In particular, block proposers wrap the `BlobTx` in the PFB namespace with the index of the first share of the blob in the data square. See [Non-interactive Default Rules](https://github.com/celestiaorg/celestia-specs/blob/master/src/rationale/message_block_layout.md#non-interactive-default-rules) for more details. +### Generating the `ShareCommitment` -Since `BlobTx`s can contain multiple blobs, the `sdk.Tx` portion of the `BlobTx` is wrapped with one share index per blob in the transaction. The index wrapped transaction is called an [IndexWrapper](https://github.com/celestiaorg/celestia-core/blob/2d2a65f59eabf1993804168414b86d758f30c383/proto/tendermint/types/types.proto#L192-L198) and this is the type that gets marshalled and written to the PayForBlobNamespace. +1. Split the blob into shares of size `appconsts.ShareSize` +1. Determine the + [`SubtreeWidth`](https://github.com/celestiaorg/celestia-app/blob/v1.0.0-rc2/pkg/shares/non_interactive_defaults.go#L94-L116) + by dividing the length in shares by the `SubtreeRootThreshold`. +1. Generate each subtree root by diving the blob shares into `SubtreeWidth` + sized sets, then take the binary [namespaced merkle tree + (NMT)](https://github.com/celestiaorg/nmt/blob/v0.16.0/docs/spec/nmt.md) root + of each set of shares. +1. Calculate the final share commitment by taking the merkle root (note: not an + NMT, just a normal binary merkle root) of the subtree roots from the previous + step. + +See +[`CreateCommitment`](https://github.com/celestiaorg/celestia-app/blob/v1.0.0-rc2/x/blob/types/payforblob.go#L169-L236) +for an implementation. See [data square +layout](../../specs/src/specs/data_square_layout.md) and +[ADR013](../../docs/architecture/adr-013-non-interactive-default-rules-for-zero-padding.md) +for details on the rational of the square layout. + +## Validity Rules + +In order for a proposal block to be considered valid, each `BlobTx`, and thus +each PFB, to be included in a block must follow a set of validity rules. + +1. Signatures: All blob transactions must have valid signatures. This is + state-dependent because correct signatures require using the correct sequence + number(aka nonce). +1. Single SDK.Msg: There must be only a single sdk.Msg in the blob transaction. +1. Namespace Validity: The namespace of each blob transaction must be valid. + This validity is determined by the following sub-rules: + 1. The namespace is not within the reserved namespace range. + 1. The namespace is not the tail padding or parity namespaces. +1. Blob Size: No blob can have a size of 0. +1. Blob Count: There must be one or more blobs included in the transaction. +1. Share Commitment Validity: Each share commitment must be valid. + 1. The size of each of the share commitments must be equal to the digest of + the hash function used (sha256 so 32 bytes). + 1. The share commitment must be calculated using the steps specified above + in [Generating the Share + Commitment](./README.md#generating-the-sharecommitment) +1. Share Versions: The versions of the shares must be supported. +1. Signer Address: The signer address must be a valid Celestia address. +1. Proper Encoding: The blob transactions must be properly encoded. +1. Size Consistency: The sizes included in the transaction must match the actual + sizes of the blobs. + +## `IndexWrappedTx` + +When a block producer is preparing a block, they must perform an extra step for +`BlobTx`s so that end-users can find the blob shares relevant to their submitted +`BlobTx`. In particular, block proposers wrap the `BlobTx` in the PFB namespace +with the index of the first share of the blob in the data square. See +[Non-interactive Default +Rules](https://github.com/celestiaorg/celestia-specs/blob/master/src/rationale/message_block_layout.md#non-interactive-default-rules) +for more details. + +Since `BlobTx`s can contain multiple blobs, the `sdk.Tx` portion of the `BlobTx` +is wrapped with one share index per blob in the transaction. The index wrapped +transaction is called an +[IndexWrapper](https://github.com/celestiaorg/celestia-core/blob/2d2a65f59eabf1993804168414b86d758f30c383/proto/tendermint/types/types.proto#L192-L198) +and this is the struct that gets marshalled and written to the +PayForBlobNamespace. ## Events @@ -79,22 +156,11 @@ The blob module emits the following events: celestia-app tx blob PayForBlobs [flags] ``` -For submitting PFB transaction via a light client's rpc, see [celestia-node's documention](https://docs.celestia.org/developers/rpc-tutorial/#submitpayforblob-arguments). +For submitting PFB transaction via a light client's rpc, see [celestia-node's +documention](https://docs.celestia.org/developers/rpc-tutorial/#submitpayforblob-arguments). -While not directly supported, the steps in the [`SubmitPayForBlob`](https://github.com/celestiaorg/celestia-app/blob/a82110a281bf9ee95a9bf9f0492e5d091371ff0b/x/blob/payforblob.go) function can be reverse engineered to submit blobs programmatically. +The steps in the +[`SubmitPayForBlobs`](https://github.com/celestiaorg/celestia-app/blob/v1.0.0-rc2/x/blob/payforblob.go#L15-L54) +function can be reverse engineered to submit blobs programmatically. - -### Generating the `ShareCommitment` - -See [`CreateCommitment`](https://github.com/celestiaorg/celestia-app/blob/ead76d2bb607ac8a2deaba552de86d4df74a116b/x/blob/types/payforblob.go#L133), [data square layout](../../specs/src/specs/data_square_layout.md), and [ADR013](../../docs/architecture/adr-013-non-interactive-default-rules-for-zero-padding.md) for rational on what the share commitment is created this way. - -1. Split the blob into shares of size `appconsts.ShareSize` -1. Determine the `SubtreeWidth` by dividing the length in shares by the - `SubtreeRootThreshold`. -1. Generate each subtree root by diving the blob shares into `SubtreeWidth` - sized sets, then take the binary [namespaced merkle - tree (NMT)](https://github.com/celestiaorg/nmt/blob/v0.16.0/docs/spec/nmt.md) root - of each set of shares. -1. Calculate the final share commitment by taking the merkle root (note: not an - NMT, just a normal binary merkle root) of the subtree roots from the previous step. From 93af08679379b8de81e49efff15d5e85cd03788c Mon Sep 17 00:00:00 2001 From: evan-forbes Date: Sun, 18 Jun 2023 21:53:58 -0500 Subject: [PATCH 03/16] docs: update event field --- x/blob/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/x/blob/README.md b/x/blob/README.md index 3e76803ed0..0ce23c5590 100644 --- a/x/blob/README.md +++ b/x/blob/README.md @@ -141,7 +141,7 @@ The blob module emits the following events: | Attribute Key | Attribute Value | |---------------|-----------------------------------------------| | signer | {bech32 encoded signer address} | -| blob_size | {size in bytes} | +| blob_sizes | {sizes of blobs in bytes} | | namespace_ids | {namespaces the blobs should be published to} | ## Parameters From 5b69d04593e814b9ad37ef3b950938f00922e3de Mon Sep 17 00:00:00 2001 From: Evan Forbes <42654277+evan-forbes@users.noreply.github.com> Date: Sun, 18 Jun 2023 22:40:32 -0500 Subject: [PATCH 04/16] docs: Apply suggestions from code review Co-authored-by: Sanaz Taheri <35961250+staheri14@users.noreply.github.com> Co-authored-by: Rootul P --- x/blob/README.md | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/x/blob/README.md b/x/blob/README.md index 3e76803ed0..582545fe67 100644 --- a/x/blob/README.md +++ b/x/blob/README.md @@ -17,20 +17,20 @@ the Celestia blockchain. Users create a single `BlobTx` that is composed of: The namespaces here must match the namespaces in the `Blob`s. 1. `ShareCommitment []byte`: a share commitment that is the root of a Merkle tree where the leaves are share commitments to each blob associated with - this BlobTx. + this `BlobTx`. After the `BlobTx` is submitted to the network, a block producer separates the -transaction from the blob. Both components get included in the data square in -different namespaces: the BlobTx gets included in the PayForBlobNamespace (one +transaction i.e., `sdk.Tx` from the blob. Both components get included in the data square in +different namespaces: the `sdk.Tx` of the original `BlobTx` together with some metadata about the separated blobs get included in the PayForBlobNamespace (one of the [reserved namespaces](../../specs/src/specs/consensus.md#reserved-namespaces)) and the -associated blob gets included in the namespace the user specified in the +associated blobs get included in the namespaces the user specified in the original `BlobTx`. Further reading: [Message Block Layout](https://github.com/celestiaorg/celestia-specs/blob/master/src/rationale/message_block_layout.md) After a block has been created, the user can verify that their data was included in a block via a blob inclusion proof. A blob inclusion proof uses the -`ShareCommitment` in the original transaction and subtree roots of the block's +`ShareCommitment` in the original `sdk.Tx` transaction and subtree roots of the block's data square to prove to the user that the shares that compose their original data do in fact exist in a particular block. @@ -38,32 +38,32 @@ data do in fact exist in a particular block. The blob module doesn't maintain it's own state outside of a single param. -When a `MsgPayForBlob` is processed, it consumes gas based on the blob size. +When a `MsgPayForBlob` of a `BlobTx` is processed, it consumes gas based on the blob size. ## Messages - [`MsgPayForBlobs`](https://github.com/celestiaorg/celestia-app/blob/v1.0.0-rc2/proto/celestia/blob/v1/tx.proto#L16-L31) - pays for a set of blobs to be included in the block. Transactions containing + pays for a set of blobs to be included in the block. Blob transactions that contain + this `sdk.Msg` are also referred to as "PFBs". this `sdk.Msg` are better known as "PFBs". + ```proto -// MsgPayForBlobs pays for the inclusion of a blob in the block. message MsgPayForBlobs { string signer = 1; - // namespaces is a list of namespaces that the blobs are associated with. A - // namespace is a byte slice of length 33 where the first byte is the - // namespaceVersion and the subsequent 32 bytes are the namespaceId. repeated bytes namespaces = 2; repeated uint32 blob_sizes = 3; - // share_commitments is a list of share commitments (one per blob). repeated bytes share_commitments = 4; - // share_versions are the versions of the share format that the blobs - // associated with this message should use when included in a block. The - // share_versions specified must match the share_versions used to generate the - // share_commitment in this message. repeated uint32 share_versions = 8; } -``` + +MsgPayForBlobs pays for the inclusion of a blob in the block and consists of the following fields: +- signer: ... +- namespace: namespace is a byte slice of length 33 where the first byte is the + namespaceVersion and the subsequent 32 bytes are the namespaceId. +- blob_sizes: ... +- share_commitments is a list of share commitments (one per blob). +- share_versions are the versions of the share format that the blobs associated with this message should use when included in a block. The share_versions specified must match the share_versions used to generate the share_commitment in this message. ### Generating the `ShareCommitment` @@ -94,8 +94,8 @@ each PFB, to be included in a block must follow a set of validity rules. 1. Signatures: All blob transactions must have valid signatures. This is state-dependent because correct signatures require using the correct sequence number(aka nonce). -1. Single SDK.Msg: There must be only a single sdk.Msg in the blob transaction. -1. Namespace Validity: The namespace of each blob transaction must be valid. +1. Single SDK.Msg: There must be only a single sdk.Msg encoded in the `sdk.Tx` field of the blob transaction `BlobTx`. +1. Namespace Validity: The namespace of each blob in a blob transaction `BlobTx` must be valid. This validity is determined by the following sub-rules: 1. The namespace is not within the reserved namespace range. 1. The namespace is not the tail padding or parity namespaces. From 10b1e41494a85950ffea52d55f14e7003a474425 Mon Sep 17 00:00:00 2001 From: evan-forbes Date: Sun, 18 Jun 2023 22:41:30 -0500 Subject: [PATCH 05/16] docs: minor update --- x/blob/README.md | 41 ++++++++++++++++++++++++++++++++--------- 1 file changed, 32 insertions(+), 9 deletions(-) diff --git a/x/blob/README.md b/x/blob/README.md index 080843a04c..d7cc0f3cd6 100644 --- a/x/blob/README.md +++ b/x/blob/README.md @@ -11,13 +11,7 @@ the Celestia blockchain. Users create a single `BlobTx` that is composed of: 1. `Data []byte`: the data to be published. 1. `ShareVersion uint32`: the version of the share format used to encode this blob into a share. -1. A single `sdk.Tx` which is composed of: - 1. `Signer string`: the transaction signer - 1. `NamespaceIds []byte`: the namespaces they wish to publish each blob to. - The namespaces here must match the namespaces in the `Blob`s. - 1. `ShareCommitment []byte`: a share commitment that is the root of a Merkle - tree where the leaves are share commitments to each blob associated with - this BlobTx. +1. A single `sdk.Tx` which contains a single `sdk.Msg` of type `MsgPayForBlobs`. After the `BlobTx` is submitted to the network, a block producer separates the transaction from the blob. Both components get included in the data square in @@ -35,9 +29,38 @@ data do in fact exist in a particular block. ## State -The blob module doesn't maintain it's own state outside of a single param. +The blob module doesn't maintain it's own state outside of a two params. Meaning +that the blob module only uses the paramstore and auth module stores. -When a `MsgPayForBlob` is processed, it consumes gas based on the blob size. +### Params + +```proto +// Params defines the parameters for the module. +message Params { + option (gogoproto.goproto_stringer) = false; + + uint32 gas_per_blob_byte = 1 + [ (gogoproto.moretags) = "yaml:\"gas_per_blob_byte\"" ]; + + uint64 gov_max_square_size = 2 + [ (gogoproto.moretags) = "yaml:\"gov_max_square_size\"" ]; +} + +``` + +#### `GasPerBlobByte` + +`GasPerBlobByte` is the amount of gas that is consumed per byte of blob data +when a `MsgPayForBlob` is processed. Currently, the Default value is 8. This +value is set below that of normal transaction gas consumption, which is 10. + +#### `GovMaxSquareSize` + +`GovMaxSquareSize` is the maximum size of a data square that is considered valid +by the validator set. This value is superseded by the `MaxSquareSize`, which is +hardcoded and cannot change without hardforking the chain. See +[ADR021](../../docs/architecture/adr-021-restricted-block-size.md) for more +details. ## Messages From 3508b36a7abeff25ba8dec5a475bac0b680782e1 Mon Sep 17 00:00:00 2001 From: evan-forbes Date: Sun, 18 Jun 2023 22:49:39 -0500 Subject: [PATCH 06/16] docs: fill out fields of PFB in new format and link to ADR014 for share verison --- x/blob/README.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/x/blob/README.md b/x/blob/README.md index 319fa48688..9d734948e2 100644 --- a/x/blob/README.md +++ b/x/blob/README.md @@ -80,14 +80,22 @@ message MsgPayForBlobs { repeated bytes share_commitments = 4; repeated uint32 share_versions = 8; } +``` + +`MsgPayForBlobs` pays for the inclusion of a blob in the block and consists of the +following fields: -MsgPayForBlobs pays for the inclusion of a blob in the block and consists of the following fields: -- signer: ... +- signer: bech32 encoded signer address - namespace: namespace is a byte slice of length 33 where the first byte is the namespaceVersion and the subsequent 32 bytes are the namespaceId. -- blob_sizes: ... +- blob_sizes: sizes of each blob in bytes. - share_commitments is a list of share commitments (one per blob). -- share_versions are the versions of the share format that the blobs associated with this message should use when included in a block. The share_versions specified must match the share_versions used to generate the share_commitment in this message. +- share_versions are the versions of the share format that the blobs associated + with this message should use when included in a block. The share_versions + specified must match the share_versions used to generate the share_commitment + in this message. See + [ADR014](../../docs/architecture/adr-014-versioned-namespaces.md) for more + details on how this effects the share encoding and when it is updated. Note that while the shares version in each protobuf encoded PFB are uint32s, the internal represantation of shares versions is always uint8s. This is because From 4338e36444725cb94055ff3c406181e29a541571 Mon Sep 17 00:00:00 2001 From: evan-forbes Date: Sun, 18 Jun 2023 23:24:23 -0500 Subject: [PATCH 07/16] docs: elaborate on size consistency --- x/blob/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/x/blob/README.md b/x/blob/README.md index 9d734948e2..d46b7619b5 100644 --- a/x/blob/README.md +++ b/x/blob/README.md @@ -140,14 +140,14 @@ each PFB, to be included in a block must follow a set of validity rules. 1. Share Commitment Validity: Each share commitment must be valid. 1. The size of each of the share commitments must be equal to the digest of the hash function used (sha256 so 32 bytes). - 1. The share commitment must be calculated using the steps specified above + 2. The share commitment must be calculated using the steps specified above in [Generating the Share Commitment](./README.md#generating-the-sharecommitment) 1. Share Versions: The versions of the shares must be supported. 1. Signer Address: The signer address must be a valid Celestia address. 1. Proper Encoding: The blob transactions must be properly encoded. -1. Size Consistency: The sizes included in the transaction must match the actual - sizes of the blobs. +1. Size Consistency: The sizes included in the PFB field `blob_sizes`, and each must match the actual + size of the respective (same index) blob in bytes. ## `IndexWrappedTx` From 6be08b48a050147b955ff66e01bc67918450db77 Mon Sep 17 00:00:00 2001 From: evan-forbes Date: Sun, 18 Jun 2023 23:28:52 -0500 Subject: [PATCH 08/16] docs: link to reserved namespaces --- x/blob/README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/x/blob/README.md b/x/blob/README.md index d46b7619b5..e62bcbca2a 100644 --- a/x/blob/README.md +++ b/x/blob/README.md @@ -134,7 +134,10 @@ each PFB, to be included in a block must follow a set of validity rules. 1. Namespace Validity: The namespace of each blob in a blob transaction `BlobTx` must be valid. This validity is determined by the following sub-rules: 1. The namespace is not within the reserved namespace range. - 1. The namespace is not the tail padding or parity namespaces. + 1. The namespace is not the + [`TAIL_PADDING_NAMESPACE`](../../specs/src/specs/consensus.md#constants) + or [RESERVED_PADDING_NAMESPACE + namespaces](../../specs/src/specs/consensus.md#constants). 1. Blob Size: No blob can have a size of 0. 1. Blob Count: There must be one or more blobs included in the transaction. 1. Share Commitment Validity: Each share commitment must be valid. From 60e4cce0113c93c386a03ce3be97a7aee6a36062 Mon Sep 17 00:00:00 2001 From: evan-forbes Date: Sun, 18 Jun 2023 23:38:47 -0500 Subject: [PATCH 09/16] docs: clarify and add link --- x/blob/README.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/x/blob/README.md b/x/blob/README.md index e62bcbca2a..7960e6be99 100644 --- a/x/blob/README.md +++ b/x/blob/README.md @@ -11,7 +11,7 @@ the Celestia blockchain. Users create a single `BlobTx` that is composed of: 1. `Data []byte`: the data to be published. 1. `ShareVersion uint32`: the version of the share format used to encode this blob into a share. -1. A single `sdk.Tx` which is composed of: +1. A single [`sdk.Tx`](https://github.com/celestiaorg/cosmos-sdk/blob/v1.15.0-sdk-v0.46.13/docs/architecture/adr-020-protobuf-transaction-encoding.md) which is composed of: 1. `Signer string`: the transaction signer 1. `NamespaceIds []byte`: the namespaces they wish to publish each blob to. The namespaces here must match the namespaces in the `Blob`s. @@ -130,14 +130,16 @@ each PFB, to be included in a block must follow a set of validity rules. 1. Signatures: All blob transactions must have valid signatures. This is state-dependent because correct signatures require using the correct sequence number(aka nonce). -1. Single SDK.Msg: There must be only a single sdk.Msg encoded in the `sdk.Tx` field of the blob transaction `BlobTx`. -1. Namespace Validity: The namespace of each blob in a blob transaction `BlobTx` must be valid. - This validity is determined by the following sub-rules: +1. Single SDK.Msg: There must be only a single sdk.Msg encoded in the `sdk.Tx` + field of the blob transaction `BlobTx`. +1. Namespace Validity: The namespace of each blob in a blob transaction `BlobTx` + must be valid. This validity is determined by the following sub-rules: + 1. The namepsace of each blob must match the respective (same index) + namespace in the `MsgPayForBlobs` `sdk.Msg`. 1. The namespace is not within the reserved namespace range. 1. The namespace is not the [`TAIL_PADDING_NAMESPACE`](../../specs/src/specs/consensus.md#constants) - or [RESERVED_PADDING_NAMESPACE - namespaces](../../specs/src/specs/consensus.md#constants). + or [RESERVED_PADDING_NAMESPACE namespaces](../../specs/src/specs/consensus.md#constants). 1. Blob Size: No blob can have a size of 0. 1. Blob Count: There must be one or more blobs included in the transaction. 1. Share Commitment Validity: Each share commitment must be valid. @@ -149,8 +151,8 @@ each PFB, to be included in a block must follow a set of validity rules. 1. Share Versions: The versions of the shares must be supported. 1. Signer Address: The signer address must be a valid Celestia address. 1. Proper Encoding: The blob transactions must be properly encoded. -1. Size Consistency: The sizes included in the PFB field `blob_sizes`, and each must match the actual - size of the respective (same index) blob in bytes. +1. Size Consistency: The sizes included in the PFB field `blob_sizes`, and each + must match the actual size of the respective (same index) blob in bytes. ## `IndexWrappedTx` From 1e600c166c8e30ecb1ffcfc5caa22e3d1e19f293 Mon Sep 17 00:00:00 2001 From: evan-forbes Date: Sun, 18 Jun 2023 23:40:18 -0500 Subject: [PATCH 10/16] docs: share commitments --- x/blob/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/x/blob/README.md b/x/blob/README.md index 7960e6be99..0882085a5b 100644 --- a/x/blob/README.md +++ b/x/blob/README.md @@ -15,7 +15,7 @@ the Celestia blockchain. Users create a single `BlobTx` that is composed of: 1. `Signer string`: the transaction signer 1. `NamespaceIds []byte`: the namespaces they wish to publish each blob to. The namespaces here must match the namespaces in the `Blob`s. - 1. `ShareCommitment []byte`: a share commitment that is the root of a Merkle + 1. `ShareCommitments [][]byte`: a share commitment that is the root of a Merkle tree where the leaves are share commitments to each blob associated with this `BlobTx`. From 695e892c0f7e6bd8d5f9ca342e0656bb23e8977c Mon Sep 17 00:00:00 2001 From: evan-forbes Date: Sun, 18 Jun 2023 23:53:44 -0500 Subject: [PATCH 11/16] docs: typo and link to reserved name space --- x/blob/README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/x/blob/README.md b/x/blob/README.md index 0882085a5b..64c77b780e 100644 --- a/x/blob/README.md +++ b/x/blob/README.md @@ -36,7 +36,7 @@ data do in fact exist in a particular block. ## State The blob module doesn't maintain it's own state outside of a two params. Meaning -that the blob module only uses the paramstore and auth module stores. +that the blob module only uses the params and auth module stores. ### Params @@ -135,17 +135,17 @@ each PFB, to be included in a block must follow a set of validity rules. 1. Namespace Validity: The namespace of each blob in a blob transaction `BlobTx` must be valid. This validity is determined by the following sub-rules: 1. The namepsace of each blob must match the respective (same index) - namespace in the `MsgPayForBlobs` `sdk.Msg`. - 1. The namespace is not within the reserved namespace range. + namespace in the `MsgPayForBlobs` `sdk.Msg` field `namespaces`. + 1. The namespace is not lexicographically less than the [MAX_RESERVED_NAMESPACE](../../specs/src/specs/consensus.md#constants) range. 1. The namespace is not the - [`TAIL_PADDING_NAMESPACE`](../../specs/src/specs/consensus.md#constants) - or [RESERVED_PADDING_NAMESPACE namespaces](../../specs/src/specs/consensus.md#constants). + [TAIL_PADDING_NAMESPACE](../../specs/src/specs/consensus.md#constants) + or [RESERVED_PADDING_NAMESPACE](../../specs/src/specs/consensus.md#constants). 1. Blob Size: No blob can have a size of 0. 1. Blob Count: There must be one or more blobs included in the transaction. 1. Share Commitment Validity: Each share commitment must be valid. 1. The size of each of the share commitments must be equal to the digest of the hash function used (sha256 so 32 bytes). - 2. The share commitment must be calculated using the steps specified above + 1. The share commitment must be calculated using the steps specified above in [Generating the Share Commitment](./README.md#generating-the-sharecommitment) 1. Share Versions: The versions of the shares must be supported. From 1db4f6e69dca881ea0a5a92c32522e9017c2d7ee Mon Sep 17 00:00:00 2001 From: evan-forbes Date: Sun, 18 Jun 2023 23:57:10 -0500 Subject: [PATCH 12/16] docs: add link to sharesize --- x/blob/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/x/blob/README.md b/x/blob/README.md index 64c77b780e..ac28eadbc5 100644 --- a/x/blob/README.md +++ b/x/blob/README.md @@ -103,15 +103,15 @@ protobuf doesn't support uint8s. ### Generating the `ShareCommitment` -1. Split the blob into shares of size `appconsts.ShareSize` -1. Determine the +1. Split the blob into shares of size [`shareSize`](../../specs/src/specs/data_structures.md#consensus-parameters) +2. Determine the [`SubtreeWidth`](https://github.com/celestiaorg/celestia-app/blob/v1.0.0-rc2/pkg/shares/non_interactive_defaults.go#L94-L116) by dividing the length in shares by the `SubtreeRootThreshold`. -1. Generate each subtree root by diving the blob shares into `SubtreeWidth` +3. Generate each subtree root by diving the blob shares into `SubtreeWidth` sized sets, then take the binary [namespaced merkle tree (NMT)](https://github.com/celestiaorg/nmt/blob/v0.16.0/docs/spec/nmt.md) root of each set of shares. -1. Calculate the final share commitment by taking the merkle root (note: not an +4. Calculate the final share commitment by taking the merkle root (note: not an NMT, just a normal binary merkle root) of the subtree roots from the previous step. From 145e3ade71560b6b47284c68360b900e2b9d6860 Mon Sep 17 00:00:00 2001 From: evan-forbes Date: Mon, 19 Jun 2023 00:02:10 -0500 Subject: [PATCH 13/16] docs: add more info for shares commitments --- x/blob/README.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/x/blob/README.md b/x/blob/README.md index ac28eadbc5..ea4bf4b832 100644 --- a/x/blob/README.md +++ b/x/blob/README.md @@ -103,15 +103,20 @@ protobuf doesn't support uint8s. ### Generating the `ShareCommitment` +The share commitment is the commitment to share encoded blobs. It can be used +for cheap inclusion checks for some data by light clients. More information and +rational can be found in the [data square layout +specs](../../specs/src/specs/data_square_layout.md). + 1. Split the blob into shares of size [`shareSize`](../../specs/src/specs/data_structures.md#consensus-parameters) -2. Determine the +1. Determine the [`SubtreeWidth`](https://github.com/celestiaorg/celestia-app/blob/v1.0.0-rc2/pkg/shares/non_interactive_defaults.go#L94-L116) by dividing the length in shares by the `SubtreeRootThreshold`. -3. Generate each subtree root by diving the blob shares into `SubtreeWidth` +1. Generate each subtree root by diving the blob shares into `SubtreeWidth` sized sets, then take the binary [namespaced merkle tree (NMT)](https://github.com/celestiaorg/nmt/blob/v0.16.0/docs/spec/nmt.md) root of each set of shares. -4. Calculate the final share commitment by taking the merkle root (note: not an +1. Calculate the final share commitment by taking the merkle root (note: not an NMT, just a normal binary merkle root) of the subtree roots from the previous step. From e0f06991e81e16244656da513b6e5942fb516c3c Mon Sep 17 00:00:00 2001 From: Evan Forbes <42654277+evan-forbes@users.noreply.github.com> Date: Mon, 19 Jun 2023 14:09:53 -0500 Subject: [PATCH 14/16] docs: review feedback Co-authored-by: Sanaz Taheri <35961250+staheri14@users.noreply.github.com> Co-authored-by: Rootul P --- x/blob/README.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/x/blob/README.md b/x/blob/README.md index ea4bf4b832..785928bba9 100644 --- a/x/blob/README.md +++ b/x/blob/README.md @@ -11,7 +11,7 @@ the Celestia blockchain. Users create a single `BlobTx` that is composed of: 1. `Data []byte`: the data to be published. 1. `ShareVersion uint32`: the version of the share format used to encode this blob into a share. -1. A single [`sdk.Tx`](https://github.com/celestiaorg/cosmos-sdk/blob/v1.15.0-sdk-v0.46.13/docs/architecture/adr-020-protobuf-transaction-encoding.md) which is composed of: +1. A single [`sdk.Tx`](https://github.com/celestiaorg/cosmos-sdk/blob/v1.15.0-sdk-v0.46.13/docs/architecture/adr-020-protobuf-transaction-encoding.md) which encapsulates a `MsgPayForBlobs` message that is composed of: 1. `Signer string`: the transaction signer 1. `NamespaceIds []byte`: the namespaces they wish to publish each blob to. The namespaces here must match the namespaces in the `Blob`s. @@ -35,7 +35,7 @@ data do in fact exist in a particular block. ## State -The blob module doesn't maintain it's own state outside of a two params. Meaning +The blob module doesn't maintain it's own state outside of two params. Meaning that the blob module only uses the params and auth module stores. ### Params @@ -54,7 +54,7 @@ message Params { #### `GasPerBlobByte` `GasPerBlobByte` is the amount of gas that is consumed per byte of blob data -when a `MsgPayForBlob` is processed. Currently, the Default value is 8. This +when a `MsgPayForBlobs` is processed. Currently, the default value is 8. This value is set below that of normal transaction gas consumption, which is 10. #### `GovMaxSquareSize` @@ -70,7 +70,6 @@ details. - [`MsgPayForBlobs`](https://github.com/celestiaorg/celestia-app/blob/v1.0.0-rc2/proto/celestia/blob/v1/tx.proto#L16-L31) pays for a set of blobs to be included in the block. Blob transactions that contain this `sdk.Msg` are also referred to as "PFBs". - this `sdk.Msg` are better known as "PFBs". ```proto message MsgPayForBlobs { @@ -82,12 +81,12 @@ message MsgPayForBlobs { } ``` -`MsgPayForBlobs` pays for the inclusion of a blob in the block and consists of the +`MsgPayForBlobs` pays for the inclusion of blobs in the block and consists of the following fields: - signer: bech32 encoded signer address -- namespace: namespace is a byte slice of length 33 where the first byte is the - namespaceVersion and the subsequent 32 bytes are the namespaceId. +- namespace: namespace is a byte slice of length 29 where the first byte is the + namespaceVersion and the subsequent 28 bytes are the namespaceId. - blob_sizes: sizes of each blob in bytes. - share_commitments is a list of share commitments (one per blob). - share_versions are the versions of the share format that the blobs associated From 920760cb312985deed1e99c8a6f681a8a4c24f29 Mon Sep 17 00:00:00 2001 From: evan-forbes Date: Mon, 19 Jun 2023 14:22:08 -0500 Subject: [PATCH 15/16] docs: use adr 007 --- x/blob/README.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/x/blob/README.md b/x/blob/README.md index 785928bba9..30f67b6e47 100644 --- a/x/blob/README.md +++ b/x/blob/README.md @@ -20,18 +20,20 @@ the Celestia blockchain. Users create a single `BlobTx` that is composed of: this `BlobTx`. After the `BlobTx` is submitted to the network, a block producer separates the -transaction i.e., `sdk.Tx` from the blob. Both components get included in the data square in -different namespaces: the `sdk.Tx` of the original `BlobTx` together with some metadata about the separated blobs get included in the PayForBlobNamespace (one -of the [reserved +transaction i.e., `sdk.Tx` from the blob. Both components get included in the +data square in different namespaces: the `sdk.Tx` of the original `BlobTx` +together with some metadata about the separated blobs get included in the +PayForBlobNamespace (one of the [reserved namespaces](../../specs/src/specs/consensus.md#reserved-namespaces)) and the associated blob gets included in the namespace the user specified in the -original `BlobTx`. Further reading: [Data Square Layout](../../specs/src/specs/data_square_layout.md) +original `BlobTx`. Further reading: [Data Square +Layout](../../specs/src/specs/data_square_layout.md) After a block has been created, the user can verify that their data was included in a block via a blob inclusion proof. A blob inclusion proof uses the -`ShareCommitment` in the original `sdk.Tx` transaction and subtree roots of the block's -data square to prove to the user that the shares that compose their original -data do in fact exist in a particular block. +`ShareCommitment` in the original `sdk.Tx` transaction and subtree roots of the +block's data square to prove to the user that the shares that compose their +original data do in fact exist in a particular block. ## State @@ -93,7 +95,7 @@ following fields: with this message should use when included in a block. The share_versions specified must match the share_versions used to generate the share_commitment in this message. See - [ADR014](../../docs/architecture/adr-014-versioned-namespaces.md) for more + [ADR007](../../docs/architecture/adr-007-universal-share-prefix.md) for more details on how this effects the share encoding and when it is updated. Note that while the shares version in each protobuf encoded PFB are uint32s, the From 5cc0296392c05e48716544da5b66b1aa86358636 Mon Sep 17 00:00:00 2001 From: evan-forbes Date: Mon, 19 Jun 2023 14:24:36 -0500 Subject: [PATCH 16/16] docs: add link --- x/blob/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/x/blob/README.md b/x/blob/README.md index 30f67b6e47..40e73bd849 100644 --- a/x/blob/README.md +++ b/x/blob/README.md @@ -35,6 +35,8 @@ in a block via a blob inclusion proof. A blob inclusion proof uses the block's data square to prove to the user that the shares that compose their original data do in fact exist in a particular block. +> TODO: link to blob inclusion (and fraud) proof + ## State The blob module doesn't maintain it's own state outside of two params. Meaning