Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore!: move blob protos to app under it's own package #2659

Merged
merged 5 commits into from
Oct 16, 2023

Conversation

cmwaters
Copy link
Contributor

@cmwaters cmwaters commented Oct 11, 2023

Ref: #2570

This PR creates a pkg/blob where the protos for Blob and BlobTx is created. The intention is that this becomes it's own go.mod to avoid needing to depend on celestia-core and our cosmos-sdk fork. Other common structures like namespace should do the same so we can achieve some cannonical representation of the structs between node and core/app

The reason for creating the types here instead of in celestia-core is so that only celestia-app becomes a multi go.mod repo as opposed to either. Additionally, Blob and BlobTx were never part of the upstream CometBFT.

I also made the decision to have a single representation of blobs rather than a proto and go native. The reason being is one of simplification and performance. With large blobs, copying the data across to another type can lead to a noticeable impact. Having a single struct however does have the dowside of potential coversion errors from uint32 to uint8 so data needs to be validated before it can be used (this is also the same as before).

This PR also ports over the proofs. I have put this all under a common/v1 proto package.

@codecov-commenter
Copy link

codecov-commenter commented Oct 11, 2023

Codecov Report

Merging #2659 (8c43cee) into main (dcb10cf) will decrease coverage by 1.22%.
The diff coverage is 3.66%.

@@            Coverage Diff             @@
##             main    #2659      +/-   ##
==========================================
- Coverage   21.11%   19.89%   -1.22%     
==========================================
  Files         138      139       +1     
  Lines       15767    16694     +927     
==========================================
- Hits         3329     3322       -7     
- Misses      12115    13050     +935     
+ Partials      323      322       -1     
Files Coverage Δ
pkg/da/data_availability_header.go 72.32% <ø> (ø)
pkg/namespace/consts.go 100.00% <ø> (ø)
pkg/proof/proof.go 70.00% <100.00%> (ø)
pkg/shares/parse_sparse_shares.go 53.44% <100.00%> (-3.70%) ⬇️
pkg/shares/share_splitting.go 48.14% <100.00%> (ø)
pkg/square/builder.go 77.85% <100.00%> (+0.82%) ⬆️
pkg/square/square.go 54.91% <100.00%> (-1.27%) ⬇️
x/blob/types/blob_tx.go 65.82% <100.00%> (+11.08%) ⬆️
app/check_tx.go 0.00% <0.00%> (ø)
app/process_proposal.go 0.00% <0.00%> (ø)
... and 6 more

@cmwaters cmwaters marked this pull request as ready for review October 12, 2023 12:24
@celestia-bot celestia-bot requested a review from a team October 12, 2023 12:24
Copy link
Collaborator

@rootulp rootulp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👏 this is so clean, great work!

I also made the decision to have a single representation of blobs rather than a proto and go native

I like this!

[discussion][not blocking] I think it may help contributors if we include a note on why there exists an x/blob module and a pkg/blob for types. I don't think there's a good place to put this yet but I guess we could add a directory overview section in the README.md. In the absence of that, we could put this note in x/blob/README.md or pkg/blob/README.md (would need to be created).

Comment on lines 18 to 20
// NamespaceVersionMaxValue is the maximum value a namespace version can be.
// This const must be updated if NamespaceVersionSize is changed.
NamespaceVersionMaxValue = math.MaxUint8
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[extremely optional] It seems like this constant could also be exported by pkg/namespace and imported here

Suggested change
// NamespaceVersionMaxValue is the maximum value a namespace version can be.
// This const must be updated if NamespaceVersionSize is changed.
NamespaceVersionMaxValue = math.MaxUint8
// NamespaceVersionMaxValue is the maximum value a namespace version can be.
// This const must be updated if NamespaceVersionSize is changed.
NamespaceVersionMaxValue = ns.NamespaceVersionMaxValue

@cmwaters
Copy link
Contributor Author

[discussion][not blocking] I think it may help contributors if we include a note on why there exists an x/blob module and a pkg/blob for types. I don't think there's a good place to put this yet but I guess we could add a directory overview section in the README.md. In the absence of that, we could put this note in x/blob/README.md or pkg/blob/README.md (would need to be created).

Yeah this makes sense. Can follow up in another PR

Copy link
Member

@evan-forbes evan-forbes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I, and probably everyone, really appreciate the refactor to only have a single struct 🔥

Comment on lines +13 to +15
// ProtoBlobTxTypeID is included in each encoded BlobTx to help prevent
// decoding binaries that are not actually BlobTxs.
const ProtoBlobTxTypeID = "BLOB"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[unrelated to this PR, just a comment no change needed]

I know that we need something like this atm, but there has to be better solution to this

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could maybe be solved with a flavor of #1105

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah although this is breaking so would have to be included in v2 protos. I don't think the BLOB part is that egregious.

As a side note, it could make more sense now having introduced the proposal version change message which isn't really a transaction as it doesn't have any signatures

@celestia-bot celestia-bot requested a review from a team October 16, 2023 11:33
@cmwaters cmwaters merged commit 4669019 into main Oct 16, 2023
32 checks passed
@cmwaters cmwaters deleted the cal/refactor-blob branch October 16, 2023 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants