feat: merk state sync chunking #267
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
finalize multi chunk with limit
implement chunk op
test chunk op encoding
convert chunk op chunk id to string
implement traversal instruction to string
add chunking error + devoid multi subtree chunk from encoding work
multi-subtree-chunk skeleton + return encoding length in multichunk
make chunks fixed size height 2
test height proof
implement height proof verifier
update documentation
verify height proof generation
add documentation
test no of chunk under chunk id fn
implement number of chunks under chunk id function
extract chunk layer function from chunk height
seperate number_of_chunk into height and layer_height functions
return multi chunk result
enforce limit without storage overhead
add test for encoding length check
implement iterator for chunk producer
remove cost from chunks
fix the error type
implement random chunk access
fixes
implement chunk height function
add traverse then build chunk function to ref walker
update comment
implement chunk producer length
init chunk producer struct
implement merk tree height function
update traversal generation instruction
add instruction traversal test
fix documentation
implement binary range function
clean up number of chunks function
given a subtree of a given height return the exit node count
documentation fixes
implement chunk_height_per_layer
verify that chunks produce expected root hash
implement and test variable depth chunk creation
restart chunking v2
remove bad test
rename files
update documentation
wip
wip
implement merk verifier + state building
implement replication from multichunk
fix chunk verification
fixed implementation of chunkid from traversal instructions
fix some tests
make chunk_id from traversal instruction test resistant to changes in underlying chunking scheme
add restoration logic test function
returning the next chunk id when you call chunk
use strings as communication interface between producer and restorer
implement chunk id from traversal instruction
add traversal instruction generation to direct string
chunk producer returns next index as string for multi chunk
clean up rewrite parent links
restoration done successfully
rough implementation of rewrite parent
implement function to extract sum from node type
wip
chunk write logic + restorer finalization + parent key tracking
new visit ref function that keeps track of traversal path
implement instruction string to traversal instruction
test child to link functionality for basic and sum merks
implement node to link include sum
wip
implement and test chunk verification
Fix layer iter function
Previous implementation made a key assumption that nodes are unique
including hash nodes, this made the layer iteration functionality
depend on the contents of the tree, which shouldn't be the case.
This adds a simpler implementation of the layer iter logic using breadth
first search.
add test to ensure chunks only contain hash and kvfeaturetype
test for avl tree during proof op execution
remove chunk_height_per_layer_lin_comb every chunk now has fixed height of 2
fominok
reviewed
Oct 11, 2023
|
|
||
| // generate as many subtree chunks as we can | ||
| // until we have exhausted all or hit a limit restriction | ||
| while current_index.is_some() { |
There was a problem hiding this comment.
there is while let syntax in Rust to cleanup some expects later
fominok
reviewed
Oct 11, 2023
| } | ||
|
|
||
| /// Returns the chunk at the given index | ||
| /// Assumes index and traversal_instructions represents the same information |
There was a problem hiding this comment.
This might be a sign of a new type needed, to ensure two field are in sync to have an invariant
5 tasks
|
Already included in #292 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
State sync specification required us to make changes to how we handle previously handled chunking.
Before, a single trunk of height = h / 2 is first sent, the node hash values for subsequent chunks are gotten and then each chunk is retrieved and applied sequentially.
Problems with this:
This work addresses those problems:
Issue being fixed or feature implemented
What was done?
How Has This Been Tested?
Breaking Changes
Checklist:
For repository code-owners and collaborators only