Primitives Module Planning and Outline

[davecgh]

This issue is intended to act as a running high-level outline and provide motivation for overall planned long-term refactors and implementation of a primitives module as time permits.

Currently, the wire module essentially acts as the fundamental building block of some data structures and other definitions for other concepts that are key to consensus. For example, block headers, transactions, blocks, and sequence lock definitions. While this has worked well over the years and is reasonable given the current architecture, it does result in the conflation of two concepts that really are distinct. Namely, that the specific details regarding the way the data structures are serialized and used in consensus is fundamentally entirely distinct from the data sharing protocol even though they are currently highly coincident.

Further, it also has led to several other fundamental consensus-related concepts such as median time sources, determination of transaction finalization, and lock times needing to be defined elsewhere since those are clearly not related to the wire protocol.

Thus, my plan is to implement a new primitives module which will become the new fundamental building block of consensus code with the following motivations and goals:

• Make the distinction between consensus primitives and data sharing much more apparent
• Consolidate some of the aforementioned related concepts
• Open the door for a wide variety of improvements at the network protocol level
• Take advantage of the major refactor to redefine the data structures in a way that significantly reduces allocations and improves cache locality
• Keep dependencies to a minimum and specifically avoid the need for deps such as leveldb
• Provide a single module for the consensus-related functionality needed by consumers that is currently spread across wire, blockchain, blockchain/standalone, and dcrutil
  • For example, block and transaction definitions, proof of work checks, and merkle root calculations
• Compute and store hashes at initialization time for lock-free access

Given that almost everything currently builds on wire either directly or indirectly, this is going to be a fairly intensive and long-running effort. Moreover, many of the individual changes will very likely look out of place and/or incomplete in isolation, so my hope is that this issue along with a high-level outline will provide insight into the end goal.

Note that some of these will very likely need to be updated as the work proceeds since unexpected things invariably come up during large development efforts such as this. They are also not in a strict order.

• Implement uint256 subpackage for all fixed-precision unsigned 256-bit integer needs -- Implemented by PR uint256: Introduce package. #2787
• Refactor code related to PoW checks from blockchain/standalone and update to use the new uint256 instead of standard library big integers -- Implemented by PR primitives: Add initial proof-of-work funcs. #2788
• Refactor code related to inclusion proofs from blockchain/standalone -- Implemented by PR primitives: Add inclusion proof funcs. #2827
• Refactor code related to subsidy from blockchain/standalone -- Implemented by PR primitives: Add subsidy calcs. #2920
• Implement exported support for compact integers (currently varints in wire)
• Implement exported support for VLQs
• Implement Tx including things such as long-term stable serialization and calculation of both witness and full hashes used in consensus
• Implement BlockHeader including things such as long-term stable serialization and calculation of the block hash as used in consensus
• Implement Block including things such as long-term stable serialization and pass through hash calculation
• Refactor code related to merkle root calculations from blockchain/standalone -- Partially implemented by primitives: Add core merkle tree root calcs. #2826
• Refactor code related to the treasury from blockchain/standalone
• Refactor code related to determining transaction types blockchain/standalone
• Remove blockchain/standalone in favor of the new primitives module
• Refactor wire to make use of the new primitives types and remove all consensus-specific definitions
• Refactor code related to MedianTimeSource from blockchain
• Refactor code related to determining transaction finalization and expiration from blockchain
• Refactor blockchain code used by external consumers related to transaction checking
• Make blockchain an internal package
• Ideally implement stake transaction type determination
  • This will probably need to be preceded by a parallel project to change the way stake transactions are identified per other discussions
  • Requires further analysis because it relies on txscript currently which is not desirable for the new module

[davecgh]

Updated issue description for PR #2787.

[davecgh]

Updated issue description for PR #2788.

[davecgh]

Updated issue description for #2826.

[davecgh]

Updated issue description for #2827.

[davecgh]

Updated issue description for #2920.

[in-eth]

Hi, @davecgh . Could you explain more details about "Implement BlockHeader including things such as long-term stable serialization and calculation of the block hash as used in consensus"? I would like to contribute with that ticket.