HIP-1056 Add block streams design #9716

edwin-greene · 2024-11-07T18:20:49Z

Description:
Adds the design for phase one, support for block stream files.

Related issue(s):

Fixes #9573

Notes for reviewer:

Checklist

Documented (Code comments, README, etc.)
Tested (unit, integration, etc.)

Signed-off-by: Edwin Greene <[email protected]>

edwin-greene · 2024-11-07T18:21:26Z

docs/design/block-streams.md

+### Block protobuf to mirror node database mapping
+
+#### Contract Create Transaction
+


More details on the missing entries still to come.

codecov · 2024-11-07T18:27:38Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.26%. Comparing base (7be3d87) to head (b900501).
Report is 160 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #9716      +/-   ##
============================================
- Coverage     92.32%   92.26%   -0.06%     
- Complexity     7413     7771     +358     
============================================
  Files           911      951      +40     
  Lines         31050    32472    +1422     
  Branches       3978     4118     +140     
============================================
+ Hits          28667    29961    +1294     
- Misses         1450     1546      +96     
- Partials        933      965      +32

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Edwin Greene <[email protected]>

edwin-greene · 2024-11-25T20:21:37Z

docs/design/block-streams.md

+```
+
+### Database
+


Currently it is expected that there is a single round per block. The round has a round_number value that currently equals the block number. In the future multiple rounds could be included in a single block. I am not sure if we want to capture this somewhere.

Rounds are sequential right? If so, can future proof by adding round_start, round_end columns.

Sounds good, I have added them.

steven-sheehy · 2024-11-14T18:31:15Z

docs/design/block-streams.md

+
+## REST API
+
+-Deprecate the REST API's State Proof Alpha. The record files and signature files will no longer be provided in the cloud buckets.


Suggested change

-Deprecate the REST API's State Proof Alpha. The record files and signature files will no longer be provided in the cloud buckets.

- Deprecate the REST API's State Proof Alpha. The record files and signature files will no longer be provided in the cloud buckets.

steven-sheehy · 2024-11-14T18:35:06Z

docs/design/block-streams.md

+
+### Data Flow
+
+![Data Flow](images/blockstream.png)


There's a StreamFileNotifier in between.

Thanks I have added it

steven-sheehy · 2024-11-26T18:44:26Z

docs/design/block-streams.md

+```java
+package com.hedera.mirror.importer.downloader.block;
+
+public class BlockStreamVerifier implements StreamFileNotifier, Closable {


The StreamFileNotifier is an existing component to batch and send stream files. The BlockStreamVerifier should leverage it, not implement it.

Got it, I have updated that part.

steven-sheehy · 2024-11-26T18:50:16Z

docs/design/block-streams.md

+public abstract class StreamPoller<StreamFile> {
+    public abstract void poll();


Should prefer interfaces

Suggested change

public abstract class StreamPoller<StreamFile> {

public abstract void poll();

public interface StreamPoller<StreamFile> {

void poll();

Sounds good, I have updated it to an interface.

steven-sheehy · 2024-11-26T18:56:13Z

docs/design/block-streams.md

+```java
+package com.hedera.mirror.common.domain.transaction;
+
+public class BlockItem implements StreamItem {


I assume this won't do any parsing like RecordItem currently does and instead just contain the Transaction, results, output, and state changes? Maybe we can show those fields and make it a simple record?

It corresponds to the protobuf BlockItem. (Maybe it should be removed in favor of just using the protobuf class directly). The block item will only have an eventTransaction or a transactionOutput or one of the other types.

steven-sheehy · 2024-11-26T19:09:42Z

docs/design/block-streams.md

+```
+
+### Database
+


Rounds are sequential right? If so, can future proof by adding round_start, round_end columns.

steven-sheehy · 2024-11-26T19:11:10Z

docs/design/block-streams.md

+```java
+package com.hedera.mirror.importer.downloader.block;
+
+public class BlockStreamVerifier implements StreamFileNotifier, Closable {


Closable is misspelled. But why is it needed? This should be a singleton bean.

Thanks I have removed that part, it will not be needed if StreamFileNotifier is used rather than implemented.

steven-sheehy · 2024-11-26T19:18:04Z

docs/design/block-streams.md

+| name               | Name of the blk file                                                                                                                |
+| node_id            |                                                                                                                                     |
+| prev_hash          | block_header.previous_block_hash                                                                                                    |
+| sidecar_count      | Count of all transaction*output.\_transaction type*.sidecars in the blk file                                                        |


Should be zero. Sidecars should go away.

Got it I have updated it with a note.

steven-sheehy · 2024-11-26T19:18:56Z

docs/design/block-streams.md

+| prev_hash          | block_header.previous_block_hash                                                                                                    |
+| sidecar_count      | Count of all transaction*output.\_transaction type*.sidecars in the blk file                                                        |
+| size               | byte size of the blk file                                                                                                           |
+| version            | Record stream version - Perhaps a new version should be added to indicate "Translated from Block"                                   |


Just assume it's 7.

Sounds good I have updated this to 7

steven-sheehy · 2024-11-26T19:22:30Z

docs/design/block-streams.md

+| Database       | Block Item                                             |
+| -------------- | ------------------------------------------------------ |
+| entity.id      | state_changes[i].state_change.map_update.value.topicId |
+| entity.deleted | state_changes[i].state_change.map_update.value.deleted |


This value doesn't exist. This should be true i map_delete present or false for any other update.

Thanks for spotting, I have updated it to map_delete/key/value

Signed-off-by: Edwin Greene <[email protected]>

sonarcloud · 2024-12-03T18:23:24Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Add block streams design

07bbf43

Signed-off-by: Edwin Greene <[email protected]>

edwin-greene added enhancement Type: New feature design labels Nov 7, 2024

edwin-greene self-assigned this Nov 7, 2024

edwin-greene linked an issue Nov 7, 2024 that may be closed by this pull request

HIP-1056: Design block stream support #9573

Open

edwin-greene commented Nov 7, 2024

View reviewed changes

edwin-greene added 8 commits November 8, 2024 13:23

Add more mapping values

c7c7113

Signed-off-by: Edwin Greene <[email protected]>

Add record file table

e7c0d94

Signed-off-by: Edwin Greene <[email protected]>

Update classes and interfaces

e3532e6

Signed-off-by: Edwin Greene <[email protected]>

Remove phase 2 details

c20aad7

Signed-off-by: Edwin Greene <[email protected]>

Updates to block steam design interfaces

a7d3352

Signed-off-by: Edwin Greene <[email protected]>

Add details on block hash verification

92d28dd

Signed-off-by: Edwin Greene <[email protected]>

Add details on cryptotransfer

4bbc577

Signed-off-by: Edwin Greene <[email protected]>

Add mapping

dd60d10

Signed-off-by: Edwin Greene <[email protected]>

edwin-greene marked this pull request as ready for review November 22, 2024 21:33

edwin-greene requested a review from a team as a code owner November 22, 2024 21:33

edwin-greene added 2 commits November 25, 2024 12:26

Add example mapping of transaction/token transfer

d8117bc

Signed-off-by: Edwin Greene <[email protected]>

Add translation image

4c5317b

Signed-off-by: Edwin Greene <[email protected]>

edwin-greene commented Nov 25, 2024

View reviewed changes

steven-sheehy requested changes Nov 26, 2024

View reviewed changes

Address feedback

b900501

Signed-off-by: Edwin Greene <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HIP-1056 Add block streams design #9716

HIP-1056 Add block streams design #9716

edwin-greene commented Nov 7, 2024

edwin-greene Nov 7, 2024

codecov bot commented Nov 7, 2024 •

edited

Loading

edwin-greene Nov 25, 2024

steven-sheehy Nov 26, 2024

edwin-greene Dec 3, 2024

steven-sheehy Nov 14, 2024

edwin-greene Dec 3, 2024

steven-sheehy Nov 14, 2024

edwin-greene Dec 3, 2024

steven-sheehy Nov 26, 2024

edwin-greene Dec 3, 2024

steven-sheehy Nov 26, 2024

edwin-greene Dec 3, 2024

steven-sheehy Nov 26, 2024

edwin-greene Dec 3, 2024

steven-sheehy Nov 26, 2024

steven-sheehy Nov 26, 2024

edwin-greene Dec 3, 2024

steven-sheehy Nov 26, 2024

edwin-greene Dec 3, 2024

steven-sheehy Nov 26, 2024

edwin-greene Dec 3, 2024

steven-sheehy Nov 26, 2024

edwin-greene Dec 3, 2024

sonarcloud bot commented Dec 3, 2024

		### Block protobuf to mirror node database mapping

		#### Contract Create Transaction


		## REST API

		-Deprecate the REST API's State Proof Alpha. The record files and signature files will no longer be provided in the cloud buckets.

	-Deprecate the REST API's State Proof Alpha. The record files and signature files will no longer be provided in the cloud buckets.
	- Deprecate the REST API's State Proof Alpha. The record files and signature files will no longer be provided in the cloud buckets.

		public abstract class StreamPoller<StreamFile> {
		public abstract void poll();

HIP-1056 Add block streams design #9716

Are you sure you want to change the base?

HIP-1056 Add block streams design #9716

Conversation

edwin-greene commented Nov 7, 2024

Choose a reason for hiding this comment

codecov bot commented Nov 7, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented Dec 3, 2024

Quality Gate passed

codecov bot commented Nov 7, 2024 •

edited

Loading