Ignited Knives

What's Changed

• Fix t_proposer_duties not being consistent with t block metrics on missed blocks by @santi1234567 in #129
• Fix/capella blocks download crashing by @santi1234567 in #130
• change the way we calculate num attestations to use block data by @santi1234567 in #135
• Fix/the first epoch downloaded has incomplete data by @santi1234567 in #136
• fix no retry on error download state by @santi1234567 in #138
• Feature/add rewards aggregation summary by @santi1234567 in #139
• Feature/add sync participation to epoch metrics by @santi1234567 in #140
• Feature/add deposits withdrawals to epoch metrics by @santi1234567 in #141
• Feature/add new slashed validators to epoch metrics by @santi1234567 in #142
• Feature/create slashings table by @santi1234567 in #143
• Prepare v3.4.0 by @santi1234567 in #144

Important notes

Two new tables were added t_validator_rewards_aggregation and t_slashings. Refer to the table docs for more information on these.

A new flag --rewards-aggregation-epoch with its corresponding .env variable GOTETH_ANALYZER_REWARDS_AGGREGATION_EPOCHS were added.

Full Changelog: v3.3.0...v3.4.0

Ignited Flame

Note: The change on t_withdrawals from #125 will create a new table and fill it again (maintaining the name). We suggest backing it up before in case something happens in the middle of the migration process.

What's Changed

• fix: dockerfile by @barnabasbusa in #120
• Update .gitmodules by @santi1234567 in #123
• Bump go to 1.21 by @santi1234567 in #124
• change t_withdrawals to order by index by @santi1234567 in #125
• fix bug when init_slot less than 32 by @santi1234567 in #126

New Contributors

• @barnabasbusa made their first contribution in #120

Full Changelog: v3.2.0...v3.3.0

Ignited Kitten

Summary

This release introduces minor refactors and major new feature: block execution rewards metrics.

• spec functions look more like Ethereum specification
• New table: block fees rewards and relay information
  • Keep in mind there can be additional rewards not tracked here, like searchers payments to MEV builders.
• Fix bug in the t_blob_sidecars_events table
• Fix bug on the max inclusion delay
• Fix bug when creating missed blocks performance stats for pools

Database changes

• There is a new table t_block_rewards
• Added three new columns in t_epoch_metrics_summary
  • f_source_att_effective_balance_eth
  • f_target_att_effective_balance_eth
  • f_head_att_effective_balance_eth
• Switch column type of column f_att_effective_balance_eth in t_epoch_metrics_summary to UInt64

What's Changed

• Refactor metrics to be more Ethereum-like specs by @tdahar in #113
• Track execution layer block rewards by @tdahar in #114
• Blob events: keep all events without overwriting by @tdahar in #115
• add attesting balance per flags in epochs table by @tdahar in #116
• Set inclusion delay to maxinclusion+1 when there is no attestation by @tdahar in #117
• Also check proposer index as proposed can be false by @tdahar in #119

Full Changelog: v3.1.0...v3.2.0

Ignited Jedi

Disclaimer

Since v3.0.0 this tool uses Clickhouse as the database server. If you wish to migrate your existing PostgreSQL database to Clickhouse please refer to https://github.com/migalabs/goteth/releases/tag/v3.0.0

Summary

Database

• Inclusion Delay is now calculated and also added to pool summaries
• Blobs are retrieved when the transactions metrics is enabled
• Fixed bug in the t_validator_last_status table, where the val_idx was always 0

Rewards system

• When calculating the maximum attestation rewards for a validator, missed blocks are now taken into account

What's Changed

• Feature/add inclusion delay column to rewards by @santi1234567 in #94
• Attestation max reward: take into account missed blocks by @tdahar in #109
• Request Txs and Blobs in bulk by @tdahar in #108
• Fix validator last status persisting wrong val idx by @tdahar in #110
• Add column avg inclusion delay to pools summary by @tdahar in #111
• v3.1.0 preparation by @tdahar in #112

Full Changelog: v3.0.0...v3.1.0

Ignited Ice

Disclaimer

PostgreSQL is not used anymore from v3.0.0 onwards. The only database supported now is Clickhouse.

Summary

This release carries the major database change we recently implemented.

• Migration from PostgreSQL to Clickhouse.
• Updated the dependencies to be compliant with the Deneb fork
• Added new metrics: head events to monitor the arrival of new blocks and the manual block reward calculation (still experimental).

Database migration

If you wish to migrate from your existing PostgreSQL database to Clickhouse, please follow this guide. Here you will find instructions and our experience migrating data from one server to the other.
Keep in mind the migration can only take place with the v3.0.0 release. After that you can upgrade to a newer version.

Docker Images

• Build migalabs/goteth:v3.0.0

What's Changed

• Feature/got eth validator summary metrics by @santi1234567 in #89
• Calculate Block Rewards manually by @tdahar in #90
• Feature/store headevent arrival timestamp by @santi1234567 in #92
• Add support for clickhouse database by @tdahar in #93
• Update API dependencies by @tdahar in #101
• Fix wrong epoch timestamp column by @tdahar in #104
• Fix tests not working as per recent upgrades by @tdahar in #105
• Reorgs should consider blocks from the reorg base to the head by @tdahar in #106
• Prepare release v3.0.0 by @tdahar in #107

New Contributors

• @santi1234567 made their first contribution in #89

Full Changelog: v2.2.0...v3.0.0

Happy Kitten

Release Notes

This release includes an important refactor of how the blocks and states are downloaded and processed.

• A cache has been added that gets cleaned with the finalized checkpoint signal
• Generic maps have been added to the cache
• Download and processer are now separate routines and functions
• Tests have been improved

A few new database migrations have been added:

• Transaction primary key -> (f_slot, f_hash): this way orphan blocks can include repeated transactions until they are deleted
• Removed primary key in validator_last_status: manual deletion as it makes the query faster

However, all the data persisted and the tool configuration remained the same. This release serves as a base for new features, which will be more atomic and easier to deploy as of the refactor.

What's Changed

• Feat/add generic map by @cortze in #85
• Refactor the main routines: Asynchronous download and processing (step0) by @tdahar in #84
• Add Finalized checker in async routine (step1) by @tdahar in #86
• Tests/improve tx query by @tdahar in #83
• Refactor/performance review (step 2) by @tdahar in #87
• Refactor main routine to handle events asynchronously (in separate goroutines) by @tdahar in #88

Full Changelog: v2.1.0...v2.2.0

Happy Jelly

Release Notes

This release fixes some errors introduced in the previous versions where the orphans table was not getting filled.
Also, it introduces a new flag in the metrics list: one can now choose whether to query API rewards (only blocks right now) or skip this.
Keep in mind, if this is skipped then rewards cannot be compared to max_reward when a validator is a proposer in the epoch.

What's Changed

• Add pending migrations to orphans table by @tdahar in #82
• Feat/optional api rewards by @tdahar in #81

Full Changelog: v2.0.1...v2.1.0

Happy Ice

Release Notes

This release contains several minor fixes.

• Repository URL dependency has been fixed in all the files
• Name has been changed in the docker files

New features

• Database tests have been added
• Snappy Compression details added for blocks

Bug Fixes:

• Fix block number and timestamp not stored in the transactions table

What's Changed

• Add DB integrity checks by @cortze in #75
• Fix block number and timestamp not stored in the transactions table by @tdahar in #74
• Allow one validator to be proposer several times in epoch by @tdahar in #77
• Test non-existent tx data for tracked blocks by @cortze in #76
• Use direct migrate.ErrNoChange for specific Error check case by @cortze in #79
• Add snappy compression details by @cortze in #78
• Refactor/update repository url by @tdahar in #80

Full Changelog: v2.0.0...v2.0.1

Happy Honey

Changelog

• The tool now supports the Deneb upcoming fork
• The tool now exposes minimal prometheus metrics (to be expanded in future PRs)

Database

• The tool now supports pool-summaries through a trigger (when an entity is inserted into the t_eth2_pubkeys table)
• The tool now stores the genesis time in the database, so as two different networks are not run in the same database
• The tool now follows a set of database migrations (following changes will mean database version upgrades)

Merge both services into 1

• Only one service
• Remove second service .env variables
• The service now download blocks and states sequentially (when we cross an epoch, download state)
• Blocks are inserted into the state structure, so more metrics can be extracted
• Rewards now take into account block rewards and withdrawals

Finality

• Finality checkpoints are stored in the database now
• Reorgs are identified and the tool rewinds and redownloads data when needed
• Orphan blocks are inserted in a separate table

Docker Images

• v2.0.0

Contributors:

• @tdahar
• @cortze
• @ocheja

v1.0.0 - Giant Panda

General Description

This first v1.0.0 release compiles a stable Ethereum CL Validator rewards indexer. It includes:

• Full Postgres integration for later data analysis
• CL rewards support for Phase0, Altair, and Bellatrix
• Indexing the max reward that a validator could have at a given epoch
• The quality of the attestation for a validator at a given epoch based on the 3 attestation flags

What's Changed

• Branch to calculate the participationRate based on BeaconState by @tdahar in #1
• Feature/add db support by @tdahar in #2
• Max Rewards in Altair by @tdahar in #3
• Create a common wrapper across specs by @tdahar in #4
• Create branch to track missing Target, source and head by @tdahar in #5
• Optimize code to calculate metrics faster by @tdahar in #6
• Refactor/standardize specs by @tdahar in #7
• Feature/additional metrics by @tdahar in #8
• Improvement/int calculation by @tdahar in #9
• Fix/stored metrics by @tdahar in #11
• update analyzer for safe-shutdown by @cortze in #13
• Stable version optimizing DB interaction, DB total size and State processing times by @cortze in #12
• Testing/add tests by @tdahar in #14
• Update README + cleanup tool commands + Add unitary tests by @cortze in #15

New Contributors

• @tdahar made their first contribution in #1
• @cortze made their first contribution in #13

Full Changelog: https://github.com/cortze/eth2-state-analyzer/commits/v1.0.0