Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance on the indexer node decreases as the number of archive databases increases. #2017

Open
oleksandrSydorenkoJ opened this issue Oct 7, 2024 · 1 comment
Open

Performance on the indexer node decreases as the number of archive databases increases. #2017

oleksandrSydorenkoJ opened this issue Oct 7, 2024 · 1 comment
Assignees
@olehnikolaiev
Labels
bug Something isn't working epic:archive-node epic:HPD
Milestone
SKALE 4.1

Comments

@oleksandrSydorenkoJ
Copy link

Describe the bug
The indexer node saves all blocks from the Skale chain to archive db.
All databases stay open during skaled work, which may lead to significant resources usage and skaled hangs.

Preconditions
Active schain medium type with connected Indexer node
Indexr node should have at least 300 million transactions of token transfer
Load tests:

  • 1000 users
  • data size 100 bytes
  • gasUsage for the transaction - 100 000 units

Skale chain on QA-net: skale_schain_massive-long-sirius

Versions
skalenetwork/schain:3.19.3-beta.0

To Reproduce

  1. Run load tests and wait for 24 hours
  2. Send 5 JSON-RPC requests "eth_getBlockByNumber("latest", false)"
    Example call
  3. Check the latest block on the indexer node
  4. Check the response time

Expected behavior
Skaled on indexer node should be fully synced with core nodes during the load
No performance warnings - skaled on indexer node should operate JSON-RPC calls less than in 1 second

Amount of open databases in skaled

schains/massive-long-sirius/28e07f34# find blocks_and_extras/ -mindepth 1 -maxdepth 1 -type d | wc -l
2080

All databases in Snapshots directory

schains/massive-long-sirius# find snapshots/*  -mindepth 1 -maxdepth 3 -type d | wc -l
6290

2024-10-07 15:00:05.365725   Performance warning: 2.821943 seconds execution time for eth_getBlockByNumber call with id=540 when called from origin http://localhost:47992 through server with index=0
[2024-10-07 15:00:06.169] [8:main] [info] 2255325:BLOCK_COMMITED: PRPSR:15:BID: 2255326:ROOT:14950692840982967621153295883338654291372433785765979208503044696449601572311:HASH:4149600e:BLOCK_TXS:17:DMSG:0:TPRPS:402:MPRPS:0:RPRPS:0:TXS:156497:TXLS:402:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:33976:PRT:0:BTA:57610440204:BSA:0:TPS:0:LWT:0:LRT:0:LWC:91:LRC:78:STAMP:1728220748.912
2024-10-07 15:00:06.117266   Block sealed #2255325 (#1666580d…)
2024-10-07 15:00:06.117323   Block stats:BN:2255324:BTS:1728220744:TXS:2487:HDRS:12:LOGS:13766:SENGS:1:TXRS:13726:BLCKS:3:ACCS:199:BQS:1:BDS:72:TSS:0:UTX:0:VTX:0:CMM:10640662:KDS:0
2024-10-07 15:00:06.121137   Block 2255325 DB usage is 1151224. Piece DB usage is 122382995 bytes
2024-10-07 15:00:06.132590   SWT:1475:BFT:1605
2024-10-07 15:00:06.132603   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
2024-10-07 15:00:06.286485   Block sealed #2255326 (#0a245d4f…)
2024-10-07 15:00:06.286508   Block stats:BN:2255325:BTS:1728220745:TXS:51:HDRS:12:LOGS:12971:SENGS:1:TXRS:12931:BLCKS:3:ACCS:199:BQS:1:BDS:73:TSS:0:UTX:0:VTX:0:CMM:15252712:KDS:0
2024-10-07 15:00:06.286643   Block 2255326 DB usage is 24668. Piece DB usage is 122407663 bytes
2024-10-07 15:00:06.294115   SWT:14:BFT:161
2024-10-07 15:00:06.294128   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
[2024-10-07 15:00:06.321] [8:main] [info] 2255326:BLOCK_COMMITED: PRPSR:1:BID: 2255327:ROOT:14950692840982967621153295883338654291372433785765979208503044696449601572311:HASH:4e5297a9:BLOCK_TXS:646:DMSG:0:TPRPS:402:MPRPS:0:RPRPS:0:TXS:156497:TXLS:402:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:34089:PRT:0:BTA:55752038912:BSA:0:TPS:0:LWT:0:LRT:0:LWC:94:LRC:80:STAMP:1728220750.138
2024-10-07 15:00:07.326347   Block sealed #2255327 (#8f14aa5d…)
2024-10-07 15:00:07.326402   Block stats:BN:2255326:BTS:1728220748:TXS:1938:HDRS:12:LOGS:14246:SENGS:1:TXRS:14206:BLCKS:3:ACCS:199:BQS:1:BDS:74:TSS:0:UTX:0:VTX:0:CMM:15252712:KDS:0
2024-10-07 15:00:07.328746   Block 2255327 DB usage is 897219. Piece DB usage is 123304882 bytes
2024-10-07 15:00:07.338275   SWT:903:BFT:1044
2024-10-07 15:00:07.338293   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
[2024-10-07 15:00:07.368] [8:main] [info] 2255327:BLOCK_COMMITED: PRPSR:1:BID: 2255328:ROOT:14950692840982967621153295883338654291372433785765979208503044696449601572311:HASH:22f9b13a:BLOCK_TXS:278:DMSG:0:TPRPS:402:MPRPS:0:RPRPS:0:TXS:156497:TXLS:402:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:34718:PRT:0:BTA:54009787729:BSA:0:TPS:0:LWT:0:LRT:0:LWC:97:LRC:82:STAMP:1728220753.66
2024-10-07 15:00:07.647972   Block sealed #2255328 (#fc0676be…)
2024-10-07 15:00:07.648004   Block stats:BN:2255327:BTS:1728220750:TXS:834:HDRS:12:LOGS:14156:SENGS:1:TXRS:14116:BLCKS:3:ACCS:199:BQS:1:BDS:75:TSS:0:UTX:0:VTX:0:CMM:15252712:KDS:0
2024-10-07 15:00:07.657688   Block 2255328 DB usage is 386443. Piece DB usage is 123691325 bytes
2024-10-07 15:00:07.665557   SWT:188:BFT:327
2024-10-07 15:00:07.665574   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
[2024-10-07 15:00:07.694] [8:main] [info] 2255328:BLOCK_COMMITED: PRPSR:3:BID: 2255329:ROOT:14950692840982967621153295883338654291372433785765979208503044696449601572311:HASH:ff24cd47:BLOCK_TXS:634:DMSG:0:TPRPS:402:MPRPS:0:RPRPS:0:TXS:156497:TXLS:402:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:34889:PRT:0:BTA:52373127505:BSA:0:TPS:0:LWT:0:LRT:0:LWC:100:LRC:84:STAMP:1728220754.447
2024-10-07 15:00:08.680928   Block sealed #2255329 (#4bb73a10…)
2024-10-07 15:00:08.680962   Block stats:BN:2255328:BTS:1728220753:TXS:1902:HDRS:12:LOGS:15146:SENGS:1:TXRS:15106:BLCKS:3:ACCS:199:BQS:1:BDS:76:TSS:0:UTX:0:VTX:0:CMM:15252712:KDS:0
2024-10-07 15:00:08.684135   Block 2255329 DB usage is 880569. Piece DB usage is 124571894 bytes
2024-10-07 15:00:08.694418   SWT:891:BFT:1028
2024-10-07 15:00:08.694437   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
[2024-10-07 15:00:08.740] [8:main] [info] 2255329:BLOCK_COMMITED: PRPSR:3:BID: 2255330:ROOT:14950692840982967621153295883338654291372433785765979208503044696449601572311:HASH:e33669e3:BLOCK_TXS:3:DMSG:0:TPRPS:402:MPRPS:0:RPRPS:0:TXS:156497:TXLS:402:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:35432:PRT:0:BTA:50832741432:BSA:0:TPS:0:LWT:0:LRT:0:LWC:103:LRC:86:STAMP:1728220756.703
2024-10-07 15:00:08.868479   Block sealed #2255330 (#1e4b0963…)
2024-10-07 15:00:08.868520   Block stats:BN:2255329:BTS:1728220754:TXS:9:HDRS:12:LOGS:14518:SENGS:1:TXRS:14478:BLCKS:3:ACCS:199:BQS:1:BDS:77:TSS:0:UTX:0:VTX:0:CMM:15252712:KDS:0
2024-10-07 15:00:08.868653   Block 2255330 DB usage is 5278. Piece DB usage is 124577172 bytes
2024-10-07 15:00:08.876631   SWT:11:BFT:182
2024-10-07 15:00:08.876643   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
[2024-10-07 15:00:08.909] [8:main] [info] 2255330:BLOCK_COMMITED: PRPSR:5:BID: 2255331:ROOT:14950692840982967621153295883338654291372433785765979208503044696449601572311:HASH:f96fd20a:BLOCK_TXS:914:DMSG:0:TPRPS:402:MPRPS:0:RPRPS:0:TXS:156497:TXLS:402:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:35528:PRT:0:BTA:49380377396:BSA:0:TPS:0:LWT:0:LRT:0:LWC:106:LRC:88:STAMP:1728220757.852
2024-10-07 15:00:09.865421   Performance warning: 4.211493 seconds execution time for eth_getBlockByNumber call with id=541 when called from origin http://localhost:47992 through server with index=0
2024-10-07 15:00:10.726525   Block sealed #2255331 (#9e6193ce…)
2024-10-07 15:00:10.726565   Block stats:BN:2255330:BTS:1728220756:TXS:2742:HDRS:12:LOGS:16343:SENGS:1:TXRS:16303:BLCKS:3:ACCS:199:BQS:1:BDS:80:TSS:0:UTX:0:VTX:0:CMM:15252712:KDS:0
2024-10-07 15:00:10.729571   Block 2255331 DB usage is 1269201. Piece DB usage is 125846373 bytes
2024-10-07 15:00:10.739439   SWT:1710:BFT:1862
2024-10-07 15:00:10.739455   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
[2024-10-07 15:00:10.769] [8:main] [info] 2255331:BLOCK_COMMITED: PRPSR:5:BID: 2255332:ROOT:14950692840982967621153295883338654291372433785765979208503044696449601572311:HASH:b8321dd7:BLOCK_TXS:56:DMSG:0:TPRPS:402:MPRPS:0:RPRPS:0:TXS:156497:TXLS:402:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:36177:PRT:0:BTA:48008700298:BSA:0:TPS:0:LWT:0:LRT:0:LWC:109:LRC:91:STAMP:1728220760.849
2024-10-07 15:00:10.895286   Block sealed #2255332 (#22f45fa4…)
2024-10-07 15:00:10.895306   Block stats:BN:2255331:BTS:1728220757:TXS:168:HDRS:12:LOGS:15541:SENGS:1:TXRS:15501:BLCKS:3:ACCS:199:BQS:1:BDS:81:TSS:0:UTX:0:VTX:0:CMM:15252712:KDS:0
2024-10-07 15:00:10.895551   Block 2255332 DB usage is 78683. Piece DB usage is 125925056 bytes
2024-10-07 15:00:10.901784   SWT:24:BFT:162
2024-10-07 15:00:10.901796   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
[2024-10-07 15:00:10.919] [8:main] [info] 2255332:BLOCK_COMMITED: PRPSR:6:BID: 2255333:ROOT:14950692840982967621153295883338654291372433785765979208503044696449601572311:HASH:6c7df1ed:BLOCK_TXS:575:DMSG:0:TPRPS:402:MPRPS:0:RPRPS:0:TXS:156497:TXLS:402:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:36189:PRT:0:BTA:46711167862:BSA:0:TPS:0:LWT:0:LRT:0:LWC:112:LRC:94:STAMP:1728220762.587
2024-10-07 15:00:11.730274   Block sealed #2255333 (#5e7c25eb…)
2024-10-07 15:00:11.730350   Block stats:BN:2255332:BTS:1728220760:TXS:1725:HDRS:12:LOGS:16635:SENGS:1:TXRS:16595:BLCKS:3:ACCS:199:BQS:1:BDS:82:TSS:0:UTX:0:VTX:0:CMM:15252712:KDS:0
2024-10-07 15:00:11.732960   Block 2255333 DB usage is 798674. Piece DB usage is 126723730 bytes
2024-10-07 15:00:11.746174   SWT:721:BFT:844
2024-10-07 15:00:11.746196   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
[2024-10-07 15:00:11.778] [8:main] [info] 2255333:BLOCK_COMMITED: PRPSR:7:BID: 2255334:ROOT:14950692840982967621153295883338654291372433785765979208503044696449601572311:HASH:ec804c9a:BLOCK_TXS:242:DMSG:0:TPRPS:402:MPRPS:0:RPRPS:0:TXS:156497:TXLS:402:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:36658:PRT:0:BTA:45481926624:BSA:0:TPS:0:LWT:0:LRT:0:LWC:115:LRC:96:STAMP:1728220764.558
2024-10-07 15:00:12.026958   Block sealed #2255334 (#a3e5f65f…)
2024-10-07 15:00:12.026997   Block stats:BN:2255333:BTS:1728220762:TXS:726:HDRS:12:LOGS:524:SENGS:1:TXRS:484:BLCKS:3:ACCS:199:BQS:1:BDS:34:TSS:0:UTX:0:VTX:0:CMM:6837:KDS:0
2024-10-07 15:00:12.027856   Block 2255334 DB usage is 336487. Piece DB usage is 127060217 bytes
2024-10-07 15:00:12.035013   SWT:137:BFT:288
2024-10-07 15:00:12.035028   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
[2024-10-07 15:00:12.065] [8:main] [info] 2255334:BLOCK_COMMITED: PRPSR:8:BID: 2255335:ROOT:14950692840982967621153295883338654291372433785765979208503044696449601572311:HASH:0eb6be56:BLOCK_TXS:658:DMSG:0:TPRPS:402:MPRPS:0:RPRPS:0:TXS:156497:TXLS:402:MGS:0:INSTS:0:BPS:0:HDRS:2:SOCK:1:FDS:36823:PRT:0:BTA:44315723385:BSA:0:TPS:0:LWT:0:LRT:0:LWC:118:LRC:98:STAMP:1728220766.400
[2024-10-07 15:00:12.065] [8:main] [info] 2255334:LEVELDB_MEM_STATS:BLOCKS:2288841:PROPS:16416:DAPS:16416:OMS:16416:PHS:16416:PVS:16416:BSS:16416:IMS:16416:RMS:16416:PCS:16416:IIN:16416:DAS:16416
[2024-10-07 15:00:12.066] [8:main] [info] 2255334:RUSAGE:USER_CPU:24:240603:PROCESS_RSS:1102492:SWAPPED_MEM:0:HEAP_MEM:1065628592:
2024-10-07 15:00:12.938309   Performance warning: 2.053201 seconds execution time for eth_getBlockByNumber call with id=543 when called from origin http://localhost:47992 through server with index=0
@DmytroNazarenko DmytroNazarenko added this to the SKALE 3.2 milestone Oct 7, 2024
@PolinaKiporenko PolinaKiporenko modified the milestones: SKALE 4.1, SKALE 4.0 Oct 23, 2024
@PolinaKiporenko
Copy link
Contributor

PolinaKiporenko commented Oct 23, 2024 •
edited
Loading

Could be divided in two parts:

  1. skaled blocks DB
  2. consensus blocks DB

@PolinaKiporenko PolinaKiporenko modified the milestones: SKALE 4.0, SKALE 4.1 Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@olehnikolaiev olehnikolaiev

Labels
bug Something isn't working epic:archive-node epic:HPD
Projects
SKALE Engineering 🚀
Status: No status
Milestone
SKALE 4.1
Development

No branches or pull requests
4 participants
@DmytroNazarenko @olehnikolaiev @oleksandrSydorenkoJ @PolinaKiporenko