Skaled crashes under significant load with MTM

[oleksandrSydorenkoJ]

Version:
skalenetwork/schain:3.16.0-stable.3

Preconditions:
Active Skale-chain medium type with enabled MTM
all nodes active and mining blocks
Amount of accounts = 1000
All accounts have 3 sFuel on balance
MTM amount = 10

Steps to reproduce:

1. Run load tests to 16 endpoints
2. Wait for 1-2 hours
3. Check skaled containers on the core nodes

Expected state:
Skaled should mining blocks without any crashes

Actual state:
Skaled crashes without errors. After restarting skaled container by admin, skaled continue mining blocks.

Logs:
crash_skaled_under_load.txt

[kladkogex]

Marking P1 as this is a crash

[kladkogex]

I think I figured out what the problem is - assigning it to myself

[kladkogex]

Reassigning this to @dimalit since Dima is saying that there is no problem with iterators.

There is something segfaulting inside the queue

[dimalit]

Need to find version on which crash was occurring, and try to fix it with
https://github.com/skalenetwork/skale-consensus/pull/800/files
or maybe with
https://github.com/skalenetwork/skale-consensus/pull/799/files

[DmytroNazarenko]

skaled: 3.17.1

[oleksandrSydorenkoJ]

should be fixed with #1588

[oleksandrSydorenkoJ]

This issue is the result of the memory leaks and should be fixed at https://github.com/skalenetwork/internal-support/issues/931