Out of memory while synchronizing the node #2624
Comments
|
Do you have logs, memory usage matrices etc from the server? |
Please see this. It has used 49GB and this number will increase till out of memory (64GB). |
|
This is not enough for us to diagnose the issue. |
|
ubuntu 22.04.3 LTS Logs: 
My RAM is 64GB |
|
Your screenshot is unreadable. Please just copy & paste the logs instead of screenshots. |
|
I have switched to 2.20. This issue still occurs . 2023-10-05 02:29:42 [Parachain] ✨ Imported #4600692 (0xc7d6…cd25) .... .... the RAM usage keeps increasing..... (the capacity is 16GB ) 2023-10-05 05:45:07 [Relaychain] ⚙️ Syncing 37.5 bps, target=#17584918 (36 peers), best: #8648566 (0x47ee…85fd), finalized #8648192 (0xd17f…ce76), ⬇ 732.9kiB/s ⬆ 23.1kiB/s |
|
Can you try a different server? Your report looks similar to this one #2576 and it is the server issue. |
|
Thanks, I will try. Now I am using Hetzner Cloud . |
|
this memory issue seems still there with other Sever (digital ocean) 23.3kiB/s ⬆ 0.4kiB/s info: 2023-11-04 02:30:22 Acala Node 2023-11-04 02:34:19 [Relaychain] 💻 Operating system: linux 2023-11-04 02:34:19 [Parachain] 💻 Operating system: linux Could you please check the reason ? |
|
If you are not build an archive mode, I will suggest enable wrap sync
|
thanks I just tried. Let's see how it works docker run -it --privileged=true acala/acala-node:2.20.0 --base-path=/acala/data --chain=acala --name=aaaa --rpc-external --rpc-cors=all --rpc-port=9933 --sync=warp -- --sync=warp "-- --sync=warp" is part of the command, right? |
|
yes. the flags before double dash is for parachain and after is for relaychain. in this case, we want to enable warp sync for both relaychain and parachain and therefore two of it |
|
Hi, this command works. Thank you. But I met another problem. root@acala-2:~/kkc/acala# sudo docker run -it --privileged=true -v /mnt/volume_fra1_01:/acala/data acala/acala-node:2.20.0 --base-path=/acala/data --chain=acala --name=aaaa --ws-external --rpc-external --rpc-cors=all --ws-port=9944 --rpc-port=9933 --sync=warp -- --sync=warp tip: a similar argument exists: '--rpc-external' Usage: acala --name --rpc-external <--chain <CHAIN_SPEC>|--dev|--base-path |--log <LOG_PATTERN>...|--detailed-log-output|--disable-log-color|--enable-log-reloading|--tracing-targets |--tracing-receiver > [-- <RELAY_CHAIN_ARGS>...] For more information, try '--help'. Now I have a node running with sudo docker run -it --privileged=true -v /mnt/volume_fra1_01:/acala/data acala/acala-node:2.20.0 --base-path=/acala/data --chain=acala --name=aaaa --rpc-external --rpc-cors=all --rpc-port=9933 --sync=warp -- --sync=warp I tried ws://172.17.0.2:9944 to connect but it does not work. How do I connect to WS? THank you again. |
|
the http port and ws port are merged so you only need one specified with |
|
We also observe an OOM while syncing a node. 
More over, the acala's finalized block went to 0, and its catching up speed doesn't look good. We run it as follows: ExecStart=/opt/validators/acala/bin/2.24.0/acala \
--base-path=/opt/validators/acala/data \
--chain=acala \
--name=chorusone-rpc-1 \
--pruning=archive \
# workaround for substrate bug, see https://github.com/paritytech/substrate/issues/9697
--state-cache-size 0 \
--rpc-external \
--rpc-cors=all \
--rpc-port=9944 \
--rpc-max-connections=2000 \
--execution=wasm \
--prometheus-port 10140 \
--prometheus-external \
--listen-addr=/ip4/0.0.0.0/tcp/30333 \
--listen-addr=/ip4/0.0.0.0/tcp/30335 \
-- \
--chain=polkadot \
--prometheus-external \
--prometheus-port 9616We also run the |
|
@qezz it is expected the finalized block to be 0 while the relaychain part is syncing |
It synced from 0 to (current head) in a day, and then after another day dropped to 0. Is this expected?
Will the logs for the OOM loop work? I stopped the node for now, so it doesn't cause issues for collocated processes. |
Sorry, I'm confusing things here. I mean finalized height for Parachain, and that's what shown on the screenshot Edit: Relaychain's finalized height was indeed 0, and that's OK. It has started growing after ~2 days of syncing, and then it went OOM after several hours |
|
ok it is not expected for the parachain finalized height goes from a non zero value to zero
that will be useful |
|
Hi, I tried to enable wrap sync with: Error: Service(Other("Warp sync enabled, but no warp sync provider configured.")) how to solve this ? or is the sync provider configured on my side ? (this is not form acala node but from parallel, could please also give some clues) Many thanks |
|
@SixoneKui this is not related to the original issue and it is not even related to this repo so please avoid posting unrelated content in the issue next time. for this particular question, my guess is that warp sync is not enabled in the client but I did not check their code and can't really tell |
|
The logs are basically empty on the OOM loop no more logs after this ^, and it's already going OOM 
|
|
This issue is very unlikely caused by Acala specific code as we don't really have any non standard code compare to other parachains. I've made a new release branch prepare for 2.25.0 release: https://github.com/AcalaNetwork/Acala/tree/release-acala-2.25.0 The docker image can be found here: https://github.com/AcalaNetwork/Acala/pkgs/container/acala-node/210655705?tag=2.25.0 |
|
Please also run with |
|
I didn't manage to make it sync from genesis. The solution for me was to rsync a datadir from one node to another |
I tried run a node on a mounted volume using:
docker run -it -v /mnt/HC_Volume_37317682:/acala/data acala/acala-node:2.21.0 --base-path=/acala/data --chain=acala
the above is got from another question.
But I see acala-node requires a large RAM, sometimes over 32GB and then causes out of memory issue (My could server has only 32GB ).
Is this normal ?
Thanks a lot.
The text was updated successfully, but these errors were encountered: