Issue: High Disk Space Usage and Ledger Ingestion Problems in Stellar Captive Core, Horizon after update to 22 version

[racconozaur]

Problem Description

After updating our Stellar Captive Core node, we noticed a significant increase in disk usage. Before the update, the node successfully ingested 22 million ledgers without issues. However, following the update, only 1 million ledgers are consuming approximately 2 TB of disk space.

We need help identifying the cause of this issue and suggestions for resolving or optimizing storage usage.

System Configuration

OS: Ubuntu 23.10 (GNU/Linux 6.5.0-44-generic x86_64)
Disk Space: 3 TB

PostgreSQL Configuration

Version: 15.7 (Ubuntu 15.7-0ubuntu0.23.10.1)

Stellar Core Configuration

Package Version: 22.0.0
Git Version: 1cd8b8dca9aeeca9ce45b129cd923992b32dc258
Ledger Protocol Version: 22
Pre-release Version: 0

rs-stellar-xdr:

Package Version: 22.0.0
Git Version: 715003372ea6380044b5a4a02907ff73e56ae9e7
Base XDR Git Version: 529d5176f24c73eeccfa5eba481d4e89c19b1181

Horizon Configuration

Version: 22.0.2
Git Version: 5df7099e675469f409ad1b8ef5bb2a8a19db7f00
Go Version: go1.22.10

Observations

1. No changes were made to PostgreSQL configurations during the update process.
2. Logs do not indicate errors, but disk utilization continues to grow at an abnormal rate.

[sreuland]

Hello, was this resolved as part of discussion on Horizon discord channel which seems to be reporting same issue and there was some further explanation on ledger ingestion sizing?

The disk space usage reported is referring to the Horizon Postgres db usage? There are a couple options to consider to mitigate horizon ingestion db size which may be worthwhile going forward if the application can accept the trade-offs. One recent additional option to reduce ingested db size is to use SKIP_TXMETA=true, it was introduced starting with Horizon 2.29.0 which just stops persisting the raw ledger transaction meta data(serialized xdr) into the horizon db, which accounts for a sizeable amount of db storage space in the history_transactions table, if the application doesn't use history_transactions.tx_meta direct from horizon db table or from the horizon API as result_meta_xdr then could be another option to consider.