Manual Compaction

Sometimes you might want to change data formats in existing MyRocks databases. Data format changes include changing compression settings (e.g. Zlib to ZSTD), block size (e.g. 16KB to 4KB), bloom filter settings (e.g. disabling to have bloom filters in the bottommost level), index formats (e.g. non partitioned index to partitioned index), and so on. By updating my.cnf and restarting mysqld instance, newly created SST files are using the new data formats. However, existing SST files continue to use old formats, and it will take a while to fully replace to use the new data formats.

Manual Compaction is one of convenient ways to forcing to rewrite the entire SST file of any column family. In MyRocks, you can trigger Manual Compaction like this.

mysql> set session rocksdb_manual_compaction_threads=16;
Query OK, 0 rows affected (0.00 sec)

mysql> set global rocksdb_compact_cf='cf_node_pk';
Query OK, 0 rows affected (25 min 13.80 sec)

"set global rocksdb_compact_cf='columnn_family_name'" is a command to trigger Manual Compaction for a specified column family. The command is blocked until the Manual Compaction ends. It may take a long time, depending on the column family and compaction throughput.

rocksdb_manual_compaction_threads is a session variable name to specify how many RocksDB threads to use for the Manual Compaction initiated from the session. By default, same as rocksdb_max_subcompactions (default is 1) thread is used.

"set global rocksdb_compact_cf='columnn_family_name'" can set only one column family at a time. To compact multiple column families, you need to execute multiple "set global rocksdb_compact_cf..." commands. In background, MyRocks serializes all Manual Compaction requests, and executes one by one, in FIFO order. In other words, MyRocks executes only one Manual Compaction at a time. You can't request more than max-manual-compactions pending+running compaction requests. rocksdb_manual_compaction_running is a status variable to tell if Manual Compaction is currently processed. rocksdb_manual_compaction_processed is a status variable to tell how many Manual Compaction requests are processed in total.

If you terminate sessions that are executing "set global rocksdb_compact_cf='columnn_family_name'", pending Manual Compaction requests are cancelled. A already running Manual Compaction is not cancelled.

If you issue normal MySQL shutdown request while running Manual Compaction, all of the pending requests and a running Manual Compaction are cancelled immediately.