merk

High-performance Merkle key/value store

Merk is a crypto key/value store - more specifically, it's a Merkle AVL tree built on top of RocksDB (Facebook's fork of LevelDB).

Its priorities are performance and reliability. While Merk was designed to be the state database for blockchains, it can also be used anywhere an auditable key/value store is needed.

FEATURES:

• Fast reads/writes - Reads have no overhead compared to a normal RocksDB store, and writes are optimized for batch operations (e.g. blocks in a blockchain).
• Fast proof generation - Since Merk implements an AVL tree rather than a trie, it is very efficient to create and verify proofs for ranges of keys.
• Concurrency - Unlike most other Merkle stores, all operations utilize all available cores - giving huge performance gains and allowing nodes to scale along with Moore's Law.
• Replication - The tree is optimized to efficiently build proofs of large chunks, allowing for nodes to download the entire state (e.g. "state syncing").
• Checkpointing - Merk can create checkpoints on disk (an immutable view of the entire store at a certain point in time) without blocking, so there are no delays in availability or liveness.
• Web-friendly - Being written in Rust means it is easy to run the proof-verification code in browsers with WebAssembly, allowing for light-clients that can verify data for themselves.
• Fits any Profile - Performant on RAM-constrained Raspberry Pi's and beefy validator rigs alike.

Usage

Install:

cargo add merk

Example:

extern crate merk;
use merk::*;

// load or create a Merk store at the given path
let mut merk = Merk::open("./merk.db").unwrap();

// apply some operations
let batch = [
    (b"key", Op::Put(b"value")),
    (b"key2", Op::Put(b"value2")),
    (b"key3", Op::Put(b"value3")),
    (b"key4", Op::Delete)
];
merk.apply(&batch).unwrap();

Status

Merk is being used in the Nomic Bitcoin Sidechain.

The codebase has not been audited but has been throroughly tested and proves to be stable.

Benchmarks

Benchmarks are measured on a 1M node tree, each node having a key length of 16 bytes and value length of 40 bytes. All tests are single-threaded (not counting RocksDB background threads).

You can test these yourself by running cargo bench.

2017 Macbook Pro

(Using 1 Merk thread and 4 RocksDB compaction threads)

Pruned (no state kept in memory)

RAM usage: ~20MB average, ~26MB max

Test	Ops per second
Random inserts	23,000
Random updates	32,000
Random deletes	26,000
Random reads	210,000
Random proof generation	133,000

Cached (all state kept in memory)

RAM usage: ~400MB average, ~1.1GB max

Test	Ops per second
Random inserts	58,000
Random updates	81,000
Random deletes	72,000
Random reads	1,565,000
Random proof generation	311,000

i9-9900K Desktop

(Using 1 Merk thread and 16 RocksDB compaction threads)

Pruned (no state kept in memory)

RAM usage: ~20MB average, ~26MB max

Test	Ops per second
Random inserts	40,000
Random updates	55,000
Random deletes	45,000
Random reads	383,000
Random proof generation	249,000

Cached (all state kept in memory)

RAM usage: ~400MB average, ~1.1GB max

Test	Ops per second
Random inserts	93,000
Random updates	123,000
Random deletes	111,000
Random reads	2,370,000
Random proof generation	497,000

Algorithm Details

The algorithms are based on AVL, but optimized for batches of operations and random fetches from the backing store. Read about the algorithms here: https://github.com/nomic-io/merk/blob/develop/docs/algorithms.md