Skip to content

Commit

Permalink
better readme
Browse files Browse the repository at this point in the history
  • Loading branch information
laskoviymishka committed Dec 20, 2024
1 parent 5ccbe28 commit 41bf402
Showing 1 changed file with 8 additions and 0 deletions.
8 changes: 8 additions & 0 deletions pkg/providers/clickhouse/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,14 @@ The core components of this implementation are:
- **Sink**: Responsible for pushing the actual data to the destination.
- **Batch**: Represents incoming data to be processed.

Key assumptions to support for exactly once in Clickhouse:

- Clickhouse is replicated, i.e. contains either Zookeeper or CHKeeper.
- `KeeperMap` table engine is enabled at target database.
- Source provide `PartID`-metadata, to split logical table into sequence of related changes
- Source provide `_offset`-column, each offset should be consistently growing.
- Source provide at-least-once delivery semantic.

The diagrams below explain the flow, structure, and interactions within the system.

---
Expand Down

0 comments on commit 41bf402

Please sign in to comment.