From 2fb0b1616536ba2b77cb38abe1a61ab5f93f1515 Mon Sep 17 00:00:00 2001
From: goodroot <9484709+goodroot@users.noreply.github.com>
Date: Tue, 26 Nov 2024 08:57:03 -0800
Subject: [PATCH 1/5] multi-primary ingestion doc
src: http://localhost:3001/docs/operations/multi-primary-ingestion/
---
.../operations/multi-primary-ingestion.md | 99 +++++++++++++++++++
documentation/sidebars.js | 6 ++
2 files changed, 105 insertions(+)
create mode 100644 documentation/operations/multi-primary-ingestion.md
diff --git a/documentation/operations/multi-primary-ingestion.md b/documentation/operations/multi-primary-ingestion.md
new file mode 100644
index 00000000..b17b3eac
--- /dev/null
+++ b/documentation/operations/multi-primary-ingestion.md
@@ -0,0 +1,99 @@
+---
+title: Multi-primary ingestion
+sidebar_label: Multi-primary ingestion
+description:
+ Instructions and advice on performing multi-primary ingestion within QuestDB Enterprise.
+---
+
+QuestDB Enterprise's multi-primary ingestion enables **strong throughput** and **high availability** via concurrent writes across multiple primaries. With the addition of **automatic failover**, it provides reliable ingestion under most failure and zero-downtime update scenarios.
+
+Multi-primary ingestion applies multiple primary nodes to handle parallel ingestion operations. Each primary node can sustain up to **5 million rows per second** as per [Time Series Benchmark Suite (TSBS)](https://github.com/questdb/tsbs) benchmarks.
+
+This document explains multi-primary ingestion, and provides details on the use of FoundationDB for metadata coordination, a sequencer for conflict resolution, and failover mechanisms for added reliability.
+
+:::note
+
+Multi-primary ingestion is only available in QuestDB Enterprise.
+
+:::
+
+## How multi-primary ingestion works
+
+### The role of the sequencer
+
+QuestDB Enterprise provides write consistency across multiple primary nodes via a **sequencer**. The sequencer coordinates transactions by assigning **monotonic transaction numbers** to writes.
+
+This approach is already used by QuestDB, even on single-instance configuration, for managing writes via **WAL files** and has been extended to support multi-primary setups:
+
+```mermaid
+sequenceDiagram
+ participant Client
+ participant Sequencer
+ participant Table
+
+ Client->>Sequencer: Write Request (Data, TxID)
+ Sequencer->>Table: Check Current Sequence
+ alt TxID matches current sequence
+ note over Sequencer,Table: Transaction is valid
+ Sequencer->>Table: Write Data
+ Table-->>Sequencer: Write Success
+ Sequencer-->>Client: Acknowledge Success
+ else TxID does not match sequence
+ note over Sequencer: Transaction conflict detected
+ Sequencer-->>Client: Conflict Error (Retry with updated TxID)
+ end
+```
+
+Broken down, here is how it works:
+
+* **Transaction sequencing:** Each write operation is assigned a unique transaction ID. The sequencer ensures that this ID is always **incremental** and consistent for the target table. Transaction sequencing is managed **per table**, so each table has an independent sequence.
+
+* **Optimistic locking:** When a write request is received, the sequencer validates the transaction number. If the transaction number is lower than the current sequence for the table, the write is rejected with a conflict error.
+
+* **Error handling:** If a transaction conflict occurs, QuestDB Enterprise notifies the client with an error message. The client can retry the transaction with updated data.
+
+### Distributed sequencer with FoundationDB
+
+In a multi-primary configuration, the sequencer operates in a **distributed mode**. It does so with [FoundationDB](https://www.foundationdb.org/) as the backend for storing transaction metadata. This enables synchronization across primaries.
+
+- **Cluster-wide coordination:** The metadata about transaction IDs for each table is maintained in FoundationDB. This allows all primaries to verify and update sequence numbers without conflicts.
+
+- **Dynamic primary selection:** Clients can send writes directly to a specific primary node or to the cluster address. When writing to the cluster address, data is automatically routed to the most optimal primary node based on the current load and availability.
+
+- **Conflict resolution:** As in single-primary setups, transaction conflicts are handled by rejecting conflicting writes and informing the client to resend the data.
+
+With FoundationDB for metadata coordination, and a robust sequencer for conflict resolution, QuestDB Enterprise provides the resiliency your team needs to scale with peace of mind.
+
+## Automatic failover
+
+QuestDB Enterprise's multi-primary architecture also supports **automatic failover** to ensure high availability in case of primary node failures. This feature minimizes downtime and maintains ingestion continuity.
+
+### Failover workflow
+
+1. **Primary and replica registration:**
+ - All **primaries** and **replicas** register with the FoundationDB coordinator on startup.
+ - This provides the cluster with a complete view of the topology, including all active primaries and replicas.
+1. **Failure detection:**
+ - The cluster continuously monitors the health of primary nodes.
+ - If a primary becomes unresponsive due to hardware issues, a restart, or other failures, the cluster initiates a failover.
+1. **Replica promotion:**
+ - A suitable replica is automatically promoted to **primary** based on its synchronization status and availability.
+ - The new primary information is propagated to all connected clients using the official QuestDB Enterprise SDK.
+ - If the client is connected to the cluster address, writes are seamlessly redirected to the new primary.
+1. **Reintegration of the failed primary:**
+ - The failed primary undergoes a recovery process, catching up with the latest transactions using the **Write Ahead Log (WAL)** from the cluster.
+ - Once synchronized, the recovered primary is reinstated, and the promoted replica returns to its previous role as a replica.
+
+## Configuration and topology awareness
+
+When configuring a QuestDB Enterprise cluster with multi-primary ingestion:
+
+- **Coordinator setup:** Both primary nodes and replicas must be configured to communicate with the FoundationDB coordinator.
+
+- **Client awareness:** Clients using the official QuestDB Enterprise SDK receive the cluster topology on connection. This allows them to:
+ - Select a specific primary for writes.
+ - Send data to the cluster address for automatic routing.
+
+- **Seamless failover:** During failover events, clients are automatically informed of topology changes, ensuring uninterrupted ingestion.
+
+
diff --git a/documentation/sidebars.js b/documentation/sidebars.js
index c9cd4f6a..1b8bd727 100644
--- a/documentation/sidebars.js
+++ b/documentation/sidebars.js
@@ -317,6 +317,12 @@ module.exports = {
label: "Enterprise",
collapsed: false,
items: [
+ {
+ id: "operations/multi-primary-ingestion",
+ type: "doc",
+ label: "Multi-primary ingestion",
+ customProps: { tag: "Enterprise" },
+ },
{
id: "operations/openid-connect-oidc-integration",
type: "doc",
From d2a369374840408e89dec3d1b37300ebb2fb91fc Mon Sep 17 00:00:00 2001
From: goodroot <9484709+goodroot@users.noreply.github.com>
Date: Tue, 26 Nov 2024 10:52:19 -0800
Subject: [PATCH 2/5] polish, interlinking
---
documentation/concept/replication.md | 58 ++++++++-----------
.../operations/multi-primary-ingestion.md | 4 +-
2 files changed, 26 insertions(+), 36 deletions(-)
diff --git a/documentation/concept/replication.md b/documentation/concept/replication.md
index aed83547..1a2ebf2e 100644
--- a/documentation/concept/replication.md
+++ b/documentation/concept/replication.md
@@ -6,8 +6,6 @@ description:
roadmap, down to limitations.
---
-import Screenshot from "@theme/Screenshot"
-
QuestDB Enterprise offers primary-replica replication with eventual consistency.
This document will teach you about the architecture, some configuration, a
@@ -58,10 +56,19 @@ The architecture consists of:
file system
- Any number of **replica** instances
-
+```mermaid
+graph TD
+ primary[Primary]
+ objectStore[Object Store]
+ replica1[Replica 1]
+ replica2[Replica 2]
+ replicaN[Replica ..N]
+
+ primary --> objectStore
+ objectStore --> replica1
+ objectStore --> replica2
+ objectStore --> replicaN
+```
## Supported Object Stores
@@ -75,26 +82,14 @@ HDFS support is on our roadmap.
We also plan to support other object stores such as Google Cloud Storage:
-
-
-
+```
Something missing? Want to see it sooner? [Contact us](/enterprise/contact)!
@@ -122,21 +117,6 @@ examples.
src="images/docs/concepts/replication-streams.webp"
/>
-
-
In addition, the same object store can be re-used for multiple replication
clusters. To do so, provide a unique `DB_INSTANCE_NAME` value.
@@ -155,3 +135,11 @@ up to date manually and are often generally just inserted during the database
tables setup phase.
This limitation will be lifted in the future.
+
+## Multi-primary ingestion
+
+[QuestDB Enterprise](/enterprise/) supports multi-primary ingestion, where
+multiple primaries can write to the same database.
+
+See the [Multi-primary ingestion](/docs/operations/multi-primary-ingestion)
+page for more information.
diff --git a/documentation/operations/multi-primary-ingestion.md b/documentation/operations/multi-primary-ingestion.md
index b17b3eac..f9707596 100644
--- a/documentation/operations/multi-primary-ingestion.md
+++ b/documentation/operations/multi-primary-ingestion.md
@@ -13,7 +13,7 @@ This document explains multi-primary ingestion, and provides details on the use
:::note
-Multi-primary ingestion is only available in QuestDB Enterprise.
+Multi-primary ingestion is coming soon to QuestDB Enterprise.
:::
@@ -23,6 +23,8 @@ Multi-primary ingestion is only available in QuestDB Enterprise.
QuestDB Enterprise provides write consistency across multiple primary nodes via a **sequencer**. The sequencer coordinates transactions by assigning **monotonic transaction numbers** to writes.
+Each primary behaves as both a primary and a replica. It will write data into the [object store](/concept/replication/#supported-object-stores) and read data from the [object store](/concept/replication/#supported-object-stores) written by the other primaries to catch up with any changes.
+
This approach is already used by QuestDB, even on single-instance configuration, for managing writes via **WAL files** and has been extended to support multi-primary setups:
```mermaid
From e04a3c435f62485c43c46d3dd6f8c3d9bcff676a Mon Sep 17 00:00:00 2001
From: goodroot <9484709+goodroot@users.noreply.github.com>
Date: Tue, 26 Nov 2024 10:52:55 -0800
Subject: [PATCH 3/5] return import
---
documentation/concept/replication.md | 2 ++
1 file changed, 2 insertions(+)
diff --git a/documentation/concept/replication.md b/documentation/concept/replication.md
index 1a2ebf2e..e5accec8 100644
--- a/documentation/concept/replication.md
+++ b/documentation/concept/replication.md
@@ -6,6 +6,8 @@ description:
roadmap, down to limitations.
---
+import Screenshot from "@theme/Screenshot"
+
QuestDB Enterprise offers primary-replica replication with eventual consistency.
This document will teach you about the architecture, some configuration, a
From b50d17fe2e106b53e92d8e357c58f945c672e332 Mon Sep 17 00:00:00 2001
From: goodroot <9484709+goodroot@users.noreply.github.com>
Date: Tue, 26 Nov 2024 10:56:03 -0800
Subject: [PATCH 4/5] aligning replication and multi-prim ingest
---
documentation/operations/multi-primary-ingestion.md | 4 ++++
documentation/sidebars.js | 10 +++++-----
2 files changed, 9 insertions(+), 5 deletions(-)
diff --git a/documentation/operations/multi-primary-ingestion.md b/documentation/operations/multi-primary-ingestion.md
index f9707596..9e49fd33 100644
--- a/documentation/operations/multi-primary-ingestion.md
+++ b/documentation/operations/multi-primary-ingestion.md
@@ -98,4 +98,8 @@ When configuring a QuestDB Enterprise cluster with multi-primary ingestion:
- **Seamless failover:** During failover events, clients are automatically informed of topology changes, ensuring uninterrupted ingestion.
+## Replication
+QuestDB Enterprise's multi-primary ingestion is built alongside its primary-replica replication capabilities.
+
+See the [Replication](/docs/operations/replication/) page for more information.
diff --git a/documentation/sidebars.js b/documentation/sidebars.js
index 1b8bd727..6d9f134e 100644
--- a/documentation/sidebars.js
+++ b/documentation/sidebars.js
@@ -318,20 +318,20 @@ module.exports = {
collapsed: false,
items: [
{
- id: "operations/multi-primary-ingestion",
+ id: "operations/openid-connect-oidc-integration",
type: "doc",
- label: "Multi-primary ingestion",
+ label: "OpenID Connect (OIDC)",
customProps: { tag: "Enterprise" },
},
{
- id: "operations/openid-connect-oidc-integration",
type: "doc",
- label: "OpenID Connect (OIDC)",
+ id: "operations/replication",
customProps: { tag: "Enterprise" },
},
{
+ id: "operations/multi-primary-ingestion",
type: "doc",
- id: "operations/replication",
+ label: "Multi-primary ingestion",
customProps: { tag: "Enterprise" },
},
{
From 45ca281f302d0fccc5598a456eb35f369ec2e33a Mon Sep 17 00:00:00 2001
From: goodroot <9484709+goodroot@users.noreply.github.com>
Date: Tue, 26 Nov 2024 11:02:17 -0800
Subject: [PATCH 5/5] interlinking
---
documentation/concept/replication.md | 2 +-
documentation/operations/multi-primary-ingestion.md | 2 +-
documentation/operations/replication.md | 8 ++++++++
3 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/documentation/concept/replication.md b/documentation/concept/replication.md
index e5accec8..ef7a0c48 100644
--- a/documentation/concept/replication.md
+++ b/documentation/concept/replication.md
@@ -143,5 +143,5 @@ This limitation will be lifted in the future.
[QuestDB Enterprise](/enterprise/) supports multi-primary ingestion, where
multiple primaries can write to the same database.
-See the [Multi-primary ingestion](/docs/operations/multi-primary-ingestion)
+See the [Multi-primary ingestion](/operations/multi-primary-ingestion)
page for more information.
diff --git a/documentation/operations/multi-primary-ingestion.md b/documentation/operations/multi-primary-ingestion.md
index 9e49fd33..fd4e5c27 100644
--- a/documentation/operations/multi-primary-ingestion.md
+++ b/documentation/operations/multi-primary-ingestion.md
@@ -102,4 +102,4 @@ When configuring a QuestDB Enterprise cluster with multi-primary ingestion:
QuestDB Enterprise's multi-primary ingestion is built alongside its primary-replica replication capabilities.
-See the [Replication](/docs/operations/replication/) page for more information.
+See the [Replication](/operations/replication/) page for more information.
diff --git a/documentation/operations/replication.md b/documentation/operations/replication.md
index c452e033..4ca86ae2 100644
--- a/documentation/operations/replication.md
+++ b/documentation/operations/replication.md
@@ -330,3 +330,11 @@ daily ones due to the computational and IO demands of applying WAL files.
For systems with high daily data injection, daily snapshots are recommended.
Infrequent snapshots or long snapshot periods, such as 60 days with 30-day WAL
expiration, may prevent successful database restoration.
+
+## Multi-primary ingestion
+
+[QuestDB Enterprise](/enterprise/) supports multi-primary ingestion, where
+multiple primaries can write to the same database.
+
+See the [Multi-primary ingestion](/operations/multi-primary-ingestion) page for
+more information.