Skip to content

Commit

Permalink
Update descriptions of modes and ttl deletes option for LDR
Browse files Browse the repository at this point in the history
  • Loading branch information
kathancox committed Dec 13, 2024
1 parent cc608ab commit 6fa0eff
Show file tree
Hide file tree
Showing 5 changed files with 11 additions and 13 deletions.
1 change: 1 addition & 0 deletions src/current/_includes/v24.3/ldr/immediate-description.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Attempts to replicate the changed row directly into the destination table, without re-running constraint validations. It does not support writing into tables with [foreign key]({% link {{ page.version.version }}/foreign-key.md %}) constraints.
1 change: 1 addition & 0 deletions src/current/_includes/v24.3/ldr/validated-description.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Attempts to apply the write in a similar way to a user-run query, which would re-run all constraint validations relevant to the destination table(s). If the change violates foreign key dependencies, unique constraints, or other constraints, the row will be put in the [dead letter queue (DLQ)]({% link {{ page.version.version }}/manage-logical-data-replication.md %}#dead-letter-queue-dlq) instead. Like the [SQL layer]({% link {{ page.version.version }}/architecture/sql-layer.md %}), `validated` mode does not recognize deletion tombstones. As a result, an update to the same key from cluster A will successfully apply on cluster B, even if that key was deleted on cluster B before the LDR job streamed the cluster A update to the key.
10 changes: 5 additions & 5 deletions src/current/v24.3/create-logical-replication-stream.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,16 +48,16 @@ Parameter | Description
Option | Description
-------+------------
`cursor` | Emits any changes after the specified timestamp. LDR will not perform an initial backfill with the `cursor` option, it will stream any changes after the specified timestamp. The LDR job will encounter an error if you specify a `cursor` timestamp that is before the configured [garbage collection]({% link {{ page.version.version }}/architecture/storage-layer.md %}#garbage-collection) window for that table. **Warning:** Apply the `cursor` option carefully to LDR streams. Using a timestamp in error could cause data loss.
`discard` | Ignore [TTL deletes]({% link {{ page.version.version }}/row-level-ttl.md %}) in an LDR stream. Use `discard = ttl-deletes`. **Note**: To ignore row-level TTL deletes in an LDR stream, it is necessary to set the [`ttl_disable_changefeed_replication`]({% link {{ page.version.version }}/row-level-ttl.md %}#ttl-storage-parameters) storage parameter on the source table. Refer to the [Ignore row-level TTL deletes](#ignore-row-level-ttl-deletes) example.
<a id="discard-ttl-deletes-option"></a>`discard` | (**Unidirectional LDR only**) Ignore [TTL deletes]({% link {{ page.version.version }}/row-level-ttl.md %}) in an LDR stream with `discard = ttl-deletes`. **Note**: To ignore row-level TTL deletes in an LDR stream, it is necessary to set the [`ttl_disable_changefeed_replication`]({% link {{ page.version.version }}/row-level-ttl.md %}#ttl-storage-parameters) storage parameter on the source table. Refer to the [Ignore row-level TTL deletes](#ignore-row-level-ttl-deletes) example.
`label` | Tracks LDR metrics at the job level. Add a user-specified string with `label`. Refer to [Metrics labels]({% link {{ page.version.version }}/logical-data-replication-monitoring.md %}#metrics-labels).
`mode` | Determines how LDR replicates the data to the destination cluster. Possible values: `immediate`, `validated`. For more details refer to [LDR modes](#ldr-modes).

## LDR modes

_Modes_ determine how LDR replicates the data to the destination cluster. There are two modes:

- `immediate` (default): Attempts to replicate the changed row directly into the destination table, without re-running constraint validations. It does not support writing into tables with [foreign key]({% link {{ page.version.version }}/foreign-key.md %}) constraints.
- `validated`: Attempts to apply the write in a similar way to a user-run query, which would re-run all constraint validations or triggers relevant to the destination table(s). It will potentially reject the change if it fails rather than writing it, which will cause the row change to enter the DLQ instead.
- `immediate` (default): {% include {{ page.version.version }}/ldr/immediate-description.md %}
- `validated`: {% include {{ page.version.version }}/ldr/validated-description.md %}

## Bidirectional LDR

Expand Down Expand Up @@ -89,14 +89,14 @@ CREATE LOGICAL REPLICATION STREAM FROM TABLES ({database.public.table_name},{dat

### Ignore row-level TTL deletes

If you would like to ignore [row-level TTL]({% link {{ page.version.version }}/row-level-ttl.md %}) deletes in the LDR stream, set the [`ttl_disable_changefeed_replication` storage parameter]({% link {{ page.version.version }}/row-level-ttl.md %}#ttl-storage-parameters) on the table. On the **source** cluster, alter the table to set the table storage parameter:
If you would like to ignore [row-level TTL]({% link {{ page.version.version }}/row-level-ttl.md %}) deletes in a **unidirectional** LDR stream, set the [`ttl_disable_changefeed_replication` storage parameter]({% link {{ page.version.version }}/row-level-ttl.md %}#ttl-storage-parameters) on the table. On the **source** cluster, alter the table to set the table storage parameter:

{% include_cached copy-clipboard.html %}
~~~ sql
ALTER TABLE {table_name} SET (ttl_disable_changefeed_replication = 'true');
~~~

When you start LDR on the **destination cluster**, include the `discard = ttl-deletes` option in the statement:
When you start LDR on the **destination cluster**, include the [`discard = ttl-deletes` option](#discard-ttl-deletes-option) in the statement:

{% include_cached copy-clipboard.html %}
~~~ sql
Expand Down
4 changes: 2 additions & 2 deletions src/current/v24.3/manage-logical-data-replication.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,11 @@ Once you have **logical data replication (LDR)** running, you will need to track

## Conflict resolution

Conflict resolution in LDR differs depending on whether the conflict occurs at the [KV]({% link {{ page.version.version }}/architecture/storage-layer.md %}) level or the [SQL]({% link {{ page.version.version }}/architecture/sql-layer.md %}) level.
In LDR, conflicts are detected at both the [KV]({% link {{ page.version.version }}/architecture/storage-layer.md %}) and the [SQL]({% link {{ page.version.version }}/architecture/sql-layer.md %}) level, which determines how LDR resolves the conflict.

### KV level conflicts

LDR uses _last write wins (LWW)_ conflict resolution based on the [MVCC timestamp]({% link {{ page.version.version }}/architecture/storage-layer.md %}#mvcc) of the replicating write. LDR will resolve conflicts by inserting the row with the latest MVCC timestamp.
LDR uses _last write wins (LWW)_ conflict resolution based on the [MVCC timestamp]({% link {{ page.version.version }}/architecture/storage-layer.md %}#mvcc) of the replicating write. LDR will resolve conflicts by inserting the row with the latest MVCC timestamp. Conflicts at the KV level are detected in both `immediate` and `validated` mode.

Conflicts at the KV level are detected when there is either:

Expand Down
8 changes: 2 additions & 6 deletions src/current/v24.3/set-up-logical-data-replication.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,12 +119,8 @@ In this step, you'll start the LDR job from the destination cluster. You can rep

_Modes_ determine how LDR replicates the data to the destination cluster. There are two modes:

- `immediate` (default): Attempts to replicate the changed row directly into the destination table, without re-running constraint validations. It does not support writing into tables with [foreign key]({% link {{ page.version.version }}/foreign-key.md %}) constraints.
- `validated`: Attempts to apply the write in a similar way to a user-run query, which would re-run all constraint validations relevant to the destination table(s). If the change violates foreign key dependencies, unique constraints, or other constraints, the row will be put in the [dead letter queue (DLQ)]({% link {{ page.version.version }}/manage-logical-data-replication.md %}#dead-letter-queue-dlq) instead.

{{site.data.alerts.callout_info}}
If you would like to ignore TTL deletes in LDR, you can use the `discard = ttl-deletes` option in the `CREATE LOGICAL REPLICATION STREAM` statement. For an example, refer to [Ignore row-level TTL deletes]({% link {{ page.version.version }}/create-logical-replication-stream.md %}#ignore-row-level-ttl-deletes).
{{site.data.alerts.end}}
- `immediate` (default): {% include {{ page.version.version }}/ldr/immediate-description.md %}
- `validated`: {% include {{ page.version.version }}/ldr/validated-description.md %}

1. From the **destination** cluster, start LDR. Use the fully qualified table name for the source and destination tables:

Expand Down

0 comments on commit 6fa0eff

Please sign in to comment.