Skip to content

Commit

Permalink
Merge branch 'main' into quick-start-fixes
Browse files Browse the repository at this point in the history
  • Loading branch information
sleto-it authored Nov 27, 2024
2 parents e9b126c + fada57d commit 593804c
Showing 1 changed file with 22 additions and 0 deletions.
22 changes: 22 additions & 0 deletions documentation/third-party-tools/kafka/questdb-kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -398,6 +398,28 @@ Note that deduplication requires designated timestamps extracted either from
message payload or Kafka message metadata. See the
[Designated timestamps](#designated-timestamps) section for more information.


#### Dead Letter Queue

When messages cannot be processed due to non-recoverable errors, such as invalid data formats or schema mismatches, the
connector can send these failed messages to a Dead Letter Queue (DLQ). This prevents the entire connector from stopping
when it encounters problematic messages. To enable this feature, configure the Dead Letter Queue in your Kafka Connect
worker configuration using the `errors.tolerance`, `errors.deadletterqueue.topic.name`, and
`errors.deadletterqueue.topic.replication.factor` properties. For example:

```properties
errors.tolerance=all
errors.deadletterqueue.topic.name=dlq-questdb
errors.deadletterqueue.topic.replication.factor=1
```

When configured, messages that fail to be processed will be sent to the specified DLQ topic along with error details,
allowing for later inspection and troubleshooting. This is particularly useful in production environments where data
quality might vary and you need to ensure continuous operation while investigating problematic messages.

See [Confluent article](https://developer.confluent.io/courses/kafka-connect/error-handling-and-dead-letter-queues/) about DLQ.


### Latency considerations

The connector waits for a batch of messages to accumulate before sending them to
Expand Down

0 comments on commit 593804c

Please sign in to comment.