Skip to content

Commit

Permalink
Merge pull request #2792 from ClickHouse/ks/update_kinesis_virtual_co…
Browse files Browse the repository at this point in the history
…lumns

Add _sequence_number as supported Kinesis virtual column
  • Loading branch information
kschlarman authored Nov 14, 2024
2 parents 24410d1 + 06e6f0f commit d74d51b
Showing 1 changed file with 7 additions and 6 deletions.
13 changes: 7 additions & 6 deletions docs/en/integrations/data-ingestion/clickpipes/kinesis.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,12 +96,13 @@ The following ClickHouse data types are currently supported in ClickPipes:

The following virtual columns are supported for Kinesis stream. When creating a new destination table virtual columns can be added by using the `Add Column` button.

| Name | Description | Recommended Data Type |
|--------------|---------------------------------------------------------------|-----------------------|
| _key | Kinesis Partition Key | String |
| _timestamp | Kinesis Approximate Arrival Timestamp (millisecond precision) | DateTime64(3) |
| _stream | Kafka Stream Name | String |
| _raw_message | Full Kinesis Message | String |
| Name | Description | Recommended Data Type |
|------------------|---------------------------------------------------------------|-----------------------|
| _key | Kinesis Partition Key | String |
| _timestamp | Kinesis Approximate Arrival Timestamp (millisecond precision) | DateTime64(3) |
| _stream | Kinesis Stream Name | String |
| _sequence_number | Kinesis Sequence Number | String |
| _raw_message | Full Kinesis Message | String |

The _raw_message field can be used in cases where only full Kinesis JSON record is required (such as using ClickHouse [`JsonExtract*`](https://clickhouse.com/docs/en/sql-reference/functions/json-functions#jsonextract-functions) functions to populate a downstream materialized
view). For such pipes, it may improve ClickPipes performance to delete all the "non-virtual" columns.
Expand Down

0 comments on commit d74d51b

Please sign in to comment.