feat: support for templating in table names #24
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This changeset introduces a simple templating language in the target
`table` configuration.
Previously, there were two options:
1. Unset Table Configuration (=default): The connector used the same QuestDB
table as the topic name from which the message originated. When the connector
was set to listen on multiple topics, it ingested into multiple QuestDB
tables.
2. Explicit Table Configuration: The connector ingested into the one configured
table, regardless of the topic from which the messages originated.
This change allows the use of templates in table configurations.
For example, setting table to `${topic}` will cause ingestion
into the table named after the topic from which the message
originated. This behavior mirrors the scenario where the table
configuration is not set.
However, it also supports more advanced scenarios, such as `${topic}_${key}``,
where key is a string representation of the message key.
Supported Placeholders:
1. ${topic}
2. ${key} - string representation of the message key or 'null'
More placeholders may be added in the future.
Caveats:
1. The key is intended for use with simple values and not with complex objects
such as structs or arrays.
2. Using ${key} can result in a large number of tables. QuestDB might require
tuning when handling thousands of tables.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This changeset introduces a simple templating language in the target
tableconfiguration.Previously, there were two options:
Unset Table Configuration (=default): The connector used the same QuestDB table as the topic name from which the message originated. When the connector was set to listen on multiple topics, it ingested into multiple QuestDB tables.
Explicit Table Configuration: The connector ingested into the one configured table, regardless of the topic from which the messages originated.
This change allows the use of templates in table configurations. For example, setting table to
${topic}will cause ingestion into the table named after the topic from which the message originated. This behavior mirrors the scenario where the table configuration is not set.However, it also supports more advanced scenarios, such as
${topic}_${key}, where key is a string representation of the message key.Supported Placeholders:
${topic}${key}- string representation of the message key or 'null'More placeholders may be added in the future.
Caveats:
${key}can result in a large number of tables. QuestDB might require tuning when handling thousands of tables.