Skip to content

Commit

Permalink
Add info about default topics prefixes to the docs (#662)
Browse files Browse the repository at this point in the history
  • Loading branch information
daniil-quix authored Dec 2, 2024
1 parent 8b4db45 commit 8fb8249
Show file tree
Hide file tree
Showing 3 changed files with 16 additions and 5 deletions.
14 changes: 10 additions & 4 deletions docs/connectors/sources/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,19 @@ For good performance, each source runs in a subprocess. Quix Streams automatical

For multiplatform support, Quix Streams starts the source process using the [spawn](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods) approach. As a side effect, each Source instance must be pickleable. If a source needs to handle unpickleable objects, it's best to initialize those in the source subprocess (in the `BaseSource.start` or `Source.run` methods).

## Customize Topic Configuration
## Topics

Sources work by sending data to intermediate Kafka topics, which StreamingDataFrames then consume and process.

By default, each Source provides a default topic based on its configuration.
To customize the topic config, pass a new `Topic` object to the `app.dataframe()` method together with the Source instance.
By default, each Source provides a default topic based on its configuration by implementing the `default_topic()` method.

!!! warning "New in 3.4.0"

Since v3.4.0, the default topics names are always prefixed with `"source__"`.


To customize the topic name or configuration, pass a new `Topic` object to the `app.dataframe()` method together with the Source instance.


**Example:**

Expand Down Expand Up @@ -109,7 +116,6 @@ To customize the topic the Source will use, create a new `Topic` and pass it to

```python
from quixstreams import Application
from quixstreams.sources import CSVSource
from quixstreams.models.topics import TopicConfig

def main():
Expand Down
3 changes: 2 additions & 1 deletion quixstreams/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -676,7 +676,8 @@ def add_source(self, source: BaseSource, topic: Optional[Topic] = None) -> Topic
:param source: a :class:`quixstreams.sources.BaseSource` instance
:param topic: the :class:`quixstreams.models.Topic` instance the source will produce to
Default: the source default
Default - the topic generated by the `source.default_topic()` method.
Note: the names of default topics are prefixed with "source__".
"""
if not topic:
# Prefix the name of the default topic generated by the Source
Expand Down
4 changes: 4 additions & 0 deletions quixstreams/sources/base/source.py
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,8 @@ def default_topic(self) -> Topic:
This method is triggered when the topic is not provided to the source.
The source must return a default topic configuration.
Note: if the default topic is used, the Application will prefix its name with "source__".
"""


Expand Down Expand Up @@ -311,6 +313,8 @@ def default_topic(self) -> Topic:
Return a default topic matching the source name.
The default topic will not be used if the topic has already been provided to the source.
Note: if the default topic is used, the Application will prefix its name with "source__".
:return: `quixstreams.models.topics.Topic`
"""
return Topic(
Expand Down

0 comments on commit 8fb8249

Please sign in to comment.