Skip to content

Commit

Permalink
WIP use_branch
Browse files Browse the repository at this point in the history
  • Loading branch information
@gwaramadze
gwaramadze committed Jul 29, 2024
1 parent a4ded95 commit 1803215
Show file tree
Hide file tree
Showing 9 changed files with 499 additions and 475 deletions.
30 changes: 20 additions & 10 deletions docs/api-reference/application.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
class Application()
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/app.py#L56)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/app.py#L56)

The main Application class.

Expand Down Expand Up @@ -63,6 +63,7 @@ def __init__(broker_address: Optional[Union[str, ConnectionConfig]] = None,
consumer_group: Optional[str] = None,
auto_offset_reset: AutoOffsetReset = "latest",
commit_interval: float = 5.0,
commit_every: int = 0,
consumer_extra_config: Optional[dict] = None,
producer_extra_config: Optional[dict] = None,
state_dir: str = "state",
Expand All @@ -83,7 +84,7 @@ def __init__(broker_address: Optional[Union[str, ConnectionConfig]] = None,
processing_guarantee: ProcessingGuarantee = "at-least-once")
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/app.py#L94)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/app.py#L94)


<br>
Expand All @@ -109,6 +110,15 @@ Default - "quixstreams-default" (set during init)
>***NOTE:*** Quix Applications will prefix it with the Quix workspace id.
- `commit_interval`: How often to commit the processed messages in seconds.
Default - 5.0.
- `commit_every`: Commit the checkpoint after processing N messages.
Use this parameter for more granular control of the commit schedule.
If the value is > 0, the application will commit the checkpoint after
processing the specified number of messages across all the assigned
partitions.
If the value is <= 0, only the `commit_interval` will be considered.
Default - 0.
>***NOTE:*** Only input offsets are counted, and the application
> may produce more results than the number of incoming messages.
- `auto_offset_reset`: Consumer `auto.offset.reset` setting
- `consumer_extra_config`: A dictionary with additional options that
will be passed to `confluent_kafka.Consumer` as is.
Expand Down Expand Up @@ -186,7 +196,7 @@ def Quix(
) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/app.py#L330)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/app.py#L342)

>***NOTE:*** DEPRECATED: use Application with `quix_sdk_token` argument instead.

Expand Down Expand Up @@ -297,7 +307,7 @@ def topic(name: str,
timestamp_extractor: Optional[TimestampExtractor] = None) -> Topic
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/app.py#L471)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/app.py#L483)

Create a topic definition.

Expand Down Expand Up @@ -378,7 +388,7 @@ topic = app.topic("input-topic", timestamp_extractor=custom_ts_extractor)
def dataframe(topic: Topic) -> StreamingDataFrame
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/app.py#L551)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/app.py#L563)

A simple helper method that generates a `StreamingDataFrame`, which is used

Expand Down Expand Up @@ -428,7 +438,7 @@ to be used as an input topic.
def stop(fail: bool = False)
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/app.py#L590)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/app.py#L602)

Stop the internal poll loop and the message processing.

Expand All @@ -455,7 +465,7 @@ to unhandled exception, and it shouldn't commit the current checkpoint.
def get_producer() -> Producer
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/app.py#L613)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/app.py#L625)

Create and return a pre-configured Producer instance.
The Producer is initialized with params passed to Application.
Expand Down Expand Up @@ -490,7 +500,7 @@ with app.get_producer() as producer:
def get_consumer(auto_commit_enable: bool = True) -> Consumer
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/app.py#L643)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/app.py#L655)

Create and return a pre-configured Consumer instance.
The Consumer is initialized with params passed to Application.
Expand Down Expand Up @@ -539,7 +549,7 @@ with app.get_consumer() as consumer:
def clear_state()
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/app.py#L690)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/app.py#L702)

Clear the state of the application.

Expand All @@ -553,7 +563,7 @@ Clear the state of the application.
def run(dataframe: StreamingDataFrame)
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/app.py#L696)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/app.py#L708)

Start processing data from Kafka using provided `StreamingDataFrame`

Expand Down
4 changes: 2 additions & 2 deletions docs/api-reference/context.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
def set_message_context(context: Optional[MessageContext])
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/context.py#L20)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/context.py#L20)

Set a MessageContext for the current message in the given `contextvars.Context`

Expand Down Expand Up @@ -55,7 +55,7 @@ sdf = sdf.update(lambda value: alter_context(value))
def message_context() -> MessageContext
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/context.py#L51)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/context.py#L51)

Get a MessageContext for the current message, which houses most of the message

Expand Down
56 changes: 28 additions & 28 deletions docs/api-reference/dataframe.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
class StreamingDataFrame(BaseStreaming)
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/dataframe.py#L63)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/dataframe.py#L63)

`StreamingDataFrame` is the main object you will use for ETL work.

Expand Down Expand Up @@ -81,7 +81,7 @@ def apply(func: Union[
metadata: bool = False) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/dataframe.py#L178)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/dataframe.py#L178)

Apply a function to transform the value and return a new value.

Expand Down Expand Up @@ -139,7 +139,7 @@ def update(func: Union[
metadata: bool = False) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/dataframe.py#L267)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/dataframe.py#L267)

Apply a function to mutate value in-place or to perform a side effect

Expand Down Expand Up @@ -207,7 +207,7 @@ def filter(func: Union[
metadata: bool = False) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/dataframe.py#L359)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/dataframe.py#L359)

Filter value using provided function.

Expand Down Expand Up @@ -259,7 +259,7 @@ def group_by(key: Union[str, Callable[[Any], Any]],
key_serializer: Optional[SerializerType] = "json") -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/dataframe.py#L445)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/dataframe.py#L445)

"Groups" messages by re-keying them via the provided group_by operation

Expand Down Expand Up @@ -324,7 +324,7 @@ a clone with this operation added (assign to keep its effect).
def contains(key: str) -> StreamingSeries
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/dataframe.py#L523)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/dataframe.py#L523)

Check if the key is present in the Row value.

Expand Down Expand Up @@ -363,7 +363,7 @@ or False otherwise.
def to_topic(topic: Topic, key: Optional[Callable[[Any], Any]] = None) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/dataframe.py#L548)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/dataframe.py#L548)

Produce current value to a topic. You can optionally specify a new key.

Expand Down Expand Up @@ -416,7 +416,7 @@ the updated StreamingDataFrame instance (reassignment NOT required).
def set_timestamp(func: Callable[[Any, Any, int, Any], int]) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/dataframe.py#L593)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/dataframe.py#L593)

Set a new timestamp based on the current message value and its metadata.

Expand Down Expand Up @@ -469,7 +469,7 @@ def set_headers(
) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/dataframe.py#L634)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/dataframe.py#L634)

Set new message headers based on the current message value and metadata.

Expand Down Expand Up @@ -518,7 +518,7 @@ a new StreamingDataFrame instance
def print(pretty: bool = True, metadata: bool = False) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/dataframe.py#L685)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/dataframe.py#L685)

Print out the current message value (and optionally, the message metadata) to

Expand Down Expand Up @@ -574,7 +574,7 @@ def compose(
) -> Dict[str, VoidExecutor]
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/dataframe.py#L727)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/dataframe.py#L727)

Compose all functions of this StreamingDataFrame into one big closure.

Expand Down Expand Up @@ -628,7 +628,7 @@ def test(value: Any,
topic: Optional[Topic] = None) -> List[Any]
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/dataframe.py#L764)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/dataframe.py#L764)

A shorthand to test `StreamingDataFrame` with provided value

Expand Down Expand Up @@ -665,7 +665,7 @@ def tumbling_window(duration_ms: Union[int, timedelta],
name: Optional[str] = None) -> TumblingWindowDefinition
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/dataframe.py#L801)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/dataframe.py#L801)

Create a tumbling window transformation on this StreamingDataFrame.

Expand Down Expand Up @@ -751,7 +751,7 @@ def hopping_window(duration_ms: Union[int, timedelta],
name: Optional[str] = None) -> HoppingWindowDefinition
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/dataframe.py#L877)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/dataframe.py#L877)

Create a hopping window transformation on this StreamingDataFrame.

Expand Down Expand Up @@ -843,7 +843,7 @@ sdf = (
def drop(columns: Union[str, List[str]]) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/dataframe.py#L969)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/dataframe.py#L969)

Drop column(s) from the message value (value must support `del`, like a dict).

Expand Down Expand Up @@ -887,7 +887,7 @@ a new StreamingDataFrame instance
class StreamingSeries(BaseStreaming)
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/series.py#L47)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/series.py#L47)

`StreamingSeries` are typically generated by `StreamingDataframes` when getting
elements from, or performing certain operations on, a `StreamingDataframe`,
Expand Down Expand Up @@ -953,7 +953,7 @@ sdf = sdf[["column_a"] & (sdf["new_sum_field"] >= 10)]
def from_apply_callback(cls, func: ApplyWithMetadataCallback) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/series.py#L107)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/series.py#L107)

Create a StreamingSeries from a function.

Expand Down Expand Up @@ -981,7 +981,7 @@ instance of `StreamingSeries`
def apply(func: ApplyCallback) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/series.py#L121)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/series.py#L121)

Add a callable to the execution list for this series.

Expand Down Expand Up @@ -1033,7 +1033,7 @@ a new `StreamingSeries` with the new callable added
def compose_returning() -> ReturningExecutor
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/series.py#L155)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/series.py#L155)

Compose a list of functions from this StreamingSeries and its parents into one

Expand Down Expand Up @@ -1064,7 +1064,7 @@ def compose(
None]] = None) -> VoidExecutor
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/series.py#L170)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/series.py#L170)

Compose all functions of this StreamingSeries into one big closure.

Expand Down Expand Up @@ -1122,7 +1122,7 @@ def test(value: Any,
ctx: Optional[MessageContext] = None) -> Any
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/series.py#L214)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/series.py#L214)

A shorthand to test `StreamingSeries` with provided value

Expand Down Expand Up @@ -1154,7 +1154,7 @@ result of `StreamingSeries`
def isin(other: Container) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/series.py#L269)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/series.py#L269)

Check if series value is in "other".

Expand Down Expand Up @@ -1199,7 +1199,7 @@ new StreamingSeries
def contains(other: Union[Self, object]) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/series.py#L296)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/series.py#L296)

Check if series value contains "other"

Expand Down Expand Up @@ -1244,7 +1244,7 @@ new StreamingSeries
def is_(other: Union[Self, object]) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/series.py#L321)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/series.py#L321)

Check if series value refers to the same object as `other`

Expand Down Expand Up @@ -1286,7 +1286,7 @@ new StreamingSeries
def isnot(other: Union[Self, object]) -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/series.py#L344)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/series.py#L344)

Check if series value does not refer to the same object as `other`

Expand Down Expand Up @@ -1329,7 +1329,7 @@ new StreamingSeries
def isnull() -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/series.py#L368)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/series.py#L368)

Check if series value is None.

Expand Down Expand Up @@ -1366,7 +1366,7 @@ new StreamingSeries
def notnull() -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/series.py#L391)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/series.py#L391)

Check if series value is not None.

Expand Down Expand Up @@ -1403,7 +1403,7 @@ new StreamingSeries
def abs() -> Self
```

[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/78b9db67a528a5423f1795cb3b5747d0f09a8768/quixstreams/dataframe/series.py#L414)
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/autogenerate-docs/quixstreams/dataframe/series.py#L414)

Get absolute value of the series value.

Expand Down
Loading

0 comments on commit 1803215

Please sign in to comment.