Skip to content

Commit

Permalink
chore: add shared client performance disclaimer
Browse files Browse the repository at this point in the history
  • Loading branch information
oliveigah committed Jul 18, 2024
1 parent b2dd8c3 commit 23dafe8
Showing 1 changed file with 15 additions and 1 deletion.
16 changes: 15 additions & 1 deletion lib/broadway_kafka/producer.ex
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,8 @@ defmodule BroadwayKafka.Producer do
* `:shared_client` - Optional. When false, it starts one client per producer.
When true, it starts a single shared client across all producers (which may reduce
memory/resource usage). Default is `false`.
memory/resource usage). May cause severe performance degradation, see
["Shared Client Performance"](#module-shared-client-performance) for details. Default is `false`.
* `:group_config` - Optional. A list of options used to configure the group
coordinator. See the ["Group config options"](#module-group-config-options) section below for a list of all available
Expand Down Expand Up @@ -215,6 +216,19 @@ defmodule BroadwayKafka.Producer do
* `[:broadway_kafka, :assignments_revoked, :start | :stop | :exception]` spans -
these events are emitted in "span style" when receiving assignments revoked call from consumer group coordinator
See `:telemetry.span/3`.
## Shared Client Performance
Enable shared client may drastically decrease performance. This happens because mutiple producers may block
each other waiting for the client response since the connection is hidden inside a process it becomes a
bottleneck.
This is more likely to be an issue if the producers on your pipeline are fetching message from multiple topics
and specially if there are very low traffic topics in the mix because of batch wait times.
In summary to mitigate this you can split your topics between multiple pipelines, but notice that this will
increase the resource usage as well creating one new client/connection for each pipeline effectively diminishing
the shared_client resource usage gains.
"""

use GenStage
Expand Down

0 comments on commit 23dafe8

Please sign in to comment.