From 23dafe891a5bd543f2d0263d1289874f2cbf4966 Mon Sep 17 00:00:00 2001 From: Gabriel Oliveira Date: Thu, 18 Jul 2024 14:04:12 -0300 Subject: [PATCH] chore: add shared client performance disclaimer --- lib/broadway_kafka/producer.ex | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/lib/broadway_kafka/producer.ex b/lib/broadway_kafka/producer.ex index a2709bb..5d19fd2 100644 --- a/lib/broadway_kafka/producer.ex +++ b/lib/broadway_kafka/producer.ex @@ -51,7 +51,8 @@ defmodule BroadwayKafka.Producer do * `:shared_client` - Optional. When false, it starts one client per producer. When true, it starts a single shared client across all producers (which may reduce - memory/resource usage). Default is `false`. + memory/resource usage). May cause severe performance degradation, see + ["Shared Client Performance"](#module-shared-client-performance) for details. Default is `false`. * `:group_config` - Optional. A list of options used to configure the group coordinator. See the ["Group config options"](#module-group-config-options) section below for a list of all available @@ -215,6 +216,19 @@ defmodule BroadwayKafka.Producer do * `[:broadway_kafka, :assignments_revoked, :start | :stop | :exception]` spans - these events are emitted in "span style" when receiving assignments revoked call from consumer group coordinator See `:telemetry.span/3`. + + ## Shared Client Performance + + Enable shared client may drastically decrease performance. This happens because mutiple producers may block + each other waiting for the client response since the connection is hidden inside a process it becomes a + bottleneck. + + This is more likely to be an issue if the producers on your pipeline are fetching message from multiple topics + and specially if there are very low traffic topics in the mix because of batch wait times. + + In summary to mitigate this you can split your topics between multiple pipelines, but notice that this will + increase the resource usage as well creating one new client/connection for each pipeline effectively diminishing + the shared_client resource usage gains. """ use GenStage