Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pubsub: sink without order #895

Open
oguzhanunlu opened this issue May 17, 2024 · 0 comments
Open

pubsub: sink without order #895

oguzhanunlu opened this issue May 17, 2024 · 0 comments

Comments

@oguzhanunlu
Copy link
Member

Currently all enrich apps sink chunks in order but we don't need order for PubSub. We need order for Kinesis and Kafka.

val sinkAndCheckpoint: Pipe[F, List[(A, Result)], Unit] =
_.parEvalMap(env.streamsSettings.concurrency.sink)(chunk =>
for {
begin <- Clock[F].realTime
result <- sinkChunk(chunk.map(_._2), env).as(chunk.map(_._1))
end <- Clock[F].realTime
_ <- Logger[F].debug(s"Chunk of size ${chunk.size} sunk in ${end - begin}")
} yield result
)
.evalMap(env.checkpoint)

We should conditionally use parEvalMapUnordered for PubSub so that resource usage (CPU and mem) and throughput can improve. It is a matter of benchmarking to see how much improvement it'll bring.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant