Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Pool limits increase latency from milliseconds to minutes in Kafka #38263

Open
1 task done
jaime0815 opened this issue Dec 6, 2024 · 1 comment
Open
1 task done

[Bug]: Pool limits increase latency from milliseconds to minutes in Kafka #38263

jaime0815 opened this issue Dec 6, 2024 · 1 comment
Assignees
@jaime0815
Labels
kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.

Comments

@jaime0815
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: master 
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

the latency behavior with pool limits:
image

the latency behavior without pool limits:
image

Encountering issues is easier in standalone deployments with limited resources.

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response
@jaime0815 jaime0815 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 6, 2024
This was referenced Dec 6, 2024
Merged
Closed
Closed
Closed
sre-ci-robot pushed a commit that referenced this issue Dec 6, 2024
@jaime0815
enhance: Remove pool from produer and consumer in kafka client (#38264)
edabfa8 
issue: #38263

Revert "fix: Move init kafka pool into once (#37786)"

Revert "enhance: Use pool to limit kafka cgo thread number (#37744)"

Signed-off-by: jaime <[email protected]>
@jaime0815 jaime0815 assigned jaime0815 and unassigned yanliang567 Dec 6, 2024
@xiaofan-luan
Copy link
Collaborator

@jaime0815
can we increate the pool?
how many threads do we have if we don't introduce a pool?

Does kafka has an internal pool?

@jaime0815 jaime0815 reopened this Dec 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jaime0815 jaime0815

Labels
kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jaime0815 @yanliang567 @xiaofan-luan