Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Insert is slow #37633

Closed
1 task done
bigsheeper opened this issue Nov 13, 2024 · 3 comments
Closed
1 task done

[Bug]: Insert is slow #37633

bigsheeper opened this issue Nov 13, 2024 · 3 comments
Assignees
@bigsheeper
Labels
kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. stale indicates no udpates for 30 days

Comments

@bigsheeper
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: master&2.4
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

When there are many collections and partitions, the AssignSegment operation becomes very slow, which in turn causes insertion to slow down significantly.
image
image
image
image

Expected Behavior

The AllocSegment function in SegmentManager is slow, likely due to lock contention.
app1Lw4ZGb

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response
@bigsheeper bigsheeper added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 13, 2024
@bigsheeper bigsheeper self-assigned this Nov 13, 2024
@bigsheeper
Copy link
Contributor Author

related to: #37630

@bigsheeper
Copy link
Contributor Author

We should reduce the lock granularity in SegmentManager.

This was referenced Nov 20, 2024
Merged
Merged
Merged
czs007 pushed a commit that referenced this issue Nov 21, 2024
@bigsheeper
enhance: [10kcp] Optimize segmentManager segments (#37884)
9e1ba07 
1. Use vchannel and partition indices for segments.
2. Replace coarse-grained mutex with concurrent map.

issue: #37633,
#37630

pr: #37836

Signed-off-by: bigsheeper <[email protected]>
@bigsheeper bigsheeper mentioned this issue Dec 2, 2024
Merged
czs007 pushed a commit that referenced this issue Dec 3, 2024
@bigsheeper
enhance: [10kcp] Update segment manger (#38153)
0c29d8f 
Use a channel level key lock for segments in segmentManager.

issue: #37633,
#37630

pr: #37836

---------

Signed-off-by: bigsheeper <[email protected]>
sre-ci-robot pushed a commit that referenced this issue Dec 6, 2024
@bigsheeper
enhance: [2.4] Reduce segmentManager lock granularity (#37869)
12cc500 
Use a channel level key lock for segments in segmentManager.

issue: #37633,
#37630

pr: #37836

---------

Signed-off-by: bigsheeper <[email protected]>
sre-ci-robot pushed a commit that referenced this issue Dec 17, 2024
@bigsheeper
enhance: Reduce segmentManager lock granularity (#37836)
d4dab3c 
Use a channel level key lock for segments in segmentManager.

issue: #37633,
#37630

---------

Signed-off-by: bigsheeper <[email protected]>
@stale Stale
Copy link

stale bot commented Dec 20, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label Dec 20, 2024
@stale stale bot closed this as completed Dec 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bigsheeper bigsheeper

Labels
kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. stale indicates no udpates for 30 days
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bigsheeper