enhance: Read metadata concurrently to accelerate recovery #38403
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: bigsheeper The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
mergify
bot
added
dco-passed
Signed-off-by: bigsheeper <[email protected]>
bigsheeper
force-pushed
the
2412-speedup-list
branch
from
1604c01 to
aaa647a
Compare
…38404) Read metadata such as segments, binlogs, and partitions concurrently at the collection level. issue: #37630 pr: #38403 Signed-off-by: bigsheeper <[email protected]>
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #38403 +/- ##
==========================================
- Coverage 81.13% 81.10% -0.04%
==========================================
Files 1385 1385
Lines 196070 196185 +115
==========================================
+ Hits 159083 159108 +25
- Misses 31397 31481 +84
- Partials 5590 5596 +6
|
…2-speedup-list
Signed-off-by: bigsheeper <[email protected]>
|
@bigsheeper go-sdk check failed, comment |
|
@bigsheeper E2e jenkins job failed, comment |
|
@bigsheeper go-sdk check failed, comment |
|
@bigsheeper cpp-unit-test check failed, comment |
Signed-off-by: bigsheeper <[email protected]>
Signed-off-by: bigsheeper <[email protected]>
|
@bigsheeper go-sdk check failed, comment |
|
rerun go-sdk |
|
@bigsheeper go-sdk check failed, comment |
|
rerun go-sdk |
internal/datacoord/meta.go
Outdated
| record := timerecord.NewTimeRecorder("datacoord") | ||
| segments, err := m.catalog.ListSegments(m.ctx) | ||
|
|
||
| resp, err := broker.ShowCollectionsInternal(m.ctx) |
There was a problem hiding this comment.
if ShowCollectionsInternal is not supported, fallback to the old way?
There was a problem hiding this comment.
Adding fallback logic would require retaining many outdated metastore interfaces. Perhaps we could instead introduce a retry mechanism to wait for the new rootcoord to come online.
|
@bigsheeper go-sdk check failed, comment |
…2-speedup-list
Signed-off-by: bigsheeper <[email protected]>
Signed-off-by: bigsheeper <[email protected]>
|
@bigsheeper go-sdk check failed, comment |
Signed-off-by: bigsheeper <[email protected]>
|
@bigsheeper E2e jenkins job failed, comment |
Read metadata such as segments, binlogs, and partitions concurrently at the collection level.
issue: #37630