Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhance: add rw/ro streaming query node replica management #38677

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

chyezh
Copy link
Contributor

@chyezh chyezh commented Dec 24, 2024

issue: #38399

  • Embed the query node into streaming node to make delegator available at streaming node.
  • The embedded query node has a special server label QUERYNODE_STREAMING-EMBEDDED.
  • Change the balance strategy to make the channel assigned to streaming node as much as possible.

@sre-ci-robot sre-ci-robot added area/internal-api size/XL Denotes a PR that changes 500-999 lines. labels Dec 24, 2024
@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: chyezh
To complete the pull request process, please assign xiaofan-luan after the PR has been reviewed.
You can assign the PR to them by writing /assign @xiaofan-luan in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mergify mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Dec 24, 2024
Copy link
Contributor

mergify bot commented Dec 24, 2024

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@chyezh chyezh force-pushed the enhance_embed_querynode_in_streamingnode branch from c3ce6ad to 0c32ff6 Compare December 24, 2024 02:44
Copy link
Contributor

mergify bot commented Dec 24, 2024

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

1 similar comment
Copy link
Contributor

mergify bot commented Dec 24, 2024

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

Copy link

codecov bot commented Dec 24, 2024

Codecov Report

Attention: Patch coverage is 77.77778% with 108 lines in your changes missing coverage. Please review.

Project coverage is 81.15%. Comparing base (dd81a13) to head (b590b13).
Report is 3 commits behind head on master.

Files with missing lines Patch % Lines
...ernal/querycoordv2/balance/multi_target_balance.go 0.00% 34 Missing ⚠️
...al/querycoordv2/balance/rowcount_based_balancer.go 68.42% 7 Missing and 5 partials ⚠️
internal/querycoordv2/ops_services.go 37.50% 5 Missing and 5 partials ⚠️
...erycoordv2/balance/channel_level_score_balancer.go 0.00% 8 Missing and 1 partial ⚠️
...ternal/querycoordv2/meta/replica_manager_helper.go 82.35% 8 Missing and 1 partial ⚠️
internal/querycoordv2/server.go 12.50% 5 Missing and 2 partials ⚠️
internal/querycoordv2/meta/replica_manager.go 87.23% 4 Missing and 2 partials ⚠️
...nternal/querycoordv2/observers/replica_observer.go 90.56% 3 Missing and 2 partials ⚠️
internal/util/streamingutil/env.go 0.00% 5 Missing ⚠️
pkg/util/contextutil/context_util.go 55.55% 3 Missing and 1 partial ⚠️
... and 3 more
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #38677      +/-   ##
==========================================
+ Coverage   81.12%   81.15%   +0.02%     
==========================================
  Files        1385     1387       +2     
  Lines      195922   196322     +400     
==========================================
+ Hits       158939   159322     +383     
+ Misses      31396    31386      -10     
- Partials     5587     5614      +27     
Components Coverage Δ
Client 79.12% <ø> (ø)
Core 69.45% <ø> (ø)
Go 83.12% <77.77%> (+0.02%) ⬆️
Files with missing lines Coverage Δ
...ernal/querycoordv2/balance/score_based_balancer.go 98.76% <100.00%> (+0.05%) ⬆️
...dv2/balance/streaming_query_node_channel_helper.go 100.00% <100.00%> (ø)
internal/querycoordv2/checkers/channel_checker.go 87.35% <100.00%> (+0.22%) ⬆️
internal/querycoordv2/meta/replica.go 100.00% <100.00%> (ø)
internal/querycoordv2/session/node_manager.go 100.00% <100.00%> (ø)
internal/querycoordv2/utils/meta.go 74.64% <100.00%> (+3.42%) ⬆️
...al/streamingcoord/server/balancer/balancer_impl.go 79.90% <100.00%> (+0.69%) ⬆️
internal/streamingcoord/server/server.go 79.41% <100.00%> (+0.62%) ⬆️
internal/util/sessionutil/session_util.go 76.51% <ø> (ø)
...ode/server/flusher/flusherimpl/channel_lifetime.go 71.27% <0.00%> (ø)
... and 12 more

... and 30 files with indirect coverage changes

Copy link
Contributor

mergify bot commented Dec 24, 2024

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

@chyezh chyezh force-pushed the enhance_embed_querynode_in_streamingnode branch from 5d38958 to 781ff46 Compare December 26, 2024 13:36
Copy link
Contributor

mergify bot commented Dec 26, 2024

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

Copy link
Contributor

mergify bot commented Dec 26, 2024

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@chyezh chyezh force-pushed the enhance_embed_querynode_in_streamingnode branch from 781ff46 to 3f613f6 Compare December 27, 2024 16:11
Copy link
Contributor

mergify bot commented Dec 27, 2024

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

@chyezh chyezh force-pushed the enhance_embed_querynode_in_streamingnode branch 2 times, most recently from 9db9bcf to 3953e7c Compare December 28, 2024 11:47
Copy link
Contributor

mergify bot commented Dec 28, 2024

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@chyezh chyezh force-pushed the enhance_embed_querynode_in_streamingnode branch 2 times, most recently from 70a9f34 to 2544e08 Compare December 29, 2024 02:46
Copy link
Contributor

mergify bot commented Dec 29, 2024

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

1 similar comment
Copy link
Contributor

mergify bot commented Dec 29, 2024

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

Copy link
Contributor

mergify bot commented Dec 29, 2024

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@sre-ci-robot sre-ci-robot added size/XXL Denotes a PR that changes 1000+ lines. and removed size/XL Denotes a PR that changes 500-999 lines. labels Dec 29, 2024
Copy link
Contributor

mergify bot commented Dec 29, 2024

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

Copy link
Contributor

mergify bot commented Dec 29, 2024

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@chyezh chyezh force-pushed the enhance_embed_querynode_in_streamingnode branch 2 times, most recently from 1546f5a to e7cebb7 Compare December 29, 2024 12:26
Copy link
Contributor

mergify bot commented Dec 29, 2024

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

@chyezh chyezh force-pushed the enhance_embed_querynode_in_streamingnode branch from 2f5fb2a to b590b13 Compare December 30, 2024 06:19
Copy link
Contributor

mergify bot commented Dec 30, 2024

@chyezh go-sdk check failed, comment rerun go-sdk can trigger the job again.

@chyezh
Copy link
Contributor Author

chyezh commented Dec 30, 2024

rerun go-sdk

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/internal-api ci-passed dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement size/XXL Denotes a PR that changes 1000+ lines.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants