[Feature] (batch write part3) Implement batch write on BE #52556
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
banmoy
force-pushed
the
be_batch
branch
2 times, most recently
from
c8b1718 to
906086d
Compare
banmoy
force-pushed
the
be_batch
branch
15 times, most recently
from
74e915b to
8fe9a95
Compare
Signed-off-by: PengFei Li <[email protected]>
Signed-off-by: PengFei Li <[email protected]>
Signed-off-by: PengFei Li <[email protected]>
Signed-off-by: PengFei Li <[email protected]>
Signed-off-by: PengFei Li <[email protected]>
Signed-off-by: PengFei Li <[email protected]>
Signed-off-by: PengFei Li <[email protected]>
Signed-off-by: PengFei Li <[email protected]>
Signed-off-by: PengFei Li <[email protected]>
Signed-off-by: PengFei Li <[email protected]>
Signed-off-by: PengFei Li <[email protected]>
Signed-off-by: PengFei Li <[email protected]>
tracymacding
approved these changes
Nov 12, 2024
Signed-off-by: PengFei Li <[email protected]>
Signed-off-by: PengFei Li <[email protected]>
wyb
reviewed
Nov 14, 2024
Signed-off-by: PengFei Li <[email protected]>
|
trueeyu
approved these changes
Nov 14, 2024
wyb
approved these changes
Nov 14, 2024
[Java-Extensions Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[BE Incremental Coverage Report]✅ pass : 513 / 626 (81.95%) file detail
|
[FE Incremental Coverage Report]✅ pass : 1 / 1 (100.00%) file detail
|
kevincai
approved these changes
Nov 14, 2024
4 tasks
decster
reviewed
Nov 14, 2024
| } | ||
| } | ||
| if (!create_if_missing) { | ||
| return Status::NotFound(""); |
There was a problem hiding this comment.
add more detailed error message?
Comment on lines
+305
to
+309
| // TODO check if the error is retryable | ||
| st = _send_rpc_request(async_ctx->data_ctx()); | ||
| int64_t wait_ts = MonotonicNanos(); | ||
| rpc_cost_ns += wait_ts - rpc_ts; | ||
| st = _wait_for_stream_load_pipe(); |
There was a problem hiding this comment.
st assigned twice without checking? is this intentional?
| _dead_stream_load_pipe_ctxs.emplace(pipe_ctx); | ||
| it = _alive_stream_load_pipe_ctxs.erase(it); | ||
| } | ||
| return st.ok() ? Status::CapacityLimitExceed("") : st; |
There was a problem hiding this comment.
add more info for error message?
| if (!_alive_stream_load_pipe_ctxs.empty()) { | ||
| return Status::OK(); | ||
| } | ||
| return Status::TimedOut(""); |
There was a problem hiding this comment.
add more info for error message?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I'm doing:
The third PR to support batching stream load on the server side. This pr mainly implements the logic on the BE side
What I'm doing:
New Components
IsomorphicBatchWrite: manages load requests with the isomorphic parameters under the same db + table. These requests can be in a batch.BatchWriteMgr: managesIsomorphicBatchWriteTimeBoundedStreamLoadPipe: a specialized implementation of theStreamLoadPipewith a time constraint. It will finish itself after a period. After the pipe finishes, appending data will getCapacityLimitExceedstatusstream_loadis the newly added brpc interface. Compared to HTTP, the brpc interface is more suitable for scenarios with high concurrency, high throughput, and asynchronous operations. Currently, the brpc interface is mainly used in conjunction with Flink connector which depends on brpc-java. Note that we can extend this brpc interface for the normal stream load in the future to improve the communication efficiency.The following diagram shows the process of BE from receiving requests to pushing data into the plan
IsomorphicBatchWriteaccording to the db + table + load parameters. The request then pushes the data to thebthread::ExecutionQueue, and waits for the data is loaded. The queue can solve the concurrency problemTimeBoundedStreamLoadPiperequestBatchWriteto FE to request a load plan, and the executor will wait for the planTimeBoundedStreamLoadPipe, and registers it toIsomorphicBatchWrite7-10. the executor pushes the data to the pipe continously
Status::CapacityLimitExceedif data is still pushed to it. After all pipes of the plan finish, there is no more data for this load, and data can be committedStatus::CapacityLimitExceedwhen pushing data to the current pipe so that the data can be pushed to the next loadFixes #51085
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: