Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

push data asynchronously at leader #590

Conversation

JacksonYao287
Copy link
Contributor

@JacksonYao287 JacksonYao287 commented Nov 13, 2024

now we always start appending log to log store at leader until all the futures of pushing data to followers are completed , which will involve unnecessary latency

this PR aims to make pushing data asynchronously, so that logs can be set to follower ASAP. it does not matter whether push_data is successful, since follower will try to fetch data from the leader if push_data fails

@codecov-commenter
Copy link

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 66.66667% with 1 line in your changes missing coverage. Please review.

Project coverage is 67.17%. Comparing base (1a0cef8) to head (dd3156d).
Report is 91 commits behind head on master.

Files with missing lines Patch % Lines
src/lib/replication/repl_dev/raft_repl_dev.cpp 66.66% 1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@             Coverage Diff             @@
##           master     #590       +/-   ##
===========================================
+ Coverage   56.51%   67.17%   +10.65%     
===========================================
  Files         108      109        +1     
  Lines       10300    10729      +429     
  Branches     1402     1466       +64     
===========================================
+ Hits         5821     7207     +1386     
+ Misses       3894     2824     -1070     
- Partials      585      698      +113     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@xiaoxichen xiaoxichen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it seems right but I would like to hold this till we solve and bake a stable version for PG move.

@JacksonYao287
Copy link
Contributor Author

it seems right but I would like to hold this till we solve and bake a stable version for PG move.

sure, will also loop other members in

@xiaoxichen
Copy link
Collaborator

one question:
the push_data_to_all_followers returns a future, we dont wait for the future at all, what is the difference compared to run_on_forget

@JacksonYao287
Copy link
Contributor Author

we dont wait for the future at all

we use folly::collectAllUnsafe to wait for the completion of all the future, no?

@xiaoxichen
Copy link
Collaborator

collectAllUnsafe

https://github.com/facebook/folly/blob/main/folly/futures/Future.h#L2360-L2385
it returns a future which satisfied when all to_be_collected features satisfied.

@JacksonYao287
Copy link
Contributor Author

JacksonYao287 commented Nov 14, 2024

collectAllUnsafe

I see , you are right. I misunderstood the meaning of collectAllUnsafe. now I think , as you said , there is no difference compared to run_on_forget and the only improvement in this PR is that allocating_buffer and releasing buffer will not block the current thread, which seems not worth a change. let me close this pr

@JacksonYao287 JacksonYao287 deleted the push-data-asyncronously-at-leader branch November 14, 2024 02:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants