Add baseline resync with read and write support #387
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hkadayam
reviewed
Apr 18, 2024
| @@ -293,7 +295,13 @@ bool HomeRaftLogStore::compact(ulong compact_lsn) { | |||
| // we rely on resrouce mgr timer to trigger truncate for all log stores in system; | |||
| // this will be friendly for multiple logstore on same logdev; | |||
|
|
|||
| // m_log_store->truncate(to_store_lsn(compact_lsn)); | |||
| #ifdef _PRERELEASE | |||
| if (iomgr_flip::instance()->test_flip("force_home_raft_log_truncate")) { | |||
There was a problem hiding this comment.
Suggest a parameterized flip, so you can force raft log truncate on specific logstore. This will trigger on every logstore truncate always.
hkadayam
reviewed
Apr 18, 2024
sanebay
changed the title
szmyd
reviewed
Aug 5, 2024
sanebay
force-pushed
the
baseline_read
branch
4 times, most recently
from
4e92c2d to
860e2cd
Compare
szmyd
reviewed
Aug 8, 2024
| LOGINFO("Leader create snapshot and truncate"); | ||
| this->create_snapshot(); | ||
| this->truncate(0); | ||
| } |
There was a problem hiding this comment.
Should we just write the snapshot_freq_distance number of logs to cause an internal snapshot like we'll experience in a real setup? We never call snapshot directly from above the library from what I know.
There was a problem hiding this comment.
We do set snapshot_freq_distance at line 946.
Add nuraft based snapshot sync across leader and followers. Implement the read and write snapshot callback. Add a generic snapshot structure to support different implemenations. Listener implements necessary function to load/save snapshot state. Listener also implements on how to read and write actual snapshot data. Added test to do basic baseline test.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add generic snapshot context and data structure.
Add callbacks for listener as discussed.
Use nuraft read and write logical snapshot callbacks to transfer the data.
Do test on both read and write for the baseline resync.
Key and values were pushed by leader and written by follower and its tested.
Currently verification of the data in the test are disabled but have been locally tested.