Baseline resync msg protocol and data structure #221
Conversation
- pg_blob_iterator is the snapshot resync context for leader(read path) - SnapshotReceiveHandler is a newly added structure as the snapshot context for follower(write path) - Comment previous implementation codes
yuwmao
requested review from
JacksonYao287,
sanebay,
yamingk,
xiaoxichen and
koujl
|
can we target the baseline resync pr to https://github.com/eBay/HomeObject/tree/baseline_resync? |
Sure. |
|
Codecov ReportAttention: Patch coverage is
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## main #221 +/- ##
==========================================
- Coverage 68.69% 63.67% -5.02%
==========================================
Files 30 31 +1
Lines 1581 1682 +101
Branches 163 178 +15
==========================================
- Hits 1086 1071 -15
- Misses 408 518 +110
- Partials 87 93 +6 ☔ View full report in Codecov by Sentry. |
| snp_batch_id_t batchId; | ||
|
|
||
| objId(shard_id_t shard_id, snp_batch_id_t batch_id) : shardId(shard_id), batchId(batch_id) { | ||
| //type_bit (1 bit) | shard_id (48 bits) | batch_id (15 bits) |
There was a problem hiding this comment.
please do overflow checking before taking the 'or' operator.
i.e if (shardid != shardid &0xffffffffffff) // error
There was a problem hiding this comment.
this is to prevent overflow batchID /shardID corrupt other field
There was a problem hiding this comment.
Good point, will add checking for them.
| user_key : [ubyte]; | ||
| data : [ubyte]; | ||
| is_deleted: bool; | ||
| data : [ubyte]; // Raw blob data loaded from drive, include BlobHeader, user_key and payload |
There was a problem hiding this comment.
where do we want to do checksum in the BR process?
i.e for each batch , how we verify the data? we can choose to use the blob crc in blob header.
There was a problem hiding this comment.
BlobHeader already has a hash, we can use that to verify every blobs in the batches.
Based on the baseline resync design doc,
PGBlobIterator is the resync context/handler for the leader(read path)
SnapshotReceiveHandler is the resync context/handler for the follower(write path)