Baseline Resync Enhancement #244
Conversation
yuwmao
force-pushed
the
baseline_resync
branch
from
b831af9 to
2a7e47a
Compare
| @@ -379,17 +379,6 @@ void HSHomeObject::on_shard_message_commit(int64_t lsn, sisl::blob const& h, hom | |||
| local_create_shard(shard_info, v_chunk_id, blkids.chunk_num(), blkids.blk_count()); | |||
| if (ctx) { ctx->promise_.setValue(ShardManager::Result< ShardInfo >(shard_info)); } | |||
|
|
|||
| // update pg's total_occupied_blk_count | |||
There was a problem hiding this comment.
Redundant, these codes are already put into the local_create_shard.
|
Codecov ReportAttention: Patch coverage is
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## baseline_resync #244 +/- ##
===================================================
- Coverage 65.52% 64.23% -1.29%
===================================================
Files 33 33
Lines 2271 2416 +145
Branches 254 280 +26
===================================================
+ Hits 1488 1552 +64
- Misses 659 719 +60
- Partials 124 145 +21 ☔ View full report in Codecov by Sentry. |
yuwmao
force-pushed
the
baseline_resync
branch
2 times, most recently
from
556376d to
f743e81
Compare
yuwmao
changed the title
yuwmao
force-pushed
the
baseline_resync
branch
2 times, most recently
from
ce6769f to
b07084c
Compare
yuwmao
force-pushed
the
baseline_resync
branch
from
b07084c to
9ef9f5f
Compare
yuwmao
changed the title
| @@ -354,6 +360,13 @@ HSHomeObject::blob_put_get_blk_alloc_hints(sisl::blob const& header, cintrusive< | |||
| return folly::makeUnexpected(homestore::ReplServiceError::FAILED); | |||
| } | |||
|
|
|||
| auto pg_iter = _pg_map.find(msg_header->pg_id); | |||
There was a problem hiding this comment.
NIT: can we change this part to is_pg_present
There was a problem hiding this comment.
Here we need to get the pg_iter as we are going to check index, so we can't use is_pg_present directly.
There was a problem hiding this comment.
Found there is already a pg_exists function, remove is_pg_present function.
This commit addresses blob duplication issues caused by resync, it mainly happens when resend snapshot mesg during baseline resync and apply log after snapshot completion. This helps avoid unnecessary GC due to duplicated data. This mechanism is effective only for duplicated blobs. Since we do not record the blks of the `shardInfo` stored in the data service, we are unable to skip data writes for duplicated shards.
yuwmao
force-pushed
the
baseline_resync
branch
2 times, most recently
from
ab2184a to
bc80900
Compare
yuwmao
force-pushed
the
baseline_resync
branch
from
bc80900 to
ab4b81d
Compare
This PR enhances baseline resync:
This mechanism is effective only for duplicated blobs. Since we do not record the blks of the
shardInfostored in the data service, we are unable to skip data writes for duplicated shards.