implement fix pg size #220
Conversation
Hooper9973
force-pushed
the
implement_fix_pg_size
branch
from
77e4908 to
823e1ff
Compare
| @@ -127,12 +136,19 @@ void HSHomeObject::on_create_pg_message_commit(int64_t lsn, sisl::blob const& he | |||
| return; | |||
| } | |||
|
|
|||
| // select chunks for pg | |||
| const auto chunk_size = chunk_selector()->get_chunk_size(); | |||
| const chunk_num_t num_chunk = sisl::round_up(pg_info.size, chunk_size) / chunk_size; | |||
There was a problem hiding this comment.
as a preparation of identical_shard_layout we need to ensure PG has same identical chunks layout( same number of chunks and same chunk size across replicas), should we include this in PGInfo (maybe HS_PGInfo) so the decision of leaders can be send to follower?
There was a problem hiding this comment.
The same chunk size is already ensured by homogeneous disks, the same number of chunks is ensured by fixed pg size, and an identical layout will be guaranteed during the create shard process by writing v_chunk_id into shard_info_superblk to ensure followers select the same v_chunk.
There was a problem hiding this comment.
yeah I am concerning about the chunk size ---- if mistaken was made, surface it up earlier.
There was a problem hiding this comment.
More comprehensive error handling and security defense code will be proposed in the Identical layout pr.
Hooper9973
force-pushed
the
implement_fix_pg_size
branch
from
823e1ff to
260d687
Compare
Hooper9973
force-pushed
the
implement_fix_pg_size
branch
2 times, most recently
from
c86f5e8 to
8514e23
Compare
|
Codecov ReportAttention: Patch coverage is
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## main #220 +/- ##
==========================================
- Coverage 68.69% 63.15% -5.54%
==========================================
Files 30 32 +2
Lines 1581 1900 +319
Branches 163 204 +41
==========================================
+ Hits 1086 1200 +114
- Misses 408 600 +192
- Partials 87 100 +13 ☔ View full report in Codecov by Sentry. |
Hooper9973
force-pushed
the
implement_fix_pg_size
branch
2 times, most recently
from
40930ce to
bf0f88b
Compare
Hooper9973
force-pushed
the
implement_fix_pg_size
branch
3 times, most recently
from
44720fc to
99c21b7
Compare
Hooper9973
force-pushed
the
implement_fix_pg_size
branch
from
99c21b7 to
1746bcc
Compare
There was a problem hiding this comment.
generally looks good. I have a final question.
m_per_dev_heap is now a heap to maintain all chunks of a pdev , right?
in our implementation, what we want is to identify whether a chunk has been assigned to any pg or not, so do we still need a heap here? maybe we can used a simpler data structure?
if I miss something, pls correct me
Thanks for bringing this up! Just to clarify a bit more: m_chunks holds all the chunks. m_per_dev_heap specifically manages only the empty chunks, and m_per_pg_heap takes care of the chunks used for specific page groups. It's important to note that m_per_dev_heap and m_per_pg_heap are mutually exclusive - a chunk will only exist in one or the other, never both. Together, the total number of chunks in m_per_dev_heap and m_per_pg_heap matches exactly with the total in m_chunks. Hope this makes things clearer~ |
Description:
This pull request introduces the initial draft for fixing the PG size. The main work includes:
pg_info_superblkto recover the state of chunkselector and the state of pg.