[Feature] Support restoring from a cluster snapshot for shared-data mode (part 3, introduce gtid for tablet metadata) #54326
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![starrocks-cr[bot]](https://avatars.githubusercontent.com/in/404334?s=60&v=4){kind=small}
fe/fe-core/src/main/java/com/starrocks/alter/LakeTableSchemaChangeJob.java
Show resolved
Hide resolved
xiangguangyxg
force-pushed
the
restore_snapshot_align_data_with_metadata
branch
2 times, most recently
from
2cda723 to
11f7c5c
Compare
kevincai
previously approved these changes
Dec 26, 2024
|
The introduction of global transaction id is a very important change. Can you explain its necessity in more detail? I hope you can give some examples. |
HangyuanLiu
reviewed
Dec 26, 2024
| @@ -70,10 +70,11 @@ public static void buildPartitionsSequentially(long dbId, OlapTable table, List< | |||
| int partitionGroupSize = Math.max(1, numBackends * 200 / Math.max(1, avgReplicasPerPartition)); | |||
| boolean enableTabletCreationOptimization = table.isCloudNativeTableOrMaterializedView() | |||
| && Config.lake_enable_tablet_creation_optimization; | |||
| long gtid = GlobalStateMgr.getCurrentState().getGtidGenerator().nextGtid(); | |||
There was a problem hiding this comment.
I think it is not a good idea to generate a global transaction id here, because it is a low-level function. It is far from the actual meaning of gtid. How can we ensure that a transaction will not call the buildPartition function multiple times? I think it should be placed in the upper-level begin transaction.
There was a problem hiding this comment.
added option arg to pass the gtid to the low-level function
…ode (part 3, introduce gtid for tablet metadata) Signed-off-by: xiangguangyxg <[email protected]>
xiangguangyxg
force-pushed
the
restore_snapshot_align_data_with_metadata
branch
from
11f7c5c to
795446b
Compare
|
[Java-Extensions Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[FE Incremental Coverage Report]✅ pass : 48 / 52 (92.31%) file detail
|
[BE Incremental Coverage Report]✅ pass : 5 / 5 (100.00%) file detail
|
HangyuanLiu
approved these changes
Dec 27, 2024
kevincai
approved these changes
Dec 27, 2024
7 tasks
|
@mergify backport branch-3.4 |
✅ Backports have been created
|
mergify bot
pushed a commit
that referenced
this pull request
Dec 27, 2024
…ode (part 3, introduce gtid for tablet metadata) (#54326) Signed-off-by: xiangguangyxg <[email protected]> (cherry picked from commit f977337) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/task/CreateReplicaTask.java # fe/fe-core/src/main/java/com/starrocks/task/TabletTaskExecutor.java # gensrc/proto/lake_types.proto # gensrc/thrift/AgentService.thrift
18 tasks
wanpengfei-git
pushed a commit
that referenced
this pull request
Dec 27, 2024
…ode (part 3, introduce gtid for tablet metadata) (backport #54326) (#54450) Signed-off-by: xiangguangyxg <[email protected]> Co-authored-by: xiangguangyxg <[email protected]>
banmoy
pushed a commit
to banmoy/starrocks
that referenced
this pull request
Dec 29, 2024
…ode (part 3, introduce gtid for tablet metadata) (StarRocks#54326) Signed-off-by: xiangguangyxg <[email protected]>
maggie-zhu
pushed a commit
to maggie-zhu/starrocks
that referenced
this pull request
Jan 6, 2025
…ode (part 3, introduce gtid for tablet metadata) (StarRocks#54326) Signed-off-by: xiangguangyxg <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I'm doing:
The dirty data must be detected when restoring from a cluster snapshot.
What I'm doing:
Introduce gtid for tablet metadata, gtid will be used for the following:
select a from b before t.Fixes #53867
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: