[GLUTEN-7028][CH][Part-5] Refactor: add NativeOutputWriter to unify CHDatasourceJniWrapper #7395
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Run Gluten Clickhouse CI |
5 tasks
|
Run Gluten Clickhouse CI |
baibaichen
force-pushed
the
feature/one-pipeline-native_out
branch
from
9a13f5f
to
def1e47
Compare
|
Run Gluten Clickhouse CI |
baibaichen
force-pushed
the
feature/one-pipeline-native_out
branch
from
def1e47
to
552e819
Compare
|
Run Gluten Clickhouse CI |
baibaichen
force-pushed
the
feature/one-pipeline-native_out
branch
from
552e819
to
c139b8d
Compare
|
Run Gluten Clickhouse CI |
baibaichen
force-pushed
the
feature/one-pipeline-native_out
branch
from
c139b8d
to
ef03755
Compare
|
Run Gluten Clickhouse CI |
baibaichen
force-pushed
the
feature/one-pipeline-native_out
branch
from
ef03755
to
17bb049
Compare
|
Run Gluten Clickhouse CI |
….mergetree.MetaSerializer
baibaichen
force-pushed
the
feature/one-pipeline-native_out
branch
from
17bb049
to
898498e
Compare
|
Run Gluten Clickhouse CI |
baibaichen
changed the title
zzcclp
approved these changes
Oct 9, 2024
baibaichen
added a commit
to Kyligence/gluten
that referenced
this pull request
Oct 15, 2024
(cherry picked from commit 94e1837a922d5a092226b195d6c3079d320878cb)
baibaichen
added a commit
that referenced
this pull request
Oct 15, 2024
* [GLUTEN-1632][CH]Daily Update Clickhouse Version (20241015) * Fix Build due to ClickHouse/ClickHouse#70135 * Resovle conflict with #7322 * gtest skip since plan is chagned due to #7395 (cherry picked from commit 94e1837a922d5a092226b195d6c3079d320878cb) --------- Co-authored-by: kyligence-git <[email protected]> Co-authored-by: Chang Chen <[email protected]>
sharkdtu
pushed a commit
to sharkdtu/gluten
that referenced
this pull request
Nov 11, 2024
…HDatasourceJniWrapper (apache#7395) * Add NativeOutputWriter * refactor CHDatasourceJniWrapper * WriteConfiguration * using hadoop Configuration to pass parameter * Implement CHMergeTreeWriterInjects::createNativeWrite * Rename datasources.clickhouse.ClickhouseMetaSerializer => datasources.mergetree.MetaSerializer * delete MergeTreeDeltaUtil and move its functionality to StorageMeta * WriteConfiguration => StorageConfigProvider * fix prefixof * WriteConfiguration => StorageConfigProvider 2 * withStorageID
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
(Fixes: #7028), This is last refactor PR, we unfiy how pass info between 3.3 and 3.5.
NativeOutputWriter, so we can unifyCHDatasourceJniWrapperConfigurationto pass config from driver to worker, this is standard way which spark used, and hence we can use the sameCHMergeTreeWriterInjects::createOutputWriterdefinition.WriteRelto pass info from jvm to cpp, see below data structure.optimizationisAny,messge Writeis added in write_optimization.protoHow was this patch tested?
UTs