DP-3392: Option to lowercase field names #64
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
afiore
approved these changes
May 21, 2024
| @@ -46,10 +47,12 @@ final class RecursiveConversion(innerConversion: ConnectConversion) extends Conn | |||
| connectValue match { | |||
| case connectValue: Struct => | |||
| val newStruct = new Struct(targetSchema) | |||
| targetSchema.fields().asScala.foreach { field => | |||
| originalSchema.fields().asScala.foreach { field => | |||
There was a problem hiding this comment.
Noticing that we now iterate on the original rather than the target schema. I guess this is necessary as we need to supply both the original and the new field schema to convertValue...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a new boolean option
connect.ems.convert.lowercase.fields, that defaults tofalse.When set to
true, all field names in the produced in the parquet will be lowercase. This will be useful in those cases where JSON input data is not clean, and field names are not always given in the same case.Details
The conversion has been implemented as a feature of our
RecursiveConverter, which now accepts an additional functional parameter to transform field names.RecursiveConverter, and therefore the lowercase transformation when set, runs as aPreConversion, which means that the conversion will run at the beginning ofRecordTransformer, before flattening and before schema evolution.