Upgrade to iceberg 1.5.2 #235
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tabmatfournier
commented
Apr 11, 2024
kafka-connect/src/main/java/io/tabular/iceberg/connect/channel/Worker.java
Outdated
Show resolved
Hide resolved
tabmatfournier
commented
Apr 11, 2024
kafka-connect/src/main/java/io/tabular/iceberg/connect/channel/Deduplicated.java
Show resolved
Hide resolved
|
Should have a branch up shortly that deals with the breaking change instead. Much better for the users --no drain mode needed. |
|
Iceberg 1.5.2 is the latest release |
tabmatfournier
commented
May 15, 2024
kafka-connect/src/main/java/io/tabular/iceberg/connect/channel/CommitterImpl.java
Show resolved
Hide resolved
fqtab
reviewed
May 17, 2024
...a-connect-events/src/test/java/io/tabular/iceberg/connect/events/EventSerializationTest.java
Show resolved
Hide resolved
kafka-connect/src/main/java/io/tabular/iceberg/connect/channel/CommitState.java
Outdated
Show resolved
Hide resolved
kafka-connect-events/src/main/java/org/apache/iceberg/avro/DeprecatedTypeToSchema.java
Show resolved
Hide resolved
kafka-connect/src/test/java/io/tabular/iceberg/connect/channel/EventDecoderTest.java
Show resolved
Hide resolved
kafka-connect/src/test/java/io/tabular/iceberg/connect/channel/EventDecoderTest.java
Outdated
Show resolved
Hide resolved
kafka-connect/src/test/java/io/tabular/iceberg/connect/channel/CommitterImplTest.java
Outdated
Show resolved
Hide resolved
kafka-connect/src/main/java/io/tabular/iceberg/connect/channel/EventDecoder.java
Outdated
Show resolved
Hide resolved
kafka-connect/src/main/java/io/tabular/iceberg/connect/channel/EventDecoder.java
Outdated
Show resolved
Hide resolved
fqtab
approved these changes
May 21, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Updates connector to Iceberg 1.5
The
iceberg-kafka-connect-eventsmodule has been deprecated in this repo: consider theselegacyEvents. Instead we rely oniceberg-kafka-connect-eventswhich has been moved into Iceberg Core. This will make it easier to move the remainder of the repository over to there this PR updates worker/coordinator/etc. to work off the (significantly) reworked event classes.The avro payloads produced by the worker/coordinator have changed in a way that is not backward compatible. This is unavoidable atm due to porting the codebase (partially) to Iceberg-core and the changes made there (see: apache/iceberg#8701 (comment)).
Encoding relies heavily on the Iceberg avro utils due to having to encode Datafiles, delete files, etc. Unfortunately changes were made upstream that break the format. This puts us in an awkward spot: we either have to port all the machinery of Avro from Iceberg to maintain the same format, or do the breaking change.
This PR ports the marchinery from Iceberg 1.4.x to have a fallback decoder if decoding fails in the event a legacy record was left behind in the control topic during the upgrade. This code should be removed in future releases.
There is no breaking change for the users because of this. Eventing now depends on donated connector code in iceberg-core, which should result in non-breaking changes for our users to migrate when the rest of the code is ported over.