Release 6.1.1 #1365
Merged
Release 6.1.1 #1365
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Iglu Scala Client has new lookupSchemasUntil function that allows to fetch list of schemas until given schema key. If we replace listSchemasLike function with lookupSchemasUntil function, RDB Loader won't rely on the list endpoint of Iglu Server. This commit makes necessary changes to use new lookupSchemasUntil function instead of listSchemasLike function.
Some of the tests are failing since windows don't have expected timestamps. In order to solve this problem, this commit makes necessary changes to read window timestamps from shredded message instead of using hard-coded values.
After starting to use lookupSchemasUntil in fetchSchemasWithSameModel, we are only getting schemas until the given schema key for every schema key. Previously, we were getting all the schemas for the same schema model. This change caused change of behavior when a message contains multiple schema keys for same schema model. When this happens, RDB Loader tries to create same table multiple times. In order to solve this problem, this commit contains the change for creating the migration for only max schema key of the same schema model.
We've seen exceptions in spark executors like: ``` java.lang.NullPointerException: Cannot invoke "scala.collection.mutable.Set.isEmpty()" because the return value of "com.snowplowanalytics.snowplow.rdbloader.transformer.batch.spark.TypesAccumulator.accum()" is null ``` The error is coming from our Spark Accumulator for accumulating Iglu types. This is similar to [an issue previously seen][1] in Spark's own `CollectionAccumulator`. That issue [was fixed in Spark][2] by making the accumulator's internal state non-final, and synchronizing access to the internal state. So here we make the exact same change to our own Accumulator. It is a rare race condition which is hard to reproduce. [1]: https://issues.apache.org/jira/browse/SPARK-20977 [2]: apache/spark#31540
istreeter
approved these changes
Oct 9, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Jira ref: PDP-1453