[mysql,postgres] Change Debezium conversion for timestamp types #317
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nathan-artie
commented
Mar 27, 2024
| return timeValue.UnixMicro(), nil | ||
| } | ||
|
|
||
| type ZonedTimestampConverter struct{} |
There was a problem hiding this comment.
Basically renamed ZonedTimestampConverter to TimestampConverter and updated the DebeziumType appropriately.
nathan-artie
commented
Mar 27, 2024
| @@ -67,18 +67,37 @@ func (DateConverter) Convert(value any) (any, error) { | |||
| return int32(timeValue.Unix() / (60 * 60 * 24)), nil | |||
| } | |||
|
|
|||
| type TimestampConverter struct{} | |||
| type MicroTimestampConverter struct{} | |||
There was a problem hiding this comment.
Added a new MicroTimestampConverter.
nathan-artie
commented
Mar 27, 2024
| _, err := converter.Convert(1234) | ||
| assert.ErrorContains(t, err, "expected time.Time got int with value: 1234") | ||
| } | ||
| { |
There was a problem hiding this comment.
I don't think we need to worry about dates > 9999 or < 0 for MicroTimestamps since it uses an int64 and won't have JSON encoding issues.
Tang8330
reviewed
Mar 28, 2024
| } | ||
| { | ||
| // Date < 0 | ||
| value, err := converter.Convert(time.Date(-1, 2, 3, 4, 5, 0, 0, time.UTC)) |
There was a problem hiding this comment.
For these tests, can we have it also run against Transfer to ensure Transfer is able to parse it correctly?
Like run it against this function: https://github.com/artie-labs/transfer/blob/52a52c9cbe92e506c7d4a24757440979d28ef8a8/lib/debezium/types.go#L72
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Logged the schema that Debezium outputs when run against Postgres:
And the data:
This changes the converters used for MySQL and Postgres to be in line with what Debezium does:
datetime->io.debezium.time.MicroTimestamptimestamp->io.debezium.time.ZonedTimestamptimestamp with time zone->io.debezium.time.ZonedTimestamptimestamp without time zone->io.debezium.time.MicroTimestamp