Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support TIMESTAMP_NTZ #962

Merged
merged 20 commits into from
Oct 17, 2024
Merged

Support TIMESTAMP_NTZ #962

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
e40dba8
WIP.
Tang8330 Oct 15, 2024
f12cb67
Update Snowflake dialect.
Tang8330 Oct 15, 2024
53d3d01
Clean up.
Tang8330 Oct 15, 2024
177a38a
Update Redshift.
Tang8330 Oct 15, 2024
5975821
Clean up.
Tang8330 Oct 15, 2024
3a2e340
Update.
Tang8330 Oct 16, 2024
ebaa657
Support this into MSSQL.
Tang8330 Oct 16, 2024
7933120
Clean up.
Tang8330 Oct 16, 2024
fa08b76
Update.
Tang8330 Oct 16, 2024
40e1520
Clean up code.
Tang8330 Oct 16, 2024
b60b6ca
Clean up.
Tang8330 Oct 16, 2024
7fc43ab
Update.
Tang8330 Oct 16, 2024
f03fc91
Merge branch 'master' into support-timestamp-ntz
Tang8330 Oct 16, 2024
8ebe8f1
Clean up.
Tang8330 Oct 16, 2024
8535aab
Adding debug stmt.
Tang8330 Oct 16, 2024
0459f2b
Clean up.
Tang8330 Oct 16, 2024
9c48682
Update.
Tang8330 Oct 16, 2024
702d5de
Rename TimezoneOffsetFormat.
Tang8330 Oct 16, 2024
e47b817
Merge branch 'master' into support-timestamp-ntz
Tang8330 Oct 16, 2024
280fe06
Merge branch 'master' into support-timestamp-ntz
Tang8330 Oct 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion clients/bigquery/dialect/dialect.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,8 @@ func (BigQueryDialect) DataTypeForKind(kindDetails typing.KindDetails, _ bool) s
// https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#datetime_type
// We should be using TIMESTAMP since it's an absolute point in time.
return "timestamp"
case ext.TimestampNTZKindType:
return "datetime"
case ext.DateKindType:
return "date"
case ext.TimeKindType:
Expand Down Expand Up @@ -103,8 +105,10 @@ func (BigQueryDialect) KindForDataType(rawBqType string, _ string) (typing.KindD
return typing.Struct, nil
case "array":
return typing.Array, nil
case "datetime", "timestamp":
case "timestamp":
return typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampTzKindType), nil
case "datetime":
return typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampNTZKindType), nil
case "time":
return typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimeKindType), nil
case "date":
Expand Down
7 changes: 5 additions & 2 deletions clients/databricks/dialect/typing.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,11 @@ func (DatabricksDialect) DataTypeForKind(kindDetails typing.KindDetails, _ bool)
case typing.ETime.Kind:
switch kindDetails.ExtendedTimeDetails.Type {
case ext.TimestampTzKindType:
// Using datetime2 because it's the recommendation, and it provides more precision: https://stackoverflow.com/a/1884088
return "TIMESTAMP"
case ext.TimestampNTZKindType:
// This is currently in public preview, to use this, the customer will need to enable [timestampNtz] in their delta tables.
// Ref: https://docs.databricks.com/en/sql/language-manual/data-types/timestamp-ntz-type.html
return "TIMESTAMP_NTZ"
case ext.DateKindType:
return "DATE"
case ext.TimeKindType:
Expand Down Expand Up @@ -73,7 +76,7 @@ func (DatabricksDialect) KindForDataType(rawType string, _ string) (typing.KindD
case "timestamp":
return typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampTzKindType), nil
case "timestamp_ntz":
return typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampTzKindType), nil
return typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampNTZKindType), nil
}

return typing.Invalid, fmt.Errorf("unsupported data type: %q", rawType)
Expand Down
2 changes: 1 addition & 1 deletion clients/databricks/dialect/typing_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ func TestDatabricksDialect_KindForDataType(t *testing.T) {
// Timestamp NTZ
kd, err := DatabricksDialect{}.KindForDataType("TIMESTAMP_NTZ", "")
assert.NoError(t, err)
assert.Equal(t, typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampTzKindType), kd)
assert.Equal(t, typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampNTZKindType), kd)
}
{
// Variant
Expand Down
4 changes: 4 additions & 0 deletions clients/mssql/dialect/dialect.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,8 @@ func (MSSQLDialect) DataTypeForKind(kindDetails typing.KindDetails, isPk bool) s
case typing.ETime.Kind:
switch kindDetails.ExtendedTimeDetails.Type {
case ext.TimestampTzKindType:
return "datetimeoffset"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like before, if we received a timestamp with tz/offset info, we discarded the tz and stored it without it. Now we'll start storing the tz. Is this change going to result in ddl changes to the column type of existing columns?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it will. I'll migrate the tables that are in MSSQL from datetime2 -> datetimeoffset once this change is in

Copy link
Contributor Author

@Tang8330 Tang8330 Oct 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I don't think this would impact existing columns.

If the existing column is created as datetime2, we will then tag the column as TIMESTAMP_NTZ when we decipher the output from describe table.

When we merge our in-memory columns with existing, existing will take precedence.

It is then on the SDK to discard the tz information, which it should be doing already.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah okay so we won't start storing the tz info for existing columns, but if a new column is created that should be tz-aware then we will store it. Seems reasonable 👍

case ext.TimestampNTZKindType:
// Using datetime2 because it's the recommendation, and it provides more precision: https://stackoverflow.com/a/1884088
return "datetime2"
case ext.DateKindType:
Expand Down Expand Up @@ -114,6 +116,8 @@ func (MSSQLDialect) KindForDataType(rawType string, stringPrecision string) (typ
case
"datetime",
"datetime2":
return typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampNTZKindType), nil
case "datetimeoffset":
return typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampTzKindType), nil
case "time":
return typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimeKindType), nil
Expand Down
6 changes: 5 additions & 1 deletion clients/redshift/dialect/typing.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@ func (RedshiftDialect) DataTypeForKind(kd typing.KindDetails, _ bool) string {
switch kd.ExtendedTimeDetails.Type {
case ext.TimestampTzKindType:
return "timestamp with time zone"
case ext.TimestampNTZKindType:
return "timestamp without time zone"
case ext.DateKindType:
return "date"
case ext.TimeKindType:
Expand Down Expand Up @@ -103,7 +105,9 @@ func (RedshiftDialect) KindForDataType(rawType string, stringPrecision string) (
}, nil
case "double precision":
return typing.Float, nil
case "timestamp with time zone", "timestamp without time zone":
case "timestamp", "timestamp without time zone":
return typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampNTZKindType), nil
case "timestamp with time zone":
return typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampTzKindType), nil
case "time without time zone":
return typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimeKindType), nil
Expand Down
13 changes: 12 additions & 1 deletion clients/redshift/dialect/typing_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,17 @@ func TestRedshiftDialect_DataTypeForKind(t *testing.T) {
}
}
}
{
// Timestamps
{
// With timezone
assert.Equal(t, "timestamp with time zone", RedshiftDialect{}.DataTypeForKind(typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampTzKindType), false))
}
{
// Without timezone
assert.Equal(t, "timestamp without time zone", RedshiftDialect{}.DataTypeForKind(typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampNTZKindType), false))
}
}
}

func TestRedshiftDialect_KindForDataType(t *testing.T) {
Expand Down Expand Up @@ -130,7 +141,7 @@ func TestRedshiftDialect_KindForDataType(t *testing.T) {
kd, err := dialect.KindForDataType("timestamp without time zone", "")
assert.NoError(t, err)
assert.Equal(t, typing.ETime.Kind, kd.Kind)
assert.Equal(t, ext.TimestampTzKindType, kd.ExtendedTimeDetails.Type)
assert.Equal(t, ext.TimestampNTZKindType, kd.ExtendedTimeDetails.Type)
}
{
kd, err := dialect.KindForDataType("time without time zone", "")
Expand Down
10 changes: 5 additions & 5 deletions clients/snowflake/dialect/dialect.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,9 @@ func (SnowflakeDialect) DataTypeForKind(kindDetails typing.KindDetails, _ bool)
case typing.ETime.Kind:
switch kindDetails.ExtendedTimeDetails.Type {
case ext.TimestampTzKindType:
// We are not using `TIMESTAMP_NTZ` because Snowflake does not join on this data very well.
// It ends up trying to parse this data into a TIMESTAMP_TZ and messes with the join order.
// Specifically, if my location is in SF, it'll try to parse TIMESTAMP_NTZ into PST then into UTC.
// When it was already stored as UTC.
return "timestamp_tz"
case ext.TimestampNTZKindType:
return "timestamp_ntz"
case ext.DateKindType:
return "date"
case ext.TimeKindType:
Expand Down Expand Up @@ -99,8 +97,10 @@ func (SnowflakeDialect) KindForDataType(snowflakeType string, _ string) (typing.
return typing.Struct, nil
case "array":
return typing.Array, nil
case "datetime", "timestamp", "timestamp_ltz", "timestamp_ntz", "timestamp_tz":
case "timestamp_ltz", "timestamp_tz":
return typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampTzKindType), nil
case "timestamp", "datetime", "timestamp_ntz":
return typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampNTZKindType), nil
case "time":
return typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimeKindType), nil
case "date":
Expand Down
22 changes: 17 additions & 5 deletions clients/snowflake/dialect/dialect_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,11 +174,23 @@ func TestSnowflakeDialect_KindForDataType(t *testing.T) {
}

func TestSnowflakeDialect_KindForDataType_DateTime(t *testing.T) {
expectedDateTimes := []string{"DATETIME", "TIMESTAMP", "TIMESTAMP_LTZ", "TIMESTAMP_NTZ(9)", "TIMESTAMP_TZ"}
for _, expectedDateTime := range expectedDateTimes {
kd, err := SnowflakeDialect{}.KindForDataType(expectedDateTime, "")
assert.NoError(t, err)
assert.Equal(t, ext.TimestampTz.Type, kd.ExtendedTimeDetails.Type, expectedDateTime)
{
// Timestamp with time zone
expectedDateTimes := []string{"TIMESTAMP_LTZ", "TIMESTAMP_TZ"}
for _, expectedDateTime := range expectedDateTimes {
kd, err := SnowflakeDialect{}.KindForDataType(expectedDateTime, "")
assert.NoError(t, err)
assert.Equal(t, ext.TimestampTz.Type, kd.ExtendedTimeDetails.Type, expectedDateTime)
}
}
{
// Timestamp without time zone
expectedDateTimes := []string{"TIMESTAMP", "DATETIME", "TIMESTAMP_NTZ(9)"}
for _, expectedDateTime := range expectedDateTimes {
kd, err := SnowflakeDialect{}.KindForDataType(expectedDateTime, "")
assert.NoError(t, err)
assert.Equal(t, ext.TimestampNTZ.Type, kd.ExtendedTimeDetails.Type, expectedDateTime)
}
}
}

Expand Down
2 changes: 1 addition & 1 deletion lib/cdc/relational/debezium_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ func (r *RelationTestSuite) TestPostgresEventWithSchemaAndTimestampNoTZ() {
r.T(),
ext.NewExtendedTime(
time.Date(2023, time.February, 2, 17, 51, 35, 175445*1000, time.UTC),
ext.TimestampTzKindType, ext.RFC3339Microsecond,
ext.TimestampNTZKindType, ext.RFC3339MicrosecondNoTZ,
),
evtData["ts_no_tz1"],
)
Expand Down
36 changes: 24 additions & 12 deletions lib/debezium/converters/timestamp.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,48 +9,60 @@ import (

type Timestamp struct{}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Timestamp
  • MicroTimestamp
  • NanoTimestamp

Do not contain any timezone information.


func (Timestamp) ToKindDetails() typing.KindDetails {
return typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampTzKindType)
func (Timestamp) layout() string {
return ext.RFC3339MillisecondNoTZ
}

func (Timestamp) Convert(value any) (any, error) {
func (t Timestamp) ToKindDetails() typing.KindDetails {
return typing.NewExtendedTimeDetails(typing.ETime, ext.TimestampNTZKindType, t.layout())
}

func (t Timestamp) Convert(value any) (any, error) {
castedValue, err := typing.AssertType[int64](value)
if err != nil {
return nil, err
}

// Represents the number of milliseconds since the epoch, and does not include timezone information.
return ext.NewExtendedTime(time.UnixMilli(castedValue).In(time.UTC), ext.TimestampTzKindType, ext.RFC3339Millisecond), nil
return ext.NewExtendedTime(time.UnixMilli(castedValue).In(time.UTC), ext.TimestampNTZKindType, t.layout()), nil
}

type MicroTimestamp struct{}

func (MicroTimestamp) ToKindDetails() typing.KindDetails {
return typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampTzKindType)
func (MicroTimestamp) layout() string {
return ext.RFC3339MicrosecondNoTZ
}

func (mt MicroTimestamp) ToKindDetails() typing.KindDetails {
return typing.NewExtendedTimeDetails(typing.ETime, ext.TimestampNTZKindType, mt.layout())
}

func (MicroTimestamp) Convert(value any) (any, error) {
func (mt MicroTimestamp) Convert(value any) (any, error) {
castedValue, err := typing.AssertType[int64](value)
if err != nil {
return nil, err
}

// Represents the number of microseconds since the epoch, and does not include timezone information.
return ext.NewExtendedTime(time.UnixMicro(castedValue).In(time.UTC), ext.TimestampTzKindType, ext.RFC3339Microsecond), nil
return ext.NewExtendedTime(time.UnixMicro(castedValue).In(time.UTC), ext.TimestampNTZKindType, mt.layout()), nil
}

type NanoTimestamp struct{}

func (NanoTimestamp) ToKindDetails() typing.KindDetails {
return typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampTzKindType)
func (nt NanoTimestamp) ToKindDetails() typing.KindDetails {
return typing.NewExtendedTimeDetails(typing.ETime, ext.TimestampNTZKindType, nt.layout())
}

func (NanoTimestamp) layout() string {
return ext.RFC3339NanosecondNoTZ
}

func (NanoTimestamp) Convert(value any) (any, error) {
func (nt NanoTimestamp) Convert(value any) (any, error) {
castedValue, err := typing.AssertType[int64](value)
if err != nil {
return nil, err
}

// Represents the number of nanoseconds since the epoch, and does not include timezone information.
return ext.NewExtendedTime(time.UnixMicro(castedValue/1_000).In(time.UTC), ext.TimestampTzKindType, ext.RFC3339Nanosecond), nil
return ext.NewExtendedTime(time.UnixMicro(castedValue/1_000).In(time.UTC), ext.TimestampNTZKindType, nt.layout()), nil
}
18 changes: 9 additions & 9 deletions lib/debezium/converters/timestamp_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import (
)

func TestTimestamp_Converter(t *testing.T) {
assert.Equal(t, typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampTzKindType), Timestamp{}.ToKindDetails())
assert.Equal(t, typing.NewExtendedTimeDetails(typing.ETime, ext.TimestampNTZKindType, ext.RFC3339MillisecondNoTZ), Timestamp{}.ToKindDetails())
{
// Invalid conversion
_, err := Timestamp{}.Convert("invalid")
Expand All @@ -19,18 +19,18 @@ func TestTimestamp_Converter(t *testing.T) {
// Valid conversion
converted, err := Timestamp{}.Convert(int64(1_725_058_799_089))
assert.NoError(t, err)
assert.Equal(t, "2024-08-30T22:59:59.089Z", converted.(*ext.ExtendedTime).String(""))
assert.Equal(t, "2024-08-30T22:59:59.089", converted.(*ext.ExtendedTime).String(""))
}
{
// ms is preserved despite it being all zeroes.
converted, err := Timestamp{}.Convert(int64(1_725_058_799_000))
assert.NoError(t, err)
assert.Equal(t, "2024-08-30T22:59:59.000Z", converted.(*ext.ExtendedTime).String(""))
assert.Equal(t, "2024-08-30T22:59:59.000", converted.(*ext.ExtendedTime).String(""))
}
}

func TestMicroTimestamp_Converter(t *testing.T) {
assert.Equal(t, typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampTzKindType), MicroTimestamp{}.ToKindDetails())
assert.Equal(t, typing.NewExtendedTimeDetails(typing.ETime, ext.TimestampNTZKindType, ext.RFC3339MicrosecondNoTZ), MicroTimestamp{}.ToKindDetails())
{
// Invalid conversion
_, err := MicroTimestamp{}.Convert("invalid")
Expand All @@ -40,18 +40,18 @@ func TestMicroTimestamp_Converter(t *testing.T) {
// Valid conversion
converted, err := MicroTimestamp{}.Convert(int64(1_712_609_795_827_923))
assert.NoError(t, err)
assert.Equal(t, "2024-04-08T20:56:35.827923Z", converted.(*ext.ExtendedTime).String(""))
assert.Equal(t, "2024-04-08T20:56:35.827923", converted.(*ext.ExtendedTime).String(""))
}
{
// micros is preserved despite it being all zeroes.
converted, err := MicroTimestamp{}.Convert(int64(1_712_609_795_820_000))
assert.NoError(t, err)
assert.Equal(t, "2024-04-08T20:56:35.820000Z", converted.(*ext.ExtendedTime).String(""))
assert.Equal(t, "2024-04-08T20:56:35.820000", converted.(*ext.ExtendedTime).String(""))
}
}

func TestNanoTimestamp_Converter(t *testing.T) {
assert.Equal(t, typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampTzKindType), NanoTimestamp{}.ToKindDetails())
assert.Equal(t, typing.NewExtendedTimeDetails(typing.ETime, ext.TimestampNTZKindType, ext.RFC3339NanosecondNoTZ), NanoTimestamp{}.ToKindDetails())
{
// Invalid conversion
_, err := NanoTimestamp{}.Convert("invalid")
Expand All @@ -61,12 +61,12 @@ func TestNanoTimestamp_Converter(t *testing.T) {
// Valid conversion
converted, err := NanoTimestamp{}.Convert(int64(1_712_609_795_827_001_000))
assert.NoError(t, err)
assert.Equal(t, "2024-04-08T20:56:35.827001000Z", converted.(*ext.ExtendedTime).String(""))
assert.Equal(t, "2024-04-08T20:56:35.827001000", converted.(*ext.ExtendedTime).String(""))
}
{
// nanos is preserved despite it being all zeroes.
converted, err := NanoTimestamp{}.Convert(int64(1_712_609_795_827_000_000))
assert.NoError(t, err)
assert.Equal(t, "2024-04-08T20:56:35.827000000Z", converted.(*ext.ExtendedTime).String(""))
assert.Equal(t, "2024-04-08T20:56:35.827000000", converted.(*ext.ExtendedTime).String(""))
}
}
14 changes: 11 additions & 3 deletions lib/debezium/schema_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -229,14 +229,22 @@ func TestField_ToKindDetails(t *testing.T) {
assert.Equal(t, typing.Invalid, kd)
}
{
// Timestamp
// Datetime (for now)
for _, dbzType := range []SupportedDebeziumType{Timestamp, TimestampKafkaConnect, MicroTimestamp, NanoTimestamp, ZonedTimestamp} {
// Timestamp with timezone
for _, dbzType := range []SupportedDebeziumType{ZonedTimestamp} {
kd, err := Field{DebeziumType: dbzType}.ToKindDetails()
assert.NoError(t, err)
assert.Equal(t, typing.NewKindDetailsFromTemplate(typing.ETime, ext.TimestampTzKindType), kd)
}
}
{
// Timestamp without timezone
for _, dbzType := range []SupportedDebeziumType{Timestamp, TimestampKafkaConnect, MicroTimestamp, NanoTimestamp} {
kd, err := Field{DebeziumType: dbzType}.ToKindDetails()
assert.NoError(t, err)
assert.Equal(t, ext.TimestampNTZKindType, kd.ExtendedTimeDetails.Type)
assert.Equal(t, typing.ETime.Kind, kd.Kind)
}
}
{
// Dates
for _, dbzType := range []SupportedDebeziumType{Date, DateKafkaConnect} {
Expand Down
8 changes: 4 additions & 4 deletions lib/debezium/types_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -302,15 +302,15 @@ func TestField_ParseValue(t *testing.T) {
field := Field{Type: Int64, DebeziumType: dbzType}
value, err := field.ParseValue(int64(1_725_058_799_000))
assert.NoError(t, err)
assert.Equal(t, "2024-08-30T22:59:59.000Z", value.(*ext.ExtendedTime).String(""))
assert.Equal(t, "2024-08-30T22:59:59.000", value.(*ext.ExtendedTime).String(""))
}
}
{
// Nano timestamp
field := Field{Type: Int64, DebeziumType: NanoTimestamp}
val, err := field.ParseValue(int64(1_712_609_795_827_000_000))
assert.NoError(t, err)
assert.Equal(t, ext.NewExtendedTime(time.Date(2024, time.April, 8, 20, 56, 35, 827000000, time.UTC), ext.TimestampTzKindType, "2006-01-02T15:04:05.000000000Z07:00"), val.(*ext.ExtendedTime))
assert.Equal(t, ext.NewExtendedTime(time.Date(2024, time.April, 8, 20, 56, 35, 827000000, time.UTC), ext.TimestampNTZKindType, "2006-01-02T15:04:05.000000000"), val.(*ext.ExtendedTime))
}
{
// Micro timestamp
Expand All @@ -319,13 +319,13 @@ func TestField_ParseValue(t *testing.T) {
// Int64
val, err := field.ParseValue(int64(1_712_609_795_827_009))
assert.NoError(t, err)
assert.Equal(t, ext.NewExtendedTime(time.Date(2024, time.April, 8, 20, 56, 35, 827009000, time.UTC), ext.TimestampTzKindType, ext.RFC3339Microsecond), val.(*ext.ExtendedTime))
assert.Equal(t, ext.NewExtendedTime(time.Date(2024, time.April, 8, 20, 56, 35, 827009000, time.UTC), ext.TimestampNTZKindType, ext.RFC3339MicrosecondNoTZ), val.(*ext.ExtendedTime))
}
{
// Float64
val, err := field.ParseValue(float64(1_712_609_795_827_001))
assert.NoError(t, err)
assert.Equal(t, ext.NewExtendedTime(time.Date(2024, time.April, 8, 20, 56, 35, 827001000, time.UTC), ext.TimestampTzKindType, ext.RFC3339Microsecond), val.(*ext.ExtendedTime))
assert.Equal(t, ext.NewExtendedTime(time.Date(2024, time.April, 8, 20, 56, 35, 827001000, time.UTC), ext.TimestampNTZKindType, ext.RFC3339MicrosecondNoTZ), val.(*ext.ExtendedTime))
}
{
// Invalid (string)
Expand Down
8 changes: 8 additions & 0 deletions lib/optimization/table_data.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -293,8 +293,16 @@ func (t *TableData) MergeColumnsFromDestination(destCols ...columns.Column) erro
inMemoryCol.KindDetails.ExtendedTimeDetails = &ext.NestedKind{}
}

// If the column in the destination is a timestamp_tz and the in-memory column is a timestamp_ntz, we should update the layout to contain timezone locale.
if foundColumn.KindDetails.ExtendedTimeDetails.Type == ext.TimestampTzKindType && inMemoryCol.KindDetails.ExtendedTimeDetails.Type == ext.TimestampNTZKindType {
if inMemoryCol.KindDetails.ExtendedTimeDetails.Format != "" {
inMemoryCol.KindDetails.ExtendedTimeDetails.Format += ext.TimezoneOffsetFormat
}
}

// Just copy over the type since the format wouldn't be present in the destination
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this comment still true?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think so, we're still not copying the layout from the destination columns.

The only logic we added is that if the destination column is TIMESTAMP_TZ and the in-memory column is TIMESTAMP_NTZ, we will append a timezone offset format to the in-memory column's format (if it's set).

inMemoryCol.KindDetails.ExtendedTimeDetails.Type = foundColumn.KindDetails.ExtendedTimeDetails.Type

}

// Copy over the decimal details
Expand Down
Loading
Loading