Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Fix parquet timestamp tz roundtrip inference #1625

Merged
merged 1 commit into from
Nov 17, 2023

Conversation

jaychia
Copy link
Contributor

@jaychia jaychia commented Nov 16, 2023

This required a fix in our arrow2 fork: Eventual-Inc/arrow2@3a23c78

Essentially when Parquet has a type that is "timezone aware", Arrow2 will always infer it as "+00:00" (not None as previously assumed). However if the embedded Arrow schema on the Parquet file tells us that the timezone should actually be something else, we need to override "+00:00" with the one provided by the embedded Arrow schema.

@jaychia jaychia enabled auto-merge (squash) November 16, 2023 23:53
@github-actions github-actions bot added the bug Something isn't working label Nov 16, 2023
Copy link

codecov bot commented Nov 17, 2023

Codecov Report

Merging #1625 (4fd301b) into main (427a4fe) will not change coverage.
Report is 4 commits behind head on main.
The diff coverage is n/a.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #1625   +/-   ##
=======================================
  Coverage   87.32%   87.32%           
=======================================
  Files          51       51           
  Lines        5040     5040           
=======================================
  Hits         4401     4401           
  Misses        639      639           

@jaychia jaychia merged commit e3a154b into main Nov 17, 2023
37 checks passed
@jaychia jaychia deleted the jay/fix-parquet-timezone-roundtrip branch November 17, 2023 00:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant