Accounting writer #72
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* replace io.IO with objstore.Bucket * move Avro schemas out of internal and support template for entries * Support for Hive catalog * Support for writing manifest files with builders and writers.
* populate requires specs so they are not nil * rename hive to hdfs
Spark complains if the field is missing
* table: SnapshotWriter * remove catalogs - Glue - Rest This removes imports of opencensus that leaks a go routine
* icebucket: used to strip paths from bucket filenames * hdfs unit test
schemas is an optional field in v1 metadata
* Partition support for manifest lists
* fix: setting upper/lower bounds from Parquet file Use the single-value serialization required by the spec * sort the schema by name * write the table schema into the manifest metadata * correctly set the upper/lower bounds of the manifest entry Using the merged table schema while this entry is being written it will leave empty fields if the field doesn't exist in the file * update the partition spec if schema changes
* added options for deleting metadata files after commit
Put snapshotwriter into it's own struct and have the hdfs snapshotwriter embed the snapshot writer to perform the generic portions of the writer This will allow future table implementations to reuse the writer.
Previously was storing 64 bit values with 32bit precision
Previously was doing a byte comparison regardless of type which doesn't work for the little endian encoded integers
If a snapshot is expired; walk the expired snapshots manifests and determine if any are orphaned and require removal
* calculate added rows for manifest * fix unit test
Then we don't have to call attributes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.