Spec: Document Snapshot Summary Optional Fields for Standardization #11660
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
Specification
6 tasks
format/spec.md
Outdated
| * `delete` -- Data files were removed and their contents logically deleted and/or delete files were added to delete rows. | ||
|
|
||
| ##### Optional Metrics | ||
| All metrics fields should have numeric string values (e.g., `"120"`). |
There was a problem hiding this comment.
I think we should note that these values may be used by engines for optimizations so they must be correct but they can be skipped if an engine doesn't want to write them. Maybe that's clear enough already though with "optional"
format/spec.md
Outdated
| * `overwrite` -- Data and delete files were added and removed in a logical overwrite operation. | ||
| * `delete` -- Data files were removed and their contents logically deleted and/or delete files were added to delete rows. | ||
|
|
||
| ##### Optional Metrics |
There was a problem hiding this comment.
As on the proposal doc I think this would be a nice appendix table
format/spec.md
Outdated
| Some of them are also used to represent partition-level metrics, in [Optional Partition-Level Summary](#optional-partition-level-summary). | ||
|
|
||
| | Field | Description | Used in Partition-Level Summary | | ||
| |-------------------------------------|-------------------------------------------------------------------|---------------------------------| |
There was a problem hiding this comment.
I'm not sure "current" is required here
format/spec.md
Outdated
| |-------------------------------------|-------------------------------------------------------------------|---------------------------------| | ||
| | **`added-data-files`** | Number of data files added in the current snapshot | Yes | | ||
| | **`deleted-data-files`** | Number of data files deleted in the current snapshot | Yes | | ||
| | **`total-data-files`** | Total number of data files in the current snapshot | No | |
There was a problem hiding this comment.
Total number of live data files?
format/spec.md
Outdated
| | **`added-data-files`** | Number of data files added in the current snapshot | Yes | | ||
| | **`deleted-data-files`** | Number of data files deleted in the current snapshot | Yes | | ||
| | **`total-data-files`** | Total number of data files in the current snapshot | No | | ||
| | **`added-delete-files`** | Number of delete files added in the current snapshot | Yes | |
There was a problem hiding this comment.
Is this a combo of position and equality deletes
format/spec.md
Outdated
| | **`added-dvs`** | Number of deletion vectors added in the current snapshot | Yes | | ||
| | **`removed-dvs`** | Number of deletion vectors removed in the current snapshot | Yes | | ||
| | **`removed-delete-files`** | Number of delete files removed in the current snapshot | Yes | | ||
| | **`total-delete-files`** | Total number of delete files in the current snapshot | No | |
There was a problem hiding this comment.
This is a combination of dv, df and eq deletes right?
format/spec.md
Outdated
| | **`added-equality-deletes`** | Number of equality delete records added in the current snapshot | Yes | | ||
| | **`removed-equality-deletes`** | Number of equality delete records removed in the current snapshot | Yes | | ||
| | **`total-equality-deletes`** | Total number of equality delete records in the current snapshot | No | | ||
| | **`deleted-duplicate-files`** | Number of duplicate files deleted in the current snapshot | No | |
There was a problem hiding this comment.
Unclear where duplicate files are coming from
format/spec.md
Outdated
| | **`removed-equality-deletes`** | Number of equality delete records removed in the current snapshot | Yes | | ||
| | **`total-equality-deletes`** | Total number of equality delete records in the current snapshot | No | | ||
| | **`deleted-duplicate-files`** | Number of duplicate files deleted in the current snapshot | No | | ||
| | **`changed-partition-count`** | Number of partitions changed in the current snapshot | No | |
There was a problem hiding this comment.
I think we need to detail, partitions with added, deleted or modified files?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces a new section, "Snapshot Summary", in the table spec under Snapshots to document optional fields in the snapshot summary, including metrics, partition-level summaries, and other fields such as Write-Audit-Publish (WAP)-related fields and ReplacePartitions indicators. The goal is to establish a clear standard for these fields, ensuring consistent naming and usage across implementations while reducing ambiguity and improving compatibility.
Proposal Here: https://docs.google.com/document/d/1Gt1ZOXVXK60IGdlmt4QlyRzaZ1iCVyYUBfMJCsiz14I/edit?usp=sharing
Marked as Draft because this is subject to change based on discussion on dev list