Spec: Minor modifications for v3 #10948
Conversation
github-actions
bot
added
the
Specification
| * Adding new supported type promotion cases, integer and long to string | ||
| * Adding default value support to struct fields | ||
| * Adding multi-argument transforms for partitioning and sorting | ||
|
|
There was a problem hiding this comment.
and specifies forwards compatibility for unknown transforms?
There was a problem hiding this comment.
If a reader doesn't recognize the transform, it will ignore it during query planning. For writing, an exception will be raised. Since this is a high-level summary, should we include these details?
There was a problem hiding this comment.
I don't think this is worth including in the summary, but we could add that. To me, this is supposed to be high level and changes like that are a bit too in depth.
|
|
||
| #### Terms | ||
| ### Terms |
There was a problem hiding this comment.
so style here is to skip "##" to avoid a table of contents entry? I wonder if it is worth a comment?
There was a problem hiding this comment.
I was trying to avoid this being at the same level as things like Schemas and data types further down. These aren't major sections of the spec. That's why I thought h3 rather than h2 was the right choice. I think we'll probably need to adjust these when we see what the site generates for the TOC.
There was a problem hiding this comment.
OK, I don't have a strong opinion, but we might want to leave a comment if possible so future readers don't try to correct what they might see as a mistake?
There was a problem hiding this comment.
TOC is not generated now for spec and its sub section. It only covers first H1 and its sub topics.
Without the table of contents for spec and its sub topics it is hard to navigate.
I have added a detailed description and fixed the same in #11067
| @@ -193,16 +204,14 @@ Supported primitive types are defined in the table below. Primitive types added | |||
|
|
|||
| Notes: | |||
|
|
|||
| 1. Decimal scale is fixed and cannot be changed by schema evolution. Precision can only be widened. | |||
There was a problem hiding this comment.
Is this being removed because we intend to change it in V3? Maybe this should be included in the summary?
There was a problem hiding this comment.
This is cleanup to make things more consistent. For decimal type promotion specifically, I removed this so that the section on schema evolution and type promotion is the source of truth. There shouldn't be two places to update so when I saw this was duplicative, I thought I should address it.
Similarly for timestamps, the type definition has the precision and there is no need to add it to the notes. I also clarified time for consistency.
| @@ -44,6 +44,15 @@ The primary change in version 2 adds delete files to encode rows that are delete | |||
|
|
|||
| In addition to row-level deletes, version 2 makes some requirements stricter for writers. The full set of changes are listed in [Appendix E](#version-2). | |||
|
|
|||
| #### Version 3: Extended Types and Capabilities | |||
There was a problem hiding this comment.
I think there is quite a bit of overlap with https://github.com/apache/iceberg/blob/main/format/spec.md#version-3. Looking at the questions that @emkornfield raised, it feels that this summary is too high level, maybe we should refer to Appendix E for details.
There was a problem hiding this comment.
My main worry is that we'll have a summary with no actual information. I think this is about the right amount since it is a high-level addition. The appendix is intended to be much more detailed.
rdblue
force-pushed
the
spec-v3-minor-changes
branch
from
72fd08e to
7576939
Compare
|
Thanks for reviewing, everyone. I've opened a thread on the dev list to merge this. |
This adds some minor changes to prepare for v3 spec work: