Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Core: Track metadata file location in ViewMetadata #8608

Merged
merged 1 commit into from
Sep 21, 2023

Conversation

nastra
Copy link
Contributor

@nastra nastra commented Sep 21, 2023

No description provided.

@nastra nastra requested a review from rdblue September 21, 2023 10:26
@nastra nastra added this to the Iceberg 1.4.0 milestone Sep 21, 2023
@github-actions github-actions bot added the core label Sep 21, 2023
@@ -90,12 +90,20 @@ static void toJson(ViewMetadata metadata, JsonGenerator gen) throws IOException
gen.writeEndObject();
}

public static ViewMetadata fromJson(String metadataLocation, String json) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these new methods will be used later on by BaseViewOperations

// when associated with a metadata file, metadata must have no changes so that the metadata
// matches exactly what is in the metadata file, which does not store changes. metadata
// location with changes is inconsistent.
Preconditions.checkArgument(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't want to fail in the check() method but rather in build(). Error messaging is aligned with TableMetadata

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with placing this here, although I would note that there's no way to read changes using ViewMetadataParser, so that should always produce metadata with a location but no changes. That means even if we did the check in ViewMetadata, we wouldn't expect a problem.

@@ -176,6 +181,7 @@ private Builder(ViewMetadata base) {
this.currentVersionId = base.currentVersionId();
this.location = base.location();
this.uuid = base.uuid();
this.metadataLocation = null;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just want to note that we're deviating from TableMetadata behavior a little in this builder, since it relates to the metadataLocation. In TableMetadata.Builder, changes are accumulated across builder instances. Basically the changes get reset to the empty set when the new table metadata is written out to an object store and reloaded from that copy. In this builder, we accumulate a new set of changes in every builder and don't carry them through. The difference is that the table metadata needs to accumulate changes across multiple operations in a transaction, while but there are no transactions for view metadata because every changes affects only the view metadata object (not other data or metadata files).

@rdblue rdblue merged commit f574743 into apache:master Sep 21, 2023
47 checks passed
@nastra nastra deleted the view-metadata-file-location branch September 22, 2023 04:47
@nastra nastra self-assigned this Nov 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

2 participants