Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Core: Fix negation in requireStrictCleanup #8675

Closed
wants to merge 1 commit into from

Conversation

Fokko
Copy link
Contributor

@Fokko Fokko commented Sep 28, 2023

Found a regression that the snapshots are still around after the commit fails: trinodb/trino#19188

io.trino.plugin.iceberg.catalog.file.TestIcebergFileMetastoreCreateTableFailure.testCreateTableFailureMetadataCleanedUp; (took: 0.0 seconds)
2023-09-28T18:24:02.9837549Z java.lang.AssertionError: [Metadata file should not exist] 
2023-09-28T18:24:02.9837847Z Expecting actual:
2023-09-28T18:24:02.9838206Z   /tmp/test_iceberg_create_table_failure482309508502886299/test_create_failure_4huakfy1pd/metadata
2023-09-28T18:24:02.9838580Z to be an empty directory but it contained:
2023-09-28T18:24:02.9839258Z   [/tmp/test_iceberg_create_table_failure482309508502886299/test_create_failure_4huakfy1pd/metadata/.snap-905595869249987831-1-7e5e14a9-79ec-449d-abf4-9cae0d693663.avro.crc,

This is a result of: #8599 It was actually enabled before, but now it is disabled.

After the fix, I checked locally, the test passes:

Found a regression that the snapshots are still around after the
commit fails: trinodb/trino#19188

After the fix, I checked locally, the test passes.
@github-actions github-actions bot added the core label Sep 28, 2023
@aokolnychyi
Copy link
Contributor

Let me check this one in a bit.

@aokolnychyi aokolnychyi added this to the Iceberg 1.4.0 milestone Sep 28, 2023
@@ -320,7 +320,7 @@ private void commitCreateTransaction() {

} catch (RuntimeException e) {
// the commit failed and no files were committed. clean up each update
if (!ops.requireStrictCleanup() || e instanceof CleanableFailure) {
Copy link
Contributor

@aokolnychyi aokolnychyi Sep 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, isn't the existing logic correct? We should not clean up in strict mode unless it is CleanableFailure? Is there a chance the test in Trino produced a commit error that we treat as unknown or unsafe to clean up?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the one that failed in Trino was testing SchemaNotFoundException here.

@Fokko
Copy link
Contributor Author

Fokko commented Sep 28, 2023

@aokolnychyi Thanks, I'm not 100% sure if it is a regression or intended. A test is failing that seems to make sense. I can take a look tomorrow with fresh eyes, but another set of eyes is very much welcome. Feel free to take over the PR.

@aokolnychyi
Copy link
Contributor

cc @rdblue @amogh-jahagirdar

@rdblue
Copy link
Contributor

rdblue commented Sep 28, 2023

I don't think this change is correct. The logic should continue to be to clean up if strict cleanup is not enabled.

However, there is a behavior change here that is probably being hit by this test. We changed behavior and enabled strict cleanup for tables unless it is turned off by the catalog. This may leak files, but it is much safer and avoids table corruption on unknown exceptions.

@aokolnychyi aokolnychyi removed this from the Iceberg 1.4.0 milestone Sep 28, 2023
@Fokko
Copy link
Contributor Author

Fokko commented Sep 29, 2023

I don't think this change is correct. The logic should continue to be to clean up if strict cleanup is not enabled.

I think it is a difference in understanding, if it is strict in cleaning up the files, I would expect it to clean it up. Anyway, good that we have the test and it is expected.

@Fokko Fokko closed this Sep 29, 2023
@Fokko Fokko deleted the fd-fix-regression branch September 29, 2023 05:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants