Remove Iceberg materialized view storage tables from metastores #18853

alexjo2144 · 2023-08-29T20:49:43Z

Description

Storage tables clutter the namespace, and do not need a separate metastore entry. The pointer to the current metadata file is instead stored in the materialized view properties.

This applies only to newly created materialized views, existing ones are not effected.

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

# Iceberg
* Newly created Materialized Views will not create a separate entry in the metastore for a storage table.

findepi

(skimming)

plugin/trino-hive/src/main/java/io/trino/plugin/hive/ViewReaderUtil.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergConfig.java

...eberg/src/main/java/io/trino/plugin/iceberg/IcebergMaterializedViewAdditionalProperties.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergTableName.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergUtil.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/MaterializedViewTableOperations.java

...no-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/AbstractIcebergTableOperations.java

...o-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/glue/GlueIcebergTableOperations.java

alexjo2144 · 2023-09-06T21:08:30Z

@findepi comments applied, PTAL when you get a chance.

alexjo2144 · 2023-09-07T14:05:54Z

A test with secrets would also be great

findepi · 2023-09-11T08:07:26Z

/test-with-secrets sha=f9840979650b7774cbe5383eb8c31e9e4aba8a58

github-actions · 2023-09-11T08:21:30Z

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/6143712396

findepi · 2023-09-12T13:49:01Z

failure looks related

alexjo2144 · 2023-09-12T16:29:34Z

failure looks related

Indeed

findepi · 2023-09-13T15:23:40Z

/test-with-secrets sha=b441b7fc458c58754ab0b8607dd488a537325f7d

github-actions · 2023-09-13T18:47:45Z

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/6175817369

alexjo2144 · 2023-09-14T01:47:40Z

Failure on the run with secrets was in the databricks delta lake suite, unrelated.

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergConfig.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/TableType.java

...iceberg/src/main/java/io/trino/plugin/iceberg/catalog/file/FileMetastoreTableOperations.java

...no-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/AbstractIcebergTableOperations.java

...o-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/glue/GlueIcebergTableOperations.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/AbstractTrinoCatalog.java

...rino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergMetastoreAccessOperations.java

findinpath · 2023-09-18T11:54:11Z

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergConfig.java

+        return hideMaterializedViewStorageTableEnabled;
+    }
+
+    @Config("iceberg.materialized-views.hide-storage-tables")


Please add a test using iceberg.materialized-views.hide-storage-tables=false.
We need to ensure that Trino can still read/refresh this kind of materialized views.

talked offline that this can be a follow-up.

findepi · 2023-10-30T15:26:07Z

(rebased)

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/AbstractTrinoCatalog.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/glue/TrinoGlueCatalog.java

findepi · 2023-11-08T16:32:12Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/BaseIcebergConnectorTest.java

@@ -6701,30 +6701,6 @@ public void testSnapshotSummariesHaveTrinoQueryIdFormatV2()
                "WHEN MATCHED THEN UPDATE SET b = t.b * 50", tableName, sourceTableName)));
    }

-    @Test
-    public void testMaterializedViewSnapshotSummariesHaveTrinoQueryId()


Why this change?

findepi · 2023-11-08T16:38:14Z

/test-with-secrets sha=28892f73b45825172bf0fa552a832753466a602e

github-actions · 2023-11-08T16:41:47Z

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/6801273645

The FILE_FORMAT_PROPERTY table property has a default value, so no need for explicit default in the code.

The helper method always returns a value

Storage tables clutter the namespace, and do not need a separate metastore entry. The pointer to the current metadata file is instead stored in the materialized view properties. This applies only to newly created materialized views, existing ones are not effected.

mosabua · 2023-11-10T17:29:43Z

Modified the release notes entry and added.

wobrycki · 2024-01-11T22:42:23Z

Hi @alexjo2144,
are the materialized view storage tables still accessible as iceberg tables as it was before?
Previously we had a case, when storage table (with name starting from st_) was referenced from spark, connected to the same iceberg configuration as trino.
For some reason we choose Spark for certain operations on iceberg tables and we would like to continue doing it.

Now, system.metadata.materiazed_views is saying the storage table name is view_name$materialized_view_storage and spark ends with error: pyspark.sql.utils.AnalysisException: Table or view not found when trying to access the storage table.

alexjo2144 · 2024-01-16T18:34:16Z

Hey @wobrycki

With the new behavior, no you won't be able to do that. I didn't think reading MV storage tables from Spark would be something anyone was doing. How do you figure out which storage table belongs to each MV? Or do you just list tables and then call a procedure like rewrite_data_files in Spark for each one?

For now, you can return to the previous behavior by setting iceberg.materialized-views.hide-storage-table=false in your Iceberg config.

On the other hand, if it's file compaction (rewrite_data_files) you are worried about, it was likely not doing much for MV storage tables in the first place. Since MVs are fully refreshed each time they are updated, the data layout should be pretty compact already.

github-actions bot added tests:hive iceberg Iceberg connector hive Hive connector labels Aug 29, 2023

cla-bot bot added the cla-signed label Aug 29, 2023

alexjo2144 requested a review from findepi August 29, 2023 20:57

findepi reviewed Aug 30, 2023

View reviewed changes

alexjo2144 force-pushed the iceberg/mv-storage-table branch 5 times, most recently from 988550d to f984097 Compare September 6, 2023 21:07

alexjo2144 requested review from raunaqmorarka and sopel39 September 7, 2023 14:40

alexjo2144 force-pushed the iceberg/mv-storage-table branch from ed9a0b0 to b441b7f Compare September 13, 2023 15:16

findinpath reviewed Sep 18, 2023

View reviewed changes

findepi mentioned this pull request Sep 29, 2023

Add ConnectorMetadata#isMaterializedView #18933

Closed

findepi force-pushed the iceberg/mv-storage-table branch from b441b7f to 5364952 Compare October 30, 2023 15:26

sopel39 removed their request for review October 31, 2023 13:00

findepi force-pushed the iceberg/mv-storage-table branch 2 times, most recently from 110cf6e to 8030954 Compare November 7, 2023 15:01

findepi reviewed Nov 7, 2023

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/AbstractTrinoCatalog.java Outdated Show resolved Hide resolved

findepi force-pushed the iceberg/mv-storage-table branch 2 times, most recently from 69c5447 to 9759483 Compare November 8, 2023 10:24

findepi reviewed Nov 8, 2023

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/AbstractTrinoCatalog.java Outdated Show resolved Hide resolved

findepi reviewed Nov 8, 2023

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/glue/TrinoGlueCatalog.java Show resolved Hide resolved

Accept Location in ForwardingOutputFile constructor

83febeb

findepi reviewed Nov 8, 2023

View reviewed changes

findepi force-pushed the iceberg/mv-storage-table branch 2 times, most recently from 01f0c4f to 28892f7 Compare November 8, 2023 16:37

findepi added 3 commits November 9, 2023 13:11

Remove redundant TableOperations construction

e24faef

Remove redundant property defaulting for MV table creation

13e292b

The FILE_FORMAT_PROPERTY table property has a default value, so no need for explicit default in the code.

Remove redundant Optional return type

1c12834

The helper method always returns a value

findepi force-pushed the iceberg/mv-storage-table branch from 28892f7 to 1457de6 Compare November 9, 2023 12:16

findepi force-pushed the iceberg/mv-storage-table branch from 1457de6 to 5293ddb Compare November 9, 2023 12:16

findepi merged commit f937d31 into trinodb:master Nov 10, 2023
50 of 52 checks passed

github-actions bot added this to the 433 milestone Nov 10, 2023

mosabua mentioned this pull request Nov 10, 2023

Add Trino 433 release notes #19660

Merged

alexjo2144 deleted the iceberg/mv-storage-table branch November 27, 2023 16:44

This was referenced May 7, 2024

Support materilized view rename across schemas in iceberg connector #9594

Open

Cannot drop Iceberg materialized view after storage table externally dropped #12503

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove Iceberg materialized view storage tables from metastores #18853

Remove Iceberg materialized view storage tables from metastores #18853

alexjo2144 commented Aug 29, 2023

findepi left a comment

alexjo2144 commented Sep 6, 2023

alexjo2144 commented Sep 7, 2023

findepi commented Sep 11, 2023

github-actions bot commented Sep 11, 2023 •

edited

Loading

findepi commented Sep 12, 2023

alexjo2144 commented Sep 12, 2023

findepi commented Sep 13, 2023

github-actions bot commented Sep 13, 2023 •

edited

Loading

alexjo2144 commented Sep 14, 2023

findinpath Sep 18, 2023

findepi Nov 10, 2023

findepi commented Oct 30, 2023

findepi Nov 8, 2023

findepi commented Nov 8, 2023

github-actions bot commented Nov 8, 2023 •

edited

Loading

mosabua commented Nov 10, 2023

wobrycki commented Jan 11, 2024

alexjo2144 commented Jan 16, 2024

Remove Iceberg materialized view storage tables from metastores #18853

Remove Iceberg materialized view storage tables from metastores #18853

Conversation

alexjo2144 commented Aug 29, 2023

Description

Additional context and related issues

Release notes

findepi left a comment

Choose a reason for hiding this comment

alexjo2144 commented Sep 6, 2023

alexjo2144 commented Sep 7, 2023

findepi commented Sep 11, 2023

github-actions bot commented Sep 11, 2023 • edited Loading

findepi commented Sep 12, 2023

alexjo2144 commented Sep 12, 2023

findepi commented Sep 13, 2023

github-actions bot commented Sep 13, 2023 • edited Loading

alexjo2144 commented Sep 14, 2023

findinpath Sep 18, 2023

Choose a reason for hiding this comment

findepi Nov 10, 2023

Choose a reason for hiding this comment

findepi commented Oct 30, 2023

findepi Nov 8, 2023

Choose a reason for hiding this comment

findepi commented Nov 8, 2023

github-actions bot commented Nov 8, 2023 • edited Loading

mosabua commented Nov 10, 2023

wobrycki commented Jan 11, 2024

alexjo2144 commented Jan 16, 2024

github-actions bot commented Sep 11, 2023 •

edited

Loading

github-actions bot commented Sep 13, 2023 •

edited

Loading

github-actions bot commented Nov 8, 2023 •

edited

Loading