Add support for creating and dropping tables using Iceberg object store

[jakelong95]

Description

Currently, Trino is only able to write to and read from tables that use Iceberg's ObjectStorageLocationProvider, but is unable to create or drop tables using the location provider.

This PR enables Trino to create tables using Iceberg's object storage by adding the following properties:

• iceberg.object-store.enabled - Corresponds with Spark's write.object-storage.enabled, which enables use of Iceberg's ObjectStorageLocationProvider
• iceberg.data-location - Corresponds to Spark's write.data.path, which sets where Iceberg data files will be written

Enabling the object store property and setting the data path will cause Iceberg to provide data file locations prefixed by a deterministic hash generated from the file name in the location specified by write.data.path, which will help reduce throttling from cloud storage systems like S3 by evenly distributing files across multiple prefixes. Iceberg's own documentation on this feature has more information.

For example, without the object store enabled you would get the following locations for data files, all under the same prefix iceberg-tables/myschema/mytable/data:

• s3://mybucket/iceberg-tables/myschema/mytable/data/file1.parquet
• s3://mybucket/iceberg-tables/myschema/mytable/data/file2.parquet
• s3://mybucket/iceberg-tables/myschema/mytable/data/file3.parquet

But, if you enable the object store, you would get the following locations, each in their own prefix:

• s3://mybucket/iceberg-tables/myschema/mytable/data/<file1 hash>/file1.parquet
• s3://mybucket/iceberg-tables/myschema/mytable/data/<file2 hash>/file2.parquet
• s3://mybucket/iceberg-tables/myschema/mytable/data/<file3 hash>/file3.parquet

Additional context and related issues

This PR maintains compatibility with Spark by using the table properties write.object-storage.enabled and write.data.path, which had previously been set up to allow Trino to write to Iceberg tables using the ObjectStorageLocationProvider in #8573

Fixes #8861

Release notes

(x) Release notes are required, with the following suggested text:

# Iceberg
* Add support for [object store file layout](https://iceberg.apache.org/docs/latest/aws/#object-store-file-layout). ({issue}`8861`)

[cla-bot]

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

[jakelong95]

Re-created from #20516

[github-actions]

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

[mosabua]

Any thoughts @amogh-jahagirdar

[github-actions]

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

[github-actions]

Closing this pull request, as it has been stale for six weeks. Feel free to re-open at any time.

[mosabua]

I added the stale-ignore label so the PR stays open. Please rebase as a next step.

[jakelong95]

Hi @mosabua, pushed the rebase in. Looks like the failed check timed out but I'm not able to restart it.
I'm also not sure what the process is for bumping Trino PRs, should I re-post in the Trino slack?

[mosabua]

Please ask for help on the #iceberg or the #core-dev channel

[anusudarsan]

why was WRITE_METADATA_LOCATION removed?

[jakelong95]

Not sure I totally follow. I'm not removing the property, just its usage here since Trino will be able to drop tables created with Iceberg's object storage file layout with this change

[grantatspothero]

Adding support for updating this table property would be useful, see IcebergMetadata#setTableProperties

It is a common to create a table and only later encounter object storage rate limiting. Not allowing updates would make this workflow to fix rate limiting a pain.

Not sure if updates should go in this PR or another one @ebyhr but heads up.

[jakelong95]

Adding support for updating this table property would be useful, see IcebergMetadata#setTableProperties

It is a common to create a table and only later encounter object storage rate limiting. Not allowing updates would make this workflow to fix rate limiting a pain.

Not sure if updates should go in this PR or another one @ebyhr but heads up.

That's a good callout. I'll add it in this PR

[ebyhr]

That's a good callout. I'll add it in this PR

@jakelong95 I already push the code to support SET PROPERTIES.

[jakelong95]

That's a good callout. I'll add it in this PR

@jakelong95 I already push the code to support SET PROPERTIES.

Yeah, saw that you added adding in object_store_enabled. I'm adding in support for modifying data_location since the two properties are related

[jakelong95]

@ebyhr Just pushed up the following changes:

1. Ability to update data_location for a table after creation, and the associated documentation
2. Test cases for adding object storage or a data location after the table was created
3. Assertions for Spark <-> Trino compatibility with the object_store_enabled and data_location properties
4. Improved assertions in some of the tests in my original commit to better validate the data file location matches the expected format.
   • For example, one of the assertions matches on the regex local:///table-location/xyz/.{6}/tpch/test_create_and_drop.* - After the data location local:///table-location/xyz it has a regex match on 6 characters, which is what Iceberg uses for the deterministic hashes based on the file name. Just wanted to be extra sure :)

Let me know if you have any more questions or concerns!

[jakelong95]

@ebyhr fixed the failing unit test that I had added.

A new failure from one of the existing unit tests TestIcebergV2.testOptimizeDuringWriteOperations came up though. I'm unable to replicate this failure on my machine, and it doesn't look like any of my changes would have affected it.