[docs] Replace examples of Hadoop catalog with JDBC catalog #11845
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -26,7 +26,11 @@ highlight some powerful features. You can learn more about Iceberg's Spark runti | |
| - [Writing Data to a Table](#writing-data-to-a-table) | ||
| - [Reading Data from a Table](#reading-data-from-a-table) | ||
| - [Adding A Catalog](#adding-a-catalog) | ||
| - [Configuring JDBC Catalog](#configuring-jdbc-catalog) | ||
| - [Configuring REST Catalog](#configuring-rest-catalog) | ||
| - [Next steps](#next-steps) | ||
| - [Adding Iceberg to Spark](#adding-iceberg-to-spark) | ||
| - [Learn More](#learn-more) | ||
|
|
||
| ### Docker-Compose | ||
|
|
||
|
|
@@ -269,42 +273,104 @@ To read a table, simply use the Iceberg table's name. | |
|
|
||
| ### Adding A Catalog | ||
|
|
||
|
There was a problem hiding this comment. weird that the guide already mention JDBC here, but the example is still hadoop |
||
| Apache Iceberg provides several catalog implementations to manage tables and enable SQL operations. | ||
| Catalogs are configured using properties under `spark.sql.catalog.(catalog_name)`. | ||
| You can configure different catalog types, such as JDBC, Hive Metastore, Glue, and REST, to manage Iceberg tables in Spark. | ||
|
|
||
| This guide covers the configuration of two popular catalog types: | ||
| * JDBC Catalog | ||
| * REST Catalog | ||
|
|
||
| To learn more, check out the [Catalog](docs/latest/spark-configuration.md#catalogs) page in the Spark section. | ||
|
|
||
| #### Configuring JDBC Catalog | ||
|
|
||
| The JDBC catalog stores Iceberg table metadata in a relational database. | ||
|
|
||
| This configuration creates a JDBC-based catalog named `local` for tables under `$PWD/warehouse` and adds support for Iceberg tables to Spark's built-in catalog. | ||
| The JDBC catalog uses file-based SQLite database as the backend. | ||
|
|
||
| === "CLI" | ||
|
|
||
| ```sh | ||
| spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }},org.xerial:sqlite-jdbc:3.46.1.3 \ | ||
|
There was a problem hiding this comment. taking on this extra dep since i dont see any iceberg specific package i can use. there is a |
||
| --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \ | ||
| --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \ | ||
| --conf spark.sql.catalog.spark_catalog.type=hive \ | ||
| --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \ | ||
| --conf spark.sql.catalog.local.type=jdbc \ | ||
| --conf spark.sql.catalog.local.uri=jdbc:sqlite:$PWD/iceberg_catalog_db.sqlite \ | ||
| --conf spark.sql.catalog.local.warehouse=$PWD/warehouse \ | ||
| --conf spark.sql.defaultCatalog=local | ||
| ``` | ||
|
|
||
| === "spark-defaults.conf" | ||
|
|
||
| ```sh | ||
| spark.jars.packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }},org.xerial:sqlite-jdbc:3.46.1.3 | ||
| spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions | ||
| spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog | ||
| spark.sql.catalog.spark_catalog.type hive | ||
| spark.sql.catalog.local org.apache.iceberg.spark.SparkCatalog | ||
| spark.sql.catalog.local.type jdbc | ||
| spark.sql.catalog.local.uri jdbc:sqlite:iceberg_catalog_db.sqlite | ||
| spark.sql.catalog.local.warehouse $PWD/warehouse | ||
|
There was a problem hiding this comment.
|
||
| spark.sql.defaultCatalog local | ||
| ``` | ||
|
|
||
| !!! note | ||
| If your Iceberg catalog is not set as the default catalog, you will have to switch to it by executing `USE local;` | ||
|
|
||
| #### Configuring REST Catalog | ||
|
|
||
| The REST catalog provides a language-agnostic way to manage Iceberg tables through a RESTful service. | ||
|
|
||
| This configuration creates a REST-based catalog named `rest` for tables under `s3://warehouse/` and adds support for Iceberg tables to Spark's built-in catalog. | ||
| The REST catalog uses the `apache/iceberg-rest-fixture` docker container from the `docker-compose.yml` above as the backend service with MinIO for S3-compatible storage. | ||
|
|
||
| === "CLI" | ||
|
|
||
| ```sh | ||
| spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }},org.xerial:sqlite-jdbc:3.46.1.3 \ | ||
| --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \ | ||
| --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \ | ||
| --conf spark.sql.catalog.spark_catalog.type=hive \ | ||
| --conf spark.sql.catalog.rest=org.apache.iceberg.spark.SparkCatalog \ | ||
| --conf spark.sql.catalog.rest.type=rest \ | ||
| --conf spark.sql.catalog.rest.uri=http://localhost:8181 \ | ||
| --conf spark.sql.catalog.rest.warehouse=s3://warehouse/ \ | ||
| --conf spark.sql.catalog.rest.io-impl=org.apache.iceberg.aws.s3.S3FileIO \ | ||
| --conf spark.sql.catalog.rest.s3.endpoint=http://localhost:9000 \ | ||
| --conf spark.sql.catalog.rest.s3.path-style-access=true \ | ||
| --conf spark.sql.catalog.rest.s3.access-key-id=admin \ | ||
| --conf spark.sql.catalog.rest.s3.secret-access-key=password \ | ||
| --conf spark.sql.catalog.rest.s3.region=us-east-1 \ | ||
| --conf spark.sql.defaultCatalog=rest | ||
| ``` | ||
|
|
||
| === "spark-defaults.conf" | ||
|
|
||
| ```sh | ||
| spark.jars.packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }} | ||
| spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions | ||
| spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog | ||
| spark.sql.catalog.spark_catalog.type hive | ||
| spark.sql.catalog.rest org.apache.iceberg.spark.SparkCatalog | ||
| spark.sql.catalog.rest.type rest | ||
| spark.sql.catalog.rest.uri http://localhost:8181 | ||
| spark.sql.catalog.rest.warehouse s3://warehouse/ | ||
| spark.sql.catalog.rest.io-impl org.apache.iceberg.aws.s3.S3FileIO | ||
| spark.sql.catalog.rest.s3.endpoint http://localhost:9000 | ||
| spark.sql.catalog.rest.s3.access-key-id admin | ||
| spark.sql.catalog.rest.s3.secret-access-key password | ||
| spark.sql.catalog.rest.s3.path-style-access true | ||
| spark.sql.catalog.rest.s3.region us-east-1 | ||
| spark.sql.defaultCatalog rest | ||
| ``` | ||
|
|
||
| !!! note | ||
| If your Iceberg catalog is not set as the default catalog, you will have to switch to it by executing `USE rest;` | ||
|
|
||
| ### Next steps | ||
|
|
||
| #### Adding Iceberg to Spark | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
note, there are two "getting started" docs
this one and
site/docs/spark-quickstart.md