| Options to be used for actions|
| `where` | ️ | string | predicate as a string used for filtering the files. Note that all files that may contain data matching the filter will be selected for rewriting|
diff --git a/site/docs/docs/nightly/docs/spark-queries.md b/site/docs/docs/nightly/docs/spark-queries.md
index 9198538f5a73..2616687cccd0 100644
--- a/site/docs/docs/nightly/docs/spark-queries.md
+++ b/site/docs/docs/nightly/docs/spark-queries.md
@@ -188,9 +188,6 @@ To inspect a table's history, snapshots, and other metadata, Iceberg supports me
Metadata tables are identified by adding the metadata table name after the original table name. For example, history for `db.table` is read using `db.table.history`.
-!!! info
- For Spark 3, prior to 3.2, the Spark [session catalog](spark-configuration.md#replacing-the-session-catalog) does not support table names with multipart identifiers such as `catalog.database.table.metadata`. As a workaround, configure an `org.apache.iceberg.spark.SparkCatalog`, or use the Spark `DataFrameReader` API.
-
### History
diff --git a/site/docs/multi-engine-support.md b/site/docs/multi-engine-support.md
index 75477ffc7106..20fd10616754 100644
--- a/site/docs/multi-engine-support.md
+++ b/site/docs/multi-engine-support.md
@@ -29,8 +29,8 @@ Connectors for Spark, Flink and Hive are maintained in the main Iceberg reposito
Processing engine connectors maintained in the iceberg repository are built for multiple versions.
For Spark and Flink, each new version that introduces backwards incompatible upgrade has its dedicated integration codebase and release artifacts.
-For example, the code for Iceberg Spark 3.1 integration is under `/spark/v3.1` and the code for Iceberg Spark 3.2 integration is under `/spark/v3.2`.
-Different artifacts (`iceberg-spark-3.1_2.12` and `iceberg-spark-3.2_2.12`) are released for users to consume.
+For example, the code for Iceberg Spark 3.4 integration is under `/spark/v3.4` and the code for Iceberg Spark 3.5 integration is under `/spark/v3.5`.
+Different artifacts (`iceberg-spark-3.4_2.12` and `iceberg-spark-3.5_2.12`) are released for users to consume.
By doing this, changes across versions are isolated.
New features in Iceberg could be developed against the latest features of an engine without breaking support of old APIs in past engine versions.
@@ -40,7 +40,7 @@ For Hive, Hive 2 uses the `iceberg-mr` package for Iceberg integration, and Hive
Iceberg provides a runtime connector jar for each supported version of Spark, Flink and Hive.
When using Iceberg with these engines, the runtime jar is the only addition to the classpath needed in addition to vendor dependencies.
-For example, to use Iceberg with Spark 3.2 and AWS integrations, `iceberg-spark-runtime-3.2_2.12` and AWS SDK dependencies are needed for the Spark installation.
+For example, to use Iceberg with Spark 3.5 and AWS integrations, `iceberg-spark-runtime-3.5_2.12` and AWS SDK dependencies are needed for the Spark installation.
Spark and Flink provide different runtime jars for each supported engine version.
Hive 2 and Hive 3 currently share the same runtime jar.
diff --git a/site/docs/spark-quickstart.md b/site/docs/spark-quickstart.md
index cd5efdb44c00..9601bcbdb0f8 100644
--- a/site/docs/spark-quickstart.md
+++ b/site/docs/spark-quickstart.md
@@ -281,7 +281,7 @@ This configuration creates a path-based catalog named `local` for tables under `
=== "CLI"
```sh
- spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.2_2.12:{{ icebergVersion }}\
+ spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }}\
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
--conf spark.sql.catalog.spark_catalog.type=hive \
@@ -294,7 +294,7 @@ This configuration creates a path-based catalog named `local` for tables under `
=== "spark-defaults.conf"
```sh
- spark.jars.packages org.apache.iceberg:iceberg-spark-runtime-3.2_2.12:{{ icebergVersion }}
+ spark.jars.packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }}
spark.sql.extensions org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.spark_catalog org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.catalog.spark_catalog.type hive
@@ -316,26 +316,26 @@ If you already have a Spark environment, you can add Iceberg, using the `--packa
=== "SparkSQL"
```sh
- spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.2_2.12:{{ icebergVersion }}
+ spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }}
```
=== "Spark-Shell"
```sh
- spark-shell --packages org.apache.iceberg:iceberg-spark-runtime-3.2_2.12:{{ icebergVersion }}
+ spark-shell --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }}
```
=== "PySpark"
```sh
- pyspark --packages org.apache.iceberg:iceberg-spark-runtime-3.2_2.12:{{ icebergVersion }}
+ pyspark --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:{{ icebergVersion }}
```
!!! note
If you want to include Iceberg in your Spark installation, add the Iceberg Spark runtime to Spark's `jars` folder.
You can download the runtime by visiting to the [Releases](releases.md) page.
-[spark-runtime-jar]: https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-spark-runtime-3.2_2.12/{{ icebergVersion }}/iceberg-spark-runtime-3.2_2.12-{{ icebergVersion }}.jar
+[spark-runtime-jar]: https://search.maven.org/remotecontent?filepath=org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/{{ icebergVersion }}/iceberg-spark-runtime-3.5_2.12-{{ icebergVersion }}.jar
#### Learn More
diff --git a/spark/build.gradle b/spark/build.gradle
index 736c47e335ff..c2bc5f8a14ed 100644
--- a/spark/build.gradle
+++ b/spark/build.gradle
@@ -20,10 +20,6 @@
// add enabled Spark version modules to the build
def sparkVersions = (System.getProperty("sparkVersions") != null ? System.getProperty("sparkVersions") : System.getProperty("defaultSparkVersions")).split(",")
-if (sparkVersions.contains("3.2")) {
- apply from: file("$projectDir/v3.2/build.gradle")
-}
-
if (sparkVersions.contains("3.3")) {
apply from: file("$projectDir/v3.3/build.gradle")
}
diff --git a/spark/v3.2/build.gradle b/spark/v3.2/build.gradle
deleted file mode 100644
index d2e2ceaf266d..000000000000
--- a/spark/v3.2/build.gradle
+++ /dev/null
@@ -1,295 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-String sparkMajorVersion = '3.2'
-String scalaVersion = System.getProperty("scalaVersion") != null ? System.getProperty("scalaVersion") : System.getProperty("defaultScalaVersion")
-
-def sparkProjects = [
- project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}"),
- project(":iceberg-spark:iceberg-spark-extensions-${sparkMajorVersion}_${scalaVersion}"),
- project(":iceberg-spark:iceberg-spark-runtime-${sparkMajorVersion}_${scalaVersion}"),
-]
-
-configure(sparkProjects) {
- configurations {
- all {
- resolutionStrategy {
- force "com.fasterxml.jackson.module:jackson-module-scala_${scalaVersion}:${libs.versions.jackson212.get()}"
- force "com.fasterxml.jackson.core:jackson-paranamer:${libs.versions.jackson212.get()}"
- force "com.fasterxml.jackson.core:jackson-databind:${libs.versions.jackson212.get()}"
- force "com.fasterxml.jackson.core:jackson-core:${libs.versions.jackson212.get()}"
- }
- }
- }
-}
-
-project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}") {
- apply plugin: 'scala'
- apply plugin: 'com.github.alisiikh.scalastyle'
-
- sourceSets {
- main {
- scala.srcDirs = ['src/main/scala', 'src/main/java']
- java.srcDirs = []
- }
- }
-
- dependencies {
- implementation project(path: ':iceberg-bundled-guava', configuration: 'shadow')
- api project(':iceberg-api')
- implementation project(':iceberg-common')
- implementation project(':iceberg-core')
- implementation project(':iceberg-data')
- implementation project(':iceberg-orc')
- implementation project(':iceberg-parquet')
- implementation(project(':iceberg-arrow')) {
- exclude group: 'io.netty', module: 'netty-buffer'
- exclude group: 'io.netty', module: 'netty-common'
- }
- implementation "org.scala-lang.modules:scala-collection-compat_${scalaVersion}:${libs.versions.scala.collection.compat.get()}"
-
- compileOnly libs.errorprone.annotations
- compileOnly libs.avro.avro
- compileOnly("org.apache.spark:spark-hive_${scalaVersion}:${libs.versions.spark.hive32.get()}") {
- exclude group: 'org.apache.avro', module: 'avro'
- exclude group: 'org.apache.arrow'
- exclude group: 'org.apache.parquet'
- exclude group: 'io.netty', module: 'netty-buffer'
- exclude group: 'io.netty', module: 'netty-common'
- exclude group: 'org.roaringbitmap'
- }
-
- implementation libs.parquet.column
- implementation libs.parquet.hadoop
-
- implementation("${libs.orc.core.get().module}:${libs.versions.orc.get()}:nohive") {
- exclude group: 'org.apache.hadoop'
- exclude group: 'commons-lang'
- // These artifacts are shaded and included in the orc-core fat jar
- exclude group: 'com.google.protobuf', module: 'protobuf-java'
- exclude group: 'org.apache.hive', module: 'hive-storage-api'
- }
-
- implementation(libs.arrow.vector) {
- exclude group: 'io.netty', module: 'netty-buffer'
- exclude group: 'io.netty', module: 'netty-common'
- exclude group: 'com.google.code.findbugs', module: 'jsr305'
- }
-
- // use netty-buffer compatible with Spark 3.2
- runtimeOnly libs.netty.buffer.compat
-
- testImplementation(libs.hadoop2.minicluster) {
- exclude group: 'org.apache.avro', module: 'avro'
- exclude group: 'io.netty', module: 'netty-buffer'
- exclude group: 'io.netty', module: 'netty-common'
- }
- testImplementation project(path: ':iceberg-hive-metastore')
- testImplementation project(path: ':iceberg-hive-metastore', configuration: 'testArtifacts')
- testImplementation project(path: ':iceberg-api', configuration: 'testArtifacts')
- testImplementation project(path: ':iceberg-core', configuration: 'testArtifacts')
- testImplementation project(path: ':iceberg-data', configuration: 'testArtifacts')
- testImplementation libs.sqlite.jdbc
- testImplementation libs.awaitility
- }
-
- tasks.withType(Test) {
- // Vectorized reads need more memory
- maxHeapSize '2560m'
- }
-}
-
-project(":iceberg-spark:iceberg-spark-extensions-${sparkMajorVersion}_${scalaVersion}") {
- apply plugin: 'java-library'
- apply plugin: 'scala'
- apply plugin: 'com.github.alisiikh.scalastyle'
- apply plugin: 'antlr'
-
- configurations {
- /*
- The Gradle Antlr plugin erroneously adds both antlr-build and runtime dependencies to the runtime path. This
- bug https://github.com/gradle/gradle/issues/820 exists because older versions of Antlr do not have separate
- runtime and implementation dependencies and they do not want to break backwards compatibility. So to only end up with
- the runtime dependency on the runtime classpath we remove the dependencies added by the plugin here. Then add
- the runtime dependency back to only the runtime configuration manually.
- */
- implementation {
- extendsFrom = extendsFrom.findAll { it != configurations.antlr }
- }
- }
-
- dependencies {
- implementation("org.scala-lang.modules:scala-collection-compat_${scalaVersion}:${libs.versions.scala.collection.compat.get()}")
-
- compileOnly "org.scala-lang:scala-library"
- compileOnly project(path: ':iceberg-bundled-guava', configuration: 'shadow')
- compileOnly project(':iceberg-api')
- compileOnly project(':iceberg-core')
- compileOnly project(':iceberg-common')
- compileOnly project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}")
- compileOnly("org.apache.spark:spark-hive_${scalaVersion}:${libs.versions.spark.hive32.get()}") {
- exclude group: 'org.apache.avro', module: 'avro'
- exclude group: 'org.apache.arrow'
- exclude group: 'org.apache.parquet'
- exclude group: 'io.netty', module: 'netty-buffer'
- exclude group: 'io.netty', module: 'netty-common'
- exclude group: 'org.roaringbitmap'
- }
-
- testImplementation project(path: ':iceberg-data')
- testImplementation project(path: ':iceberg-parquet')
- testImplementation project(path: ':iceberg-hive-metastore')
- testImplementation project(path: ':iceberg-api', configuration: 'testArtifacts')
- testImplementation project(path: ':iceberg-hive-metastore', configuration: 'testArtifacts')
- testImplementation project(path: ":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}", configuration: 'testArtifacts')
-
- testImplementation libs.avro.avro
- testImplementation libs.parquet.hadoop
-
- // Required because we remove antlr plugin dependencies from the compile configuration, see note above
- runtimeOnly libs.antlr.runtime
- antlr libs.antlr.antlr4
- }
-
- generateGrammarSource {
- maxHeapSize = "64m"
- arguments += ['-visitor', '-package', 'org.apache.spark.sql.catalyst.parser.extensions']
- }
-}
-
-project(":iceberg-spark:iceberg-spark-runtime-${sparkMajorVersion}_${scalaVersion}") {
- apply plugin: 'com.github.johnrengelman.shadow'
-
- tasks.jar.dependsOn tasks.shadowJar
-
- sourceSets {
- integration {
- java.srcDir "$projectDir/src/integration/java"
- resources.srcDir "$projectDir/src/integration/resources"
- }
- }
-
- configurations {
- implementation {
- exclude group: 'org.apache.spark'
- // included in Spark
- exclude group: 'org.slf4j'
- exclude group: 'org.apache.commons'
- exclude group: 'commons-pool'
- exclude group: 'commons-codec'
- exclude group: 'org.xerial.snappy'
- exclude group: 'javax.xml.bind'
- exclude group: 'javax.annotation'
- exclude group: 'com.github.luben'
- exclude group: 'com.ibm.icu'
- exclude group: 'org.glassfish'
- exclude group: 'org.abego.treelayout'
- exclude group: 'org.antlr'
- exclude group: 'org.scala-lang'
- exclude group: 'org.scala-lang.modules'
- }
- }
-
- dependencies {
- api project(':iceberg-api')
- implementation project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}")
- implementation project(":iceberg-spark:iceberg-spark-extensions-${sparkMajorVersion}_${scalaVersion}")
- implementation project(':iceberg-aws')
- implementation project(':iceberg-azure')
- implementation(project(':iceberg-aliyun')) {
- exclude group: 'edu.umd.cs.findbugs', module: 'findbugs'
- exclude group: 'org.apache.httpcomponents', module: 'httpclient'
- exclude group: 'commons-logging', module: 'commons-logging'
- }
- implementation project(':iceberg-gcp')
- implementation project(':iceberg-hive-metastore')
- implementation(project(':iceberg-nessie')) {
- exclude group: 'com.google.code.findbugs', module: 'jsr305'
- }
- implementation (project(':iceberg-snowflake')) {
- exclude group: 'net.snowflake' , module: 'snowflake-jdbc'
- }
-
- integrationImplementation "org.scala-lang.modules:scala-collection-compat_${scalaVersion}:${libs.versions.scala.collection.compat.get()}"
- integrationImplementation "org.apache.spark:spark-hive_${scalaVersion}:${libs.versions.spark.hive32.get()}"
- integrationImplementation libs.junit.vintage.engine
- integrationImplementation libs.slf4j.simple
- integrationImplementation libs.assertj.core
- integrationImplementation project(path: ':iceberg-api', configuration: 'testArtifacts')
- integrationImplementation project(path: ':iceberg-hive-metastore', configuration: 'testArtifacts')
- integrationImplementation project(path: ":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}", configuration: 'testArtifacts')
- integrationImplementation project(path: ":iceberg-spark:iceberg-spark-extensions-${sparkMajorVersion}_${scalaVersion}", configuration: 'testArtifacts')
- // Not allowed on our classpath, only the runtime jar is allowed
- integrationCompileOnly project(":iceberg-spark:iceberg-spark-extensions-${sparkMajorVersion}_${scalaVersion}")
- integrationCompileOnly project(":iceberg-spark:iceberg-spark-${sparkMajorVersion}_${scalaVersion}")
- integrationCompileOnly project(':iceberg-api')
- }
-
- shadowJar {
- configurations = [project.configurations.runtimeClasspath]
-
- zip64 true
-
- // include the LICENSE and NOTICE files for the shaded Jar
- from(projectDir) {
- include 'LICENSE'
- include 'NOTICE'
- }
-
- // Relocate dependencies to avoid conflicts
- relocate 'com.google.errorprone', 'org.apache.iceberg.shaded.com.google.errorprone'
- relocate 'com.google.flatbuffers', 'org.apache.iceberg.shaded.com.google.flatbuffers'
- relocate 'com.fasterxml', 'org.apache.iceberg.shaded.com.fasterxml'
- relocate 'com.github.benmanes', 'org.apache.iceberg.shaded.com.github.benmanes'
- relocate 'org.checkerframework', 'org.apache.iceberg.shaded.org.checkerframework'
- relocate 'org.apache.avro', 'org.apache.iceberg.shaded.org.apache.avro'
- relocate 'avro.shaded', 'org.apache.iceberg.shaded.org.apache.avro.shaded'
- relocate 'com.thoughtworks.paranamer', 'org.apache.iceberg.shaded.com.thoughtworks.paranamer'
- relocate 'org.apache.parquet', 'org.apache.iceberg.shaded.org.apache.parquet'
- relocate 'shaded.parquet', 'org.apache.iceberg.shaded.org.apache.parquet.shaded'
- relocate 'org.apache.orc', 'org.apache.iceberg.shaded.org.apache.orc'
- relocate 'io.airlift', 'org.apache.iceberg.shaded.io.airlift'
- relocate 'org.apache.hc.client5', 'org.apache.iceberg.shaded.org.apache.hc.client5'
- relocate 'org.apache.hc.core5', 'org.apache.iceberg.shaded.org.apache.hc.core5'
- // relocate Arrow and related deps to shade Iceberg specific version
- relocate 'io.netty', 'org.apache.iceberg.shaded.io.netty'
- relocate 'org.apache.arrow', 'org.apache.iceberg.shaded.org.apache.arrow'
- relocate 'com.carrotsearch', 'org.apache.iceberg.shaded.com.carrotsearch'
- relocate 'org.threeten.extra', 'org.apache.iceberg.shaded.org.threeten.extra'
- relocate 'org.roaringbitmap', 'org.apache.iceberg.shaded.org.roaringbitmap'
-
- archiveClassifier.set(null)
- }
-
- task integrationTest(type: Test) {
- description = "Test Spark3 Runtime Jar against Spark ${sparkMajorVersion}"
- group = "verification"
- jvmArgs += project.property('extraJvmArgs')
- testClassesDirs = sourceSets.integration.output.classesDirs
- classpath = sourceSets.integration.runtimeClasspath + files(shadowJar.archiveFile.get().asFile.path)
- inputs.file(shadowJar.archiveFile.get().asFile.path)
- }
- integrationTest.dependsOn shadowJar
- check.dependsOn integrationTest
-
- jar {
- enabled = false
- }
-}
-
diff --git a/spark/v3.2/spark-extensions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4 b/spark/v3.2/spark-extensions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4
deleted file mode 100644
index 7bd556acc5aa..000000000000
--- a/spark/v3.2/spark-extensions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4
+++ /dev/null
@@ -1,374 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- *
- * This file is an adaptation of Presto's and Spark's grammar files.
- */
-
-grammar IcebergSqlExtensions;
-
-@lexer::members {
- /**
- * Verify whether current token is a valid decimal token (which contains dot).
- * Returns true if the character that follows the token is not a digit or letter or underscore.
- *
- * For example:
- * For char stream "2.3", "2." is not a valid decimal token, because it is followed by digit '3'.
- * For char stream "2.3_", "2.3" is not a valid decimal token, because it is followed by '_'.
- * For char stream "2.3W", "2.3" is not a valid decimal token, because it is followed by 'W'.
- * For char stream "12.0D 34.E2+0.12 " 12.0D is a valid decimal token because it is followed
- * by a space. 34.E2 is a valid decimal token because it is followed by symbol '+'
- * which is not a digit or letter or underscore.
- */
- public boolean isValidDecimal() {
- int nextChar = _input.LA(1);
- if (nextChar >= 'A' && nextChar <= 'Z' || nextChar >= '0' && nextChar <= '9' ||
- nextChar == '_') {
- return false;
- } else {
- return true;
- }
- }
-
- /**
- * This method will be called when we see '/*' and try to match it as a bracketed comment.
- * If the next character is '+', it should be parsed as hint later, and we cannot match
- * it as a bracketed comment.
- *
- * Returns true if the next character is '+'.
- */
- public boolean isHint() {
- int nextChar = _input.LA(1);
- if (nextChar == '+') {
- return true;
- } else {
- return false;
- }
- }
-}
-
-singleStatement
- : statement EOF
- ;
-
-statement
- : CALL multipartIdentifier '(' (callArgument (',' callArgument)*)? ')' #call
- | ALTER TABLE multipartIdentifier ADD PARTITION FIELD transform (AS name=identifier)? #addPartitionField
- | ALTER TABLE multipartIdentifier DROP PARTITION FIELD transform #dropPartitionField
- | ALTER TABLE multipartIdentifier REPLACE PARTITION FIELD transform WITH transform (AS name=identifier)? #replacePartitionField
- | ALTER TABLE multipartIdentifier WRITE writeSpec #setWriteDistributionAndOrdering
- | ALTER TABLE multipartIdentifier SET IDENTIFIER_KW FIELDS fieldList #setIdentifierFields
- | ALTER TABLE multipartIdentifier DROP IDENTIFIER_KW FIELDS fieldList #dropIdentifierFields
- | ALTER TABLE multipartIdentifier createReplaceBranchClause #createOrReplaceBranch
- | ALTER TABLE multipartIdentifier createReplaceTagClause #createOrReplaceTag
- | ALTER TABLE multipartIdentifier DROP BRANCH (IF EXISTS)? identifier #dropBranch
- | ALTER TABLE multipartIdentifier DROP TAG (IF EXISTS)? identifier #dropTag
- ;
-
-createReplaceTagClause
- : (CREATE OR)? REPLACE TAG identifier tagOptions
- | CREATE TAG (IF NOT EXISTS)? identifier tagOptions
- ;
-
-createReplaceBranchClause
- : (CREATE OR)? REPLACE BRANCH identifier branchOptions
- | CREATE BRANCH (IF NOT EXISTS)? identifier branchOptions
- ;
-
-tagOptions
- : (AS OF VERSION snapshotId)? (refRetain)?
- ;
-
-branchOptions
- : (AS OF VERSION snapshotId)? (refRetain)? (snapshotRetention)?
- ;
-
-snapshotRetention
- : WITH SNAPSHOT RETENTION minSnapshotsToKeep
- | WITH SNAPSHOT RETENTION maxSnapshotAge
- | WITH SNAPSHOT RETENTION minSnapshotsToKeep maxSnapshotAge
- ;
-
-refRetain
- : RETAIN number timeUnit
- ;
-
-maxSnapshotAge
- : number timeUnit
- ;
-
-minSnapshotsToKeep
- : number SNAPSHOTS
- ;
-
-writeSpec
- : (writeDistributionSpec | writeOrderingSpec)*
- ;
-
-writeDistributionSpec
- : DISTRIBUTED BY PARTITION
- ;
-
-writeOrderingSpec
- : LOCALLY? ORDERED BY order
- | UNORDERED
- ;
-
-callArgument
- : expression #positionalArgument
- | identifier '=>' expression #namedArgument
- ;
-
-singleOrder
- : order EOF
- ;
-
-order
- : fields+=orderField (',' fields+=orderField)*
- | '(' fields+=orderField (',' fields+=orderField)* ')'
- ;
-
-orderField
- : transform direction=(ASC | DESC)? (NULLS nullOrder=(FIRST | LAST))?
- ;
-
-transform
- : multipartIdentifier #identityTransform
- | transformName=identifier
- '(' arguments+=transformArgument (',' arguments+=transformArgument)* ')' #applyTransform
- ;
-
-transformArgument
- : multipartIdentifier
- | constant
- ;
-
-expression
- : constant
- | stringMap
- | stringArray
- ;
-
-constant
- : number #numericLiteral
- | booleanValue #booleanLiteral
- | STRING+ #stringLiteral
- | identifier STRING #typeConstructor
- ;
-
-stringMap
- : MAP '(' constant (',' constant)* ')'
- ;
-
-stringArray
- : ARRAY '(' constant (',' constant)* ')'
- ;
-
-booleanValue
- : TRUE | FALSE
- ;
-
-number
- : MINUS? EXPONENT_VALUE #exponentLiteral
- | MINUS? DECIMAL_VALUE #decimalLiteral
- | MINUS? INTEGER_VALUE #integerLiteral
- | MINUS? BIGINT_LITERAL #bigIntLiteral
- | MINUS? SMALLINT_LITERAL #smallIntLiteral
- | MINUS? TINYINT_LITERAL #tinyIntLiteral
- | MINUS? DOUBLE_LITERAL #doubleLiteral
- | MINUS? FLOAT_LITERAL #floatLiteral
- | MINUS? BIGDECIMAL_LITERAL #bigDecimalLiteral
- ;
-
-multipartIdentifier
- : parts+=identifier ('.' parts+=identifier)*
- ;
-
-identifier
- : IDENTIFIER #unquotedIdentifier
- | quotedIdentifier #quotedIdentifierAlternative
- | nonReserved #unquotedIdentifier
- ;
-
-quotedIdentifier
- : BACKQUOTED_IDENTIFIER
- ;
-
-fieldList
- : fields+=multipartIdentifier (',' fields+=multipartIdentifier)*
- ;
-
-nonReserved
- : ADD | ALTER | AS | ASC | BRANCH | BY | CALL | CREATE | DAYS | DESC | DROP | EXISTS | FIELD | FIRST | HOURS | IF | LAST | NOT | NULLS | OF | OR | ORDERED | PARTITION | TABLE | WRITE
- | DISTRIBUTED | LOCALLY | MINUTES | MONTHS | UNORDERED | REPLACE | RETAIN | VERSION | WITH | IDENTIFIER_KW | FIELDS | SET | SNAPSHOT | SNAPSHOTS
- | TAG | TRUE | FALSE
- | MAP
- ;
-
-snapshotId
- : number
- ;
-
-numSnapshots
- : number
- ;
-
-timeUnit
- : DAYS
- | HOURS
- | MINUTES
- ;
-
-ADD: 'ADD';
-ALTER: 'ALTER';
-AS: 'AS';
-ASC: 'ASC';
-BRANCH: 'BRANCH';
-BY: 'BY';
-CALL: 'CALL';
-CREATE: 'CREATE';
-DAYS: 'DAYS';
-DESC: 'DESC';
-DISTRIBUTED: 'DISTRIBUTED';
-DROP: 'DROP';
-EXISTS: 'EXISTS';
-FIELD: 'FIELD';
-FIELDS: 'FIELDS';
-FIRST: 'FIRST';
-HOURS: 'HOURS';
-IF : 'IF';
-LAST: 'LAST';
-LOCALLY: 'LOCALLY';
-MINUTES: 'MINUTES';
-MONTHS: 'MONTHS';
-NOT: 'NOT';
-NULLS: 'NULLS';
-OF: 'OF';
-OR: 'OR';
-ORDERED: 'ORDERED';
-PARTITION: 'PARTITION';
-REPLACE: 'REPLACE';
-RETAIN: 'RETAIN';
-RETENTION: 'RETENTION';
-IDENTIFIER_KW: 'IDENTIFIER';
-SET: 'SET';
-SNAPSHOT: 'SNAPSHOT';
-SNAPSHOTS: 'SNAPSHOTS';
-TABLE: 'TABLE';
-TAG: 'TAG';
-UNORDERED: 'UNORDERED';
-VERSION: 'VERSION';
-WITH: 'WITH';
-WRITE: 'WRITE';
-
-TRUE: 'TRUE';
-FALSE: 'FALSE';
-
-MAP: 'MAP';
-ARRAY: 'ARRAY';
-
-PLUS: '+';
-MINUS: '-';
-
-STRING
- : '\'' ( ~('\''|'\\') | ('\\' .) )* '\''
- | '"' ( ~('"'|'\\') | ('\\' .) )* '"'
- ;
-
-BIGINT_LITERAL
- : DIGIT+ 'L'
- ;
-
-SMALLINT_LITERAL
- : DIGIT+ 'S'
- ;
-
-TINYINT_LITERAL
- : DIGIT+ 'Y'
- ;
-
-INTEGER_VALUE
- : DIGIT+
- ;
-
-EXPONENT_VALUE
- : DIGIT+ EXPONENT
- | DECIMAL_DIGITS EXPONENT {isValidDecimal()}?
- ;
-
-DECIMAL_VALUE
- : DECIMAL_DIGITS {isValidDecimal()}?
- ;
-
-FLOAT_LITERAL
- : DIGIT+ EXPONENT? 'F'
- | DECIMAL_DIGITS EXPONENT? 'F' {isValidDecimal()}?
- ;
-
-DOUBLE_LITERAL
- : DIGIT+ EXPONENT? 'D'
- | DECIMAL_DIGITS EXPONENT? 'D' {isValidDecimal()}?
- ;
-
-BIGDECIMAL_LITERAL
- : DIGIT+ EXPONENT? 'BD'
- | DECIMAL_DIGITS EXPONENT? 'BD' {isValidDecimal()}?
- ;
-
-IDENTIFIER
- : (LETTER | DIGIT | '_')+
- ;
-
-BACKQUOTED_IDENTIFIER
- : '`' ( ~'`' | '``' )* '`'
- ;
-
-fragment DECIMAL_DIGITS
- : DIGIT+ '.' DIGIT*
- | '.' DIGIT+
- ;
-
-fragment EXPONENT
- : 'E' [+-]? DIGIT+
- ;
-
-fragment DIGIT
- : [0-9]
- ;
-
-fragment LETTER
- : [A-Z]
- ;
-
-SIMPLE_COMMENT
- : '--' ('\\\n' | ~[\r\n])* '\r'? '\n'? -> channel(HIDDEN)
- ;
-
-BRACKETED_COMMENT
- : '/*' {!isHint()}? (BRACKETED_COMMENT|.)*? '*/' -> channel(HIDDEN)
- ;
-
-WS
- : [ \r\n\t]+ -> channel(HIDDEN)
- ;
-
-// Catch-all for anything we can't recognize.
-// We use this to be able to ignore and recover all the text
-// when splitting statements with DelimiterLexer
-UNRECOGNIZED
- : .
- ;
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergSparkSessionExtensions.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergSparkSessionExtensions.scala
deleted file mode 100644
index 455129f2c9d5..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergSparkSessionExtensions.scala
+++ /dev/null
@@ -1,77 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.iceberg.spark.extensions
-
-import org.apache.spark.sql.SparkSessionExtensions
-import org.apache.spark.sql.catalyst.analysis.AlignedRowLevelIcebergCommandCheck
-import org.apache.spark.sql.catalyst.analysis.AlignRowLevelCommandAssignments
-import org.apache.spark.sql.catalyst.analysis.CheckMergeIntoTableConditions
-import org.apache.spark.sql.catalyst.analysis.MergeIntoIcebergTableResolutionCheck
-import org.apache.spark.sql.catalyst.analysis.ProcedureArgumentCoercion
-import org.apache.spark.sql.catalyst.analysis.ResolveMergeIntoTableReferences
-import org.apache.spark.sql.catalyst.analysis.ResolveProcedures
-import org.apache.spark.sql.catalyst.analysis.RewriteDeleteFromTable
-import org.apache.spark.sql.catalyst.analysis.RewriteMergeIntoTable
-import org.apache.spark.sql.catalyst.analysis.RewriteUpdateTable
-import org.apache.spark.sql.catalyst.optimizer.ExtendedReplaceNullWithFalseInPredicate
-import org.apache.spark.sql.catalyst.optimizer.ExtendedSimplifyConditionalsInPredicate
-import org.apache.spark.sql.catalyst.parser.extensions.IcebergSparkSqlExtensionsParser
-import org.apache.spark.sql.execution.datasources.v2.ExtendedDataSourceV2Strategy
-import org.apache.spark.sql.execution.datasources.v2.ExtendedV2Writes
-import org.apache.spark.sql.execution.datasources.v2.OptimizeMetadataOnlyDeleteFromTable
-import org.apache.spark.sql.execution.datasources.v2.ReplaceRewrittenRowLevelCommand
-import org.apache.spark.sql.execution.datasources.v2.RowLevelCommandScanRelationPushDown
-import org.apache.spark.sql.execution.dynamicpruning.RowLevelCommandDynamicPruning
-
-class IcebergSparkSessionExtensions extends (SparkSessionExtensions => Unit) {
-
- override def apply(extensions: SparkSessionExtensions): Unit = {
- // parser extensions
- extensions.injectParser { case (_, parser) => new IcebergSparkSqlExtensionsParser(parser) }
-
- // analyzer extensions
- extensions.injectResolutionRule { spark => ResolveProcedures(spark) }
- extensions.injectResolutionRule { spark => ResolveMergeIntoTableReferences(spark) }
- extensions.injectResolutionRule { _ => CheckMergeIntoTableConditions }
- extensions.injectResolutionRule { _ => ProcedureArgumentCoercion }
- extensions.injectResolutionRule { _ => AlignRowLevelCommandAssignments }
- extensions.injectResolutionRule { _ => RewriteDeleteFromTable }
- extensions.injectResolutionRule { _ => RewriteUpdateTable }
- extensions.injectResolutionRule { _ => RewriteMergeIntoTable }
- extensions.injectCheckRule { _ => MergeIntoIcebergTableResolutionCheck }
- extensions.injectCheckRule { _ => AlignedRowLevelIcebergCommandCheck }
-
- // optimizer extensions
- extensions.injectOptimizerRule { _ => ExtendedSimplifyConditionalsInPredicate }
- extensions.injectOptimizerRule { _ => ExtendedReplaceNullWithFalseInPredicate }
- // pre-CBO rules run only once and the order of the rules is important
- // - metadata deletes have to be attempted immediately after the operator optimization
- // - dynamic filters should be added before replacing commands with rewrite plans
- // - scans must be planned before building writes
- extensions.injectPreCBORule { _ => OptimizeMetadataOnlyDeleteFromTable }
- extensions.injectPreCBORule { _ => RowLevelCommandScanRelationPushDown }
- extensions.injectPreCBORule { _ => ExtendedV2Writes }
- extensions.injectPreCBORule { spark => RowLevelCommandDynamicPruning(spark) }
- extensions.injectPreCBORule { _ => ReplaceRewrittenRowLevelCommand }
-
- // planner extensions
- extensions.injectPlannerStrategy { spark => ExtendedDataSourceV2Strategy(spark) }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/ProjectingInternalRow.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/ProjectingInternalRow.scala
deleted file mode 100644
index fb654b646738..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/ProjectingInternalRow.scala
+++ /dev/null
@@ -1,123 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst
-
-import org.apache.spark.sql.catalyst.util.ArrayData
-import org.apache.spark.sql.catalyst.util.MapData
-import org.apache.spark.sql.types.DataType
-import org.apache.spark.sql.types.Decimal
-import org.apache.spark.sql.types.StructType
-import org.apache.spark.unsafe.types.CalendarInterval
-import org.apache.spark.unsafe.types.UTF8String
-
-/**
- * An InternalRow that projects particular columns from another InternalRow without copying
- * the underlying data.
- */
-case class ProjectingInternalRow(schema: StructType, colOrdinals: Seq[Int]) extends InternalRow {
- assert(schema.size == colOrdinals.size)
-
- private var row: InternalRow = _
-
- override def numFields: Int = colOrdinals.size
-
- def project(row: InternalRow): Unit = {
- this.row = row
- }
-
- override def setNullAt(i: Int): Unit = {
- throw new UnsupportedOperationException("Cannot modify InternalRowProjection")
- }
-
- override def update(i: Int, value: Any): Unit = {
- throw new UnsupportedOperationException("Cannot modify InternalRowProjection")
- }
-
- override def copy(): InternalRow = {
- val newRow = if (row != null) row.copy() else null
- val newProjection = ProjectingInternalRow(schema, colOrdinals)
- newProjection.project(newRow)
- newProjection
- }
-
- override def isNullAt(ordinal: Int): Boolean = {
- row.isNullAt(colOrdinals(ordinal))
- }
-
- override def getBoolean(ordinal: Int): Boolean = {
- row.getBoolean(colOrdinals(ordinal))
- }
-
- override def getByte(ordinal: Int): Byte = {
- row.getByte(colOrdinals(ordinal))
- }
-
- override def getShort(ordinal: Int): Short = {
- row.getShort(colOrdinals(ordinal))
- }
-
- override def getInt(ordinal: Int): Int = {
- row.getInt(colOrdinals(ordinal))
- }
-
- override def getLong(ordinal: Int): Long = {
- row.getLong(colOrdinals(ordinal))
- }
-
- override def getFloat(ordinal: Int): Float = {
- row.getFloat(colOrdinals(ordinal))
- }
-
- override def getDouble(ordinal: Int): Double = {
- row.getDouble(colOrdinals(ordinal))
- }
-
- override def getDecimal(ordinal: Int, precision: Int, scale: Int): Decimal = {
- row.getDecimal(colOrdinals(ordinal), precision, scale)
- }
-
- override def getUTF8String(ordinal: Int): UTF8String = {
- row.getUTF8String(colOrdinals(ordinal))
- }
-
- override def getBinary(ordinal: Int): Array[Byte] = {
- row.getBinary(colOrdinals(ordinal))
- }
-
- override def getInterval(ordinal: Int): CalendarInterval = {
- row.getInterval(colOrdinals(ordinal))
- }
-
- override def getStruct(ordinal: Int, numFields: Int): InternalRow = {
- row.getStruct(colOrdinals(ordinal), numFields)
- }
-
- override def getArray(ordinal: Int): ArrayData = {
- row.getArray(colOrdinals(ordinal))
- }
-
- override def getMap(ordinal: Int): MapData = {
- row.getMap(colOrdinals(ordinal))
- }
-
- override def get(ordinal: Int, dataType: DataType): AnyRef = {
- row.get(colOrdinals(ordinal), dataType)
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlignRowLevelCommandAssignments.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlignRowLevelCommandAssignments.scala
deleted file mode 100644
index ad416f2a1c63..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlignRowLevelCommandAssignments.scala
+++ /dev/null
@@ -1,113 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.analysis
-
-import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.expressions.AssignmentUtils
-import org.apache.spark.sql.catalyst.plans.logical.Assignment
-import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
-import org.apache.spark.sql.catalyst.plans.logical.InsertAction
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
-import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
-import org.apache.spark.sql.catalyst.plans.logical.UpdateIcebergTable
-import org.apache.spark.sql.catalyst.rules.Rule
-
-/**
- * A rule that aligns assignments in UPDATE and MERGE operations.
- *
- * Note that this rule must be run before rewriting row-level commands.
- */
-object AlignRowLevelCommandAssignments
- extends Rule[LogicalPlan] with AssignmentAlignmentSupport {
-
- override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
- case u: UpdateIcebergTable if u.resolved && !u.aligned =>
- u.copy(assignments = alignAssignments(u.table, u.assignments))
-
- case m: MergeIntoIcebergTable if m.resolved && !m.aligned =>
- val alignedMatchedActions = m.matchedActions.map {
- case u @ UpdateAction(_, assignments) =>
- u.copy(assignments = alignAssignments(m.targetTable, assignments))
- case d: DeleteAction =>
- d
- case _ =>
- throw new AnalysisException("Matched actions can only contain UPDATE or DELETE")
- }
-
- val alignedNotMatchedActions = m.notMatchedActions.map {
- case i @ InsertAction(_, assignments) =>
- // check no nested columns are present
- val refs = assignments.map(_.key).map(AssignmentUtils.toAssignmentRef)
- refs.foreach { ref =>
- if (ref.size > 1) {
- throw new AnalysisException(
- "Nested fields are not supported inside INSERT clauses of MERGE operations: " +
- s"${ref.mkString("`", "`.`", "`")}")
- }
- }
-
- val colNames = refs.map(_.head)
-
- // check there are no duplicates
- val duplicateColNames = colNames.groupBy(identity).collect {
- case (name, matchingNames) if matchingNames.size > 1 => name
- }
-
- if (duplicateColNames.nonEmpty) {
- throw new AnalysisException(
- s"Duplicate column names inside INSERT clause: ${duplicateColNames.mkString(", ")}")
- }
-
- // reorder assignments by the target table column order
- val assignmentMap = colNames.zip(assignments).toMap
- i.copy(assignments = alignInsertActionAssignments(m.targetTable, assignmentMap))
-
- case _ =>
- throw new AnalysisException("Not matched actions can only contain INSERT")
- }
-
- m.copy(matchedActions = alignedMatchedActions, notMatchedActions = alignedNotMatchedActions)
- }
-
- private def alignInsertActionAssignments(
- targetTable: LogicalPlan,
- assignmentMap: Map[String, Assignment]): Seq[Assignment] = {
-
- val resolver = conf.resolver
-
- targetTable.output.map { targetAttr =>
- val assignment = assignmentMap
- .find { case (name, _) => resolver(name, targetAttr.name) }
- .map { case (_, assignment) => assignment }
-
- if (assignment.isEmpty) {
- throw new AnalysisException(
- s"Cannot find column '${targetAttr.name}' of the target table among " +
- s"the INSERT columns: ${assignmentMap.keys.mkString(", ")}. " +
- "INSERT clauses must provide values for all columns of the target table.")
- }
-
- val key = assignment.get.key
- val value = castIfNeeded(targetAttr, assignment.get.value, resolver)
- AssignmentUtils.handleCharVarcharLimits(Assignment(key, value))
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlignedRowLevelIcebergCommandCheck.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlignedRowLevelIcebergCommandCheck.scala
deleted file mode 100644
index d915e4f10949..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlignedRowLevelIcebergCommandCheck.scala
+++ /dev/null
@@ -1,38 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.analysis
-
-import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
-import org.apache.spark.sql.catalyst.plans.logical.UpdateIcebergTable
-
-object AlignedRowLevelIcebergCommandCheck extends (LogicalPlan => Unit) {
-
- override def apply(plan: LogicalPlan): Unit = {
- plan foreach {
- case m: MergeIntoIcebergTable if !m.aligned =>
- throw new AnalysisException(s"Could not align Iceberg MERGE INTO: $m")
- case u: UpdateIcebergTable if !u.aligned =>
- throw new AnalysisException(s"Could not align Iceberg UPDATE: $u")
- case _ => // OK
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/AssignmentAlignmentSupport.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/AssignmentAlignmentSupport.scala
deleted file mode 100644
index 14115bd3cbfe..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/AssignmentAlignmentSupport.scala
+++ /dev/null
@@ -1,202 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.analysis
-
-import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.SQLConfHelper
-import org.apache.spark.sql.catalyst.expressions.Alias
-import org.apache.spark.sql.catalyst.expressions.AnsiCast
-import org.apache.spark.sql.catalyst.expressions.AssignmentUtils._
-import org.apache.spark.sql.catalyst.expressions.Cast
-import org.apache.spark.sql.catalyst.expressions.CreateNamedStruct
-import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.expressions.GetStructField
-import org.apache.spark.sql.catalyst.expressions.Literal
-import org.apache.spark.sql.catalyst.expressions.NamedExpression
-import org.apache.spark.sql.catalyst.plans.logical.Assignment
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.internal.SQLConf.StoreAssignmentPolicy
-import org.apache.spark.sql.types.DataType
-import org.apache.spark.sql.types.StructField
-import org.apache.spark.sql.types.StructType
-import scala.collection.compat.immutable.ArraySeq
-import scala.collection.mutable
-
-trait AssignmentAlignmentSupport extends CastSupport {
-
- self: SQLConfHelper =>
-
- private case class ColumnUpdate(ref: Seq[String], expr: Expression)
-
- /**
- * Aligns assignments to match table columns.
- *
- * This method processes and reorders given assignments so that each target column gets
- * an expression it should be set to. If a column does not have a matching assignment,
- * it will be set to its current value. For example, if one passes a table with columns c1, c2
- * and an assignment c2 = 1, this method will return c1 = c1, c2 = 1.
- *
- * This method also handles updates to nested columns. If there is an assignment to a particular
- * nested field, this method will construct a new struct with one field updated
- * preserving other fields that have not been modified. For example, if one passes a table with
- * columns c1, c2 where c2 is a struct with fields n1 and n2 and an assignment c2.n2 = 1,
- * this method will return c1 = c1, c2 = struct(c2.n1, 1).
- *
- * @param table a target table
- * @param assignments assignments to align
- * @return aligned assignments that match table columns
- */
- protected def alignAssignments(
- table: LogicalPlan,
- assignments: Seq[Assignment]): Seq[Assignment] = {
-
- val columnUpdates = assignments.map(a => ColumnUpdate(toAssignmentRef(a.key), a.value))
- val outputExprs = applyUpdates(table.output, columnUpdates)
- outputExprs.zip(table.output).map {
- case (expr, attr) => handleCharVarcharLimits(Assignment(attr, expr))
- }
- }
-
- private def applyUpdates(
- cols: Seq[NamedExpression],
- updates: Seq[ColumnUpdate],
- resolver: Resolver = conf.resolver,
- namePrefix: Seq[String] = Nil): Seq[Expression] = {
-
- // iterate through columns at the current level and find which column updates match
- cols.map { col =>
- // find matches for this column or any of its children
- val prefixMatchedUpdates = updates.filter(a => resolver(a.ref.head, col.name))
- prefixMatchedUpdates match {
- // if there is no exact match and no match for children, return the column as is
- case updates if updates.isEmpty =>
- col
-
- // if there is an exact match, return the assigned expression
- case Seq(update) if isExactMatch(update, col, resolver) =>
- castIfNeeded(col, update.expr, resolver)
-
- // if there are matches only for children
- case updates if !hasExactMatch(updates, col, resolver) =>
- col.dataType match {
- case StructType(fields) =>
- // build field expressions
- val fieldExprs = fields.zipWithIndex.map { case (field, ordinal) =>
- Alias(GetStructField(col, ordinal, Some(field.name)), field.name)()
- }
-
- // recursively apply this method on nested fields
- val newUpdates = updates.map(u => u.copy(ref = u.ref.tail))
- val updatedFieldExprs = applyUpdates(
- ArraySeq.unsafeWrapArray(fieldExprs),
- newUpdates,
- resolver,
- namePrefix :+ col.name)
-
- // construct a new struct with updated field expressions
- toNamedStruct(ArraySeq.unsafeWrapArray(fields), updatedFieldExprs)
-
- case otherType =>
- val colName = (namePrefix :+ col.name).mkString(".")
- throw new AnalysisException(
- "Updating nested fields is only supported for StructType " +
- s"but $colName is of type $otherType"
- )
- }
-
- // if there are conflicting updates, throw an exception
- // there are two illegal scenarios:
- // - multiple updates to the same column
- // - updates to a top-level struct and its nested fields (e.g., a.b and a.b.c)
- case updates if hasExactMatch(updates, col, resolver) =>
- val conflictingCols = updates.map(u => (namePrefix ++ u.ref).mkString("."))
- throw new AnalysisException(
- "Updates are in conflict for these columns: " +
- conflictingCols.distinct.mkString(", "))
- }
- }
- }
-
- private def toNamedStruct(fields: Seq[StructField], fieldExprs: Seq[Expression]): Expression = {
- val namedStructExprs = fields.zip(fieldExprs).flatMap { case (field, expr) =>
- Seq(Literal(field.name), expr)
- }
- CreateNamedStruct(namedStructExprs)
- }
-
- private def hasExactMatch(
- updates: Seq[ColumnUpdate],
- col: NamedExpression,
- resolver: Resolver): Boolean = {
-
- updates.exists(assignment => isExactMatch(assignment, col, resolver))
- }
-
- private def isExactMatch(
- update: ColumnUpdate,
- col: NamedExpression,
- resolver: Resolver): Boolean = {
-
- update.ref match {
- case Seq(namePart) if resolver(namePart, col.name) => true
- case _ => false
- }
- }
-
- protected def castIfNeeded(
- tableAttr: NamedExpression,
- expr: Expression,
- resolver: Resolver): Expression = {
-
- val storeAssignmentPolicy = conf.storeAssignmentPolicy
-
- // run the type check and catch type errors
- storeAssignmentPolicy match {
- case StoreAssignmentPolicy.STRICT | StoreAssignmentPolicy.ANSI =>
- if (expr.nullable && !tableAttr.nullable) {
- throw new AnalysisException(
- s"Cannot write nullable values to non-null column '${tableAttr.name}'")
- }
-
- // use byName = true to catch cases when struct field names don't match
- // e.g. a struct with fields (a, b) is assigned as a struct with fields (a, c) or (b, a)
- val errors = new mutable.ArrayBuffer[String]()
- val canWrite = DataType.canWrite(
- expr.dataType, tableAttr.dataType, byName = true, resolver, tableAttr.name,
- storeAssignmentPolicy, err => errors += err)
-
- if (!canWrite) {
- throw new AnalysisException(
- s"Cannot write incompatible data:\n- ${errors.mkString("\n- ")}")
- }
-
- case _ => // OK
- }
-
- storeAssignmentPolicy match {
- case _ if tableAttr.dataType.sameType(expr.dataType) =>
- expr
- case StoreAssignmentPolicy.ANSI =>
- AnsiCast(expr, tableAttr.dataType, Option(conf.sessionLocalTimeZone))
- case _ =>
- Cast(expr, tableAttr.dataType, Option(conf.sessionLocalTimeZone))
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckMergeIntoTableConditions.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckMergeIntoTableConditions.scala
deleted file mode 100644
index 70f6694af60b..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckMergeIntoTableConditions.scala
+++ /dev/null
@@ -1,74 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.analysis
-
-import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.expressions.SubqueryExpression
-import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
-import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
-import org.apache.spark.sql.catalyst.plans.logical.InsertAction
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
-import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
-import org.apache.spark.sql.catalyst.rules.Rule
-
-/**
- * A rule that checks MERGE operations contain only supported conditions.
- *
- * Note that this rule must be run in the resolution batch before Spark executes CheckAnalysis.
- * Otherwise, CheckAnalysis will throw a less descriptive error.
- */
-object CheckMergeIntoTableConditions extends Rule[LogicalPlan] {
-
- override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
- case m: MergeIntoIcebergTable if m.resolved =>
- checkMergeIntoCondition("SEARCH", m.mergeCondition)
-
- val actions = m.matchedActions ++ m.notMatchedActions
- actions.foreach {
- case DeleteAction(Some(cond)) => checkMergeIntoCondition("DELETE", cond)
- case UpdateAction(Some(cond), _) => checkMergeIntoCondition("UPDATE", cond)
- case InsertAction(Some(cond), _) => checkMergeIntoCondition("INSERT", cond)
- case _ => // OK
- }
-
- m
- }
-
- private def checkMergeIntoCondition(condName: String, cond: Expression): Unit = {
- if (!cond.deterministic) {
- throw new AnalysisException(
- s"Non-deterministic functions are not supported in $condName conditions of " +
- s"MERGE operations: ${cond.sql}")
- }
-
- if (SubqueryExpression.hasSubquery(cond)) {
- throw new AnalysisException(
- s"Subqueries are not supported in conditions of MERGE operations. " +
- s"Found a subquery in the $condName condition: ${cond.sql}")
- }
-
- if (cond.find(_.isInstanceOf[AggregateExpression]).isDefined) {
- throw new AnalysisException(
- s"Agg functions are not supported in $condName conditions of MERGE operations: " + {cond.sql})
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/MergeIntoIcebergTableResolutionCheck.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/MergeIntoIcebergTableResolutionCheck.scala
deleted file mode 100644
index b3a9bda280d2..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/MergeIntoIcebergTableResolutionCheck.scala
+++ /dev/null
@@ -1,35 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.analysis
-
-import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
-
-object MergeIntoIcebergTableResolutionCheck extends (LogicalPlan => Unit) {
-
- override def apply(plan: LogicalPlan): Unit = {
- plan foreach {
- case m: UnresolvedMergeIntoIcebergTable =>
- throw new AnalysisException(s"Could not resolve Iceberg MERGE INTO statement: $m")
- case _ => // OK
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ProcedureArgumentCoercion.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ProcedureArgumentCoercion.scala
deleted file mode 100644
index 7f0ca8fadded..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ProcedureArgumentCoercion.scala
+++ /dev/null
@@ -1,56 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.analysis
-
-import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.expressions.Cast
-import org.apache.spark.sql.catalyst.plans.logical.Call
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.rules.Rule
-
-object ProcedureArgumentCoercion extends Rule[LogicalPlan] {
- override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
- case c @ Call(procedure, args) if c.resolved =>
- val params = procedure.parameters
-
- val newArgs = args.zipWithIndex.map { case (arg, index) =>
- val param = params(index)
- val paramType = param.dataType
- val argType = arg.dataType
-
- if (paramType != argType && !Cast.canUpCast(argType, paramType)) {
- throw new AnalysisException(
- s"Wrong arg type for ${param.name}: cannot cast $argType to $paramType")
- }
-
- if (paramType != argType) {
- Cast(arg, paramType)
- } else {
- arg
- }
- }
-
- if (newArgs != args) {
- c.copy(args = newArgs)
- } else {
- c
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveMergeIntoTableReferences.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveMergeIntoTableReferences.scala
deleted file mode 100644
index 63ebdef95730..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveMergeIntoTableReferences.scala
+++ /dev/null
@@ -1,153 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.analysis
-
-import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.SparkSession
-import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.plans.logical.Assignment
-import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
-import org.apache.spark.sql.catalyst.plans.logical.InsertAction
-import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
-import org.apache.spark.sql.catalyst.plans.logical.Project
-import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
-import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
-import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
-import org.apache.spark.sql.catalyst.rules.Rule
-
-/**
- * A resolution rule similar to ResolveReferences in Spark but handles Iceberg MERGE operations.
- */
-case class ResolveMergeIntoTableReferences(spark: SparkSession) extends Rule[LogicalPlan] {
-
- private lazy val analyzer: Analyzer = spark.sessionState.analyzer
-
- override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperatorsUp {
- case m @ UnresolvedMergeIntoIcebergTable(targetTable, sourceTable, context)
- if targetTable.resolved && sourceTable.resolved && m.duplicateResolved =>
-
- val resolvedMatchedActions = context.matchedActions.map {
- case DeleteAction(cond) =>
- val resolvedCond = cond.map(resolveCond("DELETE", _, m))
- DeleteAction(resolvedCond)
-
- case UpdateAction(cond, assignments) =>
- val resolvedCond = cond.map(resolveCond("UPDATE", _, m))
- // the update action can access columns from both target and source tables
- val resolvedAssignments = resolveAssignments(assignments, m, resolveValuesWithSourceOnly = false)
- UpdateAction(resolvedCond, resolvedAssignments)
-
- case UpdateStarAction(updateCondition) =>
- val resolvedUpdateCondition = updateCondition.map(resolveCond("UPDATE", _, m))
- val assignments = targetTable.output.map { attr =>
- Assignment(attr, UnresolvedAttribute(Seq(attr.name)))
- }
- // for UPDATE *, the value must be from the source table
- val resolvedAssignments = resolveAssignments(assignments, m, resolveValuesWithSourceOnly = true)
- UpdateAction(resolvedUpdateCondition, resolvedAssignments)
-
- case _ =>
- throw new AnalysisException("Matched actions can only contain UPDATE or DELETE")
- }
-
- val resolvedNotMatchedActions = context.notMatchedActions.map {
- case InsertAction(cond, assignments) =>
- // the insert action is used when not matched, so its condition and value can only
- // access columns from the source table
- val resolvedCond = cond.map(resolveCond("INSERT", _, Project(Nil, m.sourceTable)))
- val resolvedAssignments = resolveAssignments(assignments, m, resolveValuesWithSourceOnly = true)
- InsertAction(resolvedCond, resolvedAssignments)
-
- case InsertStarAction(cond) =>
- // the insert action is used when not matched, so its condition and value can only
- // access columns from the source table
- val resolvedCond = cond.map(resolveCond("INSERT", _, Project(Nil, m.sourceTable)))
- val assignments = targetTable.output.map { attr =>
- Assignment(attr, UnresolvedAttribute(Seq(attr.name)))
- }
- val resolvedAssignments = resolveAssignments(assignments, m, resolveValuesWithSourceOnly = true)
- InsertAction(resolvedCond, resolvedAssignments)
-
- case _ =>
- throw new AnalysisException("Not matched actions can only contain INSERT")
- }
-
- val resolvedMergeCondition = resolveCond("SEARCH", context.mergeCondition, m)
-
- MergeIntoIcebergTable(
- targetTable,
- sourceTable,
- mergeCondition = resolvedMergeCondition,
- matchedActions = resolvedMatchedActions,
- notMatchedActions = resolvedNotMatchedActions)
- }
-
- private def resolveCond(condName: String, cond: Expression, plan: LogicalPlan): Expression = {
- val resolvedCond = analyzer.resolveExpressionByPlanChildren(cond, plan)
-
- val unresolvedAttrs = resolvedCond.references.filter(!_.resolved)
- if (unresolvedAttrs.nonEmpty) {
- throw new AnalysisException(
- s"Cannot resolve ${unresolvedAttrs.map(_.sql).mkString("[", ",", "]")} in $condName condition " +
- s"of MERGE operation given input columns: ${plan.inputSet.toSeq.map(_.sql).mkString("[", ",", "]")}")
- }
-
- resolvedCond
- }
-
- // copied from ResolveReferences in Spark
- private def resolveAssignments(
- assignments: Seq[Assignment],
- mergeInto: UnresolvedMergeIntoIcebergTable,
- resolveValuesWithSourceOnly: Boolean): Seq[Assignment] = {
- assignments.map { assign =>
- val resolvedKey = assign.key match {
- case c if !c.resolved =>
- resolveMergeExprOrFail(c, Project(Nil, mergeInto.targetTable))
- case o => o
- }
- val resolvedValue = assign.value match {
- // The update values may contain target and/or source references.
- case c if !c.resolved =>
- if (resolveValuesWithSourceOnly) {
- resolveMergeExprOrFail(c, Project(Nil, mergeInto.sourceTable))
- } else {
- resolveMergeExprOrFail(c, mergeInto)
- }
- case o => o
- }
- Assignment(resolvedKey, resolvedValue)
- }
- }
-
- // copied from ResolveReferences in Spark
- private def resolveMergeExprOrFail(e: Expression, p: LogicalPlan): Expression = {
- val resolved = analyzer.resolveExpressionByPlanChildren(e, p)
- resolved.references.filter(!_.resolved).foreach { a =>
- // Note: This will throw error only on unresolved attribute issues,
- // not other resolution errors like mismatched data types.
- val cols = p.inputSet.toSeq.map(_.sql).mkString(", ")
- a.failAnalysis(s"cannot resolve ${a.sql} in MERGE command given columns [$cols]")
- }
- resolved
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveProcedures.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveProcedures.scala
deleted file mode 100644
index ee69b5e344f0..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveProcedures.scala
+++ /dev/null
@@ -1,190 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.analysis
-
-import java.util.Locale
-import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.SparkSession
-import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.expressions.Literal
-import org.apache.spark.sql.catalyst.plans.logical.Call
-import org.apache.spark.sql.catalyst.plans.logical.CallArgument
-import org.apache.spark.sql.catalyst.plans.logical.CallStatement
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.plans.logical.NamedArgument
-import org.apache.spark.sql.catalyst.plans.logical.PositionalArgument
-import org.apache.spark.sql.catalyst.rules.Rule
-import org.apache.spark.sql.connector.catalog.CatalogManager
-import org.apache.spark.sql.connector.catalog.CatalogPlugin
-import org.apache.spark.sql.connector.catalog.LookupCatalog
-import org.apache.spark.sql.connector.iceberg.catalog.ProcedureCatalog
-import org.apache.spark.sql.connector.iceberg.catalog.ProcedureParameter
-import scala.collection.Seq
-
-case class ResolveProcedures(spark: SparkSession) extends Rule[LogicalPlan] with LookupCatalog {
-
- protected lazy val catalogManager: CatalogManager = spark.sessionState.catalogManager
-
- override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
- case CallStatement(CatalogAndIdentifier(catalog, ident), args) =>
- val procedure = catalog.asProcedureCatalog.loadProcedure(ident)
-
- val params = procedure.parameters
- val normalizedParams = normalizeParams(params)
- validateParams(normalizedParams)
-
- val normalizedArgs = normalizeArgs(args)
- Call(procedure, args = buildArgExprs(normalizedParams, normalizedArgs).toSeq)
- }
-
- private def validateParams(params: Seq[ProcedureParameter]): Unit = {
- // should not be any duplicate param names
- val duplicateParamNames = params.groupBy(_.name).collect {
- case (name, matchingParams) if matchingParams.length > 1 => name
- }
-
- if (duplicateParamNames.nonEmpty) {
- throw new AnalysisException(s"Duplicate parameter names: ${duplicateParamNames.mkString("[", ",", "]")}")
- }
-
- // optional params should be at the end
- params.sliding(2).foreach {
- case Seq(previousParam, currentParam) if !previousParam.required && currentParam.required =>
- throw new AnalysisException(
- s"Optional parameters must be after required ones but $currentParam is after $previousParam")
- case _ =>
- }
- }
-
- private def buildArgExprs(
- params: Seq[ProcedureParameter],
- args: Seq[CallArgument]): Seq[Expression] = {
-
- // build a map of declared parameter names to their positions
- val nameToPositionMap = params.map(_.name).zipWithIndex.toMap
-
- // build a map of parameter names to args
- val nameToArgMap = buildNameToArgMap(params, args, nameToPositionMap)
-
- // verify all required parameters are provided
- val missingParamNames = params.filter(_.required).collect {
- case param if !nameToArgMap.contains(param.name) => param.name
- }
-
- if (missingParamNames.nonEmpty) {
- throw new AnalysisException(s"Missing required parameters: ${missingParamNames.mkString("[", ",", "]")}")
- }
-
- val argExprs = new Array[Expression](params.size)
-
- nameToArgMap.foreach { case (name, arg) =>
- val position = nameToPositionMap(name)
- argExprs(position) = arg.expr
- }
-
- // assign nulls to optional params that were not set
- params.foreach {
- case p if !p.required && !nameToArgMap.contains(p.name) =>
- val position = nameToPositionMap(p.name)
- argExprs(position) = Literal.create(null, p.dataType)
- case _ =>
- }
-
- argExprs
- }
-
- private def buildNameToArgMap(
- params: Seq[ProcedureParameter],
- args: Seq[CallArgument],
- nameToPositionMap: Map[String, Int]): Map[String, CallArgument] = {
-
- val containsNamedArg = args.exists(_.isInstanceOf[NamedArgument])
- val containsPositionalArg = args.exists(_.isInstanceOf[PositionalArgument])
-
- if (containsNamedArg && containsPositionalArg) {
- throw new AnalysisException("Named and positional arguments cannot be mixed")
- }
-
- if (containsNamedArg) {
- buildNameToArgMapUsingNames(args, nameToPositionMap)
- } else {
- buildNameToArgMapUsingPositions(args, params)
- }
- }
-
- private def buildNameToArgMapUsingNames(
- args: Seq[CallArgument],
- nameToPositionMap: Map[String, Int]): Map[String, CallArgument] = {
-
- val namedArgs = args.asInstanceOf[Seq[NamedArgument]]
-
- val validationErrors = namedArgs.groupBy(_.name).collect {
- case (name, matchingArgs) if matchingArgs.size > 1 => s"Duplicate procedure argument: $name"
- case (name, _) if !nameToPositionMap.contains(name) => s"Unknown argument: $name"
- }
-
- if (validationErrors.nonEmpty) {
- throw new AnalysisException(s"Could not build name to arg map: ${validationErrors.mkString(", ")}")
- }
-
- namedArgs.map(arg => arg.name -> arg).toMap
- }
-
- private def buildNameToArgMapUsingPositions(
- args: Seq[CallArgument],
- params: Seq[ProcedureParameter]): Map[String, CallArgument] = {
-
- if (args.size > params.size) {
- throw new AnalysisException("Too many arguments for procedure")
- }
-
- args.zipWithIndex.map { case (arg, position) =>
- val param = params(position)
- param.name -> arg
- }.toMap
- }
-
- private def normalizeParams(params: Seq[ProcedureParameter]): Seq[ProcedureParameter] = {
- params.map {
- case param if param.required =>
- val normalizedName = param.name.toLowerCase(Locale.ROOT)
- ProcedureParameter.required(normalizedName, param.dataType)
- case param =>
- val normalizedName = param.name.toLowerCase(Locale.ROOT)
- ProcedureParameter.optional(normalizedName, param.dataType)
- }
- }
-
- private def normalizeArgs(args: Seq[CallArgument]): Seq[CallArgument] = {
- args.map {
- case a @ NamedArgument(name, _) => a.copy(name = name.toLowerCase(Locale.ROOT))
- case other => other
- }
- }
-
- implicit class CatalogHelper(plugin: CatalogPlugin) {
- def asProcedureCatalog: ProcedureCatalog = plugin match {
- case procedureCatalog: ProcedureCatalog =>
- procedureCatalog
- case _ =>
- throw new AnalysisException(s"Cannot use catalog ${plugin.name}: not a ProcedureCatalog")
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteDeleteFromTable.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteDeleteFromTable.scala
deleted file mode 100644
index bbf7828eb4b4..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteDeleteFromTable.scala
+++ /dev/null
@@ -1,118 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.analysis
-
-import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.expressions.Alias
-import org.apache.spark.sql.catalyst.expressions.EqualNullSafe
-import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.expressions.Literal
-import org.apache.spark.sql.catalyst.expressions.Not
-import org.apache.spark.sql.catalyst.plans.logical.DeleteFromIcebergTable
-import org.apache.spark.sql.catalyst.plans.logical.Filter
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.plans.logical.Project
-import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
-import org.apache.spark.sql.catalyst.plans.logical.WriteDelta
-import org.apache.spark.sql.catalyst.util.RowDeltaUtils._
-import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
-import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.DELETE
-import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
-import org.apache.spark.sql.connector.write.RowLevelOperationTable
-import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
-
-/**
- * Assigns a rewrite plan for v2 tables that support rewriting data to handle DELETE statements.
- *
- * If a table implements SupportsDelete and SupportsRowLevelOperations, this rule assigns a rewrite
- * plan but the optimizer will check whether this particular DELETE statement can be handled
- * by simply passing delete filters to the connector. If yes, the optimizer will then discard
- * the rewrite plan.
- */
-object RewriteDeleteFromTable extends RewriteRowLevelCommand {
-
- override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
- case d @ DeleteFromIcebergTable(aliasedTable, Some(cond), None) if d.resolved =>
- EliminateSubqueryAliases(aliasedTable) match {
- case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
- val operation = buildRowLevelOperation(tbl, DELETE)
- val table = RowLevelOperationTable(tbl, operation)
- val rewritePlan = operation match {
- case _: SupportsDelta =>
- buildWriteDeltaPlan(r, table, cond)
- case _ =>
- buildReplaceDataPlan(r, table, cond)
- }
- // keep the original relation in DELETE to try deleting using filters
- DeleteFromIcebergTable(r, Some(cond), Some(rewritePlan))
-
- case p =>
- throw new AnalysisException(s"$p is not an Iceberg table")
- }
- }
-
- // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
- private def buildReplaceDataPlan(
- relation: DataSourceV2Relation,
- operationTable: RowLevelOperationTable,
- cond: Expression): ReplaceData = {
-
- // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
- val metadataAttrs = resolveRequiredMetadataAttrs(relation, operationTable.operation)
-
- // construct a read relation and include all required metadata columns
- val readRelation = buildReadRelation(relation, operationTable, metadataAttrs)
-
- // construct a plan that contains unmatched rows in matched groups that must be carried over
- // such rows do not match the condition but have to be copied over as the source can replace
- // only groups of rows
- val remainingRowsFilter = Not(EqualNullSafe(cond, Literal.TrueLiteral))
- val remainingRowsPlan = Filter(remainingRowsFilter, readRelation)
-
- // build a plan to replace read groups in the table
- val writeRelation = relation.copy(table = operationTable)
- ReplaceData(writeRelation, remainingRowsPlan, relation)
- }
-
- // build a rewrite plan for sources that support row deltas
- private def buildWriteDeltaPlan(
- relation: DataSourceV2Relation,
- operationTable: RowLevelOperationTable,
- cond: Expression): WriteDelta = {
-
- // resolve all needed attrs (e.g. row ID and any required metadata attrs)
- val rowIdAttrs = resolveRowIdAttrs(relation, operationTable.operation)
- val metadataAttrs = resolveRequiredMetadataAttrs(relation, operationTable.operation)
-
- // construct a read relation and include all required metadata columns
- val readRelation = buildReadRelation(relation, operationTable, metadataAttrs, rowIdAttrs)
-
- // construct a plan that only contains records to delete
- val deletedRowsPlan = Filter(cond, readRelation)
- val operationType = Alias(Literal(DELETE_OPERATION), OPERATION_COLUMN)()
- val requiredWriteAttrs = dedupAttrs(rowIdAttrs ++ metadataAttrs)
- val project = Project(operationType +: requiredWriteAttrs, deletedRowsPlan)
-
- // build a plan to write deletes to the table
- val writeRelation = relation.copy(table = operationTable)
- val projections = buildWriteDeltaProjections(project, Nil, rowIdAttrs, metadataAttrs)
- WriteDelta(writeRelation, project, relation, projections)
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
deleted file mode 100644
index 31b2cfe05c48..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
+++ /dev/null
@@ -1,442 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.analysis
-
-import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.ProjectingInternalRow
-import org.apache.spark.sql.catalyst.expressions.Alias
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.catalyst.expressions.AttributeReference
-import org.apache.spark.sql.catalyst.expressions.AttributeSet
-import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
-import org.apache.spark.sql.catalyst.expressions.IsNotNull
-import org.apache.spark.sql.catalyst.expressions.Literal
-import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
-import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
-import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
-import org.apache.spark.sql.catalyst.plans.FullOuter
-import org.apache.spark.sql.catalyst.plans.Inner
-import org.apache.spark.sql.catalyst.plans.LeftAnti
-import org.apache.spark.sql.catalyst.plans.LeftOuter
-import org.apache.spark.sql.catalyst.plans.RightOuter
-import org.apache.spark.sql.catalyst.plans.logical.AppendData
-import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
-import org.apache.spark.sql.catalyst.plans.logical.Filter
-import org.apache.spark.sql.catalyst.plans.logical.HintInfo
-import org.apache.spark.sql.catalyst.plans.logical.InsertAction
-import org.apache.spark.sql.catalyst.plans.logical.Join
-import org.apache.spark.sql.catalyst.plans.logical.JoinHint
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.plans.logical.MergeAction
-import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
-import org.apache.spark.sql.catalyst.plans.logical.MergeRows
-import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
-import org.apache.spark.sql.catalyst.plans.logical.NoStatsUnaryNode
-import org.apache.spark.sql.catalyst.plans.logical.Project
-import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
-import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
-import org.apache.spark.sql.catalyst.plans.logical.WriteDelta
-import org.apache.spark.sql.catalyst.util.RowDeltaUtils._
-import org.apache.spark.sql.catalyst.util.WriteDeltaProjections
-import org.apache.spark.sql.connector.expressions.FieldReference
-import org.apache.spark.sql.connector.expressions.NamedReference
-import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
-import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
-import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
-import org.apache.spark.sql.connector.write.RowLevelOperationTable
-import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
-import org.apache.spark.sql.types.IntegerType
-import org.apache.spark.sql.types.StructField
-import org.apache.spark.sql.types.StructType
-
-/**
- * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
- *
- * This rule assumes the commands have been fully resolved and all assignments have been aligned.
- * That's why it must be run after AlignRowLevelCommandAssignments.
- */
-object RewriteMergeIntoTable extends RewriteRowLevelCommand {
-
- private final val ROW_FROM_SOURCE = "__row_from_source"
- private final val ROW_FROM_TARGET = "__row_from_target"
- private final val ROW_ID = "__row_id"
-
- private final val ROW_FROM_SOURCE_REF = FieldReference(ROW_FROM_SOURCE)
- private final val ROW_FROM_TARGET_REF = FieldReference(ROW_FROM_TARGET)
- private final val ROW_ID_REF = FieldReference(ROW_ID)
-
- override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
- case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
- if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
-
- EliminateSubqueryAliases(aliasedTable) match {
- case r: DataSourceV2Relation =>
- // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
- val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
- val filteredSource = insertAction.condition match {
- case Some(insertCond) => Filter(insertCond, source)
- case None => source
- }
-
- // when there are no MATCHED actions, use a left anti join to remove any matching rows
- // and switch to using a regular append instead of a row-level merge
- // only unmatched source rows that match the condition are appended to the table
- val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
-
- val outputExprs = insertAction.assignments.map(_.value)
- val outputColNames = r.output.map(_.name)
- val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
- Alias(expr, name)()
- }
- val project = Project(outputCols, joinPlan)
-
- AppendData.byPosition(r, project)
-
- case p =>
- throw new AnalysisException(s"$p is not an Iceberg table")
- }
-
- case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
- if m.resolved && m.aligned && matchedActions.isEmpty =>
-
- EliminateSubqueryAliases(aliasedTable) match {
- case r: DataSourceV2Relation =>
- // when there are no MATCHED actions, use a left anti join to remove any matching rows
- // and switch to using a regular append instead of a row-level merge
- // only unmatched source rows that match action conditions are appended to the table
- val joinPlan = Join(source, r, LeftAnti, Some(cond), JoinHint.NONE)
-
- val notMatchedConditions = notMatchedActions.map(actionCondition)
- val notMatchedOutputs = notMatchedActions.map(actionOutput(_, Nil))
-
- // merge rows as there are multiple not matched actions
- val mergeRows = MergeRows(
- isSourceRowPresent = TrueLiteral,
- isTargetRowPresent = FalseLiteral,
- matchedConditions = Nil,
- matchedOutputs = Nil,
- notMatchedConditions = notMatchedConditions,
- notMatchedOutputs = notMatchedOutputs,
- targetOutput = Nil,
- rowIdAttrs = Nil,
- performCardinalityCheck = false,
- emitNotMatchedTargetRows = false,
- output = buildMergeRowsOutput(Nil, notMatchedOutputs, r.output),
- joinPlan)
-
- AppendData.byPosition(r, mergeRows)
-
- case p =>
- throw new AnalysisException(s"$p is not an Iceberg table")
- }
-
- case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
- if m.resolved && m.aligned =>
-
- EliminateSubqueryAliases(aliasedTable) match {
- case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
- val operation = buildRowLevelOperation(tbl, MERGE)
- val table = RowLevelOperationTable(tbl, operation)
- val rewritePlan = operation match {
- case _: SupportsDelta =>
- buildWriteDeltaPlan(r, table, source, cond, matchedActions, notMatchedActions)
- case _ =>
- buildReplaceDataPlan(r, table, source, cond, matchedActions, notMatchedActions)
- }
-
- m.copy(rewritePlan = Some(rewritePlan))
-
- case p =>
- throw new AnalysisException(s"$p is not an Iceberg table")
- }
- }
-
- // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
- private def buildReplaceDataPlan(
- relation: DataSourceV2Relation,
- operationTable: RowLevelOperationTable,
- source: LogicalPlan,
- cond: Expression,
- matchedActions: Seq[MergeAction],
- notMatchedActions: Seq[MergeAction]): ReplaceData = {
-
- // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
- val metadataAttrs = resolveRequiredMetadataAttrs(relation, operationTable.operation)
-
- // construct a scan relation and include all required metadata columns
- val readRelation = buildReadRelation(relation, operationTable, metadataAttrs)
- val readAttrs = readRelation.output
-
- // project an extra column to check if a target row exists after the join
- // project a synthetic row ID to perform the cardinality check
- val rowFromTarget = Alias(TrueLiteral, ROW_FROM_TARGET)()
- val rowId = Alias(MonotonicallyIncreasingID(), ROW_ID)()
- val targetTableProjExprs = readAttrs ++ Seq(rowFromTarget, rowId)
- val targetTableProj = Project(targetTableProjExprs, readRelation)
-
- // project an extra column to check if a source row exists after the join
- val rowFromSource = Alias(TrueLiteral, ROW_FROM_SOURCE)()
- val sourceTableProjExprs = source.output :+ rowFromSource
- val sourceTableProj = Project(sourceTableProjExprs, source)
-
- // use left outer join if there is no NOT MATCHED action, unmatched source rows can be discarded
- // use full outer join in all other cases, unmatched source rows may be needed
- // disable broadcasts for the target table to perform the cardinality check
- val joinType = if (notMatchedActions.isEmpty) LeftOuter else FullOuter
- val joinHint = JoinHint(leftHint = Some(HintInfo(Some(NO_BROADCAST_HASH))), rightHint = None)
- val joinPlan = Join(NoStatsUnaryNode(targetTableProj), sourceTableProj, joinType, Some(cond), joinHint)
-
- // add an extra matched action to output the original row if none of the actual actions matched
- // this is needed to keep target rows that should be copied over
- val matchedConditions = matchedActions.map(actionCondition) :+ TrueLiteral
- val matchedOutputs = matchedActions.map(actionOutput(_, metadataAttrs)) :+ readAttrs
-
- val notMatchedConditions = notMatchedActions.map(actionCondition)
- val notMatchedOutputs = notMatchedActions.map(actionOutput(_, metadataAttrs))
-
- val rowIdAttr = resolveAttrRef(ROW_ID_REF, joinPlan)
- val rowFromSourceAttr = resolveAttrRef(ROW_FROM_SOURCE_REF, joinPlan)
- val rowFromTargetAttr = resolveAttrRef(ROW_FROM_TARGET_REF, joinPlan)
-
- val mergeRows = MergeRows(
- isSourceRowPresent = IsNotNull(rowFromSourceAttr),
- isTargetRowPresent = if (notMatchedActions.isEmpty) TrueLiteral else IsNotNull(rowFromTargetAttr),
- matchedConditions = matchedConditions,
- matchedOutputs = matchedOutputs,
- notMatchedConditions = notMatchedConditions,
- notMatchedOutputs = notMatchedOutputs,
- targetOutput = readAttrs,
- rowIdAttrs = Seq(rowIdAttr),
- performCardinalityCheck = isCardinalityCheckNeeded(matchedActions),
- emitNotMatchedTargetRows = true,
- output = buildMergeRowsOutput(matchedOutputs, notMatchedOutputs, readAttrs),
- joinPlan)
-
- // build a plan to replace read groups in the table
- val writeRelation = relation.copy(table = operationTable)
- ReplaceData(writeRelation, mergeRows, relation)
- }
-
- // build a rewrite plan for sources that support row deltas
- private def buildWriteDeltaPlan(
- relation: DataSourceV2Relation,
- operationTable: RowLevelOperationTable,
- source: LogicalPlan,
- cond: Expression,
- matchedActions: Seq[MergeAction],
- notMatchedActions: Seq[MergeAction]): WriteDelta = {
-
- // resolve all needed attrs (e.g. row ID and any required metadata attrs)
- val rowAttrs = relation.output
- val rowIdAttrs = resolveRowIdAttrs(relation, operationTable.operation)
- val metadataAttrs = resolveRequiredMetadataAttrs(relation, operationTable.operation)
-
- // construct a scan relation and include all required metadata columns
- val readRelation = buildReadRelation(relation, operationTable, metadataAttrs, rowIdAttrs)
- val readAttrs = readRelation.output
-
- // project an extra column to check if a target row exists after the join
- val targetTableProjExprs = readAttrs :+ Alias(TrueLiteral, ROW_FROM_TARGET)()
- val targetTableProj = Project(targetTableProjExprs, readRelation)
-
- // project an extra column to check if a source row exists after the join
- val sourceTableProjExprs = source.output :+ Alias(TrueLiteral, ROW_FROM_SOURCE)()
- val sourceTableProj = Project(sourceTableProjExprs, source)
-
- // use inner join if there is no NOT MATCHED action, unmatched source rows can be discarded
- // use right outer join in all other cases, unmatched source rows may be needed
- // also disable broadcasts for the target table to perform the cardinality check
- val joinType = if (notMatchedActions.isEmpty) Inner else RightOuter
- val joinHint = JoinHint(leftHint = Some(HintInfo(Some(NO_BROADCAST_HASH))), rightHint = None)
- val joinPlan = Join(NoStatsUnaryNode(targetTableProj), sourceTableProj, joinType, Some(cond), joinHint)
-
- val deleteRowValues = buildDeltaDeleteRowValues(rowAttrs, rowIdAttrs)
- val metadataReadAttrs = readAttrs.filterNot(relation.outputSet.contains)
-
- val matchedConditions = matchedActions.map(actionCondition)
- val matchedOutputs = matchedActions.map(deltaActionOutput(_, deleteRowValues, metadataReadAttrs))
-
- val notMatchedConditions = notMatchedActions.map(actionCondition)
- val notMatchedOutputs = notMatchedActions.map(deltaActionOutput(_, deleteRowValues, metadataReadAttrs))
-
- val operationTypeAttr = AttributeReference(OPERATION_COLUMN, IntegerType, nullable = false)()
- val rowFromSourceAttr = resolveAttrRef(ROW_FROM_SOURCE_REF, joinPlan)
- val rowFromTargetAttr = resolveAttrRef(ROW_FROM_TARGET_REF, joinPlan)
-
- // merged rows must contain values for the operation type and all read attrs
- val mergeRowsOutput = buildMergeRowsOutput(matchedOutputs, notMatchedOutputs, operationTypeAttr +: readAttrs)
-
- val mergeRows = MergeRows(
- isSourceRowPresent = IsNotNull(rowFromSourceAttr),
- isTargetRowPresent = if (notMatchedActions.isEmpty) TrueLiteral else IsNotNull(rowFromTargetAttr),
- matchedConditions = matchedConditions,
- matchedOutputs = matchedOutputs,
- notMatchedConditions = notMatchedConditions,
- notMatchedOutputs = notMatchedOutputs,
- // only needed if emitting unmatched target rows
- targetOutput = Nil,
- rowIdAttrs = rowIdAttrs,
- performCardinalityCheck = isCardinalityCheckNeeded(matchedActions),
- emitNotMatchedTargetRows = false,
- output = mergeRowsOutput,
- joinPlan)
-
- // build a plan to write the row delta to the table
- val writeRelation = relation.copy(table = operationTable)
- val projections = buildMergeDeltaProjections(mergeRows, rowAttrs, rowIdAttrs, metadataAttrs)
- WriteDelta(writeRelation, mergeRows, relation, projections)
- }
-
- private def actionCondition(action: MergeAction): Expression = {
- action.condition.getOrElse(TrueLiteral)
- }
-
- private def actionOutput(
- clause: MergeAction,
- metadataAttrs: Seq[Attribute]): Seq[Expression] = {
-
- clause match {
- case u: UpdateAction =>
- u.assignments.map(_.value) ++ metadataAttrs
-
- case _: DeleteAction =>
- Nil
-
- case i: InsertAction =>
- i.assignments.map(_.value) ++ metadataAttrs.map(attr => Literal(null, attr.dataType))
-
- case other =>
- throw new AnalysisException(s"Unexpected action: $other")
- }
- }
-
- private def deltaActionOutput(
- action: MergeAction,
- deleteRowValues: Seq[Expression],
- metadataAttrs: Seq[Attribute]): Seq[Expression] = {
-
- action match {
- case u: UpdateAction =>
- Seq(Literal(UPDATE_OPERATION)) ++ u.assignments.map(_.value) ++ metadataAttrs
-
- case _: DeleteAction =>
- Seq(Literal(DELETE_OPERATION)) ++ deleteRowValues ++ metadataAttrs
-
- case i: InsertAction =>
- val metadataAttrValues = metadataAttrs.map(attr => Literal(null, attr.dataType))
- Seq(Literal(INSERT_OPERATION)) ++ i.assignments.map(_.value) ++ metadataAttrValues
-
- case other =>
- throw new AnalysisException(s"Unexpected action: $other")
- }
- }
-
- private def buildMergeRowsOutput(
- matchedOutputs: Seq[Seq[Expression]],
- notMatchedOutputs: Seq[Seq[Expression]],
- attrs: Seq[Attribute]): Seq[Attribute] = {
-
- // collect all outputs from matched and not matched actions (ignoring DELETEs)
- val outputs = matchedOutputs.filter(_.nonEmpty) ++ notMatchedOutputs.filter(_.nonEmpty)
-
- // build a correct nullability map for output attributes
- // an attribute is nullable if at least one matched or not matched action may produce null
- val nullabilityMap = attrs.indices.map { index =>
- index -> outputs.exists(output => output(index).nullable)
- }.toMap
-
- attrs.zipWithIndex.map { case (attr, index) =>
- AttributeReference(attr.name, attr.dataType, nullabilityMap(index), attr.metadata)()
- }
- }
-
- private def isCardinalityCheckNeeded(actions: Seq[MergeAction]): Boolean = actions match {
- case Seq(DeleteAction(None)) => false
- case _ => true
- }
-
- private def buildDeltaDeleteRowValues(
- rowAttrs: Seq[Attribute],
- rowIdAttrs: Seq[Attribute]): Seq[Expression] = {
-
- // nullify all row attrs that are not part of the row ID
- val rowIdAttSet = AttributeSet(rowIdAttrs)
- rowAttrs.map {
- case attr if rowIdAttSet.contains(attr) => attr
- case attr => Literal(null, attr.dataType)
- }
- }
-
- private def resolveAttrRef(ref: NamedReference, plan: LogicalPlan): AttributeReference = {
- ExtendedV2ExpressionUtils.resolveRef[AttributeReference](ref, plan)
- }
-
- private def buildMergeDeltaProjections(
- mergeRows: MergeRows,
- rowAttrs: Seq[Attribute],
- rowIdAttrs: Seq[Attribute],
- metadataAttrs: Seq[Attribute]): WriteDeltaProjections = {
-
- val outputAttrs = mergeRows.output
-
- val outputs = mergeRows.matchedOutputs ++ mergeRows.notMatchedOutputs
- val insertAndUpdateOutputs = outputs.filterNot(_.head == Literal(DELETE_OPERATION))
- val updateAndDeleteOutputs = outputs.filterNot(_.head == Literal(INSERT_OPERATION))
-
- val rowProjection = if (rowAttrs.nonEmpty) {
- Some(newLazyProjection(insertAndUpdateOutputs, outputAttrs, rowAttrs))
- } else {
- None
- }
-
- val rowIdProjection = newLazyProjection(updateAndDeleteOutputs, outputAttrs, rowIdAttrs)
-
- val metadataProjection = if (metadataAttrs.nonEmpty) {
- Some(newLazyProjection(updateAndDeleteOutputs, outputAttrs, metadataAttrs))
- } else {
- None
- }
-
- WriteDeltaProjections(rowProjection, rowIdProjection, metadataProjection)
- }
-
- // the projection is done by name, ignoring expr IDs
- private def newLazyProjection(
- outputs: Seq[Seq[Expression]],
- outputAttrs: Seq[Attribute],
- projectedAttrs: Seq[Attribute]): ProjectingInternalRow = {
-
- val projectedOrdinals = projectedAttrs.map(attr => outputAttrs.indexWhere(_.name == attr.name))
-
- val structFields = projectedAttrs.zip(projectedOrdinals).map { case (attr, ordinal) =>
- // output attr is nullable if at least one action may produce null for that attr
- // but row ID and metadata attrs are projected only in update/delete actions and
- // row attrs are projected only in insert/update actions
- // that's why the projection schema must rely only on relevant action outputs
- // instead of blindly inheriting the output attr nullability
- val nullable = outputs.exists(output => output(ordinal).nullable)
- StructField(attr.name, attr.dataType, nullable, attr.metadata)
- }
- val schema = StructType(structFields)
-
- ProjectingInternalRow(schema, projectedOrdinals)
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteRowLevelCommand.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteRowLevelCommand.scala
deleted file mode 100644
index f3f655181417..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteRowLevelCommand.scala
+++ /dev/null
@@ -1,137 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.analysis
-
-import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.ProjectingInternalRow
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.catalyst.expressions.AttributeReference
-import org.apache.spark.sql.catalyst.expressions.ExprId
-import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.rules.Rule
-import org.apache.spark.sql.catalyst.util.WriteDeltaProjections
-import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
-import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation
-import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command
-import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
-import org.apache.spark.sql.connector.write.RowLevelOperationInfoImpl
-import org.apache.spark.sql.connector.write.RowLevelOperationTable
-import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
-import org.apache.spark.sql.types.StructType
-import org.apache.spark.sql.util.CaseInsensitiveStringMap
-import scala.collection.mutable
-
-trait RewriteRowLevelCommand extends Rule[LogicalPlan] {
-
- protected def buildRowLevelOperation(
- table: SupportsRowLevelOperations,
- command: Command): RowLevelOperation = {
- val info = RowLevelOperationInfoImpl(command, CaseInsensitiveStringMap.empty())
- val builder = table.newRowLevelOperationBuilder(info)
- builder.build()
- }
-
- protected def buildReadRelation(
- relation: DataSourceV2Relation,
- table: RowLevelOperationTable,
- metadataAttrs: Seq[AttributeReference],
- rowIdAttrs: Seq[AttributeReference] = Nil): DataSourceV2Relation = {
-
- val attrs = dedupAttrs(relation.output ++ rowIdAttrs ++ metadataAttrs)
- relation.copy(table = table, output = attrs)
- }
-
- protected def dedupAttrs(attrs: Seq[AttributeReference]): Seq[AttributeReference] = {
- val exprIds = mutable.Set.empty[ExprId]
- attrs.flatMap { attr =>
- if (exprIds.contains(attr.exprId)) {
- None
- } else {
- exprIds += attr.exprId
- Some(attr)
- }
- }
- }
-
- protected def buildWriteDeltaProjections(
- plan: LogicalPlan,
- rowAttrs: Seq[Attribute],
- rowIdAttrs: Seq[Attribute],
- metadataAttrs: Seq[Attribute]): WriteDeltaProjections = {
-
- val rowProjection = if (rowAttrs.nonEmpty) {
- Some(newLazyProjection(plan, rowAttrs))
- } else {
- None
- }
-
- val rowIdProjection = newLazyProjection(plan, rowIdAttrs)
-
- val metadataProjection = if (metadataAttrs.nonEmpty) {
- Some(newLazyProjection(plan, metadataAttrs))
- } else {
- None
- }
-
- WriteDeltaProjections(rowProjection, rowIdProjection, metadataProjection)
- }
-
- // the projection is done by name, ignoring expr IDs
- private def newLazyProjection(
- plan: LogicalPlan,
- projectedAttrs: Seq[Attribute]): ProjectingInternalRow = {
-
- val projectedOrdinals = projectedAttrs.map(attr => plan.output.indexWhere(_.name == attr.name))
- val schema = StructType.fromAttributes(projectedOrdinals.map(plan.output(_)))
- ProjectingInternalRow(schema, projectedOrdinals)
- }
-
- protected def resolveRequiredMetadataAttrs(
- relation: DataSourceV2Relation,
- operation: RowLevelOperation): Seq[AttributeReference] = {
-
- ExtendedV2ExpressionUtils.resolveRefs[AttributeReference](
- operation.requiredMetadataAttributes.toSeq,
- relation)
- }
-
- protected def resolveRowIdAttrs(
- relation: DataSourceV2Relation,
- operation: RowLevelOperation): Seq[AttributeReference] = {
-
- operation match {
- case supportsDelta: SupportsDelta =>
- val rowIdAttrs = ExtendedV2ExpressionUtils.resolveRefs[AttributeReference](
- supportsDelta.rowId.toSeq,
- relation)
-
- val nullableRowIdAttrs = rowIdAttrs.filter(_.nullable)
- if (nullableRowIdAttrs.nonEmpty) {
- throw new AnalysisException(s"Row ID attrs cannot be nullable: $nullableRowIdAttrs")
- }
-
- rowIdAttrs
-
- case other =>
- throw new AnalysisException(s"Operation $other does not support deltas")
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteUpdateTable.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteUpdateTable.scala
deleted file mode 100644
index 110b38d314b7..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteUpdateTable.scala
+++ /dev/null
@@ -1,183 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.analysis
-
-import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.expressions.Alias
-import org.apache.spark.sql.catalyst.expressions.EqualNullSafe
-import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.expressions.If
-import org.apache.spark.sql.catalyst.expressions.Literal
-import org.apache.spark.sql.catalyst.expressions.Not
-import org.apache.spark.sql.catalyst.expressions.SubqueryExpression
-import org.apache.spark.sql.catalyst.plans.logical.Assignment
-import org.apache.spark.sql.catalyst.plans.logical.Filter
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.plans.logical.Project
-import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
-import org.apache.spark.sql.catalyst.plans.logical.Union
-import org.apache.spark.sql.catalyst.plans.logical.UpdateIcebergTable
-import org.apache.spark.sql.catalyst.plans.logical.WriteDelta
-import org.apache.spark.sql.catalyst.util.RowDeltaUtils._
-import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
-import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.UPDATE
-import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
-import org.apache.spark.sql.connector.write.RowLevelOperationTable
-import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
-
-/**
- * Assigns a rewrite plan for v2 tables that support rewriting data to handle UPDATE statements.
- *
- * This rule assumes the commands have been fully resolved and all assignments have been aligned.
- * That's why it must be run after AlignRowLevelCommandAssignments.
- *
- * This rule also must be run in the same batch with DeduplicateRelations in Spark.
- */
-object RewriteUpdateTable extends RewriteRowLevelCommand {
-
- override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
- case u @ UpdateIcebergTable(aliasedTable, assignments, cond, None) if u.resolved && u.aligned =>
- EliminateSubqueryAliases(aliasedTable) match {
- case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
- val operation = buildRowLevelOperation(tbl, UPDATE)
- val table = RowLevelOperationTable(tbl, operation)
- val updateCond = cond.getOrElse(Literal.TrueLiteral)
- val rewritePlan = operation match {
- case _: SupportsDelta =>
- buildWriteDeltaPlan(r, table, assignments, updateCond)
- case _ if SubqueryExpression.hasSubquery(updateCond) =>
- buildReplaceDataWithUnionPlan(r, table, assignments, updateCond)
- case _ =>
- buildReplaceDataPlan(r, table, assignments, updateCond)
- }
- UpdateIcebergTable(r, assignments, cond, Some(rewritePlan))
-
- case p =>
- throw new AnalysisException(s"$p is not an Iceberg table")
- }
- }
-
- // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
- // if the condition does NOT contain a subquery
- private def buildReplaceDataPlan(
- relation: DataSourceV2Relation,
- operationTable: RowLevelOperationTable,
- assignments: Seq[Assignment],
- cond: Expression): ReplaceData = {
-
- // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
- val metadataAttrs = resolveRequiredMetadataAttrs(relation, operationTable.operation)
-
- // construct a read relation and include all required metadata columns
- val readRelation = buildReadRelation(relation, operationTable, metadataAttrs)
-
- // build a plan with updated and copied over records
- val updatedAndRemainingRowsPlan = buildUpdateProjection(readRelation, assignments, cond)
-
- // build a plan to replace read groups in the table
- val writeRelation = relation.copy(table = operationTable)
- ReplaceData(writeRelation, updatedAndRemainingRowsPlan, relation)
- }
-
- // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
- // if the condition contains a subquery
- private def buildReplaceDataWithUnionPlan(
- relation: DataSourceV2Relation,
- operationTable: RowLevelOperationTable,
- assignments: Seq[Assignment],
- cond: Expression): ReplaceData = {
-
- // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
- val metadataAttrs = resolveRequiredMetadataAttrs(relation, operationTable.operation)
-
- // construct a read relation and include all required metadata columns
- // the same read relation will be used to read records that must be updated and be copied over
- // DeduplicateRelations will take care of duplicated attr IDs
- val readRelation = buildReadRelation(relation, operationTable, metadataAttrs)
-
- // build a plan for records that match the cond and should be updated
- val matchedRowsPlan = Filter(cond, readRelation)
- val updatedRowsPlan = buildUpdateProjection(matchedRowsPlan, assignments)
-
- // build a plan for records that did not match the cond but had to be copied over
- val remainingRowFilter = Not(EqualNullSafe(cond, Literal.TrueLiteral))
- val remainingRowsPlan = Filter(remainingRowFilter, readRelation)
-
- // new state is a union of updated and copied over records
- val updatedAndRemainingRowsPlan = Union(updatedRowsPlan, remainingRowsPlan)
-
- // build a plan to replace read groups in the table
- val writeRelation = relation.copy(table = operationTable)
- ReplaceData(writeRelation, updatedAndRemainingRowsPlan, relation)
- }
-
- // build a rewrite plan for sources that support row deltas
- private def buildWriteDeltaPlan(
- relation: DataSourceV2Relation,
- operationTable: RowLevelOperationTable,
- assignments: Seq[Assignment],
- cond: Expression): WriteDelta = {
-
- // resolve all needed attrs (e.g. row ID and any required metadata attrs)
- val rowAttrs = relation.output
- val rowIdAttrs = resolveRowIdAttrs(relation, operationTable.operation)
- val metadataAttrs = resolveRequiredMetadataAttrs(relation, operationTable.operation)
-
- // construct a scan relation and include all required metadata columns
- val readRelation = buildReadRelation(relation, operationTable, metadataAttrs, rowIdAttrs)
-
- // build a plan for updated records that match the cond
- val matchedRowsPlan = Filter(cond, readRelation)
- val updatedRowsPlan = buildUpdateProjection(matchedRowsPlan, assignments)
- val operationType = Alias(Literal(UPDATE_OPERATION), OPERATION_COLUMN)()
- val project = Project(operationType +: updatedRowsPlan.output, updatedRowsPlan)
-
- // build a plan to write the row delta to the table
- val writeRelation = relation.copy(table = operationTable)
- val projections = buildWriteDeltaProjections(project, rowAttrs, rowIdAttrs, metadataAttrs)
- WriteDelta(writeRelation, project, relation, projections)
- }
-
- // this method assumes the assignments have been already aligned before
- // the condition passed to this method may be different from the UPDATE condition
- private def buildUpdateProjection(
- plan: LogicalPlan,
- assignments: Seq[Assignment],
- cond: Expression = Literal.TrueLiteral): LogicalPlan = {
-
- // TODO: avoid executing the condition for each column
-
- // the plan output may include metadata columns that are not modified
- // that's why the number of assignments may not match the number of plan output columns
-
- val assignedValues = assignments.map(_.value)
- val updatedValues = plan.output.zipWithIndex.map { case (attr, index) =>
- if (index < assignments.size) {
- val assignedExpr = assignedValues(index)
- val updatedValue = If(cond, assignedExpr, attr)
- Alias(updatedValue, attr.name)()
- } else {
- attr
- }
- }
-
- Project(updatedValues, plan)
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/expressions/AssignmentUtils.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/expressions/AssignmentUtils.scala
deleted file mode 100644
index ce3818922c78..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/expressions/AssignmentUtils.scala
+++ /dev/null
@@ -1,86 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.expressions
-
-import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.SQLConfHelper
-import org.apache.spark.sql.catalyst.plans.logical.Assignment
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.util.CharVarcharUtils
-import org.apache.spark.sql.types.DataType
-
-object AssignmentUtils extends SQLConfHelper {
-
- /**
- * Checks whether assignments are aligned and match table columns.
- *
- * @param table a target table
- * @param assignments assignments to check
- * @return true if the assignments are aligned
- */
- def aligned(table: LogicalPlan, assignments: Seq[Assignment]): Boolean = {
- val sameSize = table.output.size == assignments.size
- sameSize && table.output.zip(assignments).forall { case (attr, assignment) =>
- val key = assignment.key
- val value = assignment.value
- val refsEqual = toAssignmentRef(attr).zip(toAssignmentRef(key))
- .forall{ case (attrRef, keyRef) => conf.resolver(attrRef, keyRef)}
-
- refsEqual &&
- DataType.equalsIgnoreCompatibleNullability(value.dataType, attr.dataType) &&
- (attr.nullable || !value.nullable)
- }
- }
-
- def toAssignmentRef(expr: Expression): Seq[String] = expr match {
- case attr: AttributeReference =>
- Seq(attr.name)
- case Alias(child, _) =>
- toAssignmentRef(child)
- case GetStructField(child, _, Some(name)) =>
- toAssignmentRef(child) :+ name
- case other: ExtractValue =>
- throw new AnalysisException(s"Updating nested fields is only supported for structs: $other")
- case other =>
- throw new AnalysisException(s"Cannot convert to a reference, unsupported expression: $other")
- }
-
- def handleCharVarcharLimits(assignment: Assignment): Assignment = {
- val key = assignment.key
- val value = assignment.value
-
- val rawKeyType = key.transform {
- case attr: AttributeReference =>
- CharVarcharUtils.getRawType(attr.metadata)
- .map(attr.withDataType)
- .getOrElse(attr)
- }.dataType
-
- if (CharVarcharUtils.hasCharVarchar(rawKeyType)) {
- val newKey = key.transform {
- case attr: AttributeReference => CharVarcharUtils.cleanAttrMetadata(attr)
- }
- val newValue = CharVarcharUtils.stringLengthCheck(value, rawKeyType)
- Assignment(newKey, newValue)
- } else {
- assignment
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExtendedV2ExpressionUtils.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExtendedV2ExpressionUtils.scala
deleted file mode 100644
index b872281d7f21..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExtendedV2ExpressionUtils.scala
+++ /dev/null
@@ -1,96 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.expressions
-
-import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.SQLConfHelper
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.connector.expressions.{Expression => V2Expression}
-import org.apache.spark.sql.connector.expressions.{SortDirection => V2SortDirection}
-import org.apache.spark.sql.connector.expressions.{NullOrdering => V2NullOrdering}
-import org.apache.spark.sql.connector.expressions.BucketTransform
-import org.apache.spark.sql.connector.expressions.DaysTransform
-import org.apache.spark.sql.connector.expressions.FieldReference
-import org.apache.spark.sql.connector.expressions.HoursTransform
-import org.apache.spark.sql.connector.expressions.IdentityTransform
-import org.apache.spark.sql.connector.expressions.MonthsTransform
-import org.apache.spark.sql.connector.expressions.NamedReference
-import org.apache.spark.sql.connector.expressions.SortValue
-import org.apache.spark.sql.connector.expressions.TruncateTransform
-import org.apache.spark.sql.connector.expressions.YearsTransform
-import org.apache.spark.sql.errors.QueryCompilationErrors
-
-/**
- * A class that is inspired by V2ExpressionUtils in Spark but supports Iceberg transforms.
- */
-object ExtendedV2ExpressionUtils extends SQLConfHelper {
- import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.MultipartIdentifierHelper
-
- def resolveRef[T <: NamedExpression](ref: NamedReference, plan: LogicalPlan): T = {
- plan.resolve(ref.fieldNames.toSeq, conf.resolver) match {
- case Some(namedExpr) =>
- namedExpr.asInstanceOf[T]
- case None =>
- val name = ref.fieldNames.toSeq.quoted
- val outputString = plan.output.map(_.name).mkString(",")
- throw QueryCompilationErrors.cannotResolveAttributeError(name, outputString)
- }
- }
-
- def resolveRefs[T <: NamedExpression](refs: Seq[NamedReference], plan: LogicalPlan): Seq[T] = {
- refs.map(ref => resolveRef[T](ref, plan))
- }
-
- def toCatalyst(expr: V2Expression, query: LogicalPlan): Expression = {
- expr match {
- case SortValue(child, direction, nullOrdering) =>
- val catalystChild = toCatalyst(child, query)
- SortOrder(catalystChild, toCatalyst(direction), toCatalyst(nullOrdering), Seq.empty)
- case IdentityTransform(ref) =>
- resolveRef[NamedExpression](ref, query)
- case BucketTransform(numBuckets, ref) =>
- IcebergBucketTransform(numBuckets, resolveRef[NamedExpression](ref, query))
- case TruncateTransform(length, ref) =>
- IcebergTruncateTransform(resolveRef[NamedExpression](ref, query), length)
- case YearsTransform(ref) =>
- IcebergYearTransform(resolveRef[NamedExpression](ref, query))
- case MonthsTransform(ref) =>
- IcebergMonthTransform(resolveRef[NamedExpression](ref, query))
- case DaysTransform(ref) =>
- IcebergDayTransform(resolveRef[NamedExpression](ref, query))
- case HoursTransform(ref) =>
- IcebergHourTransform(resolveRef[NamedExpression](ref, query))
- case ref: FieldReference =>
- resolveRef[NamedExpression](ref, query)
- case _ =>
- throw new AnalysisException(s"$expr is not currently supported")
- }
- }
-
- private def toCatalyst(direction: V2SortDirection): SortDirection = direction match {
- case V2SortDirection.ASCENDING => Ascending
- case V2SortDirection.DESCENDING => Descending
- }
-
- private def toCatalyst(nullOrdering: V2NullOrdering): NullOrdering = nullOrdering match {
- case V2NullOrdering.NULLS_FIRST => NullsFirst
- case V2NullOrdering.NULLS_LAST => NullsLast
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ExtendedReplaceNullWithFalseInPredicate.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ExtendedReplaceNullWithFalseInPredicate.scala
deleted file mode 100644
index 4d369ca0793d..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ExtendedReplaceNullWithFalseInPredicate.scala
+++ /dev/null
@@ -1,132 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.optimizer
-
-import org.apache.spark.sql.catalyst.expressions.And
-import org.apache.spark.sql.catalyst.expressions.CaseWhen
-import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.expressions.If
-import org.apache.spark.sql.catalyst.expressions.In
-import org.apache.spark.sql.catalyst.expressions.InSet
-import org.apache.spark.sql.catalyst.expressions.Literal
-import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
-import org.apache.spark.sql.catalyst.expressions.Not
-import org.apache.spark.sql.catalyst.expressions.Or
-import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
-import org.apache.spark.sql.catalyst.plans.logical.DeleteFromIcebergTable
-import org.apache.spark.sql.catalyst.plans.logical.InsertAction
-import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.plans.logical.MergeAction
-import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
-import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
-import org.apache.spark.sql.catalyst.plans.logical.UpdateIcebergTable
-import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
-import org.apache.spark.sql.catalyst.rules.Rule
-import org.apache.spark.sql.catalyst.trees.TreePattern.INSET
-import org.apache.spark.sql.catalyst.trees.TreePattern.NULL_LITERAL
-import org.apache.spark.sql.catalyst.trees.TreePattern.TRUE_OR_FALSE_LITERAL
-import org.apache.spark.sql.types.BooleanType
-import org.apache.spark.util.Utils
-
-/**
- * A rule similar to ReplaceNullWithFalseInPredicate in Spark but applies to Iceberg row-level commands.
- */
-object ExtendedReplaceNullWithFalseInPredicate extends Rule[LogicalPlan] {
-
- def apply(plan: LogicalPlan): LogicalPlan = plan.transformWithPruning(
- _.containsAnyPattern(NULL_LITERAL, TRUE_OR_FALSE_LITERAL, INSET)) {
-
- case d @ DeleteFromIcebergTable(_, Some(cond), _) =>
- d.copy(condition = Some(replaceNullWithFalse(cond)))
-
- case u @ UpdateIcebergTable(_, _, Some(cond), _) =>
- u.copy(condition = Some(replaceNullWithFalse(cond)))
-
- case m @ MergeIntoIcebergTable(_, _, mergeCond, matchedActions, notMatchedActions, _) =>
- m.copy(
- mergeCondition = replaceNullWithFalse(mergeCond),
- matchedActions = replaceNullWithFalse(matchedActions),
- notMatchedActions = replaceNullWithFalse(notMatchedActions))
- }
-
- /**
- * Recursively traverse the Boolean-type expression to replace
- * `Literal(null, BooleanType)` with `FalseLiteral`, if possible.
- *
- * Note that `transformExpressionsDown` can not be used here as we must stop as soon as we hit
- * an expression that is not [[CaseWhen]], [[If]], [[And]], [[Or]] or
- * `Literal(null, BooleanType)`.
- */
- private def replaceNullWithFalse(e: Expression): Expression = e match {
- case Literal(null, BooleanType) =>
- FalseLiteral
- // In SQL, the `Not(IN)` expression evaluates as follows:
- // `NULL not in (1)` -> NULL
- // `NULL not in (1, NULL)` -> NULL
- // `1 not in (1, NULL)` -> false
- // `1 not in (2, NULL)` -> NULL
- // In predicate, NULL is equal to false, so we can simplify them to false directly.
- case Not(In(value, list)) if (value +: list).exists(isNullLiteral) =>
- FalseLiteral
- case Not(InSet(value, list)) if isNullLiteral(value) || list.contains(null) =>
- FalseLiteral
-
- case And(left, right) =>
- And(replaceNullWithFalse(left), replaceNullWithFalse(right))
- case Or(left, right) =>
- Or(replaceNullWithFalse(left), replaceNullWithFalse(right))
- case cw: CaseWhen if cw.dataType == BooleanType =>
- val newBranches = cw.branches.map { case (cond, value) =>
- replaceNullWithFalse(cond) -> replaceNullWithFalse(value)
- }
- val newElseValue = cw.elseValue.map(replaceNullWithFalse).getOrElse(FalseLiteral)
- CaseWhen(newBranches, newElseValue)
- case i @ If(pred, trueVal, falseVal) if i.dataType == BooleanType =>
- If(replaceNullWithFalse(pred), replaceNullWithFalse(trueVal), replaceNullWithFalse(falseVal))
- case e if e.dataType == BooleanType =>
- e
- case e =>
- val message = "Expected a Boolean type expression in replaceNullWithFalse, " +
- s"but got the type `${e.dataType.catalogString}` in `${e.sql}`."
- if (Utils.isTesting) {
- throw new IllegalArgumentException(message)
- } else {
- logWarning(message)
- e
- }
- }
-
- private def isNullLiteral(e: Expression): Boolean = e match {
- case Literal(null, _) => true
- case _ => false
- }
-
- private def replaceNullWithFalse(mergeActions: Seq[MergeAction]): Seq[MergeAction] = {
- mergeActions.map {
- case u @ UpdateAction(Some(cond), _) => u.copy(condition = Some(replaceNullWithFalse(cond)))
- case u @ UpdateStarAction(Some(cond)) => u.copy(condition = Some(replaceNullWithFalse(cond)))
- case d @ DeleteAction(Some(cond)) => d.copy(condition = Some(replaceNullWithFalse(cond)))
- case i @ InsertAction(Some(cond), _) => i.copy(condition = Some(replaceNullWithFalse(cond)))
- case i @ InsertStarAction(Some(cond)) => i.copy(condition = Some(replaceNullWithFalse(cond)))
- case other => other
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ExtendedSimplifyConditionalsInPredicate.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ExtendedSimplifyConditionalsInPredicate.scala
deleted file mode 100644
index 74cf922c4a98..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ExtendedSimplifyConditionalsInPredicate.scala
+++ /dev/null
@@ -1,94 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.optimizer
-
-import org.apache.spark.sql.catalyst.expressions.And
-import org.apache.spark.sql.catalyst.expressions.CaseWhen
-import org.apache.spark.sql.catalyst.expressions.Coalesce
-import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.expressions.If
-import org.apache.spark.sql.catalyst.expressions.Literal
-import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
-import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
-import org.apache.spark.sql.catalyst.expressions.Not
-import org.apache.spark.sql.catalyst.expressions.Or
-import org.apache.spark.sql.catalyst.plans.logical._
-import org.apache.spark.sql.catalyst.rules.Rule
-import org.apache.spark.sql.catalyst.trees.TreePattern.CASE_WHEN
-import org.apache.spark.sql.catalyst.trees.TreePattern.IF
-import org.apache.spark.sql.types.BooleanType
-
-/**
- * A rule similar to SimplifyConditionalsInPredicate in Spark but applies to Iceberg row-level commands.
- */
-object ExtendedSimplifyConditionalsInPredicate extends Rule[LogicalPlan] {
-
- def apply(plan: LogicalPlan): LogicalPlan = plan.transformWithPruning(
- _.containsAnyPattern(CASE_WHEN, IF)) {
-
- case d @ DeleteFromIcebergTable(_, Some(cond), _) =>
- d.copy(condition = Some(simplifyConditional(cond)))
-
- case u @ UpdateIcebergTable(_, _, Some(cond), _) =>
- u.copy(condition = Some(simplifyConditional(cond)))
-
- case m @ MergeIntoIcebergTable(_, _, mergeCond, matchedActions, notMatchedActions, _) =>
- m.copy(
- mergeCondition = simplifyConditional(mergeCond),
- matchedActions = simplifyConditional(matchedActions),
- notMatchedActions = simplifyConditional(notMatchedActions))
- }
-
- private def simplifyConditional(e: Expression): Expression = e match {
- case And(left, right) => And(simplifyConditional(left), simplifyConditional(right))
- case Or(left, right) => Or(simplifyConditional(left), simplifyConditional(right))
- case If(cond, trueValue, FalseLiteral) => And(cond, trueValue)
- case If(cond, trueValue, TrueLiteral) => Or(Not(Coalesce(Seq(cond, FalseLiteral))), trueValue)
- case If(cond, FalseLiteral, falseValue) =>
- And(Not(Coalesce(Seq(cond, FalseLiteral))), falseValue)
- case If(cond, TrueLiteral, falseValue) => Or(cond, falseValue)
- case CaseWhen(Seq((cond, trueValue)),
- Some(FalseLiteral) | Some(Literal(null, BooleanType)) | None) =>
- And(cond, trueValue)
- case CaseWhen(Seq((cond, trueValue)), Some(TrueLiteral)) =>
- Or(Not(Coalesce(Seq(cond, FalseLiteral))), trueValue)
- case CaseWhen(Seq((cond, FalseLiteral)), Some(elseValue)) =>
- And(Not(Coalesce(Seq(cond, FalseLiteral))), elseValue)
- case CaseWhen(Seq((cond, TrueLiteral)), Some(elseValue)) =>
- Or(cond, elseValue)
- case e if e.dataType == BooleanType => e
- case e =>
- assert(e.dataType != BooleanType,
- "Expected a Boolean type expression in ExtendedSimplifyConditionalsInPredicate, " +
- s"but got the type `${e.dataType.catalogString}` in `${e.sql}`.")
- e
- }
-
- private def simplifyConditional(mergeActions: Seq[MergeAction]): Seq[MergeAction] = {
- mergeActions.map {
- case u @ UpdateAction(Some(cond), _) => u.copy(condition = Some(simplifyConditional(cond)))
- case u @ UpdateStarAction(Some(cond)) => u.copy(condition = Some(simplifyConditional(cond)))
- case d @ DeleteAction(Some(cond)) => d.copy(condition = Some(simplifyConditional(cond)))
- case i @ InsertAction(Some(cond), _) => i.copy(condition = Some(simplifyConditional(cond)))
- case i @ InsertStarAction(Some(cond)) => i.copy(condition = Some(simplifyConditional(cond)))
- case other => other
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSparkSqlExtensionsParser.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSparkSqlExtensionsParser.scala
deleted file mode 100644
index 2996ceb366e1..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSparkSqlExtensionsParser.scala
+++ /dev/null
@@ -1,391 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.parser.extensions
-
-import java.util.Locale
-import org.antlr.v4.runtime._
-import org.antlr.v4.runtime.atn.PredictionMode
-import org.antlr.v4.runtime.misc.Interval
-import org.antlr.v4.runtime.misc.ParseCancellationException
-import org.antlr.v4.runtime.tree.TerminalNodeImpl
-import org.apache.iceberg.common.DynConstructors
-import org.apache.iceberg.spark.ExtendedParser
-import org.apache.iceberg.spark.ExtendedParser.RawOrderField
-import org.apache.iceberg.spark.Spark3Util
-import org.apache.iceberg.spark.source.SparkTable
-import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.SparkSession
-import org.apache.spark.sql.catalyst.FunctionIdentifier
-import org.apache.spark.sql.catalyst.TableIdentifier
-import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
-import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
-import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.parser.ParserInterface
-import org.apache.spark.sql.catalyst.parser.extensions.IcebergSqlExtensionsParser.NonReservedContext
-import org.apache.spark.sql.catalyst.parser.extensions.IcebergSqlExtensionsParser.QuotedIdentifierContext
-import org.apache.spark.sql.catalyst.plans.logical.DeleteFromIcebergTable
-import org.apache.spark.sql.catalyst.plans.logical.DeleteFromTable
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext
-import org.apache.spark.sql.catalyst.plans.logical.MergeIntoTable
-import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
-import org.apache.spark.sql.catalyst.plans.logical.UpdateIcebergTable
-import org.apache.spark.sql.catalyst.plans.logical.UpdateTable
-import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.sql.connector.catalog.Table
-import org.apache.spark.sql.connector.catalog.TableCatalog
-import org.apache.spark.sql.execution.command.ExplainCommand
-import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.internal.VariableSubstitution
-import org.apache.spark.sql.types.DataType
-import org.apache.spark.sql.types.StructType
-import scala.jdk.CollectionConverters._
-import scala.util.Try
-
-class IcebergSparkSqlExtensionsParser(delegate: ParserInterface) extends ParserInterface with ExtendedParser {
-
- import IcebergSparkSqlExtensionsParser._
-
- private lazy val substitutor = substitutorCtor.newInstance(SQLConf.get)
- private lazy val astBuilder = new IcebergSqlExtensionsAstBuilder(delegate)
-
- /**
- * Parse a string to a DataType.
- */
- override def parseDataType(sqlText: String): DataType = {
- delegate.parseDataType(sqlText)
- }
-
- /**
- * Parse a string to a raw DataType without CHAR/VARCHAR replacement.
- */
- def parseRawDataType(sqlText: String): DataType = throw new UnsupportedOperationException()
-
- /**
- * Parse a string to an Expression.
- */
- override def parseExpression(sqlText: String): Expression = {
- delegate.parseExpression(sqlText)
- }
-
- /**
- * Parse a string to a TableIdentifier.
- */
- override def parseTableIdentifier(sqlText: String): TableIdentifier = {
- delegate.parseTableIdentifier(sqlText)
- }
-
- /**
- * Parse a string to a FunctionIdentifier.
- */
- override def parseFunctionIdentifier(sqlText: String): FunctionIdentifier = {
- delegate.parseFunctionIdentifier(sqlText)
- }
-
- /**
- * Parse a string to a multi-part identifier.
- */
- override def parseMultipartIdentifier(sqlText: String): Seq[String] = {
- delegate.parseMultipartIdentifier(sqlText)
- }
-
- /**
- * Creates StructType for a given SQL string, which is a comma separated list of field
- * definitions which will preserve the correct Hive metadata.
- */
- override def parseTableSchema(sqlText: String): StructType = {
- delegate.parseTableSchema(sqlText)
- }
-
- override def parseSortOrder(sqlText: String): java.util.List[RawOrderField] = {
- val fields = parse(sqlText) { parser => astBuilder.visitSingleOrder(parser.singleOrder()) }
- fields.map { field =>
- val (term, direction, order) = field
- new RawOrderField(term, direction, order)
- }.asJava
- }
-
- /**
- * Parse a string to a LogicalPlan.
- */
- override def parsePlan(sqlText: String): LogicalPlan = {
- val sqlTextAfterSubstitution = substitutor.substitute(sqlText)
- if (isIcebergCommand(sqlTextAfterSubstitution)) {
- parse(sqlTextAfterSubstitution) { parser => astBuilder.visit(parser.singleStatement()) }.asInstanceOf[LogicalPlan]
- } else {
- val parsedPlan = delegate.parsePlan(sqlText)
- parsedPlan match {
- case e: ExplainCommand =>
- e.copy(logicalPlan = replaceRowLevelCommands(e.logicalPlan))
- case p =>
- replaceRowLevelCommands(p)
- }
- }
- }
-
- private def replaceRowLevelCommands(plan: LogicalPlan): LogicalPlan = plan resolveOperatorsDown {
- case DeleteFromTable(UnresolvedIcebergTable(aliasedTable), condition) =>
- DeleteFromIcebergTable(aliasedTable, condition)
-
- case UpdateTable(UnresolvedIcebergTable(aliasedTable), assignments, condition) =>
- UpdateIcebergTable(aliasedTable, assignments, condition)
-
- case MergeIntoTable(UnresolvedIcebergTable(aliasedTable), source, cond, matchedActions, notMatchedActions) =>
- // cannot construct MergeIntoIcebergTable right away as MERGE operations require special resolution
- // that's why the condition and actions must be hidden from the regular resolution rules in Spark
- // see ResolveMergeIntoTableReferences for details
- val context = MergeIntoContext(cond, matchedActions, notMatchedActions)
- UnresolvedMergeIntoIcebergTable(aliasedTable, source, context)
- }
-
- object UnresolvedIcebergTable {
-
- def unapply(plan: LogicalPlan): Option[LogicalPlan] = {
- EliminateSubqueryAliases(plan) match {
- case UnresolvedRelation(multipartIdentifier, _, _) if isIcebergTable(multipartIdentifier) =>
- Some(plan)
- case _ =>
- None
- }
- }
-
- private def isIcebergTable(multipartIdent: Seq[String]): Boolean = {
- val catalogAndIdentifier = Spark3Util.catalogAndIdentifier(SparkSession.active, multipartIdent.asJava)
- catalogAndIdentifier.catalog match {
- case tableCatalog: TableCatalog =>
- Try(tableCatalog.loadTable(catalogAndIdentifier.identifier))
- .map(isIcebergTable)
- .getOrElse(false)
-
- case _ =>
- false
- }
- }
-
- private def isIcebergTable(table: Table): Boolean = table match {
- case _: SparkTable => true
- case _ => false
- }
- }
-
- private def isIcebergCommand(sqlText: String): Boolean = {
- val normalized = sqlText.toLowerCase(Locale.ROOT).trim()
- // Strip simple SQL comments that terminate a line, e.g. comments starting with `--` .
- .replaceAll("--.*?\\n", " ")
- // Strip newlines.
- .replaceAll("\\s+", " ")
- // Strip comments of the form /* ... */. This must come after stripping newlines so that
- // comments that span multiple lines are caught.
- .replaceAll("/\\*.*?\\*/", " ")
- .trim()
- normalized.startsWith("call") || (
- normalized.startsWith("alter table") && (
- normalized.contains("add partition field") ||
- normalized.contains("drop partition field") ||
- normalized.contains("replace partition field") ||
- normalized.contains("write ordered by") ||
- normalized.contains("write locally ordered by") ||
- normalized.contains("write distributed by") ||
- normalized.contains("write unordered") ||
- normalized.contains("set identifier fields") ||
- normalized.contains("drop identifier fields") ||
- isSnapshotRefDdl(normalized)))
- }
-
- private def isSnapshotRefDdl(normalized: String): Boolean = {
- normalized.contains("create branch") ||
- normalized.contains("replace branch") ||
- normalized.contains("create tag") ||
- normalized.contains("replace tag") ||
- normalized.contains("drop branch") ||
- normalized.contains("drop tag")
- }
-
- protected def parse[T](command: String)(toResult: IcebergSqlExtensionsParser => T): T = {
- val lexer = new IcebergSqlExtensionsLexer(new UpperCaseCharStream(CharStreams.fromString(command)))
- lexer.removeErrorListeners()
- lexer.addErrorListener(IcebergParseErrorListener)
-
- val tokenStream = new CommonTokenStream(lexer)
- val parser = new IcebergSqlExtensionsParser(tokenStream)
- parser.addParseListener(IcebergSqlExtensionsPostProcessor)
- parser.removeErrorListeners()
- parser.addErrorListener(IcebergParseErrorListener)
-
- try {
- try {
- // first, try parsing with potentially faster SLL mode
- parser.getInterpreter.setPredictionMode(PredictionMode.SLL)
- toResult(parser)
- }
- catch {
- case _: ParseCancellationException =>
- // if we fail, parse with LL mode
- tokenStream.seek(0) // rewind input stream
- parser.reset()
-
- // Try Again.
- parser.getInterpreter.setPredictionMode(PredictionMode.LL)
- toResult(parser)
- }
- }
- catch {
- case e: IcebergParseException if e.command.isDefined =>
- throw e
- case e: IcebergParseException =>
- throw e.withCommand(command)
- case e: AnalysisException =>
- val position = Origin(e.line, e.startPosition)
- throw new IcebergParseException(Option(command), e.message, position, position)
- }
- }
-}
-
-object IcebergSparkSqlExtensionsParser {
- private val substitutorCtor: DynConstructors.Ctor[VariableSubstitution] =
- DynConstructors.builder()
- .impl(classOf[VariableSubstitution])
- .impl(classOf[VariableSubstitution], classOf[SQLConf])
- .build()
-}
-
-/* Copied from Apache Spark's to avoid dependency on Spark Internals */
-class UpperCaseCharStream(wrapped: CodePointCharStream) extends CharStream {
- override def consume(): Unit = wrapped.consume
- override def getSourceName(): String = wrapped.getSourceName
- override def index(): Int = wrapped.index
- override def mark(): Int = wrapped.mark
- override def release(marker: Int): Unit = wrapped.release(marker)
- override def seek(where: Int): Unit = wrapped.seek(where)
- override def size(): Int = wrapped.size
-
- override def getText(interval: Interval): String = wrapped.getText(interval)
-
- // scalastyle:off
- override def LA(i: Int): Int = {
- val la = wrapped.LA(i)
- if (la == 0 || la == IntStream.EOF) la
- else Character.toUpperCase(la)
- }
- // scalastyle:on
-}
-
-/**
- * The post-processor validates & cleans-up the parse tree during the parse process.
- */
-case object IcebergSqlExtensionsPostProcessor extends IcebergSqlExtensionsBaseListener {
-
- /** Remove the back ticks from an Identifier. */
- override def exitQuotedIdentifier(ctx: QuotedIdentifierContext): Unit = {
- replaceTokenByIdentifier(ctx, 1) { token =>
- // Remove the double back ticks in the string.
- token.setText(token.getText.replace("``", "`"))
- token
- }
- }
-
- /** Treat non-reserved keywords as Identifiers. */
- override def exitNonReserved(ctx: NonReservedContext): Unit = {
- replaceTokenByIdentifier(ctx, 0)(identity)
- }
-
- private def replaceTokenByIdentifier(
- ctx: ParserRuleContext,
- stripMargins: Int)(
- f: CommonToken => CommonToken = identity): Unit = {
- val parent = ctx.getParent
- parent.removeLastChild()
- val token = ctx.getChild(0).getPayload.asInstanceOf[Token]
- val newToken = new CommonToken(
- new org.antlr.v4.runtime.misc.Pair(token.getTokenSource, token.getInputStream),
- IcebergSqlExtensionsParser.IDENTIFIER,
- token.getChannel,
- token.getStartIndex + stripMargins,
- token.getStopIndex - stripMargins)
- parent.addChild(new TerminalNodeImpl(f(newToken)))
- }
-}
-
-/* Partially copied from Apache Spark's Parser to avoid dependency on Spark Internals */
-case object IcebergParseErrorListener extends BaseErrorListener {
- override def syntaxError(
- recognizer: Recognizer[_, _],
- offendingSymbol: scala.Any,
- line: Int,
- charPositionInLine: Int,
- msg: String,
- e: RecognitionException): Unit = {
- val (start, stop) = offendingSymbol match {
- case token: CommonToken =>
- val start = Origin(Some(line), Some(token.getCharPositionInLine))
- val length = token.getStopIndex - token.getStartIndex + 1
- val stop = Origin(Some(line), Some(token.getCharPositionInLine + length))
- (start, stop)
- case _ =>
- val start = Origin(Some(line), Some(charPositionInLine))
- (start, start)
- }
- throw new IcebergParseException(None, msg, start, stop)
- }
-}
-
-/**
- * Copied from Apache Spark
- * A [[ParseException]] is an [[AnalysisException]] that is thrown during the parse process. It
- * contains fields and an extended error message that make reporting and diagnosing errors easier.
- */
-class IcebergParseException(
- val command: Option[String],
- message: String,
- val start: Origin,
- val stop: Origin) extends AnalysisException(message, start.line, start.startPosition) {
-
- def this(message: String, ctx: ParserRuleContext) = {
- this(Option(IcebergParserUtils.command(ctx)),
- message,
- IcebergParserUtils.position(ctx.getStart),
- IcebergParserUtils.position(ctx.getStop))
- }
-
- override def getMessage: String = {
- val builder = new StringBuilder
- builder ++= "\n" ++= message
- start match {
- case Origin(Some(l), Some(p)) =>
- builder ++= s"(line $l, pos $p)\n"
- command.foreach { cmd =>
- val (above, below) = cmd.split("\n").splitAt(l)
- builder ++= "\n== SQL ==\n"
- above.foreach(builder ++= _ += '\n')
- builder ++= (0 until p).map(_ => "-").mkString("") ++= "^^^\n"
- below.foreach(builder ++= _ += '\n')
- }
- case _ =>
- command.foreach { cmd =>
- builder ++= "\n== SQL ==\n" ++= cmd
- }
- }
- builder.toString
- }
-
- def withCommand(cmd: String): IcebergParseException = {
- new IcebergParseException(Option(cmd), message, start, stop)
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSqlExtensionsAstBuilder.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSqlExtensionsAstBuilder.scala
deleted file mode 100644
index 2e438de2b8cd..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSqlExtensionsAstBuilder.scala
+++ /dev/null
@@ -1,385 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.parser.extensions
-
-import java.util.Locale
-import java.util.concurrent.TimeUnit
-import org.antlr.v4.runtime._
-import org.antlr.v4.runtime.misc.Interval
-import org.antlr.v4.runtime.tree.ParseTree
-import org.antlr.v4.runtime.tree.TerminalNode
-import org.apache.iceberg.DistributionMode
-import org.apache.iceberg.NullOrder
-import org.apache.iceberg.SortDirection
-import org.apache.iceberg.expressions.Term
-import org.apache.iceberg.spark.Spark3Util
-import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.expressions.Literal
-import org.apache.spark.sql.catalyst.parser.ParserInterface
-import org.apache.spark.sql.catalyst.parser.extensions.IcebergParserUtils.withOrigin
-import org.apache.spark.sql.catalyst.parser.extensions.IcebergSqlExtensionsParser._
-import org.apache.spark.sql.catalyst.plans.logical.AddPartitionField
-import org.apache.spark.sql.catalyst.plans.logical.BranchOptions
-import org.apache.spark.sql.catalyst.plans.logical.CallArgument
-import org.apache.spark.sql.catalyst.plans.logical.CallStatement
-import org.apache.spark.sql.catalyst.plans.logical.CreateOrReplaceBranch
-import org.apache.spark.sql.catalyst.plans.logical.CreateOrReplaceTag
-import org.apache.spark.sql.catalyst.plans.logical.DropBranch
-import org.apache.spark.sql.catalyst.plans.logical.DropIdentifierFields
-import org.apache.spark.sql.catalyst.plans.logical.DropPartitionField
-import org.apache.spark.sql.catalyst.plans.logical.DropTag
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.plans.logical.NamedArgument
-import org.apache.spark.sql.catalyst.plans.logical.PositionalArgument
-import org.apache.spark.sql.catalyst.plans.logical.ReplacePartitionField
-import org.apache.spark.sql.catalyst.plans.logical.SetIdentifierFields
-import org.apache.spark.sql.catalyst.plans.logical.SetWriteDistributionAndOrdering
-import org.apache.spark.sql.catalyst.plans.logical.TagOptions
-import org.apache.spark.sql.catalyst.trees.CurrentOrigin
-import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.sql.connector.expressions
-import org.apache.spark.sql.connector.expressions.ApplyTransform
-import org.apache.spark.sql.connector.expressions.FieldReference
-import org.apache.spark.sql.connector.expressions.IdentityTransform
-import org.apache.spark.sql.connector.expressions.LiteralValue
-import org.apache.spark.sql.connector.expressions.Transform
-import scala.jdk.CollectionConverters._
-
-class IcebergSqlExtensionsAstBuilder(delegate: ParserInterface) extends IcebergSqlExtensionsBaseVisitor[AnyRef] {
-
- private def toBuffer[T](list: java.util.List[T]): scala.collection.mutable.Buffer[T] = list.asScala
- private def toSeq[T](list: java.util.List[T]): Seq[T] = toBuffer(list).toSeq
-
- /**
- * Create a [[CallStatement]] for a stored procedure call.
- */
- override def visitCall(ctx: CallContext): CallStatement = withOrigin(ctx) {
- val name = toSeq(ctx.multipartIdentifier.parts).map(_.getText)
- val args = toSeq(ctx.callArgument).map(typedVisit[CallArgument])
- CallStatement(name, args)
- }
-
- /**
- * Create an ADD PARTITION FIELD logical command.
- */
- override def visitAddPartitionField(ctx: AddPartitionFieldContext): AddPartitionField = withOrigin(ctx) {
- AddPartitionField(
- typedVisit[Seq[String]](ctx.multipartIdentifier),
- typedVisit[Transform](ctx.transform),
- Option(ctx.name).map(_.getText))
- }
-
- /**
- * Create a DROP PARTITION FIELD logical command.
- */
- override def visitDropPartitionField(ctx: DropPartitionFieldContext): DropPartitionField = withOrigin(ctx) {
- DropPartitionField(
- typedVisit[Seq[String]](ctx.multipartIdentifier),
- typedVisit[Transform](ctx.transform))
- }
-
- /**
- * Create a CREATE OR REPLACE BRANCH logical command.
- */
- override def visitCreateOrReplaceBranch(ctx: CreateOrReplaceBranchContext): CreateOrReplaceBranch = withOrigin(ctx) {
- val createOrReplaceBranchClause = ctx.createReplaceBranchClause()
-
- val branchName = createOrReplaceBranchClause.identifier()
- val branchOptionsContext = Option(createOrReplaceBranchClause.branchOptions())
- val snapshotId = branchOptionsContext.flatMap(branchOptions => Option(branchOptions.snapshotId()))
- .map(_.getText.toLong)
- val snapshotRetention = branchOptionsContext.flatMap(branchOptions => Option(branchOptions.snapshotRetention()))
- val minSnapshotsToKeep = snapshotRetention.flatMap(retention => Option(retention.minSnapshotsToKeep()))
- .map(minSnapshots => minSnapshots.number().getText.toLong)
- val maxSnapshotAgeMs = snapshotRetention
- .flatMap(retention => Option(retention.maxSnapshotAge()))
- .map(retention => TimeUnit.valueOf(retention.timeUnit().getText.toUpperCase(Locale.ENGLISH))
- .toMillis(retention.number().getText.toLong))
- val branchRetention = branchOptionsContext.flatMap(branchOptions => Option(branchOptions.refRetain()))
- val branchRefAgeMs = branchRetention.map(retain =>
- TimeUnit.valueOf(retain.timeUnit().getText.toUpperCase(Locale.ENGLISH)).toMillis(retain.number().getText.toLong))
- val create = createOrReplaceBranchClause.CREATE() != null
- val replace = ctx.createReplaceBranchClause().REPLACE() != null
- val ifNotExists = createOrReplaceBranchClause.EXISTS() != null
-
- val branchOptions = BranchOptions(
- snapshotId,
- minSnapshotsToKeep,
- maxSnapshotAgeMs,
- branchRefAgeMs
- )
-
- CreateOrReplaceBranch(
- typedVisit[Seq[String]](ctx.multipartIdentifier),
- branchName.getText,
- branchOptions,
- create,
- replace,
- ifNotExists)
- }
-
- /**
- * Create an CREATE OR REPLACE TAG logical command.
- */
- override def visitCreateOrReplaceTag(ctx: CreateOrReplaceTagContext): CreateOrReplaceTag = withOrigin(ctx) {
- val createTagClause = ctx.createReplaceTagClause()
-
- val tagName = createTagClause.identifier().getText
-
- val tagOptionsContext = Option(createTagClause.tagOptions())
- val snapshotId = tagOptionsContext.flatMap(tagOptions => Option(tagOptions.snapshotId()))
- .map(_.getText.toLong)
- val tagRetain = tagOptionsContext.flatMap(tagOptions => Option(tagOptions.refRetain()))
- val tagRefAgeMs = tagRetain.map(retain =>
- TimeUnit.valueOf(retain.timeUnit().getText.toUpperCase(Locale.ENGLISH)).toMillis(retain.number().getText.toLong))
- val tagOptions = TagOptions(
- snapshotId,
- tagRefAgeMs
- )
-
- val create = createTagClause.CREATE() != null
- val replace = createTagClause.REPLACE() != null
- val ifNotExists = createTagClause.EXISTS() != null
-
- CreateOrReplaceTag(typedVisit[Seq[String]](ctx.multipartIdentifier),
- tagName,
- tagOptions,
- create,
- replace,
- ifNotExists)
- }
-
- /**
- * Create an DROP BRANCH logical command.
- */
- override def visitDropBranch(ctx: DropBranchContext): DropBranch = withOrigin(ctx) {
- DropBranch(typedVisit[Seq[String]](ctx.multipartIdentifier), ctx.identifier().getText, ctx.EXISTS() != null)
- }
-
- /**
- * Create an DROP TAG logical command.
- */
- override def visitDropTag(ctx: DropTagContext): DropTag = withOrigin(ctx) {
- DropTag(typedVisit[Seq[String]](ctx.multipartIdentifier), ctx.identifier().getText, ctx.EXISTS() != null)
- }
-
- /**
- * Create an REPLACE PARTITION FIELD logical command.
- */
- override def visitReplacePartitionField(ctx: ReplacePartitionFieldContext): ReplacePartitionField = withOrigin(ctx) {
- ReplacePartitionField(
- typedVisit[Seq[String]](ctx.multipartIdentifier),
- typedVisit[Transform](ctx.transform(0)),
- typedVisit[Transform](ctx.transform(1)),
- Option(ctx.name).map(_.getText))
- }
-
- /**
- * Create an SET IDENTIFIER FIELDS logical command.
- */
- override def visitSetIdentifierFields(ctx: SetIdentifierFieldsContext): SetIdentifierFields = withOrigin(ctx) {
- SetIdentifierFields(
- typedVisit[Seq[String]](ctx.multipartIdentifier),
- toSeq(ctx.fieldList.fields).map(_.getText))
- }
-
- /**
- * Create an DROP IDENTIFIER FIELDS logical command.
- */
- override def visitDropIdentifierFields(ctx: DropIdentifierFieldsContext): DropIdentifierFields = withOrigin(ctx) {
- DropIdentifierFields(
- typedVisit[Seq[String]](ctx.multipartIdentifier),
- toSeq(ctx.fieldList.fields).map(_.getText))
- }
-
- /**
- * Create a [[SetWriteDistributionAndOrdering]] for changing the write distribution and ordering.
- */
- override def visitSetWriteDistributionAndOrdering(
- ctx: SetWriteDistributionAndOrderingContext): SetWriteDistributionAndOrdering = {
-
- val tableName = typedVisit[Seq[String]](ctx.multipartIdentifier)
-
- val (distributionSpec, orderingSpec) = toDistributionAndOrderingSpec(ctx.writeSpec)
-
- if (distributionSpec == null && orderingSpec == null) {
- throw new AnalysisException(
- "ALTER TABLE has no changes: missing both distribution and ordering clauses")
- }
-
- val distributionMode = if (distributionSpec != null) {
- DistributionMode.HASH
- } else if (orderingSpec.UNORDERED != null || orderingSpec.LOCALLY != null) {
- DistributionMode.NONE
- } else {
- DistributionMode.RANGE
- }
-
- val ordering = if (orderingSpec != null && orderingSpec.order != null) {
- toSeq(orderingSpec.order.fields).map(typedVisit[(Term, SortDirection, NullOrder)])
- } else {
- Seq.empty
- }
-
- SetWriteDistributionAndOrdering(tableName, distributionMode, ordering)
- }
-
- private def toDistributionAndOrderingSpec(
- writeSpec: WriteSpecContext): (WriteDistributionSpecContext, WriteOrderingSpecContext) = {
-
- if (writeSpec.writeDistributionSpec.size > 1) {
- throw new AnalysisException("ALTER TABLE contains multiple distribution clauses")
- }
-
- if (writeSpec.writeOrderingSpec.size > 1) {
- throw new AnalysisException("ALTER TABLE contains multiple ordering clauses")
- }
-
- val distributionSpec = toBuffer(writeSpec.writeDistributionSpec).headOption.orNull
- val orderingSpec = toBuffer(writeSpec.writeOrderingSpec).headOption.orNull
-
- (distributionSpec, orderingSpec)
- }
-
- /**
- * Create an order field.
- */
- override def visitOrderField(ctx: OrderFieldContext): (Term, SortDirection, NullOrder) = {
- val term = Spark3Util.toIcebergTerm(typedVisit[Transform](ctx.transform))
- val direction = Option(ctx.ASC).map(_ => SortDirection.ASC)
- .orElse(Option(ctx.DESC).map(_ => SortDirection.DESC))
- .getOrElse(SortDirection.ASC)
- val nullOrder = Option(ctx.FIRST).map(_ => NullOrder.NULLS_FIRST)
- .orElse(Option(ctx.LAST).map(_ => NullOrder.NULLS_LAST))
- .getOrElse(if (direction == SortDirection.ASC) NullOrder.NULLS_FIRST else NullOrder.NULLS_LAST)
- (term, direction, nullOrder)
- }
-
- /**
- * Create an IdentityTransform for a column reference.
- */
- override def visitIdentityTransform(ctx: IdentityTransformContext): Transform = withOrigin(ctx) {
- IdentityTransform(FieldReference(typedVisit[Seq[String]](ctx.multipartIdentifier())))
- }
-
- /**
- * Create a named Transform from argument expressions.
- */
- override def visitApplyTransform(ctx: ApplyTransformContext): Transform = withOrigin(ctx) {
- val args = toSeq(ctx.arguments).map(typedVisit[expressions.Expression])
- ApplyTransform(ctx.transformName.getText, args)
- }
-
- /**
- * Create a transform argument from a column reference or a constant.
- */
- override def visitTransformArgument(ctx: TransformArgumentContext): expressions.Expression = withOrigin(ctx) {
- val reference = Option(ctx.multipartIdentifier())
- .map(typedVisit[Seq[String]])
- .map(FieldReference(_))
- val literal = Option(ctx.constant)
- .map(visitConstant)
- .map(lit => LiteralValue(lit.value, lit.dataType))
- reference.orElse(literal)
- .getOrElse(throw new IcebergParseException(s"Invalid transform argument", ctx))
- }
-
- /**
- * Return a multi-part identifier as Seq[String].
- */
- override def visitMultipartIdentifier(ctx: MultipartIdentifierContext): Seq[String] = withOrigin(ctx) {
- toSeq(ctx.parts).map(_.getText)
- }
-
- override def visitSingleOrder(ctx: SingleOrderContext): Seq[(Term, SortDirection, NullOrder)] = withOrigin(ctx) {
- toSeq(ctx.order.fields).map(typedVisit[(Term, SortDirection, NullOrder)])
- }
-
- /**
- * Create a positional argument in a stored procedure call.
- */
- override def visitPositionalArgument(ctx: PositionalArgumentContext): CallArgument = withOrigin(ctx) {
- val expr = typedVisit[Expression](ctx.expression)
- PositionalArgument(expr)
- }
-
- /**
- * Create a named argument in a stored procedure call.
- */
- override def visitNamedArgument(ctx: NamedArgumentContext): CallArgument = withOrigin(ctx) {
- val name = ctx.identifier.getText
- val expr = typedVisit[Expression](ctx.expression)
- NamedArgument(name, expr)
- }
-
- override def visitSingleStatement(ctx: SingleStatementContext): LogicalPlan = withOrigin(ctx) {
- visit(ctx.statement).asInstanceOf[LogicalPlan]
- }
-
- def visitConstant(ctx: ConstantContext): Literal = {
- delegate.parseExpression(ctx.getText).asInstanceOf[Literal]
- }
-
- override def visitExpression(ctx: ExpressionContext): Expression = {
- // reconstruct the SQL string and parse it using the main Spark parser
- // while we can avoid the logic to build Spark expressions, we still have to parse them
- // we cannot call ctx.getText directly since it will not render spaces correctly
- // that's why we need to recurse down the tree in reconstructSqlString
- val sqlString = reconstructSqlString(ctx)
- delegate.parseExpression(sqlString)
- }
-
- private def reconstructSqlString(ctx: ParserRuleContext): String = {
- toBuffer(ctx.children).map {
- case c: ParserRuleContext => reconstructSqlString(c)
- case t: TerminalNode => t.getText
- }.mkString(" ")
- }
-
- private def typedVisit[T](ctx: ParseTree): T = {
- ctx.accept(this).asInstanceOf[T]
- }
-}
-
-/* Partially copied from Apache Spark's Parser to avoid dependency on Spark Internals */
-object IcebergParserUtils {
-
- private[sql] def withOrigin[T](ctx: ParserRuleContext)(f: => T): T = {
- val current = CurrentOrigin.get
- CurrentOrigin.set(position(ctx.getStart))
- try {
- f
- } finally {
- CurrentOrigin.set(current)
- }
- }
-
- private[sql] def position(token: Token): Origin = {
- val opt = Option(token)
- Origin(opt.map(_.getLine), opt.map(_.getCharPositionInLine))
- }
-
- /** Get the command which created the token. */
- private[sql] def command(ctx: ParserRuleContext): String = {
- val stream = ctx.getStart.getInputStream
- stream.getText(Interval.of(0, stream.size() - 1))
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/planning/RewrittenRowLevelCommand.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/planning/RewrittenRowLevelCommand.scala
deleted file mode 100644
index 4cc7a7bf2f96..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/planning/RewrittenRowLevelCommand.scala
+++ /dev/null
@@ -1,113 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.planning
-
-import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
-import org.apache.spark.sql.catalyst.plans.logical.RowLevelCommand
-import org.apache.spark.sql.catalyst.plans.logical.UpdateIcebergTable
-import org.apache.spark.sql.catalyst.plans.logical.WriteDelta
-import org.apache.spark.sql.connector.catalog.Table
-import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
-import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
-
-/**
- * An extractor for operations such as DELETE and MERGE that require rewriting data.
- *
- * This class extracts the following entities:
- * - the row-level command (such as DeleteFromIcebergTable);
- * - the read relation in the rewrite plan that can be either DataSourceV2Relation or
- * DataSourceV2ScanRelation depending on whether the planning has already happened;
- * - the current rewrite plan.
- */
-object RewrittenRowLevelCommand {
- type ReturnType = (RowLevelCommand, LogicalPlan, LogicalPlan)
-
- def unapply(plan: LogicalPlan): Option[ReturnType] = plan match {
- case c: RowLevelCommand if c.rewritePlan.nonEmpty =>
- val rewritePlan = c.rewritePlan.get
-
- // both ReplaceData and WriteDelta reference a write relation
- // but the corresponding read relation should be at the bottom of the write plan
- // both the write and read relations will share the same RowLevelOperationTable object
- // that's why it is safe to use reference equality to find the needed read relation
-
- val allowScanDuplication = c match {
- // group-based updates that rely on the union approach may have multiple identical scans
- case _: UpdateIcebergTable if rewritePlan.isInstanceOf[ReplaceData] => true
- case _ => false
- }
-
- rewritePlan match {
- case rd @ ReplaceData(DataSourceV2Relation(table, _, _, _, _), query, _, _) =>
- val readRelation = findReadRelation(table, query, allowScanDuplication)
- readRelation.map((c, _, rd))
- case wd @ WriteDelta(DataSourceV2Relation(table, _, _, _, _), query, _, _, _) =>
- val readRelation = findReadRelation(table, query, allowScanDuplication)
- readRelation.map((c, _, wd))
- case _ =>
- None
- }
-
- case _ =>
- None
- }
-
- private def findReadRelation(
- table: Table,
- plan: LogicalPlan,
- allowScanDuplication: Boolean): Option[LogicalPlan] = {
-
- val readRelations = plan.collect {
- case r: DataSourceV2Relation if r.table eq table => r
- case r: DataSourceV2ScanRelation if r.relation.table eq table => r
- }
-
- // in some cases, the optimizer replaces the v2 read relation with a local relation
- // for example, there is no reason to query the table if the condition is always false
- // that's why it is valid not to find the corresponding v2 read relation
-
- readRelations match {
- case relations if relations.isEmpty =>
- None
-
- case Seq(relation) =>
- Some(relation)
-
- case Seq(relation1: DataSourceV2Relation, relation2: DataSourceV2Relation)
- if allowScanDuplication && (relation1.table eq relation2.table) =>
- Some(relation1)
-
- case Seq(relation1: DataSourceV2ScanRelation, relation2: DataSourceV2ScanRelation)
- if allowScanDuplication && (relation1.scan eq relation2.scan) =>
- Some(relation1)
-
- case Seq(relation1, relation2) if allowScanDuplication =>
- throw new AnalysisException(s"Row-level read relations don't match: $relation1, $relation2")
-
- case relations if allowScanDuplication =>
- throw new AnalysisException(s"Expected up to two row-level read relations: $relations")
-
- case relations =>
- throw new AnalysisException(s"Expected only one row-level read relation: $relations")
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/AddPartitionField.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/AddPartitionField.scala
deleted file mode 100644
index e8b1b2941161..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/AddPartitionField.scala
+++ /dev/null
@@ -1,33 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.connector.expressions.Transform
-
-case class AddPartitionField(table: Seq[String], transform: Transform, name: Option[String]) extends LeafCommand {
- import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
-
- override lazy val output: Seq[Attribute] = Nil
-
- override def simpleString(maxFields: Int): String = {
- s"AddPartitionField ${table.quoted} ${name.map(n => s"$n=").getOrElse("")}${transform.describe}"
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/BranchOptions.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/BranchOptions.scala
deleted file mode 100644
index 2041a983700a..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/BranchOptions.scala
+++ /dev/null
@@ -1,23 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-case class BranchOptions(snapshotId: Option[Long], numSnapshots: Option[Long],
- snapshotRetain: Option[Long], snapshotRefRetain: Option[Long])
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Call.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Call.scala
deleted file mode 100644
index 9616dae5a8d3..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Call.scala
+++ /dev/null
@@ -1,33 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.util.truncatedString
-import org.apache.spark.sql.connector.iceberg.catalog.Procedure
-
-case class Call(procedure: Procedure, args: Seq[Expression]) extends LeafCommand {
- override lazy val output: Seq[Attribute] = procedure.outputType.toAttributes
-
- override def simpleString(maxFields: Int): String = {
- s"Call${truncatedString(output.toSeq, "[", ", ", "]", maxFields)} ${procedure.description}"
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/CreateOrReplaceBranch.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/CreateOrReplaceBranch.scala
deleted file mode 100644
index b7981a3c7a0d..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/CreateOrReplaceBranch.scala
+++ /dev/null
@@ -1,39 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.catalyst.expressions.Attribute
-
-case class CreateOrReplaceBranch(
- table: Seq[String],
- branch: String,
- branchOptions: BranchOptions,
- create: Boolean,
- replace: Boolean,
- ifNotExists: Boolean) extends LeafCommand {
-
- import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
-
- override lazy val output: Seq[Attribute] = Nil
-
- override def simpleString(maxFields: Int): String = {
- s"CreateOrReplaceBranch branch: ${branch} for table: ${table.quoted}"
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/CreateOrReplaceTag.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/CreateOrReplaceTag.scala
deleted file mode 100644
index 6e7db84a90fb..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/CreateOrReplaceTag.scala
+++ /dev/null
@@ -1,39 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.catalyst.expressions.Attribute
-
-case class CreateOrReplaceTag(
- table: Seq[String],
- tag: String,
- tagOptions: TagOptions,
- create: Boolean,
- replace: Boolean,
- ifNotExists: Boolean) extends LeafCommand {
-
- import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
-
- override lazy val output: Seq[Attribute] = Nil
-
- override def simpleString(maxFields: Int): String = {
- s"CreateOrReplaceTag tag: ${tag} for table: ${table.quoted}"
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/DeleteFromIcebergTable.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/DeleteFromIcebergTable.scala
deleted file mode 100644
index d1268e416c50..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/DeleteFromIcebergTable.scala
+++ /dev/null
@@ -1,49 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.catalyst.expressions.Expression
-
-case class DeleteFromIcebergTable(
- table: LogicalPlan,
- condition: Option[Expression],
- rewritePlan: Option[LogicalPlan] = None) extends RowLevelCommand {
-
- override def children: Seq[LogicalPlan] = if (rewritePlan.isDefined) {
- table :: rewritePlan.get :: Nil
- } else {
- table :: Nil
- }
-
- override def withNewRewritePlan(newRewritePlan: LogicalPlan): RowLevelCommand = {
- copy(rewritePlan = Some(newRewritePlan))
- }
-
- override protected def withNewChildrenInternal(
- newChildren: IndexedSeq[LogicalPlan]): DeleteFromIcebergTable = {
- if (newChildren.size == 1) {
- copy(table = newChildren.head, rewritePlan = None)
- } else {
- require(newChildren.size == 2, "DeleteFromIcebergTable expects either one or two children")
- val Seq(newTable, newRewritePlan) = newChildren.take(2)
- copy(table = newTable, rewritePlan = Some(newRewritePlan))
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/DropBranch.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/DropBranch.scala
deleted file mode 100644
index bee0b0fae688..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/DropBranch.scala
+++ /dev/null
@@ -1,33 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.catalyst.expressions.Attribute
-
-case class DropBranch(table: Seq[String], branch: String, ifExists: Boolean) extends LeafCommand {
-
- import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
-
- override lazy val output: Seq[Attribute] = Nil
-
- override def simpleString(maxFields: Int): String = {
- s"DropBranch branch: ${branch} for table: ${table.quoted}"
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/DropIdentifierFields.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/DropIdentifierFields.scala
deleted file mode 100644
index 29dd686a0fba..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/DropIdentifierFields.scala
+++ /dev/null
@@ -1,34 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.catalyst.expressions.Attribute
-
-case class DropIdentifierFields(
- table: Seq[String],
- fields: Seq[String]) extends LeafCommand {
- import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
-
- override lazy val output: Seq[Attribute] = Nil
-
- override def simpleString(maxFields: Int): String = {
- s"DropIdentifierFields ${table.quoted} (${fields.quoted})"
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/DropPartitionField.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/DropPartitionField.scala
deleted file mode 100644
index fb1451324182..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/DropPartitionField.scala
+++ /dev/null
@@ -1,33 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.connector.expressions.Transform
-
-case class DropPartitionField(table: Seq[String], transform: Transform) extends LeafCommand {
- import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
-
- override lazy val output: Seq[Attribute] = Nil
-
- override def simpleString(maxFields: Int): String = {
- s"DropPartitionField ${table.quoted} ${transform.describe}"
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/DropTag.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/DropTag.scala
deleted file mode 100644
index 7e4b38e74d2f..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/DropTag.scala
+++ /dev/null
@@ -1,33 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.catalyst.expressions.Attribute
-
-case class DropTag(table: Seq[String], tag: String, ifExists: Boolean) extends LeafCommand {
-
- import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
-
- override lazy val output: Seq[Attribute] = Nil
-
- override def simpleString(maxFields: Int): String = {
- s"DropTag tag: ${tag} for table: ${table.quoted}"
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeIntoIcebergTable.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeIntoIcebergTable.scala
deleted file mode 100644
index 8f84851dcda2..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeIntoIcebergTable.scala
+++ /dev/null
@@ -1,77 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.catalyst.expressions.AssignmentUtils
-import org.apache.spark.sql.catalyst.expressions.Expression
-
-case class MergeIntoIcebergTable(
- targetTable: LogicalPlan,
- sourceTable: LogicalPlan,
- mergeCondition: Expression,
- matchedActions: Seq[MergeAction],
- notMatchedActions: Seq[MergeAction],
- rewritePlan: Option[LogicalPlan] = None) extends RowLevelCommand {
-
- lazy val aligned: Boolean = {
- val matchedActionsAligned = matchedActions.forall {
- case UpdateAction(_, assignments) =>
- AssignmentUtils.aligned(targetTable, assignments)
- case _: DeleteAction =>
- true
- case _ =>
- false
- }
-
- val notMatchedActionsAligned = notMatchedActions.forall {
- case InsertAction(_, assignments) =>
- AssignmentUtils.aligned(targetTable, assignments)
- case _ =>
- false
- }
-
- matchedActionsAligned && notMatchedActionsAligned
- }
-
- def condition: Option[Expression] = Some(mergeCondition)
-
- override def children: Seq[LogicalPlan] = if (rewritePlan.isDefined) {
- targetTable :: sourceTable :: rewritePlan.get :: Nil
- } else {
- targetTable :: sourceTable :: Nil
- }
-
- override def withNewRewritePlan(newRewritePlan: LogicalPlan): RowLevelCommand = {
- copy(rewritePlan = Some(newRewritePlan))
- }
-
- override protected def withNewChildrenInternal(
- newChildren: IndexedSeq[LogicalPlan]): MergeIntoIcebergTable = {
-
- newChildren match {
- case Seq(newTarget, newSource) =>
- copy(targetTable = newTarget, sourceTable = newSource, rewritePlan = None)
- case Seq(newTarget, newSource, newRewritePlan) =>
- copy(targetTable = newTarget, sourceTable = newSource, rewritePlan = Some(newRewritePlan))
- case _ =>
- throw new IllegalArgumentException("MergeIntoIcebergTable expects either two or three children")
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeRows.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeRows.scala
deleted file mode 100644
index 3607194fe8c8..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeRows.scala
+++ /dev/null
@@ -1,56 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.catalyst.expressions.AttributeSet
-import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.util.truncatedString
-
-case class MergeRows(
- isSourceRowPresent: Expression,
- isTargetRowPresent: Expression,
- matchedConditions: Seq[Expression],
- matchedOutputs: Seq[Seq[Expression]],
- notMatchedConditions: Seq[Expression],
- notMatchedOutputs: Seq[Seq[Expression]],
- targetOutput: Seq[Expression],
- rowIdAttrs: Seq[Attribute],
- performCardinalityCheck: Boolean,
- emitNotMatchedTargetRows: Boolean,
- output: Seq[Attribute],
- child: LogicalPlan) extends UnaryNode {
-
- require(targetOutput.nonEmpty || !emitNotMatchedTargetRows)
-
- override lazy val producedAttributes: AttributeSet = {
- AttributeSet(output.filterNot(attr => inputSet.contains(attr)))
- }
-
- override lazy val references: AttributeSet = child.outputSet
-
- override def simpleString(maxFields: Int): String = {
- s"MergeRows${truncatedString(output, "[", ", ", "]", maxFields)}"
- }
-
- override protected def withNewChildInternal(newChild: LogicalPlan): LogicalPlan = {
- copy(child = newChild)
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/NoStatsUnaryNode.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/NoStatsUnaryNode.scala
deleted file mode 100644
index c21df71f069d..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/NoStatsUnaryNode.scala
+++ /dev/null
@@ -1,31 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.catalyst.expressions.Attribute
-
-case class NoStatsUnaryNode(child: LogicalPlan) extends UnaryNode {
- override def output: Seq[Attribute] = child.output
- override def stats: Statistics = Statistics(Long.MaxValue)
-
- override protected def withNewChildInternal(newChild: LogicalPlan): LogicalPlan = {
- copy(child = newChild)
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/ReplaceData.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/ReplaceData.scala
deleted file mode 100644
index 3bf726ffb719..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/ReplaceData.scala
+++ /dev/null
@@ -1,69 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.catalyst.analysis.NamedRelation
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.catalyst.expressions.AttributeSet
-import org.apache.spark.sql.catalyst.util.CharVarcharUtils
-import org.apache.spark.sql.connector.write.Write
-import org.apache.spark.sql.types.DataType
-
-/**
- * Replace data in an existing table.
- */
-case class ReplaceData(
- table: NamedRelation,
- query: LogicalPlan,
- originalTable: NamedRelation,
- write: Option[Write] = None) extends V2WriteCommandLike {
-
- override lazy val references: AttributeSet = query.outputSet
- override lazy val stringArgs: Iterator[Any] = Iterator(table, query, write)
-
- // the incoming query may include metadata columns
- lazy val dataInput: Seq[Attribute] = {
- val tableAttrNames = table.output.map(_.name)
- query.output.filter(attr => tableAttrNames.exists(conf.resolver(_, attr.name)))
- }
-
- override def outputResolved: Boolean = {
- assert(table.resolved && query.resolved,
- "`outputResolved` can only be called when `table` and `query` are both resolved.")
-
- // take into account only incoming data columns and ignore metadata columns in the query
- // they will be discarded after the logical write is built in the optimizer
- // metadata columns may be needed to request a correct distribution or ordering
- // but are not passed back to the data source during writes
-
- table.skipSchemaResolution || (dataInput.size == table.output.size &&
- dataInput.zip(table.output).forall { case (inAttr, outAttr) =>
- val outType = CharVarcharUtils.getRawType(outAttr.metadata).getOrElse(outAttr.dataType)
- // names and types must match, nullability must be compatible
- inAttr.name == outAttr.name &&
- DataType.equalsIgnoreCompatibleNullability(inAttr.dataType, outType) &&
- (outAttr.nullable || !inAttr.nullable)
- })
- }
-
- override protected def withNewChildInternal(newChild: LogicalPlan): ReplaceData = {
- copy(query = newChild)
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/ReplacePartitionField.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/ReplacePartitionField.scala
deleted file mode 100644
index 8c660c6f37b1..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/ReplacePartitionField.scala
+++ /dev/null
@@ -1,38 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.connector.expressions.Transform
-
-case class ReplacePartitionField(
- table: Seq[String],
- transformFrom: Transform,
- transformTo: Transform,
- name: Option[String]) extends LeafCommand {
- import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
-
- override lazy val output: Seq[Attribute] = Nil
-
- override def simpleString(maxFields: Int): String = {
- s"ReplacePartitionField ${table.quoted} ${transformFrom.describe} " +
- s"with ${name.map(n => s"$n=").getOrElse("")}${transformTo.describe}"
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/RowLevelCommand.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/RowLevelCommand.scala
deleted file mode 100644
index 837ee963bcea..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/RowLevelCommand.scala
+++ /dev/null
@@ -1,28 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.catalyst.expressions.Expression
-
-trait RowLevelCommand extends Command with SupportsSubquery {
- def condition: Option[Expression]
- def rewritePlan: Option[LogicalPlan]
- def withNewRewritePlan(newRewritePlan: LogicalPlan): RowLevelCommand
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/SetIdentifierFields.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/SetIdentifierFields.scala
deleted file mode 100644
index a5fa28a617e7..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/SetIdentifierFields.scala
+++ /dev/null
@@ -1,35 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.connector.expressions.Transform
-
-case class SetIdentifierFields(
- table: Seq[String],
- fields: Seq[String]) extends LeafCommand {
- import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
-
- override lazy val output: Seq[Attribute] = Nil
-
- override def simpleString(maxFields: Int): String = {
- s"SetIdentifierFields ${table.quoted} (${fields.quoted})"
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/TagOptions.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/TagOptions.scala
deleted file mode 100644
index 85e3b95f4aba..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/TagOptions.scala
+++ /dev/null
@@ -1,22 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-case class TagOptions(snapshotId: Option[Long], snapshotRefRetain: Option[Long])
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/UnresolvedMergeIntoIcebergTable.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/UnresolvedMergeIntoIcebergTable.scala
deleted file mode 100644
index 895aa733ff20..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/UnresolvedMergeIntoIcebergTable.scala
+++ /dev/null
@@ -1,45 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.catalyst.expressions.Expression
-
-/**
- * A node that hides the MERGE condition and actions from regular Spark resolution.
- */
-case class UnresolvedMergeIntoIcebergTable(
- targetTable: LogicalPlan,
- sourceTable: LogicalPlan,
- context: MergeIntoContext) extends BinaryCommand {
-
- def duplicateResolved: Boolean = targetTable.outputSet.intersect(sourceTable.outputSet).isEmpty
-
- override def left: LogicalPlan = targetTable
- override def right: LogicalPlan = sourceTable
-
- override protected def withNewChildrenInternal(newLeft: LogicalPlan, newRight: LogicalPlan): LogicalPlan = {
- copy(targetTable = newLeft, sourceTable = newRight)
- }
-}
-
-case class MergeIntoContext(
- mergeCondition: Expression,
- matchedActions: Seq[MergeAction],
- notMatchedActions: Seq[MergeAction])
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/UpdateIcebergTable.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/UpdateIcebergTable.scala
deleted file mode 100644
index 790eb9380e3d..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/UpdateIcebergTable.scala
+++ /dev/null
@@ -1,53 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.catalyst.expressions.AssignmentUtils
-import org.apache.spark.sql.catalyst.expressions.Expression
-
-case class UpdateIcebergTable(
- table: LogicalPlan,
- assignments: Seq[Assignment],
- condition: Option[Expression],
- rewritePlan: Option[LogicalPlan] = None) extends RowLevelCommand {
-
- lazy val aligned: Boolean = AssignmentUtils.aligned(table, assignments)
-
- override def children: Seq[LogicalPlan] = if (rewritePlan.isDefined) {
- table :: rewritePlan.get :: Nil
- } else {
- table :: Nil
- }
-
- override def withNewRewritePlan(newRewritePlan: LogicalPlan): RowLevelCommand = {
- copy(rewritePlan = Some(newRewritePlan))
- }
-
- override protected def withNewChildrenInternal(
- newChildren: IndexedSeq[LogicalPlan]): UpdateIcebergTable = {
- if (newChildren.size == 1) {
- copy(table = newChildren.head, rewritePlan = None)
- } else {
- require(newChildren.size == 2, "UpdateTable expects either one or two children")
- val Seq(newTable, newRewritePlan) = newChildren.take(2)
- copy(table = newTable, rewritePlan = Some(newRewritePlan))
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/V2WriteCommandLike.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/V2WriteCommandLike.scala
deleted file mode 100644
index 9192d74b7caf..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/V2WriteCommandLike.scala
+++ /dev/null
@@ -1,42 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.catalyst.analysis.NamedRelation
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.catalyst.expressions.AttributeSet
-
-// a node similar to V2WriteCommand in Spark but does not extend Command
-// as ReplaceData and WriteDelta that extend this trait are nested within other commands
-trait V2WriteCommandLike extends UnaryNode {
- def table: NamedRelation
- def query: LogicalPlan
- def outputResolved: Boolean
-
- override lazy val resolved: Boolean = table.resolved && query.resolved && outputResolved
-
- override def child: LogicalPlan = query
- override def output: Seq[Attribute] = Seq.empty
- override def producedAttributes: AttributeSet = outputSet
- // Commands are eagerly executed. They will be converted to LocalRelation after the DataFrame
- // is created. That said, the statistics of a command is useless. Here we just return a dummy
- // statistics to avoid unnecessary statistics calculation of command's children.
- override def stats: Statistics = Statistics.DUMMY
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/WriteDelta.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/WriteDelta.scala
deleted file mode 100644
index 534427cc0410..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/WriteDelta.scala
+++ /dev/null
@@ -1,126 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
-import org.apache.spark.sql.catalyst.analysis.NamedRelation
-import org.apache.spark.sql.catalyst.expressions.AttributeReference
-import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
-import org.apache.spark.sql.catalyst.expressions.NamedExpression
-import org.apache.spark.sql.catalyst.util.CharVarcharUtils
-import org.apache.spark.sql.catalyst.util.RowDeltaUtils.OPERATION_COLUMN
-import org.apache.spark.sql.catalyst.util.WriteDeltaProjections
-import org.apache.spark.sql.connector.iceberg.write.DeltaWrite
-import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
-import org.apache.spark.sql.connector.write.RowLevelOperationTable
-import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
-import org.apache.spark.sql.types.DataType
-import org.apache.spark.sql.types.IntegerType
-import org.apache.spark.sql.types.StructField
-
-/**
- * Writes a delta of rows to an existing table.
- */
-case class WriteDelta(
- table: NamedRelation,
- query: LogicalPlan,
- originalTable: NamedRelation,
- projections: WriteDeltaProjections,
- write: Option[DeltaWrite] = None) extends V2WriteCommandLike {
-
- override protected lazy val stringArgs: Iterator[Any] = Iterator(table, query, write)
-
- private def operationResolved: Boolean = {
- val attr = query.output.head
- attr.name == OPERATION_COLUMN && attr.dataType == IntegerType && !attr.nullable
- }
-
- private def operation: SupportsDelta = {
- EliminateSubqueryAliases(table) match {
- case DataSourceV2Relation(RowLevelOperationTable(_, operation), _, _, _, _) =>
- operation match {
- case supportsDelta: SupportsDelta =>
- supportsDelta
- case _ =>
- throw new AnalysisException(s"Operation $operation is not a delta operation")
- }
- case _ =>
- throw new AnalysisException(s"Cannot retrieve row-level operation from $table")
- }
- }
-
- private def rowAttrsResolved: Boolean = {
- table.skipSchemaResolution || (projections.rowProjection match {
- case Some(projection) =>
- table.output.size == projection.schema.size &&
- projection.schema.zip(table.output).forall { case (field, outAttr) =>
- isCompatible(field, outAttr)
- }
- case None =>
- true
- })
- }
-
- private def rowIdAttrsResolved: Boolean = {
- val rowIdAttrs = ExtendedV2ExpressionUtils.resolveRefs[AttributeReference](
- operation.rowId.toSeq,
- originalTable)
-
- projections.rowIdProjection.schema.forall { field =>
- rowIdAttrs.exists(rowIdAttr => isCompatible(field, rowIdAttr))
- }
- }
-
- private def metadataAttrsResolved: Boolean = {
- projections.metadataProjection match {
- case Some(projection) =>
- val metadataAttrs = ExtendedV2ExpressionUtils.resolveRefs[AttributeReference](
- operation.requiredMetadataAttributes.toSeq,
- originalTable)
-
- projection.schema.forall { field =>
- metadataAttrs.exists(metadataAttr => isCompatible(field, metadataAttr))
- }
- case None =>
- true
- }
- }
-
- private def isCompatible(projectionField: StructField, outAttr: NamedExpression): Boolean = {
- val inType = CharVarcharUtils.getRawType(projectionField.metadata).getOrElse(outAttr.dataType)
- val outType = CharVarcharUtils.getRawType(outAttr.metadata).getOrElse(outAttr.dataType)
- // names and types must match, nullability must be compatible
- projectionField.name == outAttr.name &&
- DataType.equalsIgnoreCompatibleNullability(inType, outType) &&
- (outAttr.nullable || !projectionField.nullable)
- }
-
- override def outputResolved: Boolean = {
- assert(table.resolved && query.resolved,
- "`outputResolved` can only be called when `table` and `query` are both resolved.")
-
- operationResolved && rowAttrsResolved && rowIdAttrsResolved && metadataAttrsResolved
- }
-
- override protected def withNewChildInternal(newChild: LogicalPlan): WriteDelta = {
- copy(query = newChild)
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statements.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statements.scala
deleted file mode 100644
index be15f32bc1b8..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statements.scala
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.plans.logical
-
-import org.apache.spark.sql.catalyst.expressions.Expression
-
-/**
- * A CALL statement, as parsed from SQL.
- */
-case class CallStatement(name: Seq[String], args: Seq[CallArgument]) extends LeafParsedStatement
-
-/**
- * An argument in a CALL statement.
- */
-sealed trait CallArgument {
- def expr: Expression
-}
-
-/**
- * An argument in a CALL statement identified by name.
- */
-case class NamedArgument(name: String, expr: Expression) extends CallArgument
-
-/**
- * An argument in a CALL statement identified by position.
- */
-case class PositionalArgument(expr: Expression) extends CallArgument
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/util/RowDeltaUtils.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/util/RowDeltaUtils.scala
deleted file mode 100644
index 9f8f07b77db2..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/util/RowDeltaUtils.scala
+++ /dev/null
@@ -1,27 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.util
-
-object RowDeltaUtils {
- final val OPERATION_COLUMN: String = "__row_operation"
- final val DELETE_OPERATION: Int = 1
- final val UPDATE_OPERATION: Int = 2
- final val INSERT_OPERATION: Int = 3
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/util/WriteDeltaProjections.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/util/WriteDeltaProjections.scala
deleted file mode 100644
index e4e8b147d139..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/util/WriteDeltaProjections.scala
+++ /dev/null
@@ -1,27 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.catalyst.util
-
-import org.apache.spark.sql.catalyst.ProjectingInternalRow
-
-case class WriteDeltaProjections(
- rowProjection: Option[ProjectingInternalRow],
- rowIdProjection: ProjectingInternalRow,
- metadataProjection: Option[ProjectingInternalRow])
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/connector/expressions/TruncateTransform.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/connector/expressions/TruncateTransform.scala
deleted file mode 100644
index 2a3269e2db1d..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/connector/expressions/TruncateTransform.scala
+++ /dev/null
@@ -1,38 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.connector.expressions
-
-import org.apache.spark.sql.types.IntegerType
-
-private[sql] object TruncateTransform {
- def unapply(expr: Expression): Option[(Int, FieldReference)] = expr match {
- case transform: Transform =>
- transform match {
- case NamedTransform("truncate", Seq(Ref(seq: Seq[String]), Lit(value: Int, IntegerType))) =>
- Some((value, FieldReference(seq)))
- case NamedTransform("truncate", Seq(Lit(value: Int, IntegerType), Ref(seq: Seq[String]))) =>
- Some((value, FieldReference(seq)))
- case _ =>
- None
- }
- case _ =>
- None
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/connector/write/ExtendedLogicalWriteInfoImpl.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/connector/write/ExtendedLogicalWriteInfoImpl.scala
deleted file mode 100644
index 0ae10ada5a39..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/connector/write/ExtendedLogicalWriteInfoImpl.scala
+++ /dev/null
@@ -1,31 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.connector.write
-
-import org.apache.spark.sql.connector.iceberg.write.ExtendedLogicalWriteInfo
-import org.apache.spark.sql.types.StructType
-import org.apache.spark.sql.util.CaseInsensitiveStringMap
-
-private[sql] case class ExtendedLogicalWriteInfoImpl(
- queryId: String,
- schema: StructType,
- options: CaseInsensitiveStringMap,
- rowIdSchema: StructType = null,
- metadataSchema: StructType = null) extends ExtendedLogicalWriteInfo
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/connector/write/RowLevelOperationInfoImpl.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/connector/write/RowLevelOperationInfoImpl.scala
deleted file mode 100644
index fca42feaed51..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/connector/write/RowLevelOperationInfoImpl.scala
+++ /dev/null
@@ -1,28 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.connector.write
-
-import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command
-import org.apache.spark.sql.connector.iceberg.write.RowLevelOperationInfo
-import org.apache.spark.sql.util.CaseInsensitiveStringMap
-
-case class RowLevelOperationInfoImpl(
- command: Command,
- options: CaseInsensitiveStringMap) extends RowLevelOperationInfo
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/connector/write/RowLevelOperationTable.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/connector/write/RowLevelOperationTable.scala
deleted file mode 100644
index b6cca0af15bb..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/connector/write/RowLevelOperationTable.scala
+++ /dev/null
@@ -1,54 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.connector.write
-
-import java.util
-import org.apache.spark.sql.connector.catalog.SupportsRead
-import org.apache.spark.sql.connector.catalog.SupportsWrite
-import org.apache.spark.sql.connector.catalog.Table
-import org.apache.spark.sql.connector.catalog.TableCapability
-import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
-import org.apache.spark.sql.connector.iceberg.write.ExtendedLogicalWriteInfo
-import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation
-import org.apache.spark.sql.connector.read.ScanBuilder
-import org.apache.spark.sql.types.StructType
-import org.apache.spark.sql.util.CaseInsensitiveStringMap
-
-/**
- * An internal v2 table implementation that wraps the original table during DELETE, UPDATE,
- * MERGE operations.
- */
-case class RowLevelOperationTable(
- table: Table with SupportsRowLevelOperations,
- operation: RowLevelOperation) extends Table with SupportsRead with SupportsWrite {
-
- override def name: String = table.name
- override def schema: StructType = table.schema
- override def capabilities: util.Set[TableCapability] = table.capabilities
- override def toString: String = table.toString
-
- override def newScanBuilder(options: CaseInsensitiveStringMap): ScanBuilder = {
- operation.newScanBuilder(options)
- }
-
- override def newWriteBuilder(info: LogicalWriteInfo): WriteBuilder = {
- operation.newWriteBuilder(info.asInstanceOf[ExtendedLogicalWriteInfo])
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/AddPartitionFieldExec.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/AddPartitionFieldExec.scala
deleted file mode 100644
index 55f327f7e45e..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/AddPartitionFieldExec.scala
+++ /dev/null
@@ -1,56 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import org.apache.iceberg.spark.Spark3Util
-import org.apache.iceberg.spark.source.SparkTable
-import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.connector.catalog.Identifier
-import org.apache.spark.sql.connector.catalog.TableCatalog
-import org.apache.spark.sql.connector.expressions.Transform
-
-case class AddPartitionFieldExec(
- catalog: TableCatalog,
- ident: Identifier,
- transform: Transform,
- name: Option[String]) extends LeafV2CommandExec {
- import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
-
- override lazy val output: Seq[Attribute] = Nil
-
- override protected def run(): Seq[InternalRow] = {
- catalog.loadTable(ident) match {
- case iceberg: SparkTable =>
- iceberg.table.updateSpec()
- .addField(name.orNull, Spark3Util.toIcebergTerm(transform))
- .commit()
-
- case table =>
- throw new UnsupportedOperationException(s"Cannot add partition field to non-Iceberg table: $table")
- }
-
- Nil
- }
-
- override def simpleString(maxFields: Int): String = {
- s"AddPartitionField ${catalog.name}.${ident.quoted} ${name.map(n => s"$n=").getOrElse("")}${transform.describe}"
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CallExec.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CallExec.scala
deleted file mode 100644
index f66962a8c453..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CallExec.scala
+++ /dev/null
@@ -1,40 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.catalyst.util.truncatedString
-import org.apache.spark.sql.connector.iceberg.catalog.Procedure
-import scala.collection.compat.immutable.ArraySeq
-
-case class CallExec(
- output: Seq[Attribute],
- procedure: Procedure,
- input: InternalRow) extends LeafV2CommandExec {
-
- override protected def run(): Seq[InternalRow] = {
- ArraySeq.unsafeWrapArray(procedure.call(input))
- }
-
- override def simpleString(maxFields: Int): String = {
- s"CallExec${truncatedString(output, "[", ", ", "]", maxFields)} ${procedure.description}"
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateOrReplaceBranchExec.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateOrReplaceBranchExec.scala
deleted file mode 100644
index ecf1489e0854..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateOrReplaceBranchExec.scala
+++ /dev/null
@@ -1,100 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import org.apache.iceberg.relocated.com.google.common.base.Preconditions
-import org.apache.iceberg.spark.source.SparkTable
-import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.catalyst.plans.logical.BranchOptions
-import org.apache.spark.sql.connector.catalog.Identifier
-import org.apache.spark.sql.connector.catalog.TableCatalog
-
-case class CreateOrReplaceBranchExec(
- catalog: TableCatalog,
- ident: Identifier,
- branch: String,
- branchOptions: BranchOptions,
- create: Boolean,
- replace: Boolean,
- ifNotExists: Boolean) extends LeafV2CommandExec {
-
- import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
-
- override lazy val output: Seq[Attribute] = Nil
-
- override protected def run(): Seq[InternalRow] = {
- catalog.loadTable(ident) match {
- case iceberg: SparkTable =>
- val snapshotId: java.lang.Long = branchOptions.snapshotId
- .orElse(Option(iceberg.table.currentSnapshot()).map(_.snapshotId()))
- .map(java.lang.Long.valueOf)
- .orNull
-
- val manageSnapshots = iceberg.table().manageSnapshots()
- val refExists = null != iceberg.table().refs().get(branch)
-
- def safeCreateBranch(): Unit = {
- if (snapshotId == null) {
- manageSnapshots.createBranch(branch)
- } else {
- manageSnapshots.createBranch(branch, snapshotId)
- }
- }
-
- if (create && replace && !refExists) {
- safeCreateBranch()
- } else if (replace) {
- Preconditions.checkArgument(snapshotId != null,
- "Cannot complete replace branch operation on %s, main has no snapshot", ident)
- manageSnapshots.replaceBranch(branch, snapshotId)
- } else {
- if (refExists && ifNotExists) {
- return Nil
- }
-
- safeCreateBranch()
- }
-
- if (branchOptions.numSnapshots.nonEmpty) {
- manageSnapshots.setMinSnapshotsToKeep(branch, branchOptions.numSnapshots.get.toInt)
- }
-
- if (branchOptions.snapshotRetain.nonEmpty) {
- manageSnapshots.setMaxSnapshotAgeMs(branch, branchOptions.snapshotRetain.get)
- }
-
- if (branchOptions.snapshotRefRetain.nonEmpty) {
- manageSnapshots.setMaxRefAgeMs(branch, branchOptions.snapshotRefRetain.get)
- }
-
- manageSnapshots.commit()
-
- case table =>
- throw new UnsupportedOperationException(s"Cannot create or replace branch on non-Iceberg table: $table")
- }
-
- Nil
- }
-
- override def simpleString(maxFields: Int): String = {
- s"CreateOrReplace branch: $branch for table: ${ident.quoted}"
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateOrReplaceTagExec.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateOrReplaceTagExec.scala
deleted file mode 100644
index 372cd7548632..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateOrReplaceTagExec.scala
+++ /dev/null
@@ -1,84 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import org.apache.iceberg.relocated.com.google.common.base.Preconditions
-import org.apache.iceberg.spark.source.SparkTable
-import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.catalyst.plans.logical.TagOptions
-import org.apache.spark.sql.connector.catalog._
-
-case class CreateOrReplaceTagExec(
- catalog: TableCatalog,
- ident: Identifier,
- tag: String,
- tagOptions: TagOptions,
- create: Boolean,
- replace: Boolean,
- ifNotExists: Boolean) extends LeafV2CommandExec {
-
- import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
-
- override lazy val output: Seq[Attribute] = Nil
-
- override protected def run(): Seq[InternalRow] = {
- catalog.loadTable(ident) match {
- case iceberg: SparkTable =>
- val snapshotId: java.lang.Long = tagOptions.snapshotId
- .orElse(Option(iceberg.table.currentSnapshot()).map(_.snapshotId()))
- .map(java.lang.Long.valueOf)
- .orNull
-
- Preconditions.checkArgument(snapshotId != null,
- "Cannot complete create or replace tag operation on %s, main has no snapshot", ident)
-
- val manageSnapshot = iceberg.table.manageSnapshots()
- val refExists = null != iceberg.table().refs().get(tag)
-
- if (create && replace && !refExists) {
- manageSnapshot.createTag(tag, snapshotId)
- } else if (replace) {
- manageSnapshot.replaceTag(tag, snapshotId)
- } else {
- if (refExists && ifNotExists) {
- return Nil
- }
-
- manageSnapshot.createTag(tag, snapshotId)
- }
-
- if (tagOptions.snapshotRefRetain.nonEmpty) {
- manageSnapshot.setMaxRefAgeMs(tag, tagOptions.snapshotRefRetain.get)
- }
-
- manageSnapshot.commit()
-
- case table =>
- throw new UnsupportedOperationException(s"Cannot create tag to non-Iceberg table: $table")
- }
-
- Nil
- }
-
- override def simpleString(maxFields: Int): String = {
- s"Create tag: $tag for table: ${ident.quoted}"
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropBranchExec.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropBranchExec.scala
deleted file mode 100644
index ff8f1820099a..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropBranchExec.scala
+++ /dev/null
@@ -1,56 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import org.apache.iceberg.spark.source.SparkTable
-import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.connector.catalog.Identifier
-import org.apache.spark.sql.connector.catalog.TableCatalog
-
-case class DropBranchExec(
- catalog: TableCatalog,
- ident: Identifier,
- branch: String,
- ifExists: Boolean) extends LeafV2CommandExec {
-
- import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
-
- override lazy val output: Seq[Attribute] = Nil
-
- override protected def run(): Seq[InternalRow] = {
- catalog.loadTable(ident) match {
- case iceberg: SparkTable =>
- val ref = iceberg.table().refs().get(branch)
- if (ref != null || !ifExists) {
- iceberg.table().manageSnapshots().removeBranch(branch).commit()
- }
-
- case table =>
- throw new UnsupportedOperationException(s"Cannot drop branch on non-Iceberg table: $table")
- }
-
- Nil
- }
-
- override def simpleString(maxFields: Int): String = {
- s"DropBranch branch: ${branch} for table: ${ident.quoted}"
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropIdentifierFieldsExec.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropIdentifierFieldsExec.scala
deleted file mode 100644
index dee778b474f9..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropIdentifierFieldsExec.scala
+++ /dev/null
@@ -1,65 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import org.apache.iceberg.relocated.com.google.common.base.Preconditions
-import org.apache.iceberg.relocated.com.google.common.collect.Sets
-import org.apache.iceberg.spark.source.SparkTable
-import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.connector.catalog.Identifier
-import org.apache.spark.sql.connector.catalog.TableCatalog
-
-case class DropIdentifierFieldsExec(
- catalog: TableCatalog,
- ident: Identifier,
- fields: Seq[String]) extends LeafV2CommandExec {
- import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
-
- override lazy val output: Seq[Attribute] = Nil
-
- override protected def run(): Seq[InternalRow] = {
- catalog.loadTable(ident) match {
- case iceberg: SparkTable =>
- val schema = iceberg.table.schema
- val identifierFieldNames = Sets.newHashSet(schema.identifierFieldNames)
-
- for (name <- fields) {
- Preconditions.checkArgument(schema.findField(name) != null,
- "Cannot complete drop identifier fields operation: field %s not found", name)
- Preconditions.checkArgument(identifierFieldNames.contains(name),
- "Cannot complete drop identifier fields operation: %s is not an identifier field", name)
- identifierFieldNames.remove(name)
- }
-
- iceberg.table.updateSchema()
- .setIdentifierFields(identifierFieldNames)
- .commit();
- case table =>
- throw new UnsupportedOperationException(s"Cannot drop identifier fields in non-Iceberg table: $table")
- }
-
- Nil
- }
-
- override def simpleString(maxFields: Int): String = {
- s"DropIdentifierFields ${catalog.name}.${ident.quoted} (${fields.quoted})";
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropPartitionFieldExec.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropPartitionFieldExec.scala
deleted file mode 100644
index 9a153f0c004e..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropPartitionFieldExec.scala
+++ /dev/null
@@ -1,67 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import org.apache.iceberg.spark.Spark3Util
-import org.apache.iceberg.spark.source.SparkTable
-import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.connector.catalog.Identifier
-import org.apache.spark.sql.connector.catalog.TableCatalog
-import org.apache.spark.sql.connector.expressions.FieldReference
-import org.apache.spark.sql.connector.expressions.IdentityTransform
-import org.apache.spark.sql.connector.expressions.Transform
-
-case class DropPartitionFieldExec(
- catalog: TableCatalog,
- ident: Identifier,
- transform: Transform) extends LeafV2CommandExec {
- import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
-
- override lazy val output: Seq[Attribute] = Nil
-
- override protected def run(): Seq[InternalRow] = {
- catalog.loadTable(ident) match {
- case iceberg: SparkTable =>
- val schema = iceberg.table.schema
- transform match {
- case IdentityTransform(FieldReference(parts)) if parts.size == 1 && schema.findField(parts.head) == null =>
- // the name is not present in the Iceberg schema, so it must be a partition field name, not a column name
- iceberg.table.updateSpec()
- .removeField(parts.head)
- .commit()
-
- case _ =>
- iceberg.table.updateSpec()
- .removeField(Spark3Util.toIcebergTerm(transform))
- .commit()
- }
-
- case table =>
- throw new UnsupportedOperationException(s"Cannot drop partition field in non-Iceberg table: $table")
- }
-
- Nil
- }
-
- override def simpleString(maxFields: Int): String = {
- s"DropPartitionField ${catalog.name}.${ident.quoted} ${transform.describe}"
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropTagExec.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropTagExec.scala
deleted file mode 100644
index 0a1c17c0b1b2..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropTagExec.scala
+++ /dev/null
@@ -1,56 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import org.apache.iceberg.spark.source.SparkTable
-import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.connector.catalog.Identifier
-import org.apache.spark.sql.connector.catalog.TableCatalog
-
-case class DropTagExec(
- catalog: TableCatalog,
- ident: Identifier,
- tag: String,
- ifExists: Boolean) extends LeafV2CommandExec {
-
- import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
-
- override lazy val output: Seq[Attribute] = Nil
-
- override protected def run(): Seq[InternalRow] = {
- catalog.loadTable(ident) match {
- case iceberg: SparkTable =>
- val ref = iceberg.table().refs().get(tag)
- if (ref != null || !ifExists) {
- iceberg.table().manageSnapshots().removeTag(tag).commit()
- }
-
- case table =>
- throw new UnsupportedOperationException(s"Cannot drop tag on non-Iceberg table: $table")
- }
-
- Nil
- }
-
- override def simpleString(maxFields: Int): String = {
- s"DropTag tag: ${tag} for table: ${ident.quoted}"
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ExtendedDataSourceV2Implicits.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ExtendedDataSourceV2Implicits.scala
deleted file mode 100644
index 85bda0b08d46..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ExtendedDataSourceV2Implicits.scala
+++ /dev/null
@@ -1,40 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.connector.catalog.Table
-import org.apache.spark.sql.connector.write.RowLevelOperationTable
-
-/**
- * A class similar to DataSourceV2Implicits in Spark but contains custom implicit helpers.
- */
-object ExtendedDataSourceV2Implicits {
- implicit class TableHelper(table: Table) {
- def asRowLevelOperationTable: RowLevelOperationTable = {
- table match {
- case rowLevelOperationTable: RowLevelOperationTable =>
- rowLevelOperationTable
- case _ =>
- throw new AnalysisException(s"Table ${table.name} is not a row-level operation table")
- }
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ExtendedDataSourceV2Strategy.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ExtendedDataSourceV2Strategy.scala
deleted file mode 100644
index 0a27d49287f2..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ExtendedDataSourceV2Strategy.scala
+++ /dev/null
@@ -1,159 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import org.apache.iceberg.spark.Spark3Util
-import org.apache.iceberg.spark.SparkCatalog
-import org.apache.iceberg.spark.SparkSessionCatalog
-import org.apache.spark.sql.SparkSession
-import org.apache.spark.sql.Strategy
-import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.expressions.GenericInternalRow
-import org.apache.spark.sql.catalyst.expressions.Literal
-import org.apache.spark.sql.catalyst.expressions.PredicateHelper
-import org.apache.spark.sql.catalyst.plans.logical.AddPartitionField
-import org.apache.spark.sql.catalyst.plans.logical.Call
-import org.apache.spark.sql.catalyst.plans.logical.CreateOrReplaceBranch
-import org.apache.spark.sql.catalyst.plans.logical.CreateOrReplaceTag
-import org.apache.spark.sql.catalyst.plans.logical.DeleteFromIcebergTable
-import org.apache.spark.sql.catalyst.plans.logical.DropBranch
-import org.apache.spark.sql.catalyst.plans.logical.DropIdentifierFields
-import org.apache.spark.sql.catalyst.plans.logical.DropPartitionField
-import org.apache.spark.sql.catalyst.plans.logical.DropTag
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.plans.logical.MergeRows
-import org.apache.spark.sql.catalyst.plans.logical.NoStatsUnaryNode
-import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
-import org.apache.spark.sql.catalyst.plans.logical.ReplacePartitionField
-import org.apache.spark.sql.catalyst.plans.logical.SetIdentifierFields
-import org.apache.spark.sql.catalyst.plans.logical.SetWriteDistributionAndOrdering
-import org.apache.spark.sql.catalyst.plans.logical.WriteDelta
-import org.apache.spark.sql.connector.catalog.Identifier
-import org.apache.spark.sql.connector.catalog.TableCatalog
-import org.apache.spark.sql.errors.QueryCompilationErrors
-import org.apache.spark.sql.execution.SparkPlan
-import org.apache.spark.sql.execution.datasources.DataSourceStrategy
-import scala.jdk.CollectionConverters._
-
-case class ExtendedDataSourceV2Strategy(spark: SparkSession) extends Strategy with PredicateHelper {
-
- import DataSourceV2Implicits._
-
- override def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
- case c @ Call(procedure, args) =>
- val input = buildInternalRow(args)
- CallExec(c.output, procedure, input) :: Nil
-
- case AddPartitionField(IcebergCatalogAndIdentifier(catalog, ident), transform, name) =>
- AddPartitionFieldExec(catalog, ident, transform, name) :: Nil
-
- case DropPartitionField(IcebergCatalogAndIdentifier(catalog, ident), transform) =>
- DropPartitionFieldExec(catalog, ident, transform) :: Nil
-
- case ReplacePartitionField(IcebergCatalogAndIdentifier(catalog, ident), transformFrom, transformTo, name) =>
- ReplacePartitionFieldExec(catalog, ident, transformFrom, transformTo, name) :: Nil
-
- case CreateOrReplaceBranch(
- IcebergCatalogAndIdentifier(catalog, ident), branch, branchOptions, create, replace, ifNotExists) =>
- CreateOrReplaceBranchExec(catalog, ident, branch, branchOptions, create, replace, ifNotExists) :: Nil
-
- case CreateOrReplaceTag(
- IcebergCatalogAndIdentifier(catalog, ident), tag, tagOptions, create, replace, ifNotExists) =>
- CreateOrReplaceTagExec(catalog, ident, tag, tagOptions, create, replace, ifNotExists) :: Nil
-
- case DropBranch(IcebergCatalogAndIdentifier(catalog, ident), branch, ifExists) =>
- DropBranchExec(catalog, ident, branch, ifExists) :: Nil
-
- case DropTag(IcebergCatalogAndIdentifier(catalog, ident), tag, ifExists) =>
- DropTagExec(catalog, ident, tag, ifExists) :: Nil
-
- case SetIdentifierFields(IcebergCatalogAndIdentifier(catalog, ident), fields) =>
- SetIdentifierFieldsExec(catalog, ident, fields) :: Nil
-
- case DropIdentifierFields(IcebergCatalogAndIdentifier(catalog, ident), fields) =>
- DropIdentifierFieldsExec(catalog, ident, fields) :: Nil
-
- case SetWriteDistributionAndOrdering(
- IcebergCatalogAndIdentifier(catalog, ident), distributionMode, ordering) =>
- SetWriteDistributionAndOrderingExec(catalog, ident, distributionMode, ordering) :: Nil
-
- case ReplaceData(_: DataSourceV2Relation, query, r: DataSourceV2Relation, Some(write)) =>
- // refresh the cache using the original relation
- ReplaceDataExec(planLater(query), refreshCache(r), write) :: Nil
-
- case WriteDelta(_: DataSourceV2Relation, query, r: DataSourceV2Relation, projs, Some(write)) =>
- // refresh the cache using the original relation
- WriteDeltaExec(planLater(query), refreshCache(r), projs, write) :: Nil
-
- case MergeRows(isSourceRowPresent, isTargetRowPresent, matchedConditions, matchedOutputs, notMatchedConditions,
- notMatchedOutputs, targetOutput, rowIdAttrs, performCardinalityCheck, emitNotMatchedTargetRows,
- output, child) =>
-
- MergeRowsExec(isSourceRowPresent, isTargetRowPresent, matchedConditions, matchedOutputs, notMatchedConditions,
- notMatchedOutputs, targetOutput, rowIdAttrs, performCardinalityCheck, emitNotMatchedTargetRows,
- output, planLater(child)) :: Nil
-
- case DeleteFromIcebergTable(DataSourceV2ScanRelation(r, _, output), condition, None) =>
- // the optimizer has already checked that this delete can be handled using a metadata operation
- val deleteCond = condition.getOrElse(Literal.TrueLiteral)
- val predicates = splitConjunctivePredicates(deleteCond)
- val normalizedPredicates = DataSourceStrategy.normalizeExprs(predicates, output)
- val filters = normalizedPredicates.flatMap { pred =>
- val filter = DataSourceStrategy.translateFilter(pred, supportNestedPredicatePushdown = true)
- if (filter.isEmpty) {
- throw QueryCompilationErrors.cannotTranslateExpressionToSourceFilterError(pred)
- }
- filter
- }.toArray
- DeleteFromTableExec(r.table.asDeletable, filters, refreshCache(r)) :: Nil
-
- case NoStatsUnaryNode(child) =>
- planLater(child) :: Nil
-
- case _ => Nil
- }
-
- private def buildInternalRow(exprs: Seq[Expression]): InternalRow = {
- val values = new Array[Any](exprs.size)
- for (index <- exprs.indices) {
- values(index) = exprs(index).eval()
- }
- new GenericInternalRow(values)
- }
-
- private def refreshCache(r: DataSourceV2Relation)(): Unit = {
- spark.sharedState.cacheManager.recacheByPlan(spark, r)
- }
-
- private object IcebergCatalogAndIdentifier {
- def unapply(identifier: Seq[String]): Option[(TableCatalog, Identifier)] = {
- val catalogAndIdentifier = Spark3Util.catalogAndIdentifier(spark, identifier.asJava)
- catalogAndIdentifier.catalog match {
- case icebergCatalog: SparkCatalog =>
- Some((icebergCatalog, catalogAndIdentifier.identifier))
- case icebergCatalog: SparkSessionCatalog[_] =>
- Some((icebergCatalog, catalogAndIdentifier.identifier))
- case _ =>
- None
- }
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ExtendedDistributionAndOrderingUtils.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ExtendedDistributionAndOrderingUtils.scala
deleted file mode 100644
index 8c37b1b75924..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ExtendedDistributionAndOrderingUtils.scala
+++ /dev/null
@@ -1,85 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils.toCatalyst
-import org.apache.spark.sql.catalyst.expressions.SortOrder
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.plans.logical.RepartitionByExpression
-import org.apache.spark.sql.catalyst.plans.logical.Sort
-import org.apache.spark.sql.connector.distributions.ClusteredDistribution
-import org.apache.spark.sql.connector.distributions.OrderedDistribution
-import org.apache.spark.sql.connector.distributions.UnspecifiedDistribution
-import org.apache.spark.sql.connector.write.RequiresDistributionAndOrdering
-import org.apache.spark.sql.connector.write.Write
-import org.apache.spark.sql.errors.QueryCompilationErrors
-import org.apache.spark.sql.internal.SQLConf
-import scala.collection.compat.immutable.ArraySeq
-
-/**
- * A rule that is inspired by DistributionAndOrderingUtils in Spark but supports Iceberg transforms.
- *
- * Note that similarly to the original rule in Spark, it does not let AQE pick the number of shuffle
- * partitions. See SPARK-34230 for context.
- */
-object ExtendedDistributionAndOrderingUtils {
-
- def prepareQuery(write: Write, query: LogicalPlan, conf: SQLConf): LogicalPlan = write match {
- case write: RequiresDistributionAndOrdering =>
- val numPartitions = write.requiredNumPartitions()
- val distribution = write.requiredDistribution match {
- case d: OrderedDistribution => d.ordering.map(e => toCatalyst(e, query))
- case d: ClusteredDistribution => d.clustering.map(e => toCatalyst(e, query))
- case _: UnspecifiedDistribution => Array.empty[Expression]
- }
-
- val queryWithDistribution = if (distribution.nonEmpty) {
- val finalNumPartitions = if (numPartitions > 0) {
- numPartitions
- } else {
- conf.numShufflePartitions
- }
- // the conversion to catalyst expressions above produces SortOrder expressions
- // for OrderedDistribution and generic expressions for ClusteredDistribution
- // this allows RepartitionByExpression to pick either range or hash partitioning
- RepartitionByExpression(ArraySeq.unsafeWrapArray(distribution), query, finalNumPartitions)
- } else if (numPartitions > 0) {
- throw QueryCompilationErrors.numberOfPartitionsNotAllowedWithUnspecifiedDistributionError()
- } else {
- query
- }
-
- val ordering = write.requiredOrdering.toSeq
- .map(e => toCatalyst(e, query))
- .asInstanceOf[Seq[SortOrder]]
-
- val queryWithDistributionAndOrdering = if (ordering.nonEmpty) {
- Sort(ordering, global = false, queryWithDistribution)
- } else {
- queryWithDistribution
- }
-
- queryWithDistributionAndOrdering
-
- case _ =>
- query
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ExtendedV2Writes.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ExtendedV2Writes.scala
deleted file mode 100644
index 86cab35cb36e..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ExtendedV2Writes.scala
+++ /dev/null
@@ -1,140 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import java.util.UUID
-import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.expressions.PredicateHelper
-import org.apache.spark.sql.catalyst.plans.logical.AppendData
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.plans.logical.OverwriteByExpression
-import org.apache.spark.sql.catalyst.plans.logical.OverwritePartitionsDynamic
-import org.apache.spark.sql.catalyst.plans.logical.Project
-import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
-import org.apache.spark.sql.catalyst.plans.logical.WriteDelta
-import org.apache.spark.sql.catalyst.rules.Rule
-import org.apache.spark.sql.catalyst.utils.PlanUtils.isIcebergRelation
-import org.apache.spark.sql.connector.catalog.Table
-import org.apache.spark.sql.connector.iceberg.write.DeltaWriteBuilder
-import org.apache.spark.sql.connector.write.ExtendedLogicalWriteInfoImpl
-import org.apache.spark.sql.connector.write.SupportsDynamicOverwrite
-import org.apache.spark.sql.connector.write.SupportsOverwrite
-import org.apache.spark.sql.connector.write.SupportsTruncate
-import org.apache.spark.sql.connector.write.WriteBuilder
-import org.apache.spark.sql.errors.QueryCompilationErrors
-import org.apache.spark.sql.errors.QueryExecutionErrors
-import org.apache.spark.sql.execution.datasources.DataSourceStrategy
-import org.apache.spark.sql.sources.AlwaysTrue
-import org.apache.spark.sql.sources.Filter
-import org.apache.spark.sql.types.StructType
-
-/**
- * A rule that is inspired by V2Writes in Spark but supports Iceberg transforms.
- */
-object ExtendedV2Writes extends Rule[LogicalPlan] with PredicateHelper {
-
- import DataSourceV2Implicits._
-
- override def apply(plan: LogicalPlan): LogicalPlan = plan transformDown {
- case a @ AppendData(r: DataSourceV2Relation, query, options, _, None) if isIcebergRelation(r) =>
- val writeBuilder = newWriteBuilder(r.table, query.schema, options)
- val write = writeBuilder.build()
- val newQuery = ExtendedDistributionAndOrderingUtils.prepareQuery(write, query, conf)
- a.copy(write = Some(write), query = newQuery)
-
- case o @ OverwriteByExpression(r: DataSourceV2Relation, deleteExpr, query, options, _, None)
- if isIcebergRelation(r) =>
- // fail if any filter cannot be converted. correctness depends on removing all matching data.
- val filters = splitConjunctivePredicates(deleteExpr).flatMap { pred =>
- val filter = DataSourceStrategy.translateFilter(pred, supportNestedPredicatePushdown = true)
- if (filter.isEmpty) {
- throw QueryCompilationErrors.cannotTranslateExpressionToSourceFilterError(pred)
- }
- filter
- }.toArray
-
- val table = r.table
- val writeBuilder = newWriteBuilder(table, query.schema, options)
- val write = writeBuilder match {
- case builder: SupportsTruncate if isTruncate(filters) =>
- builder.truncate().build()
- case builder: SupportsOverwrite =>
- builder.overwrite(filters).build()
- case _ =>
- throw QueryExecutionErrors.overwriteTableByUnsupportedExpressionError(table)
- }
-
- val newQuery = ExtendedDistributionAndOrderingUtils.prepareQuery(write, query, conf)
- o.copy(write = Some(write), query = newQuery)
-
- case o @ OverwritePartitionsDynamic(r: DataSourceV2Relation, query, options, _, None)
- if isIcebergRelation(r) =>
- val table = r.table
- val writeBuilder = newWriteBuilder(table, query.schema, options)
- val write = writeBuilder match {
- case builder: SupportsDynamicOverwrite =>
- builder.overwriteDynamicPartitions().build()
- case _ =>
- throw QueryExecutionErrors.dynamicPartitionOverwriteUnsupportedByTableError(table)
- }
- val newQuery = ExtendedDistributionAndOrderingUtils.prepareQuery(write, query, conf)
- o.copy(write = Some(write), query = newQuery)
-
- case rd @ ReplaceData(r: DataSourceV2Relation, query, _, None) =>
- val rowSchema = StructType.fromAttributes(rd.dataInput)
- val writeBuilder = newWriteBuilder(r.table, rowSchema, Map.empty)
- val write = writeBuilder.build()
- val newQuery = ExtendedDistributionAndOrderingUtils.prepareQuery(write, query, conf)
- rd.copy(write = Some(write), query = Project(rd.dataInput, newQuery))
-
- case wd @ WriteDelta(r: DataSourceV2Relation, query, _, projections, None) =>
- val rowSchema = projections.rowProjection.map(_.schema).orNull
- val rowIdSchema = projections.rowIdProjection.schema
- val metadataSchema = projections.metadataProjection.map(_.schema).orNull
- val writeBuilder = newWriteBuilder(r.table, rowSchema, Map.empty, rowIdSchema, metadataSchema)
- writeBuilder match {
- case builder: DeltaWriteBuilder =>
- val deltaWrite = builder.build()
- val newQuery = ExtendedDistributionAndOrderingUtils.prepareQuery(deltaWrite, query, conf)
- wd.copy(write = Some(deltaWrite), query = newQuery)
- case other =>
- throw new AnalysisException(s"$other is not DeltaWriteBuilder")
- }
- }
-
- private def isTruncate(filters: Array[Filter]): Boolean = {
- filters.length == 1 && filters(0).isInstanceOf[AlwaysTrue]
- }
-
- private def newWriteBuilder(
- table: Table,
- rowSchema: StructType,
- writeOptions: Map[String, String],
- rowIdSchema: StructType = null,
- metadataSchema: StructType = null): WriteBuilder = {
- val info = ExtendedLogicalWriteInfoImpl(
- queryId = UUID.randomUUID().toString,
- rowSchema,
- writeOptions.asOptions,
- rowIdSchema,
- metadataSchema)
- table.asWritable.newWriteBuilder(info)
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala
deleted file mode 100644
index 4fbf8a523a54..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala
+++ /dev/null
@@ -1,190 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import org.apache.spark.SparkException
-import org.apache.spark.rdd.RDD
-import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.Ascending
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.catalyst.expressions.AttributeSet
-import org.apache.spark.sql.catalyst.expressions.BasePredicate
-import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.expressions.SortOrder
-import org.apache.spark.sql.catalyst.expressions.UnsafeProjection
-import org.apache.spark.sql.catalyst.expressions.codegen.GeneratePredicate
-import org.apache.spark.sql.catalyst.util.truncatedString
-import org.apache.spark.sql.execution.SparkPlan
-import org.apache.spark.sql.execution.UnaryExecNode
-
-case class MergeRowsExec(
- isSourceRowPresent: Expression,
- isTargetRowPresent: Expression,
- matchedConditions: Seq[Expression],
- matchedOutputs: Seq[Seq[Expression]],
- notMatchedConditions: Seq[Expression],
- notMatchedOutputs: Seq[Seq[Expression]],
- targetOutput: Seq[Expression],
- rowIdAttrs: Seq[Attribute],
- performCardinalityCheck: Boolean,
- emitNotMatchedTargetRows: Boolean,
- output: Seq[Attribute],
- child: SparkPlan) extends UnaryExecNode {
-
- override def requiredChildOrdering: Seq[Seq[SortOrder]] = {
- if (performCardinalityCheck) {
- // request a local sort by the row ID attrs to co-locate matches for the same target row
- Seq(rowIdAttrs.map(attr => SortOrder(attr, Ascending)))
- } else {
- Seq(Nil)
- }
- }
-
- @transient override lazy val producedAttributes: AttributeSet = {
- AttributeSet(output.filterNot(attr => inputSet.contains(attr)))
- }
-
- @transient override lazy val references: AttributeSet = child.outputSet
-
- override def simpleString(maxFields: Int): String = {
- s"MergeRowsExec${truncatedString(output, "[", ", ", "]", maxFields)}"
- }
-
- override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan = {
- copy(child = newChild)
- }
-
- protected override def doExecute(): RDD[InternalRow] = {
- child.execute().mapPartitions(processPartition)
- }
-
- private def createProjection(exprs: Seq[Expression], attrs: Seq[Attribute]): UnsafeProjection = {
- UnsafeProjection.create(exprs, attrs)
- }
-
- private def createPredicate(expr: Expression, attrs: Seq[Attribute]): BasePredicate = {
- GeneratePredicate.generate(expr, attrs)
- }
-
- private def applyProjection(
- actions: Seq[(BasePredicate, Option[UnsafeProjection])],
- inputRow: InternalRow): InternalRow = {
-
- // find the first action where the predicate evaluates to true
- // if there are overlapping conditions in actions, use the first matching action
- // in the example below, when id = 5, both actions match but the first one is applied
- // WHEN MATCHED AND id > 1 AND id < 10 UPDATE *
- // WHEN MATCHED AND id = 5 OR id = 21 DELETE
-
- val pair = actions.find {
- case (predicate, _) => predicate.eval(inputRow)
- }
-
- // apply the projection to produce an output row, or return null to suppress this row
- pair match {
- case Some((_, Some(projection))) =>
- projection.apply(inputRow)
- case _ =>
- null
- }
- }
-
- private def processPartition(rowIterator: Iterator[InternalRow]): Iterator[InternalRow] = {
- val inputAttrs = child.output
-
- val isSourceRowPresentPred = createPredicate(isSourceRowPresent, inputAttrs)
- val isTargetRowPresentPred = createPredicate(isTargetRowPresent, inputAttrs)
-
- val matchedPreds = matchedConditions.map(createPredicate(_, inputAttrs))
- val matchedProjs = matchedOutputs.map {
- case output if output.nonEmpty => Some(createProjection(output, inputAttrs))
- case _ => None
- }
- val matchedPairs = matchedPreds zip matchedProjs
-
- val notMatchedPreds = notMatchedConditions.map(createPredicate(_, inputAttrs))
- val notMatchedProjs = notMatchedOutputs.map {
- case output if output.nonEmpty => Some(createProjection(output, inputAttrs))
- case _ => None
- }
- val nonMatchedPairs = notMatchedPreds zip notMatchedProjs
-
- val projectTargetCols = createProjection(targetOutput, inputAttrs)
- val rowIdProj = createProjection(rowIdAttrs, inputAttrs)
-
- // This method is responsible for processing a input row to emit the resultant row with an
- // additional column that indicates whether the row is going to be included in the final
- // output of merge or not.
- // 1. Found a target row for which there is no corresponding source row (join condition not met)
- // - Only project the target columns if we need to output unchanged rows
- // 2. Found a source row for which there is no corresponding target row (join condition not met)
- // - Apply the not matched actions (i.e INSERT actions) if non match conditions are met.
- // 3. Found a source row for which there is a corresponding target row (join condition met)
- // - Apply the matched actions (i.e DELETE or UPDATE actions) if match conditions are met.
- def processRow(inputRow: InternalRow): InternalRow = {
- if (emitNotMatchedTargetRows && !isSourceRowPresentPred.eval(inputRow)) {
- projectTargetCols.apply(inputRow)
- } else if (!isTargetRowPresentPred.eval(inputRow)) {
- applyProjection(nonMatchedPairs, inputRow)
- } else {
- applyProjection(matchedPairs, inputRow)
- }
- }
-
- var lastMatchedRowId: InternalRow = null
-
- def processRowWithCardinalityCheck(inputRow: InternalRow): InternalRow = {
- val isSourceRowPresent = isSourceRowPresentPred.eval(inputRow)
- val isTargetRowPresent = isTargetRowPresentPred.eval(inputRow)
-
- if (isSourceRowPresent && isTargetRowPresent) {
- val currentRowId = rowIdProj.apply(inputRow)
- if (currentRowId == lastMatchedRowId) {
- throw new SparkException(
- "The ON search condition of the MERGE statement matched a single row from " +
- "the target table with multiple rows of the source table. This could result " +
- "in the target row being operated on more than once with an update or delete " +
- "operation and is not allowed.")
- }
- lastMatchedRowId = currentRowId.copy()
- } else {
- lastMatchedRowId = null
- }
-
- if (emitNotMatchedTargetRows && !isSourceRowPresent) {
- projectTargetCols.apply(inputRow)
- } else if (!isTargetRowPresent) {
- applyProjection(nonMatchedPairs, inputRow)
- } else {
- applyProjection(matchedPairs, inputRow)
- }
- }
-
- val processFunc: InternalRow => InternalRow = if (performCardinalityCheck) {
- processRowWithCardinalityCheck
- } else {
- processRow
- }
-
- rowIterator
- .map(processFunc)
- .filter(row => row != null)
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/OptimizeMetadataOnlyDeleteFromTable.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/OptimizeMetadataOnlyDeleteFromTable.scala
deleted file mode 100644
index 54e66db91b33..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/OptimizeMetadataOnlyDeleteFromTable.scala
+++ /dev/null
@@ -1,73 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.expressions.Literal
-import org.apache.spark.sql.catalyst.expressions.PredicateHelper
-import org.apache.spark.sql.catalyst.expressions.SubqueryExpression
-import org.apache.spark.sql.catalyst.plans.logical.DeleteFromIcebergTable
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.rules.Rule
-import org.apache.spark.sql.connector.catalog.SupportsDelete
-import org.apache.spark.sql.execution.datasources.DataSourceStrategy
-import org.apache.spark.sql.sources
-import org.slf4j.LoggerFactory
-
-/**
- * Checks whether a metadata delete is possible and nullifies the rewrite plan if the source can
- * handle this delete without executing the rewrite plan.
- *
- * Note this rule must be run after expression optimization.
- */
-object OptimizeMetadataOnlyDeleteFromTable extends Rule[LogicalPlan] with PredicateHelper {
-
- val logger = LoggerFactory.getLogger(OptimizeMetadataOnlyDeleteFromTable.getClass)
-
- override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
- case d @ DeleteFromIcebergTable(relation: DataSourceV2Relation, cond, Some(_)) =>
- val deleteCond = cond.getOrElse(Literal.TrueLiteral)
- relation.table match {
- case table: SupportsDelete if !SubqueryExpression.hasSubquery(deleteCond) =>
- val predicates = splitConjunctivePredicates(deleteCond)
- val normalizedPredicates = DataSourceStrategy.normalizeExprs(predicates, relation.output)
- val dataSourceFilters = toDataSourceFilters(normalizedPredicates)
- val allPredicatesTranslated = normalizedPredicates.size == dataSourceFilters.length
- if (allPredicatesTranslated && table.canDeleteWhere(dataSourceFilters)) {
- logger.info(s"Optimizing delete expression: ${dataSourceFilters.mkString(",")} as metadata delete")
- d.copy(rewritePlan = None)
- } else {
- d
- }
- case _ =>
- d
- }
- }
-
- protected def toDataSourceFilters(predicates: Seq[Expression]): Array[sources.Filter] = {
- predicates.flatMap { p =>
- val filter = DataSourceStrategy.translateFilter(p, supportNestedPredicatePushdown = true)
- if (filter.isEmpty) {
- logWarning(s"Cannot translate expression to source filter: $p")
- }
- filter
- }.toArray
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ReplaceDataExec.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ReplaceDataExec.scala
deleted file mode 100644
index 26c652469ac4..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ReplaceDataExec.scala
+++ /dev/null
@@ -1,40 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import org.apache.spark.sql.catalyst.expressions.AttributeSet
-import org.apache.spark.sql.connector.write.Write
-import org.apache.spark.sql.execution.SparkPlan
-
-/**
- * Physical plan node to replace data in existing tables.
- */
-case class ReplaceDataExec(
- query: SparkPlan,
- refreshCache: () => Unit,
- write: Write) extends V2ExistingTableWriteExec {
-
- override lazy val references: AttributeSet = query.outputSet
- override lazy val stringArgs: Iterator[Any] = Iterator(query, write)
-
- override protected def withNewChildInternal(newChild: SparkPlan): ReplaceDataExec = {
- copy(query = newChild)
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ReplacePartitionFieldExec.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ReplacePartitionFieldExec.scala
deleted file mode 100644
index fcae0a5defc4..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ReplacePartitionFieldExec.scala
+++ /dev/null
@@ -1,72 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import org.apache.iceberg.spark.Spark3Util
-import org.apache.iceberg.spark.source.SparkTable
-import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.connector.catalog.Identifier
-import org.apache.spark.sql.connector.catalog.TableCatalog
-import org.apache.spark.sql.connector.expressions.FieldReference
-import org.apache.spark.sql.connector.expressions.IdentityTransform
-import org.apache.spark.sql.connector.expressions.Transform
-
-case class ReplacePartitionFieldExec(
- catalog: TableCatalog,
- ident: Identifier,
- transformFrom: Transform,
- transformTo: Transform,
- name: Option[String]) extends LeafV2CommandExec {
- import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
-
- override lazy val output: Seq[Attribute] = Nil
-
- override protected def run(): Seq[InternalRow] = {
- catalog.loadTable(ident) match {
- case iceberg: SparkTable =>
- val schema = iceberg.table.schema
- transformFrom match {
- case IdentityTransform(FieldReference(parts)) if parts.size == 1 && schema.findField(parts.head) == null =>
- // the name is not present in the Iceberg schema, so it must be a partition field name, not a column name
- iceberg.table.updateSpec()
- .removeField(parts.head)
- .addField(name.orNull, Spark3Util.toIcebergTerm(transformTo))
- .commit()
-
- case _ =>
- iceberg.table.updateSpec()
- .removeField(Spark3Util.toIcebergTerm(transformFrom))
- .addField(name.orNull, Spark3Util.toIcebergTerm(transformTo))
- .commit()
- }
-
- case table =>
- throw new UnsupportedOperationException(s"Cannot replace partition field in non-Iceberg table: $table")
- }
-
- Nil
- }
-
- override def simpleString(maxFields: Int): String = {
- s"ReplacePartitionField ${catalog.name}.${ident.quoted} ${transformFrom.describe} " +
- s"with ${name.map(n => s"$n=").getOrElse("")}${transformTo.describe}"
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ReplaceRewrittenRowLevelCommand.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ReplaceRewrittenRowLevelCommand.scala
deleted file mode 100644
index 414d4c0ec305..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ReplaceRewrittenRowLevelCommand.scala
+++ /dev/null
@@ -1,34 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.plans.logical.RowLevelCommand
-import org.apache.spark.sql.catalyst.rules.Rule
-
-/**
- * Replaces operations such as DELETE and MERGE with the corresponding rewrite plans.
- */
-object ReplaceRewrittenRowLevelCommand extends Rule[LogicalPlan] {
- override def apply(plan: LogicalPlan): LogicalPlan = plan transformDown {
- case c: RowLevelCommand if c.rewritePlan.isDefined =>
- c.rewritePlan.get
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/RowLevelCommandScanRelationPushDown.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/RowLevelCommandScanRelationPushDown.scala
deleted file mode 100644
index 4e89b9a1c243..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/RowLevelCommandScanRelationPushDown.scala
+++ /dev/null
@@ -1,95 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import org.apache.spark.sql.catalyst.expressions.AttributeReference
-import org.apache.spark.sql.catalyst.expressions.AttributeSet
-import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.expressions.PredicateHelper
-import org.apache.spark.sql.catalyst.expressions.SubqueryExpression
-import org.apache.spark.sql.catalyst.planning.RewrittenRowLevelCommand
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.rules.Rule
-import org.apache.spark.sql.catalyst.util.CharVarcharUtils
-import org.apache.spark.sql.connector.read.ScanBuilder
-import org.apache.spark.sql.execution.datasources.DataSourceStrategy
-import org.apache.spark.sql.sources.Filter
-import org.apache.spark.sql.types.StructType
-
-object RowLevelCommandScanRelationPushDown extends Rule[LogicalPlan] with PredicateHelper {
- import ExtendedDataSourceV2Implicits._
-
- override def apply(plan: LogicalPlan): LogicalPlan = plan transformDown {
- // push down the filter from the command condition instead of the filter in the rewrite plan,
- // which may be negated for copy-on-write operations
- case RewrittenRowLevelCommand(command, relation: DataSourceV2Relation, rewritePlan) =>
- val table = relation.table.asRowLevelOperationTable
- val scanBuilder = table.newScanBuilder(relation.options)
-
- val (pushedFilters, remainingFilters) = command.condition match {
- case Some(cond) => pushFilters(cond, scanBuilder, relation.output)
- case None => (Nil, Nil)
- }
-
- val (scan, output) = PushDownUtils.pruneColumns(scanBuilder, relation, relation.output, Nil)
-
- logInfo(
- s"""
- |Pushing operators to ${relation.name}
- |Pushed filters: ${pushedFilters.mkString(", ")}
- |Filters that were not pushed: ${remainingFilters.mkString(",")}
- |Output: ${output.mkString(", ")}
- """.stripMargin)
-
- // replace DataSourceV2Relation with DataSourceV2ScanRelation for the row operation table
- // there may be multiple read relations for UPDATEs that rely on the UNION approach
- val newRewritePlan = rewritePlan transform {
- case r: DataSourceV2Relation if r.table eq table =>
- DataSourceV2ScanRelation(r, scan, toOutputAttrs(scan.readSchema(), r))
- }
-
- command.withNewRewritePlan(newRewritePlan)
- }
-
- private def pushFilters(
- cond: Expression,
- scanBuilder: ScanBuilder,
- tableAttrs: Seq[AttributeReference]): (Seq[Filter], Seq[Expression]) = {
-
- val tableAttrSet = AttributeSet(tableAttrs)
- val filters = splitConjunctivePredicates(cond).filter(_.references.subsetOf(tableAttrSet))
- val normalizedFilters = DataSourceStrategy.normalizeExprs(filters, tableAttrs)
- val (_, normalizedFiltersWithoutSubquery) =
- normalizedFilters.partition(SubqueryExpression.hasSubquery)
-
- PushDownUtils.pushFilters(scanBuilder, normalizedFiltersWithoutSubquery)
- }
-
- private def toOutputAttrs(
- schema: StructType,
- relation: DataSourceV2Relation): Seq[AttributeReference] = {
- val nameToAttr = relation.output.map(_.name).zip(relation.output).toMap
- val cleaned = CharVarcharUtils.replaceCharVarcharWithStringInSchema(schema)
- cleaned.toAttributes.map {
- // keep the attribute id during transformation
- a => a.withExprId(nameToAttr(a.name).exprId)
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/SetIdentifierFieldsExec.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/SetIdentifierFieldsExec.scala
deleted file mode 100644
index b50550ad38ef..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/SetIdentifierFieldsExec.scala
+++ /dev/null
@@ -1,53 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import org.apache.iceberg.spark.source.SparkTable
-import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.connector.catalog.Identifier
-import org.apache.spark.sql.connector.catalog.TableCatalog
-import scala.jdk.CollectionConverters._
-
-case class SetIdentifierFieldsExec(
- catalog: TableCatalog,
- ident: Identifier,
- fields: Seq[String]) extends LeafV2CommandExec {
- import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
-
- override lazy val output: Seq[Attribute] = Nil
-
- override protected def run(): Seq[InternalRow] = {
- catalog.loadTable(ident) match {
- case iceberg: SparkTable =>
- iceberg.table.updateSchema()
- .setIdentifierFields(fields.asJava)
- .commit();
- case table =>
- throw new UnsupportedOperationException(s"Cannot set identifier fields in non-Iceberg table: $table")
- }
-
- Nil
- }
-
- override def simpleString(maxFields: Int): String = {
- s"SetIdentifierFields ${catalog.name}.${ident.quoted} (${fields.quoted})";
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/SetWriteDistributionAndOrderingExec.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/SetWriteDistributionAndOrderingExec.scala
deleted file mode 100644
index 386485b10b05..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/SetWriteDistributionAndOrderingExec.scala
+++ /dev/null
@@ -1,78 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import org.apache.iceberg.DistributionMode
-import org.apache.iceberg.NullOrder
-import org.apache.iceberg.SortDirection
-import org.apache.iceberg.TableProperties.WRITE_DISTRIBUTION_MODE
-import org.apache.iceberg.expressions.Term
-import org.apache.iceberg.spark.source.SparkTable
-import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.connector.catalog.CatalogV2Implicits
-import org.apache.spark.sql.connector.catalog.Identifier
-import org.apache.spark.sql.connector.catalog.TableCatalog
-
-case class SetWriteDistributionAndOrderingExec(
- catalog: TableCatalog,
- ident: Identifier,
- distributionMode: DistributionMode,
- sortOrder: Seq[(Term, SortDirection, NullOrder)]) extends LeafV2CommandExec {
-
- import CatalogV2Implicits._
-
- override lazy val output: Seq[Attribute] = Nil
-
- override protected def run(): Seq[InternalRow] = {
- catalog.loadTable(ident) match {
- case iceberg: SparkTable =>
- val txn = iceberg.table.newTransaction()
-
- val orderBuilder = txn.replaceSortOrder()
- sortOrder.foreach {
- case (term, SortDirection.ASC, nullOrder) =>
- orderBuilder.asc(term, nullOrder)
- case (term, SortDirection.DESC, nullOrder) =>
- orderBuilder.desc(term, nullOrder)
- }
- orderBuilder.commit()
-
- txn.updateProperties()
- .set(WRITE_DISTRIBUTION_MODE, distributionMode.modeName())
- .commit()
-
- txn.commitTransaction()
-
- case table =>
- throw new UnsupportedOperationException(s"Cannot set write order of non-Iceberg table: $table")
- }
-
- Nil
- }
-
- override def simpleString(maxFields: Int): String = {
- val tableIdent = s"${catalog.name}.${ident.quoted}"
- val order = sortOrder.map {
- case (term, direction, nullOrder) => s"$term $direction $nullOrder"
- }.mkString(", ")
- s"SetWriteDistributionAndOrdering $tableIdent $distributionMode $order"
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteDeltaExec.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteDeltaExec.scala
deleted file mode 100644
index fa4e4f648313..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteDeltaExec.scala
+++ /dev/null
@@ -1,238 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.datasources.v2
-
-import org.apache.spark.SparkEnv
-import org.apache.spark.SparkException
-import org.apache.spark.TaskContext
-import org.apache.spark.internal.Logging
-import org.apache.spark.rdd.RDD
-import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.expressions.AttributeSet
-import org.apache.spark.sql.catalyst.util.RowDeltaUtils._
-import org.apache.spark.sql.catalyst.util.WriteDeltaProjections
-import org.apache.spark.sql.connector.iceberg.write.DeltaWrite
-import org.apache.spark.sql.connector.iceberg.write.DeltaWriter
-import org.apache.spark.sql.connector.write.BatchWrite
-import org.apache.spark.sql.connector.write.DataWriter
-import org.apache.spark.sql.connector.write.DataWriterFactory
-import org.apache.spark.sql.connector.write.PhysicalWriteInfoImpl
-import org.apache.spark.sql.connector.write.WriterCommitMessage
-import org.apache.spark.sql.errors.QueryExecutionErrors
-import org.apache.spark.sql.execution.SparkPlan
-import org.apache.spark.sql.execution.metric.CustomMetrics
-import org.apache.spark.sql.execution.metric.SQLMetric
-import org.apache.spark.util.LongAccumulator
-import org.apache.spark.util.Utils
-import scala.collection.compat.immutable.ArraySeq
-import scala.util.control.NonFatal
-
-/**
- * Physical plan node to write a delta of rows to an existing table.
- */
-case class WriteDeltaExec(
- query: SparkPlan,
- refreshCache: () => Unit,
- projections: WriteDeltaProjections,
- write: DeltaWrite) extends ExtendedV2ExistingTableWriteExec[DeltaWriter[InternalRow]] {
-
- override lazy val references: AttributeSet = query.outputSet
- override lazy val stringArgs: Iterator[Any] = Iterator(query, write)
-
- override lazy val writingTask: WritingSparkTask[DeltaWriter[InternalRow]] = {
- DeltaWithMetadataWritingSparkTask(projections)
- }
-
- override protected def withNewChildInternal(newChild: SparkPlan): WriteDeltaExec = {
- copy(query = newChild)
- }
-}
-
-// a trait similar to V2ExistingTableWriteExec but supports custom write tasks
-trait ExtendedV2ExistingTableWriteExec[W <: DataWriter[InternalRow]] extends V2ExistingTableWriteExec {
- def writingTask: WritingSparkTask[W]
-
- protected override def writeWithV2(batchWrite: BatchWrite): Seq[InternalRow] = {
- val rdd: RDD[InternalRow] = {
- val tempRdd = query.execute()
- // SPARK-23271 If we are attempting to write a zero partition rdd, create a dummy single
- // partition rdd to make sure we at least set up one write task to write the metadata.
- if (tempRdd.partitions.length == 0) {
- sparkContext.parallelize(Seq.empty[InternalRow], 1)
- } else {
- tempRdd
- }
- }
- // introduce a local var to avoid serializing the whole class
- val task = writingTask
- val writerFactory = batchWrite.createBatchWriterFactory(
- PhysicalWriteInfoImpl(rdd.getNumPartitions))
- val useCommitCoordinator = batchWrite.useCommitCoordinator
- val messages = new Array[WriterCommitMessage](rdd.partitions.length)
- val totalNumRowsAccumulator = new LongAccumulator()
-
- logInfo(s"Start processing data source write support: $batchWrite. " +
- s"The input RDD has ${messages.length} partitions.")
-
- // Avoid object not serializable issue.
- val writeMetrics: Map[String, SQLMetric] = customMetrics
-
- try {
- sparkContext.runJob(
- rdd,
- (context: TaskContext, iter: Iterator[InternalRow]) =>
- task.run(writerFactory, context, iter, useCommitCoordinator, writeMetrics),
- rdd.partitions.indices,
- (index, result: DataWritingSparkTaskResult) => {
- val commitMessage = result.writerCommitMessage
- messages(index) = commitMessage
- totalNumRowsAccumulator.add(result.numRows)
- batchWrite.onDataWriterCommit(commitMessage)
- }
- )
-
- logInfo(s"Data source write support $batchWrite is committing.")
- batchWrite.commit(messages)
- logInfo(s"Data source write support $batchWrite committed.")
- commitProgress = Some(StreamWriterCommitProgress(totalNumRowsAccumulator.value))
- } catch {
- case cause: Throwable =>
- logError(s"Data source write support $batchWrite is aborting.")
- try {
- batchWrite.abort(messages)
- } catch {
- case t: Throwable =>
- logError(s"Data source write support $batchWrite failed to abort.")
- cause.addSuppressed(t)
- throw QueryExecutionErrors.writingJobFailedError(cause)
- }
- logError(s"Data source write support $batchWrite aborted.")
- cause match {
- // Only wrap non fatal exceptions.
- case NonFatal(e) => throw QueryExecutionErrors.writingJobAbortedError(e)
- case _ => throw cause
- }
- }
-
- Nil
- }
-}
-
-trait WritingSparkTask[W <: DataWriter[InternalRow]] extends Logging with Serializable {
-
- protected def writeFunc(writer: W, row: InternalRow): Unit
-
- def run(
- writerFactory: DataWriterFactory,
- context: TaskContext,
- iter: Iterator[InternalRow],
- useCommitCoordinator: Boolean,
- customMetrics: Map[String, SQLMetric]): DataWritingSparkTaskResult = {
- val stageId = context.stageId()
- val stageAttempt = context.stageAttemptNumber()
- val partId = context.partitionId()
- val taskId = context.taskAttemptId()
- val attemptId = context.attemptNumber()
- val dataWriter = writerFactory.createWriter(partId, taskId).asInstanceOf[W]
-
- var count = 0L
- // write the data and commit this writer.
- Utils.tryWithSafeFinallyAndFailureCallbacks(block = {
- while (iter.hasNext) {
- if (count % CustomMetrics.NUM_ROWS_PER_UPDATE == 0) {
- CustomMetrics.updateMetrics(ArraySeq.unsafeWrapArray(dataWriter.currentMetricsValues), customMetrics)
- }
-
- // Count is here.
- count += 1
- writeFunc(dataWriter, iter.next())
- }
-
- CustomMetrics.updateMetrics(ArraySeq.unsafeWrapArray(dataWriter.currentMetricsValues), customMetrics)
-
- val msg = if (useCommitCoordinator) {
- val coordinator = SparkEnv.get.outputCommitCoordinator
- val commitAuthorized = coordinator.canCommit(stageId, stageAttempt, partId, attemptId)
- if (commitAuthorized) {
- logInfo(s"Commit authorized for partition $partId (task $taskId, attempt $attemptId, " +
- s"stage $stageId.$stageAttempt)")
- dataWriter.commit()
- } else {
- val commitDeniedException = QueryExecutionErrors.commitDeniedError(
- partId, taskId, attemptId, stageId, stageAttempt)
- logInfo(commitDeniedException.getMessage)
- // throwing CommitDeniedException will trigger the catch block for abort
- throw commitDeniedException
- }
-
- } else {
- logInfo(s"Writer for partition ${context.partitionId()} is committing.")
- dataWriter.commit()
- }
-
- logInfo(s"Committed partition $partId (task $taskId, attempt $attemptId, " +
- s"stage $stageId.$stageAttempt)")
-
- DataWritingSparkTaskResult(count, msg)
-
- })(catchBlock = {
- // If there is an error, abort this writer
- logError(s"Aborting commit for partition $partId (task $taskId, attempt $attemptId, " +
- s"stage $stageId.$stageAttempt)")
- dataWriter.abort()
- logError(s"Aborted commit for partition $partId (task $taskId, attempt $attemptId, " +
- s"stage $stageId.$stageAttempt)")
- }, finallyBlock = {
- dataWriter.close()
- })
- }
-}
-
-case class DeltaWithMetadataWritingSparkTask(
- projs: WriteDeltaProjections) extends WritingSparkTask[DeltaWriter[InternalRow]] {
-
- private lazy val rowProjection = projs.rowProjection.orNull
- private lazy val rowIdProjection = projs.rowIdProjection
- private lazy val metadataProjection = projs.metadataProjection.orNull
-
- override protected def writeFunc(writer: DeltaWriter[InternalRow], row: InternalRow): Unit = {
- val operation = row.getInt(0)
-
- operation match {
- case DELETE_OPERATION =>
- rowIdProjection.project(row)
- metadataProjection.project(row)
- writer.delete(metadataProjection, rowIdProjection)
-
- case UPDATE_OPERATION =>
- rowProjection.project(row)
- rowIdProjection.project(row)
- metadataProjection.project(row)
- writer.update(metadataProjection, rowIdProjection, rowProjection)
-
- case INSERT_OPERATION =>
- rowProjection.project(row)
- writer.insert(rowProjection)
-
- case other =>
- throw new SparkException(s"Unexpected operation ID: $other")
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/RowLevelCommandDynamicPruning.scala b/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/RowLevelCommandDynamicPruning.scala
deleted file mode 100644
index 4b257920b66f..000000000000
--- a/spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/RowLevelCommandDynamicPruning.scala
+++ /dev/null
@@ -1,183 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.spark.sql.execution.dynamicpruning
-
-import org.apache.spark.sql.SparkSession
-import org.apache.spark.sql.catalyst.expressions.And
-import org.apache.spark.sql.catalyst.expressions.Attribute
-import org.apache.spark.sql.catalyst.expressions.AttributeMap
-import org.apache.spark.sql.catalyst.expressions.AttributeReference
-import org.apache.spark.sql.catalyst.expressions.DynamicPruningSubquery
-import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
-import org.apache.spark.sql.catalyst.expressions.Literal
-import org.apache.spark.sql.catalyst.expressions.PredicateHelper
-import org.apache.spark.sql.catalyst.expressions.SubqueryExpression
-import org.apache.spark.sql.catalyst.planning.RewrittenRowLevelCommand
-import org.apache.spark.sql.catalyst.plans.LeftSemi
-import org.apache.spark.sql.catalyst.plans.logical.DeleteFromIcebergTable
-import org.apache.spark.sql.catalyst.plans.logical.Filter
-import org.apache.spark.sql.catalyst.plans.logical.Join
-import org.apache.spark.sql.catalyst.plans.logical.JoinHint
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
-import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
-import org.apache.spark.sql.catalyst.plans.logical.Project
-import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
-import org.apache.spark.sql.catalyst.plans.logical.RowLevelCommand
-import org.apache.spark.sql.catalyst.plans.logical.Sort
-import org.apache.spark.sql.catalyst.plans.logical.Subquery
-import org.apache.spark.sql.catalyst.plans.logical.UpdateIcebergTable
-import org.apache.spark.sql.catalyst.rules.Rule
-import org.apache.spark.sql.catalyst.trees.TreePattern.PLAN_EXPRESSION
-import org.apache.spark.sql.catalyst.trees.TreePattern.SORT
-import org.apache.spark.sql.connector.read.SupportsRuntimeFiltering
-import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
-import org.apache.spark.sql.execution.datasources.v2.DataSourceV2ScanRelation
-import org.apache.spark.sql.execution.datasources.v2.ExtendedDataSourceV2Implicits
-import scala.collection.compat.immutable.ArraySeq
-
-/**
- * A rule that adds a runtime filter for row-level commands.
- *
- * Note that only group-based rewrite plans (i.e. ReplaceData) are taken into account.
- * Row-based rewrite plans are subject to usual runtime filtering.
- */
-case class RowLevelCommandDynamicPruning(spark: SparkSession) extends Rule[LogicalPlan] with PredicateHelper {
-
- import ExtendedDataSourceV2Implicits._
-
- override def apply(plan: LogicalPlan): LogicalPlan = plan transformDown {
- // apply special dynamic filtering only for plans that don't support deltas
- case RewrittenRowLevelCommand(
- command: RowLevelCommand,
- DataSourceV2ScanRelation(_, scan: SupportsRuntimeFiltering, _),
- rewritePlan: ReplaceData) if conf.dynamicPartitionPruningEnabled && isCandidate(command) =>
-
- // use reference equality to find exactly the required scan relations
- val newRewritePlan = rewritePlan transformUp {
- case r: DataSourceV2ScanRelation if r.scan eq scan =>
- // use the original table instance that was loaded for this row-level operation
- // in order to leverage a regular batch scan in the group filter query
- val originalTable = r.relation.table.asRowLevelOperationTable.table
- val relation = r.relation.copy(table = originalTable)
- val matchingRowsPlan = buildMatchingRowsPlan(relation, command)
-
- val filterAttrs = ArraySeq.unsafeWrapArray(scan.filterAttributes)
- val buildKeys = ExtendedV2ExpressionUtils.resolveRefs[Attribute](filterAttrs, matchingRowsPlan)
- val pruningKeys = ExtendedV2ExpressionUtils.resolveRefs[Attribute](filterAttrs, r)
- val dynamicPruningCond = buildDynamicPruningCond(matchingRowsPlan, buildKeys, pruningKeys)
-
- Filter(dynamicPruningCond, r)
- }
-
- // always optimize dynamic filtering subqueries for row-level commands as it is important
- // to rewrite introduced predicates as joins because Spark recently stopped optimizing
- // dynamic subqueries to facilitate broadcast reuse
- command.withNewRewritePlan(optimizeSubquery(newRewritePlan))
-
- }
-
- private def isCandidate(command: RowLevelCommand): Boolean = command.condition match {
- case Some(cond) if cond != Literal.TrueLiteral => true
- case _ => false
- }
-
- private def buildMatchingRowsPlan(
- relation: DataSourceV2Relation,
- command: RowLevelCommand): LogicalPlan = {
-
- // construct a filtering plan with the original scan relation
- val matchingRowsPlan = command match {
- case d: DeleteFromIcebergTable =>
- Filter(d.condition.get, relation)
-
- case u: UpdateIcebergTable =>
- // UPDATEs with subqueries are rewritten using a UNION with two identical scan relations
- // the analyzer clones of them and assigns fresh expr IDs so that attributes don't collide
- // this rule assigns dynamic filters to both scan relations based on the update condition
- // the condition always refers to the original expr IDs and must be transformed
- // see RewriteUpdateTable for more details
- val attrMap = buildAttrMap(u.table.output, relation.output)
- val transformedCond = u.condition.get transform {
- case attr: AttributeReference if attrMap.contains(attr) => attrMap(attr)
- }
- Filter(transformedCond, relation)
-
- case m: MergeIntoIcebergTable =>
- Join(relation, m.sourceTable, LeftSemi, Some(m.mergeCondition), JoinHint.NONE)
- }
-
- // clone the original relation in the filtering plan and assign new expr IDs to avoid conflicts
- matchingRowsPlan transformUpWithNewOutput {
- case r: DataSourceV2Relation if r eq relation =>
- val oldOutput = r.output
- val newOutput = oldOutput.map(_.newInstance())
- r.copy(output = newOutput) -> oldOutput.zip(newOutput)
- }
- }
-
- private def buildDynamicPruningCond(
- matchingRowsPlan: LogicalPlan,
- buildKeys: Seq[Attribute],
- pruningKeys: Seq[Attribute]): Expression = {
-
- val buildQuery = Project(buildKeys, matchingRowsPlan)
- val dynamicPruningSubqueries = pruningKeys.zipWithIndex.map { case (key, index) =>
- DynamicPruningSubquery(key, buildQuery, buildKeys, index, onlyInBroadcast = false)
- }
- dynamicPruningSubqueries.reduce(And)
- }
-
- private def buildAttrMap(
- tableAttrs: Seq[Attribute],
- scanAttrs: Seq[Attribute]): AttributeMap[Attribute] = {
-
- val resolver = conf.resolver
- val attrMapping = tableAttrs.flatMap { tableAttr =>
- scanAttrs
- .find(scanAttr => resolver(scanAttr.name, tableAttr.name))
- .map(scanAttr => tableAttr -> scanAttr)
- }
- AttributeMap(attrMapping)
- }
-
- // borrowed from OptimizeSubqueries in Spark
- private def optimizeSubquery(plan: LogicalPlan): LogicalPlan = plan.transformAllExpressionsWithPruning(
- _.containsPattern(PLAN_EXPRESSION)) {
- case s: SubqueryExpression =>
- val Subquery(newPlan, _) = spark.sessionState.optimizer.execute(Subquery.fromExpression(s))
- // At this point we have an optimized subquery plan that we are going to attach
- // to this subquery expression. Here we can safely remove any top level sort
- // in the plan as tuples produced by a subquery are un-ordered.
- s.withNewPlan(removeTopLevelSort(newPlan))
- }
-
- // borrowed from OptimizeSubqueries in Spark
- private def removeTopLevelSort(plan: LogicalPlan): LogicalPlan = {
- if (!plan.containsPattern(SORT)) {
- return plan
- }
- plan match {
- case Sort(_, _, child) => child
- case Project(fields, child) => Project(fields, removeTopLevelSort(child))
- case other => other
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/Employee.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/Employee.java
deleted file mode 100644
index 8918dfec6584..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/Employee.java
+++ /dev/null
@@ -1,66 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import java.util.Objects;
-
-public class Employee {
- private Integer id;
- private String dep;
-
- public Employee() {}
-
- public Employee(Integer id, String dep) {
- this.id = id;
- this.dep = dep;
- }
-
- public Integer getId() {
- return id;
- }
-
- public void setId(Integer id) {
- this.id = id;
- }
-
- public String getDep() {
- return dep;
- }
-
- public void setDep(String dep) {
- this.dep = dep;
- }
-
- @Override
- public boolean equals(Object other) {
- if (this == other) {
- return true;
- } else if (other == null || getClass() != other.getClass()) {
- return false;
- }
-
- Employee employee = (Employee) other;
- return Objects.equals(id, employee.id) && Objects.equals(dep, employee.dep);
- }
-
- @Override
- public int hashCode() {
- return Objects.hash(id, dep);
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkExtensionsTestBase.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkExtensionsTestBase.java
deleted file mode 100644
index fb9cc09567fa..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkExtensionsTestBase.java
+++ /dev/null
@@ -1,70 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import static org.apache.hadoop.hive.conf.HiveConf.ConfVars.METASTOREURIS;
-
-import java.util.Map;
-import java.util.Random;
-import java.util.concurrent.ThreadLocalRandom;
-import org.apache.iceberg.CatalogUtil;
-import org.apache.iceberg.hive.HiveCatalog;
-import org.apache.iceberg.hive.TestHiveMetastore;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
-import org.apache.iceberg.spark.SparkCatalogTestBase;
-import org.apache.iceberg.spark.SparkTestBase;
-import org.apache.spark.sql.SparkSession;
-import org.apache.spark.sql.internal.SQLConf;
-import org.junit.BeforeClass;
-
-public abstract class SparkExtensionsTestBase extends SparkCatalogTestBase {
-
- private static final Random RANDOM = ThreadLocalRandom.current();
-
- public SparkExtensionsTestBase(
- String catalogName, String implementation, Map config) {
- super(catalogName, implementation, config);
- }
-
- @BeforeClass
- public static void startMetastoreAndSpark() {
- SparkTestBase.metastore = new TestHiveMetastore();
- metastore.start();
- SparkTestBase.hiveConf = metastore.hiveConf();
-
- SparkTestBase.spark =
- SparkSession.builder()
- .master("local[2]")
- .config("spark.testing", "true")
- .config(SQLConf.PARTITION_OVERWRITE_MODE().key(), "dynamic")
- .config("spark.sql.extensions", IcebergSparkSessionExtensions.class.getName())
- .config("spark.hadoop." + METASTOREURIS.varname, hiveConf.get(METASTOREURIS.varname))
- .config("spark.sql.shuffle.partitions", "4")
- .config("spark.sql.hive.metastorePartitionPruningFallbackOnException", "true")
- .config(
- SQLConf.ADAPTIVE_EXECUTION_ENABLED().key(), String.valueOf(RANDOM.nextBoolean()))
- .enableHiveSupport()
- .getOrCreate();
-
- SparkTestBase.catalog =
- (HiveCatalog)
- CatalogUtil.loadCatalog(
- HiveCatalog.class.getName(), "hive", ImmutableMap.of(), hiveConf);
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkRowLevelOperationsTestBase.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkRowLevelOperationsTestBase.java
deleted file mode 100644
index e53c79f6c971..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/SparkRowLevelOperationsTestBase.java
+++ /dev/null
@@ -1,314 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import static org.apache.iceberg.DataOperations.DELETE;
-import static org.apache.iceberg.DataOperations.OVERWRITE;
-import static org.apache.iceberg.SnapshotSummary.ADDED_DELETE_FILES_PROP;
-import static org.apache.iceberg.SnapshotSummary.ADDED_FILES_PROP;
-import static org.apache.iceberg.SnapshotSummary.CHANGED_PARTITION_COUNT_PROP;
-import static org.apache.iceberg.SnapshotSummary.DELETED_FILES_PROP;
-import static org.apache.iceberg.TableProperties.DEFAULT_FILE_FORMAT;
-import static org.apache.iceberg.TableProperties.PARQUET_VECTORIZATION_ENABLED;
-import static org.apache.iceberg.TableProperties.WRITE_DISTRIBUTION_MODE;
-import static org.apache.iceberg.TableProperties.WRITE_DISTRIBUTION_MODE_HASH;
-import static org.apache.iceberg.TableProperties.WRITE_DISTRIBUTION_MODE_NONE;
-import static org.apache.iceberg.TableProperties.WRITE_DISTRIBUTION_MODE_RANGE;
-
-import java.io.IOException;
-import java.io.UncheckedIOException;
-import java.util.Arrays;
-import java.util.List;
-import java.util.Map;
-import java.util.Random;
-import java.util.Set;
-import java.util.concurrent.ThreadLocalRandom;
-import java.util.stream.Collectors;
-import org.apache.iceberg.DataFile;
-import org.apache.iceberg.Files;
-import org.apache.iceberg.Snapshot;
-import org.apache.iceberg.Table;
-import org.apache.iceberg.data.GenericRecord;
-import org.apache.iceberg.data.parquet.GenericParquetWriter;
-import org.apache.iceberg.io.DataWriter;
-import org.apache.iceberg.io.OutputFile;
-import org.apache.iceberg.parquet.Parquet;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
-import org.apache.iceberg.spark.SparkCatalog;
-import org.apache.iceberg.spark.SparkSessionCatalog;
-import org.apache.spark.sql.Dataset;
-import org.apache.spark.sql.Encoder;
-import org.apache.spark.sql.Encoders;
-import org.apache.spark.sql.Row;
-import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
-import org.junit.Assert;
-import org.junit.runner.RunWith;
-import org.junit.runners.Parameterized;
-import org.junit.runners.Parameterized.Parameters;
-
-@RunWith(Parameterized.class)
-public abstract class SparkRowLevelOperationsTestBase extends SparkExtensionsTestBase {
-
- private static final Random RANDOM = ThreadLocalRandom.current();
-
- protected final String fileFormat;
- protected final boolean vectorized;
- protected final String distributionMode;
-
- public SparkRowLevelOperationsTestBase(
- String catalogName,
- String implementation,
- Map config,
- String fileFormat,
- boolean vectorized,
- String distributionMode) {
- super(catalogName, implementation, config);
- this.fileFormat = fileFormat;
- this.vectorized = vectorized;
- this.distributionMode = distributionMode;
- }
-
- @Parameters(
- name =
- "catalogName = {0}, implementation = {1}, config = {2},"
- + " format = {3}, vectorized = {4}, distributionMode = {5}")
- public static Object[][] parameters() {
- return new Object[][] {
- {
- "testhive",
- SparkCatalog.class.getName(),
- ImmutableMap.of(
- "type", "hive",
- "default-namespace", "default"),
- "orc",
- true,
- WRITE_DISTRIBUTION_MODE_NONE
- },
- {
- "testhive",
- SparkCatalog.class.getName(),
- ImmutableMap.of(
- "type", "hive",
- "default-namespace", "default"),
- "parquet",
- true,
- WRITE_DISTRIBUTION_MODE_NONE
- },
- {
- "testhadoop",
- SparkCatalog.class.getName(),
- ImmutableMap.of("type", "hadoop"),
- "parquet",
- RANDOM.nextBoolean(),
- WRITE_DISTRIBUTION_MODE_HASH
- },
- {
- "spark_catalog",
- SparkSessionCatalog.class.getName(),
- ImmutableMap.of(
- "type", "hive",
- "default-namespace", "default",
- "clients", "1",
- "parquet-enabled", "false",
- "cache-enabled",
- "false" // Spark will delete tables using v1, leaving the cache out of sync
- ),
- "avro",
- false,
- WRITE_DISTRIBUTION_MODE_RANGE
- }
- };
- }
-
- protected abstract Map extraTableProperties();
-
- protected void initTable() {
- sql("ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')", tableName, DEFAULT_FILE_FORMAT, fileFormat);
- sql(
- "ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')",
- tableName, WRITE_DISTRIBUTION_MODE, distributionMode);
-
- switch (fileFormat) {
- case "parquet":
- sql(
- "ALTER TABLE %s SET TBLPROPERTIES('%s' '%b')",
- tableName, PARQUET_VECTORIZATION_ENABLED, vectorized);
- break;
- case "orc":
- Assert.assertTrue(vectorized);
- break;
- case "avro":
- Assert.assertFalse(vectorized);
- break;
- }
-
- Map props = extraTableProperties();
- props.forEach(
- (prop, value) -> {
- sql("ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')", tableName, prop, value);
- });
- }
-
- protected void createAndInitTable(String schema) {
- createAndInitTable(schema, null);
- }
-
- protected void createAndInitTable(String schema, String jsonData) {
- sql("CREATE TABLE %s (%s) USING iceberg", tableName, schema);
- initTable();
-
- if (jsonData != null) {
- try {
- Dataset ds = toDS(schema, jsonData);
- ds.writeTo(tableName).append();
- } catch (NoSuchTableException e) {
- throw new RuntimeException("Failed to write data", e);
- }
- }
- }
-
- protected void append(String table, String jsonData) {
- append(table, null, jsonData);
- }
-
- protected void append(String table, String schema, String jsonData) {
- try {
- Dataset ds = toDS(schema, jsonData);
- ds.coalesce(1).writeTo(table).append();
- } catch (NoSuchTableException e) {
- throw new RuntimeException("Failed to write data", e);
- }
- }
-
- protected void createOrReplaceView(String name, String jsonData) {
- createOrReplaceView(name, null, jsonData);
- }
-
- protected void createOrReplaceView(String name, String schema, String jsonData) {
- Dataset ds = toDS(schema, jsonData);
- ds.createOrReplaceTempView(name);
- }
-
- protected void createOrReplaceView(String name, List data, Encoder encoder) {
- spark.createDataset(data, encoder).createOrReplaceTempView(name);
- }
-
- private Dataset toDS(String schema, String jsonData) {
- List jsonRows =
- Arrays.stream(jsonData.split("\n"))
- .filter(str -> !str.trim().isEmpty())
- .collect(Collectors.toList());
- Dataset jsonDS = spark.createDataset(jsonRows, Encoders.STRING());
-
- if (schema != null) {
- return spark.read().schema(schema).json(jsonDS);
- } else {
- return spark.read().json(jsonDS);
- }
- }
-
- protected void validateDelete(
- Snapshot snapshot, String changedPartitionCount, String deletedDataFiles) {
- validateSnapshot(snapshot, DELETE, changedPartitionCount, deletedDataFiles, null, null);
- }
-
- protected void validateCopyOnWrite(
- Snapshot snapshot,
- String changedPartitionCount,
- String deletedDataFiles,
- String addedDataFiles) {
- validateSnapshot(
- snapshot, OVERWRITE, changedPartitionCount, deletedDataFiles, null, addedDataFiles);
- }
-
- protected void validateMergeOnRead(
- Snapshot snapshot,
- String changedPartitionCount,
- String addedDeleteFiles,
- String addedDataFiles) {
- validateSnapshot(
- snapshot, OVERWRITE, changedPartitionCount, null, addedDeleteFiles, addedDataFiles);
- }
-
- protected void validateSnapshot(
- Snapshot snapshot,
- String operation,
- String changedPartitionCount,
- String deletedDataFiles,
- String addedDeleteFiles,
- String addedDataFiles) {
- Assert.assertEquals("Operation must match", operation, snapshot.operation());
- validateProperty(snapshot, CHANGED_PARTITION_COUNT_PROP, changedPartitionCount);
- validateProperty(snapshot, DELETED_FILES_PROP, deletedDataFiles);
- validateProperty(snapshot, ADDED_DELETE_FILES_PROP, addedDeleteFiles);
- validateProperty(snapshot, ADDED_FILES_PROP, addedDataFiles);
- }
-
- protected void validateProperty(Snapshot snapshot, String property, Set expectedValues) {
- String actual = snapshot.summary().get(property);
- Assert.assertTrue(
- "Snapshot property "
- + property
- + " has unexpected value, actual = "
- + actual
- + ", expected one of : "
- + String.join(",", expectedValues),
- expectedValues.contains(actual));
- }
-
- protected void validateProperty(Snapshot snapshot, String property, String expectedValue) {
- String actual = snapshot.summary().get(property);
- Assert.assertEquals(
- "Snapshot property " + property + " has unexpected value.", expectedValue, actual);
- }
-
- protected void sleep(long millis) {
- try {
- Thread.sleep(millis);
- } catch (InterruptedException e) {
- throw new RuntimeException(e);
- }
- }
-
- protected DataFile writeDataFile(Table table, List records) {
- try {
- OutputFile file = Files.localOutput(temp.newFile());
-
- DataWriter dataWriter =
- Parquet.writeData(file)
- .forTable(table)
- .createWriterFunc(GenericParquetWriter::buildWriter)
- .overwrite()
- .build();
-
- try {
- for (GenericRecord record : records) {
- dataWriter.write(record);
- }
- } finally {
- dataWriter.close();
- }
-
- return dataWriter.toDataFile();
-
- } catch (IOException e) {
- throw new UncheckedIOException(e);
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAddFilesProcedure.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAddFilesProcedure.java
deleted file mode 100644
index 10426ebb5594..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAddFilesProcedure.java
+++ /dev/null
@@ -1,1136 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import java.io.File;
-import java.io.IOException;
-import java.util.List;
-import java.util.Map;
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-import java.util.stream.Collectors;
-import org.apache.avro.Schema;
-import org.apache.avro.SchemaBuilder;
-import org.apache.avro.file.DataFileWriter;
-import org.apache.avro.generic.GenericData;
-import org.apache.avro.generic.GenericDatumWriter;
-import org.apache.avro.generic.GenericRecord;
-import org.apache.avro.io.DatumWriter;
-import org.apache.iceberg.AssertHelpers;
-import org.apache.iceberg.DataFile;
-import org.apache.iceberg.TableProperties;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
-import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-import org.apache.spark.sql.Dataset;
-import org.apache.spark.sql.Row;
-import org.apache.spark.sql.RowFactory;
-import org.apache.spark.sql.types.DataTypes;
-import org.apache.spark.sql.types.Metadata;
-import org.apache.spark.sql.types.StructField;
-import org.apache.spark.sql.types.StructType;
-import org.joda.time.DateTime;
-import org.junit.After;
-import org.junit.Assert;
-import org.junit.Assume;
-import org.junit.Before;
-import org.junit.Ignore;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.TemporaryFolder;
-
-public class TestAddFilesProcedure extends SparkExtensionsTestBase {
-
- private final String sourceTableName = "source_table";
- private File fileTableDir;
-
- public TestAddFilesProcedure(
- String catalogName, String implementation, Map config) {
- super(catalogName, implementation, config);
- }
-
- @Rule public TemporaryFolder temp = new TemporaryFolder();
-
- @Before
- public void setupTempDirs() {
- try {
- fileTableDir = temp.newFolder();
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
-
- @After
- public void dropTables() {
- sql("DROP TABLE IF EXISTS %s", sourceTableName);
- sql("DROP TABLE IF EXISTS %s", tableName);
- }
-
- @Test
- public void addDataUnpartitioned() {
- createUnpartitionedFileTable("parquet");
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg";
-
- sql(createIceberg, tableName);
-
- Object result =
- scalarSql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`')",
- catalogName, tableName, fileTableDir.getAbsolutePath());
-
- Assert.assertEquals(2L, result);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT * FROM %s ORDER BY id", sourceTableName),
- sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void deleteAndAddBackUnpartitioned() {
- createUnpartitionedFileTable("parquet");
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg";
-
- sql(createIceberg, tableName);
-
- sql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`')",
- catalogName, tableName, fileTableDir.getAbsolutePath());
-
- String deleteData = "DELETE FROM %s";
- sql(deleteData, tableName);
-
- Object result =
- scalarSql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`')",
- catalogName, tableName, fileTableDir.getAbsolutePath());
- Assert.assertEquals(2L, result);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT * FROM %s ORDER BY id", sourceTableName),
- sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Ignore // TODO Classpath issues prevent us from actually writing to a Spark ORC table
- public void addDataUnpartitionedOrc() {
- createUnpartitionedFileTable("orc");
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg";
-
- sql(createIceberg, tableName);
-
- Object result =
- scalarSql(
- "CALL %s.system.add_files('%s', '`orc`.`%s`')",
- catalogName, tableName, fileTableDir.getAbsolutePath());
-
- Assert.assertEquals(2L, result);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT * FROM %s ORDER BY id", sourceTableName),
- sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void addAvroFile() throws Exception {
- // Spark Session Catalog cannot load metadata tables
- // with "The namespace in session catalog must have exactly one name part"
- Assume.assumeFalse(catalogName.equals("spark_catalog"));
-
- // Create an Avro file
-
- Schema schema =
- SchemaBuilder.record("record")
- .fields()
- .requiredInt("id")
- .requiredString("data")
- .endRecord();
- GenericRecord record1 = new GenericData.Record(schema);
- record1.put("id", 1L);
- record1.put("data", "a");
- GenericRecord record2 = new GenericData.Record(schema);
- record2.put("id", 2L);
- record2.put("data", "b");
- File outputFile = temp.newFile("test.avro");
-
- DatumWriter datumWriter = new GenericDatumWriter(schema);
- DataFileWriter dataFileWriter = new DataFileWriter(datumWriter);
- dataFileWriter.create(schema, outputFile);
- dataFileWriter.append(record1);
- dataFileWriter.append(record2);
- dataFileWriter.close();
-
- String createIceberg = "CREATE TABLE %s (id Long, data String) USING iceberg";
- sql(createIceberg, tableName);
-
- Object result =
- scalarSql(
- "CALL %s.system.add_files('%s', '`avro`.`%s`')",
- catalogName, tableName, outputFile.getPath());
- Assert.assertEquals(1L, result);
-
- List expected = Lists.newArrayList(new Object[] {1L, "a"}, new Object[] {2L, "b"});
-
- assertEquals(
- "Iceberg table contains correct data",
- expected,
- sql("SELECT * FROM %s ORDER BY id", tableName));
-
- List actualRecordCount =
- sql("select %s from %s.files", DataFile.RECORD_COUNT.name(), tableName);
- List expectedRecordCount = Lists.newArrayList();
- expectedRecordCount.add(new Object[] {2L});
- assertEquals(
- "Iceberg file metadata should have correct metadata count",
- expectedRecordCount,
- actualRecordCount);
- }
-
- // TODO Adding spark-avro doesn't work in tests
- @Ignore
- public void addDataUnpartitionedAvro() {
- createUnpartitionedFileTable("avro");
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg";
-
- sql(createIceberg, tableName);
-
- Object result =
- scalarSql(
- "CALL %s.system.add_files('%s', '`avro`.`%s`')",
- catalogName, tableName, fileTableDir.getAbsolutePath());
-
- Assert.assertEquals(2L, result);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT * FROM %s ORDER BY id", sourceTableName),
- sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void addDataUnpartitionedHive() {
- createUnpartitionedHiveTable();
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg";
-
- sql(createIceberg, tableName);
-
- Object result =
- scalarSql("CALL %s.system.add_files('%s', '%s')", catalogName, tableName, sourceTableName);
-
- Assert.assertEquals(2L, result);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT * FROM %s ORDER BY id", sourceTableName),
- sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void addDataUnpartitionedExtraCol() {
- createUnpartitionedFileTable("parquet");
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String, foo string) USING iceberg";
-
- sql(createIceberg, tableName);
-
- Object result =
- scalarSql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`')",
- catalogName, tableName, fileTableDir.getAbsolutePath());
-
- Assert.assertEquals(2L, result);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT * FROM %s ORDER BY id", sourceTableName),
- sql("SELECT id, name, dept, subdept FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void addDataUnpartitionedMissingCol() {
- createUnpartitionedFileTable("parquet");
-
- String createIceberg = "CREATE TABLE %s (id Integer, name String, dept String) USING iceberg";
-
- sql(createIceberg, tableName);
-
- Object result =
- scalarSql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`')",
- catalogName, tableName, fileTableDir.getAbsolutePath());
-
- Assert.assertEquals(2L, result);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT id, name, dept FROM %s ORDER BY id", sourceTableName),
- sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void addDataPartitionedMissingCol() {
- createPartitionedFileTable("parquet");
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String) USING iceberg PARTITIONED BY (id)";
-
- sql(createIceberg, tableName);
-
- Object result =
- scalarSql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`')",
- catalogName, tableName, fileTableDir.getAbsolutePath());
-
- Assert.assertEquals(8L, result);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT id, name, dept FROM %s ORDER BY id", sourceTableName),
- sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void addDataPartitioned() {
- createPartitionedFileTable("parquet");
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg PARTITIONED BY (id)";
-
- sql(createIceberg, tableName);
-
- Object result =
- scalarSql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`')",
- catalogName, tableName, fileTableDir.getAbsolutePath());
-
- Assert.assertEquals(8L, result);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT id, name, dept, subdept FROM %s ORDER BY id", sourceTableName),
- sql("SELECT id, name, dept, subdept FROM %s ORDER BY id", tableName));
- }
-
- @Ignore // TODO Classpath issues prevent us from actually writing to a Spark ORC table
- public void addDataPartitionedOrc() {
- createPartitionedFileTable("orc");
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg PARTITIONED BY (id)";
-
- sql(createIceberg, tableName);
-
- Object result =
- scalarSql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`')",
- catalogName, tableName, fileTableDir.getAbsolutePath());
-
- Assert.assertEquals(8L, result);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT id, name, dept, subdept FROM %s ORDER BY id", sourceTableName),
- sql("SELECT id, name, dept, subdept FROM %s ORDER BY id", tableName));
- }
-
- // TODO Adding spark-avro doesn't work in tests
- @Ignore
- public void addDataPartitionedAvro() {
- createPartitionedFileTable("avro");
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg PARTITIONED BY (id)";
-
- sql(createIceberg, tableName);
-
- Object result =
- scalarSql(
- "CALL %s.system.add_files('%s', '`avro`.`%s`')",
- catalogName, tableName, fileTableDir.getAbsolutePath());
-
- Assert.assertEquals(8L, result);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT id, name, dept, subdept FROM %s ORDER BY id", sourceTableName),
- sql("SELECT id, name, dept, subdept FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void addDataPartitionedHive() {
- createPartitionedHiveTable();
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg PARTITIONED BY (id)";
-
- sql(createIceberg, tableName);
-
- Object result =
- scalarSql("CALL %s.system.add_files('%s', '%s')", catalogName, tableName, sourceTableName);
-
- Assert.assertEquals(8L, result);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT id, name, dept, subdept FROM %s ORDER BY id", sourceTableName),
- sql("SELECT id, name, dept, subdept FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void addPartitionToPartitioned() {
- createPartitionedFileTable("parquet");
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg PARTITIONED BY (id)";
-
- sql(createIceberg, tableName);
-
- Object result =
- scalarSql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`', map('id', 1))",
- catalogName, tableName, fileTableDir.getAbsolutePath());
-
- Assert.assertEquals(2L, result);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT id, name, dept, subdept FROM %s WHERE id = 1 ORDER BY id", sourceTableName),
- sql("SELECT id, name, dept, subdept FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void deleteAndAddBackPartitioned() {
- createPartitionedFileTable("parquet");
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg PARTITIONED BY (id)";
-
- sql(createIceberg, tableName);
-
- sql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`', map('id', 1))",
- catalogName, tableName, fileTableDir.getAbsolutePath());
-
- String deleteData = "DELETE FROM %s where id = 1";
- sql(deleteData, tableName);
-
- Object result =
- scalarSql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`', map('id', 1))",
- catalogName, tableName, fileTableDir.getAbsolutePath());
- Assert.assertEquals(2L, result);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT id, name, dept, subdept FROM %s WHERE id = 1 ORDER BY id", sourceTableName),
- sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void addPartitionToPartitionedSnapshotIdInheritanceEnabledInTwoRuns() {
- createPartitionedFileTable("parquet");
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg PARTITIONED BY (id)"
- + "TBLPROPERTIES ('%s'='true')";
-
- sql(createIceberg, tableName, TableProperties.SNAPSHOT_ID_INHERITANCE_ENABLED);
-
- sql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`', map('id', 1))",
- catalogName, tableName, fileTableDir.getAbsolutePath());
-
- sql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`', map('id', 2))",
- catalogName, tableName, fileTableDir.getAbsolutePath());
-
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT id, name, dept, subdept FROM %s WHERE id < 3 ORDER BY id", sourceTableName),
- sql("SELECT id, name, dept, subdept FROM %s ORDER BY id", tableName));
-
- // verify manifest file name has uuid pattern
- String manifestPath = (String) sql("select path from %s.manifests", tableName).get(0)[0];
-
- Pattern uuidPattern = Pattern.compile("[a-f0-9]{8}(?:-[a-f0-9]{4}){4}[a-f0-9]{8}");
-
- Matcher matcher = uuidPattern.matcher(manifestPath);
- Assert.assertTrue("verify manifest path has uuid", matcher.find());
- }
-
- @Test
- public void addDataPartitionedByDateToPartitioned() {
- createDatePartitionedFileTable("parquet");
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, date Date) USING iceberg PARTITIONED BY (date)";
-
- sql(createIceberg, tableName);
-
- Object result =
- scalarSql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`', map('date', '2021-01-01'))",
- catalogName, tableName, fileTableDir.getAbsolutePath());
-
- Assert.assertEquals(2L, result);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT id, name, date FROM %s WHERE date = '2021-01-01' ORDER BY id", sourceTableName),
- sql("SELECT id, name, date FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void addDataPartitionedVerifyPartitionTypeInferredCorrectly() {
- createTableWithTwoPartitions("parquet");
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, date Date, dept String) USING iceberg PARTITIONED BY (date, dept)";
-
- sql(createIceberg, tableName);
-
- sql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`', map('date', '2021-01-01'))",
- catalogName, tableName, fileTableDir.getAbsolutePath());
-
- String sqlFormat =
- "SELECT id, name, dept, date FROM %s WHERE date = '2021-01-01' and dept= '01' ORDER BY id";
- assertEquals(
- "Iceberg table contains correct data",
- sql(sqlFormat, sourceTableName),
- sql(sqlFormat, tableName));
- }
-
- @Test
- public void addFilteredPartitionsToPartitioned() {
- createCompositePartitionedTable("parquet");
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg "
- + "PARTITIONED BY (id, dept)";
-
- sql(createIceberg, tableName);
-
- Object result =
- scalarSql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`', map('id', 1))",
- catalogName, tableName, fileTableDir.getAbsolutePath());
-
- Assert.assertEquals(2L, result);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT id, name, dept, subdept FROM %s WHERE id = 1 ORDER BY id", sourceTableName),
- sql("SELECT id, name, dept, subdept FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void addFilteredPartitionsToPartitioned2() {
- createCompositePartitionedTable("parquet");
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg "
- + "PARTITIONED BY (id, dept)";
-
- sql(createIceberg, tableName);
-
- Object result =
- scalarSql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`', map('dept', 'hr'))",
- catalogName, tableName, fileTableDir.getAbsolutePath());
-
- Assert.assertEquals(6L, result);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql(
- "SELECT id, name, dept, subdept FROM %s WHERE dept = 'hr' ORDER BY id",
- sourceTableName),
- sql("SELECT id, name, dept, subdept FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void addFilteredPartitionsToPartitionedWithNullValueFilteringOnId() {
- createCompositePartitionedTableWithNullValueInPartitionColumn("parquet");
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg "
- + "PARTITIONED BY (id, dept)";
-
- sql(createIceberg, tableName);
-
- Object result =
- scalarSql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`', map('id', 1))",
- catalogName, tableName, fileTableDir.getAbsolutePath());
-
- Assert.assertEquals(2L, result);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT id, name, dept, subdept FROM %s WHERE id = 1 ORDER BY id", sourceTableName),
- sql("SELECT id, name, dept, subdept FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void addFilteredPartitionsToPartitionedWithNullValueFilteringOnDept() {
- createCompositePartitionedTableWithNullValueInPartitionColumn("parquet");
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg "
- + "PARTITIONED BY (id, dept)";
-
- sql(createIceberg, tableName);
-
- Object result =
- scalarSql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`', map('dept', 'hr'))",
- catalogName, tableName, fileTableDir.getAbsolutePath());
-
- Assert.assertEquals(6L, result);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql(
- "SELECT id, name, dept, subdept FROM %s WHERE dept = 'hr' ORDER BY id",
- sourceTableName),
- sql("SELECT id, name, dept, subdept FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void addWeirdCaseHiveTable() {
- createWeirdCaseTable();
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, `naMe` String, dept String, subdept String) USING iceberg "
- + "PARTITIONED BY (`naMe`)";
-
- sql(createIceberg, tableName);
-
- Object result =
- scalarSql(
- "CALL %s.system.add_files('%s', '%s', map('naMe', 'John Doe'))",
- catalogName, tableName, sourceTableName);
-
- Assert.assertEquals(2L, result);
-
- /*
- While we would like to use
- SELECT id, `naMe`, dept, subdept FROM %s WHERE `naMe` = 'John Doe' ORDER BY id
- Spark does not actually handle this pushdown correctly for hive based tables and it returns 0 records
- */
- List expected =
- sql("SELECT id, `naMe`, dept, subdept from %s ORDER BY id", sourceTableName).stream()
- .filter(r -> r[1].equals("John Doe"))
- .collect(Collectors.toList());
-
- // TODO when this assert breaks Spark fixed the pushdown issue
- Assert.assertEquals(
- "If this assert breaks it means that Spark has fixed the pushdown issue",
- 0,
- sql(
- "SELECT id, `naMe`, dept, subdept from %s WHERE `naMe` = 'John Doe' ORDER BY id",
- sourceTableName)
- .size());
-
- // Pushdown works for iceberg
- Assert.assertEquals(
- "We should be able to pushdown mixed case partition keys",
- 2,
- sql(
- "SELECT id, `naMe`, dept, subdept FROM %s WHERE `naMe` = 'John Doe' ORDER BY id",
- tableName)
- .size());
-
- assertEquals(
- "Iceberg table contains correct data",
- expected,
- sql("SELECT id, `naMe`, dept, subdept FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void addPartitionToPartitionedHive() {
- createPartitionedHiveTable();
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg PARTITIONED BY (id)";
-
- sql(createIceberg, tableName);
-
- Object result =
- scalarSql(
- "CALL %s.system.add_files('%s', '%s', map('id', 1))",
- catalogName, tableName, sourceTableName);
-
- Assert.assertEquals(2L, result);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT id, name, dept, subdept FROM %s WHERE id = 1 ORDER BY id", sourceTableName),
- sql("SELECT id, name, dept, subdept FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void invalidDataImport() {
- createPartitionedFileTable("parquet");
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg";
-
- sql(createIceberg, tableName);
-
- AssertHelpers.assertThrows(
- "Should forbid adding of partitioned data to unpartitioned table",
- IllegalArgumentException.class,
- "Cannot use partition filter with an unpartitioned table",
- () ->
- scalarSql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`', map('id', 1))",
- catalogName, tableName, fileTableDir.getAbsolutePath()));
-
- AssertHelpers.assertThrows(
- "Should forbid adding of partitioned data to unpartitioned table",
- IllegalArgumentException.class,
- "Cannot add partitioned files to an unpartitioned table",
- () ->
- scalarSql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`')",
- catalogName, tableName, fileTableDir.getAbsolutePath()));
- }
-
- @Test
- public void invalidDataImportPartitioned() {
- createUnpartitionedFileTable("parquet");
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg PARTITIONED BY (id)";
-
- sql(createIceberg, tableName);
-
- AssertHelpers.assertThrows(
- "Should forbid adding with a mismatching partition spec",
- IllegalArgumentException.class,
- "is greater than the number of partitioned columns",
- () ->
- scalarSql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`', map('x', '1', 'y', '2'))",
- catalogName, tableName, fileTableDir.getAbsolutePath()));
-
- AssertHelpers.assertThrows(
- "Should forbid adding with partition spec with incorrect columns",
- IllegalArgumentException.class,
- "specified partition filter refers to columns that are not partitioned",
- () ->
- scalarSql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`', map('dept', '2'))",
- catalogName, tableName, fileTableDir.getAbsolutePath()));
- }
-
- @Test
- public void addTwice() {
- createPartitionedHiveTable();
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg PARTITIONED BY (id)";
-
- sql(createIceberg, tableName);
-
- Object result1 =
- scalarSql(
- "CALL %s.system.add_files("
- + "table => '%s', "
- + "source_table => '%s', "
- + "partition_filter => map('id', 1))",
- catalogName, tableName, sourceTableName);
- Assert.assertEquals(2L, result1);
-
- Object result2 =
- scalarSql(
- "CALL %s.system.add_files("
- + "table => '%s', "
- + "source_table => '%s', "
- + "partition_filter => map('id', 2))",
- catalogName, tableName, sourceTableName);
- Assert.assertEquals(2L, result2);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT id, name, dept, subdept FROM %s WHERE id = 1 ORDER BY id", sourceTableName),
- sql("SELECT id, name, dept, subdept FROM %s WHERE id = 1 ORDER BY id", tableName));
- assertEquals(
- "Iceberg table contains correct data",
- sql("SELECT id, name, dept, subdept FROM %s WHERE id = 2 ORDER BY id", sourceTableName),
- sql("SELECT id, name, dept, subdept FROM %s WHERE id = 2 ORDER BY id", tableName));
- }
-
- @Test
- public void duplicateDataPartitioned() {
- createPartitionedHiveTable();
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg PARTITIONED BY (id)";
-
- sql(createIceberg, tableName);
-
- scalarSql(
- "CALL %s.system.add_files("
- + "table => '%s', "
- + "source_table => '%s', "
- + "partition_filter => map('id', 1))",
- catalogName, tableName, sourceTableName);
-
- AssertHelpers.assertThrows(
- "Should not allow adding duplicate files",
- IllegalStateException.class,
- "Cannot complete import because data files to be imported already"
- + " exist within the target table",
- () ->
- scalarSql(
- "CALL %s.system.add_files("
- + "table => '%s', "
- + "source_table => '%s', "
- + "partition_filter => map('id', 1))",
- catalogName, tableName, sourceTableName));
- }
-
- @Test
- public void duplicateDataPartitionedAllowed() {
- createPartitionedHiveTable();
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg PARTITIONED BY (id)";
-
- sql(createIceberg, tableName);
-
- Object result1 =
- scalarSql(
- "CALL %s.system.add_files("
- + "table => '%s', "
- + "source_table => '%s', "
- + "partition_filter => map('id', 1))",
- catalogName, tableName, sourceTableName);
-
- Assert.assertEquals(2L, result1);
-
- Object result2 =
- scalarSql(
- "CALL %s.system.add_files("
- + "table => '%s', "
- + "source_table => '%s', "
- + "partition_filter => map('id', 1),"
- + "check_duplicate_files => false)",
- catalogName, tableName, sourceTableName);
-
- Assert.assertEquals(2L, result2);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql(
- "SELECT id, name, dept, subdept FROM %s WHERE id = 1 UNION ALL "
- + "SELECT id, name, dept, subdept FROM %s WHERE id = 1",
- sourceTableName, sourceTableName),
- sql("SELECT id, name, dept, subdept FROM %s", tableName, tableName));
- }
-
- @Test
- public void duplicateDataUnpartitioned() {
- createUnpartitionedHiveTable();
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg";
-
- sql(createIceberg, tableName);
-
- scalarSql("CALL %s.system.add_files('%s', '%s')", catalogName, tableName, sourceTableName);
-
- AssertHelpers.assertThrows(
- "Should not allow adding duplicate files",
- IllegalStateException.class,
- "Cannot complete import because data files to be imported already"
- + " exist within the target table",
- () ->
- scalarSql(
- "CALL %s.system.add_files('%s', '%s')", catalogName, tableName, sourceTableName));
- }
-
- @Test
- public void duplicateDataUnpartitionedAllowed() {
- createUnpartitionedHiveTable();
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg";
-
- sql(createIceberg, tableName);
-
- Object result1 =
- scalarSql("CALL %s.system.add_files('%s', '%s')", catalogName, tableName, sourceTableName);
- Assert.assertEquals(2L, result1);
-
- Object result2 =
- scalarSql(
- "CALL %s.system.add_files("
- + "table => '%s', "
- + "source_table => '%s',"
- + "check_duplicate_files => false)",
- catalogName, tableName, sourceTableName);
- Assert.assertEquals(2L, result2);
-
- assertEquals(
- "Iceberg table contains correct data",
- sql(
- "SELECT * FROM (SELECT * FROM %s UNION ALL " + "SELECT * from %s) ORDER BY id",
- sourceTableName, sourceTableName),
- sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testEmptyImportDoesNotThrow() {
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg";
- sql(createIceberg, tableName);
-
- // Empty path based import
- Object pathResult =
- scalarSql(
- "CALL %s.system.add_files('%s', '`parquet`.`%s`')",
- catalogName, tableName, fileTableDir.getAbsolutePath());
- Assert.assertEquals(0L, pathResult);
- assertEquals(
- "Iceberg table contains no added data when importing from an empty path",
- emptyQueryResult,
- sql("SELECT * FROM %s ORDER BY id", tableName));
-
- // Empty table based import
- String createHive =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) STORED AS parquet";
- sql(createHive, sourceTableName);
-
- Object tableResult =
- scalarSql("CALL %s.system.add_files('%s', '%s')", catalogName, tableName, sourceTableName);
- Assert.assertEquals(0L, tableResult);
- assertEquals(
- "Iceberg table contains no added data when importing from an empty table",
- emptyQueryResult,
- sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testPartitionedImportFromEmptyPartitionDoesNotThrow() {
- createPartitionedHiveTable();
-
- final int emptyPartitionId = 999;
- // Add an empty partition to the hive table
- sql(
- "ALTER TABLE %s ADD PARTITION (id = '%d') LOCATION '%d'",
- sourceTableName, emptyPartitionId, emptyPartitionId);
-
- String createIceberg =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING iceberg PARTITIONED BY (id)";
-
- sql(createIceberg, tableName);
-
- Object tableResult =
- scalarSql(
- "CALL %s.system.add_files("
- + "table => '%s', "
- + "source_table => '%s', "
- + "partition_filter => map('id', %d))",
- catalogName, tableName, sourceTableName, emptyPartitionId);
-
- Assert.assertEquals(0L, tableResult);
- assertEquals(
- "Iceberg table contains no added data when importing from an empty table",
- emptyQueryResult,
- sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- private static final List emptyQueryResult = Lists.newArrayList();
-
- private static final StructField[] struct = {
- new StructField("id", DataTypes.IntegerType, true, Metadata.empty()),
- new StructField("name", DataTypes.StringType, true, Metadata.empty()),
- new StructField("dept", DataTypes.StringType, true, Metadata.empty()),
- new StructField("subdept", DataTypes.StringType, true, Metadata.empty())
- };
-
- private static final Dataset unpartitionedDF =
- spark
- .createDataFrame(
- ImmutableList.of(
- RowFactory.create(1, "John Doe", "hr", "communications"),
- RowFactory.create(2, "Jane Doe", "hr", "salary"),
- RowFactory.create(3, "Matt Doe", "hr", "communications"),
- RowFactory.create(4, "Will Doe", "facilities", "all")),
- new StructType(struct))
- .repartition(1);
-
- private static final Dataset singleNullRecordDF =
- spark
- .createDataFrame(
- ImmutableList.of(RowFactory.create(null, null, null, null)), new StructType(struct))
- .repartition(1);
-
- private static final Dataset partitionedDF =
- unpartitionedDF.select("name", "dept", "subdept", "id");
-
- private static final Dataset compositePartitionedDF =
- unpartitionedDF.select("name", "subdept", "id", "dept");
-
- private static final Dataset compositePartitionedNullRecordDF =
- singleNullRecordDF.select("name", "subdept", "id", "dept");
-
- private static final Dataset weirdColumnNamesDF =
- unpartitionedDF.select(
- unpartitionedDF.col("id"),
- unpartitionedDF.col("subdept"),
- unpartitionedDF.col("dept"),
- unpartitionedDF.col("name").as("naMe"));
-
- private static final StructField[] dateStruct = {
- new StructField("id", DataTypes.IntegerType, true, Metadata.empty()),
- new StructField("name", DataTypes.StringType, true, Metadata.empty()),
- new StructField("ts", DataTypes.DateType, true, Metadata.empty()),
- new StructField("dept", DataTypes.StringType, true, Metadata.empty())
- };
-
- private static java.sql.Date toDate(String value) {
- return new java.sql.Date(DateTime.parse(value).getMillis());
- }
-
- private static final Dataset dateDF =
- spark
- .createDataFrame(
- ImmutableList.of(
- RowFactory.create(1, "John Doe", toDate("2021-01-01"), "01"),
- RowFactory.create(2, "Jane Doe", toDate("2021-01-01"), "01"),
- RowFactory.create(3, "Matt Doe", toDate("2021-01-02"), "02"),
- RowFactory.create(4, "Will Doe", toDate("2021-01-02"), "02")),
- new StructType(dateStruct))
- .repartition(2);
-
- private void createUnpartitionedFileTable(String format) {
- String createParquet =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING %s LOCATION '%s'";
-
- sql(createParquet, sourceTableName, format, fileTableDir.getAbsolutePath());
- unpartitionedDF.write().insertInto(sourceTableName);
- unpartitionedDF.write().insertInto(sourceTableName);
- }
-
- private void createPartitionedFileTable(String format) {
- String createParquet =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING %s PARTITIONED BY (id) "
- + "LOCATION '%s'";
-
- sql(createParquet, sourceTableName, format, fileTableDir.getAbsolutePath());
-
- partitionedDF.write().insertInto(sourceTableName);
- partitionedDF.write().insertInto(sourceTableName);
- }
-
- private void createCompositePartitionedTable(String format) {
- String createParquet =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING %s "
- + "PARTITIONED BY (id, dept) LOCATION '%s'";
- sql(createParquet, sourceTableName, format, fileTableDir.getAbsolutePath());
-
- compositePartitionedDF.write().insertInto(sourceTableName);
- compositePartitionedDF.write().insertInto(sourceTableName);
- }
-
- private void createCompositePartitionedTableWithNullValueInPartitionColumn(String format) {
- String createParquet =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) USING %s "
- + "PARTITIONED BY (id, dept) LOCATION '%s'";
- sql(createParquet, sourceTableName, format, fileTableDir.getAbsolutePath());
-
- Dataset unionedDF =
- compositePartitionedDF
- .unionAll(compositePartitionedNullRecordDF)
- .select("name", "subdept", "id", "dept")
- .repartition(1);
-
- unionedDF.write().insertInto(sourceTableName);
- unionedDF.write().insertInto(sourceTableName);
- }
-
- private void createWeirdCaseTable() {
- String createParquet =
- "CREATE TABLE %s (id Integer, subdept String, dept String) "
- + "PARTITIONED BY (`naMe` String) STORED AS parquet";
-
- sql(createParquet, sourceTableName);
-
- weirdColumnNamesDF.write().insertInto(sourceTableName);
- weirdColumnNamesDF.write().insertInto(sourceTableName);
- }
-
- private void createUnpartitionedHiveTable() {
- String createHive =
- "CREATE TABLE %s (id Integer, name String, dept String, subdept String) STORED AS parquet";
-
- sql(createHive, sourceTableName);
-
- unpartitionedDF.write().insertInto(sourceTableName);
- unpartitionedDF.write().insertInto(sourceTableName);
- }
-
- private void createPartitionedHiveTable() {
- String createHive =
- "CREATE TABLE %s (name String, dept String, subdept String) "
- + "PARTITIONED BY (id Integer) STORED AS parquet";
-
- sql(createHive, sourceTableName);
-
- partitionedDF.write().insertInto(sourceTableName);
- partitionedDF.write().insertInto(sourceTableName);
- }
-
- private void createDatePartitionedFileTable(String format) {
- String createParquet =
- "CREATE TABLE %s (id Integer, name String, date Date) USING %s "
- + "PARTITIONED BY (date) LOCATION '%s'";
-
- sql(createParquet, sourceTableName, format, fileTableDir.getAbsolutePath());
-
- dateDF.select("id", "name", "ts").write().insertInto(sourceTableName);
- }
-
- private void createTableWithTwoPartitions(String format) {
- String createParquet =
- "CREATE TABLE %s (id Integer, name String, date Date, dept String) USING %s "
- + "PARTITIONED BY (date, dept) LOCATION '%s'";
-
- sql(createParquet, sourceTableName, format, fileTableDir.getAbsolutePath());
-
- dateDF.write().insertInto(sourceTableName);
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAlterTablePartitionFields.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAlterTablePartitionFields.java
deleted file mode 100644
index 042e87c729a5..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAlterTablePartitionFields.java
+++ /dev/null
@@ -1,585 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import org.apache.iceberg.PartitionSpec;
-import org.apache.iceberg.Table;
-import org.apache.iceberg.TableProperties;
-import org.apache.iceberg.TestHelpers;
-import org.apache.iceberg.spark.SparkCatalogConfig;
-import org.apache.iceberg.spark.source.SparkTable;
-import org.apache.spark.sql.connector.catalog.CatalogManager;
-import org.apache.spark.sql.connector.catalog.Identifier;
-import org.apache.spark.sql.connector.catalog.TableCatalog;
-import org.assertj.core.api.Assertions;
-import org.junit.After;
-import org.junit.Assert;
-import org.junit.Test;
-import org.junit.runners.Parameterized;
-
-public class TestAlterTablePartitionFields extends SparkExtensionsTestBase {
-
- @Parameterized.Parameters(name = "catalogConfig = {0}, formatVersion = {1}")
- public static Object[][] parameters() {
- return new Object[][] {
- {SparkCatalogConfig.HIVE, 1},
- {SparkCatalogConfig.SPARK, 2}
- };
- }
-
- private final int formatVersion;
-
- public TestAlterTablePartitionFields(SparkCatalogConfig catalogConfig, int formatVersion) {
- super(catalogConfig.catalogName(), catalogConfig.implementation(), catalogConfig.properties());
- this.formatVersion = formatVersion;
- }
-
- @After
- public void removeTable() {
- sql("DROP TABLE IF EXISTS %s", tableName);
- }
-
- @Test
- public void testAddIdentityPartition() {
- createTable("id bigint NOT NULL, category string, ts timestamp, data string");
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assert.assertTrue("Table should start unpartitioned", table.spec().isUnpartitioned());
-
- sql("ALTER TABLE %s ADD PARTITION FIELD category", tableName);
-
- table.refresh();
-
- PartitionSpec expected =
- PartitionSpec.builderFor(table.schema()).withSpecId(1).identity("category").build();
-
- Assert.assertEquals("Should have new spec field", expected, table.spec());
- }
-
- @Test
- public void testAddBucketPartition() {
- createTable("id bigint NOT NULL, category string, ts timestamp, data string");
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assert.assertTrue("Table should start unpartitioned", table.spec().isUnpartitioned());
-
- sql("ALTER TABLE %s ADD PARTITION FIELD bucket(16, id)", tableName);
-
- table.refresh();
-
- PartitionSpec expected =
- PartitionSpec.builderFor(table.schema())
- .withSpecId(1)
- .bucket("id", 16, "id_bucket_16")
- .build();
-
- Assert.assertEquals("Should have new spec field", expected, table.spec());
- }
-
- @Test
- public void testAddTruncatePartition() {
- createTable("id bigint NOT NULL, category string, ts timestamp, data string");
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assert.assertTrue("Table should start unpartitioned", table.spec().isUnpartitioned());
-
- sql("ALTER TABLE %s ADD PARTITION FIELD truncate(data, 4)", tableName);
-
- table.refresh();
-
- PartitionSpec expected =
- PartitionSpec.builderFor(table.schema())
- .withSpecId(1)
- .truncate("data", 4, "data_trunc_4")
- .build();
-
- Assert.assertEquals("Should have new spec field", expected, table.spec());
- }
-
- @Test
- public void testAddYearsPartition() {
- createTable("id bigint NOT NULL, category string, ts timestamp, data string");
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assert.assertTrue("Table should start unpartitioned", table.spec().isUnpartitioned());
-
- sql("ALTER TABLE %s ADD PARTITION FIELD years(ts)", tableName);
-
- table.refresh();
-
- PartitionSpec expected =
- PartitionSpec.builderFor(table.schema()).withSpecId(1).year("ts").build();
-
- Assert.assertEquals("Should have new spec field", expected, table.spec());
- }
-
- @Test
- public void testAddMonthsPartition() {
- createTable("id bigint NOT NULL, category string, ts timestamp, data string");
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assert.assertTrue("Table should start unpartitioned", table.spec().isUnpartitioned());
-
- sql("ALTER TABLE %s ADD PARTITION FIELD months(ts)", tableName);
-
- table.refresh();
-
- PartitionSpec expected =
- PartitionSpec.builderFor(table.schema()).withSpecId(1).month("ts").build();
-
- Assert.assertEquals("Should have new spec field", expected, table.spec());
- }
-
- @Test
- public void testAddDaysPartition() {
- createTable("id bigint NOT NULL, category string, ts timestamp, data string");
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assert.assertTrue("Table should start unpartitioned", table.spec().isUnpartitioned());
-
- sql("ALTER TABLE %s ADD PARTITION FIELD days(ts)", tableName);
-
- table.refresh();
-
- PartitionSpec expected =
- PartitionSpec.builderFor(table.schema()).withSpecId(1).day("ts").build();
-
- Assert.assertEquals("Should have new spec field", expected, table.spec());
- }
-
- @Test
- public void testAddHoursPartition() {
- createTable("id bigint NOT NULL, category string, ts timestamp, data string");
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assert.assertTrue("Table should start unpartitioned", table.spec().isUnpartitioned());
-
- sql("ALTER TABLE %s ADD PARTITION FIELD hours(ts)", tableName);
-
- table.refresh();
-
- PartitionSpec expected =
- PartitionSpec.builderFor(table.schema()).withSpecId(1).hour("ts").build();
-
- Assert.assertEquals("Should have new spec field", expected, table.spec());
- }
-
- @Test
- public void testAddYearPartition() {
- createTable("id bigint NOT NULL, category string, ts timestamp, data string");
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assertions.assertThat(table.spec().isUnpartitioned())
- .as("Table should start unpartitioned")
- .isTrue();
-
- sql("ALTER TABLE %s ADD PARTITION FIELD year(ts)", tableName);
-
- table.refresh();
-
- PartitionSpec expected =
- PartitionSpec.builderFor(table.schema()).withSpecId(1).year("ts").build();
-
- Assertions.assertThat(table.spec()).as("Should have new spec field").isEqualTo(expected);
- }
-
- @Test
- public void testAddMonthPartition() {
- createTable("id bigint NOT NULL, category string, ts timestamp, data string");
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assertions.assertThat(table.spec().isUnpartitioned())
- .as("Table should start unpartitioned")
- .isTrue();
-
- sql("ALTER TABLE %s ADD PARTITION FIELD month(ts)", tableName);
-
- table.refresh();
-
- PartitionSpec expected =
- PartitionSpec.builderFor(table.schema()).withSpecId(1).month("ts").build();
-
- Assertions.assertThat(table.spec()).as("Should have new spec field").isEqualTo(expected);
- }
-
- @Test
- public void testAddDayPartition() {
- createTable("id bigint NOT NULL, category string, ts timestamp, data string");
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assertions.assertThat(table.spec().isUnpartitioned())
- .as("Table should start unpartitioned")
- .isTrue();
-
- sql("ALTER TABLE %s ADD PARTITION FIELD day(ts)", tableName);
-
- table.refresh();
-
- PartitionSpec expected =
- PartitionSpec.builderFor(table.schema()).withSpecId(1).day("ts").build();
-
- Assertions.assertThat(table.spec()).as("Should have new spec field").isEqualTo(expected);
- }
-
- @Test
- public void testAddHourPartition() {
- createTable("id bigint NOT NULL, category string, ts timestamp, data string");
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assertions.assertThat(table.spec().isUnpartitioned())
- .as("Table should start unpartitioned")
- .isTrue();
-
- sql("ALTER TABLE %s ADD PARTITION FIELD hour(ts)", tableName);
-
- table.refresh();
-
- PartitionSpec expected =
- PartitionSpec.builderFor(table.schema()).withSpecId(1).hour("ts").build();
-
- Assertions.assertThat(table.spec()).as("Should have new spec field").isEqualTo(expected);
- }
-
- @Test
- public void testAddNamedPartition() {
- createTable("id bigint NOT NULL, category string, ts timestamp, data string");
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assert.assertTrue("Table should start unpartitioned", table.spec().isUnpartitioned());
-
- sql("ALTER TABLE %s ADD PARTITION FIELD bucket(16, id) AS shard", tableName);
-
- table.refresh();
-
- PartitionSpec expected =
- PartitionSpec.builderFor(table.schema()).withSpecId(1).bucket("id", 16, "shard").build();
-
- Assert.assertEquals("Should have new spec field", expected, table.spec());
- }
-
- @Test
- public void testDropIdentityPartition() {
- createTable("id bigint NOT NULL, category string, data string", "category");
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assert.assertEquals(
- "Table should start with 1 partition field", 1, table.spec().fields().size());
-
- sql("ALTER TABLE %s DROP PARTITION FIELD category", tableName);
-
- table.refresh();
-
- if (formatVersion == 1) {
- PartitionSpec expected =
- PartitionSpec.builderFor(table.schema())
- .withSpecId(1)
- .alwaysNull("category", "category")
- .build();
- Assert.assertEquals("Should have new spec field", expected, table.spec());
- } else {
- Assert.assertTrue("New spec must be unpartitioned", table.spec().isUnpartitioned());
- }
- }
-
- @Test
- public void testDropDaysPartition() {
- createTable("id bigint NOT NULL, ts timestamp, data string", "days(ts)");
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assert.assertEquals(
- "Table should start with 1 partition field", 1, table.spec().fields().size());
-
- sql("ALTER TABLE %s DROP PARTITION FIELD days(ts)", tableName);
-
- table.refresh();
-
- if (formatVersion == 1) {
- PartitionSpec expected =
- PartitionSpec.builderFor(table.schema()).withSpecId(1).alwaysNull("ts", "ts_day").build();
- Assert.assertEquals("Should have new spec field", expected, table.spec());
- } else {
- Assert.assertTrue("New spec must be unpartitioned", table.spec().isUnpartitioned());
- }
- }
-
- @Test
- public void testDropBucketPartition() {
- createTable("id bigint NOT NULL, data string", "bucket(16, id)");
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assert.assertEquals(
- "Table should start with 1 partition field", 1, table.spec().fields().size());
-
- sql("ALTER TABLE %s DROP PARTITION FIELD bucket(16, id)", tableName);
-
- table.refresh();
-
- if (formatVersion == 1) {
- PartitionSpec expected =
- PartitionSpec.builderFor(table.schema())
- .withSpecId(1)
- .alwaysNull("id", "id_bucket")
- .build();
- Assert.assertEquals("Should have new spec field", expected, table.spec());
- } else {
- Assert.assertTrue("New spec must be unpartitioned", table.spec().isUnpartitioned());
- }
- }
-
- @Test
- public void testDropPartitionByName() {
- createTable("id bigint NOT NULL, category string, ts timestamp, data string");
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assert.assertTrue("Table should start unpartitioned", table.spec().isUnpartitioned());
-
- sql("ALTER TABLE %s ADD PARTITION FIELD bucket(16, id) AS shard", tableName);
-
- table.refresh();
-
- Assert.assertEquals("Table should have 1 partition field", 1, table.spec().fields().size());
-
- // Should be recognized as iceberg command even with extra white spaces
- sql("ALTER TABLE %s DROP PARTITION \n FIELD shard", tableName);
-
- table.refresh();
-
- if (formatVersion == 1) {
- PartitionSpec expected =
- PartitionSpec.builderFor(table.schema()).withSpecId(2).alwaysNull("id", "shard").build();
- Assert.assertEquals("Should have new spec field", expected, table.spec());
- } else {
- Assert.assertTrue("New spec must be unpartitioned", table.spec().isUnpartitioned());
- }
- }
-
- @Test
- public void testReplacePartition() {
- createTable("id bigint NOT NULL, category string, ts timestamp, data string");
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertTrue("Table should start unpartitioned", table.spec().isUnpartitioned());
-
- sql("ALTER TABLE %s ADD PARTITION FIELD days(ts)", tableName);
- table.refresh();
- PartitionSpec expected =
- PartitionSpec.builderFor(table.schema()).withSpecId(1).day("ts").build();
- Assert.assertEquals("Should have new spec field", expected, table.spec());
-
- sql("ALTER TABLE %s REPLACE PARTITION FIELD days(ts) WITH hours(ts)", tableName);
- table.refresh();
- if (formatVersion == 1) {
- expected =
- PartitionSpec.builderFor(table.schema())
- .withSpecId(2)
- .alwaysNull("ts", "ts_day")
- .hour("ts")
- .build();
- } else {
- expected =
- TestHelpers.newExpectedSpecBuilder()
- .withSchema(table.schema())
- .withSpecId(2)
- .addField("hour", 3, 1001, "ts_hour")
- .build();
- }
- Assert.assertEquals(
- "Should changed from daily to hourly partitioned field", expected, table.spec());
- }
-
- @Test
- public void testReplacePartitionAndRename() {
- createTable("id bigint NOT NULL, category string, ts timestamp, data string");
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertTrue("Table should start unpartitioned", table.spec().isUnpartitioned());
-
- sql("ALTER TABLE %s ADD PARTITION FIELD days(ts)", tableName);
- table.refresh();
- PartitionSpec expected =
- PartitionSpec.builderFor(table.schema()).withSpecId(1).day("ts").build();
- Assert.assertEquals("Should have new spec field", expected, table.spec());
-
- sql("ALTER TABLE %s REPLACE PARTITION FIELD days(ts) WITH hours(ts) AS hour_col", tableName);
- table.refresh();
- if (formatVersion == 1) {
- expected =
- PartitionSpec.builderFor(table.schema())
- .withSpecId(2)
- .alwaysNull("ts", "ts_day")
- .hour("ts", "hour_col")
- .build();
- } else {
- expected =
- TestHelpers.newExpectedSpecBuilder()
- .withSchema(table.schema())
- .withSpecId(2)
- .addField("hour", 3, 1001, "hour_col")
- .build();
- }
- Assert.assertEquals(
- "Should changed from daily to hourly partitioned field", expected, table.spec());
- }
-
- @Test
- public void testReplaceNamedPartition() {
- createTable("id bigint NOT NULL, category string, ts timestamp, data string");
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertTrue("Table should start unpartitioned", table.spec().isUnpartitioned());
-
- sql("ALTER TABLE %s ADD PARTITION FIELD days(ts) AS day_col", tableName);
- table.refresh();
- PartitionSpec expected =
- PartitionSpec.builderFor(table.schema()).withSpecId(1).day("ts", "day_col").build();
- Assert.assertEquals("Should have new spec field", expected, table.spec());
-
- sql("ALTER TABLE %s REPLACE PARTITION FIELD day_col WITH hours(ts)", tableName);
- table.refresh();
- if (formatVersion == 1) {
- expected =
- PartitionSpec.builderFor(table.schema())
- .withSpecId(2)
- .alwaysNull("ts", "day_col")
- .hour("ts")
- .build();
- } else {
- expected =
- TestHelpers.newExpectedSpecBuilder()
- .withSchema(table.schema())
- .withSpecId(2)
- .addField("hour", 3, 1001, "ts_hour")
- .build();
- }
- Assert.assertEquals(
- "Should changed from daily to hourly partitioned field", expected, table.spec());
- }
-
- @Test
- public void testReplaceNamedPartitionAndRenameDifferently() {
- createTable("id bigint NOT NULL, category string, ts timestamp, data string");
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertTrue("Table should start unpartitioned", table.spec().isUnpartitioned());
-
- sql("ALTER TABLE %s ADD PARTITION FIELD days(ts) AS day_col", tableName);
- table.refresh();
- PartitionSpec expected =
- PartitionSpec.builderFor(table.schema()).withSpecId(1).day("ts", "day_col").build();
- Assert.assertEquals("Should have new spec field", expected, table.spec());
-
- sql("ALTER TABLE %s REPLACE PARTITION FIELD day_col WITH hours(ts) AS hour_col", tableName);
- table.refresh();
- if (formatVersion == 1) {
- expected =
- PartitionSpec.builderFor(table.schema())
- .withSpecId(2)
- .alwaysNull("ts", "day_col")
- .hour("ts", "hour_col")
- .build();
- } else {
- expected =
- TestHelpers.newExpectedSpecBuilder()
- .withSchema(table.schema())
- .withSpecId(2)
- .addField("hour", 3, 1001, "hour_col")
- .build();
- }
- Assert.assertEquals(
- "Should changed from daily to hourly partitioned field", expected, table.spec());
- }
-
- @Test
- public void testSparkTableAddDropPartitions() throws Exception {
- createTable("id bigint NOT NULL, ts timestamp, data string");
- Assert.assertEquals(
- "spark table partition should be empty", 0, sparkTable().partitioning().length);
-
- sql("ALTER TABLE %s ADD PARTITION FIELD bucket(16, id) AS shard", tableName);
- assertPartitioningEquals(sparkTable(), 1, "bucket(16, id)");
-
- sql("ALTER TABLE %s ADD PARTITION FIELD truncate(data, 4)", tableName);
- assertPartitioningEquals(sparkTable(), 2, "truncate(data, 4)");
-
- sql("ALTER TABLE %s ADD PARTITION FIELD years(ts)", tableName);
- assertPartitioningEquals(sparkTable(), 3, "years(ts)");
-
- sql("ALTER TABLE %s DROP PARTITION FIELD years(ts)", tableName);
- assertPartitioningEquals(sparkTable(), 2, "truncate(data, 4)");
-
- sql("ALTER TABLE %s DROP PARTITION FIELD truncate(data, 4)", tableName);
- assertPartitioningEquals(sparkTable(), 1, "bucket(16, id)");
-
- sql("ALTER TABLE %s DROP PARTITION FIELD shard", tableName);
- sql("DESCRIBE %s", tableName);
- Assert.assertEquals(
- "spark table partition should be empty", 0, sparkTable().partitioning().length);
- }
-
- @Test
- public void testDropColumnOfOldPartitionFieldV1() {
- // default table created in v1 format
- sql(
- "CREATE TABLE %s (id bigint NOT NULL, ts timestamp, day_of_ts date) USING iceberg PARTITIONED BY (day_of_ts) TBLPROPERTIES('format-version' = '1')",
- tableName);
-
- sql("ALTER TABLE %s REPLACE PARTITION FIELD day_of_ts WITH days(ts)", tableName);
-
- sql("ALTER TABLE %s DROP COLUMN day_of_ts", tableName);
- }
-
- @Test
- public void testDropColumnOfOldPartitionFieldV2() {
- sql(
- "CREATE TABLE %s (id bigint NOT NULL, ts timestamp, day_of_ts date) USING iceberg PARTITIONED BY (day_of_ts) TBLPROPERTIES('format-version' = '2')",
- tableName);
-
- sql("ALTER TABLE %s REPLACE PARTITION FIELD day_of_ts WITH days(ts)", tableName);
-
- sql("ALTER TABLE %s DROP COLUMN day_of_ts", tableName);
- }
-
- private void assertPartitioningEquals(SparkTable table, int len, String transform) {
- Assert.assertEquals("spark table partition should be " + len, len, table.partitioning().length);
- Assert.assertEquals(
- "latest spark table partition transform should match",
- transform,
- table.partitioning()[len - 1].toString());
- }
-
- private SparkTable sparkTable() throws Exception {
- validationCatalog.loadTable(tableIdent).refresh();
- CatalogManager catalogManager = spark.sessionState().catalogManager();
- TableCatalog catalog = (TableCatalog) catalogManager.catalog(catalogName);
- Identifier identifier = Identifier.of(tableIdent.namespace().levels(), tableIdent.name());
- return (SparkTable) catalog.loadTable(identifier);
- }
-
- private void createTable(String schema) {
- createTable(schema, null);
- }
-
- private void createTable(String schema, String spec) {
- if (spec == null) {
- sql(
- "CREATE TABLE %s (%s) USING iceberg TBLPROPERTIES ('%s' '%d')",
- tableName, schema, TableProperties.FORMAT_VERSION, formatVersion);
- } else {
- sql(
- "CREATE TABLE %s (%s) USING iceberg PARTITIONED BY (%s) TBLPROPERTIES ('%s' '%d')",
- tableName, schema, spec, TableProperties.FORMAT_VERSION, formatVersion);
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAlterTableSchema.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAlterTableSchema.java
deleted file mode 100644
index c993c213dc5e..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAlterTableSchema.java
+++ /dev/null
@@ -1,162 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import java.util.Map;
-import org.apache.iceberg.AssertHelpers;
-import org.apache.iceberg.Table;
-import org.apache.iceberg.relocated.com.google.common.collect.Sets;
-import org.junit.After;
-import org.junit.Assert;
-import org.junit.Test;
-
-public class TestAlterTableSchema extends SparkExtensionsTestBase {
- public TestAlterTableSchema(
- String catalogName, String implementation, Map config) {
- super(catalogName, implementation, config);
- }
-
- @After
- public void removeTable() {
- sql("DROP TABLE IF EXISTS %s", tableName);
- }
-
- @Test
- public void testSetIdentifierFields() {
- sql(
- "CREATE TABLE %s (id bigint NOT NULL, "
- + "location struct NOT NULL) USING iceberg",
- tableName);
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertTrue(
- "Table should start without identifier", table.schema().identifierFieldIds().isEmpty());
-
- sql("ALTER TABLE %s SET IDENTIFIER FIELDS id", tableName);
- table.refresh();
- Assert.assertEquals(
- "Should have new identifier field",
- Sets.newHashSet(table.schema().findField("id").fieldId()),
- table.schema().identifierFieldIds());
-
- sql("ALTER TABLE %s SET IDENTIFIER FIELDS id, location.lon", tableName);
- table.refresh();
- Assert.assertEquals(
- "Should have new identifier field",
- Sets.newHashSet(
- table.schema().findField("id").fieldId(),
- table.schema().findField("location.lon").fieldId()),
- table.schema().identifierFieldIds());
-
- sql("ALTER TABLE %s SET IDENTIFIER FIELDS location.lon", tableName);
- table.refresh();
- Assert.assertEquals(
- "Should have new identifier field",
- Sets.newHashSet(table.schema().findField("location.lon").fieldId()),
- table.schema().identifierFieldIds());
- }
-
- @Test
- public void testSetInvalidIdentifierFields() {
- sql("CREATE TABLE %s (id bigint NOT NULL, id2 bigint) USING iceberg", tableName);
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertTrue(
- "Table should start without identifier", table.schema().identifierFieldIds().isEmpty());
- AssertHelpers.assertThrows(
- "should not allow setting unknown fields",
- IllegalArgumentException.class,
- "not found in current schema or added columns",
- () -> sql("ALTER TABLE %s SET IDENTIFIER FIELDS unknown", tableName));
-
- AssertHelpers.assertThrows(
- "should not allow setting optional fields",
- IllegalArgumentException.class,
- "not a required field",
- () -> sql("ALTER TABLE %s SET IDENTIFIER FIELDS id2", tableName));
- }
-
- @Test
- public void testDropIdentifierFields() {
- sql(
- "CREATE TABLE %s (id bigint NOT NULL, "
- + "location struct NOT NULL) USING iceberg",
- tableName);
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertTrue(
- "Table should start without identifier", table.schema().identifierFieldIds().isEmpty());
-
- sql("ALTER TABLE %s SET IDENTIFIER FIELDS id, location.lon", tableName);
- table.refresh();
- Assert.assertEquals(
- "Should have new identifier fields",
- Sets.newHashSet(
- table.schema().findField("id").fieldId(),
- table.schema().findField("location.lon").fieldId()),
- table.schema().identifierFieldIds());
-
- sql("ALTER TABLE %s DROP IDENTIFIER FIELDS id", tableName);
- table.refresh();
- Assert.assertEquals(
- "Should removed identifier field",
- Sets.newHashSet(table.schema().findField("location.lon").fieldId()),
- table.schema().identifierFieldIds());
-
- sql("ALTER TABLE %s SET IDENTIFIER FIELDS id, location.lon", tableName);
- table.refresh();
- Assert.assertEquals(
- "Should have new identifier fields",
- Sets.newHashSet(
- table.schema().findField("id").fieldId(),
- table.schema().findField("location.lon").fieldId()),
- table.schema().identifierFieldIds());
-
- sql("ALTER TABLE %s DROP IDENTIFIER FIELDS id, location.lon", tableName);
- table.refresh();
- Assert.assertEquals(
- "Should have no identifier field", Sets.newHashSet(), table.schema().identifierFieldIds());
- }
-
- @Test
- public void testDropInvalidIdentifierFields() {
- sql(
- "CREATE TABLE %s (id bigint NOT NULL, data string NOT NULL, "
- + "location struct NOT NULL) USING iceberg",
- tableName);
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertTrue(
- "Table should start without identifier", table.schema().identifierFieldIds().isEmpty());
- AssertHelpers.assertThrows(
- "should not allow dropping unknown fields",
- IllegalArgumentException.class,
- "field unknown not found",
- () -> sql("ALTER TABLE %s DROP IDENTIFIER FIELDS unknown", tableName));
-
- sql("ALTER TABLE %s SET IDENTIFIER FIELDS id", tableName);
- AssertHelpers.assertThrows(
- "should not allow dropping a field that is not an identifier",
- IllegalArgumentException.class,
- "data is not an identifier field",
- () -> sql("ALTER TABLE %s DROP IDENTIFIER FIELDS data", tableName));
-
- AssertHelpers.assertThrows(
- "should not allow dropping a nested field that is not an identifier",
- IllegalArgumentException.class,
- "location.lon is not an identifier field",
- () -> sql("ALTER TABLE %s DROP IDENTIFIER FIELDS location.lon", tableName));
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAncestorsOfProcedure.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAncestorsOfProcedure.java
deleted file mode 100644
index ae591821e21a..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAncestorsOfProcedure.java
+++ /dev/null
@@ -1,168 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import java.util.List;
-import java.util.Map;
-import org.apache.iceberg.AssertHelpers;
-import org.apache.iceberg.Table;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
-import org.apache.spark.sql.AnalysisException;
-import org.junit.After;
-import org.junit.Test;
-
-public class TestAncestorsOfProcedure extends SparkExtensionsTestBase {
-
- public TestAncestorsOfProcedure(
- String catalogName, String implementation, Map config) {
- super(catalogName, implementation, config);
- }
-
- @After
- public void removeTables() {
- sql("DROP TABLE IF EXISTS %s", tableName);
- }
-
- @Test
- public void testAncestorOfUsingEmptyArgs() {
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
- sql("INSERT INTO TABLE %s VALUES (1, 'a')", tableName);
- sql("INSERT INTO TABLE %s VALUES (2, 'b')", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
- Long currentSnapshotId = table.currentSnapshot().snapshotId();
- Long currentTimestamp = table.currentSnapshot().timestampMillis();
- Long preSnapshotId = table.currentSnapshot().parentId();
- Long preTimeStamp = table.snapshot(table.currentSnapshot().parentId()).timestampMillis();
-
- List output = sql("CALL %s.system.ancestors_of('%s')", catalogName, tableIdent);
-
- assertEquals(
- "Procedure output must match",
- ImmutableList.of(
- row(currentSnapshotId, currentTimestamp), row(preSnapshotId, preTimeStamp)),
- output);
- }
-
- @Test
- public void testAncestorOfUsingSnapshotId() {
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
- sql("INSERT INTO TABLE %s VALUES (1, 'a')", tableName);
- sql("INSERT INTO TABLE %s VALUES (2, 'b')", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
- Long currentSnapshotId = table.currentSnapshot().snapshotId();
- Long currentTimestamp = table.currentSnapshot().timestampMillis();
- Long preSnapshotId = table.currentSnapshot().parentId();
- Long preTimeStamp = table.snapshot(table.currentSnapshot().parentId()).timestampMillis();
-
- assertEquals(
- "Procedure output must match",
- ImmutableList.of(
- row(currentSnapshotId, currentTimestamp), row(preSnapshotId, preTimeStamp)),
- sql("CALL %s.system.ancestors_of('%s', %dL)", catalogName, tableIdent, currentSnapshotId));
-
- assertEquals(
- "Procedure output must match",
- ImmutableList.of(row(preSnapshotId, preTimeStamp)),
- sql("CALL %s.system.ancestors_of('%s', %dL)", catalogName, tableIdent, preSnapshotId));
- }
-
- @Test
- public void testAncestorOfWithRollBack() {
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
- Table table = validationCatalog.loadTable(tableIdent);
- sql("INSERT INTO TABLE %s VALUES (1, 'a')", tableName);
- table.refresh();
- Long firstSnapshotId = table.currentSnapshot().snapshotId();
- Long firstTimestamp = table.currentSnapshot().timestampMillis();
- sql("INSERT INTO TABLE %s VALUES (2, 'b')", tableName);
- table.refresh();
- Long secondSnapshotId = table.currentSnapshot().snapshotId();
- Long secondTimestamp = table.currentSnapshot().timestampMillis();
- sql("INSERT INTO TABLE %s VALUES (3, 'c')", tableName);
- table.refresh();
- Long thirdSnapshotId = table.currentSnapshot().snapshotId();
- Long thirdTimestamp = table.currentSnapshot().timestampMillis();
-
- // roll back
- sql(
- "CALL %s.system.rollback_to_snapshot('%s', %dL)",
- catalogName, tableIdent, secondSnapshotId);
-
- sql("INSERT INTO TABLE %s VALUES (4, 'd')", tableName);
- table.refresh();
- Long fourthSnapshotId = table.currentSnapshot().snapshotId();
- Long fourthTimestamp = table.currentSnapshot().timestampMillis();
-
- assertEquals(
- "Procedure output must match",
- ImmutableList.of(
- row(fourthSnapshotId, fourthTimestamp),
- row(secondSnapshotId, secondTimestamp),
- row(firstSnapshotId, firstTimestamp)),
- sql("CALL %s.system.ancestors_of('%s', %dL)", catalogName, tableIdent, fourthSnapshotId));
-
- assertEquals(
- "Procedure output must match",
- ImmutableList.of(
- row(thirdSnapshotId, thirdTimestamp),
- row(secondSnapshotId, secondTimestamp),
- row(firstSnapshotId, firstTimestamp)),
- sql("CALL %s.system.ancestors_of('%s', %dL)", catalogName, tableIdent, thirdSnapshotId));
- }
-
- @Test
- public void testAncestorOfUsingNamedArgs() {
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
- sql("INSERT INTO TABLE %s VALUES (1, 'a')", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
- Long firstSnapshotId = table.currentSnapshot().snapshotId();
- Long firstTimestamp = table.currentSnapshot().timestampMillis();
-
- assertEquals(
- "Procedure output must match",
- ImmutableList.of(row(firstSnapshotId, firstTimestamp)),
- sql(
- "CALL %s.system.ancestors_of(snapshot_id => %dL, table => '%s')",
- catalogName, firstSnapshotId, tableIdent));
- }
-
- @Test
- public void testInvalidAncestorOfCases() {
- AssertHelpers.assertThrows(
- "Should reject calls without all required args",
- AnalysisException.class,
- "Missing required parameters",
- () -> sql("CALL %s.system.ancestors_of()", catalogName));
-
- AssertHelpers.assertThrows(
- "Should reject calls with empty table identifier",
- IllegalArgumentException.class,
- "Cannot handle an empty identifier for parameter 'table'",
- () -> sql("CALL %s.system.ancestors_of('')", catalogName));
-
- AssertHelpers.assertThrows(
- "Should reject calls with invalid arg types",
- AnalysisException.class,
- "Wrong arg type for snapshot_id: cannot cast",
- () -> sql("CALL %s.system.ancestors_of('%s', 1.1)", catalogName, tableIdent));
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestBranchDDL.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestBranchDDL.java
deleted file mode 100644
index adf82d01cdc4..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestBranchDDL.java
+++ /dev/null
@@ -1,627 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import static org.assertj.core.api.Assertions.assertThat;
-
-import java.util.List;
-import java.util.Map;
-import java.util.concurrent.TimeUnit;
-import org.apache.iceberg.AssertHelpers;
-import org.apache.iceberg.Snapshot;
-import org.apache.iceberg.SnapshotRef;
-import org.apache.iceberg.Table;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
-import org.apache.iceberg.spark.SparkCatalogConfig;
-import org.apache.iceberg.spark.source.SimpleRecord;
-import org.apache.spark.sql.Dataset;
-import org.apache.spark.sql.Row;
-import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
-import org.apache.spark.sql.catalyst.parser.extensions.IcebergParseException;
-import org.assertj.core.api.Assertions;
-import org.junit.After;
-import org.junit.Assert;
-import org.junit.Before;
-import org.junit.Test;
-import org.junit.runners.Parameterized;
-
-public class TestBranchDDL extends SparkExtensionsTestBase {
- private static final String[] TIME_UNITS = {"DAYS", "HOURS", "MINUTES"};
-
- @Before
- public void before() {
- sql("CREATE TABLE %s (id INT, data STRING) USING iceberg", tableName);
- }
-
- @After
- public void removeTable() {
- sql("DROP TABLE IF EXISTS %s", tableName);
- }
-
- @Parameterized.Parameters(name = "catalogName = {0}, implementation = {1}, config = {2}")
- public static Object[][] parameters() {
- return new Object[][] {
- {
- SparkCatalogConfig.SPARK.catalogName(),
- SparkCatalogConfig.SPARK.implementation(),
- SparkCatalogConfig.SPARK.properties()
- }
- };
- }
-
- public TestBranchDDL(String catalog, String implementation, Map properties) {
- super(catalog, implementation, properties);
- }
-
- @Test
- public void testCreateBranch() throws NoSuchTableException {
- Table table = insertRows();
- long snapshotId = table.currentSnapshot().snapshotId();
-
- Integer minSnapshotsToKeep = 2;
- long maxSnapshotAge = 2L;
- long maxRefAge = 10L;
- for (String timeUnit : TIME_UNITS) {
- String branchName = "b1" + timeUnit;
- sql(
- "ALTER TABLE %s CREATE BRANCH %s AS OF VERSION %d RETAIN %d %s WITH SNAPSHOT RETENTION %d SNAPSHOTS %d %s",
- tableName,
- branchName,
- snapshotId,
- maxRefAge,
- timeUnit,
- minSnapshotsToKeep,
- maxSnapshotAge,
- timeUnit);
- table.refresh();
- SnapshotRef ref = table.refs().get(branchName);
- Assert.assertEquals(table.currentSnapshot().snapshotId(), ref.snapshotId());
- Assert.assertEquals(minSnapshotsToKeep, ref.minSnapshotsToKeep());
- Assert.assertEquals(
- TimeUnit.valueOf(timeUnit).toMillis(maxSnapshotAge), ref.maxSnapshotAgeMs().longValue());
- Assert.assertEquals(
- TimeUnit.valueOf(timeUnit).toMillis(maxRefAge), ref.maxRefAgeMs().longValue());
- }
- }
-
- @Test
- public void testCreateBranchOnEmptyTable() {
- String branchName = "b1";
- sql("ALTER TABLE %s CREATE BRANCH %s", tableName, "b1");
- Table table = validationCatalog.loadTable(tableIdent);
-
- SnapshotRef mainRef = table.refs().get(SnapshotRef.MAIN_BRANCH);
- Assertions.assertThat(mainRef).isNull();
-
- SnapshotRef ref = table.refs().get(branchName);
- Assertions.assertThat(ref).isNotNull();
- Assertions.assertThat(ref.minSnapshotsToKeep()).isNull();
- Assertions.assertThat(ref.maxSnapshotAgeMs()).isNull();
- Assertions.assertThat(ref.maxRefAgeMs()).isNull();
-
- Snapshot snapshot = table.snapshot(ref.snapshotId());
- Assertions.assertThat(snapshot.parentId()).isNull();
- Assertions.assertThat(snapshot.addedDataFiles(table.io())).isEmpty();
- Assertions.assertThat(snapshot.removedDataFiles(table.io())).isEmpty();
- Assertions.assertThat(snapshot.addedDeleteFiles(table.io())).isEmpty();
- Assertions.assertThat(snapshot.removedDeleteFiles(table.io())).isEmpty();
- }
-
- @Test
- public void testCreateBranchUseDefaultConfig() throws NoSuchTableException {
- Table table = insertRows();
- String branchName = "b1";
- sql("ALTER TABLE %s CREATE BRANCH %s", tableName, branchName);
- table.refresh();
- SnapshotRef ref = table.refs().get(branchName);
- Assert.assertEquals(table.currentSnapshot().snapshotId(), ref.snapshotId());
- Assert.assertNull(ref.minSnapshotsToKeep());
- Assert.assertNull(ref.maxSnapshotAgeMs());
- Assert.assertNull(ref.maxRefAgeMs());
- }
-
- @Test
- public void testCreateBranchUseCustomMinSnapshotsToKeep() throws NoSuchTableException {
- Integer minSnapshotsToKeep = 2;
- Table table = insertRows();
- String branchName = "b1";
- sql(
- "ALTER TABLE %s CREATE BRANCH %s WITH SNAPSHOT RETENTION %d SNAPSHOTS",
- tableName, branchName, minSnapshotsToKeep);
- table.refresh();
- SnapshotRef ref = table.refs().get(branchName);
- Assert.assertEquals(table.currentSnapshot().snapshotId(), ref.snapshotId());
- Assert.assertEquals(minSnapshotsToKeep, ref.minSnapshotsToKeep());
- Assert.assertNull(ref.maxSnapshotAgeMs());
- Assert.assertNull(ref.maxRefAgeMs());
- }
-
- @Test
- public void testCreateBranchUseCustomMaxSnapshotAge() throws NoSuchTableException {
- long maxSnapshotAge = 2L;
- Table table = insertRows();
- String branchName = "b1";
- sql(
- "ALTER TABLE %s CREATE BRANCH %s WITH SNAPSHOT RETENTION %d DAYS",
- tableName, branchName, maxSnapshotAge);
- table.refresh();
- SnapshotRef ref = table.refs().get(branchName);
- Assert.assertEquals(table.currentSnapshot().snapshotId(), ref.snapshotId());
- Assert.assertNull(ref.minSnapshotsToKeep());
- Assert.assertEquals(TimeUnit.DAYS.toMillis(maxSnapshotAge), ref.maxSnapshotAgeMs().longValue());
- Assert.assertNull(ref.maxRefAgeMs());
- }
-
- @Test
- public void testCreateBranchIfNotExists() throws NoSuchTableException {
- long maxSnapshotAge = 2L;
- Table table = insertRows();
- String branchName = "b1";
- sql(
- "ALTER TABLE %s CREATE BRANCH %s WITH SNAPSHOT RETENTION %d DAYS",
- tableName, branchName, maxSnapshotAge);
-
- AssertHelpers.assertThrows(
- "Cannot create an existing branch",
- IllegalArgumentException.class,
- "Ref b1 already exists",
- () -> sql("ALTER TABLE %s CREATE BRANCH %s", tableName, branchName));
-
- sql("ALTER TABLE %s CREATE BRANCH IF NOT EXISTS %s", tableName, branchName);
-
- table.refresh();
- SnapshotRef ref = table.refs().get(branchName);
- Assert.assertEquals(table.currentSnapshot().snapshotId(), ref.snapshotId());
- Assert.assertNull(ref.minSnapshotsToKeep());
- Assert.assertEquals(TimeUnit.DAYS.toMillis(maxSnapshotAge), ref.maxSnapshotAgeMs().longValue());
- Assert.assertNull(ref.maxRefAgeMs());
- }
-
- @Test
- public void testCreateBranchUseCustomMinSnapshotsToKeepAndMaxSnapshotAge()
- throws NoSuchTableException {
- Integer minSnapshotsToKeep = 2;
- long maxSnapshotAge = 2L;
- Table table = insertRows();
- String branchName = "b1";
-
- sql(
- "ALTER TABLE %s CREATE BRANCH %s WITH SNAPSHOT RETENTION %d SNAPSHOTS %d DAYS",
- tableName, branchName, minSnapshotsToKeep, maxSnapshotAge);
- table.refresh();
- SnapshotRef ref = table.refs().get(branchName);
- Assert.assertEquals(table.currentSnapshot().snapshotId(), ref.snapshotId());
- Assert.assertEquals(minSnapshotsToKeep, ref.minSnapshotsToKeep());
- Assert.assertEquals(TimeUnit.DAYS.toMillis(maxSnapshotAge), ref.maxSnapshotAgeMs().longValue());
- Assert.assertNull(ref.maxRefAgeMs());
-
- AssertHelpers.assertThrows(
- "Illegal statement",
- IcebergParseException.class,
- "no viable alternative at input 'WITH SNAPSHOT RETENTION'",
- () ->
- sql("ALTER TABLE %s CREATE BRANCH %s WITH SNAPSHOT RETENTION", tableName, branchName));
- }
-
- @Test
- public void testCreateBranchUseCustomMaxRefAge() throws NoSuchTableException {
- long maxRefAge = 10L;
- Table table = insertRows();
- String branchName = "b1";
- sql("ALTER TABLE %s CREATE BRANCH %s RETAIN %d DAYS", tableName, branchName, maxRefAge);
- table.refresh();
- SnapshotRef ref = table.refs().get(branchName);
- Assert.assertEquals(table.currentSnapshot().snapshotId(), ref.snapshotId());
- Assert.assertNull(ref.minSnapshotsToKeep());
- Assert.assertNull(ref.maxSnapshotAgeMs());
- Assert.assertEquals(TimeUnit.DAYS.toMillis(maxRefAge), ref.maxRefAgeMs().longValue());
-
- AssertHelpers.assertThrows(
- "Illegal statement",
- IcebergParseException.class,
- "mismatched input",
- () -> sql("ALTER TABLE %s CREATE BRANCH %s RETAIN", tableName, branchName));
-
- AssertHelpers.assertThrows(
- "Illegal statement",
- IcebergParseException.class,
- "mismatched input",
- () -> sql("ALTER TABLE %s CREATE BRANCH %s RETAIN %s DAYS", tableName, branchName, "abc"));
-
- AssertHelpers.assertThrows(
- "Illegal statement",
- IcebergParseException.class,
- "mismatched input 'SECONDS' expecting {'DAYS', 'HOURS', 'MINUTES'}",
- () ->
- sql(
- "ALTER TABLE %s CREATE BRANCH %s RETAIN %d SECONDS",
- tableName, branchName, maxRefAge));
- }
-
- @Test
- public void testDropBranch() throws NoSuchTableException {
- insertRows();
-
- Table table = validationCatalog.loadTable(tableIdent);
- String branchName = "b1";
- table.manageSnapshots().createBranch(branchName, table.currentSnapshot().snapshotId()).commit();
- SnapshotRef ref = table.refs().get(branchName);
- Assert.assertEquals(table.currentSnapshot().snapshotId(), ref.snapshotId());
-
- sql("ALTER TABLE %s DROP BRANCH %s", tableName, branchName);
- table.refresh();
-
- ref = table.refs().get(branchName);
- Assert.assertNull(ref);
- }
-
- @Test
- public void testDropBranchDoesNotExist() {
- AssertHelpers.assertThrows(
- "Cannot perform drop branch on branch which does not exist",
- IllegalArgumentException.class,
- "Branch does not exist: nonExistingBranch",
- () -> sql("ALTER TABLE %s DROP BRANCH %s", tableName, "nonExistingBranch"));
- }
-
- @Test
- public void testDropBranchFailsForTag() throws NoSuchTableException {
- String tagName = "b1";
- Table table = insertRows();
- table.manageSnapshots().createTag(tagName, table.currentSnapshot().snapshotId()).commit();
-
- AssertHelpers.assertThrows(
- "Cannot perform drop branch on tag",
- IllegalArgumentException.class,
- "Ref b1 is a tag not a branch",
- () -> sql("ALTER TABLE %s DROP BRANCH %s", tableName, tagName));
- }
-
- @Test
- public void testDropBranchNonConformingName() {
- AssertHelpers.assertThrows(
- "Non-conforming branch name",
- IcebergParseException.class,
- "mismatched input '123'",
- () -> sql("ALTER TABLE %s DROP BRANCH %s", tableName, "123"));
- }
-
- @Test
- public void testDropMainBranchFails() {
- AssertHelpers.assertThrows(
- "Cannot drop the main branch",
- IllegalArgumentException.class,
- "Cannot remove main branch",
- () -> sql("ALTER TABLE %s DROP BRANCH main", tableName));
- }
-
- @Test
- public void testDropBranchIfExists() {
- String branchName = "nonExistingBranch";
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertNull(table.refs().get(branchName));
-
- sql("ALTER TABLE %s DROP BRANCH IF EXISTS %s", tableName, branchName);
- table.refresh();
-
- SnapshotRef ref = table.refs().get(branchName);
- Assert.assertNull(ref);
- }
-
- @Test
- public void testReplaceBranchFailsForTag() throws NoSuchTableException {
- String tagName = "tag1";
-
- List records =
- ImmutableList.of(new SimpleRecord(1, "a"), new SimpleRecord(2, "b"));
- Dataset df = spark.createDataFrame(records, SimpleRecord.class);
- df.writeTo(tableName).append();
- Table table = validationCatalog.loadTable(tableIdent);
- long first = table.currentSnapshot().snapshotId();
- table.manageSnapshots().createTag(tagName, first).commit();
- df.writeTo(tableName).append();
- long second = table.currentSnapshot().snapshotId();
-
- AssertHelpers.assertThrows(
- "Cannot perform replace branch on tags",
- IllegalArgumentException.class,
- "Ref tag1 is a tag not a branch",
- () -> sql("ALTER TABLE %s REPLACE BRANCH %s AS OF VERSION %d", tableName, tagName, second));
- }
-
- @Test
- public void testReplaceBranch() throws NoSuchTableException {
- List records =
- ImmutableList.of(new SimpleRecord(1, "a"), new SimpleRecord(2, "b"));
- Dataset df = spark.createDataFrame(records, SimpleRecord.class);
- df.writeTo(tableName).append();
-
- Table table = validationCatalog.loadTable(tableIdent);
- long first = table.currentSnapshot().snapshotId();
- String branchName = "b1";
- long expectedMaxRefAgeMs = 1000;
- int expectedMinSnapshotsToKeep = 2;
- long expectedMaxSnapshotAgeMs = 1000;
- table
- .manageSnapshots()
- .createBranch(branchName, first)
- .setMaxRefAgeMs(branchName, expectedMaxRefAgeMs)
- .setMinSnapshotsToKeep(branchName, expectedMinSnapshotsToKeep)
- .setMaxSnapshotAgeMs(branchName, expectedMaxSnapshotAgeMs)
- .commit();
-
- df.writeTo(tableName).append();
- long second = table.currentSnapshot().snapshotId();
-
- sql("ALTER TABLE %s REPLACE BRANCH %s AS OF VERSION %d", tableName, branchName, second);
-
- table.refresh();
- SnapshotRef ref = table.refs().get(branchName);
- Assert.assertNotNull(ref);
- Assert.assertEquals(second, ref.snapshotId());
- Assert.assertEquals(expectedMinSnapshotsToKeep, ref.minSnapshotsToKeep().intValue());
- Assert.assertEquals(expectedMaxSnapshotAgeMs, ref.maxSnapshotAgeMs().longValue());
- Assert.assertEquals(expectedMaxRefAgeMs, ref.maxRefAgeMs().longValue());
- }
-
- @Test
- public void testReplaceBranchDoesNotExist() throws NoSuchTableException {
- List records =
- ImmutableList.of(new SimpleRecord(1, "a"), new SimpleRecord(2, "b"));
- Dataset df = spark.createDataFrame(records, SimpleRecord.class);
- df.writeTo(tableName).append();
- Table table = validationCatalog.loadTable(tableIdent);
-
- AssertHelpers.assertThrows(
- "Cannot perform replace branch on branch which does not exist",
- IllegalArgumentException.class,
- "Branch does not exist",
- () ->
- sql(
- "ALTER TABLE %s REPLACE BRANCH %s AS OF VERSION %d",
- tableName, "someBranch", table.currentSnapshot().snapshotId()));
- }
-
- @Test
- public void testReplaceBranchWithRetain() throws NoSuchTableException {
- List records =
- ImmutableList.of(new SimpleRecord(1, "a"), new SimpleRecord(2, "b"));
- Dataset df = spark.createDataFrame(records, SimpleRecord.class);
- df.writeTo(tableName).append();
-
- Table table = validationCatalog.loadTable(tableIdent);
- long first = table.currentSnapshot().snapshotId();
- String branchName = "b1";
- table.manageSnapshots().createBranch(branchName, first).commit();
- SnapshotRef b1 = table.refs().get(branchName);
- Integer minSnapshotsToKeep = b1.minSnapshotsToKeep();
- Long maxSnapshotAgeMs = b1.maxSnapshotAgeMs();
- df.writeTo(tableName).append();
- long second = table.currentSnapshot().snapshotId();
-
- long maxRefAge = 10;
- for (String timeUnit : TIME_UNITS) {
- sql(
- "ALTER TABLE %s REPLACE BRANCH %s AS OF VERSION %d RETAIN %d %s",
- tableName, branchName, second, maxRefAge, timeUnit);
-
- table.refresh();
- SnapshotRef ref = table.refs().get(branchName);
- Assert.assertNotNull(ref);
- Assert.assertEquals(second, ref.snapshotId());
- Assert.assertEquals(minSnapshotsToKeep, ref.minSnapshotsToKeep());
- Assert.assertEquals(maxSnapshotAgeMs, ref.maxSnapshotAgeMs());
- Assert.assertEquals(
- TimeUnit.valueOf(timeUnit).toMillis(maxRefAge), ref.maxRefAgeMs().longValue());
- }
- }
-
- @Test
- public void testReplaceBranchWithSnapshotRetention() throws NoSuchTableException {
- List records =
- ImmutableList.of(new SimpleRecord(1, "a"), new SimpleRecord(2, "b"));
- Dataset df = spark.createDataFrame(records, SimpleRecord.class);
- df.writeTo(tableName).append();
- String branchName = "b1";
- Table table = validationCatalog.loadTable(tableIdent);
- long first = table.currentSnapshot().snapshotId();
- table.manageSnapshots().createBranch(branchName, first).commit();
- df.writeTo(tableName).append();
- long second = table.currentSnapshot().snapshotId();
-
- Integer minSnapshotsToKeep = 2;
- long maxSnapshotAge = 2;
- Long maxRefAgeMs = table.refs().get(branchName).maxRefAgeMs();
- for (String timeUnit : TIME_UNITS) {
- sql(
- "ALTER TABLE %s REPLACE BRANCH %s AS OF VERSION %d WITH SNAPSHOT RETENTION %d SNAPSHOTS %d %s",
- tableName, branchName, second, minSnapshotsToKeep, maxSnapshotAge, timeUnit);
-
- table.refresh();
- SnapshotRef ref = table.refs().get(branchName);
- Assert.assertNotNull(ref);
- Assert.assertEquals(second, ref.snapshotId());
- Assert.assertEquals(minSnapshotsToKeep, ref.minSnapshotsToKeep());
- Assert.assertEquals(
- TimeUnit.valueOf(timeUnit).toMillis(maxSnapshotAge), ref.maxSnapshotAgeMs().longValue());
- Assert.assertEquals(maxRefAgeMs, ref.maxRefAgeMs());
- }
- }
-
- @Test
- public void testReplaceBranchWithRetainAndSnapshotRetention() throws NoSuchTableException {
- List records =
- ImmutableList.of(new SimpleRecord(1, "a"), new SimpleRecord(2, "b"));
- Dataset df = spark.createDataFrame(records, SimpleRecord.class);
- df.writeTo(tableName).append();
-
- Table table = validationCatalog.loadTable(tableIdent);
- long first = table.currentSnapshot().snapshotId();
- String branchName = "b1";
- table.manageSnapshots().createBranch(branchName, first).commit();
- df.writeTo(tableName).append();
- long second = table.currentSnapshot().snapshotId();
-
- Integer minSnapshotsToKeep = 2;
- long maxSnapshotAge = 2;
- long maxRefAge = 10;
- for (String timeUnit : TIME_UNITS) {
- sql(
- "ALTER TABLE %s REPLACE BRANCH %s AS OF VERSION %d RETAIN %d %s WITH SNAPSHOT RETENTION %d SNAPSHOTS %d %s",
- tableName,
- branchName,
- second,
- maxRefAge,
- timeUnit,
- minSnapshotsToKeep,
- maxSnapshotAge,
- timeUnit);
-
- table.refresh();
- SnapshotRef ref = table.refs().get(branchName);
- Assert.assertNotNull(ref);
- Assert.assertEquals(second, ref.snapshotId());
- Assert.assertEquals(minSnapshotsToKeep, ref.minSnapshotsToKeep());
- Assert.assertEquals(
- TimeUnit.valueOf(timeUnit).toMillis(maxSnapshotAge), ref.maxSnapshotAgeMs().longValue());
- Assert.assertEquals(
- TimeUnit.valueOf(timeUnit).toMillis(maxRefAge), ref.maxRefAgeMs().longValue());
- }
- }
-
- @Test
- public void testCreateOrReplace() throws NoSuchTableException {
- List records =
- ImmutableList.of(new SimpleRecord(1, "a"), new SimpleRecord(2, "b"));
- Dataset df = spark.createDataFrame(records, SimpleRecord.class);
- df.writeTo(tableName).append();
-
- Table table = validationCatalog.loadTable(tableIdent);
- long first = table.currentSnapshot().snapshotId();
- String branchName = "b1";
- df.writeTo(tableName).append();
- long second = table.currentSnapshot().snapshotId();
- table.manageSnapshots().createBranch(branchName, second).commit();
-
- sql(
- "ALTER TABLE %s CREATE OR REPLACE BRANCH %s AS OF VERSION %d",
- tableName, branchName, first);
-
- table.refresh();
- SnapshotRef ref = table.refs().get(branchName);
- Assert.assertNotNull(ref);
- Assert.assertEquals(first, ref.snapshotId());
- }
-
- @Test
- public void createOrReplace() throws NoSuchTableException {
- Table table = insertRows();
- long first = table.currentSnapshot().snapshotId();
- String branchName = "b1";
- insertRows();
- long second = table.currentSnapshot().snapshotId();
- table.manageSnapshots().createBranch(branchName, second).commit();
-
- sql(
- "ALTER TABLE %s CREATE OR REPLACE BRANCH %s AS OF VERSION %d",
- tableName, branchName, first);
- table.refresh();
- assertThat(table.refs().get(branchName).snapshotId()).isEqualTo(second);
- }
-
- @Test
- public void testCreateOrReplaceBranchOnEmptyTable() {
- String branchName = "b1";
- sql("ALTER TABLE %s CREATE OR REPLACE BRANCH %s", tableName, "b1");
- Table table = validationCatalog.loadTable(tableIdent);
-
- SnapshotRef mainRef = table.refs().get(SnapshotRef.MAIN_BRANCH);
- Assertions.assertThat(mainRef).isNull();
-
- SnapshotRef ref = table.refs().get(branchName);
- Assertions.assertThat(ref).isNotNull();
- Assertions.assertThat(ref.minSnapshotsToKeep()).isNull();
- Assertions.assertThat(ref.maxSnapshotAgeMs()).isNull();
- Assertions.assertThat(ref.maxRefAgeMs()).isNull();
-
- Snapshot snapshot = table.snapshot(ref.snapshotId());
- Assertions.assertThat(snapshot.parentId()).isNull();
- Assertions.assertThat(snapshot.addedDataFiles(table.io())).isEmpty();
- Assertions.assertThat(snapshot.removedDataFiles(table.io())).isEmpty();
- Assertions.assertThat(snapshot.addedDeleteFiles(table.io())).isEmpty();
- Assertions.assertThat(snapshot.removedDeleteFiles(table.io())).isEmpty();
- }
-
- @Test
- public void createOrReplaceWithNonExistingBranch() throws NoSuchTableException {
- Table table = insertRows();
- String branchName = "b1";
- insertRows();
- long snapshotId = table.currentSnapshot().snapshotId();
-
- sql(
- "ALTER TABLE %s CREATE OR REPLACE BRANCH %s AS OF VERSION %d",
- tableName, branchName, snapshotId);
- table.refresh();
- assertThat(table.refs().get(branchName).snapshotId()).isEqualTo(snapshotId);
- }
-
- @Test
- public void replaceBranch() throws NoSuchTableException {
- Table table = insertRows();
- long first = table.currentSnapshot().snapshotId();
- String branchName = "b1";
- long expectedMaxRefAgeMs = 1000;
- table
- .manageSnapshots()
- .createBranch(branchName, first)
- .setMaxRefAgeMs(branchName, expectedMaxRefAgeMs)
- .commit();
-
- insertRows();
- long second = table.currentSnapshot().snapshotId();
-
- sql("ALTER TABLE %s REPLACE BRANCH %s AS OF VERSION %d", tableName, branchName, second);
- table.refresh();
- SnapshotRef ref = table.refs().get(branchName);
- assertThat(ref.snapshotId()).isEqualTo(second);
- assertThat(ref.maxRefAgeMs()).isEqualTo(expectedMaxRefAgeMs);
- }
-
- @Test
- public void replaceBranchDoesNotExist() throws NoSuchTableException {
- Table table = insertRows();
-
- Assertions.assertThatThrownBy(
- () ->
- sql(
- "ALTER TABLE %s REPLACE BRANCH %s AS OF VERSION %d",
- tableName, "someBranch", table.currentSnapshot().snapshotId()))
- .isInstanceOf(IllegalArgumentException.class)
- .hasMessage("Branch does not exist: someBranch");
- }
-
- private Table insertRows() throws NoSuchTableException {
- List records =
- ImmutableList.of(new SimpleRecord(1, "a"), new SimpleRecord(2, "b"));
- Dataset df = spark.createDataFrame(records, SimpleRecord.class);
- df.writeTo(tableName).append();
- return validationCatalog.loadTable(tableIdent);
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java
deleted file mode 100644
index 9c2233ccb791..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCallStatementParser.java
+++ /dev/null
@@ -1,214 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import java.math.BigDecimal;
-import java.sql.Timestamp;
-import java.time.Instant;
-import java.util.List;
-import org.apache.iceberg.AssertHelpers;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
-import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-import org.apache.spark.sql.SparkSession;
-import org.apache.spark.sql.catalyst.expressions.Expression;
-import org.apache.spark.sql.catalyst.expressions.Literal;
-import org.apache.spark.sql.catalyst.expressions.Literal$;
-import org.apache.spark.sql.catalyst.parser.ParseException;
-import org.apache.spark.sql.catalyst.parser.ParserInterface;
-import org.apache.spark.sql.catalyst.parser.extensions.IcebergParseException;
-import org.apache.spark.sql.catalyst.plans.logical.CallArgument;
-import org.apache.spark.sql.catalyst.plans.logical.CallStatement;
-import org.apache.spark.sql.catalyst.plans.logical.NamedArgument;
-import org.apache.spark.sql.catalyst.plans.logical.PositionalArgument;
-import org.apache.spark.sql.types.DataType;
-import org.apache.spark.sql.types.DataTypes;
-import org.junit.AfterClass;
-import org.junit.Assert;
-import org.junit.BeforeClass;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.TemporaryFolder;
-import scala.collection.JavaConverters;
-
-public class TestCallStatementParser {
-
- @Rule public TemporaryFolder temp = new TemporaryFolder();
-
- private static SparkSession spark = null;
- private static ParserInterface parser = null;
-
- @BeforeClass
- public static void startSpark() {
- TestCallStatementParser.spark =
- SparkSession.builder()
- .master("local[2]")
- .config("spark.sql.extensions", IcebergSparkSessionExtensions.class.getName())
- .config("spark.extra.prop", "value")
- .getOrCreate();
- TestCallStatementParser.parser = spark.sessionState().sqlParser();
- }
-
- @AfterClass
- public static void stopSpark() {
- SparkSession currentSpark = TestCallStatementParser.spark;
- TestCallStatementParser.spark = null;
- TestCallStatementParser.parser = null;
- currentSpark.stop();
- }
-
- @Test
- public void testCallWithPositionalArgs() throws ParseException {
- CallStatement call =
- (CallStatement) parser.parsePlan("CALL c.n.func(1, '2', 3L, true, 1.0D, 9.0e1, 900e-1BD)");
- Assert.assertEquals(
- ImmutableList.of("c", "n", "func"), JavaConverters.seqAsJavaList(call.name()));
-
- Assert.assertEquals(7, call.args().size());
-
- checkArg(call, 0, 1, DataTypes.IntegerType);
- checkArg(call, 1, "2", DataTypes.StringType);
- checkArg(call, 2, 3L, DataTypes.LongType);
- checkArg(call, 3, true, DataTypes.BooleanType);
- checkArg(call, 4, 1.0D, DataTypes.DoubleType);
- checkArg(call, 5, 9.0e1, DataTypes.DoubleType);
- checkArg(call, 6, new BigDecimal("900e-1"), DataTypes.createDecimalType(3, 1));
- }
-
- @Test
- public void testCallWithNamedArgs() throws ParseException {
- CallStatement call =
- (CallStatement) parser.parsePlan("CALL cat.system.func(c1 => 1, c2 => '2', c3 => true)");
- Assert.assertEquals(
- ImmutableList.of("cat", "system", "func"), JavaConverters.seqAsJavaList(call.name()));
-
- Assert.assertEquals(3, call.args().size());
-
- checkArg(call, 0, "c1", 1, DataTypes.IntegerType);
- checkArg(call, 1, "c2", "2", DataTypes.StringType);
- checkArg(call, 2, "c3", true, DataTypes.BooleanType);
- }
-
- @Test
- public void testCallWithMixedArgs() throws ParseException {
- CallStatement call = (CallStatement) parser.parsePlan("CALL cat.system.func(c1 => 1, '2')");
- Assert.assertEquals(
- ImmutableList.of("cat", "system", "func"), JavaConverters.seqAsJavaList(call.name()));
-
- Assert.assertEquals(2, call.args().size());
-
- checkArg(call, 0, "c1", 1, DataTypes.IntegerType);
- checkArg(call, 1, "2", DataTypes.StringType);
- }
-
- @Test
- public void testCallWithTimestampArg() throws ParseException {
- CallStatement call =
- (CallStatement)
- parser.parsePlan("CALL cat.system.func(TIMESTAMP '2017-02-03T10:37:30.00Z')");
- Assert.assertEquals(
- ImmutableList.of("cat", "system", "func"), JavaConverters.seqAsJavaList(call.name()));
-
- Assert.assertEquals(1, call.args().size());
-
- checkArg(
- call, 0, Timestamp.from(Instant.parse("2017-02-03T10:37:30.00Z")), DataTypes.TimestampType);
- }
-
- @Test
- public void testCallWithVarSubstitution() throws ParseException {
- CallStatement call =
- (CallStatement) parser.parsePlan("CALL cat.system.func('${spark.extra.prop}')");
- Assert.assertEquals(
- ImmutableList.of("cat", "system", "func"), JavaConverters.seqAsJavaList(call.name()));
-
- Assert.assertEquals(1, call.args().size());
-
- checkArg(call, 0, "value", DataTypes.StringType);
- }
-
- @Test
- public void testCallParseError() {
- AssertHelpers.assertThrows(
- "Should fail with a sensible parse error",
- IcebergParseException.class,
- "missing '(' at 'radish'",
- () -> parser.parsePlan("CALL cat.system radish kebab"));
- }
-
- @Test
- public void testCallStripsComments() throws ParseException {
- List callStatementsWithComments =
- Lists.newArrayList(
- "/* bracketed comment */ CALL cat.system.func('${spark.extra.prop}')",
- "/**/ CALL cat.system.func('${spark.extra.prop}')",
- "-- single line comment \n CALL cat.system.func('${spark.extra.prop}')",
- "-- multiple \n-- single line \n-- comments \n CALL cat.system.func('${spark.extra.prop}')",
- "/* select * from multiline_comment \n where x like '%sql%'; */ CALL cat.system.func('${spark.extra.prop}')",
- "/* {\"app\": \"dbt\", \"dbt_version\": \"1.0.1\", \"profile_name\": \"profile1\", \"target_name\": \"dev\", "
- + "\"node_id\": \"model.profile1.stg_users\"} \n*/ CALL cat.system.func('${spark.extra.prop}')",
- "/* Some multi-line comment \n"
- + "*/ CALL /* inline comment */ cat.system.func('${spark.extra.prop}') -- ending comment",
- "CALL -- a line ending comment\n" + "cat.system.func('${spark.extra.prop}')");
- for (String sqlText : callStatementsWithComments) {
- CallStatement call = (CallStatement) parser.parsePlan(sqlText);
- Assert.assertEquals(
- ImmutableList.of("cat", "system", "func"), JavaConverters.seqAsJavaList(call.name()));
-
- Assert.assertEquals(1, call.args().size());
-
- checkArg(call, 0, "value", DataTypes.StringType);
- }
- }
-
- private void checkArg(
- CallStatement call, int index, Object expectedValue, DataType expectedType) {
- checkArg(call, index, null, expectedValue, expectedType);
- }
-
- private void checkArg(
- CallStatement call,
- int index,
- String expectedName,
- Object expectedValue,
- DataType expectedType) {
-
- if (expectedName != null) {
- NamedArgument arg = checkCast(call.args().apply(index), NamedArgument.class);
- Assert.assertEquals(expectedName, arg.name());
- } else {
- CallArgument arg = call.args().apply(index);
- checkCast(arg, PositionalArgument.class);
- }
-
- Expression expectedExpr = toSparkLiteral(expectedValue, expectedType);
- Expression actualExpr = call.args().apply(index).expr();
- Assert.assertEquals("Arg types must match", expectedExpr.dataType(), actualExpr.dataType());
- Assert.assertEquals("Arg must match", expectedExpr, actualExpr);
- }
-
- private Literal toSparkLiteral(Object value, DataType dataType) {
- return Literal$.MODULE$.create(value, dataType);
- }
-
- private T checkCast(Object value, Class expectedClass) {
- Assert.assertTrue(
- "Expected instance of " + expectedClass.getName(), expectedClass.isInstance(value));
- return expectedClass.cast(value);
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestChangelogTable.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestChangelogTable.java
deleted file mode 100644
index cc81b4b3d323..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestChangelogTable.java
+++ /dev/null
@@ -1,353 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import static org.apache.iceberg.AssertHelpers.assertThrows;
-import static org.apache.iceberg.TableProperties.FORMAT_VERSION;
-import static org.apache.iceberg.TableProperties.MANIFEST_MERGE_ENABLED;
-import static org.apache.iceberg.TableProperties.MANIFEST_MIN_MERGE_COUNT;
-
-import java.util.List;
-import java.util.Map;
-import org.apache.iceberg.DataOperations;
-import org.apache.iceberg.Snapshot;
-import org.apache.iceberg.Table;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
-import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
-import org.apache.iceberg.spark.SparkCatalogConfig;
-import org.apache.iceberg.spark.SparkReadOptions;
-import org.apache.iceberg.spark.source.SparkChangelogTable;
-import org.apache.spark.sql.DataFrameReader;
-import org.apache.spark.sql.Row;
-import org.junit.After;
-import org.junit.Assert;
-import org.junit.Test;
-import org.junit.runners.Parameterized.Parameters;
-
-public class TestChangelogTable extends SparkExtensionsTestBase {
-
- @Parameters(name = "formatVersion = {0}, catalogName = {1}, implementation = {2}, config = {3}")
- public static Object[][] parameters() {
- return new Object[][] {
- {
- 1,
- SparkCatalogConfig.SPARK.catalogName(),
- SparkCatalogConfig.SPARK.implementation(),
- SparkCatalogConfig.SPARK.properties()
- },
- {
- 2,
- SparkCatalogConfig.HIVE.catalogName(),
- SparkCatalogConfig.HIVE.implementation(),
- SparkCatalogConfig.HIVE.properties()
- }
- };
- }
-
- private final int formatVersion;
-
- public TestChangelogTable(
- int formatVersion, String catalogName, String implementation, Map config) {
- super(catalogName, implementation, config);
- this.formatVersion = formatVersion;
- }
-
- @After
- public void removeTables() {
- sql("DROP TABLE IF EXISTS %s", tableName);
- }
-
- @Test
- public void testDataFilters() {
- createTableWithDefaultRows();
-
- sql("INSERT INTO %s VALUES (3, 'c')", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
-
- Snapshot snap3 = table.currentSnapshot();
-
- sql("DELETE FROM %s WHERE id = 3", tableName);
-
- table.refresh();
-
- Snapshot snap4 = table.currentSnapshot();
-
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(
- row(3, "c", "INSERT", 2, snap3.snapshotId()),
- row(3, "c", "DELETE", 3, snap4.snapshotId())),
- sql("SELECT * FROM %s.changes WHERE id = 3 ORDER BY _change_ordinal, id", tableName));
- }
-
- @Test
- public void testOverwrites() {
- createTableWithDefaultRows();
-
- Table table = validationCatalog.loadTable(tableIdent);
-
- Snapshot snap2 = table.currentSnapshot();
-
- sql("INSERT OVERWRITE %s VALUES (-2, 'b')", tableName);
-
- table.refresh();
-
- Snapshot snap3 = table.currentSnapshot();
-
- assertEquals(
- "Rows should match",
- ImmutableList.of(
- row(2, "b", "DELETE", 0, snap3.snapshotId()),
- row(-2, "b", "INSERT", 0, snap3.snapshotId())),
- changelogRecords(snap2, snap3));
- }
-
- @Test
- public void testQueryWithTimeRange() {
- createTable();
-
- sql("INSERT INTO %s VALUES (1, 'a')", tableName);
- Table table = validationCatalog.loadTable(tableIdent);
- Snapshot snap1 = table.currentSnapshot();
- long rightAfterSnap1 = waitUntilAfter(snap1.timestampMillis());
-
- sql("INSERT INTO %s VALUES (2, 'b')", tableName);
- table.refresh();
- Snapshot snap2 = table.currentSnapshot();
- long rightAfterSnap2 = waitUntilAfter(snap2.timestampMillis());
-
- sql("INSERT OVERWRITE %s VALUES (-2, 'b')", tableName);
- table.refresh();
- Snapshot snap3 = table.currentSnapshot();
- long rightAfterSnap3 = waitUntilAfter(snap3.timestampMillis());
-
- assertEquals(
- "Should have expected changed rows only from snapshot 3",
- ImmutableList.of(
- row(2, "b", "DELETE", 0, snap3.snapshotId()),
- row(-2, "b", "INSERT", 0, snap3.snapshotId())),
- changelogRecords(rightAfterSnap2, snap3.timestampMillis()));
-
- assertEquals(
- "Should have expected changed rows only from snapshot 3",
- ImmutableList.of(
- row(2, "b", "DELETE", 0, snap3.snapshotId()),
- row(-2, "b", "INSERT", 0, snap3.snapshotId())),
- changelogRecords(snap2.timestampMillis(), snap3.timestampMillis()));
-
- assertEquals(
- "Should have expected changed rows from snapshot 2 and 3",
- ImmutableList.of(
- row(2, "b", "INSERT", 0, snap2.snapshotId()),
- row(2, "b", "DELETE", 1, snap3.snapshotId()),
- row(-2, "b", "INSERT", 1, snap3.snapshotId())),
- changelogRecords(rightAfterSnap1, snap3.timestampMillis()));
-
- assertEquals(
- "Should have expected changed rows up to the current snapshot",
- ImmutableList.of(
- row(2, "b", "INSERT", 0, snap2.snapshotId()),
- row(2, "b", "DELETE", 1, snap3.snapshotId()),
- row(-2, "b", "INSERT", 1, snap3.snapshotId())),
- changelogRecords(rightAfterSnap1, null));
-
- assertEquals(
- "Should have empty changed rows if end time is before the first snapshot",
- ImmutableList.of(),
- changelogRecords(null, snap1.timestampMillis() - 1));
-
- assertEquals(
- "Should have empty changed rows if start time is after the current snapshot",
- ImmutableList.of(),
- changelogRecords(rightAfterSnap3, null));
-
- assertEquals(
- "Should have empty changed rows if end time is before the first snapshot",
- ImmutableList.of(),
- changelogRecords(null, snap1.timestampMillis() - 1));
-
- assertEquals(
- "Should have empty changed rows if there are no snapshots between start time and end time",
- ImmutableList.of(),
- changelogRecords(rightAfterSnap2, snap3.timestampMillis() - 1));
- }
-
- @Test
- public void testTimeRangeValidation() {
- createTableWithDefaultRows();
-
- Table table = validationCatalog.loadTable(tableIdent);
-
- Snapshot snap2 = table.currentSnapshot();
-
- sql("INSERT OVERWRITE %s VALUES (-2, 'b')", tableName);
- table.refresh();
- Snapshot snap3 = table.currentSnapshot();
- long rightAfterSnap3 = waitUntilAfter(snap3.timestampMillis());
-
- assertThrows(
- "Should fail if start time is after end time",
- IllegalArgumentException.class,
- () -> changelogRecords(snap3.timestampMillis(), snap2.timestampMillis()));
- }
-
- @Test
- public void testMetadataDeletes() {
- createTableWithDefaultRows();
-
- Table table = validationCatalog.loadTable(tableIdent);
-
- Snapshot snap2 = table.currentSnapshot();
-
- sql("DELETE FROM %s WHERE data = 'a'", tableName);
-
- table.refresh();
-
- Snapshot snap3 = table.currentSnapshot();
- Assert.assertEquals("Operation must match", DataOperations.DELETE, snap3.operation());
-
- assertEquals(
- "Rows should match",
- ImmutableList.of(row(1, "a", "DELETE", 0, snap3.snapshotId())),
- changelogRecords(snap2, snap3));
- }
-
- @Test
- public void testExistingEntriesInNewDataManifestsAreIgnored() {
- sql(
- "CREATE TABLE %s (id INT, data STRING) "
- + "USING iceberg "
- + "PARTITIONED BY (data) "
- + "TBLPROPERTIES ( "
- + " '%s' = '%d', "
- + " '%s' = '1', "
- + " '%s' = 'true' "
- + ")",
- tableName, FORMAT_VERSION, formatVersion, MANIFEST_MIN_MERGE_COUNT, MANIFEST_MERGE_ENABLED);
-
- sql("INSERT INTO %s VALUES (1, 'a')", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
-
- Snapshot snap1 = table.currentSnapshot();
-
- sql("INSERT INTO %s VALUES (2, 'b')", tableName);
-
- table.refresh();
-
- Snapshot snap2 = table.currentSnapshot();
- Assert.assertEquals("Manifest number must match", 1, snap2.dataManifests(table.io()).size());
-
- assertEquals(
- "Rows should match",
- ImmutableList.of(row(2, "b", "INSERT", 0, snap2.snapshotId())),
- changelogRecords(snap1, snap2));
- }
-
- @Test
- public void testManifestRewritesAreIgnored() {
- createTableWithDefaultRows();
-
- sql("CALL %s.system.rewrite_manifests('%s')", catalogName, tableIdent);
-
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertEquals("Num snapshots must match", 3, Iterables.size(table.snapshots()));
-
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(1, "INSERT"), row(2, "INSERT")),
- sql("SELECT id, _change_type FROM %s.changes ORDER BY id", tableName));
- }
-
- @Test
- public void testMetadataColumns() {
- createTableWithDefaultRows();
- List rows =
- sql(
- "SELECT id, _file, _pos, _deleted, _spec_id, _partition FROM %s.changes ORDER BY id",
- tableName);
-
- String file1 = rows.get(0)[1].toString();
- Assert.assertTrue(file1.startsWith("file:/"));
- String file2 = rows.get(1)[1].toString();
-
- assertEquals(
- "Rows should match",
- ImmutableList.of(
- row(1, file1, 0L, false, 0, row("a")), row(2, file2, 0L, false, 0, row("b"))),
- rows);
- }
-
- private void createTableWithDefaultRows() {
- createTable();
- insertDefaultRows();
- }
-
- private void createTable() {
- sql(
- "CREATE TABLE %s (id INT, data STRING) "
- + "USING iceberg "
- + "PARTITIONED BY (data) "
- + "TBLPROPERTIES ( "
- + " '%s' = '%d' "
- + ")",
- tableName, FORMAT_VERSION, formatVersion);
- }
-
- private void insertDefaultRows() {
- sql("INSERT INTO %s VALUES (1, 'a')", tableName);
- sql("INSERT INTO %s VALUES (2, 'b')", tableName);
- }
-
- private List changelogRecords(Snapshot startSnapshot, Snapshot endSnapshot) {
- DataFrameReader reader = spark.read();
-
- if (startSnapshot != null) {
- reader = reader.option(SparkReadOptions.START_SNAPSHOT_ID, startSnapshot.snapshotId());
- }
-
- if (endSnapshot != null) {
- reader = reader.option(SparkReadOptions.END_SNAPSHOT_ID, endSnapshot.snapshotId());
- }
-
- return rowsToJava(collect(reader));
- }
-
- private List changelogRecords(Long startTimestamp, Long endTimeStamp) {
- DataFrameReader reader = spark.read();
-
- if (startTimestamp != null) {
- reader = reader.option(SparkReadOptions.START_TIMESTAMP, startTimestamp);
- }
-
- if (endTimeStamp != null) {
- reader = reader.option(SparkReadOptions.END_TIMESTAMP, endTimeStamp);
- }
-
- return rowsToJava(collect(reader));
- }
-
- private List collect(DataFrameReader reader) {
- return reader
- .table(tableName + "." + SparkChangelogTable.TABLE_NAME)
- .orderBy("_change_ordinal", "_commit_snapshot_id", "_change_type", "id")
- .collectAsList();
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCherrypickSnapshotProcedure.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCherrypickSnapshotProcedure.java
deleted file mode 100644
index 7309a176b922..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCherrypickSnapshotProcedure.java
+++ /dev/null
@@ -1,203 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import static org.apache.iceberg.TableProperties.WRITE_AUDIT_PUBLISH_ENABLED;
-
-import java.util.List;
-import java.util.Map;
-import org.apache.iceberg.AssertHelpers;
-import org.apache.iceberg.Snapshot;
-import org.apache.iceberg.Table;
-import org.apache.iceberg.exceptions.ValidationException;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
-import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
-import org.apache.spark.sql.AnalysisException;
-import org.apache.spark.sql.Dataset;
-import org.apache.spark.sql.Row;
-import org.apache.spark.sql.catalyst.analysis.NoSuchProcedureException;
-import org.junit.After;
-import org.junit.Test;
-
-public class TestCherrypickSnapshotProcedure extends SparkExtensionsTestBase {
-
- public TestCherrypickSnapshotProcedure(
- String catalogName, String implementation, Map config) {
- super(catalogName, implementation, config);
- }
-
- @After
- public void removeTables() {
- sql("DROP TABLE IF EXISTS %s", tableName);
- }
-
- @Test
- public void testCherrypickSnapshotUsingPositionalArgs() {
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
- sql("ALTER TABLE %s SET TBLPROPERTIES ('%s' 'true')", tableName, WRITE_AUDIT_PUBLISH_ENABLED);
-
- spark.conf().set("spark.wap.id", "1");
-
- sql("INSERT INTO TABLE %s VALUES (1, 'a')", tableName);
-
- assertEquals(
- "Should not see rows from staged snapshot",
- ImmutableList.of(),
- sql("SELECT * FROM %s", tableName));
-
- Table table = validationCatalog.loadTable(tableIdent);
- Snapshot wapSnapshot = Iterables.getOnlyElement(table.snapshots());
-
- List output =
- sql(
- "CALL %s.system.cherrypick_snapshot('%s', %dL)",
- catalogName, tableIdent, wapSnapshot.snapshotId());
-
- table.refresh();
-
- Snapshot currentSnapshot = table.currentSnapshot();
-
- assertEquals(
- "Procedure output must match",
- ImmutableList.of(row(wapSnapshot.snapshotId(), currentSnapshot.snapshotId())),
- output);
-
- assertEquals(
- "Cherrypick must be successful",
- ImmutableList.of(row(1L, "a")),
- sql("SELECT * FROM %s", tableName));
- }
-
- @Test
- public void testCherrypickSnapshotUsingNamedArgs() {
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
- sql("ALTER TABLE %s SET TBLPROPERTIES ('%s' 'true')", tableName, WRITE_AUDIT_PUBLISH_ENABLED);
-
- spark.conf().set("spark.wap.id", "1");
-
- sql("INSERT INTO TABLE %s VALUES (1, 'a')", tableName);
-
- assertEquals(
- "Should not see rows from staged snapshot",
- ImmutableList.of(),
- sql("SELECT * FROM %s", tableName));
-
- Table table = validationCatalog.loadTable(tableIdent);
- Snapshot wapSnapshot = Iterables.getOnlyElement(table.snapshots());
-
- List output =
- sql(
- "CALL %s.system.cherrypick_snapshot(snapshot_id => %dL, table => '%s')",
- catalogName, wapSnapshot.snapshotId(), tableIdent);
-
- table.refresh();
-
- Snapshot currentSnapshot = table.currentSnapshot();
-
- assertEquals(
- "Procedure output must match",
- ImmutableList.of(row(wapSnapshot.snapshotId(), currentSnapshot.snapshotId())),
- output);
-
- assertEquals(
- "Cherrypick must be successful",
- ImmutableList.of(row(1L, "a")),
- sql("SELECT * FROM %s", tableName));
- }
-
- @Test
- public void testCherrypickSnapshotRefreshesRelationCache() {
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
- sql("ALTER TABLE %s SET TBLPROPERTIES ('%s' 'true')", tableName, WRITE_AUDIT_PUBLISH_ENABLED);
-
- Dataset query = spark.sql("SELECT * FROM " + tableName + " WHERE id = 1");
- query.createOrReplaceTempView("tmp");
-
- spark.sql("CACHE TABLE tmp");
-
- assertEquals("View should not produce rows", ImmutableList.of(), sql("SELECT * FROM tmp"));
-
- spark.conf().set("spark.wap.id", "1");
-
- sql("INSERT INTO TABLE %s VALUES (1, 'a')", tableName);
-
- assertEquals(
- "Should not see rows from staged snapshot",
- ImmutableList.of(),
- sql("SELECT * FROM %s", tableName));
-
- Table table = validationCatalog.loadTable(tableIdent);
- Snapshot wapSnapshot = Iterables.getOnlyElement(table.snapshots());
-
- sql(
- "CALL %s.system.cherrypick_snapshot('%s', %dL)",
- catalogName, tableIdent, wapSnapshot.snapshotId());
-
- assertEquals(
- "Cherrypick snapshot should be visible",
- ImmutableList.of(row(1L, "a")),
- sql("SELECT * FROM tmp"));
-
- sql("UNCACHE TABLE tmp");
- }
-
- @Test
- public void testCherrypickInvalidSnapshot() {
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
-
- AssertHelpers.assertThrows(
- "Should reject invalid snapshot id",
- ValidationException.class,
- "Cannot cherry-pick unknown snapshot ID",
- () -> sql("CALL %s.system.cherrypick_snapshot('%s', -1L)", catalogName, tableIdent));
- }
-
- @Test
- public void testInvalidCherrypickSnapshotCases() {
- AssertHelpers.assertThrows(
- "Should not allow mixed args",
- AnalysisException.class,
- "Named and positional arguments cannot be mixed",
- () -> sql("CALL %s.system.cherrypick_snapshot('n', table => 't', 1L)", catalogName));
-
- AssertHelpers.assertThrows(
- "Should not resolve procedures in arbitrary namespaces",
- NoSuchProcedureException.class,
- "not found",
- () -> sql("CALL %s.custom.cherrypick_snapshot('n', 't', 1L)", catalogName));
-
- AssertHelpers.assertThrows(
- "Should reject calls without all required args",
- AnalysisException.class,
- "Missing required parameters",
- () -> sql("CALL %s.system.cherrypick_snapshot('t')", catalogName));
-
- AssertHelpers.assertThrows(
- "Should reject calls with empty table identifier",
- IllegalArgumentException.class,
- "Cannot handle an empty identifier",
- () -> sql("CALL %s.system.cherrypick_snapshot('', 1L)", catalogName));
-
- AssertHelpers.assertThrows(
- "Should reject calls with invalid arg types",
- AnalysisException.class,
- "Wrong arg type for snapshot_id: cannot cast",
- () -> sql("CALL %s.system.cherrypick_snapshot('t', 2.2)", catalogName));
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestConflictValidation.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestConflictValidation.java
deleted file mode 100644
index 6fcbf1f903be..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestConflictValidation.java
+++ /dev/null
@@ -1,436 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import java.util.List;
-import java.util.Map;
-import org.apache.iceberg.AssertHelpers;
-import org.apache.iceberg.IsolationLevel;
-import org.apache.iceberg.Table;
-import org.apache.iceberg.exceptions.ValidationException;
-import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-import org.apache.iceberg.spark.SparkWriteOptions;
-import org.apache.iceberg.spark.source.SimpleRecord;
-import org.apache.spark.sql.Dataset;
-import org.apache.spark.sql.Row;
-import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
-import org.apache.spark.sql.functions;
-import org.junit.After;
-import org.junit.Before;
-import org.junit.Test;
-
-public class TestConflictValidation extends SparkExtensionsTestBase {
-
- public TestConflictValidation(
- String catalogName, String implementation, Map config) {
- super(catalogName, implementation, config);
- }
-
- @Before
- public void createTables() {
- sql(
- "CREATE TABLE %s (id int, data string) USING iceberg "
- + "PARTITIONED BY (id)"
- + "TBLPROPERTIES"
- + "('format-version'='2',"
- + "'write.delete.mode'='merge-on-read')",
- tableName);
- sql("INSERT INTO %s VALUES (1, 'a'), (2, 'b'), (3, 'c')", tableName);
- }
-
- @After
- public void removeTables() {
- sql("DROP TABLE IF EXISTS %s", tableName);
- }
-
- @Test
- public void testOverwriteFilterSerializableIsolation() throws Exception {
- Table table = validationCatalog.loadTable(tableIdent);
- long snapshotId = table.currentSnapshot().snapshotId();
-
- List records = Lists.newArrayList(new SimpleRecord(1, "a"));
- spark.createDataFrame(records, SimpleRecord.class).writeTo(tableName).append();
-
- // Validating from previous snapshot finds conflicts
- Dataset conflictingDf = spark.createDataFrame(records, SimpleRecord.class);
- AssertHelpers.assertThrowsCause(
- "Conflicting new data files should throw exception",
- ValidationException.class,
- "Found conflicting files that can contain records matching ref(name=\"id\") == 1:",
- () -> {
- try {
- conflictingDf
- .writeTo(tableName)
- .option(SparkWriteOptions.VALIDATE_FROM_SNAPSHOT_ID, String.valueOf(snapshotId))
- .option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SERIALIZABLE.toString())
- .overwrite(functions.col("id").equalTo(1));
- } catch (NoSuchTableException e) {
- throw new RuntimeException(e);
- }
- });
-
- // Validating from latest snapshot should succeed
- table.refresh();
- long newSnapshotId = table.currentSnapshot().snapshotId();
- conflictingDf
- .writeTo(tableName)
- .option(SparkWriteOptions.VALIDATE_FROM_SNAPSHOT_ID, String.valueOf(newSnapshotId))
- .option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SERIALIZABLE.toString())
- .overwrite(functions.col("id").equalTo(1));
- }
-
- @Test
- public void testOverwriteFilterSerializableIsolation2() throws Exception {
- List records =
- Lists.newArrayList(new SimpleRecord(1, "a"), new SimpleRecord(1, "b"));
- spark.createDataFrame(records, SimpleRecord.class).coalesce(1).writeTo(tableName).append();
-
- Table table = validationCatalog.loadTable(tableIdent);
- long snapshotId = table.currentSnapshot().snapshotId();
-
- // This should add a delete file
- sql("DELETE FROM %s WHERE id='1' and data='b'", tableName);
- table.refresh();
-
- // Validating from previous snapshot finds conflicts
- List conflictingRecords = Lists.newArrayList(new SimpleRecord(1, "a"));
- Dataset conflictingDf = spark.createDataFrame(conflictingRecords, SimpleRecord.class);
- AssertHelpers.assertThrowsCause(
- "Conflicting new delete files should throw exception",
- ValidationException.class,
- "Found new conflicting delete files that can apply to records matching ref(name=\"id\") == 1:",
- () -> {
- try {
- conflictingDf
- .writeTo(tableName)
- .option(SparkWriteOptions.VALIDATE_FROM_SNAPSHOT_ID, String.valueOf(snapshotId))
- .option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SNAPSHOT.toString())
- .overwrite(functions.col("id").equalTo(1));
- } catch (NoSuchTableException e) {
- throw new RuntimeException(e);
- }
- });
-
- // Validating from latest snapshot should succeed
- table.refresh();
- long newSnapshotId = table.currentSnapshot().snapshotId();
- conflictingDf
- .writeTo(tableName)
- .option(SparkWriteOptions.VALIDATE_FROM_SNAPSHOT_ID, String.valueOf(newSnapshotId))
- .option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SERIALIZABLE.toString())
- .overwrite(functions.col("id").equalTo(1));
- }
-
- @Test
- public void testOverwriteFilterSerializableIsolation3() throws Exception {
- Table table = validationCatalog.loadTable(tableIdent);
- long snapshotId = table.currentSnapshot().snapshotId();
-
- // This should delete a data file
- sql("DELETE FROM %s WHERE id='1'", tableName);
- table.refresh();
-
- // Validating from previous snapshot finds conflicts
- List conflictingRecords = Lists.newArrayList(new SimpleRecord(1, "a"));
- Dataset conflictingDf = spark.createDataFrame(conflictingRecords, SimpleRecord.class);
- AssertHelpers.assertThrowsCause(
- "Conflicting deleted data files should throw exception",
- ValidationException.class,
- "Found conflicting deleted files that can contain records matching ref(name=\"id\") == 1:",
- () -> {
- try {
- conflictingDf
- .writeTo(tableName)
- .option(SparkWriteOptions.VALIDATE_FROM_SNAPSHOT_ID, String.valueOf(snapshotId))
- .option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SERIALIZABLE.toString())
- .overwrite(functions.col("id").equalTo(1));
- } catch (NoSuchTableException e) {
- throw new RuntimeException(e);
- }
- });
-
- // Validating from latest snapshot should succeed
- table.refresh();
- long newSnapshotId = table.currentSnapshot().snapshotId();
- conflictingDf
- .writeTo(tableName)
- .option(SparkWriteOptions.VALIDATE_FROM_SNAPSHOT_ID, String.valueOf(newSnapshotId))
- .option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SERIALIZABLE.toString())
- .overwrite(functions.col("id").equalTo(1));
- }
-
- @Test
- public void testOverwriteFilterNoSnapshotIdValidation() throws Exception {
- Table table = validationCatalog.loadTable(tableIdent);
-
- List records = Lists.newArrayList(new SimpleRecord(1, "a"));
- spark.createDataFrame(records, SimpleRecord.class).writeTo(tableName).append();
-
- // Validating from no snapshot id defaults to beginning snapshot id and finds conflicts
- Dataset conflictingDf = spark.createDataFrame(records, SimpleRecord.class);
- AssertHelpers.assertThrowsCause(
- "Conflicting new data files should throw exception",
- ValidationException.class,
- "Found conflicting files that can contain records matching ref(name=\"id\") == 1:",
- () -> {
- try {
- conflictingDf
- .writeTo(tableName)
- .option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SERIALIZABLE.toString())
- .overwrite(functions.col("id").equalTo(1));
- } catch (NoSuchTableException e) {
- throw new RuntimeException(e);
- }
- });
-
- // Validating from latest snapshot should succeed
- table.refresh();
- long newSnapshotId = table.currentSnapshot().snapshotId();
- conflictingDf
- .writeTo(tableName)
- .option(SparkWriteOptions.VALIDATE_FROM_SNAPSHOT_ID, String.valueOf(newSnapshotId))
- .option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SERIALIZABLE.toString())
- .overwrite(functions.col("id").equalTo(1));
- }
-
- @Test
- public void testOverwriteFilterSnapshotIsolation() throws Exception {
- List records =
- Lists.newArrayList(new SimpleRecord(1, "a"), new SimpleRecord(1, "b"));
- spark.createDataFrame(records, SimpleRecord.class).coalesce(1).writeTo(tableName).append();
-
- Table table = validationCatalog.loadTable(tableIdent);
- long snapshotId = table.currentSnapshot().snapshotId();
-
- // This should add a delete file
- sql("DELETE FROM %s WHERE id='1' and data='b'", tableName);
- table.refresh();
-
- // Validating from previous snapshot finds conflicts
- List conflictingRecords = Lists.newArrayList(new SimpleRecord(1, "a"));
- Dataset conflictingDf = spark.createDataFrame(conflictingRecords, SimpleRecord.class);
- AssertHelpers.assertThrowsCause(
- "Conflicting new delete files should throw exception",
- ValidationException.class,
- "Found new conflicting delete files that can apply to records matching ref(name=\"id\") == 1:",
- () -> {
- try {
- conflictingDf
- .writeTo(tableName)
- .option(SparkWriteOptions.VALIDATE_FROM_SNAPSHOT_ID, String.valueOf(snapshotId))
- .option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SNAPSHOT.toString())
- .overwrite(functions.col("id").equalTo(1));
- } catch (NoSuchTableException e) {
- throw new RuntimeException(e);
- }
- });
-
- // Validating from latest snapshot should succeed
- table.refresh();
- long newSnapshotId = table.currentSnapshot().snapshotId();
- conflictingDf
- .writeTo(tableName)
- .option(SparkWriteOptions.VALIDATE_FROM_SNAPSHOT_ID, String.valueOf(newSnapshotId))
- .option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SNAPSHOT.toString())
- .overwrite(functions.col("id").equalTo(1));
- }
-
- @Test
- public void testOverwriteFilterSnapshotIsolation2() throws Exception {
- Table table = validationCatalog.loadTable(tableIdent);
- long snapshotId = table.currentSnapshot().snapshotId();
-
- List records = Lists.newArrayList(new SimpleRecord(1, "a"));
- spark.createDataFrame(records, SimpleRecord.class).writeTo(tableName).append();
-
- // Validation should not fail due to conflicting data file in snapshot isolation mode
- Dataset conflictingDf = spark.createDataFrame(records, SimpleRecord.class);
- conflictingDf
- .writeTo(tableName)
- .option(SparkWriteOptions.VALIDATE_FROM_SNAPSHOT_ID, String.valueOf(snapshotId))
- .option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SNAPSHOT.toString())
- .overwrite(functions.col("id").equalTo(1));
- }
-
- @Test
- public void testOverwritePartitionSerializableIsolation() throws Exception {
- Table table = validationCatalog.loadTable(tableIdent);
- final long snapshotId = table.currentSnapshot().snapshotId();
-
- List records = Lists.newArrayList(new SimpleRecord(1, "a"));
- spark.createDataFrame(records, SimpleRecord.class).writeTo(tableName).append();
-
- // Validating from previous snapshot finds conflicts
- Dataset conflictingDf = spark.createDataFrame(records, SimpleRecord.class);
- AssertHelpers.assertThrowsCause(
- "Conflicting deleted data files should throw exception",
- ValidationException.class,
- "Found conflicting files that can contain records matching partitions [id=1]",
- () -> {
- try {
- conflictingDf
- .writeTo(tableName)
- .option(SparkWriteOptions.VALIDATE_FROM_SNAPSHOT_ID, String.valueOf(snapshotId))
- .option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SERIALIZABLE.toString())
- .overwritePartitions();
- } catch (NoSuchTableException e) {
- throw new RuntimeException(e);
- }
- });
-
- // Validating from latest snapshot should succeed
- table.refresh();
- long newSnapshotId = table.currentSnapshot().snapshotId();
- conflictingDf
- .writeTo(tableName)
- .option(SparkWriteOptions.VALIDATE_FROM_SNAPSHOT_ID, String.valueOf(newSnapshotId))
- .option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SERIALIZABLE.toString())
- .overwritePartitions();
- }
-
- @Test
- public void testOverwritePartitionSnapshotIsolation() throws Exception {
- List records =
- Lists.newArrayList(new SimpleRecord(1, "a"), new SimpleRecord(1, "b"));
- spark.createDataFrame(records, SimpleRecord.class).coalesce(1).writeTo(tableName).append();
-
- Table table = validationCatalog.loadTable(tableIdent);
- final long snapshotId = table.currentSnapshot().snapshotId();
-
- // This should generate a delete file
- sql("DELETE FROM %s WHERE data='a'", tableName);
-
- // Validating from previous snapshot finds conflicts
- Dataset conflictingDf = spark.createDataFrame(records, SimpleRecord.class);
- AssertHelpers.assertThrowsCause(
- "Conflicting deleted data files should throw exception",
- ValidationException.class,
- "Found new conflicting delete files that can apply to records matching [id=1]",
- () -> {
- try {
- conflictingDf
- .writeTo(tableName)
- .option(SparkWriteOptions.VALIDATE_FROM_SNAPSHOT_ID, String.valueOf(snapshotId))
- .option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SNAPSHOT.toString())
- .overwritePartitions();
- } catch (NoSuchTableException e) {
- throw new RuntimeException(e);
- }
- });
-
- // Validating from latest snapshot should succeed
- table.refresh();
- long newSnapshotId = table.currentSnapshot().snapshotId();
- conflictingDf
- .writeTo(tableName)
- .option(SparkWriteOptions.VALIDATE_FROM_SNAPSHOT_ID, String.valueOf(newSnapshotId))
- .option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SNAPSHOT.toString())
- .overwritePartitions();
- }
-
- @Test
- public void testOverwritePartitionSnapshotIsolation2() throws Exception {
- Table table = validationCatalog.loadTable(tableIdent);
- final long snapshotId = table.currentSnapshot().snapshotId();
-
- // This should delete a data file
- sql("DELETE FROM %s WHERE id='1'", tableName);
-
- // Validating from previous snapshot finds conflicts
- List records = Lists.newArrayList(new SimpleRecord(1, "a"));
- spark.createDataFrame(records, SimpleRecord.class).coalesce(1).writeTo(tableName).append();
- Dataset conflictingDf = spark.createDataFrame(records, SimpleRecord.class);
-
- AssertHelpers.assertThrowsCause(
- "Conflicting deleted data files should throw exception",
- ValidationException.class,
- "Found conflicting deleted files that can apply to records matching [id=1]",
- () -> {
- try {
- conflictingDf
- .writeTo(tableName)
- .option(SparkWriteOptions.VALIDATE_FROM_SNAPSHOT_ID, String.valueOf(snapshotId))
- .option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SNAPSHOT.toString())
- .overwritePartitions();
- } catch (NoSuchTableException e) {
- throw new RuntimeException(e);
- }
- });
-
- // Validating from latest snapshot should succeed
- table.refresh();
- long newSnapshotId = table.currentSnapshot().snapshotId();
- conflictingDf
- .writeTo(tableName)
- .option(SparkWriteOptions.VALIDATE_FROM_SNAPSHOT_ID, String.valueOf(newSnapshotId))
- .option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SNAPSHOT.toString())
- .overwritePartitions();
- }
-
- @Test
- public void testOverwritePartitionSnapshotIsolation3() throws Exception {
- Table table = validationCatalog.loadTable(tableIdent);
- final long snapshotId = table.currentSnapshot().snapshotId();
-
- List records = Lists.newArrayList(new SimpleRecord(1, "a"));
- spark.createDataFrame(records, SimpleRecord.class).writeTo(tableName).append();
-
- // Validation should not find conflicting data file in snapshot isolation mode
- Dataset conflictingDf = spark.createDataFrame(records, SimpleRecord.class);
- conflictingDf
- .writeTo(tableName)
- .option(SparkWriteOptions.VALIDATE_FROM_SNAPSHOT_ID, String.valueOf(snapshotId))
- .option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SNAPSHOT.toString())
- .overwritePartitions();
- }
-
- @Test
- public void testOverwritePartitionNoSnapshotIdValidation() throws Exception {
- Table table = validationCatalog.loadTable(tableIdent);
-
- List records = Lists.newArrayList(new SimpleRecord(1, "a"));
- spark.createDataFrame(records, SimpleRecord.class).writeTo(tableName).append();
-
- // Validating from null snapshot is equivalent to validating from beginning
- Dataset conflictingDf = spark.createDataFrame(records, SimpleRecord.class);
- AssertHelpers.assertThrowsCause(
- "Conflicting deleted data files should throw exception",
- ValidationException.class,
- "Found conflicting files that can contain records matching partitions [id=1]",
- () -> {
- try {
- conflictingDf
- .writeTo(tableName)
- .option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SERIALIZABLE.toString())
- .overwritePartitions();
- } catch (NoSuchTableException e) {
- throw new RuntimeException(e);
- }
- });
-
- // Validating from latest snapshot should succeed
- table.refresh();
- long snapshotId = table.currentSnapshot().snapshotId();
- conflictingDf
- .writeTo(tableName)
- .option(SparkWriteOptions.VALIDATE_FROM_SNAPSHOT_ID, String.valueOf(snapshotId))
- .option(SparkWriteOptions.ISOLATION_LEVEL, IsolationLevel.SERIALIZABLE.toString())
- .overwritePartitions();
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteDelete.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteDelete.java
deleted file mode 100644
index 22dbd9df5efa..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteDelete.java
+++ /dev/null
@@ -1,146 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import static org.apache.iceberg.TableProperties.DELETE_ISOLATION_LEVEL;
-
-import java.util.Collections;
-import java.util.Map;
-import java.util.concurrent.ExecutionException;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.Future;
-import java.util.concurrent.ThreadPoolExecutor;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicBoolean;
-import java.util.concurrent.atomic.AtomicInteger;
-import org.apache.iceberg.DataFile;
-import org.apache.iceberg.RowLevelOperationMode;
-import org.apache.iceberg.Table;
-import org.apache.iceberg.TableProperties;
-import org.apache.iceberg.data.GenericRecord;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
-import org.apache.iceberg.relocated.com.google.common.util.concurrent.MoreExecutors;
-import org.apache.iceberg.spark.Spark3Util;
-import org.apache.spark.sql.Encoders;
-import org.assertj.core.api.Assertions;
-import org.junit.Assert;
-import org.junit.Assume;
-import org.junit.Test;
-
-public class TestCopyOnWriteDelete extends TestDelete {
-
- public TestCopyOnWriteDelete(
- String catalogName,
- String implementation,
- Map config,
- String fileFormat,
- Boolean vectorized,
- String distributionMode) {
- super(catalogName, implementation, config, fileFormat, vectorized, distributionMode);
- }
-
- @Override
- protected Map extraTableProperties() {
- return ImmutableMap.of(
- TableProperties.DELETE_MODE, RowLevelOperationMode.COPY_ON_WRITE.modeName());
- }
-
- @Test
- public synchronized void testDeleteWithConcurrentTableRefresh() throws Exception {
- // this test can only be run with Hive tables as it requires a reliable lock
- // also, the table cache must be enabled so that the same table instance can be reused
- Assume.assumeTrue(catalogName.equalsIgnoreCase("testhive"));
-
- createAndInitUnpartitionedTable();
- createOrReplaceView("deleted_id", Collections.singletonList(1), Encoders.INT());
-
- sql(
- "ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')",
- tableName, DELETE_ISOLATION_LEVEL, "snapshot");
-
- sql("INSERT INTO TABLE %s VALUES (1, 'hr')", tableName);
-
- Table table = Spark3Util.loadIcebergTable(spark, tableName);
-
- ExecutorService executorService =
- MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) Executors.newFixedThreadPool(2));
-
- AtomicInteger barrier = new AtomicInteger(0);
- AtomicBoolean shouldAppend = new AtomicBoolean(true);
-
- // delete thread
- Future deleteFuture =
- executorService.submit(
- () -> {
- for (int numOperations = 0; numOperations < Integer.MAX_VALUE; numOperations++) {
- while (barrier.get() < numOperations * 2) {
- sleep(10);
- }
-
- sql("DELETE FROM %s WHERE id IN (SELECT * FROM deleted_id)", tableName);
-
- barrier.incrementAndGet();
- }
- });
-
- // append thread
- Future appendFuture =
- executorService.submit(
- () -> {
- GenericRecord record = GenericRecord.create(table.schema());
- record.set(0, 1); // id
- record.set(1, "hr"); // dep
-
- for (int numOperations = 0; numOperations < Integer.MAX_VALUE; numOperations++) {
- while (shouldAppend.get() && barrier.get() < numOperations * 2) {
- sleep(10);
- }
-
- if (!shouldAppend.get()) {
- return;
- }
-
- for (int numAppends = 0; numAppends < 5; numAppends++) {
- DataFile dataFile = writeDataFile(table, ImmutableList.of(record));
- table.newFastAppend().appendFile(dataFile).commit();
- sleep(10);
- }
-
- barrier.incrementAndGet();
- }
- });
-
- try {
- Assertions.assertThatThrownBy(deleteFuture::get)
- .isInstanceOf(ExecutionException.class)
- .cause()
- .isInstanceOf(IllegalStateException.class)
- .hasMessageContaining("the table has been concurrently modified");
- } finally {
- shouldAppend.set(false);
- appendFuture.cancel(true);
- }
-
- executorService.shutdown();
- Assert.assertTrue("Timeout", executorService.awaitTermination(2, TimeUnit.MINUTES));
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteMerge.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteMerge.java
deleted file mode 100644
index ebb445d2553b..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteMerge.java
+++ /dev/null
@@ -1,151 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import static org.apache.iceberg.TableProperties.MERGE_ISOLATION_LEVEL;
-
-import java.util.Collections;
-import java.util.Map;
-import java.util.concurrent.ExecutionException;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.Future;
-import java.util.concurrent.ThreadPoolExecutor;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicBoolean;
-import java.util.concurrent.atomic.AtomicInteger;
-import org.apache.iceberg.DataFile;
-import org.apache.iceberg.RowLevelOperationMode;
-import org.apache.iceberg.Table;
-import org.apache.iceberg.TableProperties;
-import org.apache.iceberg.data.GenericRecord;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
-import org.apache.iceberg.relocated.com.google.common.util.concurrent.MoreExecutors;
-import org.apache.iceberg.spark.Spark3Util;
-import org.apache.spark.sql.Encoders;
-import org.assertj.core.api.Assertions;
-import org.junit.Assert;
-import org.junit.Assume;
-import org.junit.Test;
-
-public class TestCopyOnWriteMerge extends TestMerge {
-
- public TestCopyOnWriteMerge(
- String catalogName,
- String implementation,
- Map config,
- String fileFormat,
- boolean vectorized,
- String distributionMode) {
- super(catalogName, implementation, config, fileFormat, vectorized, distributionMode);
- }
-
- @Override
- protected Map extraTableProperties() {
- return ImmutableMap.of(
- TableProperties.MERGE_MODE, RowLevelOperationMode.COPY_ON_WRITE.modeName());
- }
-
- @Test
- public synchronized void testMergeWithConcurrentTableRefresh() throws Exception {
- // this test can only be run with Hive tables as it requires a reliable lock
- // also, the table cache must be enabled so that the same table instance can be reused
- Assume.assumeTrue(catalogName.equalsIgnoreCase("testhive"));
-
- createAndInitTable("id INT, dep STRING");
- createOrReplaceView("source", Collections.singletonList(1), Encoders.INT());
-
- sql(
- "ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')",
- tableName, MERGE_ISOLATION_LEVEL, "snapshot");
-
- sql("INSERT INTO TABLE %s VALUES (1, 'hr')", tableName);
-
- Table table = Spark3Util.loadIcebergTable(spark, tableName);
-
- ExecutorService executorService =
- MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) Executors.newFixedThreadPool(2));
-
- AtomicInteger barrier = new AtomicInteger(0);
- AtomicBoolean shouldAppend = new AtomicBoolean(true);
-
- // merge thread
- Future mergeFuture =
- executorService.submit(
- () -> {
- for (int numOperations = 0; numOperations < Integer.MAX_VALUE; numOperations++) {
- while (barrier.get() < numOperations * 2) {
- sleep(10);
- }
-
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.value "
- + "WHEN MATCHED THEN "
- + " UPDATE SET dep = 'x'",
- tableName);
-
- barrier.incrementAndGet();
- }
- });
-
- // append thread
- Future appendFuture =
- executorService.submit(
- () -> {
- GenericRecord record = GenericRecord.create(table.schema());
- record.set(0, 1); // id
- record.set(1, "hr"); // dep
-
- for (int numOperations = 0; numOperations < Integer.MAX_VALUE; numOperations++) {
- while (shouldAppend.get() && barrier.get() < numOperations * 2) {
- sleep(10);
- }
-
- if (!shouldAppend.get()) {
- return;
- }
-
- for (int numAppends = 0; numAppends < 5; numAppends++) {
- DataFile dataFile = writeDataFile(table, ImmutableList.of(record));
- table.newFastAppend().appendFile(dataFile).commit();
- sleep(10);
- }
-
- barrier.incrementAndGet();
- }
- });
-
- try {
- Assertions.assertThatThrownBy(mergeFuture::get)
- .isInstanceOf(ExecutionException.class)
- .cause()
- .isInstanceOf(IllegalStateException.class)
- .hasMessageContaining("the table has been concurrently modified");
- } finally {
- shouldAppend.set(false);
- appendFuture.cancel(true);
- }
-
- executorService.shutdown();
- Assert.assertTrue("Timeout", executorService.awaitTermination(2, TimeUnit.MINUTES));
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteUpdate.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteUpdate.java
deleted file mode 100644
index b8c7bab78851..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteUpdate.java
+++ /dev/null
@@ -1,143 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import static org.apache.iceberg.TableProperties.UPDATE_ISOLATION_LEVEL;
-
-import java.util.Map;
-import java.util.concurrent.ExecutionException;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.Future;
-import java.util.concurrent.ThreadPoolExecutor;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicBoolean;
-import java.util.concurrent.atomic.AtomicInteger;
-import org.apache.iceberg.DataFile;
-import org.apache.iceberg.RowLevelOperationMode;
-import org.apache.iceberg.Table;
-import org.apache.iceberg.TableProperties;
-import org.apache.iceberg.data.GenericRecord;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
-import org.apache.iceberg.relocated.com.google.common.util.concurrent.MoreExecutors;
-import org.apache.iceberg.spark.Spark3Util;
-import org.assertj.core.api.Assertions;
-import org.junit.Assert;
-import org.junit.Assume;
-import org.junit.Test;
-
-public class TestCopyOnWriteUpdate extends TestUpdate {
-
- public TestCopyOnWriteUpdate(
- String catalogName,
- String implementation,
- Map config,
- String fileFormat,
- boolean vectorized,
- String distributionMode) {
- super(catalogName, implementation, config, fileFormat, vectorized, distributionMode);
- }
-
- @Override
- protected Map extraTableProperties() {
- return ImmutableMap.of(
- TableProperties.UPDATE_MODE, RowLevelOperationMode.COPY_ON_WRITE.modeName());
- }
-
- @Test
- public synchronized void testUpdateWithConcurrentTableRefresh() throws Exception {
- // this test can only be run with Hive tables as it requires a reliable lock
- // also, the table cache must be enabled so that the same table instance can be reused
- Assume.assumeTrue(catalogName.equalsIgnoreCase("testhive"));
-
- createAndInitTable("id INT, dep STRING");
-
- sql(
- "ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')",
- tableName, UPDATE_ISOLATION_LEVEL, "snapshot");
-
- sql("INSERT INTO TABLE %s VALUES (1, 'hr')", tableName);
-
- Table table = Spark3Util.loadIcebergTable(spark, tableName);
-
- ExecutorService executorService =
- MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) Executors.newFixedThreadPool(2));
-
- AtomicInteger barrier = new AtomicInteger(0);
- AtomicBoolean shouldAppend = new AtomicBoolean(true);
-
- // update thread
- Future updateFuture =
- executorService.submit(
- () -> {
- for (int numOperations = 0; numOperations < Integer.MAX_VALUE; numOperations++) {
- while (barrier.get() < numOperations * 2) {
- sleep(10);
- }
-
- sql("UPDATE %s SET id = -1 WHERE id = 1", tableName);
-
- barrier.incrementAndGet();
- }
- });
-
- // append thread
- Future appendFuture =
- executorService.submit(
- () -> {
- GenericRecord record = GenericRecord.create(table.schema());
- record.set(0, 1); // id
- record.set(1, "hr"); // dep
-
- for (int numOperations = 0; numOperations < Integer.MAX_VALUE; numOperations++) {
- while (shouldAppend.get() && barrier.get() < numOperations * 2) {
- sleep(10);
- }
-
- if (!shouldAppend.get()) {
- return;
- }
-
- for (int numAppends = 0; numAppends < 5; numAppends++) {
- DataFile dataFile = writeDataFile(table, ImmutableList.of(record));
- table.newFastAppend().appendFile(dataFile).commit();
- sleep(10);
- }
-
- barrier.incrementAndGet();
- }
- });
-
- try {
- Assertions.assertThatThrownBy(updateFuture::get)
- .isInstanceOf(ExecutionException.class)
- .cause()
- .isInstanceOf(IllegalStateException.class)
- .hasMessageContaining("the table has been concurrently modified");
- } finally {
- shouldAppend.set(false);
- appendFuture.cancel(true);
- }
-
- executorService.shutdown();
- Assert.assertTrue("Timeout", executorService.awaitTermination(2, TimeUnit.MINUTES));
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCreateChangelogViewProcedure.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCreateChangelogViewProcedure.java
deleted file mode 100644
index 1e00153e35a9..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCreateChangelogViewProcedure.java
+++ /dev/null
@@ -1,447 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import java.util.List;
-import java.util.Map;
-import org.apache.iceberg.ChangelogOperation;
-import org.apache.iceberg.Snapshot;
-import org.apache.iceberg.Table;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
-import org.apache.iceberg.spark.SparkReadOptions;
-import org.junit.After;
-import org.junit.Assert;
-import org.junit.Test;
-
-public class TestCreateChangelogViewProcedure extends SparkExtensionsTestBase {
- private static final String DELETE = ChangelogOperation.DELETE.name();
- private static final String INSERT = ChangelogOperation.INSERT.name();
- private static final String UPDATE_BEFORE = ChangelogOperation.UPDATE_BEFORE.name();
- private static final String UPDATE_AFTER = ChangelogOperation.UPDATE_AFTER.name();
-
- public TestCreateChangelogViewProcedure(
- String catalogName, String implementation, Map config) {
- super(catalogName, implementation, config);
- }
-
- @After
- public void removeTable() {
- sql("DROP TABLE IF EXISTS %s", tableName);
- }
-
- public void createTableWith2Columns() {
- sql("CREATE TABLE %s (id INT, data STRING) USING iceberg", tableName);
- sql("ALTER TABLE %s ADD PARTITION FIELD data", tableName);
- }
-
- private void createTableWith3Columns() {
- sql("CREATE TABLE %s (id INT, data STRING, age INT) USING iceberg", tableName);
- sql("ALTER TABLE %s ADD PARTITION FIELD id", tableName);
- }
-
- private void createTableWithIdentifierField() {
- sql("CREATE TABLE %s (id INT NOT NULL, data STRING) USING iceberg", tableName);
- sql("ALTER TABLE %s SET IDENTIFIER FIELDS id", tableName);
- }
-
- @Test
- public void testCustomizedViewName() {
- createTableWith2Columns();
- sql("INSERT INTO %s VALUES (1, 'a')", tableName);
- sql("INSERT INTO %s VALUES (2, 'b')", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
-
- Snapshot snap1 = table.currentSnapshot();
-
- sql("INSERT OVERWRITE %s VALUES (-2, 'b')", tableName);
-
- table.refresh();
-
- Snapshot snap2 = table.currentSnapshot();
-
- sql(
- "CALL %s.system.create_changelog_view("
- + "table => '%s',"
- + "options => map('%s','%s','%s','%s'),"
- + "changelog_view => '%s')",
- catalogName,
- tableName,
- SparkReadOptions.START_SNAPSHOT_ID,
- snap1.snapshotId(),
- SparkReadOptions.END_SNAPSHOT_ID,
- snap2.snapshotId(),
- "cdc_view");
-
- long rowCount = sql("select * from %s", "cdc_view").stream().count();
- Assert.assertEquals(2, rowCount);
- }
-
- @Test
- public void testNoSnapshotIdInput() {
- createTableWith2Columns();
- sql("INSERT INTO %s VALUES (1, 'a')", tableName);
- Table table = validationCatalog.loadTable(tableIdent);
- Snapshot snap0 = table.currentSnapshot();
-
- sql("INSERT INTO %s VALUES (2, 'b')", tableName);
- table.refresh();
- Snapshot snap1 = table.currentSnapshot();
-
- sql("INSERT OVERWRITE %s VALUES (-2, 'b')", tableName);
- table.refresh();
- Snapshot snap2 = table.currentSnapshot();
-
- List returns =
- sql(
- "CALL %s.system.create_changelog_view(" + "table => '%s')",
- catalogName, tableName, "cdc_view");
-
- String viewName = (String) returns.get(0)[0];
- assertEquals(
- "Rows should match",
- ImmutableList.of(
- row(1, "a", INSERT, 0, snap0.snapshotId()),
- row(2, "b", INSERT, 1, snap1.snapshotId()),
- row(-2, "b", INSERT, 2, snap2.snapshotId()),
- row(2, "b", DELETE, 2, snap2.snapshotId())),
- sql("select * from %s order by _change_ordinal, id", viewName));
- }
-
- @Test
- public void testTimestampsBasedQuery() {
- createTableWith2Columns();
- long beginning = System.currentTimeMillis();
-
- sql("INSERT INTO %s VALUES (1, 'a')", tableName);
- Table table = validationCatalog.loadTable(tableIdent);
- Snapshot snap0 = table.currentSnapshot();
- long afterFirstInsert = waitUntilAfter(snap0.timestampMillis());
-
- sql("INSERT INTO %s VALUES (2, 'b')", tableName);
- table.refresh();
- Snapshot snap1 = table.currentSnapshot();
-
- sql("INSERT OVERWRITE %s VALUES (-2, 'b')", tableName);
- table.refresh();
- Snapshot snap2 = table.currentSnapshot();
-
- long afterInsertOverwrite = waitUntilAfter(snap2.timestampMillis());
- List returns =
- sql(
- "CALL %s.system.create_changelog_view(table => '%s', "
- + "options => map('%s', '%s','%s', '%s'))",
- catalogName,
- tableName,
- SparkReadOptions.START_TIMESTAMP,
- beginning,
- SparkReadOptions.END_TIMESTAMP,
- afterInsertOverwrite);
-
- assertEquals(
- "Rows should match",
- ImmutableList.of(
- row(1, "a", INSERT, 0, snap0.snapshotId()),
- row(2, "b", INSERT, 1, snap1.snapshotId()),
- row(-2, "b", INSERT, 2, snap2.snapshotId()),
- row(2, "b", DELETE, 2, snap2.snapshotId())),
- sql("select * from %s order by _change_ordinal, id", returns.get(0)[0]));
-
- // query the timestamps starting from the second insert
- returns =
- sql(
- "CALL %s.system.create_changelog_view(table => '%s', "
- + "options => map('%s', '%s', '%s', '%s'))",
- catalogName,
- tableName,
- SparkReadOptions.START_TIMESTAMP,
- afterFirstInsert,
- SparkReadOptions.END_TIMESTAMP,
- afterInsertOverwrite);
-
- assertEquals(
- "Rows should match",
- ImmutableList.of(
- row(2, "b", INSERT, 0, snap1.snapshotId()),
- row(-2, "b", INSERT, 1, snap2.snapshotId()),
- row(2, "b", DELETE, 1, snap2.snapshotId())),
- sql("select * from %s order by _change_ordinal, id", returns.get(0)[0]));
- }
-
- @Test
- public void testWithCarryovers() {
- createTableWith2Columns();
- sql("INSERT INTO %s VALUES (1, 'a')", tableName);
- Table table = validationCatalog.loadTable(tableIdent);
- Snapshot snap0 = table.currentSnapshot();
-
- sql("INSERT INTO %s VALUES (2, 'b')", tableName);
- table.refresh();
- Snapshot snap1 = table.currentSnapshot();
-
- sql("INSERT OVERWRITE %s VALUES (-2, 'b'), (2, 'b'), (2, 'b')", tableName);
- table.refresh();
- Snapshot snap2 = table.currentSnapshot();
-
- List returns =
- sql(
- "CALL %s.system.create_changelog_view("
- + "remove_carryovers => false,"
- + "table => '%s')",
- catalogName, tableName, "cdc_view");
-
- String viewName = (String) returns.get(0)[0];
- assertEquals(
- "Rows should match",
- ImmutableList.of(
- row(1, "a", INSERT, 0, snap0.snapshotId()),
- row(2, "b", INSERT, 1, snap1.snapshotId()),
- row(-2, "b", INSERT, 2, snap2.snapshotId()),
- row(2, "b", DELETE, 2, snap2.snapshotId()),
- row(2, "b", INSERT, 2, snap2.snapshotId()),
- row(2, "b", INSERT, 2, snap2.snapshotId())),
- sql("select * from %s order by _change_ordinal, id, _change_type", viewName));
- }
-
- @Test
- public void testUpdate() {
- createTableWith2Columns();
- sql("ALTER TABLE %s DROP PARTITION FIELD data", tableName);
- sql("ALTER TABLE %s ADD PARTITION FIELD id", tableName);
-
- sql("INSERT INTO %s VALUES (1, 'a'), (2, 'b')", tableName);
- Table table = validationCatalog.loadTable(tableIdent);
- Snapshot snap1 = table.currentSnapshot();
-
- sql("INSERT OVERWRITE %s VALUES (3, 'c'), (2, 'd')", tableName);
- table.refresh();
- Snapshot snap2 = table.currentSnapshot();
-
- List returns =
- sql(
- "CALL %s.system.create_changelog_view(table => '%s', identifier_columns => array('id'))",
- catalogName, tableName);
-
- String viewName = (String) returns.get(0)[0];
- assertEquals(
- "Rows should match",
- ImmutableList.of(
- row(1, "a", INSERT, 0, snap1.snapshotId()),
- row(2, "b", INSERT, 0, snap1.snapshotId()),
- row(2, "b", UPDATE_BEFORE, 1, snap2.snapshotId()),
- row(2, "d", UPDATE_AFTER, 1, snap2.snapshotId()),
- row(3, "c", INSERT, 1, snap2.snapshotId())),
- sql("select * from %s order by _change_ordinal, id, data", viewName));
- }
-
- @Test
- public void testUpdateWithIdentifierField() {
- createTableWithIdentifierField();
-
- sql("INSERT INTO %s VALUES (2, 'b')", tableName);
- Table table = validationCatalog.loadTable(tableIdent);
- Snapshot snap1 = table.currentSnapshot();
-
- sql("INSERT OVERWRITE %s VALUES (3, 'c'), (2, 'd')", tableName);
- table.refresh();
- Snapshot snap2 = table.currentSnapshot();
-
- List returns =
- sql(
- "CALL %s.system.create_changelog_view(table => '%s', compute_updates => true)",
- catalogName, tableName);
-
- String viewName = (String) returns.get(0)[0];
- assertEquals(
- "Rows should match",
- ImmutableList.of(
- row(2, "b", INSERT, 0, snap1.snapshotId()),
- row(2, "b", UPDATE_BEFORE, 1, snap2.snapshotId()),
- row(2, "d", UPDATE_AFTER, 1, snap2.snapshotId()),
- row(3, "c", INSERT, 1, snap2.snapshotId())),
- sql("select * from %s order by _change_ordinal, id, data", viewName));
- }
-
- @Test
- public void testUpdateWithFilter() {
- createTableWith2Columns();
- sql("ALTER TABLE %s DROP PARTITION FIELD data", tableName);
- sql("ALTER TABLE %s ADD PARTITION FIELD id", tableName);
-
- sql("INSERT INTO %s VALUES (1, 'a'), (2, 'b')", tableName);
- Table table = validationCatalog.loadTable(tableIdent);
- Snapshot snap1 = table.currentSnapshot();
-
- sql("INSERT OVERWRITE %s VALUES (3, 'c'), (2, 'd')", tableName);
- table.refresh();
- Snapshot snap2 = table.currentSnapshot();
-
- List returns =
- sql(
- "CALL %s.system.create_changelog_view(table => '%s', identifier_columns => array('id'))",
- catalogName, tableName);
-
- String viewName = (String) returns.get(0)[0];
- assertEquals(
- "Rows should match",
- ImmutableList.of(
- row(1, "a", INSERT, 0, snap1.snapshotId()),
- row(2, "b", INSERT, 0, snap1.snapshotId()),
- row(2, "b", UPDATE_BEFORE, 1, snap2.snapshotId()),
- row(2, "d", UPDATE_AFTER, 1, snap2.snapshotId())),
- // the predicate on partition columns will filter out the insert of (3, 'c') at the planning
- // phase
- sql("select * from %s where id != 3 order by _change_ordinal, id, data", viewName));
- }
-
- @Test
- public void testUpdateWithMultipleIdentifierColumns() {
- createTableWith3Columns();
-
- sql("INSERT INTO %s VALUES (1, 'a', 12), (2, 'b', 11)", tableName);
- Table table = validationCatalog.loadTable(tableIdent);
- Snapshot snap1 = table.currentSnapshot();
-
- sql("INSERT OVERWRITE %s VALUES (3, 'c', 13), (2, 'd', 11), (2, 'e', 12)", tableName);
- table.refresh();
- Snapshot snap2 = table.currentSnapshot();
-
- List returns =
- sql(
- "CALL %s.system.create_changelog_view("
- + "identifier_columns => array('id','age'),"
- + "table => '%s')",
- catalogName, tableName);
-
- String viewName = (String) returns.get(0)[0];
- assertEquals(
- "Rows should match",
- ImmutableList.of(
- row(1, "a", 12, INSERT, 0, snap1.snapshotId()),
- row(2, "b", 11, INSERT, 0, snap1.snapshotId()),
- row(2, "b", 11, UPDATE_BEFORE, 1, snap2.snapshotId()),
- row(2, "d", 11, UPDATE_AFTER, 1, snap2.snapshotId()),
- row(2, "e", 12, INSERT, 1, snap2.snapshotId()),
- row(3, "c", 13, INSERT, 1, snap2.snapshotId())),
- sql("select * from %s order by _change_ordinal, id, data", viewName));
- }
-
- @Test
- public void testRemoveCarryOvers() {
- createTableWith3Columns();
-
- sql("INSERT INTO %s VALUES (1, 'a', 12), (2, 'b', 11), (2, 'e', 12)", tableName);
- Table table = validationCatalog.loadTable(tableIdent);
- Snapshot snap1 = table.currentSnapshot();
-
- // carry-over row (2, 'e', 12)
- sql("INSERT OVERWRITE %s VALUES (3, 'c', 13), (2, 'd', 11), (2, 'e', 12)", tableName);
- table.refresh();
- Snapshot snap2 = table.currentSnapshot();
-
- List returns =
- sql(
- "CALL %s.system.create_changelog_view("
- + "identifier_columns => array('id','age'), "
- + "table => '%s')",
- catalogName, tableName);
-
- String viewName = (String) returns.get(0)[0];
- // the carry-over rows (2, 'e', 12, 'DELETE', 1), (2, 'e', 12, 'INSERT', 1) are removed
- assertEquals(
- "Rows should match",
- ImmutableList.of(
- row(1, "a", 12, INSERT, 0, snap1.snapshotId()),
- row(2, "b", 11, INSERT, 0, snap1.snapshotId()),
- row(2, "e", 12, INSERT, 0, snap1.snapshotId()),
- row(2, "b", 11, UPDATE_BEFORE, 1, snap2.snapshotId()),
- row(2, "d", 11, UPDATE_AFTER, 1, snap2.snapshotId()),
- row(3, "c", 13, INSERT, 1, snap2.snapshotId())),
- sql("select * from %s order by _change_ordinal, id, data", viewName));
- }
-
- @Test
- public void testRemoveCarryOversWithoutUpdatedRows() {
- createTableWith3Columns();
-
- sql("INSERT INTO %s VALUES (1, 'a', 12), (2, 'b', 11), (2, 'e', 12)", tableName);
- Table table = validationCatalog.loadTable(tableIdent);
- Snapshot snap1 = table.currentSnapshot();
-
- // carry-over row (2, 'e', 12)
- sql("INSERT OVERWRITE %s VALUES (3, 'c', 13), (2, 'd', 11), (2, 'e', 12)", tableName);
- table.refresh();
- Snapshot snap2 = table.currentSnapshot();
-
- List returns =
- sql("CALL %s.system.create_changelog_view(table => '%s')", catalogName, tableName);
-
- String viewName = (String) returns.get(0)[0];
-
- // the carry-over rows (2, 'e', 12, 'DELETE', 1), (2, 'e', 12, 'INSERT', 1) are removed, even
- // though update-row is not computed
- assertEquals(
- "Rows should match",
- ImmutableList.of(
- row(1, "a", 12, INSERT, 0, snap1.snapshotId()),
- row(2, "b", 11, INSERT, 0, snap1.snapshotId()),
- row(2, "e", 12, INSERT, 0, snap1.snapshotId()),
- row(2, "b", 11, DELETE, 1, snap2.snapshotId()),
- row(2, "d", 11, INSERT, 1, snap2.snapshotId()),
- row(3, "c", 13, INSERT, 1, snap2.snapshotId())),
- sql("select * from %s order by _change_ordinal, id, data", viewName));
- }
-
- @Test
- public void testNotRemoveCarryOvers() {
- createTableWith3Columns();
-
- sql("INSERT INTO %s VALUES (1, 'a', 12), (2, 'b', 11), (2, 'e', 12)", tableName);
- Table table = validationCatalog.loadTable(tableIdent);
- Snapshot snap1 = table.currentSnapshot();
-
- // carry-over row (2, 'e', 12)
- sql("INSERT OVERWRITE %s VALUES (3, 'c', 13), (2, 'd', 11), (2, 'e', 12)", tableName);
- table.refresh();
- Snapshot snap2 = table.currentSnapshot();
-
- List returns =
- sql(
- "CALL %s.system.create_changelog_view("
- + "remove_carryovers => false,"
- + "table => '%s')",
- catalogName, tableName);
-
- String viewName = (String) returns.get(0)[0];
-
- assertEquals(
- "Rows should match",
- ImmutableList.of(
- row(1, "a", 12, INSERT, 0, snap1.snapshotId()),
- row(2, "b", 11, INSERT, 0, snap1.snapshotId()),
- row(2, "e", 12, INSERT, 0, snap1.snapshotId()),
- row(2, "b", 11, DELETE, 1, snap2.snapshotId()),
- row(2, "d", 11, INSERT, 1, snap2.snapshotId()),
- // the following two rows are carry-over rows
- row(2, "e", 12, DELETE, 1, snap2.snapshotId()),
- row(2, "e", 12, INSERT, 1, snap2.snapshotId()),
- row(3, "c", 13, INSERT, 1, snap2.snapshotId())),
- sql("select * from %s order by _change_ordinal, id, data, _change_type", viewName));
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java
deleted file mode 100644
index 46d38235293f..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java
+++ /dev/null
@@ -1,1072 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import static org.apache.iceberg.RowLevelOperationMode.COPY_ON_WRITE;
-import static org.apache.iceberg.TableProperties.DELETE_ISOLATION_LEVEL;
-import static org.apache.iceberg.TableProperties.DELETE_MODE;
-import static org.apache.iceberg.TableProperties.DELETE_MODE_DEFAULT;
-import static org.apache.iceberg.TableProperties.PARQUET_ROW_GROUP_SIZE_BYTES;
-import static org.apache.iceberg.TableProperties.SPLIT_SIZE;
-import static org.apache.spark.sql.functions.lit;
-
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.List;
-import java.util.Map;
-import java.util.concurrent.ExecutionException;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.Future;
-import java.util.concurrent.ThreadPoolExecutor;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicBoolean;
-import java.util.concurrent.atomic.AtomicInteger;
-import java.util.stream.Collectors;
-import org.apache.iceberg.AssertHelpers;
-import org.apache.iceberg.DataFile;
-import org.apache.iceberg.ManifestFile;
-import org.apache.iceberg.RowLevelOperationMode;
-import org.apache.iceberg.Snapshot;
-import org.apache.iceberg.Table;
-import org.apache.iceberg.data.GenericRecord;
-import org.apache.iceberg.exceptions.ValidationException;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
-import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
-import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-import org.apache.iceberg.relocated.com.google.common.util.concurrent.MoreExecutors;
-import org.apache.iceberg.spark.Spark3Util;
-import org.apache.iceberg.spark.data.TestHelpers;
-import org.apache.spark.SparkException;
-import org.apache.spark.sql.AnalysisException;
-import org.apache.spark.sql.Dataset;
-import org.apache.spark.sql.Encoders;
-import org.apache.spark.sql.Row;
-import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
-import org.apache.spark.sql.catalyst.parser.ParseException;
-import org.apache.spark.sql.catalyst.plans.logical.DeleteFromIcebergTable;
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan;
-import org.apache.spark.sql.execution.datasources.v2.OptimizeMetadataOnlyDeleteFromTable;
-import org.apache.spark.sql.internal.SQLConf;
-import org.assertj.core.api.Assertions;
-import org.junit.After;
-import org.junit.Assert;
-import org.junit.Assume;
-import org.junit.BeforeClass;
-import org.junit.Test;
-
-public abstract class TestDelete extends SparkRowLevelOperationsTestBase {
-
- public TestDelete(
- String catalogName,
- String implementation,
- Map config,
- String fileFormat,
- Boolean vectorized,
- String distributionMode) {
- super(catalogName, implementation, config, fileFormat, vectorized, distributionMode);
- }
-
- @BeforeClass
- public static void setupSparkConf() {
- spark.conf().set("spark.sql.shuffle.partitions", "4");
- }
-
- @After
- public void removeTables() {
- sql("DROP TABLE IF EXISTS %s", tableName);
- sql("DROP TABLE IF EXISTS deleted_id");
- sql("DROP TABLE IF EXISTS deleted_dep");
- }
-
- @Test
- public void testDeleteWithoutScanningTable() throws Exception {
- createAndInitPartitionedTable();
-
- append(new Employee(1, "hr"), new Employee(3, "hr"));
- append(new Employee(1, "hardware"), new Employee(2, "hardware"));
-
- Table table = validationCatalog.loadTable(tableIdent);
-
- List manifestLocations =
- table.currentSnapshot().allManifests(table.io()).stream()
- .map(ManifestFile::path)
- .collect(Collectors.toList());
-
- withUnavailableLocations(
- manifestLocations,
- () -> {
- LogicalPlan parsed = parsePlan("DELETE FROM %s WHERE dep = 'hr'", tableName);
-
- DeleteFromIcebergTable analyzed =
- (DeleteFromIcebergTable) spark.sessionState().analyzer().execute(parsed);
- Assert.assertTrue("Should have rewrite plan", analyzed.rewritePlan().isDefined());
-
- DeleteFromIcebergTable optimized =
- (DeleteFromIcebergTable) OptimizeMetadataOnlyDeleteFromTable.apply(analyzed);
- Assert.assertTrue("Should discard rewrite plan", optimized.rewritePlan().isEmpty());
- });
-
- sql("DELETE FROM %s WHERE dep = 'hr'", tableName);
-
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(1, "hardware"), row(2, "hardware")),
- sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testDeleteFileThenMetadataDelete() throws Exception {
- Assume.assumeFalse("Avro does not support metadata delete", fileFormat.equals("avro"));
- createAndInitUnpartitionedTable();
-
- sql("INSERT INTO TABLE %s VALUES (1, 'hr'), (2, 'hardware'), (null, 'hr')", tableName);
-
- // MOR mode: writes a delete file as null cannot be deleted by metadata
- sql("DELETE FROM %s AS t WHERE t.id IS NULL", tableName);
-
- // Metadata Delete
- Table table = Spark3Util.loadIcebergTable(spark, tableName);
- List dataFilesBefore = TestHelpers.dataFiles(table);
-
- sql("DELETE FROM %s AS t WHERE t.id = 1", tableName);
-
- List dataFilesAfter = TestHelpers.dataFiles(table);
- Assert.assertTrue(
- "Data file should have been removed", dataFilesBefore.size() > dataFilesAfter.size());
-
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(2, "hardware")),
- sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testDeleteWithPartitionedTable() throws Exception {
- createAndInitPartitionedTable();
-
- sql("INSERT INTO TABLE %s VALUES (1, 'hr'), (3, 'hr')", tableName);
- sql("INSERT INTO TABLE %s VALUES (1, 'hardware'), (2, 'hardware')", tableName);
-
- // row level delete
- sql("DELETE FROM %s WHERE id = 1", tableName);
-
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(2, "hardware"), row(3, "hr")),
- sql("SELECT * FROM %s ORDER BY id", tableName));
- List rowLevelDeletePartitions =
- spark.sql("SELECT * FROM " + tableName + ".partitions ").collectAsList();
- Assert.assertEquals(
- "row level delete does not reduce number of partition", 2, rowLevelDeletePartitions.size());
-
- // partition aligned delete
- sql("DELETE FROM %s WHERE dep = 'hr'", tableName);
-
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(2, "hardware")),
- sql("SELECT * FROM %s ORDER BY id", tableName));
- List actualPartitions =
- spark.sql("SELECT * FROM " + tableName + ".partitions ").collectAsList();
- Assert.assertEquals(
- "partition aligned delete results in 1 partition", 1, actualPartitions.size());
- }
-
- @Test
- public void testDeleteWithFalseCondition() {
- createAndInitUnpartitionedTable();
-
- sql("INSERT INTO TABLE %s VALUES (1, 'hr'), (2, 'hardware')", tableName);
-
- sql("DELETE FROM %s WHERE id = 1 AND id > 20", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertEquals("Should have 2 snapshots", 2, Iterables.size(table.snapshots()));
-
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(1, "hr"), row(2, "hardware")),
- sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testDeleteFromEmptyTable() {
- createAndInitUnpartitionedTable();
-
- sql("DELETE FROM %s WHERE id IN (1)", tableName);
- sql("DELETE FROM %s WHERE dep = 'hr'", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertEquals("Should have 2 snapshots", 2, Iterables.size(table.snapshots()));
-
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(),
- sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testExplain() {
- createAndInitUnpartitionedTable();
-
- sql("INSERT INTO TABLE %s VALUES (1, 'hr'), (2, 'hardware'), (null, 'hr')", tableName);
-
- sql("EXPLAIN DELETE FROM %s WHERE id <=> 1", tableName);
-
- sql("EXPLAIN DELETE FROM %s WHERE true", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertEquals("Should have 1 snapshot", 1, Iterables.size(table.snapshots()));
-
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(1, "hr"), row(2, "hardware"), row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
- }
-
- @Test
- public void testDeleteWithAlias() {
- createAndInitUnpartitionedTable();
-
- sql("INSERT INTO TABLE %s VALUES (1, 'hr'), (2, 'hardware'), (null, 'hr')", tableName);
-
- sql("DELETE FROM %s AS t WHERE t.id IS NULL", tableName);
-
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(1, "hr"), row(2, "hardware")),
- sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testDeleteWithDynamicFileFiltering() throws NoSuchTableException {
- createAndInitPartitionedTable();
-
- append(new Employee(1, "hr"), new Employee(3, "hr"));
- append(new Employee(1, "hardware"), new Employee(2, "hardware"));
-
- sql("DELETE FROM %s WHERE id = 2", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertEquals("Should have 3 snapshots", 3, Iterables.size(table.snapshots()));
-
- Snapshot currentSnapshot = table.currentSnapshot();
- if (mode(table) == COPY_ON_WRITE) {
- validateCopyOnWrite(currentSnapshot, "1", "1", "1");
- } else {
- validateMergeOnRead(currentSnapshot, "1", "1", null);
- }
-
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(1, "hardware"), row(1, "hr"), row(3, "hr")),
- sql("SELECT * FROM %s ORDER BY id, dep", tableName));
- }
-
- @Test
- public void testDeleteNonExistingRecords() {
- createAndInitPartitionedTable();
-
- sql("INSERT INTO TABLE %s VALUES (1, 'hr'), (2, 'hardware'), (null, 'hr')", tableName);
-
- sql("DELETE FROM %s AS t WHERE t.id > 10", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertEquals("Should have 2 snapshots", 2, Iterables.size(table.snapshots()));
-
- Snapshot currentSnapshot = table.currentSnapshot();
-
- if (fileFormat.equals("orc") || fileFormat.equals("parquet")) {
- validateDelete(currentSnapshot, "0", null);
- } else {
- if (mode(table) == COPY_ON_WRITE) {
- validateCopyOnWrite(currentSnapshot, "0", null, null);
- } else {
- validateMergeOnRead(currentSnapshot, "0", null, null);
- }
- }
-
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(1, "hr"), row(2, "hardware"), row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
- }
-
- @Test
- public void testDeleteWithoutCondition() {
- createAndInitPartitionedTable();
-
- sql("INSERT INTO TABLE %s VALUES (1, 'hr')", tableName);
- sql("INSERT INTO TABLE %s VALUES (2, 'hardware')", tableName);
- sql("INSERT INTO TABLE %s VALUES (null, 'hr')", tableName);
-
- sql("DELETE FROM %s", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertEquals("Should have 4 snapshots", 4, Iterables.size(table.snapshots()));
-
- // should be a delete instead of an overwrite as it is done through a metadata operation
- Snapshot currentSnapshot = table.currentSnapshot();
- validateDelete(currentSnapshot, "2", "3");
-
- assertEquals(
- "Should have expected rows", ImmutableList.of(), sql("SELECT * FROM %s", tableName));
- }
-
- @Test
- public void testDeleteUsingMetadataWithComplexCondition() {
- createAndInitPartitionedTable();
-
- sql("INSERT INTO TABLE %s VALUES (1, 'dep1')", tableName);
- sql("INSERT INTO TABLE %s VALUES (2, 'dep2')", tableName);
- sql("INSERT INTO TABLE %s VALUES (null, 'dep3')", tableName);
-
- sql("DELETE FROM %s WHERE dep > 'dep2' OR dep = CAST(4 AS STRING) OR dep = 'dep2'", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertEquals("Should have 4 snapshots", 4, Iterables.size(table.snapshots()));
-
- // should be a delete instead of an overwrite as it is done through a metadata operation
- Snapshot currentSnapshot = table.currentSnapshot();
- validateDelete(currentSnapshot, "2", "2");
-
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(1, "dep1")),
- sql("SELECT * FROM %s", tableName));
- }
-
- @Test
- public void testDeleteWithArbitraryPartitionPredicates() {
- createAndInitPartitionedTable();
-
- sql("INSERT INTO TABLE %s VALUES (1, 'hr')", tableName);
- sql("INSERT INTO TABLE %s VALUES (2, 'hardware')", tableName);
- sql("INSERT INTO TABLE %s VALUES (null, 'hr')", tableName);
-
- // %% is an escaped version of %
- sql("DELETE FROM %s WHERE id = 10 OR dep LIKE '%%ware'", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertEquals("Should have 4 snapshots", 4, Iterables.size(table.snapshots()));
-
- // should be an overwrite since cannot be executed using a metadata operation
- Snapshot currentSnapshot = table.currentSnapshot();
- if (mode(table) == COPY_ON_WRITE) {
- validateCopyOnWrite(currentSnapshot, "1", "1", null);
- } else {
- validateMergeOnRead(currentSnapshot, "1", "1", null);
- }
-
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(1, "hr"), row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
- }
-
- @Test
- public void testDeleteWithNonDeterministicCondition() {
- createAndInitPartitionedTable();
-
- sql("INSERT INTO TABLE %s VALUES (1, 'hr'), (2, 'hardware')", tableName);
-
- AssertHelpers.assertThrows(
- "Should complain about non-deterministic expressions",
- AnalysisException.class,
- "nondeterministic expressions are only allowed",
- () -> sql("DELETE FROM %s WHERE id = 1 AND rand() > 0.5", tableName));
- }
-
- @Test
- public void testDeleteWithFoldableConditions() {
- createAndInitPartitionedTable();
-
- sql("INSERT INTO TABLE %s VALUES (1, 'hr'), (2, 'hardware')", tableName);
-
- // should keep all rows and don't trigger execution
- sql("DELETE FROM %s WHERE false", tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(1, "hr"), row(2, "hardware")),
- sql("SELECT * FROM %s ORDER BY id", tableName));
-
- // should keep all rows and don't trigger execution
- sql("DELETE FROM %s WHERE 50 <> 50", tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(1, "hr"), row(2, "hardware")),
- sql("SELECT * FROM %s ORDER BY id", tableName));
-
- // should keep all rows and don't trigger execution
- sql("DELETE FROM %s WHERE 1 > null", tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(1, "hr"), row(2, "hardware")),
- sql("SELECT * FROM %s ORDER BY id", tableName));
-
- // should remove all rows
- sql("DELETE FROM %s WHERE 21 = 21", tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(),
- sql("SELECT * FROM %s ORDER BY id", tableName));
-
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertEquals("Should have 2 snapshots", 2, Iterables.size(table.snapshots()));
- }
-
- @Test
- public void testDeleteWithNullConditions() {
- createAndInitPartitionedTable();
-
- sql(
- "INSERT INTO TABLE %s VALUES (0, null), (1, 'hr'), (2, 'hardware'), (null, 'hr')",
- tableName);
-
- // should keep all rows as null is never equal to null
- sql("DELETE FROM %s WHERE dep = null", tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(0, null), row(1, "hr"), row(2, "hardware"), row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
-
- // null = 'software' -> null
- // should delete using metadata operation only
- sql("DELETE FROM %s WHERE dep = 'software'", tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(0, null), row(1, "hr"), row(2, "hardware"), row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
-
- // should delete using metadata operation only
- sql("DELETE FROM %s WHERE dep <=> NULL", tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(1, "hr"), row(2, "hardware"), row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
-
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertEquals("Should have 3 snapshots", 3, Iterables.size(table.snapshots()));
-
- Snapshot currentSnapshot = table.currentSnapshot();
- validateDelete(currentSnapshot, "1", "1");
- }
-
- @Test
- public void testDeleteWithInAndNotInConditions() {
- createAndInitUnpartitionedTable();
-
- sql("INSERT INTO TABLE %s VALUES (1, 'hr'), (2, 'hardware'), (null, 'hr')", tableName);
-
- sql("DELETE FROM %s WHERE id IN (1, null)", tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(2, "hardware"), row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
-
- sql("DELETE FROM %s WHERE id NOT IN (null, 1)", tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(2, "hardware"), row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
-
- sql("DELETE FROM %s WHERE id NOT IN (1, 10)", tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
- }
-
- @Test
- public void testDeleteWithMultipleRowGroupsParquet() throws NoSuchTableException {
- Assume.assumeTrue(fileFormat.equalsIgnoreCase("parquet"));
-
- createAndInitPartitionedTable();
-
- sql(
- "ALTER TABLE %s SET TBLPROPERTIES('%s' '%d')",
- tableName, PARQUET_ROW_GROUP_SIZE_BYTES, 100);
- sql("ALTER TABLE %s SET TBLPROPERTIES('%s' '%d')", tableName, SPLIT_SIZE, 100);
-
- List ids = Lists.newArrayListWithCapacity(200);
- for (int id = 1; id <= 200; id++) {
- ids.add(id);
- }
- Dataset df =
- spark
- .createDataset(ids, Encoders.INT())
- .withColumnRenamed("value", "id")
- .withColumn("dep", lit("hr"));
- df.coalesce(1).writeTo(tableName).append();
-
- Assert.assertEquals(200, spark.table(tableName).count());
-
- // delete a record from one of two row groups and copy over the second one
- sql("DELETE FROM %s WHERE id IN (200, 201)", tableName);
-
- Assert.assertEquals(199, spark.table(tableName).count());
- }
-
- @Test
- public void testDeleteWithConditionOnNestedColumn() {
- createAndInitNestedColumnsTable();
-
- sql("INSERT INTO TABLE %s VALUES (1, named_struct(\"c1\", 3, \"c2\", \"v1\"))", tableName);
- sql("INSERT INTO TABLE %s VALUES (2, named_struct(\"c1\", 2, \"c2\", \"v2\"))", tableName);
-
- sql("DELETE FROM %s WHERE complex.c1 = id + 2", tableName);
- assertEquals(
- "Should have expected rows", ImmutableList.of(row(2)), sql("SELECT id FROM %s", tableName));
-
- sql("DELETE FROM %s t WHERE t.complex.c1 = id", tableName);
- assertEquals(
- "Should have expected rows", ImmutableList.of(), sql("SELECT id FROM %s", tableName));
- }
-
- @Test
- public void testDeleteWithInSubquery() throws NoSuchTableException {
- createAndInitUnpartitionedTable();
-
- sql("INSERT INTO TABLE %s VALUES (1, 'hr'), (2, 'hardware'), (null, 'hr')", tableName);
-
- createOrReplaceView("deleted_id", Arrays.asList(0, 1, null), Encoders.INT());
- createOrReplaceView("deleted_dep", Arrays.asList("software", "hr"), Encoders.STRING());
-
- sql(
- "DELETE FROM %s WHERE id IN (SELECT * FROM deleted_id) AND dep IN (SELECT * from deleted_dep)",
- tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(2, "hardware"), row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
-
- append(new Employee(1, "hr"), new Employee(-1, "hr"));
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(-1, "hr"), row(1, "hr"), row(2, "hardware"), row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
-
- sql("DELETE FROM %s WHERE id IS NULL OR id IN (SELECT value + 2 FROM deleted_id)", tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(-1, "hr"), row(1, "hr")),
- sql("SELECT * FROM %s ORDER BY id", tableName));
-
- append(new Employee(null, "hr"), new Employee(2, "hr"));
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(-1, "hr"), row(1, "hr"), row(2, "hr"), row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
-
- sql("DELETE FROM %s WHERE id IN (SELECT value + 2 FROM deleted_id) AND dep = 'hr'", tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(-1, "hr"), row(1, "hr"), row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
- }
-
- @Test
- public void testDeleteWithMultiColumnInSubquery() throws NoSuchTableException {
- createAndInitUnpartitionedTable();
-
- append(new Employee(1, "hr"), new Employee(2, "hardware"), new Employee(null, "hr"));
-
- List deletedEmployees =
- Arrays.asList(new Employee(null, "hr"), new Employee(1, "hr"));
- createOrReplaceView("deleted_employee", deletedEmployees, Encoders.bean(Employee.class));
-
- sql("DELETE FROM %s WHERE (id, dep) IN (SELECT id, dep FROM deleted_employee)", tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(2, "hardware"), row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
- }
-
- @Test
- public void testDeleteWithNotInSubquery() throws NoSuchTableException {
- createAndInitUnpartitionedTable();
-
- append(new Employee(1, "hr"), new Employee(2, "hardware"), new Employee(null, "hr"));
-
- createOrReplaceView("deleted_id", Arrays.asList(-1, -2, null), Encoders.INT());
- createOrReplaceView("deleted_dep", Arrays.asList("software", "hr"), Encoders.STRING());
-
- // the file filter subquery (nested loop lef-anti join) returns 0 records
- sql("DELETE FROM %s WHERE id NOT IN (SELECT * FROM deleted_id)", tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(1, "hr"), row(2, "hardware"), row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
-
- sql(
- "DELETE FROM %s WHERE id NOT IN (SELECT * FROM deleted_id WHERE value IS NOT NULL)",
- tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
-
- sql("INSERT INTO TABLE %s VALUES (1, 'hr'), (2, 'hardware'), (null, 'hr')", tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(1, "hr"), row(2, "hardware"), row(null, "hr"), row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
-
- sql(
- "DELETE FROM %s WHERE id NOT IN (SELECT * FROM deleted_id) OR dep IN ('software', 'hr')",
- tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(2, "hardware")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
-
- sql(
- "DELETE FROM %s t WHERE "
- + "id NOT IN (SELECT * FROM deleted_id WHERE value IS NOT NULL) AND "
- + "EXISTS (SELECT 1 FROM FROM deleted_dep WHERE t.dep = deleted_dep.value)",
- tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(2, "hardware")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
-
- sql(
- "DELETE FROM %s t WHERE "
- + "id NOT IN (SELECT * FROM deleted_id WHERE value IS NOT NULL) OR "
- + "EXISTS (SELECT 1 FROM FROM deleted_dep WHERE t.dep = deleted_dep.value)",
- tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
- }
-
- @Test
- public void testDeleteOnNonIcebergTableNotSupported() {
- createOrReplaceView("testtable", "{ \"c1\": -100, \"c2\": -200 }");
-
- AssertHelpers.assertThrows(
- "Delete is supported only for Iceberg tables",
- AnalysisException.class,
- "DELETE is only supported with v2 tables.",
- () -> sql("DELETE FROM %s WHERE c1 = -100", "testtable"));
- }
-
- @Test
- public void testDeleteWithExistSubquery() throws NoSuchTableException {
- createAndInitUnpartitionedTable();
-
- append(new Employee(1, "hr"), new Employee(2, "hardware"), new Employee(null, "hr"));
-
- createOrReplaceView("deleted_id", Arrays.asList(-1, -2, null), Encoders.INT());
- createOrReplaceView("deleted_dep", Arrays.asList("software", "hr"), Encoders.STRING());
-
- sql(
- "DELETE FROM %s t WHERE EXISTS (SELECT 1 FROM deleted_id d WHERE t.id = d.value)",
- tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(1, "hr"), row(2, "hardware"), row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
-
- sql(
- "DELETE FROM %s t WHERE EXISTS (SELECT 1 FROM deleted_id d WHERE t.id = d.value + 2)",
- tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(2, "hardware"), row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
-
- sql(
- "DELETE FROM %s t WHERE EXISTS (SELECT 1 FROM deleted_id d WHERE t.id = d.value) OR t.id IS NULL",
- tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(2, "hardware")),
- sql("SELECT * FROM %s", tableName));
-
- sql(
- "DELETE FROM %s t WHERE "
- + "EXISTS (SELECT 1 FROM deleted_id di WHERE t.id = di.value) AND "
- + "EXISTS (SELECT 1 FROM deleted_dep dd WHERE t.dep = dd.value)",
- tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(2, "hardware")),
- sql("SELECT * FROM %s", tableName));
- }
-
- @Test
- public void testDeleteWithNotExistsSubquery() throws NoSuchTableException {
- createAndInitUnpartitionedTable();
-
- append(new Employee(1, "hr"), new Employee(2, "hardware"), new Employee(null, "hr"));
-
- createOrReplaceView("deleted_id", Arrays.asList(-1, -2, null), Encoders.INT());
- createOrReplaceView("deleted_dep", Arrays.asList("software", "hr"), Encoders.STRING());
-
- sql(
- "DELETE FROM %s t WHERE "
- + "NOT EXISTS (SELECT 1 FROM deleted_id di WHERE t.id = di.value + 2) AND "
- + "NOT EXISTS (SELECT 1 FROM deleted_dep dd WHERE t.dep = dd.value)",
- tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(1, "hr"), row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
-
- sql(
- "DELETE FROM %s t WHERE NOT EXISTS (SELECT 1 FROM deleted_id d WHERE t.id = d.value + 2)",
- tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(1, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
-
- String subquery = "SELECT 1 FROM deleted_id d WHERE t.id = d.value + 2";
- sql("DELETE FROM %s t WHERE NOT EXISTS (%s) OR t.id = 1", tableName, subquery);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
- }
-
- @Test
- public void testDeleteWithScalarSubquery() throws NoSuchTableException {
- createAndInitUnpartitionedTable();
-
- append(new Employee(1, "hr"), new Employee(2, "hardware"), new Employee(null, "hr"));
-
- createOrReplaceView("deleted_id", Arrays.asList(1, 100, null), Encoders.INT());
-
- // TODO: Spark does not support AQE and DPP with aggregates at the moment
- withSQLConf(
- ImmutableMap.of(SQLConf.ADAPTIVE_EXECUTION_ENABLED().key(), "false"),
- () -> {
- sql("DELETE FROM %s t WHERE id <= (SELECT min(value) FROM deleted_id)", tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(2, "hardware"), row(null, "hr")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
- });
- }
-
- @Test
- public void testDeleteThatRequiresGroupingBeforeWrite() throws NoSuchTableException {
- createAndInitPartitionedTable();
-
- append(new Employee(0, "hr"), new Employee(1, "hr"), new Employee(2, "hr"));
- append(new Employee(0, "ops"), new Employee(1, "ops"), new Employee(2, "ops"));
- append(new Employee(0, "hr"), new Employee(1, "hr"), new Employee(2, "hr"));
- append(new Employee(0, "ops"), new Employee(1, "ops"), new Employee(2, "ops"));
-
- createOrReplaceView("deleted_id", Arrays.asList(1, 100), Encoders.INT());
-
- String originalNumOfShufflePartitions = spark.conf().get("spark.sql.shuffle.partitions");
- try {
- // set the num of shuffle partitions to 1 to ensure we have only 1 writing task
- spark.conf().set("spark.sql.shuffle.partitions", "1");
-
- sql("DELETE FROM %s t WHERE id IN (SELECT * FROM deleted_id)", tableName);
- Assert.assertEquals("Should have expected num of rows", 8L, spark.table(tableName).count());
- } finally {
- spark.conf().set("spark.sql.shuffle.partitions", originalNumOfShufflePartitions);
- }
- }
-
- @Test
- public synchronized void testDeleteWithSerializableIsolation() throws InterruptedException {
- // cannot run tests with concurrency for Hadoop tables without atomic renames
- Assume.assumeFalse(catalogName.equalsIgnoreCase("testhadoop"));
-
- createAndInitUnpartitionedTable();
- createOrReplaceView("deleted_id", Collections.singletonList(1), Encoders.INT());
-
- sql(
- "ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')",
- tableName, DELETE_ISOLATION_LEVEL, "serializable");
-
- sql("INSERT INTO TABLE %s VALUES (1, 'hr')", tableName);
-
- ExecutorService executorService =
- MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) Executors.newFixedThreadPool(2));
-
- AtomicInteger barrier = new AtomicInteger(0);
- AtomicBoolean shouldAppend = new AtomicBoolean(true);
-
- // delete thread
- Future deleteFuture =
- executorService.submit(
- () -> {
- for (int numOperations = 0; numOperations < Integer.MAX_VALUE; numOperations++) {
- while (barrier.get() < numOperations * 2) {
- sleep(10);
- }
-
- sql("DELETE FROM %s WHERE id IN (SELECT * FROM deleted_id)", tableName);
-
- barrier.incrementAndGet();
- }
- });
-
- // append thread
- Future appendFuture =
- executorService.submit(
- () -> {
- // load the table via the validation catalog to use another table instance
- Table table = validationCatalog.loadTable(tableIdent);
-
- GenericRecord record = GenericRecord.create(table.schema());
- record.set(0, 1); // id
- record.set(1, "hr"); // dep
-
- for (int numOperations = 0; numOperations < Integer.MAX_VALUE; numOperations++) {
- while (shouldAppend.get() && barrier.get() < numOperations * 2) {
- sleep(10);
- }
-
- if (!shouldAppend.get()) {
- return;
- }
-
- for (int numAppends = 0; numAppends < 5; numAppends++) {
- DataFile dataFile = writeDataFile(table, ImmutableList.of(record));
- table.newFastAppend().appendFile(dataFile).commit();
- sleep(10);
- }
-
- barrier.incrementAndGet();
- }
- });
-
- try {
- Assertions.assertThatThrownBy(deleteFuture::get)
- .isInstanceOf(ExecutionException.class)
- .cause()
- .isInstanceOf(SparkException.class)
- .cause()
- .isInstanceOf(ValidationException.class)
- .hasMessageContaining("Found conflicting files that can contain");
- } finally {
- shouldAppend.set(false);
- appendFuture.cancel(true);
- }
-
- executorService.shutdown();
- Assert.assertTrue("Timeout", executorService.awaitTermination(2, TimeUnit.MINUTES));
- }
-
- @Test
- public synchronized void testDeleteWithSnapshotIsolation()
- throws InterruptedException, ExecutionException {
- // cannot run tests with concurrency for Hadoop tables without atomic renames
- Assume.assumeFalse(catalogName.equalsIgnoreCase("testhadoop"));
-
- createAndInitUnpartitionedTable();
- createOrReplaceView("deleted_id", Collections.singletonList(1), Encoders.INT());
-
- sql(
- "ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')",
- tableName, DELETE_ISOLATION_LEVEL, "snapshot");
-
- sql("INSERT INTO TABLE %s VALUES (1, 'hr')", tableName);
-
- ExecutorService executorService =
- MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) Executors.newFixedThreadPool(2));
-
- AtomicInteger barrier = new AtomicInteger(0);
- AtomicBoolean shouldAppend = new AtomicBoolean(true);
-
- // delete thread
- Future deleteFuture =
- executorService.submit(
- () -> {
- for (int numOperations = 0; numOperations < 20; numOperations++) {
- while (barrier.get() < numOperations * 2) {
- sleep(10);
- }
-
- sql("DELETE FROM %s WHERE id IN (SELECT * FROM deleted_id)", tableName);
-
- barrier.incrementAndGet();
- }
- });
-
- // append thread
- Future appendFuture =
- executorService.submit(
- () -> {
- // load the table via the validation catalog to use another table instance for inserts
- Table table = validationCatalog.loadTable(tableIdent);
-
- GenericRecord record = GenericRecord.create(table.schema());
- record.set(0, 1); // id
- record.set(1, "hr"); // dep
-
- for (int numOperations = 0; numOperations < 20; numOperations++) {
- while (shouldAppend.get() && barrier.get() < numOperations * 2) {
- sleep(10);
- }
-
- if (!shouldAppend.get()) {
- return;
- }
-
- for (int numAppends = 0; numAppends < 5; numAppends++) {
- DataFile dataFile = writeDataFile(table, ImmutableList.of(record));
- table.newFastAppend().appendFile(dataFile).commit();
- sleep(10);
- }
-
- barrier.incrementAndGet();
- }
- });
-
- try {
- deleteFuture.get();
- } finally {
- shouldAppend.set(false);
- appendFuture.cancel(true);
- }
-
- executorService.shutdown();
- Assert.assertTrue("Timeout", executorService.awaitTermination(2, TimeUnit.MINUTES));
- }
-
- @Test
- public void testDeleteRefreshesRelationCache() throws NoSuchTableException {
- createAndInitPartitionedTable();
-
- append(new Employee(1, "hr"), new Employee(3, "hr"));
- append(new Employee(1, "hardware"), new Employee(2, "hardware"));
-
- Dataset query = spark.sql("SELECT * FROM " + tableName + " WHERE id = 1");
- query.createOrReplaceTempView("tmp");
-
- spark.sql("CACHE TABLE tmp");
-
- assertEquals(
- "View should have correct data",
- ImmutableList.of(row(1, "hardware"), row(1, "hr")),
- sql("SELECT * FROM tmp ORDER BY id, dep"));
-
- sql("DELETE FROM %s WHERE id = 1", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertEquals("Should have 3 snapshots", 3, Iterables.size(table.snapshots()));
-
- Snapshot currentSnapshot = table.currentSnapshot();
- if (mode(table) == COPY_ON_WRITE) {
- validateCopyOnWrite(currentSnapshot, "2", "2", "2");
- } else {
- validateMergeOnRead(currentSnapshot, "2", "2", null);
- }
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(2, "hardware"), row(3, "hr")),
- sql("SELECT * FROM %s ORDER BY id, dep", tableName));
-
- assertEquals(
- "Should refresh the relation cache",
- ImmutableList.of(),
- sql("SELECT * FROM tmp ORDER BY id, dep"));
-
- spark.sql("UNCACHE TABLE tmp");
- }
-
- @Test
- public void testDeleteWithMultipleSpecs() {
- createAndInitTable("id INT, dep STRING, category STRING");
-
- // write an unpartitioned file
- append(tableName, "{ \"id\": 1, \"dep\": \"hr\", \"category\": \"c1\"}");
-
- // write a file partitioned by dep
- sql("ALTER TABLE %s ADD PARTITION FIELD dep", tableName);
- append(
- tableName,
- "{ \"id\": 2, \"dep\": \"hr\", \"category\": \"c1\" }\n"
- + "{ \"id\": 3, \"dep\": \"hr\", \"category\": \"c1\" }");
-
- // write a file partitioned by dep and category
- sql("ALTER TABLE %s ADD PARTITION FIELD category", tableName);
- append(tableName, "{ \"id\": 5, \"dep\": \"hr\", \"category\": \"c1\"}");
-
- // write another file partitioned by dep
- sql("ALTER TABLE %s DROP PARTITION FIELD category", tableName);
- append(tableName, "{ \"id\": 7, \"dep\": \"hr\", \"category\": \"c1\"}");
-
- sql("DELETE FROM %s WHERE id IN (1, 3, 5, 7)", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertEquals("Should have 5 snapshots", 5, Iterables.size(table.snapshots()));
-
- Snapshot currentSnapshot = table.currentSnapshot();
- if (mode(table) == COPY_ON_WRITE) {
- validateCopyOnWrite(currentSnapshot, "3", "4", "1");
- } else {
- validateMergeOnRead(currentSnapshot, "3", "3", null);
- }
-
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(2, "hr", "c1")),
- sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- // TODO: multiple stripes for ORC
-
- protected void createAndInitPartitionedTable() {
- sql("CREATE TABLE %s (id INT, dep STRING) USING iceberg PARTITIONED BY (dep)", tableName);
- initTable();
- }
-
- protected void createAndInitUnpartitionedTable() {
- sql("CREATE TABLE %s (id INT, dep STRING) USING iceberg", tableName);
- initTable();
- }
-
- protected void createAndInitNestedColumnsTable() {
- sql("CREATE TABLE %s (id INT, complex STRUCT) USING iceberg", tableName);
- initTable();
- }
-
- protected void append(Employee... employees) throws NoSuchTableException {
- List input = Arrays.asList(employees);
- Dataset inputDF = spark.createDataFrame(input, Employee.class);
- inputDF.coalesce(1).writeTo(tableName).append();
- }
-
- private RowLevelOperationMode mode(Table table) {
- String modeName = table.properties().getOrDefault(DELETE_MODE, DELETE_MODE_DEFAULT);
- return RowLevelOperationMode.fromName(modeName);
- }
-
- private LogicalPlan parsePlan(String query, Object... args) {
- try {
- return spark.sessionState().sqlParser().parsePlan(String.format(query, args));
- } catch (ParseException e) {
- throw new RuntimeException(e);
- }
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDuplicateSnapshotIDs.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDuplicateSnapshotIDs.java
deleted file mode 100644
index 690351840ad9..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDuplicateSnapshotIDs.java
+++ /dev/null
@@ -1,102 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import java.util.Map;
-import org.apache.iceberg.SnapshotIdGeneratorUtil;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
-import org.junit.After;
-import org.junit.Assert;
-import org.junit.Test;
-import org.mockito.MockedStatic;
-import org.mockito.Mockito;
-
-public class TestDuplicateSnapshotIDs extends SparkExtensionsTestBase {
-
- public TestDuplicateSnapshotIDs(
- String catalogName, String implementation, Map config) {
- super(catalogName, implementation, config);
- }
-
- @After
- public void removeTables() {
- sql("DROP TABLE IF EXISTS %s", tableName);
- }
-
- @Test
- public void testSameSnapshotIDBackToBack() {
- sql("DROP TABLE IF EXISTS %s ", tableName);
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
-
- try (MockedStatic utilities =
- Mockito.mockStatic(SnapshotIdGeneratorUtil.class)) {
- utilities.when(SnapshotIdGeneratorUtil::generateSnapshotID).thenReturn(42L, 42L, 43L);
- // use 42L as snapshot id for the insert
- sql("INSERT INTO TABLE %s SELECT 1, 'a' ", tableName);
- // use 42L as snapshot id for the insert and use 43L on retry.
- sql("INSERT INTO TABLE %s SELECT 2, 'b' ", tableName);
- }
- // use regular snapshot id logic for the insert
- sql("INSERT INTO TABLE %s SELECT 3, 'c' ", tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(row(1L, "a"), row(2L, "b"), row(3L, "c"));
- assertEquals(
- "should have all the rows", expectedRows, sql("SELECT * from %s order by id", tableName));
- Assert.assertEquals(sql("SELECT * from %s.snapshots", tableName).size(), 3);
- Assert.assertEquals(
- sql("SELECT * from %s.snapshots where snapshot_id = 42L", tableName).size(), 1);
- Assert.assertEquals(
- sql("SELECT * from %s.snapshots where snapshot_id = 43L", tableName).size(), 1);
- }
-
- @Test
- public void testSameSnapshotID() {
- sql("DROP TABLE IF EXISTS %s ", tableName);
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
-
- try (MockedStatic utilities =
- Mockito.mockStatic(SnapshotIdGeneratorUtil.class)) {
- utilities.when(SnapshotIdGeneratorUtil::generateSnapshotID).thenReturn(42L);
- // use 42L as snapshot id for the insert
- sql("INSERT INTO TABLE %s SELECT 1, 'a' ", tableName);
- }
- // use regular snapshot id logic for the inserts
- sql("INSERT INTO TABLE %s SELECT 2, 'b' ", tableName);
- sql("INSERT INTO TABLE %s SELECT 3, 'c' ", tableName);
- try (MockedStatic utilities =
- Mockito.mockStatic(SnapshotIdGeneratorUtil.class)) {
- utilities.when(SnapshotIdGeneratorUtil::generateSnapshotID).thenReturn(42L, 43L);
- // use 42L as snapshot id for the insert and retry with 43L.
- sql("INSERT INTO TABLE %s SELECT 4, 'd' ", tableName);
- }
- // use regular snapshot id logic for the insert
- sql("INSERT INTO TABLE %s SELECT 5, 'e' ", tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(row(1L, "a"), row(2L, "b"), row(3L, "c"), row(4L, "d"), row(5L, "e"));
- assertEquals(
- "should have all the rows", expectedRows, sql("SELECT * from %s order by id", tableName));
- Assert.assertEquals(sql("SELECT * from %s.snapshots", tableName).size(), 5);
- Assert.assertEquals(
- sql("SELECT * from %s.snapshots where snapshot_id = 42L", tableName).size(), 1);
- Assert.assertEquals(
- sql("SELECT * from %s.snapshots where snapshot_id = 43L", tableName).size(), 1);
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestExpireSnapshotsProcedure.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestExpireSnapshotsProcedure.java
deleted file mode 100644
index efb3d43668f1..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestExpireSnapshotsProcedure.java
+++ /dev/null
@@ -1,544 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import static org.apache.iceberg.TableProperties.GC_ENABLED;
-
-import java.io.File;
-import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.nio.charset.StandardCharsets;
-import java.sql.Timestamp;
-import java.time.Instant;
-import java.util.List;
-import java.util.Map;
-import java.util.UUID;
-import java.util.stream.Collectors;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-import org.apache.iceberg.AssertHelpers;
-import org.apache.iceberg.GenericBlobMetadata;
-import org.apache.iceberg.GenericStatisticsFile;
-import org.apache.iceberg.Snapshot;
-import org.apache.iceberg.StatisticsFile;
-import org.apache.iceberg.Table;
-import org.apache.iceberg.exceptions.ValidationException;
-import org.apache.iceberg.io.FileIO;
-import org.apache.iceberg.puffin.Blob;
-import org.apache.iceberg.puffin.Puffin;
-import org.apache.iceberg.puffin.PuffinWriter;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
-import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
-import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-import org.apache.iceberg.spark.SparkCatalog;
-import org.apache.iceberg.spark.data.TestHelpers;
-import org.apache.iceberg.spark.source.SimpleRecord;
-import org.apache.spark.sql.AnalysisException;
-import org.apache.spark.sql.Encoders;
-import org.apache.spark.sql.catalyst.analysis.NoSuchProcedureException;
-import org.assertj.core.api.Assertions;
-import org.junit.After;
-import org.junit.Assert;
-import org.junit.Test;
-
-public class TestExpireSnapshotsProcedure extends SparkExtensionsTestBase {
-
- public TestExpireSnapshotsProcedure(
- String catalogName, String implementation, Map config) {
- super(catalogName, implementation, config);
- }
-
- @After
- public void removeTables() {
- sql("DROP TABLE IF EXISTS %s", tableName);
- }
-
- @Test
- public void testExpireSnapshotsInEmptyTable() {
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
-
- List output = sql("CALL %s.system.expire_snapshots('%s')", catalogName, tableIdent);
- assertEquals(
- "Should not delete any files", ImmutableList.of(row(0L, 0L, 0L, 0L, 0L, 0L)), output);
- }
-
- @Test
- public void testExpireSnapshotsUsingPositionalArgs() {
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
- sql("INSERT INTO TABLE %s VALUES (1, 'a')", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
- Snapshot firstSnapshot = table.currentSnapshot();
-
- waitUntilAfter(firstSnapshot.timestampMillis());
-
- sql("INSERT INTO TABLE %s VALUES (2, 'b')", tableName);
-
- table.refresh();
-
- Snapshot secondSnapshot = table.currentSnapshot();
- Timestamp secondSnapshotTimestamp =
- Timestamp.from(Instant.ofEpochMilli(secondSnapshot.timestampMillis()));
-
- Assert.assertEquals("Should be 2 snapshots", 2, Iterables.size(table.snapshots()));
-
- // expire without retainLast param
- List output1 =
- sql(
- "CALL %s.system.expire_snapshots('%s', TIMESTAMP '%s')",
- catalogName, tableIdent, secondSnapshotTimestamp);
- assertEquals(
- "Procedure output must match", ImmutableList.of(row(0L, 0L, 0L, 0L, 1L, 0L)), output1);
-
- table.refresh();
-
- Assert.assertEquals("Should expire one snapshot", 1, Iterables.size(table.snapshots()));
-
- sql("INSERT OVERWRITE %s VALUES (3, 'c')", tableName);
- sql("INSERT INTO TABLE %s VALUES (4, 'd')", tableName);
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(3L, "c"), row(4L, "d")),
- sql("SELECT * FROM %s ORDER BY id", tableName));
-
- table.refresh();
-
- waitUntilAfter(table.currentSnapshot().timestampMillis());
-
- Timestamp currentTimestamp = Timestamp.from(Instant.ofEpochMilli(System.currentTimeMillis()));
-
- Assert.assertEquals("Should be 3 snapshots", 3, Iterables.size(table.snapshots()));
-
- // expire with retainLast param
- List output =
- sql(
- "CALL %s.system.expire_snapshots('%s', TIMESTAMP '%s', 2)",
- catalogName, tableIdent, currentTimestamp);
- assertEquals(
- "Procedure output must match", ImmutableList.of(row(2L, 0L, 0L, 2L, 1L, 0L)), output);
- }
-
- @Test
- public void testExpireSnapshotUsingNamedArgs() {
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
-
- sql("INSERT INTO TABLE %s VALUES (1, 'a')", tableName);
- sql("INSERT INTO TABLE %s VALUES (2, 'b')", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assert.assertEquals("Should be 2 snapshots", 2, Iterables.size(table.snapshots()));
-
- waitUntilAfter(table.currentSnapshot().timestampMillis());
-
- Timestamp currentTimestamp = Timestamp.from(Instant.ofEpochMilli(System.currentTimeMillis()));
-
- List output =
- sql(
- "CALL %s.system.expire_snapshots(older_than => TIMESTAMP '%s',table => '%s')",
- catalogName, currentTimestamp, tableIdent);
- assertEquals(
- "Procedure output must match", ImmutableList.of(row(0L, 0L, 0L, 0L, 1L, 0L)), output);
- }
-
- @Test
- public void testExpireSnapshotsGCDisabled() {
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
-
- sql("ALTER TABLE %s SET TBLPROPERTIES ('%s' 'false')", tableName, GC_ENABLED);
-
- AssertHelpers.assertThrows(
- "Should reject call",
- ValidationException.class,
- "Cannot expire snapshots: GC is disabled",
- () -> sql("CALL %s.system.expire_snapshots('%s')", catalogName, tableIdent));
- }
-
- @Test
- public void testInvalidExpireSnapshotsCases() {
- AssertHelpers.assertThrows(
- "Should not allow mixed args",
- AnalysisException.class,
- "Named and positional arguments cannot be mixed",
- () -> sql("CALL %s.system.expire_snapshots('n', table => 't')", catalogName));
-
- AssertHelpers.assertThrows(
- "Should not resolve procedures in arbitrary namespaces",
- NoSuchProcedureException.class,
- "not found",
- () -> sql("CALL %s.custom.expire_snapshots('n', 't')", catalogName));
-
- AssertHelpers.assertThrows(
- "Should reject calls without all required args",
- AnalysisException.class,
- "Missing required parameters",
- () -> sql("CALL %s.system.expire_snapshots()", catalogName));
-
- AssertHelpers.assertThrows(
- "Should reject calls with invalid arg types",
- AnalysisException.class,
- "Wrong arg type",
- () -> sql("CALL %s.system.expire_snapshots('n', 2.2)", catalogName));
-
- AssertHelpers.assertThrows(
- "Should reject calls with empty table identifier",
- IllegalArgumentException.class,
- "Cannot handle an empty identifier",
- () -> sql("CALL %s.system.expire_snapshots('')", catalogName));
- }
-
- @Test
- public void testResolvingTableInAnotherCatalog() throws IOException {
- String anotherCatalog = "another_" + catalogName;
- spark.conf().set("spark.sql.catalog." + anotherCatalog, SparkCatalog.class.getName());
- spark.conf().set("spark.sql.catalog." + anotherCatalog + ".type", "hadoop");
- spark
- .conf()
- .set(
- "spark.sql.catalog." + anotherCatalog + ".warehouse",
- "file:" + temp.newFolder().toString());
-
- sql(
- "CREATE TABLE %s.%s (id bigint NOT NULL, data string) USING iceberg",
- anotherCatalog, tableIdent);
-
- AssertHelpers.assertThrows(
- "Should reject calls for a table in another catalog",
- IllegalArgumentException.class,
- "Cannot run procedure in catalog",
- () ->
- sql(
- "CALL %s.system.expire_snapshots('%s')",
- catalogName, anotherCatalog + "." + tableName));
- }
-
- @Test
- public void testConcurrentExpireSnapshots() {
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
-
- sql("INSERT INTO TABLE %s VALUES (1, 'a')", tableName);
- sql("INSERT INTO TABLE %s VALUES (2, 'b')", tableName);
- sql("INSERT INTO TABLE %s VALUES (3, 'c')", tableName);
- sql("INSERT INTO TABLE %s VALUES (4, 'd')", tableName);
-
- Timestamp currentTimestamp = Timestamp.from(Instant.ofEpochMilli(System.currentTimeMillis()));
- List output =
- sql(
- "CALL %s.system.expire_snapshots("
- + "older_than => TIMESTAMP '%s',"
- + "table => '%s',"
- + "max_concurrent_deletes => %s)",
- catalogName, currentTimestamp, tableIdent, 4);
- assertEquals(
- "Expiring snapshots concurrently should succeed",
- ImmutableList.of(row(0L, 0L, 0L, 0L, 3L, 0L)),
- output);
- }
-
- @Test
- public void testConcurrentExpireSnapshotsWithInvalidInput() {
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
-
- AssertHelpers.assertThrows(
- "Should throw an error when max_concurrent_deletes = 0",
- IllegalArgumentException.class,
- "max_concurrent_deletes should have value > 0",
- () ->
- sql(
- "CALL %s.system.expire_snapshots(table => '%s', max_concurrent_deletes => %s)",
- catalogName, tableIdent, 0));
-
- AssertHelpers.assertThrows(
- "Should throw an error when max_concurrent_deletes < 0 ",
- IllegalArgumentException.class,
- "max_concurrent_deletes should have value > 0",
- () ->
- sql(
- "CALL %s.system.expire_snapshots(table => '%s', max_concurrent_deletes => %s)",
- catalogName, tableIdent, -1));
- }
-
- @Test
- public void testExpireDeleteFiles() throws Exception {
- sql(
- "CREATE TABLE %s (id bigint, data string) USING iceberg TBLPROPERTIES"
- + "('format-version'='2', 'write.delete.mode'='merge-on-read')",
- tableName);
-
- List records =
- Lists.newArrayList(
- new SimpleRecord(1, "a"),
- new SimpleRecord(2, "b"),
- new SimpleRecord(3, "c"),
- new SimpleRecord(4, "d"));
- spark
- .createDataset(records, Encoders.bean(SimpleRecord.class))
- .coalesce(1)
- .writeTo(tableName)
- .append();
- sql("DELETE FROM %s WHERE id=1", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assert.assertEquals(
- "Should have 1 delete manifest", 1, TestHelpers.deleteManifests(table).size());
- Assert.assertEquals("Should have 1 delete file", 1, TestHelpers.deleteFiles(table).size());
- Path deleteManifestPath = new Path(TestHelpers.deleteManifests(table).iterator().next().path());
- Path deleteFilePath =
- new Path(String.valueOf(TestHelpers.deleteFiles(table).iterator().next().path()));
-
- sql(
- "CALL %s.system.rewrite_data_files("
- + "table => '%s',"
- + "options => map("
- + "'delete-file-threshold','1',"
- + "'use-starting-sequence-number', 'false'))",
- catalogName, tableIdent);
- table.refresh();
-
- sql(
- "INSERT INTO TABLE %s VALUES (5, 'e')",
- tableName); // this txn moves the file to the DELETED state
- sql("INSERT INTO TABLE %s VALUES (6, 'f')", tableName); // this txn removes the file reference
- table.refresh();
-
- Assert.assertEquals(
- "Should have no delete manifests", 0, TestHelpers.deleteManifests(table).size());
- Assert.assertEquals("Should have no delete files", 0, TestHelpers.deleteFiles(table).size());
-
- FileSystem localFs = FileSystem.getLocal(new Configuration());
- Assert.assertTrue("Delete manifest should still exist", localFs.exists(deleteManifestPath));
- Assert.assertTrue("Delete file should still exist", localFs.exists(deleteFilePath));
-
- Timestamp currentTimestamp = Timestamp.from(Instant.ofEpochMilli(System.currentTimeMillis()));
- List output =
- sql(
- "CALL %s.system.expire_snapshots(older_than => TIMESTAMP '%s',table => '%s')",
- catalogName, currentTimestamp, tableIdent);
-
- assertEquals(
- "Should deleted 1 data and pos delete file and 4 manifests and lists (one for each txn)",
- ImmutableList.of(row(1L, 1L, 0L, 4L, 4L, 0L)),
- output);
- Assert.assertFalse("Delete manifest should be removed", localFs.exists(deleteManifestPath));
- Assert.assertFalse("Delete file should be removed", localFs.exists(deleteFilePath));
- }
-
- @Test
- public void testExpireSnapshotWithStreamResultsEnabled() {
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
-
- sql("INSERT INTO TABLE %s VALUES (1, 'a')", tableName);
- sql("INSERT INTO TABLE %s VALUES (2, 'b')", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assert.assertEquals("Should be 2 snapshots", 2, Iterables.size(table.snapshots()));
-
- waitUntilAfter(table.currentSnapshot().timestampMillis());
-
- Timestamp currentTimestamp = Timestamp.from(Instant.ofEpochMilli(System.currentTimeMillis()));
-
- List output =
- sql(
- "CALL %s.system.expire_snapshots("
- + "older_than => TIMESTAMP '%s',"
- + "table => '%s',"
- + "stream_results => true)",
- catalogName, currentTimestamp, tableIdent);
- assertEquals(
- "Procedure output must match", ImmutableList.of(row(0L, 0L, 0L, 0L, 1L, 0L)), output);
- }
-
- @Test
- public void testExpireSnapshotsWithSnapshotId() {
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
-
- sql("INSERT INTO TABLE %s VALUES (1, 'a')", tableName);
- sql("INSERT INTO TABLE %s VALUES (2, 'b')", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assert.assertEquals("Should be 2 snapshots", 2, Iterables.size(table.snapshots()));
-
- // Expiring the snapshot specified by snapshot_id should keep only a single snapshot.
- long firstSnapshotId = table.currentSnapshot().parentId();
- sql(
- "CALL %s.system.expire_snapshots(" + "table => '%s'," + "snapshot_ids => ARRAY(%d))",
- catalogName, tableIdent, firstSnapshotId);
-
- // There should only be one single snapshot left.
- table.refresh();
- Assert.assertEquals("Should be 1 snapshots", 1, Iterables.size(table.snapshots()));
- Assert.assertEquals(
- "Snapshot ID should not be present",
- 0,
- Iterables.size(
- Iterables.filter(
- table.snapshots(), snapshot -> snapshot.snapshotId() == firstSnapshotId)));
- }
-
- @Test
- public void testExpireSnapshotShouldFailForCurrentSnapshot() {
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
-
- sql("INSERT INTO TABLE %s VALUES (1, 'a')", tableName);
- sql("INSERT INTO TABLE %s VALUES (2, 'b')", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
- Assert.assertEquals("Should be 2 snapshots", 2, Iterables.size(table.snapshots()));
-
- AssertHelpers.assertThrows(
- "Should reject call",
- IllegalArgumentException.class,
- "Cannot expire",
- () ->
- sql(
- "CALL %s.system.expire_snapshots("
- + "table => '%s',"
- + "snapshot_ids => ARRAY(%d, %d))",
- catalogName,
- tableIdent,
- table.currentSnapshot().snapshotId(),
- table.currentSnapshot().parentId()));
- }
-
- @Test
- public void testExpireSnapshotsProcedureWorksWithSqlComments() {
- // Ensure that systems such as dbt, that inject comments into the generated SQL files, will
- // work with Iceberg-specific DDL
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
-
- sql("INSERT INTO TABLE %s VALUES (1, 'a')", tableName);
- sql("INSERT INTO TABLE %s VALUES (2, 'b')", tableName);
-
- Table table = validationCatalog.loadTable(tableIdent);
-
- Assert.assertEquals("Should be 2 snapshots", 2, Iterables.size(table.snapshots()));
-
- waitUntilAfter(table.currentSnapshot().timestampMillis());
-
- Timestamp currentTimestamp = Timestamp.from(Instant.ofEpochMilli(System.currentTimeMillis()));
-
- String callStatement =
- "/* CALL statement is used to expire snapshots */\n"
- + "-- And we have single line comments as well \n"
- + "/* And comments that span *multiple* \n"
- + " lines */ CALL /* this is the actual CALL */ %s.system.expire_snapshots("
- + " older_than => TIMESTAMP '%s',"
- + " table => '%s')";
- List output = sql(callStatement, catalogName, currentTimestamp, tableIdent);
- assertEquals(
- "Procedure output must match", ImmutableList.of(row(0L, 0L, 0L, 0L, 1L, 0L)), output);
-
- table.refresh();
-
- Assert.assertEquals("Should be 1 snapshot remaining", 1, Iterables.size(table.snapshots()));
- }
-
- @Test
- public void testExpireSnapshotsWithStatisticFiles() throws Exception {
- sql("CREATE TABLE %s (id bigint NOT NULL, data string) USING iceberg", tableName);
- sql("INSERT INTO TABLE %s VALUES (10, 'abc')", tableName);
- Table table = validationCatalog.loadTable(tableIdent);
- String statsFileLocation1 = statsFileLocation(table.location());
- StatisticsFile statisticsFile1 =
- writeStatsFile(
- table.currentSnapshot().snapshotId(),
- table.currentSnapshot().sequenceNumber(),
- statsFileLocation1,
- table.io());
- table.updateStatistics().setStatistics(statisticsFile1.snapshotId(), statisticsFile1).commit();
-
- sql("INSERT INTO %s SELECT 20, 'def'", tableName);
- table.refresh();
- String statsFileLocation2 = statsFileLocation(table.location());
- StatisticsFile statisticsFile2 =
- writeStatsFile(
- table.currentSnapshot().snapshotId(),
- table.currentSnapshot().sequenceNumber(),
- statsFileLocation2,
- table.io());
- table.updateStatistics().setStatistics(statisticsFile2.snapshotId(), statisticsFile2).commit();
-
- waitUntilAfter(table.currentSnapshot().timestampMillis());
-
- Timestamp currentTimestamp = Timestamp.from(Instant.ofEpochMilli(System.currentTimeMillis()));
- List output =
- sql(
- "CALL %s.system.expire_snapshots(older_than => TIMESTAMP '%s',table => '%s')",
- catalogName, currentTimestamp, tableIdent);
- Assertions.assertThat(output.get(0)[5]).as("should be 1 deleted statistics file").isEqualTo(1L);
-
- table.refresh();
- List statsWithSnapshotId1 =
- table.statisticsFiles().stream()
- .filter(statisticsFile -> statisticsFile.snapshotId() == statisticsFile1.snapshotId())
- .collect(Collectors.toList());
- Assertions.assertThat(statsWithSnapshotId1)
- .as(
- "Statistics file entry in TableMetadata should be deleted for the snapshot %s",
- statisticsFile1.snapshotId())
- .isEmpty();
- Assertions.assertThat(table.statisticsFiles())
- .as(
- "Statistics file entry in TableMetadata should be present for the snapshot %s",
- statisticsFile2.snapshotId())
- .extracting(StatisticsFile::snapshotId)
- .containsExactly(statisticsFile2.snapshotId());
-
- Assertions.assertThat(new File(statsFileLocation1))
- .as("Statistics file should not exist for snapshot %s", statisticsFile1.snapshotId())
- .doesNotExist();
-
- Assertions.assertThat(new File(statsFileLocation2))
- .as("Statistics file should exist for snapshot %s", statisticsFile2.snapshotId())
- .exists();
- }
-
- private StatisticsFile writeStatsFile(
- long snapshotId, long snapshotSequenceNumber, String statsLocation, FileIO fileIO)
- throws IOException {
- try (PuffinWriter puffinWriter = Puffin.write(fileIO.newOutputFile(statsLocation)).build()) {
- puffinWriter.add(
- new Blob(
- "some-blob-type",
- ImmutableList.of(1),
- snapshotId,
- snapshotSequenceNumber,
- ByteBuffer.wrap("blob content".getBytes(StandardCharsets.UTF_8))));
- puffinWriter.finish();
-
- return new GenericStatisticsFile(
- snapshotId,
- statsLocation,
- puffinWriter.fileSize(),
- puffinWriter.footerSize(),
- puffinWriter.writtenBlobsMetadata().stream()
- .map(GenericBlobMetadata::from)
- .collect(ImmutableList.toImmutableList()));
- }
- }
-
- private String statsFileLocation(String tableLocation) {
- String statsFileName = "stats-file-" + UUID.randomUUID();
- return tableLocation.replaceFirst("file:", "") + "/metadata/" + statsFileName;
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestIcebergExpressions.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestIcebergExpressions.java
deleted file mode 100644
index b1ba53455123..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestIcebergExpressions.java
+++ /dev/null
@@ -1,105 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import java.math.BigDecimal;
-import java.util.Map;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
-import org.apache.spark.sql.Column;
-import org.apache.spark.sql.Dataset;
-import org.apache.spark.sql.Row;
-import org.apache.spark.sql.catalyst.expressions.IcebergBucketTransform;
-import org.apache.spark.sql.catalyst.expressions.IcebergTruncateTransform;
-import org.junit.After;
-import org.junit.Test;
-
-public class TestIcebergExpressions extends SparkExtensionsTestBase {
-
- public TestIcebergExpressions(
- String catalogName, String implementation, Map config) {
- super(catalogName, implementation, config);
- }
-
- @After
- public void removeTables() {
- sql("DROP TABLE IF EXISTS %s", tableName);
- sql("DROP VIEW IF EXISTS emp");
- sql("DROP VIEW IF EXISTS v");
- }
-
- @Test
- public void testTruncateExpressions() {
- sql(
- "CREATE TABLE %s ( "
- + " int_c INT, long_c LONG, dec_c DECIMAL(4, 2), str_c STRING, binary_c BINARY "
- + ") USING iceberg",
- tableName);
-
- sql(
- "CREATE TEMPORARY VIEW emp "
- + "AS SELECT * FROM VALUES (101, 10001, 10.65, '101-Employee', CAST('1234' AS BINARY)) "
- + "AS EMP(int_c, long_c, dec_c, str_c, binary_c)");
-
- sql("INSERT INTO %s SELECT * FROM emp", tableName);
-
- Dataset df = spark.sql("SELECT * FROM " + tableName);
- df.select(
- new Column(new IcebergTruncateTransform(df.col("int_c").expr(), 2)).as("int_c"),
- new Column(new IcebergTruncateTransform(df.col("long_c").expr(), 2)).as("long_c"),
- new Column(new IcebergTruncateTransform(df.col("dec_c").expr(), 50)).as("dec_c"),
- new Column(new IcebergTruncateTransform(df.col("str_c").expr(), 2)).as("str_c"),
- new Column(new IcebergTruncateTransform(df.col("binary_c").expr(), 2)).as("binary_c"))
- .createOrReplaceTempView("v");
-
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(100, 10000L, new BigDecimal("10.50"), "10", "12")),
- sql("SELECT int_c, long_c, dec_c, str_c, CAST(binary_c AS STRING) FROM v"));
- }
-
- @Test
- public void testBucketExpressions() {
- sql(
- "CREATE TABLE %s ( "
- + " int_c INT, long_c LONG, dec_c DECIMAL(4, 2), str_c STRING, binary_c BINARY "
- + ") USING iceberg",
- tableName);
-
- sql(
- "CREATE TEMPORARY VIEW emp "
- + "AS SELECT * FROM VALUES (101, 10001, 10.65, '101-Employee', CAST('1234' AS BINARY)) "
- + "AS EMP(int_c, long_c, dec_c, str_c, binary_c)");
-
- sql("INSERT INTO %s SELECT * FROM emp", tableName);
-
- Dataset df = spark.sql("SELECT * FROM " + tableName);
- df.select(
- new Column(new IcebergBucketTransform(2, df.col("int_c").expr())).as("int_c"),
- new Column(new IcebergBucketTransform(3, df.col("long_c").expr())).as("long_c"),
- new Column(new IcebergBucketTransform(4, df.col("dec_c").expr())).as("dec_c"),
- new Column(new IcebergBucketTransform(5, df.col("str_c").expr())).as("str_c"),
- new Column(new IcebergBucketTransform(6, df.col("binary_c").expr())).as("binary_c"))
- .createOrReplaceTempView("v");
-
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(0, 2, 0, 4, 1)),
- sql("SELECT int_c, long_c, dec_c, str_c, binary_c FROM v"));
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java
deleted file mode 100644
index 58fbb6241e8e..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java
+++ /dev/null
@@ -1,2277 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import static org.apache.iceberg.TableProperties.MERGE_ISOLATION_LEVEL;
-import static org.apache.iceberg.TableProperties.PARQUET_ROW_GROUP_SIZE_BYTES;
-import static org.apache.iceberg.TableProperties.SPLIT_SIZE;
-import static org.apache.iceberg.TableProperties.WRITE_DISTRIBUTION_MODE;
-import static org.apache.spark.sql.functions.lit;
-
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.List;
-import java.util.Map;
-import java.util.concurrent.ExecutionException;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.Future;
-import java.util.concurrent.ThreadPoolExecutor;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicBoolean;
-import java.util.concurrent.atomic.AtomicInteger;
-import org.apache.iceberg.AssertHelpers;
-import org.apache.iceberg.DataFile;
-import org.apache.iceberg.DistributionMode;
-import org.apache.iceberg.Snapshot;
-import org.apache.iceberg.SnapshotSummary;
-import org.apache.iceberg.Table;
-import org.apache.iceberg.data.GenericRecord;
-import org.apache.iceberg.exceptions.ValidationException;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
-import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-import org.apache.iceberg.relocated.com.google.common.util.concurrent.MoreExecutors;
-import org.apache.spark.SparkException;
-import org.apache.spark.sql.AnalysisException;
-import org.apache.spark.sql.Dataset;
-import org.apache.spark.sql.Encoders;
-import org.apache.spark.sql.Row;
-import org.apache.spark.sql.catalyst.analysis.NoSuchTableException;
-import org.apache.spark.sql.internal.SQLConf;
-import org.assertj.core.api.Assertions;
-import org.junit.After;
-import org.junit.Assert;
-import org.junit.Assume;
-import org.junit.BeforeClass;
-import org.junit.Test;
-
-public abstract class TestMerge extends SparkRowLevelOperationsTestBase {
-
- public TestMerge(
- String catalogName,
- String implementation,
- Map config,
- String fileFormat,
- boolean vectorized,
- String distributionMode) {
- super(catalogName, implementation, config, fileFormat, vectorized, distributionMode);
- }
-
- @BeforeClass
- public static void setupSparkConf() {
- spark.conf().set("spark.sql.shuffle.partitions", "4");
- }
-
- @After
- public void removeTables() {
- sql("DROP TABLE IF EXISTS %s", tableName);
- sql("DROP TABLE IF EXISTS source");
- }
-
- @Test
- public void testMergeWithStaticPredicatePushDown() {
- createAndInitTable("id BIGINT, dep STRING");
-
- sql("ALTER TABLE %s ADD PARTITION FIELD dep", tableName);
-
- // add a data file to the 'software' partition
- append(tableName, "{ \"id\": 1, \"dep\": \"software\" }");
-
- // add a data file to the 'hr' partition
- append(tableName, "{ \"id\": 1, \"dep\": \"hr\" }");
-
- Table table = validationCatalog.loadTable(tableIdent);
-
- Snapshot snapshot = table.currentSnapshot();
- String dataFilesCount = snapshot.summary().get(SnapshotSummary.TOTAL_DATA_FILES_PROP);
- Assert.assertEquals("Must have 2 files before MERGE", "2", dataFilesCount);
-
- createOrReplaceView(
- "source", "{ \"id\": 1, \"dep\": \"finance\" }\n" + "{ \"id\": 2, \"dep\": \"hardware\" }");
-
- // remove the data file from the 'hr' partition to ensure it is not scanned
- withUnavailableFiles(
- snapshot.addedDataFiles(table.io()),
- () -> {
- // disable dynamic pruning and rely only on static predicate pushdown
- withSQLConf(
- ImmutableMap.of(SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED().key(), "false"),
- () -> {
- sql(
- "MERGE INTO %s t USING source "
- + "ON t.id == source.id AND t.dep IN ('software') AND source.id < 10 "
- + "WHEN MATCHED AND source.id = 1 THEN "
- + " UPDATE SET dep = source.dep "
- + "WHEN NOT MATCHED THEN "
- + " INSERT (dep, id) VALUES (source.dep, source.id)",
- tableName);
- });
- });
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1L, "finance"), // updated
- row(1L, "hr"), // kept
- row(2L, "hardware") // new
- );
- assertEquals(
- "Output should match", expectedRows, sql("SELECT * FROM %s ORDER BY id, dep", tableName));
- }
-
- @Test
- public void testMergeIntoEmptyTargetInsertAllNonMatchingRows() {
- createAndInitTable("id INT, dep STRING");
-
- createOrReplaceView(
- "source",
- "id INT, dep STRING",
- "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n"
- + "{ \"id\": 3, \"dep\": \"emp-id-3\" }");
-
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.id "
- + "WHEN NOT MATCHED THEN "
- + " INSERT *",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1, "emp-id-1"), // new
- row(2, "emp-id-2"), // new
- row(3, "emp-id-3") // new
- );
- assertEquals(
- "Should have expected rows", expectedRows, sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeIntoEmptyTargetInsertOnlyMatchingRows() {
- createAndInitTable("id INT, dep STRING");
-
- createOrReplaceView(
- "source",
- "id INT, dep STRING",
- "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n"
- + "{ \"id\": 3, \"dep\": \"emp-id-3\" }");
-
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.id "
- + "WHEN NOT MATCHED AND (s.id >=2) THEN "
- + " INSERT *",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(2, "emp-id-2"), // new
- row(3, "emp-id-3") // new
- );
- assertEquals(
- "Should have expected rows", expectedRows, sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeWithOnlyUpdateClause() {
- createAndInitTable(
- "id INT, dep STRING",
- "{ \"id\": 1, \"dep\": \"emp-id-one\" }\n" + "{ \"id\": 6, \"dep\": \"emp-id-six\" }");
-
- createOrReplaceView(
- "source",
- "id INT, dep STRING",
- "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n"
- + "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.id "
- + "WHEN MATCHED AND t.id = 1 THEN "
- + " UPDATE SET *",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1, "emp-id-1"), // updated
- row(6, "emp-id-six") // kept
- );
- assertEquals(
- "Should have expected rows", expectedRows, sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeWithOnlyDeleteClause() {
- createAndInitTable(
- "id INT, dep STRING",
- "{ \"id\": 1, \"dep\": \"emp-id-one\" }\n" + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- createOrReplaceView(
- "source",
- "id INT, dep STRING",
- "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n"
- + "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.id "
- + "WHEN MATCHED AND t.id = 6 THEN "
- + " DELETE",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1, "emp-id-one") // kept
- );
- assertEquals(
- "Should have expected rows", expectedRows, sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeWithAllCauses() {
- createAndInitTable(
- "id INT, dep STRING",
- "{ \"id\": 1, \"dep\": \"emp-id-one\" }\n" + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- createOrReplaceView(
- "source",
- "id INT, dep STRING",
- "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n"
- + "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.id "
- + "WHEN MATCHED AND t.id = 1 THEN "
- + " UPDATE SET * "
- + "WHEN MATCHED AND t.id = 6 THEN "
- + " DELETE "
- + "WHEN NOT MATCHED AND s.id = 2 THEN "
- + " INSERT *",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1, "emp-id-1"), // updated
- row(2, "emp-id-2") // new
- );
- assertEquals(
- "Should have expected rows", expectedRows, sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeWithAllCausesWithExplicitColumnSpecification() {
- createAndInitTable(
- "id INT, dep STRING",
- "{ \"id\": 1, \"dep\": \"emp-id-one\" }\n" + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- createOrReplaceView(
- "source",
- "id INT, dep STRING",
- "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n"
- + "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.id "
- + "WHEN MATCHED AND t.id = 1 THEN "
- + " UPDATE SET t.id = s.id, t.dep = s.dep "
- + "WHEN MATCHED AND t.id = 6 THEN "
- + " DELETE "
- + "WHEN NOT MATCHED AND s.id = 2 THEN "
- + " INSERT (t.id, t.dep) VALUES (s.id, s.dep)",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1, "emp-id-1"), // updated
- row(2, "emp-id-2") // new
- );
- assertEquals(
- "Should have expected rows", expectedRows, sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeWithSourceCTE() {
- createAndInitTable(
- "id INT, dep STRING",
- "{ \"id\": 2, \"dep\": \"emp-id-two\" }\n" + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- createOrReplaceView(
- "source",
- "id INT, dep STRING",
- "{ \"id\": 2, \"dep\": \"emp-id-3\" }\n"
- + "{ \"id\": 1, \"dep\": \"emp-id-2\" }\n"
- + "{ \"id\": 5, \"dep\": \"emp-id-6\" }");
-
- sql(
- "WITH cte1 AS (SELECT id + 1 AS id, dep FROM source) "
- + "MERGE INTO %s AS t USING cte1 AS s "
- + "ON t.id == s.id "
- + "WHEN MATCHED AND t.id = 2 THEN "
- + " UPDATE SET * "
- + "WHEN MATCHED AND t.id = 6 THEN "
- + " DELETE "
- + "WHEN NOT MATCHED AND s.id = 3 THEN "
- + " INSERT *",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(2, "emp-id-2"), // updated
- row(3, "emp-id-3") // new
- );
- assertEquals(
- "Should have expected rows", expectedRows, sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeWithSourceFromSetOps() {
- createAndInitTable(
- "id INT, dep STRING",
- "{ \"id\": 1, \"dep\": \"emp-id-one\" }\n" + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- createOrReplaceView(
- "source",
- "id INT, dep STRING",
- "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n"
- + "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- String derivedSource =
- "SELECT * FROM source WHERE id = 2 "
- + "UNION ALL "
- + "SELECT * FROM source WHERE id = 1 OR id = 6";
-
- sql(
- "MERGE INTO %s AS t USING (%s) AS s "
- + "ON t.id == s.id "
- + "WHEN MATCHED AND t.id = 1 THEN "
- + " UPDATE SET * "
- + "WHEN MATCHED AND t.id = 6 THEN "
- + " DELETE "
- + "WHEN NOT MATCHED AND s.id = 2 THEN "
- + " INSERT *",
- tableName, derivedSource);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1, "emp-id-1"), // updated
- row(2, "emp-id-2") // new
- );
- assertEquals(
- "Should have expected rows", expectedRows, sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeWithMultipleUpdatesForTargetRowSmallTargetLargeSource() {
- createAndInitTable(
- "id INT, dep STRING",
- "{ \"id\": 1, \"dep\": \"emp-id-one\" }\n" + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- List sourceIds = Lists.newArrayList();
- for (int i = 0; i < 10_000; i++) {
- sourceIds.add(i);
- }
- Dataset ds = spark.createDataset(sourceIds, Encoders.INT());
- ds.union(ds).createOrReplaceTempView("source");
-
- String errorMsg = "a single row from the target table with multiple rows of the source table";
- AssertHelpers.assertThrowsCause(
- "Should complain about multiple matches",
- SparkException.class,
- errorMsg,
- () -> {
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.value "
- + "WHEN MATCHED AND t.id = 1 THEN "
- + " UPDATE SET id = 10 "
- + "WHEN MATCHED AND t.id = 6 THEN "
- + " DELETE "
- + "WHEN NOT MATCHED AND s.value = 2 THEN "
- + " INSERT (id, dep) VALUES (s.value, null)",
- tableName);
- });
-
- assertEquals(
- "Target should be unchanged",
- ImmutableList.of(row(1, "emp-id-one"), row(6, "emp-id-6")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
- }
-
- @Test
- public void
- testMergeWithMultipleUpdatesForTargetRowSmallTargetLargeSourceEnabledHashShuffleJoin() {
- createAndInitTable(
- "id INT, dep STRING",
- "{ \"id\": 1, \"dep\": \"emp-id-one\" }\n" + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- List sourceIds = Lists.newArrayList();
- for (int i = 0; i < 10_000; i++) {
- sourceIds.add(i);
- }
- Dataset ds = spark.createDataset(sourceIds, Encoders.INT());
- ds.union(ds).createOrReplaceTempView("source");
-
- withSQLConf(
- ImmutableMap.of(SQLConf.PREFER_SORTMERGEJOIN().key(), "false"),
- () -> {
- String errorMsg =
- "a single row from the target table with multiple rows of the source table";
- AssertHelpers.assertThrowsCause(
- "Should complain about multiple matches",
- SparkException.class,
- errorMsg,
- () -> {
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.value "
- + "WHEN MATCHED AND t.id = 1 THEN "
- + " UPDATE SET id = 10 "
- + "WHEN MATCHED AND t.id = 6 THEN "
- + " DELETE "
- + "WHEN NOT MATCHED AND s.value = 2 THEN "
- + " INSERT (id, dep) VALUES (s.value, null)",
- tableName);
- });
- });
-
- assertEquals(
- "Target should be unchanged",
- ImmutableList.of(row(1, "emp-id-one"), row(6, "emp-id-6")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
- }
-
- @Test
- public void testMergeWithMultipleUpdatesForTargetRowSmallTargetLargeSourceNoEqualityCondition() {
- createAndInitTable("id INT, dep STRING", "{ \"id\": 1, \"dep\": \"emp-id-one\" }");
-
- List sourceIds = Lists.newArrayList();
- for (int i = 0; i < 10_000; i++) {
- sourceIds.add(i);
- }
- Dataset ds = spark.createDataset(sourceIds, Encoders.INT());
- ds.union(ds).createOrReplaceTempView("source");
-
- withSQLConf(
- ImmutableMap.of(SQLConf.PREFER_SORTMERGEJOIN().key(), "false"),
- () -> {
- String errorMsg =
- "a single row from the target table with multiple rows of the source table";
- AssertHelpers.assertThrowsCause(
- "Should complain about multiple matches",
- SparkException.class,
- errorMsg,
- () -> {
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id > s.value "
- + "WHEN MATCHED AND t.id = 1 THEN "
- + " UPDATE SET id = 10 "
- + "WHEN MATCHED AND t.id = 6 THEN "
- + " DELETE "
- + "WHEN NOT MATCHED AND s.value = 2 THEN "
- + " INSERT (id, dep) VALUES (s.value, null)",
- tableName);
- });
- });
-
- assertEquals(
- "Target should be unchanged",
- ImmutableList.of(row(1, "emp-id-one")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
- }
-
- @Test
- public void testMergeWithMultipleUpdatesForTargetRowSmallTargetLargeSourceNoNotMatchedActions() {
- createAndInitTable(
- "id INT, dep STRING",
- "{ \"id\": 1, \"dep\": \"emp-id-one\" }\n" + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- List sourceIds = Lists.newArrayList();
- for (int i = 0; i < 10_000; i++) {
- sourceIds.add(i);
- }
- Dataset ds = spark.createDataset(sourceIds, Encoders.INT());
- ds.union(ds).createOrReplaceTempView("source");
-
- String errorMsg = "a single row from the target table with multiple rows of the source table";
- AssertHelpers.assertThrowsCause(
- "Should complain about multiple matches",
- SparkException.class,
- errorMsg,
- () -> {
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.value "
- + "WHEN MATCHED AND t.id = 1 THEN "
- + " UPDATE SET id = 10 "
- + "WHEN MATCHED AND t.id = 6 THEN "
- + " DELETE",
- tableName);
- });
-
- assertEquals(
- "Target should be unchanged",
- ImmutableList.of(row(1, "emp-id-one"), row(6, "emp-id-6")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
- }
-
- @Test
- public void
- testMergeWithMultipleUpdatesForTargetRowSmallTargetLargeSourceNoNotMatchedActionsNoEqualityCondition() {
- createAndInitTable("id INT, dep STRING", "{ \"id\": 1, \"dep\": \"emp-id-one\" }");
-
- List sourceIds = Lists.newArrayList();
- for (int i = 0; i < 10_000; i++) {
- sourceIds.add(i);
- }
- Dataset ds = spark.createDataset(sourceIds, Encoders.INT());
- ds.union(ds).createOrReplaceTempView("source");
-
- String errorMsg = "a single row from the target table with multiple rows of the source table";
- AssertHelpers.assertThrowsCause(
- "Should complain about multiple matches",
- SparkException.class,
- errorMsg,
- () -> {
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id > s.value "
- + "WHEN MATCHED AND t.id = 1 THEN "
- + " UPDATE SET id = 10 "
- + "WHEN MATCHED AND t.id = 6 THEN "
- + " DELETE",
- tableName);
- });
-
- assertEquals(
- "Target should be unchanged",
- ImmutableList.of(row(1, "emp-id-one")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
- }
-
- @Test
- public void testMergeWithMultipleUpdatesForTargetRow() {
- createAndInitTable(
- "id INT, dep STRING",
- "{ \"id\": 1, \"dep\": \"emp-id-one\" }\n" + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- createOrReplaceView(
- "source",
- "id INT, dep STRING",
- "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n"
- + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- String errorMsg = "a single row from the target table with multiple rows of the source table";
- AssertHelpers.assertThrowsCause(
- "Should complain about multiple matches",
- SparkException.class,
- errorMsg,
- () -> {
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.id "
- + "WHEN MATCHED AND t.id = 1 THEN "
- + " UPDATE SET * "
- + "WHEN MATCHED AND t.id = 6 THEN "
- + " DELETE "
- + "WHEN NOT MATCHED AND s.id = 2 THEN "
- + " INSERT *",
- tableName);
- });
-
- assertEquals(
- "Target should be unchanged",
- ImmutableList.of(row(1, "emp-id-one"), row(6, "emp-id-6")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
- }
-
- @Test
- public void testMergeWithUnconditionalDelete() {
- createAndInitTable(
- "id INT, dep STRING",
- "{ \"id\": 1, \"dep\": \"emp-id-one\" }\n" + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- createOrReplaceView(
- "source",
- "id INT, dep STRING",
- "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n"
- + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.id "
- + "WHEN MATCHED THEN "
- + " DELETE "
- + "WHEN NOT MATCHED AND s.id = 2 THEN "
- + " INSERT *",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(2, "emp-id-2") // new
- );
- assertEquals(
- "Should have expected rows", expectedRows, sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeWithSingleConditionalDelete() {
- createAndInitTable(
- "id INT, dep STRING",
- "{ \"id\": 1, \"dep\": \"emp-id-one\" }\n" + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- createOrReplaceView(
- "source",
- "id INT, dep STRING",
- "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n"
- + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- String errorMsg = "a single row from the target table with multiple rows of the source table";
- AssertHelpers.assertThrowsCause(
- "Should complain about multiple matches",
- SparkException.class,
- errorMsg,
- () -> {
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.id "
- + "WHEN MATCHED AND t.id = 1 THEN "
- + " DELETE "
- + "WHEN NOT MATCHED AND s.id = 2 THEN "
- + " INSERT *",
- tableName);
- });
-
- assertEquals(
- "Target should be unchanged",
- ImmutableList.of(row(1, "emp-id-one"), row(6, "emp-id-6")),
- sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
- }
-
- @Test
- public void testMergeWithIdentityTransform() {
- for (DistributionMode mode : DistributionMode.values()) {
- createAndInitTable("id INT, dep STRING");
- sql("ALTER TABLE %s ADD PARTITION FIELD identity(dep)", tableName);
- sql(
- "ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')",
- tableName, WRITE_DISTRIBUTION_MODE, mode.modeName());
-
- append(
- tableName,
- "{ \"id\": 1, \"dep\": \"emp-id-one\" }\n" + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- createOrReplaceView(
- "source",
- "id INT, dep STRING",
- "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n"
- + "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.id "
- + "WHEN MATCHED AND t.id = 1 THEN "
- + " UPDATE SET * "
- + "WHEN MATCHED AND t.id = 6 THEN "
- + " DELETE "
- + "WHEN NOT MATCHED AND s.id = 2 THEN "
- + " INSERT *",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1, "emp-id-1"), // updated
- row(2, "emp-id-2") // new
- );
- assertEquals(
- "Should have expected rows",
- expectedRows,
- sql("SELECT * FROM %s ORDER BY id", tableName));
-
- removeTables();
- }
- }
-
- @Test
- public void testMergeWithDaysTransform() {
- for (DistributionMode mode : DistributionMode.values()) {
- createAndInitTable("id INT, ts TIMESTAMP");
- sql("ALTER TABLE %s ADD PARTITION FIELD days(ts)", tableName);
- sql(
- "ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')",
- tableName, WRITE_DISTRIBUTION_MODE, mode.modeName());
-
- append(
- tableName,
- "id INT, ts TIMESTAMP",
- "{ \"id\": 1, \"ts\": \"2000-01-01 00:00:00\" }\n"
- + "{ \"id\": 6, \"ts\": \"2000-01-06 00:00:00\" }");
-
- createOrReplaceView(
- "source",
- "id INT, ts TIMESTAMP",
- "{ \"id\": 2, \"ts\": \"2001-01-02 00:00:00\" }\n"
- + "{ \"id\": 1, \"ts\": \"2001-01-01 00:00:00\" }\n"
- + "{ \"id\": 6, \"ts\": \"2001-01-06 00:00:00\" }");
-
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.id "
- + "WHEN MATCHED AND t.id = 1 THEN "
- + " UPDATE SET * "
- + "WHEN MATCHED AND t.id = 6 THEN "
- + " DELETE "
- + "WHEN NOT MATCHED AND s.id = 2 THEN "
- + " INSERT *",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1, "2001-01-01 00:00:00"), // updated
- row(2, "2001-01-02 00:00:00") // new
- );
- assertEquals(
- "Should have expected rows",
- expectedRows,
- sql("SELECT id, CAST(ts AS STRING) FROM %s ORDER BY id", tableName));
-
- removeTables();
- }
- }
-
- @Test
- public void testMergeWithBucketTransform() {
- for (DistributionMode mode : DistributionMode.values()) {
- createAndInitTable("id INT, dep STRING");
- sql("ALTER TABLE %s ADD PARTITION FIELD bucket(2, dep)", tableName);
- sql(
- "ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')",
- tableName, WRITE_DISTRIBUTION_MODE, mode.modeName());
-
- append(
- tableName,
- "{ \"id\": 1, \"dep\": \"emp-id-one\" }\n" + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- createOrReplaceView(
- "source",
- "id INT, dep STRING",
- "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n"
- + "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.id "
- + "WHEN MATCHED AND t.id = 1 THEN "
- + " UPDATE SET * "
- + "WHEN MATCHED AND t.id = 6 THEN "
- + " DELETE "
- + "WHEN NOT MATCHED AND s.id = 2 THEN "
- + " INSERT *",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1, "emp-id-1"), // updated
- row(2, "emp-id-2") // new
- );
- assertEquals(
- "Should have expected rows",
- expectedRows,
- sql("SELECT * FROM %s ORDER BY id", tableName));
-
- removeTables();
- }
- }
-
- @Test
- public void testMergeWithTruncateTransform() {
- for (DistributionMode mode : DistributionMode.values()) {
- createAndInitTable("id INT, dep STRING");
- sql("ALTER TABLE %s ADD PARTITION FIELD truncate(dep, 2)", tableName);
- sql(
- "ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')",
- tableName, WRITE_DISTRIBUTION_MODE, mode.modeName());
-
- append(
- tableName,
- "{ \"id\": 1, \"dep\": \"emp-id-one\" }\n" + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- createOrReplaceView(
- "source",
- "id INT, dep STRING",
- "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n"
- + "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.id "
- + "WHEN MATCHED AND t.id = 1 THEN "
- + " UPDATE SET * "
- + "WHEN MATCHED AND t.id = 6 THEN "
- + " DELETE "
- + "WHEN NOT MATCHED AND s.id = 2 THEN "
- + " INSERT *",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1, "emp-id-1"), // updated
- row(2, "emp-id-2") // new
- );
- assertEquals(
- "Should have expected rows",
- expectedRows,
- sql("SELECT * FROM %s ORDER BY id", tableName));
-
- removeTables();
- }
- }
-
- @Test
- public void testMergeIntoPartitionedAndOrderedTable() {
- for (DistributionMode mode : DistributionMode.values()) {
- createAndInitTable("id INT, dep STRING");
- sql("ALTER TABLE %s ADD PARTITION FIELD dep", tableName);
- sql("ALTER TABLE %s WRITE ORDERED BY (id)", tableName);
- sql(
- "ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')",
- tableName, WRITE_DISTRIBUTION_MODE, mode.modeName());
-
- append(
- tableName,
- "{ \"id\": 1, \"dep\": \"emp-id-one\" }\n" + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- createOrReplaceView(
- "source",
- "id INT, dep STRING",
- "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n"
- + "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.id "
- + "WHEN MATCHED AND t.id = 1 THEN "
- + " UPDATE SET * "
- + "WHEN MATCHED AND t.id = 6 THEN "
- + " DELETE "
- + "WHEN NOT MATCHED AND s.id = 2 THEN "
- + " INSERT *",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1, "emp-id-1"), // updated
- row(2, "emp-id-2") // new
- );
- assertEquals(
- "Should have expected rows",
- expectedRows,
- sql("SELECT * FROM %s ORDER BY id", tableName));
-
- removeTables();
- }
- }
-
- @Test
- public void testSelfMerge() {
- createAndInitTable(
- "id INT, v STRING", "{ \"id\": 1, \"v\": \"v1\" }\n" + "{ \"id\": 2, \"v\": \"v2\" }");
-
- sql(
- "MERGE INTO %s t USING %s s "
- + "ON t.id == s.id "
- + "WHEN MATCHED AND t.id = 1 THEN "
- + " UPDATE SET v = 'x' "
- + "WHEN NOT MATCHED THEN "
- + " INSERT *",
- tableName, tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1, "x"), // updated
- row(2, "v2") // kept
- );
- assertEquals(
- "Output should match", expectedRows, sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testSelfMergeWithCaching() {
- createAndInitTable(
- "id INT, v STRING", "{ \"id\": 1, \"v\": \"v1\" }\n" + "{ \"id\": 2, \"v\": \"v2\" }");
-
- sql("CACHE TABLE %s", tableName);
-
- sql(
- "MERGE INTO %s t USING %s s "
- + "ON t.id == s.id "
- + "WHEN MATCHED AND t.id = 1 THEN "
- + " UPDATE SET v = 'x' "
- + "WHEN NOT MATCHED THEN "
- + " INSERT *",
- tableName, tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1, "x"), // updated
- row(2, "v2") // kept
- );
- assertEquals(
- "Output should match", expectedRows, sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeWithSourceAsSelfSubquery() {
- createAndInitTable(
- "id INT, v STRING", "{ \"id\": 1, \"v\": \"v1\" }\n" + "{ \"id\": 2, \"v\": \"v2\" }");
-
- createOrReplaceView("source", Arrays.asList(1, null), Encoders.INT());
-
- sql(
- "MERGE INTO %s t USING (SELECT id AS value FROM %s r JOIN source ON r.id = source.value) s "
- + "ON t.id == s.value "
- + "WHEN MATCHED AND t.id = 1 THEN "
- + " UPDATE SET v = 'x' "
- + "WHEN NOT MATCHED THEN "
- + " INSERT (v, id) VALUES ('invalid', -1) ",
- tableName, tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1, "x"), // updated
- row(2, "v2") // kept
- );
- assertEquals(
- "Output should match", expectedRows, sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public synchronized void testMergeWithSerializableIsolation() throws InterruptedException {
- // cannot run tests with concurrency for Hadoop tables without atomic renames
- Assume.assumeFalse(catalogName.equalsIgnoreCase("testhadoop"));
-
- createAndInitTable("id INT, dep STRING");
- createOrReplaceView("source", Collections.singletonList(1), Encoders.INT());
-
- sql(
- "ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')",
- tableName, MERGE_ISOLATION_LEVEL, "serializable");
-
- sql("INSERT INTO TABLE %s VALUES (1, 'hr')", tableName);
-
- ExecutorService executorService =
- MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) Executors.newFixedThreadPool(2));
-
- AtomicInteger barrier = new AtomicInteger(0);
- AtomicBoolean shouldAppend = new AtomicBoolean(true);
-
- // merge thread
- Future mergeFuture =
- executorService.submit(
- () -> {
- for (int numOperations = 0; numOperations < Integer.MAX_VALUE; numOperations++) {
- while (barrier.get() < numOperations * 2) {
- sleep(10);
- }
-
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.value "
- + "WHEN MATCHED THEN "
- + " UPDATE SET dep = 'x'",
- tableName);
-
- barrier.incrementAndGet();
- }
- });
-
- // append thread
- Future appendFuture =
- executorService.submit(
- () -> {
- // load the table via the validation catalog to use another table instance
- Table table = validationCatalog.loadTable(tableIdent);
-
- GenericRecord record = GenericRecord.create(table.schema());
- record.set(0, 1); // id
- record.set(1, "hr"); // dep
-
- for (int numOperations = 0; numOperations < Integer.MAX_VALUE; numOperations++) {
- while (shouldAppend.get() && barrier.get() < numOperations * 2) {
- sleep(10);
- }
-
- if (!shouldAppend.get()) {
- return;
- }
-
- for (int numAppends = 0; numAppends < 5; numAppends++) {
- DataFile dataFile = writeDataFile(table, ImmutableList.of(record));
- table.newFastAppend().appendFile(dataFile).commit();
- sleep(10);
- }
-
- barrier.incrementAndGet();
- }
- });
-
- try {
- Assertions.assertThatThrownBy(mergeFuture::get)
- .isInstanceOf(ExecutionException.class)
- .cause()
- .isInstanceOf(SparkException.class)
- .cause()
- .isInstanceOf(ValidationException.class)
- .hasMessageContaining("Found conflicting files that can contain");
- } finally {
- shouldAppend.set(false);
- appendFuture.cancel(true);
- }
-
- executorService.shutdown();
- Assert.assertTrue("Timeout", executorService.awaitTermination(2, TimeUnit.MINUTES));
- }
-
- @Test
- public synchronized void testMergeWithSnapshotIsolation()
- throws InterruptedException, ExecutionException {
- // cannot run tests with concurrency for Hadoop tables without atomic renames
- Assume.assumeFalse(catalogName.equalsIgnoreCase("testhadoop"));
-
- createAndInitTable("id INT, dep STRING");
- createOrReplaceView("source", Collections.singletonList(1), Encoders.INT());
-
- sql(
- "ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')",
- tableName, MERGE_ISOLATION_LEVEL, "snapshot");
-
- sql("INSERT INTO TABLE %s VALUES (1, 'hr')", tableName);
-
- ExecutorService executorService =
- MoreExecutors.getExitingExecutorService(
- (ThreadPoolExecutor) Executors.newFixedThreadPool(2));
-
- AtomicInteger barrier = new AtomicInteger(0);
- AtomicBoolean shouldAppend = new AtomicBoolean(true);
-
- // merge thread
- Future mergeFuture =
- executorService.submit(
- () -> {
- for (int numOperations = 0; numOperations < 20; numOperations++) {
- while (barrier.get() < numOperations * 2) {
- sleep(10);
- }
-
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.value "
- + "WHEN MATCHED THEN "
- + " UPDATE SET dep = 'x'",
- tableName);
-
- barrier.incrementAndGet();
- }
- });
-
- // append thread
- Future appendFuture =
- executorService.submit(
- () -> {
- // load the table via the validation catalog to use another table instance for inserts
- Table table = validationCatalog.loadTable(tableIdent);
-
- GenericRecord record = GenericRecord.create(table.schema());
- record.set(0, 1); // id
- record.set(1, "hr"); // dep
-
- for (int numOperations = 0; numOperations < 20; numOperations++) {
- while (shouldAppend.get() && barrier.get() < numOperations * 2) {
- sleep(10);
- }
-
- if (!shouldAppend.get()) {
- return;
- }
-
- for (int numAppends = 0; numAppends < 5; numAppends++) {
- DataFile dataFile = writeDataFile(table, ImmutableList.of(record));
- table.newFastAppend().appendFile(dataFile).commit();
- sleep(10);
- }
-
- barrier.incrementAndGet();
- }
- });
-
- try {
- mergeFuture.get();
- } finally {
- shouldAppend.set(false);
- appendFuture.cancel(true);
- }
-
- executorService.shutdown();
- Assert.assertTrue("Timeout", executorService.awaitTermination(2, TimeUnit.MINUTES));
- }
-
- @Test
- public void testMergeWithExtraColumnsInSource() {
- createAndInitTable(
- "id INT, v STRING", "{ \"id\": 1, \"v\": \"v1\" }\n" + "{ \"id\": 2, \"v\": \"v2\" }");
- createOrReplaceView(
- "source",
- "{ \"id\": 1, \"extra_col\": -1, \"v\": \"v1_1\" }\n"
- + "{ \"id\": 3, \"extra_col\": -1, \"v\": \"v3\" }\n"
- + "{ \"id\": 4, \"extra_col\": -1, \"v\": \"v4\" }");
-
- sql(
- "MERGE INTO %s t USING source "
- + "ON t.id == source.id "
- + "WHEN MATCHED THEN "
- + " UPDATE SET v = source.v "
- + "WHEN NOT MATCHED THEN "
- + " INSERT (v, id) VALUES (source.v, source.id)",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1, "v1_1"), // new
- row(2, "v2"), // kept
- row(3, "v3"), // new
- row(4, "v4") // new
- );
- assertEquals(
- "Output should match", expectedRows, sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeWithNullsInTargetAndSource() {
- createAndInitTable(
- "id INT, v STRING", "{ \"id\": null, \"v\": \"v1\" }\n" + "{ \"id\": 2, \"v\": \"v2\" }");
-
- createOrReplaceView(
- "source", "{ \"id\": null, \"v\": \"v1_1\" }\n" + "{ \"id\": 4, \"v\": \"v4\" }");
-
- sql(
- "MERGE INTO %s t USING source "
- + "ON t.id == source.id "
- + "WHEN MATCHED THEN "
- + " UPDATE SET v = source.v "
- + "WHEN NOT MATCHED THEN "
- + " INSERT (v, id) VALUES (source.v, source.id)",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(null, "v1"), // kept
- row(null, "v1_1"), // new
- row(2, "v2"), // kept
- row(4, "v4") // new
- );
- assertEquals(
- "Output should match", expectedRows, sql("SELECT * FROM %s ORDER BY v", tableName));
- }
-
- @Test
- public void testMergeWithNullSafeEquals() {
- createAndInitTable(
- "id INT, v STRING", "{ \"id\": null, \"v\": \"v1\" }\n" + "{ \"id\": 2, \"v\": \"v2\" }");
-
- createOrReplaceView(
- "source", "{ \"id\": null, \"v\": \"v1_1\" }\n" + "{ \"id\": 4, \"v\": \"v4\" }");
-
- sql(
- "MERGE INTO %s t USING source "
- + "ON t.id <=> source.id "
- + "WHEN MATCHED THEN "
- + " UPDATE SET v = source.v "
- + "WHEN NOT MATCHED THEN "
- + " INSERT (v, id) VALUES (source.v, source.id)",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(null, "v1_1"), // updated
- row(2, "v2"), // kept
- row(4, "v4") // new
- );
- assertEquals(
- "Output should match", expectedRows, sql("SELECT * FROM %s ORDER BY v", tableName));
- }
-
- @Test
- public void testMergeWithNullCondition() {
- createAndInitTable(
- "id INT, v STRING", "{ \"id\": null, \"v\": \"v1\" }\n" + "{ \"id\": 2, \"v\": \"v2\" }");
-
- createOrReplaceView(
- "source", "{ \"id\": null, \"v\": \"v1_1\" }\n" + "{ \"id\": 2, \"v\": \"v2_2\" }");
-
- sql(
- "MERGE INTO %s t USING source "
- + "ON t.id == source.id AND NULL "
- + "WHEN MATCHED THEN "
- + " UPDATE SET v = source.v "
- + "WHEN NOT MATCHED THEN "
- + " INSERT (v, id) VALUES (source.v, source.id)",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(null, "v1"), // kept
- row(null, "v1_1"), // new
- row(2, "v2"), // kept
- row(2, "v2_2") // new
- );
- assertEquals(
- "Output should match", expectedRows, sql("SELECT * FROM %s ORDER BY v", tableName));
- }
-
- @Test
- public void testMergeWithNullActionConditions() {
- createAndInitTable(
- "id INT, v STRING", "{ \"id\": 1, \"v\": \"v1\" }\n" + "{ \"id\": 2, \"v\": \"v2\" }");
-
- createOrReplaceView(
- "source",
- "{ \"id\": 1, \"v\": \"v1_1\" }\n"
- + "{ \"id\": 2, \"v\": \"v2_2\" }\n"
- + "{ \"id\": 3, \"v\": \"v3_3\" }");
-
- // all conditions are NULL and will never match any rows
- sql(
- "MERGE INTO %s t USING source "
- + "ON t.id == source.id "
- + "WHEN MATCHED AND source.id = 1 AND NULL THEN "
- + " UPDATE SET v = source.v "
- + "WHEN MATCHED AND source.v = 'v1_1' AND NULL THEN "
- + " DELETE "
- + "WHEN NOT MATCHED AND source.id = 3 AND NULL THEN "
- + " INSERT (v, id) VALUES (source.v, source.id)",
- tableName);
-
- ImmutableList expectedRows1 =
- ImmutableList.of(
- row(1, "v1"), // kept
- row(2, "v2") // kept
- );
- assertEquals(
- "Output should match", expectedRows1, sql("SELECT * FROM %s ORDER BY v", tableName));
-
- // only the update and insert conditions are NULL
- sql(
- "MERGE INTO %s t USING source "
- + "ON t.id == source.id "
- + "WHEN MATCHED AND source.id = 1 AND NULL THEN "
- + " UPDATE SET v = source.v "
- + "WHEN MATCHED AND source.v = 'v1_1' THEN "
- + " DELETE "
- + "WHEN NOT MATCHED AND source.id = 3 AND NULL THEN "
- + " INSERT (v, id) VALUES (source.v, source.id)",
- tableName);
-
- ImmutableList expectedRows2 =
- ImmutableList.of(
- row(2, "v2") // kept
- );
- assertEquals(
- "Output should match", expectedRows2, sql("SELECT * FROM %s ORDER BY v", tableName));
- }
-
- @Test
- public void testMergeWithMultipleMatchingActions() {
- createAndInitTable(
- "id INT, v STRING", "{ \"id\": 1, \"v\": \"v1\" }\n" + "{ \"id\": 2, \"v\": \"v2\" }");
-
- createOrReplaceView(
- "source", "{ \"id\": 1, \"v\": \"v1_1\" }\n" + "{ \"id\": 2, \"v\": \"v2_2\" }");
-
- // the order of match actions is important in this case
- sql(
- "MERGE INTO %s t USING source "
- + "ON t.id == source.id "
- + "WHEN MATCHED AND source.id = 1 THEN "
- + " UPDATE SET v = source.v "
- + "WHEN MATCHED AND source.v = 'v1_1' THEN "
- + " DELETE "
- + "WHEN NOT MATCHED THEN "
- + " INSERT (v, id) VALUES (source.v, source.id)",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1, "v1_1"), // updated (also matches the delete cond but update is first)
- row(2, "v2") // kept (matches neither the update nor the delete cond)
- );
- assertEquals(
- "Output should match", expectedRows, sql("SELECT * FROM %s ORDER BY v", tableName));
- }
-
- @Test
- public void testMergeWithMultipleRowGroupsParquet() throws NoSuchTableException {
- Assume.assumeTrue(fileFormat.equalsIgnoreCase("parquet"));
-
- createAndInitTable("id INT, dep STRING");
- sql("ALTER TABLE %s ADD PARTITION FIELD dep", tableName);
-
- sql(
- "ALTER TABLE %s SET TBLPROPERTIES('%s' '%d')",
- tableName, PARQUET_ROW_GROUP_SIZE_BYTES, 100);
- sql("ALTER TABLE %s SET TBLPROPERTIES('%s' '%d')", tableName, SPLIT_SIZE, 100);
-
- createOrReplaceView("source", Collections.singletonList(1), Encoders.INT());
-
- List ids = Lists.newArrayListWithCapacity(200);
- for (int id = 1; id <= 200; id++) {
- ids.add(id);
- }
- Dataset df =
- spark
- .createDataset(ids, Encoders.INT())
- .withColumnRenamed("value", "id")
- .withColumn("dep", lit("hr"));
- df.coalesce(1).writeTo(tableName).append();
-
- Assert.assertEquals(200, spark.table(tableName).count());
-
- // update a record from one of two row groups and copy over the second one
- sql(
- "MERGE INTO %s t USING source "
- + "ON t.id == source.value "
- + "WHEN MATCHED THEN "
- + " UPDATE SET dep = 'x'",
- tableName);
-
- Assert.assertEquals(200, spark.table(tableName).count());
- }
-
- @Test
- public void testMergeInsertOnly() {
- createAndInitTable(
- "id STRING, v STRING",
- "{ \"id\": \"a\", \"v\": \"v1\" }\n" + "{ \"id\": \"b\", \"v\": \"v2\" }");
- createOrReplaceView(
- "source",
- "{ \"id\": \"a\", \"v\": \"v1_1\" }\n"
- + "{ \"id\": \"a\", \"v\": \"v1_2\" }\n"
- + "{ \"id\": \"c\", \"v\": \"v3\" }\n"
- + "{ \"id\": \"d\", \"v\": \"v4_1\" }\n"
- + "{ \"id\": \"d\", \"v\": \"v4_2\" }");
-
- sql(
- "MERGE INTO %s t USING source "
- + "ON t.id == source.id "
- + "WHEN NOT MATCHED THEN "
- + " INSERT *",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row("a", "v1"), // kept
- row("b", "v2"), // kept
- row("c", "v3"), // new
- row("d", "v4_1"), // new
- row("d", "v4_2") // new
- );
- assertEquals(
- "Output should match", expectedRows, sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeInsertOnlyWithCondition() {
- createAndInitTable("id INTEGER, v INTEGER", "{ \"id\": 1, \"v\": 1 }");
- createOrReplaceView(
- "source",
- "{ \"id\": 1, \"v\": 11, \"is_new\": true }\n"
- + "{ \"id\": 2, \"v\": 21, \"is_new\": true }\n"
- + "{ \"id\": 2, \"v\": 22, \"is_new\": false }");
-
- // validate assignments are reordered to match the table attrs
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.id "
- + "WHEN NOT MATCHED AND is_new = TRUE THEN "
- + " INSERT (v, id) VALUES (s.v + 100, s.id)",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1, 1), // kept
- row(2, 121) // new
- );
- assertEquals(
- "Output should match", expectedRows, sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeAlignsUpdateAndInsertActions() {
- createAndInitTable("id INT, a INT, b STRING", "{ \"id\": 1, \"a\": 2, \"b\": \"str\" }");
- createOrReplaceView(
- "source",
- "{ \"id\": 1, \"c1\": -2, \"c2\": \"new_str_1\" }\n"
- + "{ \"id\": 2, \"c1\": -20, \"c2\": \"new_str_2\" }");
-
- sql(
- "MERGE INTO %s t USING source "
- + "ON t.id == source.id "
- + "WHEN MATCHED THEN "
- + " UPDATE SET b = c2, a = c1, t.id = source.id "
- + "WHEN NOT MATCHED THEN "
- + " INSERT (b, a, id) VALUES (c2, c1, id)",
- tableName);
-
- assertEquals(
- "Output should match",
- ImmutableList.of(row(1, -2, "new_str_1"), row(2, -20, "new_str_2")),
- sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeMixedCaseAlignsUpdateAndInsertActions() {
- createAndInitTable("id INT, a INT, b STRING", "{ \"id\": 1, \"a\": 2, \"b\": \"str\" }");
- createOrReplaceView(
- "source",
- "{ \"id\": 1, \"c1\": -2, \"c2\": \"new_str_1\" }\n"
- + "{ \"id\": 2, \"c1\": -20, \"c2\": \"new_str_2\" }");
-
- sql(
- "MERGE INTO %s t USING source "
- + "ON t.iD == source.Id "
- + "WHEN MATCHED THEN "
- + " UPDATE SET B = c2, A = c1, t.Id = source.ID "
- + "WHEN NOT MATCHED THEN "
- + " INSERT (b, A, iD) VALUES (c2, c1, id)",
- tableName);
-
- assertEquals(
- "Output should match",
- ImmutableList.of(row(1, -2, "new_str_1"), row(2, -20, "new_str_2")),
- sql("SELECT * FROM %s ORDER BY id", tableName));
-
- assertEquals(
- "Output should match",
- ImmutableList.of(row(1, -2, "new_str_1")),
- sql("SELECT * FROM %s WHERE id = 1 ORDER BY id", tableName));
- assertEquals(
- "Output should match",
- ImmutableList.of(row(2, -20, "new_str_2")),
- sql("SELECT * FROM %s WHERE b = 'new_str_2'ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeUpdatesNestedStructFields() {
- createAndInitTable(
- "id INT, s STRUCT,m:MAP>>",
- "{ \"id\": 1, \"s\": { \"c1\": 2, \"c2\": { \"a\": [1,2], \"m\": { \"a\": \"b\"} } } } }");
- createOrReplaceView("source", "{ \"id\": 1, \"c1\": -2 }");
-
- // update primitive, array, map columns inside a struct
- sql(
- "MERGE INTO %s t USING source "
- + "ON t.id == source.id "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.s.c1 = source.c1, t.s.c2.a = array(-1, -2), t.s.c2.m = map('k', 'v')",
- tableName);
-
- assertEquals(
- "Output should match",
- ImmutableList.of(row(1, row(-2, row(ImmutableList.of(-1, -2), ImmutableMap.of("k", "v"))))),
- sql("SELECT * FROM %s ORDER BY id", tableName));
-
- // set primitive, array, map columns to NULL (proper casts should be in place)
- sql(
- "MERGE INTO %s t USING source "
- + "ON t.id == source.id "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.s.c1 = NULL, t.s.c2 = NULL",
- tableName);
-
- assertEquals(
- "Output should match",
- ImmutableList.of(row(1, row(null, null))),
- sql("SELECT * FROM %s ORDER BY id", tableName));
-
- // update all fields in a struct
- sql(
- "MERGE INTO %s t USING source "
- + "ON t.id == source.id "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.s = named_struct('c1', 100, 'c2', named_struct('a', array(1), 'm', map('x', 'y')))",
- tableName);
-
- assertEquals(
- "Output should match",
- ImmutableList.of(row(1, row(100, row(ImmutableList.of(1), ImmutableMap.of("x", "y"))))),
- sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeWithInferredCasts() {
- createAndInitTable("id INT, s STRING", "{ \"id\": 1, \"s\": \"value\" }");
- createOrReplaceView("source", "{ \"id\": 1, \"c1\": -2}");
-
- // -2 in source should be casted to "-2" in target
- sql(
- "MERGE INTO %s t USING source "
- + "ON t.id == source.id "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.s = source.c1",
- tableName);
-
- assertEquals(
- "Output should match",
- ImmutableList.of(row(1, "-2")),
- sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeModifiesNullStruct() {
- createAndInitTable("id INT, s STRUCT", "{ \"id\": 1, \"s\": null }");
- createOrReplaceView("source", "{ \"id\": 1, \"n1\": -10 }");
-
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.id "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.s.n1 = s.n1",
- tableName);
-
- assertEquals(
- "Output should match",
- ImmutableList.of(row(1, row(-10, null))),
- sql("SELECT * FROM %s", tableName));
- }
-
- @Test
- public void testMergeRefreshesRelationCache() {
- createAndInitTable("id INT, name STRING", "{ \"id\": 1, \"name\": \"n1\" }");
- createOrReplaceView("source", "{ \"id\": 1, \"name\": \"n2\" }");
-
- Dataset query = spark.sql("SELECT name FROM " + tableName);
- query.createOrReplaceTempView("tmp");
-
- spark.sql("CACHE TABLE tmp");
-
- assertEquals(
- "View should have correct data", ImmutableList.of(row("n1")), sql("SELECT * FROM tmp"));
-
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.id "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.name = s.name",
- tableName);
-
- assertEquals(
- "View should have correct data", ImmutableList.of(row("n2")), sql("SELECT * FROM tmp"));
-
- spark.sql("UNCACHE TABLE tmp");
- }
-
- @Test
- public void testMergeWithMultipleNotMatchedActions() {
- createAndInitTable("id INT, dep STRING", "{ \"id\": 0, \"dep\": \"emp-id-0\" }");
-
- createOrReplaceView(
- "source",
- "id INT, dep STRING",
- "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n"
- + "{ \"id\": 3, \"dep\": \"emp-id-3\" }");
-
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.id "
- + "WHEN NOT MATCHED AND s.id = 1 THEN "
- + " INSERT (dep, id) VALUES (s.dep, -1)"
- + "WHEN NOT MATCHED THEN "
- + " INSERT *",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(-1, "emp-id-1"), // new
- row(0, "emp-id-0"), // kept
- row(2, "emp-id-2"), // new
- row(3, "emp-id-3") // new
- );
- assertEquals(
- "Should have expected rows", expectedRows, sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeWithMultipleConditionalNotMatchedActions() {
- createAndInitTable("id INT, dep STRING", "{ \"id\": 0, \"dep\": \"emp-id-0\" }");
-
- createOrReplaceView(
- "source",
- "id INT, dep STRING",
- "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n"
- + "{ \"id\": 3, \"dep\": \"emp-id-3\" }");
-
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.id "
- + "WHEN NOT MATCHED AND s.id = 1 THEN "
- + " INSERT (dep, id) VALUES (s.dep, -1)"
- + "WHEN NOT MATCHED AND s.id = 2 THEN "
- + " INSERT *",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(-1, "emp-id-1"), // new
- row(0, "emp-id-0"), // kept
- row(2, "emp-id-2") // new
- );
- assertEquals(
- "Should have expected rows", expectedRows, sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeResolvesColumnsByName() {
- createAndInitTable(
- "id INT, badge INT, dep STRING",
- "{ \"id\": 1, \"badge\": 1000, \"dep\": \"emp-id-one\" }\n"
- + "{ \"id\": 6, \"badge\": 6000, \"dep\": \"emp-id-6\" }");
-
- createOrReplaceView(
- "source",
- "badge INT, id INT, dep STRING",
- "{ \"badge\": 1001, \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"badge\": 6006, \"id\": 6, \"dep\": \"emp-id-6\" }\n"
- + "{ \"badge\": 7007, \"id\": 7, \"dep\": \"emp-id-7\" }");
-
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.id "
- + "WHEN MATCHED THEN "
- + " UPDATE SET * "
- + "WHEN NOT MATCHED THEN "
- + " INSERT * ",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1, 1001, "emp-id-1"), // updated
- row(6, 6006, "emp-id-6"), // updated
- row(7, 7007, "emp-id-7") // new
- );
- assertEquals(
- "Should have expected rows",
- expectedRows,
- sql("SELECT id, badge, dep FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeShouldResolveWhenThereAreNoUnresolvedExpressionsOrColumns() {
- // ensures that MERGE INTO will resolve into the correct action even if no columns
- // or otherwise unresolved expressions exist in the query (testing SPARK-34962)
- createAndInitTable("id INT, dep STRING");
-
- createOrReplaceView(
- "source",
- "id INT, dep STRING",
- "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n"
- + "{ \"id\": 3, \"dep\": \"emp-id-3\" }");
-
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON 1 != 1 "
- + "WHEN MATCHED THEN "
- + " UPDATE SET * "
- + "WHEN NOT MATCHED THEN "
- + " INSERT *",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1, "emp-id-1"), // new
- row(2, "emp-id-2"), // new
- row(3, "emp-id-3") // new
- );
- assertEquals(
- "Should have expected rows", expectedRows, sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeWithTableWithNonNullableColumn() {
- createAndInitTable(
- "id INT NOT NULL, dep STRING",
- "{ \"id\": 1, \"dep\": \"emp-id-one\" }\n" + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- createOrReplaceView(
- "source",
- "id INT NOT NULL, dep STRING",
- "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n"
- + "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n"
- + "{ \"id\": 6, \"dep\": \"emp-id-6\" }");
-
- sql(
- "MERGE INTO %s AS t USING source AS s "
- + "ON t.id == s.id "
- + "WHEN MATCHED AND t.id = 1 THEN "
- + " UPDATE SET * "
- + "WHEN MATCHED AND t.id = 6 THEN "
- + " DELETE "
- + "WHEN NOT MATCHED AND s.id = 2 THEN "
- + " INSERT *",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(
- row(1, "emp-id-1"), // updated
- row(2, "emp-id-2")); // new
- assertEquals(
- "Should have expected rows", expectedRows, sql("SELECT * FROM %s ORDER BY id", tableName));
- }
-
- @Test
- public void testMergeWithNonExistingColumns() {
- createAndInitTable("id INT, c STRUCT>");
- createOrReplaceView("source", "{ \"c1\": -100, \"c2\": -200 }");
-
- AssertHelpers.assertThrows(
- "Should complain about the invalid top-level column",
- AnalysisException.class,
- "cannot resolve t.invalid_col",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.invalid_col = s.c2",
- tableName);
- });
-
- AssertHelpers.assertThrows(
- "Should complain about the invalid nested column",
- AnalysisException.class,
- "No such struct field invalid_col",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.c.n2.invalid_col = s.c2",
- tableName);
- });
-
- AssertHelpers.assertThrows(
- "Should complain about the invalid top-level column",
- AnalysisException.class,
- "cannot resolve invalid_col",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.c.n2.dn1 = s.c2 "
- + "WHEN NOT MATCHED THEN "
- + " INSERT (id, invalid_col) VALUES (s.c1, null)",
- tableName);
- });
- }
-
- @Test
- public void testMergeWithInvalidColumnsInInsert() {
- createAndInitTable("id INT, c STRUCT>");
- createOrReplaceView("source", "{ \"c1\": -100, \"c2\": -200 }");
-
- AssertHelpers.assertThrows(
- "Should complain about the nested column",
- AnalysisException.class,
- "Nested fields are not supported inside INSERT clauses",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.c.n2.dn1 = s.c2 "
- + "WHEN NOT MATCHED THEN "
- + " INSERT (id, c.n2) VALUES (s.c1, null)",
- tableName);
- });
-
- AssertHelpers.assertThrows(
- "Should complain about duplicate columns",
- AnalysisException.class,
- "Duplicate column names inside INSERT clause",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.c.n2.dn1 = s.c2 "
- + "WHEN NOT MATCHED THEN "
- + " INSERT (id, id) VALUES (s.c1, null)",
- tableName);
- });
-
- AssertHelpers.assertThrows(
- "Should complain about missing columns",
- AnalysisException.class,
- "must provide values for all columns of the target table",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN NOT MATCHED THEN "
- + " INSERT (id) VALUES (s.c1)",
- tableName);
- });
- }
-
- @Test
- public void testMergeWithInvalidUpdates() {
- createAndInitTable("id INT, a ARRAY>, m MAP");
- createOrReplaceView("source", "{ \"c1\": -100, \"c2\": -200 }");
-
- AssertHelpers.assertThrows(
- "Should complain about updating an array column",
- AnalysisException.class,
- "Updating nested fields is only supported for structs",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.a.c1 = s.c2",
- tableName);
- });
-
- AssertHelpers.assertThrows(
- "Should complain about updating a map column",
- AnalysisException.class,
- "Updating nested fields is only supported for structs",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.m.key = 'new_key'",
- tableName);
- });
- }
-
- @Test
- public void testMergeWithConflictingUpdates() {
- createAndInitTable("id INT, c STRUCT>");
- createOrReplaceView("source", "{ \"c1\": -100, \"c2\": -200 }");
-
- AssertHelpers.assertThrows(
- "Should complain about conflicting updates to a top-level column",
- AnalysisException.class,
- "Updates are in conflict",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.id = 1, t.c.n1 = 2, t.id = 2",
- tableName);
- });
-
- AssertHelpers.assertThrows(
- "Should complain about conflicting updates to a nested column",
- AnalysisException.class,
- "Updates are in conflict for these columns",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.c.n1 = 1, t.id = 2, t.c.n1 = 2",
- tableName);
- });
-
- AssertHelpers.assertThrows(
- "Should complain about conflicting updates to a nested column",
- AnalysisException.class,
- "Updates are in conflict",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED THEN "
- + " UPDATE SET c.n1 = 1, c = named_struct('n1', 1, 'n2', named_struct('dn1', 1, 'dn2', 2))",
- tableName);
- });
- }
-
- @Test
- public void testMergeWithInvalidAssignments() {
- createAndInitTable(
- "id INT NOT NULL, s STRUCT> NOT NULL");
- createOrReplaceView(
- "source",
- "c1 INT, c2 STRUCT NOT NULL, c3 STRING NOT NULL, c4 STRUCT",
- "{ \"c1\": -100, \"c2\": { \"n1\" : 1 }, \"c3\" : 'str', \"c4\": { \"dn2\": 1, \"dn2\": 2 } }");
-
- for (String policy : new String[] {"ansi", "strict"}) {
- withSQLConf(
- ImmutableMap.of("spark.sql.storeAssignmentPolicy", policy),
- () -> {
- AssertHelpers.assertThrows(
- "Should complain about writing nulls to a top-level column",
- AnalysisException.class,
- "Cannot write nullable values to non-null column",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.id = NULL",
- tableName);
- });
-
- AssertHelpers.assertThrows(
- "Should complain about writing nulls to a nested column",
- AnalysisException.class,
- "Cannot write nullable values to non-null column",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.s.n1 = NULL",
- tableName);
- });
-
- AssertHelpers.assertThrows(
- "Should complain about writing missing fields in structs",
- AnalysisException.class,
- "missing fields",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.s = s.c2",
- tableName);
- });
-
- AssertHelpers.assertThrows(
- "Should complain about writing invalid data types",
- AnalysisException.class,
- "Cannot safely cast",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.s.n1 = s.c3",
- tableName);
- });
-
- AssertHelpers.assertThrows(
- "Should complain about writing incompatible structs",
- AnalysisException.class,
- "field name does not match",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.s.n2 = s.c4",
- tableName);
- });
- });
- }
- }
-
- @Test
- public void testMergeWithNonDeterministicConditions() {
- createAndInitTable("id INT, c STRUCT>");
- createOrReplaceView("source", "{ \"c1\": -100, \"c2\": -200 }");
-
- AssertHelpers.assertThrows(
- "Should complain about non-deterministic search conditions",
- AnalysisException.class,
- "Non-deterministic functions are not supported",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 AND rand() > t.id "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.c.n1 = -1",
- tableName);
- });
-
- AssertHelpers.assertThrows(
- "Should complain about non-deterministic update conditions",
- AnalysisException.class,
- "Non-deterministic functions are not supported",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED AND rand() > t.id THEN "
- + " UPDATE SET t.c.n1 = -1",
- tableName);
- });
-
- AssertHelpers.assertThrows(
- "Should complain about non-deterministic delete conditions",
- AnalysisException.class,
- "Non-deterministic functions are not supported",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED AND rand() > t.id THEN "
- + " DELETE",
- tableName);
- });
-
- AssertHelpers.assertThrows(
- "Should complain about non-deterministic insert conditions",
- AnalysisException.class,
- "Non-deterministic functions are not supported",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN NOT MATCHED AND rand() > c1 THEN "
- + " INSERT (id, c) VALUES (1, null)",
- tableName);
- });
- }
-
- @Test
- public void testMergeWithAggregateExpressions() {
- createAndInitTable("id INT, c STRUCT>");
- createOrReplaceView("source", "{ \"c1\": -100, \"c2\": -200 }");
-
- AssertHelpers.assertThrows(
- "Should complain about agg expressions in search conditions",
- AnalysisException.class,
- "Agg functions are not supported",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 AND max(t.id) == 1 "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.c.n1 = -1",
- tableName);
- });
-
- AssertHelpers.assertThrows(
- "Should complain about agg expressions in update conditions",
- AnalysisException.class,
- "Agg functions are not supported",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED AND sum(t.id) < 1 THEN "
- + " UPDATE SET t.c.n1 = -1",
- tableName);
- });
-
- AssertHelpers.assertThrows(
- "Should complain about agg expressions in delete conditions",
- AnalysisException.class,
- "Agg functions are not supported",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED AND sum(t.id) THEN "
- + " DELETE",
- tableName);
- });
-
- AssertHelpers.assertThrows(
- "Should complain about agg expressions in insert conditions",
- AnalysisException.class,
- "Agg functions are not supported",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN NOT MATCHED AND sum(c1) < 1 THEN "
- + " INSERT (id, c) VALUES (1, null)",
- tableName);
- });
- }
-
- @Test
- public void testMergeWithSubqueriesInConditions() {
- createAndInitTable("id INT, c STRUCT>");
- createOrReplaceView("source", "{ \"c1\": -100, \"c2\": -200 }");
-
- AssertHelpers.assertThrows(
- "Should complain about subquery expressions",
- AnalysisException.class,
- "Subqueries are not supported in conditions",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 AND t.id < (SELECT max(c2) FROM source) "
- + "WHEN MATCHED THEN "
- + " UPDATE SET t.c.n1 = s.c2",
- tableName);
- });
-
- AssertHelpers.assertThrows(
- "Should complain about subquery expressions",
- AnalysisException.class,
- "Subqueries are not supported in conditions",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED AND t.id < (SELECT max(c2) FROM source) THEN "
- + " UPDATE SET t.c.n1 = s.c2",
- tableName);
- });
-
- AssertHelpers.assertThrows(
- "Should complain about subquery expressions",
- AnalysisException.class,
- "Subqueries are not supported in conditions",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN MATCHED AND t.id NOT IN (SELECT c2 FROM source) THEN "
- + " DELETE",
- tableName);
- });
-
- AssertHelpers.assertThrows(
- "Should complain about subquery expressions",
- AnalysisException.class,
- "Subqueries are not supported in conditions",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.c1 "
- + "WHEN NOT MATCHED AND s.c1 IN (SELECT c2 FROM source) THEN "
- + " INSERT (id, c) VALUES (1, null)",
- tableName);
- });
- }
-
- @Test
- public void testMergeWithTargetColumnsInInsertConditions() {
- createAndInitTable("id INT, c2 INT");
- createOrReplaceView("source", "{ \"id\": 1, \"value\": 11 }");
-
- AssertHelpers.assertThrows(
- "Should complain about the target column",
- AnalysisException.class,
- "Cannot resolve [c2]",
- () -> {
- sql(
- "MERGE INTO %s t USING source s "
- + "ON t.id == s.id "
- + "WHEN NOT MATCHED AND c2 = 1 THEN "
- + " INSERT (id, c2) VALUES (s.id, null)",
- tableName);
- });
- }
-
- @Test
- public void testMergeWithNonIcebergTargetTableNotSupported() {
- createOrReplaceView("target", "{ \"c1\": -100, \"c2\": -200 }");
- createOrReplaceView("source", "{ \"c1\": -100, \"c2\": -200 }");
-
- AssertHelpers.assertThrows(
- "Should complain non iceberg target table",
- UnsupportedOperationException.class,
- "MERGE INTO TABLE is not supported temporarily.",
- () -> {
- sql(
- "MERGE INTO target t USING source s "
- + "ON t.c1 == s.c1 "
- + "WHEN MATCHED THEN "
- + " UPDATE SET *");
- });
- }
-
- /**
- * Tests a merge where both the source and target are evaluated to be partitioned by
- * SingePartition at planning time but DynamicFileFilterExec will return an empty target.
- */
- @Test
- public void testMergeSinglePartitionPartitioning() {
- // This table will only have a single file and a single partition
- createAndInitTable("id INT", "{\"id\": -1}");
-
- // Coalesce forces our source into a SinglePartition distribution
- spark.range(0, 5).coalesce(1).createOrReplaceTempView("source");
-
- sql(
- "MERGE INTO %s t USING source s ON t.id = s.id "
- + "WHEN MATCHED THEN UPDATE SET *"
- + "WHEN NOT MATCHED THEN INSERT *",
- tableName);
-
- ImmutableList expectedRows =
- ImmutableList.of(row(-1), row(0), row(1), row(2), row(3), row(4));
-
- List result = sql("SELECT * FROM %s ORDER BY id", tableName);
- assertEquals("Should correctly add the non-matching rows", expectedRows, result);
- }
-
- @Test
- public void testMergeEmptyTable() {
- // This table will only have a single file and a single partition
- createAndInitTable("id INT", null);
-
- // Coalesce forces our source into a SinglePartition distribution
- spark.range(0, 5).coalesce(1).createOrReplaceTempView("source");
-
- sql(
- "MERGE INTO %s t USING source s ON t.id = s.id "
- + "WHEN MATCHED THEN UPDATE SET *"
- + "WHEN NOT MATCHED THEN INSERT *",
- tableName);
-
- ImmutableList expectedRows = ImmutableList.of(row(0), row(1), row(2), row(3), row(4));
-
- List result = sql("SELECT * FROM %s ORDER BY id", tableName);
- assertEquals("Should correctly add the non-matching rows", expectedRows, result);
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMergeOnReadDelete.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMergeOnReadDelete.java
deleted file mode 100644
index b4cd6dc04616..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMergeOnReadDelete.java
+++ /dev/null
@@ -1,121 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import static org.mockito.Mockito.doAnswer;
-import static org.mockito.Mockito.spy;
-import static org.mockito.Mockito.when;
-
-import java.util.Map;
-import org.apache.iceberg.AssertHelpers;
-import org.apache.iceberg.RowDelta;
-import org.apache.iceberg.RowLevelOperationMode;
-import org.apache.iceberg.Table;
-import org.apache.iceberg.TableProperties;
-import org.apache.iceberg.exceptions.CommitStateUnknownException;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
-import org.apache.iceberg.spark.source.SparkTable;
-import org.apache.iceberg.spark.source.TestSparkCatalog;
-import org.apache.spark.SparkException;
-import org.apache.spark.sql.connector.catalog.Identifier;
-import org.junit.Test;
-import org.junit.runners.Parameterized;
-
-public class TestMergeOnReadDelete extends TestDelete {
-
- public TestMergeOnReadDelete(
- String catalogName,
- String implementation,
- Map config,
- String fileFormat,
- Boolean vectorized,
- String distributionMode) {
- super(catalogName, implementation, config, fileFormat, vectorized, distributionMode);
- }
-
- @Override
- protected Map extraTableProperties() {
- return ImmutableMap.of(
- TableProperties.FORMAT_VERSION,
- "2",
- TableProperties.DELETE_MODE,
- RowLevelOperationMode.MERGE_ON_READ.modeName());
- }
-
- @Parameterized.AfterParam
- public static void clearTestSparkCatalogCache() {
- TestSparkCatalog.clearTables();
- }
-
- @Test
- public void testCommitUnknownException() {
- createAndInitTable("id INT, dep STRING, category STRING");
-
- // write unpartitioned files
- append(tableName, "{ \"id\": 1, \"dep\": \"hr\", \"category\": \"c1\"}");
- append(
- tableName,
- "{ \"id\": 2, \"dep\": \"hr\", \"category\": \"c1\" }\n"
- + "{ \"id\": 3, \"dep\": \"hr\", \"category\": \"c1\" }");
-
- Table table = validationCatalog.loadTable(tableIdent);
-
- RowDelta newRowDelta = table.newRowDelta();
- RowDelta spyNewRowDelta = spy(newRowDelta);
- doAnswer(
- invocation -> {
- newRowDelta.commit();
- throw new CommitStateUnknownException(new RuntimeException("Datacenter on Fire"));
- })
- .when(spyNewRowDelta)
- .commit();
-
- Table spyTable = spy(table);
- when(spyTable.newRowDelta()).thenReturn(spyNewRowDelta);
- SparkTable sparkTable = new SparkTable(spyTable, false);
-
- ImmutableMap config =
- ImmutableMap.of(
- "type", "hive",
- "default-namespace", "default");
- spark
- .conf()
- .set("spark.sql.catalog.dummy_catalog", "org.apache.iceberg.spark.source.TestSparkCatalog");
- config.forEach(
- (key, value) -> spark.conf().set("spark.sql.catalog.dummy_catalog." + key, value));
- Identifier ident = Identifier.of(new String[] {"default"}, "table");
- TestSparkCatalog.setTable(ident, sparkTable);
-
- // Although an exception is thrown here, write and commit have succeeded
- AssertHelpers.assertThrowsWithCause(
- "Should throw a Commit State Unknown Exception",
- SparkException.class,
- "Writing job aborted",
- CommitStateUnknownException.class,
- "Datacenter on Fire",
- () -> sql("DELETE FROM %s WHERE id = 2", "dummy_catalog.default.table"));
-
- // Since write and commit succeeded, the rows should be readable
- assertEquals(
- "Should have expected rows",
- ImmutableList.of(row(1, "hr", "c1"), row(3, "hr", "c1")),
- sql("SELECT * FROM %s ORDER BY id", "dummy_catalog.default.table"));
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMergeOnReadMerge.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMergeOnReadMerge.java
deleted file mode 100644
index 0453a8787a33..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMergeOnReadMerge.java
+++ /dev/null
@@ -1,46 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import java.util.Map;
-import org.apache.iceberg.RowLevelOperationMode;
-import org.apache.iceberg.TableProperties;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
-
-public class TestMergeOnReadMerge extends TestMerge {
-
- public TestMergeOnReadMerge(
- String catalogName,
- String implementation,
- Map config,
- String fileFormat,
- boolean vectorized,
- String distributionMode) {
- super(catalogName, implementation, config, fileFormat, vectorized, distributionMode);
- }
-
- @Override
- protected Map extraTableProperties() {
- return ImmutableMap.of(
- TableProperties.FORMAT_VERSION,
- "2",
- TableProperties.MERGE_MODE,
- RowLevelOperationMode.MERGE_ON_READ.modeName());
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMergeOnReadUpdate.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMergeOnReadUpdate.java
deleted file mode 100644
index 2f359c7b4021..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMergeOnReadUpdate.java
+++ /dev/null
@@ -1,46 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import java.util.Map;
-import org.apache.iceberg.RowLevelOperationMode;
-import org.apache.iceberg.TableProperties;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
-
-public class TestMergeOnReadUpdate extends TestUpdate {
-
- public TestMergeOnReadUpdate(
- String catalogName,
- String implementation,
- Map config,
- String fileFormat,
- boolean vectorized,
- String distributionMode) {
- super(catalogName, implementation, config, fileFormat, vectorized, distributionMode);
- }
-
- @Override
- protected Map extraTableProperties() {
- return ImmutableMap.of(
- TableProperties.FORMAT_VERSION,
- "2",
- TableProperties.UPDATE_MODE,
- RowLevelOperationMode.MERGE_ON_READ.modeName());
- }
-}
diff --git a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMetadataTables.java b/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMetadataTables.java
deleted file mode 100644
index b11923222694..000000000000
--- a/spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMetadataTables.java
+++ /dev/null
@@ -1,724 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.iceberg.spark.extensions;
-
-import static org.apache.iceberg.types.Types.NestedField.optional;
-
-import java.io.IOException;
-import java.util.Comparator;
-import java.util.List;
-import java.util.Map;
-import java.util.stream.Collectors;
-import java.util.stream.Stream;
-import org.apache.avro.generic.GenericData.Record;
-import org.apache.commons.collections.ListUtils;
-import org.apache.iceberg.FileContent;
-import org.apache.iceberg.HasTableOperations;
-import org.apache.iceberg.ManifestFile;
-import org.apache.iceberg.Schema;
-import org.apache.iceberg.Snapshot;
-import org.apache.iceberg.Table;
-import org.apache.iceberg.TableMetadata;
-import org.apache.iceberg.avro.Avro;
-import org.apache.iceberg.io.CloseableIterable;
-import org.apache.iceberg.io.InputFile;
-import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
-import org.apache.iceberg.relocated.com.google.common.collect.Lists;
-import org.apache.iceberg.spark.Spark3Util;
-import org.apache.iceberg.spark.SparkSchemaUtil;
-import org.apache.iceberg.spark.data.TestHelpers;
-import org.apache.iceberg.spark.source.SimpleRecord;
-import org.apache.iceberg.types.Types;
-import org.apache.spark.sql.Dataset;
-import org.apache.spark.sql.Encoders;
-import org.apache.spark.sql.Row;
-import org.apache.spark.sql.RowFactory;
-import org.apache.spark.sql.catalyst.util.DateTimeUtils;
-import org.apache.spark.sql.types.StructType;
-import org.junit.After;
-import org.junit.Assert;
-import org.junit.Test;
-
-public class TestMetadataTables extends SparkExtensionsTestBase {
-
- public TestMetadataTables(String catalogName, String implementation, Map config) {
- super(catalogName, implementation, config);
- }
-
- @After
- public void removeTables() {
- sql("DROP TABLE IF EXISTS %s", tableName);
- }
-
- @Test
- public void testUnpartitionedTable() throws Exception {
- sql(
- "CREATE TABLE %s (id bigint, data string) USING iceberg TBLPROPERTIES"
- + "('format-version'='2', 'write.delete.mode'='merge-on-read')",
- tableName);
-
- List records =
- Lists.newArrayList(
- new SimpleRecord(1, "a"),
- new SimpleRecord(2, "b"),
- new SimpleRecord(3, "c"),
- new SimpleRecord(4, "d"));
- spark
- .createDataset(records, Encoders.bean(SimpleRecord.class))
- .coalesce(1)
- .writeTo(tableName)
- .append();
-
- sql("DELETE FROM %s WHERE id=1", tableName);
-
- Table table = Spark3Util.loadIcebergTable(spark, tableName);
- List expectedDataManifests = TestHelpers.dataManifests(table);
- List expectedDeleteManifests = TestHelpers.deleteManifests(table);
- Assert.assertEquals("Should have 1 data manifest", 1, expectedDataManifests.size());
- Assert.assertEquals("Should have 1 delete manifest", 1, expectedDeleteManifests.size());
-
- Schema entriesTableSchema = Spark3Util.loadIcebergTable(spark, tableName + ".entries").schema();
- Schema filesTableSchema = Spark3Util.loadIcebergTable(spark, tableName + ".files").schema();
-
- // check delete files table
- Dataset actualDeleteFilesDs = spark.sql("SELECT * FROM " + tableName + ".delete_files");
- List actualDeleteFiles = TestHelpers.selectNonDerived(actualDeleteFilesDs).collectAsList();
- Assert.assertEquals(
- "Metadata table should return one delete file", 1, actualDeleteFiles.size());
-
- List expectedDeleteFiles =
- expectedEntries(
- table, FileContent.POSITION_DELETES, entriesTableSchema, expectedDeleteManifests, null);
- Assert.assertEquals("Should be one delete file manifest entry", 1, expectedDeleteFiles.size());
- TestHelpers.assertEqualsSafe(
- TestHelpers.nonDerivedSchema(actualDeleteFilesDs),
- expectedDeleteFiles.get(0),
- actualDeleteFiles.get(0));
-
- // check data files table
- Dataset actualDataFilesDs = spark.sql("SELECT * FROM " + tableName + ".data_files");
- List actualDataFiles = TestHelpers.selectNonDerived(actualDataFilesDs).collectAsList();
- Assert.assertEquals("Metadata table should return one data file", 1, actualDataFiles.size());
-
- List expectedDataFiles =
- expectedEntries(table, FileContent.DATA, entriesTableSchema, expectedDataManifests, null);
- Assert.assertEquals("Should be one data file manifest entry", 1, expectedDataFiles.size());
- TestHelpers.assertEqualsSafe(
- TestHelpers.nonDerivedSchema(actualDataFilesDs),
- expectedDataFiles.get(0),
- actualDataFiles.get(0));
-
- // check all files table
- Dataset actualFilesDs =
- spark.sql("SELECT * FROM " + tableName + ".files ORDER BY content");
- List actualFiles = TestHelpers.selectNonDerived(actualFilesDs).collectAsList();
-
- Assert.assertEquals("Metadata table should return two files", 2, actualFiles.size());
-
- List expectedFiles =
- Stream.concat(expectedDataFiles.stream(), expectedDeleteFiles.stream())
- .collect(Collectors.toList());
- Assert.assertEquals("Should have two files manifest entries", 2, expectedFiles.size());
- TestHelpers.assertEqualsSafe(
- TestHelpers.nonDerivedSchema(actualFilesDs), expectedFiles.get(0), actualFiles.get(0));
- TestHelpers.assertEqualsSafe(
- TestHelpers.nonDerivedSchema(actualFilesDs), expectedFiles.get(1), actualFiles.get(1));
- }
-
- @Test
- public void testPartitionedTable() throws Exception {
- sql(
- "CREATE TABLE %s (id bigint, data string) "
- + "USING iceberg "
- + "PARTITIONED BY (data) "
- + "TBLPROPERTIES"
- + "('format-version'='2', 'write.delete.mode'='merge-on-read')",
- tableName);
-
- List recordsA =
- Lists.newArrayList(new SimpleRecord(1, "a"), new SimpleRecord(2, "a"));
- spark
- .createDataset(recordsA, Encoders.bean(SimpleRecord.class))
- .coalesce(1)
- .writeTo(tableName)
- .append();
-
- List recordsB =
- Lists.newArrayList(new SimpleRecord(1, "b"), new SimpleRecord(2, "b"));
- spark
- .createDataset(recordsB, Encoders.bean(SimpleRecord.class))
- .coalesce(1)
- .writeTo(tableName)
- .append();
-
- sql("DELETE FROM %s WHERE id=1 AND data='a'", tableName);
- sql("DELETE FROM %s WHERE id=1 AND data='b'", tableName);
-
- Table table = Spark3Util.loadIcebergTable(spark, tableName);
- Schema entriesTableSchema = Spark3Util.loadIcebergTable(spark, tableName + ".entries").schema();
-
- List expectedDataManifests = TestHelpers.dataManifests(table);
- List expectedDeleteManifests = TestHelpers.deleteManifests(table);
- Assert.assertEquals("Should have 2 data manifests", 2, expectedDataManifests.size());
- Assert.assertEquals("Should have 2 delete manifests", 2, expectedDeleteManifests.size());
-
- Schema filesTableSchema =
- Spark3Util.loadIcebergTable(spark, tableName + ".delete_files").schema();
-
- // Check delete files table
- List expectedDeleteFiles =
- expectedEntries(
- table, FileContent.POSITION_DELETES, entriesTableSchema, expectedDeleteManifests, "a");
- Assert.assertEquals(
- "Should have one delete file manifest entry", 1, expectedDeleteFiles.size());
-
- Dataset actualDeleteFilesDs =
- spark.sql("SELECT * FROM " + tableName + ".delete_files " + "WHERE partition.data='a'");
- List actualDeleteFiles = actualDeleteFilesDs.collectAsList();
-
- Assert.assertEquals(
- "Metadata table should return one delete file", 1, actualDeleteFiles.size());
- TestHelpers.assertEqualsSafe(
- TestHelpers.nonDerivedSchema(actualDeleteFilesDs),
- expectedDeleteFiles.get(0),
- actualDeleteFiles.get(0));
-
- // Check data files table
- List expectedDataFiles =
- expectedEntries(table, FileContent.DATA, entriesTableSchema, expectedDataManifests, "a");
- Assert.assertEquals("Should have one data file manifest entry", 1, expectedDataFiles.size());
-
- Dataset actualDataFilesDs =
- spark.sql("SELECT * FROM " + tableName + ".data_files " + "WHERE partition.data='a'");
-
- List actualDataFiles = TestHelpers.selectNonDerived(actualDataFilesDs).collectAsList();
- Assert.assertEquals("Metadata table should return one data file", 1, actualDataFiles.size());
- TestHelpers.assertEqualsSafe(
- TestHelpers.nonDerivedSchema(actualDataFilesDs),
- expectedDataFiles.get(0),
- actualDataFiles.get(0));
-
- List actualPartitionsWithProjection =
- spark.sql("SELECT file_count FROM " + tableName + ".partitions ").collectAsList();
- Assert.assertEquals(
- "Metadata table should return two partitions record",
- 2,
- actualPartitionsWithProjection.size());
- for (int i = 0; i < 2; ++i) {
- Assert.assertEquals(1, actualPartitionsWithProjection.get(i).get(0));
- }
-
- // Check files table
- List expectedFiles =
- Stream.concat(expectedDataFiles.stream(), expectedDeleteFiles.stream())
- .collect(Collectors.toList());
- Assert.assertEquals("Should have two file manifest entries", 2, expectedFiles.size());
-
- Dataset actualFilesDs =
- spark.sql(
- "SELECT * FROM " + tableName + ".files " + "WHERE partition.data='a' ORDER BY content");
- List actualFiles = TestHelpers.selectNonDerived(actualFilesDs).collectAsList();
- Assert.assertEquals("Metadata table should return two files", 2, actualFiles.size());
- TestHelpers.assertEqualsSafe(
- TestHelpers.nonDerivedSchema(actualFilesDs), expectedFiles.get(0), actualFiles.get(0));
- TestHelpers.assertEqualsSafe(
- TestHelpers.nonDerivedSchema(actualFilesDs), expectedFiles.get(1), actualFiles.get(1));
- }
-
- @Test
- public void testAllFilesUnpartitioned() throws Exception {
- sql(
- "CREATE TABLE %s (id bigint, data string) USING iceberg TBLPROPERTIES"
- + "('format-version'='2', 'write.delete.mode'='merge-on-read')",
- tableName);
-
- List records =
- Lists.newArrayList(
- new SimpleRecord(1, "a"),
- new SimpleRecord(2, "b"),
- new SimpleRecord(3, "c"),
- new SimpleRecord(4, "d"));
- spark
- .createDataset(records, Encoders.bean(SimpleRecord.class))
- .coalesce(1)
- .writeTo(tableName)
- .append();
-
- // Create delete file
- sql("DELETE FROM %s WHERE id=1", tableName);
-
- Table table = Spark3Util.loadIcebergTable(spark, tableName);
- List expectedDataManifests = TestHelpers.dataManifests(table);
- Assert.assertEquals("Should have 1 data manifest", 1, expectedDataManifests.size());
- List expectedDeleteManifests = TestHelpers.deleteManifests(table);
- Assert.assertEquals("Should have 1 delete manifest", 1, expectedDeleteManifests.size());
-
- // Clear table to test whether 'all_files' can read past files
- List results = sql("DELETE FROM %s", tableName);
- Assert.assertEquals("Table should be cleared", 0, results.size());
-
- Schema entriesTableSchema = Spark3Util.loadIcebergTable(spark, tableName + ".entries").schema();
- Schema filesTableSchema =
- Spark3Util.loadIcebergTable(spark, tableName + ".all_data_files").schema();
-
- // Check all data files table
- Dataset actualDataFilesDs = spark.sql("SELECT * FROM " + tableName + ".all_data_files");
- List actualDataFiles = TestHelpers.selectNonDerived(actualDataFilesDs).collectAsList();
-
- List expectedDataFiles =
- expectedEntries(table, FileContent.DATA, entriesTableSchema, expectedDataManifests, null);
- Assert.assertEquals("Should be one data file manifest entry", 1, expectedDataFiles.size());
- Assert.assertEquals("Metadata table should return one data file", 1, actualDataFiles.size());
- TestHelpers.assertEqualsSafe(
- TestHelpers.nonDerivedSchema(actualDataFilesDs),
- expectedDataFiles.get(0),
- actualDataFiles.get(0));
-
- // Check all delete files table
- Dataset actualDeleteFilesDs =
- spark.sql("SELECT * FROM " + tableName + ".all_delete_files");
- List actualDeleteFiles = TestHelpers.selectNonDerived(actualDeleteFilesDs).collectAsList();
- List expectedDeleteFiles =
- expectedEntries(
- table, FileContent.POSITION_DELETES, entriesTableSchema, expectedDeleteManifests, null);
- Assert.assertEquals("Should be one delete file manifest entry", 1, expectedDeleteFiles.size());
- Assert.assertEquals(
- "Metadata table should return one delete file", 1, actualDeleteFiles.size());
- TestHelpers.assertEqualsSafe(
- TestHelpers.nonDerivedSchema(actualDeleteFilesDs),
- expectedDeleteFiles.get(0),
- actualDeleteFiles.get(0));
-
- // Check all files table
- Dataset