Bump Spark 3.5.4 #11731

pan3793 · 2024-12-09T07:04:32Z

Bump Spark 3.5.4, with minimal necessary changes related to SPARK-50235 and SPARK-50463.

Note: this PR targets to 1.7.2 and 1.6.2 (for Java8 users)

pan3793 · 2024-12-10T02:46:23Z

the upgrading causes a core dump in iceberg-spark-3.5's tests

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f8529a34563, pid=83775, tid=83793
#
# JRE version: OpenJDK Runtime Environment Zulu17.54+21-CRaC-CA (17.0.13+11) (build 17.0.13+11-LTS)
# Java VM: OpenJDK 64-Bit Server VM Zulu17.54+21-CRaC-CA (17.0.13+11-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0xc34563]  ProtectionDomainEntry::object_no_keepalive()+0x3
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /home/runner/work/iceberg/iceberg/spark/v3.5/spark/core.83775)
#
# An error report file with more information is saved as:
# /home/runner/work/iceberg/iceberg/spark/v3.5/spark/hs_err_pid83775.log
#
# Compiler replay data is saved as:
# /home/runner/work/iceberg/iceberg/spark/v3.5/spark/replay_pid83775.log
#
# If you would like to submit a bug report, please visit:
#   http://www.azul.com/support/
#
> Task :iceberg-spark:iceberg-spark-3.5_2.12:test FAILED

pan3793 · 2024-12-10T03:14:09Z

the failure seems related to arrow

minimal reproduce command

./gradlew -DsparkVersions=3.5 -DscalaVersion=2.12 -DhiveVersions= -DflinkVersions= -DkafkaVersions= \
    :iceberg-spark:iceberg-spark-3.5_2.12:test --tests=TestRewriteDataFilesAction \
    -Pquick=true -x javadoc

pan3793 · 2024-12-10T15:51:02Z

update: the failure is related to SPARK-50235, and the test passed on Spark 3.5.4 RC1 with reverting that patch

viirya · 2024-12-10T17:24:40Z

Please check for some discussion there apache/spark#49131 (comment)

LuciferYang · 2024-12-10T18:13:43Z

Please check for some discussion there apache/spark#49131 (comment)

Yes, I tested it locally and found that after adding a no-op closeIfFreeable method to the IcebergArrowColumnVector.java in version 3.5, all the test cases in TestRewriteDataFilesAction were able to pass. We can further test it after the release of 3.5.4 RC2.

pan3793 · 2024-12-11T02:16:32Z

@viirya @LuciferYang thank you for the guidance, let me try

pan3793 · 2024-12-11T02:39:37Z

...5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java

+  // If a column vector is writable or constant, it should override this method and do nothing.
+  // See more details at SPARK-50235, SPARK-50463 (Fixed in Spark 3.5.4)
+  @Override
+  public void closeIfNotWritable() {}


will rename to closeIfFreeable after RC2 available

If a column vector is writable or constant, it should override this method and do nothing.

we have ConstantColumnVector as well, should we do it vector classes which absolutely requires it ?

I checked the codebase, other vectors overwrite void close() {}, so they are not affected

singhpk234

Nice catch @pan3793 !

singhpk234 · 2024-12-11T03:15:17Z

...5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java

+  // If a column vector is writable or constant, it should override this method and do nothing.
+  // See more details at SPARK-50235, SPARK-50463 (Fixed in Spark 3.5.4)
+  @Override
+  public void closeIfNotWritable() {}


If a column vector is writable or constant, it should override this method and do nothing.

we have ConstantColumnVector as well, should we do it vector classes which absolutely requires it ?

pan3793 · 2024-12-11T04:21:27Z

FYI @LuciferYang @viirya @dongjoon-hyun @wForget

CI is green now, and I think the current Spark branch-3.5 is in good shape for Iceberg.

dongjoon-hyun · 2024-12-11T16:07:29Z

Thank you for checking Apache Iceberg side, @pan3793 .

Fokko · 2024-12-17T14:54:39Z

Thanks @pan3793 for catching this early. Could you create a separate PR for just the patch? We can include this in 1.7.2.

pan3793 · 2024-12-17T16:07:41Z

@Fokko I opened #11802

...5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java

pan3793 · 2024-12-20T12:33:34Z

Spark 3.5.4 RC3 passed the vote and the jars were available on Maven Central a few minutes ago, I removed the staging repo and it's ready to go.

cc @nastra @Fokko and @jbonofre

Fokko · 2024-12-20T14:08:52Z

...5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java

@@ -59,6 +59,12 @@ public void close() {
    accessor.close();
  }

+  @Override


Did we land on adding the @Override? Now it doesn't compile against Spark 3.5.2. Leaving out the annotation is safer (it will still work with ≥3.5.4).

Did we land on adding the @Override? Now it doesn't compile against Spark 3.5.2. Leaving out the annotation is safer (it will still work with ≥3.5.4).

+1

removed.

both keeping and removing are fine to me, it has no difference in runtime

...5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java

…ectorized/IcebergArrowColumnVector.java

github-actions bot added the build label Dec 9, 2024

pan3793 mentioned this pull request Dec 10, 2024

[SPARK-50463][SQL][3.5] Fix ConstantColumnVector with Columnar to Row conversion apache/spark#49131

Closed

dongjoon-hyun mentioned this pull request Dec 10, 2024

[SPARK-50235][SQL] Clean up ColumnVector resource after processing all rows in ColumnarToRowExec apache/spark#48767

Closed

github-actions bot added the spark label Dec 11, 2024

pan3793 commented Dec 11, 2024

View reviewed changes

singhpk234 reviewed Dec 11, 2024

View reviewed changes

pan3793 force-pushed the spark-3.5.4 branch from 9144835 to 99bcc1c Compare December 16, 2024 09:11

pan3793 changed the title ~~Bump Spark 3.5.4 RC1~~ Bump Spark 3.5.4 RC2 Dec 16, 2024

pan3793 added 3 commits December 17, 2024 21:55

Bump Spark 3.5.4 RC1

c3d0b8a

3.5.4 RC2

18bb711

Spark 3.5.4 RC3

2feb6b0

pan3793 force-pushed the spark-3.5.4 branch from dfdc640 to 2feb6b0 Compare December 17, 2024 13:55

LuciferYang mentioned this pull request Dec 17, 2024

Spark 3.5: Adapt to Spark 3.5.4 #11802

Closed

nastra reviewed Dec 17, 2024

View reviewed changes

...5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java Outdated Show resolved Hide resolved

pan3793 added 2 commits December 18, 2024 10:28

comment

0c25054

release

7bb2100

pan3793 changed the title ~~Bump Spark 3.5.4 RC2~~ Bump Spark 3.5.4 Dec 20, 2024

pan3793 marked this pull request as ready for review December 20, 2024 12:31

Fokko added this to the Iceberg 1.7.2 milestone Dec 20, 2024

Fokko reviewed Dec 20, 2024

View reviewed changes

pan3793 commented Dec 20, 2024

View reviewed changes

...5/spark/src/main/java/org/apache/iceberg/spark/data/vectorized/IcebergArrowColumnVector.java Outdated Show resolved Hide resolved

Update spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/v…

dc7aea5

…ectorized/IcebergArrowColumnVector.java

huaxingao approved these changes Dec 20, 2024

View reviewed changes

viirya approved these changes Dec 20, 2024

View reviewed changes

LuciferYang approved these changes Dec 21, 2024

View reviewed changes

Fokko approved these changes Dec 21, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump Spark 3.5.4 #11731

Bump Spark 3.5.4 #11731

pan3793 commented Dec 9, 2024 •

edited

Loading

pan3793 commented Dec 10, 2024

pan3793 commented Dec 10, 2024

pan3793 commented Dec 10, 2024 •

edited

Loading

viirya commented Dec 10, 2024

LuciferYang commented Dec 10, 2024

pan3793 commented Dec 11, 2024

pan3793 Dec 11, 2024

singhpk234 Dec 11, 2024

pan3793 Dec 11, 2024 •

edited

Loading

singhpk234 left a comment

singhpk234 Dec 11, 2024

pan3793 commented Dec 11, 2024

dongjoon-hyun commented Dec 11, 2024

Fokko commented Dec 17, 2024

pan3793 commented Dec 17, 2024

pan3793 commented Dec 20, 2024

Fokko Dec 20, 2024

LuciferYang Dec 20, 2024

pan3793 Dec 20, 2024

Bump Spark 3.5.4 #11731

Are you sure you want to change the base?

Bump Spark 3.5.4 #11731

Conversation

pan3793 commented Dec 9, 2024 • edited Loading

pan3793 commented Dec 10, 2024

pan3793 commented Dec 10, 2024

pan3793 commented Dec 10, 2024 • edited Loading

viirya commented Dec 10, 2024

LuciferYang commented Dec 10, 2024

pan3793 commented Dec 11, 2024

pan3793 Dec 11, 2024

Choose a reason for hiding this comment

singhpk234 Dec 11, 2024

Choose a reason for hiding this comment

pan3793 Dec 11, 2024 • edited Loading

Choose a reason for hiding this comment

singhpk234 left a comment

Choose a reason for hiding this comment

singhpk234 Dec 11, 2024

Choose a reason for hiding this comment

pan3793 commented Dec 11, 2024

dongjoon-hyun commented Dec 11, 2024

Fokko commented Dec 17, 2024

pan3793 commented Dec 17, 2024

pan3793 commented Dec 20, 2024

Fokko Dec 20, 2024

Choose a reason for hiding this comment

LuciferYang Dec 20, 2024

Choose a reason for hiding this comment

pan3793 Dec 20, 2024

Choose a reason for hiding this comment

pan3793 commented Dec 9, 2024 •

edited

Loading

pan3793 commented Dec 10, 2024 •

edited

Loading

pan3793 Dec 11, 2024 •

edited

Loading