Spark: Test reading default values in Spark #11832
Conversation
| .isInstanceOf(IllegalArgumentException.class) | ||
| .hasMessage("Missing required field: missing_str"); | ||
| .hasRootCauseInstanceOf(IllegalArgumentException.class) | ||
| .hasMessageContaining("Missing required field: missing_str"); |
There was a problem hiding this comment.
This was needed to validate the reader failure in testMissingRequiredWithoutDefault in Spark scans because the failure happens on executors and is wrapped in SparkException when it is thrown on the driver.
| @@ -542,44 +539,4 @@ public void testPrimitiveTypeDefaultValues(Type.PrimitiveType type, Object defau | |||
|
|
|||
| writeAndValidate(writeSchema, readSchema); | |||
| } | |||
|
|
|||
| protected void withSQLConf(Map<String, String> conf, Action action) throws IOException { | |||
| @@ -96,11 +96,20 @@ public static void assertEqualsSafe(Types.StructType struct, List<Record> recs, | |||
|
|
|||
| public static void assertEqualsSafe(Types.StructType struct, Record rec, Row row) { | |||
| List<Types.NestedField> fields = struct.fields(); | |||
| for (int i = 0; i < fields.size(); i += 1) { | |||
| Type fieldType = fields.get(i).type(); | |||
| for (int readPos = 0; readPos < fields.size(); readPos += 1) { | |||
| import org.junit.jupiter.api.Test; | ||
| import org.junit.jupiter.api.io.TempDir; | ||
|
|
||
| public abstract class DataFrameWriteTestBase extends ScanTestBase { |
There was a problem hiding this comment.
New base suite for tests of data frame writes, which replaces TestDataFrameWrites and ParameterizedAvroDataTest.
| import org.junit.jupiter.api.io.TempDir; | ||
|
|
||
| /** An AvroDataScan test that validates data by reading through Spark */ | ||
| public abstract class ScanTestBase extends AvroDataTest { |
There was a problem hiding this comment.
New base class for scan tests (TestAvroScan, TestParquetScan, TestParquetVectorizedScan).
|
|
||
| @Parameters(name = "format = {0}") | ||
| public static Collection<String> parameters() { | ||
| return Arrays.asList("parquet", "avro", "orc"); |
There was a problem hiding this comment.
This was broken into DataFrameWriteTestBase and subclasses for each format:
TestAvroDataFrameWriteTestParquetDataFrameWriteTestORCDataFrameWrite
| } | ||
|
|
||
| @TestTemplate | ||
| public void testWriteWithCustomDataLocation() throws IOException { |
There was a problem hiding this comment.
Replaced by DataFrameWriteTestBase#testAlternateLocation.
| } | ||
|
|
||
| @TestTemplate | ||
| public void testNullableWithWriteOption() throws IOException { |
There was a problem hiding this comment.
Assumes Spark 2.x so this is no longer needed.
| } | ||
|
|
||
| @TestTemplate | ||
| public void testNullableWithSparkSqlOption() throws IOException { |
There was a problem hiding this comment.
Assumes Spark 2.x so this is no longer needed.
| } | ||
|
|
||
| @TestTemplate | ||
| public void testFaultToleranceOnWrite() throws IOException { |
There was a problem hiding this comment.
I dropped this test because it is testing basic Spark behavior and doesn't belong in scan and write tests for specific schemas. I didn't move it anywhere because I don't think it is a valuable test. Spark stage failures throw exceptions and don't commit. I think it was originally trying to check for side-effects, but that isn't necessary in Iceberg.
|
Thanks for the reviews, @Fokko! |
This updates Spark's tests for scans and data frame writes to validate default values.
This fixes problems found in testing:
ReassignIdswas dropping defaultsSchemaParserdid not support eitherinitial-defaultorwrite-defaultSchemaParserdid not have a test suiteThis also refactors the data frame writer tests and removes
ParameterizedAvroDataTestthat was an unnecessary copy ofAvroDataTest. To avoid needing the duplicate test suite, this updates the tests to inherit from a base class like the scan tests. Last, there were a few unnecessary tests that have been removed. One was testing basic Spark behavior (no commit if an action fails) and the others were only valid for Spark 2.x.