DB-6045 Support compression for avro external table #2065
Conversation
changli6
assigned dgomezferro, wb14123, msirek, jyuanca, OlegMazurov, wtang16 and yxia92
| } | ||
| else if (storedAs.toLowerCase().equals("a")) { | ||
| empty.write().partitionBy(partitionByCols.toArray(new String[partitionByCols.size()])) | ||
| .mode(SaveMode.Append).format("com.databricks.spark.avro").save(location); |
There was a problem hiding this comment.
why option("compression", compression) does not work for avro?
There was a problem hiding this comment.
I tried but not work.
In the Spark-Avro doc (https://docs.databricks.com/spark/latest/data-sources/read-avro.html), the example showed the compression code was accepted by spark conf settting.
You can also specify Avro compression options:
Copy to clipboardCopy
import com.databricks.spark.avro._
// configuration to use deflate compression
spark.conf.set("spark.sql.avro.compression.codec", "deflate")
Others raised the same confusion of compression code setting, but was not responded.
(databricks/spark-avro#259)
| if (compression.toLowerCase().equals("zlib")) | ||
| compression = "deflate"; | ||
| spliceSpark.conf().set("spark.sql.avro.compression.codec",compression); | ||
| } |
There was a problem hiding this comment.
spark session and configuration is shared by all spark jobs. If two "create external table" are being executed, and one wants to compress avro file, the other does not, or they try to compress in different format, setting a global configuration may not work
There was a problem hiding this comment.
Thanks @jyuanca Global session is improper, let me address it.
| int insertCount = methodWatcher.executeUpdate(String.format("insert into compressed_zlib_avro_test values ('XXXX')," + | ||
| "('YYYY')")); | ||
| Assert.assertEquals("insertCount is wrong",2,insertCount); | ||
| ResultSet rs = methodWatcher.executeQuery("select * from compressed_zlib_avro_test"); | ||
| Assert.assertEquals("COL1 |\n" + |
There was a problem hiding this comment.
The test cannot tell whether the table is compressed
|
Global session level setting of Avro compression may impact other Avro-related spark jobs, and spark-avro doesn't support DataFrame level setting for compression so far, we have to defer this feature until Spark implemented the DataFrame level setting for Avro file. |
No description provided.