AVRO-3930: Mapping org.apache.hadoop.io.ZStandardCodec to zstandard

[yaooqinn]

What is the purpose of the change

Mapping org.apache.hadoop.io.ZStandardCodec to zstandard in
HadoopCodecFactory

Verifying this change

This change added tests

Documentation

• Does this pull request introduce a new feature? (yes)
• If yes, how is the feature documented? (JavaDocs)

[martin-g]

Not related to this PR but I wonder how CodecFactory.fromString("deflate"); works ?!
In one of the tests it returns DeflateCodec.class, while in another it returns GzipCodec.class

[KalleOlaviNiemitalo]

@martin-g, what's the test in which CodecFactory.fromString("deflate") returns GZipCodec.class? To me, it looks like there is no GZipCodec class, and HadoopCodecFactory uses the DeflateCodec class for both "org.apache.hadoop.io.compress.DeflateCodec" and "org.apache.hadoop.io.compress.GZipCodec", whereas CodecFactory doesn't have this kind of aliasing built-in.

[martin-g]

@KalleOlaviNiemitalo I meant these two tests:

• avro/lang/java/mapred/src/test/java/org/apache/avro/hadoop/file/TestHadoopCodecFactory.java

  Line 29 in 61fd3e4

  void hadoopCodecFactoryDeflate() {

• avro/lang/java/mapred/src/test/java/org/apache/avro/hadoop/file/TestHadoopCodecFactory.java

  Line 50 in 61fd3e4

  void hadoopCodecFactoryGZip() {

I didn't fire an IDE to debug what is going on. It seems one is an alias to the other...

[KalleOlaviNiemitalo]

The compress method in org.apache.avro.file.DeflateCodec calls getDeflater, which then passes the nowrap flag to java.util.zip.Deflater, and the value of nowrap is always true, so it will omit the GZip header. It seems wrong to me that "org.apache.hadoop.io.compress.GZipCodec" is mapped to that implementation, especially as GZipCodec in Hadoop uses ZlibCompressor.CompressionHeader.GZIP_FORMAT. OTOH, GZipCodec in Hadoop apparently supports both formats when decompressing, so the mismatch is perhaps not causing problems in practice.

[KalleOlaviNiemitalo]

Oh and Hadoop spells "GzipCodec" with a lowercase "z", but HadoopCodecFactory in Avro registers "GZipCodec" with an uppercase "Z". Perhaps it is already not interoperable, then.

[yaooqinn]

Thanks for the review, @martin-g, and @KalleOlaviNiemitalo.

Shall I fix the GzipCodec issue here? Or with a follow-up, as it seems orthogonal?

[martin-g]

In a separate JIRA+PR. Thank you!

[yaooqinn]

Hi, @martin-g @KalleOlaviNiemitalo. Do we have any comment I missed to address for this PR？

[martin-g]

@yaooqinn I'll leave it to the Java SDK maintainers to review and merge.

[yaooqinn]

OK. I got it. Thank you @martin-g