v4.0.1

New Feature

• Initial supports to ORC input file format.
• Initial supports to RSS framework and Apache Celeborn shuffle service.

Improvement

• Optimize AggExec by supporting Implement columnar-based aggregation.
• Use custom implemented hashmap implement for aggregation.
• Supports specialized count(0).
• Optimize bloom filter by reusing same bloom filter in the same executor.
• Optimize bloom filter by supporting shrinking.
• Optimize reading parquet files by supporting parallel reading.
• Improve spill file deletion logics.

Bug fixes

• Fix file not found for path with url encoded character.
• Fix Hashaggregate convert job throwing ScalaReflectionException.
• Fix pruning error while reading parquet files with multiple row groups.
• Fix incorrect number of tasks due to missing shuffleOrigin.
• Fix record batch creating error when hash joining with empty input.

Other

• Upgrade datafusion/arrow dependency to v42/v53.
• Replace gxhash with foldhash for better compatibility on some hardwares.
• Other minor improvement & fixes.

What's Changed

• update to datafusion-v42 by @richox in #574
• doc: update tpc-h benchmark result by @richox in #614
• fix running error for Spark 3.2.0 and 3.2.1 by @XorSum in #602
• [BLAZE-287][FOLLOWUP] Use JavaUtils#newConcurrentHashMap to speed up ConcurrentHashMap#computeIfAbsent by @SteNicholas in #615
• fix pruning error while reading parquet files with multiple row groups by @richox in #616
• use custom implemented hashmap for aggregation by @richox in #617
• supports specialized count(0) by @richox in #619
• optimize bloom filter by @richox in #620
• [BLAZE-627] Make ORC and Parquet format detection more generic by @dixingxing0 in #628
• Bump object_store from 0.11.0 to 0.11.1 by @dependabot in #622
• improve memory management by @richox in #621
• Bump bytes from 1.7.2 to 1.8.0 by @dependabot in #625
• Bump tokio from 1.40.0 to 1.41.0 by @dependabot in #629
• Bump uuid from 1.10.0 to 1.11.0 by @dependabot in #618
• replace gxhash with foldhash by @richox in #624
• Bump sonic-rs from 0.3.13 to 0.3.14 by @dependabot in #623
• add new blaze logo by @richox in #633
• improve NativeOrcScan by @richox in #631
• fix Hashaggregate convert job throw ScalaReflectionException by @leizhang5s in #637
• tpcd benchmarkrunner : add orc format support by @leizhang5s in #639
• Bump bigdecimal from 0.4.5 to 0.4.6 by @dependabot in #638
• [BLAZE-573][FOLLOWUP] Bump Spark from 3.4.3 to 3.4.4 by @SteNicholas in #640
• Bump tempfile from 3.13.0 to 3.14.0 by @dependabot in #641
• parquet reading improvements by @richox in #650
• Bump tokio from 1.41.0 to 1.41.1 by @dependabot in #642
• Bump sonic-rs from 0.3.14 to 0.3.16 by @dependabot in #647
• AggExec: implement columnar accumulator states. by @richox in #646
• chore: Make spotless plugin happy by @zuston in #653
• fix(spill): Explicitly delete spill file for FileBasedSpillBuf after release by @zuston in #654
• fix(spill): Delete spill file when dropping for rust FileSpill by @zuston in #660
• [BLAZE-664] Bump Celeborn version from 0.5.1 to 0.5.2 by @SteNicholas in #665
• [MINOR] Avoid NPE when native lib is not found by @wForget in #668
• Fix build on windows by @wForget in #666
• fix(shuffle): Progagate shuffle origin to native exchange exec to make AQE rebalance valid by @zuston in #663
• improvement: Add numOfPartitions metrics for exchange exec to align with vanilla spark by @zuston in #669
• code refactoring by @richox in #658
• Introduce base blaze sql test suite by @wForget in #674
• Bump bytes from 1.8.0 to 1.9.0 by @dependabot in #671
• Bump tokio from 1.41.0 to 1.41.1 by @dependabot in #676
• code refactoring by @richox in #677
• Fix file not found for path with url encoded character by @wForget in #679
• Followup to #674, add -r for rm by @wForget in #681
• Create RecordBatch with num_rows option to avoid bhj error caused by empty output_schema by @wForget in #683
• Bump tokio from 1.41.1 to 1.42.0 by @dependabot in #680
• reduce jar package size and fix CI check error by @richox in #687

New Contributors

• @dixingxing0 made their first contribution in #628
• @leizhang5s made their first contribution in #637
• @wForget made their first contribution in #668

Full Changelog: v4.0.0...v4.0.1