Releases: kwai/blaze
Releases · kwai/blaze
v4.0.1
New Feature
- Initial supports to ORC input file format.
- Initial supports to RSS framework and Apache Celeborn shuffle service.
Improvement
- Optimize AggExec by supporting Implement columnar-based aggregation.
- Use custom implemented hashmap implement for aggregation.
- Supports specialized count(0).
- Optimize bloom filter by reusing same bloom filter in the same executor.
- Optimize bloom filter by supporting shrinking.
- Optimize reading parquet files by supporting parallel reading.
- Improve spill file deletion logics.
Bug fixes
- Fix file not found for path with url encoded character.
- Fix Hashaggregate convert job throwing ScalaReflectionException.
- Fix pruning error while reading parquet files with multiple row groups.
- Fix incorrect number of tasks due to missing shuffleOrigin.
- Fix record batch creating error when hash joining with empty input.
Other
- Upgrade datafusion/arrow dependency to v42/v53.
- Replace gxhash with foldhash for better compatibility on some hardwares.
- Other minor improvement & fixes.
What's Changed
- update to datafusion-v42 by @richox in #574
- doc: update tpc-h benchmark result by @richox in #614
- fix running error for Spark 3.2.0 and 3.2.1 by @XorSum in #602
- [BLAZE-287][FOLLOWUP] Use JavaUtils#newConcurrentHashMap to speed up ConcurrentHashMap#computeIfAbsent by @SteNicholas in #615
- fix pruning error while reading parquet files with multiple row groups by @richox in #616
- use custom implemented hashmap for aggregation by @richox in #617
- supports specialized count(0) by @richox in #619
- optimize bloom filter by @richox in #620
- [BLAZE-627] Make ORC and Parquet format detection more generic by @dixingxing0 in #628
- Bump object_store from 0.11.0 to 0.11.1 by @dependabot in #622
- improve memory management by @richox in #621
- Bump bytes from 1.7.2 to 1.8.0 by @dependabot in #625
- Bump tokio from 1.40.0 to 1.41.0 by @dependabot in #629
- Bump uuid from 1.10.0 to 1.11.0 by @dependabot in #618
- replace gxhash with foldhash by @richox in #624
- Bump sonic-rs from 0.3.13 to 0.3.14 by @dependabot in #623
- add new blaze logo by @richox in #633
- improve NativeOrcScan by @richox in #631
- fix Hashaggregate convert job throw ScalaReflectionException by @leizhang5s in #637
- tpcd benchmarkrunner : add orc format support by @leizhang5s in #639
- Bump bigdecimal from 0.4.5 to 0.4.6 by @dependabot in #638
- [BLAZE-573][FOLLOWUP] Bump Spark from 3.4.3 to 3.4.4 by @SteNicholas in #640
- Bump tempfile from 3.13.0 to 3.14.0 by @dependabot in #641
- parquet reading improvements by @richox in #650
- Bump tokio from 1.41.0 to 1.41.1 by @dependabot in #642
- Bump sonic-rs from 0.3.14 to 0.3.16 by @dependabot in #647
- AggExec: implement columnar accumulator states. by @richox in #646
- chore: Make spotless plugin happy by @zuston in #653
- fix(spill): Explicitly delete spill file for FileBasedSpillBuf after release by @zuston in #654
- fix(spill): Delete spill file when dropping for rust FileSpill by @zuston in #660
- [BLAZE-664] Bump Celeborn version from 0.5.1 to 0.5.2 by @SteNicholas in #665
- [MINOR] Avoid NPE when native lib is not found by @wForget in #668
- Fix build on windows by @wForget in #666
- fix(shuffle): Progagate shuffle origin to native exchange exec to make AQE rebalance valid by @zuston in #663
- improvement: Add numOfPartitions metrics for exchange exec to align with vanilla spark by @zuston in #669
- code refactoring by @richox in #658
- Introduce base blaze sql test suite by @wForget in #674
- Bump bytes from 1.8.0 to 1.9.0 by @dependabot in #671
- Bump tokio from 1.41.0 to 1.41.1 by @dependabot in #676
- code refactoring by @richox in #677
- Fix file not found for path with url encoded character by @wForget in #679
- Followup to #674, add -r for rm by @wForget in #681
- Create RecordBatch with num_rows option to avoid bhj error caused by empty output_schema by @wForget in #683
- Bump tokio from 1.41.1 to 1.42.0 by @dependabot in #680
- reduce jar package size and fix CI check error by @richox in #687
New Contributors
- @dixingxing0 made their first contribution in #628
- @leizhang5s made their first contribution in #637
- @wForget made their first contribution in #668
Full Changelog: v4.0.0...v4.0.1
Contributors
dixingxing0, richox, and 6 other contributors
Assets 8
v4.0.0
New features
- supports spark3.0/3.1/3.2/3.3/3.4/3.5.
- supports integrating with Apache Celeborn.
- supports native ORC input format.
- supports bloom filter join introduced in spark 3.5.
- supports forceShuffledHashJoin for running tpch/tpcds benchmarks.
- new supported native expression/functions: year, month, day, md5.
Bug fixes
- add missing UDTF.terminate() invokes.
- fix NPE while executing some native spark physical plans.
Performance
- use custom implemented hash table for faster joining, supporting SIMD, bulk searching, memory prefetching, etc.
- improve shuffle write performance.
- reuse FSDataInputStream for same input file.
What's Changed
- spark351 + supports bloom filter join by @richox in #532
- Support Spark 3.2.0 by @XorSum in #538
- Bump sonic-rs from 0.3.8 to 0.3.9 by @dependabot in #533
- Bump tonic-build from 0.12.0 to 0.12.1 by @dependabot in #525
- Bump prost from 0.13.0 to 0.13.1 by @dependabot in #521
- supports UDTF.terminate by @richox in #536
- Bump bytes from 1.6.1 to 1.7.0 by @dependabot in #539
- Bump tokio from 1.38.0 to 1.39.2 by @dependabot in #534
- Bump tempfile from 3.10.1 to 3.12.0 by @dependabot in #542
- Bump bytes from 1.7.0 to 1.7.1 by @dependabot in #540
- supports forceShuffledHashJoin. by @richox in #545
- init RSS framework by @richox in #551
- Bump radsort from 0.1.0 to 0.1.1 by @dependabot in #550
- Bump postcard from 1.0.8 to 1.0.10 by @dependabot in #552
- Bump tonic-build from 0.12.1 to 0.12.2 by @dependabot in #554
- Bump sonic-rs from 0.3.9 to 0.3.12 by @dependabot in #555
- improve shuffle performance by @richox in #559
- Bump prost from 0.13.1 to 0.13.2 by @dependabot in #556
- Bump async-trait from 0.1.81 to 0.1.82 by @dependabot in #557
- Bump tokio from 1.39.2 to 1.40.0 by @dependabot in #558
- delete walkaround_sliced_boolean_array_issue by @DDDominik in #560
- fix CI by @richox in #562
- optimize hash joins by @richox in #563
- support year/month/day functions by @richox in #565
- Support Spark 3.1.3 by @harveyyue in #561
- Bump sonic-rs from 0.3.12 to 0.3.13 by @dependabot in #566
- WIP: improve shuffle write performance by @richox in #564
- remove variables from artifacts in pom.xml by @richox in #569
- add --allow-no-vcs to cargo fix command by @richox in #570
- use memory prefetch in hash map building by @richox in #571
- support native scan orc format by @harveyyue in #544
- add spark prefix to tpcds.gen.defaultParallel to take affect by @TJX2014 in #572
- support md5 command by @KnightChess in #581
- Bump bytes from 1.7.1 to 1.7.2 by @dependabot in #578
- update doc to add protobuf installation guides by @richox in #583
- Bump prost from 0.13.2 to 0.13.3 by @dependabot in #586
- make RUSTFLAGS configurable by @XorSum in #585
- [BLAZE-587] Replace spark pattern with spark- for maven profile and shim name by @SteNicholas in #588
- use hadoop fs positioned-reading by @richox in #582
- [BLAZE-573] Support Spark 3.4 version by @SteNicholas in #589
- docs/fix: fix error message by @caicancai in #591
- reuse FSDataInputStream for same input file by @richox in #592
- make RUSTFLAGS effective before running cargo commands by @harveyyue in #594
- [BLAZE-287] Integrate Blaze with Celeborn by @RexXiong in #596
- fix spark3.1 compile error by @richox in #598
- fix timer in AggExec by @richox in #597
- update docs (celeborn related) by @richox in #599
- Change the output qualifier from var to def will avoid invoking NPE by @harveyyue in #608
- OrcExec should read the specified file range data by @harveyyue in #605
- Bump tempfile from 3.12.0 to 3.13.0 by @dependabot in #604
- Bump once_cell from 1.20.0 to 1.20.1 by @dependabot in #603
- Bump tonic-build from 0.12.2 to 0.12.3 by @dependabot in #595
- Bump async-trait from 0.1.82 to 0.1.83 by @dependabot in #590
- Bump futures-util from 0.3.30 to 0.3.31 by @dependabot in #609
- Bump futures from 0.3.30 to 0.3.31 by @dependabot in #610
- Bump once_cell from 1.20.1 to 1.20.2 by @dependabot in #611
New Contributors
- @XorSum made their first contribution in #538
- @DDDominik made their first contribution in #560
- @TJX2014 made their first contribution in #572
- @KnightChess made their first contribution in #581
- @SteNicholas made their first contribution in #588
- @caicancai made their first contribution in #591
- @RexXiong made their first contribution in #596
Full Changelog: v3.0.1...v4.0.0
Contributors
richox, TJX2014, and 8 other contributors
Assets 8
v3.0.1
v3.0.0 [yanked]
blaze-v3.0.0 [yanked]
Features
- Supports using spark.io.compression.codec for shuffle/broadcast compression
- Supports date type casting
- Refactor join implementations to support existence joins and BHJ building hash map on driver side
Performance
- Fixed performance issues when running on spark3 with default configurations
- Use cached parquet metadata
- Refactor native broadcast to avoid duplicated broadcast jobs
- Supports spark333 batch shuffle reading
Bugfix
- Fix in_list conversion in from_proto.rs
v2.0.9.1
release version 2.0.9.1 (#470) Co-authored-by: zhangli20 <[email protected]>
v2.0.9
v2.0.8
v2.0.7
update blaze version 2.0.7-SNAPSHOT (#312) Co-authored-by: zhangli20 <[email protected]>