Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Gluten-1.2] Port #10534 to Branch-1.2 for Fix hash build memory over use (#10534) #500

Merged
merged 1 commit into from
Sep 20, 2024
Merged

[Gluten-1.2] Port #10534 to Branch-1.2 for Fix hash build memory over use (#10534) #500

merged 1 commit into from
Sep 20, 2024

Commits on Sep 19, 2024

  1. Fix hash build memory over use (facebookincubator#10534) 

    Summary:
For duplicate rows memory usage, currently under parallel join build conditions, each build operator reserves memory big enough to accommodate total number of rows across all hash tables from all build operators. Instead each build operator should only reserve memory enough for its own hash table rows.

This optimization reduced hash build operator memory usage by 10x and we see total memory reduction of some queries reduced by 70%.

Pull Request resolved: facebookincubator#10534

Reviewed By: zacw7

Differential Revision: D60131886

Pulled By: tanjialiang

fbshipit-source-id: a8c1c777df557dfcfc754ef31164a116fdb917c3

(cherry picked from commit 3fb9657)
    @tanjialiang
    tanjialiang authored and liuxiang71 committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    23abe2a View commit details
    Browse the repository at this point in the history