[VL] Fix sort based shuffle oom in spill when compress was disabled

[clay4megtr]

What changes were proposed in this pull request?

Fix oom when compress was disabled in spill.

[zhztheplayer]

Thanks for working on a fix. Though this could turn the memory buffer into untracked?

cc @marin-ma

[clay4megtr]

It is, actually I'm a little confused about delegate the timing of spill trigger to spark framework, which means we can't make any other memory allocation when spilling.

[clay4megtr]

Maybe we should have a threshold to indicate whether trigger spill actively or not?

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

• Other pull requests

[marin-ma]

Thanks. Can we pre-allocate a buffer of size options_.pushBufferMaxSize, and make rssBufferOs reuse the memory each time?

[clay4megtr]

Thanks. Can we pre-allocate a buffer of size options_.pushBufferMaxSize, and make rssBufferOs reuse the memory each time?

That's a better solution, I'll update later. btw, I have another question, is it possible that sortTime which defined here include cost time of other spill type? because the trigger root is nativeShuffleWriter.write(...) ?
or spill was triggered in other code path, so the shuffleWallTime will not include the cost spent on shuffle push for that instance, which finally lead to sortTime is inaccurate, I'm not sure about this.

[zhouyuan]

related
#6896

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions]

This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks.

[marin-ma]

Thanks. Can we pre-allocate a buffer of size options_.pushBufferMaxSize, and make rssBufferOs reuse the memory each time?

@clay4megtr Let's merge this one first. Perhaps consider this change in the future. Thanks!