Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Supports pipeline event based scheduling. #54259

Open
3 tasks done
stdpain opened this issue Dec 24, 2024 · 0 comments · Fixed by #54260, #54338 or #54691
Open
3 tasks done

Supports pipeline event based scheduling. #54259

stdpain opened this issue Dec 24, 2024 · 0 comments · Fixed by #54260, #54338 or #54691

Comments

@stdpain
Copy link
Contributor

stdpain commented Dec 24, 2024

Feature request

Currently our pipeline uses poller mode for scheduling. This leads to some of the following problems.

  1. An unintended blocking implementation may result in all queries not being scheduled in a timely manner.
  2. poller may not be scheduled in time.
  3. The pipeline poller consumes an entire CPU core, resulting in a waste of resources.

Describe the solution you'd like

The main purpose of this PR (#49972) is to remove pollers through the observer notify mechanism, and we will continue this work and make better improvements.

It can save one CPU core in the scenarios such as point query/TPCDS.

image

In the new implementation, we don't need a poller thread at all, or even a block driver queue. And there is no need to introduce additional locks during scheduling.

@stdpain stdpain linked a pull request Dec 24, 2024 that will close this issue
24 tasks
@stdpain stdpain linked a pull request Dec 25, 2024 that will close this issue
24 tasks
@stdpain stdpain linked a pull request Jan 3, 2025 that will close this issue
24 tasks
@wanpengfei-git wanpengfei-git reopened this Jan 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment