PERF-#273: Scheduler with task tracking #274

arunjose696 · 2023-04-04T09:44:33Z

What do these changes do?

first commit message and PR title follow format outlined here

NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.
passes flake8 .
passes black .
signed commit with git commit -s
Resolves Track the number of tasks executing on each worker, and submit tasks to workers with minimum number of tasks #273
tests added and passing
module layout described at docs/developer/architecture.rst is up-to-date

Signed-off-by: arunjose696 <[email protected]>

unidist/core/backends/mpi/core/communication.py

unidist/core/backends/mpi/core/controller/api.py

unidist/core/backends/mpi/core/monitor.py

arunjose696 added 4 commits April 3, 2023 08:25

Tracking number of task on wach worker to schedule tasks

6c6ff7c

making scheduler class

4d137ce

removing rank from rank_to_schedule only if rank already present

2019d78

static check changes

aeb22bd

Signed-off-by: arunjose696 <[email protected]>

YarShev reviewed Apr 4, 2023

View reviewed changes

arunjose696 added 3 commits April 5, 2023 05:14

pr comments

301dd08

mend

939f2c7

changing task tracking to GC not using background thread

e360efc

arunjose696 force-pushed the scheduler_with_task_tracking branch from 79825d8 to e360efc Compare April 13, 2023 10:19

arunjose696 added 2 commits April 13, 2023 11:32

having only one mpisend from monitor

9a3ec5a

tracking tasks outside the condition

79d5bea

arunjose696 force-pushed the scheduler_with_task_tracking branch from 6c6c0f2 to 79d5bea Compare April 19, 2023 14:36

YarShev force-pushed the master branch from 28e6411 to 1e2922a Compare May 29, 2024 14:07