PERF-#367: Use std::fill to fill service shared buffer with a given value

[arunjose696]

What do these changes do?

In _allocate_shared_memory(self): function we are setting -1 to the service buffer because 0 is a valid value. This operation is a time consuming operation. Changing this to use numpy to speed up the step.

• first commit message and PR title follow format outlined here

  NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.

• passes flake8 .
• passes black .
• signed commit with git commit -s
• Resolves First put into the shared memory is slower than subsequent one #367
• tests added and passing
• module layout described at docs/developer/architecture.rst is up-to-date

[arunjose696]

Here are some results running the reproducer with and without changes.

import unidist
import numpy as np
import time

unidist.init()

t0 = time.time()
data = unidist.put(np.arange(10))
t1 = time.time()
print(f"time is {t1 - t0}")

t0 = time.time()
data2 = unidist.put(np.arange(10))
t1 = time.time()
print(f"time is {t1 - t0}")

####Current Master############
time is 4.3809590339660645
time is 0.0005385875701904297

###This pull request ##########
time is 0.38097357749938965
time is 0.0003437995910644531

removing this line to check.
time is 0.002953767776489258
time is 0.00027179718017578125

CPU tested Model name: Intel(R) Xeon(R) Platinum 8276L CPU @ 2.20GHz
Ran with
MODIN_ENGINE=unidist UNIDIST_MPI_SHARED_OBJECT_STORE=True UNIDIST_MPI_BACKOFF=0.0 UNIDIST_IS_MPI_SPAWN_WORKERS=False mpiexec -n 3 python times.py

[YarShev]

Suggested change

monitor_logger.debug("Monitor loop started")

[arunjose696]

Added this looging line here so the init() function in root ends only after the monitor.log file is created and first log is written, Or this file creation step though not very time consuming happens after init() of the root process and could make the first put() slightly slower as the monitor will be ready to recv operations only after creating the logger file.

[YarShev]

Discussed offline and decided to apply suggestion.

Aside: Is the log file created on the first write or when we call get_logger?

[YarShev]

ping

[arunjose696]

Verified again the log file is created when the "first write" is made.

[YarShev]

Thanks!

[YarShev]

Please also rename the PR title and exclude the file unidist/core/backends/mpi/core/memory/_memory.cpp (it is auto-generated by cython) out of this PR.

[YarShev]

Some perf measurements would also be helpful.