Skip to content

Commit

Permalink
Set lower CPU usage for test_shared_process_pool.py to avoid slowin…
Browse files Browse the repository at this point in the history
…g down the test (#1935)

`test_shared_process_pool.py` is taking longer than expected when running on remote-ci, causing multiple ci pipelines failed due to (1h) timeout. 
This is probably because:
- The number of workers of `SharedProcessPool` is set to `0.5 * os.sched_getaffinity(0)` by default. If the test is running on a machine that has 64 cores, it will maintain 32 sub-processes.
- For each unit test, the process pool need to create all the sub-processes and then `join()` all of them when tear down, which might be slow when the number of processes is large.
- Running the test in remote-ci is slower than running locally, probably because there are multiple ci jobs running on the same machine

The test should run faster by setting the number of workers to `0.1 * os.sched_getaffinity(0)` and it should avoid ci from being blocked by timeout.

Closes  #1936 

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md).
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - Yuchen Zhang (https://github.com/yczhang-nv)

Approvers:
  - Christopher Harris (https://github.com/cwharris)

URL: #1935
  • Loading branch information
yczhang-nv authored Oct 9, 2024
1 parent de565b3 commit 1452aaf
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions tests/utils/test_shared_process_pool.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@

import logging
import multiprocessing as mp
import os
import threading
from decimal import Decimal
from fractions import Fraction
Expand All @@ -31,6 +32,8 @@

@pytest.fixture(scope="session", autouse=True)
def setup_and_teardown():
# Set lower CPU usage for unit test to avoid slowing down the test
os.environ["MORPHEUS_SHARED_PROCESS_POOL_CPU_USAGE"] = "0.1"

pool = SharedProcessPool()

Expand All @@ -43,6 +46,7 @@ def setup_and_teardown():
# Stop the pool after all tests are done
pool.stop()
pool.join()
os.environ.pop("MORPHEUS_SHARED_PROCESS_POOL_CPU_USAGE", None)


@pytest.fixture(name="shared_process_pool")
Expand Down

0 comments on commit 1452aaf

Please sign in to comment.