Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor Morpheus unit tests and plugin to the conda recipe for per-lib testing #1933

Merged
merged 25 commits into from
Oct 12, 2024
Merged

Refactor Morpheus unit tests and plugin to the conda recipe for per-lib testing #1933

merged 25 commits into from
Oct 12, 2024

Conversation

AnuradhaKaruppiah
Copy link
Contributor

@AnuradhaKaruppiah AnuradhaKaruppiah commented Oct 8, 2024 •
edited
Loading

  • Unit tests refactor - core, dfp, llm
  • Conda recipe update to include package testing
  • Github workflow changes to require GPUs for the conda packaging stage
  • Move away from the morpheus-build-container to a slimmer base image (rapidsai/ci-conda)

@AnuradhaKaruppiah AnuradhaKaruppiah requested review from a team as code owners October 8, 2024 21:45
@copy-pr-bot copy-pr-bot
Copy link

copy-pr-bot bot commented Oct 8, 2024

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@AnuradhaKaruppiah AnuradhaKaruppiah added non-breaking Non-breaking change feature request New feature or request conda-build Enables running the conda-build step on a PR labels Oct 8, 2024
@AnuradhaKaruppiah AnuradhaKaruppiah self-assigned this Oct 8, 2024
@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

2 similar comments
@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

@AnuradhaKaruppiah AnuradhaKaruppiah force-pushed the tests-refactor branch 5 times, most recently from 0f90d26 to 6b38de4 Compare October 10, 2024 03:07
@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

@AnuradhaKaruppiah AnuradhaKaruppiah force-pushed the tests-refactor branch 2 times, most recently from 6c0b070 to b2311dc Compare October 10, 2024 19:15
@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

1 similar comment
@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

AnuradhaKaruppiah and others added 25 commits October 11, 2024 17:11
@AnuradhaKaruppiah
Organize tests by functional block
29b49b9 
Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah
Rely entirely on MORPHEUS_ROOT environment variable to setup TEST_DIRS
6fa32e6 
Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah
Make databricks tests dependent on databricks-connect being installed
4f33e7e 
databricks-connect is currently pip installed which makes it difficult
to add as a conda runtime dependency. so the tests for the functionality
have been made optional.

Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah
Replace tmpdir_path fixture with tmp_path fixture
59fe3ae 
tmpdir is a legacy fixture that seems to only be available with
pytest_kafka

Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah
Plugin per-lib tests into the morpheus conda recipe
1657e9e 
Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah
Add lxml to the morpheus-core runtime dependencies
27a20c2 
It is need by beautiful soup to fix the test failure in
test_rss_controller.py -
 bs4.FeatureNotFound: Couldn't find a tree builder with the features you
requested: xml.

Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah
GPU is needed for building the conda libraries
86dccc3 
conda buids runs the relevant unit tests to validate the library that
it just built and these tests require GPU(s).

Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah
Update the ci_pipe to require a GPU for the conda package stage
c2baf68 
GPU is a pre-requisite for testing the conda packages

Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah
Fixup checks
2b22b43 
Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah
Use rapidsai/ci-conda as base image for morpheus conda-package
429ace5 
Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah
Update morpheus conda build dependencies to work with rapidsai/ci-conda
b97c083 
Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah
Use the location of the _utils file for setting the tests_dir
7fb703b 
Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah
Add a conda_morpheus_llm key in dependencies.yaml
8cb1f91 
These are pip dependencies needed to run the llm unit tests at the end
of conda build

Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah
Add a fixture to skip databricks tests if databricks is not installed.
79f1bdd 
databricks units tests will be skipped in the morpheus-conda testing.

Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah
Update the container used for ci_pipe:package stage to rapidsai/ci-conda
dc4361d 
Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah
Change the imports in the examples dir to relative ones
b2cb54b 
This was detectected as an isort problem.

Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah
Updates related to the rapids-24.10 upgrade
a803ffc 
Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah
Change the container credentials for conda packaging
79b2a9b 
Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah
Needed for tests that import modules under tests/morpheus without ins…
bd8038e 
…tall

Example:
Without this change -
[pytest -v tests/morpheus] - works
[pytest -v tests/morpheus/pipeline] - fails

Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah
Include core dependencies in the morpheus-llm testing
bd2ed83 
Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah
Create _modules for files that need to be imported locally
7362b77 
Signed-off-by: Anuradha Karuppiah <[email protected]>
@AnuradhaKaruppiah @dagardner-nv
Update ci/scripts/run_ci_local.sh
3028977 
Co-authored-by: David Gardner <[email protected]>
@AnuradhaKaruppiah @dagardner-nv
Update tests/_utils/test_directories.py
3c1b408 
Co-authored-by: David Gardner <[email protected]>
@AnuradhaKaruppiah @dagardner-nv
Update tests/morpheus/conftest.py
3622020 
Co-authored-by: David Gardner <[email protected]>
@AnuradhaKaruppiah @dagardner-nv
Update tests/morpheus/conftest.py
d115b95 
Co-authored-by: David Gardner <[email protected]>
@AnuradhaKaruppiah
Copy link
Contributor Author

/ok to test

@AnuradhaKaruppiah
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit 3e9e02d into nv-morpheus:branch-24.10 Oct 12, 2024
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dagardner-nv dagardner-nv dagardner-nv approved these changes

Assignees

@AnuradhaKaruppiah AnuradhaKaruppiah

Labels
conda-build Enables running the conda-build step on a PR feature request New feature or request non-breaking Non-breaking change
Projects
Morpheus Boards
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@AnuradhaKaruppiah @dagardner-nv