Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document each of the Conda environment files #1932

Merged
Merged
Show file tree
Hide file tree
Changes from 250 commits
Commits
Show all changes
410 commits
Select commit Hold shift + click to select a range
e586501
Switch to using the array_pkg fixture
dagardner-nv Sep 5, 2024
f859ca0
Lint fixes
dagardner-nv Sep 5, 2024
bc6ac3d
Add a few vocabulary entries
dagardner-nv Sep 6, 2024
e898922
WIP
dagardner-nv Sep 6, 2024
16930df
WIP
dagardner-nv Sep 6, 2024
68c5f83
Passthru stage should support GPU & CPU modes
dagardner-nv Sep 6, 2024
4f92ccb
First pass at updating AppShieldSourceStage to emit ControlMessages
dagardner-nv Sep 6, 2024
1bc26a0
update tests
dagardner-nv Sep 6, 2024
0b4b7ba
WIP
dagardner-nv Sep 6, 2024
76c16ce
Set the http port to be the default since we are running in C++ mode
dagardner-nv Sep 9, 2024
a16bc90
Fix spelling error
dagardner-nv Sep 9, 2024
be15217
Remove final decorator as this conflicts with using the preallocator …
dagardner-nv Sep 9, 2024
36b0190
Deprecate AppShieldMessageMeta
dagardner-nv Sep 9, 2024
5bd82f2
Explicitly convert source dataframes to cudf
dagardner-nv Sep 9, 2024
9835e5f
Refactor to use control messages, and limit pandas<->cudf conversions
dagardner-nv Sep 9, 2024
2ad1c99
wip
dagardner-nv Sep 10, 2024
8d2f424
Use PreallocatorMixin in create features since its emitting new dataf…
dagardner-nv Sep 10, 2024
206bc4e
Use C++ mode
dagardner-nv Sep 10, 2024
45291ea
Relocate test and update
dagardner-nv Sep 10, 2024
cf4aa84
Update tests
dagardner-nv Sep 10, 2024
fc9e66b
Config.freeze enforces a compatible value for CppConfig on the first …
dagardner-nv Sep 10, 2024
f6d5513
Ensure the empty config created for boundary stages have a matching e…
dagardner-nv Sep 10, 2024
45dc802
Boundary stages should support CPU mode
dagardner-nv Sep 10, 2024
ca81a8d
Remove setting CppConfig as the Config class does this now
dagardner-nv Sep 10, 2024
89e2add
Add type hints
dagardner-nv Sep 10, 2024
51b4833
First pass at deprecating UserMessageMeta and updating AutoencoderSou…
dagardner-nv Sep 10, 2024
3370623
AppShieldSourceStage should support both cpu and gpu modes
dagardner-nv Sep 10, 2024
8baf123
Cleanup comments
dagardner-nv Sep 10, 2024
8bf6bc4
Update TrainAE to accept incoming ControlMessages
dagardner-nv Sep 10, 2024
58ab86e
Merge branch 'branch-24.10' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Sep 10, 2024
2949820
typehint cleanups
dagardner-nv Sep 10, 2024
642dcd9
Fix type-o in exception string
dagardner-nv Sep 10, 2024
e3b3f86
Minor type hint cleanups
dagardner-nv Sep 10, 2024
ac8ef0d
Replace abstractmethod decorators with NotImplementedError exceptions…
dagardner-nv Sep 10, 2024
630adf9
Add CPU support
dagardner-nv Sep 10, 2024
7c010ca
Merge branch 'branch-24.10' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Sep 11, 2024
0853542
WIP
dagardner-nv Sep 11, 2024
7852184
Fix merge error
dagardner-nv Sep 11, 2024
88c08fe
merge cleanups
dagardner-nv Sep 11, 2024
25df4e1
merge cleanups
dagardner-nv Sep 11, 2024
d6beeaf
merge cleanups
dagardner-nv Sep 11, 2024
f12b564
merge cleanups
dagardner-nv Sep 11, 2024
021189f
WIP
dagardner-nv Sep 11, 2024
100e3ae
Fix broken tests
dagardner-nv Sep 11, 2024
5628e48
Fix tests
dagardner-nv Sep 11, 2024
2d3a0d0
Fix broken test
dagardner-nv Sep 11, 2024
6a6f5eb
AE pipelines no longer need Python mode
dagardner-nv Sep 11, 2024
a001440
Enable AE tests
dagardner-nv Sep 11, 2024
aff5dc8
Fix AE tests
dagardner-nv Sep 11, 2024
bf28221
Remove redundant xfail
dagardner-nv Sep 11, 2024
832b8d0
Remove broken test, because it lives inside a directory named 'dfenco…
dagardner-nv Sep 11, 2024
25c8b88
Enable gnn tests - not currently passing
dagardner-nv Sep 11, 2024
b7f0ab6
Merge branch 'branch-24.10' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Sep 16, 2024
b63bd87
WIP
dagardner-nv Sep 16, 2024
8ed491a
Set pipeline to run in GPU mode
dagardner-nv Sep 16, 2024
6a837af
Fix mode marker
dagardner-nv Sep 16, 2024
364fb62
Remove unused import
dagardner-nv Sep 16, 2024
faab0c5
WIP
dagardner-nv Sep 16, 2024
1a5f027
First pass at replacing DFPMessageMeta with MessageMeta+ControlMessage
dagardner-nv Sep 16, 2024
f18a81e
Set DFP pipelines to run in GPU mode
dagardner-nv Sep 17, 2024
b83f342
fix dfp_training
yczhang-nv Sep 17, 2024
c3b3012
Merge branch 'fix-dfp-example-error' of github.com:yczhang-nv/Morpheu…
dagardner-nv Sep 17, 2024
169639d
Updating DFP tests
dagardner-nv Sep 17, 2024
b0c65e1
Refactor to avoid costly list comprehension
dagardner-nv Sep 17, 2024
7b01c2f
Fix test
dagardner-nv Sep 17, 2024
e8600d5
Fix test
dagardner-nv Sep 17, 2024
511dbdb
Move timezone truncation from the TrainAEStage to AutoencoderSourceStage
dagardner-nv Sep 17, 2024
c7cf687
Fix dfp tests
dagardner-nv Sep 17, 2024
379457e
Remove message_type
dagardner-nv Sep 17, 2024
53cbbef
Lint cleanups
dagardner-nv Sep 17, 2024
56ac198
WIP
dagardner-nv Sep 17, 2024
073cac1
Lint fixes
dagardner-nv Sep 17, 2024
64ab1a7
lint fixes
dagardner-nv Sep 17, 2024
acbd057
Disable the too-many-ancestors check
dagardner-nv Sep 17, 2024
db57b0f
disably cyclic check for control message
dagardner-nv Sep 17, 2024
0730407
Use 'pandas' instead of 'Pandas'
dagardner-nv Sep 18, 2024
547db64
Fix? Can't find 'action.yml' error
dagardner-nv Sep 18, 2024
2d780bb
Fix? Can't find 'action.yml' error
dagardner-nv Sep 18, 2024
e22b01a
Fix? Can't find 'action.yml' error
dagardner-nv Sep 18, 2024
66baad0
Fix? Can't find 'action.yml' error
dagardner-nv Sep 18, 2024
822c2fb
Fix? Can't find 'action.yml' error
dagardner-nv Sep 18, 2024
ba38528
Cleanups
dagardner-nv Sep 18, 2024
90792c6
Add cpu only support for testing the release container
dagardner-nv Sep 18, 2024
e779d76
Fix type-o in variable name, remove warning log
dagardner-nv Sep 18, 2024
c50c717
Warn users regarding the use of the --use_cpp flag
dagardner-nv Sep 18, 2024
c20a956
Change default triton port to the one used by the C++ impl
dagardner-nv Sep 18, 2024
3b8a0d4
Support CPU only mode in WriteToRabbitMQStage, add --use_cpu_only fla…
dagardner-nv Sep 18, 2024
e6d1935
write_simple.py now requires MORPHEUS_ROOT
dagardner-nv Sep 18, 2024
1fb03a4
Update code snippets to match source code changes
dagardner-nv Sep 18, 2024
b6ce583
Update python impls to match those from the 2_2 example
dagardner-nv Sep 18, 2024
d740423
Merge branch 'branch-24.10' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Sep 18, 2024
6fad250
Convert to pandas prior to invoking the model
dagardner-nv Sep 18, 2024
877ae49
Set columns file to be package relative, fix the probs type
dagardner-nv Sep 18, 2024
23c34c9
Update help output, remove usage of --use_cpp=False, use package rela…
dagardner-nv Sep 18, 2024
2095ebd
Change default triton port as we will only be using the C++ impl
dagardner-nv Sep 18, 2024
c87173d
Fix inferring the dataframe type
dagardner-nv Sep 18, 2024
60b0fb2
Fix inferring the dataframe type
dagardner-nv Sep 18, 2024
a22ccc9
Use the default port of the c++ triton impl
dagardner-nv Sep 18, 2024
d3eec63
Use triton http port as this is used by the C++ impl, fix path to voc…
dagardner-nv Sep 18, 2024
e224ab9
Remove unused imports
dagardner-nv Sep 18, 2024
b47ef47
update read_simple and write_simple scripts
dagardner-nv Sep 18, 2024
f70787b
Mark the LLMEngineStage as supporting CPU & GPU execution modes
dagardner-nv Sep 19, 2024
51ab625
Add type hint
dagardner-nv Sep 19, 2024
5bcd44d
Revert "Add type hint"
dagardner-nv Sep 19, 2024
cf17c9f
Revert "Mark the LLMEngineStage as supporting CPU & GPU execution modes"
dagardner-nv Sep 19, 2024
1f43837
Merge branch 'branch-24.10' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Sep 19, 2024
0539c06
Revert "Revert "Mark the LLMEngineStage as supporting CPU & GPU execu…
dagardner-nv Sep 19, 2024
5a063c2
Revert "Revert "Add type hint""
dagardner-nv Sep 19, 2024
11f58ea
WIP: not working
dagardner-nv Sep 19, 2024
3632e80
Add work-around for issue #1891
dagardner-nv Sep 23, 2024
0a9d48c
Expose thread_count as a constructor parameter to TritonInferenceStag…
dagardner-nv Sep 23, 2024
e4f0117
Merge branch 'david-cpu-only-mode-1846' into david-cpu-only-llm
dagardner-nv Sep 23, 2024
f8d2f81
Revert WIP changes
dagardner-nv Sep 23, 2024
203c56d
WIP - untested
dagardner-nv Sep 23, 2024
93597d9
Fix casting of control messages
dagardner-nv Sep 23, 2024
aec27d2
Add CPU only mode support to the simple agents pipeline
dagardner-nv Sep 23, 2024
1d811c8
Remove unused import
dagardner-nv Sep 23, 2024
4646eb2
update tests to set the payload in the metadata
dagardner-nv Sep 23, 2024
6a23f79
Fix conda path for missing llm packages
dagardner-nv Sep 23, 2024
24e41b5
Merge branch 'branch-24.10' into david-cpu-only-llm
dagardner-nv Sep 24, 2024
a64de2c
wip
dagardner-nv Sep 25, 2024
19f432d
Merge branch 'branch-24.10' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Sep 25, 2024
36f5189
Replace manually invoking CudfHelper::load(), with re-inmplementing C…
dagardner-nv Sep 25, 2024
36f441d
remove unused import
dagardner-nv Sep 25, 2024
8922dcf
Mark --use_cpp flag as deprecated, disallow combining --use_cpp with …
dagardner-nv Sep 25, 2024
395bbf9
Merge branch 'david-cpu-only-mode-1846' into david-cpu-only-llm
dagardner-nv Sep 25, 2024
27f0db2
Merge branch 'david-cpu-only-llm' into david-cpu-only-docs
dagardner-nv Sep 25, 2024
20b26db
IWYU fixes
dagardner-nv Sep 26, 2024
cd8a396
Merge branch 'david-cpu-only-mode-1846' into david-cpu-only-llm
dagardner-nv Sep 26, 2024
ab7a7c4
Merge branch 'david-cpu-only-llm' into david-cpu-only-docs
dagardner-nv Sep 26, 2024
c091271
Add explicit dependency for pynvml, this was already being pulled in …
dagardner-nv Sep 26, 2024
e8e542f
Document CPU-only example
dagardner-nv Sep 26, 2024
966fb99
Document the --use_cpu_only flag.
dagardner-nv Sep 26, 2024
9d7c343
Remove out of date documentation regarting the --use_cpp and --num_th…
dagardner-nv Sep 26, 2024
a6b75d7
Remove explicit usage of CppConfig
dagardner-nv Sep 26, 2024
9fa5c3c
Merge branch 'branch-24.10' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Sep 30, 2024
e1c8796
Remove use_cpp arg
dagardner-nv Sep 30, 2024
b7348b1
Default to CPP mode
dagardner-nv Sep 30, 2024
f1879de
Remove setting cppconfig in the grafana pipeline
dagardner-nv Sep 30, 2024
b19469d
Remove setting cppconfig in the viz pipelines
dagardner-nv Sep 30, 2024
38968de
Remove setting CppConfig
dagardner-nv Sep 30, 2024
20e355e
Remove uneeded pip install, use conda with the libmamba solver, remov…
dagardner-nv Sep 30, 2024
ad5f92a
Remove use_cpp
dagardner-nv Sep 30, 2024
d55996a
Merge branch 'branch-24.10' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Sep 30, 2024
84d6ff1
Merge branch 'david-cpu-only-mode-1846' of github.com:dagardner-nv/Mo…
dagardner-nv Sep 30, 2024
5c2542a
Merge branch 'david-cpu-only-mode-1846' of github.com:dagardner-nv/Mo…
dagardner-nv Sep 30, 2024
12c593f
Merge branch 'david-cpu-only-mode-1846-cpp-config' into david-cpu-onl…
dagardner-nv Sep 30, 2024
01fe7fd
Merge branch 'david-cpu-only-mode-1846' into david-cpu-only-llm
dagardner-nv Sep 30, 2024
52e2ad5
Merge branch 'david-cpu-only-llm' into david-cpu-only-docs
dagardner-nv Sep 30, 2024
309da4a
Replace mamba with conda using the libmamba solver, remove --use_cpp …
dagardner-nv Sep 30, 2024
71d37e0
Cleanup help string
dagardner-nv Sep 30, 2024
0a8ad97
Document the --use_cpu_only flag, use conda rather than mamba
dagardner-nv Sep 30, 2024
c342c29
Remove --use_cpp flag
dagardner-nv Sep 30, 2024
bcf7eb8
Remove --use_cpp flag, fix documented number of threads
dagardner-nv Sep 30, 2024
67c6b2b
Remove out of date documentation about additional packages being need…
dagardner-nv Sep 30, 2024
75ae25a
Remove --use_cpp flag
dagardner-nv Sep 30, 2024
16bad6f
Remove usage --use_cpp, also remove usage of the --num_threads flag w…
dagardner-nv Sep 30, 2024
f13c545
Specify filter source in the filter stage (fixes an error)
dagardner-nv Sep 30, 2024
6240b41
Update cli flags
dagardner-nv Sep 30, 2024
f692ff6
Specify stage name
dagardner-nv Sep 30, 2024
8ab412a
Document execution modes
dagardner-nv Sep 30, 2024
d4d1895
Update to document GPU execution, use the http triton server port
dagardner-nv Sep 30, 2024
2ba0c96
wip
dagardner-nv Oct 1, 2024
a60a9b9
WIP
dagardner-nv Oct 1, 2024
2d70d3d
wip
dagardner-nv Oct 1, 2024
4fddd48
Fix spelling errors
dagardner-nv Oct 1, 2024
b05bb1d
Assume pandas in the Pythin impl
dagardner-nv Oct 1, 2024
47251fb
WIP
dagardner-nv Oct 1, 2024
13ae779
WIP
dagardner-nv Oct 1, 2024
77a87d2
Remove CppConfig
dagardner-nv Oct 1, 2024
7078dc7
Fix imports of ControlMessage and MessageMeta
dagardner-nv Oct 1, 2024
eaa5198
Remove usage of --use_cpp flag
dagardner-nv Oct 1, 2024
687827f
Remove references to using mamba, just document conda with libmamba, …
dagardner-nv Oct 1, 2024
023e365
wip
dagardner-nv Oct 1, 2024
e0512f8
Add additional troubleshooting sections covering removing in-place bu…
dagardner-nv Oct 1, 2024
1f85cb2
Refer to the troubleshooting guide
dagardner-nv Oct 1, 2024
bb6de94
Merge branch 'branch-24.10' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Oct 1, 2024
1a25a8e
Merge branch 'branch-24.10' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Oct 4, 2024
c13826b
Fix merge error
dagardner-nv Oct 4, 2024
fae32d2
Merge branch 'david-cpu-only-mode-1846' into david-cpu-only-llm
dagardner-nv Oct 4, 2024
381c569
Merge branch 'david-cpu-only-llm' into david-cpu-only-docs
dagardner-nv Oct 4, 2024
f71e3b2
Document each of the conda environments
dagardner-nv Oct 4, 2024
fdd1159
superset not super-set
dagardner-nv Oct 4, 2024
79fbc88
Add 'superset' to our dictionary
dagardner-nv Oct 4, 2024
e6a46a6
Consolidate common code in the docker run scripts
dagardner-nv Oct 7, 2024
5d26520
Cleanup handling of DOCKER_ARGS, remove unused WORKSPACE_VOLUME env var
dagardner-nv Oct 7, 2024
78d5254
Merge branch 'david-cpu-only-mode-1846' into david-cpu-only-llm
dagardner-nv Oct 7, 2024
c08b7d3
Merge branch 'david-cpu-only-llm' into david-cpu-only-docs
dagardner-nv Oct 7, 2024
b4698bd
Merge branch 'david-cpu-only-docs' into david-contrib-conda-yaml-1660
dagardner-nv Oct 7, 2024
ab49477
Merge branch 'branch-24.10' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Oct 9, 2024
43c2534
Switch to using a forkserver, docs state that using fork is problemat…
dagardner-nv Oct 10, 2024
dd54ce1
Support CPU & GPU execution mode in MultiProcessingStage
dagardner-nv Oct 10, 2024
55a5b97
Test CPU and GPU execution modes
dagardner-nv Oct 10, 2024
3508628
Merge branch 'david-cpu-only-mode-1846' into david-cpu-only-llm
dagardner-nv Oct 10, 2024
492bd8a
Merge branch 'david-cpu-only-llm' into david-cpu-only-docs
dagardner-nv Oct 10, 2024
71b2616
Merge branch 'david-cpu-only-docs' into david-contrib-conda-yaml-1660
dagardner-nv Oct 10, 2024
5e50dcd
IWYU fixes
dagardner-nv Oct 10, 2024
dcbbd9e
Merge branch 'david-cpu-only-mode-1846' into david-cpu-only-llm
dagardner-nv Oct 10, 2024
807d177
Merge branch 'david-cpu-only-llm' into david-cpu-only-docs
dagardner-nv Oct 10, 2024
6adc180
Merge branch 'david-cpu-only-docs' into david-contrib-conda-yaml-1660
dagardner-nv Oct 10, 2024
60e6d95
Merge branch 'branch-24.10' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Oct 10, 2024
dd46f71
Remove unused include
dagardner-nv Oct 10, 2024
0c690ce
Merge branch 'david-cpu-only-mode-1846' into david-cpu-only-llm
dagardner-nv Oct 10, 2024
e57d14e
Merge branch 'david-cpu-only-llm' into david-cpu-only-docs
dagardner-nv Oct 10, 2024
1e0c1e5
Merge branch 'david-cpu-only-docs' into david-contrib-conda-yaml-1660
dagardner-nv Oct 10, 2024
986dbc4
Specify a cuda build of pytorch, install pytorch via conda rather tha…
dagardner-nv Oct 14, 2024
bec754a
Merge branch 'branch-24.10' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Oct 14, 2024
27cfba0
Fix the config fixture to always run in gpu mode
dagardner-nv Oct 14, 2024
6540752
Merge branch 'david-cpu-only-mode-1846' into david-cpu-only-llm
dagardner-nv Oct 14, 2024
e79203f
Merge branch 'david-cpu-only-llm' into david-cpu-only-docs
dagardner-nv Oct 14, 2024
0a7901a
Merge branch 'david-cpu-only-docs' into david-contrib-conda-yaml-1660
dagardner-nv Oct 14, 2024
14e6cdc
Revert "Specify a cuda build of pytorch, install pytorch via conda ra…
dagardner-nv Oct 14, 2024
dca1933
Merge branch 'branch-24.10' into david-cpu-only-mode-1846
dagardner-nv Oct 14, 2024
66d263c
Merge branch 'david-cpu-only-mode-1846' into david-cpu-only-llm
dagardner-nv Oct 14, 2024
829d6c2
Merge branch 'david-cpu-only-llm' into david-cpu-only-docs
dagardner-nv Oct 14, 2024
ccd288e
Merge branch 'david-cpu-only-docs' into david-contrib-conda-yaml-1660
dagardner-nv Oct 14, 2024
24ee561
Merge branch 'branch-24.10' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Oct 17, 2024
7500d37
Fix merge error
dagardner-nv Oct 17, 2024
ac2eae6
Move SupportedTypes to a class property of MonitorController
dagardner-nv Oct 17, 2024
d243108
Rename DataFrameTypeStr to DataFrameModule
dagardner-nv Oct 17, 2024
f7d4bf3
Fix overloads for get_json_reader
dagardner-nv Oct 17, 2024
1b89232
Add property methods to the mixin to wrap common type_utils uses
dagardner-nv Oct 17, 2024
9362562
Formatting fixes
dagardner-nv Oct 17, 2024
fb90067
make the get_df_pkg and get_df_class wrappers methods not properties …
dagardner-nv Oct 17, 2024
075b0fa
Update stages to use GpuAndCpuMixin helpers
dagardner-nv Oct 17, 2024
319595e
Add helper methods for getting a read_csv and read_parquet method
dagardner-nv Oct 17, 2024
ce8ea07
Use get_csv_reader
dagardner-nv Oct 17, 2024
7be0402
Fix imports
dagardner-nv Oct 17, 2024
fa67db6
Rase a ValueError instead of a KeyError to match behavior of the C++ …
dagardner-nv Oct 17, 2024
485764c
Remove inconsistent tensor reshaping issue #1955
dagardner-nv Oct 17, 2024
d903feb
Update ControlMemory tests to test the Python and C++ impls
dagardner-nv Oct 17, 2024
ef466ae
Test should be run in both cpu and gpu modes
dagardner-nv Oct 17, 2024
357a6e2
Remove unused config_only_cpp and config_no_cpp fixtures
dagardner-nv Oct 17, 2024
1998922
lint fixes
dagardner-nv Oct 17, 2024
a0f5e5b
Fix cpu-mode json tests
dagardner-nv Oct 17, 2024
80e817a
Fix misleading runtime erro messafe, remove unused inferring gpu mode…
dagardner-nv Oct 17, 2024
32fb3b8
Ignore code.visualstudio.com, these urls are being forbidden when run…
dagardner-nv Oct 17, 2024
13a0eb6
Merge branch 'david-cpu-only-mode-1846' into david-cpu-only-llm
dagardner-nv Oct 18, 2024
acff1f0
Merge branch 'david-cpu-only-llm' into david-cpu-only-docs
dagardner-nv Oct 18, 2024
341ffe0
Improved doc strings
dagardner-nv Oct 18, 2024
fe58767
Merge branch 'branch-24.10' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Oct 18, 2024
cabfc51
Merge branch 'david-cpu-only-llm' into david-cpu-only-docs
dagardner-nv Oct 18, 2024
368ea5d
Misc docstring improvements
dagardner-nv Oct 18, 2024
77c5afc
Merge branch 'branch-24.10' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Oct 18, 2024
b9d5e07
Merge branch 'david-cpu-only-docs' into david-contrib-conda-yaml-1660
dagardner-nv Oct 19, 2024
63b9fcf
Merge branch 'branch-24.10' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Oct 19, 2024
e907c67
Update docs/source/developer_guide/contributing.md
dagardner-nv Oct 21, 2024
fdc56a2
Merge branch 'branch-24.10' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Oct 21, 2024
94816a9
Expand instruction for running rapids-dependency-file-generator
dagardner-nv Oct 21, 2024
003f2f9
Merge branch 'david-contrib-conda-yaml-1660' of github.com:dagardner-…
dagardner-nv Oct 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ci/vale/styles/config/vocabularies/morpheus/accept.txt
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ pytest
[Ss]ubcard(s?)
[Ss]ubgraph(s?)
[Ss]ubword(s?)
[Ss]uperset(s?)
[Tt]imestamp(s?)
[Tt]okenization
[Tt]okenizer(s?)
Expand Down
42 changes: 36 additions & 6 deletions docs/source/developer_guide/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,35 @@ This workflow utilizes a Docker container to set up most dependencies ensuring a

### Build in a Conda Environment

If a Conda environment on the host machine is preferred over Docker, it is relatively easy to install the necessary dependencies (In reality, the Docker workflow creates a Conda environment inside the container).
If a [Conda](https://docs.conda.io/projects/conda/en/latest/) environment on the host machine is preferred over Docker, it is relatively easy to install the necessary dependencies (In reality, the Docker workflow creates a Conda environment inside the container).

#### Conda Environment YAML Files
Morpheus provides multiple Conda environment files to support different workflows. Morpheus utilizes [rapids-dependency-file-generator](https://pypi.org/project/rapids-dependency-file-generator/) to manage these multiple environment files. All of Morpheus' Conda and [pip](https://pip.pypa.io/en/stable/) dependencies along with the different environments are defined in the `dependencies.yaml` file.

The following are the available Conda environment files, all are located in the `conda/environments` directory, with the following naming convention: `<environment>_<cuda_version>_arch-<architecture>.yaml`.
| Environment | File | Description |
| --- | --- | --- |
| `all` | `all_cuda-125_arch-x86_64.yaml` | All dependencies required to build, run and test Morpheus, along with all of the examples. This is a superset of the `dev`, `runtime` and `examples` environments. |
| `dev` | `dev_cuda-125_arch-x86_64.yaml` | Dependencies required to build, run and test Morpheus. This is a superset of the `runtime` environment. |
| `examples` | `examples_cuda-125_arch-x86_64.yaml` | Dependencies required to run all examples. This is a superset of the `runtime` environment. |
| `model-utils` | `model-utils_cuda-125_arch-x86_64.yaml` | Dependencies required to train models independent of Morpheus. |
| `runtime` | `runtime_cuda-125_arch-x86_64.yaml` | Minimal set of dependencies strictly required to run Morpheus. |


##### Updating Morpheus Dependencies
Changes to Morpheus dependencies can be made in the `dependencies.yaml` file, then run `rapids-dependency-file-generator` to update the individual environment files in the `conda/environments` directory .

Install `rapids-dependency-file-generator` into the base Conda environment:
```bash
conda run -n base --live-stream pip install rapids-dependency-file-generator
```

To run:
```bash
conda run -n base --live-stream rapids-dependency-file-generator
```
dagardner-nv marked this conversation as resolved.
Show resolved Hide resolved

When ready commit the changes to the `dependencies.yaml` file and the updated environment files.
dagardner-nv marked this conversation as resolved.
Show resolved Hide resolved

#### Prerequisites

Expand All @@ -170,19 +198,21 @@ If a Conda environment on the host machine is preferred over Docker, it is relat
```bash
git submodule update --init --recursive
```
1. Create the Morpheus Conda environment
1. Create the Morpheus Conda environment using either the `dev` or `all` environment file. Refer to the [Conda Environment YAML Files](#conda-environment-yaml-files) section for more information.
```bash
conda env create --solver=libmamba -n morpheus --file conda/environments/dev_cuda-125_arch-x86_64.yaml
conda activate morpheus
```
or
```bash
conda env create --solver=libmamba -n morpheus --file conda/environments/all_cuda-125_arch-x86_64.yaml

This creates a new environment named `morpheus`, and activates that environment.
```

> **Note**: The `dev_cuda-121_arch-x86_64.yaml` Conda environment file specifies all of the dependencies required to build Morpheus and run Morpheus. However many of the examples, and optional packages such as `morpheus_llm` require additional dependencies. Alternately the following command can be used to create the Conda environment:
This creates a new environment named `morpheus`. Activate the environment with:
```bash
conda env create --solver=libmamba -n morpheus --file conda/environments/all_cuda-121_arch-x86_64.yaml
conda activate morpheus
```

1. Build Morpheus
```bash
./scripts/compile.sh
Expand Down
Loading