diff --git a/ci/vale/styles/config/vocabularies/morpheus/accept.txt b/ci/vale/styles/config/vocabularies/morpheus/accept.txt index 285a85c7d8..f31ace2e17 100644 --- a/ci/vale/styles/config/vocabularies/morpheus/accept.txt +++ b/ci/vale/styles/config/vocabularies/morpheus/accept.txt @@ -71,6 +71,7 @@ pytest [Ss]ubcard(s?) [Ss]ubgraph(s?) [Ss]ubword(s?) +[Ss]uperset(s?) [Tt]imestamp(s?) [Tt]okenization [Tt]okenizer(s?) diff --git a/docs/source/developer_guide/contributing.md b/docs/source/developer_guide/contributing.md index aa52cccda7..2b69eb7a53 100644 --- a/docs/source/developer_guide/contributing.md +++ b/docs/source/developer_guide/contributing.md @@ -151,7 +151,35 @@ This workflow utilizes a Docker container to set up most dependencies ensuring a ### Build in a Conda Environment -If a Conda environment on the host machine is preferred over Docker, it is relatively easy to install the necessary dependencies (In reality, the Docker workflow creates a Conda environment inside the container). +If a [Conda](https://docs.conda.io/projects/conda/en/latest/) environment on the host machine is preferred over Docker, it is relatively easy to install the necessary dependencies (In reality, the Docker workflow creates a Conda environment inside the container). + +#### Conda Environment YAML Files +Morpheus provides multiple Conda environment files to support different workflows. Morpheus utilizes [rapids-dependency-file-generator](https://pypi.org/project/rapids-dependency-file-generator/) to manage these multiple environment files. All of Morpheus' Conda and [pip](https://pip.pypa.io/en/stable/) dependencies along with the different environments are defined in the `dependencies.yaml` file. + +The following are the available Conda environment files, all are located in the `conda/environments` directory, with the following naming convention: `__arch-.yaml`. +| Environment | File | Description | +| --- | --- | --- | +| `all` | `all_cuda-125_arch-x86_64.yaml` | All dependencies required to build, run and test Morpheus, along with all of the examples. This is a superset of the `dev`, `runtime` and `examples` environments. | +| `dev` | `dev_cuda-125_arch-x86_64.yaml` | Dependencies required to build, run and test Morpheus. This is a superset of the `runtime` environment. | +| `examples` | `examples_cuda-125_arch-x86_64.yaml` | Dependencies required to run all examples. This is a superset of the `runtime` environment. | +| `model-utils` | `model-utils_cuda-125_arch-x86_64.yaml` | Dependencies required to train models independent of Morpheus. | +| `runtime` | `runtime_cuda-125_arch-x86_64.yaml` | Minimal set of dependencies strictly required to run Morpheus. | + + +##### Updating Morpheus Dependencies +Changes to Morpheus dependencies can be made in the `dependencies.yaml` file, then run `rapids-dependency-file-generator` to update the individual environment files in the `conda/environments` directory . + +Install `rapids-dependency-file-generator` into the base Conda environment: +```bash +conda run -n base --live-stream pip install rapids-dependency-file-generator +``` + +Then to generate update the individual environment files run: +```bash +conda run -n base --live-stream rapids-dependency-file-generator +``` + +When ready, commit both the changes to the `dependencies.yaml` file and the updated environment files into the repo. #### Prerequisites @@ -170,19 +198,21 @@ If a Conda environment on the host machine is preferred over Docker, it is relat ```bash git submodule update --init --recursive ``` -1. Create the Morpheus Conda environment +1. Create the Morpheus Conda environment using either the `dev` or `all` environment file. Refer to the [Conda Environment YAML Files](#conda-environment-yaml-files) section for more information. ```bash conda env create --solver=libmamba -n morpheus --file conda/environments/dev_cuda-125_arch-x86_64.yaml - conda activate morpheus ``` + or + ```bash + conda env create --solver=libmamba -n morpheus --file conda/environments/all_cuda-125_arch-x86_64.yaml - This creates a new environment named `morpheus`, and activates that environment. + ``` - > **Note**: The `dev_cuda-121_arch-x86_64.yaml` Conda environment file specifies all of the dependencies required to build Morpheus and run Morpheus. However many of the examples, and optional packages such as `morpheus_llm` require additional dependencies. Alternately the following command can be used to create the Conda environment: + This creates a new environment named `morpheus`. Activate the environment with: ```bash - conda env create --solver=libmamba -n morpheus --file conda/environments/all_cuda-121_arch-x86_64.yaml conda activate morpheus ``` + 1. Build Morpheus ```bash ./scripts/compile.sh