Merge branch 'branch-24.10' into devcontainers-vault

ci/vale/styles/config/vocabularies/morpheus/accept.txt

@@ -46,6 +46,7 @@ LLM(s?)
 # https://github.com/logpai/loghub/
 Loghub
 Milvus
+PyPI
 [Mm]ixin
 MLflow
 Morpheus
@@ -71,6 +72,7 @@ pytest
 [Ss]ubcard(s?)
 [Ss]ubgraph(s?)
 [Ss]ubword(s?)
+[Ss]uperset(s?)
 [Tt]imestamp(s?)
 [Tt]okenization
 [Tt]okenizer(s?)

docs/CMakeLists.txt

@@ -30,15 +30,15 @@ add_custom_target(${PROJECT_NAME}_docs
       BUILD_DIR=${CMAKE_CURRENT_BINARY_DIR} ${SPHINX_EXECUTABLE} ${SPHINX_HTML_ARGS} ${SPHINX_SOURCE} ${SPHINX_BUILD}
    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
    COMMENT "Generating documentation with Sphinx"
-   DEPENDS morpheus-package-outputs morpheus_llm-package-outputs
+   DEPENDS morpheus-package-outputs morpheus_llm-package-outputs morpheus_dfp-package-outputs
 )
 
 add_custom_target(${PROJECT_NAME}_docs_linkcheck
    COMMAND
       BUILD_DIR=${CMAKE_CURRENT_BINARY_DIR} ${SPHINX_EXECUTABLE} ${SPHINX_LINKCHECK_ARGS} ${SPHINX_SOURCE} ${SPHINX_LINKCHECK_OUT}
    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
    COMMENT "Checking documentation links with Sphinx"
-   DEPENDS morpheus-package-outputs morpheus_llm-package-outputs
+   DEPENDS morpheus-package-outputs morpheus_llm-package-outputs morpheus_dfp-package-outputs
 )
 
 list(POP_BACK CMAKE_MESSAGE_CONTEXT)

docs/source/conda_packages.md

@@ -0,0 +1,128 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Morpheus Conda Packages
+The Morpheus stages are the building blocks for creating pipelines. The stages are organized into libraries by use case. The current libraries are:
+- `morpheus-core`
+- `morpheus-dfp`
+- `morpheus-llm`
+
+The  libraries are hosted as Conda packages on the [`nvidia`](https://anaconda.org/nvidia/) channel.
+
+The split into multiple libraries allows for a more modular approach to using the Morpheus stages. For example, if you are building an application for Digital Finger Printing, you can install just the `morpheus-dfp` library. This reduces the size of the installed package. It also limits the dependencies eliminating unnecessary version conflicts.
+
+
+## Morpheus Core
+The `morpheus-core` library contains the core stages that are common across all use cases. The Morpheus core library is built from the source code in the `python/morpheus` directory of the Morpheus repository. The core library is installed as a dependency when you install any of the other Morpheus libraries.
+To set up a Conda environment with the [`morpheus-core`](https://anaconda.org/nvidia/morpheus-core) library you can run the following commands:
+### Create a Conda environment
+```bash
+export CONDA_ENV_NAME=morpheus
+conda create -n ${CONDA_ENV_NAME} python=3.10
+conda activate ${CONDA_ENV_NAME}
+```
+### Add Conda channels
+These channel are required for installing the runtime dependencies
+```bash
+conda config --env --add channels conda-forge &&\
+  conda config --env --add channels nvidia &&\
+  conda config --env --add channels rapidsai &&\
+  conda config --env --add channels pytorch
+```
+### Install the `morpheus-core` library
+```bash
+conda install -c nvidia morpheus-core
+```
+The `morpheus-core` Conda package installs the `morpheus` python package. It also pulls down all the necessary Conda runtime dependencies for the core stages including [`mrc`](https://anaconda.org/nvidia/mrc) and [`libmrc`](https://anaconda.org/nvidia/libmrc).
+### Install additional PyPI dependencies
+Some of the stages in the core library require additional dependencies that are hosted on PyPI. These dependencies are included as a requirements file in the `morpheus` python package. The requirements files can be located and installed by running the following command:
+```bash
+MORPHEUS_CORE_PKG_DIR=$(dirname $(python -c "import morpheus; print(morpheus.__file__)"))
+pip install -r ${MORPHEUS_CORE_PKG_DIR}/requirements_morpheus_core.txt
+```
+
+## Morpheus DFP
+Digital Finger Printing (DFP) is a technique used to identify anomalous behavior and uncover potential threats in the environment​. The `morpheus-dfp` library contains stages for DFP. It is built from the source code in the `python/morpheus_dfp` directory of the Morpheus repository. To set up a Conda environment with the [`morpheus-dfp`](https://anaconda.org/nvidia/morpheus-dfp) library you can run the following commands:
+### Create a Conda environment
+```bash
+export CONDA_ENV_NAME=morpheus-dfp
+conda create -n ${CONDA_ENV_NAME} python=3.10
+conda activate ${CONDA_ENV_NAME}
+```
+### Add Conda channels
+These channel are required for installing the runtime dependencies
+```bash
+conda config --env --add channels conda-forge &&\
+  conda config --env --add channels nvidia &&\
+  conda config --env --add channels rapidsai &&\
+  conda config --env --add channels pytorch
+```
+### Install the `morpheus-dfp` library
+```bash
+conda install -c nvidia morpheus-dfp
+```
+The `morpheus-dfp` Conda package installs the `morpheus_dfp` python package. It also pulls down all the necessary Conda runtime dependencies including [`morpheus-core`](https://anaconda.org/nvidia/morpheus-core).
+### Install additional PyPI dependencies
+Some of the DFP stages in the library require additional dependencies that are hosted on PyPI. These dependencies are included as a requirements file in the `morpheus_dfp` python package. And can be installed by running the following command:
+```bash
+MORPHEUS_DFP_PKG_DIR=$(dirname $(python -c "import morpheus_dfp; print(morpheus_dfp.__file__)"))
+pip install -r ${MORPHEUS_DFP_PKG_DIR}/requirements_morpheus_dfp.txt
+```
+
+## Morpheus LLM
+The `morpheus-llm` library contains stages for Large Language Models (LLM) and  Vector Databases. These stages are used for setting up Retrieval Augmented Generation (RAG) pipelines. The `morpheus-llm` library is built from the source code in the `python/morpheus_llm` directory of the Morpheus repository.
+To set up a Conda environment with the [`morpheus-llm`](https://anaconda.org/nvidia/morpheus-dfp) library you can run the following commands:
+### Create a Conda environment
+```bash
+export CONDA_ENV_NAME=morpheus-llm
+conda create -n ${CONDA_ENV_NAME} python=3.10
+conda activate ${CONDA_ENV_NAME}
+```
+### Add Conda channels
+These channel are required for installing the runtime dependencies
+```bash
+conda config --env --add channels conda-forge &&\
+  conda config --env --add channels nvidia &&\
+  conda config --env --add channels rapidsai &&\
+  conda config --env --add channels pytorch
+```
+### Install the `morpheus-llm` library
+```bash
+conda install -c nvidia morpheus-llm
+```
+The `morpheus-llm` Conda package installs the `morpheus_llm` python package. It also pulls down all the necessary Conda packages including [`morpheus-core`](https://anaconda.org/nvidia/morpheus-core).
+### Install additional PyPI dependencies
+Some of the stages in the library require additional dependencies that are hosted on PyPI. These dependencies are included as a requirements file in the `morpheus_llm` python package. And can be installed by running the following command:
+```bash
+MORPHEUS_LLM_PKG_DIR=$(dirname $(python -c "import morpheus_llm; print(morpheus_llm.__file__)"))
+pip install -r ${MORPHEUS_LLM_PKG_DIR}/requirements_morpheus_llm.txt
+```
+
+## Miscellaneous
+### Morpheus Examples
+The Morpheus examples are not included in the Morpheus Conda packages. To use them you need to clone the Morpheus repository and run the examples from source. For details refer to the [Morpheus Examples](./examples.md).
+
+### Namespace Updates
+If you were using a Morpheus release prior to 24.10 you may need to update the namespace for the DFP, LLM and vector database stages.
+
+A script, `scripts/morpheus_namespace_update.py`, has been provide to help with that and can be run as follows:
+```bash
+python scripts/morpheus_namespace_update.py --directory <directory> --dfp
+```
+```bash
+python scripts/morpheus_namespace_update.py --directory <directory> --llm
+```

docs/source/developer_guide/contributing.md

@@ -151,7 +151,35 @@ This workflow utilizes a Docker container to set up most dependencies ensuring a
 
 ### Build in a Conda Environment
 
-If a Conda environment on the host machine is preferred over Docker, it is relatively easy to install the necessary dependencies (In reality, the Docker workflow creates a Conda environment inside the container).
+If a [Conda](https://docs.conda.io/projects/conda/en/latest/) environment on the host machine is preferred over Docker, it is relatively easy to install the necessary dependencies (In reality, the Docker workflow creates a Conda environment inside the container).
+
+#### Conda Environment YAML Files
+Morpheus provides multiple Conda environment files to support different workflows. Morpheus utilizes [rapids-dependency-file-generator](https://pypi.org/project/rapids-dependency-file-generator/) to manage these multiple environment files. All of Morpheus' Conda and [pip](https://pip.pypa.io/en/stable/) dependencies along with the different environments are defined in the `dependencies.yaml` file.
+
+The following are the available Conda environment files, all are located in the `conda/environments` directory, with the following naming convention: `<environment>_<cuda_version>_arch-<architecture>.yaml`.
+| Environment | File | Description |
+| --- | --- | --- |
+| `all` | `all_cuda-125_arch-x86_64.yaml` | All dependencies required to build, run and test Morpheus, along with all of the examples. This is a superset of the `dev`, `runtime` and `examples` environments. |
+| `dev` | `dev_cuda-125_arch-x86_64.yaml` | Dependencies required to build, run and test Morpheus. This is a superset of the `runtime` environment. |
+| `examples` | `examples_cuda-125_arch-x86_64.yaml` | Dependencies required to run all examples. This is a superset of the `runtime` environment. |
+| `model-utils` | `model-utils_cuda-125_arch-x86_64.yaml` | Dependencies required to train models independent of Morpheus. |
+| `runtime` | `runtime_cuda-125_arch-x86_64.yaml` | Minimal set of dependencies strictly required to run Morpheus. |
+
+
+##### Updating Morpheus Dependencies
+Changes to Morpheus dependencies can be made in the `dependencies.yaml` file, then run `rapids-dependency-file-generator` to update the individual environment files in the `conda/environments` directory  .
+
+Install `rapids-dependency-file-generator` into the base Conda environment:
+```bash
+conda run -n base --live-stream pip install rapids-dependency-file-generator
+```
+
+Then to generate update the individual environment files run:
+```bash
+conda run -n base --live-stream rapids-dependency-file-generator
+```
+
+When ready, commit both the changes to the `dependencies.yaml` file and the updated environment files into the repo.
 
 #### Prerequisites
 
@@ -170,19 +198,21 @@ If a Conda environment on the host machine is preferred over Docker, it is relat
    ```bash
    git submodule update --init --recursive
    ```
-1. Create the Morpheus Conda environment
+1. Create the Morpheus Conda environment using either the `dev` or `all` environment file. Refer to the [Conda Environment YAML Files](#conda-environment-yaml-files) section for more information.
    ```bash
    conda env create --solver=libmamba -n morpheus --file conda/environments/dev_cuda-125_arch-x86_64.yaml
-   conda activate morpheus
    ```
+   or
+   ```bash
+   conda env create --solver=libmamba -n morpheus --file conda/environments/all_cuda-125_arch-x86_64.yaml
 
-   This creates a new environment named `morpheus`, and activates that environment.
+   ```
 
-   > **Note**: The `dev_cuda-121_arch-x86_64.yaml` Conda environment file specifies all of the dependencies required to build Morpheus and run Morpheus. However many of the examples, and optional packages such as `morpheus_llm` require additional dependencies. Alternately the following command can be used to create the Conda environment:
+   This creates a new environment named `morpheus`. Activate the environment with:
    ```bash
-   conda env create --solver=libmamba -n morpheus --file conda/environments/all_cuda-121_arch-x86_64.yaml
    conda activate morpheus
    ```
+
 1. Build Morpheus
    ```bash
    ./scripts/compile.sh

docs/source/getting_started.md

@@ -19,6 +19,7 @@ limitations under the License.
 
 There are three ways to get started with Morpheus:
 - [Using pre-built Docker containers](#using-pre-built-docker-containers)
+- [Using the Morpheus Conda packages](#using-morpheus-conda-packages)
 - [Building the Morpheus Docker container](#building-the-morpheus-container)
 - [Building Morpheus from source](./developer_guide/contributing.md#building-from-source)
 
@@ -78,6 +79,12 @@ Once launched, users wishing to launch Triton using the included Morpheus models
 
 Skip ahead to the [Acquiring the Morpheus Models Container](#acquiring-the-morpheus-models-container) section.
 
+## Using Morpheus Conda Packages
+The Morpheus stages are available as libraries that are hosted on the [`nvidia`](https://anaconda.org/nvidia) Conda channel. The Morpheus Conda packages are:
+[`morpheus-core`](https://anaconda.org/nvidia/morpheus-core), [`morpheus-dfp`](https://anaconda.org/nvidia/morpheus-dfp) and [`morpheus-llm`](https://anaconda.org/nvidia/morpheus-llm)
+
+For details on these libraries and how to use them, refer to the [Morpheus Conda Packages](./conda_packages.md) guide.
+
 ## Building the Morpheus Container
 ### Clone the Repository
 

docs/source/index.rst

@@ -54,6 +54,7 @@ Getting Started
 Using Morpheus
 ^^^^^^^^^^^^^^
  * :doc:`getting_started` - Using pre-built Docker containers, building Docker containers from source, and fetching models and datasets
+ * :doc:`Morpheus Conda Packages <conda_packages>`- Using Morpheus Libraries via the pre-built Conda Packages
  * :doc:`basics/overview` - Brief overview of the command line interface
  * :doc:`basics/building_a_pipeline` - Introduction to building a pipeline using the command line interface
  * :doc:`Morpheus Examples <examples>` - Example pipelines using both the Python API and command line interface
@@ -76,6 +77,7 @@ Deploying Morpheus
    :hidden:
 
    getting_started
+   conda_packages
    basics/overview
    basics/building_a_pipeline
    models_and_datasets

docs/source/py_api.rst

@@ -22,4 +22,5 @@ Python API
     :recursive:
 
     morpheus
+    morpheus_dfp
     morpheus_llm

docs/source/stages/morpheus_stages.md

@@ -66,7 +66,7 @@ Stages are the building blocks of Morpheus pipelines. Below is a list of the mos
 
 ## LLM
 
-- LLM Engine Stage {py:class}`~morpheus.stages.llm.llm_engine_stage.LLMEngineStage` Execute an LLM engine within a Morpheus pipeline.
+- LLM Engine Stage {py:class}`~morpheus_llm.stages.llm.llm_engine_stage.LLMEngineStage` Execute an LLM engine within a Morpheus pipeline.
 
 ## Output
 - HTTP Client Sink Stage {py:class}`~morpheus.stages.output.http_client_sink_stage.HttpClientSinkStage` Write all messages to an HTTP endpoint.