[RELEASE] Morpheus v24.10.01 (#2071)

<!--
SPDX-FileCopyrightText: Copyright (c) 2023-2024, NVIDIA CORPORATION &
AFFILIATES. All rights reserved.
SPDX-License-Identifier: Apache-2.0

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

## ❄️ Code freeze for `branch-24.10` and `v24.10` release

### What does this mean?
Only critical/hotfix level issues should be merged into `branch-24.10`
until release (merging of this PR).

All other development PRs should be retargeted towards the next release
branch: `branch-25.02`.

### What is the purpose of this PR?
- Update documentation
- Allow testing for the new release
- Enable a means to merge `branch-24.10` into `main` for the release

.github/workflows/pr.yaml

@@ -20,6 +20,7 @@ on:
     branches:
       - 'pull-request/**'
       - 'branch-*'
+      - 'main'
 
 # This allows a subsequently queued workflow run to interrupt previous runs
 concurrency:

CHANGELOG.md

@@ -15,6 +15,19 @@ See the License for the specific language governing permissions and
 limitations under the License.
 -->
 
+# Morpheus 24.10.01 (22 Nov 2024)
+
+## 🐛 Bug Fixes
+
+- Pin mlflow version to avoid breaking changes in v2.18 ([#2067](https://github.com/nv-morpheus/Morpheus/pull/2067)) [@dagardner-nv](https://github.com/dagardner-nv)
+- Execute CI on the main branch ([#2064](https://github.com/nv-morpheus/Morpheus/pull/2064)) [@dagardner-nv](https://github.com/dagardner-nv)
+
+## 📖 Documentation
+
+- Remove references to pipeline-ae in docs ([#2063](https://github.com/nv-morpheus/Morpheus/pull/2063)) [@dagardner-nv](https://github.com/dagardner-nv)
+- Document location of third party source repository ([#2059](https://github.com/nv-morpheus/Morpheus/pull/2059)) [@dagardner-nv](https://github.com/dagardner-nv)
+- Update DFP class and file paths ([#2052](https://github.com/nv-morpheus/Morpheus/pull/2052)) [@dagardner-nv](https://github.com/dagardner-nv)
+
 # Morpheus 24.10.00 (01 Nov 2024)
 
 ## 🚨 Breaking Changes

ci/conda/recipes/morpheus-libs/meta.yaml

@@ -91,7 +91,7 @@ outputs:
         - feedparser =6.0.*
         - grpcio =1.62.*
         - lxml
-        - mlflow>=2.10.0,<3
+        - mlflow>=2.10.0,<2.18
         - mrc
         - networkx=2.8.8
         - numpydoc =1.5.*
@@ -239,7 +239,7 @@ outputs:
         - scripts/fetch_data.py
         - tests/*
       script: morpheus_llm_test.sh
- 
+
 about:
   home: https://github.com/nv-morpheus/Morpheus
   license: Apache-2.0

ci/conda/recipes/morpheus/meta.yaml

@@ -97,7 +97,7 @@ outputs:
         - feedparser =6.0.*
         - grpcio =1.62.*
         - libwebp>=1.3.2 # Required for CVE mitigation: https://nvd.nist.gov/vuln/detail/CVE-2023-4863
-        - mlflow>=2.10.0,<3
+        - mlflow>=2.10.0,<2.18
         - mrc
         - networkx=2.8.8
         - numpydoc =1.5.*

ci/release/update-version.sh

@@ -113,6 +113,9 @@ sed_runner "s/${CURRENT_SHORT_TAG}/${NEXT_SHORT_TAG}/g" docs/source/getting_star
 sed_runner "s|blob/branch-${CURRENT_SHORT_TAG}|blob/branch-${NEXT_SHORT_TAG}|g" models/model-cards/*.md
 sed_runner "s|tree/branch-${CURRENT_SHORT_TAG}|tree/branch-${NEXT_SHORT_TAG}|g" models/model-cards/*.md
 
+# thirdparty
+sed_runner "s|tree/branch-${CURRENT_SHORT_TAG}|tree/branch-${NEXT_SHORT_TAG}|g" thirdparty/README.md
+
 # Update the version of the Morpheus model container
 # We need to update several files, however we need to avoid symlinks as well as the build and .cache directories
 DOCS_MD_FILES=$(find -P ./docs/source/ -type f -iname "*.md")

conda/environments/all_cuda-125_arch-x86_64.yaml

@@ -66,7 +66,7 @@ dependencies:
 - libtool
 - libwebp=1.3.2
 - libzlib >=1.3.1,<2
-- mlflow
+- mlflow>=2.10.0,<2.18
 - mrc=24.10
 - myst-parser=0.18.1
 - nbsphinx

conda/environments/dev_cuda-125_arch-x86_64.yaml

@@ -56,7 +56,7 @@ dependencies:
 - libtool
 - libwebp=1.3.2
 - libzlib >=1.3.1,<2
-- mlflow
+- mlflow>=2.10.0,<2.18
 - mrc=24.10
 - myst-parser=0.18.1
 - nbsphinx

conda/environments/examples_cuda-125_arch-x86_64.yaml

@@ -29,7 +29,7 @@ dependencies:
 - jsonpatch>=1.33
 - kfp
 - libwebp=1.3.2
-- mlflow
+- mlflow>=2.10.0,<2.18
 - mrc=24.10
 - networkx=2.8.8
 - newspaper3k=0.2

conda/environments/runtime_cuda-125_arch-x86_64.yaml

@@ -26,7 +26,7 @@ dependencies:
 - grpcio
 - grpcio-status
 - libwebp=1.3.2
-- mlflow
+- mlflow>=2.10.0,<2.18
 - mrc=24.10
 - networkx=2.8.8
 - numpydoc=1.5

dependencies.yaml

@@ -376,7 +376,7 @@ dependencies:
           - grpcio
           - grpcio-status
           # - libwebp=1.3.2 # Required for CVE mitigation: https://nvd.nist.gov/vuln/detail/CVE-2023-4863 ##
-          - mlflow #>=2.10.0,<3
+          - mlflow>=2.10.0,<2.18 # Pin version to avoid breaking change in 2.18 to thread local variable code commit id: 5541888
           - mrc=24.10
           - networkx=2.8.8
           - numpydoc=1.5

docker/Dockerfile

@@ -363,6 +363,7 @@ COPY --from=git_clone "/tmp/morpheus_repo/examples" "./examples"
 COPY --from=git_clone "/tmp/morpheus_repo/scripts" "./scripts"
 COPY --from=git_clone "/tmp/morpheus_repo/*.md" "./"
 COPY --from=git_clone "/tmp/morpheus_repo/LICENSE" "./"
+COPY --from=git_clone "/tmp/morpheus_repo/thirdparty" "./thirdparty"
 
 RUN /opt/conda/bin/conda clean -afy && \
     # Ensure the conda-bld directory is indexed even if empty

docs/source/developer_guide/guides/10_modular_pipeline_digital_fingerprinting.md

@@ -71,7 +71,7 @@ The front-end loader outputs one or more control messages that are passed to the
 
 Moreover, the updated pipeline supports human-in-the-loop workflows, such as the ability to manually trigger training or inference tasks against a specific set of data, and the capacity for real-time labeling of production inference events that can be injected back into the training pipeline.
 
-The following content will track the pipeline declared in `examples/digital_fingerprinting/production/morpheus/dfp_integrated_training_streaming_pipeline.py`
+The following content will track the pipeline declared in `examples/digital_fingerprinting/production/dfp_integrated_training_streaming_pipeline.py`
 
 ```python
 # Setup and command line argument parsing
@@ -115,7 +115,7 @@ For a full introduction to Morpheus modules, refer to the [Python Modules](7_pyt
 ## DFP Deployment
 
 
-Source: `examples/digital_fingerprinting/production/morpheus/dfp/modules/dfp_deployment.py`
+Source: `python/morpheus_dfp/morpheus_dfp/modules/dfp_deployment.py`
 
 This is the top level module that encapsulates the entire Digital Fingerprinting pipeline, it is primarily responsible for wrapping the training and inference pipelines, providing the correct module interface, and doing some configuration pre-processing. Since this module is monolithic, it supports a significant number of configuration options; however, the majority of these have intelligent defaults and are not required to be specified.
 
@@ -162,7 +162,7 @@ There are a number of modules that are used in both the training and inference p
 
 ### DFP Preprocessing
 
-Source: `examples/digital_fingerprinting/production/morpheus/dfp/modules/dfp_preproc.py`
+Source: `python/morpheus_dfp/morpheus_dfp/modules/dfp_preproc.py`
 
 The `dfp_preproc` module is a functional component within the Morpheus framework that combines multiple data filtering and processing pipeline modules related to inference and training. This module simplifies the pipeline by consolidating various modules into a single, cohesive unit. The `dfp_preproc` module supports configuration parameters such as the cache directory, timestamp column name, pre-filter options, batching options, user splitting options, and supported data loaders for various file types.
 
@@ -233,7 +233,7 @@ For a complete reference, refer to: [DataLoader Module](../../modules/core/data_
 
 ### DFP Split Users
 
-Source: `examples/digital_fingerprinting/production/morpheus/dfp/modules/dfp_split_users.py`
+Source: `python/morpheus_dfp/morpheus_dfp/modules/dfp_split_users.py`
 
 The `dfp_split_users` module is responsible for splitting the input data based on user IDs. The module provides configuration options, such as fallback username, include generic user, include individual users, and specify lists of user IDs to include or exclude in the output.
 
@@ -250,7 +250,7 @@ def dfp_split_users(builder: mrc.Builder):
 
 ### DFP Rolling Window
 
-Source: `examples/digital_fingerprinting/production/morpheus/dfp/modules/dfp_rolling_window.py`
+Source: `python/morpheus_dfp/morpheus_dfp/modules/dfp_rolling_window.py`
 
 The `dfp_rolling_window` module is responsible for maintaining a rolling window of historical data, acting as a streaming caching and batching system. The module provides various configuration options, such as aggregation span, cache directory, caching options, timestamp column name, and trigger conditions.
 
@@ -271,7 +271,7 @@ def dfp_rolling_window(builder: mrc.Builder):
 
 ### DFP Data Prep
 
-Source: `examples/digital_fingerprinting/production/morpheus/dfp/modules/dfp_data_prep.py`
+Source: `python/morpheus_dfp/morpheus_dfp/modules/dfp_data_prep.py`
 
 The `dfp_data_prep` module is responsible for preparing data for either inference or model training. The module requires a defined schema for data preparation.
 
@@ -288,7 +288,7 @@ def dfp_data_prep(builder: mrc.Builder):
 
 ## DFP Training Pipeline
 
-Source: `examples/digital_fingerprinting/production/morpheus/dfp/modules/dfp_training_pipe.py`
+Source: `python/morpheus_dfp/morpheus_dfp/modules/dfp_training_pipe.py`
 
 The DFP Training Pipe module is a consolidated module that integrates several DFP pipeline modules that are essential to the training process. This module function provides a single entry point to the training pipeline, simplifying the process of training a model. The module offers configurable parameters for various stages in the pipeline, including data batching, data preprocessing, and data encoding for model training. Additionally, the MLflow model writer options allow for the trained model to be saved for future use.
 
@@ -326,7 +326,7 @@ def dfp_training_pipe(builder: mrc.Builder):
 
 ### DFP Training
 
-Source: `examples/digital_fingerprinting/production/morpheus/dfp/modules/dfp_training.py`
+Source: `python/morpheus_dfp/morpheus_dfp/modules/dfp_training.py`
 
 The `dfp_training` module function is responsible for training the model. The `on_data` function is defined to handle incoming `ControlMessage` instances. It retrieves the user ID and the input data from the `ControlMessage`, creates an instance of the `AutoEncoder` class with the specified `model_kwargs`, and trains the model on the input data. The output message includes the trained model and metadata.
 
@@ -358,7 +358,7 @@ def mlflow_model_writer(builder: mrc.Builder):
 
 ## DFP Inference Pipeline
 
-Source: `examples/digital_fingerprinting/production/morpheus/dfp/modules/dfp_inference_pipe.py`
+Source: `python/morpheus_dfp/morpheus_dfp/modules/dfp_inference_pipe.py`
 
 The `dfp_inference_pipe` module function consolidates multiple digital fingerprinting pipeline (DFP) modules relevant to the inference process into a single module. Its purpose is to simplify the creation and configuration of an inference pipeline by combining all necessary components.
 
@@ -407,7 +407,7 @@ def dfp_inference_pipe(builder: mrc.Builder):
 
 ### DFP Inference
 
-Source: `examples/digital_fingerprinting/production/morpheus/dfp/modules/dfp_inference.py`
+Source: `python/morpheus_dfp/morpheus_dfp/modules/dfp_inference.py`
 
 The `dfp_inference` module function creates an inference module that retrieves trained models and performs inference on the input data. The module requires a `model_name_formatter` and a `fallback_username` to be configured in its parameters.
 
@@ -443,7 +443,7 @@ For a complete reference, refer to: [Filter Detections](../../modules/core/filte
 
 ### DFP Post Processing
 
-Source: `examples/digital_fingerprinting/production/morpheus/dfp/modules/dfp_postprocessing.py`
+Source: `python/morpheus_dfp/morpheus_dfp/modules/dfp_postprocessing.py`
 
 The `dfp_postprocessing` module function performs post-processing tasks on the input data.