Merge branch 'branch-24.03' of github.com:nv-morpheus/Morpheus into d…

…ependencies-updates

CHANGELOG.md

@@ -15,6 +15,17 @@ See the License for the specific language governing permissions and
 limitations under the License.
 -->
 
+# Morpheus 23.11.01 (7 Dec 2023)
+
+## 🐛 Bug Fixes
+
+- Convert `models/ransomware-models/ransomw-model-short-rf-20220126.sav` to LFS ([#1408](https://github.com/nv-morpheus/Morpheus/pull/1408)) [@mdemoret-nv](https://github.com/mdemoret-nv)
+
+## 📖 Documentation
+
+- Cloud deployment guide fixes ([#1406](https://github.com/nv-morpheus/Morpheus/pull/1406)) [@dagardner-nv](https://github.com/dagardner-nv)
+
+
 # Morpheus 23.11.00 (30 Nov 2023)
 
 ## 🚨 Breaking Changes

docs/source/cloud_deployment_guide.md

@@ -47,7 +47,6 @@ limitations under the License.
 - [Additional Documentation](#additional-documentation)
 - [Troubleshooting](#troubleshooting)
   - [Common Problems](#common-problems)
-- [The dropna stage](#the-dropna-stage)
 
 
 ## Introduction
@@ -403,7 +402,7 @@ To publish messages to a Kafka topic, we need to copy datasets to locations wher
 kubectl -n $NAMESPACE exec sdk-cli-helper -- cp -R /workspace/examples/data /common
 ```
 
-Refer to the [Morpheus CLI Overview](https://github.com/nv-morpheus/Morpheus/blob/branch-24.03/docs/source/basics/overview.rst) and [Building a Pipeline](https://github.com/nv-morpheus/Morpheus/blob/branch-24.03/docs/source/basics/building_a_pipeline.rst) documentation for more information regarding the commands.
+Refer to the [Morpheus CLI Overview](https://github.com/nv-morpheus/Morpheus/blob/branch-24.03/docs/source/basics/overview.rst) and [Building a Pipeline](https://github.com/nv-morpheus/Morpheus/blob/branch-24.03/docs/source/basics/building_a_pipeline.md) documentation for more information regarding the commands.
 
 > **Note**: Before running the example pipelines, ensure the criteria below are met:
 -   Ensure models specific to the pipeline are deployed.
@@ -445,6 +444,7 @@ helm install --set ngc.apiKey="$API_KEY" \
         --userid_filter=user123 \
         --feature_scaler=standard \
         --userid_column_name=userIdentitysessionContextsessionIssueruserName \
+        --timestamp_column_name=event_dt \
         from-cloudtrail --input_glob=/common/models/datasets/validation-data/dfp-cloudtrail-*-input.csv \
         --max_files=200 \
         train-ae --train_data_glob=/common/models/datasets/training-data/dfp-cloudtrail-*.csv \
@@ -495,7 +495,7 @@ helm install --set ngc.apiKey="$API_KEY" \
         monitor --description 'Preprocess Rate' \
         inf-triton --model_name=phishing-bert-onnx --server_url=ai-engine:8000 --force_convert_inputs=True \
         monitor --description 'Inference Rate' --smoothing=0.001 --unit inf \
-        add-class --label=pred --threshold=0.7 \
+        add-class --label=is_phishing --threshold=0.7 \
         serialize \
         to-file --filename=/common/data/<YOUR_OUTPUT_DIR>/phishing-bert-onnx-output.jsonlines --overwrite" \
     --namespace $NAMESPACE \
@@ -525,7 +525,7 @@ helm install --set ngc.apiKey="$API_KEY" \
         monitor --description 'Preprocess Rate' \
         inf-triton --force_convert_inputs=True --model_name=phishing-bert-onnx --server_url=ai-engine:8000 \
         monitor --description='Inference Rate' --smoothing=0.001 --unit inf \
-        add-class --label=pred --threshold=0.7 \
+        add-class --label=is_phishing --threshold=0.7 \
         serialize --exclude '^ts_' \
         to-kafka --output_topic <YOUR_OUTPUT_KAFKA_TOPIC> --bootstrap_servers broker:9092" \
     --namespace $NAMESPACE \
@@ -782,7 +782,7 @@ kubectl -n $NAMESPACE exec deploy/broker -c broker -- kafka-topics.sh \
 
 ## Additional Documentation
 For more information on how to use the Morpheus Python API to customize and run your own optimized AI pipelines, Refer to below documentation.
-- [Morpheus Developer Guide](https://github.com/nv-morpheus/Morpheus/tree/branch-24.03/docs/source/developer_guide)
+- [Morpheus Developer Guides](https://github.com/nv-morpheus/Morpheus/blob/branch-24.03/docs/source/developer_guide/guides.md)
 - [Morpheus Pipeline Examples](https://github.com/nv-morpheus/Morpheus/tree/branch-24.03/examples)
 
 
@@ -807,17 +807,3 @@ This section lists solutions to problems you might encounter with Morpheus or fr
   - Problem: If the standalone kafka cluster is receiving significant message throughput from the producer, this error may happen.
 
   - Solution: Reinstall the Morpheus workflow and reduce the Kafka topic's message retention time and message producing rate.
-
-## The dropna stage
-The Drop Null Attributes stage (dropna) requires the specification of a column name. This column will vary from use case (and its input data) to use case. These are the applicable columns for the pre-built pipelines provided by Morpheus.
-
-| Input | Columns |
-| ----- | ------- |
-| Azure DFP | userPrincipalName |
-| Duo DFP | username |
-| DFP Cloudtrail | userIdentitysessionContextsessionIssueruserName |
-| Email | data |
-| GNN | index, client_node, merchant_node |
-| Log Parsing | raw |
-| PCAP | data |
-| Ransomware | PID, Process, snapshot_id, timestamp, source |

docs/source/developer_guide/guides/10_modular_pipeline_digital_fingerprinting.md

@@ -527,7 +527,7 @@ From the `examples/digital_fingerprinting/production` dir, run:
 ```bash
 docker compose run morpheus_pipeline bash
 ```
-To run the DFP pipelines with the example datasets within the container, run:
+To run the DFP pipelines with the example datasets within the container, run the following from `examples/digital_fingerprinting/production/morpheus`:
 
 * Duo Training Pipeline
     ```bash
@@ -560,7 +560,7 @@ To run the DFP pipelines with the example datasets within the container, run:
         --start_time "2022-08-01" \
         --duration "60d" \
         --train_users generic \
-        --input_file "./control_messages/duo_payload_load_train_inference.json" 
+        --input_file "./control_messages/duo_payload_load_train_inference.json"
     ```
 
 * Azure Training Pipeline
@@ -594,7 +594,7 @@ To run the DFP pipelines with the example datasets within the container, run:
         --start_time "2022-08-01" \
         --duration "60d" \
         --train_users generic \
-        --input_file "./control_messages/azure_payload_load_train_inference.json" 
+        --input_file "./control_messages/azure_payload_load_train_inference.json"
     ```
 
 ### Output Fields
@@ -615,4 +615,4 @@ In addition to this, for each input feature the following output fields will exi
 | `<feature name>_z_loss` | FLOAT | The loss z-score |
 | `<feature name>_pred` | FLOAT | The predicted value |
 
-Refer to [DFPInferenceStage](6_digital_fingerprinting_reference.md#inference-stage-dfpinferencestage) for more on these fields.
+Refer to [DFPInferenceStage](6_digital_fingerprinting_reference.md#inference-stage-dfpinferencestage) for more on these fields.

docs/source/extra_info/troubleshooting.md

@@ -29,6 +29,42 @@ rm -rf ${MORPHEUS_ROOT}/build
 # Restart the build
 ./scripts/compile.sh
 ```
+
+**Incompatible MLflow Models**
+
+Models trained with a previous version of Morpheus and stored into MLflow may be incompatible with the current version. This error can be identified by the following error message occurring in an MLflow based pipeline such as DFP.
+
+```
+Error trying to get model
+
+Traceback (most recent call last):
+
+File "/workspace/examples/digital_fingerprinting/production/morpheus/dfp/stages/dfp_inference_stage.py", line 101, in on_data
+
+loaded_model = model_cache.load_model(self._client)
+```
+```
+ModuleNotFoundError: No module named 'dfencoder'
+```
+
+The work arounds available for this issue are:
+
+* Revert to the previous version of Morpheus until the models can be re-trained.
+* Re-train the model using the current version of Morpheus
+
+In the case of models trained by the DFP example, the existing models can be deleted by running the following command:
+
+```bash
+docker volume ls # list current docker volumes
+docker volume rm production_db_data production_mlflow_data
+
+# Re-build the MLflow container for DFP
+cd ${MORPHEUS_ROOT}/examples/digital_fingerprinting/production/
+docker compose build
+docker compose up mlflow
+```
+
+
 **Debugging Python Code**
 
 To debug issues in python code, several Visual Studio Code launch configurations have been included in the repo. These launch configurations can be found in `${MORPHEUS_ROOT}/morpheus.code-workspace`. To launch the debugging environment, ensure Visual Studio Code has opened the morpheus workspace file (File->Open Workspace from File...). Once the workspace has been loaded, the launch configurations should be available in the debugging tab.

examples/digital_fingerprinting/production/morpheus/benchmarks/README.md

@@ -103,11 +103,11 @@ To ensure the [file_to_df_loader.py](../../../../../morpheus/loaders/file_to_df_
 export MORPHEUS_FILE_DOWNLOAD_TYPE=dask
 ```
 
-Benchmarks for an individual workflow can be run using the following in your dev container:
+Benchmarks for an individual workflow can be run from `examples/digital_fingerprinting/production/morpheus` in your dev container:
 
 ```
 
-pytest -s --log-level=WARN --benchmark-enable --benchmark-warmup=on --benchmark-warmup-iterations=1 --benchmark-autosave test_bench_e2e_dfp_pipeline.py::<test-workflow>
+pytest -s --log-level=WARN --benchmark-enable --benchmark-warmup=on --benchmark-warmup-iterations=1 --benchmark-autosave benchmarks/test_bench_e2e_dfp_pipeline.py::<test-workflow>
 ```
 The `-s` option allows outputs of pipeline execution to be displayed so you can ensure there are no errors while running your benchmarks.
 
@@ -137,12 +137,12 @@ The `--benchmark-warmup` and `--benchmark-warmup-iterations` options are used to
 
 For example, to run E2E benchmarks on the DFP training (modules) workflow on the azure logs:
 ```
-pytest -s --benchmark-enable --benchmark-warmup=on --benchmark-warmup-iterations=1 --benchmark-autosave test_bench_e2e_dfp_pipeline.py::test_dfp_modules_azure_payload_lti_e2e
+pytest -s --benchmark-enable --benchmark-warmup=on --benchmark-warmup-iterations=1 --benchmark-autosave benchmarks/test_bench_e2e_dfp_pipeline.py::test_dfp_modules_azure_payload_lti_e2e
 ```
 
 To run E2E benchmarks on all workflows:
 ```
-pytest -s --benchmark-enable --benchmark-warmup=on --benchmark-warmup-iterations=1 --benchmark-autosave test_bench_e2e_dfp_pipeline.py
+pytest -s --benchmark-enable --benchmark-warmup=on --benchmark-warmup-iterations=1 --benchmark-autosave benchmarks/test_bench_e2e_dfp_pipeline.py
 ```
 
 Here are the benchmark comparisons for individual tests. When the control message type is "payload", the rolling window stage is bypassed, whereas when it is "streaming", the windows are created with historical data.

examples/digital_fingerprinting/production/morpheus/benchmarks/benchmark_conf_generator.py

@@ -161,6 +161,8 @@ def get_module_conf(self):
                                       source=(self.source),
                                       tracking_uri=mlflow.get_tracking_uri(),
                                       silence_monitors=True,
+                                      mlflow_experiment_name_formatter=self._get_experiment_name_formatter(),
+                                      mlflow_model_name_formatter=self._get_model_name_formatter(),
                                       train_users='generic')
         dfp_arg_parser.init()
         config_generator = ConfigGenerator(self.pipe_config, dfp_arg_parser, self.get_schema())

examples/digital_fingerprinting/production/morpheus/control_messages/azure_payload_inference.json

@@ -7,7 +7,7 @@
 			"properties": {
 			  "loader_id": "fsspec",
 			  "files": [
-				"../../../../examples/data/dfp/azure-inference-data/*.json"
+				"../../../data/dfp/azure-inference-data/*.json"
 			  ]
 			}
 		  },

examples/digital_fingerprinting/production/morpheus/control_messages/azure_payload_load_train_inference.json

@@ -7,7 +7,7 @@
 			"properties": {
 			  "loader_id": "fsspec",
 			  "files": [
-				"../../../../examples/data/dfp/azure-training-data/*.json"
+				"../../../data/dfp/azure-training-data/*.json"
 			  ]
 			}
 		  },
@@ -28,7 +28,7 @@
 			"properties": {
 			  "loader_id": "fsspec",
 			  "files": [
-				"../../../../examples/data/dfp/azure-inference-data/*.json"
+				"../../../data/dfp/azure-inference-data/*.json"
 			  ]
 			}
 		  },

examples/digital_fingerprinting/production/morpheus/control_messages/azure_payload_load_training.json

@@ -7,7 +7,7 @@
 			"properties": {
 			  "loader_id": "fsspec",
 			  "files": [
-				"../../../../examples/data/dfp/azure-training-data/*.json"
+				"../../../data/dfp/azure-training-data/*.json"
 			  ]
 			}
 		  },

examples/digital_fingerprinting/production/morpheus/control_messages/azure_payload_lti.json

@@ -7,7 +7,7 @@
 			"properties": {
 			  "loader_id": "fsspec",
 			  "files": [
-				"../../../../examples/data/dfp/azure-training-data/*.json"
+				"../../../data/dfp/azure-training-data/*.json"
 			  ]
 			}
 		  },
@@ -34,7 +34,7 @@
 			"properties": {
 			  "loader_id": "fsspec",
 			  "files": [
-				"../../../../examples/data/dfp/azure-inference-data/*.json"
+				"../../../data/dfp/azure-inference-data/*.json"
 			  ]
 			}
 		  },

examples/digital_fingerprinting/production/morpheus/control_messages/azure_payload_training.json

@@ -7,7 +7,7 @@
 			"properties": {
 			  "loader_id": "fsspec",
 			  "files": [
-				"../../../../examples/data/dfp/azure-training-data/*.json"
+				"../../../data/dfp/azure-training-data/*.json"
 			  ]
 			}
 		  },

examples/digital_fingerprinting/production/morpheus/control_messages/azure_streaming_inference.json

@@ -7,7 +7,7 @@
 			"properties": {
 			  "loader_id": "fsspec",
 			  "files": [
-				"../../../../examples/data/dfp/azure-inference-data/*.json"
+				"../../../data/dfp/azure-inference-data/*.json"
 			  ]
 			}
 		  },

examples/digital_fingerprinting/production/morpheus/control_messages/azure_streaming_lti.json

@@ -7,7 +7,7 @@
 			"properties": {
 			  "loader_id": "fsspec",
 			  "files": [
-				"../../../../examples/data/dfp/azure-training-data/*.json"
+				"../../../data/dfp/azure-training-data/*.json"
 			  ]
 			}
 		  },
@@ -34,7 +34,7 @@
 			"properties": {
 			  "loader_id": "fsspec",
 			  "files": [
-				"../../../../examples/data/dfp/azure-inference-data/*.json"
+				"../../../data/dfp/azure-inference-data/*.json"
 			  ]
 			}
 		  },

examples/digital_fingerprinting/production/morpheus/control_messages/azure_streaming_training.json

@@ -7,7 +7,7 @@
 			"properties": {
 			  "loader_id": "fsspec",
 			  "files": [
-				"../../../../examples/data/dfp/azure-training-data/*.json"
+				"../../../data/dfp/azure-training-data/*.json"
 			  ]
 			}
 		  },

examples/digital_fingerprinting/production/morpheus/control_messages/duo_payload_inference.json

@@ -7,7 +7,7 @@
 			"properties": {
 			  "loader_id": "fsspec",
 			  "files": [
-				"../../../../examples/data/dfp/duo-inference-data/*.json"
+				"../../../data/dfp/duo-inference-data/*.json"
 			  ]
 			}
 		  },

examples/digital_fingerprinting/production/morpheus/control_messages/duo_payload_load_train_inference.json

@@ -7,7 +7,7 @@
 			"properties": {
 			  "loader_id": "fsspec",
 			  "files": [
-				"../../../../examples/data/dfp/duo-training-data/*.json"
+				"../../../data/dfp/duo-training-data/*.json"
 			  ]
 			}
 		  },
@@ -28,7 +28,7 @@
 			"properties": {
 			  "loader_id": "fsspec",
 			  "files": [
-				"../../../../examples/data/dfp/duo-inference-data/*.json"
+				"../../../data/dfp/duo-inference-data/*.json"
 			  ]
 			}
 		  },

examples/digital_fingerprinting/production/morpheus/control_messages/duo_payload_lti.json

@@ -7,7 +7,7 @@
 			"properties": {
 			  "loader_id": "fsspec",
 			  "files": [
-				"../../../../examples/data/dfp/duo-training-data/*.json"
+				"../../../data/dfp/duo-training-data/*.json"
 			  ]
 			}
 		  },
@@ -34,7 +34,7 @@
 			"properties": {
 			  "loader_id": "fsspec",
 			  "files": [
-				"../../../../examples/data/dfp/duo-inference-data/*.json"
+				"../../../data/dfp/duo-inference-data/*.json"
 			  ]
 			}
 		  },

examples/digital_fingerprinting/production/morpheus/control_messages/duo_payload_only_load.json

@@ -7,7 +7,7 @@
 			"properties": {
 			  "loader_id": "fsspec",
 			  "files": [
-				"../../../../examples/data/dfp/duo-inference-data/*.json"
+				"../../../data/dfp/duo-inference-data/*.json"
 			  ]
 			}
 		  }

examples/digital_fingerprinting/production/morpheus/control_messages/duo_payload_training.json

@@ -7,7 +7,7 @@
           "properties": {
             "loader_id": "fsspec",
             "files": [
-              "../../../../examples/data/dfp/duo-training-data/*.json"
+              "../../../data/dfp/duo-training-data/*.json"
             ]
           }
         },

examples/digital_fingerprinting/production/morpheus/control_messages/duo_streaming_inference.json

@@ -7,7 +7,7 @@
 			"properties": {
 			  "loader_id": "fsspec",
 			  "files": [
-				"../../../../examples/data/dfp/duo-inference-data/*.json"
+				"../../../data/dfp/duo-inference-data/*.json"
 			  ]
 			}
 		  },

examples/digital_fingerprinting/production/morpheus/control_messages/duo_streaming_lti.json

@@ -7,7 +7,7 @@
 			"properties": {
 			  "loader_id": "fsspec",
 			  "files": [
-				"../../../../../examples/data/dfp/duo-training-data/*.json"
+				"../../../data/dfp/duo-training-data/*.json"
 			  ]
 			}
 		  },
@@ -34,7 +34,7 @@
 			"properties": {
 			  "loader_id": "fsspec",
 			  "files": [
-				"../../../../../examples/data/dfp/duo-inference-data/*.json"
+				"../../../data/dfp/duo-inference-data/*.json"
 			  ]
 			}
 		  },