From 054c5e18f5fe799e7b070e50faa1915eeac3df01 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 15 Jul 2024 08:21:33 -0700 Subject: [PATCH 1/9] Intentionally set a local relative link, this causes the documentation build to fail --- examples/llm/agents/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/llm/agents/README.md b/examples/llm/agents/README.md index 7f51397c14..19b2b33cd8 100644 --- a/examples/llm/agents/README.md +++ b/examples/llm/agents/README.md @@ -161,7 +161,7 @@ The Kafka Example in the Morpheus LLM Agents demonstrates an streaming implement facilitate the near real-time processing of LLM queries. This example is similar to the Simple example but makes use of a KafkaSourceStage to stream and retrieve messages from the Kafka topic -First, to run the Kafka example, you need to create a Kafka cluster that enables the persistent pipeline to accept queries for the LLM agents. You can create the Kafka cluster using the following guide: [Quick Launch Kafka Cluster Guide](https://github.com/nv-morpheus/Morpheus/blob/branch-23.11/docs/source/developer_guide/contributing.md#quick-launch-kafka-cluster) +First, to run the Kafka example, you need to create a Kafka cluster that enables the persistent pipeline to accept queries for the LLM agents. You can create the Kafka cluster using the following guide: [Quick Launch Kafka Cluster Guide](../../../docs/source/developer_guide/contributing.md#quick-launch-kafka-cluster) Once the Kafka cluster is running, create Kafka topic to produce input to the pipeline. From 1a642bfb274a9fee44c23142f0c6d5002af75972 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 15 Jul 2024 08:24:21 -0700 Subject: [PATCH 2/9] Fix type-o --- docs/source/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/conf.py b/docs/source/conf.py index 709e8230e0..902d7e6210 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -50,7 +50,7 @@ from github_link import make_linkcode_resolve # noqa -# Set an environment variable we can use to determine ifuncf we are building docs +# Set an environment variable we can use to determine if we are building docs os.environ["MORPHEUS_IN_SPHINX_BUILD"] = "1" # -- Project information ----------------------------------------------------- From 0dc3921e980c2178842ff605e5d2da00ee8ba4b5 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 15 Jul 2024 11:59:46 -0700 Subject: [PATCH 3/9] First pass at replacing symlinks as a method for including documents outside of the docs/ tree with stub markdown files using the {include} directive --- docs/source/examples/abp_nvsmi_detection/README.md | 5 ++++- docs/source/examples/abp_nvsmi_detection/pipeline.png | 1 - docs/source/examples/llm/agents/README.md | 5 ++++- 3 files changed, 8 insertions(+), 3 deletions(-) mode change 120000 => 100644 docs/source/examples/abp_nvsmi_detection/README.md delete mode 120000 docs/source/examples/abp_nvsmi_detection/pipeline.png mode change 120000 => 100644 docs/source/examples/llm/agents/README.md diff --git a/docs/source/examples/abp_nvsmi_detection/README.md b/docs/source/examples/abp_nvsmi_detection/README.md deleted file mode 120000 index 4a60061ddd..0000000000 --- a/docs/source/examples/abp_nvsmi_detection/README.md +++ /dev/null @@ -1 +0,0 @@ -../../../../examples/abp_nvsmi_detection/README.md \ No newline at end of file diff --git a/docs/source/examples/abp_nvsmi_detection/README.md b/docs/source/examples/abp_nvsmi_detection/README.md new file mode 100644 index 0000000000..5ec3af7b8f --- /dev/null +++ b/docs/source/examples/abp_nvsmi_detection/README.md @@ -0,0 +1,4 @@ +```{include} ../../../../examples/abp_nvsmi_detection/README.md +:relative-docs: .. +:relative-images: +``` diff --git a/docs/source/examples/abp_nvsmi_detection/pipeline.png b/docs/source/examples/abp_nvsmi_detection/pipeline.png deleted file mode 120000 index 0af176f32c..0000000000 --- a/docs/source/examples/abp_nvsmi_detection/pipeline.png +++ /dev/null @@ -1 +0,0 @@ -../../../../examples/abp_nvsmi_detection/pipeline.png \ No newline at end of file diff --git a/docs/source/examples/llm/agents/README.md b/docs/source/examples/llm/agents/README.md deleted file mode 120000 index 25df8ebb21..0000000000 --- a/docs/source/examples/llm/agents/README.md +++ /dev/null @@ -1 +0,0 @@ -../../../../../examples/llm/agents/README.md \ No newline at end of file diff --git a/docs/source/examples/llm/agents/README.md b/docs/source/examples/llm/agents/README.md new file mode 100644 index 0000000000..9035593229 --- /dev/null +++ b/docs/source/examples/llm/agents/README.md @@ -0,0 +1,4 @@ +```{include} ../../../../../examples/llm/agents/README.md +:relative-docs: .. +:relative-images: +``` From e05f563d39cd583a8d838e904727611341b7a224 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 15 Jul 2024 12:03:27 -0700 Subject: [PATCH 4/9] Insert CR header --- .../examples/abp_nvsmi_detection/README.md | 17 +++++++++++++++++ docs/source/examples/llm/agents/README.md | 17 +++++++++++++++++ 2 files changed, 34 insertions(+) diff --git a/docs/source/examples/abp_nvsmi_detection/README.md b/docs/source/examples/abp_nvsmi_detection/README.md index 5ec3af7b8f..3d1ddbab37 100644 --- a/docs/source/examples/abp_nvsmi_detection/README.md +++ b/docs/source/examples/abp_nvsmi_detection/README.md @@ -1,3 +1,20 @@ + + ```{include} ../../../../examples/abp_nvsmi_detection/README.md :relative-docs: .. :relative-images: diff --git a/docs/source/examples/llm/agents/README.md b/docs/source/examples/llm/agents/README.md index 9035593229..6b0a454800 100644 --- a/docs/source/examples/llm/agents/README.md +++ b/docs/source/examples/llm/agents/README.md @@ -1,3 +1,20 @@ + + ```{include} ../../../../../examples/llm/agents/README.md :relative-docs: .. :relative-images: From 10477c3ab7b3cb1a79982e71264eaf614a7272a5 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 15 Jul 2024 12:31:35 -0700 Subject: [PATCH 5/9] Replace all documentation symlinks with stub files --- .../examples/abp_pcap_detection/README.md | 22 ++++++++++++++++++- docs/source/examples/doca/README.md | 22 ++++++++++++++++++- .../gnn_fraud_detection_pipeline/README.md | 22 ++++++++++++++++++- docs/source/examples/llm/completion/README.md | 22 ++++++++++++++++++- docs/source/examples/llm/rag/README.md | 22 ++++++++++++++++++- .../llm/rag/images/persistent_pipeline.png | 1 - docs/source/examples/llm/vdb_upload/README.md | 22 ++++++++++++++++++- docs/source/examples/log_parsing/README.md | 22 ++++++++++++++++++- .../examples/nlp_si_detection/README.md | 22 ++++++++++++++++++- .../examples/nlp_si_detection/pipeline.png | 1 - .../examples/ransomware_detection/README.md | 22 ++++++++++++++++++- .../examples/root_cause_analysis/README.md | 22 ++++++++++++++++++- .../examples/sid_visualization/README.md | 22 ++++++++++++++++++- 13 files changed, 231 insertions(+), 13 deletions(-) mode change 120000 => 100644 docs/source/examples/abp_pcap_detection/README.md mode change 120000 => 100644 docs/source/examples/doca/README.md mode change 120000 => 100644 docs/source/examples/gnn_fraud_detection_pipeline/README.md mode change 120000 => 100644 docs/source/examples/llm/completion/README.md mode change 120000 => 100644 docs/source/examples/llm/rag/README.md delete mode 120000 docs/source/examples/llm/rag/images/persistent_pipeline.png mode change 120000 => 100644 docs/source/examples/llm/vdb_upload/README.md mode change 120000 => 100644 docs/source/examples/log_parsing/README.md mode change 120000 => 100644 docs/source/examples/nlp_si_detection/README.md delete mode 120000 docs/source/examples/nlp_si_detection/pipeline.png mode change 120000 => 100644 docs/source/examples/ransomware_detection/README.md mode change 120000 => 100644 docs/source/examples/root_cause_analysis/README.md mode change 120000 => 100644 docs/source/examples/sid_visualization/README.md diff --git a/docs/source/examples/abp_pcap_detection/README.md b/docs/source/examples/abp_pcap_detection/README.md deleted file mode 120000 index 3f20525785..0000000000 --- a/docs/source/examples/abp_pcap_detection/README.md +++ /dev/null @@ -1 +0,0 @@ -../../../../examples/abp_pcap_detection/README.md \ No newline at end of file diff --git a/docs/source/examples/abp_pcap_detection/README.md b/docs/source/examples/abp_pcap_detection/README.md new file mode 100644 index 0000000000..f6d91f484d --- /dev/null +++ b/docs/source/examples/abp_pcap_detection/README.md @@ -0,0 +1,21 @@ + + +```{include} ../../../../examples/abp_pcap_detection/README.md +:relative-docs: .. +:relative-images: +``` diff --git a/docs/source/examples/doca/README.md b/docs/source/examples/doca/README.md deleted file mode 120000 index c3e43df7d9..0000000000 --- a/docs/source/examples/doca/README.md +++ /dev/null @@ -1 +0,0 @@ -../../../../examples/doca/README.md \ No newline at end of file diff --git a/docs/source/examples/doca/README.md b/docs/source/examples/doca/README.md new file mode 100644 index 0000000000..47573d0595 --- /dev/null +++ b/docs/source/examples/doca/README.md @@ -0,0 +1,21 @@ + + +```{include} ../../../../examples/doca/README.md +:relative-docs: .. +:relative-images: +``` diff --git a/docs/source/examples/gnn_fraud_detection_pipeline/README.md b/docs/source/examples/gnn_fraud_detection_pipeline/README.md deleted file mode 120000 index 74d89a609c..0000000000 --- a/docs/source/examples/gnn_fraud_detection_pipeline/README.md +++ /dev/null @@ -1 +0,0 @@ -../../../../examples/gnn_fraud_detection_pipeline/README.md \ No newline at end of file diff --git a/docs/source/examples/gnn_fraud_detection_pipeline/README.md b/docs/source/examples/gnn_fraud_detection_pipeline/README.md new file mode 100644 index 0000000000..fdbb543a53 --- /dev/null +++ b/docs/source/examples/gnn_fraud_detection_pipeline/README.md @@ -0,0 +1,21 @@ + + +```{include} ../../../../examples/gnn_fraud_detection_pipeline/README.md +:relative-docs: .. +:relative-images: +``` diff --git a/docs/source/examples/llm/completion/README.md b/docs/source/examples/llm/completion/README.md deleted file mode 120000 index acf33313c1..0000000000 --- a/docs/source/examples/llm/completion/README.md +++ /dev/null @@ -1 +0,0 @@ -../../../../../examples/llm/completion/README.md \ No newline at end of file diff --git a/docs/source/examples/llm/completion/README.md b/docs/source/examples/llm/completion/README.md new file mode 100644 index 0000000000..215564a75c --- /dev/null +++ b/docs/source/examples/llm/completion/README.md @@ -0,0 +1,21 @@ + + +```{include} ../../../../../examples/llm/completion/README.md +:relative-docs: .. +:relative-images: +``` diff --git a/docs/source/examples/llm/rag/README.md b/docs/source/examples/llm/rag/README.md deleted file mode 120000 index 4d253e05dd..0000000000 --- a/docs/source/examples/llm/rag/README.md +++ /dev/null @@ -1 +0,0 @@ -../../../../../examples/llm/rag/README.md \ No newline at end of file diff --git a/docs/source/examples/llm/rag/README.md b/docs/source/examples/llm/rag/README.md new file mode 100644 index 0000000000..50cca84d70 --- /dev/null +++ b/docs/source/examples/llm/rag/README.md @@ -0,0 +1,21 @@ + + +```{include} ../../../../../examples/llm/rag/README.md +:relative-docs: .. +:relative-images: +``` diff --git a/docs/source/examples/llm/rag/images/persistent_pipeline.png b/docs/source/examples/llm/rag/images/persistent_pipeline.png deleted file mode 120000 index f74a9ea3f9..0000000000 --- a/docs/source/examples/llm/rag/images/persistent_pipeline.png +++ /dev/null @@ -1 +0,0 @@ -../../../../../../examples/llm/rag/images/persistent_pipeline.png \ No newline at end of file diff --git a/docs/source/examples/llm/vdb_upload/README.md b/docs/source/examples/llm/vdb_upload/README.md deleted file mode 120000 index 8e4daece9f..0000000000 --- a/docs/source/examples/llm/vdb_upload/README.md +++ /dev/null @@ -1 +0,0 @@ -../../../../../examples/llm/vdb_upload/README.md \ No newline at end of file diff --git a/docs/source/examples/llm/vdb_upload/README.md b/docs/source/examples/llm/vdb_upload/README.md new file mode 100644 index 0000000000..8acf34e387 --- /dev/null +++ b/docs/source/examples/llm/vdb_upload/README.md @@ -0,0 +1,21 @@ + + +```{include} ../../../../../examples/llm/vdb_upload/README.md +:relative-docs: .. +:relative-images: +``` diff --git a/docs/source/examples/log_parsing/README.md b/docs/source/examples/log_parsing/README.md deleted file mode 120000 index ee8517820f..0000000000 --- a/docs/source/examples/log_parsing/README.md +++ /dev/null @@ -1 +0,0 @@ -../../../../examples/log_parsing/README.md \ No newline at end of file diff --git a/docs/source/examples/log_parsing/README.md b/docs/source/examples/log_parsing/README.md new file mode 100644 index 0000000000..be0450c9f7 --- /dev/null +++ b/docs/source/examples/log_parsing/README.md @@ -0,0 +1,21 @@ + + +```{include} ../../../../examples/log_parsing/README.md +:relative-docs: .. +:relative-images: +``` diff --git a/docs/source/examples/nlp_si_detection/README.md b/docs/source/examples/nlp_si_detection/README.md deleted file mode 120000 index 8dbdbc0f30..0000000000 --- a/docs/source/examples/nlp_si_detection/README.md +++ /dev/null @@ -1 +0,0 @@ -../../../../examples/nlp_si_detection/README.md \ No newline at end of file diff --git a/docs/source/examples/nlp_si_detection/README.md b/docs/source/examples/nlp_si_detection/README.md new file mode 100644 index 0000000000..72cb1d280e --- /dev/null +++ b/docs/source/examples/nlp_si_detection/README.md @@ -0,0 +1,21 @@ + + +```{include} ../../../../examples/nlp_si_detection/README.md +:relative-docs: .. +:relative-images: +``` diff --git a/docs/source/examples/nlp_si_detection/pipeline.png b/docs/source/examples/nlp_si_detection/pipeline.png deleted file mode 120000 index 393ae7e8fd..0000000000 --- a/docs/source/examples/nlp_si_detection/pipeline.png +++ /dev/null @@ -1 +0,0 @@ -../../../../examples/nlp_si_detection/pipeline.png \ No newline at end of file diff --git a/docs/source/examples/ransomware_detection/README.md b/docs/source/examples/ransomware_detection/README.md deleted file mode 120000 index ef96606ec5..0000000000 --- a/docs/source/examples/ransomware_detection/README.md +++ /dev/null @@ -1 +0,0 @@ -../../../../examples/ransomware_detection/README.md \ No newline at end of file diff --git a/docs/source/examples/ransomware_detection/README.md b/docs/source/examples/ransomware_detection/README.md new file mode 100644 index 0000000000..8a2583d846 --- /dev/null +++ b/docs/source/examples/ransomware_detection/README.md @@ -0,0 +1,21 @@ + + +```{include} ../../../../examples/ransomware_detection/README.md +:relative-docs: .. +:relative-images: +``` diff --git a/docs/source/examples/root_cause_analysis/README.md b/docs/source/examples/root_cause_analysis/README.md deleted file mode 120000 index af7e9e0e4b..0000000000 --- a/docs/source/examples/root_cause_analysis/README.md +++ /dev/null @@ -1 +0,0 @@ -../../../../examples/root_cause_analysis/README.md \ No newline at end of file diff --git a/docs/source/examples/root_cause_analysis/README.md b/docs/source/examples/root_cause_analysis/README.md new file mode 100644 index 0000000000..3111f15a74 --- /dev/null +++ b/docs/source/examples/root_cause_analysis/README.md @@ -0,0 +1,21 @@ + + +```{include} ../../../../examples/root_cause_analysis/README.md +:relative-docs: .. +:relative-images: +``` diff --git a/docs/source/examples/sid_visualization/README.md b/docs/source/examples/sid_visualization/README.md deleted file mode 120000 index a1042736b8..0000000000 --- a/docs/source/examples/sid_visualization/README.md +++ /dev/null @@ -1 +0,0 @@ -../../../../examples/sid_visualization/README.md \ No newline at end of file diff --git a/docs/source/examples/sid_visualization/README.md b/docs/source/examples/sid_visualization/README.md new file mode 100644 index 0000000000..f429e3123b --- /dev/null +++ b/docs/source/examples/sid_visualization/README.md @@ -0,0 +1,21 @@ + + +```{include} ../../../../examples/sid_visualization/README.md +:relative-docs: .. +:relative-images: +``` From 28e100a0204cb73c71a70ca5dd853dbb4891f621 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 15 Jul 2024 12:48:20 -0700 Subject: [PATCH 6/9] Use a relative link --- examples/digital_fingerprinting/starter/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/digital_fingerprinting/starter/README.md b/examples/digital_fingerprinting/starter/README.md index 5360ad2255..75c6040707 100644 --- a/examples/digital_fingerprinting/starter/README.md +++ b/examples/digital_fingerprinting/starter/README.md @@ -22,7 +22,7 @@ We show here how to set up and run the DFP pipeline for three log types: CloudTr ## Environment Setup -Follow the instructions [here](https://github.com/nv-morpheus/Morpheus/blob/branch-24.10/docs/source/developer_guide/contributing.md) to set up your development environment in either a Docker container or conda environment. +Follow the instructions [here](../../../docs/source/developer_guide/contributing.md) to set up your development environment in either a Docker container or conda environment. ## Morpheus CLI From db38ec9e1600f9d664de26452bc437c1546d6f5e Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 15 Jul 2024 13:24:07 -0700 Subject: [PATCH 7/9] Fix anchor link --- .devcontainer/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.devcontainer/README.md b/.devcontainer/README.md index 8a91347e49..8a28296fe9 100644 --- a/.devcontainer/README.md +++ b/.devcontainer/README.md @@ -17,7 +17,7 @@ limitations under the License. # Morpheus Devcontainer -The Morpheus devcontainer is provided as a quick-to-set-up development and exploration environment for use with [Visual Studio Code](https://code.visualstudio.com) (Code). The devcontainer is a lightweight container which mounts-in a conda environment with cached packages, alleviating long conda download times on subsequent launches. It provides a simple framework for adding developer-centric [scripts](#scripts), and incorperates some helpful Code plugins, such as clangd and cmake support. +The Morpheus devcontainer is provided as a quick-to-set-up development and exploration environment for use with [Visual Studio Code](https://code.visualstudio.com) (Code). The devcontainer is a lightweight container which mounts-in a conda environment with cached packages, alleviating long conda download times on subsequent launches. It provides a simple framework for adding developer-centric [scripts](#development-scripts), and incorperates some helpful Code plugins, such as clangd and cmake support. More information about devcontainers can be found at [containers.dev](https://containers.dev/). From 353a33e31d7d2cfec5676e85c95d9b41e4799ac8 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 15 Jul 2024 13:24:32 -0700 Subject: [PATCH 8/9] replace links with relative links --- docs/source/cloud_deployment_guide.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/cloud_deployment_guide.md b/docs/source/cloud_deployment_guide.md index 31c3b9441a..0f6adc7345 100644 --- a/docs/source/cloud_deployment_guide.md +++ b/docs/source/cloud_deployment_guide.md @@ -402,7 +402,7 @@ To publish messages to a Kafka topic, we need to copy datasets to locations wher kubectl -n $NAMESPACE exec sdk-cli-helper -- cp -R /workspace/examples/data /common ``` -Refer to the [Morpheus CLI Overview](https://github.com/nv-morpheus/Morpheus/blob/branch-24.10/docs/source/basics/overview.rst) and [Building a Pipeline](https://github.com/nv-morpheus/Morpheus/blob/branch-24.10/docs/source/basics/building_a_pipeline.md) documentation for more information regarding the commands. +Refer to the [Morpheus CLI Overview](./basics/overview.rst) and [Building a Pipeline](./basics/building_a_pipeline.md) documentation for more information regarding the commands. > **Note**: Before running the example pipelines, ensure the criteria below are met: - Ensure models specific to the pipeline are deployed. @@ -782,8 +782,8 @@ kubectl -n $NAMESPACE exec deploy/broker -c broker -- kafka-topics.sh \ ## Additional Documentation For more information on how to use the Morpheus Python API to customize and run your own optimized AI pipelines, Refer to below documentation. -- [Morpheus Developer Guides](https://github.com/nv-morpheus/Morpheus/blob/branch-24.10/docs/source/developer_guide/guides.md) -- [Morpheus Pipeline Examples](https://github.com/nv-morpheus/Morpheus/tree/branch-24.10/examples) +- [Morpheus Developer Guides](./developer_guide/guides.md) +- [Morpheus Pipeline Examples](./examples.md) ## Troubleshooting From c5149972b78607dc746eb81b6e7cdcbbdcf22048 Mon Sep 17 00:00:00 2001 From: David Gardner Date: Mon, 15 Jul 2024 13:26:06 -0700 Subject: [PATCH 9/9] Replace url link to the devcontainer setup docs with a stubbed md file --- docs/source/devcontainer.md | 25 +++++++++++++++++++++++++ docs/source/examples.md | 2 +- 2 files changed, 26 insertions(+), 1 deletion(-) create mode 100644 docs/source/devcontainer.md diff --git a/docs/source/devcontainer.md b/docs/source/devcontainer.md new file mode 100644 index 0000000000..d6e7bd58a3 --- /dev/null +++ b/docs/source/devcontainer.md @@ -0,0 +1,25 @@ + + + +```{eval-rst} +:orphan: +``` +```{include} ../../.devcontainer/README.md +:relative-docs: .. +:relative-images: +``` diff --git a/docs/source/examples.md b/docs/source/examples.md index 922bf796cb..b1f7d6b9d5 100644 --- a/docs/source/examples.md +++ b/docs/source/examples.md @@ -44,4 +44,4 @@ The following are the supported environments: | [Conda](./developer_guide/contributing.md#build-in-a-conda-environment) | Morpheus is built from source by the end user, and dependencies are installed via the Conda package manager. | | [Morpheus Docker Container](./developer_guide/contributing.md#build-in-docker-container) | A Docker container that is built from source by the end user, Morpheus is then built from source from within the container. | | [Morpheus Release Container](./getting_started.md#building-the-morpheus-container) | Pre-built Docker container that is built from source by the Morpheus team, and is available for download from the [NGC container registry](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/morpheus/containers/morpheus/tags), or can be built locally from source. | -| [Dev Container](https://github.com/nv-morpheus/Morpheus/blob/branch-24.10/.devcontainer/README.md) | A [Dev Container](https://containers.dev/) that is built from source by the end user, Morpheus is then built from source from within the container. | +| [Dev Container](./devcontainer.md) | A [Dev Container](https://containers.dev/) that is built from source by the end user, Morpheus is then built from source from within the container. |