Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RELEASE] Morpheus v23.11 #1400

Merged
merged 456 commits into from
Dec 1, 2023
Merged

[RELEASE] Morpheus v23.11 #1400

merged 456 commits into from
Dec 1, 2023

Conversation

mdemoret-nv
Copy link
Contributor

❄️ Code freeze for branch-23.11 and v23.11 release

What does this mean?

Only critical/hotfix level issues should be merged into branch-23.11 until release (merging of this PR).

All other development PRs should be retargeted towards the next release branch: branch-24.03.

What is the purpose of this PR?

  • Update documentation
  • Allow testing for the new release
  • Enable a means to merge branch-23.11 into main for the release

dagardner-nv and others added 30 commits October 23, 2023 09:20
@dagardner-nv
Lint fixes
5afaa18
@dagardner-nv
linting fixes
56955bd
@dagardner-nv
Merge branch 'david-fea-sherlock-restructure' into david-fea-sherlock…
5bae140 
…-prompt-pipe
@dagardner-nv
Revert temp logging
d8d693a
@dagardner-nv
Fix merge errors
f460bf5
@dagardner-nv
Merge pull request #1296 from dagardner-nv/david-fea-sherlock-restruc…
4ff7963 
…ture

Add tests & docstrings for PromptTemplateNode
@dagardner-nv
Fix inputs [no ci]
dc79ed2
@dagardner-nv
Merge branch 'fea-sherlock' of github.com:nv-morpheus/Morpheus into d…
be86fdf 
…avid-fea-sherlock-prompt-pipe [no ci]
@dagardner-nv
Remove old comment
b0613dd
@dagardner-nv
Update concat_dataframes to support ControlMessage's
b859ca5
@dagardner-nv
Fix test to at least work for single values
3e2dcd7
@dagardner-nv
Remove old comments
d4aaad1
@dagardner-nv
Sort imports
08783b4
@dagardner-nv
Merge pull request #1308 from dagardner-nv/david-fea-sherlock-prompt-…
f0f8e93 
…pipe

Add integration test for PromptTemplateNode in a morpheus pipeline
@dagardner-nv
Move imports of packages not in the default environment into the body…
389d218 
… of the class
@dagardner-nv
First tests for ArxivSource
5a57ec6
@dagardner-nv
Finished tests for _generate_frames
4d16f6c
@dagardner-nv
Finished tests for _generate_frames [no ci]
7aabb7d
@dagardner-nv
Merge branch 'david-fea-sherlock-arxiv-source-stage' of github.com:da…
f08ac26 
…gardner-nv/Morpheus into david-fea-sherlock-arxiv-source-stage
@drobison00
vdb_upload pipeline updates. (#1309)
c616190 
Add README.md file and fix requirements.yaml file.

Authors:
  - Devin Robison (https://github.com/drobison00)

Approvers:
  - David Gardner (https://github.com/dagardner-nv)

URL: #1309
@dagardner-nv
tests for _process_pages [no ci]
68e070a
@dagardner-nv
WIP [no ci]
785ef37
@dagardner-nv
Implement max_pages feature, and complete tests [no ci]
59333a2
@dagardner-nv
Finish pipeline test
9c93fc9
@dagardner-nv
Linting fixes
be7d7f7
@dagardner-nv
Remove slow marker, this test executes in 0.05s
455361d
@dagardner-nv
Merge branch 'fea-sherlock' of github.com:nv-morpheus/Morpheus into d…
25a3137 
…avid-fea-sherlock-arxiv-source-stage
@dagardner-nv
Remove unused import
b33d949
@dagardner-nv
Remove testing of public attributes
5c64c13
@dagardner-nv
Remove testing of private attributes
4128ca8
dagardner-nv and others added 11 commits November 22, 2023 23:53
@dagardner-nv
Register DataBricksDeltaLakeSourceStage with the CLI (#1384)
bc4d40d 
Closes #1383 

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md).
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #1384
@dagardner-nv
Add missing milvus marker (#1385)
d2a962d 
Closes 

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md).
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #1385
@efajardo-nv
LLM C++ test and doc updates (#1379)
f87f16c 
- C++ unit test updates
- Split some tests into multiple tests
- Minor doc updates and fixes

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md).
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - Eli Fajardo (https://github.com/efajardo-nv)
  - Michael Demoret (https://github.com/mdemoret-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #1379
@dagardner-nv
Guard optional dependencies in try/except blocks (#1382)
119ec42 
* Add missing dependencies for `requests`, `requests-cache` and `websockets` to `meta.yaml`
* Dependencies which are only utilized by a single stage are intentionally omitted from `meta.yaml`, even if they appear in `cuda11.8_dev.yml`
* Current that list is: `bs4`, `elasticsearch`, `feedparser` and `milvus`
* Move `MILVUS_DATA_TYPE_MAP` from `morpheus/service/vdb/milvus_client.py` to `tests/test_milvus_vector_db_service.py` which was the only place it was used.

Note: Since these dependencies all exist in `cuda11.8_dev.yml`, there is no effort made to update the tests to optionally skip their associated tests.

Closes #1380

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md).
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #1382
@drobison00
Align model card requirements (#1388)
5f00e78 
Authors:
  - Devin Robison (https://github.com/drobison00)

Approvers:
  - David Gardner (https://github.com/dagardner-nv)
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #1388
@drobison00
Docs update to indicate use of conda-merge to generate install files (#…
f6537a2 
…1387)

Resolves #1369

Authors:
  - Devin Robison (https://github.com/drobison00)

Approvers:
  - Eli Fajardo (https://github.com/efajardo-nv)
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #1387
@mdemoret-nv
Fixing the conda-build with DOCA enabled and upgrading to CMake 3.25 (#…
ea188a1 
…1386)

This PR fixes the DOCA modules with building with `MORPHEUS_SUPPORT_DOCA=ON`. In addition, the following changes needed to be made:

- Upgrade CMake from 3.24 -> 3.25
   - Necessary to cleanly set `CMAKE_FIND_ROOT_PATH_MODE_INCLUDE` (and similar variables) using the `block()` function
   - Other option is to manually save off these specific variables and restore them all later which is cumbersome and error prone.
- Adds dependency to `rdma-core>=48`
   - This is necessary since DOCA requires the MLX_1.19 symbols which arent in `rdma-core=28.9`

Authors:
  - Michael Demoret (https://github.com/mdemoret-nv)
  - Pete MacKinnon (https://github.com/pdmack)

Approvers:
  - Christopher Harris (https://github.com/cwharris)

URL: #1386
@dagardner-nv
Add source & stage decorators (#1364)
ba28cff 
* Adds two new decorators `source` and `stage` allowing stages to be created from python functions.
* Accept & return types are inferred by inspecting the type annotations of the decorated function.
* This removes the need for several lines of boilerplate code needed for simple Python-only stages.
* Includes unrelated docstring fix to `morpheus/pipeline/stage_base.py`

example:
```py
from morpheus.pipeline.stage_decorator import source
from morpheus.pipeline.stage_decorator import stage

@source
def source_gen(dataframes: list[cudf.DataFrame]) -> collections.abc.Iterator[MessageMeta]:
    for df in dataframes:
        yield MessageMeta(df)


@stage
def on_data(message: MessageMeta) -> MessageMeta:
    with message.mutable_dataframe() as df:
        print(df)
    return message
```
 
Closes #1253

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md).
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Devin Robison (https://github.com/drobison00)
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #1364
@dagardner-nv
Fix race condition in the C++ impl for the pre-process fil stage (#1390)
3a02383 
* Fixes a race condition where the `fix_bad_columns` method attempts to fix the same column more than once per-DF, resulting in a failure on the second attempt.


## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md).
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - David Gardner (https://github.com/dagardner-nv)
  - Michael Demoret (https://github.com/mdemoret-nv)

Approvers:
  - Devin Robison (https://github.com/drobison00)
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #1390
@mdemoret-nv
Fix the SID Viz workflow shutdown process with the new pipeline shutd…
da3814a 
…own process (#1392)

Due to the changes in #1233, the pipeline shutdown process has been slightly altered. This updates the viz demo to shutdown correctly without error. Also added support for exiting on Ctrl+C

Fixes #1391

Authors:
  - Michael Demoret (https://github.com/mdemoret-nv)

Approvers:
  - Eli Fajardo (https://github.com/efajardo-nv)
  - David Gardner (https://github.com/dagardner-nv)

URL: #1392
@efajardo-nv
Grafana example readme update (#1393)
4ef7142 
- Remove step to log in to Grafana dashboard since example was updated to not require it anymore. 

## By Submitting this PR I confirm:
- I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md).
- When the PR is ready for review, new or existing tests cover these changes.
- When the PR is ready for review, the documentation is up to date with these changes.

Authors:
  - Eli Fajardo (https://github.com/efajardo-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #1393
@mdemoret-nv mdemoret-nv requested review from a team as code owners December 1, 2023 00:47
@mdemoret-nv mdemoret-nv added the ! - Release Related to a Major Release label Dec 1, 2023
@copy-pr-bot copy-pr-bot
Copy link

copy-pr-bot bot commented Dec 1, 2023

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@review-notebook-app Review Notebook App
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

@mdemoret-nv
Copy link
Contributor Author

/ok to test

@mdemoret-nv
CVE-2023-47248 Mitigation (#1399)
1579eaa 
This PR mitigates CVE-2023-47248 by importing `pyarrow_hotfix` as early as possible to ensure that the pyarrow hotfix is applied before any code can use it.

Can be removed after upgrading to pyarrow 14.0.1 or later (which is dictated by cudf)

Authors:
  - Michael Demoret (https://github.com/mdemoret-nv)

Approvers:
  - Christopher Harris (https://github.com/cwharris)
  - David Gardner (https://github.com/dagardner-nv)

URL: #1399
@mdemoret-nv
Copy link
Contributor Author

/ok to test

@mdemoret-nv
Copy link
Contributor Author

/ok to test

@mdemoret-nv
Copy link
Contributor Author

/ok to test

1 similar comment
@mdemoret-nv
Copy link
Contributor Author

/ok to test

@mdemoret-nv
Copy link
Contributor Author

/ok to test

@mdemoret-nv mdemoret-nv merged commit ebaeb23 into main Dec 1, 2023
7 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
! - Release Related to a Major Release
Projects
Morpheus Boards
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@mdemoret-nv @dagardner-nv @drobison00 @cwharris @bsuryadevara @efajardo-nv @pranavm7