SQL Max Connection Exhaustion and Update Control Message Utils #1238
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Forward-merge branch-23.07 to branch-23.11
Forward-merge branch-23.07 to branch-23.11
Forward-merge branch-23.07 to branch-23.11
Forward-merge branch-23.07 to branch-23.11
Forward-merge branch-23.07 to branch-23.11
Forward-merge branch-23.07 to branch-23.11
Forward-merge branch-23.07 to branch-23.11
This PR contains the changes from running the `ci/release/update-version.sh` script for a new release Authors: - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) - Tad ZeMicheal (https://github.com/tzemicheal) URL: nv-morpheus#1059
…ge-23.07 Resolve forward merger conflices for `branch-23.11`
Forward-merge branch-23.07 to branch-23.11
Forward-merge branch-23.07 to branch-23.11
* Allows building from a commit and not the head of a branch. Specifically allow running from a detached commit like a release tag. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1083
I have a few changes here. Some of them might not make it depending on how others feel.
- Move test utilities in to a library to share across tests.
- Move tests in to individual executable to enforce individual tests to not sharing code.
- Use `gtest_discover_tests` to let `ctest` run and show all tests individually.
- Use `gtest::gtest_main` and remove explicit main entry file.
- ~Build and run tests by default.~
- Ability to filter tests by `ctest --test-dir build -R "TestCuda.*"`
```
+ echo 'Running CTest...'
Running CTest...
+ ctest --test-dir build
Internal ctest changing into directory: /workspaces/morpheus/build
Test project /workspaces/morpheus/build
Start 1: TestCuda.LargeShape
1/35 Test #1: TestCuda.LargeShape ....................................... Passed 0.17 sec
Start 2: TestDataLoader.DataLoaderInitializationTest
2/35 Test nv-morpheus#2: TestDataLoader.DataLoaderInitializationTest ............... Passed 0.07 sec
Start 3: TestDataLoader.DataLoaderRegisterLoaderTest
3/35 Test nv-morpheus#3: TestDataLoader.DataLoaderRegisterLoaderTest ............... Passed 0.07 sec
Start 4: TestDataLoader.DataLoaderRemoveLoaderTest
4/35 Test nv-morpheus#4: TestDataLoader.DataLoaderRemoveLoaderTest ................. Passed 0.07 sec
Start 5: TestDataLoader.PayloadLoaderTest
5/35 Test nv-morpheus#5: TestDataLoader.PayloadLoaderTest .......................... Passed 1.42 sec
Start 6: TestDataLoader.FileLoaderTest
6/35 Test nv-morpheus#6: TestDataLoader.FileLoaderTest ............................. Passed 2.65 sec
Start 7: TestControlMessage.InitializationTest
7/35 Test nv-morpheus#7: TestControlMessage.InitializationTest ..................... Passed 0.07 sec
Start 8: TestControlMessage.SetMessageTest
8/35 Test nv-morpheus#8: TestControlMessage.SetMessageTest ......................... Passed 0.07 sec
Start 9: TestControlMessage.TaskTest
9/35 Test nv-morpheus#9: TestControlMessage.TaskTest ............................... Passed 0.07 sec
Start 10: TestControlMessage.PayloadTest
10/35 Test nv-morpheus#10: TestControlMessage.PayloadTest ............................ Passed 1.45 sec
Start 11: TestDataLoaderModule.DataLoaderModuleInitializationTest
11/35 Test nv-morpheus#11: TestDataLoaderModule.DataLoaderModuleInitializationTest ... Passed 0.07 sec
Start 12: TestDataLoaderModule.EndToEndPayloadDataLoaderTest
12/35 Test nv-morpheus#12: TestDataLoaderModule.EndToEndPayloadDataLoaderTest ........ Passed 0.12 sec
Start 13: TestDeserializers.GetIndexColCountNoIdxFromFile
13/35 Test nv-morpheus#13: TestDeserializers.GetIndexColCountNoIdxFromFile ........... Passed 1.39 sec
Start 14: TestDeserializers.GetIndexColCountWithIdxFromFile
14/35 Test nv-morpheus#14: TestDeserializers.GetIndexColCountWithIdxFromFile ......... Passed 1.37 sec
Start 15: TestDeserializers.GetIndexColCountNoIdxSimilarName
15/35 Test nv-morpheus#15: TestDeserializers.GetIndexColCountNoIdxSimilarName ........ Passed 0.19 sec
Start 16: TestDeserializers.GetIndexColCountIdx
16/35 Test nv-morpheus#16: TestDeserializers.GetIndexColCountIdx ..................... Passed 0.18 sec
Start 17: TestDeserializers.GetIndexColCountValidNameInvalidType
17/35 Test nv-morpheus#17: TestDeserializers.GetIndexColCountValidNameInvalidType .... Passed 0.16 sec
Start 18: TestDevMemInfo.RmmBufferConstructor
18/35 Test nv-morpheus#18: TestDevMemInfo.RmmBufferConstructor ....................... Passed 0.16 sec
Start 19: TestDevMemInfo.VoidPtrConstructor
19/35 Test nv-morpheus#19: TestDevMemInfo.VoidPtrConstructor ......................... Passed 0.25 sec
Start 20: TestDevMemInfo.MakeNewBuffer
20/35 Test nv-morpheus#20: TestDevMemInfo.MakeNewBuffer .............................. Passed 0.25 sec
Start 21: TestFileInOut.RoundTripCSV
21/35 Test nv-morpheus#21: TestFileInOut.RoundTripCSV ................................ Passed 2.67 sec
Start 22: TestFileInOut.RoundTripJSON
22/35 Test nv-morpheus#22: TestFileInOut.RoundTripJSON ............................... Passed 2.69 sec
Start 23: TestMatxUtil.ReduceMax1d
23/35 Test nv-morpheus#23: TestMatxUtil.ReduceMax1d .................................. Passed 0.17 sec
Start 24: TestMatxUtil.ReduceMax2dRowMajor
24/35 Test nv-morpheus#24: TestMatxUtil.ReduceMax2dRowMajor .......................... Passed 0.17 sec
Start 25: TestMatxUtil.ReduceMax2dColMajor
25/35 Test nv-morpheus#25: TestMatxUtil.ReduceMax2dColMajor .......................... Passed 1.40 sec
Start 26: TestMatxUtil.Cast
26/35 Test nv-morpheus#26: TestMatxUtil.Cast ......................................... Passed 0.16 sec
Start 27: TestMatxUtil.Threshold
27/35 Test nv-morpheus#27: TestMatxUtil.Threshold .................................... Passed 0.16 sec
Start 28: TestMatxUtil.ThresholdByRow
28/35 Test nv-morpheus#28: TestMatxUtil.ThresholdByRow ............................... Passed 0.19 sec
Start 29: TestMultiSlices.Ranges
29/35 Test nv-morpheus#29: TestMultiSlices.Ranges .................................... Passed 1.37 sec
Start 30: TestTensor.UtilsShapeString
30/35 Test nv-morpheus#30: TestTensor.UtilsShapeString ............................... Passed 0.06 sec
Start 31: TestTensor.GetElementStride
31/35 Test nv-morpheus#31: TestTensor.GetElementStride ............................... Passed 0.06 sec
Start 32: TestTensor.AsType
32/35 Test nv-morpheus#32: TestTensor.AsType ......................................... Passed 0.16 sec
Start 33: TestTensor.Create
33/35 Test nv-morpheus#33: TestTensor.Create ......................................... Passed 0.16 sec
Start 34: TestTensor.UtilsValidateShapeAndStride
34/35 Test nv-morpheus#34: TestTensor.UtilsValidateShapeAndStride .................... Passed 0.16 sec
Start 35: TestTypeUtils.DTypeCopy
35/35 Test nv-morpheus#35: TestTypeUtils.DTypeCopy ................................... Passed 0.06 sec
100% tests passed, 0 tests failed out of 35
Total Test time (real) = 19.91 sec
```
Authors:
- Christopher Harris (https://github.com/cwharris)
Approvers:
- David Gardner (https://github.com/dagardner-nv)
URL: nv-morpheus#1095
Implement RESTDataLoader. Authors: - https://github.com/yuchenz427 - Devin Robison (https://github.com/drobison00) Approvers: - Devin Robison (https://github.com/drobison00) - Bhargav Suryadevara (https://github.com/bsuryadevara) URL: nv-morpheus#986
…morpheus#1111) This PR changes the stub generation to be committed into the repo instead of as part of the build. This will allow CI steps that need the PYI files to not require a GPU as well. This works by always copying the PYI files back to the source tree so they will show up as diffs. Any diffs during the test phase will be reported as an error. Closes nv-morpheus#766 Authors: - Michael Demoret (https://github.com/mdemoret-nv) - David Gardner (https://github.com/dagardner-nv) Approvers: - Christopher Harris (https://github.com/cwharris) - David Gardner (https://github.com/dagardner-nv) URL: nv-morpheus#1111
* Add a `--viz_direction` flag to CLI * Cherry-pick'd [9532bfe](nv-morpheus@9532bfe) from @mdemoret-nv fixes nv-morpheus#643 Authors: - David Gardner (https://github.com/dagardner-nv) - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - Christopher Harris (https://github.com/cwharris) - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1119
* Publish artifact urls in job summaries fixes nv-morpheus#1105 * Remove redundant apt installs for packages that exist in the conda env fixes nv-morpheus#1106 * Checks now performs a build first. This allows pylint to accurately warn for undefined members. fixes nv-morpheus#956 * Custom pylint checker to validate correct importing of `morpheus._lib` fixes nv-morpheus#727 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1112
…heus#1088) The `gitutils.py` script often fails when you are working in a PR, or if there has been a commit to your target branch since you created your current branch. This improves the utility by using the Github CLI to query the PR information when possible and get the true target branch for the current PR. If the Github CLI is not available, or the branch isnt in a PR, it falls back to the previous functionality. Authors: - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: nv-morpheus#1088
…morpheus#1113) * Call the `manual_seed` method from within the subprocess, this ensures the subprocess runs deterministically. * Add a sleep to a busy-loop in `morpheus/models/dfencoder/multiprocessing.py` * Misc pylint fixes fixes nv-morpheus#1021 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Christopher Harris (https://github.com/cwharris) - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1113
…hen S3 is not in use (nv-morpheus#1120) * `DFPFileToDataFrameStage` is no longer an S3 specific stage, but some of the log messages and the `unique-name` still refer to S3. Fixes nv-morpheus#451 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Christopher Harris (https://github.com/cwharris) - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1120
The test reorganization resulted in a couple of problems. First, not all tests were being run due to a mistake I made in handling cmake function arguments. Second, the `test_cuda` test was producing a segfault inside of triton client while initializing curl, but only in debug builds. The fix was to the segfault was not obvious, but apparently only occured when the test utilities were compiled as a separate library. Interestingly, the other tests were not affected. Perhaps there is some global state that needs to be managed that is not managed in the simple `test_cuda` test, which only calls a Matx function (also important, the Matx function result is not inspected, so the test itself is not comprehensive, as Matx does not produce an error if the kernel is not available for the given architecture). Depends on nv-morpheus#1110 Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1118
Update the devcontainer base to 23.10 (this has no relation to cudf versioning). While there are other improvements, the main reason I'm upgrading is due to a newer version of git, which allows inline public key signing of commits - something we need to be able to commit changes while inside the container without having to manage separate container-specific signing keys. Example of signing key usage (from host, inherited by devcontainer): ``` ssh-keygen -t ed25519 -C "[email protected]" git config --global user.signingkey "key::$(cat ~/.ssh/id_ed25519.pub)" git config --global commit.gpgsign true git config --global gpg.format ssh mkdir -p ~/.config/git git config --global gpg.ssh.allowedSignersFile ~/.config/git/allowed_signers awk 'BEGIN {FS=" "; OFS=" "} {print $3, $1, $2}' <(ssh-add -L) > ~/.config/git/allowed_signers ``` Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1116
…rpheus#1137) Closes nv-morpheus#1132 Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1137
…us#1082) There were too many linting errors to fix in 23.07 burndown so a PR has been created to fix this in 23.11. Closes nv-morpheus#1050 Authors: - Michael Demoret (https://github.com/mdemoret-nv) - David Gardner (https://github.com/dagardner-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) - Tad ZeMicheal (https://github.com/tzemicheal) URL: nv-morpheus#1082
* Separate testing utilities into (`tests/_utils`) from tests for `morpheus.utils` * This required updating the imports for most tests, which then required numerous pylint fixes * Adopt updated version of yapf to fix parsing error (tuple definition with type hints raises lib2to3.pgen2.parse.ParseError: bad input google/yapf#1058) fixes nv-morpheus#1061 Authors: - David Gardner (https://github.com/dagardner-nv) - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1065
* Remove imports from `morpheus/utils/nvt/__init__.py` * Move `annotate` hack to `decorators.py` fixes nv-morpheus#1055 Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1060
…us#1134) Adding `tasks` and `metadata` properties to python `ControlMessage`, allowing for easier debugging of control messages. Closes nv-morpheus#1131 Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1134
Prefer conda package over pip dependencies Todo: - [x] build and test inside devcontainer - [x] build and test inside development container - [ ] build and test inside release container Closes nv-morpheus#1129 Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1135
* Fixes intermittent failures in `tests/test_kafka_source_stage_pipe.py::test_kafka_source_commit` * The test checked the offsets in a stage, however the C++ impl for the source stage performs a commit after calling `on_next`. This also limited us to only testing sync commits. * Updated test performs the offset check after the pipeline completes when we can be assured that all commits have completed. Fxies nv-morpheus#1217 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1212
* Test stage was updated a while ago to perform its own builds. Closes nv-morpheus#1213 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1219
- Update autoencoder `model_kwargs` in DFP training module/stage to match recently renamed constructor parameters in autoencoder. ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Eli Fajardo (https://github.com/efajardo-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) URL: nv-morpheus#1216
+ Add more detail to Triton server setup instructions Closes nv-morpheus#1201 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Eli Fajardo (https://github.com/efajardo-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1205
+ Remove multiprocess download option + Update unit tests and docs Closes nv-morpheus#1145 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Eli Fajardo (https://github.com/efajardo-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1189
- Update labels file to reflect actual model outputs: `not_phishing` and `is_phishing` - Add `AddScores` stage to Python and CLI examples - Remove `FilterDetectionsStage` because model output contains probabilities for both `not_phishing` and `is_phishing`one of which will always exceed threshold. This results in nothing ever being filtered out. Closes nv-morpheus#1208 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Eli Fajardo (https://github.com/efajardo-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1215
A so-far untested rough draft of what it looks like to add OAuth support to the KafkaSourceStage. Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1169
## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Christopher Harris (https://github.com/cwharris) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1193
Closes nv-morpheus#1199 ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) Approvers: - Michael Demoret (https://github.com/mdemoret-nv) URL: nv-morpheus#1220
2 tasks
This PR contains the changes from running the `ci/release/update-version.sh` script for a new release Authors: - Michael Demoret (https://github.com/mdemoret-nv) Approvers: - David Gardner (https://github.com/dagardner-nv) - Tad ZeMicheal (https://github.com/tzemicheal) URL: nv-morpheus#1059
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
|
Closing this PR as rebasing to branch-23.07 has not been resolved. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes [BUG]: SQL Loader Max Connections Exhaust Error #1237 [BUG]: Fix Control Message Utils #1235