Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]: DFP integrated training Azure pipeline is broken #1893

Closed
2 tasks done
yczhang-nv opened this issue Sep 17, 2024 · 0 comments · Fixed by #1894
Closed
2 tasks done

[BUG]: DFP integrated training Azure pipeline is broken #1893

yczhang-nv opened this issue Sep 17, 2024 · 0 comments · Fixed by #1894
Assignees
Labels
bug Something isn't working

Comments

@yczhang-nv
Copy link
Contributor

yczhang-nv commented Sep 17, 2024

Version

24.10

Which installation method(s) does this occur on?

Docker

Describe the bug.

DFP integrated training pipelines inexamples/digital_fingerprinting/production/ yields an error with Azure source, which indicates that user_id is not attached to ControlMessage.

Minimum reproducible example

python dfp_integrated_training_batch_pipeline.py  --tracking_uri="http://localhost:5000/" \
        --log_level DEBUG \
        --use_cpp=true \
        --source azure \
        --silence_monitors \
        --start_time "2022-08-10" \
        --duration "60d" \
        --train_users generic \
        --input_file "./control_messages/azure_payload_training.json"

Relevant log output

Click here to see error details
Exception occurred in pipeline. Rethrowing
Traceback (most recent call last):
  File "/home/dagardner/work/m3/python/morpheus/morpheus/pipeline/pipeline.py", line 405, in post_start
    await executor.join_async()
  File "/home/dagardner/work/m3/python/morpheus/morpheus/controllers/mlflow_model_writer_controller.py", line 226, in on_data
    reg_model_name = self.user_id_to_model(user_id=user)
  File "/home/dagardner/work/m3/python/morpheus/morpheus/controllers/mlflow_model_writer_controller.py", line 126, in user_id_to_model
    "user_id": self._create_safe_user_id(user_id),
  File "/home/dagardner/work/m3/python/morpheus/morpheus/controllers/mlflow_model_writer_controller.py", line 104, in _create_safe_user_id
    safe_user_id = user_id.replace('.', '_dot_')
AttributeError: 'NoneType' object has no attribute 'replace'

Full env printout

Click here to see environment details

[Paste the results of print_env.sh here, it will be hidden by default]

Other/Misc.

No response

Code of Conduct

  • I agree to follow Morpheus' Code of Conduct
  • I have searched the open bugs and have found no duplicates for this bug report
@yczhang-nv yczhang-nv added the bug Something isn't working label Sep 17, 2024
@yczhang-nv yczhang-nv self-assigned this Sep 17, 2024
@morpheus-bot-test morpheus-bot-test bot moved this from Todo to Review - Ready for Review in Morpheus Boards Sep 17, 2024
@yczhang-nv yczhang-nv changed the title [BUG]: DFP integrated training pipelines are broken [BUG]: DFP integrated training Azure pipeline is broken Sep 17, 2024
@mdemoret-nv mdemoret-nv added this to the 24.10 - Release milestone Sep 18, 2024
@rapids-bot rapids-bot bot closed this as completed in 2c1bfdc Sep 18, 2024
@github-project-automation github-project-automation bot moved this from Review - Ready for Review to Done in Morpheus Boards Sep 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants