[BUG]: Production DFP inference error when using local file path for MLflow tracking_uri

[efajardo-nv]

Version

23.11

Which installation method(s) does this occur on?

Conda

Describe the bug.

• My data resides in local Parquet files.
• I'm using a JSON control message as input and have mlflow_tracking_uri set to a local path.
• After adjusting input parameters and configuring the Schema Builder, I receive a TypeError: object of type 'NoneType' has no len() error in model_cache.py during pipeline building phase while fetching existing models.

Minimum reproducible example

python dfp_duo_pipeline.py --log_level DEBUG --train_users none --start_time "2022-08-30" --input_file="../../../data/dfp/duo-inference-data/*.json" --tracking_uri=file:/workspace/mlflow

Relevant log output

Click here to see error details

Running training pipeline with the following options:

Train generic_user: False

Skipping users: []

Start Time: 2022-08-30 00:00:00+00:00

Duration: 60 days, 0:00:00

Cache Dir: ./.cache/dfp

Tracking URI: file:/workspace/mlflow

Updating list of available models...

Exception occurred when querying the list of available models

Traceback (most recent call last):

File "/workspace/examples/digital_fingerprinting/production/morpheus/dfp/utils/model_cache.py", line 228, in _model_exists

if (len(results.token) == 0):

TypeError: object of type 'NoneType' has no len()

Traceback (most recent call last):

File "/workspace/examples/digital_fingerprinting/production/morpheus/dfp_duo_pipeline.py", line 380, in 

run_pipeline(obj={}, auto_envvar_prefix='DFP', show_default=True, prog_name="dfp")

File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/click/core.py", line 1157, in call

return self.main(*args, **kwargs)

File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/click/core.py", line 1078, in main

rv = self.invoke(ctx)

File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/click/core.py", line 1434, in invoke

return ctx.invoke(self.callback, **ctx.params)

File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/click/core.py", line 783, in invoke

return __callback(*args, **kwargs)

File "/workspace/examples/digital_fingerprinting/production/morpheus/dfp_duo_pipeline.py", line 358, in run_pipeline

pipeline.add_stage(DFPInferenceStage(config, model_name_formatter=model_name_formatter))

File "/workspace/morpheus/pipeline/stage_base.py", line 45, in inner

func(self, *args, **kwargs)

File "/workspace/examples/digital_fingerprinting/production/morpheus/dfp/stages/dfp_inference_stage.py", line 60, in init

self._model_manager = ModelManager(model_name_formatter=model_name_formatter)

File "/workspace/examples/digital_fingerprinting/production/morpheus/dfp/utils/model_cache.py", line 198, in init

self._model_exists("")

File "/workspace/examples/digital_fingerprinting/production/morpheus/dfp/utils/model_cache.py", line 228, in _model_exists

if (len(results.token) == 0):

TypeError: object of type 'NoneType' has no len()

Full env printout

Click here to see environment details

[Paste the results of print_env.sh here, it will be hidden by default]

Other/Misc.

No response

Code of Conduct

• I agree to follow Morpheus' Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report