[BUG]: databricks_cli import missing from main CLI in 24.03 release

[pdmack]

Version

24.03

Which installation method(s) does this occur on?

Docker

Describe the bug.

After building and running the 24.03 release image, the morpheus CLI tool fails to run a pipeline due to not finding the databricks_cli module. However, the relevant packages appear to be in the env:

# conda list | grep databricks
databricks-cli            0.217.0              ha8f183a_0    conda-forge
databricks-connect        14.3.1                   pypi_0    pypi
databricks-sdk            0.24.0                   pypi_0    pypi

Minimum reproducible example

morpheus run pipeline-ae

Relevant log output

Click here to see error details

File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/stages/preprocess/deserialize_stage.py", line 28, in 

from morpheus.modules.preprocess.deserialize import DeserializeLoaderFactory

File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/modules/init.py", line 26, in 

from morpheus.modules import mlflow_model_writer

File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/modules/mlflow_model_writer.py", line 20, in 

from morpheus.controllers.mlflow_model_writer_controller import MLFlowModelWriterController

File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/morpheus/controllers/mlflow_model_writer_controller.py", line 21, in 

import mlflow

File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/mlflow/init.py", line 44, in 

from mlflow import (

File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/mlflow/artifacts/init.py", line 11, in 

from mlflow.tracking import _get_store

File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/mlflow/tracking/init.py", line 8, in 

from mlflow.tracking._model_registry.utils import (

File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/mlflow/tracking/_model_registry/utils.py", line 15, in 

from mlflow.store._unity_catalog.registry.rest_store import UcModelRegistryStore

File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/mlflow/store/init.py", line 1, in 

from mlflow.store.artifact import artifact_repo

File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/mlflow/store/artifact/artifact_repo.py", line 14, in 

from mlflow.utils.file_utils import ArtifactProgressBar, create_tmp_dir

File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/mlflow/utils/file_utils.py", line 51, in 

from mlflow.utils.databricks_utils import _get_dbutils

File "/opt/conda/envs/morpheus/lib/python3.10/site-packages/mlflow/utils/databricks_utils.py", line 7, in 

from databricks_cli.configure import provider

ModuleNotFoundError: No module named 'databricks_cli'

Full env printout

Click here to see environment details

 # conda list | grep databricks
databricks-cli            0.217.0              ha8f183a_0    conda-forge
databricks-connect        14.3.1                   pypi_0    pypi
databricks-sdk            0.24.0                   pypi_0    pypi

Other/Misc.

No response

Code of Conduct

• I agree to follow Morpheus' Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report