UATs for kubeflow 1.8 fail to run with ERROR: Cannot install -r requirements.txt

[NohaIhab]

Bug Description

the UATs from branch track/1.8 are failing to run due a conflict in python dependencies between websockets and juju.

Example failed runs:
AKS
EKS

To Reproduce

run the UATs from branch track/1.8

Environment

juju 3.4/stable
k8s 1.29

Relevant Log Output

kubeflow-remote: install_deps> python -I -m pip install -r requirements.txt
ERROR: Cannot install -r requirements.txt (line 66) and websockets==8.1 because these package versions have conflicting dependencies.
Collecting anyio==4.0.0 (from -r requirements.txt (line 7))
  Downloading anyio-4.0.0-py3-none-any.whl.metadata (4.5 kB)
Collecting asttokens==2.4.1 (from -r requirements.txt (line 9))
  Downloading asttokens-2.4.1-py2.py3-none-any.whl.metadata (5.2 kB)
Collecting backcall==0.2.0 (from -r requirements.txt (line 11))
  Downloading backcall-0.2.0-py2.py3-none-any.whl.metadata (2.0 kB)
Collecting bcrypt==4.0.1 (from -r requirements.txt (line 13))
  Downloading bcrypt-4.0.1-cp36-abi3-manylinux_2_28_x86_64.whl.metadata (9.0 kB)
Collecting cachetools==5.3.2 (from -r requirements.txt (line 15))
  Downloading cachetools-5.3.2-py3-none-any.whl.metadata (5.2 kB)
Collecting certifi==2023.7.22 (from -r requirements.txt (line 17))
  Downloading certifi-2023.7.22-py3-none-any.whl.metadata (2.2 kB)
Collecting cffi==1.16.0 (from -r requirements.txt (line 23))
  Downloading cffi-1.16.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.5 kB)
Collecting charset-normalizer==3.3.2 (from -r requirements.txt (line 27))
  Downloading charset_normalizer-3.3.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (33 kB)
Collecting cryptography==41.0.5 (from -r requirements.txt (line 29))
  Downloading cryptography-41.0.5-cp37-abi3-manylinux_2_28_x86_64.whl.metadata (5.2 kB)
Collecting decorator==5.1.1 (from -r requirements.txt (line 31))
  Using cached decorator-5.1.1-py3-none-any.whl.metadata (4.0 kB)
Collecting exceptiongroup==1.1.3 (from -r requirements.txt (line 35))
  Downloading exceptiongroup-1.1.3-py3-none-any.whl.metadata (6.1 kB)
Collecting executing==2.0.1 (from -r requirements.txt (line 39))
  Downloading executing-2.0.1-py2.py3-none-any.whl.metadata (9.0 kB)
Collecting google-auth==2.23.4 (from -r requirements.txt (line 41))
  Downloading google_auth-2.23.4-py2.py3-none-any.whl.metadata (4.7 kB)
Collecting h11==0.14.0 (from -r requirements.txt (line 43))
  Using cached h11-0.14.0-py3-none-any.whl.metadata (8.2 kB)
Collecting httpcore==1.0.2 (from -r requirements.txt (line 45))
  Downloading httpcore-1.0.2-py3-none-any.whl.metadata (20 kB)
Collecting httpx==0.25.1 (from -r requirements.txt (line 47))
  Downloading httpx-0.25.1-py3-none-any.whl.metadata (7.1 kB)
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
Collecting hvac==2.0.0 (from -r requirements.txt (line 49))
  Downloading hvac-2.0.0-py3-none-any.whl.metadata (3.3 kB)
Collecting idna==3.4 (from -r requirements.txt (line 51))
  Downloading idna-3.4-py3-none-any.whl.metadata (9.8 kB)
Collecting iniconfig==2.0.0 (from -r requirements.txt (line 56))
  Using cached iniconfig-2.0.0-py3-none-any.whl.metadata (2.6 kB)
Collecting ipdb==0.13.13 (from -r requirements.txt (line 58))
  Using cached ipdb-0.13.13-py3-none-any.whl.metadata (14 kB)
Collecting ipython==8.12.3 (from -r requirements.txt (line 60))
  Downloading ipython-8.12.3-py3-none-any.whl.metadata (5.7 kB)
Collecting jedi==0.19.1 (from -r requirements.txt (line 62))
  Downloading jedi-0.19.1-py2.py3-none-any.whl.metadata (22 kB)
Collecting jinja2==3.1.2 (from -r requirements.txt (line 64))
  Downloading Jinja2-3.1.2-py3-none-any.whl.metadata (3.5 kB)
Collecting juju==3.2.3.0 (from -r requirements.txt (line 66))
  Downloading juju-3.2.3.0.tar.gz (582 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 582.0/582.0 kB 15.5 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting kubernetes==28.1.0 (from -r requirements.txt (line 70))
  Downloading kubernetes-28.1.0-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting lightkube==0.15.0 (from -r requirements.txt (line 72))
  Downloading lightkube-0.15.0-py3-none-any.whl.metadata (5.3 kB)
Collecting lightkube-models==1.28.1.4 (from -r requirements.txt (line 74))
  Downloading lightkube_models-1.28.1.4-py3-none-any.whl.metadata (1.1 kB)
Collecting macaroonbakery==1.3.1 (from -r requirements.txt (line 76))
  Downloading macaroonbakery-1.3.1.tar.gz (79 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting markupsafe==2.1.3 (from -r requirements.txt (line 78))
  Downloading MarkupSafe-2.1.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.9 kB)
Collecting matplotlib-inline==0.1.6 (from -r requirements.txt (line 80))
  Downloading matplotlib_inline-0.1.6-py3-none-any.whl.metadata (2.8 kB)
Collecting mypy-extensions==1.0.0 (from -r requirements.txt (line 82))
  Using cached mypy_extensions-1.0.0-py3-none-any.whl.metadata (1.1 kB)
Collecting oauthlib==3.2.2 (from -r requirements.txt (line 84))
  Using cached oauthlib-3.2.2-py3-none-any.whl.metadata (7.5 kB)
Collecting packaging==23.2 (from -r requirements.txt (line 88))
  Downloading packaging-23.2-py3-none-any.whl.metadata (3.2 kB)
Collecting paramiko==2.12.0 (from -r requirements.txt (line 90))
  Downloading paramiko-2.12.0-py2.py3-none-any.whl.metadata (4.6 kB)
Collecting parso==0.8.3 (from -r requirements.txt (line 92))
  Downloading parso-0.8.3-py2.py3-none-any.whl.metadata (7.5 kB)
Collecting pexpect==4.8.0 (from -r requirements.txt (line 94))
  Downloading pexpect-4.8.0-py2.py3-none-any.whl.metadata (2.2 kB)
Collecting pickleshare==0.7.5 (from -r requirements.txt (line 96))
  Downloading pickleshare-0.7.5-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting pluggy==1.3.0 (from -r requirements.txt (line 98))
  Downloading pluggy-1.3.0-py3-none-any.whl.metadata (4.3 kB)
Collecting prompt-toolkit==3.0.41 (from -r requirements.txt (line 100))
  Downloading prompt_toolkit-3.0.41-py3-none-any.whl.metadata (6.5 kB)
Collecting protobuf==3.20.3 (from -r requirements.txt (line 102))
  Downloading protobuf-3.20.3-py2.py3-none-any.whl.metadata (720 bytes)
Collecting ptyprocess==0.7.0 (from -r requirements.txt (line 104))
  Using cached ptyprocess-0.7.0-py2.py3-none-any.whl.metadata (1.3 kB)
Collecting pure-eval==0.2.2 (from -r requirements.txt (line 106))
  Downloading pure_eval-0.2.2-py3-none-any.whl.metadata (6.2 kB)
Collecting pyasn1==0.5.0 (from -r requirements.txt (line 108))
  Downloading pyasn1-0.5.0-py2.py3-none-any.whl.metadata (8.5 kB)
Collecting pyasn1-modules==0.3.0 (from -r requirements.txt (line 113))
  Downloading pyasn1_modules-0.3.0-py2.py3-none-any.whl.metadata (3.6 kB)
Collecting pycparser==2.21 (from -r requirements.txt (line 115))
  Downloading pycparser-2.21-py2.py3-none-any.whl.metadata (1.1 kB)
Collecting pygments==2.16.1 (from -r requirements.txt (line 117))
  Downloading Pygments-2.16.1-py3-none-any.whl.metadata (2.5 kB)
Collecting pymacaroons==0.13.0 (from -r requirements.txt (line 119))
  Using cached pymacaroons-0.13.0-py2.py3-none-any.whl.metadata (1.6 kB)
Collecting pynacl==1.5.0 (from -r requirements.txt (line 121))
  Using cached PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl.metadata (8.6 kB)
Collecting pyrfc3339==1.1 (from -r requirements.txt (line 126))
  Using cached pyRFC3339-1.1-py2.py3-none-any.whl.metadata (2.0 kB)
Collecting pytest==7.4.3 (from -r requirements.txt (line 130))
  Downloading pytest-7.4.3-py3-none-any.whl.metadata (7.9 kB)
Collecting pytest-asyncio==0.21.1 (from -r requirements.txt (line 135))
  Downloading pytest_asyncio-0.21.1-py3-none-any.whl.metadata (4.0 kB)
Collecting pytest-operator==0.31.0 (from -r requirements.txt (line 137))
  Downloading pytest_operator-0.31.0-py3-none-any.whl.metadata (3.5 kB)
Collecting python-dateutil==2.8.2 (from -r requirements.txt (line 139))
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl.metadata (8.2 kB)
Collecting pytz==2023.3.post1 (from -r requirements.txt (line 141))
  Downloading pytz-2023.3.post1-py2.py3-none-any.whl.metadata (22 kB)
Collecting pyyaml==6.0.1 (from -r requirements.txt (line 143))
  Downloading PyYAML-6.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
Collecting requests==2.31.0 (from -r requirements.txt (line 149))
  Downloading requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting requests-oauthlib==1.3.1 (from -r requirements.txt (line 155))
  Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl.metadata (10 kB)
Collecting rsa==4.9 (from -r requirements.txt (line 157))
  Using cached rsa-4.9-py3-none-any.whl.metadata (4.2 kB)
Collecting six==1.16.0 (from -r requirements.txt (line 159))
  Downloading six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
Collecting sniffio==1.3.0 (from -r requirements.txt (line 167))
  Downloading sniffio-1.3.0-py3-none-any.whl.metadata (3.6 kB)
Collecting stack-data==0.6.3 (from -r requirements.txt (line 171))
  Using cached stack_data-0.6.3-py3-none-any.whl.metadata (18 kB)
Collecting tenacity==8.2.3 (from -r requirements.txt (line 173))
  Downloading tenacity-8.2.3-py3-none-any.whl.metadata (1.0 kB)
Collecting tomli==2.0.1 (from -r requirements.txt (line 175))
  Downloading tomli-2.0.1-py3-none-any.whl.metadata (8.9 kB)
Collecting toposort==1.10 (from -r requirements.txt (line 179))
  Using cached toposort-1.10-py3-none-any.whl.metadata (4.1 kB)
Collecting traitlets==5.13.0 (from -r requirements.txt (line 181))
  Downloading traitlets-5.13.0-py3-none-any.whl.metadata (10 kB)
Collecting typing-extensions==4.8.0 (from -r requirements.txt (line 185))
  Downloading typing_extensions-4.8.0-py3-none-any.whl.metadata (3.0 kB)
Collecting typing-inspect==0.9.0 (from -r requirements.txt (line 189))
  Using cached typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB)
Collecting urllib3==1.26.18 (from -r requirements.txt (line 191))
  Downloading urllib3-1.26.18-py2.py3-none-any.whl.metadata (48 kB)
Collecting wcwidth==0.2.10 (from -r requirements.txt (line 195))
  Downloading wcwidth-0.2.10-py2.py3-none-any.whl.metadata (14 kB)
Collecting websocket-client==1.6.4 (from -r requirements.txt (line 197))
  Downloading websocket_client-1.6.4-py3-none-any.whl.metadata (7.7 kB)
Collecting websockets==8.1 (from -r requirements.txt (line 199))
  Downloading websockets-8.1.tar.gz (58 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
INFO: pip is looking at multiple versions of juju to determine which version is compatible with other requirements. This could take a while.

The conflict is caused by:
    The user requested websockets==8.1
    juju 3.2.3.0 depends on websockets>=10.0; python_version > "3.9"

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict

kubeflow-remote: exit 1 (9.69 seconds) /home/runner/charmed-kubeflow-uats> python -I -m pip install -r requirements.txt pid=7583
  kubeflow-remote: FAIL code 1 (9.78 seconds)
  evaluation failed :( (9.83 seconds)

Error: Process completed with exit code 1.

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6704.

This message was autogenerated

[deusebio]

I believe the issue here is that some UATs (e.g. for 1.8) are not actually using main, but for instance the track/1.8 that are not up to date with the poetry modification. track/1.8 is still based on the old dependencies (without using poetry) but they are now running in the bundle tests using 24.04 bases, with python 3.12, hence the mismatch and the error.

I suppose we could either:

1. Run UAT tests for 1.8 against the main branch as well
2. Backport poetry and dependency update to track/1.8

Of the two, 1 would be easier (also therefore removing the `track/1.8, limiting the maintainability burned), but I don't know if we can run the tests in main against the 1.8 deployment.

[misohu]

We need to backport changes from here #137