Bug: databricksruntime/python - the default python and pip python versions do not match

[marcindulak]

The version of python used by pip does not correpond to the default python version on the system.
This results in an unexpected behavior, when users install packages with pip3, to see that these packages are not available for import with python3.

For example

docker run --rm -it --name python databricksruntime/python:11.3-LTS bash -c "which pip && pip --version && which pip3 && pip3 --version"
/usr/local/bin/pip
pip 21.2.4 from /usr/local/lib/python3.9/dist-packages/pip (python 3.9)
/usr/local/bin/pip3
pip 21.2.4 from /usr/local/lib/python3.9/dist-packages/pip (python 3.9)

vs

docker run --rm -it --name python databricksruntime/python:11.3-LTS bash -c "which python3 && python3 --version"
/usr/bin/python3
Python 3.8.10

or

docker run --rm -it --name python databricksruntime/python:12.2-LTS bash -c "which pip && pip --version && which pip3 && pip3 --version"
/usr/local/bin/pip
pip 21.2.4 from /usr/local/lib/python3.9/dist-packages/pip (python 3.9)
/usr/local/bin/pip3
pip 21.2.4 from /usr/local/lib/python3.9/dist-packages/pip (python 3.9)

vs

docker run --rm -it --name python databricksruntime/python:12.2-LTS bash -c "which python3 && python3 --version"
/usr/bin/python3
Python 3.8.10

I see databricksruntime/python:13.3-LTS does not have this problem, but I've not verified whether other types of images containing pip also have this issue.

[marcindulak]

As an update, after reading https://learn.microsoft.com/en-us/azure/databricks/clusters/custom-containers, I see that instead the python and pip under /databricks/python3/bin should be used

docker run --rm -it --name python databricksruntime/python:12.2-LTS bash -c "/databricks/python3/bin/python --version && /databricks/python3/bin/pip --version"
Python 3.9.5
pip 21.2.4 from /databricks/python3/lib/python3.9/site-packages/pip (python 3.9)