Python 3.8 incurs challenges for development workflows

[ned2]

The container is currently built to use Python 3.8 for running model code. This introduces some points for development teams.

The biggest issue my team is facing at the moment is that most packages do not publish wheels for Python 3.8 onto PyPI. This means that for local development of our model, we need to build each package from source. Besides the wait time for these builds, it can be challenging getting the build environment required to build some packages on Macbooks, notably Pandas and scikit-learn.

I understand that Conda packages are designed to address this issue, however this would involve deviating from the wider engineering's team standardisation around using pip/PyPI.

This problem would be solved by moving to Python 3.9 (or greater), which Pandas, numpy, and scikit-learn all have wheels for (including for Apple Silicon).

There is also the less blocking ,but still friction-inducing, issues associated with running a much older Python version. For example, there are language features from later Python versions that we have avoid using, not to mention the substantial performance enhancements that the last few versions of Python have seen.