More extensive release validation

[laneb]

Up until the release of 2r0, we have validated new release artifacts by running the tests in a docker container against the single targeted Spark patch version with a single Python version - I think 3.6 most recently, since that appears to be what apt-get installs by default for Python 3. The image for the container is based on the Dockerfile from the release branch.

We should validate the artifact against all supported Spark minor versions and Python versions. The new make target test-thorough should satisfy this, but if we continue to run the tests in Docker, we'll need to build an image with the dev dependencies. Particularly, the official SMV image won't include all of the supported Python versions. We could install these and any future dependencies in the official SMV images, but it would come at the cost of a larger image. This image for testing would not need to be hosted.

Alternatively, we could remove Docker from the mix. I believe we started using Docker to build and test the release to ensure a standardized environment (particularly Java version). If we shift the release responsibilities into the CI, there is less need for Docker since we can standardize the environment in the CI. However, continuing to utilize Docker will ease any future transition between CI platforms.

If we do create a new image, we have the option the utilize it to install dependencies for standard dev testing. However, I'd be hesitant to do this, because I currently see degraded performance when smv-release runs the tests in Docker.

[laneb]

I’ve been trying to figure out how to get Python 3.6 installed in the SMV docker image. So far we’ve been using apt-get to install Python, but 3.6 isn’t available in the Debian stable repositories. The two approaches I've tried are

pyenv

pyenv doesn’t seem to offer a way to create a global installation. By default .pyenv installs pythons in /root, so the smv user has no permissions to the bins. We can override that behavior by exporting PYENV_ROOT to point to a directory that smv has execute permissions on. Before discovering this I tried copying /root/.pyenv into /home/smv after installing 3.6.5, but the pip bin hard codes the absolute path to the python bin:

laneb@lanes-MBP:~/twine/SMV
@> pyenv which pip
/Users/laneb/.pyenv/versions/3.6.5/bin/pip

laneb@lanes-MBP:~/twine/SMV
@> head -1 $(pyenv which pip)
#!/Users/laneb/.pyenv/versions/3.6.5/bin/python3.6

so relocating .pyenv after the fact is not an option.

Install Python from testing repository

In between unstable and stable, debian packages are marked as testing. Python3.6 is currently in Testing. By adding the testing repository to /etc/apt/sources.list and adding and removing a few packages we can get apt-get to install python3.6 and install pip with get-pip.py. I haven't grokked why I needed to remove binutils or what the consequences are. Also, we don't get any control over the patch version of Python.

Between these options I think the pyenv approach is preferrable and better reflects how we would install python in practice. I suppose we could also Python for source - don't know pros/cons there.

[jacobdr]

Have you seen this privately maintained PPA? Not quite official, but we can audit it as it’s OSS. https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa JR

…

On Jul 8, 2018, 22:42 -0700, Lane Barlow ***@***.***>, wrote: I’ve been trying to figure out how to get Python 3.6 installed in the SMV docker image. So far we’ve been using apt-get to install Python, but 3.6 isn’t available in the Debian stable repositories. The two approaches I've tried are pyenv pyenv doesn’t seem to offer a way to create a global installation. By default .pyenv installs pythons in /root, so the smv user has no permissions to the bins. We can override that behavior by exporting PYENV_ROOT to point to a directory that smv has execute permissions on. Before discovering this I tried copying /root/.pyenv into /home/smv after installing 3.6.5, but the pip bin hard codes the absolute path to the python bin: ***@***.***:~/twine/SMV @> pyenv which pip /Users/laneb/.pyenv/versions/3.6.5/bin/pip ***@***.***:~/twine/SMV @> head -1 $(pyenv which pip) #!/Users/laneb/.pyenv/versions/3.6.5/bin/python3.6 so relocating .pyenv after the fact is not an option. Install Python from testing repository In between unstable and stable, debian packages are marked as testing. Python3.6 is currently in Testing. By adding the testing repository to /etc/apt/sources.list and adding and removing a few packages we can get apt-get to install python3.6 and install pip with get-pip.py. I haven't grokked why I needed to remove binutils or what the consequences are. Also, we don't get any control over the patch version of Python. Between these options I think the pyenv approach is preferrable and better reflects how we would install python in practice. I suppose we could also Python for source - don't know pros/cons there. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[laneb]

I avoided ubuntu PPAs because Debian generally advises against it: https://wiki.debian.org/DontBreakDebian#Don.27t_make_a_FrankenDebian

The justification is that things might break in ways that aren't obvious. Debian devs want you to use the distros that were specifically designated as stable for your Debian release. I'm not worried about breaking package management ("The problems might not happen right away, but the next time you install updates.") since its just a docker image, but I am concerned that we could introduce security vulnerabilities. However, the same doc also advises against mixing stable and testing packages.

[AliTajeldin]

@laneb Let's have a discussion today about best approach for multi python version testing.

[laneb]

Based on discussion with @AliTajeldin it shouldn't be necessary to install Python 3.6. 3.5 should be sufficient.