Fix Dockerfile

[fcami]

• switch to CentOS Streams 8 for more package availability
• install s3cmd to upload test results to s3 easily
• switch to using python3-virtualenv to force Python 3.8 in venv, otherwise Python 3.6 is used. This commit uses Python 3.7, leading pip to try to install backports-datetime-fromisoformat which fails because the image has no gcc.

    gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/tmp/ods-ci/venv/include -I/usr/include/python3.6m -c backports/datetime_fromisoformat/module.c -o build/temp.linux-x86_64-3.6/backports/datetime_fromisoformat/module.o
    unable to execute 'gcc': No such file or directory
    error: command 'gcc' failed with exit status 1

[tarukumar]

Able to create container successfully on local system http://pastebin.test.redhat.com/1057656

[tarukumar]

@fcami Request you to send PR with Signature

[pablofelix]

virtualenv is used for dependency isolation. You want to prevent any dependencies or packages installed from leaking between applications. Podman/Docker achieves the same thing, it isolates your dependencies within your container and prevent leaks between containers and between applications. There are some cases where virtualenv could be used in containers mainly in multistage. Is it necessary to add the virtualenv?

[fcami]

Yes. Without that change, "venv" was used. But venv cannot create a virtualenv with a different version of Python. ods-ci uses Python 3.7+ Python features but ubi8/streams8 has 3.6 by default.

[pablofelix]

I might be missing something here, but I still don't see the need of using virtualenv, (or venv before the change), because when you use a container you can install the Python version of your choice directly using the package manager, and will still be isolated. Could you please elaborate this a bit more?

[pablofelix]

If we don't use the virtualenv we also can delete this line

[fcami]

We use it. I thought that was obvious.

[pablofelix]

Yes, I see that you use it, that's not what I meant with my comment, let me clarify it. I was just suggesting to not use it to avoid unnecessary extra layers and complexity in the docker image. Is there any chance to avoid it? That's why I asked you in my other comment if there's a reason to use it that I'm missing.

[fcami]

I don't see how.
At this point I'm just trying to make the current Dockerfile work.
My solution does the job, but I'm open to suggestions.

[pablofelix]

Yes, I'm not blocking the merge because of this, but I would like to look at it and see if the Dockerfile can work without the virtualenv. If it does, the changes can be added in a future PR.

[fcami]

I'd rather not remove the venv altogether. It doesn't hurt at all and it separates the system python packages from what's used by ods-ci. It's not exactly isolation, but it's much cleaner.

[bdattoma]

hey @fcami thanks for raising the PRs, we always appreciate that :D

I will go through all the proposed changes as soon as possible. In the meantime, some comments:

switch to CentOS Streams 8 for more package availability

I think it's better to stick with ubi images when possible, like requested to partners deliverying containers.

install s3cmd to upload test results to s3 easily

where this package will be used? We don't, so are you going to raise a PR where this package is needed?

switch to using python3-virtualenv to force Python 3.8 in venv, otherwise Python 3.6 is used. This commit uses Python 3.7, leading pip to try to install backports-datetime-fromisoformat which fails because the image has no gcc.

hm sounds strange to me. I think once python3.8 is installed, it is used as default to create the venv. We have dependencies where python 3.8 is required (e.g., tests/Resources/RHOSi.resource ), and it is working fine. Besides, while the container is being built, I see the line RUN python3 --version printing out 3.8.12. Besides, ubi minimal does not have python pre-installed

However, I will do some investigation.

[fcami]

hey @fcami thanks for raising the PRs, we always appreciate that :D

I will go through all the proposed changes as soon as possible. In the meantime, some comments:

switch to CentOS Streams 8 for more package availability

I think it's better to stick with ubi images when possible, like requested to partners deliverying containers.

Except this requires using an entitlement to get libcanberra-gtk3 (comes from CentOS or RHEL) which is required by chromium (EPEL). Do we have any partner building on ods-ci?
Using streams will make this much more portable and reusable.

install s3cmd to upload test results to s3 easily

where this package will be used? We don't, so are you going to raise a PR where this package is needed?

We (PSAP) already use it once ods-ci has run to upload the logs. The package has to be in the image.

switch to using python3-virtualenv to force Python 3.8 in venv, otherwise Python 3.6 is used. This commit uses Python 3.7, leading pip to try to install backports-datetime-fromisoformat which fails because the image has no gcc.

hm sounds strange to me. I think once python3.8 is installed, it is used as default to create the venv.

We have dependencies where python 3.8 is required (e.g., tests/Resources/RHOSi.resource ), and it is working fine. Besides, while the container is being built, I see the line RUN python3 --version printing out 3.8.12. Besides, ubi minimal does not have python pre-installed

That's not what I saw. I'd be curious to see how you can get 3.8.12 by default.

However, I will do some investigation.

Please see this log. It contains:

  Verifying        : python38-3.8.12-1.module_el8.6.0+929+89303463.x8    84/206 
(...)
STEP 26/33: RUN python3 --version
Python 3.6.8
(...)
STEP 27/33: RUN python3 -m venv venv && source venv/bin/activate &&  pip3 install --upgrade pip && venv/bin/pip3 install -r requirements.txt
...
    Running setup.py install for backports-datetime-fromisoformat: started
    Running setup.py install for backports-datetime-fromisoformat: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /tmp/ods-ci/venv/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-i2hkkrq3/backports-datetime-fromisoformat_2b14928b22ff40c49ef06932fe9b3e0e/setup.py'"'"'; __file__='"'"'/tmp/pip-install-i2hkkrq3/backports-datetime-fromisoformat_2b14928b22ff40c49ef06932fe9b3e0e/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-15zbual9/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/ods-ci/venv/include/site/python3.6/backports-datetime-fromisoformat
         cwd: /tmp/pip-install-i2hkkrq3/backports-datetime-fromisoformat_2b14928b22ff40c49ef06932fe9b3e0e/
    Complete output (17 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/backports
    copying backports/__init__.py -> build/lib.linux-x86_64-3.6/backports
    creating build/lib.linux-x86_64-3.6/backports/datetime_fromisoformat
    copying backports/datetime_fromisoformat/__init__.py -> build/lib.linux-x86_64-3.6/backports/datetime_fromisoformat
    running build_ext
    building 'backports._datetime_fromisoformat' extension
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/backports
    creating build/temp.linux-x86_64-3.6/backports/datetime_fromisoformat
    gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/tmp/ods-ci/venv/include -I/usr/include/python3.6m -c backports/datetime_fromisoformat/module.c -o build/temp.linux-x86_64-3.6/backports/datetime_fromisoformat/module.o
    unable to execute 'gcc': No such file or directory

gcc is needed because backports-datetime-fromisoformat is selected by pip ; in turn, backports-datetime-fromisoformat is selected by pip because datetime-fromisoformat is a Python 3.7+ feature and compiling the backport is necessary since this is running on 3.6.

Let me know if that helps.

[fcami]

@fcami Request you to send PR with Signature

I've added relevant signed-off-by mentions on all commits. Please let me know if you require anything else.

[bdattoma]

Except this requires using an entitlement to get libcanberra-gtk3 (comes from CentOS or RHEL) which is required by chromium (EPEL)

Are you hitting any error because of ubi image?

Do we have any partner building on ods-ci?

no we don't. I was referring to this policy

We (PSAP) already use it once ods-ci has run to upload the logs. The package has to be in the image.

Before adding this pkg I'll need to understand better this point. I'll ping you offline in the next days

Please see this log. It contains:

The directory at that link is empty. Could you share the build logs of ods-ci container please? We have tried reproducing the python version problem without success. When built on our local systems (tried by 3 of us), we got python3.8 installed as expected. Maybe more context on how/where the ods-ci image is build might help. I'll ping you offline in the next days to see together

[fcami]

Except this requires using an entitlement to get libcanberra-gtk3 (comes from CentOS or RHEL) which is required by chromium (EPEL)

Are you hitting any error because of ubi image?

Yes, libcanberra-gtk3.

Do we have any partner building on ods-ci?

no we don't. I was referring to this policy

My question becomes: do we want to /certify/ any ods-ci image?
Using Streams (which is upstream from RHEL) makes for a better community project while using UBI is the path for certification.

We (PSAP) already use it once ods-ci has run to upload the logs. The package has to be in the image.

Before adding this pkg I'll need to understand better this point. I'll ping you offline in the next days

OK.

Please see this log. It contains:

The directory at that link is empty. Could you share the build logs of ods-ci container please? We have tried reproducing the python version problem without success. When built on our local systems (tried by 3 of us), we got python3.8 installed as expected. Maybe more context on how/where the ods-ci image is build might help. I'll ping you offline in the next days to see together

OK, I guess it was auto-cleaned-up after a while.

[kpouget]

Hello @bdattoma, I'm working with @fcami in the Perf&Scale PSAP team,
let me follow up on some aspects of the discussions

The context is that we're working on an automated testing of RHODS/JupyterHub at large scale (hundreds of users), and ods-ci is a critical part of that process, as it will simulate the users connecting to RHODS dashboard -> JupyterHub -> launch a notebook -> run a simple workload in the notebook.

In this workload, ods-ci runs inside an OpenShift container (1 Job creating N ods-ci Pods). The container image is built dynamically in the cluster, in the first steps of the test, from a given ODS-CI repo/branch. See there for the BuildConfig we're currently using (pointing to a old branch forked from this repo, and customized by François and Erwan Granger).

In this context, the requirement for an entitled UBI makes it complicated to automate the build of the image in any CI, be it Github Actions, OpenShift CI (where we're working) or Quay.io.

---

Now regarding the requirement for the s3cmd binary: we use it in a wrapper of the run_robot_test.sh script: as we run ods-ci in a Pod, we need a way to export the artifacts that have been generated, out of the Pod. We found that launching Minio S3 backend in the cluster, and pushing the artifacts to this bucket was the easiest and most portable solution (a ReadWriteMany PV could have done the job, but it's more complex to setup if we want to run the testsuite in a customer's cluster)

[bdattoma]

@kpouget thank you for the context.

as we run ods-ci in a Pod, we need a way to export the artifacts that have been generated, out of the Pod

Ok we can add the package you need. Just out of my curiosity, couldn't you create a second container in the same pod and put it in charge of sharing the results? in that container you could install any packages you need to share files and change the logic independently from the main container.

Ok for the image too

[kpouget]

@kpouget thank you for the context.

as we run ods-ci in a Pod, we need a way to export the artifacts that have been generated, out of the Pod

Ok we can add the package you need. Just out of my curiosity, couldn't you create a second container in the same pod and put it in charge of sharing the results? in that container you could install any packages you need to share files and change the logic independently from the main container.

good point, I think it would make more sense this way

do you have any example of Pod using such a side-car container? I wonder what's the best way to handle the synchronization (ie, in the side-car, wait for the termination of the main container before doing anything)

as part of another work, I did it with a simple file, something like

while [[ ! -f "$SHARED_DIR/main_container_done" ]]; do
  sleep 10
done

but I wonder if there can be any cleaner way to do it ... ?

[fcami]

I'll remove the s3cmd commit.

Update: commit removed, branch re-pushed.

[bdattoma]

@kpouget thank you for the context.

as we run ods-ci in a Pod, we need a way to export the artifacts that have been generated, out of the Pod

Ok we can add the package you need. Just out of my curiosity, couldn't you create a second container in the same pod and put it in charge of sharing the results? in that container you could install any packages you need to share files and change the logic independently from the main container.

good point, I think it would make more sense this way

do you have any example of Pod using such a side-car container? I wonder what's the best way to handle the synchronization (ie, in the side-car, wait for the termination of the main container before doing anything)

as part of another work, I did it with a simple file, something like

while [[ ! -f "$SHARED_DIR/main_container_done" ]]; do
  sleep 10
done

but I wonder if there can be any cleaner way to do it ... ?

I think there's not an out of the box functionality to wait until the other container is finished. Your solution should work but I suggest you to add a timeout for safety. With a similar logic, you could check for the ods-ci container status in the pod, and once the status is "terminated" (and the output files are generated) then you publish the results. Unfortunately I don't have an example :(

[fcami]

After more testing:

• untentitled UBI does not work as there are missing packages.
• switching from entitled UBI or CentOS Streams does not make any difference: in both cases everything works.
• adding any kind of rpm-packaged Python application like s3cmd pulls the default system Python interpreter (3.6), which in turn breaks the build with venv, because it uses the system Python (3.6) and ods-ci is now 3.7+
• switching from venv to virtualenv fixes this.

I'll remove the virtualenv commit and open a separate PR with it that you may choose to close/wontfix ; in my mind I'd use virtualenv because ods-ci requires a different interpreter than the default, making the build vulnerable.

[fcami]

I've removed the virtualenv commit and updated both the issue #406 and created a separate PR with it: #418

All that remains here is switching to CentOS Streams to remove the dependency on an entitled UBI.

[kpouget]

After more testing:

* untentitled UBI does not work as there are missing packages.

* switching from entitled UBI or CentOS Streams does not make any difference: in both cases everything works.

* adding any kind of rpm-packaged Python application like s3cmd pulls the default system Python interpreter (3.6), which in turn breaks the build with venv, because it uses the system Python (3.6) and ods-ci is now 3.7+

* switching from venv to virtualenv fixes this.

I'll remove the virtualenv commit and open a separate PR with it that you may choose to close/wontfix ; in my mind I'd use virtualenv because ods-ci requires a different interpreter than the default, making the build vulnerable.

thanks for the explanations François, makes sense to me

[tarukumar]

LGTM