Remove build and publish base and base-with-services to registry

[unkcpz]

After aiidateam/aiida-core#6080, the use cases of base and base-with-services are all covered. I propose to remove those for simplicity. It also saves the CI build resource and time.

• how to arrange the tests.
• readme.

[danielhollas]

@unkcpz can you please comment on how much total build time this saves? My gut feeling would be that it does not save that much since these images needs to be built anyway, and that's the expensive step, publishing should not be that expensive. But maybe I am missing something.

[unkcpz]

Compare between two new CI runs:

• 1h 27m 1s (with base, the current main) https://github.com/aiidalab/aiidalab-docker-stack/actions/runs/6563217261 and
• 1h 8m 34s (this PR) https://github.com/aiidalab/aiidalab-docker-stack/actions/runs/6533815664?pr=412

I'd say save quite much. But as you said the image needs to be built anyway for lab/full-stack. The time saved is mostly from the slow upload docker image as an artifact step, which even takes a longer time than building the image.

[danielhollas]

The time saved is mostly from the slow upload docker image as an artifact step, which even takes a longer time than building the image.

That is strange. Is this true for both amd64 and arm runners, or only on of them?
over 1h time is brutal, compared to what we had before. Is this all coming from the arm64 builds? Are we reusing all the caches properly?

[unkcpz]

over 1h time is brutal, compared to what we had before.

It is 1h for all images not for one, and not all are run in parallel because if lab is failed it is pointless to run full-stack. Originally all builds ran in parallel and produced 4 images.
Without the changes in this PR, it will finally generate 8 images, 4 for amd64 and 4 for arm64.
If you look at the time (https://github.com/aiidalab/aiidalab-docker-stack/actions/runs/6533815664?pr=412), every image build+test+uplead costs 10~15mins. AMR64 and AMD64 runs in parallel.
It is sad that the upload of artifacts is slow and uses 50% of the time, which is a GitHub action issue (actions/upload-artifact#199) and they aren't keen to solve it.

The artifact is essential to separate the image build and different arch combining so can not be skipped if we want to keep a clear separation of different arch/images.

Are we reusing all the caches properly?

There is no cache used (the artifacts are reused but not github action cache ). I think no cache should be used since every change we make potentially influences the final image and we want to get it tested.

[danielhollas]

Interesting, thank you for the write-up!
The last comment in that issue seems to imply that things should significantly improve in the upcoming v4 release of the artifact-upload action, so we should keep an eye on that.

[danielhollas]

I am unsure here, partly perhaps because I am not that familiar with the new aiida-core image and what exactly it covers. I get the benefits of not publishing unneeded images, but am unsure if we are not getting rid of useful functionality. I can imagine for example that the base-with-services image that includes jupyter stack might be beneficial for tutorial / demo purposes.

I would like at least @yakutovicha to weigh in. Not sure who else might be using these images? @unkcpz feel free to ping somebody else who you think might be qualified to make this decision.

I did not look at the changes here in too much detail but they look fine to me at a glance.

[unkcpz]

Not sure who else might be using these images? @unkcpz feel free to ping somebody else who you think might be qualified to make this decision.

Thanks for pointing this out, I didn't consider this. If I remember correctly during AiiDA tutorial the student is running tutorial in a JupyterLab environment. Thus I guess base-with-services was used at that time?
In fact, the full-stack also can be used with JupyterLab as a server engine, only need to change the default server engine when deploying (and JupyterLab is the default actually). I pinning @mbercx who deployed the Azure services for previous tutorial, do you find base image is still useful if full-stack/lab can cover the use case (Need to mention a downside I can foresee is the full-stack/lab contains /home/jovyan/apps which is not useful and confusing maybe?).

Another option to accelerate the build time is make base/base-with-serveces and lab/full-stack build in parallel, downside is if the base is break the lab/full-stack will still build and waste some resources but I think it is acceptable.

[unkcpz]

As mentioned in the AiiDAlab, @danielhollas mentioned the artifact upload is not necessary, which I didn't realize. I 'll try it and that'll save a lot build/test time for the images.

[yakutovicha]

As mentioned in the AiiDAlab, @danielhollas mentioned the artifact upload is not necessary, which I didn't realize. I 'll try it and that'll save a lot build/test time for the images.

So is there anything for me to review at the moment? If you still need to work on the PR, then I prefer to have a look afterwards. Maybe make it a draft also.

[unkcpz]

@yakutovicha for this PR it tries to remove the release of base and base-with-services, I think it is not necessary because I think hard and can not come up with cases that can not covered by lab and full-stack.

[danielhollas]

Note that there was a plan to decouple the various services into separate containers (e.g. separate container for pgsql) and use docker-compose to run them. Which might overall help to manage the dependencies. So I would be against removing base and base-with-services for now, I think we should be able to speed up the build times in other way as discussed. Dne čt 9. 11. 2023 15:44 uživatel Jusong Yu ***@***.***> napsal:

…

@yakutovicha <https://github.com/yakutovicha> for this PR it tries to remove the release of base and base-with-services, I think it is not necessary because I think hard and can not come up with cases that can not covered by lab and full-stack. — Reply to this email directly, view it on GitHub <#412 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACIY64NSG75ZKTBM333INX3YDT24TAVCNFSM6AAAAAA6CD4HCSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBUGA3TMNJQGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[unkcpz]

into separate containers (e.g. separate container for pgsql) and use docker-compose to
run them.

make sense to me if there are potential use cases of those images. I close this PR.