Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a bundle package and environment to build a basic "externals" stack #676

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

tmadlener
Copy link
Contributor

@tmadlener tmadlener commented Nov 26, 2024

BEGINRELEASENOTES

  • Add a key4hep-externals bundle package that can serve as a minimal external base on top of which most of the Key4hep packages can be built.
    • Most importantly this should include all the dependencies necessary for Gaudi, podio and DD4hep.
    • An additional constraint is given by a roughly 6 hour build time on github hosted runners, which makes it possible to build this into an external base image for further layering.

ENDRELEASENOTES

This is a proposal to add a BundlePackage and an environment that can be used to build a minimal external base on top of which most (all?) Key4hep packages can be built. The current dependencies are mainly collected from trying to build Gaudi and podio on top of it, there are almost certainly packages missing for other "low level" Key4hep packages (e.g. I haven't checked DD4hep).

One of the potential use cases for this is container building where a basis container can be built up-front and only has to change infrequently, while nightly builds or release build without changes to this basis container can be layered on top of it. We (well mostly @madbaron) have been exercising this workflow a bit for building containers for the muon collider, and so far it works out quite well.

I am opening this PR for a general discussion to see what others think and also to see which packages should still go into this package.

The naming is also still up for debate, currently its base but it could just as well be external or minimal if we want to reserve base for something that already has the main building blocks of Key4hep as well (e.g. k4FWCore, DD4hep, EDM4hep, maybe k4geo).

@madbaron
Copy link

Just one note to keep in mind if we end up changing the packages included in "base": to keep the multi-step image build working on the github runners, we should keep the compilation time (including setup, checkout and upload to the registry) of each image/layer below 6 hours.

The current set takes about 5h 40m (see for example https://github.com/madbaron/mucoll-spack/actions/runs/12018805366)

@tmadlener tmadlener changed the title [WIP] Add a bundle package and environment to build a basic "externals" stack Add a bundle package and environment to build a basic "externals" stack Dec 17, 2024
@tmadlener
Copy link
Contributor Author

@madbaron quick heads up that I have renamed the package (and the environment) to use external instead of base. So MuonColliderSoft/mucoll-spack#17 will require some changes.

As dicussed on some occasions, we would like this to be added here, and we would also take on the maintenance for this. We also plan to port some of the muon collider specific github actions that build an image from this to here in a follow up PR unless something speaks strictly against that.

I am not sure if / which CI checks we should / could already add now.

compiler: []

boost:
require: +atomic+chrono+context+date_time+exception+fiber+filesystem+graph+iostreams+locale+log+math+multithreaded+program_options+python+random+regex+serialization+shared+signals+system+test+thread+timer+wave
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jmcarcell should this go into the common packages.yaml?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants