New image type: Fedora iot-bootable-container

[achilleas-k]

Add new image type and its image function called "bootc-container". The image function is very similar to the iot commit but enables the BootContainer flag so it exports a bootable base container instead of a tarball of the ostree repo.

The package set for the container mostly replicates the package selection from centos-bootc [1]. Comments are also retained from the original source files to assist in future decisions for modifications.

The two differences from the centos-bootc package set are:

• Kernel not specified: it is selected automatically along with user package selection.
• The packages specified in the user-experience [2] file aren't added. We want to keep the image definition minimal, and the existing set is already bigger than minimal.

[1] https://github.com/CentOS/centos-bootc/tree/4d5e1d86fbe1028f47e08d5533b4825e3b42dc68
[2] https://github.com/CentOS/centos-bootc/blob/4d5e1d86fbe1028f47e08d5533b4825e3b42dc68/tier-1/user-experience.yaml

DRAFT

Keeping draft until I include a boot test for this.

[cgwalters]

There's a bigger picture issue here too in that I think it'd be most useful to have a base image build process which works as part of a Containerfile...executed as part of podman build at least. That process would just much more naturally fit into existing pipelines.

[cgwalters]

Previous build-base-image-as-container discussion in coreos/rpm-ostree#4688

[achilleas-k]

There's a bigger picture issue here too in that I think it'd be most useful to have a base image build process which works as part of a Containerfile...executed as part of podman build at least. That process would just much more naturally fit into existing pipelines.

The plan here is to build Fedora bootc containers using image builder in Koji and we need it soon. There's certainly room for alternative, easier ways, but right now we have all the bits right here and can get the builds building (and iterating) with not a lot of effort in the existing pipelines.

[cgwalters]

The plan here is to build Fedora bootc containers using image builder in Koji and we need it soon.

Is that the plan? There was extensive prior work on
https://pagure.io/pungi/pull-request/1699
which also builds in Koji.

Don't get me wrong, I understand the thought behind this but...at a baseline forking the image definition is introducing a quite serious problem at this point in time.

I'd be much more interested in seeing an approach for sustainably exposing the ability to build base images for "external" users. The rpm-ostree YAML is...okay, but not great. We have issues like e.g. coreos/rpm-ostree#4701 which this effectively re-introduces - and again more generally the lack of (obvious) integration with container-native pipeline tooling (e.g. stitching together a manifest listed image).

This hardcoded-package-lists-in-Go is not something I can see us taking down a path for giving to "external" folks for a nice way to build base images.

[achilleas-k]

forking the image definition is introducing a quite serious problem at this point in time

Is it? We can keep it in sync. Question is: Who owns the definition, who will want to request changes for it, and where will that happen? Having the definition be a specification with potentially multiple implementations isn't unsustainable as long as everyone agrees on where the spec lives and who "owns" it.

I'd be much more interested in seeing an approach for sustainably exposing the ability to build base images for "external" users.

Sure. But that's a different conversation. The sustainability of a given solution is something we can theorise about. We can talk about all sorts of ways to enable external users to build base image. At the same time we can build base images with the tooling we have that can already do that. I don't see why it needs to be one or the other.

We have issues like e.g. coreos/rpm-ostree#4701 which this effectively re-introduces

Not sure I understand the linked issue fully. Is it that it doesn't implement the full container config spec? Then yeah we don't. I wouldn't say we're introducing a problem here, we're enabling an image type that we can build which exposes some customisation but is ultimately a specific configuration of a base image that we want to build. A template of sorts. If and when we need more, we can iterate, but we don't have to expose the full range of every kind of bootable container that can be built, just like when we build "Fedora 39 qcow2" we don't expose every single possible configuration of a Fedora 39 VM image. We do err on the side of limiting flexibility in favour of maintainability and predictability (and limiting the possibility of broken or unusable images), maybe a bit too far, but that's something we're considering more globally in the context of image builder and isn't specific to containers (bootable or otherwise).

more generally the lack of (obvious) integration with container-native pipeline tooling

IB isn't "container-native". You might even say it's not even "container-friendly" (bootc-image-builder made this quite clear). We all know this and we've discussed the reasons multiple times. I still don't understand why building a container image in our own infra with tooling that isn't container friendly is a problem though.

This hardcoded-package-lists-in-Go is not something I can see us taking down a path for giving to "external" folks for a nice way to build base images.

There are a lot of ideas for how to externalise this configuration. We never landed on one we all felt comfortable maintaining or distributing, but there are some good candidates. Other priorities pushed this down the list. It might be something we're going to change for a whole distro definition at some point (maybe Fedora and not RHEL, for example). Again though, it's a bigger story about how images are defined in IB and how much flexibility we want to enable from the outside. I don't think this is something we're going to change because of the introduction of a single new image definition, but it might be something we change as a result of a bigger integration effort, which might start from here.

[cgwalters]

Is it? We can keep it in sync. Question is: Who owns the definition, who will want to request changes for it, and where will that happen?

I would be the very first to say with a baseline statement that having content shipped by Fedora come from where the definition is currently (in https://github.com/CentOS/centos-bootc ) is weird. I know others have expressed this.

There is just a fundamental tension here in that the OS/distribution needs to be capable of owning the things it ships...but there also is today a somewhat dedicated team that is intending to manage this base image - and crucially that team has a somewhat strong interest in keeping things in sync across Fedora and C9S (and C10S in the near future). And that team operates cross-distribution.

This issue of course runs very much into the bigger picture infrastructural divergence between Fedora and CentOS Stream. Having things in this github organization doesn't from my PoV improve that (but doesn't make it worse either).

There's been a lot of discussions about aligning things with https://gitlab.com/redhat/centos-stream/ - which this effectively replaces. There's not small implications to that.

Having the definition be a specification with potentially multiple implementations isn't unsustainable as long as everyone agrees on where the spec lives and who "owns" it.

Given that we know we want to also make it very convenient for users/customers to build custom base images, ultimately I am less concerned with trying to define a "spec" for the single base image versus just ensuring that we aren't splitting up development efforts and CI and build systems too much in the short term.

None of this is a new issue...we of course have divergence between the way things like how the cloud images are built across the different distributions today.

Anyways I guess bottom line from me: Given that https://github.com/CentOS/centos-bootc/ is still actively maintained right at this moment, let's please continue to maintain it as Source Of Truth in the short term (i.e. sync the Go code from the YAML) as we iterate towards rethinking the larger long term architecture.

[cgwalters]

Another big thing this just drops is all the content we inject that isn't RPM. Now again...I feel the tension here because 98% of the content is RPM. But it's rather annoying to do configuration-management-via-RPM.

We have some important things in there today (some of which is already duplicated in osbuild, but definitely not all of it).

I am not happy myself with generating static files via scripts, and the direction I'd like to take some of these config files is a more container-native flow that looks like COPY.

[achilleas-k]

We have some important things in there today (some of which is already duplicated in osbuild, but definitely not all of it).

Things like these that aren't covered by existing osbuild stages should become new stages. This is what they're made for.

[achilleas-k]

I expect the tests will pass now so marking this as ready.

[cgwalters]

I guess as a framing thing here...ultimately the current centos-bootc configs were forked from FCOS, which actually in turn distantly inherits from RHEL Atomic Host. I've been maintaining variants of this thing for many years.

And we're pretty committed to continuing to maintain Fedora CoreOS for quite some time, alongside RHEL Atomic Host.

I had an outstanding PR in coreos/fedora-coreos-config#2610 which would add some inheritance.

Anyways, from my PoV the expected way the bootc ecosystem involves is container-native buildsystems, and ultimately at least some baseline system that works as part of a Containerfile.

[ondrejbudai]

@cgwalters I think these are good questions for @rhatdan and @nullr0ute. We got asked to build a minimal image for Fedora IoT that would include kernel, systemd and bootc. Thus, we are leveraging our experience of building Fedora IoT and RHEL for Edge to deliver this as soon as possible, because the deadlines are tight.

My suggestion is to keep things as they are in this PR, so we can start integrating this in Fedora asap. When the dust settles, we can certainly discuss how we can improve the setup. :)

Regarding container-native buildsystems: It would be quite cool to have this in bootc-image-builder, we should explore this option later.

[cgwalters]

Today though, the technology involved here doesn't actually use bootc (though, it is intending to be compatible with the current "ostree container" code for sure, but this is hindering the ability for bootc to do more than that). In fact, I'm chasing multiple bugs derived from that. There's a core tension here in that what we're building isn't RHEL for Edge - though it does absolutely make sense to not rewrite the world too! I'm not fundamentally opposed to experimentation in this direction, but let's be very clear: this is basically a fork. And that has quite not-small implications.

I'd kindly ask given the state of things that the work here not be branded bootc.

[cgwalters]

One very specific thing this change currently drops is CentOS/centos-bootc#98 - and that's not a small thing! It's actually a notable part of what we're trying to change here with bootc - as a distinct thing from ostree.

Hmm, now that I look...I'm quite surprised to see no one involved here as a watcher on https://github.com/CentOS/centos-bootc/watchers ...if you're not at least skimming and understanding changes in the source repository that you're forking code from, that's a recipe for problems exactly like this.

Somewhat less important, but notable given the above...there was some extensive debate in ostreedev/ostree#3114 (and also in ostreedev/ostree#3120 ) that it'd certainly be useful to have a smaller set of people watching there.

To rephrase this: what's key to the team maintaining the "bootc" experience is not rooted in making disk images from a filesystem tree. What's important is day 2 update consistency - hence all the debate in these things.

Going to other repos, as of right now we have one other participant as a watcher in bootc - I expect more than that.

[achilleas-k]

Today though, the technology involved here doesn't actually use bootc (though, it is intending to be compatible with the current "ostree container" code for sure, but this is hindering the ability for bootc to do more than that). In fact, I'm chasing multiple bugs derived from that. There's a core tension here in that what we're building isn't RHEL for Edge - though it does absolutely make sense to not rewrite the world too! I'm not fundamentally opposed to experimentation in this direction, but let's be very clear: this is basically a fork. And that has quite not-small implications.

I'd kindly ask given the state of things that the work here not be branded bootc.

I think this is the core of the issue. The bootc naming implies a connection with the centos-bootc project. I admit we didn't give the name much thought and I'm fine with calling it anything else. Keep in mind that merging here does not make the image configuration available to osbuild-composer, IB, or osbuild in Koji automatically, so there's room for changing the configuration and the name before it becomes "official".

This relates to my previous comment about who owns the image and where the specification lives. We were asked to produce a bootc compatible bootable container image for Fedora (not a direct quote, but more or less how I understood it). The initial request was to include bootc, a kernel, and systemd (and beyond that, anything require to actually make the image usable). I based it on the package lists in centos-bootc because it seemed like a good place to start. It could have just as well been based on the Fedora IoT base image, but that was considered a bit "less minimal" than what we needed.

One very specific thing this change currently drops is CentOS/centos-bootc#98 - and that's not a small thing! It's actually a notable part of what we're trying to change here with bootc - as a distinct thing from ostree.

The capability needs to be added to osbuild. We decided things like this can come after the original PoC.

Hmm, now that I look...I'm quite surprised to see no one involved here as a watcher on https://github.com/CentOS/centos-bootc/watchers ...if you're not at least skimming and understanding changes in the source repository that you're forking code from, that's a recipe for problems exactly like this.

Somewhat less important, but notable given the above...there was some extensive debate in ostreedev/ostree#3114 (and also in ostreedev/ostree#3120 ) that it'd certainly be useful to have a smaller set of people watching there.

To rephrase this: what's key to the team maintaining the "bootc" experience is not rooted in making disk images from a filesystem tree. What's important is day 2 update consistency - hence all the debate in these things.

Going to other repos, as of right now we have one other participant as a watcher in bootc - I expect more than that.

That's fair, you're right. We (or at least I) haven't been keeping up with the changes in all the parts that are changing here. That's primarily because it's often hard to understand what's an internal technology change and what affects the artifact that's coming out of all this. Also, until now, we were merely consuming the artifact, not producing anything resembling it. But beyond that, I see this as one more image type that's more or less controlled by the people who asked for it. Much like the R4E and IoT artifacts, we (IB) enable the configuration and building, and the edge/iot team define the configuration: packages, default fileystems, services, configurations, etc. Now that doesn't absolve us from knowing anything about it of course. But whether this image configuration tracks the centos-bootc one closely or not is something I expect we'll figure out in the process after we (IB and Edge/IoT) start building and testing. At least that's how we've been doing it so far, even though we never made this process explicit.
Again though, you're right. We should be watching the upstream bits more closely if we're going to be using them to define images.

[achilleas-k]

Troubleshooting that installer build failure for F39 aarch64.

[achilleas-k]

Troubleshooting that installer build failure for F39 aarch64.

Fixed in #388.

[ondrejbudai]

We agreed on continuing with this effort until we figure out something better. Ideally, it should be possible to build the base container via a multi-stage Containerfile, but such a file doesn't exist yet, thus pursuing the goal of Fedora IoT bootable container images with Image Builder sounds like the only immediately actionable way.