Roadmap to Fedora Bootable Containers

[travier]

Notes

• This is a proposed roadmap that is subject to change and refinement
• While not complete nor matching the current Fedora bootable container images, you can already use Fedora CoreOS with container images, but we currently don't recommend that as it comes with important caveats.

Roadmap

Building and publishing Bootable Container images

• We are currently building bootable container images in our pipeline
• Those images are published to Quay.io: https://quay.io/repository/fedora/fedora-coreos?tab=tags

Switching to Bootable Container images by default

• Create container repo tags for each FCOS release:
  • Create container repo tags for each FCOS release #1367
• Migrate Fedora CoreOS users to update via a container image by default:
  • Move from OSTree to OCI for updates #1823
• Zincati needs to be adapted for the container update flow:
  • Making Cincinnati updates work with ostree containers #1263
• (Optional but desired) Use zstd:chunked container images

DNF5 integration

• Adding dnf5 to the images:
  • https://fedoraproject.org/wiki/Changes/DNFAndBootcInImageModeFedora
  • Ship dnf in FCOS and RHCOS #1687
  • Add dnf5 on Fedora 41+ fedora-coreos-config#2915
• Better error handling / messages in dnf (on running systems) would make this less confusing to our users
  • Give a more useful error message on image-mode systems rpm-software-management/dnf5#1727
• A lot of testing needed, especially regarding alternative kernels, custom kernel modules, /var and /opt handling, etc.

bootc integration

• Integration of bootc will require integration in Zincati
• Bootc is currently root only: no unprivileged interface, no DBus interface
• Related discussions:
  • cli: Add --json support to --check containers/bootc#472
  • bootc status does not report unlocked status containers/bootc#474

Bootimages

• Using bootc install to-filesystem when building container images: Use bootc install to-filesystem to build our bootimages #1827
• Making it easier to build custom bootimages. Consider alignment with b-i-b.

Local package layering

• Figure out a solution for users that want to locally layer packages
• Similar to what's needed for Fedora Atomic Desktops
• Tracked in https://gitlab.com/fedora/bootc/tracker/-/issues/4

Rebasing on Fedora Bootc manifests

• https://gitlab.com/fedora/bootc/base-images/-/merge_requests/48
• Inherit from fedora-bootc's tier-x on Fedora 42+ fedora-coreos-config#3177

Rebasing on Fedora Bootc container images

• Needs better support for container builds in the Fedora Infrastructure (via gitlab.com/fedora?)
• Support for container deltas / zstd:chunked:
  • Add support for zstd:chunked ostreedev/ostree-rs-ext#608
  • https://gitlab.com/fedora/bootc/tracker/-/issues/9
• Investigate if rebasing to a container based workflow for those image would bring benefits
  • Needs a fleshed out CI story for the base images and layered variants
• Investigate the new Experimental Base Images Builder: https://gitlab.com/fedora/bootc/base-images-experimental

Anaconda

• See: Consider supporting Anaconda to install FCOS/RHCOS in the future #1646

Investigate Konflux CI/CD

See if we can build FCOS using Konflux: https://gist.github.com/ralphbean/a3644111a549e8cedb0b207f90d42dc9

Documentation updates

• We will likely have to update the documentation to link to the Fedora Bootable Containers docs.

Issues that needs to be triaged / refocused

• Create tooling to take a FCOS-derived container and generate bootable images from it #1151
• Tracker for bootc integration #1446
• Complete composefs integration in Fedora CoreOS #1718
• Investigate UKI support in Fedora CoreOS #1719

See also all the issues tagged with bootable-containers: https://github.com/coreos/fedora-coreos-tracker/issues?q=is%3Aopen+label%3Aarea%2Fbootable-containers+sort%3Aupdated-desc

References

See:

• https://fedoramagazine.org/get-involved-with-fedora-bootable-containers/
• https://fedoraproject.org/wiki/Initiatives/Fedora_bootc

See for Fedora Atomic Desktops: https://gitlab.com/fedora/ostree/sig/-/issues/26

[cgwalters]

Thanks so much for putting this together!

[cgwalters]

Any objections if we let the fedora-bootc related discussion "double up" as part of the current fcos community meeting? Basically maybe ~40 minutes for this topic which would also include e.g. things related to https://gitlab.com/fedora/ostree/sig/-/issues/26 etc?

[jlebon]

Any objections if we let the fedora-bootc related discussion "double up" as part of the current fcos community meeting?

I wonder actually if we should schedule a community video meeting to discuss this. I guess it's too late for today but could be next week's.

[travier]

What do you mean by double up? I would prefer we keep the Fedora CoreOS meeting distinct for now.

We can however include the Atomic SIG topics into the bootc one as we are not having Atomic SIG meetings right now.

[jasonbrooks]

What do you mean by double up? I would prefer we keep the Fedora CoreOS meeting distinct for now.

We can however include the Atomic SIG topics into the bootc one as we are not having Atomic SIG meetings right now.

I think @cgwalters just meant double up for next week, or something, inviting additional people to talk about bootc at the same time that FCOS folks do.

[travier]

As I'm working on Live ISO support for Atomic Desktops, I'm experimenting with using a container layer to include all the tools needed for the Live ISO / installer instead of including them in the base image by default: https://github.com/travier/fedora-kinoite/blob/main/fedora-kinoite-live/Containerfile

Thinking about this more, we could use the same approach for Ignition and related first boot elements. We would include those in a container layer and rebuild the initrd and use this "derived" container to generate all the disk images. Systems installed this way would be pointed to the image without the first boot layer, that will thus "disappear" from the system on the first update.

We would thus have two container tags for each release, once with and one without the first boot layer.

Ideally we would also produce a container image that does not include an initrd, then we layer the initrd to create the "normal" base image which is delivered on updates.

Same idea but for podman/moby-engine as well: #1723

- FCOS image with no-initrd, no first boot tools, no podman & moby-engine (base/core)
    |
     ---> layer podman & moby-engine
    |     |
    |      ---> layer initrd (full)
    |     |
    |      ---> layer first boot tools & build initrd (full-first-boot)
    |
     ---> layer initrd (minimal)
    |
     ---> layer first boot tools & build initrd (minimal-first-boot)

[travier]

You may encounter issues when layering packages that comes with their own SELinux policy modules in container layers. See: ostreedev/ostree-rs-ext#510 (tracked in https://gitlab.com/fedora/ostree/sig/-/issues/45 for the Atomic Desktops)

[lmilbaum]

Do we want to add a CI/CD section to the roadmap? Onboarding to Konflux might be a relevant task. WDYT?