tracker: Rebase onto Fedora 36

[dustymabe]

Rebase to a new version of Fedora (N=36)

At Branching

Branching is when a new stream is "branched" off of rawhide. This eventually becomes the next major Fedora (N).

Release engineering changes

• Verify that a few tags were created when branching occurred:

• f${N+1}-coreos-signing-pending

• f${N+1}-coreos-continuous

• Add the N+1 signing key short hash (usually found here) to the tag info for the coreos-pool tag. The following commands view the current settings and then update the list to the 32/33/34/35 keys. You'll most likely have to get someone from releng to run the second command (edit-tag).

  • koji taginfo coreos-pool
  • koji edit-tag coreos-pool -x tag2distrepo.keys="12c944d0 9570ff31 45719a39 9867c58f"

coreos-installer changes

• Update coreos-installer to know about the signing key used for the future new major version of Fedora (N+1).
• Drop the signing key for the obsolete stable release (N-2).

Update rawhide stream

• Update manifest.yaml to list N+1 as the releasever.

Enable branched stream

• Update manifest.yaml to list N as the releasever.
• Update streams.groovy to include the branched stream in the list of mechanical refs.

At Fedora (N) Beta

Update fedora-coreos-config next-devel

• Bump releasever in manifest.yaml
• Update the repos in manifest.yaml if needed
• Run cosa fetch --update-lockfile
• PR the result

Ship rebased next

• Ship next
• Set a new update barrier for N-2 on next. In the barrier entry set a link to the docs. See discussion.

Preparing for Fedora (N) GA

Update fedora-coreos-config testing-devel

• Bump releasever in manifest.yaml
• Update the repos in manifest.yaml if needed
• Run cosa fetch --update-lockfile
• Bump the base Fedora version in ci/buildroot/Dockerfile
• PR the result

At Fedora (N) GA

Ship rebased testing

• Ship testing
• Set a new update barrier for N-2 on testing. In the barrier entry set a link to the docs. See discussion.

Disabled branched stream

• Update streams.groovy to remove the branched stream in the list of mechanical refs.

Untag old packages

koji untag N-2 packages from the pool (at some point we'll have GC in place to do this for us, but for now we must remember to do this manually or otherwise distRepo will fail once the signed packages are GC'ed). For example the following snippet finds all RPMs signed by the Fedora 32 key and untags them. Use this process:

• Find the key short hash. Usually found here. Then:

f32key=12c944d0
key=$f32key
untaglist=''
for build in $(koji list-tagged --quiet coreos-pool | cut -f1 -d' '); do
    if koji buildinfo $build | grep $key 1>/dev/null; then
        untaglist+="${build} "
        echo "Adding $build to untag list"
    fi
done

• Now we have a list of builds to untag. But we need one more sanity check. Let's make sure none of those are actually being used. Fire up the latest FCOS testing-devel and run:

f32key=12c944d0
key=$f32key
rpm -qai | grep -B 8 $key

If there are any RPMs signed by the old key they'll need to be investigated. Maybe they shouldn't be used any longer. Or maybe they're still needed.

• After verifying the list looks good:

koji untag-build coreos-pool $untaglist

• Now that untagging is done, give a heads up to rpm-ostree developers that N-2 packages have been untagged and that they may need to update their CI compose tests to freeze on a newer FCOS commit.

• Remove the N-2 signing key from the tag info for the coreos-pool tag. The following commands view the current settings and then update the list to the 33/34/35 keys. You'll most likely have to get someone from releng to run the second command (edit-tag).

  • koji taginfo coreos-pool
  • koji edit-tag coreos-pool -x tag2distrepo.keys="9570ff31 45719a39 9867c58f"

Disable next-devel stream

We prefer to disable next-devel when there is no difference between testing-devel and next-devel. This allows us to prevent wasting a bunch of resources (bandwidth, storage, compute) for no reason. After the switch to N if next-devel and testing-devel are in lockstep, then disable next-devel.

• Follow the instructions here to disable next-devel

After Fedora (N) GA

Ship rebased stable

• Ship stable
• Set a new update barrier for N-2 on stable. In the barrier entry set a link to the docs. See discussion.

Miscellaneous container updates

These are various containers in use throughout our ecosystem. We should update or open a ticket to track updating them once a new Fedora release is out. If you open a ticket instead of doing the update add a link to the ticket as comment.

• Update coreos-assembler or open ticket to update:
  • Dockerfile
• Update coreos-installer
  • Dockerfile
• Update Ignition
  • Dockerfile.validate
• Update Butane
  • Dockerfile
• Update fedora-coreos-cincinnati
  • Dockerfile
• Update config-bot
  • Dockerfile
• Update coreos-koji-tagger
  • Dockerfile
  • ImageStream
  • BuildConfig
• Update coreos-ostree-importer
  • Dockerfile
  • ImageStream
  • BuildConfig

[dustymabe]

request for releng to add the f37 signing key to the taginfo for coreos-pool: https://pagure.io/releng/issue/10635

[dustymabe]

• signing-keys: regular Fedora cycle rotation of keys coreos-installer#770

[dustymabe]

• [branched] manifest: bump to F36 fedora-coreos-config#1506
• [rawhide] manifest: rawhide is now F37 fedora-coreos-config#1507

[dustymabe]

• streams: re-enable branched stream fedora-coreos-pipeline#476

[dustymabe]

• [next-devel] Move to Fedora Linux 36 fedora-coreos-config#1605

[dustymabe]

In general you may see us fast-tracking packages in next-devel before they reach bodhi stable. In general this is to prevent downgrades (newer packages in f35 than in f36) when comparing testing* streams to next* streams.

[dustymabe]

Just noting we updated the release barrier for next in coreos/fedora-coreos-streams@40fde3e

[dustymabe]

We were migrating to a new OpenShift cluster so I moved the coreos-koji-tagger, coreos-ostree-importer, and fedora-ostree-pruner to Fedora 36 in:

• https://pagure.io/fedora-infra/ansible/pull-request/1034
• https://pagure.io/fedora-infra/ansible/pull-request/1038
• Updates for tagger and importer fedora-coreos-releng-automation#151

[lucab]

For coreos-cincinnati, we are artificially keeping it on an older Fedora base image due to https://pagure.io/fedora-infrastructure/issue/10111. I crossed it out for now.

[dustymabe]

For coreos-cincinnati, we are artificially keeping it on an older Fedora base image due to https://pagure.io/fedora-infrastructure/issue/10111. I crossed it out for now.

Hmm. We're migrating to a new (OCP4) cluster, though. Doesn't that mean the issue goes away?

[lucab]

At least not until we have migrated out of the old OCP3 cluster, which won't happen before F36 GA due to the infra freeze.
At that point, yes I hope the issue will be solved and we'll be able to update the base image.

[bgilbert]

Dockerfile updates:

• Dockerfile.validate: build with Fedora 36 ignition#1364
• Dockerfile: update to Fedora 36 coreos-installer#857
• Dockerfile: update to Fedora 36 butane#345

[dustymabe]

• PR to switch testing-devel
  • migrate to Fedora 36 fedora-coreos-config#1732

[dustymabe]

• streams: disable next-devel fedora-coreos-pipeline#522

[dustymabe]

• Move to Fedora 36! coreos-assembler#2856

[lucab]

• dist/Dockerfile: bump base image to Fedora 36 fedora-coreos-cincinnati#65

[dustymabe]

• config-bot/Dockerfile: update to F36 fedora-coreos-releng-automation#152

[dustymabe]

I'm going through and untagging F34 packages from coreos-pool today.

[jlebon]

Now that untagging is done, give a heads up to rpm-ostree developers that N-2 packages have been untagged and that they may need to update their CI compose tests to freeze on a newer FCOS commit.

coreos/rpm-ostree#3700

[dustymabe]

• [stable] tree: promote changes from testing at ad362a929591fc5f1bb97d6fac2572b3ea82ee16 fedora-coreos-config#1752

[lucab]

We had to revert the F36 bump for fedora-coreos-cincinnati as it was causing service disruption. We will attempt it again, and we are running on F35 for the moment. The ticket for this is coreos/fedora-coreos-cincinnati#70.

[dustymabe]

Closing this out since we've already shipped F37 and completed all tasks.