updates alternatives don't work

[dustymabe]

However much you might like or dislike alternatives. There are significant pieces of our stack that still use it. Let's go through some effort to get it working with rpm-ostree systems so we dont hit the recent issue with iptables again (#676). There is an open issue upstream (fedora-sysv/chkconfig#27) asking if alternatives can be updated to use /etc/ for state instead of /var/. Let's try to work with the maintainer upstream to fix this problem so all RPM-OSTree systems don't have this same problem where intended defaults don't get applied.

[travier]

Stumbled upon this one again investigating the following log:

systemd[1]: var.mount: Directory /var to mount over is not empty, mounting anyway.

Which is from:

ls -R /sysroot/ostree/deploy/fedora-coreos/deploy/*/var/
/sysroot/ostree/deploy/fedora-coreos/deploy/1b7e0d318f5e78013434236ad17edb8b65297f0e8de3e765562fdedc1a7e52ab.0/var/:
lib  mnt

/sysroot/ostree/deploy/fedora-coreos/deploy/1b7e0d318f5e78013434236ad17edb8b65297f0e8de3e765562fdedc1a7e52ab.0/var/lib:
alternatives  vagrant

/sysroot/ostree/deploy/fedora-coreos/deploy/1b7e0d318f5e78013434236ad17edb8b65297f0e8de3e765562fdedc1a7e52ab.0/var/mnt:

For reference: coreos/rpm-ostree#1614 & coreos/rpm-ostree#1861

Code: https://github.com/coreos/rpm-ostree/blob/12cc4e5f5bbe8da2b59e10fef05b9d30156fc39e/rust/src/composepost.rs#L707-L713

[LorbusChris]

related: fedora-sysv/chkconfig#9

[travier]

fedora-sysv/chkconfig#135 has been merged. I have not checked the implications for existing and new systems yet.

[travier]

With https://bodhi.fedoraproject.org/updates/FEDORA-2024-4960de2b0f in F41, this should be fixed in next. Will need some testing. Maybe CC @HuijingHei ?

[travier]

If it's indeed fixed, we'll need a release notes entry and probably some docs.

[HuijingHei]

Do testing according to fedora-sysv/chkconfig#135 (comment) in fedora-bootc container. Not sure this is enough, let me know if there is any other test, thanks!

$cat Containerfile 
FROM quay.io/fedora/fedora-bootc:41
RUN dnf -y install golang
RUN ls /etc/alternatives /etc/alternatives-admindir
RUN go help > /dev/null

$podman build -t test .
STEP 1/4: FROM quay.io/fedora/fedora-bootc:41
STEP 2/4: RUN dnf -y install golang
--> Using cache 7fa164769841b1d89377ec95d9caac5f1d246bc96b35fe53689b13a9ed70f706
--> 7fa164769841
STEP 3/4: RUN ls /etc/alternatives /etc/alternatives-admindir
/etc/alternatives:
arptables
arptables-helper
arptables-man
arptables-restore
arptables-restore-man
arptables-save
arptables-save-man
cifs-idmap-plugin
ebtables
ebtables-man
ebtables-restore
ebtables-save
go
gofmt
ip6tables
ip6tables-restore
ip6tables-save
iptables
iptables-restore
iptables-save
ld
libnssckbi.so.x86_64
soelim
soelim.1.gz

/etc/alternatives-admindir:
arptables
cifs-idmap-plugin
ebtables
go
iptables
ld
libnssckbi.so.x86_64
soelim
--> 5cdd6d809582
STEP 4/4: RUN go help > /dev/null
COMMIT test
--> 0023f91c90b6
Successfully tagged localhost/test:latest
0023f91c90b6124c9652d559a96fdea9fc77ca4b120c13f14ee39389e59ad566

Check in container alternatives --verbose --display go works.

$ podman run --rm -it test bash
bash-5.2# alternatives --verbose --display go
reading /etc/alternatives-admindir/go
go - status is auto.
 link currently points to /usr/lib/golang/bin/go
/usr/lib/golang/bin/go - priority 90
 follower gofmt: /usr/lib/golang/bin/gofmt
Current `best' version is /usr/lib/golang/bin/go.

[travier]

Can you test on a running system if the alternatives commands work? Maybe something like what's in https://www.redhat.com/en/blog/alternatives-command. Those won't work "live" as they modify /usr .

We should then update the docs in https://docs.fedoraproject.org/en-US/fedora-coreos/alternatives/ with the "normal" alternatives commands.

[travier]

From https://www.mankier.com/8/alternatives:

• The install & remove commands should work in a container build
• The config & set ones should work at runtime.

[HuijingHei]

Try on silverblue, seems can not create new at runtime, if using existing files works.

fedora@fedora:~$ sudo alternatives --install /usr/bin/emacs emacs /usr/bin/vi 1
fedora@fedora:~$ sudo alternatives --set emacs /usr/bin/vi
fedora@fedora:~$ alternatives --display emacs
emacs - status is manual.
 link currently points to /usr/bin/vi
/usr/bin/emacs-pgtk - priority 80
/usr/bin/vi - priority 1
Current `best' version is /usr/bin/emacs-pgtk.

fedora@fedora:~$ sudo alternatives --remove emacs /usr/bin/vi
fedora@fedora:~$ alternatives --display emacs
emacs - status is auto.
 link currently points to /usr/bin/emacs-pgtk
/usr/bin/emacs-pgtk - priority 80
Current `best' version is /usr/bin/emacs-pgtk.

fedora@fedora:~$ rpm-ostree status
State: idle
Deployments:
● fedora:fedora/41/x86_64/silverblue
                  Version: 41.20241015.n.0 (2024-10-15T08:12:05Z)
               BaseCommit: 86d8b03190580c77eb08fad2e695ba3e255d6942f8ac74aa0acbed819fb09759
             GPGSignature: Valid signature by 466CF2D8B60BC3057AA9453ED0622462E99D6AD1
          LayeredPackages: emacs

  fedora:fedora/41/x86_64/silverblue
                  Version: 41.20241015.n.0 (2024-10-15T08:12:05Z)
                   Commit: 86d8b03190580c77eb08fad2e695ba3e255d6942f8ac74aa0acbed819fb09759
             GPGSignature: Valid signature by 466CF2D8B60BC3057AA9453ED0622462E99D6AD1

[HuijingHei]

Build FCOS image with next (as it has fixed alternatives-1.30-1.fc41.x86_64) and removing the related alternatives in https://github.com/coreos/fedora-coreos-config/blob/testing-devel/manifests/fedora-coreos-base.yaml#L74, do testing refer to coreos/fedora-coreos-config#264. By default iptables points to /usr/sbin/iptables-legacy, use alternatives to update iptables points to /usr/sbin/iptables-nft.

[core@cosa-devsh ~]$ sudo alternatives --display iptables
iptables - status is auto.
 link currently points to /usr/sbin/iptables-legacy
/usr/sbin/iptables-legacy - priority 10
 follower ip6tables: /usr/sbin/ip6tables-legacy
 follower ip6tables-restore: /usr/sbin/ip6tables-legacy-restore
 follower ip6tables-save: /usr/sbin/ip6tables-legacy-save
 follower iptables-restore: /usr/sbin/iptables-legacy-restore
 follower iptables-save: /usr/sbin/iptables-legacy-save
/usr/sbin/iptables-nft - priority 10
 follower ip6tables: /usr/sbin/ip6tables-nft
 follower ip6tables-restore: /usr/sbin/ip6tables-nft-restore
 follower ip6tables-save: /usr/sbin/ip6tables-nft-save
 follower iptables-restore: /usr/sbin/iptables-nft-restore
 follower iptables-save: /usr/sbin/iptables-nft-save
Current `best' version is /usr/sbin/iptables-legacy.

[core@cosa-devsh ~]$ pfx=/usr/sbin/iptables
pfx6=/usr/sbin/ip6tables
sudo alternatives --install $pfx iptables $pfx-nft 15 \
  --slave $pfx6 ip6tables $pfx6-nft \
  --slave $pfx-restore iptables-restore $pfx-nft-restore \
  --slave $pfx-save iptables-save $pfx-nft-save \
  --slave $pfx6-restore ip6tables-restore $pfx6-nft-restore \
  --slave $pfx6-save ip6tables-save $pfx6-nft-save
sudo alternatives --auto iptables

[core@cosa-devsh ~]$ sudo alternatives --display iptables
iptables - status is auto.
 link currently points to /usr/sbin/iptables-nft
/usr/sbin/iptables-legacy - priority 10
 follower ip6tables: /usr/sbin/ip6tables-legacy
 follower ip6tables-restore: /usr/sbin/ip6tables-legacy-restore
 follower ip6tables-save: /usr/sbin/ip6tables-legacy-save
 follower iptables-restore: /usr/sbin/iptables-legacy-restore
 follower iptables-save: /usr/sbin/iptables-legacy-save
/usr/sbin/iptables-nft - priority 15
 follower ip6tables: /usr/sbin/ip6tables-nft
 follower ip6tables-restore: /usr/sbin/ip6tables-nft-restore
 follower ip6tables-save: /usr/sbin/ip6tables-nft-save
 follower iptables-restore: /usr/sbin/iptables-nft-restore
 follower iptables-save: /usr/sbin/iptables-nft-save
Current `best' version is /usr/sbin/iptables-nft.

[HuijingHei]

The related doc is updated in coreos/fedora-coreos-docs#673

[travier]

So, coreos/fedora-coreos-config#264 is not "correct" in the sense that it does something that should only be done when creating an RPM or when you want to add your own config.

I think what we want instead is to document:

[core@cosa-devsh ~]$ alternatives --display iptables
iptables - status is manual.
 link currently points to /usr/sbin/iptables-nft
/usr/sbin/iptables-legacy - priority 10
 follower ip6tables: /usr/sbin/ip6tables-legacy
 follower ip6tables-restore: /usr/sbin/ip6tables-legacy-restore
 follower ip6tables-save: /usr/sbin/ip6tables-legacy-save
 follower iptables-restore: /usr/sbin/iptables-legacy-restore
 follower iptables-save: /usr/sbin/iptables-legacy-save
/usr/sbin/iptables-nft - priority 10
 follower ip6tables: /usr/sbin/ip6tables-nft
 follower ip6tables-restore: /usr/sbin/ip6tables-nft-restore
 follower ip6tables-save: /usr/sbin/ip6tables-nft-save
 follower iptables-restore: /usr/sbin/iptables-nft-restore
 follower iptables-save: /usr/sbin/iptables-nft-save
Current `best' version is /usr/sbin/iptables-legacy.
[core@cosa-devsh ~]$ iptables --version
iptables v1.8.10 (nf_tables)
[core@cosa-devsh ~]$ sudo alternatives --set iptables /usr/sbin/iptables-legacy
[core@cosa-devsh ~]$ alternatives --display iptables
iptables - status is manual.
 link currently points to /usr/sbin/iptables-legacy
/usr/sbin/iptables-legacy - priority 10
 follower ip6tables: /usr/sbin/ip6tables-legacy
 follower ip6tables-restore: /usr/sbin/ip6tables-legacy-restore
 follower ip6tables-save: /usr/sbin/ip6tables-legacy-save
 follower iptables-restore: /usr/sbin/iptables-legacy-restore
 follower iptables-save: /usr/sbin/iptables-legacy-save
/usr/sbin/iptables-nft - priority 10
 follower ip6tables: /usr/sbin/ip6tables-nft
 follower ip6tables-restore: /usr/sbin/ip6tables-nft-restore
 follower ip6tables-save: /usr/sbin/ip6tables-nft-save
 follower iptables-restore: /usr/sbin/iptables-nft-restore
 follower iptables-save: /usr/sbin/iptables-nft-save
Current `best' version is /usr/sbin/iptables-legacy.
[core@cosa-devsh ~]$ iptables --version
iptables v1.8.10 (legacy)

See also: #1818

[travier]

If I understand correctly, the --install and --remove commands are there to create & remove new alternatives configs so we should probably not use them to alter the existing ones.

[HuijingHei]

Thanks for the pointing out, sorry for my misunderstanding, update in the doc PR and remove the command output.

[dustymabe]

The fix for this went into next stream release 41.20240916.1.0. Please try out the new release and report issues.

[dustymabe]

The fix for this went into testing stream release 41.20241027.2.0. Please try out the new release and report issues.

[dustymabe]

The fix for this went into stable stream release 41.20241027.3.0.