SB with bootc: systemd-remount-fs.service : mount: /: fsconfig system call failed: overlay: No changes allowed in reconfigure.

[TheGreatDeadOne]

Describe the bug

Hello, I'm testing the unofficial image of SB with bootc and I noticed something strange in the status of systemd. The systemd-omount-fs.service fails to mount / (!?), But it still does not lake the operation of the OS. I should add that I tested some sysext images (I used using mkosi+erofs and dnf --installroot), following Flatcar (Fedora-Sysext's too) recipe, but I can't say that the error came up after the image was mounted because I only checked the loading failure Unit soon after performing systemd-sysext unmerge.

To Reproduce
Please describe the steps needed to reproduce the bug:

1. systemctl status
2. service failed

Screenshots

dmesg:

[   10.785509] systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE

OS version:

State: idle
BootedDeployment:
● ostree-unverified-registry:quay.io/fedora-ostree-desktops/silverblue:rawhide
                   Digest: sha256:647a6450a8405017f63596c081416a4d4609b1d44ce45bbcde9803e7259b4614
                  Version: 42.20241014.0 (2024-10-14T01:14:03Z)

bootc status

apiVersion: org.containers.bootc/v1
kind: BootcHost
metadata:
  name: host
spec:
  image:
    image: quay.io/fedora-ostree-desktops/silverblue:rawhide
    transport: registry
  bootOrder: default
status:
  staged: null
  booted:
    image:
      image:
        image: quay.io/fedora-ostree-desktops/silverblue:rawhide
        transport: registry
      version: 42.20241014.0
      timestamp: null
      imageDigest: sha256:647a6450a8405017f63596c081416a4d4609b1d44ce45bbcde9803e7259b4614
    cachedUpdate: null
    incompatible: false
    pinned: false
    store: ostreeContainer
    ostree:
      checksum: 9f2ac59fac0fc7df9d1e74ff4d76b236f50af80f233fa3f96a36bb12a051cead
      deploySerial: 0
  rollback:
    image: null
    cachedUpdate: null
    incompatible: false
    pinned: true
    store: null
    ostree:
      checksum: 72d19057a29cd39983644bcc0422028740e79b65d8a739c06cf7a64d4575ec9a
      deploySerial: 0
  rollbackQueued: false
  type: bootcHost

PS: The official ostree image does not have this service failure.

[TheGreatDeadOne]

sudo systemctl service-log-level systemd-remount-fs.service debug:

 Unit systemd-remount-fs.service doesn't declare BusName=.
Hint: the service must declare BusName= and implement the appropriate D-Bus interface.
      See the org.freedesktop.LogControl1(5) man page for details.

SYSTEMD_LOG_LEVEL=debug systemctl restart -f systemd-remount-fs.service:

Bus n/a: changing state UNSET → OPENING
sd-bus: starting bus by connecting to /run/systemd/private...
Bus n/a: changing state OPENING → AUTHENTICATING
Successfully forked off '(sd-askpwagent)' as PID 6844.
Executing dbus call org.freedesktop.systemd1.Manager RestartUnit(systemd-remount-fs.service, replace)
Bus n/a: changing state AUTHENTICATING → RUNNING
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=RestartUnit cookie=1 reply_cookie=0 signature=ss error-name=n/a error-message=n/a
Got message type=method_return sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a  cookie=4 reply_cookie=1 signature=o error-name=n/a error-message=n/a
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=GetUnit cookie=2 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Got message type=method_return sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a  cookie=11 reply_cookie=2 signature=o error-name=n/a error-message=n/a
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1/unit/systemd_2dremount_2dfs_2eservice interface=org.freedesktop.DBus.Properties member=Get cookie=3 reply_cookie=0 signature=ss error-name=n/a error-message=n/a
Got message type=method_return sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a  cookie=12 reply_cookie=3 signature=v error-name=n/a error-message=n/a
Adding /org/freedesktop/systemd1/job/10111 to the set
Got message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dremount_2dfs_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged  cookie=1 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Got message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dremount_2dfs_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged  cookie=2 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Got message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=JobNew  cookie=3 reply_cookie=0 signature=uos error-name=n/a error-message=n/a
Got message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dremount_2dfs_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged  cookie=5 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Got message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dremount_2dfs_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged  cookie=6 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Got message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged  cookie=7 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Got message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dremount_2dfs_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged  cookie=8 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Got message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dremount_2dfs_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged  cookie=9 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Got message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/job/10111 interface=org.freedesktop.DBus.Properties member=PropertiesChanged  cookie=10 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Got message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dremount_2dfs_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged  cookie=13 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Got message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dremount_2dfs_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged  cookie=14 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Got message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged  cookie=15 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Got message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dremount_2dfs_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged  cookie=16 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Got message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dremount_2dfs_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged  cookie=17 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Got message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=JobRemoved  cookie=18 reply_cookie=0 signature=uoss error-name=n/a error-message=n/a
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1/unit/systemd_2dremount_2dfs_2eservice interface=org.freedesktop.DBus.Properties member=Get cookie=4 reply_cookie=0 signature=ss error-name=n/a error-message=n/a
Got message type=method_return sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a  cookie=21 reply_cookie=4 signature=v error-name=n/a error-message=n/a
Job for systemd-remount-fs.service failed because the control process exited with error code.
See "systemctl status systemd-remount-fs.service" and "journalctl -xeu systemd-remount-fs.service" for details.
Unexpected job result 'failed' for unit 'systemd-remount-fs.service', assuming server side newer than us.
Got result failed/Mídia não encontrada for job systemd-remount-fs.service.
Bus n/a: changing state RUNNING → CLOSED

journalctl -xe :

out 14 02:56:43 firelink audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=fprintd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? ter>
out 14 02:56:46 firelink audit[5250]: USER_AUTH pid=5250 uid=1000 auid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=pam_unix acct="gabrielelyas" exe=>
out 14 02:56:46 firelink audit[5250]: USER_ACCT pid=5250 uid=1000 auid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:accounting grantors=pam_unix acct="gabrielelyas" exe="/us>
out 14 02:56:46 firelink systemd[1]: Starting systemd-remount-fs.service - Remount Root and Kernel File Systems...
░░ Subject: Unidade systemd-remount-fs.service sendo iniciado
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A unidade systemd-remount-fs.service está sendo iniciada.
out 14 02:56:46 firelink systemd-remount-fs[5270]: mount: /: fsconfig system call failed: overlay: No changes allowed in reconfigure.
out 14 02:56:46 firelink systemd-remount-fs[5270]:        dmesg(1) may have more information after failed mount system call.
out 14 02:56:46 firelink systemd-remount-fs[5263]: /usr/bin/mount for / exited with exit status 32.
out 14 02:56:46 firelink systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ An ExecStart= process belonging to unit systemd-remount-fs.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
out 14 02:56:46 firelink systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit systemd-remount-fs.service has entered the 'failed' state with result 'exit-code'.
out 14 02:56:46 firelink systemd[1]: Failed to start systemd-remount-fs.service - Remount Root and Kernel File Systems.
░░ Subject: A unidade systemd-remount-fs.service falhou
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A unidade systemd-remount-fs.service falhou.

I think the problem is here:

out 14 02:56:46 firelink systemd-remount-fs[5263]: /usr/bin/mount for / exited with exit status 32 ??

[travier]

This is likely because you have the / mount point configured in your /etc/fstab and systemd tries to re-apply the config from there on the running system but it does not make sense anymore with composefs (did not make much sense before but was mostly harmless IIRC).

[travier]

We'll have to figure out a way to fix / migrate systems for that 🤔 . This is related to https://gitlab.com/fedora/ostree/sig/-/issues/35.