Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docker exec format error after image commit #868

Open
ChGen opened this issue Nov 13, 2024 · 3 comments
Open

Docker exec format error after image commit #868

ChGen opened this issue Nov 13, 2024 · 3 comments

Comments

@ChGen
Copy link

ChGen commented Nov 13, 2024 •
edited
Loading

After committing finished container, which was executed via sysbox, resulting image cannot be run in docker with exec format error errror. Reproduction steps are below:

$ docker run -it ubuntu:22.04
root@9c7b6a965eb1:/# echo $HOSTNAME > test1
root@9c7b6a965eb1:/# cat test1
9c7b6a965eb1

$ docker commit 9c test1
sha256:fc5cba69c60daa6c93d4f60e45e8c867a09bc919c958d9129ece3bc49be494e2
$ docker run test1 cat test1
9c7b6a965eb1

$ docker run --runtime=sysbox-runc -it ubuntu:22.04
root@4d15241bcdff:/# echo $HOSTNAME > test1
root@4d15241bcdff:/# cat test1
4d15241bcdff
root@4d15241bcdff:/# exit
exit

$ docker commit 4d152 test2
sha256:43f577f64b8af01426fd721d71a8d95c3a5643e0d7ac9feb2872215666c0c4dd
$ docker run test2 cat test1
exec /usr/bin/cat: exec format error
$ docker run --runtime=sysbox-runc test2 cat test1
standard_init_linux.go:344: exec user process: name = /usr/bin/cat, args = [cat test1], environ = [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=04844fee8813 HOME=] caused: exec format error

$ docker run --runtime=sysbox-runc test1 cat test1
9c7b6a965eb1

$ docker images
REPOSITORY   TAG       IMAGE ID       CREATED              SIZE
test2        latest    43f577f64b8a   56 seconds ago       160MB
<none>       <none>    90a864902a28   About a minute ago   160MB
test1        latest    fc5cba69c60d   2 minutes ago        77.9MB
ubuntu       22.04     97271d29cb79   2 months ago         77.9MB

System information and logs:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.5 LTS
Release:        22.04
Codename:       jammy
$ uname -a
Linux EGRAMM16 5.15.153.1-microsoft-standard-WSL2 #1 SMP Fri Mar 29 23:14:13 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
$ docker --version
Docker version 24.0.7, build 24.0.7-0ubuntu2~22.04.1
sysbox-mgr --version
sysbox-mgr
		edition:        Community Edition (CE)
		version:        0.6.5
		commit:         1159d228eac8402efa63bd2cb18cdf9e404ea130
		built at:       Sat Nov  9 06:10:05 UTC 2024
		built by:       Rodny Molina

sudo systemctl status sysbox -n20
Failed to dump process list for 'sysbox.service', ignoring: Input/output error
● sysbox.service - Sysbox container runtime
     Loaded: loaded (/lib/systemd/system/sysbox.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-11-13 14:11:33 MSK; 33s ago
       Docs: https://github.com/nestybox/sysbox
   Main PID: 12265 (sh)
      Tasks: 4 (limit: 19000)
     Memory: 948.0K
        CPU: 15ms
     CGroup: /system.slice/sysbox.service

Nov 13 14:11:33 EGRAMM16 systemd[1]: Started Sysbox container runtime.
Nov 13 14:11:33 EGRAMM16 sh[12266]: sysbox-runc
Nov 13 14:11:33 EGRAMM16 sh[12266]:         edition:         Community Edition (CE)
Nov 13 14:11:33 EGRAMM16 sh[12266]:         version:         0.6.5
Nov 13 14:11:33 EGRAMM16 sh[12266]:         commit:         1b440ff266841f3d2d296e664122a9e29ceb9fd8
Nov 13 14:11:33 EGRAMM16 sh[12266]:         built at:         Sat Nov  9 06:09:34 UTC 2024
Nov 13 14:11:33 EGRAMM16 sh[12266]:         built by:         Rodny Molina
Nov 13 14:11:33 EGRAMM16 sh[12266]:         oci-specs:         1.1.0+dev
Nov 13 14:11:33 EGRAMM16 sh[12273]: sysbox-mgr
Nov 13 14:11:33 EGRAMM16 sh[12273]:         edition:         Community Edition (CE)
Nov 13 14:11:33 EGRAMM16 sh[12273]:         version:         0.6.5
Nov 13 14:11:33 EGRAMM16 sh[12273]:         commit:         1159d228eac8402efa63bd2cb18cdf9e404ea130
Nov 13 14:11:33 EGRAMM16 sh[12273]:         built at:         Sat Nov  9 06:10:05 UTC 2024
Nov 13 14:11:33 EGRAMM16 sh[12273]:         built by:         Rodny Molina
Nov 13 14:11:33 EGRAMM16 sh[12279]: sysbox-fs
Nov 13 14:11:33 EGRAMM16 sh[12279]:         edition:         Community Edition (CE)
Nov 13 14:11:33 EGRAMM16 sh[12279]:         version:         0.6.5
Nov 13 14:11:33 EGRAMM16 sh[12279]:         commit:         aeba775e52cc6385fa4807c594fc7ee164ad624c
Nov 13 14:11:33 EGRAMM16 sh[12279]:         built at:         Sat Nov  9 06:10:01 UTC 2024
Nov 13 14:11:33 EGRAMM16 sh[12279]:         built by:         Rodny Molina


docker info | grep -i runtime
 Runtimes: io.containerd.runc.v2 runc sysbox-runc
 Default Runtime: runc

cat /etc/docker/daemon.json
{
    "runtimes": {
        "sysbox-runc": {
            "path": "/usr/bin/sysbox-runc"
        }
    },
    "bip": "172.20.0.1/16",
    "default-address-pools": [
        {
            "base": "172.25.0.0/16",
            "size": 24
        }
    ]
}

Also reproduced on Ubuntu 24.04.1 LTS under WSL2.
@ctalledo
Copy link
Member

ctalledo commented Dec 5, 2024 •
edited
Loading

Hi @ChGen, thanks for filing the issue and apologies for the late reply.

I am not able to repro on my Ubuntu Jammy host (kernel 6.8) with Sysbox v0.6.5.

Can you show me the kernel version you have? (i.e., uname -a).

Also can you paste the output of findmnt from inside the Sysbox container?

$ docker run --runtime=sysbox-runc -it ubuntu:22.04
root@4d15241bcdff:/# findmnt

Finally, please show me the sysbox-mgr logs (journalctl -u sysbox-mgr), particularly the first log lines after sysbox starts. For example:

Nov 25 22:20:47 lenovo systemd[1]: Starting sysbox-mgr (part of the Sysbox container runtime)...
Nov 25 22:20:47 lenovo sysbox-mgr[1086]: time="2024-11-25 22:20:47" level=info msg="Starting ..."
Nov 25 22:20:47 lenovo sysbox-mgr[1086]: time="2024-11-25 22:20:47" level=info msg="Sysbox data root: /var/lib/sysbox"
Nov 25 22:20:47 lenovo sysbox-mgr[1086]: time="2024-11-25 22:20:47" level=info msg="Shiftfs module found in kernel: no"
Nov 25 22:20:47 lenovo sysbox-mgr[1086]: time="2024-11-25 22:20:47" level=info msg="Shiftfs works properly: no"
Nov 25 22:20:47 lenovo sysbox-mgr[1086]: time="2024-11-25 22:20:47" level=info msg="Shiftfs-on-overlayfs works properly: no"
Nov 25 22:20:47 lenovo sysbox-mgr[1086]: time="2024-11-25 22:20:47" level=info msg="ID-mapped mounts supported by kernel: yes"
Nov 25 22:20:47 lenovo sysbox-mgr[1086]: time="2024-11-25 22:20:47" level=info msg="Overlayfs on ID-mapped mounts supported by kernel: yes"
Nov 25 22:20:47 lenovo sysbox-mgr[1086]: time="2024-11-25 22:20:47" level=info msg="Operating in system container mode."
Nov 25 22:20:47 lenovo sysbox-mgr[1086]: time="2024-11-25 22:20:47" level=info msg="Relaxed read-only mode disabled."
Nov 25 22:20:47 lenovo sysbox-mgr[1086]: time="2024-11-25 22:20:47" level=info msg="Inner container image preloading enabled."
Nov 25 22:20:47 lenovo sysbox-mgr[1086]: time="2024-11-25 22:20:47" level=info msg="Listening on /run/sysbox/sysmgr.sock"
Nov 25 22:20:47 lenovo sysbox-mgr[1086]: time="2024-11-25 22:20:47" level=info msg="Ready ..."

Thanks.

@ChGen
Copy link
Author

ChGen commented Dec 6, 2024

@ctalledo , thank you for response.
Here's some extra info:

1. Kernel and its command line:

$ uname -a
Linux EGRAMM16 5.15.153.1-microsoft-standard-WSL2 #1 SMP Fri Mar 29 23:14:13 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

$ cat /proc/cmdline
initrd=\initrd.img WSL_ROOT_INIT=1 panic=-1 nr_cpus=16 hv_utils.timesync_implicit=1 console=hvc0 debug pty.legacy_count=0 WSL_ENABLE_CRASH_DUMP=1 mitigations=off cgroup_no_v1=all sysctl.vm.swappiness=10 sysctl.vm.vfs_cache_pressure=50

2. Also, I've uploaded corrupted docker image, in case it may help somehow:

docker pull chgena/sysbox-test2

3. findmnt output:

$ docker run --runtime=sysbox-runc -it ubuntu:22.04
root@eba774de1316:/# findmnt
TARGET                   SOURCE                                        FSTYPE   OPTIONS
/                        overlay                                       overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/MXNJVGZ4AS7GPLKQANM
|-/sys                   sysfs                                         sysfs    rw,nosuid,nodev,noexec,relatime
| |-/sys/firmware        tmpfs                                         tmpfs    ro,relatime,uid=165536,gid=165536
| |-/sys/fs/cgroup       cgroup                                        cgroup2  rw,nosuid,nodev,noexec,relatime,nsdelegate
| |-/sys/devices/virtual sysboxfs[/sys/devices/virtual]                fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,a
| |-/sys/kernel          sysboxfs[/sys/kernel]                         fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,a
| `-/sys/module/nf_conntrack/parameters
|                        sysboxfs[/sys/module/nf_conntrack/parameters] fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,a
|-/proc                  proc                                          proc     rw,nosuid,nodev,noexec,relatime
| |-/proc/bus            proc[/bus]                                    proc     ro,nosuid,nodev,noexec,relatime
| |-/proc/fs             proc[/fs]                                     proc     ro,nosuid,nodev,noexec,relatime
| |-/proc/irq            proc[/irq]                                    proc     ro,nosuid,nodev,noexec,relatime
| |-/proc/acpi           tmpfs                                         tmpfs    ro,relatime,uid=165536,gid=165536
| |-/proc/keys           none[/null]                                   devtmpfs rw,nosuid,relatime,size=8106836k,nr_inodes=2026709,mode=755
| |-/proc/timer_list     none[/null]                                   devtmpfs rw,nosuid,relatime,size=8106836k,nr_inodes=2026709,mode=755
| |-/proc/swaps          sysboxfs[/proc/swaps]                         fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,a
| |-/proc/sys            sysboxfs[/proc/sys]                           fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,a
| `-/proc/uptime         sysboxfs[/proc/uptime]                        fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,a
|-/dev                   tmpfs                                         tmpfs    rw,nosuid,size=65536k,mode=755,uid=165536,gid=165536
| |-/dev/console         devpts[/0]                                    devpts   rw,nosuid,noexec,relatime,gid=165541,mode=620,ptmxmode=666
| |-/dev/mqueue          mqueue                                        mqueue   rw,nosuid,nodev,noexec,relatime
| |-/dev/pts             devpts                                        devpts   rw,nosuid,noexec,relatime,gid=165541,mode=620,ptmxmode=666
| |-/dev/shm             shm                                           tmpfs    rw,nosuid,nodev,noexec,relatime,size=65536k,uid=165536,gid=165536
| |-/dev/null            none[/null]                                   devtmpfs rw,nosuid,relatime,size=8106836k,nr_inodes=2026709,mode=755
| |-/dev/random          none[/random]                                 devtmpfs rw,nosuid,relatime,size=8106836k,nr_inodes=2026709,mode=755
| |-/dev/kmsg            none[/null]                                   devtmpfs rw,nosuid,relatime,size=8106836k,nr_inodes=2026709,mode=755
| |-/dev/full            none[/full]                                   devtmpfs rw,nosuid,relatime,size=8106836k,nr_inodes=2026709,mode=755
| |-/dev/tty             none[/tty]                                    devtmpfs rw,nosuid,relatime,size=8106836k,nr_inodes=2026709,mode=755
| |-/dev/zero            none[/zero]                                   devtmpfs rw,nosuid,relatime,size=8106836k,nr_inodes=2026709,mode=755
| `-/dev/urandom         none[/urandom]                                devtmpfs rw,nosuid,relatime,size=8106836k,nr_inodes=2026709,mode=755
|-/var/lib/docker        /dev/sdd[/var/lib/sysbox/docker/eba774de1316ef1a00920196b1d0fc7f5143be70b917993b340e8b6ee47c2e47]
|                                                                      ext4     rw,relatime,discard,errors=remount-ro,data=ordered
|-/etc/resolv.conf       /dev/sdd[/var/lib/docker/containers/eba774de1316ef1a00920196b1d0fc7f5143be70b917993b340e8b6ee47c2e47/resolv.conf]
|                                                                      ext4     rw,relatime,idmapped,discard,errors=remount-ro,data=ordered
|-/etc/hostname          /dev/sdd[/var/lib/docker/containers/eba774de1316ef1a00920196b1d0fc7f5143be70b917993b340e8b6ee47c2e47/hostname]
|                                                                      ext4     rw,relatime,idmapped,discard,errors=remount-ro,data=ordered
|-/etc/hosts             /dev/sdd[/var/lib/docker/containers/eba774de1316ef1a00920196b1d0fc7f5143be70b917993b340e8b6ee47c2e47/hosts]
|                                                                      ext4     rw,relatime,idmapped,discard,errors=remount-ro,data=ordered
|-/var/lib/rancher/k3s   /dev/sdd[/var/lib/sysbox/rancher-k3s/eba774de1316ef1a00920196b1d0fc7f5143be70b917993b340e8b6ee47c2e47]
|                                                                      ext4     rw,relatime,discard,errors=remount-ro,data=ordered
|-/var/lib/rancher/rke2  /dev/sdd[/var/lib/sysbox/rancher-rke2/eba774de1316ef1a00920196b1d0fc7f5143be70b917993b340e8b6ee47c2e47]
|                                                                      ext4     rw,relatime,discard,errors=remount-ro,data=ordered
|-/var/lib/kubelet       /dev/sdd[/var/lib/sysbox/kubelet/eba774de1316ef1a00920196b1d0fc7f5143be70b917993b340e8b6ee47c2e47]
|                                                                      ext4     rw,relatime,discard,errors=remount-ro,data=ordered
|-/var/lib/k0s           /dev/sdd[/var/lib/sysbox/k0s/eba774de1316ef1a00920196b1d0fc7f5143be70b917993b340e8b6ee47c2e47]
|                                                                      ext4     rw,relatime,discard,errors=remount-ro,data=ordered
|-/var/lib/buildkit      /dev/sdd[/var/lib/sysbox/buildkit/eba774de1316ef1a00920196b1d0fc7f5143be70b917993b340e8b6ee47c2e47]
|                                                                      ext4     rw,relatime,discard,errors=remount-ro,data=ordered
|-/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs
|                        /dev/sdd[/var/lib/sysbox/containerd/eba774de1316ef1a00920196b1d0fc7f5143be70b917993b340e8b6ee47c2e47]
|                                                                      ext4     rw,relatime,discard,errors=remount-ro,data=ordered
`-/usr/lib/modules/5.15.153.1-microsoft-standard-WSL2
                         none                                          overlay  ro,nosuid,nodev,noatime,lowerdir=/modules,upperdir=/lib/modules/5.1
root@eba774de1316:/#

4. sysbox-mgr logs:

$ journalctl -u sysbox-mgr|cat
Nov 13 14:11:33 EGRAMM16 systemd[1]: Starting sysbox-mgr (part of the Sysbox container runtime)...
Nov 13 14:11:33 EGRAMM16 sysbox-mgr[12245]: time="2024-11-13 14:11:33" level=info msg="Starting ..."
Nov 13 14:11:33 EGRAMM16 sysbox-mgr[12245]: time="2024-11-13 14:11:33" level=info msg="Sysbox data root: /var/lib/sysbox"
Nov 13 14:11:33 EGRAMM16 sysbox-mgr[12245]: time="2024-11-13 14:11:33" level=warning msg="No kernel-headers found in host filesystem at /usr/src/linux-headers-5.15.153.1-microsoft-standard-WSL2. No headers will be mounted inside any of the containers."
Nov 13 14:11:33 EGRAMM16 sysbox-mgr[12245]: time="2024-11-13 14:11:33" level=info msg="Shiftfs module found in kernel: no"
Nov 13 14:11:33 EGRAMM16 sysbox-mgr[12245]: time="2024-11-13 14:11:33" level=info msg="Shiftfs works properly: no"
Nov 13 14:11:33 EGRAMM16 sysbox-mgr[12245]: time="2024-11-13 14:11:33" level=info msg="Shiftfs-on-overlayfs works properly: no"
Nov 13 14:11:33 EGRAMM16 sysbox-mgr[12245]: time="2024-11-13 14:11:33" level=info msg="ID-mapped mounts supported by kernel: yes"
Nov 13 14:11:33 EGRAMM16 sysbox-mgr[12245]: time="2024-11-13 14:11:33" level=info msg="Overlayfs on ID-mapped mounts supported by kernel: no"
Nov 13 14:11:33 EGRAMM16 sysbox-mgr[12245]: time="2024-11-13 14:11:33" level=info msg="Operating in system container mode."
Nov 13 14:11:33 EGRAMM16 sysbox-mgr[12245]: time="2024-11-13 14:11:33" level=info msg="Relaxed read-only mode disabled."
Nov 13 14:11:33 EGRAMM16 sysbox-mgr[12245]: time="2024-11-13 14:11:33" level=info msg="Inner container image preloading enabled."
Nov 13 14:11:33 EGRAMM16 sysbox-mgr[12245]: time="2024-11-13 14:11:33" level=info msg="Listening on /run/sysbox/sysmgr.sock"
Nov 13 14:11:33 EGRAMM16 sysbox-mgr[12245]: time="2024-11-13 14:11:33" level=info msg="Ready ..."
Nov 13 14:11:33 EGRAMM16 systemd[1]: Started sysbox-mgr (part of the Sysbox container runtime).
Nov 13 14:14:17 EGRAMM16 systemd[1]: sysbox-mgr.service: Failed to kill control group /system.slice/sysbox-mgr.service, ignoring: Input/output error
Nov 13 14:14:17 EGRAMM16 systemd[1]: Stopping sysbox-mgr (part of the Sysbox container runtime)...
Nov 13 14:14:17 EGRAMM16 sysbox-mgr[12245]: time="2024-11-13 14:14:17" level=info msg="Caught OS signal: terminated"
Nov 13 14:14:17 EGRAMM16 sysbox-mgr[12245]: time="2024-11-13 14:14:17" level=info msg="Stopping (gracefully) ..."
Nov 13 14:14:17 EGRAMM16 sysbox-mgr[12245]: time="2024-11-13 14:14:17" level=info msg=Stopped.
Nov 13 14:14:17 EGRAMM16 sysbox-mgr[12245]: time="2024-11-13 14:14:17" level=info msg=Exiting.
Nov 13 14:14:17 EGRAMM16 systemd[1]: sysbox-mgr.service: Failed to kill control group /system.slice/sysbox-mgr.service, ignoring: Input/output error
Nov 13 14:14:17 EGRAMM16 systemd[1]: sysbox-mgr.service: Failed to kill control group /system.slice/sysbox-mgr.service, ignoring: Input/output error
Nov 13 14:14:17 EGRAMM16 systemd[1]: sysbox-mgr.service: Deactivated successfully.
Nov 13 14:14:17 EGRAMM16 systemd[1]: Stopped sysbox-mgr (part of the Sysbox container runtime).
Nov 13 14:14:17 EGRAMM16 systemd[1]: sysbox-mgr.service: Consumed 1.400s CPU time.
Nov 13 14:14:17 EGRAMM16 systemd[1]: Starting sysbox-mgr (part of the Sysbox container runtime)...
Nov 13 14:14:17 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:14:17" level=info msg="Starting ..."
Nov 13 14:14:17 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:14:17" level=info msg="Sysbox data root: /var/lib/sysbox"
Nov 13 14:14:17 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:14:17" level=warning msg="No kernel-headers found in host filesystem at /usr/src/linux-headers-5.15.153.1-microsoft-standard-WSL2. No headers will be mounted inside any of the containers."
Nov 13 14:14:17 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:14:17" level=info msg="Shiftfs module found in kernel: no"
Nov 13 14:14:17 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:14:17" level=info msg="Shiftfs works properly: no"
Nov 13 14:14:17 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:14:17" level=info msg="Shiftfs-on-overlayfs works properly: no"
Nov 13 14:14:17 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:14:17" level=info msg="ID-mapped mounts supported by kernel: yes"
Nov 13 14:14:17 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:14:17" level=info msg="Overlayfs on ID-mapped mounts supported by kernel: no"
Nov 13 14:14:17 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:14:17" level=info msg="Operating in system container mode."
Nov 13 14:14:17 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:14:17" level=info msg="Relaxed read-only mode disabled."
Nov 13 14:14:17 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:14:17" level=info msg="Inner container image preloading enabled."
Nov 13 14:14:17 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:14:17" level=info msg="Listening on /run/sysbox/sysmgr.sock"
Nov 13 14:14:17 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:14:17" level=info msg="Ready ..."
Nov 13 14:14:17 EGRAMM16 systemd[1]: Started sysbox-mgr (part of the Sysbox container runtime).
Nov 13 14:20:19 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:20:19" level=info msg="registered new container 4d15241bcdff"
Nov 13 14:20:34 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:20:34" level=info msg="unregistered container 4d15241bcdff"
Nov 13 14:21:30 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:21:30" level=info msg="registered new container 04844fee8813"
Nov 13 14:21:31 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:21:31" level=info msg="unregistered container 04844fee8813"
Nov 13 14:22:27 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:22:27" level=info msg="registered new container 9fd0de2429e0"
Nov 13 14:22:28 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:22:28" level=info msg="unregistered container 9fd0de2429e0"
Nov 13 14:24:33 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:24:33" level=info msg="registered new container a5d5cce99635"
Nov 13 14:24:34 EGRAMM16 sysbox-mgr[13217]: time="2024-11-13 14:24:34" level=info msg="unregistered container a5d5cce99635"
-- Boot 5ed96abf6e294ed6b8c9ae2f1e50d77f --
Dec 06 18:28:20 EGRAMM16 systemd[1]: Starting sysbox-mgr (part of the Sysbox container runtime)...
Dec 06 18:28:21 EGRAMM16 sysbox-mgr[259]: time="2024-12-06 18:28:21" level=info msg="Starting ..."
Dec 06 18:28:21 EGRAMM16 sysbox-mgr[259]: time="2024-12-06 18:28:21" level=info msg="Sysbox data root: /var/lib/sysbox"
Dec 06 18:28:22 EGRAMM16 sysbox-mgr[259]: time="2024-12-06 18:28:22" level=warning msg="No kernel-headers found in host filesystem at /usr/src/linux-headers-5.15.153.1-microsoft-standard-WSL2. No headers will be mounted inside any of the containers."
Dec 06 18:28:22 EGRAMM16 sysbox-mgr[259]: time="2024-12-06 18:28:22" level=info msg="Shiftfs module found in kernel: no"
Dec 06 18:28:22 EGRAMM16 sysbox-mgr[259]: time="2024-12-06 18:28:22" level=info msg="Shiftfs works properly: no"
Dec 06 18:28:22 EGRAMM16 sysbox-mgr[259]: time="2024-12-06 18:28:22" level=info msg="Shiftfs-on-overlayfs works properly: no"
Dec 06 18:28:22 EGRAMM16 sysbox-mgr[259]: time="2024-12-06 18:28:22" level=info msg="ID-mapped mounts supported by kernel: yes"
Dec 06 18:28:22 EGRAMM16 sysbox-mgr[259]: time="2024-12-06 18:28:22" level=info msg="Overlayfs on ID-mapped mounts supported by kernel: no"
Dec 06 18:28:22 EGRAMM16 sysbox-mgr[259]: time="2024-12-06 18:28:22" level=info msg="Operating in system container mode."
Dec 06 18:28:22 EGRAMM16 sysbox-mgr[259]: time="2024-12-06 18:28:22" level=info msg="Relaxed read-only mode disabled."
Dec 06 18:28:22 EGRAMM16 sysbox-mgr[259]: time="2024-12-06 18:28:22" level=info msg="Inner container image preloading enabled."
Dec 06 18:28:22 EGRAMM16 sysbox-mgr[259]: time="2024-12-06 18:28:22" level=info msg="Listening on /run/sysbox/sysmgr.sock"
Dec 06 18:28:22 EGRAMM16 sysbox-mgr[259]: time="2024-12-06 18:28:22" level=info msg="Ready ..."
Dec 06 18:28:22 EGRAMM16 systemd[1]: Started sysbox-mgr (part of the Sysbox container runtime).
Dec 06 22:22:27 EGRAMM16 sysbox-mgr[259]: time="2024-12-06 22:22:27" level=info msg="registered new container eba774de1316"
Dec 06 22:23:15 EGRAMM16 sysbox-mgr[259]: time="2024-12-06 22:23:15" level=info msg="unregistered container eba774de1316"

@ChGen
Copy link
Author

ChGen commented Dec 7, 2024 •
edited
Loading

Extra info:
I've updated my WSL2 and Ubuntu 22.04 to the latest components versions (see below), but this bug is still reproducible for me.

  1. Bug reproduction:
evgeny@EGRAMM16:~$ docker run --name t4 --runtime=sysbox-runc ubuntu:22.04
evgeny@EGRAMM16:~$ docker commit t4 test4
sha256:9c759f9b74b32574ad7ef2a7cc2c35a624d50e7fa1646531ae6a054b392f9cc1
evgeny@EGRAMM16:~$ docker run test4
exec /bin/bash: exec format error
evgeny@EGRAMM16:~$ docker run --name t5 ubuntu:22.04
evgeny@EGRAMM16:~$ docker commit t5 test5
sha256:18bb704ad4babc38737625d253522d119be2d2b8be7617a1151b8a84389e57ca
evgeny@EGRAMM16:~$ docker run test5
evgeny@EGRAMM16:~$ echo $?
0
  1. Versions:
evgeny@EGRAMM16:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.5 LTS
Release:        22.04
Codename:       jammy

evgeny@EGRAMM16:~$ uname -a
Linux EGRAMM16 5.15.167.4-microsoft-standard-WSL2 #1 SMP Tue Nov 5 00:21:55 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

evgeny@EGRAMM16:~$ wsl.exe -v
WSL version: 2.3.26.0
Kernel version: 5.15.167.4-1
WSLg version: 1.0.65
MSRDC version: 1.2.5620
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.26100.1-240331-1435.ge-release
Windows version: 10.0.26100.2314

evgeny@EGRAMM16:~$ sysbox-mgr --version
sysbox-mgr
        edition:        Community Edition (CE)
        version:        0.6.5
        commit:         1159d228eac8402efa63bd2cb18cdf9e404ea130
        built at:       Sat Nov  9 06:10:05 UTC 2024
        built by:       Rodny Molina
  1. sysbox-mgr log:
-- Boot 554426091ab149fda93fb0587e4e4752 --
Dec 07 12:39:00 EGRAMM16 systemd[1]: Starting sysbox-mgr (part of the Sysbox container runtime)...
Dec 07 12:39:00 EGRAMM16 sysbox-mgr[239]: time="2024-12-07 12:39:00" level=info msg="Starting ..."
Dec 07 12:39:00 EGRAMM16 sysbox-mgr[239]: time="2024-12-07 12:39:00" level=info msg="Sysbox data root: /var/lib/sysbox"
Dec 07 12:39:00 EGRAMM16 sysbox-mgr[239]: time="2024-12-07 12:39:00" level=warning msg="No kernel-headers found in host filesystem at /usr/src/linux-headers-5.15.167.4-microsoft-standard-WSL2. No headers will be mounted inside any of the containers."
Dec 07 12:39:00 EGRAMM16 sysbox-mgr[239]: time="2024-12-07 12:39:00" level=info msg="Shiftfs module found in kernel: no"
Dec 07 12:39:00 EGRAMM16 sysbox-mgr[239]: time="2024-12-07 12:39:00" level=info msg="Shiftfs works properly: no"
Dec 07 12:39:00 EGRAMM16 sysbox-mgr[239]: time="2024-12-07 12:39:00" level=info msg="Shiftfs-on-overlayfs works properly: no"
Dec 07 12:39:00 EGRAMM16 sysbox-mgr[239]: time="2024-12-07 12:39:00" level=info msg="ID-mapped mounts supported by kernel: yes"
Dec 07 12:39:00 EGRAMM16 sysbox-mgr[239]: time="2024-12-07 12:39:00" level=info msg="Overlayfs on ID-mapped mounts supported by kernel: no"
Dec 07 12:39:00 EGRAMM16 sysbox-mgr[239]: time="2024-12-07 12:39:00" level=info msg="Operating in system container mode."
Dec 07 12:39:00 EGRAMM16 sysbox-mgr[239]: time="2024-12-07 12:39:00" level=info msg="Relaxed read-only mode disabled."
Dec 07 12:39:00 EGRAMM16 sysbox-mgr[239]: time="2024-12-07 12:39:00" level=info msg="Inner container image preloading enabled."
Dec 07 12:39:00 EGRAMM16 sysbox-mgr[239]: time="2024-12-07 12:39:00" level=info msg="Listening on /run/sysbox/sysmgr.sock"
Dec 07 12:39:00 EGRAMM16 sysbox-mgr[239]: time="2024-12-07 12:39:00" level=info msg="Ready ..."
Dec 07 12:39:00 EGRAMM16 systemd[1]: Started sysbox-mgr (part of the Sysbox container runtime).
Dec 07 12:39:45 EGRAMM16 sysbox-mgr[239]: time="2024-12-07 12:39:45" level=info msg="registered new container 9076f602f0b2"
Dec 07 12:39:57 EGRAMM16 sysbox-mgr[239]: time="2024-12-07 12:39:57" level=info msg="unregistered container 9076f602f0b2"
Dec 07 12:42:07 EGRAMM16 sysbox-mgr[239]: time="2024-12-07 12:42:07" level=info msg="registered new container 5329106bd39f"
Dec 07 12:42:07 EGRAMM16 sysbox-mgr[239]: time="2024-12-07 12:42:07" level=info msg="unregistered container 5329106bd39f"
  1. findmnt output:
evgeny@EGRAMM16:~$ docker run --runtime=sysbox-runc ubuntu:22.04 findmnt
TARGET                                                       SOURCE
                            FSTYPE   OPTIONS
/                                                            overlay
                            overlay  rw,relatime,lowerdir=/var/lib/docker/overlay2/l/YTGKTTQBWEYCYL24TV75BPEYZG:/var/lib/docker/overlay2/l/QNBNV7VWQW72CTWPAMAYN6VKCT,upperdir=/var/lib/sysbox/rootfs/e4e43aa883191921b2e1d2da34dcc7138a92cb93842efcc34226f46c51577529/overlay2/diff,workdir=/var/lib/sysbox/rootfs/e4e43aa883191921b2e1d2da34dcc7138a92cb93842efcc34226f46c51577529/overlay2/work,metacopy=on
|-/sys                                                       sysfs
                            sysfs    rw,nosuid,nodev,noexec,relatime
| |-/sys/firmware                                            tmpfs
                            tmpfs    ro,relatime,uid=165536,gid=165536
| |-/sys/fs/cgroup                                           cgroup
                            cgroup2  rw,nosuid,nodev,noexec,relatime,nsdelegate
| |-/sys/devices/virtual                                     sysboxfs[/sys/devices/virtual]
                            fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
| |-/sys/kernel                                              sysboxfs[/sys/kernel]
                            fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
| `-/sys/module/nf_conntrack/parameters                      sysboxfs[/sys/module/nf_conntrack/parameters]
                            fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
|-/proc                                                      proc
                            proc     rw,nosuid,nodev,noexec,relatime
| |-/proc/bus                                                proc[/bus]
                            proc     ro,nosuid,nodev,noexec,relatime
| |-/proc/fs                                                 proc[/fs]
                            proc     ro,nosuid,nodev,noexec,relatime
| |-/proc/irq                                                proc[/irq]
                            proc     ro,nosuid,nodev,noexec,relatime
| |-/proc/acpi                                               tmpfs
                            tmpfs    ro,relatime,uid=165536,gid=165536
| |-/proc/keys                                               none[/null]
                            devtmpfs rw,nosuid,relatime,size=8106836k,nr_inodes=2026709,mode=755
| |-/proc/timer_list                                         none[/null]
                            devtmpfs rw,nosuid,relatime,size=8106836k,nr_inodes=2026709,mode=755
| |-/proc/swaps                                              sysboxfs[/proc/swaps]
                            fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
| |-/proc/sys                                                sysboxfs[/proc/sys]
                            fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
| `-/proc/uptime                                             sysboxfs[/proc/uptime]
                            fuse     rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
|-/dev                                                       tmpfs
                            tmpfs    rw,nosuid,size=65536k,mode=755,uid=165536,gid=165536
| |-/dev/mqueue                                              mqueue
                            mqueue   rw,nosuid,nodev,noexec,relatime
| |-/dev/pts                                                 devpts
                            devpts   rw,nosuid,noexec,relatime,gid=165541,mode=620,ptmxmode=666
| |-/dev/shm                                                 shm
                            tmpfs    rw,nosuid,nodev,noexec,relatime,size=65536k,uid=165536,gid=165536
| |-/dev/null                                                none[/null]
                            devtmpfs rw,nosuid,relatime,size=8106836k,nr_inodes=2026709,mode=755
| |-/dev/random                                              none[/random]
                            devtmpfs rw,nosuid,relatime,size=8106836k,nr_inodes=2026709,mode=755
| |-/dev/kmsg                                                none[/null]
                            devtmpfs rw,nosuid,relatime,size=8106836k,nr_inodes=2026709,mode=755
| |-/dev/full                                                none[/full]
                            devtmpfs rw,nosuid,relatime,size=8106836k,nr_inodes=2026709,mode=755
| |-/dev/tty                                                 none[/tty]
                            devtmpfs rw,nosuid,relatime,size=8106836k,nr_inodes=2026709,mode=755
| |-/dev/zero                                                none[/zero]
                            devtmpfs rw,nosuid,relatime,size=8106836k,nr_inodes=2026709,mode=755
| `-/dev/urandom                                             none[/urandom]
                            devtmpfs rw,nosuid,relatime,size=8106836k,nr_inodes=2026709,mode=755
|-/var/lib/docker                                            /dev/sdc[/var/lib/sysbox/docker/e4e43aa883191921b2e1d2da34dcc7138a92cb93842efcc34226f46c51577529]                 ext4     rw,relatime,discard,errors=remount-ro,data=ordered
|-/etc/resolv.conf                                           /dev/sdc[/var/lib/docker/containers/e4e43aa883191921b2e1d2da34dcc7138a92cb93842efcc34226f46c51577529/resolv.conf] ext4     rw,relatime,idmapped,discard,errors=remount-ro,data=ordered
|-/etc/hostname                                              /dev/sdc[/var/lib/docker/containers/e4e43aa883191921b2e1d2da34dcc7138a92cb93842efcc34226f46c51577529/hostname]    ext4     rw,relatime,idmapped,discard,errors=remount-ro,data=ordered
|-/etc/hosts                                                 /dev/sdc[/var/lib/docker/containers/e4e43aa883191921b2e1d2da34dcc7138a92cb93842efcc34226f46c51577529/hosts]       ext4     rw,relatime,idmapped,discard,errors=remount-ro,data=ordered
|-/var/lib/rancher/k3s                                       /dev/sdc[/var/lib/sysbox/rancher-k3s/e4e43aa883191921b2e1d2da34dcc7138a92cb93842efcc34226f46c51577529]            ext4     rw,relatime,discard,errors=remount-ro,data=ordered
|-/var/lib/rancher/rke2                                      /dev/sdc[/var/lib/sysbox/rancher-rke2/e4e43aa883191921b2e1d2da34dcc7138a92cb93842efcc34226f46c51577529]           ext4     rw,relatime,discard,errors=remount-ro,data=ordered
|-/var/lib/kubelet                                           /dev/sdc[/var/lib/sysbox/kubelet/e4e43aa883191921b2e1d2da34dcc7138a92cb93842efcc34226f46c51577529]                ext4     rw,relatime,discard,errors=remount-ro,data=ordered
|-/var/lib/k0s                                               /dev/sdc[/var/lib/sysbox/k0s/e4e43aa883191921b2e1d2da34dcc7138a92cb93842efcc34226f46c51577529]                    ext4     rw,relatime,discard,errors=remount-ro,data=ordered
|-/var/lib/buildkit                                          /dev/sdc[/var/lib/sysbox/buildkit/e4e43aa883191921b2e1d2da34dcc7138a92cb93842efcc34226f46c51577529]               ext4     rw,relatime,discard,errors=remount-ro,data=ordered
|-/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs /dev/sdc[/var/lib/sysbox/containerd/e4e43aa883191921b2e1d2da34dcc7138a92cb93842efcc34226f46c51577529]             ext4     rw,relatime,discard,errors=remount-ro,data=ordered
`-/usr/lib/modules/5.15.167.4-microsoft-standard-WSL2        none
                            overlay  ro,nosuid,nodev,noatime,lowerdir=/modules,upperdir=/lib/modules/5.15.167.4-microsoft-standard-WSL2/rw/upper,workdir=/lib/modules/5.15.167.4-microsoft-standard-WSL2/rw/work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ctalledo @ChGen