[BUG] toolbox broken after 2:3.2.0-0.1.rc1.fc34 update #166

bluebluesilver · 2021-05-07T15:39:33Z

Describe the bug
toolbox enter fails with Error: invalid entry point PID of container fedora-toolbox-34 on silverblue/34. Works on silverblue/rawhide and workstation/34.

To Reproduce
Please describe the steps needed to reproduce the bug:

Open terminal
run toolbox enter

Expected behavior
Toolbox entered

OS version:

$ rpm-ostree status -b
State: idle
BootedDeployment:
● ostree://fedora:fedora/34/x86_64/silverblue
                   Version: 34.20210507.0 (2021-05-07T00:45:36Z)
                BaseCommit: a0ca5647b4c7cead00648c21d9ed42ef9ddaaca8303b3162223dbb8fe586ca71
              GPGSignature: Valid signature by 8C5BA6990BDB26E19F2A1A801161AE6945719A39
           LayeredPackages: fedora-workstation-repositories gnome-boxes kitty kitty-terminfo
                            virt-manager

Additional context
Some coreOS issues (see here) suggested that failure to start might be because of the /etc/localtime file or link, so:

$ ls /etc | grep localtime; echo $?
1

And debug info for trying to enter in a gist.

The text was updated successfully, but these errors were encountered:

travier · 2021-05-07T15:58:34Z

Can you try:

$ podman start --attach fedora-toolbox-34
$ podman exec -ti fedora-toolbox-34

ma3yta · 2021-05-08T02:27:43Z

@travier I got problem with toolbox containers after last Silverblue OSTree update too.

ma3yta@localhost ~]$ podman start --attach dotnet
Error: unable to start container b2a64907dbefb2a22240b94bc88bd0c92540895b7a392370efa0fc7dc601dceb: 
opening file `` for writing: No such file or directory: OCI not found

[ma3yta@localhost ~]$ rpm-ostree db diff
ostree diff commit from: rollback deployment (603b88fc3a1131565b1818387da74fba898a052bd12088473e6529a71f050b49)
ostree diff commit to:   booted deployment (a2383e4c5ba7d99cd96fe016cec0aa529af6d27a88d33279b9afb3d87e083c14)
Upgraded:
  glibc 2.33-5.fc34 -> 2.33-8.fc34
  glibc-all-langpacks 2.33-5.fc34 -> 2.33-8.fc34
  glibc-common 2.33-5.fc34 -> 2.33-8.fc34
  glibc-devel 2.33-5.fc34 -> 2.33-8.fc34
  glibc-doc 2.33-5.fc34 -> 2.33-8.fc34
  glibc-headers-x86 2.33-5.fc34 -> 2.33-8.fc34
  ibus-typing-booster 2.11.2-4.fc34 -> 2.11.4-1.fc34
  kernel 5.11.17-300.fc34 -> 5.11.18-300.fc34
  kernel-core 5.11.17-300.fc34 -> 5.11.18-300.fc34
  kernel-devel 5.11.17-300.fc34 -> 5.11.18-300.fc34
  kernel-modules 5.11.17-300.fc34 -> 5.11.18-300.fc34
  kernel-modules-extra 5.11.17-300.fc34 -> 5.11.18-300.fc34
  less 575-2.fc34 -> 581.2-1.fc34
  nautilus 40.1-1.fc34 -> 40.1-2.fc34
  nautilus-extensions 40.1-1.fc34 -> 40.1-2.fc34
  podman 2:3.1.2-1.fc34 -> 2:3.2.0-0.1.rc1.fc34
  podman-plugins 2:3.1.2-1.fc34 -> 2:3.2.0-0.1.rc1.fc34
  zchunk-libs 1.1.9-2.fc34 -> 1.1.11-1.fc34

[ma3yta@localhost ~]$ toolbox --version
toolbox version 0.0.99.1
[ma3yta@localhost ~]$ podman --version
podman version 3.2.0-rc1

containers/podman#10274

bluebluesilver · 2021-05-08T10:34:59Z

$ podman start --attach fedora-toolbox-34 
{"msg":"exec container process `/usr/bin/toolbox`: Permission denied","level":"error","time":"2021-05-08T10:32:32.000150692Z"}
$ podman exec -ti fedora-toolbox-34
Error: must provide a non-empty command to start an exec session: invalid argument
$ podman exec -ti fedora-toolbox-34 bash
Error: can only create exec sessions on running containers: container state improper
$ toolbox --version
toolbox version 0.0.99.1
$ podman --version
podman version 3.1.2

juhp · 2021-05-09T18:57:57Z

(TBH, for this kind of reason I have a SB vm to test toolbox before rebooting my host into newer podman deployment.)

tpopela · 2021-05-10T09:33:11Z

Closing as it's reported as containers/podman#10274 - until it's fixed please downgrade the Toolbox with:

rpm-ostree override replace https://kojipkgs.fedoraproject.org//packages/podman/3.1.2/1.fc34/x86_64/podman-3.1.2-1.fc34.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/podman/3.1.2/1.fc34/x86_64/podman-plugins-3.1.2-1.fc34.x86_64.rpm

bluebluesilver · 2021-05-10T09:49:15Z

Thanks for the workaround.
Once the issue is fixed, how do I revert to non-overridden?
Is this sufficient:

rpm-ostree override reset podman podman-plugins

ondrejholy · 2021-05-10T09:51:51Z

sudo rpm-ostree override reset podman podman-plugins can be used for reversion.

bluebluesilver · 2021-05-10T10:14:51Z

I'd like to reopen this issue, as it's still a problem for me.

$ toolbox rm --all; toolbox enter -y
Error: invalid entry point PID of container fedora-toolbox-34

$ rpm-ostree status -b
State: idle
BootedDeployment:
● ostree://fedora:fedora/34/x86_64/testing/silverblue
                   Version: 34.20210508.0 (2021-05-08T02:04:12Z)
                BaseCommit: 7b460da9c2ccd2519aa6756435eaf8bb6fb1c1253f29e870ded830419e37d3c6
              GPGSignature: Valid signature by 8C5BA6990BDB26E19F2A1A801161AE6945719A39
      ReplacedBasePackages: podman podman-plugins 2:3.2.0-0.1.rc1.fc34 -> 2:3.1.2-1.fc34
           LayeredPackages: kitty kitty-terminfo

$ ls /etc/localtime
ls: cannot access '/etc/localtime': No such file or directory

$ podman start --attach fedora-toolbox-34
{"msg":"exec container process `/usr/bin/toolbox`: Permission denied","level":"error","time":"2021-05-10T10:10:48.000626951Z"}

$ podman exec -ti fedora-toolbox-34 bash
Error: can only create exec sessions on running containers: container state improper

bluebluesilver · 2021-05-10T10:40:31Z

Tried going back to a previos version with rpm-ostree deploy 34.20210420.0 where

$ rpm -qa podman*
podman-plugins-3.1.1-2.fc34.x86_64
podman-3.1.1-2.fc34.x86_64

Same exact issue.

bluebluesilver · 2021-05-10T10:59:17Z

This seems to be a bug in the 34 toolbox.

$ for n in 33 34 35; do echo -n "$n "; podman run -ti fedora-toolbox:$n echo WORKING; done
33 WORKING
34 Error: open executable: Permission denied: OCI permission denied
35 WORKING

Additional info
Tested with #166 (comment), #166 (comment) and

$ rpm -qa podman*
podman-plugins-3.2.0-0.1.rc1.fc34.x86_64
podman-3.2.0-0.1.rc1.fc34.x86_64

$ rpm-ostree status -b
State: idle
BootedDeployment:
● ostree://fedora:fedora/34/x86_64/testing/silverblue
                   Version: 34.20210509.0 (2021-05-09T01:48:12Z)
                BaseCommit: 4eef9f48a5f529ed13ecac714c32624122b431dba194b3b09018d082fee35734
              GPGSignature: Valid signature by 8C5BA6990BDB26E19F2A1A801161AE6945719A39
           LayeredPackages: kitty kitty-terminfo

tpopela · 2021-05-10T11:58:31Z

I'd like to reopen this issue, as it's still a problem for me.

This is a different issue, please report in https://github.com/containers/toolbox/issues

travier · 2021-05-10T16:50:41Z

Opened containers/podman#10296 so that hopefully at some point we don't get those regressions anymore

JamesBelchamber · 2021-05-14T08:55:30Z

Should this be closed while the issue still exists? I can see that the related Red Hat bug is open: https://bugzilla.redhat.com/show_bug.cgi?id=1958546

The bug has been fixed upstream but a release has not been made, which means no fix in Fedora/Silverblue. But the rc shouldn't have landed up in Fedora anyway 😅

Keeping this bug open will allow people to track it, and know when it should be fixed :)

tpopela · 2021-05-14T09:17:59Z

From my point of view yes. This issue tracker serves as a proxy. What is reported somewhere else (preferably in upstream or in RHBZ) we try to link it from there and close the issue. One might say that what you are proposing is the right way, but we do have limited resources.

juhp · 2021-05-17T03:10:37Z

podman-3.1.2-3.fc34 was pushed stable

tpopela closed this as completed May 10, 2021

tpopela mentioned this issue May 10, 2021

[BUG] broken release candidate of podman in stable repo #168

Closed

tpopela changed the title ~~[BUG] toolbox broken~~ [BUG] toolbox broken after 2:3.2.0-0.1.rc1.fc34 update May 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] toolbox broken after 2:3.2.0-0.1.rc1.fc34 update #166

[BUG] toolbox broken after 2:3.2.0-0.1.rc1.fc34 update #166

bluebluesilver commented May 7, 2021 •

edited

Loading

travier commented May 7, 2021

ma3yta commented May 8, 2021 •

edited

Loading

bluebluesilver commented May 8, 2021

juhp commented May 9, 2021 •

edited

Loading

tpopela commented May 10, 2021

bluebluesilver commented May 10, 2021 •

edited

Loading

ondrejholy commented May 10, 2021 •

edited

Loading

bluebluesilver commented May 10, 2021

bluebluesilver commented May 10, 2021

bluebluesilver commented May 10, 2021

tpopela commented May 10, 2021

travier commented May 10, 2021

JamesBelchamber commented May 14, 2021

tpopela commented May 14, 2021

juhp commented May 17, 2021

[BUG] toolbox broken after 2:3.2.0-0.1.rc1.fc34 update #166

[BUG] toolbox broken after 2:3.2.0-0.1.rc1.fc34 update #166

Comments

bluebluesilver commented May 7, 2021 • edited Loading

travier commented May 7, 2021

ma3yta commented May 8, 2021 • edited Loading

bluebluesilver commented May 8, 2021

juhp commented May 9, 2021 • edited Loading

tpopela commented May 10, 2021

bluebluesilver commented May 10, 2021 • edited Loading

ondrejholy commented May 10, 2021 • edited Loading

bluebluesilver commented May 10, 2021

bluebluesilver commented May 10, 2021

bluebluesilver commented May 10, 2021

tpopela commented May 10, 2021

travier commented May 10, 2021

JamesBelchamber commented May 14, 2021

tpopela commented May 14, 2021

juhp commented May 17, 2021

bluebluesilver commented May 7, 2021 •

edited

Loading

ma3yta commented May 8, 2021 •

edited

Loading

juhp commented May 9, 2021 •

edited

Loading

bluebluesilver commented May 10, 2021 •

edited

Loading

ondrejholy commented May 10, 2021 •

edited

Loading