toolbox failing when there is a container running on another terminal with different $HOME

[yilkalargaw]

I sometimes run toolbox in a specific directory to try out things without touching the home directory. To do this I usually change $HOME. When ever I do this and try to open my old toolboxes in another terminal window, toolbox fails with toolbox: failed to create /run/.toolboxenv in container dnfy-fedora.

I think it might have something todo with fuse-overlays because restarting the containers would not help.

How to recreate

mkdir tmptrial
$HOME=$HOME/tmptrial
toolbox create -c tmp_container
toolbox enter -c tmp_container

# now on a new terminal

cd ~ / toolbox enter -c good_old_container 

[HarryMichal]

Hi @yilkalargaw! I have a suspicion that this is a problem of Podman and not of Toolbox. I'll try to investigate a bit later on.

[juanje]

I saw this issue and I tried to reproduce it. I kinda did.
I couldn't enter into the container after the change of the $HOME, but it also happened without other toolbox container running.
Also, I end up with a container in where I can't enter, even from another terminal with the proper $HOME value.

Here are my podman and toolbox versions:

• podman: 2.0.2
• toolbox: 0.0.92

The OS is Fedora 32.20200712.0 (Silverblue).

Steps to reproduce the error:

• Create a new toolbox's container (and try it). This is not necessary, it fails with the default container as well, but makes the test cleaner:

[juanje@xps ~]$ toolbox create --container poc
Created container: poc
Enter with: toolbox enter --container poc
[juanje@xps ~]$ toolbox enter --container poc
⬢[juanje@toolbox ~]$ 

• Create a directory and set the $HOME value:

[juanje@xps ~]$ mkdir tmphome
[juanje@xps ~]$ cd tmphome
[juanje@xps tmphome]$ export HOME=$PWD
[juanje@xps ~]$ 

• Try to enter the poc container from that terminal:

[juanje@xps ~]$ toolbox enter --container poc
No toolbox containers found. Create now? [y/N] 
A container can be created later with the 'create' command.
Run 'toolbox --help' for usage.

NOTE: I pressed enter, so it didn't create a new container. It should exist.

• Now nor podman nor toolbox can see the previously created containers, at that terminal:

[juanje@xps ~]$ toolbox list
[juanje@xps ~]$ podman ps -a
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

But it can in another terminal:

[juanje@xps copr]$ toolbox list
IMAGE ID      IMAGE NAME                   CREATED
dc930c1469f5  localhost/fedora-toolbox:32  4 weeks ago

CONTAINER ID  CONTAINER NAME     CREATED         STATUS   IMAGE NAME
889cd51f190e  fedora-toolbox-32  15 hours ago    running  localhost/fedora-toolbox:32
57f36e6b3050  poc                7 minutes ago   running  localhost/fedora-toolbox:32
[juanje@xps copr]$ podman ps -a
CONTAINER ID  IMAGE                        COMMAND               CREATED         STATUS             PORTS   NAMES
57f36e6b3050  localhost/fedora-toolbox:32  toolbox --verbose...  7 minutes ago   Up 7 minutes ago           poc
889cd51f190e  localhost/fedora-toolbox:32  toolbox --verbose...  15 hours ago    Up 42 minutes ago          fedora-toolbox-32

• Now I change the HOME back to the normal value and try to enter the container:

[juanje@xps ~]$ export HOME=/var/home/juanje/
[juanje@xps tmphome]$ toolbox enter --container poc

Nothing happend... so I run with verbose the toolbox enter for that container (which before was working fine) I see this:

[juanje@xps tmphome]$ toolbox --verbose enter --container poc
DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Checking if /etc/subgid and /etc/subuid have entries for user juanje 
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Toolbox config directory is /var/home/juanje//.config/toolbox 
DEBU Current Podman version is 2.0.2              
DEBU Old Podman version is 2.0.2                  
DEBU Migration not needed: Podman version 2.0.2 is unchanged 
DEBU Resolving container and image names          
DEBU Container: 'poc'                             
DEBU Image: ''                                    
DEBU Release: ''                                  
DEBU Resolved container and image names           
DEBU Container: 'poc'                             
DEBU Image: 'fedora-toolbox:32'                   
DEBU Release: '32'                                
DEBU Checking if container poc exists             
DEBU Calling org.freedesktop.Flatpak.SessionHelper.RequestSession 
DEBU Starting container poc                       
DEBU Inspecting entry point of container poc      
DEBU Entry point PID is a float64                 
DEBU Entry point of container poc is toolbox (PID=175870) 
DEBU Waiting for container poc to finish initializing 
DEBU Checking if initialization stamp /run/user/1000/toolbox/container-initialized-175870 exists 
DEBU Container poc is initialized                 
DEBU Looking for command /bin/bash in container poc 
Error: unable to find user juanje: no matching entries in passwd file
DEBU command /bin/bash not found in container poc; using /bin/bash instead 
DEBU Creating list of environment variables to forward 
DEBU COLORTERM=truecolor                          
DEBU DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus 
DEBU DBUS_SYSTEM_BUS_ADDRESS is unset             
DEBU DESKTOP_SESSION=gnome-xorg                   
DEBU DISPLAY=:0                                   
DEBU LANG=es_ES.UTF-8                             
DEBU SHELL=/bin/bash                              
DEBU SSH_AUTH_SOCK=/run/user/1000/keyring/ssh     
DEBU TERM=xterm-256color                          
DEBU TOOLBOX_PATH=/usr/bin/toolbox                
DEBU VTE_VERSION=6003                             
DEBU WAYLAND_DISPLAY is unset                     
DEBU XDG_CURRENT_DESKTOP=GNOME                    
DEBU XDG_DATA_DIRS=/var/home/juanje/.local/share/flatpak/exports/share/:/var/lib/flatpak/exports/share/:/usr/local/share/:/usr/share/ 
DEBU XDG_MENU_PREFIX=gnome-                       
DEBU XDG_RUNTIME_DIR=/run/user/1000               
DEBU XDG_SEAT is unset                            
DEBU XDG_SESSION_DESKTOP=gnome-xorg               
DEBU XDG_SESSION_ID is unset                      
DEBU XDG_SESSION_TYPE=x11                         
DEBU XDG_VTNR is unset                            
DEBU Running in container poc:                    
DEBU podman                                       
DEBU --log-level                                  
DEBU error                                        
DEBU exec                                         
DEBU --interactive                                
DEBU --tty                                        
DEBU --user                                       
DEBU juanje                                       
DEBU --workdir                                    
DEBU /var/home/juanje/tmphome                     
DEBU --env=COLORTERM=truecolor                    
DEBU --env=DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus 
DEBU --env=DESKTOP_SESSION=gnome-xorg             
DEBU --env=DISPLAY=:0                             
DEBU --env=LANG=es_ES.UTF-8                       
DEBU --env=SHELL=/bin/bash                        
DEBU --env=SSH_AUTH_SOCK=/run/user/1000/keyring/ssh 
DEBU --env=TERM=xterm-256color                    
DEBU --env=TOOLBOX_PATH=/usr/bin/toolbox          
DEBU --env=VTE_VERSION=6003                       
DEBU --env=XDG_CURRENT_DESKTOP=GNOME              
DEBU --env=XDG_DATA_DIRS=/var/home/juanje/.local/share/flatpak/exports/share/:/var/lib/flatpak/exports/share/:/usr/local/share/:/usr/share/ 
DEBU --env=XDG_MENU_PREFIX=gnome-                 
DEBU --env=XDG_RUNTIME_DIR=/run/user/1000         
DEBU --env=XDG_SESSION_DESKTOP=gnome-xorg         
DEBU --env=XDG_SESSION_TYPE=x11                   
DEBU poc                                          
DEBU capsh                                        
DEBU --caps=                                      
DEBU --                                           
DEBU -c                                           
DEBU exec "$@"                                    
DEBU /bin/sh                                      
DEBU /bin/bash                                    
Error: unable to find user juanje: no matching entries in passwd file

• Now I try to remove the container and I can't.
  Right now I'm in the ~/tmphome directory with the proper $HOME set, but I can't remove the container. The same happens if I try from another terminal:

[juanje@xps tmphome]$ toolbox rm -f poc
Error: failed to remove container poc
[juanje@xps tmphome]$ toolbox --verbose rm -f poc
DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Checking if /etc/subgid and /etc/subuid have entries for user juanje 
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Toolbox config directory is /var/home/juanje//.config/toolbox 
DEBU Current Podman version is 2.0.2              
DEBU Old Podman version is 2.0.2                  
DEBU Migration not needed: Podman version 2.0.2 is unchanged 
DEBU Removing container poc                       
2020-07-14T16:33:45.000774986Z: kill process 175870: Operation not permitted
Error: cannot remove container 57f36e6b3050d79089e23b3213518515e5b8becb1c9864a14cfe7d3788b592cc as it could not be stopped: operation not permitted
Error: failed to remove container poc

Final thoughts:
I think the main issue is that toolbox set the config directory to $HOME/.config/toolbox and change that variable can mess things up. For example, I realized that, toolbox, after the first change of $HOME, creates the following directories:

[juanje@xps tmphome]$ tree .config/
.config/
└── toolbox
    └── podman-system-migrate

1 directory, 1 file
[juanje@xps tmphome]$ tree .local/
.local/
└── share
    └── containers
        └── storage
            ├── libpod
            │   └── bolt_state.db
            ├── mounts
            ├── overlay
            │   └── l
            ├── overlay-containers
            │   └── containers.lock
            ├── overlay-images
            │   └── images.lock
            ├── overlay-layers
            │   └── layers.lock
            ├── storage.lock
            ├── tmp
            └── userns.lock

11 directories, 6 files

Right now I can't use (or remove) the tested containers 😞, but I can create new ones and use them.

I hope this give you more info about the issue and sorry for all the verboseness.

[juanje]

Also, I don't know if it's a separated bug, but it's weird that when I tried the container (after all the $HOME changes) it didn't tell me that it wasn't working.
At the beginning, I wasn't sure if toolbox had entered the container or not. It said nothing. The only hint was that the typical purple point before the prompt wasn't there.

[juanje]

One more thing. I've just rebooted the system and all the toolbox's containers work fine.
I tried many things to run or remove the containers, before the reboot, but nothing. After the reboot, everything works just fine.

[yilkalargaw]

@juanje you are right. I had the same problems regarding not being able to access my other containers from a fresh terminal with a proper home. To solve the issue without rebooting you have to kill every process associated with podman and then kill every process that is related to fuse overlays. After that restarting containers in a clean terminal will solve the issue. That is why I suggested fuse-overlays may be part of the issue.

[juanje]

@yilkalargaw Thanks for the tip! 😄
It wasn't fun to not been able to work with containers after testing, without rebooting 😞

[yilkalargaw]

I figured out a new thing about this issue. And I thought I might add it because it might help in the debugging.

the following works perfectly

mkdir tmp1 && cd tmp1 && HOME=$PWD

podman pull ubuntu && podman run -it --entrypoint /bin/bash docker.io/library/ubuntu

# do something

exit

cd .. && mkdir tmp2 && cd tmp2 && HOME=$PWD

podman pull ubuntu && podman run -it --entrypoint /bin/bash docker.io/library/ubuntu

Even after switching back and forth between the two directories access of the two containers work perfectly. But if you open a clean new terminal and try to start your toolbox containers they fail to start. This indicates it might not be a problem with podman or may be volume sharing might be the problem. I have not tried a simple volume sharing though and I might try it in the future.

[yilkalargaw]

Yep it is a volume sharing issue. I wonder if it is a fuse or toolbox issue.

[debarshiray]

This seems a like a duplicate of #183

I remember @yilkalargaw had come up with the idea of redefining $HOME in one of the duplicates of #183

While it's a neat hack, it's just that - a hack.

Changing $HOME on-the-fly like that is not supported by Toolbox. In fact, that's something that should generally be avoided beyond exceptions where it's meant to work or as quick hack demos.

[debarshiray]

Duplicate of #183