seg fault when running bundle with libnvidia-container-tools installed in container

[dbkinghorn]

I hope you are doing well!

I'm creating enroot bundles and installing libnvidia-container-tools in the bundle container to avoid that as a dependency where the bundle will be run.

I know I've done this successfully before but I'm now getting a seg fault when starting a bundle. I have verified that the lib is installed.

Extracting [####################] 100%
/tmp/tgi-1.0.3.RzpNQ8ymu6/.enroot/lib/common.sh: line 211: 84533 Segmentation fault      (core dumped) "${file}"
[ERROR] /tmp/tgi-1.0.3.RzpNQ8ymu6/.enroot/etc/system/hooks.d/98-nvidia.sh exited with return code 139

Installing libnvidia-container-tools on the local system does allow the bundle to run.

Any ideas?

[3XX0]

Which version of libnvidia-container? This might be a regression in the lib, if you manage to have an easy reproducer we could move this issue there so they can look at it.

[dbkinghorn]

Sorry, I intended to include that. This is indeed a new release. I'll put together a simple test case. Thanks!

kinghorn@ai2:~$ nvidia-container-cli --version
cli-version: 1.14.1
lib-version: 1.14.1
build date: 2023-09-07T16:05+00:00
build revision: 1eb5a30
build compiler: x86_64-linux-gnu-gcc-7 7.5.0
build platform: x86_64
build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fplan9-extensions -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections

[dbkinghorn]

I tried using libnvidia-container* 1.13 and 1.12 but no luck. I also tried with the just released 1.14.2

Here's a hackish script that would create a bundle with the library installed in the container that fails on a host system that does not have the library installed.

#!/usr/bin/env bash
# Testing enroot bundle with installed libnividia-container-tools

LOCAL_DIR=~/.local/share/enroot

# Create enroot container
enroot import docker://nvcr.io#nvidia/cuda:12.2.0-runtime-ubuntu22.04

enroot create --force --name cuda12.2 nvidia+cuda+12.2.0-runtime-ubuntu22.04.sqsh

# Make sure needed packages are installed
enroot start --rw --root cuda12.2 apt-get update
enroot start --rw --root cuda12.2 apt-get install --yes curl gnupg2 lsb-release
# Write script for installing nvidia container toolkit to /root/install-nvidia-container-toolkit.sh
cat <<EOF >$LOCAL_DIR/cuda12.2/root/install-nvidia-container-toolkit.sh
#!/bin/bash
# Install nvidia container toolkit
curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey |  apt-key add -
curl -s -L https://nvidia.github.io/libnvidia-container/ubuntu22.04/libnvidia-container.list | tee /etc/apt/sources.list.d/libnvidia-container.list

apt-get update
apt-get install --yes libnvidia-container-tools libnvidia-container1
apt-get install --yes nvidia-container*
EOF

# Make script executable
chmod +x $LOCAL_DIR/cuda12.2/root/install-nvidia-container-toolkit.sh

# Start the container and run the script
enroot start --rw --root cuda12.2 /root/install-nvidia-container-toolkit.sh

# Now export the container and create a new bundle
enroot export --force cuda12.2
enroot bundle --force cuda12.2.sqsh

The error output from starting the bundle is;

kinghorn@ai2:~$ ./cuda12.2.run 
Extracting [####################] 100%
nvidia-container-cli: initialization error: load library failed: libnvidia-container-go.so.1: cannot open shared object file: no such file or directory
[ERROR] /tmp/cuda12.2.ryKPEotDDu/.enroot/etc/system/hooks.d/98-nvidia.sh exited with return code 1

The test is not throwing a seg fault. But I don't know what should be providing libnvidia-container-go.so.1

This is not a show stopper since I can install the library on systems under test but it is an extra cleanup step after the testing. (I'm running LLM benchmarks for validation on systems that are going out to customers.)

[dbkinghorn]

... and just checked. libnvidia-container-go.so.1 is in the bundle where it's supposed to be.

[3XX0]

Moving the issue since this isn't specific to Enroot per-se. It looks like a regression since Golang dependencies were added. I think there was a plan to build static builds as well which could help in this case.

[dbkinghorn]

Thanks Jonathon, best wishes! --Don

[3XX0]

BTW, regarding the libnvidia-container-go.so.1 not being found, you can probably work around it by defining the following in /etc/system/hooks.d/98-nvidia.sh before generating the bundle:

export LD_LIBRARY_PATH="${ENROOT_ROOTFS}/lib/x86_64-linux-gnu/"