rocm: llama-cli killed by SIGSEGV

[Chevek]

I have this GPU: AMD RADEON RX 7900 XTX
ramalama pull "quay.io/ramalama/rocm:latest"

When I try a model, there is always a crash.

e.g.
$ ramalama run llama3.2
"llama-cli killed by SIGSEGV"
cmdline:
llama-cli -m /var/lib/ramalama/models/ollama/llama3.2:latest --in-prefix "" --in-suffix "" --no-display-prompt -p $'You are a helpful assistant' -c 2048 -ngl 99 -cnv

os_release:
Fedora release 40 (Forty)
x86_64
CPU:
AMD Ryzen 7 7700 8-Core Processor

[Chevek]

$ ramalama list
NAME MODIFIED SIZE
ollama://llama3.2:latest 10 minutes ago 1.88 GB

[ericcurtin]

I have a an almost identical setup to you, Fedora 40 host. My GPU is this one though:

AMD Radeon RX 7600

llama3.2:latest runs fine.

I'd recommend searching for similar issues here:

https://github.com/ggerganov/llama.cpp/issues

and if you don't fine anything logging an issue there.

llama.cpp does most of the heavy lifting for ramalama so a lot of the issues stem from something closer to llama.cpp like this one:

#247

[ericcurtin]

Other debug steps you can do is build/run against various version of ROCm and llama.cpp in and outside the container.

This is how it is done:

https://github.com/containers/ramalama/blob/main/container-images/rocm/Containerfile

Like does your setup work on any version of ROCm and/or llama.cpp? That would be interesting...

[bmahabirbu]

@ericcurtin I can check into this! I recently got a 7800xt. It would be nice to know how you both setup amd GPU support on Podman as well maybe there are some differences?

Also I know that for any kernel release under 6.8 it doesn't play nice with any rx 7000s graphics cards but I believe Fedora 40 is at 6.8.

[Chevek]

Fedora 40 is on a rolling kernel. At the moment, this is the one I have:
Linux fedora 6.11.3-200.fc40.x86_64

AFAICT I do not touch podman configuration in any way, it's just ramalama.

As I use my setup for gaming/multimedia use case, I've rpm fusion enabled and some stuff like this:
sudo dnf swap -y mesa-va-drivers mesa-va-drivers-freeworld
sudo dnf swap -y mesa-vdpau-drivers mesa-vdpau-drivers-freeworld
sudo dnf swap -y mesa-va-drivers.i686 mesa-va-drivers-freeworld.i686
sudo dnf swap -y mesa-vdpau-drivers.i686 mesa-vdpau-drivers-freeworld.i686
rocm:
sudo usermod -a -G render,video $LOGNAME
sudo dnf -y install rocm-opencl rocminfo rocm-clinfo rocm-hip rocm-runtime rocm-smi rocm-hip-devel

I'll give a try with a stock Fedora 40 installation next week.

[ericcurtin]

@ericcurtin I can check into this! I recently got a 7800xt. It would be nice to know how you both setup amd GPU support on Podman as well maybe there are some differences?

Also I know that for any kernel release under 6.8 it doesn't play nice with any rx 7000s graphics cards but I believe Fedora 40 is at 6.8.

SGTM.

I don't do anything special to be honest, I just install podman and RamaLama on Fedora 40 and it just seems to work fine GPU accelerated on my machine.

Do you use podman, docker or no containers here @Chevek ?

[ericcurtin]

Funnily enough the GPU acceleration on my AMD machine stopped working, I dunno if that's because of an upgrade I did to packages, the kernel, etc. A containers change or what.

[bmahabirbu]

Interesting, does rocminfo still work on the terminal? Ive had issues where the rocm didn't pass permissions correctly to podman and it seemed like functionality wasn't there among others.

I have a local llama.cpp build that I test for a sanity check when im messing with container settings