Don't install DCGM if the driver has been blacklisted

[aieri]

If the sysadmin wants to pass the gpu to a virtual instance via pci
passthrough, they will need to make the gpu unavailable to the host
system by blacklisting the kernel driver. On such a system DCGM would
not be able to function and should therefore not be deployed.

This commit makes the NVIDIA gpu verifier more strict by only marking
DCGM as an available tool if both an NVIDIA gpu is detected and the
kernel module is not blacklisted.

Fixes: #362

Testing setup

Given a server with an NVIDIA gpu, drivers installed, and modules blacklisted via modprobe:

$ cat /etc/modprobe.d/blacklist-nvidia.conf 
blacklist nvidia
blacklist nvidiafb
blacklist nouveau
blacklist nvidia_drm

Following the dev-environment.md file, set up a local controller deploy an older stable charm (e.g. rev 84), setting up resources if necessary.

Next pack the charm from this commit, scp it to the server, and install it:

$ juju refresh --switch ./hardware-observer_ubuntu-24.04-amd64_ubuntu-22.04-amd64_ubuntu-20.04-amd64.charm  hardware-observer

Expected result:

• DCGM is not installed
• the nvidia driver remains not loaded
• charm remains in active idle

[jneo8]

Thank you for the implementation!

Based on the code review (note: I haven’t tested it), I believe we’re at a good point to start adding unit tests.

[gabrielcocenza]

Small suggestion, but LGTM.

[jneo8]

LGTM if we follow Gabriel's suggestion on unit test.

[Deezzir]

LGTM

[aieri]

not merging yet because manual tests show that updating to this charm version forces the driver to be installed and loaded, despite the blacklisting

[aieri]

one possible hacky way to handle the upgrade issue would be to do something like this in _on_install_or_upgrade:

if HWTool.DCGM in self.stored_tools and not nvidia_gpu_verifier():
    self.stored_tools.remove(HWTool.DCGM)

(basically a gpu-specific hardware redetection round)

[jneo8]

one possible hacky way to handle the upgrade issue would be to do something like this in _on_install_or_upgrade

Need to be very careful about this proposal. This somehow means run the detect function every time when hook is triggered.

not merging yet because manual tests show that updating to this charm version forces the driver to be installed and loaded, despite the blacklisting

I believe this issue occurs because the list of HWTool included the GPU prior to the charm upgrade.

It’s a bit of a chicken-and-egg problem—you’ll need a way to clean the local state to prevent this from happening.

New deployed unit won't encounter this.(If I am correct)

I have couple proposals:

• Ask people to redeploy the juju unit. Since the origin issue is a edge case and new deployment won't encounter this after merging. It can be a lowest effort option for us.
• Provide --clean-resource argument to the re-detect action, which will remove the unused resource on the machine. User encounter this issue can be simple fixed by running the action.
• Run re-detect every time on install/upgrade hook, this may be lowest priority option since changing the life-cycle make it become more not stable.

[samuelallan72]

Drive by comment: how expensive is it to detect the tools? Perhaps we can build the tools list on every charm hook, rather than using stored state?

[jneo8]

Drive by comment: how expensive is it to detect the tools? Perhaps we can build the tools list on every charm hook, rather than using stored state?

We did this in the past and encounter an issue that some unstable hardware just you different result every time you run re-detect. I will suggestion not dig into the same hole again.

[aieri]

ok, I've tested that at least the upgrade scenario from rev 84 (pre-dcgm) works fine. Further cleanups would require changing how and when we do hw redetection, which is a much bigger endeavor than this change