GPU utilization

[NikoYura]

Hello JUDI developers,

I’m encountering an issue where the built-in JUDI tests show GPU utilization correctly after setting the environment variables in my notebook, but when I try to run my own tasks such as forward modeling or Reverse Time Migration (RTM), only the CPU is being utilized. Could you help me find the potential cause of this discrepancy?

Julia Version 1.10.5
JUDI v3.4.6

I would appreciate guidance on the paths I should investigate or potential misconfigurations that could be leading to this behavior.

Thank you in advance for your help!

[mloubout]

How do you monitor the gpuy usage, and what's the problem size you are running?

[NikoYura]

Thank you for your suggestion!

During the tests, memory consuming processes appear (screenshot attached), with the CPU behaving inactive. But during my tasks (forward modeling, RTM with adjoint counting), there are no processes in nvidia-smi, and the CPU is fully loaded.

[mloubout]

Can you run your own scripts/... with DEVITO_LOGGING=DEBUG and show the log?

[NikoYura]

Thanks for the tip, I found the problem with the wrong call thanks to debug. A little bit not how I expected the initial behavior of my program.

I would like to clarify, does splitting the algorithm into workers with distributed.jl package work similarly with GPU? Are there any nuances?
Is it possible to set environment parameters in the notebook instead of exporting? For some reason some variables work, but GPU variables or for example debug do not work in this form ENV[“DEVITO_LOGGING”] = “DEBUG”?

[mloubout]

Is it possible to set environment parameters in the notebook instead of exporting?

There is a utility function set_devito_config that allows you to modify the devito configuration from a script or notebook. For example you can do set_devito_config("log-level", "DEBUG")

does splitting the algorithm into workers with distributed.jl package work similarly with GPU?

Yes it does, you might need to look at the CUDA.jl documentation to make sure each worker is pinned to its own GPU.