How to troubleshoot CUDA memory issues

[ajkessel]

I've been trying to get this to work on WSL via local install, Docker, as well as the VoiceCraft_API fork. In all cases, I get stuck at something like the following after a long pause:

    concated_audio, gen_audio = inference_one_sample(model, Namespace(**config), phn2num, text_tokenizer, audio_tokenizer, audio_fn, target_transcript, device, decode_config, prompt_end_frame)
  File "/opt/conda/envs/voicecraft/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/adam/work/inference_tts_scale.py", line 76, in inference_one_sample
    concat_frames, gen_frames = model.inference_tts_batch(
  File "/home/adam/work/models/voicecraft.py", line 1358, in inference_tts_batch
    past = torch.cat([past, present.to(past.dtype)], dim=-2) if past.ndim > 3 else present.to(past.dtype)
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

This is just with the default settings in inference_tts.ipynb that comes with the Docker image.

The device is a 2024 ThinkPad P1G7 which I think should have enough resources to run the test. Specs include 32GB physical RAM, 48GB virtual RAM, NVIDIA GeForce RTX 4070 with 8GB RAM.

Any suggestions for how to troubleshoot? Is this just not usable in my environment?