P2P and IB communications in RTX 3090 disabled?

[Nero10578]

I just updated to the latest commit and now axolotl warns that on RTX 3090 and 4090 P2P and IB communications are disabled?

As far as I know RTX 3090 can do P2P as long as you run two with NVLink, so can we get that enabled if there's only 2 RTX 3090s? Because that's what me and a lot of other home users run.

[NanoCode012]

Do you recall what the error looked like? I don't recall axolotl setting these settings by default.

[Nero10578]

The following values were not passed to `accelerate launch` and had defaults used instead:
        `--num_processes` was set to a value of `2`
                More than one GPU was found, enabling multi-GPU training.
                If this was unintended please pass in `--num_processes=1`.
        `--num_machines` was set to a value of `1`
        `--mixed_precision` was set to a value of `'no'`
        `--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
Using RTX 3090 or 4000 series which doesn't support faster communication speedups. Ensuring P2P and IB communications are disabled.
[2024-02-24 05:49:13,926] [INFO] [datasets.<module>:58] [PID:2238647] PyTorch version 2.1.2+cu121 available.
[2024-02-24 05:49:13,926] [INFO] [datasets.<module>:58] [PID:2238646] PyTorch version 2.1.2+cu121 available.
[2024-02-24 05:49:15,723] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-02-24 05:49:15,723] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-02-24 05:49:17,343] [INFO] [axolotl.normalize_config:176] [PID:2238646] [RANK:0] GPU memory usage baseline: 0.000GB (+1.107GB misc)
[2024-02-24 05:49:17,343] [INFO] [axolotl.normalize_config:176] [PID:2238647] [RANK:1] GPU memory usage baseline: 0.000GB (+1.107GB misc)

This is what it spits out at the beginning.

[NanoCode012]

Oh, I don’t recall that warning before. How’s the actual training?

[Nero10578]

It trains just fine but I have no idea how to check if it's using P2P via NVlink or not. Nvidia-smi does show that both cards are linked via NVlink while running in WSL.

[NanoCode012]

There’s a nccl doc on the docs folder in the repo. May I ask if any of those commands can help you?