In multiple GPUs: RuntimeError: vk::PhysicalDevice::createDeviceUnique: ErrorInitializationFailed #20

[DelinQu]

Hi 👋, I caught the error "RuntimeError: vk::PhysicalDevice::createDeviceUnique: ErrorInitializationFailed #20" when starting the environment in a non-zero device. The simulator works well when using the default CUDA device 0, but it failed in any other devices in 1,2,3,4,5,6,7:

I tried the solution at #20 and haosulab/ManiSkill#73, but it doesn't work for me.

[xuanlinli17]

You can try ManiSkill3 version of the Bridge envs. @StoneT2000 will migrate the Google Robot envs later.

I think a fix for ManiSkill2 is to set DISPLAY="" CUDA_VISIBLE_DEVICES=x python {}

[xuanlinli17]

See e.g.., haosulab/ManiSkill#79 (from old ManiSkill2)

[StoneT2000]

for maniskill 3 the fix should work proposed by xuanlin should work

[DelinQu]

Thanks for your replies, the DISPLAY has already been unset in https://github.com/simpler-env/SimplerEnv/blob/d55e19162be86794875839725fd484b768e25873/simpler_env/main_inference.py#L21C2-L21C31, I have no idea why it doesn't work for me. So I will migrate the environment to maniskiill3, does it cause any difference to the evaluation results, compared with mainiskill2? I must make the evaluation fair.

[xuanlinli17]

It should be very similar.

[DelinQu]

Weird. Simpler start-up successfully if I set the CUDA_VISIBLE_DEVICES=x,0 python {}. The Memory and utils of CUDA:0 are almost zero, but it's critical for setup maniskill2 environments:

[StoneT2000]

If you only need to measure success / partial success rate in the main study (not the texture randomization ablations, which have not been ported over) I'd just recommend moving to ManiSkill3, it has better support and less bugs related to rendering/gpus.