The size of tensor a (32) must match the size of tensor b (40) at non-singleton dimension 2

[babbahotep]

Using Ubuntu Deepstack docker 2022.01.1 on esxi with grid Nvidia GPU. RTX A5000
After sometime Deepstack seems to crash with:

File "/app/intelligencelayer/shared/detection.py", line 81, in run_task
det = detector.predict(img_path, threshold)
File "/app/intelligencelayer/shared/./process.py", line 63, in predict
pred = self.model(img, augment=False)[0]
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/app/intelligencelayer/shared/./models/yolo.py", line 149, in forward
return self.forward_once(x, profile) # single-scale inference, train
File "/app/intelligencelayer/shared/./models/yolo.py", line 176, in forward_once
x = m(x) # run
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/app/intelligencelayer/shared/./models/yolo.py", line 75, in forward
y[..., 0:2] * 2.0 - 0.5 + self.grid[i].to(x[i].device)
RuntimeError: The size of tensor a (32) must match the size of tensor b (40) at non-singleton dimension 2

Anyway increase logging level to get timestamps?
Anyone have an idea what’s causing this crash?
Any options to catch this and restart deepstack docker instance automatically?

[MisterEkole]

Hi @babbahotep, this error is as a result of shape mismatch. I'd suggest you make sure the shape of both input tensors match, so in place addition can be executed hitch free.

[babbahotep]

Input is blue iris i have no control over the input.
this only happens on esxi docker with vgpu. DS on windows bare metal works fine.