Issues running the SDK examples with OAK-D-SR

[G-Levine]

I'm trying to run the vio_visu.py example with the OAK-D-SR and running into some problems.

1. The OV9782 cameras on the SR support 720p and 800p resolution, but "800p" can't be used as a value for inputResolution in the VIO pipeline config.

Here's the error message:

SpectacularAI ERROR: invalid or unsupported color input resolution 800p

My workaround was to set it manually back to 800p after constructing the VIO pipeline with 1200p but before starting the DepthAI pipeline:

config = spectacularAI.depthai.Configuration()
config.useColor = False
config.useStereo = False
config.inputResolution = "1200p"
config.useColorStereoCameras = True
vio_pipeline = spectacularAI.depthai.Pipeline(pipeline, config)
vio_pipeline.colorRight.setResolution(depthai.ColorCameraProperties.SensorResolution.THE_800_P)
vio_pipeline.colorPrimary.setResolution(depthai.ColorCameraProperties.SensorResolution.THE_800_P)
vio_pipeline.colorRight.setVideoSize(1280, 800)
vio_pipeline.colorPrimary.setVideoSize(1280, 800)

2. The VIO pipeline tries to call dai::CalibrationHandler::getCameraExtrinsics and dai::CalibrationHandler::getDistortionCoefficients on CAM_A, which isn't present on the SR (this device only has CAM_B and CAM_C).

The error message:
There is no Camera data available corresponding to the the requested source cameraId

My hacky GDB workaround to force it to never call those functions with CAM_A:

gdb --args python3 vio_visu.py
(gdb) set non-stop on
(gdb) break getCameraExtrinsics
(gdb) commands
(gdb) set $rdx=1
(gdb) c
(gdb) end
(gdb) break getDistortionCoefficients
(gdb) commands
(gdb) set $rdx=1
(gdb) c
(gdb) end
(gdb) run

3. After working around the above issues, I'm running into this one. I'm not exactly sure what the cause is, but it seems like it could be related to this.

Exception in thread Thread-1 (reader_loop):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/sdk-examples/python/oak/vio_visu.py", line 151, in reader_loop
    for vio_out in vio_source:
  File "/home/ubuntu/sdk-examples/python/oak/vio_visu.py", line 43, in live_vio_reader
    vio_pipeline.startSession(device) as vio_session:
IndexError: _Map_base::at

[kaatrasa]

Hey,

Thanks for letting us know about the issue. Next release will have a fix where the middle RGB camera (CAM_A) is optional, and useColorStereoCameras-mode will support 720p and 800p resolution. Note that we don't have a OAK-D-SR to test with, so there could be some additional issues.

I will let you know when the next release is available.

[kaatrasa]

Hey, SDK version 1.34.0 is now available if you wish to try it out. Feel free to let me know if you run into further issues with OAK-D-SR.

[G-Levine]

Works now. Here is the correct IMU-to-camera matrix for the OAK-D-SR (simply using the default value results in a ton of drift):

vio_pipeline.imuToCameraLeft = [
   [-1, 0, 0, 0],
   [0, -1, 0, 0],
   [0, 0, 1, 0],
   [0, 0, 0, 1]
]

[kaatrasa]

Thanks, your imu->camera matrix will be set by default for OAK-D-SR in next release (1.35.0).