Pointcloud constructed from rgbd_sensor depth+rgb image has different orientation to points output

[heuristicus]

I'm using the subt stack for the reproduction because I already have it set up. Given that this is an issue with the sensor itself I think this would also reproduce in a basic setup with just a camera.

To reproduce:

• Run ign launch tunnel_circuit_practice.ign robotName1:=explorer_x1 robotConfig1:=EXPLORER_X1_SENSOR_CONFIG_1 localModel:=true
• Run rviz
• Change the fixed frame to world
• Display the /explorer_x1/front_rgbd/points topic
• Add a DepthCloud display by adding from topic and selecting the depthcloud on the /explorer_x1/front_rgbd/image_raw topic. Selecting the depth/image_raw topic will not work (and does not display anything for some reason, even with the same topics as will be input below)
• Change the depth map topic of the DepthCloud to /explorer_x1/front_rgbd/depth/image_raw
• You will now see an additional uncoloured pointcloud
• To colour the pointcloud, set color image topic to /explorer_x1/front_rgbd/image_raw

You can use this rviz config file to reproduce the rviz setup I have. You will have to change the depth map topic of the DepthCloud to an invalid topic, like /explorer_x1/front_rgbd/image_raw and then change it back to /explorer_x1/front_rgbd/depth/image_raw in order for the depth cloud to display correctly. I do not think this weirdness is connected to the issue as I have displayed the pointcloud in another visualisation tool and it displays the same behaviour.

The issue

The rgdb_sensor provides the topics sensor/image_raw, sensor/depth/image_raw and sensor/points. When displaying the sensor/points pointcloud in rviz the cloud appears in the expected orientation.

The cloud that is displayed by the DepthCloud generated by combining the depth and rgb points is in a different orientation to that which is displayed by the pointcloud display, despite the fact that they should be in the same orientation.

From the images above it appears that the cloud generated from the depth/rgb image expects that the positive z-axis points out into the world from the camera lens, whereas the points cloud assumes that is the case for the positive x axis.

This problem is not present in gazebo. I believe this is because the rgb and depth images are published in a separate frame from the base frame of the camera, the sensor optical_frame, where the positive z axis is in the expected orientation:

Here you can see the incorrect orientation of the clouds relative to each other:

[iche033]

for subt, we recently added an optical frame so that DepthCloud can be visualized with the desired orientation, see osrf/subt#413.

As for visualization not showing up the first time, this comment in the pull request describes a workaround to make the visualizations appear.

[heuristicus]

Thanks for that information - it fixes the issue for me. An upstream fix on the sensor itself would still be preferable though.

[Danilrivero]

Hello, we have encountered the same issue recently when migrating our stack. Has any upstream fix been made? It seems the issue still hasn't been resolved.

Warm regards

[azeey]

A quick way to add an optical frame in ROS is to create a static transform (from gazebosim/ros_gz#634 (review)):

<node pkg="tf2_ros" exec="static_transform_publisher" args="--frame-id camera_link  --child-frame-id camera_link_optical --roll 1.570796327 --yaw 1.570796327" />

[JorgePH]

A quick way to add an optical frame in ROS is to create a static transform (from gazebosim/ros_gz#634 (review)):

<node pkg="tf2_ros" exec="static_transform_publisher" args="--frame-id camera_link  --child-frame-id camera_link_optical --roll 1.570796327 --yaw 1.570796327" />

Even if you set a new frame, there is no consistency between the pointcloud msg and the rgbd image. The frame_id in the ros msgs is the same for both, but the data itself follow two different conventions. Wouldn't it make sense to have two different sdf tags to define each frame_id (optical and pointcloud)?