Visualization and Interfacing Documentation

Raghav Pillai

The visualization sub-directory contains the frame for exploring live or recorded (ROS bag) Voltron data.

Rework of Console Logs

Much of our new in-vehicle data visualization will still require the operator to interpret raw output data from various nodes. Our current implementation of output simply outputs all data to the output stream, leading it to be quite difficult to interpret data that matters. With this in mind, it’s imperative that we work on creating a method of passing output along easily, this being by creating a new output topic on our system. All this requires is our output to communicate the relevant data on the console topic, as well as other classification data like node group and name, as well as priority level. All of these would give the operator control to view exactly what is needed.

Console Output

An easy way of implementing this would be to use the console reporting mechanism built into ROS, ‘rosout’ (roscpp’s implementation is rosconsole). We can use this with rqt_console to easily display and filter ROS messages, and use this as a stepping stone until we get to our full web-based layout. Possible issues with ROS2, but we’ll have to see.

Assigning Tags

With each output from a certain node, we will assign certain tags to these outputs to be filtered in the end console log view. At the moment, the planned tags consist of three status enumerators (debug, warning, error) to handle different types of output, tags for node groups (different vision nodes, behavior nodes, control nodes, etc), and a tag for which individual node created the output.

Assuming we keep the JSON format, we could also create a unique ID for each transmitted log. For example, a pointcloud object has too many fields to display properly, so being able to take this ID and either expand it or put the ID in a separate view to see the expanded data could be something that we would need to do depending on how the layout turns out.

Example

A quick example of this implementation is as below: Lidar integration has several different possible outputs. Normal outputs run every cycle, and if an error is detected, those show up between normal outputs. Instead of these directly communicating with the standard output stream, these would be published to the console topic with our aforementioned parameters -- node group, node name, and priority level. The operator can cycle through priority levels and node groups, which would allow the operator to see consistent errors, or be able to see specific data easier.

Seeing Node Data and Statuses

Another important part of being able to interpret data better will be seeing what nodes are active, and the status of those nodes. With the aforementioned console rework, we’ll be able to see the status of each individual node, as well as all outputs from that node. This would not have to be complicated or cluttered, simple the status enumerator (error, running or inactive) and the logs themselves. Errors pertaining to the node could be easily viewed, and persist to easily see what is going wrong.

Better Interpretation of CAN Bus Data

Something that will be imperative going into the future is being able to see data directly from the CAN bus. This includes things like speed, brake/throttle engagement, current gear, blinker status, headlight status, and steering angles.

Implementation

We can subscribe to the topics created by our CAN Interface Node (“voltron_can”) module to get raw data from the car’s CAN bus, and display this. This data is (pending further review) isolated from the other ROS nodes, so in the event of a failure, this should remain operational for manual control of the car. This should remain compartmentalized as much as possible to preserve these systems from fault from other node issues.

Data Analysis & Statistics

Creating a window to see the status and statistics of observed data is something that could help diagnose runtime issues or drops in data transmission.

Implementation

To reduce overhead, a data analysis implementation would have to sit at the client receiver and receive all data at first before it is given to the other data modules. Through this, it would filter all events, and provide statistics of the frequency of data per second, the delay between the client and ROS timestamps, and other possible statistics that we could use in the future to debug.

3D and Map Data

3D and Map data will be the most difficult portion of the new functionality. With our goal to have live 3D and Map data with minimal overhead, it’s imperative that we use low resource-intensive libraries and build on these.

Implementation

The core interface will be developed using react so that we are able to scale this to other possible methods of interfacing or remote devices in the future. This will take longer than embedded scripts but will allow us to create a dynamic user interface that can be adjusted on the fly. This will mean that the creation of reusable React hooks will be necessary, which will also allow us to adjust the interface easily in the future, and fork older hooks if needed for newer features.

All three-dimensional data will be interpreted using regl-worldview. Worldview is built off WebGL as opposed to OpenGL ES, which allows us to create easily scalable 3D models without any native driver support, and runs with fairly low overhead. Using regl on top of this simplifies WebGL to draw optimized JS code, and it runs on a lower level than other engines that were in candidacy. We also get better error handling and performance through regl, since it automatically uses dynamic code generation and partial evaluation to be extremely lightweight.

This implementation allows us to interpret our pointcloud and vector data from the 3D and map views easily and map these relative to the car. Map view specifically will need several different layers and filters, but through worldview’s library, we can easily create objects around the target object. Worldview works around a camera system, where it tracks the target object (in this case, our vehicle), and we can control the properties of the said camera to set the position and distance of the camera, as well as offset.

Designing a More Intuitive Layout

Creating an intuitive layout is important for the general cleanliness of the interface, as well as being able to interpret data more quickly and easily. As seen in Waymo’s design library, much of this is done through having a clean layout that provides only the most useful data. As we’re in a heavy period of testing and development at the moment, it’s important to have a hybrid of these, where we can see an overview of all the data, and be able to see specifics.

Implementation

We’ll need to work closely with the classification team and our artists to create a clean view, post-classification so we can see easily what is going on with the car. Obviously, in the event that the operator wants to see the raw pointcloud data, they’ll be able to, but otherwise, we’ll want to keep this as clean as possible.

The webview will all be done through react, as stated previously, and have a focus on cleanliness and modularity. Like webviz (from Cruise), the goal is to create easily interactable visual modules, so the operator can traverse between modular panels easily to see specific data.

It’ll also be interesting to use audio cues as a way to communicate possible errors and actions to the operator, so this will be a possible way in the future that we de-clutter our main interface.

Core Communication Protocol

The basic communication between our ROS nodes and our backend will be supported by the ros2-web-bridge. With the rclnodejs library, we can use this Node.js library to create a JSON API that will use websockets to deliver real-time communication between the client and server applications (while they’re both connected). Using the rosbridge v2.0 protocol, we can relay regular and status messages, ROS operations and responses between the client and server. In syntax, these are similar to the messaging service in roscpp. << Question: will other team's nodes be able to send messages to the backend? I think a visualization of candidate paths and the target pose from the behavior planner could be cool. --Jim >>

Encoding and Message Transformation

Creating a lightweight messaging system is very important to keep latency down, and to keep the websocket stream as close to realtime as possible without any dropped data. We’re given some experimental operations from rosbridge for encoding and transformation such as fragmentation of messages for easier reading by the client, compressing PNG files and maps and CBOR encoding for faster compression of large data sets. This would entail creating a CBOR encoder (for most data requests) which would output a binary message instead of a regular JSON string, and the client would have a decoder to convert back to a normal protocol message.

Embedded Status Messages

Rosbridge is able to give us status messages for successes and failures of several protocol commands. These statuses are info (successes), warning (possible error), error (invalid request), and none. These would be used in regular operations to indicate loss of data, incorrect data requests and other possible client commands that could result in bad data.

User Authentication

User Authentication isn’t too much of an issue for now, but rosbridge allows us to create a user profile for authentication, which would verify the information with rosauth (not tied to ROS, library with rosbridge) to see if the authentication is good or invalid. If good, then proceed to keep the connection.

Messaging

ROS messages can be sent using the rosbridge protocol, and can interact with ROS. These both allow us to do familiar ROS functions like advertise/unadvertise, publish, subscribe/unsubscribe and call services straight from the client. Since our rosbridge implementation is a 2-way street, we could use this as a future way to insert data into ROS manually from the web client.

Client Interpretation

The client-side of our interface would receive the websocket connection and use the roslibjs to deliver the data to one of the viewports modules. As previously stated, the data would be handled separately on entry, creating a buffer between the frontend interpretation and the data events themselves. Internal data would be fed to frontend interpreters (such as our 3D view, console view, etc) instead of creating through an internal protocol rather than continuing to use the ROS client protocol.

Listening to the data is fairly simple. Communicating with the rosbridge server only requires a Ros node object (connecting to localhost for now), with the use of an ActionClient to send goals and receive responses from the webserver. The same methodology can be used for receiving video and messages and publishing, subscribing, and performing service calls.

ROS Bag and Recorded Data

The bag format allows us to store ROS message data easily, and therefore will provide an important tool for us to debug and view testing sessions after the live session. Rosbag are binary data files, and therefore we need to create an interpreter to view these files, as well as enable our encoder in VDE. This means our web visualization tool must be able to run separately from the VDE instance, and will be developed separately.

Interpreter

Using the rosbag.js module, we can develop a way of interpreting our ROS data on the client like webviz. We can read the data records for specific topics, and using the framework developed for our live sessions, we can read the same topics and interpret all our topics the same way, the only difference would be the actual ingestion. Because our ingestion is centralized and our individual visualization modules don’t use raw data from the bridge, we can simply switch how we ingest our data, and the rest of the environment would not be affected.

To interpret our ROS bag data, we would create a rosbag reader from the rosbagjs module, creating a localized Bag instance. We would simply consume the messages from this bag instance using the bag.readMessages method, which would allow us to interpret our topic data directly into our interpreter.

Encoder

It seems we already have a rosbag encoder in our VDE environment, so we would just have to enable this, and we’d be able to record all our topic data from our ROS system and output it to a .bag file in the root folder of our instance.