Tool and study data for Generating Realistic and Diverse Tests for LiDAR-Based Perception Systems.
We propose a new technique in testing LiDAR-based perception systems that takes as input existing real-world labeled LiDAR test cases, as shown below, and then mutates them to generate novel, realistic test cases that can expand the diversity of tests to explore the long-tail of possible scenes.
Example point cloud and labelling
Example mutations
| Adding Entities | Removing Entities | Altering Entities |
|---|---|---|
 
|
 
|
 
|
Instructions and troubleshooting available in the INSTALL.md file.
The ./tool directory contains the source code for the tool as well as a script (./tool_demo.sh) for running a minimal complete example of the mutations implemented in the tool and explored in the study.
The tool begins by performing Resource Collection (Approach in Section II-B, Implementation in Section III-A), then generates the 7 mutations discussed in the Section VI Table III.
The demo scripts use Docker and Docker Compose and these must be installed.
The docker script will download a small subset of the SemantiKITTI dataset, run Resource Collection on this subset, and then generate 5 mutations of each type.
The selected data will be downloaded in ./selected_data/ and tool demo output will be produced in ./sample_tool_output/.
The output contains the raw bin and label files, along with visualizations in the final_viz folders.
Within the ./selected_data/ folder, the bin files are the raw LiDAR point clouds with intensities, and the label files contain the ground truth semantic labeling with instances included in the SemanticKITTI format.
After running the Resource Collection phase, the label files will be modified in place to contain the additional entities identified (Section II-B).
Within the ./sample_tool_output/ folder, the new mutations will each be in separate folders of the form <mutation name>_<timestamp>/.
Each mutation folder will contain a /output/done/ folder that contains a set of
To run the tool demo, use the following command. Building the Docker containers will take ~10 minutes, downloading and running Resource Collection will take ~10 more minutes, and then running the mutations will take ~10 more. There will be progress announcements throughout.
cd ./tool/
./tool_demo.shSimilar Docker scripts to recreate the figures and tables from the paper are available in the ./study directory. Please refer to that README for more information.
As noted in the paper, humans are not well suited to judge realism for LiDAR point clouds. Examples of false positive and true positive failures as determined by our voting mechanism discussed in Section IV-C-2 are available in the study false positives folder.
As noted in Section III, the implementation of only three of the basic mutation operations were selected for discussion. Complete discussion of all of the basic mutation operations are available in this supplement.
The software contained in this repository is covered under the included LICENSE file. The paper included in this repository is not covered by this license and is instead covered by the IEEE Copyright. The paper is included under the terms of the IEEE copyright to be included on the author's personal servers.