Skip to content

Latest commit

 

History

History
75 lines (66 loc) · 3.02 KB

LYFT_PREPROCESSING.md

File metadata and controls

75 lines (66 loc) · 3.02 KB
Raw

Download Lyft dataset and convert it to KITTI format

Adapted from MODEST (https://github.com/YurongYou/MODEST) Lyft-to-KITTI conversion.

Download the Lyft Level 5 AV Dataset (Perception) from https://level5.lyft.com/dataset/ and decompress it into LYFT_ROOT. Install Lyft Dataset SDK (https://github.com/lyft/nuscenes-devkit):

pip install -U git+https://github.com/lyft/nuscenes-devkit

We have been experimenting with an earlier version of lyft dataset (obtained on Aug 16 2019, now this version seems not available from the official website), which contains fewer sequences than the lastest version and slightly different localization data. The sample tokens of the old version were dumped in data_preprocessing/lyft/lyft_2019_train_sample_tokens.txt.

Convert the lyft dataset into KITTI format to LYFT_KITTI_FORMAT by

ln -s LYFT_ROOT/train/train_maps LYFT_ROOT/train/maps
ln -s LYFT_ROOT/train/train_lidars LYFT_ROOT/train/lidars
ln -s LYFT_ROOT/train/train_images LYFT_ROOT/train/images
cd data_processing/lyft
python lyft2kitti.py --store_dir LYFT_KITTI_FORMAT --lyft_dataroot LYFT_ROOT/train \
    --table_folder LYFT_ROOT/train/train_data \
    --sample_token_list ./lyft_2019_train_sample_tokens.txt --meta_info_prefix trainset_

The indices of traversals were dumped in data_preprocessing/lyft/meta_data/lyft_2019_train_sample_tracks.pkl.

[Optional] Obtain the train/test split

You can skip the following by directly using the uploaded files in data_preprocessing/lyft/meta_data/. We include the corresponding scripts to generate the splits as follows.

The train/test split of the traversals by their geo-location is generated by the following commands.

cd data_processing/lyft
python split_traintest.py --data_root LYFT_KITTI_FORMAT --track_list_file meta_data/lyft_2019_train_sample_tracks.pkl

It will generate

data_preprocessing/lyft/meta_data/fw70_2m_train_track_list.pkl
data_preprocessing/lyft/meta_data/fw70_2m_valid_train_idx_info.pkl
data_preprocessing/lyft/meta_data/fw70_2m_train_idx.txt # 11873 samples
data_preprocessing/lyft/meta_data/fw70_2m_full_test_idx.txt # 4901 samples

Generate ground planes for detection training

cd data_processing
python RANSAC.py --calib_dir LYFT_KITTI_FORMAT/training/calib \
    --lidar_dir LYFT_KITTI_FORMAT/training/velodyne \
    --planes_dir LYFT_KITTI_FORMAT/training/planes --min_h 1.5 --max_h 2.5

Setup for detection training

Create soft links to lyft dataset:

cd downstream/OpenPCDet/data/lyft
mkdir training
ln -s LYFT_KITTI_FORMAT/training/velodyne
ln -s LYFT_KITTI_FORMAT/training/label_2
ln -s LYFT_KITTI_FORMAT/training/calib
ln -s LYFT_KITTI_FORMAT/training/planes
ln -s LYFT_KITTI_FORMAT/training/image_2
ln -s LYFT_KITTI_FORMAT/training/oxts
ln -s LYFT_KITTI_FORMAT/training/l2e

After installing OpenPCDet, run

cd downstream/OpenPCDet/
python -m pcdet.datasets.kitti.kitti_dataset create_kitti_infos tools/cfgs/dataset_configs/lyft_dataset.yaml True