Jan. 15th, 2024: 🚀 🚀 🚀 The official implementation of our PivotNet is released now. Enjoy it!Jul. 14th, 2023: 👏 Our PivotNet is accepted by ICCV 2023 ! Refer to the Long-version Paper for more details.May. 26th, 2023: 🏆 Our team win the Championship and the Innovation-Award of the CVPR23 Online HD Map Construction Challenge ! [ Leaderboard / Tech-Report]Feb. 28th, 2023: 👏 Our BeMapNet is accepted by CVPR 2023 ! Refer to the Paper for more details. [Long version on Arxiv / Code]
Vectorized high-definition map (HD-map) construction has garnered considerable attention in the field of autonomous driving research. Towards precise map element learning, we propose a simple yet effective architecture named PivotNet, which adopts unified pivot-based map representations and is formulated as a direct set prediction paradigm. Concretely, we first propose a novel Point-to-Line Mask module to encode both the subordinate and geometrical point-line priors in the network. Then, a well-designed Pivot Dynamic Matching module is proposed to model the topology in dynamic point sequences by introducing the concept of sequence matching. Furthermore, to supervise the position and topology of the vectorized point predictions, we propose a Dynamic Vectorized Sequence loss. PivotNet contains four primary components: Camera Feature Extractor, BEV Feature Decoder, Line-aware Point Decoder, and Pivotal Point Predictor. It takes the RGB images as inputs and generates flexible and compact vectorized representation without any post-processing.
We build the released version of PivotNet upon BeMapNet project. Therefore, this project supports the reproduction of both PivotNet and BeMapNet.
Step-by-step Installation
<\br>-
a. Check Environment
Python >= 3.8 CUDA 11.1 # other versions of python/cuda have not been fully tested, but I think they should work as well.
-
b. Create a conda virtual environment and activate it. (Optional)
conda create -n pivotnet python=3.8 -y conda activate pivotnet
-
c. Install PyTorch and torchvision following the official instructions.
pip3 install torch==1.10.1+cu111 torchvision==0.11.2+cu111 -f https://download.pytorch.org/whl/torch_stable.html
-
d. Install MMCV following the official instructions. (need GPU)
pip3 install -U openmim mim install mmcv==1.7.1
-
e. Install Detectron2 following the official instructions.
python3 -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu111/torch1.10/index.html
-
f. Compiling CUDA operators for Deformable Transformer.
cd mapmaster/models/bev_decoder/deform_transformer/ops bash make.sh -
g. Install PivotNet.
git clone [email protected]:wenjie710/PivotNet.git cd pivotnet pip3 install -r requirement.txt
Material Preparation
<\br>-
a. Data: NuScenes
- Download&Unzip the NuScenes dataset into your server and link it to desirable path.
cd /path/to/pivotnet mkdir data ln -s /any/path/to/your/nuscenes data/nuscenes - Generate Pivot/Bezier-annotations from NuScenes's raw-annotations.
cd /path/to/pivotnet python3 tools/anno_converter/nuscenes/convert.py -d ./data # generate pivot-annotation only OR python3 tools/anno_converter/nuscenes/convert.py -d ./data --bezier # generate both pivot and bezier annotatation (more time needed)
- Download&Unzip the NuScenes dataset into your server and link it to desirable path.
-
b. Weights: Public-Pretrain-Models
- Download public pretrain-weights as backbone initialization.
cd /path/to/pivotnet cd assets/weights wget https://github.com/wenjie710/PivotNet/releases/download/v1.0/efficientnet-b0-355c32eb.pth . wget https://github.com/wenjie710/PivotNet/releases/download/v1.0/resnet50-0676ba61.pth . wget https://github.com/wenjie710/PivotNet/releases/download/v1.0/upernet_swin_tiny_patch4_window7_512x512.pth .
- Download public pretrain-weights as backbone initialization.
-
c. Check: Project-Structure
- Your project directory should be,
assets | -- weights (resnet, swin-t, efficient-b0, ...) | -- mapmaster configs data | -- nuscenes | -- samples (CAM_FRONT, CAM_FRONT_LEFT, CAM_FRONT_RIGHT, ...) | -- annotations | -- v1.0-trainval | -- ... | -- customer | -- pivot-bezier | -- *.npz tools
- Your project directory should be,
Training and Evluation
<\br>-
a. Model Training
bash run.sh train pivotnet_nuscenes_swint 30 # pivotnet, default: 8GPUs, bs=1, epochs=30 bash run.sh train bemapnet_nuscenes_swint 30 # bemapnet, default: 8GPUs, bs=1, epochs=30
-
b. Model Evaluation
bash run.sh test pivotnet_nuscenes_swint ${checkpoint-path} # for pivotnet bash run.sh test bemapnet_nuscenes_swint ${checkpoint-path} # for bemapnet
-
c. Reproduce with one command
bash run.sh reproduce pivotnet_nuscenes_swint # for pivotnet bash run.sh reproduce bemapnet_nuscenes_swint # for bemapnet
Results on NuScenes Val Set
<\br>-
a. Easy-Setting --> AP-threshold is
0.5m, 1.0m, 1.5m(same as VectorMapNet / MapTR)Model Config Schd mAPdivider mAPpedcross mAPboundary mAPavg Download PivotNet-Effb0 config 30ep 59.3 54.1 60.0 57.8 model/train and eval PivotNet-Res50 config 30ep 58.0 53.5 59.7 57.1 model/train and eval PivotNet-SwinT config 30ep 62.9 57.9 64.0 61.6 model/train and eval PivotNet-SwinT config 110ep 67.8 62.1 69.2 66.4 model/train and eval -
b. Hard-Setting --> AP-threshold is
0.2m, 0.5m, 1.0m(Recommended as a more practical HD map evaluation protocol)Model Config Schd mAPdivider mAPpedcross mAPboundary mAPavg Download PivotNet-Effb0 config 30ep 44.0 35.9 39.7 39.8 model/train and eval PivotNet-Res50 config 30ep 43.5 35.6 40.4 39.8 model/train and eval PivotNet-SwinT config 30ep 47.7 39.4 43.7 43.6 model/train and eval PivotNet-SwinT config 110ep 54.1 43.3 50.3 49.3 model/train and eval
If you find PivotNet/BeMapNet/MachMap is useful in your research or applications, please consider giving us a star ⭐ and citing them by the following BibTeX entries:
@inproceedings{ding2023pivotnet,
title={Pivotnet: Vectorized pivot learning for end-to-end hd map construction},
author={Ding, Wenjie and Qiao, Limeng and Qiu, Xi and Zhang, Chi},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
pages={3672--3682},
year={2023}
}
@InProceedings{Qiao_2023_CVPR,
author = {Qiao, Limeng and Ding, Wenjie and Qiu, Xi and Zhang, Chi},
title = {End-to-End Vectorized HD-Map Construction With Piecewise Bezier Curve},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2023},
pages = {13218-13228}
}
@article{qiao2023machmap,
author={Limeng Qiao and Yongchao Zheng and Peng Zhang and Wenjie Ding and Xi Qiu and Xing Wei and Chi Zhang},
title={MachMap: End-to-End Vectorized Solution for Compact HD-Map Construction},
journal={arXiv preprint arXiv:2306.10301},
year={2023},
}