Skip to content

Latest commit

 

History

History
200 lines (161 loc) · 10.9 KB

README.md

File metadata and controls

200 lines (161 loc) · 10.9 KB
Raw

NEWS !!!

  • Jan. 15th, 2024: 🚀 🚀 🚀 The official implementation of our PivotNet is released now. Enjoy it!
  • Jul. 14th, 2023: 👏 Our PivotNet is accepted by ICCV 2023 ! Refer to the Long-version Paper for more details.
  • May. 26th, 2023: 🏆 Our team win the Championship and the Innovation-Award of the CVPR23 Online HD Map Construction Challenge ! [ Leaderboard / Tech-Report]
  • Feb. 28th, 2023: 👏 Our BeMapNet is accepted by CVPR 2023 ! Refer to the Paper for more details. [Long version on Arxiv / Code]

Introduction

Vectorized high-definition map (HD-map) construction has garnered considerable attention in the field of autonomous driving research. Towards precise map element learning, we propose a simple yet effective architecture named PivotNet, which adopts unified pivot-based map representations and is formulated as a direct set prediction paradigm. Concretely, we first propose a novel Point-to-Line Mask module to encode both the subordinate and geometrical point-line priors in the network. Then, a well-designed Pivot Dynamic Matching module is proposed to model the topology in dynamic point sequences by introducing the concept of sequence matching. Furthermore, to supervise the position and topology of the vectorized point predictions, we propose a Dynamic Vectorized Sequence loss. PivotNet contains four primary components: Camera Feature Extractor, BEV Feature Decoder, Line-aware Point Decoder, and Pivotal Point Predictor. It takes the RGB images as inputs and generates flexible and compact vectorized representation without any post-processing.

Documentation

We build the released version of PivotNet upon BeMapNet project. Therefore, this project supports the reproduction of both PivotNet and BeMapNet.

Step-by-step Installation <\br>

  • a. Check Environment

    Python >= 3.8
CUDA 11.1
# other versions of python/cuda have not been fully tested, but I think they should work as well.

  • b. Create a conda virtual environment and activate it. (Optional)

    conda create -n pivotnet python=3.8 -y
conda activate pivotnet

  • c. Install PyTorch and torchvision following the official instructions.

    pip3 install torch==1.10.1+cu111 torchvision==0.11.2+cu111 -f https://download.pytorch.org/whl/torch_stable.html

  • d. Install MMCV following the official instructions. (need GPU)

    pip3 install -U openmim
mim install mmcv==1.7.1

  • e. Install Detectron2 following the official instructions.

    python3 -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu111/torch1.10/index.html

  • f. Compiling CUDA operators for Deformable Transformer.

    cd mapmaster/models/bev_decoder/deform_transformer/ops
bash make.sh

  • g. Install PivotNet.

    git clone [email protected]:wenjie710/PivotNet.git
cd pivotnet
pip3 install -r requirement.txt
Material Preparation <\br>

  • a. Data: NuScenes

    • Download&Unzip the NuScenes dataset into your server and link it to desirable path. 
      cd /path/to/pivotnet
mkdir data
ln -s /any/path/to/your/nuscenes data/nuscenes
    • Generate Pivot/Bezier-annotations from NuScenes's raw-annotations. 
      cd /path/to/pivotnet
python3 tools/anno_converter/nuscenes/convert.py -d ./data # generate pivot-annotation only
OR python3 tools/anno_converter/nuscenes/convert.py -d ./data --bezier   # generate both pivot and bezier annotatation (more time needed)

  • b. Weights: Public-Pretrain-Models

    • Download public pretrain-weights as backbone initialization. 
      cd /path/to/pivotnet
cd assets/weights
wget https://github.com/wenjie710/PivotNet/releases/download/v1.0/efficientnet-b0-355c32eb.pth .
wget https://github.com/wenjie710/PivotNet/releases/download/v1.0/resnet50-0676ba61.pth .
wget https://github.com/wenjie710/PivotNet/releases/download/v1.0/upernet_swin_tiny_patch4_window7_512x512.pth .

  • c. Check: Project-Structure

    • Your project directory should be, 
        assets
    | -- weights (resnet, swin-t, efficient-b0, ...)
    | -- 
  mapmaster
  configs
  data
    | -- nuscenes
      | -- samples (CAM_FRONT, CAM_FRONT_LEFT, CAM_FRONT_RIGHT, ...)
      | -- annotations
      | -- v1.0-trainval
      | -- ...
      | -- customer
        | -- pivot-bezier
          | -- *.npz
  tools
Training and Evluation <\br>

  • a. Model Training

    bash run.sh train pivotnet_nuscenes_swint 30  # pivotnet, default: 8GPUs, bs=1, epochs=30
bash run.sh train bemapnet_nuscenes_swint 30  # bemapnet, default: 8GPUs, bs=1, epochs=30

  • b. Model Evaluation

    bash run.sh test pivotnet_nuscenes_swint ${checkpoint-path}  # for pivotnet
bash run.sh test bemapnet_nuscenes_swint ${checkpoint-path}  # for bemapnet

  • c. Reproduce with one command

    bash run.sh reproduce pivotnet_nuscenes_swint  # for pivotnet
bash run.sh reproduce bemapnet_nuscenes_swint  # for bemapnet

Models & Results

Results on NuScenes Val Set <\br>

Citation

If you find PivotNet/BeMapNet/MachMap is useful in your research or applications, please consider giving us a star ⭐ and citing them by the following BibTeX entries:

@inproceedings{ding2023pivotnet,
  title={Pivotnet: Vectorized pivot learning for end-to-end hd map construction},
  author={Ding, Wenjie and Qiao, Limeng and Qiu, Xi and Zhang, Chi},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={3672--3682},
  year={2023}
}

@InProceedings{Qiao_2023_CVPR,
    author    = {Qiao, Limeng and Ding, Wenjie and Qiu, Xi and Zhang, Chi},
    title     = {End-to-End Vectorized HD-Map Construction With Piecewise Bezier Curve},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {13218-13228}
}

@article{qiao2023machmap,
    author={Limeng Qiao and Yongchao Zheng and Peng Zhang and Wenjie Ding and Xi Qiu and Xing Wei and Chi Zhang},
    title={MachMap: End-to-End Vectorized Solution for Compact HD-Map Construction}, 
    journal={arXiv preprint arXiv:2306.10301},
    year={2023},
}