Skip to content

Latest commit

 

History

History
executable file
·
112 lines (88 loc) · 3.87 KB

README.md

File metadata and controls

executable file
·
112 lines (88 loc) · 3.87 KB
Raw

NeRF4SeRe: Neural Radiance Fields for Scene Reconstruction

Project of AI3604 Computer Vision, SJTU.
Wei Jiang * · Kangrui Mao * · Gu Zhang * · Haoyu Zhen *
\star = equal contribution

Paper PDF Pre PyTorch

TODO

  • installation guideline
  • training and testing code for NeRF, INeRF and Instant-NGP
  • implementation of NeRFusion
  • NICE-SLAM integration
  • visualizations

Installation

git clone --recursive https://github.com/NeRF-SeRe/SeRe.git
cd SeRe

  1. Environment setup You should install google-sparsehash first following the instructions in torchsparse (e.g., sudo apt-get install libsparsehash-dev on Ubuntu and brew install google-sparsehash on Mac OS).

    Then install the dependencies:

    conda create -c conda-forge -n sere python=3.9
pip install -r requirements.txt
pip install git+https://github.com/facebookresearch/pytorch3d.git
pip install --upgrade git+https://github.com/mit-han-lab/[email protected]
export PYTHONPATH="/path/to/sere:$PYTHONPATH"

  2. Dataset preparation Please refer to https://github.com/cvg/nice-slam to download the Replica Dataset. For SJTUers, you could download it from Replica-JBox.

    Then link the dataset

    ln -s /path/to/Replica data/replica

Training Pipeline

Adding --device x to specify the GPU device (default: 0).

  1. Vanilla NeRF

    python run/main_nerf.py --config configs/nerf/replica/room0.txt

  2. Instant-NGP

    python run/main_ngp.py data/replica/room0/ --workspace output/ngp/replica/room0/ --fp16 --cuda_ray

  3. NeRFusion

    python run/main_nerfusion.py data/replica/room0 --fusion_path data/replica --workspace output/nerfusion/replica/room0 --cuda_ray --fp16 

python run/main_nerfusion.py data/replica/room0 --fusion_path data/replica --fusion_scenes office0 office1 --workspace output/nerfusion/replica/room0 --cuda_ray --fp16 # change fusion_scenes to specify the scenes to be trained

  4. Refine camera pose (INeRF)

    After you have trained a NeRF model, please put the checkpoints (its name should keep the same as model_name in config) under /ckpts/inerf. Then run:

    python run/main_inerf.py --config configs/inerf/replica/room0.txt

Evaluation

If you have already trained a NeRF model, just add --test to test it. eg.

python run/main_ngp.py data/replica/room0/ --workspace output/ngp/replica/room0/ --fp16 --cuda_ray --test # write images

python run/main_ngp.py data/replica/room0/ --workspace output/ngp/replica/room0/ --fp16 --cuda_ray --test --video # generate video

Acknowledgement

For TA

Our latest report and appendices are in docs.