Omniverse NeRF Extension

Neural Radiance Field (NeRF) extension for Omniverse.

Prerequisites

• Hardware:
  • CPU: x86
  • GPU: NVIDIA RTX GPU
  • See this page for more details.
• Operating System: Ubuntu 20.04/22.04.
• Software:
  • NVIDIA Driver
  • Docker
  • NVIDIA Container Toolkit
  • Omniverse Isaac Sim (through NGC Container)

Demo

2024-04-24.05-01-10.mp4

At the exterior of the National Tsing Hua University (NTHU) swimming pool. Collected by @muyishen2040, which is also used in the DriveEnv-NeRF project.

2024-04-24.18-03-01.mp4

Exterior of the Delta Building in National Tsing Hua University (NTHU). Collected by @Howardkhh.

2024-04-24.18-41-01.mp4

The UR5 Robotic Arm in ElsaLab.

2024-04-24.18-59-47.mp4

The Husky UGV in ElsaLab.

Note: The datasets for these demos are not provided in this repository as they are casually collected. The purpose of these demos is to illustrate that this repository can be readily applied to a variety of custom datasets with extremely low effort. The following guide will use the Poster dataset for simplicity.

Setup

git clone https://github.com/j3soon/omni-nerf-extension
cd omni-nerf-extension

Download assets:

wget https://github.com/j3soon/omni-nerf-extension/releases/download/v0.0.2/assets.zip
unzip assets.zip

Prepare assets for nerfstudio_renderer:

# change the DATE_TIME to the name of the placeholder
DATE_TIME=2023-12-30_111633
CHECKPOINT_NAME=step-000029999
cp -r ./assets/outputs/poster/nerfacto/$DATE_TIME ./assets/outputs/poster/nerfacto/DATE_TIME
mv ./assets/outputs/poster/nerfacto/DATE_TIME/nerfstudio_models/$CHECKPOINT_NAME.ckpt ./assets/outputs/poster/nerfacto/DATE_TIME/nerfstudio_models/CHECKPOINT_NAME.ckpt

The following assumes that you are running the commands from the root of the repository.

Managing Containers

Login to NGC and pull the image nvcr.io/nvidia/isaac-sim:2023.1.1 by following this guide. Then build the docker images for the extension:

docker pull nvcr.io/nvidia/isaac-sim:2023.1.1
docker compose build

Launch the containers:

# You might want to use `tmux` for exec-ing into the containers later
xhost +local:docker
docker compose up

Then follow the remaining sections.

To remove and stop the containers, run:

docker compose down

Nerfstudio Renderer

Code: nerfstudio_renderer

The renderer server would be listening on port 10001 upon successful startup:

INFO SLAVE/10001[MainThread]: server started on [0.0.0.0]:10001

After seeing the above logs, no additional steps are required for the renderer server.

PyGame Viewer

Code: pygame_viewer

Attach to the container and run the testing script:

docker exec -it pygame-viewer /src/run.sh

The script may fail at the first run due to the cold start of the renderer server. If it fails, try run the script again.

2024-04-23.21-35-58.mp4

Isaac Sim Viewer

Code: extension

docker exec -it isaac-sim-viewer bash
# in container
/isaac-sim/runapp.sh --ext-folder /src/exts --enable omni.nerf.viewport

1. Select the folder /workspace/usd
2. Open the file example_scene.usd
3. Click the mesh that you added in Step 2.
4. Press the button in NeRF Viewport to update the input mesh of NeRF.

2024-04-23.21-39-22.mp4

Known Issues:

• The Omniverse UI seems to block the use of double-clicking when the extension is enabled. I believe this is due to the rendering updates interrupting the determination of the double-click event. This issue can be bypassed by using single left-clicks and right-clicks instead. Alternatively, move the NeRF Viewport to cover the default Viewport (such that the rendering stops).
• Cannot correctly handling non-uniform scaling of the object mesh yet.

Development Notes

Nerfstudio Renderer

After modifying code, you need to remove and recreate the container to apply changes. This is because the container will copy and install the code upon startup.

PyGame Viewer

After modifying code, you need to re-run the testing script. The docker container can be re-used since the code is mounted as a volume.

Isaac Sim Viewer

Setup VSCode intellisense for the extension:

cd extension
./link_app.sh --path "$HOME/.local/share/ov/pkg/code-2022.3.3"
# open the `extension` folder in VSCode

After modifying code, you can restart Isaac Sim to apply changes. The docker container can be re-used since the code is mounted as a volume. If the change is small, it is often faster to disable and re-enable the extension in the Isaac Sim UI. This can be done through Window > Extensions > NVIDIA > General, search nerf, and then un-toggle and re-toggle the extension.

Future Directions

• Include 3D Gaussian Splatting (3DGS) backend for real-time rendering, which is supported by a newer version of Nerfstudio, see Splatfacto for more details.
• Include SDFStudio Neuralangelo for improved mesh extraction. However, I've heard that SDFStudio does not perform as well as the original Neuralangelo implementation.
• Support multiple Nerf renderings in a single scene potentially through Compositioning or Block-NeRF.

Related Works

This project focuses on the simplest integration of a NeRF renderer with Omniverse, which diverges from more advanced usages, such as those proposed in a talk from GTC Spring 2023. The formal way to integrate NeRF with Omniverse is to first prototype with custom USD attributes. The NeRF rendering can then be done by reading the custom USD attributes, using special MDL materials, or other clever techniques. Although this allows more advanced usages, this approach is much more difficult to implement. Moreover, the rasterization techniques used in 3DGS is fundamentally different from the volume-rendering techniques used in NeRF, which may require using two different rendering engines in the same scene.

Acknowledgement

This project has been made possible through the support of ElsaLab and NVIDIA AI Technology Center (NVAITC).

Special thanks to @tigerpaws01 for the initial implementation of the NeRF renderer backend and PyGame viewer. This project wouldn't have come to light if it weren't for his early contributions. Fun fact: this project was initiated during one of our dinner conversation.

I would also like to thank the NeRF Study Group members, @muyishen2040, @AndreaHsu, @Howardkhh, and VickyHuang1113. Numerous insights and experiences have been gained through the collaboration on the DriveEnv-NeRF project, which has significantly accelerated the development process of this project.

For a complete list of contributors to the code of this repository, please visit the contributor list.

Disclaimer: this is not an official NVIDIA product.