Cannot optimize with velocities

[jerredchen]

Hi,

Thank you for this codebase! I would like to test the velocity optimization for the method by running python train.py --dataset=synthetic-mb --case=9. However, I'm running into the following error:

File "/home/jerred/3dgs-deblur/nerfstudio/nerfstudio/models/splatfacto.py", line 491, in after_train
    assert self.xys.absgrad is not None  # type: ignore
AttributeError: 'Tensor' object has no attribute 'absgrad'

The full error message:

Traceback (most recent call last):
  File "/home/jerred/miniconda3/envs/nerfstudio_copy/bin/ns-train", line 8, in <module>
    sys.exit(entrypoint())
  File "/home/jerred/3dgs-deblur/nerfstudio/nerfstudio/scripts/train.py", line 262, in entrypoint
    main(
  File "/home/jerred/3dgs-deblur/nerfstudio/nerfstudio/scripts/train.py", line 247, in main
    launch(
  File "/home/jerred/3dgs-deblur/nerfstudio/nerfstudio/scripts/train.py", line 189, in launch
    main_func(local_rank=0, world_size=world_size, config=config)
  File "/home/jerred/3dgs-deblur/nerfstudio/nerfstudio/scripts/train.py", line 100, in train_loop
    trainer.train()
  File "/home/jerred/3dgs-deblur/nerfstudio/nerfstudio/engine/trainer.py", line 264, in train
    callback.run_callback_at_location(
  File "/home/jerred/3dgs-deblur/nerfstudio/nerfstudio/engine/callbacks.py", line 115, in run_callback_at_location
    self.run_callback(step=step)
  File "/home/jerred/3dgs-deblur/nerfstudio/nerfstudio/engine/callbacks.py", line 105, in run_callback
    self.func(*self.args, **self.kwargs, step=step)
  File "/home/jerred/3dgs-deblur/nerfstudio/nerfstudio/models/splatfacto.py", line 491, in after_train
    assert self.xys.absgrad is not None  # type: ignore
AttributeError: 'Tensor' object has no attribute 'absgrad'
Traceback (most recent call last):
  File "train.py", line 353, in <module>
    process(args.input_folder, args)
  File "train.py", line 193, in process
    subprocess.check_call(cmd)
  File "/home/jerred/miniconda3/envs/nerfstudio_copy/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ns-train', 'splatfacto', '--data', 'data/inputs-processed/synthetic-mb/cozyroom', '--viewer.quit-on-train-completion', 'True', '--pipeline.model.rasterize-mode', 'antialiased', '--pipeline.model.use-scale-regularization', 'True', '--pipeline.model.num-downscales', '0', '--pipeline.model.background-color', 'auto', '--pipeline.model.cull-scale-thresh', '2.0', '--pipeline.model.optimize-eval-velocities=False', '--pipeline.model.blur-samples=10', '--max-num-iterations', '20000', '--vis=tensorboard', '--pipeline.model.rolling-shutter-compensation=False', '--pipeline.model.min-rgb-level=10', '--pipeline.model.camera-velocity-optimizer.enabled=True', '--output-dir', 'data/outputs/synthetic-mb/motion_blur-velocity_opt', 'nerfstudio-data', '--orientation-method', 'none', '--eval-mode', 'interval', '--eval-interval', '8', '--optimize-eval-cameras', 'True']' returned non-zero exit status 1.

To reproduce, I am running on Ubuntu 22.04, Python 3.8, CUDA 11.8 with an Nvidia 3090 RTX. I followed the instructions from the README for installing a conda environment with the dependencies to install Nerfstudio.

[oseiskar]

Did you have a previous Nerfstudio installation in the same Conda environment or did you start from scratch? This error could happen if you have the wrong version of Nerfstudio or gsplat installed. Could you try again with a fresh Conda environment and running ./scripts/install.sh, please?

[jerredchen]

Thanks for your reply. I actually did try with a fresh conda environment following here followed by ./scripts/install.sh (did not run pip install nerfstudio).

I also noticed that the velocity optimization is fine for using the downloaded SAI data, is there a difference in between the synthetic and the real data that causes this?

[oseiskar]

Ok. Thanks for reporting. This actually has nothing to do with installation issues.

It appears there's currently a bug/limitation in the implementation: you cannot enable velocity optimization without also enabling pose optimization.

So this does not work: python train.py --dataset=synthetic-mb --case=9
but this does: python train.py --dataset=synthetic-mb --case=25.

(for reference here's the output of python train.py --dataset=synthetic-mb showing the options)

▹ python train.py --dataset=synthetic-mb
valid cases
1:	baseline	data/inputs-processed/synthetic-mb/cozyroom
2:	baseline	data/inputs-processed/synthetic-mb/factory
3:	baseline	data/inputs-processed/synthetic-mb/pool
4:	baseline	data/inputs-processed/synthetic-mb/tanabata
5:	motion_blur	data/inputs-processed/synthetic-mb/cozyroom
6:	motion_blur	data/inputs-processed/synthetic-mb/factory
7:	motion_blur	data/inputs-processed/synthetic-mb/pool
8:	motion_blur	data/inputs-processed/synthetic-mb/tanabata
9:	motion_blur-velocity_opt	data/inputs-processed/synthetic-mb/cozyroom
10:	motion_blur-velocity_opt	data/inputs-processed/synthetic-mb/factory
11:	motion_blur-velocity_opt	data/inputs-processed/synthetic-mb/pool
12:	motion_blur-velocity_opt	data/inputs-processed/synthetic-mb/tanabata
13:	motion_blur-velocity_opt-zero_init	data/inputs-processed/synthetic-mb/cozyroom
14:	motion_blur-velocity_opt-zero_init	data/inputs-processed/synthetic-mb/factory
15:	motion_blur-velocity_opt-zero_init	data/inputs-processed/synthetic-mb/pool
16:	motion_blur-velocity_opt-zero_init	data/inputs-processed/synthetic-mb/tanabata
17:	pose_opt	data/inputs-processed/synthetic-mb/cozyroom
18:	pose_opt	data/inputs-processed/synthetic-mb/factory
19:	pose_opt	data/inputs-processed/synthetic-mb/pool
20:	pose_opt	data/inputs-processed/synthetic-mb/tanabata
21:	pose_opt-motion_blur	data/inputs-processed/synthetic-mb/cozyroom
22:	pose_opt-motion_blur	data/inputs-processed/synthetic-mb/factory
23:	pose_opt-motion_blur	data/inputs-processed/synthetic-mb/pool
24:	pose_opt-motion_blur	data/inputs-processed/synthetic-mb/tanabata
25:	pose_opt-motion_blur-velocity_opt	data/inputs-processed/synthetic-mb/cozyroom
26:	pose_opt-motion_blur-velocity_opt	data/inputs-processed/synthetic-mb/factory
27:	pose_opt-motion_blur-velocity_opt	data/inputs-processed/synthetic-mb/pool
28:	pose_opt-motion_blur-velocity_opt	data/inputs-processed/synthetic-mb/tanabata
29:	pose_opt-motion_blur-velocity_opt-zero_init	data/inputs-processed/synthetic-mb/cozyroom
30:	pose_opt-motion_blur-velocity_opt-zero_init	data/inputs-processed/synthetic-mb/factory
31:	pose_opt-motion_blur-velocity_opt-zero_init	data/inputs-processed/synthetic-mb/pool
32:	pose_opt-motion_blur-velocity_opt-zero_init	data/inputs-processed/synthetic-mb/tanabata