adaptive for blender 4.0

README.md

@@ -1,50 +1,169 @@
 # ESIM: an Open Event Camera Simulator
 
-[![ESIM: an Open Event Camera Simulator](http://rpg.ifi.uzh.ch/esim/img/youtube_preview.png)](https://youtu.be/ytKOIX_2clo)
+This repository contains the source code for the improved [ESIM](https://rpg.ifi.uzh.ch/esim.html) event camera simulator used in **Robust *e*-NeRF** <sub>[![Project Page](https://img.shields.io/badge/Project_Page-black
+)](https://wengflow.github.io/robust-e-nerf) [![arXiv](https://img.shields.io/badge/arXiv-black)](https://arxiv.org/abs/2309.08596) [![Code](https://img.shields.io/badge/Code-black)](https://github.com/wengflow/robust-e-nerf) [![Dataset](https://img.shields.io/badge/Dataset-black
+)](https://huggingface.co/datasets/wengflow/robust-e-nerf)</sub>. In particular, we incorporate the following changes:
+
+1. Event simulation model
+   - Improve the overall event simulation accuracy by accounting for additional edge cases 
+   - Improve the refractory period model by explicitly resetting the pixel reference intensity and timestamp at the end of the refractory period
+   - Desynchronize initial event generation across pixels by randomly initializing pixel reference timestamps
+   - Model junction leakage, which increases the rate of ON events and decreases the rate of OFF events
+   - Modify the pixel-to-pixel contrast threshold variation model to be time-independent
+   - Merge `feature/color` branch to support color event cameras
+2. Rendering engine
+   - Support [Blender](https://www.blender.org) as a rendering engine
+   - Support [Unreal Engine](https://www.unrealengine.com) 4.27.2 with a modified [UnrealCV](https://unrealcv.org/) plugin
+3. Camera trajectory
+   - Circumvent singularities in interpolating quaternion orientations by supporting rotation vector/angle-axis orientation representation in the trajectory CSV for interpolation
+4. Miscellaneous
+   - Fix various bugs & installation errors
+
+If you use this improved version of ESIM for your work, please cite:
+
+```bibtex
+@inproceedings{low2023_robust-e-nerf,
+  title = {Robust e-NeRF: NeRF from Sparse & Noisy Events under Non-Uniform Motion},
+  author = {Low, Weng Fei and Lee, Gim Hee},
+  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
+  year = {2023}
+}
+```
 
-This is the code for the 2018 CoRL paper **ESIM: an Open Event Camera Simulator** by [Henri Rebecq](http://henri.rebecq.fr), [Daniel Gehrig](https://danielgehrig18.github.io/) and [Davide Scaramuzza](http://rpg.ifi.uzh.ch/people_scaramuzza.html):
 ```bibtex
-@Article{Rebecq18corl,
-  author        = {Henri Rebecq and Daniel Gehrig and Davide Scaramuzza},
-  title         = {{ESIM}: an Open Event Camera Simulator},
-  journal       = {Conf. on Robotics Learning (CoRL)},
-  year          = 2018,
-  month         = oct
+@inproceedings{rebecq18_esim,
+  author = {Henri Rebecq and Daniel Gehrig and Davide Scaramuzza},
+  title = {{ESIM}: an Open Event Camera Simulator},
+  journal = {Conf. on Robotics Learning (CoRL)},
+  year = 2018
 }
 ```
-You can find a pdf of the paper [here](http://rpg.ifi.uzh.ch/docs/CORL18_Rebecq.pdf). If you use any of this code, please cite this publication.
 
-## Python Bindings
-Python bindings for the event camera simulator can be found [here](https://github.com/uzh-rpg/rpg_vid2e). 
-We now also support GPU support for fully parallel event generation!
+## Installation
+
+The following installation steps were tested on Ubuntu 20.04 and 22.04 with GTX 1080 Ti, RTX 3090 and RTX A5000 GPUs.
+
+### ESIM
+
+We recommend creating a new Catkin workspace specifically for ESIM. We name the workspace as `esim_ws` and place it under the home directory, as follows:
+```bash
+mkdir -p ~/esim_ws/src
+```
+
+Clone this repository into the source space of the workspace with:
+```bash
+cd ~/esim_ws/src
+git clone https://github.com/wengflow/rpg_esim.git
+```
+
+We also recommend using [Conda](https://docs.conda.io/en/latest/) to set up an environment with the appropriate dependencies for running ESIM, as follows:
+1. Install [Mamba](https://mamba.readthedocs.io/en/latest/index.html), an improved re-implementation of Conda, according to the [official instructions](https://mamba.readthedocs.io/en/latest/mamba-installation.html#mamba-install)
+2. Create the `esim` environment with:
+   ```bash
+   mamba env create -f rpg_esim/environment.yml
+   ```
+3. Activate the environment and initialize `rosdep` with:
+   ```bash
+   conda activate esim
+   rosdep init
+   rosdep update
+   ```
+
+Initialize and configure the workspace with:
+```bash
+catkin init
+catkin config --cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo
+vcs-import < rpg_esim/dependencies.yaml
+
+cd ze_oss
+touch imp_3rdparty_cuda_toolkit/CATKIN_IGNORE \
+      imp_app_pangolin_example/CATKIN_IGNORE \
+      imp_benchmark_aligned_allocator/CATKIN_IGNORE \
+      imp_bridge_pangolin/CATKIN_IGNORE \
+      imp_cu_core/CATKIN_IGNORE \
+      imp_cu_correspondence/CATKIN_IGNORE \
+      imp_cu_imgproc/CATKIN_IGNORE \
+      imp_ros_rof_denoising/CATKIN_IGNORE \
+      imp_tools_cmd/CATKIN_IGNORE \
+      ze_data_provider/CATKIN_IGNORE \
+      ze_geometry/CATKIN_IGNORE \
+      ze_imu/CATKIN_IGNORE \
+      ze_trajectory_analysis/CATKIN_IGNORE
+```
+We set `-DCMAKE_BUILD_TYPE=RelWithDebInfo`, instead of `-DCMAKE_BUILD_TYPE=Release`, to preserve the ease of debugging while still enabling code optimization during compilation.
+
+Build the `esim_ros` package with:
+```bash
+catkin build esim_ros
+```
+
+Lastly, add the following alias to your `.bashrc` file for activating the `esim` Conda environment and sourcing the [ROS](https://www.ros.org/) environment setup files:
+```bash
+alias cae='conda activate esim; source ~/esim_ws/devel/setup.bash'
+```
+
+The installation steps described above were consolidated from the following sources, with some modifications:
+1. https://github.com/uzh-rpg/rpg_esim/wiki/Installation
+2. https://github.com/uzh-rpg/rpg_esim/wiki/Installation-(ROS-Melodic)
+3. https://robostack.github.io/GettingStarted.html
+
+### Blender
+
+To use Blender as the rendering engine, we require [Blender as a Python module](https://docs.blender.org/api/current/info_advanced_blender_as_bpy.html) being installed in a separate `blender` Conda environment with a compatible Python version and [PyZMQ](https://pyzmq.readthedocs.io/en/latest/) also installed.
+
+We provide a Python wheel for Blender 3.4.0 at [this link](https://github.com/wengflow/rpg_esim/releases/download/v1.0/bpy-3.4.0a0-cp310-cp310-manylinux_2_31_x86_64.whl), which is compiled with [CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit) 11.4 and [NVIDIA OptiX](https://developer.nvidia.com/rtx/ray-tracing/optix) 7.3.0 support (requires NVIDIA R465.84 driver or newer), for Linux and Python 3.10. The `blender` environment may be created with all the necessary dependencies, including this wheel, with the following:
+   ```bash
+   conda env create -f rpg_esim/blender_environment.yml
+   ```
 
+Alternatively, build Blender as a Python module from source, according to the [official instructions](https://wiki.blender.org/w/index.php?title=Building_Blender/Other/BlenderAsPyModule), and install it via `pip` in the `blender` environment.
 
-## Features
+### UnrealEngine and UnrealCV
 
-- Accurate event simulation, guaranteed by the tight integration between the rendering engine and the event simulator
-- Inertial Measurement Unit (IMU) simulation
-- Support for multi-camera systems
-- Ground truth camera poses, IMU biases, angular/linear velocities, depth maps, and optic flow maps
-- Support for camera distortion (only planar and panoramic renderers)
-- Different C+/C- contrast thresholds
-- Basic noise simulation for event cameras (based on additive Gaussian noise on the contrast threshold)
-- Motion blur simulation
-- Publish to ROS and/or save data to rosbag
+To use UnrealEngine as the rendering engine, first install UnrealEngine 4.27 ([Windows/Mac](https://docs.unrealengine.com/4.27/en-US/Basics/InstallingUnrealEngine/), [Linux](https://docs.unrealengine.com/4.27/en-US/SharingAndReleasing/Linux/BeginnerLinuxDeveloper/SettingUpAnUnrealWorkflow/)). Then, build a compatible UnrealCV Plugin from [this modified source code](https://github.com/wengflow/unrealcv/tree/esim), according to the [official instructions](https://docs.unrealcv.org/en/master/plugin/install.html#compile-from-source-code).
 
-## Install
+## Using Blender as the Rendering Engine
 
-Installation instructions can be found in [our wiki](https://github.com/uzh-rpg/rpg_esim/wiki/Installation).
+### Configuration Options
+Configuration options for the Blender rendering engine are detailed at the top of `blender_renderer.cpp` of the `imp_blender_renderer` ROS package. Note that with N GPUs, the valid `blender_render_device_type` and `blender_render_device_id` combinations are as follows:
 
-## Run
+| `blender_render_device_type` | `blender_render_device_id` |
+| :---: | :---: |
+| 0 (CPU) | N |
+| 1 (CUDA) | 0 to N-1 |
+| 2 (OptiX) | N+1 to 2N |
 
-Specific instructions to run the simulator depending on the chosen rendering engine can be found in [our wiki](https://github.com/uzh-rpg/rpg_esim/wiki).
+A sample usage is provided in `cfg/blender.conf` of the `esim_ros` package.
 
-## Acknowledgements
+### Running ESIM with Blender
 
-We thank Raffael Theiler and Dario Brescianini for their contributions to ESIM.
-This research was supported by by Swiss National Center of Competence Research Robotics (NCCR), Qualcomm (through the Qualcomm Innovation Fellowship Award 2018), the SNSF-ERC Starting Grant and DARPA FLA.
+First, start the Blender rendering server on a given port (*e.g.* 5555) with:
+```bash
+cae
+roscd imp_blender_renderer
+conda activate blender
+python scripts/blender_bridge.py --port 5555
+```
+
+Then, run ESIM under a given configuration (*e.g.* `cfg/blender.conf`) in another terminal with:
+```bash
+cae
+roscd esim_ros
+roslaunch esim_ros esim.launch config:=cfg/blender.conf
+```
+
+To visualize the output of the simulator, you can open `rviz` (from a new terminal) as follows:
+```bash
+cae
+roscd esim_visualization
+rviz -d cfg/esim.rviz
+```
+
+You can also open `rqt` for more visualizations, as follows:
+```bash
+cae
+roscd esim_visualization
+rqt --perspective-file cfg/esim.perspective
+```
 
-A significant part of ESIM uses components (spline trajectories, inertial measurement unit simulation, various utility functions) from the [ze_oss](https://github.com/zurich-eye/ze_oss) project.
-ESIM depends on [UnrealCV](https://github.com/unrealcv/unrealcv) for the photorealistic rendering engine.
-We also reused some [code samples](https://github.com/JoeyDeVries/LearnOpenGL.git) from the excellent [Lean OpenGL](https://learnopengl.com/) tutorial in our OpenGL rendering engine.
-Finally, ESIM depends on the [Open Asset Import Library (assimp)](https://github.com/assimp/assimp) to load 3D models and Blender scenes within the OpenGL rendering engine.
+Please refer to the [ESIM wiki](https://github.com/uzh-rpg/rpg_esim/wiki) for further details on the usage of other rendering engines.

blender_environment.yml

@@ -0,0 +1,10 @@
+name: blender
+channels:
+  - conda-forge
+  - defaults
+dependencies:
+  - python=3.10.2
+  - pip=22.2.2
+  - pyzmq=24.0.1
+  - pip:
+    - https://github.com/wengflow/rpg_esim/releases/download/v1.0/bpy-3.4.0a0-cp310-cp310-manylinux_2_31_x86_64.whl

blender_environment_4_0.yml

@@ -0,0 +1,10 @@
+name: blender_4_0
+channels:
+  - conda-forge
+  - defaults
+dependencies:
+  - python=3.10.2
+  - pip=22.2.2
+  - pyzmq=24.0.1
+  - pip:
+    - https://github.com/wengflow/rpg_esim/releases/download/v1.0/bpy-3.4.0a0-cp310-cp310-manylinux_2_31_x86_64.whl

dependencies.yaml

@@ -1,45 +1,45 @@
 repositories:
   catkin_simple:
     type: git
-    url: git@github.com:catkin/catkin_simple.git
+    url: https://github.com/catkin/catkin_simple.git
     version: master
   ze_oss:
     type: git
-    url: git@github.com:uzh-rpg/ze_oss.git
-    version: master
+    url: https://github.com/wengflow/ze_oss.git
+    version: esim
   gflags_catkin:
     type: git
-    url: git@github.com:ethz-asl/gflags_catkin.git
+    url: https://github.com/ethz-asl/gflags_catkin.git
     version: master
   glog_catkin:
     type: git
-    url: git@github.com:ethz-asl/glog_catkin.git
-    version: master
+    url: https://github.com/wengflow/glog_catkin.git
+    version: esim
   eigen_catkin:
     type: git
-    url: git@github.com:ethz-asl/eigen_catkin.git
+    url: https://github.com/ethz-asl/eigen_catkin.git
     version: master
   eigen_checks:
     type: git
-    url: git@github.com:ethz-asl/eigen_checks.git
+    url: https://github.com/ethz-asl/eigen_checks.git
     version: master
   minkindr:
     type: git
-    url: git@github.com:ethz-asl/minkindr.git
+    url: https://github.com/ethz-asl/minkindr.git
     version: master
   minkindr_ros:
     type: git
-    url: git@github.com:ethz-asl/minkindr_ros.git
+    url: https://github.com/ethz-asl/minkindr_ros.git
     version: master
   yaml_cpp_catkin:
     type: git
-    url: git@github.com:ethz-asl/yaml_cpp_catkin.git
+    url: https://github.com/ethz-asl/yaml_cpp_catkin.git
     version: master
   rpg_dvs_ros:
     type: git
-    url: git@github.com:uzh-rpg/rpg_dvs_ros.git
-    version: master
+    url: https://github.com/wengflow/rpg_dvs_ros.git
+    version: esim
   assimp_catkin:
     type: git
-    url: git@github.com:uzh-rpg/assimp_catkin.git
+    url: https://github.com/uzh-rpg/assimp_catkin.git
     version: master

environment.yml

@@ -0,0 +1,31 @@
+name: esim
+channels:
+  - conda-forge
+  - robostack
+  - robostack-experimental
+  - defaults
+dependencies:
+  # Robostack ROS Noetic Dependencies
+  - python=3.8
+  - ros-noetic-desktop=1.5.0
+  - compilers=1.4.2
+  - cmake=3.23.1
+  - pkg-config=0.29.2
+  - make=4.3
+  - ninja=1.11.0
+  - colcon-common-extensions=0.3.0
+  - catkin_tools=0.8.5
+  - rosdep=0.21.0
+  - py-opencv=4.5.2   # Fix OpenCV version for identical imread() RGB-to-grayscale behaviour
+
+  # ESIM Dependencies
+  - vcstool=0.3.0
+  - ros-noetic-pcl-ros=1.7.2
+  - ros-noetic-hector-trajectory-server=0.5.2
+  - glfw=3.3.7
+  - glm=0.9.9.4
+  - cppzmq=4.8.1
+
+  # `glog_catkin` Dependencies
+  - automake=1.16.5   # Reference: https://github.com/buffer/pylibemu/issues/24#issuecomment-434268519
+  - autoconf=2.69     # Reference: https://github.com/ethz-asl/glog_catkin/issues/27#issuecomment-384325459

event_camera_simulator/esim/include/esim/esim/camera_simulator.hpp

@@ -13,23 +13,23 @@ class ImageBuffer
 
   struct ImageData
   {
-    ImageData(Image img, Time stamp, Duration exposure_time)
+    ImageData(ColorImage img, Time stamp, Duration exposure_time)
       : image(img),
         stamp(stamp),
         exposure_time(exposure_time) {}
 
-    Image image;
+    ColorImage image;
     Time stamp;
     Duration exposure_time; // timestamp since last image (0 if this is the first image)
   };
 
-  using ExposureImage = std::pair<Duration, Image>;
+  using ExposureImage = std::pair<Duration, ColorImage>;
 
   // Rolling image buffer of mazimum size 'buffer_size_ns'.
   ImageBuffer(Duration buffer_size_ns)
     : buffer_size_ns_(buffer_size_ns) {}
 
-  void addImage(Time t, const Image& img);
+  void addImage(Time t, const ColorImage& img);
 
   std::deque<ImageData> getRawBuffer() const { return data_; }
 
@@ -61,8 +61,8 @@ class CameraSimulator
     buffer_.reset(new ImageBuffer(exposure_time_));
   }
 
-  bool imageCallback(const Image& img, Time time,
-                     const ImagePtr &camera_image);
+  bool imageCallback(const ColorImage& img, Time time,
+                     const ColorImagePtr &camera_image);
 
 private:
   ImageBuffer::Ptr buffer_;

event_camera_simulator/esim/include/esim/esim/event_simulator.hpp

@@ -25,27 +25,33 @@ class EventSimulator
     double sigma_Cp;
     double sigma_Cm;
     Duration refractory_period_ns;
+    double leak_rate_hz;
     bool use_log_image;
     double log_eps;
+    bool simulate_color_events;
   };
 
-  using TimestampImage = cv::Mat_<ze::real_t>;
+  using TimestampImage = Eigen::Matrix<Time, Eigen::Dynamic, Eigen::Dynamic>;
 
   EventSimulator(const Config& config)
     : config_(config),
       is_initialized_(false),
-      current_time_(0)
+      last_time_(0)
   {}
 
   void init(const Image& img, Time time);
-  Events imageCallback(const Image& img, Time time);
+  Events imageCallback(const ColorImage& img, Time time);
 
 private:
+
   bool is_initialized_;
-  Time current_time_;
-  Image ref_values_;
   Image last_img_;
-  TimestampImage last_event_timestamp_;
+  Time last_time_;
+  Image ref_it_;
+  TimestampImage ref_timestamp_;
+  Image per_pixel_Cp_;
+  Image per_pixel_Cm_;
+  FloatType leak_gradient_;
   cv::Size size_;
 
   Config config_;