neuromorphic_drivers is a library to interact with USB Neuromorphic devices. The drivers were written from scratch with portability and performance in mind.
| Name | Type | Resolution | Data types | Mask | Synchronize | Rate limiter | Temperature | Illuminance |
|---|---|---|---|---|---|---|---|---|
| Prophesee EVK4 | Camera | 1280 × 720 | DVS, trigger | ✓ | ✓ | ✓ | ✓ | ✓ |
| Prophesee EVK3 HD | Camera | 1280 × 720 | DVS, trigger | ✓ | - | ✓ | - | - |
This table lists fratures supported by this library. Some devices support unlisted features or features marked as "no" that have yet to be added to neuromorphic_drivers.
| Name | Links |
|---|---|
| Prophesee EVK4 | https://www.prophesee.ai/event-camera-evk4/ |
| Prophesee EVK3 HD | https://www.prophesee.ai/event-based-evk-3/ |
pip install neuromorphic_driversOn Linux, run the following comman after installing the package to install UDEV rules.
neuromorphic-drivers-install-udev-rulesThe following script reads data from a connected device.
import neuromorphic_drivers as nd
nd.print_device_list() # print a table that lists connected devices
with nd.open() as device:
for status, packet in device:
if "dvs_events" in packet:
# packet["dvs_events"] is a structured numpy array
# with dtype [("t", "<u8"), ("x", "<u2"), ("y", "<u2"), ("on", "?")])
pass
if "trigger_events" in packet:
# packet["trigger_events"] is a structured numpy array
# with dtype [("t", "<u8"), ("id, "<u1"), ("rising", "?")])
passThe keys in a packet depend on the device and are only present if the associated array is not empty. Packets contain variable numbers of events (typically a few thousand to a few hundred thousand) covering a variable amount of time (typically a few dozen microseconds to a few milliseconds).
The configuration, specific to each device, defines all the device's parameters – from biases to region of interest. It can be specified when opening a device and it can be updated at any time. The default configuration for device d is defined in python/neuromorphic_drivers/generated/devices/d.py.
import neuromorphic_drivers as nd
configuration = nd.prophesee_evk4.Configuration(
biases=nd.prophesee_evk4.Biases(
diff_off=170,
diff_on=130,
)
)
with nd.open(configuration=configuration) as device:
for status, packet in device:
...
if not_sensitive_enough:
configuration.biases.diff_on -= 10
configuration.biases.diff_off -= 10
device.update_configuration(configuration)nd.open may open any supported device. However, it only considers matching devices if a configuration is specified. Passing the wrong configuration type to update_configuration raises an error.
Prophesee's EVK4 has a hardware event-rate limiter that randomly drops events when the sensor generates more than a given number per second. The limiter has two parameters. reference_period_us defines a count period in µs (maximum 200). maximum_events_per_period defines the number of events per period beyond which the limiter activates. The effective limit is maximum_events_per_period / reference_period_us in events/µs.
import neuromorphic_drivers as nd
configuration = nd.prophesee_evk4.Configuration(
rate_limiter=nd.prophesee_evk4.RateLimiter(
reference_period_us=200,
maximum_events_per_period=4000,
)
)
with nd.open(configuration=configuration) as device:
...Converting the raw USB data into events can be an expensive operation if the data rate is high. Raw mode skips parsing and can be useful if one simply wishes to store the data into a file to be processed offline.
import neuromorphic_drivers as nd
with nd.open(raw=True) as device:
for status, packet in device:
# in raw mode, packet is a "bytes" objectdef open(
# initial device configuration
configuration: typing.Optional[Configuration] = None,
# timeout for each iteration (for status, packet in device)
# By default, the iterator blocks until data is available.
# In some cases (other time-driven calculations, graphics...),
# it can be desirable to run an iteration of the loop even if no data is available.
# If iterator_timeout is not None, packet may be None.
iterator_timeout: typing.Optional[float] = None,
# whether to skip data parsing
raw: bool = False,
# device serial number, None selects the first available device.
# Use nd.list_devices() to get a list of connected devices and their serials.
serial: typing.Optional[str] = None,
# USB software ring configuration, None falls back to default.
usb_configuration: typing.Optional[UsbConfiguration] = None,
# maximum number of raw USB packets merged to create a packet returned by the iterator.
# Under typical conditions, each iterator packet is built from one USB packet.
# However, if more than one USB packet has already been received,
# the library builds the iterator packet by merging the USB packets,
# to reduce the number of GIL (Global Interpreter Lock) operations.
# While merging speeds up processing in such cases, it can lead to large latency spikes if too many packets are merged at once.
# iterator_maximum_raw_packets limits the number of packets that are merged, even if more USB packets have been received.
# The remaning USB packets are not dropped but simply returned on the next iteration(s).
iterator_maximum_raw_packets: int = 64,
): ...The fields of the USB configuration are identical across devices, but the default values depend on the device. The values below are for a Prophesee EVK4.
@dataclasses.dataclass
class UsbConfiguration:
buffer_length: serde.type.uint64 = 131072 # size of each buffer in the ring, in bytes
ring_length: serde.type.uint64 = 4096 # number of buffers in the ring, the total size is ring_length * buffer_length
transfer_queue_length: serde.type.uint64 = 32 # number of libusb transfers submitted in parallel
allow_dma: bool = False # whether to enable Direct Memory AccessSee python/examples for different usage examples.
python/examples/any_display.py implements a live event viewer with exponential decays caculated by the GPU. It requires vispy and glfw (pip install vispy glfw pyopengl).
python/examples/evk4_plot_hot_pixels generates plots and require Plotly (pip install plotly pandas kaleido).
Local build (first run).
cd python
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install maturin numpy
maturin develop # or maturin develop --release to build with optimizationsLocal build (subsequent runs).
cd python
source .venv/bin/activate
maturin develop # or maturin develop --release to build with optimizationsBefore pushing new code, run the following to lint and format it.
cd python
isort .; black .; pyright .The files in python/python/neuromorphic_drivers/generated are generated by the build script (python/build.rs) and should not be modified directly. They are included in this repository for documentation purposes.
When making changes to the Rust library and the Python wrappers at the same time, change neuromorphic-drivers = "x.y" to neuromorphic-drivers = {path = "../drivers"} to use the most recent version of the Rust code.
See https://docs.rs/neuromorphic-drivers/latest/neuromorphic_drivers/ for documentation.
-
Write the following content to /etc/udev/rules.d/65-neuromorphic-drivers.rules.
SUBSYSTEM=="usb", ATTRS{idVendor}=="152a",ATTRS{idProduct}=="84[0-1]?", MODE="0666" SUBSYSTEM=="usb", ATTRS{idVendor}=="04b4",ATTRS{idProduct}=="00f[4-5]", MODE="0666" -
Run the following commands (or reboot the machine).
sudo udevadm control --reload-rules sudo udevadm trigger
Run a specific test.
cd drivers
cargo test --release read -- --nocapturePublish on crates.io.
cargo publish -p neuromorphic-types
cargo publish -p neuromorphic-driversRecent event-based cameras, such as the Prophesee EVK4, can generate more data than can be processed in real-time under certain circumstances. While the data can always be moved to the computer's memory in real-time, the simplest algorithms (including converting the raw USB bytes to a {t, x, y, polarity} event representation) struggle to keep up during data rate peaks. This library uses seperate threads for reading (USB to memory) and processing (memory to memory or disk) with a circular buffer (ring) at the interface. Short data bursts are seemlessly absorbed by the ring and are typically not an issue, even though they brifely cause a spike in latency. However, persistent high data rates cause the ring to slowly fill and cause the program to eventually crash. Depending on the use-case, one of the following work arounds can be applied:
- reduce the camera's sensitivity (usually by changing
diff_offanddiff_on) - enable the event rate limiter if the device supports it (the limiter randomly drops events before sensing them to the computer, reducing bandwidth issues but significantly degrading the quality of transient bursts)
- reduce the camera's spatial resolution by masking rows and columns
- change the environment if possible (avoid flickering lights, reduce the optical flow, remove background clutter, keep large and fast objects out of the field of view)
- call
device.clear_backlog(until=0)whenever thebacklogbecomes too large (the maximum backlog is the size of the ring minus the size of the transfer queue) - use
nd.open(raw=true)to skip the parser and directly access the USB bytes, typically to save them to a file
This library relies on libusb for all USB communications. libusb supports Direct Memory Access (Linux only for now, though other platforms may be added in the future), which allows the USB controller to directly write packets to memory without requiring CPU (and OS kernel) intervention. While this can increase performance and reduce CPU usage, DMA comes without caveats that we disabled it by default. Users when knoweldge of their USB controller and its limitations may want to enable it to increase performance on embedded systems.
USB drivers have a limited number of DMA file objects (128 / 256 / 512 / 1024). This is typically not enough to accomodate event bursts (we use 4096 buffers by default for the EVK4, with 131072 bytes per buffer). The code fall backs to non-DMA buffers if all the DMA buffers are used (for instance the first 128 buffers would be DMA and the rest would be non-DMA) and may result in variable performance over time.
Using all the available DMA buffers can cause other USB transfers to fail (including control transfers to configure the device).
Using DMA thus requires one of the following workarounds:
- Use a small number of DMA buffers and copy packets to a larger ring before processing (this somewhat defeats the purpose of DMA and increases memory copies).
- Use larger buffers (around 1 MB) to inccrease the ring size without increasing the number of buffers, at the cost of increased latency.
- Ensure that processing can always keep up with the data rate (sparse scenes / low-sensitivity biases / event-rate limiter / simple processing). Note that simply expanding vectorized EVT3 events to 13-bytes DVS events is not real-time during data peaks.