Skip to content

Latest commit

 

History

History
209 lines (135 loc) · 10.2 KB

README.md

File metadata and controls

209 lines (135 loc) · 10.2 KB
Raw

Quokka + VR Streaming Project

Quick and dirty integration prototype of Raspberry Pi, FPGA, and Unity3D for image streaming from camera to headset with axis control - a weekend project to hack around.

Watch demo on YouTube - https://www.youtube.com/watch?v=7mvU3ZHIpUw

Demo

Setup

We are going to build integrated system like that

Setup

What is it doing?

  • Camera fixture is made of 3 servos to give 3 axis control for direction of view
  • Pi is running web server API to serve images and receive servos data
  • VR headset displays image from Pi camera by periodically polling for latest frame with augmented face detection layer.
  • VR headset periodically sends internal camera direction to Pi as json payload e.g. { x: 45, y: -45, z: 0 }
  • Pi translates incoming camera direction into servo rotation data.
  • Pi sends servos data to FPGA using UART with ad-hoc protocol as byte array [marker, servo1, servo2, servo3] e.g. [255, 135, 45, 90]
  • FPGA reads data from UART and sets values for each servo
  • FPGA runs parallel servo drivers and each driver sends PWM signal to attached servo motor

Prerequisites

To make this project, you will need bunch of hardware, please run through setup and make sure you got all required bits and pieces.

Choose FPGA vendor and dev board. There are plenty around, wide range on boards from proprietary and open source projects.

I designed my own board for Quokka Toolkit and using it in this project. (https://www.kickstarter.com/projects/quokka-robotics/quokka-fpga-iot-controller)

Quokka Board

Sample design will also work on any device of your choice.

Installing and Assembling

FPGA

PDB

PDBImg

Choose some pins on your board to be used for servo control and connect things together.

FPGA

Once you have all connected, you need to make sample design for FPGA.

I am creating a toolkit for quick FPGA development using C# language instead of traditional HDL languages. (https://github.com/EvgenyMuryshkin/QuokkaEvaluation)

FPGA part is built using components that already available as part of my toolkit. All required drivers are included in this tutorial, so no need to download anything in addition. Please refer to main toolkit page for more examples and drivers

NOTE: example is using Verilog. VHDL is also supported with configuration change in quokka cli

  • Check out PC setup section for software installation
  • Clone this repo into some directory e.g. qvr
  • Open solution from qvr\fpga\qvr.sln in Visual Studio
  • Edit qvr\verilog.json configuration file and set your board clock rate in Hz
  • Run cli project, it is already configured for sample parameters. You should see console output like next

CLI

  • Create new project in qvr\fpga\Verilog folder using EDA software for your device
  • Add all *.v files from Verilog folder and from generated subfolder (sample project for Quokka board already provided for reference, if you use similar device, project can be reused).

There are two controller - Main and Test.

Main controller is reading data from UART and applying it to servos

Test controller is self contained, driving servos from side to side, can be used to test FPGA setup independently

  • Set top-level controller, run compilation, pin assignment, synthesys and programming. You should get working FPGA device if everything was setup properly. If it does not work - please refer to troubleshooting section and have fun debugging :)

  • Use Test controller to check your setup, then switch to main contoller and carry on with the tutorial

Watch test controller demo on YouTube - https://www.youtube.com/watch?v=JQwfiyl2v78

Demo

Raspberry Pi

My final setup looks like that PI

Most of the required packages already come with NOOBS disto, which I am using. If you have another setup, you need to install missing packages as you go.

cd ~
git clone https://github.com/EvgenyMuryshkin/qvr.git
  • Run setup to download nets and configure environment
cd ~/qvr/pi
chmod 777 setup
./setup
source ~/.bashrc

Setup script downloads nets and creates alias snn to run streaming server. If all went well, you should be able to run it from terminal

snn
  • Run snn to start web server and image processing, you should see next console output SNN
  • Test connection from PC browser by navigating to << PI_ADDRESS >>:8000 (http://192.168.1.201:8000)

Pi is all set up and ready to go

Camera fixture

Assembly something like that with parts from ROT2U kit. I tried to keep all axis close together so it moves similar to head movement. Make sure it is balanced and does not tip over when rotates.

Fixture

PC

Install and configure all required software, make sure you are familiar with how to open projects, build and run tools etc.

I am using VSCode Remote SHH on PC to edit and run software on Pi side, it is super awesome and time saving tool. VSCode

If you don't have a VR headset, please refer to instructions on how to emulate it inside Mixed Reality Portal.

That is about it, hope you have fun and learn new thing while doing this small exercise

Challenges

Challenge 1

Disable face detection to increase frame rate.

Replace polling of image with direct handling of mjpg to get smooth video stream from Raspberry Pi

See links for reference

(https://answers.unity.com/questions/1228247/unity-5-c-simple-html-viewer.html)

(https://github.com/DanielArnett/SampleUnityMjpegViewer)

Challenge 2

Modify servo angle adjustment logic in FPGA to make it as real time as possible to actual head movement

Challenge 3

Add VR controllers handling and wheel base to Pi and FPGA so they can be moved around.

Troubleshooting

Most of the issues can be found on Stack Overflow. Many of them related to missing packages, lack of permissions, different hardware configuration. If you encounter any issues, please create PR so we can update this section accordigly.

Problem Solution
No UART communications Make sure that RXD-TXD pairs are connected property, try to make loopback (connect it to itself) to test it locally
Pi server does not receive data on UART If you don't have permission to serial port, it might just silently not work, no errors, no crashes, no data. One of provided solutions might work for you (https://lb.raspberrypi.org/forums/viewtopic.php?t=197823)
No image from camera appears in Unity snn server is not running in Pi - run server and check stream in browser
No image from camera appears in Unity Make sure that you have correct IP address of Pi in component. Open Assets/AxisComponent.cs and edit Pi address if is not correct

Contacts

Follow me on Twitter - https://twitter.com/ITMayWorkDev

E-mail: [email protected]