Research

Research and Development

Below is a synopsis of future research and development related to Motorcar and its subsystems. All of these come from discussions with Forrest in this Pre-Mailing List Discussion Archive. Please reference the link and search for key terms to see more of the details, thought they should all be stated here.

Build system

Moving from qmake and distributed make files to CMake

Separation of Motorcar sub-components

Involves defining an API for the Motorcar compositor and sub-components. This can be broken into:

• Compositor
• Windowing Library
• Input subsystem

Remove dependency on QTWayland

Forest's Comments

This might qualify better as future development since its a pretty colossal undertaking QtWayland is really cool because it makes it really easy to write Wayland compositors, but its not really designed to interact with third party extensions and doesn't expose any of the Wayland resources over the public API Since Motorcar relies on a set of protocol extensions for 3D windowing and 3D input events, it has to access some of the private QtWayland headers, which creates a really nasty build dependency Eliminating this would require replacing QtWayland with a raw Wayland compositor interface and some way of getting an OpenGL/EGL context in the compositor (which could be normal Qt or something like SDL2 or more raw Wayland). This is doable, and would be a gigantic benefit to Motorcar, but would be a huge amount of code to write. This is on my long term todo list

Creating Windowing Toolkit

Forrest's Comments

Client side code for 3D clients Right now the motorcar demo client is a monolithic 1600 line file of mixed C and C++ because I was running out of time when I started working on it so I didn't take the time to do it right. This is basically awful in every way possible. Fixing this is going to be the first major development item I'm planning on tackling, I just want to make a better video first I'm planning on restructuring it as a client side toolkit which provides basic 3D windowing services to clients over a simple interface, so that clients, and UI toolkits wont have to interact directly with Wayland.

• Integration with VRUI
• This is partially dependent on the client-side code cleanup
• VRUI provides a 3D UI framework for immersive 3D applications, and there are several applications written for it
• Driving VRUI's device backend from the Motorcar client library would allow VRUI applications to run inside of Motorcar, which would be really tight
• VRUI also has a fairly mature device backend with support for many devices, so driving the motorcar compositor off of VRUI's device backend could also be really valuable
• I don't know what this would actually involve, I believe VRUI has significant X dependencies that could make it very difficult to integrate with wayland and therefore motorcar. It could get really nasty

Items to consider

Determine the path forward for a windowing toolkit

• VRUI, could we port this to Wayland and run within Motorcar?
• Familiarity with OpenGL required.

Augmented Reality

Forrest's Comments

This is what I'm really excited for, I just want to make sure the existing code is reasonably well structured before I jump into it.

Basically the idea I like is to put pass through cameras on the front of the rift, along with a depth camera (or two) and try and mix the client applications with reality.

I also want to try and do 6DoF tracking with computer vision (basically some kind of stereo odometry or real-time RGB-D SLAM) so that the system could be run on a completely mobile platform

Compositor Optimizations

Forrest's Comments

One of the critical problems facing Motorcar. Basically the way Motorcar embeds 3D content from clients in the 3D compositor space (by having the client redraw its content with the new viewpoint information each frame) creates a synchronization problem as soon as any client fails to finish drawing the frame before the viewpoint updates, which is of course more likely with heavier clients. There are several different ways to handle this. You can wait for all of the clients to finish drawing their content before you update, but this means that a single slow client can cause the compositor to drop frames and, even worse, a client that hangs can cause the compositor to hang. You can always draw the most recent content from each client but then if a client starts to drop frames its content appears to move relative to its window bounds (this is basically what I’m doing now). The approach suggested to me by some of the devs on the Wayland IRC is to set a timeout for new content and have the compositor wait for all clients until the timeout occurs, and then use the most recent frame from clients that failed to finish by the timeout. This allows the compositor to set a minimum frame rate and not drop below it, but also prevents it from dropping frames from a client thats just barely behind schedule. I think this makes a lot of sense and I’d be interested in implementing this, but its not super high on the list because the results aren’t bad right now (and I think a lot of what makes clients slow is the EGL depth buffer thing.)

Input System

Forrest's Comments

Yeah the input problem is really interesting, and input abstraction is a huge part of the potential that Motorcar holds. Not just for motion controllers like the hydra, but also for skeletal input (like finger tracking) and using it to drive touch input to 2D applications. Also gesture input is a really interesting problem, since you need a general way to represent gestures and specific gesture events that happen when those gestures are recognized. There is a lot of interesting work here.

Skeletal Input

Basically there's a bunch of devices on the market that give you skeletal pose data in one form or another, and it would be really cool to bring that into Motorcar and deliver it as input to 3D applications (and use it to drive touch input in 2D applications)

Handling this input abstractly is paramount to keeping the system modular, and in many cases it may be necessary to combine data from multiple sensors into a single skeletal model, which is not a trivial task

A friend of mine in grad school did his thesis on this, resulting in a system called Jester, which I think is pretty well suited for being a device layer.

I don't think hes particularly interested in getting involved in motorcar but if we start using Jester we could probably drag him into it

Gesture Input

This is related to the former, but not necessarily dependent on it Basically I think it would be really cool to use gestures both as application input and also as window control, but doing this right is pretty complicated

• You need some kind of gesture recognizer on a system level that takes input from the device agnostic skeleton model

• You need some way of representing gestures abstractly so that they can be passed around between different pieces of software (not only so the comopsitor can send clients gesture events but also so clients can request that the compositor recognize a gesture on their behalf)

• I have several novel length reddit discussions about this that I could copy-paste in here if you want to get into this

More of Forrest's comments regarding mouse input

I haven’t gotten far, I was working on it early on before I had the rift and the hydra, and it made a lot of sense because with a single viewport its really straightforward to project the mouse position on the screen into 3D, but as soon as you start working in the rift there isn't really an obvious way to take the mouse’s two degrees of freedom and derive 3 degree of freedom interaction from it.

I was thinking you could have the mouse move a cursor on the surface of a sphere centered on the head position, and then project a ray through the center of the sphere and the mouse position to get 3D selection, but even then you still only have 2 degrees of freedom. You could control the radial degree of freedom with the scroll wheel, but then you don't have a way to scroll in applications.

Basically the mouse is fundamentally a 2D input device, so using it for 3D input requires that you come up with at least one degree of freedom from something else, and this is basically why I just stopped dealing with it in favor of using the hydra. If you have any ideas I’d love to hear them, because I don’t think the other people working on this have hydras either, so they’re kind of in the same boat as you. This could be something you look into if you like, but it will probably involve some nasty Qt bullshit to constrain the mouse, I haven't really looked into it.

EGL Depth buffer extensions

Forrest's Comments

The process of encoding the client depth buffers into the EGL color buffers is a huge amount of GPU overhead and I would imagine that eliminating it would be one of the most significant single optimizations possible. That being said, I really don't have the slightest idea where to start with this, I would probably go hit up the wayland IRC channel to see if I could get some pointers to good resources on the subject and then work from there. Basically the only reason I think this problem could be solved with EGL extensions is because someone on the Wayland channel told me it might be possible, there could be huge obstacles that I'm completely unaware of. If you’re interested in researching this I could try and find some info about it, I don’t know what documentation (if any) exists on implementing EGL extensions and if so where this documentation would hypothetically exist.

In summary:

Issue of concern is that encoding the client depth buffers into ELG color buffers involves a lot of GPU intensive work. Eliminating this requirement would be very beneficial, and possibly one of the most single beneficial optimization regarding the Motorcar compositor.

It is unclear where to start but some Ideas involve:

• Open communication with the Wayland team via IRC
• EGL extensions are a hypothetical proposition at solving a problem at this time. More information from the wayland team and investigation of this space would be required.

Promote too and find others interested in Motorcar and related subject matter.

PR is a big thing when it comes to any project and it sustainability. Someone who can help maintain interest in the subject and recruit others to help out that is a great win for Motorcar.

• Be a promoter and point of contact on various VR related forums and sites including: Oculus VR forums, reddit, Wayland and more. Any more would go under an Outreach page within the Motorcar Wiki.