Support for streaming capture cards

[alexmi256]

Is your enhancement request related to a problem? Please describe.

No
Describe the solution you'd like

I'd like to know if I can use a USB capture card as the video source instead of the current desktop.

Describe alternative implementation you have seen

I know I could use OBS or similar to display the capture card on the desktop and then have selkies-gstreamer transmit the desktop but this seems inefficient (since cheap capture cards already encoded the video) and could introduce additional latency.
Additional context

The reason why I want to do this is so that I can play console content on my VR device. Thus, having the controller not connected to the PC is fine since it will be connected directly to the console.

• I confirm that this issue is relevant to the scope of this project. If you know that upstream projects are the cause of this problem, please raise the issue there.
• I confirm that I have read other open and closed issues and that duplicates do not exist.
• I confirm that I have described the solution as well as the problem in detail, and if possible, explained how to solve the problem.
• I confirm that no portion of this issue contains credentials or other private information, and it is my own responsibility to protect my privacy.
• I confirm that the authors of this issue does not willfully breach or infringe legal regulations, in any and all global law, regarding trademarks, trade names, logos, patents, or any and all other forms of external intellectual property, as well as adhering to software license terms of open-source and proprietary software projects.

[PMohanJ]

I don't have exposure to these VR devices so I didn't understand what you're trying to achieve by this. But, to answer your question: No, right now we can't use USB capture card as video source.
Though technically it's possible, I'm not sure if this feature aligns with the sellkies-gstreamer actual intent. It's just an opinion of mine.

[alexmi256]

I'm trying to get the output of a game console (HDMI output) to stream remotely.
I can use a capture card to get the video of the console but I don't know how to share that video output to another device while minimizing latency and hopefully retaining the same picture quality that the capture card gives.

Are there any projects or tools that you can recommend to achieve this?

[ehfd]

This is a very tricky use case that was raised in #103.

Few resources:
https://github.com/LizardByte/PyStream
https://github.com/LizardByte/RetroArcher
https://github.com/LizardByte/libdisplaydevice
https://github.com/darthcloud/BlueRetro
m1k1o/neko#371

For this, two things would have to be implemented:

• Video4Linux or UVC interface from the external physical device to GStreamer or FFmpeg in a PC for stream video processing - This alone can be done with OBS as well as Selkies with some dozen hours of code change if no control is required
• An API to send user control received from webrtc_input.py into the PC to the physical device

While the first is trivial, the second is multitudes trickier than the first and likely requires a certain implementation for every single console one wants to use. This is why I am hesitant to put this into the scope of our project.

The best way would be to create an interface that connects a PC to the device, and an interface to control the device if needed, without remote streaming, and instead use a remote desktop like Sunshine or Selkies to control or view from the PC which the console connects to into a client.

If you don't need remote controlling, latency is much less important, so something like OBS works.

[ehfd]

For VR, I recommend using WiVRn, ALVR, Virtual Desktop, etc. to stream from PC to VR.

Anyways, in order to "play", it is critical that some API to relay a virtual controller from PC to console is implemented (we cannot do this for you because of the wide variance in the scope of our project). Else, you are much better off with an emulator instead.

[PMohanJ]

Now I understand what this #103 is all about, back then I didn't understand it a bit.
I agree, streaming a USB video source is trivial, but the controlling part of it is tough if the controlling interface needs to support wide variety of retro devices. As explained in #103, a unified interface is required.

[ehfd]

In your case @alexmi256

If your objective is to use the VR device as just the display, close to the console device, and use a real controller to the console device yourself, you can surely use Selkies, Sunshine, or better, WiVRn or ALVR, after connecting a capture card into a PC. If the controller is directly connected to the console device, the second component can be worked around.

[alexmi256]

Thanks @ehfd for the detailed resources and alternative options, I will start looking those.
I 100% agree that this project (and others like Moonlight, Sunshine) should not care or try to implement support for custom controllers. Piping whatever the client sends and then letting the server deal with it seems like the most hassle free way of dealing with this.

The main part I'm trying to see is if I can re-stream whatever a capture card gives (50ms latency) which is usually MJPEG or YUY2 without re-encoding if possible since it will further degrade picture quality and potentially add more latency. For this I found webrtc-streamer but I didn't get to try it out.

[ehfd]

@alexmi256
Perhaps https://github.com/Glimesh/broadcast-box from the creators of Pion or https://github.com/CESNET/UltraGrid.

WebRTC doesn't typically support MJPEG or raw video, so it needs re-encoding.

[ehfd]

Please post here or reopen if there are addition questions or suggestions.