feat(input/linux): add support for more virtual input devices

[Hazer]

Description

This is a direct continuation of #2315 @ABeltramo 's amazing work.

• Rebased to master (f0a00ae)
• Adopted new vue components
• Broken the inputtino.cpp into namespaces by kind
• Renamed input.cpp to legacy_input.cpp

I had to rename the branch for CI to work, this is a continuation of #2261

Added support for the followings new virtual devices:

• Nintendo Switch pro joypad

• DualSense (PS5) joypad, including:

  • Gyroscope
  • Accelerometer
  • Touchpad
  • LED
  • Battery status
  • Force feedback? (is this supported in Moonlight already? I have to check..)
  • Microphone and Headset jack? (we should be able to plug into that as well, to be checked..)

And generally completely refactored all virtual input code using inputtino as a library

Requirements

The DualSense implementation requires access to /dev/uhid, (more on this later..) this is easily done by setting the right permissions, similar to how we deal with udev already:

First we’ll add our user to the input group:

sudo usermod -a -G input $USER

Then we add an udev rule to allow access to /dev/uhid:

echo 'KERNEL=="uhid", GROUP="input", MODE="0660"' | sudo tee /etc/udev/rules.d/99-uhid.rules

Finally, we have to make sure that the uhid kernel module is loaded at boot:

echo "uhid" | sudo tee /etc/modules-load.d/uhid.conf

Code details

The main file is src/platform/linux/inputtino.cpp I've kept the original input.cpp so that we can easily compare the two implementations and possibly add what's missing but the idea would be to completely replace that implementation with this. To switch from one implementation to the other you can just set the Cmake variable -DSUNSHINE_USE_INPUTTINO=OFF.

Unfortunately we have to deal with uhid in order to properly emulate gyro, acceleration and touchpad; there's a bit of a rationale into why this is not possible with udev here. This might be a deal breaker for you guys but I hope it's not; if that's the case, we can still emulate a PS5 joypad using udev without supporting gyro and acceleration.

I think bringing in uhid has also other benefits, since it sits at a lower level than uinput, we can really emulate any kind of usb device and possibly even completely integrate usb over IP if this will ever be considered as an addition to the protocol.

As for the code itself, Inputtino has got unit tests using libinput for mouse, keyboard, pen and touchscreen and SDL2 for testing joypads. These aren't complete (yet) but should cover most of the codebase.

Future plans

• I'd like to give it a go at implementing gyro and acceleration support for a Nintendo joypad too.

• Ideally, since we are emulating the real joystick USB messages, this should also be a good foundation for a Windows virtual driver implementation. I know nothing about this but I'd like to do more research on it and possibly add a multiplatform implementation to inputtino.

  • I know of the EV license and all the requirements, for some reason it's hard to convince companies to invest in this without a functioning prototype first 😅

• Add support for trackpads in Moonlight? Generally it seems to not be the same as a touchscreen: it moves relatively instead of absolutely and has got different gestures.

What's missing?

Most notably the XTest fallback that is present in input.cpp if this is something that is used we can easily add it here as well. Everything else should be implemented, please let me know if something is missing or misbeaving.

I'm probably forgetting something else..

Issues Fixed or Closed

Might help with #1720

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Dependency update (updates to dependencies)
• Documentation update (changes to documentation)
• Repository update (changes to repository files, e.g. .github/...)

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added or updated the in code docstring/documentation-blocks for new or existing methods/components

Branch Updates

LizardByte requires that branches be up-to-date before merging. This means that after any PR is merged, this branch
must be updated before it can be merged. You must also
Allow edits from maintainers.

• I want maintainers to keep my branch updated

[codecov]

Codecov Report

Attention: Patch coverage is 3.57143% with 378 lines in your changes missing coverage. Please review.

Project coverage is 8.70%. Comparing base (5f1a17f) to head (920cca6).
Report is 152 commits behind head on master.

Files with missing lines	Patch %	Lines
src/platform/linux/input/inputtino_gamepad.cpp	2.66%	144 Missing and 2 partials ⚠️
src/platform/linux/input/inputtino.cpp	3.22%	60 Missing ⚠️
src/platform/linux/input/inputtino_keyboard.cpp	0.00%	33 Missing ⚠️
src/platform/linux/input/inputtino_pen.cpp	0.00%	29 Missing ⚠️
src/platform/linux/input/inputtino_mouse.cpp	0.00%	26 Missing ⚠️
src/input.cpp	0.00%	22 Missing ⚠️
src/platform/linux/input/inputtino_touch.cpp	0.00%	20 Missing ⚠️
src/platform/linux/input/inputtino_common.h	0.00%	17 Missing ⚠️
src/config.cpp	0.00%	5 Missing and 5 partials ⚠️
src/platform/windows/input.cpp	40.00%	7 Missing and 2 partials ⚠️
... and 1 more

Additional details and impacted files

@@            Coverage Diff            @@
##           master   #2606      +/-   ##
=========================================
- Coverage    9.88%   8.70%   -1.19%     
=========================================
  Files          88      94       +6     
  Lines       18070   17366     -704     
  Branches     8605    8251     -354     
=========================================
- Hits         1787    1512     -275     
+ Misses      13374   12994     -380     
+ Partials     2909    2860      -49     

Flag	Coverage Δ
Linux	6.49% <1.63%> (-1.94%)	⬇️
Windows	3.82% <15.55%> (+0.04%)	⬆️
macOS-12	?
macOS-13	9.72% <5.12%> (-0.01%)	⬇️
macOS-14	10.02% <5.12%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/input.h	0.00% <ø> (ø)
src/platform/common.h	34.56% <100.00%> (+0.81%)	⬆️
src/platform/macos/input.cpp	36.57% <14.28%> (-0.22%)	⬇️
src/platform/windows/input.cpp	1.06% <40.00%> (+0.65%)	⬆️
src/config.cpp	4.31% <0.00%> (-0.53%)	⬇️
src/platform/linux/input/inputtino_common.h	0.00% <0.00%> (ø)
src/platform/linux/input/inputtino_touch.cpp	0.00% <0.00%> (ø)
src/input.cpp	0.23% <0.00%> (-0.01%)	⬇️
src/platform/linux/input/inputtino_mouse.cpp	0.00% <0.00%> (ø)
src/platform/linux/input/inputtino_pen.cpp	0.00% <0.00%> (ø)
... and 3 more

... and 4 files with indirect coverage changes

[ReenigneArcher]

@ABeltramo any idea how to solve these errors on Fedora 40? https://github.com/LizardByte/Sunshine/actions/runs/9295282809/job/25581974710?pr=2606#step:10:3266

We're using gcc-14 on that distro, I'm guessing that's part of the problem.

[ABeltramo]

I've quickly skimmed thru the logs:

error: 'clamp' is not a member of 'std'

sounds like it's missing #include <algorithm> I've just pushed a potential fix, can you upgrade the submodule to bc6afda ?

[Hazer]

sounds like it's missing #include <algorithm> I've just pushed a potential fix, can you upgrade the submodule to bc6afda ?

@ABeltramo Seems to work! But now it needs the same fix in uinput/pentablet.cpp too.

[azuwis]

I was testing this PR in my setup, and found some problems of the virtual motion sensor.

Sunshine ad93e89
Moonlight 5.0.1
DualSense Wireless Controller 054C:0CE6

After setup uhid, on the server side (sunshine), I have:

/dev/input/event30:	Sunshine DualSense (virtual) pad
/dev/input/event31:	Sunshine DualSense (virtual) pad Motion Sensors
/dev/input/event256:	Sunshine DualSense (virtual) pad Touchpad

on the client side (moonlight), I have:

/dev/input/event15:	DualSense Wireless Controller
/dev/input/event16:	DualSense Wireless Controller Motion Sensors
/dev/input/event17:	DualSense Wireless Controller Touchpad

But in the game tested, the motion sensor is unreasonablely fast to use.

After compare the output of evtest /dev/input/event*, it seems there are some problems with ABS_RX ABS_RY ABS_RZ.

So, I run evtest on the server and client at the some time to capture ABS_RX:

On the server side:

# Sunshine DualSense (virtual) pad Motion Sensors
$ evtest /dev/input/event31 | awk -F '( |,)' '/ABS_RX/ {print $3, $14}' > sunshine.txt

On the client side:

# DualSense Wireless Controller Motion Sensors
$ evtest /dev/input/event16 | awk -F '( |,)' '/ABS_RX/ {print $3, $14}' > local.txt

Edit sunshine.txt and local.txt to align the timestamp, and use gnuplot the render them:

set term png
set timefmt "%s"
set xdata time
set grid

set output "sunshine.png"
plot "sunshine.txt" using 1:2 with lines

set output "local.png"
plot "local.txt" using 1:2 with lines

The 2 plots represent the same movement of the dualsense controller, the server side seems wrong to me.

[ABeltramo]

This is probably the most in-depth bug report I've ever received, thanks @azuwis !
I know there's something wrong with the actual values of gyro and accelerometer as I was mentioning here but thanks for confirming it, I have to check where the error is between Moonlight reported values and the virtual pad implementation

[ABeltramo]

sounds like it's missing #include <algorithm> I've just pushed a potential fix, can you upgrade the submodule to bc6afda ?

@ABeltramo Seems to work! But now it needs the same fix in uinput/pentablet.cpp too.

My bad, I've added that too!

[Hazer]

I have to check where the error is between Moonlight reported values and the virtual pad implementation

@azuwis thanks for your report!

@ABeltramo I'm splitting in smaller parts the upper input layer too, so it will be easier to understand, here is the PR #2594 if you want to use it

And the (hopefully) last changes missing to build on all platforms: games-on-whales/inputtino#4

[ABeltramo]

@Hazer I've pushed a fix for Gyro and Acceleration, it should now be very close to the original:

gyro_capture.mp4

You have to update inputtino to the latest and change the following (I'm afraid I can't push to this PR, right?):

Sunshine/src/platform/linux/input/inputtino_gamepad.cpp

Line 210 in 6cf7f3d

std::get<inputtino::PS5Joypad>(*gamepad->joypad).set_motion(inputtino::PS5Joypad::GYROSCOPE, motion.x, motion.y, motion.z);

to:

std::get<inputtino::PS5Joypad>(*gamepad->joypad).set_motion(inputtino::PS5Joypad::GYROSCOPE, deg2rad(motion.x), deg2rad(motion.y), deg2rad(motion.z));

(add the deg2rad conversion).

It would be great if @azuwis can give it another go after this change is in!

[Hazer]

@ABeltramo sent you an invite for my fork, if you want to change directly.
Another fix for fedora: games-on-whales/inputtino#5

Updated with latest ps5 changes, can you give it another go @azuwis?

[ABeltramo]

@ABeltramo sent you an invite for my fork, if you want to change directly. Another fix for fedora: games-on-whales/inputtino#5

Updated with latest ps5 changes, can you give it another go @azuwis?

Damn you Fedora! I'm going to install it in a VM to quickly test this compilation issues..
I don't see the deg2rad change that I've mentioned before in the commit that you pushed, could it be because I'm on mobile?
Thanks for the invite, I'll double-check everything again tomorrow!

[Hazer]

@ABeltramo a few developers VS a hat, who would win?

So far the hat is beating both of us 🤣

I don't know, maybe it's because it's on mobile. I've updated now to games-on-whales/inputtino@99c9840, so I guess everything should be there.

PS: Fedora is building with latest release. So far.

[azuwis]

Sunshine: azuwis@4855e09, not the version in this PR, have to apply the patch @ABeltramo mentioned:

You have to update inputtino to the latest and change the following (I'm afraid I can't push to this PR, right?):

Sunshine/src/platform/linux/input/inputtino_gamepad.cpp

Line 210 in 6cf7f3d

std::get<inputtino::PS5Joypad>(*gamepad->joypad).set_motion(inputtino::PS5Joypad::GYROSCOPE, motion.x, motion.y, motion.z);

to:

std::get<inputtino::PS5Joypad>(*gamepad->joypad).set_motion(inputtino::PS5Joypad::GYROSCOPE, deg2rad(motion.x), deg2rad(motion.y), deg2rad(motion.z));

Gnuplot of ABS_RX:

set term png
set timefmt "%s"
set xdata time
set grid

set output "all.png"
plot "sunshine.txt" using 1:2 with lines linecolor rgb "red" title "Sunshine", \
     "local.txt" using 1:2 with lines linecolor rgb "blue" title "Local"

Great work! 👍

[ABeltramo]

Thanks @azuwis for confirming! @Hazer I've pushed the missing bit to your branch, I think there are only a few things left for this PR:

• Fix the Fedora build
• Fix the PS5 trackpad integration (it seems to not move properly)
• Decide if we want to add the automatic failover (mentioned here) when uhid/uinput aren't available and implement it

Anything else missing?

[ReenigneArcher]

For your third checklist, I don't know that it's worth it. I don't think we'll be keeping both code paths for long.

[ABeltramo]

For your third checklist, I don't know that it's worth it. I don't think we'll be keeping both code paths for long.

That sounds good to me! Shall we clean it up then? I mean, it's always going to be present in git history if anyone ever is going to need it..

[ReenigneArcher]

I'm fine with that, or handling it in a separate PR right after the next stable release.

I'm just a little worried about a big change like this with so few testers.

[ABeltramo]

That's a valid concern; whilst the DualSense stuff is definitely experimental and lightly tested users can always force an xbox joypad from the config page and that's been in use for more than a year in Wolf. I'd assume that any big issue would have been reported by now..

[VoxelTek]

Hey! Just did a test of the latest version. I can confirm, the rumble issue is indeed fixed, however, the "low battery" message still appears.

[ABeltramo]

Hey! Just did a test of the latest version. I can confirm, the rumble issue is indeed fixed, however, the "low battery" message still appears.

Thanks for testing it out! How far off is the battery from what is reported in the client? I'm not sure how reliable is Moonlight (SDL I guess?) in getting that value.. It would be nice to see if it's the Moonlight control packet reporting that or something wrong in the virtual device in Sunshine.

[ReenigneArcher]

@ABeltramo it looks like the percentage we give is divided by 10. Is that the issue? https://github.com/games-on-whales/inputtino/blob/2739465690c7bbd1a27cb4e285ff08f486a208e3/src/uhid/joypad_ps5.cpp#L383

[VoxelTek]

Thanks for testing it out! How far off is the battery from what is reported in the client? I'm not sure how reliable is Moonlight (SDL I guess?) in getting that value.. It would be nice to see if it's the Moonlight control packet reporting that or something wrong in the virtual device in Sunshine.
@ABeltramo

I'll have to do more investigation, I've been primarily testing using a modded Switch running a Moonlight client. I have tried with other proper controllers, however, and I do believe that those also presented the "low battery" message. I'll double-check in a bit.

When I received those errors, the Switch was most certainly on somewhere around 80-90%, or more.

[cgutman]

Maybe @cgutman can shed some light on which HW is currently supported as a drawing tablet?

SDL2 doesn't have proper pen support, so you'd need to test on Android or iOS using a stylus. Those should support all of the fancy pen features (pressure, tilt, hover, buttons) assuming your hardware has them.