🐛 Bug: Gestures ignored on right-side landscape

[mark-omarov]

Description:

Gestures ignored when the device is stored on its right-side w/ landscape mode on, as soon as the device is rotated to its left-side gestures work correctly again.

Noticed on devices w/ versions: Android 11, Android 12 (Android 12 also has a problem with minicap in the landscape mode).
Minitouch consumer/environment: STF

Steps To Reproduce

• Connect an Android 11/12 (sdk 30/31?) to the STF
• Start using the device in portrait mode
  • If autorotate is off - switch on
  • Open Settings (or any application supports landscape mode)
• Rotate the actual device on its right-side (the window should be in landscape mode)
  • Try to manipulate via the browser - commands not reflected
• Rotate the actual device on its left-side (the window should be in landscape mode)
  • Try to manipulate via the browser - now it works

[stoefln]

We are experiencing the same problem. This could be related: #2

[mark-omarov]

@stoefln it might be the case indeed.

[mark-omarov]

Hello @koral-- , could I humbly ask you to check on this issue? I went through the minitouch.c file trying to figure out what may cause the problem, also debugged with STF, it seems values are sent correctly (and are being counted from the screen's pivot point) to the minitouch, I wanted to dive deeper to the libevdev library but it's completely out of my experience.
I have zero experience working with android, and very humble experience with C, therefore, if you could help out with this one, we'd really appreciate.

[mark-omarov]

It seems like the mapping is not performing correctly.

When a device is in portrait mode being connected through STF - it works fine, even if switched to the landscape mode (I suppose android rotates to its left-side, because on the right-side it doesn't work at all).

However, when a device being connected while in the landscape mode, two scenarios are possible:

• If the device is on its right-side in the landscape mode - manipulations are ignored, it's impossible to perform any kind of action with the screen
• If the device is on its left-side in the landscape mode - it's technically possible to manipulate the device, however, the mapping seems to be odd, the contact point is incorrect, if you open settings you can see it easily while trying to select an item - another item is selected

While it's possible to workaround this issue by connecting the device in the portrait mode and once everything is done rotate on its left-side (or keep in portrait, whatever best suits), it's not resolving the underlaying
issue, therefore, this is still a problem.
This problem affects both Android 11, Android 12, and possible lower versions as well.

@stoefln you might want to check it out as well. 👆

[stoefln]

@mark-omarov thanks a lot for your insights!
It seems this issue occurs only on Android 11 and up because of the necessity for the STFService + Agent, right?
Landscape works really well on older phones for us.
I wanted to dig into the STFService code base and do some debugging, but haven't found the time. We built a fallback solution using Appiums UIAutomator server. It works for portrait and landscape, but it's much slower...

[mark-omarov]

Hi @stoefln!
That's right, it seems to happen for those versions. Unfortunately, I don't have an Android 10 at hands, so can't say for sure about that, but tested on both Android 11/12 and it's positive.

I've performed a few further investigations of minitouch with STF and can say that for those versions Android InputManager is being used, so all the commands are forwarded to the STFService. And it seems that the mapping itself is actually handled there, since I could not find any related logic for mapping in the minitouch itself.

I think it's something that should be handled on the STFService side, unfortunately, I'm not an Android Developer, and can't dig into it deeper, I'll see if I can get an actual Android Dev to take a look at the problem.

Thanks for sharing your fallback solution!

[koral--]

Hi all.

I am answering since I was summoned. Here are my thoughts:

• the hypothesis about incorrect coordinates calculation sounds reasonable
• I recall an issue Portrait/Landscape inconsitence on large tablets openstf/stf#711 about incorrect orientation indicator displayed on web UI - that may be related or it is analogous. I mean formulae themselves are correct but they are receiving an incorrect current rotation data

At the moment I don't have capacity to investigate the issue further but I will be happy to review PR.

[stoefln]

@mark-omarov did you find any solution for the landscape problem?

[mark-omarov]

Hey @stoefln,

I don't have the capacity to work on this issue furthermore. But, yes I kinda dealt with it using a workaround.
Basically, it never works if the device is placed on the right side, so the workaround would be these two steps:

1. Connect the device via the USB while holding it in portrait mode (even tablets)
2. Place the device either on its back or on its left side.

I've had an android dev take a look, unfortunately, he didn't find a solution to this problem, his attempt was to remove calculations from the STFService, so the inputs are 1:1 mapped to the device pivot point. I've tried to play around with this updating the frontend part, but obviously, since devices earlier than android 10 versions are using a different input method - this is a breaking change for them.
There's a theoretical solution to it as well - keep only using input through STFService, but I'm not aware of all of the devices will handle it properly. As far as my tests went - all good, but again, I'm not an android developer to figure this out all the way through.

[stoefln]

Thanks so much @mark-omarov for the update. I kind of gave up on it too. Will try to switch to ws-scrcpy instead.
If I find any solution for minitouch, I will let you know.

[pcrepieux]

Sorry to come late to the party (didn't get any notification or possibly missed it).
I've just fixed it. Will push it shortly. That actually is an issue with STFService.apk

[mark-omarov]

Hey @pcrepieux , I'm curious if I was nearly right with the cause of the problem? Thanks for taking time to get it fixed, much obliged! I'll test it as soon as STFService is updated.

[stoefln]

Exciting news 🥳
Looking forward, @pcrepieux let me know if I can help!

[koral--]

Fixed in DeviceFarmer/STFService.apk#68

[stoefln]

For me it's still not working. Actually landscape is not working at all above Android 10, no matter which side.

[mark-omarov]

Hey @stoefln , if I remember correctly - you don't use STF, rather implement your own solution, is that right?
If that's the case, taking a peek at the usage in STF may help. I've never tried to standalone, after this fix the problem with right-side landscape is gone for me.

[stoefln]

Thanks a lot @mark-omarov for letting me know. Yes, it's my own solution. Most probably we will switch to scrcpy if everything is going well. Otherwise I will take a closer look how OpenSTF is communicating with StfService...

[pcrepieux]

In case it might help, for touchscreen interactions, STF still communicate with minitouch as it used to do. But, starting with Android 10, minitouch will simply forward the commands (which are text based) to the STF agent by mean of a dedicated local socket ("minitouchagent").
I don't know how minitouch is involved in your custom solution but you maybe only need to install STFService.apk and start the agent beforehand.
That mostly guesses. Maybe you already tried this.

[stoefln]

@pcrepieux thanks a lot for trying to help! I am making use of the agent, and it works great for portrait (actually default orientation), so I assume it's not related to the way I set up the service and the agent. But it doesn't work for landscape (90 and 270 degrees) on Android 10 devices.

Below you will find the class I wrote for launching service, agent and sending commands. I think I am missing some piece of information, because like you said OpenStf is able to handle landscape and I am not.

Is maybe my coordinate mapping wrong? I tried to find the relevant piece in the openstf codebase many months back, but without success....
I would be very happy if somebody could check out the sendClick() and orientationTransform() methods, maybe it's a very obvious problem:
https://gist.github.com/stoefln/ede709c0ec07d158d8533229f22705ae

[mark-omarov]

Hey @stoefln , I can't compare your code to the STF version, don't have access to a laptop now, but here's the file that's responsible for calculations in STF, maybe it will help you out.

https://github.com/DeviceFarmer/stf/blob/master/res/app/components/stf/screen/scaling/scaling-service.js

[stoefln]

@mark-omarov thanks a lot. I just tried to figure out what OpenStf is doing differently, but it's complex. So it will take me more time I guess.