Epub decorators are not accessible

[sdsantos]

Bug Report

What happened?

Decorations on an Epub can't be found through the Android Talkback accessibility tool. This happens even if you directly tap them.

Expected behavior

When talkback is on, users should be able to tap a decoration on an epub to focus on it, if there's content or an action available inside the decorator. The decorations should also be focused on before or after the content they refer to when navigating through an epub with Talkback.

How to reproduce?

This is reproducible with the sample app provided:

• Highlight a section of an epub
• Turn talkback on
• Tap on the highlighted section or icon to focus on it (nothing is selected)

Environment

• Readium version: 2.3.0

Development environment

• OS: MacBook Air M1 202
• IDE: Android Studio Electric Eel | 2022.1.1 Patch 2

Testing device

• Android version: 8.1.0
• Model: Google Nexus X
• Is it an emulator? No

Additional context

I'm able to tap to select decorations if I revert the pointer-events: none set in the decorator container. For example, running this javascript code after adding decorations:

Array.prototype.forEach.call(document.querySelectorAll("div"), function(elm) {
    if (elm.style.pointerEvents == "none") {
        elm.style.pointerEvents = "auto";
    }
});

Nevertheless, all decorations are lumped together separately from the content, likely because they are positioned with position: absolute, but I'm not sure on that.

[mickael-menu]

I'm able to tap to select decorations if I revert the pointer-events: none set in the decorator container.

The reason there is a pointer-events: none on decorations is because we don't want to prevent the user from clicking on a link or other interactive element under the highlight. Right now, activating a decoration is handled as a global click handler fallback on the root element, if nothing else intercepted the click/tap.

Maybe this should be disabled when entering screen reader mode. Having the highlights work is probably more important than the links behind a highlight, in this particular case.

Nevertheless, all decorations are lumped together separately from the content, likely because they are positioned with position: absolute, but I'm not sure on that.

We opted for absolute positioning instead of inline decorations to prevent disrupting the publication’s DOM elements. If we inserted the decorations in the document directly, position: absolute might be broken as well.

But maybe a compromise could be made for screen readers, what about using both:

• An absolute-positioned overlay decoration for rendering the clickable highlight.
• A discrete span element at the start (and/or the end) of the highlighted portion to notify the screen reader.
  • A single span should not be too disruptive as we don't need to break down DOM elements.
  • Will this really work? How can a user use Talkback to interact with the highlight in this situation? I'm not very familiar with screen readers.

[sdsantos]

The reason there is a pointer-events: none on decorations is because we don't want to prevent the user from clicking on a link or other interactive element under the highlight. Right now, activating a decoration is handled as a global click handler fallback on the root element, if nothing else intercepted the click/tap.

Maybe this should be disabled when entering screen reader mode. Having the highlights work is probably more important than the links behind a highlight, in this particular case.

FYI: I've added pointer-events: auto to the decoration html elements that had data-activable="1" (they are icons on the side of the text) and it was enough to make them tappable, even by leaving their containers as pointer-events: none.

We opted for absolute positioning instead of inline decorations to prevent disrupting the publication’s DOM elements. If we inserted the decorations in the document directly, position: absolute might be broken as well.

But maybe a compromise could be made for screen readers, what about using both:

• An absolute-positioned overlay decoration for rendering the clickable highlight.
• A discrete span element at the start (and/or the end) of the highlighted portion to notify the screen reader.
  • A single span should not be too disruptive as we don't need to break down DOM elements.
  • Will this really work? How can a user use Talkback to interact with the highlight in this situation? I'm not very familiar with screen readers.

There are 2 ways I know of interacting with elements with Talkover.

1. You can hover them by dragging your fingers across the screen, or tapping them directly. That will trigger Talkover to read them. (I imagine this is useful when you can see or know that something's there, but you can't tell what it is)
2. You can swipe the screen left or right to change the element you are focusing on, basically traversing the entire screen to read it from start to end.

So ideally we would want to enable those two behaviors. The decorator should be focused before or after the text it refers to if users are swiping through the screen. But if decorators have a visible element they should be able to tap it as well. Bonus points if those 2 behaviors are enabled by the same element, because that would make it more consistent. I could tap a decorator to learn what it's about, and then swipe to the previous/next expected section of text, and not the previous/next decorator.

Now I know that making it all consistent is a tall order. It would probably require some sort of CSS reset that would make the position: absolute still work, even if right before or after the block of content it refers to.

[mickael-menu]

Before making any decision, I'd like to get input from accessibility UI experts and users of screen readers. Maybe the highlights should be accessible through a dedicated screen instead. And how can screen reader users highlight a portion of text in the first place?

@gautierchomel Maybe you could help us with these discussions?

[ctechdev]

Has there been any feedback on this issue?

[mickael-menu]

No feedback received yet.

[HadrienGardeur]

This feature has been recently implemented in Thorium, maybe @danielweck could chime in on how this was handled.

[danielweck]

In Thorium, inline text highlight and margin indicators are not accessible via the keyboard, and consequently to screen reader users. They are minimally interactive affordances, via pointing devices (mouse, trackpad, etc.) for hover and click (and click only via touch).
The primary development focus has been to provide a structured navigation experience via the paginated list of annotations / bookmarks / search results.
With time and effort, I am sure a keyboard-driven UX could be implemented for inline and margin "decorations", but that's not on the roadmap for now.

[HadrienGardeur]

@danielweck what about the creation of highlights/notes?

[danielweck]

Once a native DOM Selection / Range is created by the user inside a publication's XHTML document, the add/edit/remove UX is realised entirely inside the application's GUI, there are no affordances in the document's webview. Essentially, when the user hovers+clicks on an in-text highlight or a margin decoration, the event is dispatched to the application's shell which decides how to implement subsequent interactions. There is no visual co-location of the click site (highlight x/y coordinates) and of the editor panel. This works well in Thorium as there is a dedicated toolbar icon button which is used as the anchoring site for the GUI popout dialog.