Lift autoplay restriction on a muted AudioContext

[hoch]

Branched from #2411 (comment):

On a muted context, the autoplay restriction should not be applied.

Chrome v111 which is the only browser implementing setSinkId(sinkId: { type: "none" }) so far still enforces the autoplay policy on a muted AudioContext.

[chrisguttandin]

This is the corresponding bug for Chromium: https://bugs.chromium.org/p/chromium/issues/detail?id=1467591.

Should the spec explicitly say that the autoplay policy should not affect a muted AudioContext?

[padenot]

Probably worth it to make explicit yes.

[hoch]

@chrisguttandin Could you take a look at the possible spec change for this? The impl change might be trivial, but we need a written spec to initiate the work.

[mjwilson-google]

There are two benefits I see to the autoplay restriction:

• It stops websites from playing annoying sounds.
• It stops websites from using WebAudio for user fingerprinting on sites they haven't interacted with.

Muted AudioContexts don't play annoying sounds, but they could still potentially be used for fingerprinting by running audio graphs and looking at the results. Note that microphone input should be gated behind a permission, so direct audio capture shouldn't be possible without user interaction.

Is this something we should be concerned about? My guess is that the autoplay restriction wasn't supposed to be a privacy or security feature so we should handle anti-fingerprinting another way, but I wanted to bring this up just in case.

[danrossi]

I've been putting together some tests. There seems to be an oversight here. For web admin consoles behind logins that need autoplay players muted for live streaming with analyser working to show an audio meter. It requires some kind of modal requesting gesture each load. Which will enable context and unmute the video element to get analyser data. Even with gain attached and the gain muted, the video requires to be unmuted to get data, which breaks autoplay. I'm unsure how to get the analyser working with muted video elements.

The gain node is supposed to take over volume control and the video element stay unmuted with volume set to 1. So to override volume control with a gain node because a muted video element gets no data, a context with no sink just for the analyser isn't helpful unless multiple contexts can be setup one for volume control and another for analyser but then I still run into the problem of requiring a muted video element for autoplay and analyser gets no data,

I found a hack by temporarily requesting microphone device permissions like with webrtc features completely works around autoplay mute and context restrictions and with gain muted also. unmuted video element still fails to play in safari, it has to go back to muted. The autoplay should somehow identify the gain is muted rather than the video element to keep receiving data. For both video playback and audio context.

There should have been a permission feature so there is no requirement to show a modal dialog to keep requesting user gesture and permission saved in the site settings. Like with camera and microphone. I may have to implement such a hack showing modal dialog in my feature to turn on context and unmute the video element and may be annoying for the end user as it's impossible to save the setting. And provide the mediadevice hack as another option.

1. mediadevice work around https://electroteque.org/dev/audio/video-mediadevice.html
2. click to enable context and unmute video element per page load https://electroteque.org/dev/audio/video.html
3. same as above but unmute and play to work around browser restriction with unmuted video element - https://electroteque.org/dev/audio/video-unmuteplay.html
4. mediadevice option has no restriction - https://electroteque.org/dev/audio/webcam.html

[chrisguttandin]

@hoch I think the only place that would need to be edited would be the definition of allowed to start.

https://webaudio.github.io/web-audio-api/#allowed-to-start

This is the current version:

An AudioContext is said to be allowed to start if the user agent allows the context state to transition from "suspended" to "running". A user agent may disallow this initial transition, and to allow it only when the AudioContext's relevant global object has sticky activation.

Should I go ahead and create a PR to change this paragraph to explicitly disallow the user agent to block the transition if the sinkId is set to { type: 'none' }?

@mjwilson-google I might be wrong but I think it should not provide any additional fingerprinting capabilities compared to an OfflineAudioContext.

@danrossi In theory there is an option to get a MediaStream from a muted audio element using captureStream(). It should completely ignore the volume of the audio element. But as far as I can tell current implementations are very incosistent.

It's the second to last paragraph in here: https://w3c.github.io/mediacapture-fromelement/#dom-htmlmediaelement-capturestream.

... Muting the audio on a media element does not cause the capture to produce silence, ... Similarly, the audio level or volume of the media element does not affect the volume of captured audio.

[danrossi]

@chrisguttandin captureStream also requires an unmuted video element to analyse audio. Like with canvas capture it's software encoding so incurs unnecessary cpu. Maybe there should be an option to just capture audio not video. I've implemented features to show the requirement of a permission system like mediadevices so acceptance of audio contexts is stored in site settings. Why I think it's an oversight. You can see here it requires to display a modal dialog to capture gestures per page load as nothing is saved or remembered.

I did find however clicking from another page in chrome, will work around unmuted video and context autoplay restrictions. So the modal will only show up when visiting the page directly and suspended context states. I developed a hack requesting microphone temporarily with a dialog to notify what is going on. Works around all autoplay restructions. And the permission is saved to keep requesting it only if the context is suspended.

I guess with a context with an empty sink can be used for the analyser, and due to the problem requiring unmuted video elements to get audio data. A second context is needed to mute audio through a gain node which I have a feature for so muting keeps audio analysers going by keeping the element unmuted with volume set to 1. Autoplay will still fail due to muted restrictions. Webkit is more touchy with this and will immediately stop playback if the element is unmuted.

Should I try and make a report about captureStream with chrome or provide a way to get audio data with muted video elements ?

https://electroteque.org/plugins/videojs/audio-visualiser/demos/autoplay/
https://electroteque.org/plugins/videojs/audio-visualiser/demos/autoplay-mediadevice/

Chrome remembers clicks from another page so no autoplay restriction. Firefox and Safari does not.

https://electroteque.org/plugins/videojs/audio-visualiser/demos/