PCM delay synchronization

[borine]

PCM delay reporting for audio/video synchronization came up in a recent discussion, and it got me thinking about how BlueALSA could offer better support for sink devices that do not report their delay values.

At present BlueALSA offers a single default fixed adjustment (of 0 frames) that can be overridden for a specific PCM by using its delay parameter. This for me is sufficient, except when using the default 00:00:00:00:00:00 PCM.

I would like to propose that BlueALSA could offer a persistent delay adjustment for a PCM that is remembered, so once set that same value would be used for that PCM no matter if it is opened by using its actual address or the default address.

There are a few specification details that need to be agreed, such as:

• should the value be shared only by ALSA clients (and therefore stored within the ALSA config); or should it be shared with all D-Bus clients (and therefore stored in the BlueALSA persistent state and made available via the D-Bus API) ?
• if part of the D-Bus API, should the adjustment be applied to the Delay property by the bluealsa daemon, or should it be a distinct property that individual clients can choose to apply if they wish?
• should it be possible for the value to be overridden when opening a PCM, and if so should the override value become the new stored value or should it be used for that one stream only?
• what units should the value be expressed in? (frames, milliseconds, etc). The current PCM delay parameter is the number of frames to be added or subtracted.

I have a proof-of-concept implementation which creates a new org.bluealsa.PCM1 property called "DelaySync" which is an integer defaulting to 0 and is writable by clients. This value is distinct from "Delay" and so a D-Bus client can choose whether or not to apply it. The ALSA PCM plugin does apply this adjustment. The ALSA CTL plugin has a new extended control called "Sync" which can be used to modify the adjustment for a PCM. So a user can use alsamixer to adjust the audio sync, for example while watching a video, and then that adjusted value will be remembered for that PCM, just like the last set volume is remembered. In this implementation the PCM delay parameter also updates the stored DelaySync property. The unit I have chosen for the property is 1/10 milliseconds (same as the Delay property), but the CTL plugin and bluealsa-cli use milliseconds. The step value of the ALSA control is 25ms, with a range of -3250ms to +3250ms.

Before raising a PR for review, I would be interested in the thoughts of any users on:

• would such a feature be useful in practice?
• would the feature cause problems for any use cases I have overlooked?
• have you any related ideas that should be considered before too much detailed design or implementation work is done on this feature ?

[arkq]

I like the idea! So here are my thoughts:

should the value be shared only by ALSA clients or should it be shared with all D-Bus clients

shared with all clients, IMHO it should be indistinguishable from delay coming from device from the PCM point of view. What I mean is that maybe this value should be set by the user, but user should read the "Delay" property anyway (this user adjusted delay value should be added to the "Delay").

should it be possible for the value to be overridden when opening a PCM

maybe we should keep the current ALSA delay as it is right now, ALSA delay will be added to all other delays, but we can deprecate the ALSA delay anyway (it will be dropped someday)

what units should the value be expressed in

same as the "Delay"

called "DelaySync"

I'm not sure about the name, not yet, but sync suggests that we need to sync something... but in fact there is nothing to sync if there is no video...

The ALSA CTL plugin has a new extended control called "Sync"

OK with new control, not sure about the name (see the above answer)

would the feature cause problems for any use cases I have overlooked?

There is one potential issue (but maybe it can be somehow addressed). The delay depends on the used codec. With my Jabra headset, I have to use one delay value for AAC and different one for SBC. Anyway, I'm not switching codecs for other things than testing, so one value is OK for me, but it might be annoying that the value set for one codec is lost when switching back to it from a different codec (which needs different delay).

have you any related ideas

I think that this manual delay should be used only in case where headset lacks delay reporting (this feature availability is reported during A2DP connection).

[borine]

this value should be set by the user, but user should read the "Delay" property anyway (this user adjusted delay value should be added to the "Delay"

That was my first thought, but I could not devise a suitable interface. I guess the adjustment would be a method, not a property. I was uncomfortable with not knowing how much of the delay was "real" (ie reported by the remote device and measured internally by bluealsa) and how much of it was arbitrary adjustment.

maybe we should keep the current ALSA delay as it is right now, ALSA delay will be added to all other delays, but we can deprecate the ALSA delay anyway (it will be dropped someday)

OK. In implementation terms that is a minor detail and can be decided once the bigger picture has become more focused.

what units should the value be expressed in> same as the "Delay"

Agreed, as far as the D-Bus API is concerned. For user interfaces I think that is too fine, so for the bluealsa-cli "delay-sink" command I used milliseconds (but allow for example 500.3 for those users who really believe they need sub-millisecond precision).

For the ALSA CTL plugin, it would be very tedious using the alsamixer interface to scroll up at 1/10 millisecond intervals, so I chose a step size of 25ms (this choice was influenced by kodi which uses a 25ms interval for its own audio sync control); and its just easier to display "ms" as the unit rather than "1/10 ms"

I'm not sure about the name, not yet, but sync suggests that we need to sync something... but in fact there is nothing to sync if there is no video...

I originally used the name "DelayOffset", and also considered "DelayAdjustment" before going with "DelaySync". My thinking is that this property is only ever needed when an application needs to sync the audio stream with some other stream, typically a video stream. For simple audio applications the Delay property is ignored anyway.

The delay depends on the used codec. With my Jabra headset, I have to use one delay value for AAC and different one for SBC.

OK. I use a raspberry pi sink implemented using bluealsa-aplay. This reports a delay of 0 for all codecs. On my bluealsa source device, I set "DelaySync" to 250ms and this gives me very good AV synchronization using SBC (48000Hz) aptX (48000 Hz) and AAC (96000Hz). Perhaps you need different delay arguments only because of the different sample rates when the delay argument uses frames as its unit?

I presume it must be possible to store a different delay adjustment value for each codec used on a PCM, although I do not have a clear picture of how the user interface, or the API, would look. If this feature necessary then more creative thought is required.

I think that this manual delay should be used only in case where headset lacks delay reporting (this feature availability is reported during A2DP connection).

AFAIAA, a Bluez sink (BlueALSA, PulseAudio or Pipewire) claims that it does provide A2DP delay reporting feature, but in fact the reported value is always zero. So I'm not so sure about this constraint. My feeling is that the feature should always be available, and the user can decide whether to apply it or not for a particular device.

[arkq]

Perhaps you need different delay arguments only because of the different sample rates when the delay argument uses frames as its unit

No, both are using 44100 Hz sampling rate. The reason for bigger delay in case of AAC is entirely on the headset side. I guess that somehow the internal buffer for AAC is bigger. But maybe other headsets behave differently. However there are headsets which support "normal" codecs, and low latency ones. I guess, that when the low latency is selected, buffers are also smaller. However, such headsets for sure report delay via AVDTP.

AFAIAA, a Bluez sink (BlueALSA, PulseAudio or Pipewire) claims that it does provide A2DP delay reporting feature, but in fact the reported value is always zero.

I'm not sure what you mean by BlueZ sink. If you mean BT sink implemented by BlueALSA, PulseAudio or any other app on top of BlueZ, then you are out of luck. BlueZ does not support delay reporting for sinks. The app can claim that it will report delay, and this will be exported in A2DP profile, but later on updates will not be sent.

[borine]

The reason for bigger delay in case of AAC is entirely on the headset side

In that case, a different delay adjustment for each codec is definitely desirable. That would seem to be another reason why this feature should be a D-Bus method rather than a property. I'll have a think on how an interface for this might look.

BlueZ does not support delay reporting for sinks

That is exactly what I mean. A sink built using Bluez does not send any delay updates even though it claims to support Delay reporting (A2DP 1.3.1+). So my "headset" (built with Bluez) claims to support delay reporting. If my source (running kodi with bluealsa) accepts this claim and therefore disables delay adjustment the audio will lag behind the video. So I think my source must always enable the delay adjustment feature whether the sink claims to provide delay reporting or not.

[borine]

Here's some thoughts on revised naming, assuming we also switch to using D-Bus methods instead of a property and have the adjustment applied by the org.bluealsa service to the Delay property:

org.bluealsa.PCM1 methods

void SetDelayAdjustment(string codec, int16 adjustment)

       Set an arbitrary adjustment (+/-) to the reported Delay
       in 1/10 of millisecond for a specific codec. This
       adjustment is applied to the Delay property when that
       codec is selected, and can be used to compensate for
       devices that do not report accurate Delay values.

array{string, int16} GetDelayAdjustments()

       Return the array of currently set delay adjustments.
       Each entry of the array gives the name of a codec and
       the adjustment that the PCM will apply to Delay
       property when that codec is selected.

bluealsa-cli command

bluealsa-cli delay-adjustment PCM-PATH [CODEC] [ADJUSTMENT]

    - If two optional args are given, apply the given adjustment to the given codec
    - If one optional arg is given, if its a valid codec name print the current adjustment for that codec,
        or if its a valid number set the adjustment for the currently selected codec (there is a potential 
        race here if some other client meanwhile changes the codec, but hopefully we can tolerate that)
    - If no optional args are given, print the current adjustment for all codecs.

ALSA CTL plugin

For this to work well with alsamixer, the label has to be short (4-5 characters), and at present "Sync" is still the best I have come up with. Any alternative suggestions are welcome !!

[borine]

I've uploaded my POC implementation to a branch delay-sync here:
https://github.com/borine/bluez-alsa/tree/delay-sync

Please anyone interested do clone and build this to play around with. Its not necessarily very robust yet, but it does enough to illustrate the concept. Manual pages have been edited to document the feature.

[EDIT]
That branch is now defunct. See next comment.

[borine]

I've gone ahead and created a draft PR #653. Please post any comments/suggestions to that PR

[borine]

This functionality was added to bluealsa some time ago. It is in the master branch and will therefore be available in the next release. Therefore I am closing this discussion.