Sound delay in raspberry pi midi keyboard project

[PJBrs]

Hi, I'm trying to build a raspberrypi project where I play may midi keyboard and listen to the sound via bluetooth. Main reason for this: my children keep destroying my headphones by tripping over the wires.

I've tried various combinations now of fluidsynth, timidity, bluealsa, and jackd, but I haven't found any way to reduce the delay between striking a key on my keyboard and hearing sound via bluealsa. I estimate the delay to be about 250ms, which unfortunately is way too long for playing the piano. I hope you can help me get rid of that delay!

First, I did notice quite some open issues on github that mention delays with bluealsa:
#15
#124
#156

However, I don't understand if, and how I can reduce it. I have the following questions:
Are there tunables in the latest bluealsa code that I should try? Have any solutions been found already? I'm currently on raspbian stretch with bluealsa 1.3.0. Incidentally, I couldn't figure out what the "defaults.bluealsa.delay" setting was about, but it didn't seem to affect the delay... Really hope that I can get some advice!

Kind regards,

PJBrs

[PJBrs]

Since I reported my issue here, I've managed to compile the latest git version of bluealsa to try again. I think the following messages may hold some clues, especially that bit about fluidsynth requesting a different amount and size of periods (don't know what they are, but I reckon that they determine buffer size, and I imagine that buffer size is related to delay...?):

FluidSynth version 1.1.6
Copyright (C) 2000-2012 Peter Hanappe and others.
Distributed under the LGPL license.
SoundFont(R) is a registered trademark of E-mu Systems, Inc.

../shared/ctl-client.c:107: Connecting to socket: /var/run/bluealsa/hci0
../shared/ctl-client.c:237: Getting transport for FC:58:FA:B1:F3:C5 type 0x41
bluealsa-pcm.c:534: Setting constraints
fluidsynth: warning: Requested a period size of 64, got 220 instead
fluidsynth: warning: Requested 8 periods, got 19 instead
bluealsa-pcm.c:282: Initializing HW
../shared/ctl-client.c:396: Requesting PCM open for FC:58:FA:B1:F3:C5
bluealsa-pcm.c:305: FIFO buffer size: 4096
bluealsa-pcm.c:311: Selected HW buffer: 20 periods x 1924 bytes <= 38484 bytes
bluealsa-pcm.c:326: Initializing SW
bluealsa-pcm.c:326: Initializing SW
bluealsa-pcm.c:326: Initializing SW
bluealsa-pcm.c:326: Initializing SW
bluealsa-pcm.c:342: Prepared
bluealsa-pcm.c:326: Initializing SW
bluealsa-pcm.c:342: Prepared
bluealsa-pcm.c:214: Starting

Any suggestions?

[PJBrs]

Also, if perhaps I'm under the wrong impression that bluetooth audio will be an option for my usecase at all, please let me know as well. At my fastest, I can now play an entire scale on my electric piano before hearing it on my bluetooth speaker. Also, I worry a bit about this comment:

	 /* In order to prevent audio tearing and minimize CPU utilization, we're
	 * going to setup buffer size constraints. These limits are derived from
	 * the transport sampling rate and the number of channels, so the buffer
	 * "time" size will be constant. The minimal period size and buffer size
	 * are respectively 10 ms and 200 ms. Upper limits are not constraint. */
	unsigned int min_p = pcm->transport->sampling * 10 / 1000 * pcm->transport->channels * 2;
	unsigned int min_b = pcm->transport->sampling * 200 / 1000 * pcm->transport->channels * 2;

[nekarkedoc]

Hi PJBrs,
I'm not one of the devs here, but I am quite interested in this project, and have spent quite some time with midi and PC/software synths. In my experience, bluetooth does introduce enough latency (time between controller midi on and audible sound produced) to make it unsuitable for live midi performance use. Latency is critically important for live performance because once it gets around 50ms or more, the delay between what the mind expects to hear at note press time and when the note is actually heard becomes great enough to make playing anything live virtually impossible as the ear/mind expect that instantaneous feedback of audible sound. On Windows or Linux, without bluetooth, I have been able to tweak an average of around 35-40ms latency, leaving very little headroom for addititional latency, and bluetooth, because of it's own buffering requirements, puts the latency very noticeably above 50ms.
Additional signal processing/effects each take their own toll on the overall latency experienced, like some of the reverb/chorus effects Qsynth provides, or external DSP software like Calf.
I have actually successfully used a Pi3 as a midi soundsource for an maudio controller, with qsynth and Jackd, but had to use wired phones or speaker due to the latency. I found I had to tweak the audio buffers down in Jack to get the tolerable 30-4ms latency, but it would do a decent job as long as you didn't throw too many simultaneous notes at it, like a full run with the sus pedal on; It would become garbled, which is a sign you don't have enough buffers. But if you add buffers, your latency is diminished, making the Pi ok for practice, but not a performance.

[arkq]

Also, if perhaps I'm under the wrong impression that bluetooth audio will be an option for my usecase at all, please let me know as well.

It might be impossible to reduce delay to 50ms (as Steven has mentioned, that such a delay is an upper acceptable limit) with bluealsa. In general it should be possible to go to something around 25-50 ms with Bluetooth technology, however it would require special software dedicated for low latency. Bluetooth specification for HFP says that audio delay should not exceed 50ms (if I remember correctly), because higher delay might be inconvenient for online gaming when two ppl are seating in the same room. This specification does not say anything about software which produces sound, it only states that the communication delay and encoding/decoding should be within 50ms. BUT this is for HFP/HSP. The problem with A2DP (if you require better audio quality) is that it encodes bigger chunks of data at once and decodes it. This process alone adds significant amount of delay. There is a codec for A2DP which should produce low delay - AptX Low Latency. Unfortunately, bluealsa does not support such a codec - there is no much online info how it works. Anyway, it might be possible to reduce the delay for standard SBC codec and for AptX. The trick is, that one could send data earlier, without waiting for full MTU. With AptX it is possible to transfer 4 PCM frames - AptX encoder encodes 4 frames (4 samples for right channel and 4 samples for left channel) at once. For SBC the number is around 200 frames. See this: https://github.com/Arkq/bluez-alsa/blob/master/src/io.c#L498 https://github.com/Arkq/bluez-alsa/blob/master/src/io.c#L1202

[PJBrs]

Okay, thanks @nekarkedoc and @arkq for your reactions!

As far as I now understand, it should be able to get lower latencies if I try HSP and HFP protocols instead of A2DP, so that's the first thing I'll try. The other thing that I understand is that the best option would be the A2DP profile with the aptx(ll) (ll for low latency) codec, which has (way) better latency than normal aptx and aptx(hd), but that current bluez-alsa doesn't support that codec. Is there any bluez command that can show me which A2DP codecs are supported? But then, for this option to be useful it should first be supported in bluez-alsa. Finally, I could start tinkering myself and reduce the number of frames sent each time to reduce latency with the SBC codec... That might not be too much beyond my capabilities ;-)

I did notice that the latest pulseaudio might support aptx, but not the ll version. Also, when I tried pulseaudio, that alone took about 25% cpu. If I hit many many notes I only get very rare garbling using bluez-alsa, so that should be usable on the Pi zero. With pulseaudio, I might try again later.

Thanks for now, I'll let you know about any progress!

[arkq]

For supported codecs, run: bluealsa -h. For master build wit all codecs enabled you should see something like this:

Usage:
  bluealsa [OPTION]...

Options:
  -h, --help		print this help and exit
[...]

Available BT profiles:
  - a2dp-source	Advanced Audio Source (LDAC, APT-X, AAC, SBC)
  - a2dp-sink	Advanced Audio Sink (AAC, SBC)
[...]