Audio quality > 16 kHz

[dreimer1986]

Checklist

• I have filled out the template to the best of my ability.
• This only contains 1 feature request (if you have multiple feature requests, open one feature request for each feature request).
• This issue is not a duplicate feature request of previous feature requests.

Is your feature request related to a problem? Please describe.

Not really a problem, just muffled audio because of the limit to 16000 Hz. I am very sure I read somewhere about a few reasons for using this limit. I just don't find it anymore.

Describe the solution you'd like

44100 Hz sounds like a nice solution, especially if the media player features are used more often.

Describe alternatives you've considered

Of course I just can use another device as media player, but I want less than more and Onju voice should make my Echo obsolete which is not much left todo for.

Additional context

[jherby2k]

The issue is that the external/experimental esphome_audio component needed to get esp-idf working with a media player (rather than just a speaker) has some duplex limitations. Specifically, input and output get the same sample rate. There is a resampler, but for whatever reason it doesn't seem to work with the input (mic). So that means the sample rate has to be defined as exactly what the mic can handle. Which is 16kHz.

You can try it yourself! Set the rate to 48000 (lines 147, 160) and the mic will stop working. Different values for input and output won't work. Add the resampler to the input (see lines 181, 171) and the mic will still not work. This is the only setting that currently works.

Hopefully they get the resampler working with input, then you'll be able to set 48000 or whatever.

IMO, the microwakeword version is not ready for use just yet.

[dreimer1986]

In my case I don't even use the microwakeword one as it does not allow me to use my own wakeword. The default one has the same audio "problem" even though there is nothing set up for 16 kHz inside the config. The wakeword one is not ready for use maybe, yes. When I try to use it I get funny errors, like:

adf_pipeline.i2s_audio: [source /data/packages/2ab3d759/esphome/onju-voice-microwakeword.yaml:153]

Unknown value 'audioin', did you mean 'audio_in', 'audio_out'?.

[tetele]

@jherby2k do you know if there's an issue we can track regarding this limitation on gnumpi/esphome_audio? I can't seem to find one at first glance.

[jherby2k]

There isn't - i added a comment to the existing thread you had there, but it was ignored. I've opened one now: gnumpi/esphome_audio#42

[tetele]

Thank you!

[dreimer1986]

Did someone already try it out with the new branch? I am not at home right now and can't test before tomorrow. I am not sure if my audioin renaming to audio_in is really correct, too. (As I said, I use the config without microwakeword right now as my personalized sarcastic catgirl OpenAI is called Luna ^^) I really wonder if I can create a wake word file for microwakeword too...

[jherby2k]

I won’t have time until tomorrow

[dreimer1986]

I tried my luck and if there is no fault on my side it does NOT work. I used this config:

substitutions:
  name: onju-voice-e27688
  friendly_name: Onju Voice Satellite e27688
packages:
  tetele.onju_voice_satellite:
    url: https://github.com/dreimer1986/onju-voice-satellite
    file: esphome/onju-voice-microwakeword_1.yaml
    ref: main
    refresh: 0d

esphome:
  name: ${name}
  name_add_mac_suffix: false
  friendly_name: ${friendly_name}
api:
  encryption:
    key: jkgjhkulh
wifi:
  ssid: xxx
  password: xxx
ota:
  platform: esphome

and this package https://github.com/dreimer1986/onju-voice-satellite/blob/main/esphome/onju-voice-microwakeword_1.yaml

and here the log:
logs_onju-voice-e27688_run.txt

P.S.
Github once more casues MADNESS in line 154 where by magic my "audio-in" written there suddently becomes "audio"+"In" after a bit. Even in here I am UNABLE to type this line as one word in without it being replaced by "audio". O_o I can build a few times and then Github magically changed it back. So take this one with a bit of salt.

[dreimer1986]

Somehow I am too stupid to make things work. According to the report over there it should work now, but something I do is wrong I guess?

[dwyschka]

i was testing with gnumpi. please test the following config:

https://github.com/dwyschka/esphome-configs/blob/main/onjou-voice.yml

Adjust it to your needs pls

[dreimer1986]

@dwyschka thx for the help. It was me setting input and output to 16 Bits... Now all is fine again. Music Assistant still is a mess and most of the time does not play as MP3 is not my main source and the only one it sems to like, but the normal media play feature does fine where I can select the right files easily. So... Seems to be way better :D

[dwyschka]

enforce lossy mp3 in music assistant on your device, it works better.

[dreimer1986]

Tried that, but nope. Even then the stream seems to be sort of flac alike: http://192.168.181.42:8098/flow/media_player.onju_voice_e27688_onju_voice_satellite_e27688/87bd58754b47491b800608f96ed48eef.flac?ts=1718660677 it works in browser though, so this is fine, but Onju stays silent. I think that my beta fetish is in the way again ^^ I try to use 2.1.0b6 here.

[dreimer1986]

Did anyone try to correct the non microwakeword version, too? I tried to (see my fork) but it seems like my copy together tinkering is not working at all.

[dwyschka]

Tried that, but nope. Even then the stream seems to be sort of flac alike: http://192.168.181.42:8098/flow/media_player.onju_voice_e27688_onju_voice_satellite_e27688/87bd58754b47491b800608f96ed48eef.flac?ts=1718660677 it works in browser though, so this is fine, but Onju stays silent. I think that my beta fetish is in the way again ^^ I try to use 2.1.0b6 here.

Which audio provider are you using? i tested with spotify and yt. but, i set the audio quality on the onju to "44100 mhz 24bit". maybe thats the fix for you?

[dreimer1986]

I tried changing the supported audio output in Music Assistant to 44100 @24bit, but still have a flac stream inside onju's logs. Do you see a mp3 there?

[denisjoshua]

Subscribe cause I'm interested to make also my 7 ghome mini around my home.
Noroc :-)

[tbrasser]

Subscribe cause I'm interested to make also my 7 ghome mini around my home.
Noroc :-)

Same

[tetele]

I've tried upping the sample rate of the i2s_audio components, but since the voice_assistant sample rate is hard-coded to 16kHz and the buffer was recently cut in half, I get these warnings

[17:39:40][W][micro_wake_word:103]: Not enough free bytes in ring buffer to store incoming audio data (free bytes=896, incoming bytes=1024). Resetting the ring buffer. Wake word detection accuracy will be reduced.
[17:39:40][W][micro_wake_word:103]: Not enough free bytes in ring buffer to store incoming audio data (free bytes=896, incoming bytes=1024). Resetting the ring buffer. Wake word detection accuracy will be reduced.
[17:39:41][W][micro_wake_word:103]: Not enough free bytes in ring buffer to store incoming audio data (free bytes=896, incoming bytes=1024). Resetting the ring buffer. Wake word detection accuracy will be reduced.
[17:39:41][W][micro_wake_word:103]: Not enough free bytes in ring buffer to store incoming audio data (free bytes=896, incoming bytes=1024). Resetting the ring buffer. Wake word detection accuracy will be reduced.
[17:39:42][W][micro_wake_word:103]: Not enough free bytes in ring buffer to store incoming audio data (free bytes=640, incoming bytes=1024). Resetting the ring buffer. Wake word detection accuracy will be reduced.

[dreimer1986]

Hm, maybe this explains why the microphone still is a weak spot on the Onju I tinker with? I have the same warnings of course and sometimes I have problems getting the wake word detected, especially if I am more than 2m away and/or speak slower. If so, maybe both settings should be added to configuration? Shall I open some issues or are you planning to do something in that direction already?

Btw, just to be safe it's not my modding around making things worse:
I use my own lil tinkering fork for this: https://github.com/dreimer1986/onju-voice-satellite/tree/main

Only added:

• MicroWakeWord v2
• updated the sample rates and added resampler for mic
• added codec: auto to make Music Assistant happy
• added -DBOARD_HAS_PSRAM build parameter (No clue if this is of any use at all.)

This one builds fine even with the new Beta, but fails to link due to some Beta problems (esphome/issues#6036), so try it on stable version for now. (As I prepared for 2024.7 already you likely need the external components block from 1d83037 to make the older ESPHome happy again)

[dreimer1986]

OK, regarding the mic is weak problem. I tinkered with all kind of settings the last few days and it did not get that much better until... I talked to heinzpeda on ESPHome Discord. He recommended to use 48000 Hz for the pipeline and the result is day and night to before! I am back to accidental and funny activations and starting a sarcastic discussion about some unrelated phrase it caught up. That's how good the mic is now again. Media Player works flawless, too. I added some experimental routine that auto disables the wake word on playback and reactivates it on pause and stop. All in all I have what I want now except some stuff like a way to reply to a question the Assistant asks, and a custom micro wake word, but this is nothing we can do yet.

[jherby2k]

hey @dreimer1986 i tried your fork and had some success. These two settings seem to be causing inaccuracies though:

  auto_gain: 31dBFS
  volume_multiplier: 8

removing them massively improved whisper accuracy (i assume the mic was picking up lots of noise / clipping) yet it can still hear me from a ways off.

also looks like you removed this under microphone, which seems necessary:

gain_log2: 3

I had to disable your auto-disable wake word stuff as it tends to leave me in a broken state after one or two questions, and left my mute switch backwards.

I'm also having general instability, but i'll fiddle some more.

[jherby2k]

Unfortunately despite playing back at 48khz/32bit w/resampling and decoding directly from FLAC via Music Assistant, it still sounds like crap. I have an unmodified Nest Mini right next to it and it sounds way, way better on the same tracks. Sounds like a $5 bluetooth speaker / AM radio. Not sure this is something that can be fixed in software.

[jherby2k]

I've fiddled with a bunch of settings to see what works, what's redundant etc and this is what i've found. Of note, use_apll seems to improve things somewhat, although the Nest Mini still sounds better.

adf_pipeline:
  - platform: i2s_audio
    type: audio_out
    id: adf_i2s_out
    i2s_audio_id: i2s_shared
    i2s_dout_pin: GPIO12
    fixed_settings: true
    use_apll: true              # Quality - Seems to improve high frequency response to my ears at least
    channel: left
    sample_rate: 48000          # Quality improvement
    #adf_alc: true              # implicit - if set to false, volume can't be controlled
    #alc_max: 0.5               # This just makes everything quieter. I've left it out
    #bits_per_sample: 32bit     # Implicit - 32-bit is now the default

  - platform: i2s_audio
    type: audio_in
    id: adf_i2s_in
    i2s_audio_id: i2s_shared
    i2s_din_pin: GPIO17
    fixed_settings: true
    use_apll: true              # Set here to match the output
    channel: left 
    sample_rate: 48000          # Set here to match the output
    pdm: false                  # required to compile
    #bits_per_sample: 32bit     # Implicit - 32-bit is now the default

microphone:
  - platform: adf_pipeline
    id: onju_microphone
    keep_pipeline_alive: true
    gain_log2: 3
    pipeline:
      - adf_i2s_in
      - resampler               # Quality - lets you use > 16 kHz sample rates
      - self

media_player:
  - platform: adf_pipeline
    id: onju_out
    name: None
    internal: false
    keep_pipeline_alive: true
    codec: auto                 # Quality - Lets Music Assistant stream FLAC directly, instead of using the trancoding to MP3 option
    pipeline:
      - self
      - resampler
      - adf_i2s_out

[jherby2k]

Where / how this would get integrated is above my skill level, but we might be able to improve ADC in software with something like this:

https://github.com/G6EJD/ESP32-ADC-Accuracy-Improvement-function

would ideally take some measurements off an actual Onju and tweak as necesary.

[dreimer1986]

I thought it would be a bad idea to use this issue for my stuff, but as this is meant to help in the original code, I think it's fine to continue. If not @tetele please just tell us.

Sooo... Many things to reply to:

Regarding auto_gain: 31dBFS and volume_multiplier: 8. Yes, the 8 was a bit too much and was a remnant when the microphone did a terrible job when the pipeline is set on 44100 Hz, but I see no reason to remove it altogether. I never used Piper and Whisper, so no idea if it dislikes them, but the official cloud solution does not. So I went down to volume_multiplier: 2 like the official S3 board configs did. Seems to be still fine over here.

Regarding gain_log2: 3 removed. Yes, I did remove it as it did not do anything positive, but negative over here. Once more I took the official config as example and removed it. Never had a situation where it was needed for working. In my case it almost completely disables any text detection and after the wake word it's impossible to get any more information into it. Just added it again, had the same flaws and removed it again which made it working fine once more. So over here the setting is fatal to be added.

The auto disable wake word stuff works fine here, so I keep it there for now and do some more testing.

Regarding the audio quality. Tbh it was below my expectations, too. But as I never heard the original I just thought it's how it is. Tbh it's a small device and I thought that I cannot expect much more off it. So you say it should sound better huh? I will try your APLL precision timer now as this one is new to me and give feedback regarding it. All in all the quality maxed out is the reason for all the complex media player usage in this config, so we should do all things possible to make it do the best job possible.

EDIT: OK one of them is quick to answer... use_apll is a GODSEND when things are more complex in the music playback. Just listened to Sternhagelvoll from In Extremo and was surprised how good the bagpipes sounds now. So yes, this one is a MUST HAVE!

EDIT2: And another new one... If I remove the auto_gain and volume_multiplier settings then I can set gain_log2: 3 and it works fine. I have no clue which one would be to prefer here... yet.

[tetele]

Do go on, please. I'll collect this stuff when i finally get the chance to tackle the issue.

My only question is how have you guys overcome the ring buffer overflow issue i've mentioned?

[dreimer1986]

Well... I did nothing at all. It just works fine here. I saw it a few times in my logs, but it seems like it has no negative effect here. Sometimes my Onju is a bit moody and does not want to work on 1st try, but most of the time it works just fine. Compared to the moodyness of my Echo devices it's still waay more reliable. I more often have a problem with micro wake word telling me that it detected my wake word, but the vad says no. Exact wording is difficult right now as I cannot make it output it right now... I will add it as soon as I managed to persuade it to do that again ^^

EDIT:

[02:18:03][D][micro_wake_word:162]: The 'okay nabu' model sliding average probability is 0.987 and most recent probability is 0.973
[02:18:03][D][micro_wake_word:331]: Wake word model predicts okay nabu, but VAD model doesn't.

[jherby2k]

I only saw that ring buffer issue when not using the resampler for input and/or without the PSRAM. Not an issue for a while.

[jhbruhn]

I still don't understand why gain_log2: 3 does cause harm for dreimer1986, but for other devices (like mine) it is required for the Onju Voice Mics to deliver usable audio. Could there be a cache-mismatch between esphome_audio versions somewhere? The ESPHome version is the most recent one IIRC.

Generally I think setting gain_log2 is better than using volume_multiplier if one is needed because gain_log2 will affect the microphones output directly, which feeds into both micro_wake_word and voice_assistant, while volume_multiplier will only affect voice_assistant.

gain_log2 = 1 should be equal to volume_multiplier = 2, gain_log2 = 2 -> volume_multiplier = 4 etc.

Perhaps PCBWay used a different microphone for some reason? My quoted BOM mentions SPH0645LM4H-B.

[dreimer1986]

The dIfference is that I kept volume_multiplier and auto gain. Thus I bet it was WAAAY too loud. I went your way by now btw: dreimer1986@9747209

[jherby2k]

i'm very curious what you guys think about use_apll: true. Listen to some music with high frequencies like lots of cymbals. I feel like they're cleaner using the high-frequency clock, but i can't really do an ABX test here easily.

[dreimer1986]

I already had my test. Not side by side as I just have one Onju Voice right now. (Enough PCBs left to even sell a few, but no Nest to take apart) But the results are clearly there and noticeable and thus worth the change I would say. I have my standard testing song with medieval instruments, bagpipes and singing. (In Extremo - Sternhagelvoll) It starts with a bagpipes solo with echo effect and then continues with electrical guitar and drums. The first 15 sec already show a big difference. The solo is not much difference, but as soon as the guitar starts to play the bagpipes became sort of "muffled". APLL reduces this IMO and the bagpipes are clearly better to differentiate than before. I would say the quality clearly is improving with APLL. Still not to a audiophile level, but this thing is a small handful of sound. Of course it's nothing to compare to my Teufel Ultima 40 hell of a sound system.

[witold-gren]

Hey guys, I don't know ESPHome that well (I'm just learning it) but I noticed a few changes in the latest version that I think may make it easier to improve the sound quality. esphome/esphome#7306

[witold-gren]

I did some tests on my devices and discovered something interesting.. in HA i enabled debug mode for my all 5 devices..

assist_pipeline:
  debug_recording_dir: /share/assist_pipeline

1. I install this update in all 5 devices and also I disabled wake words (@jherby2k version).
2. I just tap on top of device and I start talking something.
3. I go to check te result in HA. And to my surprise 3 devices works correct and 2 of them I heard a constant screeching noise.
4. I decided to return to the previous version of the software and it turned out that (@tetele version).
5. In one device this screeching noise is gone, but In second device is it incredibly laugh.
6. In my other 3 devices, when I back to original configuration from @tetele the the recording was of lower quality then configuration from @jherby2k .
7. I currently have @jherby2k version on 3 devices, and @tetele version in 2 of devices..

This is very strange because I thought they would be exactly the same PCB..Do you have similar ones?

[jhbruhn]

I've had issues on one of my PCBs with one Mic being defective (or just badly soldered). Luckily I was able to change the mic channel which then used the second mic on the PCB.

At least that was my theory for that PCB...

[witold-gren]

Another discovery, turning off the devices that were beeping for 5 minutes caused the squeaking to disappear. Now I was able to use the version from @jherby2k again on each of the devices. However, after checking, part of devices have better sound quality than others.

[witold-gren]

I continue debugging my devices and I discover new problems:

1. I uploaded new version of software: https://soundcloud.com/witold-gren/1-device1-1?in=witold-gren/sets/onju-voice&si=74a51a224d10414a96dfc65ae74e335d&utm_source=clipboard&utm_medium=text&utm_campaign=social_sharing
2. I disabled power for 1 min: https://soundcloud.com/witold-gren/2-device1-1?in=witold-gren/sets/onju-voice&si=82c39daf916d44f38b15ab9f317f06d5&utm_source=clipboard&utm_medium=text&utm_campaign=social_sharing
3. I disabled power for all night and after I run devices the voice was inaudible: https://soundcloud.com/witold-gren/3-device1-1?in=witold-gren/sets/onju-voice&si=25d118c721eb41fdb4f6007d4b3bb2cf&utm_source=clipboard&utm_medium=text&utm_campaign=social_sharing
4. When I turned the devices off again for 1 minute, everything returned to normal: https://soundcloud.com/witold-gren/4-device1-1?in=witold-gren/sets/onju-voice&si=0b2682cad72947f9a22dc7b5a25dceff&utm_source=clipboard&utm_medium=text&utm_campaign=social_sharing

A similar situation occurs in other devices:

1. https://soundcloud.com/witold-gren/5-device2-1?in=witold-gren/sets/onju-voice&si=e956f5f51c094c0eae766c29e0e6996c&utm_source=clipboard&utm_medium=text&utm_campaign=social_sharing
2. https://soundcloud.com/witold-gren/6-device2-1?in=witold-gren/sets/onju-voice&si=c696c2e8801945a79d4486a475e3fb20&utm_source=clipboard&utm_medium=text&utm_campaign=social_sharing
3. https://soundcloud.com/witold-gren/7-device2-1?in=witold-gren/sets/onju-voice&si=e61d28da450f4cf7bc0b894f9d7e99fc&utm_source=clipboard&utm_medium=text&utm_campaign=social_sharing
4. https://soundcloud.com/witold-gren/8-device2-1?in=witold-gren/sets/onju-voice&si=160f56c539df4b99a343b08d6ae14d12&utm_source=clipboard&utm_medium=text&utm_campaign=social_sharing

I also have devices that sound very good:

1. Device 1 (after restart) - https://soundcloud.com/witold-gren/device1?in=witold-gren/sets/onju-voice&si=a913b2729ae54aa690ff1279738f2139&utm_source=clipboard&utm_medium=text&utm_campaign=social_sharing
2. Device 2 (after restart) - https://soundcloud.com/witold-gren/device2?in=witold-gren/sets/onju-voice&si=fe43b878ec8747d49aeebd61d3eb849a&utm_source=clipboard&utm_medium=text&utm_campaign=social_sharing
3. Device 3 (after restart) - https://soundcloud.com/witold-gren/device3?in=witold-gren/sets/onju-voice&si=9785e708a7b74cf6b3c12b0deaa62d0a&utm_source=clipboard&utm_medium=text&utm_campaign=social_sharing

I don't understand what's going on here at all. Sometimes the devices have very good sound and sometimes it is unclear or there is only a squeaking noise. ¯_(ツ)_/¯

[TheStigh]

Seems there is something going on with your pcb's. I've just tested 2 of the 5 and they work great.

[TheStigh]

I've had issues on one of my PCBs with one Mic being defective (or just badly soldered). Luckily I was able to change the mic channel which then used the second mic on the PCB.

At least that was my theory for that PCB...

You're in Norway?

[ashald]

Not sure if this makes sense, but in my experience Onju can sound much better if we could adjust the equalizer. To me, the medium frequencies stand out too much, while hi and low are lacking. In Arduino-based version it's super trivial to add support for equalizer, so I did some quick patching in my local version of the component and I think it sounded much better with +6/-12/+6 gain adjustments.

The ADF Pipeline version sounds... terrible to me "out of the box" but it gets notably better with use_apll enabled as in @jherby2k 's config (#52 (comment)). But as stated by others, it helps a lot with hi frequencies while there's still to much of med and too few of low. IDK if there's any other falg similar to use_apll that can help with it, but I found that ADF supports equalizer (https://docs.espressif.com/projects/esp-adf/en/latest/api-reference/audio-processing/equalizer.html) and tried to follow an example https://github.com/espressif/esp-adf/blob/master/examples/audio_processing/pipeline_equalizer/main/equalizer_example.c to adjust @gnumpi's code to add an equalizer pipeline component (ashald/esphome_audio@448ba10).

Unfortunately, I couldn't make it work [at least yet]. But that's exactly why I'm writing here as I'm hoping someone who's more knowledgeable in this domain can help by pointing out what's wrong. I don't think I structured the code "in the right way" (eg the equalizer component should not be under i2s) but as I'm not familiar with the framework, was just trying to make things work with minimum amount of changes.

The way I tried it in my config was:

external_components:
  - components: [adf_pipeline, i2s_audio]
    source:
      type: git
      url: https://github.com/ashald/esphome_audio
      ref: eq

adf_pipeline:
    # ... rest from @jherby2k's config
  - id: onju_out_equalizer
    type: equalizer
    platform: i2s_audio         # This should've been idf_pipeline... if code was in the right place

media_player:
  - platform: adf_pipeline
    id: onju_out
    name: "${name}"
    internal: false
    keep_pipeline_alive: true
    codec: auto
    pipeline:
      - self
      - resampler
      - onju_out_equalizer  # That's the only change withing media_player config
      - adf_i2s_out

Then the intent was to add numeric inputs to adjust each band to real-time tuning, and extend config to provide defaults, but I left it out of the above sample for brevity.

As to how it doesn't work - the sound appears distorted with EQ enabled, hear the recordings: https://soundcloud.com/user-430926493-669422671/sets/adf-eq

I also tried adding onju_out_equalizer to the mic pipeline (just in case if for whatever reasons pipelines need to be similar - I don't fully understand how ADF works), but it crashes that way.. seemingly because it adds 2 equalizers to the pipeline, which I don't understand why (as in, I don't understand why eg that doesn't happen to the resampler).

Hoping someone can help to figure this out so we can get our Onjus sounding even better while running on ADF with MWW.
Thanks!