[BUG]: The sound is captured back to the microphone

[MaximKurbanov]

Package version

3.0.0-pre.6

Environment

* OS: Android
* Unity version:2021.3

Hi!
I have big problem with my voice chat App. When 2 people voice each other by mobile phone. The voice sound of the speaker is recorded back to the microphone, and causing echo. After many times the sound is echoed and resonant with each other, causing loud noises. I've been looking for solutions like AEC(Acoustic Echo Cancellation), Echo Suppression, but it's not work for Unity.
Have you ever had this problem or has any solutions, please let me know. Thanks so much!!!

Steps To Reproduce

Run app on 2 mobile devices

Current Behavior

No response

Expected Behavior

No response

Anything else?

No response

[karasusan]

@MaximKurbanov

#651
Is this the same issue?

[Joshua-Douglas]

Hello,

Thank you so much for your tireless efforts with this package @karasusan! My team has been using this package extensively and we've found that it works very well. I'm wondering if this bug might be a good opportunity for our team to contribute back to this repository.

I've noticed that the current implementation hard-codes the AudioOptions within the WebRTCPlugin Context object.

rtc::scoped_refptr<AudioSourceInterface> Context::CreateAudioSource()
    {
        // avoid optimization specially for voice
        cricket::AudioOptions audioOptions; // echo_cancellation will default to nullptr
        audioOptions.auto_gain_control = false;
        audioOptions.noise_suppression = false;
        audioOptions.highpass_filter = false;
        return UnityAudioTrackSource::Create(audioOptions);
    }

Do you think noise_suppression, echo_cancellation, etc would work as expected if these options were set to true? If so, then we might be able to close out this issue by exposing the AudioOptions and allowing end-users to specify echo_cancellation if they need it (this would also close #587 I believe).

If you think this would be a good approach, and if you are open to contributions, then I would be happy to submit a fix. I'd be happy to outline the changes in a design document before submitting a PR, so you could approve the changes before I begin implementing.

[karasusan]

@Joshua-Douglas
Hi, always we welcome to your contribution. Making these audio options to public is a first step to solve this issue. I'm not sure these opitions can fix it, therefore we need to test these options.

If you make a PR, we will be glad to review and test.

[Joshua-Douglas]

Hello @karasusan, that is great to hear - thanks!

Would you rather I submit the PR directly, or provide you with an outline of my planned changes before I submit? I don't mind sharing my outlined design before implementing, since these changes will impact the public API.

Either option works for me. I'll keep you updated.

[karasusan]

@Joshua-Douglas
I'd like to see your planned changes of the public API.
Maybe we need to make a new class to express AudioOptions.

[karasusan]

@Joshua-Douglas
Have you already develop for this issue?
I'm going to start this task before long.

[Joshua-Douglas]

@karasusan I have not started. If you are ready to start soon, then that would be great! I have another open source project than I'm supporting, so you would probably get it done much faster.

Can't wait to see your changes! It would be great if this does enable echo cancellation🤞!

[karasusan]

@Joshua-Douglas
I have already started this issue. This is a draft PR.

• feat: Add AudioOptions class #978

Can you try that If I make a package file to test this fix?

[MaximKurbanov]

@Joshua-Douglas I have already started this issue. This is a draft PR.

• feat: Add AudioOptions class #978

Can you try that If I make a package file to test this fix?

I can.

[karasusan]

I found the issue in our implementation for audio input from Unity. We need to fix it to use these audio options. Sorry but this fix is dropped from the next release version pre.7.

We will focus on this issue in the next version.

[Shushpancheak]

We have tried to implement AudioProcessingModule in native code, and added the following code in what I believe the two points at which your program either receives the audio, or sends it:

UnityAudioTrackSource.cpp

        rtc::scoped_refptr<::webrtc::AudioProcessing> audioProcessingModule = GetAudioProcessingModule();

        if (audioProcessingModule) {
            const size_t sliceDurationMs = 10;
            const size_t sliceLength = (nSampleRate * nNumChannels * sliceDurationMs) / 1000;
            const ::webrtc::StreamConfig streamConfig = StreamConfig(nSampleRate, nNumChannels);

            auto cfg = GetAdditionalAudioProcessingConfig();
            bool invertStreams = cfg.invertStreams;

            for (size_t offset = 0; offset < nNumFrames; offset += sliceLength) {
                size_t sliceLength = std::min(sliceLength, nNumFrames - offset);
                
                if (!invertStreams) {
                    audioProcessingModule->ProcessStream(&_convertedAudioData[offset], streamConfig, streamConfig, &_convertedAudioData[offset]);
                } else {
                    audioProcessingModule->ProcessReverseStream(&_convertedAudioData[offset], streamConfig, streamConfig, &_convertedAudioData[offset]);
                }
            }
        }

AudioTrackSinkAdapter.cpp

        rtc::scoped_refptr<::webrtc::AudioProcessing> audioProcessingModule = GetAudioProcessingModule();

        if (audioProcessingModule) {
            const size_t sliceDurationMs = 10;
            const size_t sliceLength = (sampleRate * channels * sliceDurationMs) / 1000;
            const ::webrtc::StreamConfig streamConfig = StreamConfig(sampleRate, channels);

            auto cfg = GetAdditionalAudioProcessingConfig();
            bool invertStreams = cfg.invertStreams;

            for (size_t offset = 0; offset < readLength; offset += sliceLength) {
                size_t sliceLength = std::min(sliceLength, readLength - offset);
                
                if (!invertStreams) {
                    audioProcessingModule->ProcessReverseStream(&_bufferIn[offset], streamConfig, streamConfig, &_bufferIn[offset]);
                } else {
                    audioProcessingModule->ProcessStream(&_bufferIn[offset], streamConfig, streamConfig, &_bufferIn[offset]);
                }
            }
        }

The audioProcessingModule is a Singleton to which we apply some AudioOptions, including AcousticEchoCancellation.
However, we still get the echo. @karasusan Maybe you can pinpoint us to the location of the native code where we should look at to fix the problem with your implementation for audio input/output?

[karasusan]

@Shushpancheak
I think UnityAudioTrackSource is a right place to process noise cancellation. Maybe there are bugs in the code.

[Mahesh218]

Could you please let us know if there has been a delivery date set for this fix? We're currently facing some difficulties due to this issue. If possible, could you kindly provide us with a tentative date? Thank you very much for your assistance @karasusan

[Shushpancheak]

Are there any updates?

[DanMcPete]

We are still held back by this problem on ios and android mobile devices. webrtc package is rendered unusable on mobile devices when microphone picks up speaker audio without filter. Can you give any update on this fix? Thank you for your help.

[LoopIssuer]

Are there any updates?
@karasusan

[LetMeKillYA]

We are facing the same issue on the Android platform any updates would be helpful. @karasusan

[VanIseghemThomas]

Same issue, also following for updates @karasusan

[Gerson2017]

When can we solve it? It's been so long@karasusan