Add support for OpenVino noise suppression model in the SOF plugin #8847

ranj063 · 2024-02-09T00:42:58Z

Introduce a new module that performs noise suppression. The module loads the noise suppression model using the OpenVino Runtime plugin, processes the input samples to produce output samples with clean speech. Currently, the module is hard-coded to compile the model to be run on the CPU only and will be extended for other devices like the NPU in the future.

To prevent the following with C++ compilation. error: invalid conversion from ‘uint32_t’ {aka ‘unsigned int’} to ‘sof_ipc_frame’ [-fpermissive] Signed-off-by: Ranjani Sridharan <[email protected]>

Add the missing return in the hw_free callback. Signed-off-by: Ranjani Sridharan <[email protected]>

For now, only process widgets that don't need the basecfg extension are supported. Signed-off-by: Ranjani Sridharan <[email protected]>

Set both ibs/obs to be based on the ALSA period_size to make sure that the intermediate buffers are large enough. Signed-off-by: Ranjani Sridharan <[email protected]>

Set the host widget correctly and use the capture specific functions during prepare/free. Signed-off-by: Ranjani Sridharan <[email protected]>

This will be needed for the plugin voice PCM in preparation to add the noise suppression module in the host pipeline. Signed-off-by: Ranjani Sridharan <[email protected]>

lyakh · 2024-02-09T07:58:49Z

posix/include/rtos/panic.h

+/* hack for now to prevent compilation warning
+ * warning: ISO C++ forbids converting a string constant to ‘char*’ [-Wwrite-strings]
+ */
+#define assert(cond) (void)(cond)


just as an idea - maybe you could use

#ifdef __cplusplus

I fixed it by changing the filename argument in __panic() to const char *

singalsu · 2024-02-09T16:46:51Z

tools/plugin/modules/noise_suppression_interface.cpp

+		for (ch = 0; ch < NS_MAX_SOURCE_CHANNELS; ch++) {
+			/* split each channel samples and convert to floating point */
+			for (i = ch, j = 0; j < frame_count; i+=2,j++)
+				inp_wave_fp32[j] = (float)input_data[i] * scale;


Is there no need to check input and output buffer for wrap? I can't see a module interface struct populate, but this is still legacy process type and not sink/source API, right?

yes, legacy process. Do you think sink/source makes more sense with this?

As long as legacy is not deprecated should be no need. The sink/source API allows resources savings in DSP but it's not as critical on the host side. Also we don't yet have those savings implemented so currently there's no difference. The no-wrap check linear buffers will be a feature in sink/source API.

I was asking if it's ensured in plugin that buffers can be operated as linear even in legacy. It might work so too in FW but I'm not sure it's ensured. The components have still the samples/frame before wrap functions in use.

@singalsu I've added the checks for input/output wrap now.

Make the filename argument be a const char * to avoid the following warning with c++ compliation: warning: ISO C++ forbids converting a string constant to ‘char*’ [-Wwrite-strings] Signed-off-by: Ranjani Sridharan <[email protected]>

Introduce a new module that performs noise suppression. The module loads the noise suppression model using the OpenVino Runtime plugin, processes the input samples to produce output samples with clean speech. CUrrently, the module is hard-coded to compile the model to be run on the CPU only and will be extended for other devices like the NPU in the future. Signed-off-by: Ranjani Sridharan <[email protected]>

Add the conf file for the OpenVino noise suppression module and introduce the module in the capture path in the plugin topology when noise suppression is enabled. ATM, only 16K capture is supported by the model, so change the input/output rates for the host/dai modules in the plugin capture pipeline. Signed-off-by: Ranjani Sridharan <[email protected]>

Fix the stale instruction for install and add the steps for setting up OpenVino/OpenCV to test the noise suppression model from the open model zoo repository. Signed-off-by: Ranjani Sridharan <[email protected]>

singalsu · 2024-02-14T08:06:09Z

tools/plugin/modules/ov_noise_suppression/noise_suppression_interface.cpp

+				/* wrap if needed */
+				if (inp >= source->end_addr)
+			                inp = (char *)source->addr +
+			                        ((char *)inp - (char *)source->end_addr);


This works but it's not efficient. The linear buffer can be filled in two steps with help from audio stream functions for samples or frames before wrap. We've fixed a lot of these during last year for better performance.

@singalsu lets take this as an incremental fix. Can you give a pointer to good example code to copy.

Yes it's ok, this is not as consuming as read/write frag.

dbaluta · 2024-02-14T09:38:15Z

@ranj063 do you have SOF running as an userspace plugin in ALSA? And this is an addition to that plugin for NS model?

lgirdwood · 2024-02-14T14:44:03Z

src/include/sof/audio/audio_stream.h

@@ -365,7 +365,7 @@ static inline int audio_stream_set_params(struct audio_stream *buffer,
 	if (!params)
 		return -EINVAL;

-	buffer->runtime_stream_params.frame_fmt = params->frame_fmt;
+	buffer->runtime_stream_params.frame_fmt = (enum sof_ipc_frame)params->frame_fmt;


Good catch by G++, these should be fixed at source though, but sadly I dont think all C compilers allow this

diff --git a/src/include/ipc/stream.h b/src/include/ipc/stream.h index c397f34a1..6466b2cbc 100644 --- a/src/include/ipc/stream.h +++ b/src/include/ipc/stream.h @@ -75,7 +75,7 @@ struct sof_ipc_stream_params { struct sof_ipc_hdr hdr; struct sof_ipc_host_buffer buffer; uint32_t direction; /**< enum sof_ipc_stream_direction */ - uint32_t frame_fmt; /**< enum sof_ipc_frame */ + enum sof_ipc_frame frame_fmt:32; /**< enum sof_ipc_frame */ uint32_t buffer_fmt; /**< enum sof_ipc_buffer_format */ uint32_t rate; uint16_t stream_tag;

lgirdwood · 2024-02-14T14:46:30Z

tools/plugin/modules/ov_noise_suppression/noise_suppression_interface.cpp

+				/* wrap if needed */
+				if (inp >= source->end_addr)
+			                inp = (char *)source->addr +
+			                        ((char *)inp - (char *)source->end_addr);


@singalsu lets take this as an incremental fix. Can you give a pointer to good example code to copy.

lgirdwood · 2024-02-14T14:48:01Z

@ranj063 do you have SOF running as an userspace plugin in ALSA? And this is an addition to that plugin for NS model?

Yes. Exactly. Still all WIP code, not production ready yet.

marc-hb · 2024-02-14T18:49:59Z

tools/plugin/modules/ov_noise_suppression/CMakeLists.txt

+# Noise suppression module
+find_package(OpenVINO COMPONENTS Runtime)
+
+if(OpenVINO_FOUND)


Suggested change

if(OpenVINO_FOUND)

if(OpenVINO_FOUND)

Can you move the find_package() and if() to the parent CMakeLists.txt? This avoids the entire file being guarded.

marc-hb · 2024-02-14T18:52:25Z

Wow, both ACE https://sof-ci.01.org/sofpr/PR8847/build2695/devicetest/index.html and CAVS https://sof-ci.01.org/sofpr/PR8847/build2696/devicetest/index.html are 100% green! This OpenVino stuff is miraculous, it fixes suspend/resume failures! :-D

singalsu · 2024-02-15T09:25:06Z

Wow, both ACE https://sof-ci.01.org/sofpr/PR8847/build2695/devicetest/index.html and CAVS https://sof-ci.01.org/sofpr/PR8847/build2696/devicetest/index.html are 100% green! This OpenVino stuff is miraculous, it fixes suspend/resume failures! :-D

A plugin build and run CI step would be good.

ranj063 · 2024-02-15T19:36:51Z

Wow, both ACE https://sof-ci.01.org/sofpr/PR8847/build2695/devicetest/index.html and CAVS https://sof-ci.01.org/sofpr/PR8847/build2696/devicetest/index.html are 100% green! This OpenVino stuff is miraculous, it fixes suspend/resume failures! :-D

aww i dont want to fix the find_package stuff now. I like all greens!

ranj063 added 6 commits February 8, 2024 16:40

audio: audio_stream: Explicitly cast to enum

d7f13a1

To prevent the following with C++ compilation. error: invalid conversion from ‘uint32_t’ {aka ‘unsigned int’} to ‘sof_ipc_frame’ [-fpermissive] Signed-off-by: Ranjani Sridharan <[email protected]>

tools: plugin: pcm: Add missing return

0d1fa25

Add the missing return in the hw_free callback. Signed-off-by: Ranjani Sridharan <[email protected]>

tools: plugin: tplg: Fix parsing of process widgets

0b541e9

For now, only process widgets that don't need the basecfg extension are supported. Signed-off-by: Ranjani Sridharan <[email protected]>

tools: plugin: tplg: Fix ibs/obs for all widgets

ee1f8e2

Set both ibs/obs to be based on the ALSA period_size to make sure that the intermediate buffers are large enough. Signed-off-by: Ranjani Sridharan <[email protected]>

tools: plugin: Fix prepare/free widgets for capture

49fb17d

Set the host widget correctly and use the capture specific functions during prepare/free. Signed-off-by: Ranjani Sridharan <[email protected]>

tools: plugin: 50-sof: Add new config for 16KHz stereo 16-bit format

26aaaad

This will be needed for the plugin voice PCM in preparation to add the noise suppression module in the host pipeline. Signed-off-by: Ranjani Sridharan <[email protected]>

ranj063 requested review from marc-hb, jsarha, lgirdwood, plbossart, mmaka1, lbetlej, dbaluta and kv2019i as code owners February 9, 2024 00:42

lyakh reviewed Feb 9, 2024

View reviewed changes

singalsu reviewed Feb 9, 2024

View reviewed changes

ranj063 force-pushed the plugin-noise-suppression branch 2 times, most recently from e3a6e2d to ca595ae Compare February 9, 2024 19:35

src: panic: Modify the signature of __panic()

1d61283

Make the filename argument be a const char * to avoid the following warning with c++ compliation: warning: ISO C++ forbids converting a string constant to ‘char*’ [-Wwrite-strings] Signed-off-by: Ranjani Sridharan <[email protected]>

ranj063 force-pushed the plugin-noise-suppression branch from ca595ae to 814a7de Compare February 9, 2024 22:29

ranj063 added 3 commits February 13, 2024 09:14

tools: plugin: README: Add instructions for testing noise suppression

c790f60

Fix the stale instruction for install and add the steps for setting up OpenVino/OpenCV to test the noise suppression model from the open model zoo repository. Signed-off-by: Ranjani Sridharan <[email protected]>

ranj063 force-pushed the plugin-noise-suppression branch from 814a7de to c790f60 Compare February 13, 2024 17:14

ranj063 requested review from singalsu and lyakh February 13, 2024 17:19

singalsu reviewed Feb 14, 2024

View reviewed changes

lgirdwood approved these changes Feb 14, 2024

View reviewed changes

singalsu approved these changes Feb 14, 2024

View reviewed changes

marc-hb reviewed Feb 14, 2024

View reviewed changes

kv2019i approved these changes Feb 16, 2024

View reviewed changes

kv2019i merged commit 48462a4 into thesofproject:main Feb 16, 2024
38 of 44 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for OpenVino noise suppression model in the SOF plugin #8847

Add support for OpenVino noise suppression model in the SOF plugin #8847

ranj063 commented Feb 9, 2024

lyakh Feb 9, 2024

ranj063 Feb 9, 2024

singalsu Feb 9, 2024

ranj063 Feb 9, 2024

singalsu Feb 9, 2024

ranj063 Feb 13, 2024

singalsu Feb 14, 2024

lgirdwood Feb 14, 2024

singalsu Feb 14, 2024

dbaluta commented Feb 14, 2024

lgirdwood Feb 14, 2024

lgirdwood Feb 14, 2024

lgirdwood commented Feb 14, 2024

marc-hb Feb 14, 2024

marc-hb commented Feb 14, 2024 •

edited

Loading

singalsu commented Feb 15, 2024

ranj063 commented Feb 15, 2024

Add support for OpenVino noise suppression model in the SOF plugin #8847

Add support for OpenVino noise suppression model in the SOF plugin #8847

Conversation

ranj063 commented Feb 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dbaluta commented Feb 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lgirdwood commented Feb 14, 2024

Choose a reason for hiding this comment

marc-hb commented Feb 14, 2024 • edited Loading

singalsu commented Feb 15, 2024

ranj063 commented Feb 15, 2024

marc-hb commented Feb 14, 2024 •

edited

Loading