AudioWorklet issue with Vite.

[huwprosser]

This is a really cool project. Massive respect.

I'm trying to add this to a Vite/React project but I'm getting a
DOMException: The user aborted a request.

Error whenever I run the code. Any help or advice would be massively appreciated. I think it's something to do with the audio worklet in real-time-vad.ts but don't know enough about this to diagnose.

[ricky0123]

Thanks for the kind words!

I have used it in a react project with success so far by using setState to store the MIcVAD object and useEffect to initialize it. Also, you will have to serve the onnx/wasm/worklet files (see the webpack config in the README here). I plan to create a new package called ricky0123/vad-react with a custom hook to use the vad in react projects in the coming days.

You can try thisas a custom hook to get started

export function useVAD(args: Partial<RealTimeVADOptions>) {
  const [vadRunning, setVadRunning] = useState<boolean>(false)
  const [vad, setVAD] = useState<MicVAD | null>(null)
  const pauseVAD = () => {
    vad?.pause()
    setVadRunning(false)
  }
  const startVAD = () => {
    vad?.start()
    setVadRunning(true)
  }
  useEffect(() => {
    ;(async () => {
      const myvad = await MicVAD.new(args)
      setVAD(myvad)
      myvad.start()
      setVadRunning(true)
    })()
    return function cleanUp() {
      pauseVAD()
    }
  }, [])
  return {
    vadRunning,
    pauseVAD,
    startVAD,
  }
}

I definitely won't guarantee it's bug free, but it has been working so far for me.

[ricky0123]

To clarify about serving the onnx/wasm/worklet files, basically you have to

1. serve the onnx file that comes distributed with my package (see the file ending in .onnx in the dist folder here)
2. serve the vad.worklet.js file (as seen in the dist folder I pointed to above
3. serve the wasm files that come distributed with the npm package onnxruntime-web
4. this is redundant from the last point, but I think you need to add onnxruntime-web as a dependency, because I've made it an "external" dependency of my package (installing @ricky0123/vad doesn't install onnxruntime-web)

I'm not familiar with vite, but maybe there are analogous plugins to the ones I use in the webpack config snippet in the README.

[ricky0123]

Gonna close this for now. Let me know if you have any further issues. FYI here's now a package @ricky0123/vad-react for react. I've expanded the documentation a bit for bundling. I'll try to add an example for Vite at some point but I don't have time right now. The two easiest options are probably adding a bash script to add the worklet and other necessary files to your dist folder, or using something like vite-plugin-static-copy.

[Craigtut]

Adding some info for others that come across this thread. Use the plugin mentioned above with config like this:

vite.config.ts:

export default defineConfig({
  plugins: [
    viteStaticCopy({
      targets: [
        {
          src: 'node_modules/@ricky0123/vad-web/dist/vad.worklet.bundle.min.js',
          dest: '/'
        },
        {
          src: 'node_modules/@ricky0123/vad-web/dist/silero_vad.onnx',
          dest: '/'
        },
        {
          src: 'node_modules/onnxruntime-web/dist/*.wasm',
          dest: '/'
        }
      ]
    })],
})

[cipherhell]

Adding some info for others that come across this thread. Use the plugin mentioned above with config like this:

vite.config.ts:

export default defineConfig({
  plugins: [
    viteStaticCopy({
      targets: [
        {
          src: 'node_modules/@ricky0123/vad-web/dist/vad.worklet.bundle.min.js',
          dest: '/'
        },
        {
          src: 'node_modules/@ricky0123/vad-web/dist/silero_vad.onnx',
          dest: '/'
        },
        {
          src: 'node_modules/onnxruntime-web/dist/*.wasm',
          dest: '/'
        }
      ]
    })],
})

This should be added as suggestion in the documentation, that would be very helpful...

[xiaoqiang1999]

Adding some info for others that come across this thread. Use the plugin mentioned above with config like this:

vite.config.ts:

export default defineConfig({
  plugins: [
    viteStaticCopy({
      targets: [
        {
          src: 'node_modules/@ricky0123/vad-web/dist/vad.worklet.bundle.min.js',
          dest: '/'
        },
        {
          src: 'node_modules/@ricky0123/vad-web/dist/silero_vad.onnx',
          dest: '/'
        },
        {
          src: 'node_modules/onnxruntime-web/dist/*.wasm',
          dest: '/'
        }
      ]
    })],
})

It's very useful, thank you!

[xiaoqiang1999]

@ricky0123
Also, could you consider allowing users to specify the path to the model file, onnxruntime, and js themselves?
Just like this library: @vladmandic/human
It is a face recognition library based on Tensorflow. It only supports manually specifying the model root path, otherwise it will not be available.

Users can manually copy files such as models to DevServer and specify the path. The experience is good.

[xiaoqiang1999]

Adding some info for others that come across this thread. Use the plugin mentioned above with config like this:
vite.config.ts:

export default defineConfig({
  plugins: [
    viteStaticCopy({
      targets: [
        {
          src: 'node_modules/@ricky0123/vad-web/dist/vad.worklet.bundle.min.js',
          dest: '/'
        },
        {
          src: 'node_modules/@ricky0123/vad-web/dist/silero_vad.onnx',
          dest: '/'
        },
        {
          src: 'node_modules/onnxruntime-web/dist/*.wasm',
          dest: '/'
        }
      ]
    })],
})

This should be added as suggestion in the documentation, that would be very helpful...

I have added it to the document and made some adjustments:
User guide for browser use

[ricky0123]

@ricky0123 Also, could you consider allowing users to specify the path to the model file, onnxruntime, and js themselves? Just like this library: @vladmandic/human It is a face recognition library based on Tensorflow. It only supports manually specifying the model root path, otherwise it will not be available.

Users can manually copy files such as models to DevServer and specify the path. The experience is good.

Hi @xiaoqiang1999 I had not considered that, but yeah, it might not be a bad idea. This seems to be the single biggest source of frustration for users. Thanks for the suggestion!

[Louise-Huang]

Hi @ricky0123
I'm trying to add this to a Vite/Vue project, and I also use the plugin in vite.config.ts, but I'm still getting a
DOMException: The user aborted a request.

vite.config.ts:

import { viteStaticCopy } from "vite-plugin-static-copy";
export default defineConfig({
  plugins: [
    viteStaticCopy({
      targets: [
        {
          src: "node_modules/@ricky0123/vad-web/dist/vad.worklet.bundle.min.js",
          dest: "./",
        },
        {
          src: "node_modules/@ricky0123/vad-web/dist/silero_vad.onnx",
          dest: "./",
        },
        {
          src: "node_modules/onnxruntime-web/dist/*.wasm",
          dest: "./",
        },
      ],
    }),
  ]
});

[ricky0123]

Hi @Louise-Huang can you look in browser devtools in the network tab and see where the site is trying to get these files?

[Louise-Huang]

Hi @ricky0123 I added all the comments mentioned in issue #128 that should be added, and it works for me!