The web-platform-tests tracking issue

[orottier]

The wpt suite was introduced to this lib in #42. It runs without crashing now and our aim is to improve the test score over time.

This tracking issue lists the high level TODOs. Single test failures are probably best reported and fixed with separate issues (either in the node bindings or the rust library).

• get rid of the wpt fork (how to deal with webaudio/resources/buffer-loader.js, reinstate 8 deleted files)
• how to handle the crashtest tests?
• look into idlharness suite How to run the idl-harness from the wpt suite? #86
• collect results of the browser implementations to determine baseline performance
• implement major missing functionality (Support addEventListener() #31 AudioWorklet support #28 ...)
• improve reported errors (see Error handling #8 Improve error handling #54 )
• check reporter, it seems to count some log lines as errors e.g. × Executing "test"
• focus on foundational test failures (the-audionode-interface, the-audioparam-interface, the-audiocontext-interface, the-offlineaudiocontext-interface, processing-model, the-audiobuffer-interface)
• focus on other test failures
• re-enable 'the-audiocontext-interface/suspend-with-navigation.html'
• re-enable 'the-audionode-interface/audionode-disconnect-audioparam.html'
• re-enable 'the-audionode-interface/audionode-disconnect.html'

[b-ma]

Thanks for the follow up, two notes:

get rid of the wpt fork (how to deal with webaudio/resources/buffer-loader.js, reinstate 8 deleted files)

I think this can be done by

1. Implementing the context.decodeAudioData(response, successCallback, errorCallback) signature, which should be rather simple to do on the JS side
2. Providing a mock to XMLHttpRequest to the runner using raw fs.readFileSync internally

collect results of the browser implementations to determine baseline performance

Maybe a bit old but there is this report: https://webaudio.github.io/web-audio-api/implementation-report.html

[orottier]

I have submitted a PR over at wpt for the buffer-loader.js: web-platform-tests/wpt#43824
Let's see where that gets us

[b-ma]

results from #42

RESULTS:
- # pass: 3413 (+27)
- # fail: 1353 (-27)
- # type error issues: 235 (=)

files ignored: 8 (=)

[b-ma]

results with #61

  RESULTS:
  - # pass: 3605 (+192)
  - # fail: 1284 (-69)
  - # type error issues: 234 (-1)

[b-ma]

results with #66

RESULTS:
  - # pass: 3778 (+173)
  - # fail: 1290 (+6)
  - # type error issues: 59 (-175)

(the SNR is improving)

[orottier]

Results with v0.41.1 (no PR needed here due to patch release, just cargo update)

  RESULTS:
  - # pass: 3858 (+80)
  - # fail: 1216 (-74)
  - # type error issues: 62 (+3)

[orottier]

Results from #70

  RESULTS:
  - # pass: 3856 (-2) 
  - # fail: 1226 (+10)
  - # type error issues: 63 (+1)
  - # files ignored: 3 (-5)

[orottier]

Results from #76 (webaudio-rs v0.42 mainly):

  RESULTS:
  - # pass: 4304 (+448)
  - # fail: 782 (-444)
  - # type error issues: 59 (-4)
  - # files ignored: 3 (=)

[b-ma]

Results from 4b2dc95

RESULTS:
  - # pass: 4394 (+90)
  - # fail: 712 (-80)
  - # type error issues: 57 (-2)

[orottier]

Results from other browsers.
Interesting to see they hit so many more assertions. Probably makes sense if there are worklet-node assertions for example. For more detail I will collect results per subdirectory (the-blabla-interface) and make another comparison.

Chrome 122.0.6261

./wpt run --log-mach=- --log-mach-level=info --headless --binary /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome chrome webaudio/

web-platform-test
~~~~~~~~~~~~~~~~~
Ran 8278 checks (8021 subtests, 257 tests)
Expected results: 8208
Unexpected results: 70
  test: 3 (2 error, 1 timeout)
  subtest: 67 (64 fail, 1 notrun, 1 precondition_failed, 1 timeout)


Firefox 123.0

./wpt run --log-mach=- --log-mach-level=info --headless --binary /Applications/Firefox\ Nightly.app/Contents/MacOS/firefox firefox webaudio/

web-platform-test
~~~~~~~~~~~~~~~~~
Ran 7662 checks (7405 subtests, 257 tests)
Expected results: 7097
Unexpected results: 565
  test: 10 (9 error, 1 timeout)
  subtest: 555 (526 fail, 24 notrun, 5 timeout)


Safari 17.3.1

./wpt run --log-mach=- --log-mach-level=info --headless --channel stable safari webaudio/

web-platform-test
~~~~~~~~~~~~~~~~~
Ran 8268 checks (8011 subtests, 257 tests)
Expected results: 8158
Unexpected results: 110
  test: 14 (5 error, 9 timeout)
  subtest: 96 (66 fail, 24 notrun, 6 timeout)

[orottier]

Results from #82 #85 and orottier/web-audio-api-rs#455 orottier/web-audio-api-rs#454 orottier/web-audio-api-rs#452 orottier/web-audio-api-rs#456

  RESULTS:
  - # pass: 4467 (+73)
  - # fail: 736 (+24)
  - # type error issues: 58 (+1)
  - # files ignored: 2 (-1)

[orottier]

I have made the initial quantitative comparison now per directory (per node)

Methodology:

• our score = # pass + # type error
• total = the reported number of subtests via the official wpt runner

I have filed an issue over at wpt to remove the empty dirs: web-platform-tests/wpt#44749

[b-ma]

Cool, quite interesting

[b-ma]

With #87, #88 and web-audio-api-rs#1.0.0-rc.1

  RESULTS:
  - # pass: 4772 (+305)
  - # fail: 1052 (+316)
  - # type error issues: 54 (-4)
  - # files ignored: 2 (Ø)

[orottier]

With rust AudioParam PRs merged orottier/web-audio-api-rs#459 orottier/web-audio-api-rs#460

  RESULTS:
  - # pass: 5009 (+237)
  - # fail: 659 (-393)
  - # type error issues: 54 (=)
  - # files ignored: 2 (=)

[b-ma]

w/ #90

RESULTS:
  - # pass: 5278
  - # fail: 591
  - # type error issues: 46

[b-ma]

w/ update rc.3 and #103

  RESULTS:
  - # pass: 6240
  - # fail: 1072
  - # type error issues: 24

[b-ma]

w/ #104

RESULTS:
  - # pass: 6739
  - # fail: 800
  - # type error issues: 24

[b-ma]

w/ #112

RESULTS:
  - # pass: 6927
  - # fail: 677
  - # type error issues: 4
> wpt duration: 1:50.123 (m:ss.mmm)

[b-ma]

v0.20.0

  RESULTS:
  - # pass: 7005
  - # fail: 618
  - # type error issues: 4
> wpt duration: 2:15.084 (m:ss.mmm)

[orottier]

The results for the AudioWorkletNode on my machine (with the latest fix for zero-output nodes) are

npm run wpt -- --filter the-audioworklet-interface

  RESULTS:
  - # pass: 259
  - # fail: 41
  - # type error issues: 0
> wpt duration: 2:57.474 (m:ss.mmm)

But they are plagued with messages Cannot read properties of undefined (reading 'getElementsByTagName'). Should we port any browser method to NodeJs again?

[b-ma]

/w #132

RESULTS:
  - # pass: 7316
  - # fail: 528
  - # type error issues: 3
> wpt duration: 3:00.313 (m:ss.mmm)