Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Better Intelligibility metric for CAD2 Task1 #419

Merged
merged 44 commits into from
Oct 15, 2024
Merged

Better Intelligibility metric for CAD2 Task1 #419

merged 44 commits into from
Oct 15, 2024

Conversation

groadabike
Copy link
Contributor

In CAD2 task1, the original intelligibility metric was using whisper for transcription and jiwer for computing correctness.
However, jiwer was used without any normalisation resulting in lower score.
For example, capitalised vs non-capitalised words, punctuations not excluded.

The changes are:

Replace jiwer by alt-eval which performs several normalisation before calling jiwer.
Add verbose option True or False for MSGB HL model to reduce unnecessary prints. Warnings will still be printed regardless the verbose option.

pre-commit-ci bot and others added 30 commits July 29, 2024 23:08
@pre-commit-ci
[pre-commit.ci] pre-commit-autoupdate
9bb51c8 
updates:
- [github.com/asottile/pyupgrade: v3.16.0 → v3.17.0](asottile/pyupgrade@v3.16.0...v3.17.0)
- [github.com/pre-commit/mirrors-mypy: v1.10.1 → v1.11.0](pre-commit/mirrors-mypy@v1.10.1...v1.11.0)
- [github.com/astral-sh/ruff-pre-commit: v0.5.1 → v0.5.5](astral-sh/ruff-pre-commit@v0.5.1...v0.5.5)
- [github.com/pycqa/pylint: v3.2.5 → v3.2.6](pylint-dev/pylint@v3.2.5...v3.2.6)
@groadabike
Update README.md
2cc4c09 
Add CAD2 to the README
@groadabike
Merge pull request #400 from claritychallenge/pre-commit-ci-update-co…
d5d09a1 
…nfig

[pre-commit.ci] pre-commit-autoupdate
@pre-commit-ci
[pre-commit.ci] pre-commit-autoupdate
8575c4b 
updates:
- [github.com/psf/black: 24.4.2 → 24.8.0](psf/black@24.4.2...24.8.0)
- [github.com/pycqa/flake8.git: 7.1.0 → 7.1.1](https://github.com/pycqa/flake8.git/compare/7.1.0...7.1.1)
- [github.com/nbQA-dev/nbQA: 1.8.5 → 1.8.7](nbQA-dev/nbQA@1.8.5...1.8.7)
- [github.com/pre-commit/mirrors-mypy: v1.11.0 → v1.11.1](pre-commit/mirrors-mypy@v1.11.0...v1.11.1)
- [github.com/astral-sh/ruff-pre-commit: v0.5.5 → v0.5.7](astral-sh/ruff-pre-commit@v0.5.5...v0.5.7)
@groadabike
Merge pull request #403 from claritychallenge/pre-commit-ci-update-co…
9fcdf8b 
…nfig

[pre-commit.ci] pre-commit-autoupdate
@groadabike
Add explicit hydra.job.chdir=True needed for version 1.2
76e732b 
Signed-off-by: Gerardo Roa <[email protected]>
@groadabike
add version_base=None in @hydra.main(), needed for version=1.2
d0fa83d 
Signed-off-by: Gerardo Roa <[email protected]>
@pre-commit-ci
[pre-commit.ci] Fixing issues with pre-commit
a4cfc57
@groadabike
Place imports at top of cell
ddb3b22 
Signed-off-by: Gerardo Roa <[email protected]>
@jonbarker68
Merge pull request #408 from claritychallenge/407-pre-commit-raises-s…
b383605 
…everal-errors-in-jupyter-notebooks

Place imports at top of cell
@jonbarker68
Merge pull request #406 from claritychallenge/grd-hydra-set-version-b…
c191b4a 
…ase-and-chdir

Hydra explicit params
@jonbarker68
Merge pull request #402 from claritychallenge/groadabike-patch-1
783f34f 
Update README.md
@pre-commit-ci
[pre-commit.ci] pre-commit-autoupdate
1f3116b 
updates:
- [github.com/DavidAnson/markdownlint-cli2: v0.13.0 → v0.14.0](DavidAnson/markdownlint-cli2@v0.13.0...v0.14.0)
- [github.com/pre-commit/mirrors-mypy: v1.11.1 → v1.11.2](pre-commit/mirrors-mypy@v1.11.1...v1.11.2)
- [github.com/astral-sh/ruff-pre-commit: v0.5.7 → v0.6.4](astral-sh/ruff-pre-commit@v0.5.7...v0.6.4)
- [github.com/pycqa/pylint: v3.2.6 → v3.2.7](pylint-dev/pylint@v3.2.6...v3.2.7)
@groadabike
sample rate in config
11c7769
@groadabike
correct use of whisper
27d14f2 
Signed-off-by: Gerardo Roa <[email protected]>
@groadabike
config
73ea121 
Signed-off-by: Gerardo Roa <[email protected]>
@groadabike
fix ruff reported errors
7e9a5bd 
Signed-off-by: Gerardo Roa <[email protected]>
@pre-commit-ci
[pre-commit.ci] Fixing issues with pre-commit
a07895b
@jonbarker68
Merge pull request #412 from claritychallenge/411-pre-commit-errors-w…
c8d4a22 
…ith-notebooks

New errors in Jupyter Notebook
@jonbarker68
Merge remote-tracking branch 'origin/main' into pre-commit-ci-update-…
2be5175 
…config
@jonbarker68
Merge pull request #404 from claritychallenge/pre-commit-ci-update-co…
dade982 
…nfig

[pre-commit.ci] pre-commit-autoupdate
@jonbarker68
Increment min support python to 3.9
c46ad4a
@jonbarker68
Reduced sensitivity of a small number of tests that were failing on t…
c008cf4 
…he mac
@jonbarker68
Fixed to use new scipy window module
6c124a1
@jonbarker68
Updated min versions in project dependencies
8b5d63c
@jonbarker68
Fixed a brittle test that was broken by numpy upgrade
d0038b4
@jonbarker68
second attempt to fix a broken test
666bb10
@jonbarker68
Merge pull request #410 from claritychallenge/cad2-fix-in-main
a526a3a 
Cad2 fix in main
@pre-commit-ci
[pre-commit.ci] pre-commit-autoupdate
36cc73e 
updates:
- [github.com/astral-sh/ruff-pre-commit: v0.6.4 → v0.6.5](astral-sh/ruff-pre-commit@v0.6.4...v0.6.5)
@groadabike
Merge pull request #413 from claritychallenge/jpb/support-for-python-312
649c76c 
Jpb/support for python 312
groadabike and others added 13 commits September 20, 2024 10:47
@groadabike
Update run_tests.yml
1421a8f 
change version of test
@groadabike
Merge pull request #414 from claritychallenge/pre-commit-ci-update-co…
8b2be0d 
…nfig

[pre-commit.ci] pre-commit-autoupdate
@pre-commit-ci
[pre-commit.ci] pre-commit-autoupdate
6b66e86 
updates:
- [github.com/pre-commit/pre-commit-hooks: v4.6.0 → v5.0.0](pre-commit/pre-commit-hooks@v4.6.0...v5.0.0)
- [github.com/psf/black: 24.8.0 → 24.10.0](psf/black@24.8.0...24.10.0)
- [github.com/astral-sh/ruff-pre-commit: v0.6.5 → v0.6.9](astral-sh/ruff-pre-commit@v0.6.5...v0.6.9)
- [github.com/pycqa/pylint: v3.2.7 → v3.3.1](pylint-dev/pylint@v3.2.7...v3.3.1)
@groadabike
Merge pull request #415 from claritychallenge/pre-commit-ci-update-co…
745fef9 
…nfig

[pre-commit.ci] pre-commit-autoupdate
@pre-commit-ci
[pre-commit.ci] pre-commit-autoupdate
46af3a4 
updates:
- [github.com/asottile/pyupgrade: v3.17.0 → v3.18.0](asottile/pyupgrade@v3.17.0...v3.18.0)
@groadabike
Merge pull request #417 from claritychallenge/pre-commit-ci-update-co…
12f0a5f 
…nfig

[pre-commit.ci] pre-commit-autoupdate
@groadabike
add alt_eval into requirements
e019ef9 
Signed-off-by: Gerardo Roa <[email protected]>
@groadabike
replace jiwer for alt-eval
f5543bc 
Signed-off-by: Gerardo Roa <[email protected]>
@groadabike
add verbose into msgb
820f03c 
Signed-off-by: Gerardo Roa <[email protected]>
@groadabike
correct discrepancy in branch
b7cadfb 
Signed-off-by: Gerardo Roa <[email protected]>
@groadabike
correct discrepancy
adf64af 
Signed-off-by: Gerardo Roa <[email protected]>
@groadabike
Merge branch 'main' into alt-eval-cad2-task1
13978a3
@pre-commit-ci
[pre-commit.ci] Fixing issues with pre-commit
9f4df5b
@groadabike groadabike marked this pull request as ready for review October 15, 2024 14:30
@groadabike groadabike marked this pull request as draft October 15, 2024 14:31
@groadabike groadabike changed the base branch from main to v0.6 October 15, 2024 14:31
@groadabike groadabike marked this pull request as ready for review October 15, 2024 14:36
@groadabike groadabike merged commit 85ed818 into v0.6 Oct 15, 2024
1 check was pending
@sgraetzer
Copy link
Contributor

In CAD2 task1, the original intelligibility metric was using whisper for transcription and jiwer for computing correctness. However, jiwer was used without any normalisation resulting in lower score. For example, capitalised vs non-capitalised words, punctuations not excluded.

The changes are:

Replace jiwer by alt-eval which performs several normalisation before calling jiwer. Add verbose option True or False for MSGB HL model to reduce unnecessary prints. Warnings will still be printed regardless the verbose option.

Excellent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@groadabike @sgraetzer @jonbarker68