Use Speech Recognition to Transcribe Oral Argument Audio

[mlissner]

We currently have about 7500 hours of oral argument audio without transcriptions. We need to go through these audio files and run a speech to text tool on them. This would have massive benefits:

• Alerts based on things said in a court!
• Transcription search
• Written transcriptions

The research in this area seems to be taking a few different paths from what I've gathered. The tech industry mostly needs this for when people are talking to Siri, so most of the research is making it better able to hear little phrases rather than complex transcriptions like we have.

The other fission is between cloud-based APIs and software that you can install. Cloud based APIs have the best quality, and tend to be fairly turnkey. OTOH, installable software can be tuned to the corpus we have (legal audio), and doesn't have API limits or costs associated with it.

The good news seems to be that unified APIs seem to be bubbling to the surface. For example, here's a Python library that lets you use:

• CMU Sphinx (an installable)
• Google Speech Recognition
• Wit.ia (some thing that Facebook apparently now owns)
• IBM Speech to Text (1000 hours/month free, I think)
• AT&T Speech to Text

Pretty solid number of choices in a single library. On top of these, there are a few other providers that only do speech recognition, and even YouTube (where @brianwc now works) does captions on videos. We've talked to a few of the speech-to-text startups, but none have had any interest in helping out a non-profit. Start-ups, am I right?

Anyway, there's clearly a lot to do here. An MVP might be to figure out the API limits and start pushing to the cloud using as many of the APIs as needed, though that probably brings a lot of complexity and variance in quality. Even using IBM's free tool, we could knock out our current collection in about eight or nine months. More comments on this over on hacker news too.

PS: I swear there used to be a bug for this, but I can't find it, so I'm loading this one with keywords like transcription, audio, oral arguments, transcribe, recognition...

[mlissner]

I'm told by a friend that Trint might be an option to look at. Looks more integrated than we probably want though.

[waldoj]

FWIW, I tested the quality of court audio transcription (using Virginia state court audio), and posted my conclusions here. Speechmatics offered the best bang for the buck.

[mlissner]

Really good to know where the quality is, @waldoj. Their prices are crazy though. Our 7500 hours of content at the price you mentioned (6¢/minute) comes to $27k. We'd need some sort of non-profit agreement...

[mlissner]

...or money.

[djeraseit]

I sent a request to Google Cloud Speech for beta access as it would probably be the most accurate with their natural language processing system. Unfortunately, each audio clip can be a maximum of 2 minutes long.

[djeraseit]

This project looks promising for long term viability:
https://github.com/pannous/tensorflow-speech-recognition

Google has open sourced the software for neural network and Facebook open sourced the hardware.

[mlissner]

@djeraseit We've got beta access to Google Cloud Speech and we're playing with it. It seems to work, but the quality is actually very low. Right now we're attempting to process as much audio as possible before Google makes it a pay service.

The max of 2 minutes was also lifted, btw.

[djeraseit]

IBM Watson has speech to text API. First 1,000 minutes per month are free.

http://www.ibm.com/smarterplanet/us/en/ibmwatson/developercloud/speech-to-text.html

[mlissner]

Yep, this is in my original ticket, above:

IBM Speech to Text (1000 hours/month free, I think)

[mlissner]

Big news here. We've signed a contract with Google, and they are giving us credits to transcribe all our existing oral argument audio recordings using their Speech to Text API. They will also be giving us credits going forward to do this on an ongoing basis.

We'll be announcing this properly once we've wrapped up this bug, but for now I want to get to work building this feature. I've outlined the process below, but could really use help getting it done, as I'm stretched quite thin:

• Add a new field for the transcripts in the database. Name it transcript_google or similar, so we know where the field's value came from.
• Add a new search field to the Audio search endpoint.
  • Make this work for oral argument alerts (should happen automatically)
• Figure out how to use the speech to text API to create celery tasks that send the correct audio files to the API.
  • Limit of 80 minutes per track.
  • Hard limit of 1.72 million seconds per day, and a rate limit of 25 seconds of uploaded audio per second in the day, which works out to 2.16 million seconds per day. Therefore, we'll hit the daily cap before the rate cap. We currently have 50,540,000 seconds of audio, so this will take about 30 days.
  • Files must be formatted as LINEAR16. Sample rates can be between 8000 Hz and 48000 Hz. with 16k recommended (but do not resample)
  • Poll for results (open question: how long are results available?)
  • Files must be in the Google cloud (cannot be uploaded as base64 or with a public URL)
  • Phrases can be provided, but only 500 per request, 10,000 chars per request, and 100 chars per phrase.
• Send all existing data to Google to generate transcripts.
• Update the scraper to transcribe any new content that comes in.
• Expose the transcripts on the oral argument pages (but probably hide it behind a button/warning: "Transcripts automatically generated by Google and will have errors or be difficult to read.").
• Write blog post, announce it, etc.

[waldoj]

Yay! Are you worried about the transcript quality? (Or is that question mooted by the fact that you can get_ a transcript with Google, so comparing it to a hypothetical more expensive system is an exercise in futility?)

[mlissner]

My concerns are mostly mooted by getting transcripts from Google in the first place. But also, the other pretty awesome part of this deal is what's in it for Google: We're giving them all of our audio as a zip (it's like 700GB or something), and they're going to use that for their machine learning, because they need big data sources with multiple speakers. So, I'm hopeful that they'll use our data to train their system, and boom, quality will be really good.

Anyway, even if that weren't true, this is an amazing offer. Everything I've heard is that they've got the best quality speech to text of anybody.

[mlissner]

Some big progress here that will be landing as soon as tests complete successfully:

1. Our database models are updated.
2. I've set this up to run as a celery "chain". This means that things will be processed by a series of async tasks:
   • The first task will re-encode the file in LINEAR16 format, then upload it to google storage.
   • The second task will request that google do speech to text on the uploaded file.
   • The third task will poll Google for results to the second task, with an exponential back off. It will begin looking for results after five minutes, then ten, twenty, fourty, etc. It will give up after waiting 320 minutes. (This API takes about as long to run as the uploaded files and our longest is about 240 minutes). Once the poll completes successfully, it will be saved to the DB, and from there to Solr.
   • A final task comes through and does cleanup (though this isn't strictly necessary, because Google is set to auto-delete all content after seven days).
3. I've updated Solr to have the transcript results in search results.

There's still more to do here, but this is the bulk of it. The missing pieces are:

• Add snippets to search results (these were terrible before, should be better now).
• Kick off a script to do all the old content (or just do it in a screen session). For this, it looks like we should wait for some backend upgrades from Google. In particular, they plan to soon have timestamps in the results, which they don't currently.
• Figure out how to handle API limits, if they happen (I don't think we'll hit these unless we further optimize the pipeline).
• Update the scraper.
• Expose the transcripts on the oral argument pages (I'll want to look at the quality some more before doing this though).
• Update the FAQ to mention transcripts.

[ghost]

Hi @mlissner, I just heard about this issue from @azeemba at the supreme court transcripts project. Would love to contribute. What's the current state of generating transcripts?

[mlissner]

Honestly...I'm not sure! I haven't looked at this in a while, but there's a lot of code for doing text-to-speech. I think where it wound up was that the quality wasn't good enough to read as a human, but it probably would be good enough for alerts and search. I think we still have credits with Google to convert stuff, so if you wanted to pick this up and run with it, that could be pretty great.

[mlissner]

FWIW, I just yanked a bunch of this code in 293b3b3. That's not to say we shouldn't do this, but I'm a code-deleting kick and I'm deleting any code that has lingered without use.

[ghost]

Hi @mlissner got it. I'm thinking that trying to get really good transcripts is beyond the scope of my NLP knowledge at this point, but would be great to come back to this later on as I work on that!

[mlissner]

Man oh man, I haven't updated this ticket in a while. This is now quite feasible and at high quality by using Whisper. This would be a great project for a volunteer to run with.

[waldoj]

I love it when enough time passes that a long-standing open issue goes from implausible to highly plausible.

[waldoj]

FWIW, if the files are stereo, you'd want to drop them to mono. And you might also experiment with reducing the sample rate. As low as 8 kHz may get equally-good results.

(I'm sorry if I'm just driving by and dropping in advice that's quite obvious to you!)

[flooie]

@waldoj

oh thats fantastic - Im not an audiophile or well informed on audio formats. But reducing to mono 8k would probably get everything under the magic 25mb threshold

[grossir]

On the OpenAI forums there are comments about doing that, and from the feedback, it seems to work

I take a 64k stereo mp3 and mash it with OPUS in an OGG container down to 12kbps mono, also using the speech optimizations. Command line is below:
ffmpeg -i audio.mp3 -vn -map_metadata -1 -ac 1 -c:a libopus -b:a 12k -application voip audio.ogg

I haven't tried it myself yet, but @flooie sent me a 5 hours - 20MB file that returned an InternalServerError from the API, repeteadly:
InternalServerError: Error code: 500 - {'error': {'message': 'The server had an error processing your request. Sorry about that! You can retry your request, or contact us through our help center at help.openai.com if you keep seeing this error. (Please include the request ID req_b3716aa50fcb212e6a813a807215a382 in your email.)', 'type': 'server_error', 'param': None, 'code': None}}

Maybe there was a problem with the re-encoding, maybe there is an undocumented audio length limit?

[grossir]

Some datapoints from testing a draft of the transcription command #4102

Using the case name as prompt helps getting names right.
For this example case "Tafolla v. S.C.D.A.O.
". Without case name:

Tofolla, I'm not sure of the pronunciation, versus Heilig. Thank you. So Mr. Bergstein, before we start, pronounce your client's last name. Tofolla. Tofolla. That's how I do it. OK, well, you know better than we do. All right, so you have 10 minutes, but you reserve three minutes for rebuttal. Correct. So you may proceed. OK, we have two reasonable accommodation periods relevant to this case. The first one involved plaintiff's office interaction with Joseph Carroll on January 7, 2014, when we argue the jury can find that defendants violated the ADA when Carroll ordered plaintiff to archive the closed files because the order contradicted the medical note that plaintiff's doctor prepared a couple days ago. Let me ask you about that, Mr. Tofolla, because as I was reading the briefs on this, it just seemed like this case- He's Bergstein. I'm sorry. Bergstein. Thank you. You got Tofolla in my head. This case seems to be a disagreement over what the doctor notes said. It seems to me that the

With case name:

or Tafolla, I'm not sure of the pronunciation, versus Heilig. So Mr. Bergstein, before we start, pronounce your client's last name. Tafolla. Tafolla, that's how I do it. Okay, well, you know better than we do. All right, so you have ten minutes, but you reserve three minutes for rebuttal. Correct. So you may proceed. Okay, we have two reasonable accommodation periods relevant to this case. The first one involved plaintiff's office interaction with Joseph Carroll on January 7, 2014, when we argue the jury can find that defendants violated the ADA when Carroll ordered plaintiff to archive the closed files because the order contradicted the medical note that plaintiff's doctor prepared a couple days earlier. Let me ask you about that, Mr. Tafolla, because as I was reading the briefs on this, it just seemed like this case- He's Bergstein. I'm sorry. Bergstein. You got Tafolla in my head. This case seems to be a disagreement over what the doctor notes said. It seems to me that they were wil

---

Response time is fast, in a direct linear relationship with the audio duration. As a caveat, they were sequential requests

---

I did some manual testing, I think around ~30 requests. Also, I tested the command by running it against 97 audio files. However, we got billed 319 API requests. I don't know the reason for the difference

[mlissner]

Also, I tested the command by running it against 97 audio files. However, we got billed 319 API requests. I don't know the reason for the difference

I went and looked. No idea either, but we should keep an eye on this and make sure we understand it before we run up a 3× larger bill than we expect.

[waldoj]

That's a great observation on priming the transcription with the case name! That would seem to indicate that it would be helpful to include any metadata in the prompt that might appear within the audio.

[mlissner]

Maybe a bunch of legalese, but I can't think of a good list? I guess we'll have to see what's bad and plug holes as we see them.

[waldoj]

I was only thinking in terms of the name of the judge, the names of the lawyers, that kind of thing, but that's a good point! If there were any specialized legal terms that transcription consistently got wrong, the prompt could prime for them.

[flooie]

What about using the final opinion text for priming the audio

[mlissner]

We won't have that when we get recordings in the future, but even at this point we don't have them linked (though we should). V2, perhaps!

[flooie]

You know what are posted with oral arguments are briefs

[mlissner]

Just to follow up here: We've generated transcripts for nearly every oral argument file in CL. We are doing some cleanup for:

• 868 audio files that didn't get processed the first time (status 0)
• 103 that failed (status 2)
• 1481 that hallucinated (status 3)
• 1893 that are > 25MB (status 4)
• 182 with missing files (status 5)

[mlissner]

OK, this is getting closer to done:

• In fix(audio.transcribe): add argument to retry different stt_status #4137, we tweaked things to reduce the hallucination problem by removing the case name as a prompt. The remaining "hallucinations" are probably better described as OpenAI doing its darnedest to make sense of horrible recordings. A human could do better, but some of these are very bad.

• In feat(audio): Add helper method to downsize audio records doctor#194, added support to doctor to downsample mp3s and convert them to ogg, so that most of our large files are small enough, and in feat(audio): Add logic to transcribe big files #4141, we added it to CourtListener.

A few remaining things:

1. We still have 154 files, (0.16%) that are too big to process.

   There seems to be a two hour limit in the OpenAI API, so we either need to either:

   • cut some of these into smaller pieces that we then merge back together;
   • only do the first two hours of these and then tell the user as such;
   • we need to try speeding them up to see if we still get good enough quality; or
   • just throw an error for users that says, "Sorry, this file was too big to transcribe."

   I think I lean towards trying a sped-up version, then trying just the first two hours, then throwing an error if none of that works.

2. We haven't set up a process to do this for every new item we scrape. It cost about $21k to do 90k files, so that's 23¢ each. I think we can commit to that — unless we go and get thousands of additional files!

3. We need a new issue to discuss how we'll add these files to the UI (and we need to do so).

4. We need to do a blog post, etc.

What else?