Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speeding up mapping with HuggingFace datasets #82

Open
MaveriQ opened this issue Jun 17, 2021 · 5 comments
Open

Speeding up mapping with HuggingFace datasets #82

MaveriQ opened this issue Jun 17, 2021 · 5 comments

Comments

@MaveriQ
Copy link

MaveriQ commented Jun 17, 2021

Hi. I am trying to convert corpora from HF to their IPA form with the following snippet. But I am getting really slow speeds.. only a couple of examples per second. Do you know how it can be sped up? Thanks

import epitran
from datasets import load_dataset

bookscorpus = load_dataset('bookcorpus',split='train')
epi = epitran.Epitran('eng-Latn')

def transliterate(x):
    return {'trans': epi.transliterate(x['text'])}

tokenized = bookscorpus.map(lambda x: transliterate(x),num_proc=32)
@dmort27
Copy link
Owner

dmort27 commented Nov 10, 2021

Hi @MaveriQ , I seem to have missed this message when you originally sent it. It is true that the Python implementation of Epitran is very slow. If you can take advantage of concurrency, you can do better. I am in the process of rewriting parts of Epitran in Rust, which will increase performance—based on initial tests—by about 100 times.

@dmort27 dmort27 closed this as completed Nov 10, 2021
@dmort27 dmort27 reopened this Nov 10, 2021
@dmort27
Copy link
Owner

dmort27 commented Nov 10, 2021

However, for English this will not help as much since English support is provided by Flite (written in C) and the only python part converts the ARPAbet representation from Flite to IPA.

@MaveriQ
Copy link
Author

MaveriQ commented Nov 12, 2021

Hi. If by concurrency you meant multiprocessing, I already tried that, but it's still pretty slow. Can you recommend anything else for English? Thanks

@dmort27
Copy link
Owner

dmort27 commented Nov 12, 2021

As I look at this issue, the real problem is that Epitran spawns a shell every time it calls lex_lookup to convert a word to IPA. This is expensive (although lex_lookup itself is quite efficient). The solutions would be to:

  • Create Python bindings to the Flite libraries, so the relevant functions could be called directly (rather than via the shell)
  • Create a method to epitran.flite.Flite that passes inputs to lex_lookup by the batch.

The second solution would be easier, but ultimately less satisfying.

@juice500ml
Copy link
Contributor

FYI for future reference, extremely dirty quickfix would be:

import re

words = [...]  # words to be transliterated
with open("eng_words.sh", "w") as f:
    f.write("\n".join([f"lex_lookup {w} | head -n 1" for w in words]))

!bash eng_words.sh > eng_lex.txt

lexs = open("eng_lex.txt").readlines()
arpa_to_ipa = {'ey': 'ej', 'ae': 'æ', 'iy': 'i', 'eh': 'ɛ', 'ay': 'aj', 'ih': 'ɪ', 'ow': 'ow', 'aa': 'ɑ', 'ao': 'ɔ', 'aw': 'aw', 'oy': 'oj', 'ah': 'ʌ', 'ax': 'ə', 'uw': 'u', 'uh': 'ʊ', 'er': 'ɹ̩', 'b': 'b', 'ch': 't͡ʃ', 'd': 'd', 'dx': 'ɾ', 'f': 'f', 'g': 'ɡ', 'hh': 'h', 'jh': 'd͡ʒ', 'k': 'k', 'l': 'l', 'em': 'm̩', 'm': 'm', 'en': 'n̩', 'n': 'n', 'ng': 'ŋ', 'p': 'p', 'q': 'ʔ', 'r': 'ɹ', 's': 's', 'sh': 'ʃ', 't': 't', 'dh': 'ð', 'th': 'θ', 'v': 'v', 'w': 'w', 'y': 'j', 'z': 'z', 'zh': 'ʒ'}

ipas = []
for lex in lexs:
    lex = lex.strip()[1:-1].split()
    lex = map(lambda d: re.sub(r'\d', '', d), lex)
    ipa = map(lambda d: arpa_to_ipa[d], lex)
    ipas.append("".join(list(ipa)))

word_to_ipa = dict(zip(words, ipas))  # now this dict has key: word, value: transliteration result

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants