Training a Tokenizer from large texts read from memory

[nlpravi]

I have around 800M files where the text field is one of the fields in the file. I would like to train a new tokenizer only on the text field. I cannot extract the text and write to new files because of the huge volume of files. Is there anyway to train a new tokenizer on this data by reading only the text fields from each of these files and passing them to the training process?

Thanks,
Ravi.

[Narsil]

Sorry no, not at the moment. How much does you 800M files represent of memory ? How much of that will your "text"represent ?

Training algorithms need to have the whole dataset readable to be able to learn (the preprocessing helps not having the whole array in memory if it's possible to do) so you would have to fit the whole thing in memory anyway. If it fits in memory, how come it can't fit on disk ? Usually disk is cheaper than RAM, no ? Just trying to understand your use case so we might design training API better in the future.

[nlpravi]

Thanks Narsil. The 800M represent around 800G on disk. I'm not sure about the memory. Since there is no way to load them in memory directly, We can definitely write the files to disk and train a tokenizer.

Thanks again for your quick response.

[Narsil]

Also quick tip, usually you don't need to train the tokenizer on the whole huge dataset.

1Go - 10Go should be more than enough to get good heuristics for your tokenizer. Going to the full dataset will only yield very marginally better results.
Better to do more runs and choose carefully the normalizers,pre_tokenizers and so on..

[nlpravi]

Thanks for the tip Narsil. That would really help.

[n1t0]

Duplicate of #198