Efficient Replace normalizer

[rlrs]

The existing Replace normalizer, used for example in the Llama and Mistral tokenizers, is implemented very inefficiently.
This results in normalization taking orders of magnitude longer than it should, making it very time consuming to tokenize long sequences. I've seen a few issues that probably refer to this, for example huggingface/transformers#25873.

This PR replaces the existing implementation -- which seems to scale quadratically with sequence length and number of matches -- with an implementation that scales linearly, while (hopefully) retaining the exact same semantics.
In my benchmarks with real long-sequence data, tokenizing with the Llama tokenizer is more than two orders of magnitude faster.

[ArthurZucker]

I'll have a look, sounds really interesting !

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

(I am planning on remove the normalizer of Llama and mistral families in favor of pre_tokenizer, but will still check this out!)

[rlrs]

(I am planning on remove the normalizer of Llama and mistral families in favor of pre_tokenizer, but will still check this out!)

That's fair, until then this fix will let people tokenize as long sequences as they want.
The code should work with any non-overlapping matches (I'm not sure overlapping matches were ever supported), so it might help with other tokenizers as well, but I don't know if any others use the feature.

[rlrs]

I see this got closed from inactivity. Do you need anything from me in order to merge this? Profiling, documentation?

[ArthurZucker]

Sorry I'll take some time to review!

[ArthurZucker]

Will be my priority review!

[Narsil]

LGTM. Sorry I've just seen this , seems quite important change indeed.

[Narsil]

I just got pinged internally on this.

Before

After

from matplotlib import pyplot as plt
import time
from tqdm import tqdm
from tokenizers import Tokenizer

import numpy as np
lens = np.arange(0, 100000, 100)

with open("data/big.txt") as f:
    TEXT = f.read()

times_fast = []
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
for ll in tqdm(lens):
    text = TEXT[:ll]
    start = time.perf_counter()
    tokenizer.encode(text)
    times_fast.append(time.perf_counter() - start)



timesgpt2 = []
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2", use_fast=True)
for ll in tqdm(lens):
    text = TEXT[:ll]
    start = time.perf_counter()
    tokenizer.encode(text)
    timesgpt2.append(time.perf_counter() - start)

times = []
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1", use_fast=False)
for ll in tqdm(lens):
    text = TEXT[:ll]
    start = time.perf_counter()
    tokenizer.encode(text)
    times.append(time.perf_counter() - start)

plt.plot(lens, times_fast)
plt.plot(lens, timesgpt2)
plt.plot(lens, times)
plt.legend(["mistral (tokenizers)", "gpt2 (tokenizers)", "mistral(spm)"])
plt.xlabel("Length in chars")
plt.ylabel("tokenization time (seconds)")
plt.show()

[ArthurZucker]

Thanks a lot @rlrs ! Really sorry about the delay and shoutout to you for this clean piece of work!