Tiny improvement

[Narsil]

Rational is as follows, before we used to hash the sequence, fail to find in cache, then lookup in vocab, hit and return.
This ignore_merges doesn't insert in cache (it's already in the vocab so no need to duplicate the data), we can put it beforehand limiting the amount of cache reads.

Before:

==============
num_threads: 8, data size: 24.04 MB, documents: 10000 Avg Length: 1659
tiktoken 	61.73 MB  / s
huggingface 	23.32 MB / s
==============
num_threads: 8, data size: 1.11 MB, documents: 10000 Avg Length: 116
tiktoken 	6.65 MB  / s
huggingface 	20.20 MB / s

After:

==============
num_threads: 8, data size: 24.04 MB, documents: 10000 Avg Length: 1659
tiktoken 	59.51 MB  / s
huggingface 	25.36 MB / s
==============
num_threads: 8, data size: 1.11 MB, documents: 10000 Avg Length: 116
tiktoken 	7.48 MB  / s
huggingface 	20.93 MB / s

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

long is non english only, good that you added the average length as well

[Narsil]

Yes but the principal factor affecting speed seems to be document length (I tried user arabic and had similar results)