Tokenizer does not split text according to newly added input tokens

[jiongjiongli]

System Info

• transformers version: 4.47.1
• Platform: Linux-6.1.85+-x86_64-with-glibc2.35
• Python version: 3.10.12
• Huggingface_hub version: 0.27.0
• Safetensors version: 0.4.5
• Accelerate version: 1.2.1
• Accelerate config: not found
• PyTorch version (GPU?): 2.5.1+cu121 (False)
• Tensorflow version (GPU?): 2.17.1 (False)
• Flax version (CPU?/GPU?/TPU?): 0.8.5 (cpu)
• Jax version: 0.4.33
• JaxLib version: 0.4.33

Who can help?

@ArthurZucker and @itazap

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Repro Steps:

1. Initialize a tokenizer with use_fast=False
2. Add new tokens "red" and "e" to the tokenizer vocabulary using add_tokens()
3. Try to tokenize the word "read"

• Expected: The tokenizer should split "read" into ['r', 'e', 'ad'] since "e" is now a token
• Actual: The tokenizer keeps "read" as a single token, ignoring the newly added vocabulary

Repro Code:

from transformers import AutoTokenizer


def test(text, input_tokens):
    tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False)
    tokenizer.add_tokens(input_tokens)
    output_tokens = tokenizer.tokenize(text)
    print(f"Output tokens: {output_tokens}")


model_id = "google-bert/bert-base-cased"
test("read", ["red", "e"])

• Expected Output: Output tokens: ['r', 'e', 'ad']
• Actual Output: Output tokens: ['read']

Expected behavior

The tokenizer should split "read" into ['r', 'e', 'ad'] since "e" is now a token.