[Tokenizer] Fix slow and fast serialization

[ArthurZucker]

What does this PR do?

• sets the defaults for AddedToken instances where needed to match what is pushed to the hub
• sets the default for AddedToken to not strip left and right to match the fast tokenizers
• fixes the added_tokens.json file: a recent push made it save all the added tokens encoder, but it should only save the indexes greater than the vocab size for forward compatibility.
• fixes the list of additionnal_special_tokens that were added twice / overwritten
• fixes add_tokens : if the added tokens is a string we check if it's not already in the added vocab instead of always defaulting to strip left or right.
• fixes saving: the added_tokens_decoder should not add a "__type ":"AddedToken" field to the added tokens otherwise the previous versions of transformers will try to load them.

fixes #26732, fixes #26775, fixes #26773, fixes #26768, fixes #26859

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[LysandreJik]

Feel free to merge when ready as seen offline with you

[tongyx361]

I ran into the error below

Traceback (most recent call last):
  File ".../src/train_flash_attn_2.py", line 11, in <module>
    train()
  File ".../src/train.py", line 157, in train
    tokenizer = transformers.AutoTokenizer.from_pretrained(
  File ".../lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 751, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File ".../lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2017, in from_pretrained
    return cls._from_pretrained(
  File ".../lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2243, in _from_pretrained
    init_kwargs[key] = added_tokens_map.get(init_kwargs[key], init_kwargs[key])
TypeError: unhashable type: 'dict'

So I added some prints and get this intermediate values:

cls.SPECIAL_TOKENS_ATTRIBUTES: (list)['bos_token', 'eos_token', 'unk_token', 'sep_token', 'pad_token', 'cls_token', 'mask_token', 'additional_special_tokens']
added_tokens_map: (dict){'<unk>': AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), '<s>': AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), '</s>': AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True)}
init_kwargs: (dict){'add_bos_token': True, 'add_eos_token': False, 'bos_token': {'__type': 'AddedToken', 'content': '<s>', 'lstrip': False, 'normalized': True, 'rstrip': False, 'single_word': False}, 'clean_up_tokenization_spaces': False, 'eos_token': {'__type': 'AddedToken', 'content': '</s>', 'lstrip': False, 'normalized': True, 'rstrip': False, 'single_word': False}, 'legacy': None, 'model_max_length': 1024, 'pad_token': None, 'sp_model_kwargs': {}, 'unk_token': {'__type': 'AddedToken', 'content': '<unk>', 'lstrip': False, 'normalized': True, 'rstrip': False, 'single_word': False}, 'vocab_file': '.../tokenizer.model', 'tokenizer_file': '.../tokenizer.json', 'name_or_path': '...'}
key: (streos_token
init_kwargs[key]: (dict){'__type': 'AddedToken', 'content': '</s>', 'lstrip': False, 'normalized': True, 'rstrip': False, 'single_word': False}

According to the output, I made a fix, which seemed to work out:

# Passing AddedTokens and not strings to the class to prevent it from casting the string to a different AddedToken
for key in cls.SPECIAL_TOKENS_ATTRIBUTES & init_kwargs.keys():
    if added_tokens_map != {} and init_kwargs[key] is not None:
        if key != "additional_special_tokens":
            # >>> debug
            def print_info(name, obj):
                print(f"{name}: ({type(obj).__name__}){obj}")
            print_info("cls.SPECIAL_TOKENS_ATTRIBUTES", cls.SPECIAL_TOKENS_ATTRIBUTES)
            print_info("added_tokens_map", added_tokens_map)
            print_info("init_kwargs", init_kwargs)
            print_info("key", key)
            print_info("init_kwargs[key]", init_kwargs[key])
            # <<< debug
-            init_kwargs[key] = added_tokens_map.get(init_kwargs[key], init_kwargs[key])
+            init_kwargs[key] = added_tokens_map.get(key, init_kwargs[key]) # fix

[ArthurZucker]

Could you share a reproducer? Would help me a lot as well!

[tongyx361]

Could you share a reproducer? Would help me a lot as well!

Sorry that I'm too busy to do so right now 😭

But this only happened when I loaded the tokenizer of Llemma-7B.

I hope this description could help you reproduce the error.