GemmaTokenizerFast word_ids() returns only zeros

[Alienmaster]

System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

• transformers version: 4.41.2
• Platform: Linux-5.15.0-86-generic-x86_64-with-glibc2.31
• Python version: 3.10.13
• Huggingface_hub version: 0.23.1
• Safetensors version: 0.4.2
• Accelerate version: 0.28.0
• Accelerate config: not found
• PyTorch version (GPU?): 2.2.1+cu121 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: no

Who can help?

@ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

The method word_ids() does only return a list of zeros instead of the correct word_ids.

sentence = "I love my cat"
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("google/Gemma-7b") #-version a0eac5b
encoded = tokenizer(sentence, return_tensors="pt")
print(encoded.word_ids())
# [None, 0, 0, 0, 0]

I tried several variations of configurations stated in the linked issues in #28881 , but for Gemma it doesn't change the result. The llama3 tokenizer outputs the correct values with this code.

Expected behavior

The output of word_ids should look like
[None, 0, 1, 2, 3]

[ArthurZucker]

Hey! Will have a look thanks for reporting

[ArthurZucker]

It seems that we need this:

tokenizer._tokenizer.pre_tokenizer = Sequence([Split("▁","merged_with_next")])
encoded = tokenizer(sentence, return_tensors="pt")
print(encoded.word_ids())
[None, 0, 1, 2, 3]

[ArthurZucker]

#32191 should fix this !

[xenova]

Not included in #32191 since the proposed fix breaks encoding. Will need to do in a follow-up PR :)

[ArthurZucker]

Small update:

pre_tokenzier = pre_tokenizers.Split(Regex('(?<!▁)▁'), "merged_with_next")
pre_tokenzier.pre_tokenize_str(sentence.replace(" ", "▁"))
Out[41]: [('I', (0, 1)), ('▁love', (1, 6)), ('▁my', (6, 9)), ('▁cat', (9, 13))]
tokenizer(sentence).tokens()
Out[48]: ['<bos>', 'I', '▁love', '▁my', '▁cat']
tokenizer(sentence).word_ids()
Out[49]: [None, 0, 1, 2, 3]

Gives somewhat acceptable results,

[ArthurZucker]

But I don't recommend word ids, separations are "brittle" and this changes the output of the tokenization

[Alienmaster]

Is there a better way to link the tokens to words than word_ids?

[ArthurZucker]

Offsets!

[ArthurZucker]

The offset mapping give you the exact place where the token corresponds to in the original string

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.