Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gemma's Tokenizer fails to split on spaces #30416

Closed
4 tasks
vasqu opened this issue Apr 23, 2024 · 0 comments · Fixed by #28881
Closed
4 tasks

Gemma's Tokenizer fails to split on spaces #30416

vasqu opened this issue Apr 23, 2024 · 0 comments · Fixed by #28881
Labels
Core: Tokenization Internals of the library; Tokenization.

Comments

@vasqu
Copy link
Contributor

vasqu commented Apr 23, 2024

Preface

This is related to #29617, similar issue as described in there so this is kind of a placeholder. Should be fixed by #28881.

Who can help?

@ArthurZucker

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('google/gemma-7b', use_auth=True)
sentence = 'Sampel to demonstrate the issue'

input_encoding = tokenizer(sentence, add_special_tokens=True)
tokens = input_encoding.tokens()
word_ids = input_encoding.word_ids()

print(f'Tokens produced: {tokens}\nReferenced word ids: {word_ids}')

Expected behavior

Correct split(s) assigned.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Core: Tokenization Internals of the library; Tokenization.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants