You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In my use-case I wish to calculate the number of tokens for a particular string of text without padding or truncation. However, I find that the tokenizer.json may by default have set to true for both. Is there a way to programmatically override it before encoding a text using this library?
Thanks
The text was updated successfully, but these errors were encountered:
Hello:
In my use-case I wish to calculate the number of tokens for a particular string of text without padding or truncation. However, I find that the tokenizer.json may by default have set to true for both. Is there a way to programmatically override it before encoding a text using this library?
Thanks
The text was updated successfully, but these errors were encountered: