Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
XLM-R tokenizer, return correct unk id for corrupted input
This should never happen, but we returned the incorrect unknown piece identifier in the worst-case fallback (where tokenization doesn't return any pieces).
- Loading branch information