Skip to content

Commit

Permalink
Merge branch 'main' into fix-decode
Browse files Browse the repository at this point in the history
  • Loading branch information
ArthurZucker committed Jul 12, 2024
2 parents 1bd8d96 + f2a44dc commit bf8f664
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion tokenizers/src/tokenizer/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -851,7 +851,8 @@ where
.iter()
.filter_map(|id| {
self.added_vocabulary
.id_to_token(*id, &self.model)
.simple_id_to_token(*id)
.or_else(|| self.model.id_to_token(*id))
.filter(|token| {
!skip_special_tokens || !self.added_vocabulary.is_special_token(token)
})
Expand Down

0 comments on commit bf8f664

Please sign in to comment.