Skip to content

Commit

Permalink
Handle Ġ as a space replacement too
Browse files Browse the repository at this point in the history
  • Loading branch information
Rocketknight1 committed Feb 14, 2024
1 parent bec98bf commit 254fa2d
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/transformers/generation/stopping_criteria.py
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ def get_matching_positions(vocab, stop_strings):
matching_positions = []
possible_end_lengths = []
for i in range(1 - len(token), len(stop_string)):
tok = token[::-1].replace("▁", " ")
tok = token[::-1].replace("▁", " ").replace("Ġ", " ")
stop = stop_string[::-1]
if i < 0:
tok = tok[-i:]
Expand Down

0 comments on commit 254fa2d

Please sign in to comment.