Skip to content

Commit

Permalink
🚨🚨🚨 Limit backtracking in Nougat regexp (#35264)
Browse files Browse the repository at this point in the history
* Limit backtracking in regexp

* Update

* [run-slow] nougat

* Update
  • Loading branch information
qubvel authored Dec 17, 2024
1 parent d29a06e commit deac971
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/transformers/models/nougat/tokenization_nougat_fast.py
Original file line number Diff line number Diff line change
Expand Up @@ -514,7 +514,7 @@ def post_process_single(self, generation: str, fix_markdown: bool = True) -> str
generation = generation.replace("\n* [leftmargin=*]\n", "\n")
# Remove lines with markdown headings starting with #, with numerals,
# and possibly roman numerals with trailing spaces and newlines
generation = re.sub(r"^#+ (?:\.?(?:\d|[ixv])+)*\s*(?:$|\n\s*)", "", generation, flags=re.M)
generation = re.sub(r"^#+ (?:[\d+\.]+|[ixv\.]+)?\s*(?:$|\n\s*)", "", generation, flags=re.M)
# most likely hallucinated titles
lines = generation.split("\n")
if lines[-1].startswith("#") and lines[-1].lstrip("#").startswith(" ") and len(lines) > 1:
Expand Down

0 comments on commit deac971

Please sign in to comment.