Prevent attempts at nesting character sets in regex filter #389
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Not sure how I didn't catch this when testing a couple years ago. Basically, nesting character sets is impossible in Python's regex, which is what my filter code tried to do because it wasn't checking for IN tokens. This adds that check and performs confusable insertion in place for those sets. Also cleans up some of the code in that area.
Also missed this. "a-z" in [a-z] is tokenized as (RANGE, (97, 122)). This commit adds support for those tokens.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Not sure how I didn't catch this when testing the initial regex feature a couple years ago. Basically, nesting character sets is impossible in Python's regex, which is what my filter code tried to do because it wasn't checking for IN tokens, so those confusable glyphs would be discarded by the regex compiler (thankfully it doesn't outright throw an error). This PR adds that check and performs confusable insertion in place for those sets. Also cleans up some of the code in that area.
Example of the issue:
h[ea]lpwould effectively be converted into this, which is invalid regex:instead of this, which is valid and desired:
(** only added for emphasis)