Skip to content

Commit

Permalink
[MINOR:TYPO] Fix docstrings (#1653)
Browse files Browse the repository at this point in the history
* [MINOR:TYPO] Update pre_tokenizers.rs

* [MINOR:TYPO] Update __init__.pyi
  • Loading branch information
cakiki authored Nov 5, 2024
1 parent 5e223ce commit 57884eb
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -421,7 +421,7 @@ class Split(PreTokenizer):
Args:
pattern (:obj:`str` or :class:`~tokenizers.Regex`):
A pattern used to split the string. Usually a string or a a regex built with `tokenizers.Regex`.
A pattern used to split the string. Usually a string or a regex built with `tokenizers.Regex`.
If you want to use a regex pattern, it has to be wrapped around a `tokenizer.Regex`,
otherwise we consider is as a string pattern. For example `pattern="|"`
means you want to split on `|` (imagine a csv file for example), while
Expand Down
2 changes: 1 addition & 1 deletion bindings/python/src/pre_tokenizers.rs
Original file line number Diff line number Diff line change
Expand Up @@ -334,7 +334,7 @@ impl PyWhitespaceSplit {
///
/// Args:
/// pattern (:obj:`str` or :class:`~tokenizers.Regex`):
/// A pattern used to split the string. Usually a string or a a regex built with `tokenizers.Regex`.
/// A pattern used to split the string. Usually a string or a regex built with `tokenizers.Regex`.
/// If you want to use a regex pattern, it has to be wrapped around a `tokenizer.Regex`,
/// otherwise we consider is as a string pattern. For example `pattern="|"`
/// means you want to split on `|` (imagine a csv file for example), while
Expand Down

0 comments on commit 57884eb

Please sign in to comment.