Skip to content

Commit

Permalink
update doc
Browse files Browse the repository at this point in the history
  • Loading branch information
ArthurZucker committed Aug 5, 2024
1 parent a010f6b commit 7da7397
Showing 1 changed file with 1 addition and 2 deletions.
3 changes: 1 addition & 2 deletions bindings/python/src/pre_tokenizers.rs
Original file line number Diff line number Diff line change
Expand Up @@ -324,8 +324,7 @@ impl PyWhitespaceSplit {
///
/// Args:
/// pattern (:obj:`str` or :class:`~tokenizers.Regex`):
/// A pattern used to split the string. Usually a string or a a regex built with `tokenizers.Regex`
///
/// A pattern used to split the string. Usually a string or a a regex built with `tokenizers.Regex`. If you want to use a regex pattern, it has to be wrapped around a `tokenizer.Regex`, otherwise we consider is as a string pattern.
/// behavior (:class:`~tokenizers.SplitDelimiterBehavior`):
/// The behavior to use when splitting.
/// Choices: "removed", "isolated", "merged_with_previous", "merged_with_next",
Expand Down

0 comments on commit 7da7397

Please sign in to comment.