Skip to content

v0.3.0

Compare
Choose a tag to compare
@umarbutler umarbutler released this 18 May 12:06
· 36 commits to main since this release

Added

  • Introduced the chunkerify() function, which constructs a chunker from a tokenizer or token counter that can be reused and can also chunk multiple texts in a single call. The resulting chunker speeds up chunking by 40.4% thanks, in large part, to a token counter that avoid having to count the number of tokens in a text when the number of characters in the text exceed a certain threshold, courtesy of @R0bk (#3) (337a186).