fix(deps): update rust crate tokenizers to 0.21.0

[renovate]

This PR contains the following updates:

Package	Type	Update	Change
tokenizers	dependencies	minor	0.20.0 -> 0.21.0

---

Release Notes

huggingface/tokenizers (tokenizers)

v0.21.0

Compare Source

Release v0.20.4 v0.21.0

• More cache options. by @​Narsil in https://github.com/huggingface/tokenizers/pull/1675
• Disable caching for long strings. by @​Narsil in https://github.com/huggingface/tokenizers/pull/1676
• Testing ABI3 wheels to reduce number of wheels by @​Narsil in https://github.com/huggingface/tokenizers/pull/1674
• Adding an API for decode streaming. by @​Narsil in https://github.com/huggingface/tokenizers/pull/1677
• Decode stream python by @​Narsil in https://github.com/huggingface/tokenizers/pull/1678
• Fix encode_batch and encode_batch_fast to accept ndarrays again by @​diliop in https://github.com/huggingface/tokenizers/pull/1679

We also no longer support python 3.7 or 3.8 (similar to transformers) as they are deprecated.

Full Changelog: huggingface/tokenizers@v0.20.3...v0.21.0

v0.20.4

Compare Source

v0.20.3

Compare Source

What's Changed

There was a breaking change in 0.20.3 for tuple inputs of encode_batch!

• fix pylist by @​ArthurZucker in https://github.com/huggingface/tokenizers/pull/1673
• [MINOR:TYPO] Fix docstrings by @​cakiki in https://github.com/huggingface/tokenizers/pull/1653

New Contributors

• @​cakiki made their first contribution in https://github.com/huggingface/tokenizers/pull/1653

Full Changelog: huggingface/tokenizers@v0.20.2...v0.20.3

v0.20.2

Compare Source

Release v0.20.2

Thanks a MILE to @​diliop we now have support for python 3.13! 🥳

What's Changed

• Bump cookie and express in /tokenizers/examples/unstable_wasm/www by @​dependabot in https://github.com/huggingface/tokenizers/pull/1648
• Fix off-by-one error in tokenizer::normalizer::Range::len by @​rlanday in https://github.com/huggingface/tokenizers/pull/1638
• Arg name correction: auth_token -> token by @​rravenel in https://github.com/huggingface/tokenizers/pull/1621
• Unsound call of set_var by @​sftse in https://github.com/huggingface/tokenizers/pull/1664
• Add safety comments by @​Manishearth in https://github.com/huggingface/tokenizers/pull/1651
• Bump actions/checkout to v4 by @​tinyboxvk in https://github.com/huggingface/tokenizers/pull/1667
• PyO3 0.22 by @​diliop in https://github.com/huggingface/tokenizers/pull/1665
• Bump actions versions by @​tinyboxvk in https://github.com/huggingface/tokenizers/pull/1669

New Contributors

• @​rlanday made their first contribution in https://github.com/huggingface/tokenizers/pull/1638
• @​rravenel made their first contribution in https://github.com/huggingface/tokenizers/pull/1621
• @​sftse made their first contribution in https://github.com/huggingface/tokenizers/pull/1664
• @​Manishearth made their first contribution in https://github.com/huggingface/tokenizers/pull/1651
• @​tinyboxvk made their first contribution in https://github.com/huggingface/tokenizers/pull/1667
• @​diliop made their first contribution in https://github.com/huggingface/tokenizers/pull/1665

Full Changelog: huggingface/tokenizers@v0.20.1...v0.20.2

v0.20.1

Compare Source

What's Changed

The most awaited offset issue with Llama is fixed 🥳

• Update README.md by @​ArthurZucker in https://github.com/huggingface/tokenizers/pull/1608
• fix benchmark file link by @​152334H in https://github.com/huggingface/tokenizers/pull/1610
• Bump actions/download-artifact from 3 to 4.1.7 in /.github/workflows by @​dependabot in https://github.com/huggingface/tokenizers/pull/1626
• [ignore_merges] Fix offsets by @​ArthurZucker in https://github.com/huggingface/tokenizers/pull/1640
• Bump body-parser and express in /tokenizers/examples/unstable_wasm/www by @​dependabot in https://github.com/huggingface/tokenizers/pull/1629
• Bump serve-static and express in /tokenizers/examples/unstable_wasm/www by @​dependabot in https://github.com/huggingface/tokenizers/pull/1630
• Bump send and express in /tokenizers/examples/unstable_wasm/www by @​dependabot in https://github.com/huggingface/tokenizers/pull/1631
• Bump webpack from 5.76.0 to 5.95.0 in /tokenizers/examples/unstable_wasm/www by @​dependabot in https://github.com/huggingface/tokenizers/pull/1641
• Fix documentation build by @​ArthurZucker in https://github.com/huggingface/tokenizers/pull/1642
• style: simplify string formatting for readability by @​hamirmahal in https://github.com/huggingface/tokenizers/pull/1632

New Contributors

• @​152334H made their first contribution in https://github.com/huggingface/tokenizers/pull/1610
• @​hamirmahal made their first contribution in https://github.com/huggingface/tokenizers/pull/1632

Full Changelog: huggingface/tokenizers@v0.20.0...v0.20.1

---

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.