Release ver2.2
What's Changed
- Improve for gpt neox by @hiroshi-matsuda-rit in #3
- Release hf-slow ver2.1 alpha1 by @hiroshi-matsuda-rit in #4
- Strictly distinguishing slow and fast tokenizers by @hiroshi-matsuda-rit in #5
- update docstrings by @hiroshi-matsuda-rit in #6
- Release hf_fast.a2 by @hiroshi-matsuda-rit in #7
- Release hf_slow.a2 by @hiroshi-matsuda-rit in #8
- add trust_remote_code setting files by @hiroshi-matsuda-rit in #9
- Apply Apache 2.0 by @hiroshi-matsuda-rit in #10
- Release ver2.2 by @hiroshi-matsuda-rit in #11
New Contributors
- @hiroshi-matsuda-rit made their first contribution in #3
Full Changelog: https://github.com/llm-jp/llm-ja-tokenizer/commits/v2.2