v0.1.0

daulet released this 01 Apr 05:21

· 61 commits to main since this release

3188ded

Initial release with:

Load tokenizer from file
Encode with add_special_tokens support
Decode with skip_special_tokens support
Vocab size

Assets 2