Skip to content

v0.1.96
Compare
Choose a tag to compare
@taku910 taku910 released this 17 Jun 16:55
v0.1.96
d8711f5
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Updates

  • Improves the performance of unigram training
  • Updated the nfkc normalization with the latest ICU module.
  • Stop handling zero-width-joiner string as whitespace.

New features

  • added new sampling algorithm without replacement.
  • added API for new sampling and perplexity calculation.
  • added allow_whitespace_only_pieces mode.
Assets 46
Loading