Releases · google/sentencepiece

08 Jun 09:05

taku910

v0.1.92

Bug fix

Assets 32

21 May 03:25

taku910

v0.1.91

[Python] Added a feature to feed training data as Python's iterable object.
https://github.com/google/sentencepiece/tree/master/python#training-without-local-filesystem
[Python] Added a feature to set model writer to emit the output model to any non-local devices.
https://github.com/google/sentencepiece/tree/master/python#training-without-local-filesystem
[C++] Add an API to returns the trained model directly as std::string.

Ignores nbest parameter in BPE-dropout
fixed build error when SPM_ENABLE_NFKC_COMPILE=ON
fixed the cost computation around user_defined_symbol and faster encoding introduced in the previous release.

Assets 32

13 May 06:20

taku910

v0.1.90

Renamed v0.1.9 to v0.1.90 because PyPI doesn't recognize 0.1.9 as the latest release.

Assets 32

13 May 02:52

taku910

v0.1.9

--byte_fallback: fallback UNK token into UTF-8 byte sequences. 256 byte symbols are reserved in advance.
https://arxiv.org/pdf/1909.03341.pdf Note that you need to set --character_coverage less than 1.0, otherwise byte-fall-backed token may not appear in the training data.
BPE-dropout: Implemented BPE dropout. https://arxiv.org/abs/1910.13267
Sampling API is available for the BPE.
https://github.com/google/sentencepiece/blob/master/src/sentencepiece_processor.h#L287
--required_chars=chars: Specify the set of Unicode chars that must be included in the final vocab.
--split_digits: Split all digits (0-9) into separate pieces (disabled by default)
Denormalization: Apply extra normalization rule after decoding. We can specify the rule as TSV via --denormalization_rule_tsv=file flag. Note that offset information may not always be preserved.
--train_extremely_large_corpus: Train the unigram model from extremely large corpus (> 10M sentences) to avoid integer overflow. Note that it will increase the memory usage. 300GB or larger memory might be necessary.

30%-50% performance improvement is obtained in the default unigram one-best tokenization.

[Python] Added Python friendly API. New API allows to feed any chars to user_defined_symbols during the training. The old methods are still available.
https://github.com/google/sentencepiece/tree/master/python#segmentation
[C++] Added the interface to feed training data via arbitrary iterator object.
https://github.com/google/sentencepiece/blob/master/src/sentencepiece_trainer.h#L40
[C++] Added the interface to set set a pre-tokenizer to specify the word boundary. This is used as a word-boundary constraint to set the seed vocabulary, and not used in the inference time.
https://github.com/google/sentencepiece/blob/master/src/pretokenizer_for_training.h