Skip to content

Commit

Permalink
Add Building Without FastTokenizer Info (openvinotoolkit#247)
Browse files Browse the repository at this point in the history
  • Loading branch information
apaniukov authored Aug 30, 2024
1 parent d98f727 commit 6336aa3
Showing 1 changed file with 16 additions and 0 deletions.
16 changes: 16 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,22 @@ By default, all available ICU locales are supported, which significantly increas

By following these instructions, you can effectively reduce the size of the ICU libraries in your final package.

### Build OpenVINO Tokenizers without FastTokenizer Library

If a tokenizer doesn't use `CaseFold`, `UnicodeNormalization` or `Wordpiece` operations, you can drastically reduce package binary size by building OpenVINO Tokenizers without FastTokenizer dependency with this flag:
```bash
-DENABLE_FAST_TOKENIZERS=OFF
```
This option can also help with building for platform that is supported by FastTokenizer, for example `Android x86_64`.
Example for a pip installation path:
```bash
pip install git+https://github.com/openvinotoolkit/openvino_tokenizers.git --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly --config-settings=override=cmake.options.ENABLE_FAST_TOKENIZERS=OFF
```
## Usage
:warning: OpenVINO Tokenizers can be inferred on a `CPU` device only.
Expand Down

0 comments on commit 6336aa3

Please sign in to comment.