-
Notifications
You must be signed in to change notification settings - Fork 199
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
1 changed file
with
1 addition
and
1 deletion.
There are no files selected for viewing
Submodule openvino_tokenizers
updated
27 files
| +2 −0 | .github/workflows/labeler.yml | |
| +17 −6 | .github/workflows/linux.yml | |
| +6 −2 | .github/workflows/mac.yml | |
| +6 −3 | .github/workflows/windows.yml | |
| +12 −3 | CMakeLists.txt | |
| +169 −126 | README.md | |
| +2 −1 | pyproject.toml | |
| +11 −0 | python/openvino_tokenizers/constants.py | |
| +9 −1 | python/openvino_tokenizers/convert_tokenizer.py | |
| +169 −50 | python/openvino_tokenizers/hf_parser.py | |
| +1 −17 | python/openvino_tokenizers/tiktoken_parser.py | |
| +91 −58 | python/openvino_tokenizers/tokenizer_pipeline.py | |
| +52 −7 | python/openvino_tokenizers/utils.py | |
| +21 −4 | src/CMakeLists.txt | |
| +52 −0 | src/byte_fallback.cpp | |
| +36 −0 | src/byte_fallback.hpp | |
| +1 −1 | src/icu4c.patch | |
| +1 −0 | src/ov_extension.cpp | |
| +3 −3 | src/ragged_to_dense.cpp | |
| +28 −21 | src/sentence_piece.cpp | |
| +2 −0 | src/sentence_piece.hpp | |
| +1 −0 | src/tokenizer.hpp | |
| +21 −20 | src/vocab_decoder.cpp | |
| +47 −1 | tests/conftest.py | |
| +1 −1 | tests/pass_rates.json | |
| +12,885 −0 | tests/stats.json | |
| +115 −69 | tests/tokenizers_test.py |