Skip to content

Commit

Permalink
Updated tokenizers
Browse files Browse the repository at this point in the history
  • Loading branch information
@ilya-lavrenov
ilya-lavrenov committed Aug 1, 2024
1 parent 1b64f77 commit d10bf1b
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion thirdparty/openvino_tokenizers
Submodule openvino_tokenizers updated 13 files
+16 −16 README.md
+128 −15 python/openvino_tokenizers/hf_parser.py
+1 −14 python/openvino_tokenizers/tokenizer_pipeline.py
+27 −13 src/CMakeLists.txt
+1 −1 src/regex_normalization.hpp
+134 −47 src/regex_split.cpp
+23 −8 src/regex_split.hpp
+1 −1 src/tensorflow_translators.cpp
+30 −5 src/utils.cpp
+2 −0 src/utils.hpp
+1 −1 tests/pass_rates.json
+2,000 −2,000 tests/stats.json
+2 −1 tests/tokenizers_test.py

0 comments on commit d10bf1b

Please sign in to comment.