Skip to content

Commit

Permalink
revert
Browse files Browse the repository at this point in the history
  • Loading branch information
Wovchena committed Jun 7, 2024
1 parent a8726e0 commit dd6d19a
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 5 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/genai_package.yml
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ jobs:
- run: call ov\setupvars.bat && python -m pip install ./thirdparty/openvino_tokenizers/[transformers] --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/pre-release
- run: call ov\setupvars.bat && python -m pip install --upgrade-strategy eager -r ./samples/requirements.txt
# optimum randomly fails with TinyLlama/TinyLlama-1.1B-Chat-v1.0 in that particular workflow.
- run: call ov\setupvars.bat && optimum-cli export openvino --trust-remote-code --model katuni4ka/tiny-random-phi3 tiny-random-phi3
- run: call ov\setupvars.bat && "${{ github.workspace }}/samples_install/samples_bin/greedy_causal_lm" .\tiny-random-phi3\ ""
- run: call ov\setupvars.bat && ./ov/samples/python/multinomial_causal_lm/multinomial_causal_lm.py ./tiny-random-phi3/ 0
- run: call ov\setupvars.bat && ooptimum-cli export openvino --trust-remote-code --weight-format fp16 --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 TinyLlama-1.1B-Chat-v1.0
- run: call ov\setupvars.bat && "${{ github.workspace }}/samples_install/samples_bin/greedy_causal_lm" .\TinyLlama-1.1B-Chat-v1.0\ ""
- run: call ov\setupvars.bat && ./ov/samples/python/multinomial_causal_lm/multinomial_causal_lm.py ./TinyLlama-1.1B-Chat-v1.0/ 0
if: ${{ 'Release' == matrix.build-type }} # Python bindings can be built in Release only
2 changes: 0 additions & 2 deletions src/cpp/src/tokenizer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -124,8 +124,6 @@ class Tokenizer::TokenizerImpl {

TokenizerImpl(std::filesystem::path tokenizer_path) {
ov::Core core;
// Disable mmap to allow constructing multiple Tokenizers from the same file.
core.set_property(ov::enable_mmap(false));

if (tokenizer_path.extension() == ".xml")
OPENVINO_THROW("ov_tokenizers_path should be a path to a dir not a xml file");
Expand Down

0 comments on commit dd6d19a

Please sign in to comment.