Merge releases/2024/3 into master #720

Wovchena · 2024-07-31T20:15:20Z

No description provided.

Workaround Python_VERSION_MAJOR and MINOR not being set by replasing Python3 with Python Disable generation of some of the COMPONENTs not needed for GenAI. There are still unwanted empty archives, but they are generated uncounditionally by rapidjson.

…envinotoolkit#604) That allows LLMPipeline to create ContinuousBatchingPipeline as a backend. There's also a constructor accepting ireq, which can be used if the model was already transformed appropriately for ContinuousBatchingPipeline. But it feels it's going to be misleading and it simpler just to throw if such constructor is called with ContinuousBatchingPipeline backend.

) Port of PR: openvinotoolkit#615

Updated default configurations based on results from CVS-143530. (cherry picked from commit f460002)

Remove unwanted archives

…t#637)

Co-authored-by: Yaroslav Tarkan <[email protected]>

…#642) OpenVINOGenAITargets.cmake was excluded from packaging because CPACK_COMPONENTS_ALL is custom now and doesn't install Unspecified component

Co-authored-by: Pavel Esir <[email protected]>

…oop for greedy sampling (openvinotoolkit#607) Searching for max element in a custom loop gives better performance than using std::max_element

Cherry picked from master

@Wovchena

@Wovchena, retarget to OV 24.3 release branch

- Added Readme for python tests - Added `--model_ids` option to run selectively only on specific models --------- Co-authored-by: Zlobin Vladimir <[email protected]>

…Readme

Symbols that cause errors: - `\u0643` - `\u25aa`

… optional plugin_config in tokenizer (openvinotoolkit#669) This improves performance of CB lib when tested within OVMS. Already merged to master: openvinotoolkit#651 This is cherry-pick

…oolkit#670) [mixtral-8x7b-instruct-v0.1-int4-ov](https://huggingface.co/OpenVINO/mixtral-8x7b-instruct-v0.1-int4-ov/) didn't have `generation_config.json` therefore generation continued ininitely. EOS_TOKEN_ID was red correctly but during generation it was not met. Updated docs so in every generate call max_new_tokens is set either in arguments or via default generation config `pipe.set_generation_config({'max_new_tokens': 100, 'num_beam_groups': 3, ...)` tickets: CVS-146933 CVS-146324

Co-authored-by: Anastasiia Pnevskaia <[email protected]>

Docstring for generation time metrics Ticket: CVS-132859

Co-authored-by: Zlobin Vladimir <[email protected]>

src/README.md

samples/python/benchmark_genai/README.md

ilya-lavrenov · 2024-08-01T08:46:55Z

samples/cpp/benchmark_genai/benchmark_genai.cpp

do we have CI runs for these new samples? I don't see it

No. It was last hour merge

do we have a task for it?

Created 148650

src/README.md

src/cpp/include/openvino/genai/llm_pipeline.hpp

ilya-lavrenov · 2024-08-01T08:52:36Z

src/cpp/include/openvino/genai/perf_metrics.hpp

Are they applicable only for Stateful pipeline or works will for static / CB as well?

Only for Stateful. I suspect uninitialized perf metrics are returned for CB backend.

Let's create a task for it.
We need unified response for all backends.

Created 148651

ilya-lavrenov · 2024-08-01T08:53:15Z

src/cpp/include/openvino/genai/scheduler_config.hpp

@@ -16,7 +16,7 @@ struct SchedulerConfig {
    std::size_t num_kv_blocks = 0;

    // total size of KV cache in GB
-    std::size_t cache_size = 0;
+    std::size_t cache_size = 1;


why? it's explicitly set to 0, so users have to set some meaningful values on their own

It's now possible to construct the struct with all the default values which is used to create CB backend in LLMPipeline.

ok, let's fix it together with CB device name via passing scheduler config as device property

@olpipi, please, address it

ilya-lavrenov · 2024-08-01T08:58:37Z

src/docs/BUILD.md

- Microsoft Visual Studio 2019 or higher, version 16.3 or later
- Python 3.8 or higher
- Git for Windows
+    #### Option 2 - setting environment variables manually:


why do we need to recommend such approach? setupvars is default and we don't need to expose variables manually (which is implementation detail), because we will have to support it in future in case of any changes in OpenVINO scripts.

Developers tend to stop at build step and skip installation part. @TolyaTalamanov is an example.

it's a developer problems that they cannot read readme step by step

They can. But they aren't explained that it's not recommended to skip it and why.

Wovchena and others added 30 commits July 15, 2024 13:48

Workaround (openvinotoolkit#618)

e4637b3

Workaround Python_VERSION_MAJOR and MINOR not being set by replasing Python3 with Python Disable generation of some of the COMPONENTs not needed for GenAI. There are still unwanted empty archives, but they are generated uncounditionally by rapidjson.

Revert to python3

423c8e3

Revert to python3 (openvinotoolkit#622)

8ad336c

Fix cmake Python var name (openvinotoolkit#624)

1b1b2f0

Clear beam search info when generate() is finished. (openvinotoolkit#630

f0c2677

) Port of PR: openvinotoolkit#615

Update nncf_utils.py (openvinotoolkit#616) (openvinotoolkit#633)

73badf6

Updated default configurations based on results from CVS-143530. (cherry picked from commit f460002)

Workaround cmake packaging (openvinotoolkit#634)

25655e3

Remove unwanted archives

Save licensing_genai into docs to align with OpenVINO (openvinotoolki…

754f6d7

…t#637)

Update submodule (openvinotoolkit#638)

e5247e0

Add Llama3 (openvinotoolkit#620)

2d1fa3b

Co-authored-by: Yaroslav Tarkan <[email protected]>

nightly->rc1 (openvinotoolkit#621)

489a87d

Add OpenVINOGenAITargets to core_genai_dev COMPONENT (openvinotoolkit…

67f0467

…#642) OpenVINOGenAITargets.cmake was excluded from packaging because CPACK_COMPONENTS_ALL is custom now and doesn't install Unspecified component

Apply todo, initialize detokenizer's cache (openvinotoolkit#647)

1969160

Cherry-pick static LLM pipeline changes (openvinotoolkit#654)

0e0f6a9

Co-authored-by: Pavel Esir <[email protected]>

[Continuous batching] Replace standard max_element call with custom l…

cb100cb

…oop for greedy sampling (openvinotoolkit#607) Searching for max element in a custom loop gives better performance than using std::max_element

wip

f0e4190

add detokenization metric; refactor split to perf_conter & perf_metrics

7cab496

refactor structure, add python sample

bb1113c

Cherry-pick custom max_element loop (openvinotoolkit#662)

7bf42f1

Cherry picked from master

add more preicise durations

0a8f0d9

Add note for pybind ov::Tensor issue (openvinotoolkit#659)

bad01b9

[OV 24.3]Fix multinomial sample CMakeList (openvinotoolkit#658)

cb0da0a

@Wovchena, retarget to OV 24.3 release branch

add Readme for tests (openvinotoolkit#664)

bc92248

- Added Readme for python tests - Added `--model_ids` option to run selectively only on specific models --------- Co-authored-by: Zlobin Vladimir <[email protected]>

add cpp Readme, ensured correct batch processing, add PerfMetrics to …

90320f4

…Readme

use MeanStdPair

aeec730

[2024.3] Fix symbol encode error (openvinotoolkit#629)

56eeafc

Symbols that cause errors: - `\u0643` - `\u25aa`

[release branch] Add infer request queue for tokenizers and allow for…

8934a0e

… optional plugin_config in tokenizer (openvinotoolkit#669) This improves performance of CB lib when tested within OVMS. Already merged to master: openvinotoolkit#651 This is cherry-pick

Add CB naive chat (openvinotoolkit#644)

f9e45e1

Co-authored-by: Anastasiia Pnevskaia <[email protected]>

Wovchena and others added 15 commits July 31, 2024 01:43

Disable broken test (openvinotoolkit#707)

2a80828

update optimum commit for releases/2024/3 (openvinotoolkit#711)

d89cdcb

change commit for optimum

2428a3a

Merge branch 'releases/2024/3' into ea/upd_opt_commit

1473e7f

change commit for optimum (openvinotoolkit#714)

8cb12b2

Add perf metric docstrings (openvinotoolkit#713)

2f778f3

Docstring for generation time metrics Ticket: CVS-132859

rc1->rc2 (openvinotoolkit#695)

2dc6b64

Docs for version compatibility (openvinotoolkit#692)

3bfdd3f

Co-authored-by: Zlobin Vladimir <[email protected]>

Merge branch 'releases/2024/3' into merge-releases/2024/3-into-master

a9d6541

coorect after git merge conflict resolution

e76f9f9

coorect after git merge conflict resolution

9a0b7e9

coorect after git merge conflict resolution

a88cfc8

coorect after git merge conflict resolution

2311f6e

cache_size

45937bc

skip

4962039

Wovchena requested review from ilya-lavrenov, pavel-esir and yatarkan August 1, 2024 07:19

yatarkan requested changes Aug 1, 2024

View reviewed changes

src/README.md Outdated Show resolved Hide resolved

ilya-lavrenov reviewed Aug 1, 2024

View reviewed changes

Correct links and typos

b5f21dc

ilya-lavrenov approved these changes Aug 1, 2024

View reviewed changes

yatarkan approved these changes Aug 1, 2024

View reviewed changes

pavel-esir approved these changes Aug 1, 2024

View reviewed changes

Wovchena added this pull request to the merge queue Aug 1, 2024

Wovchena removed this pull request from the merge queue due to a manual request Aug 1, 2024

Wovchena added this pull request to the merge queue Aug 1, 2024

Merged via the queue into openvinotoolkit:master with commit 47fbb5e Aug 1, 2024
27 checks passed

pavel-esir mentioned this pull request Aug 6, 2024

cherry-pick from release: Add custom streamer doc #749

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge releases/2024/3 into master #720

Merge releases/2024/3 into master #720

Wovchena commented Jul 31, 2024

ilya-lavrenov Aug 1, 2024

Wovchena Aug 1, 2024

ilya-lavrenov Aug 1, 2024

Wovchena Aug 1, 2024

ilya-lavrenov Aug 1, 2024

Wovchena Aug 1, 2024

ilya-lavrenov Aug 1, 2024

Wovchena Aug 1, 2024

ilya-lavrenov Aug 1, 2024

Wovchena Aug 1, 2024

ilya-lavrenov Aug 1, 2024

Wovchena Aug 1, 2024

ilya-lavrenov Aug 1, 2024

Wovchena Aug 1, 2024

ilya-lavrenov Aug 1, 2024

Wovchena Aug 1, 2024

Merge releases/2024/3 into master #720

Merge releases/2024/3 into master #720

Conversation

Wovchena commented Jul 31, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment