Accept buffer in LLMPipeline ctor #1262

pavel-esir · 2024-11-27T12:02:34Z

Ticket: CVS-158144, CVS-158142

src/cpp/src/tokenizer.cpp

src/cpp/src/utils.hpp

src/cpp/src/utils.cpp

src/cpp/src/tokenizer.cpp

samples/cpp/chat_sample/chat_sample.cpp

src/cpp/include/openvino/genai/tokenizer.hpp

src/cpp/src/continuous_batching_pipeline.cpp

src/cpp/src/speculative_decoding/speculative_decoding_impl.hpp

src/cpp/src/continuous_batching_pipeline.cpp

src/cpp/src/llm_pipeline.cpp

samples/cpp/text_generation/README.md

samples/cpp/text_generation/CMakeLists.txt

samples/cpp/text_generation/generate_with_encrypted_model.cpp

src/cpp/include/openvino/genai/tokenizer.hpp

src/cpp/src/speculative_decoding/speculative_decoding_impl.cpp

src/cpp/src/llm_pipeline.cpp

ilya-lavrenov · 2024-12-04T09:12:25Z

src/cpp/src/tokenizer.cpp

+    ov::Core core;
+    const char* ov_tokenizer_path = getenv(ScopedVar::ENVIRONMENT_VARIABLE_NAME);
+    OPENVINO_ASSERT(ov_tokenizer_path, "openvino_tokenizers path is not set");
+    core.add_extension(ov_tokenizer_path);


I still think that default singleton from utils.hpp can include tokenizer extension to avoid duplication of logic of multiples Cores creation.

src/cpp/src/llm_pipeline.cpp

… LLMPipeline ctor

…ctors

samples/cpp/text_generation/encrypted_model_causal_lm.cpp

ilya-lavrenov · 2024-12-04T13:42:57Z

samples/cpp/text_generation/encrypted_model_causal_lm.cpp

+    // This is necessary only for NPU, for other plugins can be ommited.
+    ov::AnyMap model_descr_properties = {{"name_or_path", "meta-llama/Llama-2-7b-chat-hf"}, 
+                                         {"type", "llama"}, 
+                                         {"num_key_value_heads", 32}};


maybe we can avoid demonstrating such NPU hacks in samples?

I hope that @dmatveev and @TolyaTalamanov can come up with more generic approach which does not require reading this information from config.json. E.g. num_key_value_heads can be extracted from model itself

openvino.genai/src/cpp/src/utils/paged_attention_transformations.cpp

Lines 23 to 25 in e2fa0d0

ov::PartialShape k_shape = parameters[kv_caches_inputs_offset]->get_partial_shape();

OPENVINO_ASSERT(k_shape.rank().get_length() == 3, "KV cache shape is expected to have rank 3, while shape is ", k_shape);

size_t num_kv_heads = k_shape[1].get_length(), head_size = k_shape[2].get_length();

Removed this from the sample. In order not to forget about that created a ticket for NPU to search for a better solution.
@TolyaTalamanov @dmatveev

src/cpp/src/llm_pipeline_static.hpp

src/cpp/src/llm_pipeline.cpp

…ctors

Ticket: CVS-158144, CVS-158142

- ~#1302 (didn't port this PR because of the issue CVS-159227) - #1262 - #1336 - #1331 --------- Co-authored-by: Andrei Kochin <[email protected]> Co-authored-by: Vladimir Zlobin <[email protected]> Co-authored-by: Ilya Lavrenov <[email protected]>

- ~openvinotoolkit#1302 (didn't port this PR because of the issue CVS-159227) - openvinotoolkit#1262 - openvinotoolkit#1336 - openvinotoolkit#1331 --------- Co-authored-by: Andrei Kochin <[email protected]> Co-authored-by: Vladimir Zlobin <[email protected]> Co-authored-by: Ilya Lavrenov <[email protected]>

- ~openvinotoolkit/openvino.genai#1302 (didn't port this PR because of the issue CVS-159227) - openvinotoolkit/openvino.genai#1262 - openvinotoolkit/openvino.genai#1336 - openvinotoolkit/openvino.genai#1331 --------- Co-authored-by: Andrei Kochin <[email protected]> Co-authored-by: Vladimir Zlobin <[email protected]> Co-authored-by: Ilya Lavrenov <[email protected]>

github-actions bot added category: LLM LLM pipeline (stateful, static) category: sampling Sampling / Decoding algorithms category: tokenizers Tokenizer class or submodule update category: samples GenAI samples category: GenAI C++ API Changes in GenAI C++ public headers labels Nov 27, 2024

pavel-esir changed the title ~~initial~~ Accept buffer in LLMPipeline ctor Nov 27, 2024

ilya-lavrenov assigned ilya-lavrenov and Wovchena Nov 27, 2024

pavel-esir added 2 commits November 27, 2024 13:13

initial

624d9f5

use string and ov::Tensor instead of a raw buffer

9ed8f6e

ilya-lavrenov reviewed Nov 27, 2024

View reviewed changes

src/cpp/src/tokenizer.cpp Outdated Show resolved Hide resolved

src/cpp/src/utils.hpp Outdated Show resolved Hide resolved

src/cpp/src/utils.cpp Show resolved Hide resolved

src/cpp/src/tokenizer.cpp Outdated Show resolved Hide resolved

ilya-lavrenov added this to the 2024.6 milestone Nov 27, 2024

ilya-lavrenov reviewed Nov 27, 2024

View reviewed changes

samples/cpp/chat_sample/chat_sample.cpp Outdated Show resolved Hide resolved

ilya-lavrenov reviewed Nov 27, 2024

View reviewed changes

src/cpp/include/openvino/genai/tokenizer.hpp Show resolved Hide resolved

github-actions bot added category: continuous batching Continuous batching category: speculative decoding Speculative decoding labels Nov 28, 2024

continuous batching ctor with model from buffer

dd69db2

github-actions bot removed the category: samples GenAI samples label Nov 28, 2024

revert chat sample

3856770

pavel-esir force-pushed the add_new_ctors branch from 413c015 to 2cdf0d3 Compare November 28, 2024 23:00

pavel-esir added 3 commits November 29, 2024 00:12

add CTOR with model_str and ov::Tensor buffers to NPU/StaticLLMPipeline

2cdf0d3

fix win build, fix chat template patching

5b73eb4

fix speculative decoding

0f45144

ilya-lavrenov mentioned this pull request Nov 29, 2024

[VLM] Support compile OV model and weight from buffer #1264

Merged

improve TokenizerImpl

cb7f55e

pavel-esir marked this pull request as ready for review November 29, 2024 10:59

pavel-esir requested a review from ilya-lavrenov November 29, 2024 10:59

fix typos

add6268

ilya-lavrenov reviewed Nov 29, 2024

View reviewed changes

pavel-esir commented Nov 29, 2024

View reviewed changes

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

andrei-kochin added the Code Freeze label Dec 4, 2024

github-actions bot added no-match-files and removed category: sampling Sampling / Decoding algorithms labels Dec 4, 2024

fix path greedy_causal_lm -> text_generation

4b7c4c2

ilya-lavrenov reviewed Dec 4, 2024

View reviewed changes

pavel-esir added 2 commits December 4, 2024 13:14

update encrypted_model_causal_lm sample, made model_desr setable from…

0fc3bbe

… LLMPipeline ctor

Merge remote-tracking branch 'upstream/releases/2024/5' into add_new_…

62ba450

…ctors

pavel-esir requested a review from ilya-lavrenov December 4, 2024 12:26

add ctor with Properties

4a45257

pavel-esir requested a review from apaniukov December 4, 2024 13:27

ilya-lavrenov reviewed Dec 4, 2024

View reviewed changes

apaniukov reviewed Dec 4, 2024

View reviewed changes

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

apaniukov approved these changes Dec 4, 2024

View reviewed changes

fix "Yoda style" if statements, some other corrections

2457b89

ilya-lavrenov reviewed Dec 4, 2024

View reviewed changes

src/cpp/src/llm_pipeline.cpp Outdated Show resolved Hide resolved

pavel-esir added 2 commits December 4, 2024 19:22

simplify a bit TokenizerImpl construction

9befd0c

fix plugin_config -> properties for NPY

bbe1b7b

ilya-lavrenov approved these changes Dec 4, 2024

View reviewed changes

ilya-lavrenov added the port to master PR needs to be ported to master from release branch label Dec 4, 2024

Merge remote-tracking branch 'upstream/releases/2024/5' into add_new_…

c05febe

…ctors

andrei-kochin added this pull request to the merge queue Dec 5, 2024

Merged via the queue into openvinotoolkit:releases/2024/5 with commit 2d4496a Dec 5, 2024
52 checks passed

pavel-esir deleted the add_new_ctors branch December 5, 2024 08:30

pavel-esir pushed a commit to pavel-esir/openvino.genai that referenced this pull request Dec 10, 2024

Accept buffer in LLMPipeline ctor (openvinotoolkit#1262)

0d0ff4a

Ticket: CVS-158144, CVS-158142

pavel-esir mentioned this pull request Dec 10, 2024

Port from 24.6 release to master #1356

Merged

pavel-esir removed the port to master PR needs to be ported to master from release branch label Dec 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Accept buffer in LLMPipeline ctor #1262

Accept buffer in LLMPipeline ctor #1262

pavel-esir commented Nov 27, 2024 •

edited

Loading

ilya-lavrenov Dec 4, 2024

ilya-lavrenov Dec 4, 2024

pavel-esir Dec 4, 2024

	ov::PartialShape k_shape = parameters[kv_caches_inputs_offset]->get_partial_shape();
	OPENVINO_ASSERT(k_shape.rank().get_length() == 3, "KV cache shape is expected to have rank 3, while shape is ", k_shape);
	size_t num_kv_heads = k_shape[1].get_length(), head_size = k_shape[2].get_length();

Accept buffer in LLMPipeline ctor #1262

Accept buffer in LLMPipeline ctor #1262

Conversation

pavel-esir commented Nov 27, 2024 • edited Loading

ilya-lavrenov Dec 4, 2024

Choose a reason for hiding this comment

ilya-lavrenov Dec 4, 2024

Choose a reason for hiding this comment

pavel-esir Dec 4, 2024

Choose a reason for hiding this comment

pavel-esir commented Nov 27, 2024 •

edited

Loading