Update streaming in LM Encoding & CB #1377
Conversation
src/cpp/src/lm_encoding.cpp
Outdated
| @@ -126,7 +126,7 @@ std::pair<EncodedResults, int32_t> get_lm_encoded_results( | |||
| get_active_sequence_groups), | |||
| active_sequence_groups.end()); | |||
|
|
|||
| while (active_sequence_groups.size() > 0) { | |||
| do { | |||
| size_t total_num_tokens = 0; | |||
|
|
|||
| for (auto& sequence_group : active_sequence_groups) { | |||
There was a problem hiding this comment.
BTW
@sbalandi should we use active_sequence_groups instead of sequence_groups when compute beam_offets ?
There was a problem hiding this comment.
it looks like yes, we can use active_sequence_groups here, but that variant is ok too, as in non active num_running_seqs will be 0
I'll check it to be sure and fix in #1215
…nai into streaming_lm_encoding
iefode
changed the title
| if (streamer_ptr && generations.at(0).get()->can_read()) { | ||
| std::unordered_map<uint64_t, GenerationOutput> token = generations.at(0).get()->back(); | ||
| for (const auto& gen_token : token.begin()->second.generated_ids) { | ||
| if (!streamer_ptr->put(gen_token)) { |
There was a problem hiding this comment.
copy-pasted incorrect code?
According to documentation
openvino.genai/src/cpp/include/openvino/genai/streamer_base.hpp
Lines 18 to 20 in 9bcadf7
we need to break when
put returns true
| OPENVINO_ASSERT(1 == token.size()); | ||
| OPENVINO_ASSERT(1 == token.begin()->second.generated_ids.size()); | ||
| continue_generation = !streamer_ptr->put(token.begin()->second.generated_ids.at(0)); | ||
| for (const auto& gen_token : token.begin()->second.generated_ids) { |
There was a problem hiding this comment.
continue_generation assignment is dropped here and hence, we can have abandoned requests with allocated block tables as drop_requests() is not called below.
No description provided.