Skip to content

Commit

Permalink
Updated docstring.
Browse files Browse the repository at this point in the history
  • Loading branch information
popovaan committed Jul 24, 2024
1 parent 7febd46 commit f33f10e
Showing 1 changed file with 6 additions and 1 deletion.
7 changes: 6 additions & 1 deletion src/cpp/include/openvino/genai/scheduler_config.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,12 @@ struct SchedulerConfig {
// max number of scheduled sequences (you can think of it as "max batch size")
std::size_t max_num_seqs = 256;

// enable caching of KV-blocks and reusage in next generations.
// Enable caching of KV-blocks.
// When turned on all previously calculated KV-caches are kept in memory for future usages.
// KV-caches can be rewritten if KV-cache limit is reached, but blocks are not released.
// This results in more RAM usage, maximum RAM usage is determined by cache_size or num_kv_blocks parameters.
// When turend off only KV-cache required for batch calculation is kept in memory and
// when a sequence has finished genegartion its cache is released.
bool enable_prefix_caching = false;
};
}

0 comments on commit f33f10e

Please sign in to comment.