Skip to content

Commit

Permalink
Fix max seq len
Browse files Browse the repository at this point in the history
  • Loading branch information
vshampor committed Dec 20, 2024
1 parent ea64dc5 commit 0a753e2
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/cpp/src/continuous_batching_impl.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ void ContinuousBatchingPipeline::ContinuousBatchingImpl::init(
m_rotation_deltas_stores.push_back(store);
}

size_t max_sequence_cache_occupation_length_in_blocks = scheduler_config.max_num_batched_tokens + 1;
size_t max_sequence_cache_occupation_length_in_blocks = scheduler_config.max_num_batched_tokens / m_scheduler->get_block_size() + 1;
size_t embedding_size = device_config.get_head_size();
m_cache_rotation_calculator = std::make_shared<CacheRotationCalculator>(
m_scheduler->get_block_size(),
Expand Down

0 comments on commit 0a753e2

Please sign in to comment.