Skip to content

Commit

Permalink
Don't enable hlfb for KV cache in OSS.
Browse files Browse the repository at this point in the history
PiperOrigin-RevId: 693441376
  • Loading branch information
haozha111 authored and copybara-github committed Nov 5, 2024
1 parent 29aa17f commit cdba806
Showing 1 changed file with 3 additions and 0 deletions.
3 changes: 3 additions & 0 deletions ai_edge_torch/generative/layers/kv_cache.py
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,9 @@ def update(
Returns:
KVCacheEntry: The updated KVCache entry based on the passed inputs.
"""
# Don't enable HLFB for kv cache op for now, since it won't work with LLM
# inference engine. Remove this part once we ship a new LLM inference engine.
enable_hlfb=False
update_func = _update_kv_hlfb_impl if enable_hlfb else _update_kv_base_impl
return update_func(cache, input_pos, k_slice, v_slice)

Expand Down

0 comments on commit cdba806

Please sign in to comment.