feat: support attention sinks

[OlivierDehaene]

See https://arxiv.org/abs/2309.17453

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[CEDIDataVault]

Hello, may I ask if attention sinks are still rather unstable currently? How should I use it?

[Bec-k]

Some test is failing. Test should be changed or it's a bug?

[OlivierDehaene]

The implemetation is not correct yet as we do not cache before RoPE. I need to modify the paged attention kernels but I did not find the time yet to do it.

[Bec-k]

Any update on this one?

[zfang]

Looking forward to this as a better alternative than RoPE interpolation. In my experience RoPE interpolation techniques such as NTK tend to cause hallucination.

[Bec-k]

Strange that this is not yet implemented, because i haven't seen or heard a better attentions technique so far. Please prove me wrong.

[Narsil]

Reason is simple, this implementation doesn't work exactly like sinks, the reason is KV cache handling because attention sinks require to slide everything except the sinks themselves, which makes AFAIK the KV cache invalid (since as soon as you slide once, the attention of the first non sink token is wrong, since it had been attending to non sinks before).
Prove me wrong.

@Bec-k No one is saying you're wrong. That doesn't mean we can implement the feature easily (or that it's worth the engineering time or maintenance cost) atm.

[xuan1905]

Attention Sinks caching has been implemented in transformers. Really looking forward to the release of this feature on huggingface.

[Bec-k]

@OlivierDehaene attention cache have been added into the huggingface transformers package, it is implemented as a custom cache:
https://huggingface.co/docs/transformers/en/internal/generation_utils#transformers.SinkCache
https://github.com/huggingface/transformers/blob/main/src/transformers/cache_utils.py#L172
PR of integration into transformers library:
https://github.com/huggingface/transformers/pull/26681/files

Can you refactor "text-generation-inference" code to adapt this new feature? Does this solves initial problem?

[Bec-k]

Possibly you can forward that new Sinks Cache implementation in transformers and request additional changes for it to be properly implemented into your runtime?