Issue: UI is missing option to change Flash Attention KV quantization setting

[GabeAl]

I accidentally posted a bug in the cli version of the bug tracker,

Bug fix: Flash Attention - KV cache quantization is stuck at FP16 with no way to revert to Q4_0

The gist of it is, no way to set flash attention quants = no way to fit large contexts on the GPU = regression.

This leads to a large series of usability regressions.

1. Impossible to load 128k context on GPU
2. Slower performance even with the reduced context that fits on the GPU
3. Inability to analyze long documents with the same accuracy as before (which is ironic considering 0.3's aim to enable this)
4. All issues associated with rolling back to 0.2X series, including chat history import.

Feel free to close the other one I made earlier:
lmstudio-ai/lms#70

[dmatora]

I had to switch to ollama to use 70B 128K.
Please add support for Q4/Q8 KV cache quantization

[yagil]

@dmatora @GabeAl is the specific ask here a way to set the KV cache quantization level?

[dmatora]

yes

[GabeAl]

Yes absolutely. Many users have commented including on Discord about this. It is one major bottleneck holding users back.

I depend on large contexts, and large contexts need quantization (q4) to be feasible on commodity hardware.