support for W4A8KV8/16 and other models

[KKwanhee]

Thank you for making the code publicly available to evaluate various quantization configurations.

I have a few questions:

First, using these codes, is it possible to evaluate W4A8KV8 or W4A8KV16 instead of W4A8KV4 in lmquant/projects/llm/scripts/qoq.sh?

Second, can I run accuracy evaluations for other models like Qwen2.5-32B or Gemma2? If possible, are there default values for alpha and beta of qoq that do not result in significant accuracy loss?

Thanks!

[synxlin]

Hi,

For your first question, you can evaluate W4A8KV8 or W4A8KV16 by directly settings the dtype parameter for opts field in the configuration (see here).

For your second question, we currently do not have smooth alpha and beta for Qwen2.5 and Gemma2. You may try with the default settings here and here.