Model Accuracy AutoGPTQ VS GPTQ-for-LLaMa VS ExLlama? Is there a difference?

[viperwasp]

I have sort of a two part question and discussion.

1. If we take any GPTQ model lets say Wizard Vicuna 13B.

I was told that if we quantize this model into five different final models.
In order for their Accuracy or perplexity whatever you want to call it.
They appear something like this.
--Best--
GGML Wizard Vicuna 13B 5_1
GGML Wizard Vicuna 13B 5_0
GPTQ Wizard Vicuna 13B 4bit
GGML Wizard Vicuna 13B 4_1
GGML Wizard Vicuna 13B 4_0
--Worst--
Of course there are other variants. But lets just use those five.
Is there any truth to this? Have you heard and or seen confirmation to yes or no to this?
The difference may be slim to none or marginally noticeable I guess depending on the model and prompt etc.

2. Now lets focus only on GPTQ Wizard Vicuna 13B 4bit.

Does anyone know is there is a difference in perplexity/accuracy between AutoGPTQ, GPTQ-for-LLaMa or ExLlama?
What have you heard? What does the Data show? How sure are you? Sharing source would be awesome.

For example I've only heard rumours. I can confirm that certain modes or models are faster or slower of course. But I have not personally checked accuracy or read anywhere that AutoGPT is better or worse in accuracy VS GPTQ-forLLaMA. But I did hear a few people say that GGML 4_0 is generally worse than GPTQ. And GGML 5_0 is generally better than GPTQ. However I have no real proof to back this up. Which is why I'm asking for opinions, facts and theories? If there is a difference between back-ends like these three modes is it like you know virtually nothing? Like 0.001% etc. If you have anything else relevant to add even if it's not directly related? Let us know. Thanks.

Just to be clear I am talking about the same exact models being compared. Same model name and Paramaters. I understand that in some cases it might actually depend on the model too. But as a rule of thumb what is the expected conclusion.

[AeonBlack]

For point 2.
After the current bugfixes as of c5641b6 and turboderp/exllama@890c258 I'm noticing not just blinding speed with ExLlama but massive accuracy jumps compared to AutoGPTQ on TheBloke_Wizard-Vicuna-13B-Uncensored-GPTQ.
This is a massive contrast to earlier today when ExLlama support was added, it was fast but heavily inaccurate when using the API with Silly Tavern.
Right now I couldn't suggest past ExLlama.

[viperwasp]

Your saying past ExLlama was kinda bad? So this does or can mean different backends can effect accuracy be it a bug or just how they work. Okay but you think the current ExLlama is the best out of all three backends? How much better? Is either AutoGPTQ or Llama-to-GPTQ bad?
Thanks for the info and how sure are you? You saw notable improvements?

Interesting I read the change log. I see. Okay yes it seems that ExLlama was bugged. Likely fix now or at least better. But I am still wondering if it has better accuracy than current AutoGPTQ or Llama-to-GPTQ if anyone knows.

[AeonBlack]

I can only say from my current usage as it has not long been since the updates, I have not used Llama-to-GPTQ previously.
And yah, I seen very notable improvements immediately, like before the coherence was shoddy, not fully talking in memory, answering wrongly, or not doing what was asked when using a complex prompt with multiple steps. Now it's able to fully carry out the prompt's requests taking in the current context.
And the speed improvement is night and day. At least 5-6x faster, if not more.

[viperwasp]

Sorry just to clarify one thing. This different your just mentioned now. Is only comparing ExLlama to former ExLlama. And has nothing to do with AutoGPTQ or Llama-to-GPTQ? I already know you said you never used Llama-to-GPTQ. But the accuracy bonus your referring to is not AutoGPTQ VS Exllama? Or is it? lol
By the way thanks AeonBlack.

I totally want someone to link like one of those preplexity tests using the same model across all three backends.

[AeonBlack]

Ah sorry, comparing AutoGPTQ to ExLlama.

[viperwasp]

Okay interesting. Thanks. So now I just need to wonder if how ExLlama compares to Llama-to-GPTQ. So far it seems like out of the two you have used ExLlama is faster and more accurate. If the same is true for ExLlama over Llama-to-GPTQ than that is absolutely amazing! Of course I know it's possible for you to get this wrong or it may somehow vary from model to model. But for now I'm going to assume your possibly correct.

P.S
If possible can anyone of you pros out there please benchmark this for us if the benchmarks would actually show which is better!

[jianmomo]

Hello, I want to know if you have found the answer to this question. I am also confused. Different loading methods mean different effects. I am curious whether the results are different because of the different models.

[viperwasp]

I don't know if your talking to me or not. But I almost feel like even experts are often confused. Few true experts around who know this crap. lol But there is just to much. Since asking this there is like so much more released and it changes weekly. Not only with how the models are run. But various settings and types of models. Even the entire way AI models are graded or tested as been thrown into question. Not in a sinister way but in a do we really even know? We can make great guesses but a lot of information that goes around is flawed and user error only adds to that. I'm just going to sit back and use the top known models. Try a few things here or there. One day we might have more concrete answers. Some seemingly simple questions I've had extreme trouble figuring out.