Problem with training LoRA for Model "TheBloke/Pygmalion-2-13B-GPTQ"

[DasBinNichtIch]

When I train the LoRA with my own format, then I can create this LoRA but before it finishes I get this error:

That means it cannot find the {'v_proj', 'q_proj'} modules. Is that only because Pygmalion doesnt have them or is there another problem?

[araleza]

You can perform LoRA training on 4 bit GPTQ models, but you have to load them with the Transformers model loader, not any of the other ones. If you load the model with (e.g.) ExLlamav2_HF, you'll get that error message that you've shown here.

The docs say you should tick the 'auto-devices' and 'disable_exllama' options when Loading the model with the Transformers loader in order to perform LoRA training.

[araleza]

There's a bug where you can't apply LoRAs to models loaded with Transformers (although it'll say it successfully applied the LoRA, it has no effect). If you can load your model with ExLlamav2 instead, you can apply the LoRA there.

Or you can try the code change I discovered to make applying LoRAs to Transformers work:
#5182 (comment)

In summary: For 4-bit GPTQ, train with the model loaded with Transformers, apply the resulting LoRA to the model loaded with ExLlamav2.

[jaqenwang]

thanke reply, actually seem still no effect with ExLlamav2
This is my model: "Llama-2-13B-GPTQ " main; I am using raw txt as LORA input

Anyway, i will try your code change on Transformers;

Thanks
Hui

[jaqenwang]

it always doesn't take effect ... looks something get wrong
i can raise a thread for the issue..

[araleza]

First, I'd try increasing your training rate and then making the loss go really low in your LoRA. If the loss isn't low enough I don't see much effect from LoRAs I train at all.

[jaqenwang]

thanks for the suggestions! it helps, i will double check the data

[DasBinNichtIch]

Thank you so much :)

[DasBinNichtIch]

Another quick question, if I train the LoRA with Transformers and want to apply it with ExLlamav2_HF should that work or can I only use it with the Transformers Model Loader?

[araleza]

You can definitely train with Transformers and apply the resulting LoRA to the model reloaded with ExLlamav2_HF (or ExLlamav2), cause that's what I do, cause the inference is faster that way.

I can only get the 4 bit GPTQ quants to work reliably though, the 8 bit ones don't seem to work that way.

[DasBinNichtIch]

Really? Cause I was doing that and my AIs output was absolutely corrupted:

I was using my LoRA which somehow had a train loss of 4.5308. After training a new LoRA with a shorter dataset, I had a train loss of 2.4669 and suddenly using ExLlamav2_HF worked. Anything you know about that or should I just try again with different modifications?

EDIT: Maybe that was because I only diabled "disable_exllamav2" and not "disable_exllama" too for the Transformers Model Loader.