lora : error message if new token is added in the adapter

[ngxson]

Related to #9778

Explanation from TRL team member:

There can be situations where PEFT will automatically save the embedding layer when it detects that it has been changed (e.g. new tokens being added). However, this should save the full embedding layer, not lora_A and lora_B.

In llama.cpp, we simply can't support this case because it will break adapter hot-reload (cannot switch lm_head or embd_tokens at runtime). Also, if multiple adapters each has its own set of additional token, we can't mix and match these adapters together.

Solution

When fine-tuning your model using TRL, please make sure:

• Not to add any token
• Not to call setup_chat_format()

---

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low

[slaren]

In llama.cpp, we simply can't support this case because it will break adapter hot-reload

I think we can support this by loading the full tensor from the LoRA and replacing it in the model, it shouldn't be a problem. Keep a copy to the previous tensor pointer to restore it when unloading the LoRA, and reject applying another LoRA if it also has a full copy of the same tensor. Or better, just modify the LoRA applying functions like llm_build_lora_mm to use the replacement tensor. Dealing with the changes to the tokenizer may be a more difficult problem, however.

[ngxson]

In fact, I'm thinking of the case where each adapter has its own set of added tokens. This will be impossible because now each adapter has its own definition of the added token.

In any cases, I don't think user should add a new tokens to LoRA since the their embeddings will not be trained. In fact, these embeddings will be initialized to random vectors.

(I'm merging this PR once @Victoran0 confirms that removing setup_chat_format fixes the problem)

[Victoran0]

Yes @ngxson , removing setup_chat_format resolved the error, the lora adapter's size is now less than 100mb.
This commit is optimal as it tells the user where the problem is coming from and what to fix. Great job!