Finetuning (Lora)

[kevinintel]

No description provided.

[endomorphosis]

can you be specific about what implementation you want to use for finetuning?

@nisten had finetuned https://huggingface.co/nisten/Biggie-SmoLlm-0.4B on the dual sapphire rapids system using https://github.com/axolotl-ai-cloud/axolotl?tab=readme-ov-file#train he got fp8 training working but i was unable to confirm with him whether the accelerate library was using AMX or AVX as the underlying instruction, and whether there are more performance optimizations that can be made.

The intention is that we will eventually need some sort of guardrail model, but he reports that right now the model training is "10x slower than a nvidia L4 24GB", and he will try to see what further optimizations he can make, and then write a readme for the purposes of OPEA.

[endomorphosis]

[nisten]

I used code that would work with tim dettmers old bnb8bit linear optimizer but bits and bytes only supports that on GPU not cpu.
accelerate config supports mixed-precision float8_bfloat16 and it ran with it but i think it was just running bf16.
Training CPU only opens up someinteresting possibilities with using very very large batch sizes (like 256) in the training parameters.

[kevinintel]

we want to add fine tuning components and it can support LoRA.
But thanks for your sharing, we will create a basic fine tuning components first, you can raise a PR to add functionalities after it.
Thanks

[XinyuYe-Intel]

Finetuning with LoRA is supported in this PR.