Error while fine tuning with peft, lora, accelerate, SFTConfig and SFTTrainer #3230
Comments
|
I'm pretty sure that if you use |
|
The last working code I have (for only 1 GPU) is that (only the model part for clarity): I ran the code with 1 or 4 GPUs without any change (I run the code on a distant server where "slurm" is used so I can easily ask 1 or 4 GPUs for different tasks): 1 GPU output: 4 GPUs output: We can see when I ask 1 GPU it will take 20h of training and when I use 4 GPUs it will take 18h so not a lot of differences. I would hope with 4 GPUs it will last almost 4X less time than with 1 GPU. Also we can see only one GPU is used when I ask 4 GPUs. So apparently SFTTrainer doesn't know how to use all GPUs when there is more than 1 GPU. I also tried that where people advise to add for when they use more than 1 GPU: I have the same result, the train will approximatively take 18-20h like when I use only 1 GPU |
|
After several attempts trying different options, I noticed that my code is indeed using multiple GPUs, but I'm observing some strange behavior. Specifically, when I run my code with 1 GPU, it takes about 19.5hours. When using 4 GPUs, the time drops only slightly to 17.5hours. However, when using 2 GPUs, the runtime is significantly better, around 9 hours, which is actually faster than when I use 3 (13h) or 4 GPUs. The outputs I have: 4 GPU output (~17h30) 3 GPU output (~13h) 2 GPUs output (~9h) 1 GPU output (~19h30) and this is the code used to show VRAM usage: I'm trying to understand why my code performs best with 2 GPUs instead of 4. Additionally, based on my console outputs, it seems that only the first GPU is being used during trainer.train(). I'm wondering how I can verify the GPU utilization from my Python code, since I'm in an environment ( |
|
Glad that you got it running, but I'm not sure why you see the bad scaling behavior. Just minor issue I spotted in your code, but that's unlikely to be the cause: When you pass |
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
System Info
Information
Tasks
no_trainerscript in theexamplesfolder of thetransformersrepo (such asrun_no_trainer_glue.py)Reproduction
Hi,
I try to parallelize training on 4 GPU (v100 32GB VRAM). I have a working code for 1 GPU using lora, peft, SFTConfig and SFTTrainer. I tried to add some lines from accelerate (the lib) as I saw on some tutorials to achieve my goal without success.
This is the error I get (I get it 4 times due to the parallelization, but for more clarity, I put only one occurence):
This is my code (I don’t put all, just the code about the model itself for more clarity):
The only code I added between the 1 GPU and 4 GPU versions is:
And I run the code via a bash script:
The 1 GPU version of this script was:
python script.py --model_path $1 --output $2I also tried at the end of the main function (deleting ‘model = accelerator.prepare_model(model)’):
But this time I have this error:
I tried to do some fixes as discussed on this https://discuss.huggingface.co/t/multiple-gpu-in-sfttrainer/91899
Unfortunately I still have some errors:
This is my code now:
I removed
and I modified my bash script:
Expected behavior
I would like to use accelerate to make my 1 GPU code working with more GPUs
The text was updated successfully, but these errors were encountered: