Adding FSDP Support to Training Library#213

Merged

Maxusmusti merged 47 commits intomainfrom ap/accelerate-fsdp-tmp2

Sep 26, 2024

+660-333

Commits on Sep 25, 2024

increased performance of data process and made a flag for the number of parallel processes

aldo-pareja
authored and
Maxusmusti
committed
Refactor special tokens handling and improve chat template flexibility

aldo-pareja
authored and
Maxusmusti
committed
Refactor data processing and tokenization

aldo-pareja
authored and
Maxusmusti
committed
added MixtralForCausalLM as another supported model

aldo-pareja
authored and
Maxusmusti
committed
iterim branch exploring accelerate from fix_mistral_template

aldo-pareja
authored and
Maxusmusti
committed
accelerate testing

aldo-pareja
authored and
Maxusmusti
committed
fixed data process

aldo-pareja
authored and
Maxusmusti
committed
accelerate works on deepspeed

aldo-pareja
authored and
Maxusmusti
committed
fsdp first try

aldo-pareja
authored and
Maxusmusti
committed
testing both fsdp and deepspeed

aldo-pareja
authored and
Maxusmusti
committed
made deepspeed the default

aldo-pareja
authored and
Maxusmusti
committed
black formatting

aldo-pareja
authored and
Maxusmusti
committed
fixed typo on batch size logging

aldo-pareja
authored and
Maxusmusti
committed
added samples seen to the logging

aldo-pareja
authored and
Maxusmusti
committed
fsdp needs a different optimizer workflow

aldo-pareja
authored and
Maxusmusti
committed
black formatting

aldo-pareja
authored and
Maxusmusti
committed
removed weight decay from the fsdp optimizer

aldo-pareja
authored and
Maxusmusti
committed
fixed a merge conflict

aldo-pareja
authored and
Maxusmusti
committed
aligning with the latest

aldo-pareja
authored and
Maxusmusti
committed
fixing linting errors

aldo-pareja
authored and
Maxusmusti
committed
update: rename arguments and add config options in trainingargs

RobotSail
authored and
Maxusmusti
committed
update init.py

RobotSail
authored and
Maxusmusti
committed
bug fixes

RobotSail
authored and
Maxusmusti
committed
Added cpu offloading for fsdp
Maxusmusti
committed
Move sharding strategy to fsdp options
Maxusmusti
committed
Add lora ckpt saving (first pass)
Maxusmusti
committed
Re-add save per epoch
Maxusmusti
committed
Adding lora/qlora saving pt2
Maxusmusti
committed
Connect new lora save patch
Maxusmusti
committed
Fix lora ckpt saving
Maxusmusti
committed
Load/save fixes assuming rc is allowed
Maxusmusti
committed
Fix for deepspeed accelerate reqs
Maxusmusti
committed
Add accelerate 0.34 patch for saving
Maxusmusti
committed
Re-add dolomite conversion for ckpts
Maxusmusti
committed
Clean up extraneous stuff
Maxusmusti
committed
Remove breaking check
Maxusmusti
committed
Feedback round 1
Maxusmusti
committed
Feedback round 2
Maxusmusti
committed
Make sure model is passed back in case it is updated in setup_optimizer
Maxusmusti
committed
Remove branching prepare statements
Maxusmusti
committed
Add note about commented data process code
Maxusmusti
committed
Adding more notes for code readers
Maxusmusti
committed
minor bug fixes & improvements
RobotSail
committed
update docs to include fsdp info, rename TrainingArgs.distributed_training_backend to TrainingArgs.distributed_backend and DistributedTrainingBackend to DistributedBackend
RobotSail
committed
Lower max shard size
Maxusmusti
committed
Remove extra comment
Maxusmusti
committed

Commits on Sep 26, 2024

Enum security
Maxusmusti
committed