Skip to content

Commit

Permalink
Update src/transformers/training_args.py
Browse files Browse the repository at this point in the history
Co-authored-by: Zach Mueller <[email protected]>
  • Loading branch information
helloworld1 and muellerzr authored Apr 18, 2024
1 parent 5443e5b commit 6404b3f
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions src/transformers/training_args.py
Original file line number Diff line number Diff line change
Expand Up @@ -483,8 +483,8 @@ class TrainingArguments:
If `"True"`, each individually wrapped FSDP unit will broadcast module parameters from rank 0 to
ensure they are the same across all ranks after initialization
- cpu_ram_efficient_loading (`bool`, *optional*, defaults to `False`)
If `"True"`, only the first process loads the pretrained model checkpoint while all other processes
have empty weights. When this setting is True, `sync_module_states` also must to be True,
If `True`, only the first process loads the pretrained model checkpoint while all other processes
have empty weights. When this setting is `True`, `sync_module_states` also must to be `True`,
otherwise all the processes except the main process would have random weights leading to unexpected
behaviour during training.
- activation_checkpointing (`bool`, *optional*, defaults to `False`):
Expand Down

0 comments on commit 6404b3f

Please sign in to comment.