Skip to content

Commit

Permalink
add brief introduction
Browse files Browse the repository at this point in the history
  • Loading branch information
jiachunp committed Nov 15, 2024
1 parent 9118612 commit 2479b21
Showing 1 changed file with 6 additions and 1 deletion.
7 changes: 6 additions & 1 deletion deepspeed/runtime/zero/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -304,7 +304,12 @@ class DeepSpeedZeroConfig(DeepSpeedConfigModel):
"""
zeropp_loco_param: Optional[Dict[str, Any]] = None
"""
loco-zero++ params
This dictionary contains parameters for using LoCo-Zero++, with two key parameters:
- `err_beta`: A coefficient for the moving average of quantization errors before and after gradient computation.
It ranges between 0 and 1, with a default value of 0.8.
- `reset_T`: The number of steps after which the moving average error buffer is cleared. Its default value is 1024.
These parameters can be adjusted based on performance needs. Example configuration in ds config:
"zeropp_loco_param": { "err_beta": 0.8, "reset_T": 1024}
"""

mics_shard_size: int = Field(-1, new_param="mics_shard_size")
Expand Down

0 comments on commit 2479b21

Please sign in to comment.