Skip to content

Commit

Permalink
fix docstrings in zamba config
Browse files Browse the repository at this point in the history
  • Loading branch information
pglorio authored Oct 4, 2024
1 parent fabaaec commit 75f0d89
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/transformers/models/zamba/configuration_zamba.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,9 @@ class ZambaConfig(PretrainedConfig):
Number of hidden layers in the model.
num_attention_heads (`int`, *optional*, defaults to 16):
Number of attention heads for each attention layer in the Transformer decoder.
num_key_value_heads (`int`, *optional*, defaults to 16):
attention_head_dim (`int`, *optional*):
Dimension of the attention head in the Transformer decoder.
num_key_value_heads (`int`, *optional*, defaults to 16):
This is the number of key_value heads that should be used to implement Grouped Query Attention. If
`num_key_value_heads=None`, the model will use Multi Head Attention (MHA), if
`num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
Expand Down

0 comments on commit 75f0d89

Please sign in to comment.