add modules_in_block_to_quantize arg in GPTQconfig
#27956
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
amyeroberts
reviewed
Dec 12, 2023
There was a problem hiding this comment.
Thanks for adding and linking to the corresponding PR in optimum!
Overall looks OK - agree with @ArthurZucker that docstring needs some clarity
SunMarc
changed the title
modules_in_block_to_quantize arg in GPTQconfig
Co-authored-by: Arthur <[email protected]>
amyeroberts
reviewed
Dec 13, 2023
Comment on lines
383
to
386
| modules_in_block_to_quantize (`List[List[str]]`, *optional*): | ||
| List list of module names to quantize in the block specified. This argument is useful to exclude certain linear modules from being quantized. | ||
| The block to quantize can be specified by setting `block_name_to_quantize`. We will quantize each list sequentially. If not set, we will quantize all linear layers. | ||
| Example: `inside_layer_modules=[["self_attention.query_key_value"], ["mlp.dense_h_to_4h"]]` |
There was a problem hiding this comment.
What does each list in the list represent here? E.g. is the first element in the list - modules_in_block_to_quantize[0] - the list of layers to quantize for the first block? How does that match with block_name_to_quantize which seems to only take a single string i.e. I would expect a single block.
There was a problem hiding this comment.
Yes, I will detail a little bit more why we are using a list of list. See comment above.
amyeroberts
approved these changes
Dec 13, 2023
Co-authored-by: amyeroberts <[email protected]>
ArthurZucker
approved these changes
Dec 13, 2023
iantbutler01
pushed a commit
to BismuthCloud/transformers
that referenced
this pull request
Dec 16, 2023
* add inside_layer_modules arg * fix * change to modules_to_quantize_inside_block * fix * remane again * Apply suggestions from code review Co-authored-by: Arthur <[email protected]> * better docsting * fix again with less explanation * Update src/transformers/utils/quantization_config.py Co-authored-by: amyeroberts <[email protected]> * style --------- Co-authored-by: Arthur <[email protected]> Co-authored-by: amyeroberts <[email protected]>
staghado
pushed a commit
to staghado/transformers
that referenced
this pull request
Jan 15, 2024
* add inside_layer_modules arg * fix * change to modules_to_quantize_inside_block * fix * remane again * Apply suggestions from code review Co-authored-by: Arthur <[email protected]> * better docsting * fix again with less explanation * Update src/transformers/utils/quantization_config.py Co-authored-by: amyeroberts <[email protected]> * style --------- Co-authored-by: Arthur <[email protected]> Co-authored-by: amyeroberts <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR adds the
modules_in_block_to_quantizequantization arg for gptq. This is necessary for converting specific layers to quantized layers. With this PR, we should be able to run the gptq mixtral model. See related PR in optimum.