-
Notifications
You must be signed in to change notification settings - Fork 188
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add gemlite kernel option to autoquant #1449
Conversation
Summary: att Test Plan: python generate.py --checkpoint_path $CHECKPOINT_PATH/$MODEL_REPO/model.pth --compile --compile_prefill --quantization autoquant-gemlite-int4 --output_json_path result.json Reviewers: Subscribers: Tasks: Tags:
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/1449
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit dacec8b with merge base 1beb6d0 (): This comment was automatically generated by Dr. CI and updates every 15 minutes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you Jerry! Overall looks ok to me, I haven't tested it though and I am not familar with autoquant api, so would make sense if Charles could look at it as well.
@@ -46,15 +46,14 @@ def get_gemlite_quant_kwargs(bit_width, group_size): | |||
return kwargs | |||
|
|||
|
|||
def apply_gemlite_quant( | |||
def get_gemlite_aqt_kwargs( | |||
weight, | |||
group_size=64, | |||
bit_width=4, | |||
packing_bitwidth=8, | |||
contiguous=None, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this one is using packing_bitwidth=8
by default, while the other functions are using 32. Ideally this one should be using 32 too
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will change, looks like it should be OK
Summary:
Refactored gemlite quant api a bit and added gemlite as an option to autoquant:
GEMLITE_INT4_AUTOQUANT_CLASS_LIST
and alsoALL_AUTOQUANT_CLASS_LIST
Test Plan:
python generate.py --checkpoint_path $CHECKPOINT_PATH/$MODEL_REPO/model.pth --compile --compile_prefill --precision float16 --quantization autoquant-gemlite-int4
(gemlite only works for float16)
Reviewers:
Subscribers:
Tasks:
Tags: