-
Notifications
You must be signed in to change notification settings - Fork 116
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add hybrid quantization for StableDiffusion pipelines #584
Conversation
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot for your work @l-bat
compressed_model = _weight_only_quantization(model, quantization_config) | ||
|
||
quantized_model = nncf.quantize( | ||
compressed_model, | ||
dataset, | ||
model_type=nncf.ModelType.TRANSFORMER, | ||
ignored_scope=nncf.IgnoredScope(**ptq_ignored_scope), | ||
advanced_parameters=nncf.AdvancedQuantizationParameters(AdvancedSmoothQuantParameters(matmul=-1)), | ||
subset_size=quantization_config.subset_size, | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why do we need to do this ? could you explain a bit by adding comments + explaining what is meant by hybrid quantization ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added docstring to _hybrid_quantization
.
On the one hand, post-training quantization of the UNet model requires leads to accuracy drop. On the other hand, the weight compression doesn't improve performance when applying to Stable Diffusion models, because the size of activations is comparable to weights. That is why the proposal is to apply quantization in hybrid mode which means that we quantize: (1) weights of MatMul and Embedding layers and (2) activations of other layers.
raise NotImplementedError(f"Quantization in hybrid mode is not supported for {cls.__name__}") | ||
|
||
num_inference_steps = 4 if isinstance(cls, OVLatentConsistencyModelPipeline) else 50 | ||
quantization_config.dataset = dataset |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here you modify the value in the quantization_config
object that the user passed to the API. This is not good, IMO. I would recommend passing dataset
explicitly to _hybrid_quantization()
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
created deepcopy of quantization_config
to pass it to _hybrid_quantization
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think it is safe enough to create a deep copy. Imagine someone passes a huge dataset inside the config. We will copy it as far as I understand.
This looks very interesting, thanks @l-bat ! Can you add this to the documentation? https://github.com/huggingface/optimum-intel/blob/main/docs/source/optimization_ov.mdx It would also be good to add a note with a link to this example https://github.com/huggingface/optimum-intel/tree/main/examples/openvino/stable-diffusion (for people who find that when looking for quantizing stable diffusion, to let them know about this option) |
docs/source/optimization_ov.mdx
Outdated
@@ -69,6 +69,23 @@ from optimum.intel import OVModelForCausalLM | |||
model = OVModelForCausalLM.from_pretrained(model_id, load_in_8bit=True) | |||
``` | |||
|
|||
## Hybrid quantization | |||
|
|||
Traditional optimization methods like post-training 8-bit quantization do not work for Stable Diffusion models because accuracy drops significantly. On the other hand, weight compression does not improve performance when applied to Stable Diffusion models, as the size of activations is comparable to weights. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Traditional optimization methods like post-training 8-bit quantization do not work for Stable Diffusion models because accuracy drops significantly. On the other hand, weight compression does not improve performance when applied to Stable Diffusion models, as the size of activations is comparable to weights. | |
Traditional optimization methods like post-training 8-bit quantization do not work well for Stable Diffusion models and can lead to poor generation results. On the other hand, weight compression does not improve performance significantly when applied to Stable Diffusion models, as the size of activations is comparable to weights. |
docs/source/optimization_ov.mdx
Outdated
|
||
Traditional optimization methods like post-training 8-bit quantization do not work for Stable Diffusion models because accuracy drops significantly. On the other hand, weight compression does not improve performance when applied to Stable Diffusion models, as the size of activations is comparable to weights. | ||
The UNet model takes up most of the overall execution time of the pipeline. Thus, optimizing just one model brings substantial benefits in terms of inference speed while keeping acceptable accuracy without fine-tuning. Quantizing the rest of the diffusion pipeline does not significantly improve inference performance but could potentially lead to substantial degradation of accuracy. | ||
Therefore, the proposal is to apply quantization in hybrid mode for the UNet model and weight-only quantization for other pipeline components. The hybrid mode involves the quantization of weights in MatMul and Embedding layers, and activations of other layers, facilitating accuracy preservation post-optimization while reducing the model size. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Therefore, the proposal is to apply quantization in hybrid mode for the UNet model and weight-only quantization for other pipeline components. The hybrid mode involves the quantization of weights in MatMul and Embedding layers, and activations of other layers, facilitating accuracy preservation post-optimization while reducing the model size. | |
Therefore, the proposal is to apply quantization in *hybrid mode* for the UNet model and weight-only quantization for the rest of the pipeline components. The hybrid mode involves the quantization of weights in MatMul and Embedding layers, and activations of other layers, facilitating accuracy preservation post-optimization while reducing the model size. |
docs/source/optimization_ov.mdx
Outdated
Traditional optimization methods like post-training 8-bit quantization do not work for Stable Diffusion models because accuracy drops significantly. On the other hand, weight compression does not improve performance when applied to Stable Diffusion models, as the size of activations is comparable to weights. | ||
The UNet model takes up most of the overall execution time of the pipeline. Thus, optimizing just one model brings substantial benefits in terms of inference speed while keeping acceptable accuracy without fine-tuning. Quantizing the rest of the diffusion pipeline does not significantly improve inference performance but could potentially lead to substantial degradation of accuracy. | ||
Therefore, the proposal is to apply quantization in hybrid mode for the UNet model and weight-only quantization for other pipeline components. The hybrid mode involves the quantization of weights in MatMul and Embedding layers, and activations of other layers, facilitating accuracy preservation post-optimization while reducing the model size. | ||
For optimizing the Stable Diffusion pipeline, utilize the `quantization_config` to define optimization parameters. To enable hybrid quantization, specify the quantization dataset in the `quantization_config`; otherwise, weight-only quantization in specified precisions will be applied to UNet. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For optimizing the Stable Diffusion pipeline, utilize the `quantization_config` to define optimization parameters. To enable hybrid quantization, specify the quantization dataset in the `quantization_config`; otherwise, weight-only quantization in specified precisions will be applied to UNet. | |
The `quantization_config` is utilized to define optimization parameters for optimizing the Stable Diffusion pipeline. To enable hybrid quantization, specify the quantization dataset in the `quantization_config`. Otherwise, weight-only quantization to a specified data type (8 tr 4 bits) is applied to UNet model. |
ratio (`float`, *optional*, defaults to 1.0): | ||
dataset (`str or List[str]`, *optional*): | ||
The dataset used for data-aware compression or quantization with NNCF. You can provide your own dataset | ||
in a list of string or just use the the one from the list ['wikitext2','c4','c4-new','ptb','ptb-new'] for LLLMs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in a list of string or just use the the one from the list ['wikitext2','c4','c4-new','ptb','ptb-new'] for LLLMs | |
in a list of strings or just use the one from the list ['wikitext2','c4','c4-new','ptb','ptb-new'] for LLLMs |
dataset (`str or List[str]`, *optional*): | ||
The dataset used for data-aware compression or quantization with NNCF. You can provide your own dataset | ||
in a list of string or just use the the one from the list ['wikitext2','c4','c4-new','ptb','ptb-new'] for LLLMs | ||
or ['conceptual_captions','laion/220k-GPT4Vision-captions-from-LIVIS','laion/filtered-wit'] for SD models. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
or ['conceptual_captions','laion/220k-GPT4Vision-captions-from-LIVIS','laion/filtered-wit'] for SD models. | |
or ['conceptual_captions','laion/220k-GPT4Vision-captions-from-LIVIS','laion/filtered-wit'] for diffusion models. |
raise ValueError( | ||
f"""You have entered a string value for dataset. You can only choose between | ||
['wikitext2','c4','c4-new','ptb','ptb-new'], but we found {self.dataset}""" | ||
{llm_datasets} for LLLMs or {stable_diffusion_datasets} for SD models, but we found {self.dataset}""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
{llm_datasets} for LLLMs or {stable_diffusion_datasets} for SD models, but we found {self.dataset}""" | |
{llm_datasets} for LLLMs or {stable_diffusion_datasets} for diffusion models, but we found {self.dataset}""" |
# load the UNet model uncompressed to apply hybrid quantization further | ||
unet = cls.load_model(unet_path) | ||
# Apply weights compression to other `components` without dataset | ||
quantization_config.dataset = None |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the error-prone approach when you change the values of the input argument, IMO. Please think about how we can make it more safe.
QAT example is outdated and will be revised in the next PR. After that I will add a note with a link to this example. |
q_config_params = quantization_config.__dict__ | ||
wc_params = {param: value for param, value in q_config_params.items() if param != "dataset"} | ||
wc_quantization_config = OVWeightQuantizationConfig.from_dict(wc_params) | ||
else: | ||
wc_quantization_config = quantization_config | ||
unet = cls.load_model(unet_path, wc_quantization_config) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
q_config_params = quantization_config.__dict__ | |
wc_params = {param: value for param, value in q_config_params.items() if param != "dataset"} | |
wc_quantization_config = OVWeightQuantizationConfig.from_dict(wc_params) | |
else: | |
wc_quantization_config = quantization_config | |
unet = cls.load_model(unet_path, wc_quantization_config) | |
weight_quantization_params = {param: value for param, value in quantization_config.__dict__.items() if param != "dataset"} | |
weight_quantization_config = OVWeightQuantizationConfig.from_dict(weight_quantization_params) | |
else: | |
weight_quantization_config = quantization_config | |
unet = cls.load_model(unet_path, weight_quantization_config) |
@echarlaix, PR is ready and we will update the SD example from QAT to the new approach in the follow-up PR. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great thanks @l-bat
* Add hybrid quantization for StableDiffusion pipelines * apply black * fix tests * fix ruff * fix lcm bug * apply review comments * rework dataset processing * Add doc * remove SDXL test * Apply comments * reformat
…tension-for-transformers. (#455) * Support weight-only quantization with quantized operators in intel-extension-for-transformers * Update code style * Update readme for weight-only quantization example * Update code * Adapt intel-extension-for-transformers 1.3 API change Signed-off-by: Cheng, Penghui <[email protected]> * Support weight-only quantization with quantized operators in intel-extension-for-transformers * Update code * rebase code on main branch Signed-off-by: Cheng, Penghui <[email protected]> * Update example Signed-off-by: Cheng, Penghui <[email protected]> * Update optimum/intel/neural_compressor/quantization.py Co-authored-by: Ella Charlaix <[email protected]> * [OV]: Fixed inference after 4 bit weight compression (#569) * [OV]: Fixed inferece after 4 bit weight compression * Fixed issue * Update optimum/intel/openvino/modeling_decoder.py Co-authored-by: Ella Charlaix <[email protected]> * Applied comments * Fixed issue when request is None --------- Co-authored-by: Ella Charlaix <[email protected]> * Updated docs with load_in_4bit (#558) * Updated docs with load_in_4bit * Update documentation * Update documentation * typo --------- Co-authored-by: Ella Charlaix <[email protected]> * Update Transformers dependency requirements (#571) * Fix compatibility for latest transformers release (#570) * fix compatibility for latest transformers release * update setup * update setup * fix test input size * fix prepare generation for llama models * Deprecate compression options (#565) * deprecate compression options * style * fix configuration * Update CLI argument * update documentation * deprecate torch nn modules for ov quantizer * fix ov config for fp32 models * fix format * update documentation * Add check for configuration * fix ratio default value for SD models * add quantization_config argument for OVModel * remove commented line * Update docs/source/inference.mdx Co-authored-by: Alexander Kozlov <[email protected]> * add default config for causal LM * fix warning message --------- Co-authored-by: Alexander Kozlov <[email protected]> * Add default quantization int4 config for Mixtral-8x7B (#576) * Update stable diffusion example requirements (#579) * Fix collecting duplicate tensors in quantization calibration dataset (#577) * Added deepcopying of inputs collected by InferRequestWrapper. Added a test covering the fixed issue. * Phrasing tweaks * Add soundfile to test requirements * Added librosa to test requirements * Added copying to other data cache appends * Remove the need for real test data * Process __call__ call properly * Addressed suggested changes * Save an openvino config summarizing all information related to quantization when saving model (#578) * fix doc * remove default compression value * set default compression config when not provided * save openvino config to include quantization configuration * fix style * add test * update setup * style * remove from quantization_config key from ov_config * add test * update setup * modify method name * Fix warning (#582) * Fix warning * fix message warning * Add reference to the temporary directory for windows fix (#581) * Fix documentation (#583) * Fix documentation * fix * Add llama test model to cover MQA (#585) * change llama test model to cover MQA * keep llama and llama2 in tests * fix code style * Include nncf in openvino extra (#586) * Fix title documentation (#588) * Update OpenVINO documentation links in README.md (#587) * Update OpenVINO documentation links in README.md The links are now aligned with OpenVINO 2024.0 documentation, and include permalinks instead of direct links, when possible. * Update inference.mdx * Update index.mdx * Update installation.mdx * Update README.md * Fix default int8 quantization for CLI (#592) * Change model output parameter to last_hidden_states for IPEXModel (#589) * change model output parameter to last_hidden_states * update ipex model testiong * update testing * add output name to ipex model * Add IPEX model patcher (#567) * llama model patcher * fix jit model * fix jit model * rm autocast in model * add llama model patcher * support assisted decoding and add reorder cache function * add comment for _prepare_past_key_values * rebase main * fix model_dtype * rm useless comments * fix llama * add comments for ipex_rope and ipex_scale_dot_product * fix comments * add enable_tpp comments * fix import * fix review aroun2 * add torch.no_grad to avoid auto_kernel_selection issue * use torch.no_grad in jit trace * fix ipex model testing * add tests for ipex model generation with multi inputs * fix code style * remove __get__(self) as _reorder_cache is static method for the class * fix reorder_cache * use model_type * check if reorder_cache is a static method * fix _reorder_cache * fix raise import error * test ipex patching * fix comments * update API name and testing * disable untill ipex version 2.5.0 * update testing name * Update optimum/intel/ipex/modeling_base.py Co-authored-by: Ella Charlaix <[email protected]> * Update tests/ipex/test_modeling.py Co-authored-by: Ella Charlaix <[email protected]> * fix tests --------- Co-authored-by: Ella Charlaix <[email protected]> * Updates weight quantization section in the docs (#593) * Remove accelerate and onnxruntime from required dependencies (#590) * Remove accelerate dependency * Add accelerate to import backend mapping * Add eval method to OVModels * add onnxruntime install for OV test * fix test expected int8 * Fix OpenVINO image classification examples (#598) * Fix weights compression for OPenVINO models (#596) * hot fix for weights compression * rewrite mcok tests * Fix default ov config (#600) * Add warning for transformers>=4.38 and OpenVINO 2024.0 (#599) * Add warning for transformers>=4.38 and OpenVINO 2024.0 * Use is_openvino_version to compare versions * Show version warning only for llama and gpt-bigcode * Fix style, show OpenVINO version * Include affected model types in warning message * Add hybrid quantization for StableDiffusion pipelines (#584) * Add hybrid quantization for StableDiffusion pipelines * apply black * fix tests * fix ruff * fix lcm bug * apply review comments * rework dataset processing * Add doc * remove SDXL test * Apply comments * reformat * Show device name in _print_compiled_model_properties (#541) * Show device name in _print_compiled_model_properties Enable CACHE_DIR also for devices like "GPU:0" * Update optimum/intel/openvino/modeling_seq2seq.py Co-authored-by: Ella Charlaix <[email protected]> * Change check for gpu device --------- Co-authored-by: Ella Charlaix <[email protected]> * Update code with comments Signed-off-by: Cheng, Penghui <[email protected]> * Fixed pylint error Signed-off-by: Cheng, Penghui <[email protected]> * Update optimum/intel/neural_compressor/configuration.py Co-authored-by: Ella Charlaix <[email protected]> * Fixed example and UT for weight-only quantization Signed-off-by: Cheng, Penghui <[email protected]> * Fixed pre-ci test error Signed-off-by: Cheng, Penghui <[email protected]> * Fixed pre-ci test error Signed-off-by: Cheng, Penghui <[email protected]> * Fixed UT and examples error Signed-off-by: Cheng, Penghui <[email protected]> * Fixed pre-CI error Signed-off-by: Cheng, Penghui <[email protected]> * Fixed UT error Signed-off-by: Cheng, Penghui <[email protected]> * Update tests/openvino/test_modeling_basic.py Co-authored-by: Ella Charlaix <[email protected]> * Update examples/neural_compressor/language-modeling/README.md Co-authored-by: Ella Charlaix <[email protected]> * Update examples/neural_compressor/language-modeling/run_clm.py Co-authored-by: Ella Charlaix <[email protected]> * Update examples/neural_compressor/language-modeling/run_clm.py Co-authored-by: Ella Charlaix <[email protected]> * Update examples/neural_compressor/language-modeling/run_clm.py Co-authored-by: Ella Charlaix <[email protected]> * Update examples/neural_compressor/language-modeling/run_clm.py Co-authored-by: Ella Charlaix <[email protected]> * Update examples/neural_compressor/language-modeling/run_clm.py Co-authored-by: Ella Charlaix <[email protected]> * Load weight-only quantized model with INCModelForCausalLM Signed-off-by: Cheng, Penghui <[email protected]> * Changed parameters name for GPTQ in example Signed-off-by: Cheng, Penghui <[email protected]> * Changed parameters order in INCQuantizer.quantize Signed-off-by: Cheng, Penghui <[email protected]> * Fixed UT error Signed-off-by: Cheng, Penghui <[email protected]> * Update examples/neural_compressor/text-generation/run_generation.py Co-authored-by: Ella Charlaix <[email protected]> * Update optimum/intel/neural_compressor/quantization.py Co-authored-by: Ella Charlaix <[email protected]> * Update optimum/intel/neural_compressor/quantization.py Co-authored-by: Ella Charlaix <[email protected]> * Update import message Signed-off-by: Cheng, Penghui <[email protected]> * Limit intel-extension-for-transformers version Signed-off-by: Cheng, Penghui <[email protected]> * Limit torch version for weight-only quantization Signed-off-by: Cheng, Penghui <[email protected]> * Fixed doc building error Signed-off-by: Cheng, Penghui <[email protected]> --------- Signed-off-by: Cheng, Penghui <[email protected]> Co-authored-by: Ella Charlaix <[email protected]> Co-authored-by: Alexander Kozlov <[email protected]> Co-authored-by: Ella Charlaix <[email protected]> Co-authored-by: Lyalyushkin Nikolay <[email protected]> Co-authored-by: Helena Kloosterman <[email protected]> Co-authored-by: Nikita Savelyev <[email protected]> Co-authored-by: jiqing-feng <[email protected]> Co-authored-by: Ekaterina Aidova <[email protected]> Co-authored-by: Karol Blaszczak <[email protected]> Co-authored-by: Liubov Talamanova <[email protected]>
What does this PR do?
OVStableDiffusionPipeline
,OVStableDiffusionXLPipeline
andOVLatentConsistencyModelPipeline
, when the part of the model is fully quantized and the weights of another part are just compressed['conceptual_captions','laion/220k-GPT4Vision-captions-from-LIVIS','laion/filtered-wit']
Before submitting