update test_run_compressed

[horheynm]

Contingent on merge of huggingface/transformers#34719

SUMMARY:
Update run_compressed tests from decompression tests to run_comrpressed tests -> test if run_compressed True/False models generate the same output

Add decompress tests that copies attrs from the source dir path's model to the target model.

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

[dsikka]

The generations compared are decompressed and decompressed on the forward pass ie compressed linear. So they should be close within a tolerance

[dsikka]

I think you've changed the goal of the test which is compressed linear vs decompressing the whole model. We still want run_compressed to be True for one of the models

[horheynm]

I think you've changed the goal of the test which is compressed linear vs decompressing the whole model. We still want run_compressed to be True for one of the models

Honestly, idk what the original test was doing. If you want to add compressed linear, decompressed model and its model.generate thats cool.

original test is comparing a quantized model and a non-touched base model, that will always fail

[dsikka]

I think you've changed the goal of the test which is compressed linear vs decompressing the whole model. We still want run_compressed to be True for one of the models

Honestly, idk what the original test was doing. If you want to add compressed linear, decompressed model and its model.generate thats cool.

original test is comparing a quantized model and a non-touched base model, that will always fail

1. We load a compressed model:

llm-compressor/tests/llmcompressor/transformers/compression/test_run_compressed.py

Line 29 in 606aab2

cls.compressed_model = AutoModelForCausalLM.from_pretrained(

2. We create an empty model to hold the decompressed weights:

llm-compressor/tests/llmcompressor/transformers/compression/test_run_compressed.py

Line 37 in 606aab2

cls.uncompressed_model = AutoModelForCausalLM.from_pretrained(

3. We then use lines 42-50 to decompress the whole model, using the empty model as the skeleton. This is loading the same checkpoint as what is used by the compressed model.
   

   llm-compressor/tests/llmcompressor/transformers/compression/test_run_compressed.py

   Line 42 in 606aab2

   config = AutoConfig.from_pretrained(cls.model_stub)

We then run generations and compare the two cases. You're comparing decompressed in both cases, one is just decompressing on the forward pass. It's the same checkpoint in both cases, which is why this test passes.

We just want to update such that the uncompressed model can be decompressed using your new transformer changes.
The goal of the test is that compressed linear decompression is the same as the entire model being decompressed up-front.

[horheynm]

Ok i see, we should update the test name, run_compressed_configs is confusing

[dsikka]

Ok i see, we should update the test name, test_run_config is confusing

Yeah cuz of the flag/arg name. We can call it test_compressed_linear_decompress or something

[horheynm]

/ready