Gptq tokenized dataset

[SunMarc]

What does this PR do ?

This PR allow to pass tokenized dataset for gptq quantization

[fxmarty]

#1585 supersede this, right?

[SunMarc]

Thanks for having a look @fxmarty ! No this is a functionality requested by @TheBloke. The quantization in transformers is quite slow compared to AutoGPTQ, maybe because of the dataset processing. So we allow tokenized dataset. My hunch personally is that with modules_in_block_to_quantize, we should have the same speed now. This is because we are quantizing one layer at a time when we don't set modules_in_block_to_quantize

[TheBloke]

Yeah I asked for it so I could have complete control over the dataset when using Transformers to make a GPTQ.

With AutoGPTQ, I have this control, because I can tokenise the dataset myself and then pass this to AutoGPTQ to use.

I use this to pick context-length appropriate samples. Eg for a 4096 model, I will pass 128 x 4096 token samples.

With Transformers I could never do this, I just have to pass List[str], and I wasn't sure what data exactly was being used. So I just passed 5000 x strings of various lengths.

Transformers was also much slower at making GPTQs than AutoGPTQ, and I thought these facts might be connected - although based on what Marc said here, maybe that's for other reasons?

Anyway, even if it's not the cause of the speed difference, it's great that I'll now be able to have full control over the dataset so I can ensure I send enough data for long context models, but not more than I need. And also now I can bulk tokenise the dataset myself, which I can do very fast.