Llama-2-7B-QServe model doesn't give the expected output #11
Comments
|
Hi @MuYu-zhi, Your observation is probably relate to the fact that Llama-2-7B is not an instruction-tuned model. Besides, we currently apply greedy decoding in our implementation for simplicity. Adding repetition penalty to the sampler will definitely alleviate this problem. Regarding other problems, I cannot guarantee that the last hidden states of the quantized model will look similar to the original model. This is because output error incurred by quantization will propagate across the model, from the first layer to the end, but I will expect that the output distribution should be similar. Best, |
|
@kentang-mit thanks for your reply. Hope you can help clarify a few more questions：
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I run
qserve_e2e_generation.pywith my own prompts, rather than the originalWildChatdataset, the outputs seem just to repeat the input prompt until the size ofmax_token.Two input prompts among the all five are:
and
sampling_params = SamplingParams(n=1, top_p=1.0, top_k=50, temperature=1.0, stop_token_ids=[128001, 128009], max_tokens=1024, )A output log snippet is as follow:
The text was updated successfully, but these errors were encountered: