Bug: phi 3.5 mini produces garbage past 4096 context

[patw]

What happened?

Phi 3.5 mini doesn't produce <|end|> or <|endoftext|> when the context is set higher than 4096, just endless garbage tokens. Possible rope scale issue?

Name and Version

llama-server, recent compile

What operating system are you seeing the problem on?

No response

Relevant log output

No response

[themanyone]

For conversation, the server is working fine with phi-3.5 quantized to 4 bits. But after a while it started outputting tons of blank lines and garbage when told to make a simple HTML page. Hitting the [Reset] button on the chat server's Gradio page, localhost:8080 fixed it for now. It makes great web pages.

The only thing I can guess is that unusual prompt formats from using other models corrupted the chat history somehow. But I have no way to look into the (now cleared) chat history to see. Will keep testing!

[bartowski1182]

Are you using flash attention or not? I've seen that without flash attention the output is garbage, but with its coherent

[patw]

I found with -fa turned on it was running super slow, but also outputting garbage. Right now it's off and stable at 4096.

[ThiloteE]

To do: Test, if fixed by #9396

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.