AI Prompts that slowly wither into repeating nonsense or gibberish.

[viperwasp]

I'm likely going to post this as an actual bug report with actual logs and examples. In a more professional way with less rambling. However for now I'm just going to ramble. Only read if your interested or having the same issue etc.

I have a problem I've been dealing with for about 1-2 weeks. Ever since a recent update. I've updated to current version of posting this and it's not resolved. Using Mixtral 8x7B Instruct. Also present in nous-hermes-2-mixtral-8x7b-dpo.Q5_K_M.

Basically 80% of the time everything goes fairly well. I do notice longer general prompts and more repeating or redundant answers. But particularly when asking it to write a story. It start off great. And than eventually gets stuck into a rut.

It will start just describing someones thoughts. Like he was upset, he was seeing red, he was angry, he was not okay,.... bla bla bla... and sometimes it gets so bad it literally repeats the last 2-3 words over and over. So yeah in general I feel quality of all responses has gone down a tiny bit. But it's majorly noticeable when writing a story but usually in longer conversations.

Even if I re-do the answer it usually breaks down about the same length of tokens into the whatever post started to mess up. But if I copy last answer. And cut the good stuff before it starts to mess up. And than paste it and hit continue it sometime is able to go on from there and re-write the end without the issue. Other times it immediately starts doing it again.

So yes I would describe the issue as longer and more redundant answers to most or all questions. At 20% of the time a long story will just start to melt down. It almost feels like the AI wants to stop typing but a stopping token or something is missing. So it feels it has to go on. And than it melts down either that or it's forgetting what it just typed. Like it's memory is being consumed.

I use "Simple1" but I've tried others none of it really helps. Changed settings I may have "fixed" it once without knowing what I did but it could have just been luck too.

[araleza]

This is just an issue that current LLMs have. I know you said that it seemed to start happening in the last 1-2 weeks, but it's been true for everyone since way back. Bigger models seem to be less susceptible to the issue, and the longer the context, the more likely it seems to be that it starts happening.

There are people that have been investigating how to stop it, like here in this Reddit thread:

https://old.reddit.com/r/LocalLLaMA/comments/1ap8mxh/what_causes_llms_to_fall_into_repetitions_while/

[viperwasp]

Thank you. I'm still sure that something is still different but I can't rule out that I've just been lucky all this time? Yes I've dealt with the rare repeating issue used to be more common. But this feels different. These new models I installed from first use to just a few weeks ago I never had this issue one single time.

I save all of my conversations etc. I can pull up 110 long conversations in row without this. Now depending on how I word things etc. It can happen quite frequently. More I want to say but I can't quite describe it. It's like a thought on the tip of my tongue.

This is different than old repeating issues I've had. But I can still get perfect and long conversations even now. But I looked though that link and thank you very much Araleza. You might be correct and I somehow never ran into it for a long time. If you or anyone else notices anything off with my settings let me k

Windows 10, RTX 4080.

[viperwasp]

I know it's like a 8K or 32K model but I manually lower it to 4K. I've always done that. But it's entirely possible that a bug or something can come along that only affects a specific range of settings. Perhaps I'm just using some different settings. I also notice that presence_penalty, and frequency_penalty are new? How long have those been around for llama.cpp?

[TheLounger]

I'm with you on this, something changed significantly for me as well.
This doesn't seem to be the same thing as araleza mentioned, which is something I had as well ever since I started using LLMs almost a year ago.
I think it started around the beginning of February. Llama-based seems more affected than some more recent Mixtral flavor.
The main issue is that it's extremely repetitive, often even after 2 messages. It copies things verbatim, even within the same message with only 3 sentences. It repeating things over and over again in the same message. It copies your messages verbatim as well.

I'm looking into a way to compare different builds and/or llama-cpp-python versions, and will try to show some examples as well.

[viperwasp]

Thanks and yes I'm bad at explaining things but I am confident we are having the same issue. The same issue but perhaps not the same cause. I kind of hope it is the same cause and that we can figure it out or it can fix it self. But the way you describe it is exactly how I experience it. Sometimes the issue is less present. Sometimes it's absolutely terrible. And like you said it can copy or like reference what I said or asked. Like 2-6 times in it's response. If you look over my settings in the screenshots. Those are fairly default settings I think. Anything that stands out that we are doing similar? Is it pretty much all similar? Have you experienced the issue with like Ctransforms or any model type? Is it only Llama.cpp? Finally do you use alternatives to Oobabooga that are better right now?