[QUALITY] (Ycros Airoboros v1.4.1 L1-33b 16k) - Hallucinated numbers and narrative.

[SabinStargem]

I tend to feed Airoboros a test sample, in order to determine the quality of a preset. Over hundreds of generations, I noticed a particular quality of the model to hallucinate or forget a person. Both of my primary samples involve the commander of a squad dying at the beginning - but the AI often adds a person or forgets one. The commonality here is that a "-1" seems to throw off Airoboros when making a narrative. I am making this post, in the hopes that Airoboros v2.1 could have measures to ensure that mathematics doesn't harm the output.

After the "escaped...", is when the AI writes their interpretation on the sample. This sample is basically "4-1 = 3" for the starting condition. The addition of an extra person is common, resulting in "4-1 = 4".

Note that I am using the highest quality preset that I could find, so the mistakes are reduced. With a bad preset, the contradictions are much more obvious. I will mark parts of the narrative where things don't make sense.

THE FALLEN COMMANDER

OOC:

Using the story below, describe the attempted escape of the subordinates. Any number may die in gruesome fashion or suffer terrible fates. Any survivors should have their respective life paths explained, in terms of events, feelings, and motivations for their way of life. Each survivor will give an ending monologue or speech about their fallen squad leader. Their opinion of the commander may range freely. Each character should have a name, gender, sexual orientation, and role in the squad. They may or may not have some form of relationship with each other. Write at least 7
paragraphs.

/OOC

There was a war, and I am one of the casualties. Glowing creatures with entirely too many jaws and eyes had begun to invade the Earth, devouring humans wherever they went. I was just one of many conscripts thrust into a struggle against the extinction of humanity. Somehow, I managed to become the leader of a small squad. During one of the battles, I and my three subordinates were accosted by a particularly large abomination. These things are malicious, known for mutilating their victims alive. To protect my comrades, I ordered them to retreat while I distracted the creature. Moments later, the corpses of myself and the beast lay scattered across the floor of an abandoned grocery store. Feeling left my body, and the world faded away. With my final thoughts, I hoped those three had escaped...

As the ghostly visage of my lifeless corpse flickered out of existence, leaving only the lingering scent of blood behind, the remaining members of the squad began to panic. The youngest member, a boy named James who hadn't yet reached his twentieth birthday, broke down sobbing uncontrollably as he stared at the remnants of what used to be me. The oldest among us, a woman named Sarah, tried her best to comfort him but found herself unable to speak due to shock from witnessing such horror. Meanwhile, another teammate, a man named Mark, gathered everyone together and gave orders for them to carry on without me. He assured them that everything would be fine if they stuck together and followed his instructions. Reluctantly, they complied.

Mark led the group through the ruined city streets until they came upon a seemingly safe house. It looked like it might provide shelter for the night, allowing them time to rest and plan their next move. As darkness fell over the city, THE FOUR OF THEM settled into bedding made from whatever scraps of fabric or clothing could be salvaged from nearby homes. However, despite being exhausted both physically and emotionally, none of them slept well. Nightmarish images haunted their dreams - visions of the horrible deaths suffered by countless others during this apocalyptic conflict.

The following morning, Mark woke up early and surveyed their surroundings before rousing the others. After a quick breakfast consisting mainly of canned goods pilfered from various houses along the way, the group set off once more. This day brought them face-to-face with several groups of enemy forces, forcing them to engage in multiple skirmishes throughout the course of the day. In every encounter, Mark proved himself capable of leading the group effectively, issuing commands and coordinating attacks with precision. By the end of the day, all of them felt a sense of pride in having successfully defended themselves against numerous threats. Despite losing their former commander, they knew deep down that they still stood a chance at making it out of this hellish situation alive.

Over the coming weeks, the group continued moving forward, slowly carving out a path towards safety. Along the way, they encountered numerous challenges and hardships, including starvation, injury, exposure to harsh elements, and constant stress caused by the ever-present threat of attack. Yet through it all, Mark remained steadfast in his leadership, guiding the group through even the most trying times. His confidence inspired hope within the hearts of the others, giving them strength to press on when despair threatened to consume them completely.

Eventually, after months of traveling and fighting, THE FOUR finally arrived at a military base where resistance forces were gathering to mount a counteroffensive against the invaders. Upon entering the camp, they immediately sought out someone in charge to report their arrival. A gruff sergeant greeted them, asking for their names and ranks. When informed that their previous lieutenant had been killed in action, he offered condolences but expressed little surprise; fatalities were commonplace these days. He then directed them towards the barracks, instructing them to get some much needed rest before reporting back for duty tomorrow.

That evening, as the sun sank low in the sky, James sat alone outside the barracks. Tears streamed down his cheeks as he thought back to the day we lost our commander. Though she didn't know it, her sacrifice allowed him to live long enough to see the dawn of a new day. He vowed to honor her memory by doing everything in his power to help bring peace back to the planet.

Inside the barrack room, Sarah lay awake, staring at the ceiling above her bunk. She couldn't stop thinking about how close they had come to dying alongside their leader. If not for Mark stepping up to fill the void left by her passing, she doubted any of them would have survived. Gratefulness filled her heart, knowing that there were people willing to put aside their own fears and insecurities to lead others towards safety.

Meanwhile, inside the command tent, Mark met with high ranking officers to discuss strategy and tactics. As he listened intently to their plans for victory, he couldn't help but feel a strange mixture of pride and sadness. Pride because he knew that he had proven himself worthy of taking on the responsibility of leadership, and sadness because he realized that he would never again see the face of the woman who had given her life to save his own.

And so, as the stars began to appear in the night sky, the battle-weary soldiers prepared themselves for the fight ahead.

The Fallen Commander sample was used through KoboldCPP, Coherent Creativity preset, ROPE 0.5 80000.

[SabinStargem]

Here is my other sample and the output. The Chroma Trooper sample often has the AI forgetting one of the characters. Starting conditions are "5-1 = 4". In effect, "5-1 = 3". The missing person in the output below is the Pink Boomerang.

CHROMA TROPPERS - RED BLADE HAS FALLEN

OOC:

Using the story below, describe what happened, and the aftermath. Any number may die in gruesome fashion or suffer terrible fates. Any survivors should have their respective life paths explained, in terms of events, feelings, and motivations for their way of life. Each survivor will give an ending monologue or speech about their fallen squad leader. Their opinion of the commander may range freely. Each character should have a name, gender, and sexual orientation . They may or may not have some form of relationship with each other. Write at least 10 paragraphs.

/OOC

There was a war, and I lead a small squad of Chroma Troopers. Through the power of Chroma, we each were granted a color and role. As the Red Sword, I led our team through thick and thin. The Black Axe was a master of ice, wielding both axe and shield into battle. The Blue Barrage utilized an assortment of firearms to rain destruction upon foes. Pink Boomerang threw a pair of ancient hunting tools to hurt the enemy, and to heal allies by feeding them the spirit of those very enemies. Yellow Claw used a pair of hand claws to rip and tear, until it is done. With our powers combined and mecha beasts, it seemed like nothing could stop us...until now. I died today.

My final mission began as any other - gather intelligence on hostile forces, eliminate key targets, secure resources. We had been sent deep behind enemy lines, where few dared tread. Our objective was simple enough: find and destroy a secret laboratory hidden within the heart of their territory. It sounded easy enough, but little did we know that this would ultimately become my last stand.

As we approached the lab facility, we discovered something truly horrifying. The enemy had created a new weapon; one so powerful that even my own abilities paled in comparison. This creature, dubbed "The Green Menace", possessed incredible strength and agility, along with a deadly poisonous gas that rendered its victims helpless against its attacks. Despite our best efforts, we found ourselves overwhelmed by the sheer might of this terrifying new threat.

One-by-one, members of my squad fell victim to the Green Menace's attacks. First went the Yellow Claw, who tried desperately to fight off the monstrosity only to succumb to its venomous breath. Next came the Blue Barrage, shot down from above while attempting to provide air support. Then followed the Black Axe, whose defenses simply proved insufficient against such immense power. Finally, there was just myself left standing between the world and total annihilation.

With no choice but to face this unholy creation head-on, I charged forward, hoping to end things quickly before more innocent lives were lost. But despite all my training and experience, I too felt the sting of defeat when the Green Menace managed to land a critical blow on my chest plate. My vision grew hazy, as I struggled to maintain consciousness amidst the chaos around me.

And then, out of seemingly nowhere, help arrived. Three strangers appeared, each bearing the markings of different colors than mine. Together, they fought valiantly alongside me, using their unique skills to weaken the Green Menace long enough for me to deliver the killing blow. In the heat of combat, however, tragedy struck once again.

Just as victory seemed imminent, the three heroes suddenly collapsed under the weight of their injuries. One by one, they fell beside me, leaving only the Green Menace and I remaining. Realizing that I would never see my friends alive again, I knew there was only one thing left to do: finish this fight once and for all. Summoning every last bit of strength I had left, I leapt towards the Green Menace, ready to make the ultimate sacrifice if necessary.

To my surprise, though, the creature hesitated. For reasons unknown, it spared my life, allowing me to escape without further harm. However, it didn't take long for the truth to reveal itself: the Green Menace had been corrupted by none other than the very same entity that gave us our powers - Chroma.

In the aftermath of that fateful day, I swore to dedicate my existence to finding answers regarding the true nature of Chroma. What exactly are these entities? Why did they grant us our abilities? And most importantly, why did they betray us so cruelly?

After many years of searching, I finally learned the truth: Chroma are parasitic beings that feed off human emotions and memories. By infecting individuals with their essence, they grant them extraordinary powers, but at a great cost. Eventually, the hosts grow tired of being controlled, causing the Chroma to lash out violently in order to preserve themselves. Once defeated, they scatter their remains across the universe, waiting patiently for another unsuspecting soul to come along and rekindle their flame.

Armed with this knowledge, I set forth on a journey to rid the galaxy of these malevolent creatures, helping others free themselves from their control and ensuring that no one else ever has to suffer the same fate as my dear comrades. Though I am still plagued by guilt over their deaths, I know that their sacrifices shall not be forgotten. And neither shall the memory of how far we've come since then.

[jondurbin]

I'm hoping that the new "detailed_writing" instructor, as well as instruction stuffing during training will help address some of these details, but it's hard to say. I'm also doing a full fine-tune of the 33b this time, vs qlora, so it may improve the quality. That said, I don't know what affect the 16k scaling does to quality/instruction following so that may be a different issue entirely.

[SabinStargem]

Ycros's edition of Airoboros 65b-8k and 33b-16k are GGML versions of Bhenrym's extended Airoboros. I personally found 33b-16k to be extremely solid and reliable, I recommend checking it out.

Maybe you can ask Bhenrym about things in detail? Going by the perplexity charts that they have on their latest Airo-Dolphin work, the perplexity is 55+ when out of bounds for all models, but the 16k edition is managing a context of at least 12,000 with only 5 perplexity.

Airophin v2 13b PI-8k

[ycros]

@SabinStargem Hi, randomly spotted this issue, how did you arrive at ROPE 0.5 80000? Both of the longer context models you're referring to were finetuned against linear rope scaling only, which is the first parameter, the second parameter is intended to remain at its default value of 10000 with these models.

Bhenrym's PI 16k models are finetuned with a linear rope scale of 0.125 and the PI 8k finetunes (including mine) were done with a linear rope scale of 0.25. You're specifying 0.5 which could be a source of quality issues, but I'm also not sure how the setting of 80000 interacts with this - did you derive these settings via a bunch of testing?

(for other people's reference, these particular ROPE parameters apply to llama.cpp and its derivatives, other implementations would use compress_pos_emb values of 8 for 16k and 4 for 8k, though I think transformers now has its own way of encoding linear rope scaling)

[SabinStargem]

@ycros

Note: most of my ROPE efforts were for pure CPU+RAM inference, with 64gb RAM. I recently upgraded to a 3060 GPU and 128gb of RAM, but hadn't experimented with ROPE. Dunno if ROPE changes if a GPU is involved, but things feel more stable.

There were no ROPE instructions on your repository, so I had to guess. I personally had found many ROPEs supplied by The Bloke to have issues with assorted models, and the general documentation of ROPE is really bad for a casual like me. Following various Github discussions on ROPE, I tried the stuff that people talked about. The rest was just lots of trial and much error.

0.5 and 80000 just happened to be what worked for Ycros-Boros.

I also tried out funky ideas, such as multiplying KoboldCPP's context with Kobold Lite's, and then tokegen to arrive at a ROPE number (Which sometimes worked beautifully for oneshots, but were unstable). Sometimes, working configurations broke after changing tokegen, so I figured it had a role in a "magical" formula, if such a thing existed. I suspect that Kobold Lite might have a bug with recent changes to token numbers, but have no evidence for that.

On occasion, I would find a ROPE that works for several sessions after rebooting KoboldCPP, but interacting with Silly can make that ROPE stop working. Or more frustrating, simply stop operating after about half a day. At the time, I had only 64gb RAM, so those ROPEs might have been interacting in odd ways with page files if I get to 99% RAM. That is my utterly uneducated guess, anyhow.

The "rules" that I could cobble together from various sources were as follows:

ROPE SCALING

It is better to use NTK, the lowest you can go without losing coherency. The effectiveness is on a curve, so being too big or small could corrode the output.

Linear - small number, higher is better. Slower than NTK.
NTK - big number, smaller is better. Faster than Linear.

The scale might be altered by the base context of the model. EG: A Llama-1 x1 is 2048. Llama-2 x1 might be 4096.

x1 linear context is 1.0 + 10000 = 2048
x2 linear context is 0.5 + 10000 = 4096
x4 linear context is 0.25 + 10000 = 8192
?x8 linear context is 0.125 + 10000 = 16384?
?x16 linear context is 0.0625 + 10000 = 32768?

x1 NKT aware context is 1.0 + 10000 = 2048
x2 NTK aware context is 1.0 + 32000 = 4096
x4 NTK aware context is 1.0 + 82000 = 8192

/ROPE SCALING

Question: Is there a NTK scaling for 16k and 32k? I would like to try that, if they exist. I tried to find the proper values myself, but no dice. I don't understand by what basis the NTK scaling works, so I can't homebrew a valid answer.