Generation never starts: "context is null"

[sharpy66]

Load model,
Type hello
Press send
Open log
Log says:
[isolate 08:13:02] llama loaded
[isolate 08:13:02] main found: true
[isolate 08:13:02] trying main
[isolate 08:13:02] trying main DONE Instance of 'llama_context_params'
[isolate 08:13:02] context is null

Generation never starts

Samsung Z Flip4
8gb of ram
Snapdragon 8+ Gen1
The demo shows a oneplus device, probably an issue with Samsung phones, I'll check later on a different device.

[andsofine]

same here

[GeorvityLabs]

same issue @maxime-guerin-biprep @ThibautLEAUX any solution to this problem.
I'm getting the same problem.
I'm using a device with MediaTek Dimensity 1200 and 8gb Ram.

[maxime-guerin-biprep]

did you use the release 1.1 or 2.0 ?
Did you have the permission pop up ?
which version of android do you have ?
Which model did you use ?

[sharpy66]

Newest release, no permission pop up, android 13, One UI 5.1, Llama-7b

[GeorvityLabs]

did you use the release 1.1 or 2.0 ?
Did you have the permission pop up ?
which version of android do you have ?
Which model did you use ?

Release 2.0
Permission pop up did come I clicked on open file.
I used llama7b-ggml model 4bit quantized

[GeorvityLabs]

did you use the release 1.1 or 2.0 ?
Did you have the permission pop up ?
which version of android do you have ?
Which model did you use ?

Release 2.0
Permission pop up did come I clicked on open file.
I used llama7b-ggml model 4bit quantized

[sharpy66]

did you use the release 1.1 or 2.0 ?
Did you have the permission pop up ?
which version of android do you have ?
Which model did you use ?

Release 2.0
Permission pop up did come I clicked on open file.
I used llama7b-ggml model 4bit quantized

Same problem same steps.

[maxime-guerin-biprep]

Okay, we will check it today

[maxime-guerin-biprep]

Hello, we changed the targetSdkVersion to 31 it should work now.
We added more log from the loading of the model.

https://github.com/Bip-Rep/sherpa/releases/tag/2.0.1

[andsofine]

https://github.com/Bip-Rep/sherpa/releases/tag/2.0.1

now it crashes after some time when I enter something. (maybe because of 6gb ram)

also the opening of the model is very slow. it would be great to have a download progress indicator

Device: Pixel 4 xl
OS: GrapheneOS A13 (TP1A.221005.002.B2)

[maxime-guerin-biprep]

We only succeded to run it on a 8gb device, so it migth be why you crash.
Do you have another device with 8gb ?

[GeorvityLabs]

We only succeded to run it on a 8gb device, so it migth be why you crash.
Do you have another device with 8gb ?

@maxime-guerin-biprep
I'm getting the following error :

[maxime-guerin-biprep]

We did got this error when we loaded an alpaca model instead of a llama.
Are you using an llama or alpaca model ?

[GeorvityLabs]

@maxime-guerin-biprep
i tried llama model from here and here , then i tried alpaca and now i tried gpt4all trained model.
all gave similar errors.

is it possible to mention how you went about converting and getting the model. or linking the exact model you have been using.if you provide a temp link i'll download it and try out.
any possible way this could also get alpaca support?

[maxime-guerin-biprep]

We used Dalai a node js implementation here is a link

[maxime-guerin-biprep]

It seems that there is 2 format of models, you can go to Llama repo under Using GPT4All, their is a python script to convert models

[GeorvityLabs]

@maxime-guerin-biprep can you link the ggml-model.bin file here temporarily, if possible.
dalai is having some issues on my local machine.

[GeorvityLabs]

It seems that there is 2 format of models, you can go to Llama repo under Using GPT4All, their is a python script to convert models

Using the gpt4all to ggml conversion script to generate the .bin file should work?

[maxime-guerin-biprep]

i'am downloading the gpt4all rigth now, i will try to convert it later

[maxime-guerin-biprep]

So i just try the script convert-unversioned-ggml-to-ggml.py of the llama repo and succeded to use an old alpaca model using this file to convert it with this command
python3 convert-unversioned-ggml-to-ggml.py /path/to/bin .okenizer.model

[maxime-guerin-biprep]

I just tried the script convert-gpt4all-to-ggml.py and it worked

[GeorvityLabs]

@maxime-guerin-biprep could you upload the model bin files to huggingface temporarily so that i can test it at my end?

[GeorvityLabs]

@maxime-guerin-biprep i wrote this colab for conversion , but it doesn't work as intended , could you have a look if there is any issues : https://colab.research.google.com/drive/1F7ITFw7MAqEsYUN7ce7sG6rN-eAd8mnd?usp=sharing

[maxime-guerin-biprep]

you need to put the .bin in a folder and make the first argumen the folder, you will have a .bin and .orig after, use the .bin

[sharpy66]

updating my models to get rid of the bothersome "invalid model files" error, I now seem to get close, but the app just crashes after a moment, the last log is "trying main DONE instance of 'llama-context-params'

I'm going to try using a different script to update my models, I'll let you know if that works.

[GeorvityLabs]

you need to put the .bin in a folder and make the first argumen the folder, you will have a .bin and .orig after, use the .bin

@maxime-guerin-biprep alpaca model now works.
But it has hallucinations and the generation keeps on going and never stops.

Can you add a "Stop Generation" button to the UI?

Do you have any idea why the output keeps on going @maxime-guerin-biprep , it keeps on going and then crashes.

[ThibautLEAUX]

@GeorvityLabs the default preprompt is not that perfect at all, it was just a simple one, we hope that people will make better ones so that it hallucinates less.
We can try to add a stop generation button

[GeorvityLabs]

@ThibautLEAUX yea it would be great to add a stop generation button so that we could manually stop when it starts to hallucinate?
Is there any way to algorithmically detect when the model starts hallucinating? and maybe we can cut off / stop the generation at that point?
@maxime-guerin-biprep

[GeorvityLabs]

@ThibautLEAUX @maxime-guerin-biprep also was curious is the gtp4all working for you guys.
I tried converting the .bin model form the gpt4all repo using the llama.cpp conversion script, but it doesn't seem to run , the app sorta crashes.
does it work on your end?
also , would be great to have a reset option as i mentioned before.

[maxime-guerin-biprep]

Did you use the convert-gpt4all-to-ggml.py ? Or the same as alpaca ?
Because with the convert-gpt4all-to-ggml.py one it is not the folder you have to give but the file itself

[GeorvityLabs]

@maxime-guerin-biprep
it works now. yea, i had used the other script first.
it works fine now.
hope you guys add a reset button so that we can stop model when it starts hallucinating.

[maxime-guerin-biprep]

We will also had an option to do the same as the instruct mode on llama.cop main.
With this there will be no use of prepompt and reverse prompt for the alpaca and gpt4all model

[GeorvityLabs]

@maxime-guerin-biprep that is good to hear! looking forward to the update.

[GeorvityLabs]

@maxime-guerin-biprep
I just noticed that with the gpt4all model.
When I ask the question :
Who is obamas wife?
It gives me the right answer but then it adds another question related to Michelle Obama after.

Do you know why the model doesn't stop soon after giving the answer?

Any ideas on how to fix this issue?
I only asked one question, but it appends another question and keeps generating stuff related to the topic.

[GeorvityLabs]

@maxime-guerin-biprep
I just noticed that with the gpt4all model.
When I ask the question :
Who is obamas wife?
It gives me the right answer but then it adds another question related to Michelle Obama after.

Do you know why the model doesn't stop soon after giving the answer?

Any ideas on how to fix this issue?
I only asked one question, but it appends another question and keeps generating stuff related to the topic.

[Wingie]

Hmmm upto which version are supported?

[GeorvityLabs]

@maxime-guerin-biprep @ThibautLEAUX any updates on the stop generation button?
reset chat button etc?

[maxime-guerin-biprep]

we will try to do it this week

[GeorvityLabs]

@maxime-guerin-biprep great, looking forward to testing it out

[ThibautLEAUX]

Hmmm upto which version are supported?

we just released an update, so it is working with recent models

[maxime-guerin-biprep]

@GeorvityLabs we released the stop button

[GeorvityLabs]

@GeorvityLabs we released the stop button

Cool , I'll do some testing and check how it is working.