Switch model to Llama 3

[mirix]

Thanks for this great project. It works quite well.

However, I would like to change the model to:

https://huggingface.co/mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF

I have been trying to adapt the prompt template. It kind of works, even without changing the template, but eventually it goes into an endless response loop. I guess that the issue are the stop tokens or something like that.

Has anyone tried this?

[KoljaB]

Quite sure you are right, the reason will most probably be the different prompt formats ("chat templates") the models use. This project is a bit outdated and therefore still uses the raw template format.

Meanwhile most of this can be abstracted away. Today most LLM providers offer servers with chat endpoints, like these:

• lmstudio: http://localhost:1234/v1/chat/completions
• openai: https://api.openai.com/v1/chat/completions
• anthropic: https://api.anthropic.com/v1/messages
• ollama: http://localhost:11434/api/chat

Or you can directly use huggingface transformers.pipeline, which abstracts the model chat template away.

If you want to use the chat endpoints, the LocalEmotionalAIVoiceChat project has example code that does it this way.

Alternatively you can also use then python OpenAI library to wrap the base endpoints (http://localhost:1234/v1). For example code, Linguflex project does some LLM requests in this way.