use existing llama.cpp install

[scalar27]

I've been using llama.cpp for quite a while (M1 Mac). Is there a way I can get ai_voicetalk_local.py to point to that installation instead of reinstalling it here? Sorry, newbie question...

[KoljaB]

Just leave out step 2 of installation. I think coqui engine does not run in realtime on a Mac though.

[scalar27]

I did leave out step 2 but then I get an error when I try to run:
ModuleNotFoundError: No module named 'llama_cpp'

[KoljaB]

Python import of llama_cpp fails, that means your environment does not have working python bindings for your llama.cpp.
Please look here for Mac bindings, probably Metal (MPS).

[scalar27]

Thank you. I did get it to work following your comment. Like the other M1 person, I do get stuttering. It's a shame because the voice quality is excellent and the latency is rather short. Hope a future update might solve this for us!

[scalar27]

I managed to get this working with the Gemma 2 model. However, I am having trouble setting the parameters. It's working but doesn't seem optimal. I see them in creation_params.json, and also in coqui_engine.py. Would it be possible for LocalAiVoiceChat to utilize Llama.cpp's server endpoint (instead)? Or would that require a lot of rewriting of the code?

[KoljaB]

I like that idea, I'll have to look into that.

[scalar27]

Great. It seems like a more standard approach these days. I'd be happy to test whatever. As mentioned above I'm on a M1 Mac so this isn't the fastest setup but it's now working pretty well with no stuttering.