server : add OAI compat for /v1/completions

[ngxson]

Supersede #10645

Ref documentation: https://platform.openai.com/docs/api-reference/completions/object

The /v1/completions endpoint can now be OAI-compatible (not to be confused with /completion endpoint, without /v1 prefix)

Also regrouped the docs to have 2 dedicated sections: one for OAI-compat API and one for non-OAI API

TODO:

• add test
• add docs

[ericcurtin]

Would this make llama-server compatible with this client?

https://github.com/open-webui/open-webui

if yes can we please get this in? 😄

I'm also curious for anyone in the know, it seems like a lot of the openai clients (like open-webui) expect the functionality of being able to switch models per request. Does llama-server support this and if not, what would be the effort to add that roughly?

[ggerganov]

Does llama-server support this and if not, what would be the effort to add that roughly?

This is not supported atm. But this logic seems like something more suitable for a proxy/routing layer rather than implementing it in llama-server.

[ngxson]

@ericcurtin I have no idea if they support 3rd party openai-compatible server or not. Judging from they README, they kinda support it via :ollama docker image tag, but I'm not sure if that means "image with ollama built-in" or "bring your own ollama server"

In either case, I think they rely on /v1/chat/completions, which we already have in llama.cpp. So it's not related to the current PR.

[mostlygeek]

This is not supported atm. But this logic seems like something more suitable for a proxy/routing layer rather than implementing it in llama-server.

I wrote llama-swap for just this purpose. It’s a transparent proxy that will swap llama-server based on the model name in the api call. It’s a single golang binary with no dependencies so it is easy to deploy.