Fix the incompatibility of ollama and groq json's response and update default model selection #87
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The changes included in this commit:
The incompatibility of Ollama and Groq JSON's response
This problem has been mentioned in issue 1, issue 2, issue 3.
It's mainly caused by the function calling compatibility required by instructor JSON's response, which seems not to work very well in Groq's llama and Ollama's model(looks like a bug of litellm) when using expert search and generates related queries.
To solve this problem, it's suggested that JSON mode rather than tools be used in the instructor response model when the model is Groq and Ollama.
Based on my tests, this change will stabilize Groq and Ollama’s structured generation.
Update the default model selection
Third-party Openai-proxy server
add the support for third-party Openai-proxy server by including the "OPENAI_API_BASE" env variable in the docker-compose file.