Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix the incompatibility of ollama and groq json's response and update default model selection #87

Merged
merged 2 commits into from
Sep 6, 2024

Conversation

init0xyz
Copy link
Contributor

The changes included in this commit:

  • fix the incompatibility of Ollama and Groq JSON's response
  • update the default model selection
  • add the support for third-party Openai-proxy server

The incompatibility of Ollama and Groq JSON's response

This problem has been mentioned in issue 1, issue 2, issue 3.
It's mainly caused by the function calling compatibility required by instructor JSON's response, which seems not to work very well in Groq's llama and Ollama's model(looks like a bug of litellm) when using expert search and generates related queries.
To solve this problem, it's suggested that JSON mode rather than tools be used in the instructor response model when the model is Groq and Ollama.
Based on my tests, this change will stabilize Groq and Ollama’s structured generation.

Update the default model selection

  1. change the default "fast" model from GPT-3.5-turbo to GPT-4o-mini, using a more powerful model without increasing cost.
  2. update the Groq model from llama3-70b to llama3.1-70b, also changed the Ollama llama3 to llama3.1.

Third-party Openai-proxy server

add the support for third-party Openai-proxy server by including the "OPENAI_API_BASE" env variable in the docker-compose file.

init0xyz added 2 commits July 23, 2024 21:49
add OPENAI_API_BASE variable in env-template and docker-compose file
Copy link

vercel bot commented Aug 13, 2024

@init0xyz is attempting to deploy a commit to the rashadphil's projects Team on Vercel.

A member of the Team first needs to authorize it.

@init0xyz init0xyz changed the title Fix the incompatibility of ollama and groq json's response and update default model selection. Fix the incompatibility of ollama and groq json's response and update default model selection Aug 14, 2024
@rashadphz
Copy link
Owner

thanks!!

@rashadphz rashadphz merged commit 883003f into rashadphz:main Sep 6, 2024
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants