Skip to content

Commit

Permalink
LLM micro service extract input model name
Browse files Browse the repository at this point in the history
No input model from UI.
vLLM and OLLAMA get the model name from the envrionment
TGI mode model form ModelID

Signed-off-by: Wang, Xigui <[email protected]>
  • Loading branch information
xiguiw committed Aug 12, 2024
1 parent c45f8f0 commit 8b09b79
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion comps/llms/text-generation/ollama/llm.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
def llm_generate(input: LLMParamsDoc):
ollama = Ollama(
base_url=ollama_endpoint,
model=input.model,
model=input.model if input.model else model_name,
num_predict=input.max_new_tokens,
top_k=input.top_k,
top_p=input.top_p,
Expand Down Expand Up @@ -49,4 +49,5 @@ async def stream_generator():

if __name__ == "__main__":
ollama_endpoint = os.getenv("OLLAMA_ENDPOINT", "http://localhost:11434")
model_name = os.getenv("OLLAMA_MODEL", "meta-llama/Meta-Llama-3-8B-Instruct")
opea_microservices["opea_service@llm_ollama"].start()

0 comments on commit 8b09b79

Please sign in to comment.