Setup model routing config and plan routing to o1 #6189
Conversation
| Router that routes the prompt that is judged by a LLM as complex and requires a step-by-step plan. | ||
| """ | ||
|
|
||
| JUDGE_MODEL = 'gpt-4o' |
There was a problem hiding this comment.
Would be interesting to see if we can experiment with cheaper model for that 🤔
| * Translating high-level requirements into detailed implementation steps and ensuring consistency. | ||
|
|
||
| === BEGIN USER MESSAGE === | ||
| {message} |
There was a problem hiding this comment.
We could also experiment sending O1 with the last 5/10 action/observation 🤔 in case there's some deep reasoning required to figure out the error, etc.
| ) | ||
|
|
||
| # Replace the model with the reasoning model | ||
| kwargs['model'] = self.model_routing_config.reasoning_model |
There was a problem hiding this comment.
Is model enough, or also: custom provider, base URL?
There was a problem hiding this comment.
We could design the reasoning model not as a part of an LLM instance, but as a second LLM instance in the agent?
| [model_routing] | ||
|
|
||
| # The reasoning model to use for plan generation | ||
| reasoning_model = "o1-preview-2024-09-12" |
There was a problem hiding this comment.
| reasoning_model = "o1-preview-2024-09-12" | |
| [llm.reasoning_model] | |
| model = "o1-preview-2024-09-12" | |
| ... |
There was a problem hiding this comment.
I'm so happy to see this, thank you! I do think we are missing some minimal framework to experiment with reasoning models.
About the way to choose another model:
We already have the ability to choose, configure, and use a random model, for example in evals: we can write the model configuration in toml, in a custom named LLM config section, [llm.o1], load it with an utility function, and instantiate an LLM from it.
We can use that here. Names are user-defined, and we can, if we want, set in stone a particular name for the reasoning model, e.g. [llm.reasoning_model], or [llm.oh_reasoning_model], or [llm.blueberry] (or strawberry for that matter), whatever name.
| @@ -0,0 +1,42 @@ | |||
| ANALYZE_PROMPT = """Analyze this prompt to see if it requires a detailed plan generation. | |||
There was a problem hiding this comment.
I'm a bit curious, is this prompt handmade, is it llm-generated, or what is the source of this prompt?
End-user friendly description of the problem this fixes or functionality that this introduces
Give a summary of what the PR does, explaining any non-trivial design decisions
This PR is to:
Link of any specific issues this addresses