"openai.BadRequestError: Invalid role" error while using Ollama + LiteLLM + Autogen

[itsba2]

I will try to elaborate on my issue and would appreciate any kind of help!

Introduction

I am running an ollama server with LiteLLM proxy. I aim to design a group chat using Autogen. The designated agents are:

User_Proxy: ask question
SQL_Generator: generate SQL
SQL_Runner: run SQL (tool calling)
Result_Validator: validate result

Problems

1. The group chat works fine up until SQL_Runner returns the data from DB. But when the next speaker is expected to be the Result_Validator, SQL_Generator is assigned as next speaker, although I explicitly state in the system message that SQL_Runner should pass the result to Result_Validator.

2. For some reason, SQL_Generator agent's role remains as tool following the activity of SQL_Runner instead of one of [system, user, assistant] and it leads to this error:

***** Response from calling tool (call_6493f510-d318-4e85-b552-8963bece5fcb) *****
[
    // data from DB
]
**********************************************************************************
Next speaker: SQL_Generator
Traceback (most recent call last):
  File "/home/ba2/dev/softlab24/tum-softlab-2024/test_autogen_sql.py", line 232, in <module>
    main()
  File "/home/ba2/dev/softlab24/tum-softlab-2024/test_autogen_sql.py", line 116, in main
    user_proxy.initiate_chat(manager, clear_history=True, message=question)
  File "/home/ba2/.cache/pypoetry/virtualenvs/tum-softlab-2024-LrHmNXK2-py3.11/lib/python3.11/site-packages/autogen/agentchat/conversable_agent.py", line 1102, in initiate_chat
    self.send(msg2send, recipient, silent=silent)
  File "/home/ba2/.cache/pypoetry/virtualenvs/tum-softlab-2024-LrHmNXK2-py3.11/lib/python3.11/site-packages/autogen/agentchat/conversable_agent.py", line 738, in send
    recipient.receive(message, self, request_reply, silent)
  File "/home/ba2/.cache/pypoetry/virtualenvs/tum-softlab-2024-LrHmNXK2-py3.11/lib/python3.11/site-packages/autogen/agentchat/conversable_agent.py", line 902, in receive
    reply = self.generate_reply(messages=self.chat_messages[sender], sender=sender)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ba2/.cache/pypoetry/virtualenvs/tum-softlab-2024-LrHmNXK2-py3.11/lib/python3.11/site-packages/autogen/agentchat/conversable_agent.py", line 2056, in generate_reply
    final, reply = reply_func(self, messages=messages, sender=sender, config=reply_func_tuple["config"])
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ba2/.cache/pypoetry/virtualenvs/tum-softlab-2024-LrHmNXK2-py3.11/lib/python3.11/site-packages/autogen/agentchat/groupchat.py", line 1131, in run_chat
    reply = speaker.generate_reply(sender=self)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ba2/.cache/pypoetry/virtualenvs/tum-softlab-2024-LrHmNXK2-py3.11/lib/python3.11/site-packages/autogen/agentchat/conversable_agent.py", line 2056, in generate_reply
    final, reply = reply_func(self, messages=messages, sender=sender, config=reply_func_tuple["config"])
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ba2/.cache/pypoetry/virtualenvs/tum-softlab-2024-LrHmNXK2-py3.11/lib/python3.11/site-packages/autogen/agentchat/conversable_agent.py", line 1424, in generate_oai_reply
    extracted_response = self._generate_oai_reply_from_client(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ba2/.cache/pypoetry/virtualenvs/tum-softlab-2024-LrHmNXK2-py3.11/lib/python3.11/site-packages/autogen/agentchat/conversable_agent.py", line 1443, in _generate_oai_reply_from_client
    response = llm_client.create(
               ^^^^^^^^^^^^^^^^^^
  File "/home/ba2/.cache/pypoetry/virtualenvs/tum-softlab-2024-LrHmNXK2-py3.11/lib/python3.11/site-packages/autogen/oai/client.py", line 831, in create
    response = client.create(params)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ba2/.cache/pypoetry/virtualenvs/tum-softlab-2024-LrHmNXK2-py3.11/lib/python3.11/site-packages/autogen/oai/client.py", line 400, in create
    response = completions.create(**params)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ba2/.cache/pypoetry/virtualenvs/tum-softlab-2024-LrHmNXK2-py3.11/lib/python3.11/site-packages/openai/_utils/_utils.py", line 274, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ba2/.cache/pypoetry/virtualenvs/tum-softlab-2024-LrHmNXK2-py3.11/lib/python3.11/site-packages/openai/resources/chat/completions.py", line 815, in create
    return self._post(
           ^^^^^^^^^^^
  File "/home/ba2/.cache/pypoetry/virtualenvs/tum-softlab-2024-LrHmNXK2-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 1277, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ba2/.cache/pypoetry/virtualenvs/tum-softlab-2024-LrHmNXK2-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 954, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "/home/ba2/.cache/pypoetry/virtualenvs/tum-softlab-2024-LrHmNXK2-py3.11/lib/python3.11/site-packages/openai/_base_client.py", line 1058, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': 'invalid role: tool, role must be one of [system, user, assistant]', 'type': 'api_error', 'param': None, 'code': None}}

Code

Please let me know if you need extra pieces of code.

LLM Configs

llama3_config_list = [
    {
        "model": "llama3:latest",
        "base_url": f"{OLLAMA_BASE_URL}:{OLLAMA_PORT}/v1",
        "api_key": "ollama",
    }
]

litellm_config_list = [
    {
        "model": "NotRequired",
        "api_key": "NotRequired",
        "base_url": f"{LITELLM_BASE_URL}:{LITELLM_PORT}",
        "price": [0, 0],
    }
]

local_llm_config = {"config_list": litellm_config_list, "cache_seed": None}

GroupChat

user_proxy = autogen.UserProxyAgent(
    name=Role.USER_PROXY,
    system_message=Prompt.USER_PROXY,
    human_input_mode="NEVER",
    is_termination_msg=is_termination_msg,
    code_execution_config={"use_docker": False, "work_dir": "test"},
)

sql_generator = autogen.AssistantAgent(
    name=Role.SQL_GENERATOR,
    system_message=Prompt.SQL_GENERATOR,
    llm_config=get_custom_llm_config(config_list=llama3_config_list),
)

sql_runner = autogen.AssistantAgent(
    name=Role.SQL_RUNNER,
    system_message=Prompt.SQL_RUNNER,
    llm_config=local_llm_config,
)

@user_proxy.register_for_execution()
@sql_runner.register_for_llm(description="SQL query runner")
def run_sql(sql: Annotated[str, "SQL query string to run"]) -> str:
    with PostgresManager() as db:
        db.get_rds_connection()
        return db.run_sql(sql)

result_validator = autogen.AssistantAgent(
    name=Role.RESULT_VALIDATOR,
    system_message=Prompt.RESULT_VALIDATOR,
    llm_config=get_custom_llm_config(config_list=llama3_config_list)
)

groupchat = autogen.GroupChat(
    agents=[
        user_proxy,
        sql_generator,
        sql_runner,
        result_validator,
    ],
    messages=[],
)

manager = autogen.GroupChatManager(groupchat=groupchat)

user_proxy.initiate_chat(manager, message=question)

For the problem number 1, I have already tried to set autogen.GroupChat's speaker_selection_method to "round_robin" but the result did not change at all.

For the problem number 2, I am guessing that it might be overcome if I can somehow prevent agent role from being cached, but I cannot really figure out how.

[itsba2]

I was encountering this error since my ollama version did not support tool calling.
After upgrading ollama server version (to 0.4.2) now the role "tool" is available.