OpenAI agent cannot properly parse various complex function argument types

[mykola-mokhnach-parloa]

Steps to reproduce:

1. Define a new function similar to the example in

   agents/examples/multimodal_agent.py

   Line 29 in c5897c1

   fnc_ctx = llm.FunctionContext()

2. Set the function argument type to Optional[str] or Union[int, type(None)] or float | None
3. Run the pipeline

Actual result:
An exception stack trace is raised because the build_oai_function_description type support is limited to only (str, int, float, bool), even though the FunctionContext class is capable of recognizing the above nullable types.

Expected result:
The build_oai_function_description mentioned above should be able to properly parse nullable types like ones above.
For example, we expect Optional[str] to be transformed into ["string", "null"] OpenAI type definition.

[davidzhao]

fixed in #1211

[mykola-mokhnach-parloa]

Unfortunately the above change does not completely fix the original issue. The build_oai_function_description function remains the same and is still unable to parse optional type definitions. It would be great to have an actual unit test that verifies parsing of above cases and confirms the actual fix.

[jayeshp19]

Hi @mykola-mokhnach-parloa,

I’m unable to reproduce this issue. Could you please share the parameters you're passing or a screenshot of the error you're encountering?

Fetching the first argument does not work for type definitions like Union[None, str] or None | str, where None is passed as the first argument. The validation should check if the argument length is between 1 and 2, and then take the first non-None type argument.

I believe these cases are already handled in this function:

agents/livekit-agents/livekit/agents/llm/function_context.py

Lines 291 to 307 in 6b4e903

	def _is_optional_type(typ) -> Tuple[bool, Any]:
	"""return is_optional, inner_type"""
	origin = typing.get_origin(typ)
	if origin in {typing.Union, getattr(__builtins__, "UnionType", typing.Union)}:
	args = typing.get_args(typ)
	is_optional = type(None) in args
	inner_arg = None
	for arg in args:
	if arg is not type(None):
	inner_arg = arg
	break
	return is_optional, inner_arg
	return False, None

cc: @theomonnom

[mykola-mokhnach-parloa]

thanks for your reply @jayeshp19

I expect the below unit tests to pass (currently some of them fail):

import pytest
from typing import Union, Optional
from inspect import _empty

from livekit.agents.llm import FunctionInfo, FunctionArgInfo
from livekit.agents.llm.function_context import _is_optional_type
from livekit.agents import llm


def test_typing():
    assert _is_optional_type(Optional[int]) == (True, int)
    assert _is_optional_type(Union[str, None]) == (True, str)
    assert _is_optional_type(None | float) == (True, float)
    assert _is_optional_type(Union[str, int]) == (False, None)


@pytest.mark.parametrize(
    ("arg_typ", "oai_type"),
    [
        (int, "number"),
        (Optional[int], ["number", "null"]),
        (None | int, ["number", "null"]),
        (Union[None, int], ["number", "null"]),
        (Union[str, None], ["string", "null"]),
    ]
)
def test_description_building(arg_typ: type, oai_type: str | list[str]):
    fi = FunctionInfo(
        name='foo',
        description='foo',
        auto_retry=False,
        callable=lambda: None,
        arguments={
            'arg': FunctionArgInfo(
                name='foo',
                description='foo',
                type=arg_typ,
                default=_empty,
                choices=(),
            ),
        }
    )
    assert llm._oai_api.build_oai_function_description(
        fi
    )['function']['parameters']['properties']['foo']['type'] == oai_type