Support chat response format

[drbh]

This PR adds support for the response_format parameter in the chat endpoint. This callers users to constrain chat results to a specified grammar.

Important

This feature is mutually exclusive with tools, as they both constrain the output via a grammar internally.

reference in openai api: https://platform.openai.com/docs/api-reference/chat/create#chat-create-response_format

example usage

from openai import OpenAI
from pydantic import BaseModel, conint
from typing import List
import json


class Weather(BaseModel):
    location: str
    days: List[str]
    weather: List[str]
    temperature: List[str]


client = OpenAI(
    base_url="http://localhost:3000/v1",
    api_key="_",
)


chat_completion = client.chat.completions.create(
    model="tgi",
    messages=[
        {
            "role": "system",
            "content": f"[Date: Mon Jun 10th]\nRespond to the users questions and answer them in the following format: {Weather.schema()}",
        },
        {
            "role": "user",
            "content": "What's the weather like the next 3 days in Brooklyn, NY?",
        },
    ],
    seed=42,
    max_tokens=500,
    response_format={"type": "json_object", "value": Weather.schema()},
)

json_response = chat_completion.choices[0].message.content
json_response = json.loads(json_response)
print(json.dumps(json_response, indent=2))
# {
#   "days": [
#     "Today",
#     "Tomorrow",
#     "Day After Tomorrow"
#   ],
#   "location": "Brooklyn, NY",
#   "temperature": [
#     "65",
#     "72",
#     "78"
#   ],
#   "weather": [
#     "Mostly Cloudy",
#     "Partly Cloudy",
#     "Sunny"
#   ]
# }

[aymeric-roucher]

Thank you @drbh , this will be a great addition! 🔥

Is it possible to use a regex as the grammar, like response_format={"type": "regex", "value": "..."} ?

[drbh]

@aymeric-roucher yes regex should work as well! The response_format is using the same mechanics as grammar in the generate endpoint (so both will accept the "type"s even if we add more going forward)