feat: supports openai chat completions API

[drbh]

This PR adds support to make TGI a drop in replacement for OpenAI clients by exposing the same HTTP interface.

Notes

• TGI inits a single model at startup so the model field is unused in HTTP requests.
• max_tokens and stream should work as expected but other params may be (unimplemented or not supported)

General approach

• fetch the tokenizer_config at startup from the hub
• pass tokenizer_config into Infer so we have it at request time
• use the chat_template on the config to format chat request
• parse jinja template and render chat string
• pass inputs into existing generate function
• wrap generation output in expected structure before returning

How to test

Streaming curl

curl localhost:3000/v1/chat/completions \
    -X POST \
    -d '{
  "model": "tgi",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user",
      "content": "What is deep learning?"
    }
  ],
  "stream": true,
  "max_tokens": 20
}' \
    -H 'Content-Type: application/json'

It is also possible to use the openai python library and change the base url

🌊 STREAMING REQUEST

from openai import OpenAI

# init the client but point it to TGI
client = OpenAI(
    base_url="http://localhost:3000/v1",
    api_key="not needed for a local LLM"
)

chat_completion = client.chat.completions.create(
    model="tgi",
    messages=[
        {"role": "system", "content": "You are a helpful assistant." },
        {"role": "user", "content": "What is deep learning?"}
    ],
    stream=True
)

# iterate and print stream
for message in chat_completion:
    print(message)

# ChatCompletionChunk(id='', choices=[Choice(delta=ChoiceDelta(content=' that', function_call=None, role='assistant', tool_calls=None), finish_reason=None, index=2, logprobs=None)], created=1704486761, model='', object='text_completion', system_fingerprint='')

🚗 SYNCHRONOUS REQUEST

from openai import OpenAI

# init the client but point it to TGI
client = OpenAI(
    base_url="http://localhost:3000/v1",
    api_key="not needed for a local LLM"
)

chat_completion = client.chat.completions.create(
    model="tgi",
    messages=[
        {"role": "system", "content": "You are a helpful assistant." },
        {"role": "user", "content": "What is deep learning?"}
    ],
    stream=False
)

print(chat_completion)
# ChatCompletion(id='', choices=[Choice(finish_reason=None, index=0, logprobs=None, message=ChatCompletionMessage(content='\nDeep learning is a new field of research that has been gaining traction in the last ...', role='assistant', function_call=None, tool_calls=None))], created=1704486762, model='', object='text_completion', system_fingerprint='', usage=CompletionUsage(completion_tokens=100, prompt_tokens=76, total_tokens=176))

How to run dev

cd text-generation-inference/server
MASTER_ADDR=127.0.0.1 MASTER_PORT=5555 text-generation-server serve --trust-remote-code gpt2

***note many of the existing chat_templates use non standard jinja (ie. adding a raise to the template) which will throw an error when parsing; hence using upstage/SOLAR-10.7B-Instruct-v1.0 since it has a valid template

cd text-generation-inference/router
cargo run -- --tokenizer-name upstage/SOLAR-10.7B-Instruct-v1.0

trigger

curl localhost:3000/v1/chat/completions \
    -X POST \
    -d '{ "model": "gpt-3.5-turbo", "messages": [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "What is the IP address of the Google DNS servers?" } ], "stream": true, "max_tokens": 20, "logprobs": true }' \
    -H 'Content-Type: application/json'

^ supports stream: true and stream: false requests

[Narsil]

Fixes #735

[OlivierDehaene]

Nice!
A few comments here and there but nothing big.

[drbh]

**commits above are from a rebase to resolve merge conflicts

[Narsil]

Pre-emptive LGTM so you can merge once you used the prefill.len() instead of the validated length.

It's more correct (since it goes through the real python tokenizer used by the model) and avoids creating that weird duplication in the struct (that I'm going to add anyway here: https://github.com/huggingface/text-generation-inference/pull/1436/files but for different reasons)

[drbh]

**commits above are from a rebase to resolve merge conflicts

[drbh]

update: the latest commit adds support for loading the config from a local file. in order to support as many configurations as possible a new cli argument was added --tokenizer-config-path. If this argument is supplied we will load the config from the file, otherwise we'll fallback to look for tokenizer_config.json in the working directory or attempt to fetch it from hf_hub

cargo run --y -tokenizer-name upstage/SOLAR-10.7B-Instruct-v1.0 --tokenizer-config-path ./valid_tokenizer_config.json

Possible Config States

Condition	Local Model Available	Local Config Available	Revision Specified	Action for Tokenizer/Model Info	Action for Tokenizer Config
1	Yes	Yes	-	Load locally	Load locally
2	Yes	No	-	Load locally	Load from API
3	No	Yes	Yes	Load from API	Load locally
4	No	No	Yes	Load from API	Load from API
5	Yes	Yes	Yes	Load locally	Load locally
6	Yes	No	Yes	Load locally	Load from API
7	No	Yes	No	Load from API	Load locally
8	No	No	No	Load from API	Load from API

[philschmid]

Would it be possible to configure the "route" for that new chat completion API via an environment variable/cli arg? I am asking since sagemaker only exposes on route with /invocations. By using an env var, users could enable the chat API when they want to use it, e.g. we could have --oai-chat-route /invocations.

cc @jeffboudier

[Narsil]

@drbh we can do this in a follow-up. We're talking change this: https://github.com/huggingface/text-generation-inference/blob/main/router/src/server.rs#L697 Through a CLI argument (both in launcher and router I think)

[paulcx]

@drbh I can confirm the local tokenizer_config file is found by setting tokenizer-config-path to launcher args in #1495. I'm wondering something wrong about default path which cause "Could not find tokenizer config locally and no revision specified" error?

[UniverseFly]

Have the same issue with the quantized deepseek model

[drbh]

Hi @paulcx and @UniverseFly, unfortunately I cannot reproduce this issues on main.

I'm using the following command

text-generation-launcher \
  --model-id TheBloke/deepseek-coder-6.7B-instruct-AWQ \
  --quantize awq \
  --tokenizer-config-path ~/deepseek-coder-tokenizer-config.json

where deepseek-coder-tokenizer-config.json contents is

{
  "add_bos_token": true,
  "add_eos_token": false,
  "bos_token": "<｜begin▁of▁sentence｜>",
  "clean_up_tokenization_spaces": false,
  "eos_token": "<|EOT|>",
  "legacy": true,
  "model_max_length": 16384,
  "pad_token": "<｜end▁of▁sentence｜>",
  "sp_model_kwargs": {},
  "unk_token": null,
  "tokenizer_class": "LlamaTokenizerFast",
  "chat_template": "{%- set found_item = false -%}\n{%- for message in messages -%}\n    {%- if message['role'] == 'system' -%}\n        {%- set found_item = true -%}\n    {%- endif -%}\n{%- endfor -%}\n{%- if not found_item -%}\n{{'You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer.\\n'}}\n{%- endif %}\n{%- for message in messages %}\n    {%- if message['role'] == 'system' %}\n{{ message['content'] }}\n    {%- else %}\n        {%- if message['role'] == 'user' %}\n{{'### Instruction:\\n' + message['content'] + '\\n'}}\n        {%- else %}\n{{'### Response:\\n' + message['content'] + '\\n<|EOT|>\\n'}}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{{'### Response:\\n'}}\n"
}

request

curl localhost:3000/v1/chat/completions \
    -X POST \
    -d '{ "model": "gpt-3.5-turbo", "messages": [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "What is the IP address of the Google DNS servers?" } ], "stream": false, "max_tokens": 20, "seed": 0 }' \
    -H 'Content-Type: application/json'

response

{"id":"","object":"text_completion","created":1706571866,"model":"TheBloke/deepseek-coder-6.7B-instruct-AWQ","system_fingerprint":"1.4.0-native","choices":[{"index":0,"message":{"role":"assistant","content":"As an AI Programming Assistant, I primarily focus on providing information and answering questions related to computer science"},"logprobs":null,"finish_reason":"length"}],"usage":{"prompt_tokens":89,"completion_tokens":20,"total_tokens":109}}

Additionally the following will fail to load the tokenizer_config from the hub as expected, because the tokens specified in the config are not strings as mentioned above.

link to config

 text-generation-launcher \
  --model-id TheBloke/deepseek-coder-6.7B-instruct-AWQ \
  --quantize awq 

I hope this information is helpful! Thanks!

[paulcx]

@drbh I loaded the model from the local directory in launcher arguments (--model-id /data/deepseek-coder-6.7B-instruct-AWQ --quantize awq). In this instance, the model directory is "/data/deepseek-coder-6.7B-instruct-AWQ," within docker env and I'm curious if the launcher code will automatically locate the tokenizer_config.json in this directory by default or anything else. Could this be the cause of the error mentioned?

[UniverseFly]

@drbh thanks for your response. I was trying the v1.4.0 docker image. Unfortunately I do not have the privilege to install required dependencies to build the main branch from my server, but I am trying to build its docker image and will report back. Is it expected that this doesn't work on v1.4.0?

[UniverseFly]

@drbh thanks for your response. I was trying the v1.4.0 docker image. Unfortunately I do not have the privilege to install required dependencies to build the main branch from my server, but I am trying to build its docker image and will report back. Is it expected that this doesn't work on v1.4.0?

Oh I made a silly mistake.. Since I was using docker, the --tokenizer-config-path should point to the docker side. After I corrected the path it worked! Thank you @drbh

[paulcx]

@drbh I loaded the model from the local directory in launcher arguments (--model-id /data/deepseek-coder-6.7B-instruct-AWQ --quantize awq). In this instance, the model directory is "/data/deepseek-coder-6.7B-instruct-AWQ," within docker env and I'm curious if the launcher code will automatically locate the tokenizer_config.json in this directory by default or anything else. Could this be the cause of the error mentioned?

@drbh Would you please verify the issue for local model launcher args?

[spew]

I commented about the local model's tokenizer config not being found here: #1427 (comment)

I think that the server / launcher should look in the model directory instead of the working directory (or in addition to the working directory).

[drbh]

Hi @paulcx I believe this PR will resolve you're issue: #1518

originally the code was looking for the tokenizer_config in the local directory but now it should correctly check the same directory as the model files.

Would you be able to try the latest changes and let me know if that resolves it for you? Thanks!

[paulcx]

Hi @paulcx I believe this PR will resolve you're issue: #1518

originally the code was looking for the tokenizer_config in the local directory but now it should correctly check the same directory as the model files.

Would you be able to try the latest changes and let me know if that resolves it for you? Thanks!

cool, I can confirm that pr solved issue. @drbh

BTW 1. would you think it's better to put chat_template into /info api? 2. So far I can not debug the chat_template and can not see if it works as expected.

[paulcx]

@drbh I attempted to switch many plugins that support the OpenAI API (such as genieAI, sider) to the TGI API, but the result was that the conversations could not be stopped. Specifically, the tgi openai api (stream mode) seems to end differently compared to the output (length?) from the OpenAI API. I used to create custom openai api server and yield ServerSentEvent("[DONE]") at the end of stream output. Not sure if this help?

[dongs0104]

Hi @drbh
Thanks for this PR,
When i use the Llama 3 instruction model on TGI, the chat_template inside the tokenizer contains the bos token twice, am i right? it is hard to debug in rust code, i am not good at rust code sorry ;(

huggingface/trl#1114