Add support for chat templates

[xenova]

This PR adds support for chat templates, making it possible to generate well-formatted prompts in a generic fashion. This will also be useful to users of other libraries, like TGI, as this means they can generating prompts on the client-side (link to some discussion).

Since the python library's implementation uses Jinja, it required a minimalistic JavaScript reimplementation of the templating engine. Special thanks to Tyler Laceby for his amazing "Guide to Interpreters" tutorial series, which provided the basis for this implementation.

Example: Applying a chat template to a chat history

import { AutoTokenizer } from '@xenova/transformers';

const tokenizer = await AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1");

const chat = [
  { "role": "user", "content": "Hello, how are you?" },
  { "role": "assistant", "content": "I'm doing great. How can I help you today?" },
  { "role": "user", "content": "I'd like to show off how chat templating works!" },
]

const text = await tokenizer.apply_chat_template(chat, { tokenize: false });
// "<s>[INST] Hello, how are you? [/INST]I'm doing great. How can I help you today?</s> [INST] I'd like to show off how chat templating works! [/INST]"

const input_ids = await tokenizer.apply_chat_template(chat, { tokenize: true, return_tensor: false });
// [1, 733, 16289, 28793, 22557, 28725, 910, 460, 368, 28804, 733, 28748, 16289, 28793, 28737, 28742, 28719, 2548, 1598, 28723, 1602, 541, 315, 1316, 368, 3154, 28804, 2, 28705, 733, 16289, 28793, 315, 28742, 28715, 737, 298, 1347, 805, 910, 10706, 5752, 1077, 3791, 28808, 733, 28748, 16289, 28793]

cc @OlivierDehaene (original requester of the feature)

TODOs

• Add support for built-in string functions (e.g., .strip())
• Add support for logical operators
• Add all loop variables (like loop.last)
• Add full support for nested for-loops (currently, the loop index is reused)
• Auto-generate unit tests
• Add support for array slicing (e.g., messages[1:])
• Do trim_blocks=True, lstrip_blocks=True (needed by HuggingFaceH4/zephyr-7b-beta)
• Add test cases for invalid templates

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.