Skip to content

Commit

Permalink
Add Claude 3 + Mistral to tokenizer playground
Browse files Browse the repository at this point in the history
  • Loading branch information
xenova authored Mar 4, 2024
1 parent ce4fd62 commit d525c8b
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions examples/tokenizer-playground/src/App.jsx
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,8 @@ function App() {
<option value="Xenova/gpt-4">gpt-4 / gpt-3.5-turbo / text-embedding-ada-002</option>
<option value="Xenova/text-davinci-003">text-davinci-003 / text-davinci-002</option>
<option value="Xenova/gpt-3">gpt-3</option>
<option value="Xenova/claude-tokenizer">Claude 3</option>
<option value="Xenova/mistral-tokenizer">Mistral</option>
<option value="Xenova/gemma-tokenizer">Gemma</option>
<option value="Xenova/llama-tokenizer">LLaMA / Llama 2</option>
<option value="Xenova/t5-small">T5</option>
Expand Down

0 comments on commit d525c8b

Please sign in to comment.