From 682e551326e5e725f6f5da6b70199d7ec20022fd Mon Sep 17 00:00:00 2001 From: Arthur <48595927+ArthurZucker@users.noreply.github.com> Date: Fri, 9 Aug 2024 10:19:43 +0200 Subject: [PATCH] Update README.md --- README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/README.md b/README.md index bac865508..ecde6a1b6 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,11 @@ versatility. original sentence that corresponds to a given token. - Does all the pre-processing: Truncate, Pad, add the special tokens your model needs. +## Performances +Performances can vary depending on hardware, but running the [~/bindings/python/benches/test_tiktoken.py](https://github.com/huggingface/tokenizers/bindings/python/benches/test_tiktoken.py) should give the following on a g6 aws instance: +![image](https://github.com/user-attachments/assets/2b913d4b-e488-4cbc-b542-f90a6c40643d) + + ## Bindings We provide bindings to the following languages (more to come!):