Results for 14 pretrained models coming soon #16
Replies: 3 comments 1 reply
-
|
Any news about this experiment? 👀 |
Beta Was this translation helpful? Give feedback.
-
|
https://github.com/alasdairforsythe/tokenmonster/blob/main/benchmark/pretrain.md |
Beta Was this translation helpful? Give feedback.
-
Hooray!!! |
Beta Was this translation helpful? Give feedback.
-
|
Any follow ups on this @alasdairforsythe ? For 80% one-word tokens and 20% multi-word tokens? |
Beta Was this translation helpful? Give feedback.
-
I've been pretraining from scratch NanoGPT Small and Medium sized models with different TokenMonster vocabularies, and comparing against GPT-2 Tokenizer and TikToken p50k_base.
Results will be up in 1-2 weeks.
Beta Was this translation helpful? Give feedback.
All reactions