What’s the limitations?

[yukiarimo]

Are there any limitations on LLaMA 2 models I can run in the browser? Like, if I have a ~4 GB model (7B), can I run it on my iPhone/Mac through WASM?

[rahuldshetty]

I haven't tested but it might be too buggy (and slow) to run more than 2GB models with llm.js. This directly leverages the CPU w/WASM engine on the browser without any WebGPU acceleration. Also due to 32bit memory address limitations on the browser side, this library will not work (yet).

I'd highly recommend checking out these 2 projects if you are looking to run models over 4GB:

• https://github.com/mlc-ai/web-llm
• https://huggingface.co/docs/transformers.js