Mixture of Depths

Comparison "Mixture-of-Depths" with default Llama architecture. Inspired by meta-llama and sramshetty/mixture-of-depths

Start

1. Look how to setup env for Llama 2 here

2. Install reqirements

pip install -r "reqirements.txt"

3. Try out the notebook training.ipynb

Result:

The value of the loss function is up to 40% lower for the MoD architecture than for the default Llama.