Cannot reproduce the model

[mina58]

I downloaded the dataset and followed the cleaning steps in the paper (removing files with >= 20000 characters) but this yielded a dataset of size 220MB. I am trying to reproduce the model using the exact steps in the finetuning repo but I get very different results.

Is there something I am doing wrong?