Axolotl support for Stable LM 2 Zephyr 1.6B

[interstellarninja]

Hi,
I'm working on finetuning stability's new 1.6B parameter model stablelm-2-zephyr-1_6b.

When I tried training with axolotl.cli.train or preprocessing dataset with axolotl.cli.preprocess I'm getting the following error:

TypeError: cannot pickle 'builtins.CoreBPE' object

I have already tried upgrading datasets to the latest version as suggested by stability team but the error seems to persist. The same config and dataset had worked for stability's previous model Stablelm-Zephyr-3b model.

One thing that has changed is that they are using a new tokenizer for this model called "Arcade100kTokenizer" based on tiktoken.

Has anyone else tried finetuning this model yet and if anybody knows how to fix this error?

[NanoCode012]

Is this not supported still? I would recommend moving this to an Issue (if there's not one already).

[interstellarninja]

hey i got it to work later after they updated their tokenizer script thanks.