Which config parameters are fixed during preprocessing?

[kheyer]

I want to test different hyperparameters over several training runs. To save time, I'd like to preprocess the dataset once, then use the preprocessed dataset for multiple runs.

I wanted to check which config parameters are fixed during the preprocessing stage in case that conflicts with some of the parameters I want to change. For example, I imagine sequence_len and sample_packing parameters can't be changed without rerunning the preprocessing. Are there other parameters that behave in that way? For example, can batch size be changed after preprocessing, or is that set during preprocessing?

[NanoCode012]

Please check this section: https://github.com/OpenAccess-AI-Collective/axolotl/blob/main/src/axolotl/utils/data.py#L143