How to keep long sequences?

[kheyer]

I'm trying to finetune on a dataset with lots of long sequences. I see the phrase "dropping long sequences" printed during preprocessing.

What config parameter controls the drop length?

Is it possible to keep long sequences?

Ideally I'd like long sequences to be chunked into smaller sequences based on some max length parameter. Is there a way to do this?

[NanoCode012]

• Increase the sequence_len
• Chunk yourself (type: completion does this automatically)

That message happens even if no sequences are dropped.