What happened when load_dataset() recieves the .tar files?

[TomSuen]

accelerate launch examples/brushnet/train_brushnet.py
--pretrained_model_name_or_path runwayml/stable-diffusion-v1-5
--output_dir runs/logs/brushnet_segmentationmask
--train_data_dir data/BrushData \

I noticed that the train_data_dir parameter passed in contained a .tar file. What happened? Does the load_dataset method automatically unpack the .tar file?

[a-r-r-o-w]

This question is more well-suited for https://github.com/huggingface/datasets. You can open an issue/discussion there for more help.

Some links for reference:

• https://huggingface.co/docs/datasets/en/loading
• https://huggingface.co/docs/datasets/en/about_dataset_load