Difference between data in webdataset_avg_new and webdataset_avg_split files

[EvelynChengG]

Hello! Thank you very much for your excellent work!
In the database connection you provided for huggingface, there are two folders webdataset_avg_new and webdataset_avg_split files, which contain the same file names, but the file contents and sizes are different, is there any difference between the files in the two folders? Is there any difference between the files in the two folders? I hope to get your answer!

[PaulScotti]

Thanks!

The difference is that the original authors of the Natural Scenes Dataset withheld 3 sessions of data for the Algonauts challenge. After that challenge ended, they released those 3 sessions. webdataset_avg_new contains those recently released 3 sessions whereas webdataset_avg_split does not contain those sessions

Also note that this repository is out of date as we've released a new repository and paper for MindEye2 here: https://github.com/MedARC-AI/MindEyeV2/tree/main