[NeurIPS2022] Mind Reader: Reconstructing complex images from brain activities [Mobile friendly version] [Slides]
TL;DR
We include additional text modality for reconstructing image stimuli from fMRI activities. Pretrained CLIP embedding space and a conditional generative model are utilized to counter data scarcity.
Pipeline overview
Code
Mapping model is defined as fmri_clip_res in fmri_clip.py
Conditional generative model is modified in StyleGAN2 folder, main files changed: train.py, files under training folder, and files in torch_utils/ops because of compatibility issues (see here).
To use the trained models for generation, see this notebook.
We used wandb for hyperparameter tracking & tuning.
Sample results
For each two rows: top is ground truth, bottom is our reconstruction.
