MedSyn

Official PyTorch implementation for paper MedSyn: Text-guided Anatomy-aware Synthesis of High-Fidelity 3D CT Images, accepted by IEEE Transactions on Medical Imaging.

This code is made by Yanwu Xu and Li Sun.

[Paper] [Project]

Table of Contents

1. Environment Setup
2. Pretrained Checkpoint
3. Pre-processing Data
4. Training
5. Inference
6. Additional Scripts
7. Generated Samples
8. Citation
9. License and Copyright
10. Contact

Environment Setup

Before running and doing inference based on our code, we highly recommend preparing at least two GPUs with 48G GPU memory each.

conda create -n medsyn python==3.9

In addition to this, you need to also install several packages by:

pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu121
pip install monai==0.8.0
pip install accelerate
pip install einops
pip install einops_exts

Pretrained Checkpoint

Refer to the src folder

Our checkpoint for pre-trained language model is available here. Our checkpoint for model pre-trained on UPMC dataset is available here (Application required).

Pre-processing Data

Refer to the preprocess folder

Training

Refer to the src folder

This is a one-key running bash, which will run both low-res and high-res. But the training can be done independently

sh run_train.sh

Inference

Refer to the src folder

sh run_inference.sh

Additional Scripts

We give the inference for our text conditional generation in "prompt.ipynb" and the conditional generation with segmentation in "seg_conditional.ipynb"

Generated Samples

Low-Res	High-Res

Comparisons

Generation Conditioned on Reports

Generation Conditioned on Segmentation Mask

Citation

@ARTICLE{medsyn2024,
  author={Xu, Yanwu and Sun, Li and Peng, Wei and Jia, Shuyue and Morrison, Katelyn and Perer, Adam and Zandifar, Afrooz and Visweswaran, Shyam and Eslami, Motahhare and Batmanghelich, Kayhan},
  journal={IEEE Transactions on Medical Imaging}, 
  title={MedSyn: Text-guided Anatomy-aware Synthesis of High-Fidelity 3D CT Images}, 
  year={2024},
  doi={10.1109/TMI.2024.3415032}}

License and Copyright

CC-BY-NC

Contact

Yanwu Xu [[email protected]], Li Sun [[email protected]], Kayhan Batmanghelich [[email protected]]