MSAF: Multimodal Split Attention Fusion

Code for the paper MSAF: Multimodal Split Attention Fusion. This is our implementation of the MSAF module and the three MSAF-powered multimodal networks.

If you use this code, please cite our paper:

@misc{su2020msaf,
      title={MSAF: Multimodal Split Attention Fusion}, 
      author={Lang Su and Chuqing Hu and Guofa Li and Dongpu Cao},
      year={2020},
      eprint={2012.07175},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Installation

Clone this repo along with submodules

git clone --recurse-submodules https://github.com/anita-hu/MSAF.git

Install dependencies

Method 1: Using environment.yml (installs dependencies for all three datasets)

With this method, you can skip dependency installation steps from the dataset specific README files

conda env create -f environment.yml

Method 2: Without environment.yml

This code was developed with Python 3.6, PyTorch 1.7.1 in Ubuntu 20.04.

• Basic dependencies (needed for all datasets): Pytorch, Tensorboard
• Dataset specific dependencies: see README file in each dataset folder

Usage

• The MSAF module is implemented in MSAF.py
• The README file in each dataset folder has details on data preprocessing, training and evaluation (pretrained weights are available)
  • RAVDESS
  • CMU-MOSEI
  • NTU RGB+D