Hyperformer

This is the official implementation of our paper Hypergraph Transformer for Skeleton-based Action Recognition.

Excellent Efficiency

Model	Parameters	FLOPs	Training Time/Epoch	Accuracy
ST-TR	12.1M	259.4G	11m48s	89.9
DSTA	4.1M	64.7G	10m36s	91.5
Efficient-GCN-B4 *	2.0M	15.2G*	15m37s*	91.7*
Hyperformer	2.6M	14.8G	6m52s	92.9

Remark: Comparing computation budget under the same setting (1 Tesla V100-32GB, 64 batch size, etc.) on NTU RGB+D 60 Cross-Subject Benchmark. *EfficientNet-B4 requires 4 times higher temporal resolution (288 vs 64 frames) and thus 2 GPUs to fit into memory and reach their reported accuracy.

Attention on Hypergraph

Hyperformer architecture

Preparation

Install torchlight

Run pip install -e torchlight

Download datasets.

There are 3 datasets to download:

• NTU RGB+D 60 Skeleton
• NTU RGB+D 120 Skeleton
• NW-UCLA

NTU RGB+D 60 and 120

1. Request dataset here: https://rose1.ntu.edu.sg/dataset/actionRecognition
2. Download the skeleton-only datasets:
   1. nturgbd_skeletons_s001_to_s017.zip (NTU RGB+D 60)
   2. nturgbd_skeletons_s018_to_s032.zip (NTU RGB+D 120)
   3. Extract above files to ./data/nturgbd_raw

NW-UCLA

1. Download dataset from CTR-GCN
2. Move all_sqe to ./data/NW-UCLA

Data Processing

Directory Structure

Put downloaded data into the following directory structure:

- data/
  - NW-UCLA/
    - all_sqe
      ... # raw data of NW-UCLA
  - ntu/
  - ntu120/
  - nturgbd_raw/
    - nturgb+d_skeletons/     # from `nturgbd_skeletons_s001_to_s017.zip`
      ...
    - nturgb+d_skeletons120/  # from `nturgbd_skeletons_s018_to_s032.zip`
      ...

Generating Data

• Generate NTU RGB+D 60 or NTU RGB+D 120 dataset:

 cd ./data/ntu # or cd ./data/ntu120
 # Get skeleton of each performer
 python get_raw_skes_data.py
 # Remove the bad skeleton 
 python get_raw_denoised_data.py
 # Transform the skeleton to the center of the first frame
 python seq_transformation.py

Evaluation

We provide the pretrained model weights for NTURGB+D 60 and NTURGB+D 120 benchmarks.

To use the pretrained weights for evaluation, please run the following command:

bash evaluate.sh

Training & Testing

Training

bash train.sh

Please check the configuration in the config directory.

Testing

bash evaluate.sh

To ensemble the results of different modalities, run the following command:

bash ensemble.sh

Acknowledgements

This repo is based on 2s-AGCN and CTR-GCN. The data processing is borrowed from SGN and HCN, and the training strategy is based on InfoGCN.

Thanks to the original authors for their work!

Citation

Please cite this work if you find it useful:

 @article{zhou2022hypergraph,
  title={Hypergraph Transformer for Skeleton-based Action Recognition},
  author={Zhou, Yuxuan and Cheng, Zhi-Qi and Li, Chao and Geng, Yifeng and Xie, Xuansong and Keuper, Margret},
  journal={arXiv preprint arXiv:2211.09590},
  year={2022}
}

Contact

For any questions, feel free to contact: [email protected]