Rethinking the Evaluation of Visible and Infrared Image Fusion

Dayan Guan¹, Yixuan Wu², Tianzhu Liu¹, Alex C. Kot³, Yanfeng Gu¹

¹Harbin Institute of Technology ²Northeastern University ³Nanyang Technological University

Examples

Abstract

Visible and Infrared Image Fusion (VIF) has garnered significant interest across a wide range of high-level vision tasks, such as object detection and semantic segmentation. However, the evaluation of VIF methods remains challenging due to the absence of ground truth. This paper proposes a Segmentation-oriented Evaluation Approach (SEA) to assess VIF methods by incorporating the semantic segmentation task and leveraging segmentation labels available in latest VIF datasets. Specifically, SEA utilizes universal segmentation models, capable of handling diverse images and classes, to predict segmentation outputs from fused images and compare these outputs with segmentation labels. Our evaluation of recent VIF methods using SEA reveals that their performance is comparable or even inferior to using visible images only, despite nearly half of the infrared images demonstrating better performance than visible images. Further analysis indicates that the two metrics most correlated to our SEA are the gradient-based fusion metric $Q_{\text{ABF}}$ and the visual information fidelity metric $Q_{\text{VIFF}}$ in conventional VIF evaluation metrics, which can serve as proxies when segmentation labels are unavailable. We hope that our evaluation will guide the development of novel and practical VIF methods.

Getting Started

Installation

1. Environment

git clone https://github.com/Yixuan-2002/SEA.git
cd SEA
pip install -r requirements.txt

2. Install Segment-Everything-Everywhere-All-At-Once and Grounded-SAM

3. Download Pretrained Checkpoints to the path ./checkpoints/

wget https://huggingface.co/xdecoder/X-Decoder/resolve/main/xdecoder_focalt_last.pt
wget https://huggingface.co/xdecoder/X-Decoder/resolve/main/xdecoder_focall_last.pt
wget https://huggingface.co/xdecoder/SEEM/blob/main/seem_focalt_v1.pt
wget https://huggingface.co/xdecoder/SEEM/blob/main/seem_focall_v1.pt

Dataset

The data includes the FMB Dataset and the MVSeg Dataset.

Download them from the above links, and organize them as follows.

├── SEA
│   ├── VIF_Results_FMB
│   │   ├── 2018_DenseFuse
│   │   ├── 2019_FusionGAN
│   │   └── ...
│   ├── VIF_Results_MVSeg
│   │   ├── 2018_DenseFuse
│   │   ├── 2019_FusionGAN
│   │   └── ...
│   ├── ...

Evaluation

• Traditional Evaluation Methods

sh VIFB.sh

Selecting different Datasets by modifying the results_folder in the python code in ./VIFB

• SEA Evaluation Method

sh SEEM.sh
sh xdecoder.sh
sh GroundedSAM.sh

Acknowledgements

• This work is built upon the SEEM, X-Decoder and Grounded SAM.

Citation

Please consider citing if you find our work useful:

@article{guan2024rethinking,
  title={Rethinking the Evaluation of Visible and Infrared Image Fusion},
  author={Guan, Dayan and Wu, Yixuan and Liu, Tianzhu and Kot, Alex and Gu, Yanfeng},
  journal={arXiv preprint arXiv:2410.06811},
  year={2024}
}