RbA: Segmenting Unknown Regions Rejected by All (ICCV 2023)

state-of-the-art on SegmentMeIfYouCan Benchmark

Nazir Nayal, Mısra Yavuz, João Henriques, Fatma Güney

[arXiv][BibTeX] [Project]

Updates

• 25.10.2024: Model Weights are available now as a release!
• code with training and evaluation scripts are available

Installation

See installation instructions for necessary installations and setup

Datasets Preparation

See Dataset Preparation for details on downloading and preparing datasets for both training and evaluation.

Model Zoo and Baselines

We provide the checkpoints and config files used to train RbA under different configurations in the RbA Model Zoo.

Training

In order to replicate any of our experiments, we provide the config files for all the models in the RbA Model Zoo. Given the config file, a training experiment can be run using the following command:

python train_net.py \
--config-file PATH_TO_CONFIG_FILE \
--num-gpus    NUM_GPUS \
OUTPUT_DIR    PATH_TO_STORE_CKPTS_AND_LOGS

For example, to train a Swin-B model with 1 decoder layer, on 4 gpus and store the outputs in a folder named model_logs/swin_b_1dl you can use the following:

python train_net.py \
--config-file configs/cityscapes/semantic-segmentation/swin/single_decoder_layer/maskformer2_swin_base_IN21k_384_bs16_90k_1dl.yaml \
--num-gpus 4 \
OUTPUT_DIR  model_logs/swin_b_1dl/

NOTE: some experiments require the model to be initialized from pretrained weights, make sure the required weights are available under the pretrained/ folder. Details about the require pretrained weights can be found in RbA Model Zoo.

Evaluation

We provide evaluate_ood.py for evaluating on OoD datasets. A simple usage for the script is as follows:

python evaluate_ood.py 
  --out_path results_test/ \  # folder to store results as pkl files
  --models_folder ckpts/ \
  --datasets_folder PATH_TO_DATASETS_ROOT \
  --model_mode all \  # evaluates all models in the models_folder
  --dataset_mode all \ # evaluate on all datasets (RA & FS LaF)

The script assumes the following:

• The OoD datasets are setup as described in Datasets Prepration
• The parameter --models_folder is a path to a folder that contains multiple folders, where each folder corresponds to a model. In a model's folder the scripts expects to files: 1) config.yaml and its checkpoint 2) model_final.pth. Setting up the models is explained in RbA Model Zoo Introduction

The scripts supports more finegrained options like selecting subsets of the models in a folder or the datasets. Please check evaluate_ood.py for descriptions of the options.

Citing RbA

If you use RbA in your research, please use the following BibTeX entry.

@InProceedings{nayal2023ICCV,
  author    = {Nazir Nayal and Mısra Yavuz and João F. Henriques and Fatma Güney},
  title     = {RbA: Segmenting Unknown Regions Rejected by All},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year      = {2023},
}

Acknowledgement

This repo is built mainly on top Mask2Former repo: (https://github.com/facebookresearch/Mask2Former).

Different code snippets are also adapted from the following repos:

• PEBAL
• DenseHybrid
• EOSPN

We thank all of the authors of these repos for their contributions.