How do use the pretrained models of ICNet that are provided?

[bjchen666]

@meetshah1995 @adam9500370 The pretrained models of ICNet come from #81. I do not how to use.

[adam9500370]

Hi, @gshdong .
After you downloaded the pretrained models, you can modify the corresponding settings in config file (refer to configs/fcn8s_pascal.yml),
e.g., using cfg['training']['resume'] to load pretrained models.

model:
    arch: icnet
...
training:
...
    resume: icnet_cityscapes_trainval_90k.pth

Then run python train.py --config your_config.yml for training
or run python validate.py --config your_config.yml --model_path icnet_cityscapes_trainval_90k.pth for validation.

[bjchen666]

@adam9500370 I run python validate.py --config icnet_cityscapes.yml --model_path icnet_cityscapes_train_30k.pth that is provided by you for validation. However, I get only 0.0198757906223 miou. How do I get your 65.86 miou. I'm confused.

[adam9500370]

https://github.com/meetshah1995/pytorch-semseg/blob/4778145c7179bd14be034a35baf7dca3b85c81f2/validate.py#L35-L41
Due to original data preprocessing, you need to set img_norm=False and version="pascal" arguments in data_loader.

And replace https://github.com/meetshah1995/pytorch-semseg/blob/6efd0b5baf11cb11f828390c24a5d10808a6e51c/validate.py#L40 with cfg['data']['img_cols']

[bjchen666]

@adam9500370 I set img_norm=False and version="pascal" arguments in cityscapes_loader.py. However, I get 0.0516289067284 miou. I don't know what's wrong. Can you send me your successful code, my email is [email protected]. Thank you very much!

[adam9500370]

https://github.com/meetshah1995/pytorch-semseg/blob/6efd0b5baf11cb11f828390c24a5d10808a6e51c/ptsemseg/models/icnet.py#L182
https://github.com/meetshah1995/pytorch-semseg/blob/6efd0b5baf11cb11f828390c24a5d10808a6e51c/ptsemseg/models/icnet.py#L196
If you are using pytorch 0.4.x, you need to replace these lines interp(x, output_size=get_interp_size(x, s_factor=2)) with the corresponding form F.interpolate(x, size=get_interp_size(x, s_factor=2), mode='bilinear', align_corners=True), because it seems that F.grid_sample in pytorch 0.4.x is not the same as that one in pytorch 0.3.x (need to set tensor type as float).
In addition, set align_corners=True for F.upsample(...). (at [L575, L589, L795] in ptsemseg/models/utils.py and [L213, L223] in ptsemseg/models/icnet.py)

And you can create a config file (icnet_cityscapes.yml) as the followings:

model:
    arch: icnet
    version: cityscapes
    with_bn: False
data:
    dataset: cityscapes
    train_split: train
    val_split: val
    img_rows: 1025
    img_cols: 2049
    path: cityscapes_dataset_root_path
training:
    batch_size: 1
    resume: icnet_cityscapes_train_30k.pth

[bjchen666]

@adam9500370 Thank you very much for helping me solve this problem. This problem is caused by different pytorch versions. I change my pytorch from 0.4 to 0.3. I get 66+% miou. Thank you again！

[bjchen666]

@adam9500370 Do you train the ICNet models by yourself now？ Could you please share how you trained ICNet models?

[adam9500370]

No, I don't train ICNet models by myself.

Maybe you can refer to original paper for more details.

1. Train half-compressed PSPNet50 baseline model.
2. Load the baseline model weights at low-resolution branch and train the whole ICNet model.

[bjchen666]

Ok, thank you very much!

[ambareeshsrja16]

HI @adam9500370

Thank you so much for the pretrained models and the implementation. I wanted to use the same for an interim step in another learning task. I proceeded as follows:

1. Downloaded pretrained weights from Update PSPNet and ICNet #81 (icnetBN_cityscapes_trainval_90k.pth)
2. Proceeded to run with the following command:

python3 test.py --model_path /home/ambareesh/Project_276C/gym_torcs/DDPG_PyTorch_Vanilla/pytorch-semseg/icnetBN_cityscapes_trainval_90k_bn.pth --dataset camvid --img_path _11.jpeg

I got the following error:

Read Input Image from : _11.jpeg
/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:482: FutureWarning: Conversion of the second argument of issubdtype from int to np.signedinteger is deprecated. In future, it will be treated as np.int64 == np.dtype(int).type.
if issubdtype(ts, int):
/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:485: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
elif issubdtype(type(size), float):
Traceback (most recent call last):
File "test.py", line 162, in
test(args)
File "test.py", line 65, in test
model.load_state_dict(state)
File "/home/ambareesh/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 839, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for icnet:
size mismatch for classification.weight: copying a param with shape torch.Size([19, 128, 1, 1]) from checkpoint, the shape in current model is torch.Size([12, 128, 1, 1]).
size mismatch for classification.bias: copying a param with shape torch.Size([19]) from checkpoint, the shape in current model is torch.Size([12]).
size mismatch for cff_sub24.low_classifier_conv.weight: copying a param with shape torch.Size([19, 256, 1, 1]) from checkpoint, the shape in current model is torch.Size([12, 256, 1, 1]).
size mismatch for cff_sub24.low_classifier_conv.bias: copying a param with shape torch.Size([19]) from checkpoint, the shape in current model is torch.Size([12]).
size mismatch for cff_sub12.low_classifier_conv.weight: copying a param with shape torch.Size([19, 128, 1, 1]) from checkpoint, the shape in current model is torch.Size([12, 128, 1, 1]).
size mismatch for cff_sub12.low_classifier_conv.bias: copying a param with shape torch.Size([19]) from checkpoint, the shape in current model is torch.Size([12]).

I'm guessing there was a size mismatch (saved model has 19 classes whereas the data loader is trying to get 12?) because of the "--dataset" option. So I hardcoded this in test.py:
# n_classes = loader.n_classes
n_classes = 19

and ran the same command again to get the following error

/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:482: FutureWarning: Conversion of the second argument of issubdtype from int to np.signedinteger is deprecated. In future, it will be treated as np.int64 == np.dtype(int).type.
if issubdtype(ts, int):
/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:485: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
elif issubdtype(type(size), float):
Traceback (most recent call last):
File "test.py", line 162, in
test(args)
File "test.py", line 70, in test
outputs = model(images)
File "/home/ambareesh/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/home/ambareesh/Project_276C/gym_torcs/DDPG_PyTorch_Vanilla/pytorch-semseg/ptsemseg/models/icnet.py", line 202, in forward
x_sub4 = self.pyramid_pooling(x_sub4)
File "/home/ambareesh/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/home/ambareesh/Project_276C/gym_torcs/DDPG_PyTorch_Vanilla/pytorch-semseg/ptsemseg/models/utils.py", line 571, in forward
out = F.avg_pool2d(x, k_sizes[i], stride=strides[i], padding=0)
RuntimeError: Given input size: (1024x3x3). Calculated output size: (1024x0x-1). Output size is too small

Question:
My image (_11.jpeg) is small (64*64 RGB), I'm not sure if that is causing the error. I assumed since ICUnet is doing image segmentation, the size of the image didn't matter. Does it?

If so, where do I change the image size in the model? (to make sure that one of your pre-trained models works out of the box for me)

I am not very concerned about the accuracy of the model, I am using it for feature extraction, planning to extract the encoded layer in between, after everything starts to work.

Thank you!

[adam9500370]

Because n_classes in CamVid data loader is defined as 12, if you run with --dataset camvid, you will get the first error. You can run with --dataset cityscapes without the first error.

You can refer to my previous comment to solve the second error.

[erichhhhho]

@gshdong Hi, have you trained the ICNet yourself? And would you mind sharign how's the performance?

[ambareeshsrja16]

@adam9500370 Thank you for getting back! I tried your suggestion, but I seem to have some trouble with the dataloader part. The error seems to be the one mentioned here, so I'm assuming I haven't configured the dataset path clearly.

I had downloaded this dataset leftImg8bit_trainvaltest.zip (11GB) [md5] from the specified link. Is there some path variable I need to modify?

I am using the model for test i.e a single forward pass(or eval mode), and hence the command I'm using (from the repo readme) is:
python test.py [-h] [--model_path [MODEL_PATH]] [--dataset [DATASET]] [DCRF]] [--img_path [IMG_PATH]] [--out_path [OUT_PATH]]
which doesn't gave a config file or option to specify data path.

P.S - The error specifically is in this line from cityscapes_loader.py (line 82)
self.images_base = os.path.join(self.root, "leftImg8bit", self.split)
where self.root is None, and hence os.path.join is throwing an error

[adam9500370]

Because of this commit, it seems that test_mode variable in cityscapes_loader.py is not used.
You can try to replace these lines
https://github.com/meetshah1995/pytorch-semseg/blob/801fb200547caa5b0d91b8dde56b837da029f746/ptsemseg/loader/cityscapes_loader.py#L82-L85
with the following lines to solve root=None error:

    if not self.test_mode:
        self.images_base = os.path.join(self.root, "leftImg8bit", self.split)
        self.annotations_base = os.path.join(self.root, "gtFine", self.split)

        self.files[split] = recursive_glob(rootdir=self.images_base, suffix=".png")
    else:
        self.files[split] = [""] # to enable testine without loading original dataset