Update PSPNet and ICNet

[adam9500370]

Modifications:

1. In cityscapes_loader.py, add args for mean version and img_norm, and input type for scipy.misc.imresize needs to be uint8 with RGB mode (since original pretrained model uses pascal mean image, and image doesn't normalize to [0,1])
2. Fix wrong number n_blocks for bottoleNeckIdentityPSP in residualBlockPSP function (n_blocks -> n_blocks-1)
3. Add auxiliary training layers for training PSPNet
4. Modify tile_predict with flip arg and support for batch with tensor type
5. Add PSPNet support for training and testing (extra args: img_norm, eval_flip, measure_time)

Validation results on cityscapes validation set (mIoU/pAcc):
Without flip: 78.65/96.31
With flip: 78.80/96.34
I feed images into network input size: 1025x2049 and single scale (since model input is odd numbers (713x713))
Run on single GTX 1080TI, time is about 1.2~1.3 fps
python validate.py --model_path checkpoints/pspnet_101_cityscapes.pth --dataset cityscapes --img_rows 1025 --img_cols 2049 --no-img_norm --eval_flip --measure_time --batch_size 2 --split val

Pretrained models in pytorch:
pspnet_50_ade20k.pth
pspnet_101_cityscapes.pth
pspnet_101_pascalvoc.pth

[adam9500370]

Thank you for your nice implementations.
I think PSPNet weight loaded from caffemodel is ok.
By the way, I also implement near real-time (avg ~23 fps) ICNet based on this PSPNet, the followings are results on cityscapes validation set (without flip) (mIoU/pAcc):
ICNet_train_30k: 65.86/94.21
ICNet_trainval_90k: 79.18/96.31
Do you want to merge my ICNet implementation in the current repo?

[meetps]

@adam9500370 - Thanks for the PR!

Could you update the PR to have the ICNet as well? It'd be great to have ICNet in the suite as well.

[meetps]

Perhaps it would be faster if we used torch.nn.functional.upsample_nearest instead of scipy imresize.

[adam9500370]

I will rewrite this part and update ICNet.
I will create auxiliary loss function for PSPNet and ICNet, and model outputs adjusted by training or eval mode to make train.py, validate.py, and test.py scripts clean).

[meetps]

This will break the test script for all other dataloaders, which don't have the img_norm argument.

[adam9500370]

Since I just add this arg in cityscapes dataloader, I will add it in other dataloaders.

[meetps]

Looks good to me, need to make very minor some changes in the dataloader to incorporate the img_norm.

Looking forward to the ICNet Implementation!. I will merge asap.

[adam9500370]

Update:

1. Add img_norm arg to all dataloaders and change order for scipy.misc.imresize
2. Add auxiliary loss func (multi_scale_cross_entropy2d)
3. Add loss and modify model outputs adjusted by training or eval mode in PSPNet
4. Make train.py, validate.py, test.py scripts clean
5. Update ICNet support

Pretrained models:
ICNet_train_30k
ICNet_train_30k_bn
ICNet_trainval_90k
ICNet_trainval_90k_bn

[nichalin]

Hi adam9500370, thanks for sharing. Could you please also share how you trained those PSPnet and ICNet models?

[adam9500370]

Hi, @nichalin .
I don't really train those models.
The validation results above are obtained by converting from original pretrained caffe models to pytorch models.

[nichalin]

Got it. Thanks, @adam9500370.

[lxtGH]

Hi !!! @nichalin Did you do the crop prediction to get 78.80 mIOU /96.34
I use the model you provided but I only got 77.32 mIOU. I think crop test may be better.

[lxtGH]

OK, I found the reason, because I use the latest pytorch, I need to set align_corners=True to get your answer, I will do the crop test

[erichhhhho]

@adam9500370 @meetshah1995 @lxtGH
Hi guys, when I was doing inference using the pretrained model pspnet_101_cityscapes.pth of PSPNet on Cityscape dataset. (With input image size 1025x2049 as @adam9500370 mentioned)

I met the following problem:

• RuntimeError: CUDA out of memory. Tried to allocate 130.00 MiB (GPU 0; 10.91 GiB total capacity; 10.12 GiB already allocated; 115.38 MiB free; 85.29 MiB cached)

May I ask about how much GPU memory is needed to conduct full resolution inference? Thx

[lxtGH]

@erichhhhho I guess you should inference with the code like this:
with torch.no_grad():
out = model(img)

[erichhhhho]

@erichhhhho I guess you should inference with the code like this:
with torch.no_grad():
out = model(img)

Thank you!!

[erichhhhho]

@lxtGH @adam9500370 @meetshah1995
Hi, I think the valset mIoU on PSPNet Cityscape in here(78.65) we can get is only on 713*173. However, the original implementation is reported on 1024*2048 resolution(79.70). And in here, if the inference size is 1024*2048, the mIoU is 0.6759.

I think the performance gap is mainly because the inference method. It is related to the sliding prediction scheme discussed here

Has anyone tried this before?