Average Precision and Recall negative (-1.000) and No Prediction Results

[jwnz]

I have two major issues.

1. In the training process, the Average Recall and Precision for small and medium are both negative (-1).
2. After training, regardless of the value of Average Precision (area= Large), I am unable to produce a single bounding box. This also applies to when I try to infer a bounding box on an image in the training set as well.

These problems seems to be common as there are many related issues:
negative precision/recall

• Average Recall and Average Precision are -1 #233
• Average Precision and recall are -1 #164
• mAP is negative during trainning #286

My input image sizes vary slightly but the vast majority of each image is of size 1920x1080

Below is an example of the validation output.
Note: I am using the same images for validation and training.

DONE (t=0.01s).
IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.030
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.030
2021-04-09 11:56:55,228 train.py[line:447] INFO: Created checkpoint directory

Below is the command I'm using to train. I have tried several learning rates. It has no effect on the small and medium recall.

python train.py -l 0.25 -g 0 -pretrained ./pytorch-YOLOv4/yolov4.conv.137.pth -classes 1 -dir ../images -train_label_path ./data/simple.txt

The input for the bounding boxes is correct; Here is an example or my train.txt:

./data/368.jpg 272,391,1290,929,0
./data/8.jpg 322,114,1293,909,0
./data/737.jpg 681,46,1152,1079,0
...

In the cfg.py file I have changed the variables as folllows:

• Cfg.batch=4
• Cfg.subdivisions=1
• Cfg.width=416
• Cfg.height=416
• Cfg.max_batches=2000 # I'm only training a single new class
• Cfg.steps=1600,1800
  - Cfg.boxes=1 # only one box per image

I modified the yolov4-custom.cfg file to match the cfg.py file as well as made the changes mentioned in THIS link.
I changed the image_size variable in the Yolo_loss class to match the width and height (416) in the cfg file.
I created my own get_image_id function in the dataset.py file to return an integer representing a particular image.

Here is an example of my prediction output:

Loading weights from Yolov4_epoch3.pth... Done!
-----------------------------------
           Preprocess : 0.000637
      Model Inference : 0.818682
-----------------------------------
/???/???/???/pytorch-YOLOv4/tool/utils.py:199: RuntimeWarning: invalid value encountered in greater
  argwhere = max_conf[i] > conf_thresh
-----------------------------------
       max and argmax : 0.000091
                  nms : 0.000212
Post processing total : 0.000302
-----------------------------------
[]
-----------------------------------
           Preprocess : 0.000371
      Model Inference : 0.729752
-----------------------------------
-----------------------------------
       max and argmax : 0.000088
                  nms : 0.000046
Post processing total : 0.000135
-----------------------------------
[]
./data/368.jpg: Predicted in 0.732061 seconds.
save plot results to predictions.jpg

If I modify the thresholds in demo.py file in the detect_cv2 funrtion from 0.4 and 0.6 to a negative number, I get bounding boxes, but here is a rough example of the results: [0.927, 0.927, 0.927, 0.927, 0, 0, 0]

I have absolutely no clue where I'm going wrong here. Am I missing something?

[missFuture]

I also have the problem.

my result
continue training from last 100 epoch train-weights

tensorboard:

we can get that the loss is down to hundred(200) from a high value

AP AR is almost zero

but when i test each validation or test image, i can get nearly correct prediction.like this

what i have done under this repo?

• my train and val images are 608*608, which fits the input size(darknet)
• modify my yolov4-custom.cfg, and set Cfg.use_darknet_cfg = false in cfg.py . refer to RuntimeError: shape '[1, 3, 29, 76, 76]' is invalid for input of size 1472880 #138
• train for 100 epoch from the pretrain model yolov4.conv.137.pth
• set batch = 4 and subdivision=1 as Tianxiaomo did.

i dont know how to solve this problem. please help me

[jwnz]

I believe the issue I'm having is related to the size of the training images. Due to the memory constraints of my GPU (CPU training just doesn't work on my machine), I was forced to train the model with an image size of 416x416. However, the item (and coincidentally the bounding boxes) I was trying to detect are significantly larger then the width and height of 416.

My hypothesis is that by increasing the height and width , the two issues I mentioned in my original post above may be solved. Unfortunately, I don't have the resources to test this hypothesis.

I hope that this helps someone. However, I have managed to solve the issues I was having regarding this task by simply using another repository.

[BruceDai003]

I also have the problem.

my result
continue training from last 100 epoch train-weights

tensorboard:

we can get that the loss is down to hundred(200) from a high value

AP AR is almost zero

but when i test each validation or test image, i can get nearly correct prediction.like this

what i have done under this repo?

• my train and val images are 608*608, which fits the input size(darknet)
• modify my yolov4-custom.cfg, and set Cfg.use_darknet_cfg = false in cfg.py . refer to RuntimeError: shape '[1, 3, 29, 76, 76]' is invalid for input of size 1472880 #138
• train for 100 epoch from the pretrain model yolov4.conv.137.pth
• set batch = 4 and subdivision=1 as Tianxiaomo did.

i dont know how to solve this problem. please help me

I like your sesame-paste. What's your command to train?

[tylister]

I also have the problem.

my result
continue training from last 100 epoch train-weights

tensorboard:

we can get that the loss is down to hundred(200) from a high value

AP AR is almost zero

but when i test each validation or test image, i can get nearly correct prediction.like this

what i have done under this repo?

• my train and val images are 608*608, which fits the input size(darknet)
• modify my yolov4-custom.cfg, and set Cfg.use_darknet_cfg = false in cfg.py . refer to RuntimeError: shape '[1, 3, 29, 76, 76]' is invalid for input of size 1472880 #138
• train for 100 epoch from the pretrain model yolov4.conv.137.pth
• set batch = 4 and subdivision=1 as Tianxiaomo did.

i dont know how to solve this problem. please help me

what's the format of your dataset?