Distance calculation between two pedestrians from real-time video feeds

[sayakpaul]

Goal

To come up with a Python script that can take a video feed in real-time, detect pedestrians, and spit out their distances from each other.

Deliverables

• A Colab Notebook that demonstrates the distance calculation algorithm
• A Python script that can take a video feed in real-time, detect pedestrians, and spit out their distances from each other.

Considerations

Before starting to work on this issue, please discuss how you do you plan to implement the algorithm via the comments.

[hritik5102]

Implementation details : Docs

please can you go through it, if any changes are required please let me know
Thank you :)

[sayakpaul]

@hritik5102 thanks for providing it, we appreciate your efforts. But now that the program has started it would be a bit difficult for us to prioritize this review. But we will try our best. I hope you'd understand.

[hritik5102]

@hritik5102 thanks for providing it, we appreciate your efforts. But now that the program has started it would be a bit difficult for us to prioritize this review. But we will try our best. I hope you'd understand.

Yeah, I do understand.

[Sudarshana2000]

@rishiraj @sayakpaul Please review my work till now, provided in the colab notebook.
The visualisation of social distancing is provided in this video.

[sayakpaul]

@Sudarshana2000 amazing work! I am really impressed here. I love the way you have presented the entire material. Kudos!

[SubhasmitaSw]

@sayakpaul @rishiraj
Following is the colab Notebook containing all of my work. With Distance Calculation Algorithm shown explicitly and the visualization also shows all possible permutations of distances of pedestrains from each other.

I had tested the real-time detection of the model on NVIDIA Jetson Nano on my Local Machine with 24 FPS .

Here's the trained output on Sample Video and the BEV visualization

[sayakpaul]

@Ask-Subhasmita a couple of the pointers -

• We first need to see a Colab Notebook that can run entirely on Colab without requiring any Drive access. Please go through the other issues to avail yourself of a couple of options for doing this.
• I would reformat the markdown a bit. Since we are using markdown it's possible to hyperlink something. So, I would mention paper/blog/tutorial titles and then hyperlink them. Here's an example - A Battle of Text Detectors for Mobile Deployments: CRAFT vs. EAST and NOT https://sayak.dev/optimizing-text-detectors/.
• Regarding the FPS information, could you attach the output of that? You can refer to the same PyImageSearch blog post to see a good way to display the FPS information. Additionally, which Nano did you use? v1 or v2? Did you further optimize the SSD MobileNet model graph with TensorRT?
• You mentioned two object detectors - SSD MobileNet and YOLO. What's the purpose?
• Your BeV implementation and its use look really good. I would make the visualization a bit more explicit, though. From the current form, it's a bit problematic to understand what's really happening. Maybe consider including some warning messages or something similar? You would want to display all those necessary information to a regulatory body, right?

[SubhasmitaSw]

@sayakpaul

• Since I had messed up with my OS It will take some time to deploy the working of the entire code. Further, I'm in the mid of my final exams so I would request mentors to avail me some extended time.
• I would reformat every link along with proper documentation.
• The colab notebook would contain all the information along with other performance metrics displayed as graphs. NVIDIA Jetson Nano v2 was used and optimizing the model wasn't successful, encountered a lot of errors.
• Both Object Detectors were experimented with for some reason SSD model was a bit faster than YOLO. For future work I would train my own custom model specifically for the detection of pedestrians.
• I would rightfully Include all the violation warnings as explicit messages on Screen In the updated Colab Notebook.

[sayakpaul]

Since I had messed up with my OS It will take some time to deploy the working of the entire code. Further, I'm in the mid of my final exams so I would request mentors to avail me some extended time.

Not sure about the second part @rishiraj can clarify further. But you can always experiment with Colab. For now, I would suggest that only.

The colab notebook would contain all the information along with other performance metrics displayed as graphs. NVIDIA Jetson Nano v2 was used and optimizing the model wasn't successful, encountered a lot of errors.

What kind? TRT conversion is supposed to be pretty seamless. Check out the following resources if necessary -

• https://github.com/sayakpaul/MIRNet-TFLite-TRT/blob/main/MIRNet_TRT.ipynb
• https://sayak.dev/tf.keras/tensorrt/tensorflow/2020/07/01/accelerated-inference-trt.html

I believe with further optimizations the FPS can be further improved.

Both Object Detectors were experimented with for some reason SSD model was a bit faster than YOLO. For future work I would train my own custom model specifically for the detection of pedestrians.

SSDs would generally be faster than YOLO primarily because of the architectural benefits. And moreover, the particular variant you used makes use of MobileNet as the backbone, so, it's going to be faster. You're more than welcome to train your own model. Since time may be limited for WoC, I would keep it to pre-trained models only. But if you can wrap up model training within a feasible timeline, it'd be great.

[SubhasmitaSw]

Since time may be limited for WoC, I would keep it to pre-trained models only. But if you can wrap up model training within a feasible timeline, it'd be great.

I would try to do that alongside pre-trained models.
And will it be in the process to check for real-time using nano? or with sample video only?

[rishiraj]

@SubhasmitaSw since you had mentioned in your proposal about the end semester exam before the first evaluation, you can have some extension. But please adhere to your timeline in phase two. And as Sayak Paul mentioned, it is best to use Collab for now.

[SubhasmitaSw]

@rishiraj I will keep these in mind. Thank You.

[SubhasmitaSw]

WhatsApp.Video.2020-12-25.at.00.09.10.mp4

@sayakpaul , @rishiraj , Here's a bit update, warning msgs displayed for real-time, using Tiny-YOLO and next will share the codes along with evaluation metrics.

[sayakpaul]

@SubhasmitaSw that's definitely better. Could you also try out SSD MobileNet or MobileDet as the detector and see if the false positives could be reduced? You can check out the following thread for more details on the model - #5.

[SubhasmitaSw]

@sayakpaul Sure, I'll try both of them and let you know the results. However I'm facing a bit problem while accessing my webcam for realtime during coding on colab, some guidance will be helpful.

[sayakpaul]

You can use a video and run your pipeline when using Colab. Here's an example: https://github.com/sayakpaul/MIRNet-TFLite-TRT/blob/main/MIRNet_TRT.ipynb.