Possibility of choosing quality versus speed trade-offs to decrease frame processing times?

[mathieu-b]

Hi

First of all, thanks for the great work you are doing with 4dface and all the other related libraries.

I ran the latest pre-built release and it works correctly, however the example runs at about 5 FPS.

I have the following question: what would be the best ways / parameters that could be tweaked to decrease processing times, at the expense of losing some of the accuracy?

For example:

• would input image down-scaling help reduce times?
• could a simpler face mesh (with less vertices) help decrease mesh fitting times?

These the specs of the machine I'm using:

• Microsoft Windows 10 Pro
• NVIDIA GeForce GTX 1080
• Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz, 4001 Mhz, 4 main processors, 8 logic processors
• 32.0 GB of RAM
• Logitech C930e Webcam (USB 2.0, max FPS seems to be 30)

Regards

[patrikhuber]

Hi Mathieu,

Thank you very much! And apologies for the late reply.

I would suggest that you profile the code (e.g. run the Visual Studio profiler), and see in what function it spends most of the time.
From the top of my memory, the landmark detection and texture extraction would be the slow parts.
For the landmark detection, the slowest part is the HOG feature extraction. You could theoretically use a different, faster detector, like dlib. Or you could retrain an rcr-detector with fewer landmarks, so that fewer HOG features have to be extracted. I believe there's some other closed tickets either here or in the superviseddescent repo with a number of suggestions.
For the texture extraction, you could just program that in OpenGL, that would make it a lot faster. I think there's some (closed?) issues discussing this as well.

The fitting from eos, and the mesh, shouldn't be a slow-down, that should be less than 5ms per frame. If that's an issue there's some parameters that you can tune when calling the fit_shape_and_pose function but I think it'll be useless because the detector and texture extraction use up much more time.

If you're interested in a commercial solution, you could drop me an email, we're working on most of these things (or have already solved some of them).

[mathieu-b]

Hi Patrik

Ok, thanks for your answer :)
Good to know that you are working on a commercial option.

Regards