Function opimization routines + tests

[mzhurovich]

What has been done:

• Function optimization routines were moved to fncas_optimize.h
• These routines are now classes Optimizer<F>, which use F::compute() in their mainOptimize() function
• In addition to the original naive gradient descent, implemented:
  1. gradient descent with backtracking
  2. simple conjugate gradients method (backtracking + Polak–Ribière)
• A number of tests, including algorithm comparison

[mzhurovich]

@dkorolev Look at this, please :)

[dkorolev]

We'll make these class members later on, no need to parse by name and vice versa.

[dkorolev]

Hi @mzhurovich ,

Looks really good! I left a few minor comments. The top-level takeaway message is:

1. A few more math-related methods can go into mathutils.h, it would make the math optimization code a bit more readable.
2. I personally vote for much stricter NaN and INF checking. From my experience, those issues will show up soon enough on real-life functions, since exponents are common in ML costs.
3. More comments on the methods and functions would be great to have. Links to Wikipedia/Wolfram/MathWorld would often be enough.

Thanks!
Dima

[mzhurovich]

@dkorolev Hey, just added latest changes, PTAL