fathom-train output is perhaps a little too stochastic

[danielhertenstein]

Sometimes, you may run fathom-train twice in a row and get significantly different output metrics (You'll also always get different coefficients, but this is to be expected due to random initialization of the weights and the size of the learning rate). We currently think this is due to the optimizer getting stuck in a local minimum. We should find ways to prevent that.

This has the practical consequence in that when you are close to "done" with a model and trying to find the best performing set of rules or positive weighting factor or decision threshold, the slight changes in the coefficients from fathom-train can have a significant influence on the output metrics of fathom-test. It would be nice to dampen this variability in fathom-train coefficients so comparing models is easier/possible.

A first possible solution is changing the early stopping mechanism to let the loss go back up for a certain number of iterations before stopping and saying the previously encountered minimum is the best. This could also be combined with a larger learning rate to allow the optimizer to escape a local minimum faster, though we should be wary of the model not converging because of this.