Asynchronous validation

[jindrahelcl]

Inspired by Nematus, the validation could be done asynchronously.

Do you think it's a good idea?
How do you think this should be implemented?
Are there any non-backward-compatible aspects? (I don't think there are, not sure though)

[jlibovicky]

Generally, I think it is a good idea, but I am not sure how to implement it properly.

Maybe we can have a session that is entirely on CPU, and a separate thread will load the current variables and do the validation. We can also do the validation continuously, regardless on the exact number of the training data - just start a new validation when the previous one has ended. Is it possible in TF?

In any case, I would wait with it until I merge the workflow_refactor branch (and untli the other never-ending branches are merged).