Fix NaNs during training on 5-appliances

[JackKelly]

Ideas:

• Find a minimal example which still fails (something that trains fast)
• Is it:
  • The new nntools code (check by re-running e82)
  • dodgy data (check using old_X and old_y)
  • initialisation?
  • learning rate?
  • different learning algorithm?
    • Try adagrad?
    • Try skaae's adadelta_normscaled
    • "rescale the learning rate according to the difference in cost normalization"
  • try concat layer (see this comment)
  • Different cost function.
    • Read this about categorical_crossentropy. Perhaps I should be using the binary_crossentropy (their simple auto-associator sounds vaguely similar to what I'm doing). - oops, we are already using the binary_crossentropy!
    • Try MSE
  • doesn't like producing 5 outputs?
    try single-appliance experts.
  • smaller seq length?
  • smaller batch size?
  • doesn't like range of targets (maybe shouldn't get right to 1?)
  • quantized inputs
  • do we get NaNs if we remove the Conv1D layers?
  • do we get NaNs if we swap from BLSTM to LSTM?
  • do we get NaNs if we remove (B)LSTM layers?
• Hacks:
  • Save, say, last 5 training examples. When we hit a bad patch, examine these examples.
  • Save network weights. If we get a bad bit of training then revert network to, say, 5 steps ago.