Loss and performance

[dongli96]

Hello,

Thank you for your nice code.

I noticed that in the training process, the batch loss always oscillated within the range of 0.02~0.07 and did not descent a lot even the model is trained for like 100 epochs. But I indeed saw the improved quality of generated images as the training proceeds.

I know you alreay discussed a similar issue in #40. But it is still my curiosity that why the generation quality improves with the loss plateauing. My guess is that in the first epochs the model makes small mistakes for nearly all the samples while in the later epochs the model predicts exactly for most samples while making big mistakes at some minor samples, which makes the overall loss value close but the performance different.

I looking forward to hearing your insightful ideas.