Ensure BlockNeuralAutoregressiveTransform maps real->real #1897
Conversation
|
Maybe using LeakyReLU likes in the paper instead? By the way, is there a reference for LeakyTanh? |
|
I believe that all the results from the paper use tanh. From personal experience (on admittedly a very limited set of applications), leaky relu performs far worse than either tanh or leaky tanh. No, I don't have a reference for it, I just chose it to solve the problem to ensure dy/dx is at least a threshold value, whilst maintaining similar results on a couple toy density estimation tasks. I completely understand if you'd prefer to keep it to how it is defined in the paper, but IMO the other issues are bound to arise some point e.g. people expecting (sensibly defined) BNAF guides to be supported on the same set as the true posterior, or expecting normalised densities when transforming with the inverse. |
| """ | ||
| An implementation of Block Neural Autoregressive neural network. | ||
|
|
||
| In contrast to the original paper, by default, we use ``LeakyTanh`` as the | ||
| activation, defined as :math:`y=Tanh(x) + cx` with :math:`c` being a small constant, |
There was a problem hiding this comment.
nit: c being a small constant (default to 0.01)
|
LGTM! Could you double check if the change still works for https://num.pyro.ai/en/stable/examples/covtype.html and https://num.pyro.ai/en/stable/examples/neutra.html tutorials? |
This fixes #1655 by changing BNAF activation to$y=\text{tanh}(x) + cx$ , by default (with c=0.01). This ensures the transform maps from real->real, which e.g.
BlockNeuralAutoregressiveTransformwill have the correct codomain (currently set asreal_vector)Let me know if you think there are any problems with this solution.