Question about KFACLaplace for BatchNorm

[jeffwillette]

I couldn't many implementations for KFACLaplace so I was implementing it from your baseline in this repo. I noticed a few things about the handling of batchnorm that I am curious about...

• At this line https://github.com/wjmaddox/swa_gaussian/blob/master/swag/posteriors/laplace.py#L190 you give a rather strong hardcoded value for Tikhonov regularization on BatchNorm Layers. Is there a reason for this extreme value above the epsilon values for the rest of the layers?
• A few lines later it seems that the Cholesky of the Hessian for BN is taken, but then it is stored in the state as the inverse Choleksy. I don't think the inverse of the BN Hessian or its Cholesky is ever taken. Is this correct? https://github.com/wjmaddox/swa_gaussian/blob/master/swag/posteriors/laplace.py#L202