Skip to content

Commit

Permalink
Fix broken link in NovoGrad docstring (#2123)
Browse files Browse the repository at this point in the history
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
  • Loading branch information
1 parent 84f03d6 commit 81529ff
Showing 1 changed file with 10 additions and 10 deletions.
20 changes: 10 additions & 10 deletions tensorflow_addons/optimizers/novograd.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,16 +23,16 @@

@tf.keras.utils.register_keras_serializable(package="Addons")
class NovoGrad(tf.keras.optimizers.Optimizer):
"""The NovoGrad Optimizer was first proposed in [Stochastic Gradient
Methods with Layerwise Adaptvie Moments for training of Deep
Networks](https://arxiv.org/pdf/1905.11286.pdf)
NovoGrad is a first-order SGD-based algorithm, which computes second
moments per layer instead of per weight as in Adam. Compared to Adam,
NovoGrad takes less memory, and has been found to be more numerically
stable. More specifically we compute (for more information on the
computation please refer to this
[link](https://nvidia.github.io/OpenSeq2Seq/html/optimizers.html):
"""Optimizer that implements NovoGrad.
The NovoGrad Optimizer was first proposed in [Stochastic Gradient
Methods with Layerwise Adaptive Moments for training of Deep
Networks](https://arxiv.org/pdf/1905.11286.pdf) NovoGrad is a
first-order SGD-based algorithm, which computes second moments per
layer instead of per weight as in Adam. Compared to Adam, NovoGrad
takes less memory, and has been found to be more numerically stable.
(For more information on the computation please refer to this
[link](https://nvidia.github.io/OpenSeq2Seq/html/optimizers.html))
Second order moment = exponential moving average of Layer-wise square
of grads:
Expand Down

0 comments on commit 81529ff

Please sign in to comment.