chainer.optimizer_hooks.GradientNoise

class chainer.optimizer_hooks.GradientNoise(eta, noise_func=<function exponential_decay_noise>)

Optimizer/UpdateRule hook function for adding gradient noise.

This hook function simply adds noise generated by the noise_func to the gradient. By default it adds time-dependent annealed Gaussian noise to the gradient at every training step:

\[g_t \leftarrow g_t + N(0, \sigma_t^2)\]

where

\[\sigma_t^2 = \frac{\eta}{(1+t)^\gamma}\]

with \(\eta\) selected from {0.01, 0.3, 1.0} and \(\gamma = 0.55\).

Parameters:

• eta (float) – Parameter that defines the scale of the noise, which for the default noise function is recommended to be either 0.01, 0.3 or 1.0.
• noise_func (function) – Noise generating function which by default is given by Adding Gradient Noise Improves Learning for Very Deep Networks.

Variables:

timing (string) – Specifies when this hook should be called by the Optimizer/UpdateRule. Valid values are ‘pre’ (before any updates) and ‘post’ (after any updates).

New in version 4.0.0: The timing parameter.

Methods

__call__(rule, param)

Call self as a function.

Attributes

call_for_each_param = True

name = 'GradientNoise'

timing = 'pre'