chainer.link_hooks.WeightStandardization

class chainer.link_hooks.WeightStandardization(*, eps=1e-05, weight_name='W', name=None)

Weight Standardization (WS) link hook implementation.

This hook standardizes a weight by weight statistics.

This link hook implements a WS which computes the mean and variance along axis “output channels”, then normalizes by these statistics. WS improves training by reducing the Lipschitz constants of the loss and the gradients like batch normalization (BN) but without relying on large batch sizes during training. Specifically, the performance of WS with group normalization (GN) trained with small-batch is able to match or outperforms that of BN trained with large-batch. WS is originally proposed for 2D convolution layers followed by mainly GN and sometimes BN. Note that this hook is able to handle layers such as N-dimensional convolutional, linear and embedding layers but there is no guarantee that this hook helps training.

See: Siyuan Qiao et. al., Weight Standardization

Parameters

• eps (float) – Numerical stability in standard deviation calculation. The default value is 1e-5.

• weight_name (str) – Link’s weight name to appky this hook. The default value is 'W'.

• name (str or None) – Name of this hook. The default value is 'WeightStandardization'.

Methods

__enter__()

__exit__()

added(link)

Callback function invoked when the link hook is registered

Parameters

link (Link) – Link object to which the link hook is registered. None if the link hook is registered globally.

deleted(link: Optional[chainer.link.Link]) → None

Callback function invoked when the link hook is unregistered

Parameters

link (Link) – Link object to which the link hook is unregistered. None if the link hook had been registered globally.

forward_postprocess(cb_args)

Callback function invoked after a forward call of a link.

Parameters

args –

Callback data. It has the following attributes:

• link (Link)

  Link object.

• forward_name (str)

  Name of the forward method.

• args (tuple)

  Non-keyword arguments given to the forward method.

• kwargs (dict)

  Keyword arguments given to the forward method.

• out

  Return value of the forward method.

forward_preprocess(cb_args)

Callback function invoked before a forward call of a link.

Parameters

args –

Callback data. It has the following attributes:

• link (Link)

  Link object.

• forward_name (str)

  Name of the forward method.

• args (tuple)

  Non-keyword arguments given to the forward method.

• kwargs (dict)

  Keyword arguments given to the forward method.

__eq__(value, /)

Return self==value.

__ne__(value, /)

Return self!=value.

__lt__(value, /)

Return self<value.

__le__(value, /)

Return self<=value.

__gt__(value, /)

Return self>value.

__ge__(value, /)

Return self>=value.

Attributes

name = 'WeightStandardization'