chainer.link_hooks.WeightStandardization¶
-
class
chainer.link_hooks.
WeightStandardization
(*, eps=1e-05, weight_name='W', name=None)[source]¶ Weight Standardization (WS) link hook implementation.
This hook standardizes a weight by weight statistics.
This link hook implements a WS which computes the mean and variance along axis “output channels”, then normalizes by these statistics. WS improves training by reducing the Lipschitz constants of the loss and the gradients like batch normalization (BN) but without relying on large batch sizes during training. Specifically, the performance of WS with group normalization (GN) trained with small-batch is able to match or outperforms that of BN trained with large-batch. WS is originally proposed for 2D convolution layers followed by mainly GN and sometimes BN. Note that this hook is able to handle layers such as N-dimensional convolutional, linear and embedding layers but there is no guarantee that this hook helps training.
See: Siyuan Qiao et. al., Weight Standardization
- Parameters
Methods
-
added
(link)[source]¶ Callback function invoked when the link hook is registered
- Parameters
link (Link) – Link object to which the link hook is registered.
None
if the link hook is registered globally.
-
deleted
(link: Optional[chainer.link.Link]) → None[source]¶ Callback function invoked when the link hook is unregistered
- Parameters
link (Link) – Link object to which the link hook is unregistered.
None
if the link hook had been registered globally.
-
__eq__
()¶ Return self==value.
-
__ne__
()¶ Return self!=value.
-
__lt__
()¶ Return self<value.
-
__le__
()¶ Return self<=value.
-
__gt__
()¶ Return self>value.
-
__ge__
()¶ Return self>=value.
Attributes
-
name
= 'WeightStandardization'¶