chainer.link_hooks.SpectralNormalization

class chainer.link_hooks.SpectralNormalization(n_power_iteration=1, eps=1e-06, use_gamma=False, factor=None, weight_name='W', name=None)

Spectral Normalization link hook implementation.

This hook normalizes a weight using max singular value and this value is computed via power iteration method. Currently, this hook is supposed to be added to chainer.links.Linear, chainer.links.EmbedID, chainer.links.Convolution2D, chainer.links.ConvolutionND, chainer.links.Deconvolution2D, and chainer.links.DeconvolutionND. However, you can use this to other links like RNNs by specifying weight_name. It is highly recommended to add this hook before optimizer setup because this hook add a scaling parameter gamma if use_gamma is True. Otherwise, the registered gamma will not be updated.

\[\begin{split}\bar{\mathbf{W}} &=& \dfrac{\mathbf{W}}{\sigma(\mathbf{W})} \\ \text{, where} \ \sigma(\mathbf{W}) &:=& \max_{\mathbf{h}: \mathbf{h} \ne 0} \dfrac{\|\mathbf{W} \mathbf{h}\|_2}{\|\mathbf{h}\|_2} = \max_{\|\mathbf{h}\|_2 \le 1} \|\mathbf{W}\mathbf{h}\|_2\end{split}\]

See: T. Miyato et. al., Spectral Normalization for Generative Adversarial Networks

Parameters

• n_power_iteration (int) – Number of power iteration. The default value is 1.

• eps (float) – Numerical stability in norm calculation. The default value is 1e-6 for the compatibility with mixed precision training. The value used in the author’s implementation is 1e-12.

• use_gamma (bool) – If True, weight scaling parameter gamma which is initialized by initial weight’s max singular value is introduced.

• factor (float, None) – Scaling parameter to divide maximum singular value. The default value is 1.0.

• weight_name (str) – Link’s weight name to apply this hook. The default value is 'W'.

• name (str or None) – Name of this hook. The default value is 'SpectralNormalization'.

Variables

• ~SpectralNormalization.vector_name (str) – Name of the approximate first left singular vector registered in the target link. the target link.

• ~SpectralNormalization.axis (int) – Axis of weight represents the number of output feature maps or output units (out_channels and out_size, respectively).

Example

There are almost the same but 2 ways to apply spectral normalization (SN) hook to links.

1. Initialize link and SN separately. This makes it easy to handle buffer and parameter of links registered by SN hook.

>>> l = L.Convolution2D(3, 5, 3)
>>> hook = chainer.link_hooks.SpectralNormalization()
>>> _ = l.add_hook(hook)
>>> # Check the shape of the first left singular vector.
>>> getattr(l, hook.vector_name).shape
(5,)
>>> # Delete SN hook from this link.
>>> l.delete_hook(hook.name)

2. Initialize both link and SN hook at one time. This makes it easy to define your original Chain.

>>> # SN hook handles lazy initialization!
>>> layer = L.Convolution2D(
...     5, 3, stride=1, pad=1).add_hook(
...         chainer.link_hooks.SpectralNormalization())

Methods

__enter__()

__exit__()

added(link)

Callback function invoked when the link hook is registered

Parameters

link (Link) – Link object to which the link hook is registered. None if the link hook is registered globally.

deleted(link)

Callback function invoked when the link hook is unregistered

Parameters

link (Link) – Link object to which the link hook is unregistered. None if the link hook had been registered globally.

forward_postprocess(cb_args)

Callback function invoked after a forward call of a link.

Parameters

args –

Callback data. It has the following attributes:

• link (Link)

  Link object.

• forward_name (str)

  Name of the forward method.

• args (tuple)

  Non-keyword arguments given to the forward method.

• kwargs (dict)

  Keyword arguments given to the forward method.

• out

  Return value of the forward method.

forward_preprocess(cb_args)

Callback function invoked before a forward call of a link.

Parameters

args –

Callback data. It has the following attributes:

• link (Link)

  Link object.

• forward_name (str)

  Name of the forward method.

• args (tuple)

  Non-keyword arguments given to the forward method.

• kwargs (dict)

  Keyword arguments given to the forward method.

normalize_weight(link)

Normalize target weight before every single forward computation.

reshape_W(W)

Reshape & transpose weight into 2D if necessary.

__eq__(value, /)

Return self==value.

__ne__(value, /)

Return self!=value.

__lt__(value, /)

Return self<value.

__le__(value, /)

Return self<=value.

__gt__(value, /)

Return self>value.

__ge__(value, /)

Return self>=value.

Attributes

name = 'SpectralNormalization'