Chainer
stable

Tutorials

  • Chainer at a Glance
  • Concepts Walkthrough

Examples

  • Neural Net Examples
  • Colab Notebook Examples
  • Awesome Chainer

References

  • API Reference
    • Variable and Parameter
    • Functions
    • Link and Chains
      • Learnable connections
      • Activation/loss/normalization functions with parameters
      • Machine learning models
      • Pre-trained models
      • Link and Chain base classes
      • Link hooks
        • chainer.link_hooks.SpectralNormalization
        • chainer.link_hooks.TimerHook
        • chainer.link_hooks.WeightStandardization
        • chainer.LinkHook
    • Probability Distributions
    • Optimizers
    • Weight Initializers
    • Snapshot Writers
    • Training Tools
    • Datasets
    • Iterator
    • Serializers
    • Backends and Devices
    • Utilities
    • Configuring Chainer
    • Debug Mode
    • Visualization of Computational Graph
    • Static Subgraph Optimizations: Usage
    • Static Subgraph Optimizations: Design Notes
    • Caffe Model Support
    • Assertion and Testing
  • Installation
  • ChainerX Documentation
  • Distributed Deep Learning with ChainerMN
  • Export Chainer to ONNX

Other

  • API Compatibility Policy
  • Contribution Guide
  • Tips and FAQs
  • Performance Best Practices
  • Upgrade Guide
  • License

Community

  • Slack Chat
  • Forums
Chainer
  • Docs »
  • API Reference »
  • Link and Chains »
  • chainer.link_hooks.SpectralNormalization
  • Edit on GitHub

chainer.link_hooks.SpectralNormalization¶

class chainer.link_hooks.SpectralNormalization(n_power_iteration=1, eps=1e-06, use_gamma=False, factor=None, weight_name='W', name=None)[source]¶

Spectral Normalization link hook implementation.

This hook normalizes a weight using max singular value and this value is computed via power iteration method. Currently, this hook is supposed to be added to chainer.links.Linear, chainer.links.EmbedID, chainer.links.Convolution2D, chainer.links.ConvolutionND, chainer.links.Deconvolution2D, and chainer.links.DeconvolutionND. However, you can use this to other links like RNNs by specifying weight_name. It is highly recommended to add this hook before optimizer setup because this hook add a scaling parameter gamma if use_gamma is True. Otherwise, the registered gamma will not be updated.

\[\begin{split}\bar{\mathbf{W}} &=& \dfrac{\mathbf{W}}{\sigma(\mathbf{W})} \\ \text{, where} \ \sigma(\mathbf{W}) &:=& \max_{\mathbf{h}: \mathbf{h} \ne 0} \dfrac{\|\mathbf{W} \mathbf{h}\|_2}{\|\mathbf{h}\|_2} = \max_{\|\mathbf{h}\|_2 \le 1} \|\mathbf{W}\mathbf{h}\|_2\end{split}\]

See: T. Miyato et. al., Spectral Normalization for Generative Adversarial Networks

Parameters
  • n_power_iteration (int) – Number of power iteration. The default value is 1.

  • eps (float) – Numerical stability in norm calculation. The default value is 1e-6 for the compatibility with mixed precision training. The value used in the author’s implementation is 1e-12.

  • use_gamma (bool) – If True, weight scaling parameter gamma which is initialized by initial weight’s max singular value is introduced.

  • factor (float, None) – Scaling parameter to divide maximum singular value. The default value is 1.0.

  • weight_name (str) – Link’s weight name to apply this hook. The default value is 'W'.

  • name (str or None) – Name of this hook. The default value is 'SpectralNormalization'.

Variables
  • vector_name (str) – Name of the approximate first left singular vector registered in the target link. the target link.

  • axis (int) – Axis of weight represents the number of output feature maps or output units (out_channels and out_size, respectively).

Example

There are almost the same but 2 ways to apply spectral normalization (SN) hook to links.

1. Initialize link and SN separately. This makes it easy to handle buffer and parameter of links registered by SN hook.

>>> l = L.Convolution2D(3, 5, 3)
>>> hook = chainer.link_hooks.SpectralNormalization()
>>> _ = l.add_hook(hook)
>>> # Check the shape of the first left singular vector.
>>> getattr(l, hook.vector_name).shape
(5,)
>>> # Delete SN hook from this link.
>>> l.delete_hook(hook.name)

2. Initialize both link and SN hook at one time. This makes it easy to define your original Chain.

>>> # SN hook handles lazy initialization!
>>> layer = L.Convolution2D(
...     5, 3, stride=1, pad=1).add_hook(
...         chainer.link_hooks.SpectralNormalization())

Methods

__enter__()[source]¶
__exit__()[source]¶
added(link)[source]¶

Callback function invoked when the link hook is registered

Parameters

link (Link) – Link object to which the link hook is registered. None if the link hook is registered globally.

deleted(link)[source]¶

Callback function invoked when the link hook is unregistered

Parameters

link (Link) – Link object to which the link hook is unregistered. None if the link hook had been registered globally.

forward_postprocess(cb_args)[source]¶

Callback function invoked after a forward call of a link.

Parameters

args –

Callback data. It has the following attributes:

  • link (Link)

    Link object.

  • forward_name (str)

    Name of the forward method.

  • args (tuple)

    Non-keyword arguments given to the forward method.

  • kwargs (dict)

    Keyword arguments given to the forward method.

  • out

    Return value of the forward method.

forward_preprocess(cb_args)[source]¶

Callback function invoked before a forward call of a link.

Parameters

args –

Callback data. It has the following attributes:

  • link (Link)

    Link object.

  • forward_name (str)

    Name of the forward method.

  • args (tuple)

    Non-keyword arguments given to the forward method.

  • kwargs (dict)

    Keyword arguments given to the forward method.

normalize_weight(link)[source]¶

Normalize target weight before every single forward computation.

reshape_W(W)[source]¶

Reshape & transpose weight into 2D if necessary.

__eq__(value, /)¶

Return self==value.

__ne__(value, /)¶

Return self!=value.

__lt__(value, /)¶

Return self<value.

__le__(value, /)¶

Return self<=value.

__gt__(value, /)¶

Return self>value.

__ge__(value, /)¶

Return self>=value.

Attributes

name = 'SpectralNormalization'¶
Next Previous

© Copyright 2015, Preferred Networks, inc. and Preferred Infrastructure, inc. Revision e9da1423.

Built with Sphinx using a theme provided by Read the Docs.
Read the Docs v: stable
Versions
latest
stable
v7.8.1
v7.8.0
v7.7.0
v7.4.0
v7.2.0
v7.1.0
v7.0.0
v6.7.0
v6.6.0
v6.5.0
v6.4.0
v6.3.0
v6.2.0
v6.1.0
v6.0.0
v6
v5.4.0
v5.3.0
v5.2.0
v5.1.0
v5.0.0
v4.5.0
v4.4.0
v4.3.1
v4.3.0
v4.2.0
v4.1.0
v4.0.0
v3.5.0
v3.4.0
v3.3.0
v3.2.0
v3.1.0
v3.0.0
v2.1.0
v2.0.2
v1.24.0
v1.23.0
v1.22.0
v1.21.0
v1.20.0.1
v1.19.0
v1.18.0
v1.17.0
v1.16.0
v1.15.0.1
v1.14.0
v1.13.0
v1.12.0
v1.11.0
v1.10.0
v1.9.1
v1.8.2
v1.7.2
v1.6.2.1
v1.5.1
v1.4.1
v1.3.2
v1.2.0
v1.1.2
v1.0.1
Downloads
pdf
html
epub
On Read the Docs
Project Home
Builds

Free document hosting provided by Read the Docs.