Chainer
stable

Tutorials

  • Chainer at a Glance
  • Concepts Walkthrough

Examples

  • Neural Net Examples
  • Colab Notebook Examples
  • Awesome Chainer

References

  • API Reference
    • Variable and Parameter
    • Functions
    • Link and Chains
      • Learnable connections
      • Activation/loss/normalization functions with parameters
      • Machine learning models
      • Pre-trained models
      • Link and Chain base classes
      • Link hooks
        • chainer.link_hooks.SpectralNormalization
        • chainer.link_hooks.TimerHook
        • chainer.link_hooks.WeightStandardization
        • chainer.LinkHook
    • Probability Distributions
    • Optimizers
    • Weight Initializers
    • Snapshot Writers
    • Training Tools
    • Datasets
    • Iterator
    • Serializers
    • Backends and Devices
    • Utilities
    • Configuring Chainer
    • Debug Mode
    • Visualization of Computational Graph
    • Static Subgraph Optimizations: Usage
    • Static Subgraph Optimizations: Design Notes
    • Caffe Model Support
    • Assertion and Testing
  • Installation
  • ChainerX Documentation
  • Distributed Deep Learning with ChainerMN
  • Export Chainer to ONNX

Other

  • API Compatibility Policy
  • Contribution Guide
  • Tips and FAQs
  • Performance Best Practices
  • Upgrade Guide
  • License

Community

  • Slack Chat
  • Forums
Chainer
  • Docs »
  • API Reference »
  • Link and Chains »
  • chainer.link_hooks.WeightStandardization
  • Edit on GitHub

chainer.link_hooks.WeightStandardization¶

class chainer.link_hooks.WeightStandardization(*, eps=1e-05, weight_name='W', name=None)[source]¶

Weight Standardization (WS) link hook implementation.

This hook standardizes a weight by weight statistics.

This link hook implements a WS which computes the mean and variance along axis “output channels”, then normalizes by these statistics. WS improves training by reducing the Lipschitz constants of the loss and the gradients like batch normalization (BN) but without relying on large batch sizes during training. Specifically, the performance of WS with group normalization (GN) trained with small-batch is able to match or outperforms that of BN trained with large-batch. WS is originally proposed for 2D convolution layers followed by mainly GN and sometimes BN. Note that this hook is able to handle layers such as N-dimensional convolutional, linear and embedding layers but there is no guarantee that this hook helps training.

See: Siyuan Qiao et. al., Weight Standardization

Parameters
  • eps (float) – Numerical stability in standard deviation calculation. The default value is 1e-5.

  • weight_name (str) – Link’s weight name to appky this hook. The default value is 'W'.

  • name (str or None) – Name of this hook. The default value is 'WeightStandardization'.

Methods

__enter__()[source]¶
__exit__()[source]¶
added(link)[source]¶

Callback function invoked when the link hook is registered

Parameters

link (Link) – Link object to which the link hook is registered. None if the link hook is registered globally.

deleted(link: Optional[chainer.link.Link]) → None[source]¶

Callback function invoked when the link hook is unregistered

Parameters

link (Link) – Link object to which the link hook is unregistered. None if the link hook had been registered globally.

forward_postprocess(cb_args)[source]¶

Callback function invoked after a forward call of a link.

Parameters

args –

Callback data. It has the following attributes:

  • link (Link)

    Link object.

  • forward_name (str)

    Name of the forward method.

  • args (tuple)

    Non-keyword arguments given to the forward method.

  • kwargs (dict)

    Keyword arguments given to the forward method.

  • out

    Return value of the forward method.

forward_preprocess(cb_args)[source]¶

Callback function invoked before a forward call of a link.

Parameters

args –

Callback data. It has the following attributes:

  • link (Link)

    Link object.

  • forward_name (str)

    Name of the forward method.

  • args (tuple)

    Non-keyword arguments given to the forward method.

  • kwargs (dict)

    Keyword arguments given to the forward method.

__eq__(value, /)¶

Return self==value.

__ne__(value, /)¶

Return self!=value.

__lt__(value, /)¶

Return self<value.

__le__(value, /)¶

Return self<=value.

__gt__(value, /)¶

Return self>value.

__ge__(value, /)¶

Return self>=value.

Attributes

name = 'WeightStandardization'¶
Next Previous

© Copyright 2015, Preferred Networks, inc. and Preferred Infrastructure, inc. Revision e9da1423.

Built with Sphinx using a theme provided by Read the Docs.
Read the Docs v: stable
Versions
latest
stable
v7.8.1
v7.8.0
v7.7.0
v7.4.0
v7.2.0
v7.1.0
v7.0.0
v6.7.0
v6.6.0
v6.5.0
v6.4.0
v6.3.0
v6.2.0
v6.1.0
v6.0.0
v6
v5.4.0
v5.3.0
v5.2.0
v5.1.0
v5.0.0
v4.5.0
v4.4.0
v4.3.1
v4.3.0
v4.2.0
v4.1.0
v4.0.0
v3.5.0
v3.4.0
v3.3.0
v3.2.0
v3.1.0
v3.0.0
v2.1.0
v2.0.2
v1.24.0
v1.23.0
v1.22.0
v1.21.0
v1.20.0.1
v1.19.0
v1.18.0
v1.17.0
v1.16.0
v1.15.0.1
v1.14.0
v1.13.0
v1.12.0
v1.11.0
v1.10.0
v1.9.1
v1.8.2
v1.7.2
v1.6.2.1
v1.5.1
v1.4.1
v1.3.2
v1.2.0
v1.1.2
v1.0.1
Downloads
pdf
html
epub
On Read the Docs
Project Home
Builds

Free document hosting provided by Read the Docs.