chainer.UpdateRule

class chainer.UpdateRule(parent_hyperparam=None)

Base class of all update rules.

Update rule is an object that implements how to update one parameter variable using the gradient of a loss function. This class provides the interface and the common features of any update rules.

An update rule can be set to a Variable object that represents a parameter array of a model. An Optimizer instance defines which parameters to update, and the update rule instance of each parameter defines how to update it.

Hook functions can be set to any update rule instance. The hook function is called just before or after any updates (configurable) in the order of registrations.

An implementation of update rule should override update_core() or its device-dependent variants (i.e., update_core_cpu() and update_core_gpu()).

The state (e.g. a moving average of the gradient) of the update rule is stored into the state dictionary. An implementation of update rule using state should also override init_state() to initialize the state at the first update. The values of the state dictionary are automatically copied to the appropriate device before the update based on the data and grad arrays.

Parameters:

parent_hyperparam (Hyperparameter) – Hyperparameter that provides the default values.

Variables:

• enabled (bool) – Flag to configure if this update rule is active. If the update rule is not active (i.e., enabled = False), the update() method does not update the parameter.
• hyperparam (Hyperparameter) – Hyperparameter of the update rule.
• t (int) – Number of updates made by this update rule.

Methods

add_hook(hook, name=None, timing='auto')

Adds a hook function.

The hook function is called before or after any updates (see the timing attribute).

Parameters:

• hook (callable) – Hook function to be added. It takes two arguments: the update rule object and the parameter variable.
• name (str) – Name of the hook function. The name attribute of the hook function is used by default.
• timing (str) – Specifies when the hook is called. If ‘auto’, the timimg property of the hook will decide the timing. If ‘pre’, the hook will be called before any updates. If ‘post’, the hook will be called after any updates. If ‘auto’ and the timing property of the hook is not available, timing will default to ‘pre’.

init_state(param)

Initializes the state.

Any implementations that use the state should override this mehtod. This method is called at the first update.

Parameters:param (Variable) – Parameter variable. It can be used to extract the shape and the data type of the parameter.

remove_hook(name)

Removes the specified hook function.

Parameters:name (str) – Name of the hook function to be removed. The hook function registered with this name will be removed.

serialize(serializer)

Serializes the update rule state.

Be careful that this method only saves/loads the state of the update rule. The parameters of the target link is not saved/loaded by this method, and so you need to serialize the target link separately if you want to fully recover the training state including parameters.

Parameters:serializer (AbstractSerializer) – Serializer object.

update(param)

Invokes hook functions and updates the parameter.

Parameters:param (Variable) – Variable to be updated.

update_core(param)

Updates the parameter.

Implementation of UpdateRule should override this method or both of update_core_cpu() and update_core_gpu().

Parameters:param (Variable) – Variable to be updated.

update_core_cpu(param)

Updates the parameter on CPU.

See update_core() for details.

Parameters:param (Variable) – Variable to be updated.

update_core_gpu(param)

Updates the parameter on GPU.

See update_core() for details.

Parameters:param (Variable) – Variable to be updated.

use_fp32_update(flag=True)

Enables use of parameter update in fp32.

This method enables use of parameter update in fp32. When it is enabled and data type of original parameter variable is fp16, fp32 copy of parameter variable is automatically created and retained at self.fp32_param. And the parameter is update in fp32 in the following way.

1. copys the grad of original parameter variable to the grad of fp32 parameter variable, converting its data type from fp16 to fp32.
2. updates the parameter in fp32.
3. copys the data of fp32 parameter variable to the data of original parameter variable, converting its data type from fp32 to fp16.

See meth:update for details.

Attributes

state

State dictionary.