WeightDecayStep¶
Chainable step that applies weight decay (analogous to l2 regularization) to the parameters.
This is typically applied after gradients have been scaled by criteria such as past gradient updates, but is applied before scaling by the learning rate, so that the learning rate does not change the ratio of the weight decay to the gradient update. The weight decay can be used in conjunction with a mask data structure that has the same nested structure as the weights being optimized, but which contains booleans indicating which weights should be decayed. Version 0.2.0
Ports/Properties¶
gradients¶
Gradients to be transformed.
- verbose name: Gradients
- default value: None
- port type: DataPort
- value type: object (can be None)
- data direction: INOUT
weights¶
Optional current weights.
- verbose name: Weights
- default value: None
- port type: DataPort
- value type: object (can be None)
- data direction: IN
state¶
Explicit state of the node.
- verbose name: State
- default value: None
- port type: DataPort
- value type: object (can be None)
- data direction: INOUT
weight_decay_mask¶
Mask structure for the weight decay.
- verbose name: Weight Decay Mask
- default value: None
- port type: DataPort
- value type: object (can be None)
- data direction: IN
decay_rate¶
Weight decay rate. This is typically a small value, such as 1e-4.
- verbose name: Decay Rate
- default value: 0.0001
- port type: FloatPort
- value type: float (can be None)
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
- verbose name: Set Breakpoint (Debug Only)
- default value: False
- port type: BoolPort
- value type: bool (can be None)
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.
- verbose name: Metadata
- default value: {}
- port type: DictPort
- value type: dict (can be None)