AdaFactorStep¶
The AdaFactor optimizer step.
Based on Shazeer and Stern, 2018, this is an adaptive optimizer that is designed for fast training of large-scale networks (might be overkill the small networks usually used with biosignals). The approach is to saves memory by using a factored representation of the second moment gradient estimates, and only applies to matrix/tensor-shaped parameters that meet a minimum axis size (see the min_size_to_factor setting). Like all step nodes, this node only processes gradients, and the resulting updates must be applied manually to the weights (this can be accomplished using the Add node). However, you can also pass it to the StepSolver node which implements the full optimization loop. The learning rate can instead be given as a schedule, by wiring one of the Schedule nodes into the learning_rate_schedule port. The weight decay can be used in conjunction with a mask data structure that has the same nested structure as the weights being optimized, but which contains booleans indicating which weights should be decayed. More Info... Version 0.2.0
Ports/Properties¶
gradients¶
Gradients to be transformed.
- verbose name: Gradients
- default value: None
- port type: DataPort
- value type: object (can be None)
- data direction: INOUT
weights¶
Optional current weights.
- verbose name: Weights
- default value: None
- port type: DataPort
- value type: object (can be None)
- data direction: IN
state¶
Explicit state of the node.
- verbose name: State
- default value: None
- port type: DataPort
- value type: object (can be None)
- data direction: INOUT
learning_rate_schedule¶
Optional learning rate schedule.
- verbose name: Learning Rate Schedule
- default value: None
- port type: DataPort
- value type: BaseNode (can be None)
- data direction: IN
weight_decay_mask¶
Mask structure for the weight decay (optional).
- verbose name: Weight Decay Mask
- default value: None
- port type: DataPort
- value type: object (can be None)
- data direction: IN
learning_rate¶
Learning rate. Must be provided. If a learning rate schedule is provided, this value should be left unspecified.
- verbose name: Learning Rate
- default value: None
- port type: FloatPort
- value type: float (can be None)
min_size_to_factor¶
Only factor statistics if two dimensions of some weight are larger than this value.
- verbose name: Min Size To Factor
- default value: 128
- port type: IntPort
- value type: int (can be None)
decay_rate¶
Controls second-moment exponential decay schedule.
- verbose name: Decay Rate
- default value: 0.8
- port type: FloatPort
- value type: float (can be None)
decay_offset¶
Starting step when the fine-tuning phase begins.
- verbose name: Decay Offset
- default value: 0
- port type: IntPort
- value type: int (can be None)
multiply_by_parameter_scale¶
Scale learning rate by parameter norm. If False, provided learning rate is absolute step size.
- verbose name: Multiply By Parameter Scale
- default value: True
- port type: BoolPort
- value type: bool (can be None)
clipping_threshold¶
Optional gradient clipping threshold (norm). If set to None, this is disabled. This is per parameter vector/matrix.
- verbose name: Optional Clipping Threshold
- default value: 1.0
- port type: FloatPort
- value type: float (can be None)
momentum¶
Optional exponential decay rate for momentum.
- verbose name: Optional Momentum
- default value: None
- port type: FloatPort
- value type: float (can be None)
momentum_precision¶
Numeric precision for the momentum buffer. Keep resolves to the precision of the inputs.
- verbose name: Momentum Precision
- default value: keep
- port type: EnumPort
- value type: str (can be None)
weight_decay¶
Optional rate at which to decay weights. This is usually a small number, like e.g., 1e-4.
- verbose name: Optional Weight Decay
- default value: None
- port type: FloatPort
- value type: float (can be None)
epsilon¶
Regularization constant for RMS gradient.
- verbose name: Epsilon
- default value: 1e-30
- port type: FloatPort
- value type: float (can be None)
factored¶
Whether to use factored second-moment estimates. This can be turned off to disable the factorization (e.g., to mimick a simpler optimizer).
- verbose name: Factored
- default value: True
- port type: BoolPort
- value type: bool (can be None)
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
- verbose name: Set Breakpoint (Debug Only)
- default value: False
- port type: BoolPort
- value type: bool (can be None)
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.
- verbose name: Metadata
- default value: {}
- port type: DictPort
- value type: dict (can be None)