FromageStep¶
The Fromage (Frobenius Matched Gradient Descent) optimizer step.
Based on Bernstein et al, 2020, this optimizer requires no or little learning rate tuning or scheduling, and can work across a range of neural network topologies, including transformers and GANs, with the same setting. A minimal degree of adaptation, such as dividing by 10 when the loss plateaus, can be helpful, however. Like all step nodes, this node only processes gradients, and the resulting updates must be applied manually to the weights (this can be accomplished using the Add node). However, you can also pass it to the StepSolver node which implements the full optimization loop. More Info... Version 0.2.0
Ports/Properties¶
gradients¶
Gradients to be transformed.
- verbose name: Gradients
- default value: None
- port type: DataPort
- value type: object (can be None)
- data direction: INOUT
weights¶
Optional current weights.
- verbose name: Weights
- default value: None
- port type: DataPort
- value type: object (can be None)
- data direction: IN
state¶
Explicit state of the node.
- verbose name: State
- default value: None
- port type: DataPort
- value type: object (can be None)
- data direction: INOUT
learning_rate¶
Learning rate.
- verbose name: Learning Rate
- default value: 0.01
- port type: FloatPort
- value type: float (can be None)
min_norm¶
Minimum gradient norm. This can be used to avoid dividing by zero when rescaling; small gradients are rescaled to at least this value.
- verbose name: Min Norm
- default value: 1e-06
- port type: FloatPort
- value type: float (can be None)
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
- verbose name: Set Breakpoint (Debug Only)
- default value: False
- port type: BoolPort
- value type: bool (can be None)
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.
- verbose name: Metadata
- default value: {}
- port type: DictPort
- value type: dict (can be None)