WarmupExponentialDecaySchedule¶
A linear warmup followed by an exponential decay parameter schedule.
This schedule is a combination of a linear ramp (from initial value to peak value over the first warmup_steps), followed by an optional constant plateau (if decay_begin>-9), followed by an exponential decay (from peak value to final value at a rate governed by decay_rate and decay_steps, where decay_steps is the number of steps over which the value decays by a factor of decay_rate). This is one of the most robust schedules for training neural networks, and is the current state of the art, but note that none of the default values should be considered anywhere near optimal for a given setup -- these are just example settings that help show how the schedule is configured. The linear warmup ensures that the model does not diverge during early training where weights can be assumed to be random, the peak portion helps the model escape local minima (as in simulated annealing), and the exponential decay phase helps the model converge to a good solution with high precision. Schedule nodes in NeuroPype are used for fine-grained control over how parameters, like the learning rate, should change over time during optimization. Most Step nodes offer a learning_rate_schedule port, into which a Schedule node can be wired to override the otherwise default constant learning rate. However, any other optimizer step parameter can be controlled by a schedule, simply by wiring the schedule node's output into the respective parameter of the Step nodes, and passing the schedule the current iteration (step) count of the optimization process. Version 0.2.0
Ports/Properties¶
step¶
Current step (iteration) count.
- verbose name: Step
- default value: None
- port type: DataPort
- value type: object (can be None)
- data direction: IN
value¶
Schedule value at current step count.
- verbose name: Value
- default value: None
- port type: DataPort
- value type: object (can be None)
- data direction: OUT
init_value¶
Initial parameter value. This is the value at the beginning of the schedule. Note that the default may be application specific. The parameter is then linearly ramped up for warmup_steps and remains constant until transition_begin.
- verbose name: Initial Value
- default value: 0.0
- port type: FloatPort
- value type: float
peak_value¶
Peak parameter value. This is the value at the peak plateau of the warmup schedule, before it is annealed again exponentially.
- verbose name: Peak Value
- default value: 1.0
- port type: FloatPort
- value type: float (can be None)
warmup_steps¶
Number of steps over which to ramp up from the initial value to the peak value. After this, the parameter is held constant until transition_begin, after which it is then exponentially annealed until it reaches the final value.
- verbose name: Warmup Steps
- default value: 100
- port type: IntPort
- value type: int (can be None)
plateau_steps¶
Step count after which to begin the transition from the peak value to the final value. The parameter is held at the peak value for this many steps.
- verbose name: Plateau Steps
- default value: 0
- port type: IntPort
- value type: int (can be None)
decay_steps¶
The number of steps over which the parameter decays by decay_rate. Note that this is not the total duration of the decay portion; the decay will only finish one the value has reached the specified final value. The basic formula is value = initial_value * decay_rate ^ (count_since_decay_begin / decay_steps), followed by clipping according to the final_value.
- verbose name: Decay Steps
- default value: 10
- port type: IntPort
- value type: int (can be None)
decay_rate¶
Decay rate. The parameter value decays by this factor for every decay_steps. This can be between 0 and 1 for a regular decay schedule, or greater than 1 for an exponential growth schedule.
- verbose name: Decay (Or Growth) Rate
- default value: 0.9
- port type: FloatPort
- value type: float
final_value¶
Final parameter value. Once the schedule reaches this value, it will remain at this value for the remainder of the optimization process. (if the decay rate is < 1, this is effectively a lower bound on the parameter value, and if the decay rate is > 1, this is an upper bound)
- verbose name: Final Value
- default value: 0.0
- port type: FloatPort
- value type: float
staircase¶
If True, the parameter value is decayed in a staircase fashion, i.e ., the parameter changed by exactly decay_rate every decay_steps steps. If False, the parameter value is decayed in a continuous fashion according to the formula given in the docs for decay_steps.
- verbose name: Staircase
- default value: False
- port type: BoolPort
- value type: bool (can be None)
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
- verbose name: Set Breakpoint (Debug Only)
- default value: False
- port type: BoolPort
- value type: bool (can be None)
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.
- verbose name: Metadata
- default value: {}
- port type: DictPort
- value type: dict (can be None)
step_multiplier¶
Multiplier for the step count. This value is multiplied with each of the step counts to uniformly speed up or slow down the schedule through a single parameter. When used to define an optimizer used by the DeepModel node, this can also be set to 0.0, in which case the multiplier is chosen such that the schedule reaches its final value at the end of the training process, but note that this is not always possible, namely for schedules that are never reach a final value. Otherwise, to make a schedule dependent on the number of steps done by a node, you may normalize your schedule to eg 1000 steps and then wire a formula that calculates the steps done by some process divided by 1000 into this node.
- verbose name: Step Multiplier
- default value: 1.0
- port type: FloatPort
- value type: float (can be None)