WarmupCosineDecaySchedule¶
A linear warmup followed by cosine decay schedule.
Similarly to the "Linear Warmup Exponential Decay" Schedule, this schedule begins with a linear ramp from the initial value to the peak value over the course of warmup_steps steps, and then follows a cosine function (from initial peak to first trough) down to the final value over the course of decay_steps steps. This is one of the most robust learning rate schedules and represents the state of the art along with the linear warmup then exponential decay schedule. However, as for all schedules note that none of the defaults should be used without either making good educated guesses, experimentation, or consuling literature that you are aiming to replicate. Schedule nodes in NeuroPype are used for fine-grained control over how parameters, like the learning rate, should change over time during optimization. Most Step nodes offer a learning_rate_schedule port, into which a Schedule node can be wired to override the otherwise default constant learning rate. However, any other optimizer step parameter can be controlled by a schedule, simply by wiring the schedule node's output into the respective parameter of the Step nodes, and passing the schedule the current iteration (step) count of the optimization process. Version 0.2.0
Ports/Properties¶
step¶
Current step (iteration) count.
- verbose name: Step
- default value: None
- port type: DataPort
- value type: object (can be None)
- data direction: IN
value¶
Schedule value at current step count.
- verbose name: Value
- default value: None
- port type: DataPort
- value type: object (can be None)
- data direction: OUT
init_value¶
Initial parameter value. This is the value at the beginning of the schedule.
- verbose name: Initial Value
- default value: 0.0
- port type: FloatPort
- value type: float
peak_value¶
Peak parameter value. This is the value at the peak following the initial warmup, before it is lowered again following a cosine function.
- verbose name: Peak Value
- default value: 1.0
- port type: FloatPort
- value type: float (can be None)
final_value¶
Final parameter value. Once the schedule reaches this value, it will remain at this value for the remainder of the optimization process.
- verbose name: Final Value
- default value: 0.0
- port type: FloatPort
- value type: float
warmup_steps¶
Number of steps over which to ramp up from the initial value to the peak value. After this, the parameter is lowered again following the shape of a cosine function down to the desired final value value.
- verbose name: Warmup Steps
- default value: 100
- port type: IntPort
- value type: int (can be None)
decay_steps¶
The number of steps over which the cosine decay takes place. This is a soft transition following a raised- cosine function from the peak value down to the final value.
- verbose name: Decay Steps
- default value: 100
- port type: IntPort
- value type: int (can be None)
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
- verbose name: Set Breakpoint (Debug Only)
- default value: False
- port type: BoolPort
- value type: bool (can be None)
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.
- verbose name: Metadata
- default value: {}
- port type: DictPort
- value type: dict (can be None)
step_multiplier¶
Multiplier for the step count. This value is multiplied with each of the step counts to uniformly speed up or slow down the schedule through a single parameter. When used to define an optimizer used by the DeepModel node, this can also be set to 0.0, in which case the multiplier is chosen such that the schedule reaches its final value at the end of the training process, but note that this is not always possible, namely for schedules that are never reach a final value. Otherwise, to make a schedule dependent on the number of steps done by a node, you may normalize your schedule to eg 1000 steps and then wire a formula that calculates the steps done by some process divided by 1000 into this node.
- verbose name: Step Multiplier
- default value: 1.0
- port type: FloatPort
- value type: float (can be None)