LinearOneCycleSchedule¶
A one-cycle linear ramp up/down parameter schedule.
This schedule is a linear ramp up to a peak value, followed by a linear ramp down to the initial value, and finally a linear ramp down to the final value that is held until the end of the schedule (transition_steps). The parameter is held at the final value after that. Only the peak value is specified directly, while the initial value (aka base value) is given as ratio to the peak value, as is the final value. Also, the upslope and downslope durations are given as fractions of the total transition_steps. The fraction of the remaining final slope is the remainder of the transition_steps after the upslope and downslope durations have been subtracted. This schedule is inspired by Smith and Topin's 2018 paper, "Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates" (see URL). Schedule nodes in NeuroPype are used for fine-grained control over how parameters, like the learning rate, should change over time during optimization. Most Step nodes offer a learning_rate_schedule port, into which a Schedule node can be wired to override the otherwise default constant learning rate. However, any other optimizer step parameter can be controlled by a schedule, simply by wiring the schedule node's output into the respective parameter of the Step nodes, and passing the schedule the current iteration (step) count of the optimization process. More Info... Version 0.2.0
Ports/Properties¶
step¶
Current step (iteration) count.
- verbose name: Step
- default value: None
- port type: DataPort
- value type: object (can be None)
- data direction: IN
value¶
Schedule value at current step count.
- verbose name: Value
- default value: None
- port type: DataPort
- value type: object (can be None)
- data direction: OUT
peak_value¶
Parameter value at peak. This is the maximum value that the parameter will attain over the course of the schedule.
- verbose name: Peak Value
- default value: 1.0
- port type: FloatPort
- value type: float (can be None)
transition_steps¶
Step count at which to end the transition from the initial value to the final value. The parameter is held at the final value after this step count is reached. This is the duration of the full scaling cycle.
- verbose name: Transition Steps
- default value: 100
- port type: IntPort
- value type: int (can be None)
peak_base_ratio¶
Ratio of the peak to the base value. This is the ratio of the peak value to the initial value, and the value that will be attained after the downslope. Note that there's a final slope that reduces the parameter to the final minimum that is usually considerably lower than the base.
- verbose name: Peak Base Ratio
- default value: 25
- port type: FloatPort
- value type: float (can be None)
peak_final_ratio¶
Ratio of the peak to the final value at the end of the cycle. The parameter will be held at the final value after transition_steps have been reached.
- verbose name: Peak Final Ratio
- default value: 10000.0
- port type: FloatPort
- value type: float (can be None)
upslope_fraction¶
Fraction of the transition_steps that will be used for the upslope. That is, the peak is reached after transition_steps * upslope_fraction steps.
- verbose name: Upslope Step Fraction
- default value: 0.3
- port type: FloatPort
- value type: float (can be None)
downslope_fraction¶
Fraction of the transition_steps that will be used for the downslope back to the initial value. After this, there is a final slope going to the final value that continues over transition_steps * (1 - upslope_fraction - downslope_fraction) steps. Note this is parameterized slightly less confusingly than the underlying optax linear_onecycle_schedule function.
- verbose name: Downslope Step Fraction
- default value: 0.55
- port type: FloatPort
- value type: float (can be None)
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
- verbose name: Set Breakpoint (Debug Only)
- default value: False
- port type: BoolPort
- value type: bool (can be None)
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.
- verbose name: Metadata
- default value: {}
- port type: DictPort
- value type: dict (can be None)
step_multiplier¶
Multiplier for the step count. This value is multiplied with each of the step counts to uniformly speed up or slow down the schedule through a single parameter. When used to define an optimizer used by the DeepModel node, this can also be set to 0.0, in which case the multiplier is chosen such that the schedule reaches its final value at the end of the training process, but note that this is not always possible, namely for schedules that are never reach a final value. Otherwise, to make a schedule dependent on the number of steps done by a node, you may normalize your schedule to eg 1000 steps and then wire a formula that calculates the steps done by some process divided by 1000 into this node.
- verbose name: Step Multiplier
- default value: 1.0
- port type: FloatPort
- value type: float (can be None)