Skip to content

← deep_learning package

SGDStep

The Stochastic Gradient Descent (SGD) optimizer step.

Popularized in its modern incarnation by Sutskever et al. 2013, SGD is a simple yet powerful optimizer that that both can serve as a baseline and sometimes outperforms more complex optimizers, e.g., on reasonably benign network topologies. This implementation includes optional support for momentum and Nesterov acceleration, which are standard practice when optimizing DNNs. Like all step nodes, this node only processes gradients, and the resulting updates must be applied manually to the weights (this can be accomplished using the Add node). However, you can also pass it to the StepSolver node which implements the full optimization loop. The learning rate can instead be given as a schedule, by wiring one of the Schedule nodes into the learning_rate_schedule port. More Info... Version 0.2.0

Ports/Properties

gradients

Gradients to be transformed.

  • verbose name: Gradients
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT

weights

Optional current weights.

  • verbose name: Weights
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN

state

Explicit state of the node.

  • verbose name: State
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT

learning_rate_schedule

Optional learning rate schedule.

  • verbose name: Learning Rate Schedule
  • default value: None
  • port type: DataPort
  • value type: BaseNode (can be None)
  • data direction: IN

learning_rate

Learning rate. A typical choice may be 0.001 here, but this is problem dependent. If a learning rate schedule is provided, this value should be left unspecified.

  • verbose name: Learning Rate
  • default value: None
  • port type: FloatPort
  • value type: float (can be None)

momentum

Optional exponential decay rate for momentum.

  • verbose name: Optional Momentum
  • default value: None
  • port type: FloatPort
  • value type: float (can be None)

nesterov

Whether to use Nesterov acceleration.

  • verbose name: Use Nesterov Acceleration
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)

mu_precision

Numeric precision for the first-order accumulator. Keep resolves to the precision of the inputs.

  • verbose name: Mu Precision
  • default value: keep
  • port type: EnumPort
  • value type: str (can be None)

set_breakpoint

Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)

metadata

User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)