Skip to content

← deep_learning package

MomentumStep

Chainable step that adds momentum based on one of several formulations, including classic momentum, Nesterov acceleration, and exponential moving average.

This can improve the convergence behavior of the optimizer, including the convergence rate (esp. when nesterov acceleration is used) and the ability to escape saddle points. Version 0.2.0

Ports/Properties

gradients

Gradients to be transformed.

  • verbose name: Gradients
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT

weights

Optional current weights.

  • verbose name: Weights
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN

state

Explicit state of the node.

  • verbose name: State
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT

decay

Exponential decay rate for past values.

  • verbose name: Decay
  • default value: 0.9
  • port type: FloatPort
  • value type: float (can be None)

type

Type of momentum to use. Momentum is classic momentum, Nesterov is Nesterov acceleration, and EMA is exponential moving average.

  • verbose name: Type
  • default value: momentum
  • port type: EnumPort
  • value type: str (can be None)

mu_precision

Numeric precision for the first-order accumulator. Keep resolves to the precision of the inputs.

  • verbose name: Mu Precision
  • default value: keep
  • port type: EnumPort
  • value type: str (can be None)

set_breakpoint

Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)

metadata

User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)