Skip to content

← deep_learning package

AggregateStep

A modifier for the step node that is wired into it which accumulates k successive gradient evaluations and passes them to the optimizer for use in one summed (or averaged) update.

This can be used for things like multi-task learning (if the k successive gradients stem from multiple tasks that are visited round-robin), or for varying the batch size over the course of training (since k can be controlled by a schedule), or for simulating large-batch training with batch sizes that otherwise would not fit in memory. Note this is not simply chained after another step using the ChainedStep node, but rather it is a modifier of the step node that is wired into it. Version 0.2.0

Ports/Properties

gradients

Gradients to be transformed.

  • verbose name: Gradients
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT

weights

Optional current weights.

  • verbose name: Weights
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN

state

Explicit state of the node.

  • verbose name: State
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT

modify_step

Step to modify.

  • verbose name: Modify Step
  • default value: None
  • port type: DataPort
  • value type: BaseNode (can be None)
  • data direction: IN

num_ministeps_schedule

Optional schedule for the k parameter.

  • verbose name: Num Ministeps Schedule
  • default value: None
  • port type: DataPort
  • value type: BaseNode (can be None)
  • data direction: IN

num_ministeps

Number of successive updates to accumulate. If not provided, a schedule must be wired in. This is the k parameter in the documentation.

  • verbose name: Num Ministeps
  • default value: None
  • port type: IntPort
  • value type: int (can be None)

reduction

Whether to sum or mean the updates.

  • verbose name: Reduction Operation
  • default value: sum
  • port type: EnumPort
  • value type: str (can be None)

skip_ministep_if

Optional criteria under which to skip mini-steps as if they did not happen. Note that using this on multi-task problems may cause the stepping to go out of sync between tasks.

  • verbose name: Skip Mini-Step If
  • default value: never
  • port type: ComboPort
  • value type: str (can be None)

set_breakpoint

Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)

metadata

User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)