Skip to content

← deep_learning package

GradientClippingStep

Chainable step that clips incoming gradients based on their norm, ensuring that the gradient norm does not exceed a provided threshold.

This implements several variants, including elementwise clipping, layerwise clipping, global clipping (as in Pascanu et al, 2012), and adaptive clipping (relative to the prior parameter norm). The adaptive clipping follows Brock, Smith, De, and Simonyan (2021), "High-Performance Large-Scale Image Recognition Without Normalization.". Unlike the end-to-end steps (named after specific published algorithms), this is a chainable step (to be used with the ChainedStep node) that takes in a gradient and outputs a modified gradient, and it would usually be combined with other steps (like scaling by the learning rate) to yield a full optimizer step. More Info... Version 0.2.0

Ports/Properties

gradients

Gradients to be transformed.

  • verbose name: Gradients
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT

weights

Optional current weights.

  • verbose name: Weights
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: IN

state

Explicit state of the node.

  • verbose name: State
  • default value: None
  • port type: DataPort
  • value type: object (can be None)
  • data direction: INOUT

variant

Variant of gradient clipping to apply. Per-weight means that each individual weight update is limited to +/- threshold, per-layer means that the gradient for each parameter vector/matrix is scaled such that its norm is at most threshold, and global means that the gradient for the entire parameter set is scaled such that its norm is at most threshold. Per-weight-relative means that each individual weight update is limited to +/- threshold times the norm of the corresponding parameter before the update (requires prior parameter values to be provided to the StepUpdate node).

  • verbose name: Variant
  • default value: per-parameter
  • port type: EnumPort
  • value type: str (can be None)

threshold

Threshold value. A typical value is 1.0, but depending on the network and data, other values, may be explored.

  • verbose name: Threshold
  • default value: 1.0
  • port type: FloatPort
  • value type: float (can be None)

epsilon

Small value to prevent clipping of zero-initialized parameters. Only used for per-weight-relative.

  • verbose name: Epsilon (If Per-Weight-Relative)
  • default value: 0.001
  • port type: FloatPort
  • value type: float (can be None)

set_breakpoint

Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)

metadata

User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)