Skip to content

← deep_learning package

VarianceScalingInitializer

Initialize weights from a distribution whose scale is adapted to the shape of the initialized array.

This can be configured to obtain a variety of standard initializers, including Glorot (scale=1, mode=fan_avg), LeCun (scale=1, mode=fan_in), and He (scale=2, mode=fan_in), in combination with distribution=uniform or truncated_normal. Version 0.5.0

Ports/Properties

scale

Scale to multiply the variance by. This is most commonly 1 (e.g., Glorot and LeCun initialization), but can also be 2 (e.g., for He initialization).

  • verbose name: Scale Multiplier
  • default value: 1
  • port type: FloatPort
  • value type: float (can be None)

mode

Scale the variance based on one of: the number of input units (fan_in, used in LeCun or He initialization), number of output units (fan_out), or the average of the number of input and output units (fan_avg, used in Glorot initialization).

  • verbose name: Scale By
  • default value: fan_in
  • port type: EnumPort
  • value type: str (can be None)

distribution

Distribution from which to draw the initial weights. Typically, this is either truncated normal or uniform. The parameters of the distribution are computed as follows. First, the scale s is computed as: s = sqrt(scale / n) where n is the number of input units for fan_in, the number of output units for fan_out, or the average of the number of input and output units for fan_avg. Then the mean is 0 and the standard deviation is sqrt(s) for normal, and sqrt(adjs) for truncated normal, where adj is an adjustment factor to compensate for the truncation. For uniform, the bounds are [-sqrt(3s), sqrt(3*s)].

  • verbose name: Distribution Type
  • default value: truncated_normal
  • port type: EnumPort
  • value type: str (can be None)

fan_in_axes

Axes to use for computing the number of input units. If None, all but the last dimension are used (default for e.g., convolutional kernels). Otherwise this may need to be set to a list of axis indices (counting from 0). The number of output units is always the remaining axes.

  • verbose name: Fan-In Axis Indices
  • default value: None
  • port type: ListPort
  • value type: list (can be None)

set_breakpoint

Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)

metadata

User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)