Skip to content

← deep_learning package

GroupNorm

Apply group normalization to the given data.

This splits the feature axis into groups, and z-scores the data in each group separately, computing statistics across typically the feature axis wihin the group and any spatial axes. This can be a useful alternative to other norms such as batch norm, layer norm, or instance norm, when there are enough features to split into groups and ideally when those features have some groupwise structure (for example outputs of a grouped convolution). The normalization is applied per instance and no cross-instance statistics are taken. Like most normalizations, group normalization typically includes a learned scale and bias parameter, whose shape (and thus dimensionality) can be configured; these can also be optionally overridden with externally generated values. If packet data is given, this node ensures that the instance axes come first and the feature axes come last and are flattened into a single feature axis. More Info... Version 0.2.0

Ports/Properties

data

Data to process.

  • verbose name: Data
  • default value: None
  • port type: DataPort
  • value type: AnyNumeric (can be None)
  • data direction: INOUT

scale_init

Initializer for the trainable scale.

  • verbose name: Scale Init
  • default value: None
  • port type: DataPort
  • value type: BaseNode (can be None)
  • data direction: IN

bias_init

Initializer for the trainable bias.

  • verbose name: Bias Init
  • default value: None
  • port type: DataPort
  • value type: BaseNode (can be None)
  • data direction: IN

scale_prior

Optional prior distribution for the scale.

  • verbose name: Scale Prior
  • default value: None
  • port type: DataPort
  • value type: Distribution (can be None)
  • data direction: IN

bias_prior

Optional prior distribution for the bias.

  • verbose name: Bias Prior
  • default value: None
  • port type: DataPort
  • value type: Distribution (can be None)
  • data direction: IN

num_groups

Number of feature groups to use for the normalization. If this is set to 1, this becomes nearly equivalent to layer normalization. If this is set to the number of features, this becomes equivalent to instance normalization.

  • verbose name: Number Of Feature Groups
  • default value: 32
  • port type: IntPort
  • value type: int (can be None)

epsilon

Small value to add to the variance to avoid division by zero.

  • verbose name: Epsilon
  • default value: 1e-05
  • port type: FloatPort
  • value type: float (can be None)

learnable_scale

Whether to learn a trainable scale parameter. Normalizations typically include such a parameter in order to drive the subsequent activation function in a regime that is desirable for downstream computations (e.g., saturating or linear). Note the shape (and thus dimensionality) of the learned parameter is governed by the param_axes (learn scale/bias across axes) parameter.

  • verbose name: Learnable Scale
  • default value: True
  • port type: BoolPort
  • value type: bool (can be None)

learnable_bias

Whether to learn a trainable bias parameter. See the learnable scale for more details.

  • verbose name: Learnable Bias
  • default value: True
  • port type: BoolPort
  • value type: bool (can be None)

data_format

Format of the input data. This is only respected when working with plain arrays and is ignored for packet data, which always normalizes the data to 'channels_last' layout. If 'channels_last', the data is assumed to be in the format ({batch}, ..., channels). If 'channels_first', the data is assumed to be in the format ({batch}, channels, ...).

  • verbose name: Array Data Format
  • default value: auto
  • port type: EnumPort
  • value type: str (can be None)

scale_initializer

Choice of scale initializer. This can either be one of the provided initializers, or the value "custom", in which case one of the Initializer nodes must be wired into the respective input port. For beginners it is recommended to stick to the defaults, since initialization of deep net layers is nuanced and can be tricky, otherwise be prepared to experiment with different choices. In general, the variance-scaling (lecun, glorot/xavier, he/kaiming) initializers are recommended, except for very simple/small layers where you may have a good default assumption as to the distribution of the weights (e.g., truncated_normal or uniform). Bias layers are typically zero-initialized. For initializers that take arguments, you can also type out the arguments positionally as in "truncated_normal(1.0,0.0)" (note reversed order of stddev, mean). The following initializers have arguments (here listed with their defaults): constant(value), those ending in normal(stddev=1, mean=0), those ending in uniform(min=0,max=1), orthogonal(scale=1,axis=-1), identity(gain=1), and variance_scaling(scale=1, "fan_in" (default)/"fan_avg"/"fan_out", "truncated_normal"(default)/"normal"/"uniform",optional-axis-indices=auto). Note that glorot and xavier are aliases for each other, and likewise he and kaiming are aliases for each other.

  • verbose name: Scale Initializer
  • default value: ones
  • port type: ComboPort
  • value type: str (can be None)

bias_initializer

Choice of bias initializer. This can either be one of the provided initializers, or the value "custom", in which case one of the Initializer nodes must be wired into the respective input port. For beginners it is recommended to stick to the defaults, since initialization of deep net layers is nuanced and can be tricky, otherwise be prepared to experiment with different choices. In general, the variance-scaling (lecun, glorot/xavier, he/kaiming) initializers are recommended, except for very simple/small layers where you may have a good default assumption as to the distribution of the weights (e.g., truncated_normal or uniform). Bias layers are typically zero-initialized. For initializers that take arguments, you can also type out the arguments positionally as in "truncated_normal(1.0,0.0)" (note reversed order of stddev, mean). The following initializers have arguments (here listed with their defaults): constant(value), those ending in normal(stddev=1, mean=0), those ending in uniform(min=0,max=1), orthogonal(scale=1,axis=-1), identity(gain=1), and variance_scaling(scale=1, "fan_in" (default)/"fan_avg"/"fan_out", "truncated_normal"(default)/"normal"/"uniform",optional-axis-indices=auto). Note that glorot and xavier are aliases for each other, and likewise he and kaiming are aliases for each other.

  • verbose name: Bias Initializer
  • default value: zeros
  • port type: ComboPort
  • value type: str (can be None)

layername

Name of the layer. Used for naming of the trainable parameters.

  • verbose name: Layer Name
  • default value: groupnorm
  • port type: StringPort
  • value type: str (can be None)

set_breakpoint

Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)

metadata

User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)