GatedRecurrentUnitLayer¶
A gated recurrent unit (GRU) recurrent core, based on Chung et al.
(2014). This is a recurrent layer that conceptually retains internal network activations across successive time steps when the node is used to process time series data (see below for more details). In contrast to LSTMs, GRUs have a simpler structure and their output size (length of emitted feature axis) equals the number of hidden units. The node is highly competitive in practice in its ability to learn long-term dependencies using a gating mechanism that controls Like all recurrent layers, the node does not usually store and carry over these activations, but depends on either a special loop node (the RecurrentLoop node, see docs for more details) to step aross the time axis of some given data array, or requires that the user manually passes in and retrieves that carry state via the carry in/out port. Also like all recurrent nodes, this node will move the instance axis of the input first, optionally retain any axes listed in the parallel_axes port, and and flatten any other axes into a single feature axis at the end. When managing the carry state manually, the state for this node can be obtained via get_initial() or be constructed as either an array or packet (depending on the data that it is used with), where the first axis is of size 2, the last axis is of size units, and the middle axis is sized to be the product of the instance axes and other parallel axes of the data. Also as with most built-in layers, you can override the initializer for the input and hidden weights and/or the bias, which default to Lecun Normal and zero, respectively. More Info... Version 0.2.0
Ports/Properties¶
data¶
Data to process at current time step.
- verbose name: Data
- default value: None
- port type: DataPort
- value type: AnyNumeric (can be None)
- data direction: INOUT
carry¶
Carried-over activations from previous time step.
- verbose name: Carry
- default value: None
- port type: DataPort
- value type: AnyNumeric (can be None)
- data direction: INOUT
wi_init¶
Initializer for the input weights.
- verbose name: Wi Init
- default value: None
- port type: DataPort
- value type: BaseNode (can be None)
- data direction: IN
wh_init¶
Initializer for the hidden weights.
- verbose name: Wh Init
- default value: None
- port type: DataPort
- value type: BaseNode (can be None)
- data direction: IN
b_init¶
Initializer for the bias.
- verbose name: B Init
- default value: None
- port type: DataPort
- value type: BaseNode (can be None)
- data direction: IN
wi_prior¶
Optional prior distribution for the input weights.
- verbose name: Wi Prior
- default value: None
- port type: DataPort
- value type: Distribution (can be None)
- data direction: IN
wh_prior¶
Optional prior distribution for the hidden weights.
- verbose name: Wh Prior
- default value: None
- port type: DataPort
- value type: Distribution (can be None)
- data direction: IN
b_prior¶
Optional prior distribution for the bias.
- verbose name: B Prior
- default value: None
- port type: DataPort
- value type: Distribution (can be None)
- data direction: IN
units¶
Number of hidden units.
- verbose name: Hidden Units
- default value: 256
- port type: IntPort
- value type: int (can be None)
wi_initializer¶
Choice of input weight initializer. This can either be one of the provided initializers, or the value "custom", in which case one of the Initializer nodes must be wired into the respective input port. For beginners it is recommended to stick to the defaults, since initialization of deep net layers is nuanced and can be tricky, otherwise be prepared to experiment with different choices. In general, the variance-scaling (lecun, glorot/xavier, he/kaiming) initializers are recommended, except for very simple/small layers where you may have a good default assumption as to the distribution of the weights (e.g., truncated_normal or uniform). Bias layers are typically zero-initialized. For initializers that take arguments, you can also type out the arguments positionally as in "truncated_normal(1.0,0.0)" (note reversed order of stddev, mean). The following initializers have arguments (here listed with their defaults): constant(value), those ending in normal(stddev=1, mean=0), those ending in uniform(min=0,max=1), orthogonal(scale=1,axis=-1), identity(gain=1), and variance_scaling(scale=1, "fan_in" (default)/"fan_avg"/"fan_out", "truncated_normal"(default)/"normal"/"uniform",optional-axis-indices=auto). Note that glorot and xavier are aliases for each other, and likewise he and kaiming are aliases for each other.
- verbose name: Input Weight Initializer
- default value: lecun_normal
- port type: ComboPort
- value type: str (can be None)
wh_initializer¶
Choice of hidden weight initializer. This can either be one of the provided initializers, or the value "custom", in which case one of the Initializer nodes must be wired into the respective input port. For beginners it is recommended to stick to the defaults, since initialization of deep net layers is nuanced and can be tricky, otherwise be prepared to experiment with different choices. In general, the variance-scaling (lecun, glorot/xavier, he/kaiming) initializers are recommended, except for very simple/small layers where you may have a good default assumption as to the distribution of the weights (e.g., truncated_normal or uniform). Bias layers are typically zero-initialized. For initializers that take arguments, you can also type out the arguments positionally as in "truncated_normal(1.0,0.0)" (note reversed order of stddev, mean). The following initializers have arguments (here listed with their defaults): constant(value), those ending in normal(stddev=1, mean=0), those ending in uniform(min=0,max=1), orthogonal(scale=1,axis=-1), identity(gain=1), and variance_scaling(scale=1, "fan_in" (default)/"fan_avg"/"fan_out", "truncated_normal"(default)/"normal"/"uniform",optional-axis-indices=auto). Note that glorot and xavier are aliases for each other, and likewise he and kaiming are aliases for each other.
- verbose name: Hidden Weight Initializer
- default value: lecun_normal
- port type: ComboPort
- value type: str (can be None)
b_initializer¶
Choice of bias initializer. This can either be one of the provided initializers, or the value "custom", in which case one of the Initializer nodes must be wired into the respective input port. For beginners it is recommended to stick to the defaults, since initialization of deep net layers is nuanced and can be tricky, otherwise be prepared to experiment with different choices. In general, the variance-scaling (lecun, glorot/xavier, he/kaiming) initializers are recommended, except for very simple/small layers where you may have a good default assumption as to the distribution of the weights (e.g., truncated_normal or uniform). Bias layers are typically zero-initialized. For initializers that take arguments, you can also type out the arguments positionally as in "truncated_normal(1.0,0.0)" (note reversed order of stddev, mean). The following initializers have arguments (here listed with their defaults): constant(value), those ending in normal(stddev=1, mean=0), those ending in uniform(min=0,max=1), orthogonal(scale=1,axis=-1), identity(gain=1), and variance_scaling(scale=1, "fan_in" (default)/"fan_avg"/"fan_out", "truncated_normal"(default)/"normal"/"uniform",optional-axis-indices=auto). Note that glorot and xavier are aliases for each other, and likewise he and kaiming are aliases for each other.
- verbose name: Bias Initializer
- default value: zeros
- port type: ComboPort
- value type: str (can be None)
layername¶
Name of the layer. Used for naming of weights.
- verbose name: Layer Name
- default value: gru
- port type: StringPort
- value type: str (can be None)
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
- verbose name: Set Breakpoint (Debug Only)
- default value: False
- port type: BoolPort
- value type: bool (can be None)
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.
- verbose name: Metadata
- default value: {}
- port type: DictPort
- value type: dict (can be None)
parallel_axes¶
Optionally an axis or comma-separated list of axes that shall be processed in parallel by the layer, using the same learned weights. This is useful if you have multiple parallel processes that you wish to process separately but you assume that they are all ultimately governed by the same rules, you can learn a single kernel for all of them. Note that, for packet data, the instance axis (if present) and for plain-array data the first axis is always treated as a parallel axis, so you don't need to list it here.
- verbose name: Parallel Axes
- default value:
- port type: ComboPort
- value type: str (can be None)