EmbeddingLayer¶
A trainable layer for mapping categorical (integer) data to low-dimensional vectors.
This is used because neural networks are not inherently well adapted to mapping categorical data to continuous representations, so for small to mid-sized categorical data, one typically has to convert such data to a one-hot encoding, which can then be used with a Dense layer. However, in the case of many categories, this can be inefficient, and if categories inherently share separable features, this node provides a scalable solution that works for tens of thousands of categories (e.g., when the inputs are indices corresponding to a large tabularized vocabulary of words or other tokens, e.g., event markers) by mapping these to a learned lower-dimensional feature vectors. The maximum integer that can be encountered as well as the dimensionality of the vector embedding are pre-specified as parameters. The node has a few options for performance tuning (op_precision and internal_mapping), and can also be initialized with a pretrained embedding matrix (w_pretrained) as an alternative to random initialization. In this case the input data may only contain one stream. For Packet data, the output of this node has the same axes as the input, except for an additional feature axis (of length embedding_features) at the end. Note that this node does not automatically work on event marker data in packets, since this node, like all other DNN nodes, ignores marker streams. Instead, you will have to define a fixed tabulated vocabulary of event markers and convert any markers you want to process into indices into this vocabulary, which would go into the data array of a regular non-marker stream (which can and most likely should still be indexed by an instance axis etc.) More Info... Version 0.2.0
Ports/Properties¶
data¶
Data to process.
- verbose name: Data
- default value: None
- port type: DataPort
- value type: AnyNumeric (can be None)
- data direction: INOUT
w_init¶
Initializer for the weights.
- verbose name: W Init
- default value: None
- port type: DataPort
- value type: BaseNode (can be None)
- data direction: IN
w_pretrained¶
Optionally a pretrained embedding matrix.
- verbose name: W Pretrained
- default value: None
- port type: DataPort
- value type: AnyArray (can be None)
- data direction: IN
w_prior¶
Optional prior distribution for the weights.
- verbose name: W Prior
- default value: None
- port type: DataPort
- value type: Distribution (can be None)
- data direction: IN
num_categories¶
Number of discrete categories. This is equivalent to the largest integer index that the node may receive plus 1. This is also known as the vocabulary size if the inputs are indices into a dictionary when processing text data.
- verbose name: Num Categories
- default value: 1000
- port type: IntPort
- value type: int (can be None)
embedding_features¶
Number of features in the embedding.
- verbose name: Embedding Features
- default value: 100
- port type: IntPort
- value type: int (can be None)
internal_mapping¶
How the node internally implements the lookup operation. The external behavior of the node is the same in either case, but performance characteristics may differ depending on the implementation and the other settings. The default is currently indices, but this may change in the future.
- verbose name: Internal Mapping
- default value: default
- port type: EnumPort
- value type: str (can be None)
w_initializer¶
Choice of weight initializer. This can either be one of the provided initializers, or the value "custom", in which case one of the Initializer nodes must be wired into the respective input port. For beginners it is recommended to stick to the defaults, since initialization of deep net layers is nuanced and can be tricky, otherwise be prepared to experiment with different choices. In general, the variance-scaling (lecun, glorot/xavier, he/kaiming) initializers are recommended, except for very simple/small layers where you may have a good default assumption as to the distribution of the weights (e.g., truncated_normal or uniform). Bias layers are typically zero-initialized. For initializers that take arguments, you can also type out the arguments positionally as in "truncated_normal(1.0,0.0)" (note reversed order of stddev, mean). The following initializers have arguments (here listed with their defaults): constant(value), those ending in normal(stddev=1, mean=0), those ending in uniform(min=0,max=1), orthogonal(scale=1,axis=-1), identity(gain=1), and variance_scaling(scale=1, "fan_in" (default)/"fan_avg"/"fan_out", "truncated_normal"(default)/"normal"/"uniform",optional-axis-indices=auto). Note that glorot and xavier are aliases for each other, and likewise he and kaiming are aliases for each other.
- verbose name: Weight Initializer
- default value: truncated_normal
- port type: ComboPort
- value type: str (can be None)
op_precision¶
Operation precision. This is a compute performance optimization. See jax documentation for details on these options. Note that this only applies to the operation, while the storage precision may be separately configurable depending on the node in question.
- verbose name: Operation Precision
- default value: default
- port type: EnumPort
- value type: str (can be None)
layername¶
Name of the layer. Used for naming of weights.
- verbose name: Layer Name
- default value: embedding
- port type: StringPort
- value type: str (can be None)
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
- verbose name: Set Breakpoint (Debug Only)
- default value: False
- port type: BoolPort
- value type: bool (can be None)
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.
- verbose name: Metadata
- default value: {}
- port type: DictPort
- value type: dict (can be None)