Package: deep_learning

Deep learning nodes.

These nodes can be used to implement deep learning workflows, including training and inference. The main node here is the Deep Model node, which is a simple "one-stop shop" for controlling the training and use of a neural network in a workflow, similarly to conventional machine learning nodes such as LinearDiscriminantAnalysis. This node receives a network (composed of, among others, Layer nodes and other math operations) and may also receive an optimizer step. However, one may also build deep learning workflows from scratch using the Net nodes, Step nodes, and the Gradient and/oror Jacobian nodes (found in the optimization category). Deep learning nodes use by convention to instance axis to index observations (e.g., in a mini-batch) and the feature axis for what are typically known as "channels" in convolution operations. Other axes, such as space, time, frequency and so forth may be used to encode other spatial or temporal dimensions of the data as needed, and will propagate through the nodes as one would expect. The following sets of nodes are provided: - Nodes ending in Layer: these are the traditional neural network layers, which are characterized by containing implicit trainable parameters. - Nodes ending in Initializer: these can be used to initialize the parameters of a layer, but they are less frequently needed in practice, since the initializer can also be specified from a drop-down menu per node (as a string in Python). - Nodes ending in Norm: these are normalization stages that can be interspersed between layers. Some have (non-trainable) state, which needs to be explicitly managed when using low-level optimization primitives. - Node starting with Net: these ar ethe high-level network management nodes, which act on a whole network module (i.e. a set of layers). These are used to define a module, materialize or share it in a larger computational graph, and to obtain initialization and forward-pass functions to perform training. - nodes ending in Step: these are the optimization steps that can be used to train a network. There are two categories: end-to-end optimizer steps such as AdamStep, and partial gradient processing steps such as CenteringStep. - node with Core in the name: these pertain to recurrent cores, that is, the portions of networks that receive (part) of their past output as input. - nodes ending in Schedule: these are used to schedule the learning rate and other hyperparameters during training, which are typically annealed. - other nodes are stateless (pure math) operations that are frequently used in neural networks, e.g., pooling, activation functions, gradient, and so forth. Note that many other nodes from other categories, especially any mathematical operation nodes that have a "backend" parameter can be used in neural nets.

Nodes in this package:

• AMSGrad Step
  The AMSGrad optimizer step.

• Activation Function (Net)
  Apply an elementwise non-linear neural network activation function to the given data.

• AdaBelief Step
  The AdaBelief optimizer step.

• AdaFactor Step
  The AdaFactor optimizer step.

• Adagrad Step
  The Adagrad optimizer step.

• Adam Step
  The Adam optimizer step.

• AdamW Step
  The AdamW optimizer step (adam with weight decay).

• Adamax Step
  The Adamax optimizer step.

• AdamaxW Step
  The AdamaxW optimizer step (adamax with weight decay).

• Additive Noise (Augmentation)
  Add univariate noise drawn from a given distribution to the data.

• Additive Noise Step
  Chainable step that adds Gaussian noise to the gradients.

• Aggregate Step
  A modifier for the step node that is wired into it which accumulates k successive gradient evaluations and passes them to the optimizer for use in one summed (or averaged) update.

• Apply If Finite Step
  A modifier of the step node that is wired into it, which prevents NaN or infinite or inf updates from going through unless the max_consecutive_errors has been exceeded, in which case the update goes through.

• Batch Flatten (Net)
  Flatten all axes of the input tensor except for the batch dimension (or more generally n leading dimensions).

• Batch Normalization (Net)
  Apply batch normalization to the given data.

• Batch Reshape (Net)
  Reshape input tensor preserving the batch dimension.

• Centering Step
  Chainable step that centers the gradients (subtracts their mean).

• Chained Step
  Compose a sequence of gradient processing steps into a single update step.

• Constant Initializer (Net)
  An initializer that always returns the same constant value.

• Constant Schedule
  A constant parameter schedule.

• Constraint Step
  Chainable step that allows parameters (or in some cases, gradients) to be constrained by projecting them into the desired form.

• Convolution Layer (Net)
  A 1/2/3/N-D standard convolution layer.

• Cosine Decay Schedule
  A cosine decay schedule.

• Cosine One-Cycle Ramp Schedule
  A one-cycle cosine ramp up/down parameter schedule.

• Custom Initializer (Net)
  A custom initializer that is defined by a computational graph that takes in a shape and data type (string) and returns an array of the given shape/type.

• Custom Schedule
  A custom schedule that is defined by a graph function that takes in a step count and returns a parameter value.

• Custom Step
  A custom optimizer step that is defined by a graph applied to gradients and, optionally, current weights.

• Cyclic Linear Warmup Cosine Decay (SGDR) Schedule
  A cyclic linear warmup followed by cosine decay schedule.

• Differentially Private SGD Step
  The differentially private SGD (DPSGD) optimizer step.

• Deep Model
  A deep-learning based machine learning model.

• Dense Layer (Net)
  Dense neural network layer.

• Depthwise Convolution Layer (Net)
  A 1/2/3/N-D depthwise convolution layer.

• Depthwise Separable Convolution Layer (Net)
  A 1/2/3/N-D depthwise separable convolution layer.

• Dropout (Net)
  Apply dropout regularization to the data.

• Categorical Embedding Layer (Net)
  A trainable layer for mapping categorical (integer) data to low-dimensional vectors.

• Exponential Decay Schedule
  An exponential decay (or growth) schedule.

• Fromage Step
  The Fromage (Frobenius Matched Gradient Descent) optimizer step.

• Gated Recurrent Unit (GRU) Layer (Net)
  A gated recurrent unit (GRU) recurrent core, based on Chung et al.

• Gradient Clipping Step
  Chainable step that clips incoming gradients based on their norm, ensuring that the gradient norm does not exceed a provided threshold.

• Group Normalization (Net)
  Apply group normalization to the given data.

• Identity Initializer (Net)
  An initializer that initializes the array to an identity matrix or stacks thereof (where the last two dimensions are the identity matrix).

• Instance Normalization (Net)
  Apply instance normalization to the given data.

• LAMB Step
  The LAMB optimizer step.

• LARS Step
  The LARS optimizer step.

• Layer Normalization (Net)
  Apply layer normalization to the given data.

• Linear One-Cycle Ramp Schedule
  A one-cycle linear ramp up/down parameter schedule.

• Linear Schedule
  A linear parameter schedule.

• Long Short-Term Memory (LSTM) Layer (Net)
  A long short-term memory (LSTM) recurrent layer, based on Hochreiter and Schmidhuber (1997).

• Mixup (Augmentation)
  Interpolate between training exemplars, including class labels.

• Momentum Step
  Chainable step that adds momentum based on one of several formulations, including classic momentum, Nesterov acceleration, and exponential moving average.

• Multi-Head Attention Layer (Net)
  A multi-head attention layer.

• Multiplicative Noise (Augmentation)
  Scale the given data by noise drawn from some provided distribution.

• Define Net
  Define a neural network module (subnet).

• Apply Net (Forward Pass)
  Define a neural network module (subnet).

• Initialize Net
  Get initial weights and optionally state for a network that has been transformed to functional form.

• Materialize Net
  Materialize a network module (subnet), which allows it to be wired into a larger computational graph along with input placeholders and other nodes (incl.

• Share Net
  Reuse a materialized network in a computational graph.

• Transform Net
  Transform a computational graph that involves neural net nodes into a functional form, with weights threaded out.

• Net Weight Array
  Define network weight array.

• NoisySGD Step
  The NoisySGD optimizer step.

• Normal Initializer (Net)
  An initializer that draws initial weights from a Gaussian normal distribution with a given mean and standard deviation.

• Novograd Step
  The Novograd optimizer step.

• Null Step
  A no-op gradient update step that can be used to explicitly freeze (a subset of) weights, e.g

• Optimistic Gradient Descent Step
  The Optimistic gradient descent optimizer step.

• Orthogonal Initializer (Net)
  An initializer that generates a random matrix of orthogonal vectors.

• Partitioned Step
  Apply a set of steps to different labeled subsets of the parameters, using a separately provided labeling dictionary, yielding a composite update step.

• Piecewise Constant Schedule (Factor-based)
  A piecewise-constant parameter schedule.

• Piecewise Interpolated Schedule (Factor-based)
  A piecewise interpolated parameter schedule.

• Polynomial Schedule
  A polynomial parameter schedule.

• Pooling (Net)
  Perform an N-dimensional spatial pooling operation (average or max) on the given data.

• Rectified Adam Step
  The Rectified Adam optimizer step.

• RMS Normalization (Net)
  Apply RMS normalization to the given data.

• RMSProp Step
  The RMSProp optimizer step.

• Randomize Cap Rotation (Augmentation)
  Simulate small rotations of the cap montage to augment neural data.

• Random Time Slice (Augmentation)
  Extract a randomly offset time slice of the given length from the (already-segmented) input data.

• Recurrent Loop (Net)
  Loop a graph representing a recurrent neural network across an input array (e.g

• Stochastic Gradient Descent (SGD) Step
  The Stochastic Gradient Descent (SGD) optimizer step.

• SM3 Step
  The SM3 optimizer step.

• Scaling Step
  Chainable step that scales the gradients by a fixed factor and/or a factor that varies on a schedule.

• Sequence of Schedules
  A composite schedule that is a sequence of multiple provided schedules, each with a given starting step count.

• Apply Step
  Apply an optimizer step to given gradients, prior state, and optionally prior weights.

• Init Step
  Get initial state for an optimization step.

• Transposed Convolution Layer (Net)
  A 1/2/3/N-D "transposed" (upscaling) convolution layer.

• Truncated Normal Initializer (Net)
  An initializer that draws initial weights from a truncated Gaussian distribution with a given mean and standard deviation.

• Trust Ratio Scaling Step
  A chainable step that scales gradients by the trust ratio (ratio of parameter norm to update norm).

• Uniform Initializer (Net)
  An initializer that draws initial weights from a uniform distribution with a given minimum and maximum.

• Variance Scaling Initializer (Net)
  Initialize weights from a distribution whose scale is adapted to the shape of the initialized array.

• Linear Warmup Cosine Decay Schedule
  A linear warmup followed by cosine decay schedule.

• Linear Warmup Exponential Decay Schedule
  A linear warmup followed by an exponential decay parameter schedule.

• Weight Decay Step
  Chainable step that applies weight decay (analogous to l2 regularization) to the parameters.

• Yogi Step
  The Yogi optimizer step.