Package: optimization

Optimization nodes.

This module contains nodes related to optimization algorithms. The most important specialty nodes are Gradient, Jacobian, Performance Metric, and Value And Gradient. Besides these, the following types of nodes are found here: - Solve nodes: these solve a specific category of optimization problem and output the solution - Loss nodes: these measure the discrepancy between desired (target) and predicted values, and output that "loss" (per sample) - Penalty nodes: these implement a specific type of (usually non-differentiable) cost in a fashion that is usable with the Proximal Gradient Descent Solve and the Convex Model node (in the ML category). These represent proximal operators. - Constraint nodes: these nodes accept a (usually vector) value and apply a constraint to it, such as bounding it to a specific range; these nodes are usable with the Proximal Gradient Descent Solve node.

Nodes in this package:

• Box Constraint
  A constraint projection forcing the given solution to lie inside an n-dimensional box.

• Clustered Penalty
  A penalty encouraging the patterns along individual rows of the solution matrix to condense into few similar stereotypical patterns; implemented as a proximal operator.

• Cosine Distance (Loss)
  Calculate the cosine distance between vector predictions and vector targets.

• Function Transformation (Internal)
  Apply a library-based transformation to a graph and invoke the resulting graph with some arguments.

• Gradient
  Calculate the gradient for a given graph with respect to one or more of its parameters.

• Uniform Group-Sparse Penalty
  A penalty encouraging group-sparse solutions with groups corresponding to the entries along one or more axes; implemented as a proximal operator.

• Hinge Loss
  Calculate the Hinge loss between predictions and target values.

• Huber Loss
  Calculate the robust Huber loss between predictions and regression target values.

• Jacobian
  Calculate the Jacobian matrix for a given graph with respect to one or more of its parameters.

• KL Divergence (Loss)
  Calculates the KL divergence between a predicted/proposed discrete probability distribution (in log-space) and a target/known distribution (by default given as probabilities but optionally in log-space).

• Smooth Nonlinear Solve (L-BFGS)
  Iteratively solve a smooth nonlinear optimization problem using the L-BFGS method.

• Linear Equality Constraint
  A constraint projection forcing the given solution vector to satisfy a linear matrix equation Ax=b; the data will be flattened to a vector before applying the projection.

• Linear Solve (Operator)
  Solve a linear system of equations of the form Ax=b.

• Log-Cosh Loss
  Calculate the robust log-cosh loss between predictions and regression targets.

• Low-Rank Penalty
  A penalty encouraging solution matrices or tensors that are low rank with respect to two or more axes.

• Non-negative Constraint
  A constraint projection forcing the given solution to have no negative elements.

• Non-negative Sum Constraint
  A constraint projection forcing the given solution to sum to a constant value, with all values being non-negative.

• Nonuniform Group-Sparse Penalty
  A penalty encouraging group-sparse (few nonzero groups of elements) solutions along one or more axes with non-uniform group sizes, implemented as a proximal operator.

• Norm Constraint
  A constraint projection forcing the given solution to have a norm of at most or exactly the given scale.

• Overlapping Group-Sparse Penalty
  A penalty encouraging group-sparse (few nonzero groups of elements) solutions along one or more axes with overlapping and non-uniform group sizes.

• Parameter Optimization
  Optimize hyper-parameters of a pipeline given a dataset.

• Performance Metric
  Calculate one of a set of performance metrics between predictions and target values.

• Plane Constraint
  A constraint projection forcing the given solution to lie on a plane in N-dimensional space (hyper-plane) or in a half-space bounded by the plane (on one side of the plane).

• Proximal Gradient Descent Solve
  Iteratively solve a convex optimization problems using the proximal gradient descent method.

• Sigmoid Binary Cross-Entropy Loss
  Calculate a cross-entropy loss with either binary labels or multi-class (non-exclusive) labels and logit predictions (unnormalized log probabilities).

• Softmax Cross-Entropy Loss
  Calculate a softmax cross-entropy loss between logit predictions (unnormalized log probabilities) and integer target labels or a label distribution.

• Sparse Penalty
  A penalty encouraging sparse (few nonzeros) solutions, as a proximal operator for the classic l1 norm.

• Squared or L2 Loss
  Calculate a squared error between predictions and regression target values.

• Value and Gradient
  Calculate the function value and gradient for a given graph with respect to one or more of its parameters.