← optimization package

LBFGSSolve

Iteratively solve a smooth nonlinear optimization problem using the L-BFGS method.

This node is suitable for solving optimization problems of intermediate dimensionality and data size efficiently and with relatively few evaluations of the cost function. Typically, this is used on convex problems, but it may also be used on non-convex problems at the risk of finding only a local optimum. For an alternative that supports non-smooth problems, see the ProximalGradientSolve node. The L-BFGS method is a quasi-Newton method that uses second-order (curvature) information of the cost function by maintaining a low-rank approximation of the inverse Hessian. This comes with additional tracking and memory overhead per iteration compared to first-order methods, but if the optimum is approximately quadratically shaped (e.g., due to a dominant quadratic term in the cost function), the method can reach the optimum in a smaller number of iterations. The node provides a wide range of tuning parameters to maximize the performance on specific types of data; however, the method tends to be sufficiently robust that these parameters often do not have to be tuned. In simple cases, the initial weights can be omitted, but in the general case, initial weights (e.g., an appropriately shaped all-zeros array or packet) must be provided since otherwise the shape is unknown. Like with all solve nodes, the solution is differentiable, meaning that the node can be used in a place where gradients are taken, for example in the network graph a DeepModel node. More Info... Version 0.5.0

Ports/Properties

data

Data to process.

• verbose name: Data
• default value: None
• port type: DataPort
• value type: AnyNumeric (can be None)
• data direction: IN

weights

Initial and final weights. If not set, will be initialized to a packet equivalent to a single all-zeroes instance of the training-data.

• verbose name: Weights
• default value: None
• port type: DataPort
• value type: AnyNumeric (can be None)
• data direction: INOUT

state

Optional initial and final solver state.

• verbose name: State
• default value: None
• port type: DataPort
• value type: object (can be None)
• data direction: INOUT

cost

Smooth cost function given some data.

• verbose name: Cost
• default value: None
• port type: GraphPort
• value type: Graph

cost__signature

Signature for the "cost" input. This represents the signature for the subgraph that is wired into the "cost" port. This is formatted as in (a,b,c) where a,b,c are names of placeholders that are expected in the subgraph that goes into the "cost" port. Alternatively, it can also be provided in data structure form as a list of lists, as in: [['a','b','c']].

• verbose name: Cost [Signature]
• default value: (w,D)
• port type: Port
• value type: object (can be None)

hyper_params

Hyper-parameters for the cost function. This is a dictionary of arbitrary key-value pairs that can be used to configure the cost function. The graph may then declare and use placeholders named the same as of the dictionary keys. This is mainly such that the solver does not have to be recompiled when these values change; for constants, you may alternatively wire them directly into the cost function.

• verbose name: Additional Cost Parameters
• default value: {}
• port type: DictPort
• value type: dict (can be None)

max_iter

Maximum number of iterations.

• verbose name: Max Iter
• default value: 500
• port type: IntPort
• value type: int (can be None)

abstol

Absolute convergence tolerance. If weights change less than this (after normalization by step size), the optimization terminates. Note that this depends on the data scale.

• verbose name: Abstol
• default value: 0.001
• port type: FloatPort
• value type: float (can be None)

memory_size

Number of previous gradients to remember to approximate curvature. This configures the value of L in L-BFGS (i.e., the limited memory size). Typical values are 6 to 10.

• verbose name: Memory Size
• default value: 10
• port type: IntPort
• value type: int (can be None)

stepsize

Optional step size override. If unspecified, the step size is adapted automatically using back-tracking line search.

• verbose name: Stepsize
• default value: None
• port type: FloatPort
• value type: float (can be None)

max_stepsize

Maximum step size to use. Only used if step size is dynamically adapted.

• verbose name: Max Stepsize
• default value: 1.0
• port type: FloatPort
• value type: float (can be None)

min_stepsize

Minimum step size to use. Only used if step size is dynamically adapted.

• verbose name: Min Stepsize
• default value: 1e-06
• port type: FloatPort
• value type: float (can be None)

linesearch_type

Line-search strategy to use to adapt step sizes.

• verbose name: Linesearch Type
• default value: zoom
• port type: EnumPort
• value type: str (can be None)

linesearch_init

Strategy to initialize the line-search step size for a new iteration. The least aggressive is current, which starts from the previous step size. Max sets it to the max_stepsize at each iteration, and increase increases it by a factor of increase_factor from the previous iteration (or min_stepsize if smaller than that).

• verbose name: Linesearch Init
• default value: increase
• port type: EnumPort
• value type: str (can be None)

max_backtrack

Maximum number of line search steps per iteration. Only used if step size is left to automatic.

• verbose name: Max Backtrack
• default value: 15
• port type: IntPort
• value type: int (can be None)

backtrack_factor

Backtracking line search factor. Only used if stepsize is 0 and if a backtracking linesearch type is used.

• verbose name: Backtrack Factor
• default value: 0.8
• port type: FloatPort
• value type: float (can be None)

increase_factor

Factor by which to increase the step size if the line search succeeds.

• verbose name: Increase Factor
• default value: 1.5
• port type: FloatPort
• value type: float (can be None)

use_jit

If enabled, attempt to use JIT compilation for the inner loop. This incurs a one-time compilation cost, but the actual solving will be greatly accelerated if using the GPU.

• verbose name: Use Jit
• default value: auto
• port type: EnumPort
• value type: str (can be None)

unroll

Whether to unroll the optimization loop. This can save some per-iteration overhead, but the benefit depends on the specific use case.

• verbose name: Unroll
• default value: auto
• port type: EnumPort
• value type: str (can be None)

implicit_diff

Whether to use implicit differentiation for computing gradients. The alternative is to unroll the solver iterations.

• verbose name: Implicit Diff
• default value: True
• port type: BoolPort
• value type: bool (can be None)

implicit_diff_solver

Solver to use for implicit differentiation. Auto is currently defaulting to normal-cg, which is a fast generic solver for high-dimensional problems. The options bicgstab and gmres can be more robust on badly conditioned problems, but are slower. The lu method is likely fastest on very low-dimensional problems, but note that the cost of implicit differentiation is likely marginal compared to the cost of the overall solve.

• verbose name: Implicit Diff Solver
• default value: auto
• port type: EnumPort
• value type: str (can be None)

verbosity

Verbosity level. 0: no output, 1: per-iteration summary. Note that JIT will be disabled if verbosity is used.

• verbose name: Verbosity
• default value: 0
• port type: IntPort
• value type: int (can be None)

stop_if_linesearch_fails

Whether to stop if the line search fails. This can be set to true to either debug bad settings of the min/max step-size values or line-search strategy, or to catch issues early with problematic data if one has strong assumptions about the cost function.

• verbose name: Stop If Linesearch Fails
• default value: False
• port type: BoolPort
• value type: bool (can be None)

use_gamma

Whether to use the gamma parameter in the initialization of the inverse Hessian.

• verbose name: Init Hessian Using Gamma
• default value: True
• port type: BoolPort
• value type: bool (can be None)

incremental_updates

Whether to use incremental updates from the previous state. If not set, the previous or wired-in state is ignored.

• verbose name: Incremental Updates
• default value: False
• port type: BoolPort
• value type: bool (can be None)

set_breakpoint

Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

• verbose name: Set Breakpoint (Debug Only)
• default value: False
• port type: BoolPort
• value type: bool (can be None)

metadata

User-definable meta-data associated with the node. Usually reserved for technical purposes.

• verbose name: Metadata
• default value: {}
• port type: DictPort
• value type: dict (can be None)