← optimization package

ParameterOptimization

Optimize hyper-parameters of a pipeline given a dataset.

The main use case of this node is with machine-learning pipelines that have free parameters (e.g., regularization parameters). The node can auto-tune these parameters given a dataset and optionally an evaluation function (which defaults to an inner cross-validation on the dataset). This works as follows: the user specifies for each free parameter of the pipeline a search range, which can be discrete or continuous (see parameter documentation for the supported formats). The node then samples multiple combinations of parameter settings, and for each such setting it configures the pipeline with the setting and passes the pipeline to the evaluator, which assess the performance of the configured pipeline on the given dataset. Once training is completed, one can retrieve either the optimal parameter setting (via the optimal_params output), or the fully trained method graph (with parameters bound to optimal values) via the optimized output. Furthermore, the Parameter Optimization node can also be used as a proxy for the method and generate training-set predictions (via the data output) on the initial invocation or test-set predictions on subsequent invocations (if initialize_once is set). One can also save and re-load the node's trained model via the model input/output. The pipeline is typically a subgraph with a placeholder node with slotname "data" that represents the input data, and any number of other placeholder nodes representing the tunable parameters (e.g., "lambda", etc.). Note that the node represents a training strategy, and not a validation strategy. To obtain cross-validation scores, you need to pass the entire parameter search procedure with all its inputs, except for the data replaced by a "data" placeholder, into the "method" input of an outer cross-validation node, which then receives the dataset to use for validation. A key parameter is the compute budget, which specifies how many parameter settings to try; the default is a low value sufficient only for a single parameter. The node supports a number of different search strategies, including randomized grid searches and other gradient-free optimization strategies. The node also supports optional parallel computation via the num_procs argument, which can be useful for long-running optimization tasks. Note that the default evaluator is a parallel 5x-cross-validation, so it will by default already use multiple processes, which are further multiplied by the num_procs argument. You can override the evaluator by providing a subgraph with a data and pype argument placeholder, which typically, wire into a cross-validation node. The node will by default log into your NeuroPype logs folder to allow recovery of partial results in case of crashes or long-running tasks. More Info... Version 2.0.1

Ports/Properties

data

Data to process.

• verbose name: Data
• default value: None
• port type: DataPort
• value type: object (can be None)
• data direction: INOUT

method

Pipeline to optimize.

• verbose name: Method
• default value: None
• port type: GraphPort
• value type: Graph

method__signature

Arguments for the pipeline. Note that you do not need to list the placeholders corresponding to the hyper-parameters here, since these are best thought of as optional parameters that should have defaults. The method may have a second argument named is_training, which, if present, receives a True value when training and otherwise False.

• verbose name: Method [Signature]
• default value: (data)
• port type: Port
• value type: object (can be None)

evaluator

Optional evaluation function.

• verbose name: Evaluator
• default value: None
• port type: GraphPort
• value type: Graph

evaluator__signature

Arguments for the optional evaluation function. The job of this function is to take a dataset, a pipeline representing the method to evaluate, and to return a loss measure that shall be minimized. The default graph is simply a Placeholder named data and a GraphPlaceholder named method, which are wired into the data and method inputs of a Crossvalidation node. The default configuration of that crossvalidation is configured to run in serial using the configured settings below, and if the wired-in method graph has a Segmentation node, then it will have its trial_size parameter set to equal the time bounds of that segmentation. This function may be provided in order to have more complete control over how the nested cross-validation is performed (e.g., parallelism, repeats etc), or to replace the cross-validation process by a different evaluation strategy altogether.

• verbose name: Evaluator [Signature]
• default value: (data,method)
• port type: Port
• value type: object (can be None)

optimized

Optimally parameterized pipeline or trained model.

• verbose name: Optimized
• default value: None
• port type: GraphPort
• value type: Graph

optimal_params

Set of optimal parameters.

• verbose name: Optimal Params
• default value: None
• port type: DataPort
• value type: dict (can be None)
• data direction: OUT

searcher

Search method. The following choices are available. For few (under 3) and relatively well-behaved regularization parameters, sampling strategies like Scrambled Hammersley go a long way, and most ML practitioners look no further. For higher- dimensional search spaces, it is typically not possible to exhaustively explore the space, and in such cases the ideal optimizer performs local search in reasonable directions with the abilty to escape local minima; the advanced optimizers are different ways of doing this given different assumptions about the objective function. Despite their sophistication, many of these optimizers require a hefty compute budget well beyond the default setting. Typically, practitioners will try a few different optimizers early on and pick the one that works best. It is not advisable to aggressively tune the choice or parameters of the optimizers in an attempt to boost the generalization error for a paper, since this will yield unreliable (artificially good) results. Choices are:

• Hammersley Sampling: sample the search space in a regular grid; note that the grid is typically not aligned with the parameter axes, but it ensures regular spacing of sample points.
• Scrambled Hammersley Sampling: this is essentially Hammersley with additional randomization of the grid points, which has a better chance of exploring values for individual parameters, since it will not repetitively sample the same values along a given axis.
• Two-Points Crossover Differential Evolution: this is a variant of differential evolution that uses two points to generate a new candidate solution. It is recommended for continuous search spaces.
• Test-Based Population Size Adaptation: performs well in noisy situations.
• Particle-Swarm Optimization: a fairly robust optimization strategy.
• OnePlusOne (generic): a simple and robust optimization strategy for continuous parameters. Has numerous options.
• Covariance Matrix Adaptation (generic): a capable optimization strategy for low-noise siutations, which requires a high compute budget. Has a number of options.
• Bayesian Optimization (generic): a classic Bayesian method, note the presence of options.
• Differential Evolution (generic): a powerful evolutionary strategy; also features additional settings.
• Estimation of Multivariate Normal Algorithm (generic): a fairly efficient optimizer in parallel settings (i.e., high num_procs).
• Latin Hypercube Sampling: This is another pseudo-random sampling pattern with fairly good coverage, but which it is considerably less regular than Hammersley sampling.

• verbose name: Search Strategy
• default value: Hammersley Sampling
• port type: EnumPort
• value type: str (can be None)

budget

Number of parameter settings to evaluate. This is the main parameter that determines how long the optimization will take, and it governs how many parameter combinations will be passed to the given evaluator for evaluation. Note that the budget required to maintain a given sampling density increases exponentially with the number of parameters to optimize.

• verbose name: Compute Budget
• default value: 10
• port type: IntPort
• value type: int (can be None)

param1

Search range for parameter 1. The following formats are supported: name=[1,2,3] (list of values), name=(1...3) (continuous range), name=2**(1...3) (log-spaced continuous range).

• verbose name: Parameter 1
• default value: alpha=[1,2,3]
• port type: StringPort
• value type: str (can be None)

param2

Search range for parameter 2. See param1 for supported formats.

• verbose name: Parameter 2
• default value:
• port type: StringPort
• value type: str (can be None)

param3

Search range for parameter 3. See param1 for supported formats.

• verbose name: Parameter 3
• default value:
• port type: StringPort
• value type: str (can be None)

param4

Search range for parameter 4. See param1 for supported formats.

• verbose name: Parameter 4
• default value:
• port type: StringPort
• value type: str (can be None)

param5

Search range for parameter 5. See param1 for supported formats.

• verbose name: Parameter 5
• default value:
• port type: StringPort
• value type: str (can be None)

searcher_config

Additional configuration for the searcher. This can be used to pass additional parameters to the chosen searcher, but is rarely necessary. The searchers are part of the nevergrad package, and the documentation for the searchers can be found at https://facebookresearch.github.io/nevergrad/optimizers_ref.html.

• verbose name: Searcher Configuration
• default value: None
• port type: DictPort
• value type: dict (can be None)

enabled

Whether to enable parameter search. If disabled, the node will use the midpoint of the respective search ranges as the parameter value.

• verbose name: Enable Search
• default value: True
• port type: BoolPort
• value type: bool (can be None)

default_settings

Default settings when search is NOT enabled. Formatted like a Python dictionary, i.e., {'alpha': 10, 'beta': 0.1}. This is useful when rapidly prototyping parts of a method that requires sensible parameter values without incurring the time cost of a parameter search. Note however that this field should not be used to dial in the final parameter values and then report the resulting performance scores as if they represented the generalization error of the method, since this would be cherry-picking and falls under scientific malpractice -- unless the final parameters were determined on an entirely different dataset that one evaluates performance on, or were educated a priori guesses made well before any data was seen. Instead, when final scores (e.g., for a paper or study) are to be reported, one should leave the parameter search enabled and measure performance in this fashion.

• verbose name: Default Settings
• default value: {}
• port type: DictPort
• value type: dict (can be None)

search_metric

Scoring metric with regards to which to optimize hyper-parameters. Note that which metric is being optimized matters, and the resulting model is likely to perform better with regards to that metric than others, although typically other metrics will also generally improve as a side effect of optimizing one metric; good candidates are BAcc (balanced accuracy) or AUC (area under ROC curve) for classification methods, MSE (mean squared error) or the more robust MAE (mean absolute error) for regression pipelines. It is also possible to optimize for metrics like sensitivity or specificity in limited circumstances. See also the cross-validation node for more details.

• verbose name: Metric To Optimize
• default value: BAcc
• port type: EnumPort
• value type: str (can be None)

num_folds

Number of nested cross-validation folds for parameter search. Note that it is also possible to wire in a subgraph into the evaluator input that gives full control over the nested cross-validation setup. If so, it is recommended to set this parameter to 0 to avoid confusion over which setting is in effect. Cross-validation proceeds by splitting up the data into this many blocks of trials, and then tests the method on each block. This happens on the respective training set (e.g., of an outer cross-validation). For each fold, the method is re-trained on all the other blocks, excluding the test block. This is not a randomized cross-validation, but a blockwise cross-validation, which is usually the correct choice if the data stem from a time series. If there are few trials in the data, one can use a higher number here (e.g., 10) to ensure that more data is available for training. It is also possible to set this to 1, which means leave-one-out CV; typically this is excessive, except in conjunction with the grouping field, which causes one group (of trials that have the same group id) to be held out at a time. If a number between 0 and 1 is used, this is p-holdout cross-validation, where the respective fixed fraction of trials is held out. See also the cross-validation node for additional details and options.

• verbose name: Number Of Cross-Validation Folds
• default value: 5
• port type: FloatPort
• value type: float (can be None)

cv_cond_field

The name of the instance data field that contains the values that the method should be optimized to predict. This parameter will be ignored if the input data packet has previously been processed by a BakeDesignMatrix node.

• verbose name: Condition Field (Cross-Validaiton)
• default value: TargetValue
• port type: StringPort
• value type: str (can be None)

cv_group_field

Optionally a field indicating the group from which each trial is sourced. If given, then data will be split such that test sets contain unseen groups. Examples groups are Subject_id, Session_id, etc.

• verbose name: Grouping Field (Cross-Validation)
• default value:
• port type: StringPort
• value type: str (can be None)

cv_stratified

Optionally perform stratified cross-validation. This means that all the folds have the same relative percentage of trials with each label. Note that this requires labels to be quantized or binned to be meaningful.

• verbose name: Stratified Cross-Validation
• default value: False
• port type: BoolPort
• value type: bool (can be None)

num_procs

Number of processes to use for parallel computation. If None, the global setting NUM_PROC, which defaults to the number of CPUs on the system, will be used.

• verbose name: Max Parallel Processes
• default value: 1
• port type: IntPort
• value type: int (can be None)

num_threads_per_proc

Number of threads to use for each process. This can be used to limit the number of threads by each process to mitigate potential churn. This can be further overridden by the used evaluator.

• verbose name: Threads Per Process
• default value: 4
• port type: IntPort
• value type: int (can be None)

multiprocess_backend

Backend to use for farming out computation across multiple (CPU) processes. Multiprocessing is the simple Python default, which is not a bad start. Pebble can be more robust than multi-pprocessing and can be a useful alternative to multiprocessing if you are running out of memory after a while or if tasks otherwise appear to hang or crash unexpectedly. Serial means to not run things in parallel but instead in series (even if num_procs is >1), which can help with debugging. Threading uses Python threads in the same process, but this is not recommended for most use cases due to what is known as GIL contention.

• verbose name: Multiprocess Backend
• default value: multiprocessing
• port type: EnumPort
• value type: str (can be None)

assign_gpus

Assign GPUs for the parameter search. If enabled, the parameter search will make sure that each process gets a different GPU (potentially giving the same GPU to more than one process if there are more processes than GPUs on the system).

• verbose name: Assign Gpus
• default value: False
• port type: BoolPort
• value type: bool (can be None)

serial_if_debugger

If True, then if the Python debugger is detected, the node will run in serial mode, even if multiprocess_backend is set to something else. This is useful for debugging, since the debugger does not work well with parallel processes. This can be disabled if certain steps should nevertheless run in parallel (e.g., to reach a breakpoint more quickly).

• verbose name: Serial If Debugger
• default value: True
• port type: BoolPort
• value type: bool (can be None)

loss_if_error

Loss value to return if an error occurs during evaluation. If None, the error will be raised. This can be useful to prevent the optimization from crashing if a parameter setting causes an error, but it can also mask errors that would otherwise be caught. This value may be set to the worst-case loss according to the measure used by the evaluator (e.g., 1.0 for accuracy); potentially larger values can also be used, but note that their effect on the search process depends on the selected search method (some may be more likely to steer cleer of error-causing parameters while others might become unstable).

• verbose name: Loss Value If Error
• default value: None
• port type: FloatPort
• value type: float (can be None)

straggler_cutoff

Cutoff time for stragglers in standard deviations. If a parameter setting takes more than this many standard deviations longer than the average time, it will be discarded. This helps mitigate hangs due to bad parameter settings hat take excessively long or hangs with individual runs. In rare cases this will trim off runs that would have been useful, in which case you may increase this parameter.

• verbose name: Long-Running Task Cutoff
• default value: 5
• port type: FloatPort
• value type: float (can be None)

random_seed

Random seed (int or None). Different values will give somewhat different outcomes.

• verbose name: Random Seed
• default value: 12345
• port type: IntPort
• value type: int (can be None)

log_folder

Folder to store optimization logs. This defaults to NeuroPype's log folder. This can be useful to recover partial results if your parameter search takes very long and has to be terminated (or crashes mid-way). If you set it to empty, no logs will be stored.

• verbose name: Log Folder
• default value: None
• port type: StringPort
• value type: str (can be None)

initialize_once

Calibrate the model only once. If set to False, then this node will recalibrate itself whenever a non-streaming data chunk is received that has both training labels and associated training instances.

• verbose name: Calibrate Only Once
• default value: True
• port type: BoolPort
• value type: bool (can be None)

dont_reset_model

Do not reset the model when the preceding graph is changed. Normally, when certain parameters of preceding nodes are being changed, the model will be reset. If this is enabled, the model will persist, but there is a chance that the model is incompatible when input data format to this node has changed.

• verbose name: Do Not Reset Model
• default value: False
• port type: BoolPort
• value type: bool (can be None)

verbose

Print verbose output.

• verbose name: Verbose
• default value: True
• port type: BoolPort
• value type: bool (can be None)

set_breakpoint

Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

• verbose name: Set Breakpoint (Debug Only)
• default value: False
• port type: BoolPort
• value type: bool (can be None)

metadata

User-definable meta-data associated with the node. Usually reserved for technical purposes.

• verbose name: Metadata
• default value: {}
• port type: DictPort
• value type: dict (can be None)