← machine_learning package

EnsemblePredictor

A model that combines outputs from multiple models trained on subsets of the data.

The node accepts a wired-in base machine-learning pipeline (via the method input), which is instantiated multiple times and which is trained with different subsets of the data and/or random seeds. This node then combines the predictions of the submodels using the specified rule to form a consensus prediction. Such models tend to be less prone to overfitting the training data. Note that this node can be configured to reproduce both the strategy commonly known as bagging (when setting sampling with replacement to True, using a subset size of 1, and using the mean or voting integration rule), as well as the strategy known as pasting (when setting sampling with replacement to False, using a subset size of 0.5-0.8 and using either the mean or voting rule), as well as a robust approach wherein the median is used in conjunction with a subset size of 0.2-0.4 (this is conceptually similar to Theil-Sen estimation); however, note that this strategy is often outperformed by natively robust methods and is therefore mainly useful if there is no other robust option available. Since the computational cost is higher by a factor of the number of models, this node is often used only in the final stages of model development, when the underlying base model model is largely finalized; however, it can also be used during development if the base model is otherwise prone to overfitting or has robustness issues. Version 0.7.1

Ports/Properties

data

Data to process.

• verbose name: Data
• default value: None
• port type: DataPort
• value type: AnyNumeric (can be None)
• data direction: INOUT

method

Underlying base method.

• verbose name: Method
• default value: None
• port type: GraphPort
• value type: Graph

method__signature

Argument names of an underlying base model. Your base model is a subgraph that must contain at least one Placeholder node whose slotname must match the argument name listed here. The placeholder then acts as the entry point for any data that is passed into the pipeline when it is invoked by the ensembling node. Your pipeline's final node (which typically produces the predictions) is then wired to the ensembling node's "method" input port. In graphical UIs, this edge will be displayed in dotted style to indicate that this is not normal forward data flow, but that a subgraph (your pipeline) runs under the control of the ensembling node. In summary, your pipeline starts with a Placeholder that is followed by some processing nodes (in the simplest case just a single machine-learning node, such as Linear Discriminant Analysis). The final node of your pipeline is the one whose outputs are taken to be the pipeline's predictions, and this node is wired into the "method" input of this node. As always in NeuroPype, any "loose ends" downstream of your placeholder are also considered to be part of the pipeline but do not contribute to the result (they may be used for other purposes, such as printing progress information). Your pipeline will generally be instantiated multiple times, and at training time, each instance will be given a different part of the data. Your pipeline may optionally have a additional placeholders, as follows: you may have an is_training placeholder (used to indicate whether your pipeline is currently being called on training data or test data), a random_seed placeholder (which receives a different random seed than the respective other models), and an index parameter (indicating the 0-based index of the current model).

• verbose name: Method [Signature]
• default value: (data)
• port type: Port
• value type: object (can be None)

enabled

Whether to enable ensembling. If this is unset, the node will have essentially no effect (other than passing the configured random seed directly to the model, if the model has such a placeholder).

• verbose name: Enabled
• default value: True
• port type: BoolPort
• value type: bool (can be None)

num_models

Number of models to train.

• verbose name: Num Models
• default value: 10
• port type: IntPort
• value type: int (can be None)

data_subset

Fraction of the data to use for training each model. Can also be given as a negative integer, in which case the value encodes the number of data points to hold out. The typical setup when using sampling without replacement is 50-80% of the data, which helps primarily with overfitting when used in conjunction with the mean or voting rule. It is possible to configure the approach to be additionally robust to a small number of outliers in the data by using the median integration rule and choosing a dataset subset that is strictly smaller than the breakdown point of the median estimator (i.e., less than 0.5), for example 0.33. This is the same idea also exploited in the Theil-Sen estimator (when used with random subsets).

• verbose name: Data Subset
• default value: 0.75
• port type: FloatPort
• value type: float (can be None)

rule

The rule used to combine predictions across models. The mean is the typical approach for regression or probabilistic outputs, which helps with overfitting. The median is a robust alternative to the mean that is less affected by outliers, but note that you may need a low subset fraction to make this work well (e.g., 0.25-0.33). Voting predicts the most likely class label according to each submodel and then takes the majority vote; this approach is also fairly robust to outliers, but note that it erases the probabilistic information in the predictions.

• verbose name: Rule
• default value: mean
• port type: EnumPort
• value type: str (can be None)

with_replacement

Whether to sample with replacement. If set, this implements bootstrap aggregation ("bagging"), and if unset, it implements a technique that has been called "pasting". Both techniques can have pros and cons, and bagging is more commonly used, but can cause problems with certain types of models, for example those that perform internal cross-validations (e.g., ParameterOptimization, ProbabilityCalibration) or ones that aggregate triasl within subject (e.g., TrialAggregatePredictor).

• verbose name: Sample With Replacement
• default value: False
• port type: BoolPort
• value type: bool (can be None)

stratified

Whether to stratify the sampling by class labels. This helps ensure that the proportion of class labels in each subset is representative of the proportion on the full dataset. This is only implemented for the case where each item (e.g., packet) has a single label. This can be useful for small datasets.

• verbose name: Stratified
• default value: False
• port type: BoolPort
• value type: bool (can be None)

cond_field

The name of the instance data field that contains the conditions (classes) to be discriminated; used for stratified sampling.

• verbose name: Condition Field
• default value: TargetValue
• port type: StringPort
• value type: str (can be None)

num_procs

Number of processes to use for parallel computation. If None, the global setting NUM_PROC, which defaults to the number of CPUs on the system, will be used.

• verbose name: Max Parallel Processes
• default value: 1
• port type: IntPort
• value type: int (can be None)

num_threads_per_proc

Number of threads to use for each process. This can be used to limit the number of threads by each process to mitigate potential churn.

• verbose name: Threads Per Process
• default value: 4
• port type: IntPort
• value type: int (can be None)

compute_backends

GPU compute backends that may be used by the pipeline. If you include GPU compute backends here, workloads using those backends will be farmed out across multiple GPUs (if available) when running cross-validation folds in parallel. The 'auto' mode will attempt to auto-detect any backend settings in the given pipeline's nodes, but note that this will only catch nodes where this is explicit in the node's properties, and GPU workloads missed in this fashion will run by default on GPU 0.

• verbose name: Compute Backends
• default value: ['auto']
• port type: SubsetPort
• value type: list (can be None)

num_procs_per_gpu

Number of processes to use per GPU. This is only relevant if you have GPU compute backends enabled. If your GPU(s) are under-utilized during cross-validation, you can increase this to run this many models on each GPU.

• verbose name: Processes Per Gpu
• default value: 1
• port type: IntPort
• value type: int (can be None)

multiprocess_backend

Backend to use for farming out computation across multiple (CPU) processes. Multiprocessing is the simple Python default, which is not a bad start. Nestable is a version of multiprocessing that allows your pipeline to itself use parallel computation. Loky is a fast and fairly stable backend, but it does not support nested parallelism and has different limitations than multiprocessing. It can be helpful to try either if you are running into an issue trying to run something in parallel. Serial means to not run things in parallel but instead in series (even if num_procs is >1), which can help with debugging. Threading uses Python threads in the same process, but this is not recommended for most use cases due to what is known as GIL contention.

• verbose name: Multiprocess Backend
• default value: serial
• port type: EnumPort
• value type: str (can be None)

serial_if_debugger

If True, then if the Python debugger is detected, the node will run in serial mode, even if multiprocess_backend is set to something else. This is useful for debugging, since the debugger does not work well with parallel processes. This can be disabled if certain steps should nevertheless run in parallel (e.g., to reach a breakpoint more quickly).

• verbose name: Serial If Debugger
• default value: True
• port type: BoolPort
• value type: bool (can be None)

initialize_once

Calibrate the model only once. If set to False, then this node will recalibrate itself whenever a non-streaming data chunk is received that has both training labels and associated training instances.

• verbose name: Calibrate Only Once
• default value: True
• port type: BoolPort
• value type: bool (can be None)

dont_reset_model

Do not reset the model when the preceding graph is changed. Normally, when certain parameters of preceding nodes are being changed, the model will be reset. If this is enabled, the model will persist, but there is a chance that the model is incompatible when input data format to this node has changed.

• verbose name: Do Not Reset Model
• default value: False
• port type: BoolPort
• value type: bool (can be None)

random_seed

Seed for any pseudo-random choices during training. This can be either a splittable seed as generated by Create Random Seed or a plain integer seed.

• verbose name: Random Seed
• default value: 12345
• port type: Port
• value type: AnyNumeric

verbosity

Verbosity level for diagnostics.

• verbose name: Verbosity
• default value: 1
• port type: IntPort
• value type: int (can be None)

set_breakpoint

Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

• verbose name: Set Breakpoint (Debug Only)
• default value: False
• port type: BoolPort
• value type: bool (can be None)

metadata

User-definable meta-data associated with the node. Usually reserved for technical purposes.

• verbose name: Metadata
• default value: {}
• port type: DictPort
• value type: dict (can be None)