← machine_learning package

TrialAggregatePredictor

A model that makes aggregate predictions based on multiple trials.

The node accepts a wired-in single-trial machine-learning pipeline (via the method input), which is trained and tested on single trials, and which may output probabilities or other continuous-valued outputs. This node then aggregates the single-trial outputs into a single prediction for each group of trials, based on the provided group field and integration rule. The node furthermore uses probability calibration to ensure that the single-trial outputs are valid probabilities that accurately quantify the confidence at the trial level, which is not available from the raw single-trial predictions. Currently this node is only suitable for binary classification tasks. For technical reasons, probability calibration is usually not be performed on the same data that a classifier was trained on, since the classifier will generate confidence scores on its own training data that will not be representative of the classifier's outputs on new data. Therefore, the calibration generally uses a cross-validation approach to train the underlying classifier on different data than what is used to fit the calibration mapping (see also cross-validation node for full details on cross-validation is generally done). Version 0.5.0

Ports/Properties

data

Data to process.

• verbose name: Data
• default value: None
• port type: DataPort
• value type: Packet (can be None)
• data direction: INOUT

method

Single-trial scoring model.

• verbose name: Method
• default value: None
• port type: GraphPort
• value type: Graph

method__signature

Argument names of an underlying single-trial pipeline. Your pipeline is a subgraph that must contain at least one Placeholder node whose slotname must match the argument name listed here. The placeholder then acts as the entry point for any data that is passed into the pipeline when it is invoked by the aggregation node. Your pipeline's final node (which typically produces the predictions) is then wired to the aggregation node's "method" input port. In graphical UIs, this edge will be displayed in dotted style to indicate that this is not normal forward data flow, but that a subgraph (your pipeline) runs under the control of the aggregation node. In summary, your pipeline starts with a Placeholder that is followed by some processing nodes (in the simplest case just a single machine-learning node, such as Linear Discriminant Analysis). The final node of your pipeline is the one whose outputs are taken to be the pipeline's predictions, and this node is wired into the "method" input of this node. As always in NeuroPype, any "loose ends" downstream of your placeholder are also considered to be part of the pipeline but do not contribute to the result (they may be used for other purposes, such as printing progress information). Your pipeline may optionally have a second placeholder, which should by convention have slotname set to is_training, and then is_training must be listed as the second argument here. This second placeholder is used to indicate whether your pipeline is currently being called on training data or test data. During training, the aggregation node will execute your pipeline like a cross-validation, meaning that, for each fold in the cross-validation, your pipeline graph is instantiated from its default (uninitialized) state, and is then called with the training set of that fold. Then, the same graph is called again, but this time with the test set of that fold; it is then up to any adaptive nodes in your pipeline (e.g., machine learning nodes) to adapt themselves on the first call and to make predictions (usually without adapting again) on the second call. The pipeline is discarded after each fold and a new pipeline graph is instantiated (to avoid any unintended train/test leakage).

• verbose name: Method [Signature]
• default value: (data)
• port type: Port
• value type: object (can be None)

group_field

The name of the instance data field that contains the group identifiers. Trials with the same group identifier will be aggregated together at prediction time. It is assumed that within a group the condition is the same.

• verbose name: Group Field
• default value: SubjectID
• port type: StringPort
• value type: str (can be None)

cond_field

The name of the instance data field that contains the conditions to be discriminated. This parameter will be ignored if the packet has previously been processed by a DescribeStatisticalDesign node.

• verbose name: Cond Field
• default value: TargetValue
• port type: StringPort
• value type: str (can be None)

class_weights

Per-class weights. Optionally this is a mapping from class label to weight. The weights represent the a priori ("prior") probability of encountering a specific class that the model shall assume. The weights will be renormalized so that they add up to one. Example syntax: {'0': 0.5, '1': 0.5} (note the quotes before the colons).

• verbose name: Per-Class Weight
• default value: None
• port type: Port
• value type: object (can be None)

aggregate

Whether to enable aggregation. If disabled, the single-trial responses of the underlying model will be passed through.

• verbose name: Perform Aggregation
• default value: True
• port type: BoolPort
• value type: bool (can be None)

rule

The rule used to integrate the single-trial responses; these mostly control a statistical efficiency vs robustness (to outlier trials) tradeoff. The default is to average the probabilities, but other methods are available that may be more robust to outliers or more sensitive to the tails of the distribution. The 'mean' and median rules are the arithmetic mean, and median, respectively. The 'huber', 'trim_mean' and 'winsorize' options are the Huber mean, trimmed mean, and winsorized mean, respectively. These use the outlier cutoff, which may need to be set appropriately.

• verbose name: Rule
• default value: trim_mean
• port type: EnumPort
• value type: str (can be None)

outlier_cutoff

Outlier cutoff when aggregating robustly across trials. When using the trim_mean or winsorize rules, this is the proportion of trials to trim (default 0.1), and when using the huber rule, this is the cutoff point in (robust) standard deviations from the median (also known as the Huber constant); default 1.345.

• verbose name: Outlier Cutoff
• default value: None
• port type: FloatPort
• value type: float (can be None)

model_is_probabilistic

Whether the underlying model outputs probabilities. If left to auto, the node will attempt to auto-detect this and if it fails, will report an error.

• verbose name: Model Is Probabilistic
• default value: auto
• port type: EnumPort
• value type: str (can be None)

max_probability

Maximum probability value from the underlying model. This is used to clip the probabilities to avoid numerical instability when the model outputs very high probabilities. Only used if the model is probabilistic.

• verbose name: Max Probability
• default value: 0.99
• port type: FloatPort
• value type: float (can be None)

calibrate_probabilities

Whether to perform probability calibration of the groupwise outputs. If this is not enabled, the generated probabilities will typically be wildly conservative, as if a single-trial prediction was made.

• verbose name: Calibrate Probabilities
• default value: True
• port type: BoolPort
• value type: bool (can be None)

num_folds

Number of cross-validation folds for out-of-sample calibration. Note that this cross-validation generally acts on the groups of trials, not on individual trials; the default is to leave one group out, but if there are many groups, this can be set to an explicit number of folds such as 5 or 10. Cross-validation proceeds by splitting up the data into this many blocks of trials, and then tests the method on each block. For each fold, the method is re-trained on all the other blocks, excluding the test block (therefore, the total running time is proportional to the number of folds). This is not a randomized cross-validation, but a blockwise cross-validation, which is usually the correct choice if the data stem from a time series. If there are few trials in the data, one can use a higher number here (e.g., 10) to ensure that more data is available for training. This can also be set to 0 if no cross-validation is desired, in which case the method and calibration mapping are trained on all the data, or to 1 to use a (perhaps costly) leave-one-out cross-validation.

• verbose name: Number Of Cross-Validation Folds
• default value: 1
• port type: IntPort
• value type: int (can be None)

cv_stratified

Optionally perform stratified cross-validation. This means that all the folds have the same relative percentage of trials in each class.

• verbose name: Stratified Cross-Validation
• default value: True
• port type: BoolPort
• value type: bool (can be None)

refit_bias

Whether to refit the bias term in the calibration mapping. When this is disabled, only the confidence of the classifier is recalibrated, which is usually what is desired when switching from single trials to groupwise predictions. However, one may additionally rebias the classifier by enabling this option, but note that the number of data points available for this is typically much lower than the number of trials, so the bias is likely to be somewhat less accurately estimated than by the underlying single-trial model. Nevertheless, if the underlying model is biased in out-of-sample predictions, or very close to chance level, then this this may have to be enabled since otherwise the calibration may end up predicting the opposite of the underlying model (this can be spotted by enabling the sanity checks).

• verbose name: Refit Bias Term
• default value: False
• port type: BoolPort
• value type: bool (can be None)

calibrator

Type of classifier to use. Logreg is a simple logistic regression, and bayes-logreg is a Bayesian logistic regression. The latter has more sensible priors on the weights and can therefore be more robust, especially when the underlying single-trial predictor is very weak (e.g., close to chance level).

• verbose name: Calibrator
• default value: logreg
• port type: EnumPort
• value type: str (can be None)

sanity_checks

Check the accuracy of the underlying model on the training data. If the model is worse than chance, a warning will be issued. This is useful to catch bugs in the underlying model.

• verbose name: Sanity Checks
• default value: False
• port type: BoolPort
• value type: bool (can be None)

num_procs

Number of processes to use for parallel computation. If None, the global setting NUM_PROC, which defaults to the number of CPUs on the system, will be used.

• verbose name: Max Parallel Processes
• default value: 1
• port type: IntPort
• value type: int (can be None)

num_threads_per_proc

Number of threads to use for each process. This can be used to limit the number of threads by each process to mitigate potential churn.

• verbose name: Threads Per Process
• default value: 4
• port type: IntPort
• value type: int (can be None)

compute_backends

GPU compute backends that may be used by the pipeline. If you include GPU compute backends here, workloads using those backends will be farmed out across multiple GPUs (if available) when running cross-validation folds in parallel. The 'auto' mode will attempt to auto-detect any backend settings in the given pipeline's nodes, but note that this will only catch nodes where this is explicit in the node's properties, and GPU workloads missed in this fashion will run by default on GPU 0.

• verbose name: Compute Backends
• default value: ['auto']
• port type: SubsetPort
• value type: list (can be None)

num_procs_per_gpu

Number of processes to use per GPU. This is only relevant if you have GPU compute backends enabled. If your GPU(s) are under-utilized during cross-validation, you can increase this to run this many CV folds on each GPU.

• verbose name: Processes Per Gpu
• default value: 1
• port type: IntPort
• value type: int (can be None)

multiprocess_backend

Backend to use for farming out computation across multiple (CPU) processes. Multiprocessing is the simple Python default, which is not a bad start. Nestable is a version of multiprocessing that allows your pipeline to itself use parallel computation. Loky is a fast and fairly stable backend, but it does not support nested parallelism and has different limitations than multiprocessing. It can be helpful to try either if you are running into an issue trying to run something in parallel. Serial means to not run things in parallel but instead in series (even if num_procs is >1), which can help with debugging. Threading uses Python threads in the same process, but this is not recommended for most use cases due to what is known as GIL contention.

• verbose name: Multiprocess Backend
• default value: serial
• port type: EnumPort
• value type: str (can be None)

serial_if_debugger

If True, then if the Python debugger is detected, the node will run in serial mode, even if multiprocess_backend is set to something else. This is useful for debugging, since the debugger does not work well with parallel processes. This can be disabled if certain steps should nevertheless run in parallel (e.g., to reach a breakpoint more quickly).

• verbose name: Serial If Debugger
• default value: True
• port type: BoolPort
• value type: bool (can be None)

initialize_once

Calibrate the model only once. If set to False, then this node will recalibrate itself whenever a non-streaming data chunk is received that has both training labels and associated training instances.

• verbose name: Calibrate Only Once
• default value: True
• port type: BoolPort
• value type: bool (can be None)

dont_reset_model

Do not reset the model when the preceding graph is changed. Normally, when certain parameters of preceding nodes are being changed, the model will be reset. If this is enabled, the model will persist, but there is a chance that the model is incompatible when input data format to this node has changed.

• verbose name: Do Not Reset Model
• default value: False
• port type: BoolPort
• value type: bool (can be None)

verbosity

Verbosity level for diagnostics.

• verbose name: Verbosity
• default value: 2
• port type: IntPort
• value type: int (can be None)

set_breakpoint

Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

• verbose name: Set Breakpoint (Debug Only)
• default value: False
• port type: BoolPort
• value type: bool (can be None)

metadata

User-definable meta-data associated with the node. Usually reserved for technical purposes.

• verbose name: Metadata
• default value: {}
• port type: DictPort
• value type: dict (can be None)