← machine_learning package

Crossvalidation

Perform a cross-validation and return a loss measure that quantifies how well the pipeline performed when generalizing to previously unseen data.

This node receives an evaluation data set (packet , or list of packets for multi-session data) and a reference to a processing pipeline. The pipeline is typically a chain of nodes that starts with a Placeholder node (whose slotname is set to "data"), and whose final node's "this" output is wired into the cross-validation's "method" input. This means that the the pipeline is a graph with one input (the data) and it's passed to the cross-validation node (you can learn more about this way of passing graphs in NeuroPype's docs on graph-accepting nodes). The pipeline may also accept a second placeholder, which is expected to be a boolean representing whether the pipeline is used for training or testing (i.e., an is_training flag). While used with a cross-validation, the pipeline will "experience" being called more than once -- the first time, it receives training data (corresponding to the training portion of a fold), and from then on it will receive only test data (corresponding to the test portion of a fold). The pipeline is expected to adapt itself only on the first call (this is accomplished by ensuring the initialize_once flag is set in all adaptive nodes, which is the default for all machine-learning nodes, but it is not necessarily the default for all otherwise adaptive nodes that may occur in the pipeline, such as statistics (e.g., Standardization) or adaptive feature extraction, so you may have to set this flag there. The reason the pipeline only "experiences" training data once is that the pipeline will be discarded after each fold is complete, and a fresh copy of the untrained pipeline is used with the next fold (this is to ensure that there can be no train-test leakage). The pipeline may generally output results in one of three permitted forms: a) the predictions on the input data it received in some format (typically a Packet, possibly an array), b) a two-element list where the first entry is the aforementioned predictions, and the second entry is a data structure representing a model (e.g., this can be the "model" output of the ML node, if any, or a combination of model outputs of the ML node and any preceding adaptive nodes such as Standardization) and c) the pipeline may also return a new graph that represents the trained pipeline for subsequent use with test data. Optionally, you can also specify a custom scoring function to the Crossvalidation node, the simplest of which is a Placeholder named pred followed by MeasureLoss, wired into the "scoring" port. The cross-validation will partition the dataset into non-overlapping training and test subsets, and repeatedly invoke the pipeline first on a training subset, then on a corresponding test subset to obtain out-of-sample predictions. The predictions are then compared to the true labels by the scoring function, and one or more specified loss metrics are computed, and returned via the "loss" output. The loss is either a simple float, or can also be a packet with a statistic axis that reports a number of stats besides the mean value, depending on the selected loss_format. The node supports the majority of standard cross-validation schemes, and defaults to a sane 5-fold blockwise (non-randomized) CV, which is appropriate for time-series data. You can also specify a grouping structure (which is implicit if you are passing a list of packets) to perform grouped CV (where whole held-out groups of trials appear in the test set). If your pipeline includes adaptive signal processing (e.g., whitening, PCA, ICA, etc), you will need to pass continuous data with event markers into the cross-validation node, and include those adaptive steps in your piepline. The node can run computations for multiple folds in parallel, if you specify num_procs as either None (default all cores) or some number >1. You may also experiment with the number of threads that each process may use (num_threads_per_proc), since numpy can easily cause excessive thread churn (100% utilization). If you have multiple GPUs and your pipeline uses one or more GPU backends (e.g., jax or torch), then the node can spread the folds out over the GPUs, depending on num_procs_per_gpu and compute_backends. See the "return trained model" flag for how to obtain a model trained on the whole data from the cross-validation. More Info... Version 2.2.1

Ports/Properties

data

Data for evaluation.

• verbose name: Data
• default value: None
• port type: DataPort
• value type: object (can be None)
• data direction: IN

method

Pipeline to evaluate.

• verbose name: Method
• default value: None
• port type: GraphPort
• value type: Graph

method__signature

Argument names of the pipeline being cross-validated. Your pipeline is a subgraph that must contain at least one Placeholder node whose slotname must match the argument name listed here. The placeholder then acts as the entry point for any data that is passed into the pipeline when it is invoked by the cross-validation. Your pipeline's final node (which typically produces the predictions) is then wired to the cross-validation's "method" input port. In graphical UIs, this edge will be displayed in dotted mode to indicate that this is not normal forward data flow, but that a subgraph (your pipeline) is being passed to the cross-validation node as a whole, which will then repeatedly invoke your pipeline graph. In summary, your pipeline starts with a Placeholder that is followed by some processing nodes (in the simplest case just a single machine-learning node, such as Linear Discriminant Analysis). The final node of your pipeline is the one whose outputs are taken to be the pipeline's predictions, and this node is wired into the "method" input of the Cross-validation. Any "loose ends" downstream of your placeholder are also considered to be part of the pipeline but do not contribute to the result (they may be used for other purposes, such as printing progress information). Your pipeline may optionally have a second placeholder, which should by convention have slotname set to is_training, and then is_training must be listed as the second argument here. This second placeholder is used to indicate whether your pipeline is currently being called on training data or test data. Regardless of whether you expose this parameter or not, the way your pipeline is executed by the cross-validation is as follows. For each fold in the cross-validation, your pipeline graph is instantiated from its default (uninitialized) state, and is then called with the training set of that fold. Then, the same graph is called again, but this time with the test set of that fold; it is then up to any adaptive nodes in your pipeline (e.g., machine learning nodes) to adapt themselves on the first call and to make predictions (usually without adapting again) on the second call. The pipeline is discarded after each fold and a new pipeline graph is instantiated (to avoid any unintended train/test leakage). Your pipeline MAY also return as its final output a two-element list whose first element is the predictions and whose second element is a model data structure (by convention usually a dictionary). If the cross-validation node has the return_model option enabled, then this output will be used to populate the "model" output port of the cross-validation node, and the "trained" output port will remain empty (for this your pipeline will be run once on the entire dataset, with no second invocation since there is no more test data).

• verbose name: Method [Signature]
• default value: (data)
• port type: Port
• value type: object (can be None)

scoring

Optional custom scoring rule.

• verbose name: Scoring
• default value: None
• port type: GraphPort
• value type: Graph (can be None)

scoring__signature

List of arguments of an optional scoring rule graph. This is an optional graph that implements the scoring (evaluation) rule of the CrossValidation node. If no graph is provided, the default behavior is equivalent to having wired in a graph here that consists of a Placeholder node (with slotname set to pred) connected to a MeasureLoss node (configured in accordance with the loss_metrics option), which is then wired into the "scoring" input of the cross-validation node. To override this default, you instead wire in your own graph, starting with a placeholder and followed by some computation that outputs a performance measure. This graph may also have an additional placeholder named verbose, which the Crossvalidation will use to print a desired subset of results.

• verbose name: Scoring [Signature]
• default value: (pred)
• port type: Port
• value type: object (can be None)

loss

Aggregate loss measure (or multiple measures).

• verbose name: Loss
• default value: None
• port type: DataPort
• value type: AnyNumeric (can be None)
• data direction: OUT

loss_std

Loss standard deviation across CV splits. This is only set if loss_format is 'plain'; otherwise it will be reported in the axes of the loss output.

• verbose name: Loss Std
• default value: None
• port type: DataPort
• value type: AnyNumeric (can be None)
• data direction: OUT

losses

Per-fold loss measures across CV folds.

• verbose name: Losses
• default value: None
• port type: DataPort
• value type: Packet (can be None)
• data direction: OUT

predictions

Concatenated per-sample predictions across all CV folds.

• verbose name: Predictions
• default value: None
• port type: DataPort
• value type: Packet (can be None)
• data direction: OUT

trained

Trained pipeline.

• verbose name: Trained
• default value: None
• port type: GraphPort
• value type: Graph

loss_metrics

Loss metrics. This is used if no custom scoring function was provided. The following measures can be calculated both offline and online: MCR is mis-classification rate (aka error rate), MSE is mean-squared error, MAE is mean absolute error, Sign is the sign error, Bias is the prediction bias. The following measures are currently only available during offline computations: SMAE is the standardized mean absolute error, SMSE is the standardized mean-squared error, max is the maximum error, RMS is the root mean squared error, MedSE is the median squared error, MedAE is the median absolute error, SMedAE is the standardized median absolute error, AUC is negative area under ROC curve, R2 is the R-squared loss, CrossEnt is the cross-entropy loss, ExpVar is the negative explained variance.

• verbose name: Output Metrics
• default value: ['MCR']
• port type: SubsetPort
• value type: list (can be None)

loss_format

Format of the loss output packet. If set to stats-axis, a statistics axis and a feature axis (one entry per measure) will be used. If set to 2-feature-axes, instead of the stats axis, a second feature axis will be created. If set to plain, loss have the mean loss and loss_std the standard deviation, and in case of legacy, they will be floats if only a single metric was selected.

• verbose name: Output Format
• default value: legacy
• port type: EnumPort
• value type: str (can be None)

folds

Number of cross-validation folds. If >1 this is k-fold CV. If set to 1, this is leave-one-out CV. If folds is between 0 and 1, this is p-holdout CV (p being the percentage to hold out). If set to 0, the training-set error is computed. If a negative integer, this is p-holdout CV (p being the number of samples to hold out, where all permutations are generated; one may optionally set repeats to 0 to hold out exactly the last |p| sessions once).

• verbose name: Cross-Validation Folds
• default value: 5
• port type: FloatPort
• value type: float (can be None)

randomized

Whether to perform randomized or blockwise cross-validation. Blockwise is preferred for data stemming from a time series.

• verbose name: Randomized
• default value: False
• port type: BoolPort
• value type: bool (can be None)

stratified

Use stratified CV. This means that all the folds have the same relative percentage of trials in each class.

• verbose name: Stratified
• default value: False
• port type: BoolPort
• value type: bool (can be None)

repeats

Number of repetitions. This is only useful in case of randomized CV.

• verbose name: Number Of Repetitions
• default value: 1
• port type: IntPort
• value type: int (can be None)

group_field

Optionally a field indicating the group from which each trial is sourced. If given, then data will be split such that test sets contain unseen groups. Examples groups are SubjectID, SessionID, etc.

• verbose name: Group Field
• default value:
• port type: StringPort
• value type: str (can be None)

cond_field

The name of the instance data field that contains the conditions (classes) to be discriminated. This parameter will be ignored if the input data packet has previously been processed by a BakeDesignMatrix node.

• verbose name: Condition Field
• default value: TargetValue
• port type: StringPort
• value type: str (can be None)

censor_labels

Whether to censor labels from the model at prediction time as a safeguard against data leakage. Note that this feature requires the input and output data structures of the model to be identical at prediction time, and the input data cannot have multiple sets of labels in different parts of the data structure. Setting this to False does not imply that data leakage will happen (it should not in any case); this simply provides an additional safeguard. One case where you will need to send this to False is if, during development, you want to be able to debug test-time predictions inside the model (i.e., by setting a breakpoint), so that you can compare the predictions with their respective labels. Note that this setting only supports already segmented data (i.e., trials with labels), not continuous data with segmentation happening inside the cross validation method.

• verbose name: Censor Labels
• default value: True
• port type: BoolPort
• value type: bool (can be None)

return_model

Whether to also compute a model trained on the entire dataset. This will do one additional fold where the training data is the whole dataset and there is no test data (this can be done in parallel to the other fold if the number of processes setting is set to high enough). The cross-validation node will then populate the "trained" output port with a graph representing the trained pipeline, which can be subsequently used with e.g., the "Call" node to invoke it on test data. Alternatively, if the pipeline was configured to return a two-element list whose second element is a model data structure, then the model returned in this fashion on that (training-only) fold will be used to populate the "model" output port of the cross-validation node instead. In this case, the trained output will remain empty.

• verbose name: Return Trained Model
• default value: False
• port type: BoolPort
• value type: bool (can be None)

trial_size

Assumed size of a trial. Formatted as in the Segmentation node's time_bounds argument, and given in seconds relative to the respective markers. This is used to be able to perform continuous-time selection around markers. Should be large enough to encompass the segment size used in the Segmentation step of the given method, if any. You can also make it longer to retain more continuous data around the ends of your training or test data, e.g., to make available to continuous-time filters etc. If this is left unspecified, it will be inferred from the pipeline's Segmentation node.

• verbose name: Segment Span
• default value: []
• port type: ListPort
• value type: list (can be None)

exclude_margin

Exclude any training trials that fall within this many seconds of test trials.

• verbose name: Exclude Margin
• default value: 5.0
• port type: FloatPort
• value type: float (can be None)

tight_selection

If True, the selected stretches of continuous data (if any) will be tightly enclosing. Otherwise, the selection will run to the edges of the dataset where possible. This only applies if continuous (non-segmented) data is passed into the cross-validation node.

• verbose name: Tight Selection
• default value: False
• port type: BoolPort
• value type: bool (can be None)

num_procs

Number of processes to use for parallel computation. If None, the global setting NUM_PROC, which defaults to the number of CPUs on the system, will be used.

• verbose name: Max Parallel Processes
• default value: 1
• port type: IntPort
• value type: int (can be None)

num_threads_per_proc

Number of threads to use for each process. This can be used to limit the number of threads by each process to mitigate potential churn.

• verbose name: Threads Per Process
• default value: 4
• port type: IntPort
• value type: int (can be None)

compute_backends

GPU compute backends that may be used by the pipeline. If you include GPU compute backends here, workloads using those backends will be farmed out across multiple GPUs (if available) when running cross-validation folds in parallel. The 'auto' mode will attempt to auto-detect any backend settings in the given pipeline's nodes, but note that this will only catch nodes where this is explicit in the node's properties, and GPU workloads missed in this fashion will run by default on GPU 0.

• verbose name: Compute Backends
• default value: ['auto']
• port type: SubsetPort
• value type: list (can be None)

num_procs_per_gpu

Number of processes to use per GPU. This is only relevant if you have GPU compute backends enabled. If your GPU(s) are under-utilized during cross-validation, you can increase this to run this many CV folds on each GPU.

• verbose name: Processes Per Gpu
• default value: 1
• port type: IntPort
• value type: int (can be None)

multiprocess_backend

Backend to use for farming out computation across multiple (CPU) processes. Multiprocessing is the simple Python default, which is not a bad start. Nestable is a version of multiprocessing that allows your pipeline to itself use parallel computation. Loky is a fast and fairly stable backend, but it does not support nested parallelism and has different limitations than multiprocessing. It can be helpful to try either if you are running into an issue trying to run something in parallel. Serial means to not run things in parallel but instead in series (even if num_procs is >1), which can help with debugging. Threading uses Python threads in the same process, but this is not recommended for most use cases due to what is known as GIL contention.

• verbose name: Multiprocess Backend
• default value: loky
• port type: EnumPort
• value type: str (can be None)

serial_if_debugger

If True, then if the Python debugger is detected, the node will run in serial mode, even if multiprocess_backend is set to something else. This is useful for debugging, since the debugger does not work well with parallel processes. This can be disabled if certain steps should nevertheless run in parallel (e.g., to reach a breakpoint more quickly).

• verbose name: Serial If Debugger
• default value: True
• port type: BoolPort
• value type: bool (can be None)

compute_order

order in which to compute the cross-validation folds. This has no impact on the results (aside from potential random seed variation). In the default order, the held-out test set will sweep through the data in order, and at the end the full model is trained on the whole data if requested. Reversed runs this process in reverse order, for example to debug issues that may occur near the end of a training run (such as out-of-memory errors with the full model).

• verbose name: Compute Order
• default value: default
• port type: EnumPort
• value type: str (can be None)

clear_memory_per_fold

If enabled, then the memory will be cleared after each fold. This is useful if you are running out of memory during cross-validation, but it will slow down the computation somewhat. The auto mode will attempt to auto-determine a sensible setting depending on the model used. The aggressive mode will force each task to run in a separate subprocess, and will reclaim the subprocess after the fold, which guarantees that memory will be reclaimed; this should be considered a last resort, since it will slow down the computation.

• verbose name: Clear Memory Per Fold
• default value: auto
• port type: EnumPort
• value type: str (can be None)

run_name

Name of the run, for logging purposes.

• verbose name: Run Name
• default value: Crossvalidation
• port type: StringPort
• value type: str (can be None)

ignore_bad_packets

If input is given as a list of packets, ignore packets that are bad (e.g ., due to missing labels). This will print a warning; otherwise an exception will be raised.

• verbose name: Ignore Bad Packets
• default value: False
• port type: BoolPort
• value type: bool (can be None)

trial_padding

Padding around the specified trial length. This is to avoid cutting the signal time axis too short near the edge of segments. Note that for very low-sampling rate data, e.g., NIRS, you may need a larger margin.

• verbose name: Trial Padding
• default value: 0.3
• port type: FloatPort
• value type: float (can be None)

random_seed

Random seed (int or None). Different values will give somewhat different outcomes.

• verbose name: Random Seed
• default value: 12345
• port type: IntPort
• value type: int (can be None)

with_fold_num

Whether to include FoldNum and RepeatNum axis fields in the outputs where applicable. FoldNum is a zero-based index of the fold across all repeats, while RepeatNum is a zero-based index of the repeat (for the extra train-only split when return_model is True, RepeatNum is -1).

• verbose name: With Fold Num
• default value: True
• port type: BoolPort
• value type: bool (can be None)

verbose

Whether to print verbose output.

• verbose name: Verbose
• default value: True
• port type: BoolPort
• value type: bool (can be None)

loss_metric

Legacy loss metric. This has been superseded by loss_metrics, which can take multiple metrics.

• verbose name: Loss Metric (Legacy)
• default value: MCR
• port type: EnumPort
• value type: str (can be None)

set_breakpoint

Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

• verbose name: Set Breakpoint (Debug Only)
• default value: False
• port type: BoolPort
• value type: bool (can be None)

metadata

User-definable meta-data associated with the node. Usually reserved for technical purposes.

• verbose name: Metadata
• default value: {}
• port type: DictPort
• value type: dict (can be None)