PerformanceMetric¶
Calculate one of a set of performance metrics between predictions and target values.
Performance metrics return a kind of performance score for a given model, and a fairly large number of metrics have been proposed across various statistical and machine-learning areas. This node supports a number of the most common metrics for classification and regression problems; see the documentation for the metric port for details on specific metrics. Beware that for classification metrics including accuracy, precision, recall, and f1, the predictions should be the class labels (not the class probabilities). The node will attempt to detect such cases and convert the predictions to classes if they are not already, but this may not be reliable in all cases, especially when there are few samples. Version 0.5.0
Ports/Properties¶
preds¶
Predictions.
- verbose name: Preds
- default value: None
- port type: DataPort
- value type: AnyNumeric (can be None)
- data direction: IN
targs¶
Target values.
- verbose name: Targs
- default value: None
- port type: DataPort
- value type: AnyNumeric (can be None)
- data direction: IN
output¶
Output.
- verbose name: Output
- default value: None
- port type: DataPort
- value type: AnyNumeric (can be None)
- data direction: OUT
perf_metric¶
Performance metric to evaluate. Note that, in constrast to loss functions (nodes ending in Loss), higher values are better for performance metrics, an in the case of metrics that are technically loss functions (e.g., the mean squared error), the negative of the underlying loss is returned. Generally some of the available metrics are suitable for classification problems (accuracy, precision/recall/f1, and roc metrics) while others are for use in regression problems (r2, explained_variance, max_error and the neg_error metrics). These metrics are all available in the scikit-learn package, and the documentation for them can be found on the web. In quick summary, balanced_accuracy is a good choice for generic classification problems on balanced or unbalanced data, precision/recall are sometimes asked for in detection problems of a relatively rare class (e.g., clinical diagnosis), and the roc metrics are useful both for measuring performance on unbalanced data (although on extremely unbalanced data, the measures can break down), and for measuring the performance in detection problems across a range of decision thresholds, e.g., when the relative importance of type-1 and type-2 errors is not fixed at the time the ML work is done. The F1 scores are a specific blend of precision and recall across classes that can be used as a general-purpose (albeit perhaps not very interpretable) performance measure on some signal detection tasks. For regression, the r2 score is a fair choice for generic signal regression and is relative to the scale of the target variable; explained variance is similar but specifically does not score systematic offset (i.e., bias) in the predictions (usually a deficiency of the metric but sometimes useful, e.g. while working on a model that will still receive a bias correction later). The neg_error metrics are the most common tools in regression problems; of those, the mean_absolute and median_absolute variants are perhaps the most interpretable as the errors are in the same units as the target variable (e.g., meters), and the latter is robust if there are outliers in either the targets or predictions. The mean_squared error is the default choice under a Gaussian noise assumption, and the mean_squared_log error is useful if the data are log-normally distributed (e.g., data from exponential growth processes). Somewhat similarly, the neg_mean_absolute_percentage_error is also adequate for measuring performance on data where the target variable ranges across several orders of magnitude in scale, and the error is relative to each individual target value (but note that the score is not measured in percent but is a fraction). Note that, unlike the loss measures, most of the metrics are not differentiable, so they cannot be directly optimized for using e.g., gradient descent. However, model hyper-parameters can be optimized with respect to these metrics when using, e.g., the Parameter Optimization node (although at significant computational cost).
- verbose name: Perf Metric
- default value: balanced_accuracy
- port type: ComboPort
- value type: str (can be None)
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
- verbose name: Set Breakpoint (Debug Only)
- default value: False
- port type: BoolPort
- value type: bool (can be None)
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.
- verbose name: Metadata
- default value: {}
- port type: DictPort
- value type: dict (can be None)
axis_pairing¶
How to pair axes of the preds and targs operands. In 'positional' mode, axes are paired by their position according to a right alignment, that is, the last axis of the first operand is paired with the last axis of the second operand, and so on, while any missing axes behave as if they were unnamed axes of length 1 (this is the same way plain n-dimensional arrays pair in Python/numpy). In 'matched' mode, axes are paired by their type and optionally label, where the axis order of the first operand is preserved in the output, optionally with additional axes that only occur in the second operand prepended on the left. The other operand then has its axes reordered to match. All axis classes are treated as distinct, except for the plain axis, which is treated as a wildcard axis that can pair with any other axis. The 'default' value resolves to a value that may be overridden in special contexts (e.g., an ambient Inference node) and otherwise resolves to the setting of the configuration variable default_axis_pairing, which is set to 'positional' in 2024.x. See also the 'label_handling' property for how labels are treated in this mode. Note that axis pairing can be subtle, and it is recommended to not blindly trust that the default behavior is always what the user intended.
- verbose name: Axis Pairing
- default value: default
- port type: EnumPort
- value type: str (can be None)
label_pairing¶
How to treat axis labels when pairing axes in 'matched' mode. In 'always' mode, labels are always considered significant, and axes with different labels are always considered distinct, which means that, if the two operands each have an axis of same type but with different labels, each operand will have a singleton axis inserted to pair with the respective axis in the other operand. In 'ignore' mode, labels are entirely ignored when pairing axes; this means that, if multiple axes of the same type occur in one or more operands, the last space axis in the first operand is paired with the last space axis in the second operand, etc. as in positional mode. In 'auto' mode, labels are only considered significant if they are necessary for distinguishing two or more axes of the same type in any of the operands, or if they occur on a plain axis.
- verbose name: Label Pairing
- default value: auto
- port type: EnumPort
- value type: str (can be None)
lists_as_arrays¶
Whether to treat lists as numeric arrays, as opposed to a recursive data structure. This is equivalent to converting any list operand(s) to arrays before passing them to the node. Broadly, enabling this allows for more efficient processing of large lists of either numbers, or lists of many smaller arrays, with some limitations and caveats, as follows. If one operand was a list and the other an array, an array will be returned, and if all were lists, a list will be returned (if the operand lists themselves contained arrays, the result is still a "pure" list of potentially nested lists of numbers). A limitation is that input lists may contain ONLY numbers or arrays of the same shape (e..g., no lists of blocks or packets, dictionaries, or irregular array shapes in this mode). In contrast, when the option is disabled, then if one operand is a list and the other is not a list (e.g., a number, array, block, packet, etc), then each element of the list is separately undergoing the operation while the other operand is held fixed. This has the consequence that a) the result will generally be a list if at least one operand is a list and b) the result will be mathematically different when one is a list and the other is an array, because the array will be separately combined with each list element. Note the default for this option was enabled by default in NeuroPype 2023. The option can also be set globally or in a context using the WithOptions node, for example for pipeline-wide backwards compatibility.
- verbose name: Lists As Arrays
- default value: False
- port type: BoolPort
- value type: bool (can be None)