← machine_learning package

MeasureLoss

Measure discrepancy between some predicted labels and true labels.

This is a fully-featured statistics node that is typically used to measure performance of a model, which can compute a number of error metrics, aggregate them over successive ticks, and generally expects target label information to be encoded in fields of the instance axis. For pure mathematical loss functions (which can also be directly optimized for), see the other Loss nodes in this category. This node accepts as input some data that has both estimates or predictions (e.g., from a machine learning node), as well as ground-truth labels (desired outputs). Using these inputs, the node computes a loss function that measures the disagreement between the predictions and the labels. The node supports a large variety of losses that can be used in different circumstances (based on different assumptions about the type of estimates or the severity of different kinds of errors), such as misclassification rate, mean squared error, or area under receiver-operator curve. Be sure to inform yourself about what is the most appropriate loss metric for your use case before trusting the numbers blindly, or e.g., accidentally relying on misleading results (e.g., inapplicable metrics) to make decisions. The node is capable of accumulating results across multiple trials or data chunks, for instance during real-time processing. It is also possible to do the same across multiple datasets in offline or batch processing, by setting the appropriate parameters in the node. This node passes the computed loss out through its loss port (so wire that one into the next node). This will contain either a statistics axis or a feature axis (depending on the output_format parameter) with the following six items: 'N Batches', 'N Trials', 'value', 'p_val', 'st_stat', 't_stat', where 'value' will be the selected loss metric. If more than one loss metric is selected, the data tensor is a matrix with a feature axis (or second feature axis depending on the output_format) will hold the values for each metric. See the documentation for the output_format parameter for more details. (Deprecation notice: ) More Info... Version 1.6.0

Ports/Properties

loss

Loss estimates.

• verbose name: Loss
• default value: None
• port type: DataPort
• value type: Packet (can be None)
• data direction: OUT

data

Data to process. Deprecation notice: The incoming data used to be passed through unchanged through this same port, but that behavior has changed and this port is set to IN only.

• verbose name: Data
• default value: None
• port type: DataPort
• value type: Packet (can be None)
• data direction: IN

loss_metrics

Loss metrics. The following measures can be calculated both offline and online: MCR is misclassification rate (aka error rate), MSE is mean-squared error, MAE is the mean absolute error, Sign is the sign error, Bias is the prediction bias. The following measures are currently only available during offline computations: SMAE is the standardized mean absolute error, SMSE is the standardized mean-squared error, max is the maximum error, RMSE is the root mean squared error, MedSE is the median squared error. MedAE is the median absolute error, SMedAE is the standardized median absolute error, AUC is negative area under ROC curve, R2 is the R-squared loss, CrossEnt is the cross-entropy loss, ExpVar is the negative explained variance, CM is the confusion matrix.

• verbose name: Loss Metrics
• default value: []
• port type: SubsetPort
• value type: list (can be None)

batching

Batch size for computing measures. If set to None (default), the trials are batched as they come in. If set to 0, all trials are put into a single batch. Other values will use batches of the given trial count. Batching is mostly useful if you are interested in the standard deviation of some measures, or if a measure cannot be computed reliably from few instances (e.g., in case of median, AUC, etc). Mutually exclusive with batch_axis.

• verbose name: Batch Size
• default value: None
• port type: IntPort
• value type: int (can be None)

batch_field

Instance axis field name to use for batching. If specified, instances are grouped by unique values of this field (e.g., 'BlockNumber', 'FoldNum'), and loss is computed separately for each group. Groups are ordered by first appearance in the data. Mutually exclusive with batching and batch_axis.

• verbose name: Batch Field
• default value: None
• port type: StringPort
• value type: str (can be None)

batch_axis

Axis over which to batch loss calculations. If specified, computes loss separately for each position along this axis, then reports mean and std across those positions. Mutually exclusive with batching and batch_field. Accepts axis shorthands like 'time', 'frequency', etc.

• verbose name: Batch Axis
• default value: None
• port type: ComboPort
• value type: object (can be None)

output_format

Format of the loss output packet. If set to stats-axis, the loss packet will contain a statistics axis with with six items ('N Batches', 'N Trials', 'value', 'p_val', 'st_stat', 't_stat') and a feature axis with one item per selected metric in loss_metrics (with the same name, i.e., 'MSE'). If set to 2-feature-axes, a feature axis with label statistics_types will be created in place of the statistics axis described above, with the same items, and a second feature axis with label explanatory_variables will be created with one item per computed measure. In legacy mode a single feature axis will be created with one item per computed measure plus a N Trials item, and no statistics.

• verbose name: Output Format
• default value: legacy
• port type: EnumPort
• value type: str (can be None)

accumulate_offline

Accumulate statistics across offline data packets, as well. Normally this node will only accumulate statistics on real-time (streaming) data. By enabling this, it will also accumulate across multiple non-streaming data sets, e.g., in batch processing.

• verbose name: Accumulate Offline
• default value: False
• port type: BoolPort
• value type: bool (can be None)

ignore_resets

Keep accumulating throughout state resets of the pipeline. For instance, after rewiring or changing files. This is most relevant in batch processing, where a pipeline iterates over multiple files. If you intend to aggregate statistics across multiple files, you need to enable this option.

• verbose name: Ignore Resets
• default value: False
• port type: BoolPort
• value type: bool (can be None)

cond_field

The name of the instance data field that contains the true target values. This parameter will be ignored if the packet has previously been processed by a DescribeStatisticalDesign node.

• verbose name: Cond Field
• default value: TargetValue
• port type: StringPort
• value type: str (can be None)

ttest_level

Loss value against which to test for significant differences in a t-test. This only applies if there were multiple batches.

• verbose name: Ttest Level
• default value: 0.5
• port type: FloatPort
• value type: float (can be None)

significance_level

Significance level for t-test.

• verbose name: Significance Level
• default value: 0.05
• port type: FloatPort
• value type: float (can be None)

log_results

Log results.

• verbose name: Log Results
• default value: True
• port type: BoolPort
• value type: bool (can be None)

metric_can_throw

Whether a calculation error in the metric can raise an exception vs return a NaN loss value.

• verbose name: Metric Can Throw
• default value: False
• port type: BoolPort
• value type: bool (can be None)

suppress_nans

Impute NaN loss values with neutral values. Depending on the loss, a NaN prediction may raise an exception; this option will impute a chance-level or otherwise neutral value.

• verbose name: Impute Nans
• default value: False
• port type: BoolPort
• value type: bool (can be None)

include_raw

Include raw per-batch metric values in the output, appended after the summary statistics. This is useful for paired comparisons across methods (e.g., paired t-tests). Only applies when output_format is 'stats-axis' or '2-feature-axes' and there are multiple batches.

• verbose name: Include Raw
• default value: False
• port type: BoolPort
• value type: bool (can be None)

output_for_stats_table

Output the data in such a way that it can be parsed by the ParseStatsTable node, for creating stats tables. This creates two feature axes instead of one. You can now use the output_format to get more format choices, and additional metrics.

• verbose name: Output For Stats Table
• default value: False
• port type: BoolPort
• value type: bool (can be None)

loss_metric

Legacy loss metric. Will become deprecated in favor of loss_metrics (allowing multiple metrics). Note that you cannot specify both loss_metric and loss_metrics.

• verbose name: Loss Metric (Legacy)
• default value:
• port type: EnumPort
• value type: str (can be None)

set_breakpoint

Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

• verbose name: Set Breakpoint (Debug Only)
• default value: False
• port type: BoolPort
• value type: bool (can be None)

metadata

User-definable meta-data associated with the node. Usually reserved for technical purposes.

• verbose name: Metadata
• default value: {}
• port type: DictPort
• value type: dict (can be None)