← machine_learning package

ClassifierThresholdTuning

Tune the classification threshold of a predictive model so that the resulting class labels maximise a user-defined performance metric.

This can be used to, for example, balance sensitivity and specificity of a binary classifier as needed. The node performs an internal cross-validation, evaluates a series of candidate thresholds and selects the one that gives the highest score. It can be used with any pipeline that outputs either class probabilities or decision scores. Note that this node will not produce its outputs in the form of probabilities but rather outputs the hard class labels. More Info... Version 0.1.0

Ports/Properties

data

Data to process.

• verbose name: Data
• default value: None
• port type: DataPort
• value type: Packet (can be None)
• data direction: INOUT

method

Base classifier whose decision threshold is to be tuned.

• verbose name: Method
• default value: None
• port type: GraphPort
• value type: Graph

method__signature

Argument names of the pipeline being tuned. The wired-in graph must start with a Placeholder (slotname usually data) and end with a node producing per-sample decision scores or probabilities. The pipeline is instantiated afresh inside each cross-validation split to avoid train / test leakage.

• verbose name: Method [Signature]
• default value: (data)
• port type: Port
• value type: object (can be None)

preds_group_field

Method groups predictions by the specified field. This is rarely used, but can be employed if the method used emits predictions not per input trial but per each member of a group (e.g. a session or subject).

• verbose name: Grouping Field (Predictions)
• default value:
• port type: StringPort
• value type: str (can be None)

enabled

Whether to enable this node. If this is unset, the node will have essentially no effect.

• verbose name: Enabled
• default value: True
• port type: BoolPort
• value type: bool (can be None)

scoring

Performance metric that the node will maximise when choosing the threshold (e.g . balanced_accuracy, f1, roc_auc). The choice of metric determines how the trade-off between different types of classification errors is handled when selecting the optimal decision threshold from the classifier's continuous output scores (or probabilities). Available options: - 'accuracy': Standard accuracy. Maximizes the overall percentage of correct predictions. Can be misleading on imbalanced datasets, as it might favor thresholds that simply predict the majority class frequently. - 'balanced_accuracy': Accuracy adjusted for class imbalance. Averages the recall (sensitivity) for each class. Favors thresholds that perform equally well across all classes, regardless of their frequency. - 'top_k_accuracy': Checks if the true label is among the top k highest-scored predictions. Less common for simple threshold tuning which usually relies on a single score per instance. - 'average_precision': Summarizes the precision-recall curve, focusing on the positive class. Favors thresholds achieving high precision (few false positives) across various recall levels. Good for rare positive classes. - 'f1': Harmonic mean of precision and recall for the positive class. Favors thresholds that balance finding positive samples (recall) and the accuracy of positive predictions (precision). - 'f1_macro': Unweighted average of the F1 score across all classes. Favors thresholds balancing precision/recall equally for all classes. - 'f1_weighted': Average F1 score across classes, weighted by class frequency. Favors thresholds balancing precision/recall, giving more weight to performance on common classes. - 'precision': Fraction of predicted positives that are actually positive (TP / (TP + FP)). Favors higher thresholds, making the classifier more conservative about predicting positive to minimize false alarms. - 'recall': Fraction of actual positives correctly identified (TP / (TP + FN)). Also known as Sensitivity or True Positive Rate. Favors lower thresholds to minimize missed positive cases (false negatives). - 'roc_auc': Area Under the Receiver Operating Characteristic curve (plots Recall vs. False Positive Rate). Measures overall discrimination ability across all thresholds. When used for selection, it favors a threshold corresponding to a good balance between high recall and low false positive rate. - 'roc_auc_ovr': For multi-class, computes AUC for each class vs. the rest, then averages (unweighted). Favors thresholds balancing TPR/FPR well on average for each class. - 'roc_auc_ovo': For multi-class, computes AUC for each pair of classes, then averages (unweighted). Favors thresholds discriminating well between pairs of classes. - 'roc_auc_ovr_weighted': Like 'roc_auc_ovr', but weighted by class frequency. - 'roc_auc_ovo_weighted': Like 'roc_auc_ovo', but weighted by class frequency. - 'gmean_sens_spec': Geometric mean of Sensitivity (Recall) and Specificity (True Negative Rate). Favors thresholds that balance the correct identification of both positive and negative classes, useful for imbalanced data.

• verbose name: Scoring
• default value: balanced_accuracy
• port type: EnumPort
• value type: str (can be None)

thresholds

Either the number of candidate thresholds to test or an explicit list of thresholds. If an integer is given the node generates this many equally spaced thresholds covering the observed score / probability range.

• verbose name: Threshold Grid
• default value: 100
• port type: Port
• value type: object (can be None)

refit

Re-train the underlying model on the full data after the optimal threshold has been selected.

• verbose name: Refit
• default value: True
• port type: BoolPort
• value type: bool (can be None)

num_folds

Number of cross-validation folds. Cross-validation proceeds by splitting the data into num_folds blocks, iteratively holding out each block as validation data while training on the remaining blocks. The total runtime is therefore proportional to this value. • Set to 0 to disable CV and tune on the entire dataset (not recommended – prone to over-fitting). Set to 1 for leave-one-out CV (can be slow).

• verbose name: Number Of Cv Folds
• default value: 5
• port type: IntPort
• value type: int (can be None)

cv_group_field

Optional grouping field to ensure that samples from the same group do not appear in both train and validation sets (e.g . SubjectID, SessionID).

• verbose name: Grouping Field (Cv)
• default value:
• port type: StringPort
• value type: str (can be None)

cv_stratified

Perform stratified splits so that the class proportions are similar across folds. Recommended when classes are imbalanced.

• verbose name: Stratified Cv
• default value: True
• port type: BoolPort
• value type: bool (can be None)

cv_randomized

Use randomized cross-validation.

• verbose name: Randomized Cv
• default value: False
• port type: BoolPort
• value type: bool (can be None)

n_jobs

Number of processor cores to use during the tuning process (-1 means all available cores).

• verbose name: Parallel Jobs
• default value: 1
• port type: IntPort
• value type: int (can be None)

random_seed

Random seed controlling the CV shuffling and threshold sampling.

• verbose name: Random Seed
• default value: 12345
• port type: IntPort
• value type: int (can be None)

store_cv_results

If True the node records every tested threshold and its corresponding score. Useful for diagnostics but increases memory usage.

• verbose name: Store Cv Results
• default value: False
• port type: BoolPort
• value type: bool (can be None)

cond_field

Name of the instance-level data field that contains the class labels. Ignored if a DescribeStatisticalDesign node already processed the packet.

• verbose name: Cond Field
• default value: TargetValue
• port type: StringPort
• value type: str (can be None)

out_format

Output format of the predictions. If set to 'classes', the node will output a single feature that has the class index of the predicted class for each instance. If set to 'pseudo-probabilities', the node will output two features that contain the pseudo-probabilities of the predicted class and the other class; these will always be 0 or 1, depending on the predicted class. This can be useful for inserting this node into an existing pipeline that expects this format.

• verbose name: Output Format
• default value: classes
• port type: EnumPort
• value type: str (can be None)

initialize_once

If False the node will re-tune itself whenever a new non-streaming data chunk with targets arrives.

• verbose name: Calibrate Only Once
• default value: True
• port type: BoolPort
• value type: bool (can be None)

dont_reset_model

If True the model persists even if upstream parameters change. Use with caution – the model might become incompatible with the new data format.

• verbose name: Do Not Reset Model
• default value: False
• port type: BoolPort
• value type: bool (can be None)

set_breakpoint

Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

• verbose name: Set Breakpoint (Debug Only)
• default value: False
• port type: BoolPort
• value type: bool (can be None)

metadata

User-definable meta-data associated with the node. Usually reserved for technical purposes.

• verbose name: Metadata
• default value: {}
• port type: DictPort
• value type: dict (can be None)