ProbabilityCalibration¶
Calibrate continuous outputs of a method to be valid probabilities.
This node acts as a "wrapper" for a machine learning pipeline that some sort of continuous output that has some monotonic relationship with the probability of a class. The node will then learn a one-dimensional mapping from the continuous output to a probability, and the calibration node then acts as a drop-in replacement for the original method. The node is used by wiring a pipeline into the "method" input port, where the pipeline is a subgraph that usually starts with a Placeholder with slotname set to "data"; see also the documentation tooltip for the method signature parameter. For technical reasons, probability calibration should not be performed on the same data that a classifier was trained on, since the classifier will generate untypically high confidence scores on its own training data that will not be representative of the classifier's outputs on new data. Therefore, the calibration node generally uses a cross-validation approach to train the underlying classifier on different data than what is used to fit the calibration mapping (see also cross-validation node for full details on cross-validation is generally done). Since this naturally results in k calibrated classifiers (for k folds), a straightfoward way to combine these classifiers is to average their outputs, which is the default behavior of the node (ensemble set to True). Alternatively, to reduce computational cost at prediction time, one can disable the ensemble option, which will then use a single classifier that is trained on all the data but still uses the calibration mapping trained via the cross-validation, but note that using a classifier trained no more data can introduce some bias (e.g., in test-set confidence) into the predictions and also does not enjoy the robustness conferred by the ensemble approach. Like all machine learning methods, this method needs to be calibrated ("trained") before it can make any predictions on data. For this, the method requires training instances and associated training labels. The typical way to get such labels associated with time-series data is to make sure that a marker stream is included in the data, which is usually imported together with the data using one of the Import nodes, or received over the network alongside with the data, e.g., using the LSL Input node (with a non-empty marker query). These markers are then annotated with target labels using the Assign Targets node. To generate instances of training data for each of the training markers, one usually uses the Segmentation node to extract segments from the continuous time series around each marker. Since this machine learning method is not capable of being trained incrementally on streaming data, the method requires a data packet that contains the entire training data; this training data packet can either be accumulated online and then released in one shot using the Accumulate Calibration Data node, or it can be imported from a separate calibration recording and then spliced into the processing pipeline using the Inject Calibration Data, where it passes through the same nodes as the regular data until it reaches the machine learning node, where it is used for calibration. Once this node is calibrated, the trainable state of this node can be saved to a model file and later loaded for continued use. More Info... Version 0.5.0
Ports/Properties¶
data¶
Data to process.
- verbose name: Data
- default value: None
- port type: DataPort
- value type: Packet (can be None)
- data direction: INOUT
method¶
Method to calibrate.
- verbose name: Method
- default value: None
- port type: GraphPort
- value type: Graph
method__signature¶
Argument names of the pipeline being calibrated. Your pipeline is a subgraph that must contain at least one Placeholder node whose slotname must match the argument name listed here. The placeholder then acts as the entry point for any data that is passed into the pipeline when it is invoked by the calibration node. Your pipeline's final node (which typically produces the predictions) is then wired to the calibration node's "method" input port. In graphical UIs, this edge will be displayed in dotted style to indicate that this is not normal forward data flow, but that a subgraph (your pipeline) runs under the control of the calibration node. In summary, your pipeline starts with a Placeholder that is followed by some processing nodes (in the simplest case just a single machine-learning node, such as Linear Discriminant Analysis). The final node of your pipeline is the one whose outputs are taken to be the pipeline's predictions, and this node is wired into the "method" input of the Cross-validation. Any "loose ends" downstream of your placeholder are also considered to be part of the pipeline but do not contribute to the result (they may be used for other purposes, such as printing progress information). Your pipeline may optionally have a second placeholder, which should by convention have slotname set to is_training, and then is_training must be listed as the second argument here. This second placeholder is used to indicate whether your pipeline is currently being called on training data or test data. Regardless of whether you expose this parameter or not. The calibration node will execute your pipeline like a cross-validation, meaning that, for each fold in the cross-validation, your pipeline graph is instantiated from its default (uninitialized) state, and is then called with the training set of that fold. Then, the same graph is called again, but this time with the test set of that fold; it is then up to any adaptive nodes in your pipeline (e.g., machine learning nodes) to adapt themselves on the first call and to make predictions (usually without adapting again) on the second call. The pipeline is discarded after each fold and a new pipeline graph is instantiated (to avoid any unintended train/test leakage).
- verbose name: Method [Signature]
- default value: (data)
- port type: Port
- value type: object (can be None)
mapping¶
Type of output mapping. The default 'sigmoid' mapping is a logistic function that maps scores scores to probabilities (logistic regression). The alternative 'isotonic' mapping is a non-parametric method that fits a piecewise constant, strictly increasing function to the data. The latter requires a fairly large number of data points to work well (e.g., >1000), but is more flexible and can capture more complex relationships between scores and true probabilities.
- verbose name: Probability Mapping
- default value: sigmoid
- port type: EnumPort
- value type: str (can be None)
ensemble¶
Use an ensemble of classifiers for prediction. If this is set, the method will train one classifier per fold, calibrate it on the respective test fold, and at prediction time this will average the predictions of all the classifiers. This is the recommended setting as it can result in somewhat better calibration (e.g. lower bias) and robustness, but is also more costly at prediction time. If disabled, a single classifier is used that is trained on all the data, a single calibration mapping trained via the cross-validation.
- verbose name: Use Ensemble
- default value: True
- port type: BoolPort
- value type: bool (can be None)
cond_field¶
The name of the instance data field that contains the conditions to be discriminated. This parameter will be ignored if the packet has previously been processed by a DescribeStatisticalDesign node.
- verbose name: Cond Field
- default value: TargetValue
- port type: StringPort
- value type: str (can be None)
enabled¶
Whether to enable probability claibration. If disabled, the predictions of the underlying model will be passed through unmodified.
- verbose name: Enabled
- default value: True
- port type: BoolPort
- value type: bool (can be None)
num_folds¶
Number of cross-validation folds for parameter search. Cross-validation proceeds by splitting up the data into this many blocks of trials, and then tests the method on each block. For each fold, the method is re-trained on all the other blocks, excluding the test block (therefore, the total running time is proportional to the number of folds). This is not a randomized cross-validation, but a blockwise cross-validation, which is usually the correct choice if the data stem from a time series. If there are few trials in the data, one can use a higher number here (e.g., 10) to ensure that more data is available for training. This can also be set to 0 if no cross-validation is desired, in which case the method and calibration mapping are trained on all the data, or to 1 to use a (perhaps costly) leave-one-out cross-validation.
- verbose name: Number Of Cross-Validation Folds
- default value: 5
- port type: IntPort
- value type: int (can be None)
cv_group_field¶
Optionally a field indicating the group from which each trial is sourced. If given, then data will be split such that test sets contain unseen groups. Examples groups are SubjectID, SessionID, etc.
- verbose name: Grouping Field (Cross-Validation)
- default value:
- port type: StringPort
- value type: str (can be None)
cv_stratified¶
Optionally perform stratified cross-validation. This means that all the folds have the same relative percentage of trials in each class.
- verbose name: Stratified Cross-Validation
- default value: True
- port type: BoolPort
- value type: bool (can be None)
initialize_once¶
Calibrate the model only once. If set to False, then this node will recalibrate itself whenever a non-streaming data chunk is received that has both training labels and associated training instances.
- verbose name: Calibrate Only Once
- default value: True
- port type: BoolPort
- value type: bool (can be None)
dont_reset_model¶
Do not reset the model when the preceding graph is changed. Normally, when certain parameters of preceding nodes are being changed, the model will be reset. If this is enabled, the model will persist, but there is a chance that the model is incompatible when input data format to this node has changed.
- verbose name: Do Not Reset Model
- default value: False
- port type: BoolPort
- value type: bool (can be None)
set_breakpoint¶
Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.
- verbose name: Set Breakpoint (Debug Only)
- default value: False
- port type: BoolPort
- value type: bool (can be None)
metadata¶
User-definable meta-data associated with the node. Usually reserved for technical purposes.
- verbose name: Metadata
- default value: {}
- port type: DictPort
- value type: dict (can be None)