← machine_learning package

StochasticGradientDescentClassification

Classify data instances using models trained via stochastic gradient descent.

Stochastic gradient descent is an optimization technique that is very fast for very large amounts of training data, since it does not have to touch all the data in each update iteration. This node is rather flexible in that it can implement a variety of data terms, including logistic regression, support vector machines, certain robust classifiers, and some rarely-used variants. These can be combined with a variety of regularization terms to control the complexity of the solution, including shrinkage/ridge regularization (l2), sparsity ( l1), and elastic net (l1+l2). In this sense, this node can mimic the features of several other nodes in NeuroPype, with different tradeoffs in running time that depend on the number of data points and the number of features, where this node is mainly of interest for very large numbers of data points (i.e., trials). See also tooltips of the parameters for toggling these features for more information on the involved tradeoffs, as well as the documentation of the other nodes ( Logistic Regression, and Support Vector Classification). One unique feature of this node is the ability to implement a robust classifier that theoretically can perform better than most other methods if the data do indeed contain outliers. For this reason, removing artifacts from the raw data will make less of a difference when this method is used. Like the other methods, this node includes tuning parameters that are automatically tuned using cross-validation. If there are very few trials, or some extensive stretches of the data exhibit only one class, the procedure used to find the regularization parameters (cross-validation) can fail with an error that there were too few or no trials of a given class present. By default, this method will return not the most likely class label for each trial it is given, but instead the probabilies for each class (=category of labels), that the trial is actually of that class. Like all machine learning methods, this method needs to be calibrated ("trained") before it can make any predictions on data. For this, the method requires training instances and associated training labels. The typical way to get such labels associated with time-series data is to make sure that a marker stream is included in the data, which is usually imported together with the data using one of the Import nodes, or received over the network alongside with the data, e.g., using the LSL Input node (with a non-empty marker query). These markers are then annotated with target labels using the Assign Targets node. To generate instances of training data for each of the training markers, one usually uses the Segmentation node to extract segments from the continuous time series around each marker. Since this machine learning method is not capable of being trained incrementally on streaming data, the method requires a data packet that contains the entire training data; this training data packet can either be accumulated online and then released in one shot using the Accumulate Calibration Data node, or it can be imported from a separate calibration recording and then spliced into the processing pipeline using the Inject Calibration Data, where it passes through the same nodes as the regular data until it reaches the machine learning node, where it is used for calibration. Once this node is calibrated, the trainable state of this node can be saved to a model file and later loaded for continued use. More Info... Version 1.2.0

Ports/Properties

data

Data to process.

• verbose name: Data
• default value: None
• port type: DataPort
• value type: Packet (can be None)
• data direction: INOUT

loss

Loss function to use. This selects the data term, i.e., what assumptions are being imposed on the data. Hinge yields an SVM, log_loss yields logistic regression, modified_huber is another smooth loss that enables probability outputs and tolerance to outliers, squared_hinge is like hinge but is quadratically penalized. Perceptron is the loss used by the perceptron algorithm. Epsilon_insensitive is equivalent to support vector regression, and squared_epsilon_insensitive is a rarely used hybrid between linear and support vector regression. These regression techniques are here used to classify data instances, though.

• verbose name: Data Term
• default value: log_loss
• port type: EnumPort
• value type: str (can be None)

regularizer

Regularization term. Selecting l2 (default) is the most commonly used regularization, and is very effective in preventing over-fitting to random details in the training data, thus helping generalization to new data. The l1 method is applicable when only a small number of features in the data are relevant and most features are irrelevant, and this regularization is capable of identifying and selecting the relevant features (although if the optimal model is not sparse, then this will work less well than l2). The elasticnet version is a weighted combination of the l1 and l2, which addresses an issue with l1 where highly correlated features would receive rather arbitrary relative weights.

• verbose name: Regularization Term
• default value: l2
• port type: EnumPort
• value type: str (can be None)

alphas

Regularization strength. This is a list of candidate values, the best of which is found via an exhaustive search (i.e., each value is tested one by one, and therefore the total running time is proportional to the number of values listed here). The details of the parameter search can be controlled via the search metric and number of folds parameters. Larger values cause stronger regularization, that is, less risk of over-fitting. A value of 0 means no regularization, and there is no upper limit to how large the values that can be given here may be -- however, depending on the scale of the data and the number of trials, there is a cutoff beyond which all features are weighted by zero, and are thus unused. Often one covers a range between 0.1 and 10, and at times 0.01 to 100. Typically the values here are not linearly spaced, but follow an exponential progression ( e.g., 0.25, 0.5, 1, 2, 4, 8, ... etc). The default search range is intentionally coarse for quick running times; refine it to smaller steps to obtain potentially better solutions, but do not expect massive gains from refining.

• verbose name: Regularization Strength
• default value: [0.1, 0.5, 1.0, 5, 10.0]
• port type: ListPort
• value type: list (can be None)

l1_ratio

Tradeoff parameter between between l1 and l2 penalties when using the elasticnet regularization term. This parameter controls the balance between the l1 regularization term (as in LASSO) and the l2 regularization term (as in ridge regression). A value of 0 leads to exclusive use of l2, and a value of 1 leads to exlusively use of l1. This is a list of candidate values, the best of which is found via an exhaustive search (i.e., each value is tested one by one, and therefore the total running time is proportional to the number of values listed here). In fact, for each setting of this parameter, the entire list of possible values for the regularization strength is tested, so the running is also proportional to the number of those values. The details of the parameter search can be controlled via the search metric and number of folds parameters. By default this is not seached, but a reasonable choice would be [0.1, 0.5, 0.7, 0.9, 0.95, 0.99, 1].

• verbose name: Regularization Type Tradeoff Parameter
• default value: [0.5]
• port type: ListPort
• value type: list (can be None)

feature_scaling

Feature scaling to use. If set to auto, then scaling will default to robust if a robust loss is used, and otherwise std. If features are not already standardized beforehand, enabling this tends to ensure that features are treated equally by the regularization instead of it being scale-dependent.

• verbose name: Feature Scaling
• default value: none
• port type: EnumPort
• value type: str (can be None)

tolerance

Convergence tolerance. This is the desired errors tolerance or acceptable inaccuracy in the solution. Using larger values gives less accurate results, but will lead to faster compute times. Note that, for biosignal-driven machine learning systems, one often does not need very small tolerances.

• verbose name: Tolerance
• default value: 0.001
• port type: FloatPort
• value type: float (can be None)

max_iter

Maximum number of iterations. This is one of the stopping criteria to limit the compute time. The default is usually fine, and gains from increasing the number of iterations will be minimal (it can be worth experimenting with lower iteration numbers if the algorithm must finish in a fixed time budget, at a cost of potentially less accurate solutions).

• verbose name: Maximum Number Of Iterations
• default value: 1000
• port type: IntPort
• value type: int (can be None)

num_jobs

Number of parallel compute jobs. This value only affects the running time and not the results. Values between 1 and twice the number of CPU cores make sense to expedite computation, but may temporarily reduce the responsiveness of the machine. The value of -1 stands for all available CPU cores.

• verbose name: Number Of Parallel Jobs
• default value: 1
• port type: IntPort
• value type: int (can be None)

search_metric

Parameter search metric. When the regularization parameter is given as a list of values, then the method will run a cross-validation for each possible parameter value and use this metric to score how well the method performs in each case, in order to select the best parameter. While 'accuracy' is usually a good default, some other metrics can be useful under special circumstances, e.g., roc_auc for highly imbalanced ratios of trials from different classes.

• verbose name: Scoring Metric If Searching Parameters
• default value: accuracy
• port type: EnumPort
• value type: str (can be None)

num_folds

Number of cross-validation folds for parameter search. Cross-validation proceeds by splitting up the data into this many blocks of trials, and then tests the method on each block. For each fold, the method is re-trained on all the other blocks, excluding the test block (therefore, the total running time is proportional to the number of folds). This is not a randomized cross-validation, but a blockwise cross-validation, which is usually the correct choice if the data stem from a time series. If there are few trials in the data, one can use a higher number here (e.g., 10) to ensure that more data is available for training.

• verbose name: Number Of Cross-Validation Folds
• default value: 5
• port type: IntPort
• value type: int (can be None)

cv_group_field

Optionally a field indicating the group from which each trial is sourced. If given, then data will be split such that test sets contain unseen groups. Examples groups are SubjectID, SessionID, etc.

• verbose name: Grouping Field (Cross-Validation)
• default value:
• port type: StringPort
• value type: str (can be None)

cv_stratified

Optionally perform stratified cross-validation. This means that all the folds have the same relative percentage of trials in each class.

• verbose name: Stratified Cross-Validation
• default value: False
• port type: BoolPort
• value type: bool (can be None)

shuffle

Shuffle data after each epoch. This is useful if the data comes from a time series with highly correlated successive trials, and is worth experimenting with to reach faster convergence. The main reason why it is disabled by default is that traditional SGD does not enable this by default.

• verbose name: Shuffle Data
• default value: False
• port type: BoolPort
• value type: bool (can be None)

probabilistic

Use probabilistic outputs. If enabled, the node will output for each class the probability that a given trial is of that class; otherwise it will output the most likely class label.

• verbose name: Output Probabilities
• default value: True
• port type: BoolPort
• value type: bool (can be None)

warm_start

Start from previous solution. This allows for online updating of the model.

• verbose name: Warm Start
• default value: False
• port type: BoolPort
• value type: bool (can be None)

verbosity

Verbosity level. Higher numbers will produce more extensive diagnostic output.

• verbose name: Verbosity Level
• default value: 0
• port type: IntPort
• value type: int (can be None)

averaging

Use averaging. This is an alternative way of performing stochastic gradient descent, which can lead to faster convergence. Can also be an integer greater than 1; in this case, this is the number of samples seen after which averaging kicks in.

• verbose name: Averaging
• default value: False
• port type: Port
• value type: object (can be None)

include_bias

Include a bias term. If false, your features need to be centered, or include a dummy feature set to 1.

• verbose name: Include Bias Term
• default value: True
• port type: BoolPort
• value type: bool (can be None)

epsilon

Epsilon for epsilon-insensitive and Huber data terms. Note that this depends strongly on the scale of the data. Can be interpreted as the cutoff in data units beyond which data values are treated more robustly (i.e., as potential outliers).

• verbose name: Epsilon
• default value: 0.1
• port type: FloatPort
• value type: float (can be None)

learning_rate_schedule

Schedule for adapting the learning rate. This is a highly technical setting that allows to implement different forms of stochastic gradient descent. The options correspond to the following rules for computing the learning rate (step size) eta of the method dependent on the current iteration number t, where 'constant' means eta=eta0, 'optimal' yields the formula eta=1.0/(t+t0), and 'invscaling' corresponds to the formula eta=eta0/pow(t,power_t). The main reason to change this would be to replicate some other published implementation.

• verbose name: Learning Rate Schedule
• default value: invscaling
• port type: EnumPort
• value type: str (can be None)

eta0

Initial learning rate. This is the parameter eta0 in the learning rate schedule formulate.

• verbose name: Initial Learning Rate (Step Size)
• default value: 0.01
• port type: FloatPort
• value type: float (can be None)

power_t

Exponent in invscaling schedule. This is the exponent power_t in the invscaling learning rate schedule.

• verbose name: Exponential Falloff (Invscaling Only)
• default value: 0.25
• port type: FloatPort
• value type: float (can be None)

random_seed

Random seed. Different values may give slightly different outcomes.

• verbose name: Random Seed
• default value: 12345
• port type: IntPort
• value type: int (can be None)

initialize_once

Calibrate the model only once. If set to False, then this node will recalibrate itself whenever a non-streaming data chunk is received that has both training labels and associated training instances.

• verbose name: Calibrate Only Once
• default value: True
• port type: BoolPort
• value type: bool (can be None)

class_weights

Per-class weights. Optionally this is a mapping from class label to weight. The weights represent the a priori ("prior") probability of encountering a specific class that the model shall assume. The weights will be renormalized so that they add up to one. Example syntax: {'0': 0.5, '1': 0.5} (note the quotes before the colons).

• verbose name: Per-Class Weight
• default value: None
• port type: Port
• value type: object (can be None)

dont_reset_model

Do not reset the model when the preceding graph is changed. Normally, when certain parameters of preceding nodes are being changed, the model will be reset. If this is enabled, the model will persist, but there is a chance that the model is incompatible when input data format to this node has changed.

• verbose name: Do Not Reset Model
• default value: False
• port type: BoolPort
• value type: bool (can be None)

set_breakpoint

Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

• verbose name: Set Breakpoint (Debug Only)
• default value: False
• port type: BoolPort
• value type: bool (can be None)

metadata

User-definable meta-data associated with the node. Usually reserved for technical purposes.

• verbose name: Metadata
• default value: {}
• port type: DictPort
• value type: dict (can be None)