← machine_learning package

RegularizedLogisticRegression

Classify data instances using regularized Logistic Regression with complex regularization terms.

The logistic regression method is a versatile and principled statistical method that learns a generalized linear mapping from input data to the probability of that data belonging to one of several possible classes. This method is highly competitive with other linear or generalized linear methods, such as LDA and SVM. In comparison to the simple logistic regression node, the emphasis of this implementation is on the support for complex regularization terms, in particular group sparsity and trace-norm regularization. When these features are not needed, it is recommended to consider using the simple node instead, as the training time may be shorter, and the parameter settings may allow for more timely or robust convergence on a wide range of data. Logistic regression makes relatively weak assumptions on the distribution of the data, so that it is moderately tolerant to outliers, especially when compared to relatively brittle methods like LDA. Logistic regression is not considered to be quite as robust as SVM (therefore it can make sense to remove artifacts beforehand using the appropriate nodes if the data are very noisy) -- however, the main benefit of logistic regression over SVM is that the outputs can be straightforwardly interpreted as probabilities without having to resort to 'tricks' such as probability calibration. This implementation uses regularization by default, which allows it to handle large numbers of features very well. This implementation supports several types of regularization, including l1, l2, l1/l2, and trace. The basic l2 regularization is a fine choice for most data (it is closely related to shrinkage in LDA or a Gaussian prior in Bayesian methods). The alternative l1 regularization is unique in that it can learn to identify a sparse subset of features that is relevant while pruning out all other features as irrelevant. This sparsity regularization is statistically very efficient and can deal with an extremely large number of irrelevant features. The l1/l2 regularization is a combination of the two, which can identify groups of features that are irrelevant (also known as group sparsity). In this implementation, sparsity acts on the features as if they formed a matrix, and will prune entire rows or columns of the matrix. In order to be able to do this, the input data tensor should have more than one axis that can be used as features, and these axes need to be identified via the group_axis parameter. The trace regularization mode also views the features as a matrix, but it will learn a weighting such that the corresponding matrix of weights per feature is low rank. As such, it will learn a weighting that can be decomposed into a sum of a small number of pairs of weight profiles for the two matrix axes, e.g., spatial and temporal profiles if features were space by time, which are multiplied together to each form a rank-1 matrix. To determine the optimal regularization strength, the a list of candidate parameter settings can be given, which is then searched by the method using an internal cross-validation on the data to find the best value. If there are very few trials, or some extensive stretches of the data exhibit only one class, this cross-validation can fail with an error that there were too few or no trials of a given class present. Also, the default search grid for regularization (i.e., the list of candidate values) is deliberately rather coarse to keep the method fast. For higher-quality results, use a more fine grained list of values (which will be correspondingly slower). There are also several other implementations of logistic regression with different regularization terms or different performance characteristics in other nodes. This method can be implemented using a number of different numerical approaches which have different running times depending on the number of data points and features. If you are re-solving the problem a lot, it can make sense to try out the various solvers to find the fastest one. By default, this method will return not the most likely class label for each trial it is given, but instead the probabilies for each class (=category of labels), that the trial is actually of that class. Like all machine learning methods, this method needs to be calibrated ("trained") before it can make any predictions on data. For this, the method requires training instances and associated training labels. The typical way to get such labels associated with time-series data is to make sure that a marker stream is included in the data, which is usually imported together with the data using one of the Import nodes, or received over the network alongside with the data, e.g., using the LSL Input node (with a non-empty marker query). These markers are then annotated with target labels using the Assign Targets node. To generate instances of training data for each of the training markers, one usually uses the Segmentation node to extract segments from the continuous time series around each marker. Since this machine learning method is not capable of being trained incrementally on streaming data, the method requires a data packet that contains the entire training data; this training data packet can either be accumulated online and then released in one shot using the Accumulate Calibration Data node, or it can be imported from a separate calibration recording and then spliced into the processing pipeline using the Inject Calibration Data, where it passes through the same nodes as the regular data until it reaches the machine learning node, where it is used for calibration. Once this node is calibrated, the trainable state of this node can be saved to a model file and later loaded for continued use. More Info... Version 1.0.2

Ports/Properties

data

Data to process.

• verbose name: Data
• default value: None
• port type: DataPort
• value type: Packet (can be None)
• data direction: INOUT

penalty

Regularization type. The default trace regularization is a good choice for most data. The trace norm yields low-rank solutions, and l1/l2 yields group-wise sparse solutions. The others are mostly included for completeness; it is more efficient to use LogisticRegression to apply these.

• verbose name: Regularization Type
• default value: trace
• port type: EnumPort
• value type: str (can be None)

group_axes

Axes over which the penalty shall group. For the trace norm, 2 axes must be given. WARNING: this parameter is not safe to use with untrusted strings from the network.

• verbose name: Group Axes
• default value: space, time
• port type: StringPort
• value type: str (can be None)

lambdas

Regularization strength. Stronger regularization makes it possible to fit a more complex model (more parameters), or use less data to fit a model of same complexity.

• verbose name: Regularization Trength
• default value: [0.1, 1.0, 5.0, 10.0]
• port type: Port
• value type: list (can be None)

search_metric

Parameter search metric. When the regularization parameter is given as a list of values, then the method will run a cross-validation for each possible parameter value and use this metric to score how well the method performs in each case, in order to select the best parameter. While 'accuracy' is usually a good default, some other metrics can be useful under special circumstances, e.g., roc_auc for highly imbalanced ratios of trials from different classes.

• verbose name: Scoring Metric If Searching Parameters
• default value: accuracy
• port type: EnumPort
• value type: str (can be None)

num_folds

Number of cross-validation folds for parameter search. Cross-validation proceeds by splitting up the data into this many blocks of trials, and then tests the method on each block. For each fold, the method is re-trained on all the other blocks, excluding the test block (therefore, the total running time is proportional to the number of folds). This is not a randomized cross-validation, but a blockwise cross-validation, which is usually the correct choice if the data stem from a time series. If there are few trials in the data, one can use a higher number here (e.g., 10) to ensure that more data is available for training.

• verbose name: Number Of Cross-Validation Folds
• default value: 5
• port type: IntPort
• value type: int (can be None)

cv_group_field

Optionally a field indicating the group from which each trial is sourced. If given, then data will be split such that test sets contain unseen groups. Examples groups are SubjectID, SessionID, etc.

• verbose name: Grouping Field (Cross-Validation)
• default value:
• port type: StringPort
• value type: str (can be None)

cv_stratified

Optionally perform stratified cross-validation. This means that all the folds have the same relative percentage of trials in each class.

• verbose name: Stratified Cross-Validation
• default value: False
• port type: BoolPort
• value type: bool (can be None)

num_jobs

Number of parallel compute jobs. This value only affects the running time and not the results. Values between 1 and twice the number of CPU cores make sense to expedite computation, but may temporarily reduce the responsiveness of the machine. The value of -1 stands for all available CPU cores.

• verbose name: Number Of Parallel Jobs
• default value: 1
• port type: IntPort
• value type: int (can be None)

probabilistic

Use probabilistic outputs. If enabled, the node will output for each class the probability that a given trial is of that class; otherwise it will output the most likely class label.

• verbose name: Output Probabilities
• default value: True
• port type: BoolPort
• value type: bool (can be None)

verbosity

Verbosity level. Higher numbers will produce more extensive diagnostic output.

• verbose name: Verbosity Level
• default value: 0
• port type: IntPort
• value type: int (can be None)

max_iter

Maximum number of iterations. This is one of the stopping criteria to limit the compute time. The default is usually fine, and gains from increasing the number of iterations will be minimal (it can be worth experimenting with lower iteration numbers if the algorithm must finish in a fixed time budget, at a cost of potentially less accurate solutions).

• verbose name: Maximum Number Of Iterations
• default value: 1000
• port type: IntPort
• value type: int (can be None)

inner_gtol

"LBFGS convergence tolerance. Larger values give less accurate results but faster solution time.

• verbose name: Inner Gtol
• default value: 0.0001
• port type: FloatPort
• value type: float (can be None)

inner_max_iter

LBFGS maximum number of iterations for inner solver. Additional stopping criterion to limit compute time.

• verbose name: Inner Max Iter
• default value: 10
• port type: IntPort
• value type: int (can be None)

abs_tol

Absolute convergence tolerance. Smaller values will lead the solver run to a closer approximation of the optimal solution, at the cost of increased running time. See also relative tolerance.

• verbose name: Absolute Tolerance
• default value: 1e-05
• port type: FloatPort
• value type: float (can be None)

rel_tol

Relative convergence tolerance. Smaller values will lead the solver run to a closer approximation of the optimal solution, at the cost of increased running time. In contrast to the absolute tolerance, this value is relative to the magnitude of the regression weights. Note that the used method works best when the desired accuracy is not excessive, and merely a good approximation is sought.

• verbose name: Relative Tolerance
• default value: 0.01
• port type: FloatPort
• value type: float (can be None)

lbfgs_memory

LBFGS memory length. This is the maximum number of variable-metric corrections tracked to approximate the inverse Hessian matrix.

• verbose name: Lbfgs Memory
• default value: 10
• port type: IntPort
• value type: int (can be None)

init_rho

Initial value of augmented Lagrangian parameter. This parameter, which is specific to the used solver, can be auto-tuned, which makes the method is relatively robust to the initial value. However, slight adjustments in the 1 to 30 range can reduce the number of iterations required for convergence and thereby the running time. However, choosing a grossly inappropriate value can cause the method to fail to converge, which is easily diagnosed by having a solution that is essentially non-sparse.

• verbose name: Solver Rho
• default value: 4
• port type: FloatPort
• value type: float (can be None)

update_rho

Auto-tune rho parameter. Whether to update the solver's rho parameter dynamically. This can be used to achieve faster convergence times in highly time-sensitive setups, but there is a modest risk that on some data the solution can 'blow up', although this could potentially be overcome by tuning the other solver parameters related to the rho update logic.

• verbose name: Auto-Tune Solver Rho
• default value: True
• port type: BoolPort
• value type: bool (can be None)

rho_threshold

Rho update trigger threshold. This determines how frequently updates to the solver rho parameter can be triggered. A larger value will lead to rho changing less frequently. This parameter is essentially a tradeoff between the solver adapting more quickly to settings that are optimal for convergence (using a lower value) versus preventing settings from changing too erratically (using a higher value) and thus prompting stalls or convergence failures on difficult data.

• verbose name: Solver Rho Update Threshold
• default value: 10
• port type: FloatPort
• value type: float (can be None)

rho_incr

Rho update increment factor. When rho is being increased, this is the factor by which it is changed. Larger values can lead to quicker adaptation if the initial value is off or solutions change rapidly, at an increased risk of overshooting.

• verbose name: Solver Rho Increment Factor
• default value: 2
• port type: FloatPort
• value type: float (can be None)

rho_decr

Rho update decrement factor. When rho is being decreased, this is the factor by which it is divided. Larger values can lead to quicker adaptation if the initial value is off or solutions change rapidly, at an increased risk of overshooting.

• verbose name: Solver Rho Decrement Factor
• default value: 2
• port type: FloatPort
• value type: float (can be None)

over_relaxation

ADMM over-relaxation parameter. 1.0 is no over-relaxation.

• verbose name: Over Relaxation
• default value: 1.0
• port type: FloatPort
• value type: float (can be None)

initialize_once

Calibrate the model only once. If set to False, then this node will recalibrate itself whenever a non-streaming data chunk is received that has both training labels and associated training instances.

• verbose name: Calibrate Only Once
• default value: True
• port type: BoolPort
• value type: bool (can be None)

dont_reset_model

Do not reset the model when the preceding graph is changed. Normally, when certain parameters of preceding nodes are being changed, the model will be reset. If this is enabled, the model will persist, but there is a chance that the model is incompatible when input data format to this node has changed.

• verbose name: Do Not Reset Model
• default value: False
• port type: BoolPort
• value type: bool (can be None)

set_breakpoint

Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

• verbose name: Set Breakpoint (Debug Only)
• default value: False
• port type: BoolPort
• value type: bool (can be None)

metadata

User-definable meta-data associated with the node. Usually reserved for technical purposes.

• verbose name: Metadata
• default value: {}
• port type: DictPort
• value type: dict (can be None)