← deep_learning package

DeepModel

A deep-learning based machine learning model.

This can be used as a drop-in replacement for off-the-shelf ML nodes such as the "Linear Discriminant Analysis" node. The node is configured by wiring a network graph, starting with an input placeholder, and containing some mix of Layer nodes, other deep learning nodes (in the deep learning category) and pure math nodes, and any other data restructuring nodes. Stateful nodes like filters (e.g., in the signal processing category) are not currently supported. The last node of the network graph should produce an output (i.e., a prediction) for each instance (trial) in the original input data. The output can be packet or a plain array. The input placeholder name should then be listed in the network signature of the deep model node. Alternatively you can also end the network graph with a Define Net node, which gives the network (and its weights) a name prefix; in this case the input placeholder needs to be listed in the network signature of the Define Net node (and you will need to use a (*) signature for the network graph in the deep model node. Optionally your network may also end in a NetTransform node, but this is not necessary (the deep model will automatically do the equivalent of that if needed). The predictions are then passed into the selected loss function along with the target labels; the loss can either be chosen from among a set of predefined losses, or you can specify a custom loss graph. The latter needs to have two input placeholders, both listed in the loss signature (defaulting to preds and targs). The first listed input must receive the predictions and the second must be the one for the target values. Typically the graph will then use one of the predefined loss nodes (nodes ending in Loss), but it can also implement, for example, custom per-class weights or per-instance weights. This node then takes the gradient of the provided loss with respect to the weights, and uses the selected optimizer to update the weights. The optimizer can be chosen from a set of predefined optimizers, or you can wire in a custom optimizer step node (node ending in Step). Of these, the ChainedStep and CustomStep nodes allow you to build fully custom optimizer graphs, which can be used to implement techniques such as learning rate schedules, gradient clipping, weight decay, weight constraints via projected gradient descent, or different optimizers for different weights (e.g., excluding layers or treating biases differently, etc). The full step will optionally be compiled to run efficiently, although not every combination of nodes used may be compilable. The learning behavior of the node can be configured in terms of the used batch size, number of steps (measured in instances, batches, or epochs), data shuffling behavior, and what kind of input data shall be used for training (e.g., only offline data or both offline and streaming, etc) vs only for prediction. You can also pass a 2-element list of training and validation data to the node, in which case a validation step will be executed on the validation data with some specified frequency, and which will evaluate the specified metric on predictions vs. targets. You can also provide a custom validation step as a graph, which can be used for custom progress tracking such as specific output requirements, custom metrics, custom validation frequency, and so forth. It is recommended to benchmark your GPU utilization and memory usage using a tool like nvitop while running, in order to ensure that you're using the GPU(s) optimally. There are be pros/cons to moving the data to the GPU using the MoveToBackend node before passing to this node; this may use more GPU memory (especially with larger datasets) but will typically achieve higher utilization. Alternatively, you can keep the data on the host but place two (or rarely more) tasks on the same GPU, for example by setting your cross validation node to run in parallel; this can restore any lost utilization. Also, while validation data is a protection against overfitting, it can be a good idea to instead limit the epoch count to the point where the model starts to overfit and to disable the validation split. This will typically be more efficient and use less GPU memory, often allowing you to place more than one task on the GPU. Like all machine learning methods, this method needs to be calibrated ("trained") before it can make any predictions on data. For this, the method requires training instances and associated training labels. The typical way to get such labels associated with time-series data is to make sure that a marker stream is included in the data, which is usually imported together with the data using one of the Import nodes, or received over the network alongside with the data, e.g., using the LSL Input node (with a non-empty marker query). These markers are then annotated with target labels using the Assign Targets node. To generate instances of training data for each of the training markers, one usually uses the Segmentation node to extract segments from the continuous time series around each marker. Since this machine learning method is not capable of being trained incrementally on streaming data, the method requires a data packet that contains the entire training data; this training data packet can either be accumulated online and then released in one shot using the Accumulate Calibration Data node, or it can be imported from a separate calibration recording and then spliced into the processing pipeline using the Inject Calibration Data, where it passes through the same nodes as the regular data until it reaches the machine learning node, where it is used for calibration. Once this node is calibrated, the trainable state of this node can be saved to a model file and later loaded for continued use. More Info... Version 0.5.2

Ports/Properties

data

Data to process. This can be a packet or a two- element list of training and validation data.

• verbose name: Data
• default value: None
• port type: DataPort
• value type: object (can be None)
• data direction: INOUT

net

Neural network graph.

• verbose name: Net
• default value: None
• port type: GraphPort
• value type: Graph

net__signature

Argument names of network graph. This is a listing of the names of input arguments of your neural network, which is typically a single argument conventionally named "inputs" (although you can choose your own name). Your network is then a graph that begins with a Placeholder node whose slotname must match the name listed here, and which is followed by a series of NN nodes (e.g., Layers, normalization, etc), possibly interleaved with other mathematical operations and/or data formatting nodes. The final output of your network is expected to be predictions given the inputs (without a link function applied, i.e., if you are doing classification, the final output should be logits as produced by a Dense layer without a trailing Activation node, instead of probabilities). This final output node is then wired into the DeepModel's "net" input port. Note that in graphical UIs, the edge that goes into the "net" input will be drawn in dotted style to indicate that this is not normal forward data flow, but that the network graph runs under the control of the DeepModel node. The DeepModel node will act on your network in various ways, among others taking derivatives to optimize the weights, and invoking it to make predictions. The final predictions are ideally in a form that can be directly wired into one of the Loss nodes (e.g., SquaredLoss) without any further processing; a Dense node trivially satisfies this. Your network graph may also optionally contain an additional Placeholder node with slotname set to is_training, which can also be listed here in the signature. This placeholder will receive True if the network is called on training data or on prediction data.

• verbose name: Net [Signature]
• default value: (inputs)
• port type: Port
• value type: object (can be None)

loss

Optional custom loss function graph.

• verbose name: Loss
• default value: None
• port type: GraphPort
• value type: Graph

loss__signature

Optional argument names accepted by loss function graph. This is only used if you aim to fully override the loss function by a completely custom graph; in all other cases you can simply select the desired loss in the loss_function port. This is a graph with usually at least two placeholders (one for predictions and one for targets) whose slotnames match those listed here, and which usually ends in one of the Loss nodes (note that the loss returned by this graph is per prediction rather than a single overall scalar). The default loss is structured like this, where the Loss node is chosen and configured according to the loss_function setting. Your may also accept a third input that receives per-sample weights. For certain unsupervised losses you may also build for targets being passed in as all-zeros. The final output of the loss graph is then wired into the "loss" input of the DeepModel node. Note that in graphical UIs, the edge that goes into the "loss" input will be drawn in dotted style to indicate that this is not normal forward data flow, but that the loss graph runs under the control of the DeepModel node.

• verbose name: Loss [Signature]
• default value: (preds,targs)
• port type: Port
• value type: object (can be None)

optstep

Optional optimizer step node.

• verbose name: Optstep
• default value: None
• port type: GraphPort
• value type: Graph

valstep

Optional validation step graph.

• verbose name: Valstep
• default value: None
• port type: GraphPort
• value type: Graph

valstep__signature

Arguments for the optional validation step graph. This is a graph with usually three placeholders (iteration count, predictions, and optionally targets) that receive their input from the supplied validation data. This graph can do anything (e.g., compute and print a performance metric), may re-batch the data (e.g., to compute a running average), could optionally only emit outputs once every few calls, and may invoke a Break or Continue node to either stop the optimization (e.g., as a form of early stopping) or to reject the current optimizer update (e.g. for backtracking to the state at the previous validation update), which can be used if loss spikes or other signs of divergence occur. The final output of the graph should be a performance measure, e.g., as obtained from the PerformanceMetric node. The default graph has three placeholders named iter, preds, and targs that are wired into the update, preds, and targs ports of the PerformanceMetric node. The node's perf_metric setting defaults to the DeepModel's eval_metric setting. To replace this graph, you can start with a custom graph that replicates this recipe and then modify it to your needs.

• verbose name: Valstep [Signature]
• default value: (iter,preds,targs)
• port type: Port
• value type: object (can be None)

trainfeed

Optional training feed graph.

• verbose name: Trainfeed
• default value: None
• port type: GraphPort
• value type: Graph

trainfeed__signature

Arguments for the optional training feed graph. The purpose of this graph is to emit shuffled mini-batches of training data, and it can be overridden to realize highly customized input pipelines, which is however rarely necessary. The basic function of the graph is to take the inputs (e.g., a Packet), and according to the iter parameter, which is the 0-based iteration number, emit a mini-batch of data; the default behavior is fairly complex and works as follows: first, an integer range is created that indexes all trials in the input dataset. Then, in a collecting fold loop loop over the number of epochs (passes through the training set), the range is successively shuffled and the shuffled ranges are then concatenated and flattened into a single index array. Then, a length-based index range starting at the current iter and of length batch size is selected from that index sequence and then used to index the input dataset, which is then returned. Due to the need to shuffle each range differently, the fold loop uses the split random seed node to successively advance the current random key while splitting off a key for use with the permutation. Note that the graph only sees data meant for training and neither test-only nor validation data, and that all inputs except for the iter are constant for the duration of the training process.The total_insts arguments is the total number of instances to emit across the training process, i.e., this will be larger than the number of instances in the dataset of multiple passes over the dataset are to be made.

• verbose name: Trainfeed [Signature]
• default value: (inputs,randkey,batch_size,total_insts,iter)
• port type: Port
• value type: object (can be None)

augmentations

Optional data augmentation graph.

• verbose name: Augmentations
• default value: None
• port type: GraphPort
• value type: Graph

augmentations__signature

Arguments for the optional data augmentation graph. This graph applies to the output of the training feed, i.e., a single mini-batch. The graph may increase/decrease or otherwise recombine the instances in the batch, and may also add noise or other perturbations. One technique is to first use the RepeatAlongAxis node to replicate the batch a number of times, then to apply the augmentations, and finally to select a random subset (driven by the randkey) to bring the batch back to its original size. Another variant defines a larger than usual incoming batch size (e.g., 5x) in the DeepModel node, augments this, and finally reduces the output to a normal batch size. This allows the augmentation to operate on a greater diversity of data than it otherwise would; however, this has the side effect that a pass through a dataset will only use a relatively small percentage of the trials, so the epoch count has to also be increased accordingly (e.g., 5x). The graph may also have a third argument that receives the is_training flag, which is True when the graph is called on training data and False when it is called on test data; if this is not present, the graph is only applied to training data and never test or validation data. This is mostly useful for augmentations that produce biased results.

• verbose name: Augmentations [Signature]
• default value: (batch,randkey,~)
• port type: Port
• value type: object (can be None)

loss_function

Loss function to optimize for. The two cross-entropy losses are the default losses to use for classification problems. The sigmoid loss is for two-class classification and assumes that your pipeline typically ends in a single Dense Layer node (or equivalent) that has one output unit and no trailing Activation node (the loss function applies the sigmoid). The softmax loss is for multi-class classification and instead assumes that you use a Dense Layer node with as many output units as there are classes. Again, the Dense Layer should not be followed by an Activation node, as a softmax non-linearity is applied by the loss function. The hinge loss is a robust but non-probabilistic loss for two-class classification (used in support vector machines). The squared, huber, and log_cosh are all losses for use in regression problems, i.e., if a continuous-valued quantity with normally-distributed (in case of squared loss) or heavy-tailed noise (with huber and log_cosh) is being predicted. The latter two losses are therefore usable for robust regression, but note that the robustness of Huber critically depends on a parameter. The cosine_distance loss is usable for vector-valued predictions where the correlation against some target vectors is being optimized. Additional information can also be found in the node of similar name ending in Loss. You may also specify an entirely custom loss via a graph, in which case the loss function may be set to 'custom' for clarity. To specify a custom loss, wire a graph into the "loss" port that has a 'preds' and 'targs' placeholder node and which emits a single scalar loss for the given batch, which in the simplest case is one of the existing Loss nodes followed by a Sum node (that drops the summed-over axis). Overriding the loss allows you to use, for example, class-weighted losses or instance-weighted losses (for the latter your loss graph needs to accept a third positional argument that receives the per-instance weights).

• verbose name: Loss Function
• default value: sigmoid_binary_crossentropy
• port type: ComboPort
• value type: str (can be None)

optimizer

Optimizer to use. Optimizers differ in a number of characteristics, most importantly in the speed of convergence and the tendency to either diverge ("blow up", resulting in NaNs or infinites) or to fail to learn anything. Other characteristics are how much memory they use, and to some extent how likely they are to overfit the training data. Good starting choices are adam and adamw, or one of adabelief, amsgrad, fromage or yogi if adam fails (e.g., diverges or fails to learn), but all optimizers have scenarios where they outperform all others. More advanced users may experiment with rmsprop, novograd, sgd, or noisysgd. For very large batch sizes, lamb and lars are well-adapted. Also adafactor and sm3 can be useful for very large models to conserve memory. For information on the available optimizers, see the individual Step nodes in NeuroPype's deep learning category. Optimizers can be configured with parameters here, by appending them in parentheses after the name and separated by commas, in the order of appearance in the respective step node. For example, 'adam(0.001, 0.9, 0.999)' specifies the Adam optimizer with a learning rate of 0.001, beta1 of 0.9, and beta2 of 0.999. Note that in most optimizers the learning rate is potentially problem-specific, and may have to be tuned. Beyond basic scenarios and to achieve maximum accuracy, the learning rate would typically be decayed over time following a schedule to improve robustness, convergence and final-solution accuracy. To specify such a learning rate schedule you need use a custom optimizer. This is done by choosing the "custom" optimizer setting and then wiring one of the Step nodes corresponding to your optimizer into the "optstep" port (using the step node's "this" output port). Finally you wire one of the Schedule nodes (e.g., "Linear Warmup Exponential Decay Schedule" (WarmupExponentialDecaySchedule)) into the into the step node's 'learning_rate_schedule' port. This also allows you to override parameters of the optimizer in your pipeline graph rather than in textual form. When doing so note that not all Step node are end-to-end optimizers - if in doubt, go by the optimizers in this listing or read the optimizer's documentation text. It is also possible to use a fully custom optimizer using either the ChainedStep node or the CustomStep node, which gives fine-grained control over how different layers are updated, weight-decay regularization and constraints, and other aspects of the optimization process. (see documentation of these two nodes for more details on their use)

• verbose name: Optimizer
• default value: adam(0.001)
• port type: ComboPort
• value type: str (can be None)

batch_size

Number of samples to process in each (mini-)batch. This is the number of instances that will be processed at a time by the network. The batch size is a tradeoff between computational speed (larger batches can be faster on sufficiently parallel hardware) vs stochastic noise in the gradient estimate (smaller batches are more noisy) and memory usage (large batches require more memory). Note that noisy gradients are not necessarily a bad thing, since they tend to act as a form of regularization, which may protect against overfitting. The batch size also interacts with the learning rate, so after changing one the other may have to be readjusted. For very large batch sizes (1000s), you may also need to select a different type of optimizer step, for example LAMB or LARS.

• verbose name: Batch Size
• default value: 32
• port type: IntPort
• value type: int (can be None)

maxsteps

Maximum number of training steps to perform, measured in the given step unit (see stepunit). Must be specified. If set to 0, this will suppress training.

• verbose name: Maxsteps
• default value: None
• port type: IntPort
• value type: int (can be None)

stepunit

Unit in which maxsteps is given. Important: the 'batches' and 'instances' units are the total number of steps to train, even if training on streaming data or on successively provided offline datasets, or if the data has more instances than used as per maxsteps. In contrast, the 'epochs' unit always refers to the training data on the current invocation of the node; that is, if epochs is set to N, then the node will make exactly N passes through each given training data packet, whether that is the first and only packet or wether this is the m'th such packet received during eg training on streaming data or on successive offline data (as governed by the train_on setting).

• verbose name: Maxsteps Unit
• default value: epochs
• port type: EnumPort
• value type: str (can be None)

shuffle

Whether and when to shuffle the training data. Shuffling is necessary when training on time-series data (or other serially correlated data) to implement stochastic gradient descent. If set to "none", no shuffling is performed, which usually requires that your training data is preshuffled. If set to "once", then the training dataset is shuffled once at the start of training. If set to "per-epoch", then the training data is shuffled at the start of each epoch (this behaves the same way regardless of whether your maxsteps is counted in terms of epochs or iterations). This option is ignored and may be set to 'custom' when a custom trainfeed graph is specified, which takes over shuffling.

• verbose name: Shuffle
• default value: per-epoch
• port type: ComboPort
• value type: str (can be None)

train_on

Update the model on the specified data. Note that generally the model will output predictions on any data that it receives whether it is training on it or not. In the "initial offline" mode - and only if the model is not already trained (i.e., if no previously learned model was loaded in) - the model is trained on the first non-streaming packet that it receives, which should thus be the training set. This can be used when the first packet is the training data and any subsequent packets are test data only, OR when a pretrained model is loaded that should not be further updated. In the "successive offline" mode, the model is trained on any non-streaming packet that it receives, whether it is already pretrained or not (in such case it will be further fine-tuned given the new training data). Any streaming data is taken as test-only data; this is a typical scenario for real-time processing when the model is first trained or fine-tuned on some pre-recorded data, but it can also be used to just train a model on multiple successive datasets. If max_steps is given in epochs, it means that each successive dataset will be passed through the model for this many epochs. The last mode, "offline and streaming" will train on any data, whether it is offline or streaming. This can be used for either real-time training or fine-tuning, i.e., while data is being collected, or for "out-of-core" training on a very large dataset that would not fit in memory if loaded up-front, and which is therefore streamed through the model in chunks. The parameter can also be changed at runtime, to switch from one mode to another.

• verbose name: Train On
• default value: initial offline
• port type: EnumPort
• value type: str (can be None)

dont_reset_model

Do not reset the model when the input data graph is changed. Normally, when certain parameters of preceding nodes are being changed, the model will be reset. If this is enabled, the model will persist, but there is a chance that the model is incompatible when input data format to this node has changed.

• verbose name: Do Not Reset Model
• default value: False
• port type: BoolPort
• value type: bool (can be None)

validation_split

Fraction of the training data to use for validation. Setting this to a number between 0 and 1 will split off the specified fraction from the training data (always from the end of the data) and use it as validation data. This data will not be used for training but to support the early-stopping technique, wherein training stops when the validation performance no longer improves. This can also be omitted (set to zero) to disable validation and early stopping, in which case the maximum number of steps needs to be set more carefully. As an alternative to using this parameter, one may also provide validation data along with training data through the data input.

• verbose name: Validation Split
• default value: 0.2
• port type: FloatPort
• value type: float (can be None)

eval_metric

Metric to use for reporting progress. This can be either a built-in validation metric or a fully custom validator step (which must be wired into the valstep port). The metric is evaluated both on the training data and on the validation data if any was passed in. Note that ONLY the validation metric will track generalization performance, while the training-set metric mainly serves to spot potential optimizer pathologies or model inadequacy. One training-set diagnostic is whether the model learned anything at all from the training data vs flatlining at chance performance level, and another is as an upper bound on the best-case generalization error under ideal circumstances, where an unsatisfactory score would indicate that the model capacity or architecture or learning process failed to adequately model the training data. Generally some of the metrics are suitable for classification problems (accuracy, precision/recall/f1, and roc metrics) while others are for use in regression problems (r2, explained_variance, max_error and the neg_error metrics). These metrics are all available in the scikit-learn package, and the documentation for them can be found on the web. In quick summary, balanced_accuracy is a good choice for generic classification problems on balanced or unbalanced data, precision/recall are sometimes asked for in detection problems of a relatively rare class (e.g., clinical diagnosis), and the roc metrics are useful both for measuring performance on unbalanced data (although on extremely unbalanced data, the measures can break down), and for measuring the performance in detection problems across a range of decision thresholds, e.g., when the relative importance of type-1 and type-2 errors is not fixed at the time the ML work is done. The F1 scores are a specific blend of precision and recall across classes that can be used as a general-purpose (albeit perhaps not very interpretable) performance measure on some signal detection tasks. For regression, the r2 score is a fair choice for generic signal regression and is relative to the scale of the target variable; explained variance is similar but specifically does not score systematic offset (i.e., bias) in the predictions (usually a deficiency of the metric but sometimes useful, e.g. while working on a model that will still receive a bias correction later). The neg_error metrics are the most common tools in regression problems; of those, the mean_absolute, median_absolute variants are perhaps the most interpretable as the errors are directly in the same units as the target variable (e.g., meters), and the latter is robust if there are outliers in either the targets or predictions. The mean_squared error is the default choice under a Gaussian noise assumption, and the mean_squared_log error is useful if the data are log-normally distributed (e.g., data from exponential growth processes). Somewhat similarly, the neg_mean_absolute_percentage_error is also adequate for measuring performance on data where the target variable ranges across several orders of magnitude in scale, and the error is relative to each individual target value (but note that the score is not measured in percent but is a fraction). Note that in evaluation scores, higher values are better (unlike loss measures). Also unlike the loss measures, most of the metrics are not differentiable, so they cannot be directly plugged into the optimizer. However, model hyper-parameters can be optimized with respect to these metrics when using the Parameter Optimization node, at significant computational overhead (requiring many model training runs for different hyper parameter values).

• verbose name: Evaluation Metric
• default value: balanced_accuracy
• port type: ComboPort
• value type: str (can be None)

verbosity

Verbosity level. Higher numbers will produce more extensive diagnostic output.

• verbose name: Verbosity Level
• default value: 1
• port type: EnumPort
• value type: str (can be None)

random_seed

Seed for any pseudo-random choices during training. This can be either a splittable seed as generated by Create Random Seed or a plain integer seed. If left unspecified, this will resolve to either the current 0-based "task number" if used in a context where such a thing is defined (for example cross-validation or a parallel for loop), or 12345 as a generic default.

• verbose name: Random Seed
• default value: None
• port type: Port
• value type: AnyNumeric (can be None)

compile_model

When to compile the model used for training. This can be set to 'never' to run the model in what is known as "eager mode", which is slow but allows for potentially easier debugging. The difference between always and cached is that the latter is a hint to attempt to save the compiled model to disk and reuse it later when possible. However, whether and in what situations that is actually done depends on DeepModel and the underlying implementation.

• verbose name: Compile Model
• default value: cached
• port type: EnumPort
• value type: str (can be None)

skip_incomplete_batch

Whether to skip the last batch encountered during training if it is smaller than the batch size. This is applicable when the dataset size is not a multiple of the batch size. Processing this batch will require a separate compilation for the odd batch size, but leaves no data samples unprocessed. When training for typical number of epochs and trials, the trials in this batch tend to be insignificant to the overall training progress, which typically involves thousands of batches, so this option should generally be left enabled.

• verbose name: Skip Incomplete Batch
• default value: True
• port type: BoolPort
• value type: bool (can be None)

retain_model

What model to retain for output. This can be either the model with the best loss or validation score, or the model that corresponds to the last training step. If no validation data are provided, this will always be the last model.

• verbose name: Retain Model
• default value: lowest-loss
• port type: EnumPort
• value type: str (can be None)

canonicalize_output_axes

Whether to canonicalize the output axes of the model to match the expected output axes of other machine-learning nodes. This can be turned off if your model emits a handcrafted feature or statistic axis to describe its predictions that you'd like to retain. Note though that some downstream nodes, like MeasureLoss might not work as expected.

• verbose name: Canonicalize Output Axes
• default value: True
• port type: BoolPort
• value type: bool (can be None)

no_compile_in_debug

Do not compile the model when running in debug mode.

• verbose name: No Compile In Debug
• default value: True
• port type: BoolPort
• value type: bool (can be None)

conserve_memory_every

Conserve memory by clearing caches etc. every this many steps (in the respective stepunit). This is a hint to the system that may not actually be honored, depending on the implementation and settings such as whether the model is compiled or not. The default is to NOT clear any caches. A good choice is to clear caches every 3-5 epochs for heavy models. For extremely memory-constrained setups, this can also be set to -1, in which case the cache will be cleared before training a new epoch and before predictions for a new epoch.

• verbose name: Conserve Memory Every
• default value: 0
• port type: IntPort
• value type: int (can be None)

set_breakpoint

Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

• verbose name: Set Breakpoint (Debug Only)
• default value: False
• port type: BoolPort
• value type: bool (can be None)

metadata

User-definable meta-data associated with the node. Usually reserved for technical purposes.

• verbose name: Metadata
• default value: {}
• port type: DictPort
• value type: dict (can be None)