Skip to content

← machine_learning package

EmpiricalBayesClassification

Sparse Bayesian logistic regression using the empirical Bayes (type-II maximum likelihood) framework.

The model places independent zero-mean Gaussian priors on each feature weight with per-feature precision hyperparameters alpha. Evidence maximization drives many alpha values to infinity, effectively pruning the corresponding features and yielding a sparse classifier with hard zeros.

Two algorithm variants are available. The fast method (Tipping & Faul, 2003) starts from an empty model and sequentially adds, deletes, or re-estimates features based on their marginal likelihood contribution. This is efficient for high-dimensional problems where only a few features are relevant. The full method uses the MacKay ARD update rule, starting with all features and iteratively pruning those whose precision exceeds a threshold. It is simpler and can be more thorough, but has higher per-iteration cost for large feature sets.

The underlying inference uses a Laplace approximation (iteratively reweighted least squares) to handle the logistic likelihood, combined with closed-form alpha updates in the outer evidence-maximization loop. This approach is deterministic (no sampling), fast, and produces exact zeros for irrelevant features.

More Info... Version 0.7.0

Ports/Properties

data

Data to process.

  • verbose name: Data
  • default value: None
  • port type: DataPort
  • value type: Packet (can be None)
  • data direction: INOUT

cond_field

Instance data field that contains the class labels to discriminate. The field must be binary (exactly two distinct values).

  • verbose name: Cond Field
  • default value: TargetValue
  • port type: StringPort
  • value type: str (can be None)

feature_scaling

Feature scaling applied before fitting. Standardization ('std') is recommended because the ARD prior is scale-sensitive. 'robust' uses the median and interquartile range instead of mean and standard deviation.

  • verbose name: Feature Scaling
  • default value: std
  • port type: EnumPort
  • value type: str (can be None)

method

Method variant.

'fast' uses the Tipping & Faul (2003) sequential add/delete algorithm that maintains a small active set of relevant features and is efficient for high-dimensional problems with sparse solutions. It starts from an empty model and greedily adds features that most increase the marginal likelihood.

'full' uses the MacKay ARD update rule starting from all features and progressively pruning irrelevant ones. It explores the full feature space more thoroughly but has higher per-iteration cost (O(p3) when p > n, mitigated via the Woodbury identity).

  • verbose name: Method
  • default value: fast
  • port type: EnumPort
  • value type: str (can be None)

include_bias

Include a bias (intercept) term in the logistic regression model. The bias receives a weak prior (small alpha) and is always retained.

  • verbose name: Include Bias Term
  • default value: True
  • port type: BoolPort
  • value type: bool (can be None)

pruning_threshold

Alpha threshold for feature pruning. Features whose ARD precision hyperparameter alpha exceeds this value are considered irrelevant and removed. Larger values are more permissive. Primarily used by the 'full' method; the 'fast' method uses its own marginal-likelihood-based deletion criterion.

  • verbose name: Pruning Threshold
  • default value: 100000000.0
  • port type: FloatPort
  • value type: float (can be None)

max_iterations

Maximum outer iterations for the evidence-maximization loop. For 'fast', this is the maximum number of sequential add/delete/re-estimate steps. For 'full', it is the maximum number of alpha-update rounds. The 'fast' method typically converges in a few dozen steps; the 'full' method may use up to 200 for large feature sets.

  • verbose name: Maximum Iterations
  • default value: 500
  • port type: IntPort
  • value type: int (can be None)

convergence_tolerance

Convergence tolerance on the change in log marginal likelihood between successive outer iterations. The default of 1e-4 is well suited for both method variants.

  • verbose name: Convergence Tolerance
  • default value: 0.0001
  • port type: FloatPort
  • value type: float (can be None)

irls_max_iterations

Maximum IRLS (iteratively reweighted least squares) iterations per outer step. The IRLS loop finds the MAP estimate of the logistic regression coefficients given the current alpha values via the Laplace approximation.

  • verbose name: Irls Max Iterations
  • default value: 25
  • port type: IntPort
  • value type: int (can be None)

irls_tolerance

Convergence tolerance for the IRLS inner loop on the maximum absolute weight change.

  • verbose name: Irls Tolerance
  • default value: 1e-06
  • port type: FloatPort
  • value type: float (can be None)

initialize_once

Calibrate the model only once on the first data that arrives. Subsequent data chunks are only used for prediction. Uncheck to recalibrate whenever new labeled data arrives.

  • verbose name: Calibrate Only Once
  • default value: True
  • port type: BoolPort
  • value type: bool (can be None)

dont_reset_model

Do not reset the model when the preceding graph structure is changed. Enable this to keep a previously calibrated model across graph edits.

  • verbose name: Do Not Reset Model
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)

probabilistic

Output class probabilities rather than hard class labels. When enabled, the output has two columns (one per class) with the predicted probabilities.

  • verbose name: Output Probabilities
  • default value: True
  • port type: BoolPort
  • value type: bool (can be None)

verbosity

Verbosity level. Set to 1 for summary output, 2 for per-iteration diagnostics.

  • verbose name: Verbosity Level
  • default value: 0
  • port type: IntPort
  • value type: int (can be None)

set_breakpoint

Set a breakpoint on this node. If this is enabled, your debugger (if one is attached) will trigger a breakpoint.

  • verbose name: Set Breakpoint (Debug Only)
  • default value: False
  • port type: BoolPort
  • value type: bool (can be None)

metadata

User-definable meta-data associated with the node. Usually reserved for technical purposes.

  • verbose name: Metadata
  • default value: {}
  • port type: DictPort
  • value type: dict (can be None)