Parameters Reference¶
Comprehensive documentation for all hyperparameters, knobs, and training controls in mltreelib.
Inspired by the parameter architecture of CatBoost and LightGBM, every knob in mltreelib is typed, deterministic, and mapped directly to the underlying mltreelib-core Rust engine.
Quick Navigation¶
- 1. Core Tree Architecture & Sizing
- 2. Splitting & Search Space Controls
- 3. Regularization, Purity Gates & Split Penalties
- 4. Statistical Hypothesis Stopping & Pre-Selection
- 5. Categorical Handling & Monotonicity Constraints
- 6. Pruning, Honesty & Missing Value Routing
- 7. Leaf Models, GLM Families & Penalized Solvers
- 8. Decoupled Leaf Controls & Hierarchical Retraining
- 9. Telemetry & Execution Controls
1. Core Tree Architecture & Sizing¶
Controls how deep and wide the tree can grow before termination.
| Parameter | Type | Default | Valid Range | Description |
|---|---|---|---|---|
max_depth |
int |
5 |
[1, 64] |
Maximum tree depth from root to any leaf. Set to 1 for decision stumps; increase to 8-12 to capture deep multi-way interactions. |
max_leaf_nodes |
int \| None |
None |
[2, \infty) |
Maximum number of terminal leaf nodes. If set, growth follows best-first priority queue until leaf limit is reached. |
min_rows |
int |
50 |
[1, \infty) |
Minimum number of samples required to form a terminal leaf (minbucket in R / min_child_samples in LightGBM). Prevents overfitting to tiny sub-populations. |
minsplit |
int \| None |
None |
[2, \infty) |
Minimum number of samples required to attempt a split. If None, defaults to 2 * min_rows. |
minprob |
float |
0.02 |
(0.0, 0.5) |
Minimum fraction of training rows required in a terminal node (\(N_{\text{leaf}} / N \ge \text{minprob}\)). Safeguards against isolated slivers. |
min_gain |
float |
1e-9 |
[0.0, \infty) |
Minimum impurity reduction required to justify creating child nodes. Analogous to min_split_gain in LightGBM or gamma in XGBoost. |
2. Splitting & Search Space Controls¶
Governs candidate generation, feature subsampling, and candidate evaluations.
| Parameter | Type | Default | Options | Description |
|---|---|---|---|---|
gain |
str \| None |
None |
"entropy", "gini", "variance" |
Split evaluation objective. If None, automatically resolves to "entropy" for classification and "variance" for regression. |
split_eval |
str |
"fast" |
"fast", "exact" |
Split evaluation engine. "fast" utilizes SIMD vectorization and streaming histogram accumulators; "exact" re-evaluates all continuous thresholds without binning. |
split_precision |
int \| None |
4 |
[1, 16] |
Decimal digit rounding applied to candidate thresholds. Rounds continuous splits (e.g. 3.1416 instead of 3.1415926535) to prevent floating-point overfitting and improve explainability. |
split_cols |
list[int] \| list[str] \| None |
None |
Feature subset | Whitelist of features permitted to form splits. Useful when reserving specific features solely for leaf linear models. |
col_sample |
float \| None |
None |
(0.0, 1.0] |
Fraction of features randomly subsampled at each split decision (colsample_bytree in XGBoost / mtry ratio). Injects stochastic decorrelation across ensemble trees. |
col_weights |
list[float] \| None |
None |
Non-negative weights | Prior weights determining the sampling probability of each feature during candidate selection. Higher weights bias search toward key business drivers. |
interaction_groups |
list[list[int \| str]] \| None |
None |
Feature sets | Enforces interaction constraints: features may only appear on the same root-to-leaf path if they belong to a shared interaction group. Prevents disallowed cross-variable leakage. |
presort |
bool |
False |
True, False |
Whether to pre-sort continuous features at the root. Reduces per-node sort overhead at the cost of initial fit memory. |
3. Regularization, Purity Gates & Split Penalties¶
Structural regularization mechanisms designed to suppress noisy split boundaries.
| Parameter | Type | Default | Valid Range | Description |
|---|---|---|---|---|
purity_gate |
float \| None |
None |
(0.0, 1.0) |
Purity stopping criterion. If a node's target standard deviation or Gini impurity falls below this fraction of global variance, node splitting halts early. |
purity_gate_metric |
str |
"sd" |
"sd", "iqr", "mad" |
Metric for evaluating node purity: standard deviation ("sd"), interquartile range ("iqr"), or median absolute deviation ("mad"). |
tail_quantile |
float \| None |
None |
(0.0, 0.25) |
Extrema protection threshold. Penalizes splits that isolate extreme target quantiles into tiny leaves. |
imbalance_penalty |
float |
0.0 |
[0.0, 1.0] |
Penalizes asymmetric splits where $ |
alpha |
float |
0.0 |
[0.0, \infty) |
Structural complexity penalty subtracted from the candidate split gain (\(G' = G - \alpha\)). Forces simpler tree topologies. |
4. Statistical Hypothesis Stopping & Pre-Selection¶
Replaces heuristic gain thresholds with formal statistical hypothesis testing (analogous to CHAID and partykit::ctree).
| Parameter | Type | Default | Options | Description |
|---|---|---|---|---|
split_test |
str |
"none" |
"none", "f_test", "permutation" |
Formal statistical test evaluated on prospective splits. Halts recursion when null hypothesis of independence cannot be rejected. |
split_alpha |
float |
0.05 |
(0.0, 1.0) |
Significance level threshold (\(p < \alpha\)) required to accept a split candidate. |
bonferroni |
bool |
False |
True, False |
Applies Bonferroni multiplicity correction (\(p < \alpha / P\)) adjusting for the number of tested feature candidates. Eliminates selection bias toward high-cardinality features. |
preselect |
str |
"none" |
"none", "variance", "correlation" |
Heuristic screening step executed before split scans to discard uninformative columns. |
min_relevance |
float \| None |
None |
[0.0, 1.0] |
Minimum correlation/mutual information required to include a feature in split search. |
5. Categorical Handling & Monotonicity Constraints¶
Native handling of categorical variables and directional domain knowledge.
| Parameter | Type | Default | Options | Description |
|---|---|---|---|---|
cat_order |
str \| None |
None |
"target", "frequency", "natural" |
Categorical level ordering strategy. "target" orders levels by target mean/rate, converting combinatorial \(2^{K-1}\) subset search into linear \(O(K)\) boundary scans (Fisher's exact theorem). |
cat_search |
str |
"cumulative" |
"cumulative", "exhaustive" |
Partition search algorithm across ordered levels. "cumulative" checks consecutive level subsets; "exhaustive" evaluates all partition subsets (small \(K\) only). |
monotone |
list[int] \| None |
None |
[-1, 0, +1] |
Monotonicity constraints per feature: +1 forces non-decreasing target response, -1 forces non-increasing, 0 unconstrained. |
tree_monotone |
list[int] \| None |
None |
[-1, 0, +1] |
Enforces monotonic split step direction across parent-child tree partitions. |
leaf_monotone |
list[int] \| None |
None |
[-1, 0, +1] |
Enforces non-negative/non-positive sign constraints on leaf GLM linear coefficients. |
6. Pruning, Honesty & Missing Value Routing¶
Controls post-fit pruning, missing value handling, and sample splitting.
| Parameter | Type | Default | Options | Description |
|---|---|---|---|---|
prune |
str |
"cp" |
"cp", "cost_complexity", "bic", "aic", "none" |
Post-pruning algorithm. "cp" matches CART/rpart complexity pruning; "bic" penalizes number of active parameters using Bayesian Information Criterion. |
cp_alpha |
float |
5.0 |
[0.0, \infty) |
Complexity pruning penalty multiplier. Higher values produce shallower, more general trees. |
honesty |
bool |
False |
True, False |
Enables honest estimation (Wager & Athey 2018). Tree splits are learned on a training sub-sample, while leaf models are fitted on a held-out estimation sub-sample. Guarantees asymptotic normality. |
honesty_fraction |
float |
0.5 |
(0.0, 1.0) |
Proportion of rows allocated to the split-learning sub-sample when honesty=True. |
maxsurrogate |
int |
0 |
[0, 10] |
Number of surrogate split variables stored per node (CART style). When a primary split feature is missing during prediction, the highest-agreement surrogate variable routes the sample. |
7. Leaf Models, GLM Families & Penalized Solvers¶
Defines the parametric or non-parametric local model fitted within each terminal leaf.
| Parameter | Type | Default | Options | Description |
|---|---|---|---|---|
leaf |
str |
"linear" |
"linear", "mean", "median", "quantile", "huber", "mars", "knn" |
Mathematical formulation of the leaf model: piecewise linear OLS ("linear"), constant mean ("mean"), median ("median"), Huber robust regression ("huber"), adaptive splines ("mars"), or non-parametric nearest neighbors ("knn"). |
knn_k |
int |
5 |
[1, \infty) |
Number of nearest neighbors queried at leaf inference time. |
knn_variant |
str |
"uniform" |
"uniform", "distance", "exponential", "dual", "double" |
Weighting scheme for leaf KNN: "uniform" (unweighted), "distance" (inverse distance \(1/(d+\epsilon)^p\)), "exponential" (\(\exp(-\gamma d)\)), "dual" (distance \(\times\) compactness), or "double" (DW-KNN: distance consistency \(\times\) neighbor reliability \(v_i\)). |
knn_metric |
str |
"euclidean" |
"euclidean", "manhattan", "chebyshev", "minkowski", "cosine", "correlation", "canberra", "braycurtis", "gower", "mahalanobis" |
Distance proximity metric across leaf feature coordinates. "gower" normalizes mixed continuous/categorical features and handles missing entries; "mahalanobis" applies diagonal variance scaling. |
knn_k_validity |
int |
5 |
[1, \infty) |
Secondary neighborhood size for precomputing neighbor reliability \(v_i\) (DW-KNN) or local compactness during fit(). |
knn_gamma |
float \| None |
None |
(0.0, \infty) |
Exponential spatial decay rate parameter. If None, automatically calibrated by the inverse median neighbor distance within the queried leaf. |
knn_p |
float |
2.0 |
(0.0, \infty) |
Power parameter \(p\) for Minkowski metric or inverse distance weighting (\(1 / (d + \epsilon)^p\)). |
leaf_family |
str |
"gaussian" |
"gaussian", "binomial", "poisson", "tweedie", "tobit", "softmax" |
Target distribution family for Generalized Linear Model (GLM) leaves. "gaussian" uses identity link, "binomial" uses logit link, "poisson" uses log link. |
leaf_solver |
str |
"irls" |
"irls", "lbfgs", "ccd", "cmaes", "shade", "nes", "ga" |
Local optimization solver for leaf regression models: Iteratively Reweighted Least Squares / exact normal Cholesky ("irls"), Quasi-Newton L-BFGS with Armijo line search ("lbfgs"), Cyclic Coordinate Descent with active set ("ccd"), or SOTA continuous evolutionary algorithms: Covariance Matrix Adaptation Evolution Strategy ("cmaes"), Success-History Adaptive Differential Evolution ("shade"), Separable Natural Evolution Strategies ("nes"), or Real-Coded Genetic Algorithm with SBX crossover ("ga"). |
leaf_cols |
str |
"all" |
"all", "path_used" |
Feature policy for leaf linear models. "all" fits over all features (with L1/L2 shrinkage); "path_used" restricts the leaf model strictly to features split along that leaf's ancestry path (winnowing). |
l1 |
float |
0.0 |
[0.0, \infty) |
Lasso regularization penalty on leaf linear coefficients. Drives irrelevant leaf weights strictly to zero. |
l2 |
float |
0.5 |
[0.0, \infty) |
Ridge regularization penalty on leaf linear coefficients. Stabilizes collinear inversion matrices in high-dimensional leaves. |
tau |
float |
0.5 |
(0.0, 1.0) |
Quantile level when leaf="quantile". Optimizes asymmetric pinball loss for conditional quantile forecasting. |
tweedie_p |
float |
1.5 |
[1.0, 2.0] |
Power parameter \(p\) for Tweedie exponential dispersion modeling. Ideal for zero-inflated insurance claims (\(1 < p < 2\)). |
tobit_threshold |
float |
0.0 |
\(\mathbb{R}\) | Censoring threshold \(y_c\) when leaf_family="tobit". Models left-censored Gaussian variables via expectation-maximization. |
leaf_digest |
bool |
True |
True, False |
Maintains streaming \(t\)-digests in leaves. Enables fast empirical prediction intervals (predict_interval(coverage=0.9)) without distributional assumptions. |
mars_degree |
int |
1 |
[1, 3] |
Maximum interaction degree for MARS spline hinge functions within leaves. |
mars_terms |
int |
10 |
[2, 50] |
Maximum number of basis terms retained in leaf spline models. |
mars_penalty |
float |
2.0 |
[0.0, 10.0] |
Generalized cross-validation (GCV) penalty per knot for MARS leaf models. |
8. Decoupled Leaf Controls & Hierarchical Retraining¶
Enables independent configuration of leaf models and decoupled retraining without changing tree topology.
| Parameter | Type | Default | Description |
|---|---|---|---|
control / leaf_control |
LeafControl \| dict \| None |
None |
Dedicated LeafControl configuration object decoupling leaf model architecture (family, l1, l2, leaf_features) from split tree parameters. |
train_groups |
bool |
True |
Whether to preserve grouped hierarchical entities during leaf retraining. |
train_ids |
bool |
False |
Whether to index training entity row keys in leaf sample caches (leaf_samples()). |
9. Telemetry & Execution Controls¶
| Parameter | Type | Default | Options | Description |
|---|---|---|---|---|
verbose |
int |
0 |
0, 1, 2, 3 |
Training verbosity. 0 = silent; 1 = live depth progress, per-depth summaries, and fit complete summary; 2 = stream per-node split candidates, winning features, threshold, gain, and leaf reasons; 3 = stream leaf solver optimization (GLM convergence iterations, deviance step, linear fit stats). |
seed |
int |
42 |
int |
Deterministic random seed controlling feature subsampling (col_sample) and honest partition splits. |