Skip to content

Evaluate and monitor a model

Path: Choose a baseline → Choose the split → Select metrics and costs → Tune on validation data → Report once on test data → Monitor outcomes and drift.

Task Start with Also check
Regression MAE and RMSE against a mean or operational baseline Residuals, large errors, segments, time periods
Classification Confusion matrix and precision/recall at a chosen threshold Probability calibration, PR curve, costs, subgroup outcomes
Clustering Segment sizes, profiles and stability Out-of-sample assignment, usefulness and sensitivity to scaling
Anomaly Reviewed precision at a fixed review budget Score drift, alert volume, missed cases when labels are available
Survival Censoring-aware discrimination and calibration Risk sets and reliability at the chosen horizon
Uplift Randomized held-out policy value Arm counts, uncertainty, cost and overlap

A complete held-out example

The data and metrics example uses the same raw mixed inputs for regression and classification. It generates 4,000 synthetic rows with a fixed seed: 2,000 for training, 1,000 for validation and 1,000 for test. It uses raw category labels and missing numeric values without encoding or imputation.

uv run python examples/data_metrics_pathway.py

Regression: compare with a baseline

The script compares test MAE with a constant equal to the training outcome mean. For this synthetic run, model MAE is 2.505 and baseline MAE is 5.098. Those numbers describe this generated sample, not a general benchmark. Use residual plots to look for error patterns rather than relying only on an average.

Held-out regression residuals

Classification: ranking, calibration and decisions

The script chooses among four thresholds on validation data. Its illustrative cost assigns three units to a missed event and one unit to a false alert. The selected threshold is 0.2. The test confusion matrix is [[6, 486], [1, 507]], with actual classes in rows and predicted classes in columns. Test average precision is 0.643.

Held-out precision-recall curve

A precision–recall curve shows the tradeoff as the score threshold changes. Compare it with the positive event rate and the capacity of the review team. It does not establish calibration.

Held-out calibration plot

Calibration compares predicted probabilities with observed event frequencies in bins. Small bins make the plot unstable. The 1,000 test rows come from a synthetic distribution; the plot alone does not establish calibration in production.

Held-out confusion matrix

The confusion matrix describes decisions at one chosen threshold. Change the cost assumptions or capacity and repeat threshold selection on validation data. Keep the test set independent of that selection.

Inspect the tree

Regression tree for the mixed data example

The tree is rendered from ModelTree.to_dot(). Node conditions describe the fitted partitions. Check feature names, missing routing and leaf support before using them in a report. Categorical conditions can display internal codes, so check their meaning before sharing a rule. A prediction path does not establish the effect of an intervention.

Regression split importance

Split importance summarizes gain used by the fitted tree. It is not a causal effect and does not measure every source of predictive value. The visualization guide shows how to select and render each view.

What remains

The current metrics guide catalogs many report functions, but it does not provide a complete task-by-task evaluation workflow. Survival, uplift, clustering and anomaly examples still need validated task-specific evaluation and uncertainty examples. Check the API before assuming that a recommended metric is implemented by this package.

Metrics and plots · Metrics reference · Model pathways