Evaluate and monitor a model¶
Path: Choose a baseline → Choose the split → Select metrics and costs → Tune on validation data → Report once on test data → Monitor outcomes and drift.
| Task | Start with | Also check |
|---|---|---|
| Regression | MAE and RMSE against a mean or operational baseline | Residuals, large errors, segments, time periods |
| Classification | Confusion matrix and precision/recall at a chosen threshold | Probability calibration, PR curve, costs, subgroup outcomes |
| Clustering | Segment sizes, profiles and stability | Out-of-sample assignment, usefulness and sensitivity to scaling |
| Anomaly | Reviewed precision at a fixed review budget | Score drift, alert volume, missed cases when labels are available |
| Survival | Censoring-aware discrimination and calibration | Risk sets and reliability at the chosen horizon |
| Uplift | Randomized held-out policy value | Arm counts, uncertainty, cost and overlap |
A complete held-out example¶
The data and metrics example uses the same raw mixed inputs for regression and classification. It generates 4,000 synthetic rows with a fixed seed: 2,000 for training, 1,000 for validation and 1,000 for test. It uses raw category labels and missing numeric values without encoding or imputation.
Regression: compare with a baseline¶
The script compares test MAE with a constant equal to the training outcome mean. For this synthetic run, model MAE is 2.505 and baseline MAE is 5.098. Those numbers describe this generated sample, not a general benchmark. Use residual plots to look for error patterns rather than relying only on an average.
Classification: ranking, calibration and decisions¶
The script chooses among four thresholds on validation data. Its illustrative cost assigns three units to a missed event and one unit to a false alert. The selected threshold is 0.2. The test confusion matrix is [[6, 486], [1, 507]], with actual classes in rows and predicted classes in columns. Test average precision is 0.643.
A precision–recall curve shows the tradeoff as the score threshold changes. Compare it with the positive event rate and the capacity of the review team. It does not establish calibration.
Calibration compares predicted probabilities with observed event frequencies in bins. Small bins make the plot unstable. The 1,000 test rows come from a synthetic distribution; the plot alone does not establish calibration in production.
The confusion matrix describes decisions at one chosen threshold. Change the cost assumptions or capacity and repeat threshold selection on validation data. Keep the test set independent of that selection.
Inspect the tree¶
The tree is rendered from ModelTree.to_dot(). Node conditions describe the fitted partitions. Check feature names, missing routing and leaf support before using them in a report. Categorical conditions can display internal codes, so check their meaning before sharing a rule. A prediction path does not establish the effect of an intervention.
Split importance summarizes gain used by the fitted tree. It is not a causal effect and does not measure every source of predictive value. The visualization guide shows how to select and render each view.
What remains¶
The current metrics guide catalogs many report functions, but it does not provide a complete task-by-task evaluation workflow. Survival, uplift, clustering and anomaly examples still need validated task-specific evaluation and uncertainty examples. Check the API before assuming that a recommended metric is implemented by this package.