Skip to content

API reference — Metrics

Complete evaluation reports at once. Guides: Metrics, Evaluation reports.

Metrics

pythonclass Metrics(()) One metrics class computing complete reports at once — the yardstick every learner is measured with. Task: both. Pairs with a missing/NaN side are dropped from the math and reported in n_missing, never hidden.

Metrics.brier_score

pythonbrier_score((self, /, y_true, probs)) Brier score of probability predictions. Task: classification.

Args: y_true: targets (positive = > 0.5). probs: predicted probabilities.

Returns: mean squared probability error.

Metrics.calibration

pythoncalibration((self, /, y_true, scores, n_bins=10)) Calibration (reliability) bins: mean predicted score vs observed frequency per bin. Task: classification.

Args: y_true: targets (positive = > 0.5). scores: predicted scores/probabilities. n_bins: equal-width bins over [0, 1] (default 10).

Returns: {"bins": [{"left", "right", "mean_score", "observed", "n"}]} (empty bins omitted; scores clip into edge bins).

Metrics.classification

pythonclassification((self, /, y_true, y_pred, probabilities=None)) Complete classification report: accuracy, macro averages, per-class precision/recall/f1/support, confusion matrix — and with probabilities (P(class 1)) also logloss, brier and ROC-AUC.

Args: y_true: integer class labels (None/NaN = missing). y_pred: predicted labels (None/NaN = missing). probabilities: P(class 1) per row; enables the probabilistic metrics.

Returns: the full report as a dict.

Raises: ValueError: mismatched lengths, non-integer labels or no observed pairs.

Metrics.classification_by_group

pythonclassification_by_group((self, /, keys, y_true, y_pred)) Classification report per group key (sorted by key). Task: classification.

Args: keys: group label per row (rows with missing key/labels skip). y_true: integer class labels. y_pred: integer class labels.

Returns: {group: classification_report}.

Metrics.classification_scores

pythonclassification_scores((self, /, y_true, y_pred)) Single-label classification scorecard (binary or multiclass). Task: classification.

Args: y_true: integer-like labels. y_pred: integer-like labels.

Returns: {"n", "n_classes", "accuracy", "balanced_accuracy", "precision_macro/micro/weighted", "recall_*", "f1_*", "mcc", "jaccard_macro/micro/weighted", "hamming_loss", "zero_one_loss"}.

Metrics.competing_risks

pythoncompeting_risks((self, /, durations, events, cif_curves=None, risk_scores=None, times=None)) Complete competing risks report for multi-cause survival endpoints: cause-specific Harrell's C, Uno's IPCW C, cause-specific IBS and time-dependent Brier scores over specified or sample decile evaluation horizons.

Args: durations: observed follow-up times. events: event indicators (0 = censored, 1..J = mutually exclusive causes). cif_curves: optional CIF curves mapping {cause: (N, T)} or 3D list (N, n_causes, T). risk_scores: optional risk scores mapping {cause: (N,)} or 2D list (N, n_causes). times: optional evaluation horizon times grid.

Metrics.competing_risks_calibration

pythoncompeting_risks_calibration((self, /, durations, events, predicted_cif, horizon)) Competing risks calibration report across causes (Alberge et al. 2026).

Metrics.d_calibration

pythond_calibration((self, /, durations, events, surv_at_durations, n_bins=10)) D-Calibration test (Haider et al. 2020) for survival distribution uniformity.

Metrics.decile_table

pythondecile_table((self, /, y_true, scores, n_buckets=10)) Decile profile (kds/PerfScoreCard decile table; modelplotpy response at decile granularity). Task: classification.

Args: y_true: targets (positive = > 0.5). scores: predicted scores/probabilities. n_buckets: equal-count buckets by score descending (default 10).

Returns: [{"decile", "n", "avg_score", "events", "event_rate", "cum_capture", "cum_event_rate", "lift"}, ...] (decile 1 = top).

Metrics.double_lift

pythondouble_lift((self, /, y_true, scores_a, scores_b, n_buckets=10)) Double-lift table: actual vs model A vs model B along A's ranking (CAS monograph chart; additive vs ratio display is a plotting concern). Task: regression or classification.

Args: y_true: targets. scores_a: model A predictions (defines the ranking). scores_b: model B predictions. n_buckets: equal-count buckets (default 10).

Returns: [{"bucket", "n", "actual", "pred_a", "pred_b"}, ...].

Metrics.gain_curve

pythongain_curve((self, /, y_true, scores)) Cumulative gains curve. Task: classification.

Args: y_true: targets (positive = > 0.5). scores: predicted scores/probabilities.

Returns: {"points": [[population_fraction, capture_rate], ...]} (lift at a point = capture / fraction).

Metrics.hinge_loss

pythonhinge_loss((self, /, y_true, decision)) Hinge loss over signed decision margins. Task: classification.

Args: y_true: targets (positive = > 0.5). decision: decision-function margins (signed).

Returns: mean hinge loss.

Metrics.ici

pythonici((self, /, durations, events, predicted_survival, horizon)) Alias for integrated_calibration_index.

Metrics.integrated_calibration_index

pythonintegrated_calibration_index((self, /, durations, events, predicted_survival, horizon)) Integrated Calibration Index (Austin et al. 2020) at fixed horizon.

Metrics.ks_statistic

pythonks_statistic((self, /, y_true, scores)) KS sweep + statistic + Score@KS. Task: classification.

Args: y_true: targets (positive = > 0.5). scores: predicted scores/probabilities.

Returns: {"points": [[fpr, tpr, score], ...], "ks", "score_at_ks", "tpr_at_ks", "fpr_at_ks"}.

Metrics.lift_curve

pythonlift_curve((self, /, y_true, scores)) Cumulative lift curve. Task: classification.

Args: y_true: targets (positive = > 0.5). scores: predicted scores/probabilities.

Returns: {"points": [[population_fraction, lift], ...]} (lift = capture / population fraction).

Metrics.log_loss

pythonlog_loss((self, /, y_true, probs)) Log loss of probability predictions. Task: classification.

Args: y_true: targets (positive = > 0.5). probs: predicted probabilities.

Returns: mean log loss.

Metrics.multilabel_scores

pythonmultilabel_scores((self, /, y_true, y_scores)) Multilabel scorecard: rows are instances, columns are labels, entries

0.5 are positive (raw entries double as ranking scores). Task: multilabel classification.

Args: y_true: label matrix (rows x labels, 0/1). y_scores: prediction/score matrix (same shape).

Returns: {"n", "n_labels", "subset_accuracy", "hamming_loss", "precision_micro/macro/samples", "recall_*", "f1_*", "label_ranking_ap", "coverage_error"}.

Metrics.one_calibration

pythonone_calibration((self, /, durations, events, predicted_survival, horizon)) 1-Calibration slope and intercept (Austin et al. 2020) at fixed horizon.

Metrics.pairwise_distances

pythonpairwise_distances((self, /, a, b, metric='euclidean', p=2.0)) Pairwise distances between two row sets. Task: any.

Args: a: first rows (rows x features). b: second rows (same width). metric: "euclidean" | "manhattan" | "chebyshev" | "minkowski" | "cosine" | "correlation" | "hamming" | "jaccard". p: Minkowski power (default 2.0).

Returns: len(a) x len(b) distance matrix.

Metrics.pr_curve

pythonpr_curve((self, /, y_true, scores)) Precision-recall curve + average precision. Task: classification.

Args: y_true: targets (positive = > 0.5). scores: predicted scores/probabilities.

Returns: {"points": [[recall, precision], ...], "average_precision": float}.

Metrics.regression

pythonregression((self, /, y_true, y_pred)) Complete regression report: n, n_missing, mean_error (actual - fitted), bias (mean_pred / mean_true), mae, mad, mape (non-zero targets), rmse, r2, max_error, explained_variance and deciles — the DAR per-target-decile error profile (the bathtub view).

Args: y_true: actual values (None/NaN = missing). y_pred: fitted values (None/NaN = missing).

Returns: the full report as a dict.

Raises: ValueError: mismatched lengths or no observed pairs.

Metrics.regression_by_group

pythonregression_by_group((self, /, keys, y_true, y_pred)) Regression report per group (keys: dates, categories, ...) — the legacy per-key summary lineage. Rows with a missing key are ignored.

Args: keys: group label per row (None = ignored). y_true: actual values (None/NaN = missing). y_pred: fitted values (None/NaN = missing).

Returns: {group: regression report}, sorted by group.

Raises: ValueError: mismatched lengths or a group with no observed pairs.

Metrics.roc_curve

pythonroc_curve((self, /, y_true, scores)) ROC curve + AUC. Task: classification.

Args: y_true: targets (positive = > 0.5). scores: predicted scores/probabilities.

Returns: {"points": [[fpr, tpr], ...], "auc": float}.

Metrics.stability_index

pythonstability_index((self, /, reference, current, n_bins=10)) Stability index (PSI on scores, CSI on a feature column). Task: classification or regression monitoring.

Args: reference: reference-period values (bins derive from their range). current: current-period values. n_bins: equal-width bins (default 10).

Returns: {"psi": float, "bins": [{"left", "right", "reference_share", "current_share", "contribution"}, ...]}.

Metrics.survival

pythonsurvival((self, /, durations, events, risk_scores, surv_curves=None, times=None)) Complete survival analysis report (Harrell C-index, Uno IPCW C-index, Integrated Brier Score).

Args: durations: observed duration / time-to-event per row. events: event indicator (1.0 = event observed, 0.0 = right-censored). risk_scores: predicted risk score (higher score = shorter expected survival). surv_curves: optional individual survival probability curves S(t|x) matrix (N, K). times: optional evaluation horizon times grid.

Metrics.time_series

pythontime_series((self, /, y_true, y_pred, y_train=None, seasonal_period=None, std_pred=None)) Complete time series forecasting report (MAE, RMSE, MASE, sMAPE, WAPE, MDA, CRPS).

Args: y_true: actual time series targets. y_pred: predicted values. y_train: optional in-sample training series (used to scale MASE). seasonal_period: optional seasonality m (e.g. 7 for daily, 12 for monthly). std_pred: optional predicted standard deviation per period (used for Gaussian CRPS).

Metrics.uplift

pythonuplift((self, /, y_true, treatment, tau_pred)) Complete causal uplift and Qini curve report.

Args: y_true: observed outcome per row. treatment: binary treatment indicator (1.0 = treated, 0.0 = control). tau_pred: predicted treatment effect / uplift score per row.

Returns: dict with ATE, Qini coefficient, normalized Qini, AUUC, decile uplifts, Qini curve.

Metrics.weighted_kappa

pythonweighted_kappa((self, /, y_true, y_pred, weights='quadratic')) Cohen's kappa with optional weights over the label scale. Task: classification.

Args: y_true: integer-like labels. y_pred: integer-like labels. weights: "none" | "linear" | "quadratic" (default "quadratic").

Returns: kappa (1 = perfect agreement).