ELO Benchmark Arenas¶
One scoreboard, one protocol, every model scored on the same prepared data. The arena evaluates models across Rust, Python, and R under a strict 300s process-group timeout.
Dedicated Arena Boards¶
- Overall Arena (56 Datasets): Combined cumulative Elo across 18 Regression + 38 Classification datasets.
- Regression Arena (18 Datasets): Interactive RMSE leaderboard with sticky column freezes and winner badges.
- Classification Arena (38 Datasets): Interactive leaderboards broken down by Accuracy, LogLoss, Binary-only, and Multiclass-only.
- Benchmark Datasets Index: OpenML suites, row/column counts, and feature metadata.
The Protocols¶
1. Regression Arena¶
- Data: 18 regression benchmark datasets, one stratified 80/20 split (seed 7), prepared once by
examples/elo_prepare.py. - Matchups: Per dataset, every unordered pair of models "plays" once; lower test RMSE wins (tie = draw).
- Scores: Evaluates 36 models across Rust, Python, and R (
benchmarks/elo_scores.csv). - Reproduce:
uv run python examples/elo_report.py(orbash examples/elo_run_resume.sh).
2. Classification Arena¶
- Data: 38 TabArena datasets (30 binary, 8 multiclass) from OpenML suite 457, stratified 80/20 split within class (seed 7), prepared once by
examples/elo_prepare_cla.pywith raw strings for categoricals and<ds>.meta.json. - Matchups: Per dataset, round-robin pairwise comparisons based on test Accuracy and Log Loss (cross-entropy), K=32, starting rating 1000.
- Scores: Evaluates 32 models across Python, Rust, and R (
benchmarks/elo_cla_scores.csv), including RuleQuestC5.0 (tree)andC5.0 (rules)in R,HistGradientBoosting,sklearn GBM,LightGBM,CatBoost,CTBoost,ChimeraBoost,EBM,ranger,bart,glmtree, and all 4ModelTreevariants (all,path,mean,winnow). - Reproduce:
uv run python examples/elo_report_cla.py(orbash examples/elo_run_cla.sh).
Metadata Taxonomy¶
Every score tracked in the arena includes two core architectural dimensions:
- lang: Model implementation language — Rust, Python, or R.
- cat_handling:
- internal (native / as-is): Direct native consumption of categoricals without driver encoding (ModelTree, LightGBM, CatBoost, CTBoost, ChimeraBoost, EBM, HistGradientBoosting, modeltrees, R factors like rpart, cubist, C5.0, ctree, glmtree).
- external (driver-encoded): Requires driver-side ordinal encoding fit on train (sklearn RF, sklearn GBM, ExtraTrees, sklearn tree, forust, perpetual, MARS).
- none (numeric-only): Strictly requires numeric matrices; refuses categorical columns (statsmodels, trust-free, R grf).
The Regression Board¶
== Elo (round-robin on per-dataset RMSE, K=32, start 1000) ==
+-----------+---------------------+------------------------+--------+------------+-----------+
| ELO | FIT TIME (±sd) | MODEL | LANG | CAT | DATA |
+-----------+---------------------+------------------------+--------+------------+-----------+
| 1690.0 | 12.01 s (±17.76) | ChimeraBoost | Python | internal | 18/18 |
| 1622.3 | 18.24 s (±23.58) | ranger | R | internal | 16/18 |
| 1618.1 | 19.20 s (±33.35) | sklearn RF | Python | external | 18/18 |
| 1568.1 | 13.02 s (±33.28) | ExtraTrees | Python | external | 18/18 |
| 1532.3 | 23.17 s (±31.44) | grf_llf | R | none | 16/18 |
| 1470.6 | 0.35 s (±0.22) | LightGBM(200) | Python | internal | 18/18 |
| 1439.3 | 7.20 s (±23.69) | perpetual (Rust) | Rust | external | 17/18 |
| 1425.5 | 1.04 s (±1.10) | CatBoost(200) | Python | internal | 18/18 |
| 1350.1 | 0.95 s (±1.91) | HistGradientBoosting | Python | internal | 17/18 |
| 1319.3 | 65.08 s (±68.98) | EBM | Python | internal | 15/18 |
| 1299.2 | 1.22 s (±0.86) | forust (Rust) | Rust | external | 18/18 |
| 1210.8 | 0.76 s (±1.83) | CTBoost | Python | internal | 17/18 |
| 1168.4 | 27.91 s (±39.58) | bart | R | internal | 17/18 |
| 1166.7 | 18.06 s (±23.80) | grf_rf | R | none | 18/18 |
| 1050.1 | 13.77 s (±20.73) | ModelTree(all) | Rust | internal | 16/18 |
| 1030.9 | 7.00 s (±13.24) | sklearn GBM | Python | external | 18/18 |
| 976.4 | 20.22 s (±47.48) | lmtree | R | internal | 14/18 |
| 972.9 | 15.29 s (±51.54) | modeltrees lmtree | Python | internal | 14/18 |
| 958.1 | 29.02 s (±57.11) | glmtree | R | internal | 14/18 |
| 928.3 | 5.95 s (±8.90) | ModelTree(winnow) | Rust | internal | 16/18 |
| 892.7 | 6.23 s (±15.19) | cubist | R | internal | 17/18 |
| 873.1 | 20.49 s (±31.78) | MARS (mars-earth) | Python | external | 10/18 |
| 761.5 | 20.60 s (±51.49) | mboost | R | internal | 15/18 |
| 697.6 | 0.33 s (±0.68) | statsmodels OLS | Python | none | 8/18 |
| 690.7 | 10.19 s (±27.50) | earth | R | internal | 17/18 |
| 681.8 | 0.09 s (±0.10) | KNN | Python | external | 18/18 |
| 655.7 | 21.17 s (±59.09) | ctree | R | internal | 17/18 |
| 611.2 | 14.37 s (±7.33) | trust-free | Python | none | 4/18 |
| 584.5 | 0.09 s (±0.23) | sklearn Linear | Python | external | 18/18 |
| 547.3 | 10.80 s (±21.74) | ModelTree(path) | Rust | internal | 18/18 |
| 539.8 | 0.34 s (±0.57) | sklearn tree | Python | external | 18/18 |
| 527.1 | 17.48 s (±43.02) | statsmodels RLM | Python | none | 8/18 |
| 482.0 | 5.34 s (±20.14) | ModelTree(mean) | Rust | internal | 18/18 |
| 368.3 | 22.52 s (±59.15) | statsmodels QuantReg | Python | none | 7/18 |
| 289.3 | 1.34 s (±3.71) | rpart | R | internal | 17/18 |
+-----------+---------------------+------------------------+--------+------------+-----------+
Interactive Regression Board (18 Datasets × 35 Models)¶
Click any column header to sort. The left key columns stay frozen while scrolling horizontally. Green cells indicate the best RMSE on that dataset; blue indicates the fastest model; orange indicates the Elo leader.
{% include-markdown "benchmarks/elo_board.md" %}
The Classification Board¶
== Elo (round-robin on per-dataset LogLoss & Accuracy, K=32, start 1000) ==
+---------------+------------+----------------+--------------------------+---------+----------+
| ELO (LogLoss) | ELO (Acc) | FIT TIME (±sd) | MODEL | LANG | DATASETS |
+---------------+------------+----------------+--------------------------+---------+----------+
| 1473.4 | 1357.9 | 31.47 s | EBM | Python | 22/24 |
| 1468.1 | 1380.6 | 8.02 s | perpetual (Rust) | Rust | 15/24 |
| 1440.8 | 1294.4 | 2.56 s | CatBoost(200) | Python | 23/24 |
| 1435.0 | 1108.4 | 2.37 s | ChimeraBoost | Python | 23/24 |
| 1429.7 | 1236.3 | 8.57 s | bart | R | 15/24 |
| 1360.8 | 1230.8 | 0.33 s | CTBoost | Python | 21/24 |
| 1343.8 | 1042.3 | 8.04 s | sklearn GBM | Python | 23/24 |
| 1292.2 | 903.4 | 10.37 s | grf_rf | R | 7/24 |
| 1288.2 | 1244.2 | 7.60 s | ranger | R | 21/24 |
| 1226.1 | 1032.0 | 35.68 s | MARS (mars-earth) | Python | 8/24 |
| 1170.4 | 996.1 | 2.10 s | ctree | R | 20/24 |
| 1138.0 | 894.8 | 8.07 s | mboost | R | 15/24 |
| 1084.6 | 884.1 | 0.41 s | rpart | R | 22/24 |
| 1065.6 | 1002.2 | 1.15 s | forust (Rust) | Rust | 19/24 |
| 1049.8 | 1096.1 | 2.69 s | sklearn RF | Python | 23/24 |
| 995.1 | 759.4 | 0.03 s | statsmodels MNLogit | Python | 5/24 |
| 994.5 | 1317.2 | 0.71 s | HistGradientBoosting | Python | 21/24 |
| 965.3 | 787.4 | 3.27 s | c50_tree | R | 24/24 |
| 954.9 | 934.2 | 16.04 s | modeltrees multinomtree | Python | 9/24 |
| 950.7 | 989.9 | 1.21 s | ExtraTrees | Python | 23/24 |
| 921.6 | 1280.8 | 0.60 s | LightGBM(200) | Python | 23/24 |
| 891.5 | 737.2 | 2.44 s | sklearn LogReg | Python | 23/24 |
| 853.5 | 1142.3 | 0.70 s | ModelTree(path) | Rust | 18/24 |
| 787.4 | 1173.3 | 0.43 s | ModelTree(mean) | Rust | 19/24 |
| 784.0 | 760.9 | 26.71 s | modeltrees glmtree | Python | 14/24 |
| 763.2 | 674.4 | 1.74 s | earth | R | 16/24 |
| 762.9 | 700.1 | 22.37 s | glmtree | R | 14/24 |
| 745.1 | 997.1 | 6.07 s | ModelTree(winnow) | Rust | 18/24 |
| 698.5 | 1056.1 | 7.42 s | ModelTree(all) | Rust | 16/24 |
| 379.4 | 683.6 | 0.13 s | KNN | Python | 23/24 |
| 307.8 | 838.1 | 3.27 s | c50_rules | R | 24/24 |
| -21.7 | 464.4 | 0.21 s | sklearn tree | Python | 23/24 |
+---------------+------------+----------------+--------------------------+---------+----------+
Interactive Classification Board (24 Datasets × 32 Models)¶
Click any column header to sort. The left key columns stay frozen while scrolling horizontally. Green cells indicate the best LogLoss on that dataset; blue indicates the fastest model; orange indicates the Elo leader.
{% include-markdown "benchmarks/elo_cla_board.md" %}
Reading the Boards¶
- Stock configs only: All engines evaluated under their default configurations across all datasets.
- Single Model Trees vs Ensembles:
ModelTree(all)scores 1050.1 in regression andModelTree(path)scores 853.5 in classification, clearly outperforming single-tree model tree baselines like Rglmtree(762.9, 22.37s) and Pythonmodeltrees glmtree(784.0, 26.71s) while fitting in 0.70s (38× faster).- In pure accuracy,
ModelTree(mean)achieves Elo Acc 1173.3, outperformingsklearn RF(1096.1),sklearn GBM(1042.3), andc50_tree(787.4). - Time Tiers:
FIT TIME: Average fit seconds across all scored datasets.DATASETS: Datasets where the model completed within the strict 180s budget (timeouts or numerical failures forfeit matchups).
Development rounds (ours vs the frozen field)¶
The external field (R packages, LightGBM, CatBoost, CTBoost, forust,
perpetual, MARS, sklearn, modeltrees) is scored once and frozen in
benchmarks/elo_baselines.csv — where we started. Every development round
re-scores only the ModelTree variants on the same prepared splits:
uv run python examples/elo_round.py # all datasets
ELO_DATASETS=abalone,boston uv run python examples/elo_round.py # a subset
uv run python examples/elo_round.py --freeze # deliberately re-anchor
Each round prints started (frozen baseline) -> reached Elo for every
ModelTree flavor, with per-fit timings in the scores file — so progress across
iterations is visible instead of silently overwritten. The full from-scratch
rebuild (all models, groups of 5 datasets, shareable intermediate state after
each group) is bash examples/elo_run_groups.sh.
The field's raw scores (per dataset RMSE/MAE/fit time) live in
benchmarks/elo_scores.csv;
the longer research log is in docs/field-notes.md.
Running a round (repeatable, with provenance)¶
One command re-scores our four flavors against the frozen field:
uv run python examples/elo_round.py # all 18 datasets
ELO_DATASETS=abalone,boston uv run python examples/elo_round.py # subset
Every round records what created its numbers:
benchmarks/rounds/<stamp>/settings.json— the exact settings dict each flavor was fit with (single-sourced: the manifest records the same dictvariants()builds from, so record and reality cannot drift), the git SHA, the dataset list, every result row, and Elo before → after.benchmarks/elo_round.log— stdlibloggingoutput (file + console), one line per round:timestamp, stamp, sha, dataset count, per-model Elo before->after. This is the first place to look for "what did we change and what did it move".benchmarks/elo_scores.csv— the live board (our rows upserted);benchmarks/elo_baselines.csv— the frozen anchor ("where we started").
Tuned settings from examples/hpo_optuna.py print in the same dict shape —
paste them into SETTINGS in elo_round.py to score a tuned flavor.