multabench 0.0.1__tar.gz → 0.0.2__tar.gz

{multabench-0.0.1 → multabench-0.0.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: multabench
-Version: 0.0.1
+Version: 0.0.2
 Summary: Benchmarking multimodal tabular learning: tables whose columns include text or images
 Author: Alan Arazi, Eilam Shapira, Shoham Grunblat, Mor Ventura, Elad Hoffer, Gioia Blayer, David Holzmüller, Lennart Purucker, Gaël Varoquaux, Frank Hutter, Roi Reichart
 License-Expression: MIT
@@ -42,9 +42,17 @@ Requires-Dist: tabdpt>=1.1.9; extra == "tabdpt"
 Provides-Extra: tabpfn
 Requires-Dist: tabpfn>=9.2.0; extra == "tabpfn"
 Provides-Extra: tabstar
-Requires-Dist: tabstar>=1.1.15; extra == "tabstar"
+Requires-Dist: tabstar>=1.2.1; extra == "tabstar"
+Provides-Extra: tabfm
+Requires-Dist: tabpfn-extensions[many_class]>=0.6.1; extra == "tabfm"
+Provides-Extra: causilo
+Requires-Dist: causilo==1.0.2; extra == "causilo"
+Provides-Extra: tabdpt13
+Requires-Dist: tabdpt==1.3.1; extra == "tabdpt13"
 Provides-Extra: autogluon-mm
 Requires-Dist: autogluon.multimodal; extra == "autogluon-mm"
+Provides-Extra: leaderboard
+Requires-Dist: bencheval==0.1.1.dev20261015010754; extra == "leaderboard"
 Provides-Extra: dev
 Requires-Dist: matplotlib; extra == "dev"
 Requires-Dist: pytest; extra == "dev"
@@ -76,32 +84,51 @@ than used frozen.
 
 ## Leaderboard
 
+The leaderboard lives at [huggingface.co/spaces/multabench/leaderboard](https://huggingface.co/spaces/multabench/leaderboard),
+with per-dataset results and cost; `space/` holds its app and data.
+
 Default hyperparameters, `--size 10k`, 6 folds, frozen `e5-small` text and `dino-small` image embeddings
-(ConTextTab reads raw text itself). Elo is fit on one battle per dataset and model pair, with RandomForest
-at 1000 and 95% intervals from bootstrapping the datasets; a model with no result on a dataset has no
-battles there. TabPFN-v2 and v2.5 take at most 10 classes, which leaves out 3 of the 80 datasets.
+(ConTextTab and TabSTAR read raw text themselves; TabSTAR loads, for each dataset it was pretrained on, the
+checkpoint that held that dataset out). TabPFN-v2 and v2.5 take at most 10 classes, which leaves out 3 of the 80 datasets.
+
+**How Elo is computed.** With TabArena's own evaluation package, [bencheval](https://pypi.org/project/bencheval/),
+pinned in the `leaderboard` extra. Every fold is its own battle: on each of a dataset's 6 folds, each pair of models
+plays one battle, won by the lower test error on that fold (a tie counts half). The battles are fit as a Bradley–Terry
+model, with RandomForest anchored at 1000. The 95% intervals come from 1,000 bootstrap resamples of the datasets, each
+drawn with all 6 of its folds, so the intervals reflect how many datasets there are rather than how many folds. They
+are bootstrapped before anchoring, so each model's interval is its own, RandomForest's included, rather than relative
+to RandomForest. A model with no result on a fold has no battles there; nothing is imputed. Win rate, average rank, normalized
+error and best-on count are per fold as well. Until October 2026 the leaderboard, like the paper, played one battle
+per dataset on the mean error over its folds. That makes each dataset winner-take-all, which stretches the scale: on
+the then 17 models, Kumo-Tabular was at 2670 rather than 2347, with the same ranking.
 
 | # | Model | Elo, 80 datasets | 95% CI | Text, 40 | Images, 40 | Datasets |
 |---|---|---|---|---|---|---|
-| 1 | TabPFN-3.5 | 2221 | +230 / −176 | 2080 | 2458 | 80 |
-| 2 | Kumo-Tabular-Small | 2139 | +190 / −151 | 2094 | 2268 | 80 |
-| 3 | TabPFN-3.5-Fast | 2010 | +195 / −149 | 1906 | 2199 | 80 |
-| 4 | TabICLv2 | 1745 | +137 / −115 | 1637 | 1941 | 80 |
-| 5 | TabPFN-3 | 1666 | +145 / −117 | 1554 | 1866 | 80 |
-| 6 | TabPFN-v2p5 | 1661 | +136 / −114 | 1647 | 1743 | 77 |
-| 7 | TabPFN-v2 | 1407 | +117 / −97 | 1322 | 1559 | 77 |
-| 8 | TabM | 1366 | +98 / −89 | 1348 | 1432 | 80 |
-| 9 | ConTextTab | 1353 | +113 / −102 | 1279 | 1490 | 80 |
-| 10 | CatBoost | 1298 | +66 / −55 | 1307 | 1321 | 80 |
-| 11 | TabDPT | 1251 | +112 / −93 | 1116 | 1455 | 80 |
-| 12 | RealMLP | 1082 | +81 / −71 | 1103 | 1059 | 80 |
-| 13 | LightGBM | 1058 | +63 / −61 | 1093 | 1013 | 80 |
-| 14 | RandomForest | 1000 | anchor | 1000 | 1000 | 80 |
-| 15 | XGBoost | 988 | +67 / −65 | 1024 | 936 | 80 |
+| 1 | Kumo-Tabular | 2279 | +108 / −84 | 2310 | 2334 | 80 |
+| 2 | Kumo-Tabular-Medium | 2126 | +86 / −72 | 2141 | 2190 | 80 |
+| 3 | TabFM | 2057 | +86 / −63 | 2044 | 2141 | 80 |
+| 4 | TabPFN-3.5 | 1991 | +78 / −69 | 1913 | 2132 | 80 |
+| 5 | Kumo-Tabular-Small | 1937 | +56 / −46 | 1915 | 2026 | 80 |
+| 6 | Causilo | 1878 | +51 / −43 | 1826 | 1993 | 80 |
+| 7 | TabPFN-3.5-Fast | 1861 | +61 / −54 | 1782 | 2002 | 80 |
+| 8 | TabICLv2 | 1648 | +37 / −35 | 1564 | 1792 | 80 |
+| 9 | TabPFN-3 | 1586 | +37 / −36 | 1488 | 1747 | 80 |
+| 10 | TabPFN-v2p5 | 1586 | +34 / −38 | 1565 | 1660 | 77 |
+| 11 | TabPFN-v2 | 1373 | +45 / −50 | 1291 | 1505 | 77 |
+| 12 | ConTextTab | 1339 | +56 / −63 | 1270 | 1455 | 80 |
+| 13 | TabM | 1328 | +41 / −45 | 1311 | 1378 | 80 |
+| 14 | CatBoost | 1268 | +42 / −48 | 1274 | 1283 | 80 |
+| 15 | TabDPT | 1231 | +62 / −65 | 1106 | 1400 | 80 |
+| 16 | TabSTAR | 1201 | +68 / −71 | 1241 | 1165 | 80 |
+| 17 | RealMLP | 1093 | +57 / −69 | 1100 | 1090 | 80 |
+| 18 | LightGBM | 1052 | +64 / −73 | 1081 | 1016 | 80 |
+| 19 | RandomForest | 1000 | +80 / −87 | 1000 | 1000 | 80 |
+| 20 | XGBoost | 997 | +69 / −78 | 1029 | 953 | 80 |
 
 Each model's runs are in `multabench/leaderboard/models/<model>.csv`, one row per (dataset, fold), failed
-runs included. `python -m multabench.leaderboard.elo` prints the table, with win rate, average rank and
-normalized error; `--modalities text` or `--modalities image,image-text` restricts it to those datasets.
+runs included. `python -m multabench.leaderboard.elo` prints the table, with win rate, average rank, normalized
+error and the number of folds each model is best on; `--modalities text` or `--modalities image,image-text` restricts
+it to those datasets.
 
 Leaderboard runs pass `--official`, which also requires a clean checkout of a commit on `master`, so the `git`
 column of every row names code anyone can check out. The image runs of the first 11 models are the one exception: they
@@ -116,11 +143,18 @@ pip install multabench
 ```
 
 ```python
-from multabench import Size, load_embeddings, load_split
+from multabench import Size, list_datasets, load_embeddings, load_published_results, load_split, score
+datasets = list_datasets(modality="text", task="regression", tier="core")  # DatasetInfo: id, hf_repo, modality, ...
 split = load_split("MUL_IMAGE_PETFINDER", fold=0, size=Size.TEN_K)  # split.x_train, split.y_train, split.x_test, ...
 embeddings = load_embeddings("MUL_TEXT_WINE_REVIEW", fold=0, size=Size.TEN_K, encoder="e5-small")
+result = score(split.task_type, y_true=y_test, y_pred=proba)  # metric, test_score, test_error, as the leaderboard
+published = load_published_results()  # every leaderboard run, in the columns of multabench/leaderboard/models/
 ```
 
+Each package version reads every dataset and embeddings repo at the commit pinned in
+`multabench/datasets/revisions.json`, and ships the leaderboard's runs as of its release. After uploading to one of
+those repos, `python -m multabench.datasets.revisions` pins them all to their current `main`.
+
 To run the benchmark:
 
 ```bash
@@ -132,8 +166,11 @@ python benchmark.py --model light --dataset_name MUL_IMAGE_PETFINDER --fold 0 --
 
 `pyproject.toml` lists the core dependencies, a `bench` extra for `benchmark.py` and one extra per model package, so
 `pip install -e ".[bench,catboost]"` installs what one model needs; each wrapper imports its package only when it runs.
-ConTextTab and Kumo are pinned to a git commit, which PyPI doesn't allow in extras, so `requirements-git.txt` lists them.
-`init.sh` sets up the shared `.venv` from `requirements.txt`, which locks the core, every extra and those two.
+ConTextTab, Kumo and TabFM are pinned to a git commit, which PyPI doesn't allow in extras, so `requirements-git.txt`
+lists them. `init.sh` sets up the shared `.venv` from `requirements.txt`, which locks the core, every extra and those
+three. A model whose requirements conflict with that lock declares its own venv in its wrapper (`ENV`), locked in
+`requirements-<ENV>.txt` and set up with `source init.sh <ENV>`: Causilo, which needs a newer torch, runs in
+`.venv-causilo`. `sweep.py` writes each model's venv python into its jobs, so one sweep covers every model.
 
 `benchmark.py` is the single entry point: it evaluates one model on one dataset fold and writes the
 result to `runs/<model>_<dataset>_<text_encoder>_<image_encoder>_<size>_<fold>.json` (`--output_dir` to change), with
@@ -265,7 +302,7 @@ MulTaBench is not an official TabArena leaderboard.
 | | TabArena | MulTaBench | Status |
 |---|---|---|---|
 | Metrics | ROC AUC (binary), log loss (multiclass), RMSE (regression), via AutoGluon's scorers | Same | ✅ |
-| Leaderboard | Elo via `bencheval`, RandomForest (default) anchored at 1000, missing results imputed with RandomForest and flagged, bootstrapped confidence intervals | Elo as in the paper, RandomForest anchored at 1000, bootstrapped confidence intervals; missing results are left out, neither imputed nor dropping the model, so a model is ranked on the datasets it ran on, which its Datasets column shows | |
+| Leaderboard | Elo via `bencheval`, one battle per task fold, RandomForest (default) anchored at 1000, missing results imputed with RandomForest and flagged, bootstrapped confidence intervals | Same Elo, one battle per dataset fold; missing results are left out, neither imputed nor dropping the model, so a model is ranked on the datasets it ran on, which its Datasets column shows | |
 | Splits | Fixed and part of the task definition | Same: a seeded function of the data | ✅ |
 | Group and time structure | Audited per dataset; group-aware or forward-in-time splits where needed | Not audited: every dataset is split IID. Some datasets may have group or time structure, in which case IID splits can overestimate performance | |
 | Runtime | Train and inference time per 1K rows | Same, with the time spent embedding text and images reported separately | ✅ |

{multabench-0.0.1 → multabench-0.0.2}/README.md

@@ -23,32 +23,51 @@ than used frozen.
 
 ## Leaderboard
 
+The leaderboard lives at [huggingface.co/spaces/multabench/leaderboard](https://huggingface.co/spaces/multabench/leaderboard),
+with per-dataset results and cost; `space/` holds its app and data.
+
 Default hyperparameters, `--size 10k`, 6 folds, frozen `e5-small` text and `dino-small` image embeddings
-(ConTextTab reads raw text itself). Elo is fit on one battle per dataset and model pair, with RandomForest
-at 1000 and 95% intervals from bootstrapping the datasets; a model with no result on a dataset has no
-battles there. TabPFN-v2 and v2.5 take at most 10 classes, which leaves out 3 of the 80 datasets.
+(ConTextTab and TabSTAR read raw text themselves; TabSTAR loads, for each dataset it was pretrained on, the
+checkpoint that held that dataset out). TabPFN-v2 and v2.5 take at most 10 classes, which leaves out 3 of the 80 datasets.
+
+**How Elo is computed.** With TabArena's own evaluation package, [bencheval](https://pypi.org/project/bencheval/),
+pinned in the `leaderboard` extra. Every fold is its own battle: on each of a dataset's 6 folds, each pair of models
+plays one battle, won by the lower test error on that fold (a tie counts half). The battles are fit as a Bradley–Terry
+model, with RandomForest anchored at 1000. The 95% intervals come from 1,000 bootstrap resamples of the datasets, each
+drawn with all 6 of its folds, so the intervals reflect how many datasets there are rather than how many folds. They
+are bootstrapped before anchoring, so each model's interval is its own, RandomForest's included, rather than relative
+to RandomForest. A model with no result on a fold has no battles there; nothing is imputed. Win rate, average rank, normalized
+error and best-on count are per fold as well. Until October 2026 the leaderboard, like the paper, played one battle
+per dataset on the mean error over its folds. That makes each dataset winner-take-all, which stretches the scale: on
+the then 17 models, Kumo-Tabular was at 2670 rather than 2347, with the same ranking.
 
 | # | Model | Elo, 80 datasets | 95% CI | Text, 40 | Images, 40 | Datasets |
 |---|---|---|---|---|---|---|
-| 1 | TabPFN-3.5 | 2221 | +230 / −176 | 2080 | 2458 | 80 |
-| 2 | Kumo-Tabular-Small | 2139 | +190 / −151 | 2094 | 2268 | 80 |
-| 3 | TabPFN-3.5-Fast | 2010 | +195 / −149 | 1906 | 2199 | 80 |
-| 4 | TabICLv2 | 1745 | +137 / −115 | 1637 | 1941 | 80 |
-| 5 | TabPFN-3 | 1666 | +145 / −117 | 1554 | 1866 | 80 |
-| 6 | TabPFN-v2p5 | 1661 | +136 / −114 | 1647 | 1743 | 77 |
-| 7 | TabPFN-v2 | 1407 | +117 / −97 | 1322 | 1559 | 77 |
-| 8 | TabM | 1366 | +98 / −89 | 1348 | 1432 | 80 |
-| 9 | ConTextTab | 1353 | +113 / −102 | 1279 | 1490 | 80 |
-| 10 | CatBoost | 1298 | +66 / −55 | 1307 | 1321 | 80 |
-| 11 | TabDPT | 1251 | +112 / −93 | 1116 | 1455 | 80 |
-| 12 | RealMLP | 1082 | +81 / −71 | 1103 | 1059 | 80 |
-| 13 | LightGBM | 1058 | +63 / −61 | 1093 | 1013 | 80 |
-| 14 | RandomForest | 1000 | anchor | 1000 | 1000 | 80 |
-| 15 | XGBoost | 988 | +67 / −65 | 1024 | 936 | 80 |
+| 1 | Kumo-Tabular | 2279 | +108 / −84 | 2310 | 2334 | 80 |
+| 2 | Kumo-Tabular-Medium | 2126 | +86 / −72 | 2141 | 2190 | 80 |
+| 3 | TabFM | 2057 | +86 / −63 | 2044 | 2141 | 80 |
+| 4 | TabPFN-3.5 | 1991 | +78 / −69 | 1913 | 2132 | 80 |
+| 5 | Kumo-Tabular-Small | 1937 | +56 / −46 | 1915 | 2026 | 80 |
+| 6 | Causilo | 1878 | +51 / −43 | 1826 | 1993 | 80 |
+| 7 | TabPFN-3.5-Fast | 1861 | +61 / −54 | 1782 | 2002 | 80 |
+| 8 | TabICLv2 | 1648 | +37 / −35 | 1564 | 1792 | 80 |
+| 9 | TabPFN-3 | 1586 | +37 / −36 | 1488 | 1747 | 80 |
+| 10 | TabPFN-v2p5 | 1586 | +34 / −38 | 1565 | 1660 | 77 |
+| 11 | TabPFN-v2 | 1373 | +45 / −50 | 1291 | 1505 | 77 |
+| 12 | ConTextTab | 1339 | +56 / −63 | 1270 | 1455 | 80 |
+| 13 | TabM | 1328 | +41 / −45 | 1311 | 1378 | 80 |
+| 14 | CatBoost | 1268 | +42 / −48 | 1274 | 1283 | 80 |
+| 15 | TabDPT | 1231 | +62 / −65 | 1106 | 1400 | 80 |
+| 16 | TabSTAR | 1201 | +68 / −71 | 1241 | 1165 | 80 |
+| 17 | RealMLP | 1093 | +57 / −69 | 1100 | 1090 | 80 |
+| 18 | LightGBM | 1052 | +64 / −73 | 1081 | 1016 | 80 |
+| 19 | RandomForest | 1000 | +80 / −87 | 1000 | 1000 | 80 |
+| 20 | XGBoost | 997 | +69 / −78 | 1029 | 953 | 80 |
 
 Each model's runs are in `multabench/leaderboard/models/<model>.csv`, one row per (dataset, fold), failed
-runs included. `python -m multabench.leaderboard.elo` prints the table, with win rate, average rank and
-normalized error; `--modalities text` or `--modalities image,image-text` restricts it to those datasets.
+runs included. `python -m multabench.leaderboard.elo` prints the table, with win rate, average rank, normalized
+error and the number of folds each model is best on; `--modalities text` or `--modalities image,image-text` restricts
+it to those datasets.
 
 Leaderboard runs pass `--official`, which also requires a clean checkout of a commit on `master`, so the `git`
 column of every row names code anyone can check out. The image runs of the first 11 models are the one exception: they
@@ -63,11 +82,18 @@ pip install multabench
 ```
 
 ```python
-from multabench import Size, load_embeddings, load_split
+from multabench import Size, list_datasets, load_embeddings, load_published_results, load_split, score
+datasets = list_datasets(modality="text", task="regression", tier="core")  # DatasetInfo: id, hf_repo, modality, ...
 split = load_split("MUL_IMAGE_PETFINDER", fold=0, size=Size.TEN_K)  # split.x_train, split.y_train, split.x_test, ...
 embeddings = load_embeddings("MUL_TEXT_WINE_REVIEW", fold=0, size=Size.TEN_K, encoder="e5-small")
+result = score(split.task_type, y_true=y_test, y_pred=proba)  # metric, test_score, test_error, as the leaderboard
+published = load_published_results()  # every leaderboard run, in the columns of multabench/leaderboard/models/
 ```
 
+Each package version reads every dataset and embeddings repo at the commit pinned in
+`multabench/datasets/revisions.json`, and ships the leaderboard's runs as of its release. After uploading to one of
+those repos, `python -m multabench.datasets.revisions` pins them all to their current `main`.
+
 To run the benchmark:
 
 ```bash
@@ -79,8 +105,11 @@ python benchmark.py --model light --dataset_name MUL_IMAGE_PETFINDER --fold 0 --
 
 `pyproject.toml` lists the core dependencies, a `bench` extra for `benchmark.py` and one extra per model package, so
 `pip install -e ".[bench,catboost]"` installs what one model needs; each wrapper imports its package only when it runs.
-ConTextTab and Kumo are pinned to a git commit, which PyPI doesn't allow in extras, so `requirements-git.txt` lists them.
-`init.sh` sets up the shared `.venv` from `requirements.txt`, which locks the core, every extra and those two.
+ConTextTab, Kumo and TabFM are pinned to a git commit, which PyPI doesn't allow in extras, so `requirements-git.txt`
+lists them. `init.sh` sets up the shared `.venv` from `requirements.txt`, which locks the core, every extra and those
+three. A model whose requirements conflict with that lock declares its own venv in its wrapper (`ENV`), locked in
+`requirements-<ENV>.txt` and set up with `source init.sh <ENV>`: Causilo, which needs a newer torch, runs in
+`.venv-causilo`. `sweep.py` writes each model's venv python into its jobs, so one sweep covers every model.
 
 `benchmark.py` is the single entry point: it evaluates one model on one dataset fold and writes the
 result to `runs/<model>_<dataset>_<text_encoder>_<image_encoder>_<size>_<fold>.json` (`--output_dir` to change), with
@@ -212,7 +241,7 @@ MulTaBench is not an official TabArena leaderboard.
 | | TabArena | MulTaBench | Status |
 |---|---|---|---|
 | Metrics | ROC AUC (binary), log loss (multiclass), RMSE (regression), via AutoGluon's scorers | Same | ✅ |
-| Leaderboard | Elo via `bencheval`, RandomForest (default) anchored at 1000, missing results imputed with RandomForest and flagged, bootstrapped confidence intervals | Elo as in the paper, RandomForest anchored at 1000, bootstrapped confidence intervals; missing results are left out, neither imputed nor dropping the model, so a model is ranked on the datasets it ran on, which its Datasets column shows | |
+| Leaderboard | Elo via `bencheval`, one battle per task fold, RandomForest (default) anchored at 1000, missing results imputed with RandomForest and flagged, bootstrapped confidence intervals | Same Elo, one battle per dataset fold; missing results are left out, neither imputed nor dropping the model, so a model is ranked on the datasets it ran on, which its Datasets column shows | |
 | Splits | Fixed and part of the task definition | Same: a seeded function of the data | ✅ |
 | Group and time structure | Audited per dataset; group-aware or forward-in-time splits where needed | Not audited: every dataset is split IID. Some datasets may have group or time structure, in which case IID splits can overestimate performance | |
 | Runtime | Train and inference time per 1K rows | Same, with the time spent embedding text and images reported separately | ✅ |

multabench-0.0.2/multabench/__init__.py

@@ -0,0 +1,7 @@
+from multabench.benchmark.load import load_embeddings, load_split
+from multabench.benchmark.metrics import Score, score
+from multabench.benchmark.splits import Size
+from multabench.datasets.all_datasets import Modality
+from multabench.datasets.info import DatasetInfo, Tier, list_datasets
+from multabench.datasets.objects import SupervisedTask
+from multabench.leaderboard.published import load_published_results

{multabench-0.0.1 → multabench-0.0.2}/multabench/baselines/abstract_model.py

@@ -16,7 +16,8 @@ from multabench.baselines.preprocessing.image_embeddings import (
 )
 from multabench.baselines.training.metrics import calculate_metric
 from multabench.datasets.all_datasets import MulTaBenchDatasetID
-from multabench.baselines.preprocessing.categorical import fit_categorical_encoders, transform_categorical_features
+from multabench.baselines.preprocessing.categorical import (fit_categorical_encoders, fit_category_dtypes,
+                                                            transform_categorical_features)
 from multabench.baselines.preprocessing.numerical import fit_numerical_median, transform_numerical_features
 from multabench.baselines.preprocessing.target import transform_preprocess_y, fit_preprocess_y
 from multabench.dino.constants import DINOV3_SMALL
@@ -35,7 +36,11 @@ class TabularModel:
     USE_CATEGORICAL_ENCODING: bool
     USE_TEXT_EMBEDDINGS: bool
     USE_TARGET_ENCODER: bool
+    # Categorical columns as pandas' category dtype, with the training values as categories, for models that read it.
+    USE_CATEGORY_DTYPE = False
     NEEDS_GPU = True
+    # A model whose requirements conflict with the shared .venv runs in its own, .venv-<ENV>, from requirements-<ENV>.txt.
+    ENV: Optional[str] = None
 
     def __init__(self, problem_type: SupervisedTask, device: torch.device,
                  dataset: MulTaBenchDatasetID | None = None, verbose: bool = False, image_folder: str | None = None,
@@ -117,6 +122,8 @@ class TabularModel:
             )
             self._train_embeddings.update(text_embeddings)
             self.vprint(f"📝 Detected {len(self.text_transformers)} text features: {sorted(self.text_transformers)}")
+        if self.USE_CATEGORY_DTYPE:
+            self.category_dtypes = fit_category_dtypes(x=x_train, categorical_features=self.categorical_features)
         self.fit_internal_preprocessor(x=x_train, y=y_train)
 
     def transform_preprocessor(self, x: DataFrame, y: Optional[Series],
@@ -149,6 +156,8 @@ class TabularModel:
                 device=self.device,
                 train_embeddings=train_embeddings,
             )
+        if self.USE_CATEGORY_DTYPE:
+            x = x.astype(self.category_dtypes)
         return self.transform_internal_preprocessor(x=x, y=y)
 
     def fit_internal_preprocessor(self, x: DataFrame, y: Series) -> Tuple[DataFrame, Series]:

multabench-0.0.2/multabench/baselines/causilo.py

@@ -0,0 +1,26 @@
+from pandas import DataFrame, Series
+
+from multabench.baselines.abstract_model import TabularModel
+from multabench.constants import SEED
+
+
+class Causilo(TabularModel):
+    """Nums AI's Causilo at TabArena's default; it needs torch>=2.13, so it runs in its own venv."""
+
+    MODEL_NAME = "Causilo"
+    SHORT_NAME = "causilo"
+    ENV = "causilo"
+    USE_VAL_SPLIT = False
+    USE_MEDIAN_FILLING = False
+    USE_CATEGORICAL_ENCODING = False
+    USE_TEXT_EMBEDDINGS = True
+    USE_TARGET_ENCODER = True
+    USE_CATEGORY_DTYPE = True
+
+    def initialize_model(self):
+        from causilo import CausiloClassifier, CausiloRegressor
+        cls = CausiloClassifier if self.is_cls else CausiloRegressor
+        return cls(n_estimators=8, random_state=SEED, device=str(self.device))
+
+    def fit_model(self, x_train: DataFrame, y_train: Series, x_val: DataFrame, y_val: Series):
+        self.model_.fit(x_train, y_train)

multabench-0.0.2/multabench/baselines/exaone_tabular.py

@@ -0,0 +1,41 @@
+import numpy as np
+from pandas import DataFrame, Series
+
+from multabench.baselines.abstract_model import TabularModel
+from multabench.constants import SEED
+
+# TabArena reads the weights at main; this is main as of TabArena's runs and ours.
+REVISION = "093ad1c2613e1e95d26df8610451d2e02ed53cbe"
+
+
+def _float32(x: DataFrame) -> np.ndarray:
+    values = x.to_numpy(dtype=np.float32)
+    values[~np.isfinite(values)] = np.nan
+    return values
+
+
+class EXAONETabular(TabularModel):
+    """LG's EXAONE Tabular at TabArena's default: 8 ensemble members in float16, in its own venv
+    (requirements-exaone.in)."""
+
+    MODEL_NAME = "EXAONE-Tabular"
+    SHORT_NAME = "exaone"
+    ENV = "exaone"
+    USE_VAL_SPLIT = False
+    USE_MEDIAN_FILLING = False
+    USE_CATEGORICAL_ENCODING = True
+    USE_TEXT_EMBEDDINGS = True
+    USE_TARGET_ENCODER = True
+
+    def initialize_model(self) -> None:
+        return None
+
+    def fit_model(self, x_train: DataFrame, y_train: Series, x_val: DataFrame, y_val: Series):
+        from exaonetabular import EXAONETabularClassifier, EXAONETabularRegressor
+        model_cls = EXAONETabularClassifier if self.is_cls else EXAONETabularRegressor
+        self.model_ = model_cls.from_pretrained(device=str(self.device), revision=REVISION, ensemble_count=8,
+                                                compute_dtype="float16", seed=SEED)
+        self.model_.fit(_float32(x_train), y_train.to_numpy())
+
+    def predict_from_processed(self, x: DataFrame) -> np.ndarray:
+        return super().predict_from_processed(_float32(x))

{multabench-0.0.1 → multabench-0.0.2}/multabench/baselines/kumo_tabular.py

@@ -1,3 +1,5 @@
+from unittest.mock import patch
+
 import numpy as np
 import torch
 from pandas import DataFrame, Series
@@ -24,6 +26,7 @@ class KumoEstimator:
 
     def fit(self, x: DataFrame, y: Series):
         import sdm
+        from sdm.cache import Cache
         self.model = sdm.models.KumoTabular(task=self.task, size=self.size, device=self.device)
         self.stypes = sdm.infer_stypes(x, _low_cardinality="infer")
         x_context = sdm.TableTensor.from_pandas(df=x, stypes=self.stypes, device=self.device)
@@ -34,7 +37,10 @@ class KumoEstimator:
         # NVIDIA's adapter fits one ensemble member at a time above these sizes, to bound GPU memory.
         batch_size = 1 if rows > 2_000 or rows * cols >= 50_000 else self.num_estimators
         generator = torch.Generator(self.device).manual_seed(SEED)
-        with torch.inference_mode(), self._inference():
+        # sdm copies each estimator's fitted cache to pinned CPU memory, up to 56 GB on Hearthstone, past the 32 GB of
+        # RAM; Cache._apply_tensor does only that copy in fit, so skipping it keeps the caches on the GPU, where predict
+        # reads them anyway.
+        with torch.inference_mode(), self._inference(), patch.object(Cache, "_apply_tensor", lambda c, fn: c):
             self.model.fit(x=x_context, y=y_context, num_estimators=self.num_estimators,
                            estimator_batch_size=batch_size, generator=generator)
         return self

{multabench-0.0.1 → multabench-0.0.2}/multabench/baselines/preprocessing/categorical.py

@@ -1,6 +1,6 @@
 from typing import Set, Dict
 
-from pandas import Series, DataFrame
+from pandas import CategoricalDtype, Series, DataFrame
 from sklearn.preprocessing import LabelEncoder
 
 from multabench.baselines.preprocessing.nulls import MISSING_VALUE
@@ -21,4 +21,8 @@ def transform_categorical_features(x: DataFrame, categorical_encoders: Dict[str,
         s = x[col].apply(lambda v: v if v in encoder.classes_ else MISSING_VALUE)
         s = encoder.transform(s).astype(int)
         x[col] = s
-    return x
+    return x
+
+def fit_category_dtypes(x: DataFrame, categorical_features: Set[str]) -> Dict[str, CategoricalDtype]:
+    """The train values of each categorical column, for models that read pandas' category dtype; unseen ones are NaN."""
+    return {col: CategoricalDtype(sorted(x[col].unique())) for col in categorical_features}

multabench-0.0.2/multabench/baselines/tabdpt13.py

@@ -0,0 +1,45 @@
+import numpy as np
+import torch
+from pandas import DataFrame, Series
+
+from multabench.baselines.abstract_model import TabularModel
+from multabench.constants import SEED
+
+# The weights TabArena benchmarked; tabdpt 1.3 can't load the older TabDPT's, nor 1.1 these.
+HF_REPO = "Layer6/TabDPT"
+WEIGHTS = "tabdpt1_3.safetensors"
+REVISION = "a5ca6e01c0fa09ec68c73e958e5199d1932abb3a"
+
+
+class TabDPT13(TabularModel):
+    """TabDPT 1.3 at TabArena's default; tabdpt 1.3 conflicts with the 1.1 TabDPT needs, so it has its own venv."""
+
+    MODEL_NAME = "TabDPT-1.3"
+    SHORT_NAME = "dpt13"
+    ENV = "tabdpt13"
+    USE_VAL_SPLIT = False
+    USE_MEDIAN_FILLING = False
+    USE_CATEGORICAL_ENCODING = True
+    USE_TEXT_EMBEDDINGS = True
+    USE_TARGET_ENCODER = True
+
+    def initialize_model(self) -> None:
+        return None
+
+    def fit_model(self, x_train: DataFrame, y_train: Series, x_val: DataFrame, y_val: Series):
+        from huggingface_hub import hf_hub_download
+        from tabdpt import TabDPTClassifier, TabDPTRegressor
+        model_cls = TabDPTClassifier if self.is_cls else TabDPTRegressor
+        self.model_ = model_cls(device=str(self.device), use_flash=True, compile=False, verbose=False,
+                                model_weight_path=hf_hub_download(HF_REPO, WEIGHTS, revision=REVISION),
+                                normalizer="standard", missing_indicators=False, clip_sigma=8,
+                                feature_reduction="pca", context_reduction="subsample", faiss_metric="l2")
+        # tabdpt reduces wide tables with torch.pca_lowrank, which draws from torch's global generator.
+        torch.manual_seed(SEED)
+        self.model_.fit(x_train.to_numpy(), y_train.to_numpy())
+
+    def predict_from_processed(self, x: DataFrame) -> np.ndarray:
+        if not self.is_cls:
+            return self.model_.predict(x.to_numpy(), seed=SEED)
+        probs = self.model_.ensemble_predict_proba(x.to_numpy(), seed=SEED)
+        return probs[:, 1] if self.d_output == 2 else probs

multabench-0.0.2/multabench/baselines/tabfm.py

@@ -0,0 +1,60 @@
+import numpy as np
+from pandas import DataFrame, Series
+
+from multabench.baselines.abstract_model import TabularModel
+from multabench.constants import SEED
+
+HEAD_CLASSES = 10
+
+
+class TabFMCodeRow:
+    """ManyClassClassifier's base estimator, as in TabArena: a TabFM classifier per codebook row, sharing the network
+    and rebuilding the frame from the arrays the codebook passes."""
+
+    def __init__(self, network, dtypes: Series):
+        self.network = network
+        self.dtypes = dtypes
+
+    def __sklearn_clone__(self):
+        return TabFMCodeRow(network=self.network, dtypes=self.dtypes)
+
+    def _frame(self, x) -> DataFrame:
+        return DataFrame(x, columns=self.dtypes.index).astype(self.dtypes)
+
+    def fit(self, x, y):
+        from tabfm import TabFMClassifier
+        self.model = TabFMClassifier(model=self.network, random_state=SEED).fit(self._frame(x), np.asarray(y))
+        self.classes_ = self.model.classes_
+        return self
+
+    def predict_proba(self, x) -> np.ndarray:
+        return self.model.predict_proba(self._frame(x))
+
+
+class TabFM(TabularModel):
+    """Google's TabFM at TabArena's default, with tabpfn-extensions' output coding above its 10-class head."""
+
+    MODEL_NAME = "TabFM"
+    SHORT_NAME = "tabfm"
+    USE_VAL_SPLIT = False
+    USE_MEDIAN_FILLING = False
+    USE_CATEGORICAL_ENCODING = False
+    USE_TEXT_EMBEDDINGS = True
+    USE_TARGET_ENCODER = True
+    USE_CATEGORY_DTYPE = True
+
+    def initialize_model(self) -> None:
+        return None
+
+    def fit_model(self, x_train: DataFrame, y_train: Series, x_val: DataFrame, y_val: Series):
+        from tabfm import TabFMClassifier, TabFMRegressor, tabfm_v1_0_0_pytorch
+        network = tabfm_v1_0_0_pytorch.load(model_type="classification" if self.is_cls else "regression",
+                                            device=str(self.device))
+        if self.d_output > HEAD_CLASSES:
+            from tabpfn_extensions.many_class import ManyClassClassifier
+            self.model_ = ManyClassClassifier(estimator=TabFMCodeRow(network=network, dtypes=x_train.dtypes),
+                                              alphabet_size=HEAD_CLASSES, random_state=SEED)
+        else:
+            cls = TabFMClassifier if self.is_cls else TabFMRegressor
+            self.model_ = cls(model=network, random_state=SEED)
+        self.model_.fit(x_train, y_train)

multabench-0.0.2/multabench/baselines/tabldm.py

@@ -0,0 +1,37 @@
+from pandas import DataFrame, Series
+
+from multabench.baselines.abstract_model import TabularModel
+from multabench.constants import SEED
+from multabench.utils.devices import CPU_CORES
+
+# The Hub revision of TabArena's results; the regression checkpoint was replaced after it.
+HF_REPO = "occams/Xiaomi-TabLDM"
+REVISION = "0e10427273bc862fb60fe7df84c89e5298e16cc7"
+# The classifier defaults of the commit TabArena's results ran, which upstream changed since.
+CLASSIFIER_DEFAULTS = {"n_estimators": 8, "feat_shuffle_method": "latin", "cat_random_encode": False, "use_amp": "auto"}
+
+
+class XiaomiTabLDM(TabularModel):
+    """Xiaomi's TabLDM at TabArena's default: its plain ensemble, without the candidate enhancement, whose rank
+    normalizer fails when a predict batch has more rows than the training set."""
+
+    MODEL_NAME = "Xiaomi-TabLDM"
+    SHORT_NAME = "tabldm"
+    USE_VAL_SPLIT = False
+    USE_MEDIAN_FILLING = False
+    USE_CATEGORICAL_ENCODING = False
+    USE_TEXT_EMBEDDINGS = True
+    USE_TARGET_ENCODER = True
+    USE_CATEGORY_DTYPE = True
+
+    def initialize_model(self) -> None:
+        return None
+
+    def fit_model(self, x_train: DataFrame, y_train: Series, x_val: DataFrame, y_val: Series):
+        from huggingface_hub import hf_hub_download
+        from tabldm import TabLDMClassifier, TabLDMRegressor
+        checkpoint = "checkpoints/clf_default.ckpt" if self.is_cls else "checkpoints/reg_default.ckpt"
+        model_cls, defaults = (TabLDMClassifier, CLASSIFIER_DEFAULTS) if self.is_cls else (TabLDMRegressor, {})
+        self.model_ = model_cls(device=str(self.device), n_jobs=CPU_CORES, random_state=SEED, enhance_candidates=False,
+                                model_path=hf_hub_download(HF_REPO, checkpoint, revision=REVISION), **defaults)
+        self.model_.fit(x_train, y_train)