preorder4mlc 1.0.0__tar.gz

preorder4mlc-1.0.0/CITATION.cff

@@ -0,0 +1,57 @@
+# Citation File Format - https://citation-file-format.github.io/
+cff-version: 1.2.0
+title: "preorder4MLC: Pre-Order Based Multi-Label Classification"
+message: "If you use this software, please cite the accompanying paper."
+type: software
+authors:
+  - family-names: Hoang
+    given-names: Xuan-Truong
+    email: hxtruong@jaist.ac.jp
+    affiliation: "School of Knowledge Science, Japan Advanced Institute of Science and Technology, Japan"
+    orcid: "https://orcid.org/0009-0003-4350-5148"
+  - family-names: Nguyen
+    given-names: Vu-Linh
+    email: vu-linh.nguyen@hds.utc.fr
+    affiliation: "Heudiasyc Laboratory, University of Technology of Compiègne, France"
+  - family-names: Destercke
+    given-names: Sébastien
+    email: sebastien.destercke@hds.utc.fr
+    affiliation: "Heudiasyc Laboratory, University of Technology of Compiègne, France"
+  - family-names: de Campos
+    given-names: Cassio
+    email: c.decampos@tue.nl
+    affiliation: "Uncertainty in Artificial Intelligence Group, Eindhoven University of Technology, The Netherlands"
+  - family-names: Huynh
+    given-names: Van-Nam
+    email: huynh@jaist.ac.jp
+    affiliation: "School of Knowledge Science, Japan Advanced Institute of Science and Technology, Japan"
+repository-code: "https://github.com/hxtruong6/pre-order-for-mlc"
+license: MIT
+version: "1.0.0"
+date-released: "2026-05-15"
+keywords:
+  - multi-label classification
+  - preference learning
+  - noisy and imbalanced labels
+  - robustness
+  - partial-abstention
+  - integer linear programming
+preferred-citation:
+  type: article
+  title: "Robust Multi-Label Classification via Preference Learning"
+  authors:
+    - family-names: Hoang
+      given-names: Xuan-Truong
+      orcid: "https://orcid.org/0009-0003-4350-5148"
+    - family-names: Nguyen
+      given-names: Vu-Linh
+    - family-names: Destercke
+      given-names: Sébastien
+    - family-names: de Campos
+      given-names: Cassio
+    - family-names: Huynh
+      given-names: Van-Nam
+  journal: "Machine Learning"
+  publisher: "Springer"
+  year: 2026
+  doi: "<TODO: DOI once assigned>"

preorder4mlc-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Hoàng Xuân Trường and Vu-Linh Nguyen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

preorder4mlc-1.0.0/MANIFEST.in

@@ -0,0 +1,8 @@
+include LICENSE
+include README.md
+include REPRODUCE.md
+include CITATION.cff
+include requirements.txt
+include requirements-core.txt
+recursive-include docs *.md *.yaml
+prune docs/superpowers

preorder4mlc-1.0.0/PKG-INFO

@@ -0,0 +1,213 @@
+Metadata-Version: 2.4
+Name: preorder4mlc
+Version: 1.0.0
+Summary: Robust multi-label classification via preference learning: pairwise classifiers and ILP-based Bayes-optimal order-structure prediction.
+Author-email: Xuan-Truong Hoang <hxtruong@jaist.ac.jp>, Vu-Linh Nguyen <vu-linh.nguyen@hds.utc.fr>, Sébastien Destercke <sebastien.destercke@hds.utc.fr>, Cassio de Campos <c.decampos@tue.nl>, Van-Nam Huynh <huynh@jaist.ac.jp>
+License: MIT License
+        
+        Copyright (c) 2026 Hoàng Xuân Trường and Vu-Linh Nguyen
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Repository, https://github.com/hxtruong6/pre-order-for-mlc
+Keywords: multi-label classification,preference learning,order structures,integer linear programming,partial abstention,robustness
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.26
+Requires-Dist: scipy>=1.12
+Requires-Dist: scikit-learn>=1.4
+Requires-Dist: pandas>=2.0
+Requires-Dist: joblib>=1.3
+Requires-Dist: liac-arff>=2.5
+Requires-Dist: lightgbm>=4.0
+Requires-Dist: highspy>=1.7
+Requires-Dist: cvxopt>=1.3
+Requires-Dist: scikit-multilearn>=0.2
+Provides-Extra: viz
+Requires-Dist: matplotlib>=3.8; extra == "viz"
+Requires-Dist: seaborn>=0.13; extra == "viz"
+Requires-Dist: openpyxl>=3.1; extra == "viz"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: pytest-cov>=4; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Requires-Dist: build>=1.0; extra == "dev"
+Requires-Dist: twine>=5.0; extra == "dev"
+Dynamic: license-file
+
+# preorder4MLC: Pre-Order Based Multi-Label Classification
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+[![Python](https://img.shields.io/badge/python-3.12-blue)](https://www.python.org/)
+
+In **multi-label classification (MLC)** each instance can carry several
+labels at once (a song that is both *happy* and *relaxed* — as in the
+`emotions` dataset; an email tagged both *business* and *legal* — as in the
+`enron` dataset, both used in this paper). This repository introduces a method that predicts
+labels by first learning **how labels compare** rather than deciding each
+label in isolation — and that can **abstain** on labels it is unsure about
+instead of guessing.
+
+> 📄 Code and full experimental results for:
+> **Robust Multi-Label Classification via Preference Learning** —
+> Vu-Linh Nguyen, Xuan-Truong Hoang, Sébastien Destercke, Cassio de Campos,
+> Van-Nam Huynh (Nguyen and Hoang contributed equally).
+> *Machine Learning* (Springer), 2026. \<TODO: DOI once assigned\>
+
+> **Abstract.** We explore how multi-label classification (MLC) tasks can
+> be cast into order structure learning, exploiting the rich structure of
+> orders to improve and robustify MLC. We formalise the transformation of
+> MLC into an order structure learning and prediction task, study the
+> prediction of Bayes-optimal order structures, and run experiments in
+> settings where orders are especially beneficial: robust MLC under noisy
+> and imbalanced labels, and MLC prediction with partial abstention.
+
+## What it does
+
+The method is built around **Bipartite Ordered Preference Orders (BOPOs)**.
+Instead of predicting each label independently, it works in three steps:
+
+1. **Learn pairwise preferences.** For every pair of labels, a calibrated
+   classifier estimates the probability of their relative ordering.
+2. **Search for the best order.** Per instance, an integer linear program
+   (ILP) combines those pairwise probabilities into a single coherent
+   **preference order** — a *pre-order* or a *partial-order* — that
+   minimises an expected loss (Hamming or Subset 0/1).
+3. **Derive a prediction.** The order is turned into one of three outputs:
+   a plain binary vector, the preference order itself, or a
+   **partial-abstention** vector that marks uncertain labels as "abstain".
+
+An optional *height* constraint on the order yields **eight inference
+algorithms** in total (pre-/partial-order × Hamming/Subset × height 2/∅).
+We compare against four standard baselines: **BR, CC, CLR, ECC**.
+
+```
+              ┌──────────────────────┐
+   training → │ K(K−1)/2 pairwise    │ → pairwise probabilities pᵢⱼ
+              │ calibrated classif.  │
+              └──────────────────────┘
+                         │
+                         ▼
+              ┌──────────────────────┐
+              │ per-instance ILP     │ → preference order
+              │ search (cvxopt+GLPK) │   (pre- or partial-order)
+              └──────────────────────┘
+                         │
+                         ▼
+              ┌──────────────────────┐
+              │ derive prediction    │ → BinaryVector
+              │                      │   PreferenceOrder
+              │                      │   PartialAbstention
+              └──────────────────────┘
+```
+
+## Quick start
+
+```bash
+# Install (editable, so scripts/ can import the package)
+python -m venv .venv && source .venv/bin/activate
+pip install -r requirements.txt
+pip install -e .
+
+# Run one dataset end-to-end
+python scripts/train.py    --dataset emotions --results_dir results/run-dev
+python scripts/evaluate.py --dataset emotions --results_dir results/run-dev
+```
+
+To reproduce **every** number and figure in the paper, see
+[REPRODUCE.md](REPRODUCE.md) — it covers the full pipeline, expected wall
+times, and the exact environment.
+
+## Datasets
+
+Ten multi-label datasets are used:
+`chd_49`, `emotions`, `scene`, `yeast`, `water_quality`, `humanpseaac`,
+`gpositivepseaac`, `plantpseaac`, `viruspseaac`, and `enron`.
+
+The first nine ARFFs are bundled under `data/`, so the pipeline runs
+end-to-end right after `pip install`. `enron.arff` (K=53) is downloaded
+separately — see [REPRODUCE.md §2](REPRODUCE.md). CLI keys match
+`preorder4mlc.config::ConfigManager.DATASET_CONFIGS`.
+
+## Partial abstention
+
+A partial-abstention prediction is a vector in `{0, 1, −1}` where `−1`
+means **"abstain"** on that label. This lets the model stay silent where
+it is uncertain instead of forcing a 0/1 call. Two pairs of metrics
+capture the trade-off — *recovery* (did the abstentions cover the truth?)
+and *abstention rate* (how often did it abstain?):
+
+```
+ŷ = [1, 0, 1, −1, 0, −1]      y = [0, 0, 1, 1, 0, 0]
+
+AREC = (0 + 1 + 1 + 1 + 1 + 1) / 6 = 4/6      # −1 counts as covering {0,1}
+AABS = 2 / 6                                   # fraction abstained
+```
+
+Full definitions (`AREC`, `AABS`, `REC`, `ABS`) live in
+`preorder4mlc.evaluation_metric`.
+
+## Repository layout
+
+```
+preorder4mlc/            # Library package
+├── config.py            # Run configuration + dataset registry
+├── datasets4experiments.py   # ARFF loading, k-fold splits, label noise
+├── base_classifiers.py  # Pairwise / calibrated classifier factory
+├── estimator.py         # Uniform interface over RF / ETC / XGBoost / LightGBM
+├── inference_models.py  # PredictBOPOs (BOPOs + BR / CC / CLR baselines)
+├── searching_algorithms.py   # ILP search for pre- and partial-orders
+├── training_orchestrator.py  # Training loop over learners × folds × algorithms
+├── evaluation_metric.py # Example-, label-, ranking-, abstention-metrics
+└── utils/               # Summaries, statistical tests, figures
+
+scripts/                 # CLI entry points
+├── train.py / evaluate.py          # BOPOs + CLR / BR / CC
+├── train_ecc.py / evaluate_ecc.py  # ECC baseline
+└── smoke_predict_bopos.py          # Behavior-preservation smoke test
+
+data/                    # 9 bundled ARFFs (enron downloaded separately)
+results/                 # Per-fold CSVs + aggregated tables
+run.sh                   # End-to-end reproduction driver
+REPRODUCE.md             # Step-by-step reproduction recipe
+```
+
+## Citation
+
+If you use this code, please cite the paper (see
+[CITATION.cff](CITATION.cff)):
+
+```bibtex
+@article{nguyen2026robust,
+  title     = {Robust Multi-Label Classification via Preference Learning},
+  author    = {Nguyen, Vu-Linh and Hoang, Xuan-Truong and Destercke, S{\'e}bastien and de Campos, Cassio and Huynh, Van-Nam},
+  journal   = {Machine Learning},
+  publisher = {Springer},
+  year      = {2026},
+}
+```
+
+## License
+
+Released under the [MIT License](LICENSE).

preorder4mlc-1.0.0/README.md

@@ -0,0 +1,155 @@
+# preorder4MLC: Pre-Order Based Multi-Label Classification
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+[![Python](https://img.shields.io/badge/python-3.12-blue)](https://www.python.org/)
+
+In **multi-label classification (MLC)** each instance can carry several
+labels at once (a song that is both *happy* and *relaxed* — as in the
+`emotions` dataset; an email tagged both *business* and *legal* — as in the
+`enron` dataset, both used in this paper). This repository introduces a method that predicts
+labels by first learning **how labels compare** rather than deciding each
+label in isolation — and that can **abstain** on labels it is unsure about
+instead of guessing.
+
+> 📄 Code and full experimental results for:
+> **Robust Multi-Label Classification via Preference Learning** —
+> Vu-Linh Nguyen, Xuan-Truong Hoang, Sébastien Destercke, Cassio de Campos,
+> Van-Nam Huynh (Nguyen and Hoang contributed equally).
+> *Machine Learning* (Springer), 2026. \<TODO: DOI once assigned\>
+
+> **Abstract.** We explore how multi-label classification (MLC) tasks can
+> be cast into order structure learning, exploiting the rich structure of
+> orders to improve and robustify MLC. We formalise the transformation of
+> MLC into an order structure learning and prediction task, study the
+> prediction of Bayes-optimal order structures, and run experiments in
+> settings where orders are especially beneficial: robust MLC under noisy
+> and imbalanced labels, and MLC prediction with partial abstention.
+
+## What it does
+
+The method is built around **Bipartite Ordered Preference Orders (BOPOs)**.
+Instead of predicting each label independently, it works in three steps:
+
+1. **Learn pairwise preferences.** For every pair of labels, a calibrated
+   classifier estimates the probability of their relative ordering.
+2. **Search for the best order.** Per instance, an integer linear program
+   (ILP) combines those pairwise probabilities into a single coherent
+   **preference order** — a *pre-order* or a *partial-order* — that
+   minimises an expected loss (Hamming or Subset 0/1).
+3. **Derive a prediction.** The order is turned into one of three outputs:
+   a plain binary vector, the preference order itself, or a
+   **partial-abstention** vector that marks uncertain labels as "abstain".
+
+An optional *height* constraint on the order yields **eight inference
+algorithms** in total (pre-/partial-order × Hamming/Subset × height 2/∅).
+We compare against four standard baselines: **BR, CC, CLR, ECC**.
+
+```
+              ┌──────────────────────┐
+   training → │ K(K−1)/2 pairwise    │ → pairwise probabilities pᵢⱼ
+              │ calibrated classif.  │
+              └──────────────────────┘
+                         │
+                         ▼
+              ┌──────────────────────┐
+              │ per-instance ILP     │ → preference order
+              │ search (cvxopt+GLPK) │   (pre- or partial-order)
+              └──────────────────────┘
+                         │
+                         ▼
+              ┌──────────────────────┐
+              │ derive prediction    │ → BinaryVector
+              │                      │   PreferenceOrder
+              │                      │   PartialAbstention
+              └──────────────────────┘
+```
+
+## Quick start
+
+```bash
+# Install (editable, so scripts/ can import the package)
+python -m venv .venv && source .venv/bin/activate
+pip install -r requirements.txt
+pip install -e .
+
+# Run one dataset end-to-end
+python scripts/train.py    --dataset emotions --results_dir results/run-dev
+python scripts/evaluate.py --dataset emotions --results_dir results/run-dev
+```
+
+To reproduce **every** number and figure in the paper, see
+[REPRODUCE.md](REPRODUCE.md) — it covers the full pipeline, expected wall
+times, and the exact environment.
+
+## Datasets
+
+Ten multi-label datasets are used:
+`chd_49`, `emotions`, `scene`, `yeast`, `water_quality`, `humanpseaac`,
+`gpositivepseaac`, `plantpseaac`, `viruspseaac`, and `enron`.
+
+The first nine ARFFs are bundled under `data/`, so the pipeline runs
+end-to-end right after `pip install`. `enron.arff` (K=53) is downloaded
+separately — see [REPRODUCE.md §2](REPRODUCE.md). CLI keys match
+`preorder4mlc.config::ConfigManager.DATASET_CONFIGS`.
+
+## Partial abstention
+
+A partial-abstention prediction is a vector in `{0, 1, −1}` where `−1`
+means **"abstain"** on that label. This lets the model stay silent where
+it is uncertain instead of forcing a 0/1 call. Two pairs of metrics
+capture the trade-off — *recovery* (did the abstentions cover the truth?)
+and *abstention rate* (how often did it abstain?):
+
+```
+ŷ = [1, 0, 1, −1, 0, −1]      y = [0, 0, 1, 1, 0, 0]
+
+AREC = (0 + 1 + 1 + 1 + 1 + 1) / 6 = 4/6      # −1 counts as covering {0,1}
+AABS = 2 / 6                                   # fraction abstained
+```
+
+Full definitions (`AREC`, `AABS`, `REC`, `ABS`) live in
+`preorder4mlc.evaluation_metric`.
+
+## Repository layout
+
+```
+preorder4mlc/            # Library package
+├── config.py            # Run configuration + dataset registry
+├── datasets4experiments.py   # ARFF loading, k-fold splits, label noise
+├── base_classifiers.py  # Pairwise / calibrated classifier factory
+├── estimator.py         # Uniform interface over RF / ETC / XGBoost / LightGBM
+├── inference_models.py  # PredictBOPOs (BOPOs + BR / CC / CLR baselines)
+├── searching_algorithms.py   # ILP search for pre- and partial-orders
+├── training_orchestrator.py  # Training loop over learners × folds × algorithms
+├── evaluation_metric.py # Example-, label-, ranking-, abstention-metrics
+└── utils/               # Summaries, statistical tests, figures
+
+scripts/                 # CLI entry points
+├── train.py / evaluate.py          # BOPOs + CLR / BR / CC
+├── train_ecc.py / evaluate_ecc.py  # ECC baseline
+└── smoke_predict_bopos.py          # Behavior-preservation smoke test
+
+data/                    # 9 bundled ARFFs (enron downloaded separately)
+results/                 # Per-fold CSVs + aggregated tables
+run.sh                   # End-to-end reproduction driver
+REPRODUCE.md             # Step-by-step reproduction recipe
+```
+
+## Citation
+
+If you use this code, please cite the paper (see
+[CITATION.cff](CITATION.cff)):
+
+```bibtex
+@article{nguyen2026robust,
+  title     = {Robust Multi-Label Classification via Preference Learning},
+  author    = {Nguyen, Vu-Linh and Hoang, Xuan-Truong and Destercke, S{\'e}bastien and de Campos, Cassio and Huynh, Van-Nam},
+  journal   = {Machine Learning},
+  publisher = {Springer},
+  year      = {2026},
+}
+```
+
+## License
+
+Released under the [MIT License](LICENSE).

preorder4mlc-1.0.0/REPRODUCE.md

@@ -0,0 +1,192 @@
+# Reproducing the paper results
+
+This document records the exact steps required to reproduce every number
+and figure reported in the journal-revision paper from a clean clone of
+this repository.
+
+## 1. Environment
+
+Tested on Python 3.12 with the pinned versions in `requirements.txt`.
+GLPK headers must be present on the system so that `cvxopt.glpk` can
+solve the per-instance ILP.
+
+```bash
+# Ubuntu / Debian
+sudo apt-get install libglpk-dev
+
+# Python deps
+python -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+pip install -e .
+```
+
+## 2. Data
+
+Ten multi-label datasets are used in the paper. Nine ARFFs are tracked
+under `data/` for one-click reproduction; `enron.arff` (K=53) must be
+downloaded separately from COMETA / MULAN and placed at
+`data/enron.arff`.
+
+| Key (CLI) | File | Labels | Source |
+|---|---|---|---|
+| `chd_49` | `CHD_49.arff` | 6 | Coronary heart disease |
+| `emotions` | `emotions.arff` | 6 | Mulan repository |
+| `scene` | `scene.arff` | 6 | Mulan repository |
+| `yeast` | `Yeast.arff` | 14 | Mulan repository |
+| `water_quality` | `Water-quality.arff` | 14 | UCI |
+| `humanpseaac` | `HumanPseAAC.arff` | 14 | Pse-AAC encoding |
+| `gpositivepseaac` | `GpositivePseAAC.arff` | 4 | Pse-AAC encoding |
+| `plantpseaac` | `PlantPseAAC.arff` | 12 | Pse-AAC encoding |
+| `viruspseaac` | `VirusPseAAC.arff` | 6 | Pse-AAC encoding |
+| `enron` | `enron.arff` | 53 | COMETA / MULAN (not bundled) |
+
+## 3. Determinism
+
+All splits, folds, and label-noise draws are seeded by
+`preorder4mlc.constants.RANDOM_STATE = 6`. The per-fold parallel
+training step uses `joblib.Parallel(n_jobs=-1)`; sklearn ensemble
+estimators are seeded through `RANDOM_STATE` as well so identical
+splits produce identical predictions.
+
+## 4. Reproducing all results
+
+The repository ships **two drivers** for the same pipeline. Both call the
+same four `scripts/*.py` entry points per dataset; they differ only in how
+the work is orchestrated. Pick one:
+
+| | `run.sh` | `scripts/slurm/submit_all.sh` |
+|---|---|---|
+| Target | a single machine | a SLURM HPC cluster |
+| Execution | datasets run sequentially | hundreds of array tasks in parallel |
+| Resume | re-runs everything from scratch | skips cells already complete on disk |
+| Best for | small/quick runs, a few datasets | full reproduction incl. `enron` (K=53) |
+
+Whichever driver you use, each dataset runs the same four steps in order:
+
+1. `python scripts/train.py --dataset <key> --results_dir <dir>` —
+   train pairwise classifiers, BOPOs (pre- and partial-order), and the
+   CLR / BR / CC baselines.
+2. `python scripts/evaluate.py --dataset <key> --results_dir <dir>` —
+   write per-fold evaluation CSVs for the BOPOs / CLR / BR / CC pickles.
+3. `python scripts/train_ecc.py --dataset <key> --algorithm ecc
+   --results_dir <dir>` — train the ECC (Ensemble of Classifier Chains)
+   baseline.
+4. `python scripts/evaluate_ecc.py --dataset <key> --algorithm ecc
+   --results_dir <dir>` — write per-fold evaluation CSVs for ECC.
+
+### 4a. Single machine — `run.sh`
+
+```bash
+RESULTS_DIR=results/run-$(date +%Y%m%d) bash run.sh
+```
+
+Runs all ten datasets sequentially with the default base learner (Random
+Forest). Per-dataset stdout/stderr lands in `logs/run-<date>/<dataset>.log`.
+To reproduce the LightGBM variant, pass `--base_learner lgbm` to both
+`train.py` and `train_ecc.py`.
+
+`run.sh` does **not** skip completed work and has no `enron` throttle, so it
+suits local runs of the lighter datasets. For the full matrix — especially
+`enron`, which needs an HPC node and the HiGHS solver (see §7) — use the
+SLURM driver below.
+
+### 4b. HPC cluster — `scripts/slurm/submit_all.sh`
+
+```bash
+# Submit the full RF + LightGBM matrix; results land in
+# results/rerun_<date>_rf and results/rerun_<date>_lgbm by default.
+bash scripts/slurm/submit_all.sh
+
+# Preview the sbatch plan without submitting:
+bash scripts/slurm/submit_all.sh --dry-run
+
+# Restrict learners / datasets via env vars:
+LEARNERS="RF"            bash scripts/slurm/submit_all.sh   # RF only
+DATASETS="viruspseaac"   bash scripts/slurm/submit_all.sh   # one dataset
+```
+
+The controller expands the full matrix (10 datasets × 4 noise levels × 5
+algorithms × {RF, LightGBM}) into array jobs, **skips any cell already
+complete on disk**, and submits eval jobs with
+`--dependency=afterok:<train-jobids>`. It respects the cluster QOS submit
+cap (`SUBMIT_CAP`) with backoff, and throttles the heavy `enron` BOPOS
+arrays (`ENRON_QUEUE_FRACTION`, default 0.4) so other datasets keep ~60% of
+the queue; the throttle lifts automatically once every non-`enron` cell is
+done. Logs land in `slurm_logs/`; watch with `squeue -u $USER`.
+
+`merge_split_results.py` reassembles the per-repeat sub-arrays that the
+`enron` throttle splits jobs into. After all eval jobs finish, aggregate
+with `summarize_metrics` (§5).
+
+## 5. Summarising into the paper tables
+
+```bash
+python -m preorder4mlc.utils.summarize_metrics \
+    --results_dir results/run-<date> \
+    --output_dir  results/run-<date>_summary
+```
+
+Writes one `<Dataset>_<PredictionType>_summary.csv` per dataset to the
+output directory. `PredictionType` is one of `BinaryVector`,
+`PartialAbstention`, or `ScoreVector`.
+
+## 6. Statistical tests and figures
+
+```bash
+python -m preorder4mlc.utils.statistical_tests \
+    --results_dir results/run-<date>_summary \
+    --output_dir  results/run-<date>_summary/stats
+
+python -m preorder4mlc.utils.plot_figures \
+    --results_dir     results/run-<date>_summary \
+    --raw_results_dir results/run-<date>
+```
+
+The figure script writes critical-difference diagrams, average-rank vs
+noise curves, abstention plots, Hamming/Subset trade-offs, and rank
+heatmaps to `results/run-<date>_summary/figures/`.
+
+## 7. Compute budget
+
+Single-machine wall time (5 repeats x 5 folds, joblib across all CPUs),
+per dataset, with Random Forest as the base learner:
+
+| Dataset | Wall time |
+|---|---|
+| chd_49 | ~4 min |
+| gpositivepseaac | ~5 min |
+| viruspseaac | ~5 min |
+| emotions | ~6 min |
+| plantpseaac | ~30 min |
+| scene | ~40 min |
+| water_quality | ~60 min |
+| yeast | ~60 min |
+| humanpseaac | ~2.5 h |
+| enron | per-fold ~18 min on an HPC node (K=53; uses the HiGHS solver — set `PREORDER_SOLVER=highs` — and ~300 GB RAM; GLPK is intractable at this label count) |
+
+ECC adds at most ~30 min per dataset (typically far less). For HPC
+re-runs, dataset jobs can be submitted in parallel.
+
+## 8. Output bundle
+
+After step 5 the run directory contains:
+
+* `results/run-<date>/*.pkl` — per-fold training records (not tracked
+  by git; recreated by step 1).
+* `results/run-<date>/evaluation_*.csv` — per-fold metric CSVs.
+* `results/run-<date>_summary/*_summary.csv` — per-dataset aggregate
+  tables (one per `<Dataset>_<PredictionType>`).
+
+The canonical aggregate tables that produced the paper numbers ship as
+CSV under:
+
+* `results/final_rf_summary/` — Random Forest (paper default).
+* `results/final_lgbm_summary/` — LightGBM variant.
+
+Each also keeps its raw `results/final_rf/` and `results/final_lgbm/`
+per-fold pickles + `evaluation_*.csv` on disk (pickles untracked) so the
+summaries can be re-aggregated via step 5 without retraining. New runs
+can be diffed against these CSVs to confirm equivalence. The optional
+statistical tests and figures (step 6) are not shipped; regenerate them
+from the summary CSVs as needed.

preorder4mlc-1.0.0/docs/CONVENTIONS.md

@@ -0,0 +1,9 @@
+# Conventions
+
+- **Package layout:** flat `preorder4mlc/` package; CLIs live in `scripts/`.
+- **Datasets:** registered in `preorder4mlc/config.py` (`ConfigManager.DATASET_CONFIGS`);
+  loaded by `preorder4mlc/datasets4experiments.py`.
+- **Results:** per-fold CSVs under `results/<run>/`; aggregated by
+  `preorder4mlc/utils/summarize_metrics.py`.
+- **Solvers:** ILP search uses HiGHS (`highspy`) with a CVXOPT fallback (`solvers.py`).
+- **Versioning:** keep `pyproject.toml`, `.zenodo.json`, `CITATION.cff` in lockstep.

preorder4mlc-1.0.0/docs/EXTENDING.md

@@ -0,0 +1,19 @@
+# Extending preorder4mlc
+
+## Add a dataset
+Register it in `preorder4mlc/config.py` (`ConfigManager.DATASET_CONFIGS`) with its label
+count and file orientation, and ensure `preorder4mlc/datasets4experiments.py` can load its
+ARFF/CSV. Then add its key to the `DATASETS` array in `run.sh`.
+
+## Add a base classifier
+Implement it in `preorder4mlc/base_classifiers.py` following the existing pairwise /
+label-ranking interface (must expose `fit` / `predict`-style methods consumed by
+`training_orchestrator.py`).
+
+## Add an evaluation metric
+Add the metric to `preorder4mlc/evaluation_metric.py`; it is then available to the
+evaluation scripts and the summary tables.
+
+## Change the ILP search
+The search objective and constraints live in `preorder4mlc/searching_algorithms.py` and
+`preorder4mlc/solvers.py` (HiGHS via `highspy`, CVXOPT fallback).