threshold-opt 0.1.0__tar.gz

threshold_opt-0.1.0/.gitignore

@@ -0,0 +1,93 @@
+# Python
+__pycache__/
+*.py[cod]
+*.pyo
+*.pyd
+.Python
+*.egg
+*.egg-info/
+dist/
+build/
+eggs/
+parts/
+var/
+sdist/
+wheels/
+.installed.cfg
+lib/
+lib64/
+
+# Virtual environments
+.venv/
+venv/
+env/
+ENV/
+
+# uv
+.uv/
+
+# pytest
+.pytest_cache/
+.cache/
+htmlcov/
+.coverage
+coverage.xml
+
+# Jupyter
+.ipynb_checkpoints/
+*.ipynb
+
+# Results / data (large files)
+results/
+experiments/results/
+data/
+*.parquet
+*.csv
+*.h5
+*.pkl
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Secrets
+.env
+.env.*
+*.pem
+*.key
+
+# AI
+.claude/
+
+# catboost
+catboost_info/
+
+# tex audit outputs
+tex/scripts/forward_refs_report.json
+
+# LaTeX build artifacts
+*.aux
+*.fdb_latexmk
+*.fls
+*.log
+*.out
+*.synctex.gz
+*.toc
+*.bbl
+*.blg
+*.pdf
+# biblatex/biber + makeindex + list-of artifacts (IME-USP template)
+*.bcf
+*.run.xml
+*.idx
+*.ilg
+*.ind
+*.lof
+*.lot
+tex/_backup_pre_argmax_rename_20260519_231422/

threshold_opt-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Carlo Lemos
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

threshold_opt-0.1.0/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.5
+Name: threshold-opt
+Version: 0.1.0
+Summary: Threshold optimization for binary and multiclass classification
+Project-URL: Repository, https://github.com/vitaliset/threshold-opt
+Project-URL: Issues, https://github.com/vitaliset/threshold-opt/issues
+Author-email: Carlo Lemos <vitaliset@users.noreply.github.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: binary,calibration,classification,cost-sensitive,decision-making,multiclass,threshold,threshold-optimization
+Classifier: Development Status :: 3 - Alpha
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.11
+Requires-Dist: joblib>=1.3
+Requires-Dist: numpy>=1.26
+Requires-Dist: scikit-learn>=1.6
+Requires-Dist: scipy>=1.11
+Description-Content-Type: text/markdown
+
+# threshold_opt
+
+Threshold optimization for binary and multiclass classification.
+
+Given a trained classifier and a cost matrix (or metric objective), `threshold_opt` finds the decision rule that minimizes expected cost. It supports binary and multiclass problems, with strategies ranging from closed-form analytical solutions to empirical search and calibration-based approaches.
+
+This library is part of a [master's dissertation](https://github.com/vitaliset/threshold-opt/blob/main/tex/carlo_master.tex) on multiclass threshold optimization. The full experiment pipeline lives in [`experiments/`](https://github.com/vitaliset/threshold-opt/blob/main/experiments/README.md).
+
+## Install
+
+```bash
+pip install threshold-opt
+```
+
+The package is published as `threshold-opt` and imported as `threshold_opt`.
+
+For development (includes experiment dependencies):
+
+```bash
+git clone https://github.com/vitaliset/threshold-opt.git
+cd threshold-opt
+uv sync
+```
+
+## Quick example
+
+```python
+import numpy as np
+from sklearn.linear_model import LogisticRegression
+from sklearn.model_selection import train_test_split
+from sklearn.datasets import load_breast_cancer
+from threshold_opt import binary_strategies
+
+# Train any sklearn-compatible classifier
+X, y = load_breast_cancer(return_X_y=True)
+X_train, X_val, y_train, y_val = train_test_split(
+    X, y, test_size=0.3, random_state=42
+)
+model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
+
+# Define a cost matrix: missing cancer (FN) costs 10x more than a false alarm (FP)
+cost_matrix = np.array([
+    [0.0, 1.0],   # TN=0, FP=1
+    [10.0, 0.0],  # FN=10, TP=0
+])
+
+# Find the optimal threshold analytically (no validation data needed)
+strategy = binary_strategies.AnalyticalThreshold(model)
+strategy.fit(np.array([]), np.array([]), cost_matrix)
+y_pred = strategy.predict(X_val)
+
+# Or search empirically on a validation set
+strategy = binary_strategies.EmpiricalThreshold(model)
+strategy.fit(X_val, y_val, cost_matrix)
+y_pred = strategy.predict(X_val)
+```
+
+## What's inside
+
+| Module | Description |
+|--------|-------------|
+| `binary_strategies` | 11 strategies for binary classification: default, analytical, empirical search, and calibration variants (isotonic, Platt, temperature, Dirichlet) |
+| `multiclass_strategies` | 12 strategies for multiclass: argmax, risk minimization, threshold ratios, Voronoi tuning, sequential selection, risk partitioning, and calibration variants |
+| `calibration` | `DirichletCalibrator` for post-hoc probability calibration in log-space |
+| `metrics` | Cost matrix utilities, metric objectives (`MetricObjective`), vectorized scoring, analytical threshold derivation |
+| `optimization` | Grid search and random search optimizers for strategy parameter tuning |
+| `base` | `DecisionStrategy` abstract base class |
+
+All strategies follow a unified interface: `fit(X, y, objective)` then `predict(X)`. Objectives can be cost matrices (numpy arrays), `MetricObjective` instances, or callable factories.
+
+### Public API
+
+`threshold_opt` ships its public API as importable submodules; there is no flat top-level namespace re-export. Stable surface:
+
+- **Strategy contract** ([`base.DecisionStrategy`](https://github.com/vitaliset/threshold-opt/blob/main/src/threshold_opt/base.py)). All strategies subclass `DecisionStrategy` and expose:
+    - `fit(X, y, optimization_objective)` — fit the decision layer on validation data and return `self`.
+    - `predict(X)` — emit class predictions.
+    - `needs_validation_set: bool` — `True` if the strategy consumes a held-out validation split (calibrators, single-split tuners), `False` otherwise (default, analytical, CV-tuners).
+- **Optimization objectives** ([`metrics.OptimizationObjective`](https://github.com/vitaliset/threshold-opt/blob/main/src/threshold_opt/metrics.py)). Either a numpy `(K, K)` cost matrix (cell `[i, j]` = cost of predicting `j` when the truth is `i`), or a `metrics.MetricObjective(metric_fn, higher_is_better, ...)` wrapping any `(y_true, y_pred) -> float` callable. `MetricObjective.vectorized_key` is the optional fast path used by binary strategies; binary metric objectives that omit it still work via a per-threshold loop.
+- **Calibrator** ([`calibration.DirichletCalibrator`](https://github.com/vitaliset/threshold-opt/blob/main/src/threshold_opt/calibration.py)). Standalone `fit(probs, y)` / `predict_proba(probs)` calibrator with optional ODIR penalties (`reg_lambda`, `reg_mu`). Used by `DirichletCalibrationArgmax` / `DirichletCalibrationAnalytical` and freely composable.
+- **Optimizers** ([`optimization.RandomSearchOptimizer`, `GridSearchOptimizer`](https://github.com/vitaliset/threshold-opt/blob/main/src/threshold_opt/optimization.py)). Pluggable parameter optimizers for tuning strategies; subclass `optimization.ParameterOptimizer` to add new ones.
+- **Shared CV seed** ([`base.CV_RANDOM_STATE`](https://github.com/vitaliset/threshold-opt/blob/main/src/threshold_opt/base.py)). Every K-fold loop in the library uses this constant, so external precompute caches that key on it stay in sync with strategies' lazy refits.
+
+Internal symbols (prefixed with `_`, e.g. `base._CVFoldPredictions`, `binary_strategies._positive_class_probs`) are not part of the public API and may change without notice.
+
+## Dissertation and experiments
+
+This repo also contains the experiment pipeline and LaTeX source for the dissertation. See [`experiments/README.md`](https://github.com/vitaliset/threshold-opt/blob/main/experiments/README.md) for how to run experiments, generate tables, and build the PDF.
+
+## Development
+
+```bash
+make test      # run pytest
+make lint      # ruff + ruff-format + chktex
+make help      # all available targets
+```
+
+## Releasing
+
+The package is uploaded to [PyPI](https://pypi.org/project/threshold-opt/) by [`.github/workflows/publish.yml`](https://github.com/vitaliset/threshold-opt/blob/main/.github/workflows/publish.yml), through [trusted publishing](https://docs.pypi.org/trusted-publishers/). To release a new version:
+
+1. Set `version` in `pyproject.toml`.
+2. Publish a GitHub release whose tag is `v` followed by that version (`v0.1.0` for version `0.1.0`).
+
+The workflow checks that the tag matches the version, builds the package and uploads it. Releases with any other tag publish nothing.
+
+## Project structure
+
+```
+src/threshold_opt/     Library: strategies, calibration, metrics, optimization
+experiments/           Experiment pipeline, analysis, cloud scripts
+tex/                   LaTeX dissertation source
+```

threshold_opt-0.1.0/README.md

@@ -0,0 +1,113 @@
+# threshold_opt
+
+Threshold optimization for binary and multiclass classification.
+
+Given a trained classifier and a cost matrix (or metric objective), `threshold_opt` finds the decision rule that minimizes expected cost. It supports binary and multiclass problems, with strategies ranging from closed-form analytical solutions to empirical search and calibration-based approaches.
+
+This library is part of a [master's dissertation](https://github.com/vitaliset/threshold-opt/blob/main/tex/carlo_master.tex) on multiclass threshold optimization. The full experiment pipeline lives in [`experiments/`](https://github.com/vitaliset/threshold-opt/blob/main/experiments/README.md).
+
+## Install
+
+```bash
+pip install threshold-opt
+```
+
+The package is published as `threshold-opt` and imported as `threshold_opt`.
+
+For development (includes experiment dependencies):
+
+```bash
+git clone https://github.com/vitaliset/threshold-opt.git
+cd threshold-opt
+uv sync
+```
+
+## Quick example
+
+```python
+import numpy as np
+from sklearn.linear_model import LogisticRegression
+from sklearn.model_selection import train_test_split
+from sklearn.datasets import load_breast_cancer
+from threshold_opt import binary_strategies
+
+# Train any sklearn-compatible classifier
+X, y = load_breast_cancer(return_X_y=True)
+X_train, X_val, y_train, y_val = train_test_split(
+    X, y, test_size=0.3, random_state=42
+)
+model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
+
+# Define a cost matrix: missing cancer (FN) costs 10x more than a false alarm (FP)
+cost_matrix = np.array([
+    [0.0, 1.0],   # TN=0, FP=1
+    [10.0, 0.0],  # FN=10, TP=0
+])
+
+# Find the optimal threshold analytically (no validation data needed)
+strategy = binary_strategies.AnalyticalThreshold(model)
+strategy.fit(np.array([]), np.array([]), cost_matrix)
+y_pred = strategy.predict(X_val)
+
+# Or search empirically on a validation set
+strategy = binary_strategies.EmpiricalThreshold(model)
+strategy.fit(X_val, y_val, cost_matrix)
+y_pred = strategy.predict(X_val)
+```
+
+## What's inside
+
+| Module | Description |
+|--------|-------------|
+| `binary_strategies` | 11 strategies for binary classification: default, analytical, empirical search, and calibration variants (isotonic, Platt, temperature, Dirichlet) |
+| `multiclass_strategies` | 12 strategies for multiclass: argmax, risk minimization, threshold ratios, Voronoi tuning, sequential selection, risk partitioning, and calibration variants |
+| `calibration` | `DirichletCalibrator` for post-hoc probability calibration in log-space |
+| `metrics` | Cost matrix utilities, metric objectives (`MetricObjective`), vectorized scoring, analytical threshold derivation |
+| `optimization` | Grid search and random search optimizers for strategy parameter tuning |
+| `base` | `DecisionStrategy` abstract base class |
+
+All strategies follow a unified interface: `fit(X, y, objective)` then `predict(X)`. Objectives can be cost matrices (numpy arrays), `MetricObjective` instances, or callable factories.
+
+### Public API
+
+`threshold_opt` ships its public API as importable submodules; there is no flat top-level namespace re-export. Stable surface:
+
+- **Strategy contract** ([`base.DecisionStrategy`](https://github.com/vitaliset/threshold-opt/blob/main/src/threshold_opt/base.py)). All strategies subclass `DecisionStrategy` and expose:
+    - `fit(X, y, optimization_objective)` — fit the decision layer on validation data and return `self`.
+    - `predict(X)` — emit class predictions.
+    - `needs_validation_set: bool` — `True` if the strategy consumes a held-out validation split (calibrators, single-split tuners), `False` otherwise (default, analytical, CV-tuners).
+- **Optimization objectives** ([`metrics.OptimizationObjective`](https://github.com/vitaliset/threshold-opt/blob/main/src/threshold_opt/metrics.py)). Either a numpy `(K, K)` cost matrix (cell `[i, j]` = cost of predicting `j` when the truth is `i`), or a `metrics.MetricObjective(metric_fn, higher_is_better, ...)` wrapping any `(y_true, y_pred) -> float` callable. `MetricObjective.vectorized_key` is the optional fast path used by binary strategies; binary metric objectives that omit it still work via a per-threshold loop.
+- **Calibrator** ([`calibration.DirichletCalibrator`](https://github.com/vitaliset/threshold-opt/blob/main/src/threshold_opt/calibration.py)). Standalone `fit(probs, y)` / `predict_proba(probs)` calibrator with optional ODIR penalties (`reg_lambda`, `reg_mu`). Used by `DirichletCalibrationArgmax` / `DirichletCalibrationAnalytical` and freely composable.
+- **Optimizers** ([`optimization.RandomSearchOptimizer`, `GridSearchOptimizer`](https://github.com/vitaliset/threshold-opt/blob/main/src/threshold_opt/optimization.py)). Pluggable parameter optimizers for tuning strategies; subclass `optimization.ParameterOptimizer` to add new ones.
+- **Shared CV seed** ([`base.CV_RANDOM_STATE`](https://github.com/vitaliset/threshold-opt/blob/main/src/threshold_opt/base.py)). Every K-fold loop in the library uses this constant, so external precompute caches that key on it stay in sync with strategies' lazy refits.
+
+Internal symbols (prefixed with `_`, e.g. `base._CVFoldPredictions`, `binary_strategies._positive_class_probs`) are not part of the public API and may change without notice.
+
+## Dissertation and experiments
+
+This repo also contains the experiment pipeline and LaTeX source for the dissertation. See [`experiments/README.md`](https://github.com/vitaliset/threshold-opt/blob/main/experiments/README.md) for how to run experiments, generate tables, and build the PDF.
+
+## Development
+
+```bash
+make test      # run pytest
+make lint      # ruff + ruff-format + chktex
+make help      # all available targets
+```
+
+## Releasing
+
+The package is uploaded to [PyPI](https://pypi.org/project/threshold-opt/) by [`.github/workflows/publish.yml`](https://github.com/vitaliset/threshold-opt/blob/main/.github/workflows/publish.yml), through [trusted publishing](https://docs.pypi.org/trusted-publishers/). To release a new version:
+
+1. Set `version` in `pyproject.toml`.
+2. Publish a GitHub release whose tag is `v` followed by that version (`v0.1.0` for version `0.1.0`).
+
+The workflow checks that the tag matches the version, builds the package and uploads it. Releases with any other tag publish nothing.
+
+## Project structure
+
+```
+src/threshold_opt/     Library: strategies, calibration, metrics, optimization
+experiments/           Experiment pipeline, analysis, cloud scripts
+tex/                   LaTeX dissertation source
+```

threshold_opt-0.1.0/pyproject.toml

@@ -0,0 +1,94 @@
+[project]
+name = "threshold-opt"
+version = "0.1.0"
+description = "Threshold optimization for binary and multiclass classification"
+readme = "README.md"
+requires-python = ">=3.11"
+license = "MIT"
+authors = [{ name = "Carlo Lemos", email = "vitaliset@users.noreply.github.com" }]
+keywords = [
+    "threshold-optimization",
+    "threshold",
+    "classification",
+    "binary",
+    "multiclass",
+    "cost-sensitive",
+    "decision-making",
+    "calibration",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Programming Language :: Python :: 3.11",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+    # Lower bounds: numpy>=1.26 for Python 3.11 wheels and the array API
+    # changes used by the calibration code; scipy>=1.11 for the numpy 1.26
+    # ABI; scikit-learn>=1.6 for sklearn.frozen.FrozenEstimator (used in
+    # binary_strategies._CalibratedDefaultThreshold and friends).
+    "numpy>=1.26",
+    "scipy>=1.11",
+    "scikit-learn>=1.6",
+    "joblib>=1.3",
+]
+
+[project.urls]
+Repository = "https://github.com/vitaliset/threshold-opt"
+Issues = "https://github.com/vitaliset/threshold-opt/issues"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+# The distribution is the library alone: the experiment pipeline and the
+# dissertation source stay out of both artifacts. The tests live next to
+# the modules and ship in the sdist, not in the wheel.
+[tool.hatch.build.targets.sdist]
+include = ["src/threshold_opt"]
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/threshold_opt"]
+exclude = ["*_test.py"]
+
+[tool.pytest.ini_options]
+testpaths = ["src", "experiments"]
+python_files = ["*_test.py"]
+
+[dependency-groups]
+experiments = [
+    "pandas",
+    "pyarrow",          # pandas parquet backend
+    "lightgbm",
+    "xgboost",
+    "catboost",
+    "tabpfn",
+    "tabicl",           # TabICLv2 tabular foundation model
+    "pytabkit",         # RealMLP_TD_Classifier
+    "tabm",             # TabM deep tabular model (ICLR 2025)
+    "imbalanced-learn", # geometric_mean_score
+    "tqdm",
+]
+dev = [
+    "pytest",
+    "matplotlib",
+    "seaborn",
+    "ipykernel>=7.2.0",
+    "jinja2",
+    "ruff",
+    "pre-commit>=4.5.1",
+]
+
+[tool.uv]
+default-groups = ["dev", "experiments"]
+
+[tool.ruff]
+line-length = 79
+
+[tool.ruff.lint]
+select = ["E", "W", "F", "I", "RUF"]
+ignore = [
+    "E731",   # lambda assignment fine in research code
+    "E741",   # allow short math names (l, I, O)
+    # ruff format conflict rules
+    "W191", "E111", "E114", "E117",
+]

threshold_opt-0.1.0/src/threshold_opt/__init__.py

@@ -0,0 +1 @@
+"""Threshold optimization for binary and multiclass classification."""

threshold_opt-0.1.0/src/threshold_opt/base.py

@@ -0,0 +1,84 @@
+"""Base classes for decision strategies."""
+
+import abc
+import dataclasses
+from typing import Self
+
+import numpy as np
+import numpy.typing as npt
+
+from threshold_opt import metrics
+
+# Seed shared by every K-fold loop in the library and by the experiments
+# pipeline's precompute (see ``experiments.common.pipeline``). Both call
+# sites must agree on this value; otherwise the precompute cache desyncs
+# from the strategy's own lazy refit and the experiment is broken in a
+# hard-to-diagnose way.
+CV_RANDOM_STATE = 42
+
+
+@dataclasses.dataclass(frozen=True)
+class _CVFoldPredictions:
+    """Labels and predicted probabilities for a single CV validation fold."""
+
+    y: np.ndarray
+    probs: np.ndarray
+
+
+def _coerce_integer_labels(y: npt.ArrayLike) -> np.ndarray:
+    """Cast *y* to int when it's already integer-valued (even if float dtype).
+
+    OpenML can return class labels encoded as float64 (values like 0.0, 1.0,
+    2.0). Downstream multiclass strategies use labels as array indices, so
+    we canonicalise here. Non-integer-valued floats (e.g. 0.5, regression
+    targets) pass through untouched; the strategy-level validator rejects
+    them before this runs.
+    """
+    arr = np.asarray(y)
+    if np.issubdtype(arr.dtype, np.integer):
+        return arr
+    return arr.astype(int)
+
+
+class DecisionStrategy(abc.ABC):
+    """Abstract base class for decision strategies."""
+
+    def __init__(self, model):
+        self.model = model
+
+    @property
+    def needs_validation_set(self) -> bool:
+        """Whether the strategy requires a validation set for calibration."""
+        return True
+
+    def fit(
+        self,
+        X: npt.ArrayLike,
+        y: npt.ArrayLike,
+        optimization_objective: metrics.OptimizationObjective,
+    ) -> Self:
+        """Fit the strategy using validation data."""
+        self._validate_fit_input(X, y, optimization_objective)
+        return self._fit(X, _coerce_integer_labels(y), optimization_objective)
+
+    def _validate_fit_input(
+        self,
+        X: npt.ArrayLike,
+        y: npt.ArrayLike,
+        optimization_objective: metrics.OptimizationObjective,
+    ) -> None:
+        """Validates input data before fitting."""
+        pass
+
+    @abc.abstractmethod
+    def _fit(
+        self,
+        X: npt.ArrayLike,
+        y: npt.ArrayLike,
+        optimization_objective: metrics.OptimizationObjective,
+    ) -> Self:
+        """Implementation of the fit logic."""
+
+    @abc.abstractmethod
+    def predict(self, X: npt.ArrayLike) -> npt.ArrayLike:
+        """Predict class labels for X using the fitted strategy."""