optimal-classification-cutoffs 0.1.0__py3-none-any.whl

optimal_classification_cutoffs-0.1.0.dist-info/METADATA

@@ -0,0 +1,114 @@
+Metadata-Version: 2.4
+Name: optimal-classification-cutoffs
+Version: 0.1.0
+Summary: Utilities for computing optimal classification cutoffs for binary and multi-class
+Author-email: Gaurav Sood <contact@gsood.com>
+License: MIT License
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Mathematics
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: scipy
+Requires-Dist: scikit-learn
+Provides-Extra: examples
+Requires-Dist: matplotlib; extra == "examples"
+Requires-Dist: pandas; extra == "examples"
+Dynamic: license-file
+
+# Optimal Classification Cut-Offs
+
+Probabilistic classifiers output per-class probabilities, and fixed cutoffs such as ``0.5`` rarely maximize metrics like accuracy or the F\ :sub:`1` score.
+This package provides utilities to **select optimal probability cutoffs for each class**, supporting both multi-class and binary classifiers.
+Optimization methods include brute-force search, numerical techniques, and gradient-based approaches.
+Binary thresholding at a single cutoff remains fully supported as a special case.
+
+## Quick start
+
+```python
+from optimal_cutoffs import ThresholdOptimizer
+
+# true binary labels and predicted probabilities
+y_true = ...
+y_prob = ...
+
+optimizer = ThresholdOptimizer(objective="f1")
+optimizer.fit(y_true, y_prob)
+y_pred = optimizer.predict(y_prob)
+```
+
+## API
+
+### `get_confusion_matrix(true_labs, pred_prob, threshold)`
+- **Purpose:** Compute confusion-matrix counts for a threshold.
+- **Args:** arrays of true labels and probabilities, plus the decision threshold.
+- **Returns:** `(tp, tn, fp, fn)` counts.
+
+### `register_metric(name=None, func=None)`
+- **Purpose:** Add a metric function to the global registry.
+- **Args:** optional metric name and callable; can also be used as a decorator.
+- **Returns:** the registered function or decorator.
+
+### `register_metrics(metrics)`
+- **Purpose:** Register multiple metric functions at once.
+- **Args:** dictionary mapping names to callables.
+- **Returns:** `None`.
+
+### `get_probability(true_labs, pred_prob, objective='accuracy', verbose=False)`
+- **Purpose:** Brute-force search for the threshold that maximizes accuracy or F\ :sub:`1`.
+- **Args:** true labels, predicted probabilities, metric name, and verbosity flag.
+- **Returns:** optimal threshold.
+
+### `get_optimal_threshold(true_labs, pred_prob, metric='f1', method='smart_brute')`
+- **Purpose:** Optimize any registered metric using different strategies
+  (brute force, ``minimize``, or ``gradient``).
+- **Args:** true labels, probabilities, metric name, and optimization method.
+- **Returns:** optimal threshold.
+
+### `cv_threshold_optimization(true_labs, pred_prob, metric='f1', method='smart_brute', cv=5, random_state=None)`
+- **Purpose:** Estimate thresholds via cross-validation and report per-fold scores.
+- **Returns:** arrays of thresholds and scores.
+
+### `nested_cv_threshold_optimization(true_labs, pred_prob, metric='f1', method='smart_brute', inner_cv=5, outer_cv=5, random_state=None)`
+- **Purpose:** Perform nested cross-validation for threshold estimation and
+  unbiased performance evaluation.
+- **Returns:** arrays of outer-fold thresholds and scores.
+
+### `ThresholdOptimizer(objective='accuracy', verbose=False)`
+- **Purpose:** High-level wrapper with ``fit``/``predict`` methods.
+- **Args:** metric name and verbosity flag.
+- **Returns:** fitted instance with ``threshold_`` attribute after calling ``fit``.
+
+## Examples
+
+- [Cross-validation and gradient methods](examples/comscore.ipynb)
+
+## Authors
+
+Suriyan Laohaprapanon and Gaurav Sood

optimal_classification_cutoffs-0.1.0.dist-info/RECORD

@@ -0,0 +1,10 @@
+optimal_classification_cutoffs-0.1.0.dist-info/licenses/LICENSE,sha256=UIp30ue1HZit7tMmSK0SS3swJBqOcLLnLJn5LY5YdNE,1036
+optimal_cutoffs/__init__.py,sha256=bWxw96ELKUsgmIY6wpmz90xzZN1lT_bEUaMw0hztsGY,628
+optimal_cutoffs/cv.py,sha256=qTKzENz4sXDybwZlF1rMHqnpa5PO_d4KvQhD2jy54vA,3130
+optimal_cutoffs/metrics.py,sha256=DEPz4y55paCoidz-NkybA0nNY-lwai37JM-93-XYOAA,3103
+optimal_cutoffs/optimizers.py,sha256=jEia55zi5xtIcYUCaqfDPKQ4Q-Do7bTr-5YcN5eREok,4545
+optimal_cutoffs/wrapper.py,sha256=QQveckCCAKcARIbczWmUUjyVXHTb-Qqu-ww-r4jvsLU,1824
+optimal_classification_cutoffs-0.1.0.dist-info/METADATA,sha256=zDqlXvnZbLHDcKfD77WdYQBy0S6GtM7wBxaNnDgtBro,5137
+optimal_classification_cutoffs-0.1.0.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+optimal_classification_cutoffs-0.1.0.dist-info/top_level.txt,sha256=Vjgzvn80t_BEJJZ1ERgMsXbrg_AYLIyPDoLwx9yFdpQ,16
+optimal_classification_cutoffs-0.1.0.dist-info/RECORD,,

optimal_classification_cutoffs-0.1.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (80.9.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

optimal_classification_cutoffs-0.1.0.dist-info/licenses/LICENSE

@@ -0,0 +1,19 @@
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

optimal_classification_cutoffs-0.1.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+optimal_cutoffs

optimal_cutoffs/__init__.py

@@ -0,0 +1,23 @@
+"""Top-level package for optimal classification cutoff utilities."""
+
+from .metrics import (
+    get_confusion_matrix,
+    METRIC_REGISTRY,
+    register_metric,
+    register_metrics,
+)
+from .optimizers import get_probability, get_optimal_threshold
+from .cv import cv_threshold_optimization, nested_cv_threshold_optimization
+from .wrapper import ThresholdOptimizer
+
+__all__ = [
+    "get_confusion_matrix",
+    "METRIC_REGISTRY",
+    "register_metric",
+    "register_metrics",
+    "get_probability",
+    "get_optimal_threshold",
+    "cv_threshold_optimization",
+    "nested_cv_threshold_optimization",
+    "ThresholdOptimizer",
+]

optimal_cutoffs/cv.py

@@ -0,0 +1,107 @@
+"""Cross-validation helpers for threshold optimization."""
+
+from __future__ import annotations
+
+import numpy as np
+from sklearn.model_selection import KFold
+
+from .optimizers import get_optimal_threshold, _metric_score
+
+
+def cv_threshold_optimization(
+    true_labs,
+    pred_prob,
+    metric="f1",
+    method="smart_brute",
+    cv=5,
+    random_state=None,
+):
+    """Estimate an optimal threshold using cross-validation.
+
+    Parameters
+    ----------
+    true_labs:
+        Array of true binary labels.
+    pred_prob:
+        Predicted probabilities from a classifier.
+    metric:
+        Metric name to optimize; must exist in the metric registry.
+    method:
+        Optimization strategy passed to
+        :func:`~optimal_cutoffs.optimizers.get_optimal_threshold`.
+    cv:
+        Number of folds for :class:`~sklearn.model_selection.KFold` cross-validation.
+    random_state:
+        Seed for the cross-validator shuffling.
+
+    Returns
+    -------
+    tuple[numpy.ndarray, numpy.ndarray]
+        Arrays of per-fold thresholds and scores.
+    """
+
+    kf = KFold(n_splits=cv, shuffle=True, random_state=random_state)
+    thresholds = []
+    scores = []
+    for train_idx, test_idx in kf.split(true_labs):
+        thr = get_optimal_threshold(
+            true_labs[train_idx], pred_prob[train_idx], metric=metric, method=method
+        )
+        thresholds.append(thr)
+        score = _metric_score(true_labs[test_idx], pred_prob[test_idx], thr, metric)
+        scores.append(score)
+    return np.array(thresholds), np.array(scores)
+
+
+def nested_cv_threshold_optimization(
+    true_labs,
+    pred_prob,
+    metric="f1",
+    method="smart_brute",
+    inner_cv=5,
+    outer_cv=5,
+    random_state=None,
+):
+    """Nested cross-validation for threshold optimization.
+
+    Parameters
+    ----------
+    true_labs:
+        Array of true binary labels.
+    pred_prob:
+        Predicted probabilities from a classifier.
+    metric:
+        Metric name to optimize.
+    method:
+        Optimization strategy passed to
+        :func:`~optimal_cutoffs.optimizers.get_optimal_threshold`.
+    inner_cv:
+        Number of folds in the inner loop used to estimate thresholds.
+    outer_cv:
+        Number of outer folds for unbiased performance assessment.
+    random_state:
+        Seed for the cross-validators.
+
+    Returns
+    -------
+    tuple[numpy.ndarray, numpy.ndarray]
+        Arrays of outer-fold thresholds and scores.
+    """
+
+    outer = KFold(n_splits=outer_cv, shuffle=True, random_state=random_state)
+    outer_thresholds = []
+    outer_scores = []
+    for train_idx, test_idx in outer.split(true_labs):
+        inner_thresholds, _ = cv_threshold_optimization(
+            true_labs[train_idx],
+            pred_prob[train_idx],
+            metric=metric,
+            method=method,
+            cv=inner_cv,
+            random_state=random_state,
+        )
+        thr = float(np.mean(inner_thresholds))
+        outer_thresholds.append(thr)
+        score = _metric_score(true_labs[test_idx], pred_prob[test_idx], thr, metric)
+        outer_scores.append(score)
+    return np.array(outer_thresholds), np.array(outer_scores)

optimal_cutoffs/metrics.py

@@ -0,0 +1,113 @@
+"""Metric registry, confusion matrix utilities, and built-in metrics."""
+
+from typing import Callable, Dict
+import numpy as np
+
+METRIC_REGISTRY: Dict[str, Callable[[int, int, int, int], float]] = {}
+
+
+def register_metric(name: str = None, func: Callable[[int, int, int, int], float] = None):
+    """Register a metric function.
+
+    Parameters
+    ----------
+    name:
+        Optional key under which to store the metric. If not provided the
+        function's ``__name__`` is used.
+    func:
+        Metric callable accepting ``tp, tn, fp, fn``. When supplied the
+        function is registered immediately. If omitted, the returned decorator
+        can be used to annotate a metric function.
+
+    Returns
+    -------
+    Callable
+        The registered function or decorator.
+    """
+    if func is not None:
+        METRIC_REGISTRY[name or func.__name__] = func
+        return func
+
+    def decorator(f: Callable[[int, int, int, int], float]):
+        METRIC_REGISTRY[name or f.__name__] = f
+        return f
+
+    return decorator
+
+
+def register_metrics(metrics: Dict[str, Callable[[int, int, int, int], float]]):
+    """Register multiple metric functions.
+
+    Parameters
+    ----------
+    metrics:
+        Mapping of metric names to callables that accept ``tp, tn, fp, fn``.
+
+    Returns
+    -------
+    None
+        This function mutates the global :data:`METRIC_REGISTRY` in-place.
+    """
+    METRIC_REGISTRY.update(metrics)
+
+
+@register_metric("f1")
+def f1_score(tp: int, tn: int, fp: int, fn: int) -> float:
+    """Compute the F\ :sub:`1` score.
+
+    Parameters
+    ----------
+    tp, tn, fp, fn:
+        Elements of the confusion matrix.
+
+    Returns
+    -------
+    float
+        The harmonic mean of precision and recall.
+    """
+    precision = tp / (tp + fp) if tp + fp > 0 else 0.0
+    recall = tp / (tp + fn) if tp + fn > 0 else 0.0
+    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
+
+
+@register_metric("accuracy")
+def accuracy_score(tp: int, tn: int, fp: int, fn: int) -> float:
+    """Compute classification accuracy.
+
+    Parameters
+    ----------
+    tp, tn, fp, fn:
+        Elements of the confusion matrix.
+
+    Returns
+    -------
+    float
+        Ratio of correct predictions to total samples.
+    """
+    total = tp + tn + fp + fn
+    return (tp + tn) / total if total > 0 else 0.0
+
+
+def get_confusion_matrix(true_labs, pred_prob, prob):
+    """Compute confusion-matrix counts for a given threshold.
+
+    Parameters
+    ----------
+    true_labs:
+        Array of true binary labels.
+    pred_prob:
+        Array of predicted probabilities in ``[0, 1]``.
+    prob:
+        Decision threshold applied to ``pred_prob``.
+
+    Returns
+    -------
+    tuple[int, int, int, int]
+        Counts ``(tp, tn, fp, fn)``.
+    """
+    pred_labs = pred_prob > prob
+    tp = np.sum(np.logical_and(pred_labs == 1, true_labs == 1))
+    tn = np.sum(np.logical_and(pred_labs == 0, true_labs == 0))
+    fp = np.sum(np.logical_and(pred_labs == 1, true_labs == 0))
+    fn = np.sum(np.logical_and(pred_labs == 0, true_labs == 1))
+    return tp, tn, fp, fn

optimal_cutoffs/optimizers.py

@@ -0,0 +1,126 @@
+"""Threshold search strategies for optimizing classification metrics."""
+
+from __future__ import annotations
+
+import numpy as np
+from scipy import optimize
+
+from .metrics import METRIC_REGISTRY, get_confusion_matrix
+
+
+def _accuracy(prob, true_labs, pred_prob, verbose=False):
+    tp, tn, fp, fn = get_confusion_matrix(true_labs, pred_prob, prob[0])
+    accuracy = (tp + tn) / (tp + tn + fp + fn)
+    if verbose:
+        print(f"Probability: {prob[0]:0.4f} Accuracy: {accuracy:0.4f}")
+    return 1 - accuracy
+
+
+def _f1(prob, true_labs, pred_prob, verbose=False):
+    tp, tn, fp, fn = get_confusion_matrix(true_labs, pred_prob, prob[0])
+    precision = tp / (tp + fp) if tp + fp > 0 else 0.0
+    recall = tp / (tp + fn) if tp + fn > 0 else 0.0
+    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
+    if verbose:
+        print(f"Probability: {prob[0]:0.4f} F1 score: {f1:0.4f}")
+    return 1 - f1
+
+
+def get_probability(true_labs, pred_prob, objective="accuracy", verbose=False):
+    """Brute-force search for a simple metric's best threshold.
+
+    Parameters
+    ----------
+    true_labs:
+        Array of true binary labels.
+    pred_prob:
+        Predicted probabilities from a classifier.
+    objective:
+        Metric to optimize. Supported values are ``"accuracy"`` and ``"f1"``.
+    verbose:
+        If ``True``, print intermediate metric values during the search.
+
+    Returns
+    -------
+    float
+        Threshold that maximizes the specified metric.
+    """
+    if objective == "accuracy":
+        prob = optimize.brute(
+            _accuracy, (slice(0.1, 0.9, 0.1),), args=(true_labs, pred_prob, verbose), disp=verbose
+        )
+    elif objective == "f1":
+        prob = optimize.brute(
+            _f1, (slice(0.1, 0.9, 0.1),), args=(true_labs, pred_prob, verbose), disp=verbose
+        )
+    else:
+        raise ValueError(f"Unknown objective: {objective}")
+    return float(prob[0] if isinstance(prob, np.ndarray) else prob)
+
+
+def _metric_score(true_labs, pred_prob, threshold, metric="f1"):
+    """Compute a metric score for a given threshold using registry metrics."""
+    tp, tn, fp, fn = get_confusion_matrix(true_labs, pred_prob, threshold)
+    try:
+        metric_func = METRIC_REGISTRY[metric]
+    except KeyError as exc:
+        raise ValueError(f"Unknown metric: {metric}") from exc
+    return float(metric_func(tp, tn, fp, fn))
+
+
+def get_optimal_threshold(true_labs, pred_prob, metric="f1", method="smart_brute"):
+    """Find the threshold that optimizes a metric.
+
+    Parameters
+    ----------
+    true_labs:
+        Array of true binary labels.
+    pred_prob:
+        Predicted probabilities from a classifier.
+    metric:
+        Name of a metric registered in :data:`~optimal_cutoffs.metrics.METRIC_REGISTRY`.
+    method:
+        Strategy used for optimization: ``"smart_brute"`` evaluates all unique
+        probabilities, ``"minimize"`` uses ``scipy.optimize.minimize_scalar``,
+        and ``"gradient"`` performs a simple gradient ascent.
+
+    Returns
+    -------
+    float
+        The threshold that maximizes the chosen metric.
+    """
+    if method == "smart_brute":
+        thresholds = np.unique(pred_prob)
+        scores = [_metric_score(true_labs, pred_prob, t, metric) for t in thresholds]
+        return float(thresholds[int(np.argmax(scores))])
+
+    if method == "minimize":
+        res = optimize.minimize_scalar(
+            lambda t: -_metric_score(true_labs, pred_prob, t, metric),
+            bounds=(0, 1),
+            method="bounded",
+        )
+        # ``minimize_scalar`` may return a threshold that is suboptimal for
+        # piecewise-constant metrics like F1. To provide a more robust
+        # solution, also evaluate all unique predicted probabilities and pick
+        # whichever threshold yields the highest score.
+        candidates = np.unique(np.append(pred_prob, res.x))
+        scores = [_metric_score(true_labs, pred_prob, t, metric) for t in candidates]
+        return float(candidates[int(np.argmax(scores))])
+
+    if method == "gradient":
+        threshold = 0.5
+        lr = 0.1
+        eps = 1e-5
+        for _ in range(100):
+            grad = (
+                _metric_score(true_labs, pred_prob, threshold + eps, metric)
+                - _metric_score(true_labs, pred_prob, threshold - eps, metric)
+            ) / (2 * eps)
+            threshold = np.clip(threshold + lr * grad, 0.0, 1.0)
+        return float(threshold)
+
+    raise ValueError(f"Unknown method: {method}")
+
+
+__all__ = ["get_probability", "get_optimal_threshold"]

optimal_cutoffs/wrapper.py

@@ -0,0 +1,62 @@
+"""High-level wrapper for threshold optimization."""
+
+from __future__ import annotations
+
+from .optimizers import get_probability
+
+
+class ThresholdOptimizer:
+    """Brute-force optimizer for classification thresholds.
+
+    The class wraps :func:`optimal_cutoffs.optimizers.get_probability` and
+    exposes a scikit-learn style ``fit``/``predict`` API.
+    """
+
+    def __init__(self, objective: str = "accuracy", verbose: bool = False):
+        """Create a new optimizer.
+
+        Parameters
+        ----------
+        objective:
+            Metric to optimize, e.g. ``"accuracy"`` or ``"f1"``.
+        verbose:
+            If ``True``, print progress during threshold search.
+        """
+        self.objective = objective
+        self.verbose = verbose
+        self.threshold_ = None
+
+    def fit(self, true_labs, pred_prob):
+        """Estimate the optimal threshold from labeled data.
+
+        Parameters
+        ----------
+        true_labs:
+            Array of true binary labels.
+        pred_prob:
+            Predicted probabilities from a classifier.
+
+        Returns
+        -------
+        ThresholdOptimizer
+            Fitted instance with ``threshold_`` attribute set.
+        """
+        self.threshold_ = get_probability(true_labs, pred_prob, self.objective, self.verbose)
+        return self
+
+    def predict(self, pred_prob):
+        """Convert probabilities to class predictions using the learned threshold.
+
+        Parameters
+        ----------
+        pred_prob:
+            Array of predicted probabilities to be thresholded.
+
+        Returns
+        -------
+        numpy.ndarray
+            Boolean array of predicted class labels.
+        """
+        if self.threshold_ is None:
+            raise RuntimeError("ThresholdOptimizer has not been fitted.")
+        return pred_prob > self.threshold_