jevkit-calibrate 0.1.0__tar.gz

jevkit_calibrate-0.1.0/.gitignore

@@ -0,0 +1,12 @@
+__pycache__/
+*.py[cod]
+.venv/
+venv/
+dist/
+build/
+*.egg-info/
+.pytest_cache/
+.coverage
+htmlcov/
+.ruff_cache/
+uv.lock

jevkit_calibrate-0.1.0/PKG-INFO

@@ -0,0 +1,113 @@
+Metadata-Version: 2.5
+Name: jevkit-calibrate
+Version: 0.1.0
+Summary: Verify TypeSafe Jev's calibration on your own data. Reliability diagrams, ECE, Brier, and confidence thresholds derived from labeled outcomes.
+Project-URL: Homepage, https://github.com/pjdurden/jevkit-py
+Project-URL: Issues, https://github.com/pjdurden/jevkit-py/issues
+Author: Prajjwal Chittori
+License-Expression: MIT
+Keywords: calibration,ece,jev,reliability,system-one,thresholds,typesafe
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Requires-Dist: jevkit-core>=0.2.0
+Description-Content-Type: text/markdown
+
+# jevkit-calibrate
+
+Calibration is Jev's central claim: the probabilities are meant to track real
+frequencies, so that among answers given 0.8, about 80% are right. TypeSafe
+measures this across groups of predictions and says plainly that it does not
+guarantee any individual answer.
+
+This package checks the claim on **your** data, which is the part nobody else
+can do for you, and turns the result into a threshold you can defend.
+
+> Unofficial and unaffiliated with TypeSafe.
+
+```bash
+pip install jevkit-calibrate
+```
+
+Pure standard library. No numpy, no pandas, no plotting stack. A calibration
+check that needs a build toolchain is a calibration check that does not get run.
+
+## Measure
+
+```python
+from jevkit_core import read_records
+from jevkit_calibrate import calibrate, observations_from_records
+
+observations = observations_from_records(read_records("labeled.jevl"))
+report = calibrate(observations)
+
+print(report.summary())
+print(report.diagram())
+```
+
+```
+observations: 4000
+accuracy:     0.7485
+mean claimed: 0.7469  (underconfident by 0.0016)
+ECE:          0.0094  (over 10 bins)
+MCE:          0.0167
+Brier:        0.1681
+log loss:     0.5043
+```
+
+`ECE` is the average gap between claimed and observed, weighted by how many
+observations fall in each bin. `MCE` is the worst gap in any one bin, which is
+what catches a healthy-looking ECE hiding one badly wrong region. `Brier` and
+`log loss` are proper scoring rules: they reward being calibrated **and**
+decisive, so a model that always says 0.5 scores badly even though it is
+perfectly calibrated.
+
+## Pick a threshold
+
+TypeSafe's confidence page recommends three bands and says where you draw them
+depends on your data. This draws them from the data.
+
+```python
+from jevkit_calibrate import recommend_for_accuracy
+
+point = recommend_for_accuracy(observations, target_accuracy=0.95)
+if point is None:
+    print("no threshold reaches 95% on this data")
+else:
+    print(f"threshold {point.threshold:.2f}: "
+          f"covers {point.coverage:.1%} at {point.accuracy:.1%}, "
+          f"{point.errors} wrong answers acted on")
+```
+
+`None` is a real answer, not a failure. It means this question cannot be
+automated at that bar, and the honest move is to change the question rather
+than lower the threshold.
+
+## CLI
+
+```bash
+jevkit-calibrate labeled.jevl
+jevkit-calibrate labeled.jevl --target-accuracy 0.95
+jevkit-calibrate labeled.jevl --min-coverage 0.80
+jevkit-calibrate labeled.jevl --max-ece 0.05     # CI gate
+jevkit-calibrate labeled.jevl --format json
+```
+
+## Which quantity gets calibrated
+
+By default, the probability mass on the chosen outcome, which is what "when it
+says 0.8, is it right 80% of the time" means.
+
+`--use-confidence` calibrates the API's `confidence` statistic instead. That is
+a different question, and the one to ask when you want to know whether your
+*routing* threshold sits in the right place. Nouls carry no confidence, so they
+fall back to probability either way.
+
+## License
+
+MIT

jevkit_calibrate-0.1.0/README.md

@@ -0,0 +1,93 @@
+# jevkit-calibrate
+
+Calibration is Jev's central claim: the probabilities are meant to track real
+frequencies, so that among answers given 0.8, about 80% are right. TypeSafe
+measures this across groups of predictions and says plainly that it does not
+guarantee any individual answer.
+
+This package checks the claim on **your** data, which is the part nobody else
+can do for you, and turns the result into a threshold you can defend.
+
+> Unofficial and unaffiliated with TypeSafe.
+
+```bash
+pip install jevkit-calibrate
+```
+
+Pure standard library. No numpy, no pandas, no plotting stack. A calibration
+check that needs a build toolchain is a calibration check that does not get run.
+
+## Measure
+
+```python
+from jevkit_core import read_records
+from jevkit_calibrate import calibrate, observations_from_records
+
+observations = observations_from_records(read_records("labeled.jevl"))
+report = calibrate(observations)
+
+print(report.summary())
+print(report.diagram())
+```
+
+```
+observations: 4000
+accuracy:     0.7485
+mean claimed: 0.7469  (underconfident by 0.0016)
+ECE:          0.0094  (over 10 bins)
+MCE:          0.0167
+Brier:        0.1681
+log loss:     0.5043
+```
+
+`ECE` is the average gap between claimed and observed, weighted by how many
+observations fall in each bin. `MCE` is the worst gap in any one bin, which is
+what catches a healthy-looking ECE hiding one badly wrong region. `Brier` and
+`log loss` are proper scoring rules: they reward being calibrated **and**
+decisive, so a model that always says 0.5 scores badly even though it is
+perfectly calibrated.
+
+## Pick a threshold
+
+TypeSafe's confidence page recommends three bands and says where you draw them
+depends on your data. This draws them from the data.
+
+```python
+from jevkit_calibrate import recommend_for_accuracy
+
+point = recommend_for_accuracy(observations, target_accuracy=0.95)
+if point is None:
+    print("no threshold reaches 95% on this data")
+else:
+    print(f"threshold {point.threshold:.2f}: "
+          f"covers {point.coverage:.1%} at {point.accuracy:.1%}, "
+          f"{point.errors} wrong answers acted on")
+```
+
+`None` is a real answer, not a failure. It means this question cannot be
+automated at that bar, and the honest move is to change the question rather
+than lower the threshold.
+
+## CLI
+
+```bash
+jevkit-calibrate labeled.jevl
+jevkit-calibrate labeled.jevl --target-accuracy 0.95
+jevkit-calibrate labeled.jevl --min-coverage 0.80
+jevkit-calibrate labeled.jevl --max-ece 0.05     # CI gate
+jevkit-calibrate labeled.jevl --format json
+```
+
+## Which quantity gets calibrated
+
+By default, the probability mass on the chosen outcome, which is what "when it
+says 0.8, is it right 80% of the time" means.
+
+`--use-confidence` calibrates the API's `confidence` statistic instead. That is
+a different question, and the one to ask when you want to know whether your
+*routing* threshold sits in the right place. Nouls carry no confidence, so they
+fall back to probability either way.
+
+## License
+
+MIT

jevkit_calibrate-0.1.0/pyproject.toml

@@ -0,0 +1,33 @@
+[project]
+name = "jevkit-calibrate"
+version = "0.1.0"
+description = "Verify TypeSafe Jev's calibration on your own data. Reliability diagrams, ECE, Brier, and confidence thresholds derived from labeled outcomes."
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+authors = [{ name = "Prajjwal Chittori" }]
+keywords = ["jev", "typesafe", "system-one", "calibration", "ece", "reliability", "thresholds"]
+classifiers = [
+  "Development Status :: 3 - Alpha",
+  "Intended Audience :: Developers",
+  "License :: OSI Approved :: MIT License",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Topic :: Software Development :: Testing",
+]
+dependencies = ["jevkit-core>=0.2.0"]
+
+[project.scripts]
+jevkit-calibrate = "jevkit_calibrate.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/pjdurden/jevkit-py"
+Issues = "https://github.com/pjdurden/jevkit-py/issues"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/jevkit_calibrate"]

jevkit_calibrate-0.1.0/src/jevkit_calibrate/__init__.py

@@ -0,0 +1,25 @@
+"""Verify Jev's calibration on your own data, and pick thresholds from it.
+
+Calibration is the central claim behind a System One model, and it is the one
+thing a user cannot check without tooling. This package computes reliability
+diagrams, ECE, MCE, Brier and log loss over labeled answers, and turns a
+labeled set into a defensible confidence threshold.
+
+Pure standard library. A calibration check that needs a numpy build is a
+calibration check that does not get run.
+"""
+
+from .metrics import (Bin, CalibrationReport, Observation, brier_score, calibrate,
+                      expected_calibration_error, log_loss,
+                      maximum_calibration_error, reliability_bins)
+from .records import observations_from_records
+from .thresholds import (ThresholdPoint, recommend_for_accuracy,
+                         recommend_for_coverage, sweep)
+
+__version__ = "0.1.0"
+
+__all__ = ["calibrate", "CalibrationReport", "Observation", "Bin",
+           "reliability_bins", "expected_calibration_error",
+           "maximum_calibration_error", "brier_score", "log_loss",
+           "observations_from_records", "sweep", "ThresholdPoint",
+           "recommend_for_accuracy", "recommend_for_coverage", "__version__"]

jevkit_calibrate-0.1.0/src/jevkit_calibrate/cli.py

@@ -0,0 +1,108 @@
+"""``jevkit-calibrate`` command line interface."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+
+from jevkit_core import RecordFormatError, read_records
+
+from .metrics import calibrate
+from .records import observations_from_records
+from .thresholds import recommend_for_accuracy, recommend_for_coverage
+
+EXIT_OK = 0
+EXIT_FAILED_GATE = 1
+EXIT_USAGE = 2
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(
+        prog="jevkit-calibrate",
+        description="Measure how well Jev's probabilities match outcomes on your labeled "
+                    "data, and pick a confidence threshold from it. Never calls the API.",
+    )
+    parser.add_argument("files", nargs="+", help=".jevl files containing labeled records")
+    parser.add_argument("--bins", type=int, default=10, help="reliability bins (default 10)")
+    parser.add_argument("--question", action="append", dest="questions",
+                        help="restrict to this question id (repeatable)")
+    parser.add_argument("--use-confidence", action="store_true",
+                        help="calibrate the API's confidence rather than the probability "
+                             "on the chosen outcome")
+    parser.add_argument("--target-accuracy", type=float,
+                        help="recommend the lowest threshold reaching this accuracy")
+    parser.add_argument("--min-coverage", type=float,
+                        help="recommend the highest threshold still covering this fraction")
+    parser.add_argument("--max-ece", type=float,
+                        help="exit non-zero if ECE exceeds this")
+    parser.add_argument("--format", choices=("text", "json"), default="text")
+    parser.add_argument("--no-diagram", action="store_true")
+    args = parser.parse_args(argv)
+
+    records = []
+    try:
+        for path in args.files:
+            records.extend(read_records(path))
+    except (OSError, RecordFormatError) as exc:
+        print(f"jevkit-calibrate: {exc}", file=sys.stderr)
+        return EXIT_USAGE
+
+    observations = observations_from_records(
+        records, question_ids=args.questions, use_confidence=args.use_confidence
+    )
+    if not observations:
+        print("jevkit-calibrate: no labeled observations found. Records need a 'label' "
+              "object keyed by question id.", file=sys.stderr)
+        return EXIT_USAGE
+
+    report = calibrate(observations, n_bins=args.bins)
+
+    recommendation = None
+    if args.target_accuracy is not None:
+        recommendation = ("accuracy", args.target_accuracy,
+                          recommend_for_accuracy(observations, args.target_accuracy))
+    elif args.min_coverage is not None:
+        recommendation = ("coverage", args.min_coverage,
+                          recommend_for_coverage(observations, args.min_coverage))
+
+    if args.format == "json":
+        payload = report.to_dict()
+        if recommendation:
+            kind, target, point = recommendation
+            payload["recommendation"] = {
+                "for": kind, "target": target,
+                "threshold": point.threshold if point else None,
+                "coverage": point.coverage if point else None,
+                "accuracy": point.accuracy if point else None,
+                "errors": point.errors if point else None,
+            }
+        print(json.dumps(payload, indent=2))
+    else:
+        print(report.summary())
+        if not args.no_diagram:
+            print()
+            print(report.diagram())
+        if recommendation:
+            kind, target, point = recommendation
+            print()
+            if point is None:
+                if kind == "accuracy":
+                    print(f"No threshold reaches {target:.1%} accuracy on this data. The "
+                          f"question cannot be automated at that bar; change the question "
+                          f"rather than the threshold.")
+                else:
+                    print(f"No threshold covers {target:.1%} of cases.")
+            else:
+                print(f"threshold {point.threshold:.2f}: covers {point.coverage:.1%} "
+                      f"({point.covered}/{point.total}) at {point.accuracy:.1%} accuracy, "
+                      f"{point.errors} wrong answer(s) acted on, {point.escalated} escalated")
+
+    if args.max_ece is not None and report.ece > args.max_ece:
+        print(f"jevkit-calibrate: ECE {report.ece:.4f} exceeds {args.max_ece}", file=sys.stderr)
+        return EXIT_FAILED_GATE
+    return EXIT_OK
+
+
+if __name__ == "__main__":  # pragma: no cover
+    raise SystemExit(main())

jevkit_calibrate-0.1.0/src/jevkit_calibrate/metrics.py

@@ -0,0 +1,264 @@
+"""Calibration metrics over labeled jev answers.
+
+Calibration is Jev's central claim: the probabilities are meant to reflect
+real-world frequencies, so that among answers given 0.8, about 80% are right.
+TypeSafe measures this across groups of predictions and says plainly that it
+does not guarantee any individual answer. This module is how you check the claim
+on *your* data, which is the part nobody else can do for you.
+
+Everything here is pure arithmetic over recorded answers. Nothing calls the API,
+and there are no dependencies beyond the standard library: a calibration check
+that requires a numpy build is a calibration check that does not get run.
+"""
+
+from __future__ import annotations
+
+import math
+from dataclasses import dataclass, field
+from typing import Iterable, Sequence
+
+__all__ = ["Observation", "Bin", "CalibrationReport", "reliability_bins",
+           "expected_calibration_error", "maximum_calibration_error",
+           "brier_score", "log_loss", "calibrate"]
+
+
+@dataclass(frozen=True)
+class Observation:
+    """One labeled prediction.
+
+    ``probability`` is the model's stated probability for the outcome it chose;
+    ``correct`` is whether that outcome matched the label.
+    """
+
+    probability: float
+    correct: bool
+    question_id: str = ""
+    request_id: str = ""
+
+    def __post_init__(self) -> None:
+        if not 0.0 <= self.probability <= 1.0:
+            raise ValueError(f"probability must be in [0, 1], got {self.probability}")
+
+
+@dataclass
+class Bin:
+    """One bucket of a reliability diagram."""
+
+    lower: float
+    upper: float
+    observations: list[Observation] = field(default_factory=list)
+
+    @property
+    def count(self) -> int:
+        return len(self.observations)
+
+    @property
+    def mean_probability(self) -> float:
+        """What the model claimed, on average."""
+        if not self.observations:
+            return 0.0
+        return sum(o.probability for o in self.observations) / self.count
+
+    @property
+    def accuracy(self) -> float:
+        """What actually happened."""
+        if not self.observations:
+            return 0.0
+        return sum(1 for o in self.observations if o.correct) / self.count
+
+    @property
+    def gap(self) -> float:
+        """Claimed minus observed. Positive means overconfident."""
+        return self.mean_probability - self.accuracy
+
+    def label(self) -> str:
+        return f"[{self.lower:.2f}, {self.upper:.2f}{']' if self.upper == 1.0 else ')'}"
+
+
+def reliability_bins(observations: Sequence[Observation], n_bins: int = 10) -> list[Bin]:
+    """Bucket observations into equal-width probability bins.
+
+    Equal-width rather than equal-count, because the question being asked is
+    "when the model says 0.9, is it right 90% of the time", and that question is
+    about fixed probability ranges. Empty bins are kept so the diagram does not
+    silently mislead about coverage.
+    """
+    if n_bins < 1:
+        raise ValueError("n_bins must be at least 1")
+    edges = [i / n_bins for i in range(n_bins + 1)]
+    bins = [Bin(lower=edges[i], upper=edges[i + 1]) for i in range(n_bins)]
+    for obs in observations:
+        # The top bin is closed so that a probability of exactly 1.0 lands in it.
+        index = min(int(obs.probability * n_bins), n_bins - 1)
+        bins[index].observations.append(obs)
+    return bins
+
+
+def expected_calibration_error(observations: Sequence[Observation], n_bins: int = 10) -> float:
+    """ECE: average gap between claimed and observed, weighted by bin population.
+
+    0 is perfect. Note that ECE depends on the bin count, so compare ECE values
+    only when they were computed with the same ``n_bins``.
+    """
+    if not observations:
+        return 0.0
+    total = len(observations)
+    return sum(
+        (b.count / total) * abs(b.gap) for b in reliability_bins(observations, n_bins) if b.count
+    )
+
+
+def maximum_calibration_error(observations: Sequence[Observation], n_bins: int = 10) -> float:
+    """MCE: the worst gap in any populated bin.
+
+    ECE can look healthy while one region is badly wrong. MCE is what catches
+    that, and it is the number that matters when a single region is where your
+    high-stakes decisions live.
+    """
+    if not observations:
+        return 0.0
+    return max((abs(b.gap) for b in reliability_bins(observations, n_bins) if b.count), default=0.0)
+
+
+def brier_score(observations: Sequence[Observation]) -> float:
+    """Mean squared error between probability and outcome. Lower is better.
+
+    Unlike ECE this is a proper scoring rule: it rewards being both calibrated
+    and decisive, so a model that always says 0.5 scores poorly even though it
+    is perfectly calibrated.
+    """
+    if not observations:
+        return 0.0
+    return sum((o.probability - (1.0 if o.correct else 0.0)) ** 2 for o in observations) / len(
+        observations
+    )
+
+
+def log_loss(observations: Sequence[Observation], eps: float = 1e-15) -> float:
+    """Mean negative log likelihood. Lower is better.
+
+    Punishes confident mistakes far harder than Brier does. ``eps`` clamps the
+    probability away from 0 and 1, since an unclamped confident miss is infinite
+    and would swamp every other observation.
+    """
+    if not observations:
+        return 0.0
+    total = 0.0
+    for o in observations:
+        p = min(max(o.probability, eps), 1.0 - eps)
+        total -= math.log(p) if o.correct else math.log(1.0 - p)
+    return total / len(observations)
+
+
+@dataclass
+class CalibrationReport:
+    observations: list[Observation]
+    n_bins: int = 10
+
+    @property
+    def count(self) -> int:
+        return len(self.observations)
+
+    @property
+    def accuracy(self) -> float:
+        if not self.observations:
+            return 0.0
+        return sum(1 for o in self.observations if o.correct) / self.count
+
+    @property
+    def mean_probability(self) -> float:
+        if not self.observations:
+            return 0.0
+        return sum(o.probability for o in self.observations) / self.count
+
+    @property
+    def bins(self) -> list[Bin]:
+        return reliability_bins(self.observations, self.n_bins)
+
+    @property
+    def ece(self) -> float:
+        return expected_calibration_error(self.observations, self.n_bins)
+
+    @property
+    def mce(self) -> float:
+        return maximum_calibration_error(self.observations, self.n_bins)
+
+    @property
+    def brier(self) -> float:
+        return brier_score(self.observations)
+
+    @property
+    def log_loss(self) -> float:
+        return log_loss(self.observations)
+
+    @property
+    def overconfident(self) -> bool:
+        """Claimed probability exceeds observed accuracy overall."""
+        return self.mean_probability > self.accuracy
+
+    def diagram(self, width: int = 28) -> str:
+        """A text reliability diagram.
+
+        Each row shows a bin's claimed probability against what actually
+        happened, so a miscalibrated region is visible without plotting.
+        """
+        lines = [
+            f"{'bin':<14} {'n':>5} {'claimed':>8} {'actual':>8} {'gap':>7}",
+            "-" * 46,
+        ]
+        for b in self.bins:
+            if not b.count:
+                lines.append(f"{b.label():<14} {0:>5} {'-':>8} {'-':>8} {'-':>7}")
+                continue
+            bar_len = int(b.accuracy * width)
+            marker = int(b.mean_probability * width)
+            cells = ["#" if i < bar_len else " " for i in range(width)]
+            if 0 <= marker < width:
+                cells[marker] = "|" if cells[marker] == " " else "+"
+            lines.append(
+                f"{b.label():<14} {b.count:>5} {b.mean_probability:>8.3f} "
+                f"{b.accuracy:>8.3f} {b.gap:>+7.3f}  {''.join(cells)}"
+            )
+        lines.append("")
+        lines.append("  # observed accuracy, | claimed probability, + both")
+        return "\n".join(lines)
+
+    def summary(self) -> str:
+        if not self.count:
+            return "no labeled observations"
+        direction = "overconfident" if self.overconfident else "underconfident"
+        return "\n".join([
+            f"observations: {self.count}",
+            f"accuracy:     {self.accuracy:.4f}",
+            f"mean claimed: {self.mean_probability:.4f}  ({direction} "
+            f"by {abs(self.mean_probability - self.accuracy):.4f})",
+            f"ECE:          {self.ece:.4f}  (over {self.n_bins} bins)",
+            f"MCE:          {self.mce:.4f}",
+            f"Brier:        {self.brier:.4f}",
+            f"log loss:     {self.log_loss:.4f}",
+        ])
+
+    def to_dict(self) -> dict[str, object]:
+        return {
+            "count": self.count,
+            "accuracy": self.accuracy,
+            "mean_probability": self.mean_probability,
+            "overconfident": self.overconfident,
+            "ece": self.ece,
+            "mce": self.mce,
+            "brier": self.brier,
+            "log_loss": self.log_loss,
+            "n_bins": self.n_bins,
+            "bins": [
+                {
+                    "lower": b.lower, "upper": b.upper, "count": b.count,
+                    "mean_probability": b.mean_probability, "accuracy": b.accuracy,
+                    "gap": b.gap,
+                }
+                for b in self.bins
+            ],
+        }
+
+
+def calibrate(observations: Iterable[Observation], n_bins: int = 10) -> CalibrationReport:
+    return CalibrationReport(observations=list(observations), n_bins=n_bins)

jevkit_calibrate-0.1.0/src/jevkit_calibrate/records.py

@@ -0,0 +1,59 @@
+"""Turning labeled `.jevl` records into calibration observations."""
+
+from __future__ import annotations
+
+from typing import Iterable
+
+from jevkit_core import Record, parse_answers
+
+from .metrics import Observation
+
+__all__ = ["observations_from_records", "SKIPPED_UNLABELED"]
+
+SKIPPED_UNLABELED = "unlabeled"
+
+
+def observations_from_records(
+    records: Iterable[Record],
+    *,
+    question_ids: Iterable[str] | None = None,
+    use_confidence: bool = False,
+) -> list[Observation]:
+    """Extract one observation per labeled answer.
+
+    Records with no ``label`` are skipped: calibration needs ground truth, and
+    silently treating an unlabeled record as correct or incorrect would poison
+    every number downstream.
+
+    ``use_confidence`` picks which quantity is being calibrated. The default,
+    ``False``, uses the probability mass on the chosen outcome, which is what
+    "when it says 0.8, is it right 80% of the time" means. Setting it to ``True``
+    calibrates the API's ``confidence`` statistic instead, which is a different
+    question and answers whether your *routing* threshold is well placed. Nouls
+    have no confidence, so they fall back to the probability either way.
+    """
+    wanted = set(question_ids) if question_ids is not None else None
+    out: list[Observation] = []
+
+    for record in records:
+        if not record.label:
+            continue
+        answers = parse_answers(record.answers)
+        for qid, label in record.label.items():
+            if wanted is not None and qid not in wanted:
+                continue
+            answer = answers.get(qid)
+            if answer is None:
+                continue
+            correct = answer.is_correct(label)
+            if use_confidence and answer.confidence is not None:
+                probability = answer.confidence
+            else:
+                probability = answer.top_probability
+            out.append(Observation(
+                probability=probability,
+                correct=correct,
+                question_id=qid,
+                request_id=record.request_id,
+            ))
+    return out

jevkit_calibrate-0.1.0/src/jevkit_calibrate/thresholds.py

@@ -0,0 +1,112 @@
+"""Choosing a confidence threshold from labeled data instead of guessing.
+
+TypeSafe's confidence page recommends three bands: act automatically, proceed
+with caution, escalate. It also says plainly that where you draw those lines
+depends on your domain and your data. This module draws them from the data.
+
+The trade is always the same. Raising the threshold means acting on fewer cases
+but being right more often on the ones you do act on. That is a curve, not a
+number, so ``sweep`` returns the whole curve and the ``recommend_*`` functions
+pick a point on it against a constraint you state.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Sequence
+
+from .metrics import Observation
+
+__all__ = ["ThresholdPoint", "sweep", "recommend_for_accuracy", "recommend_for_coverage"]
+
+
+@dataclass(frozen=True)
+class ThresholdPoint:
+    """What happens if you auto-handle everything at or above ``threshold``."""
+
+    threshold: float
+    covered: int
+    total: int
+    correct: int
+
+    @property
+    def coverage(self) -> float:
+        """Fraction of cases handled automatically."""
+        return self.covered / self.total if self.total else 0.0
+
+    @property
+    def accuracy(self) -> float:
+        """Accuracy on the automatically handled cases.
+
+        Defined as 1.0 when nothing is covered: a threshold that acts on nothing
+        is never wrong. Read it together with ``coverage``, never alone.
+        """
+        return self.correct / self.covered if self.covered else 1.0
+
+    @property
+    def escalated(self) -> int:
+        return self.total - self.covered
+
+    @property
+    def errors(self) -> int:
+        """Wrong answers you acted on. Usually the number that actually costs."""
+        return self.covered - self.correct
+
+
+def sweep(observations: Sequence[Observation], steps: int = 101) -> list[ThresholdPoint]:
+    """Coverage and accuracy at every threshold from 0 to 1.
+
+    Uses the probability the model put on the outcome it chose, so this works
+    for Choice and Score confidence and for a Noul's distance from 0.5 alike, as
+    long as the caller is consistent about which it fed in.
+    """
+    if steps < 2:
+        raise ValueError("steps must be at least 2")
+    total = len(observations)
+    points: list[ThresholdPoint] = []
+    for i in range(steps):
+        threshold = i / (steps - 1)
+        covered = [o for o in observations if o.probability >= threshold]
+        points.append(ThresholdPoint(
+            threshold=threshold,
+            covered=len(covered),
+            total=total,
+            correct=sum(1 for o in covered if o.correct),
+        ))
+    return points
+
+
+def recommend_for_accuracy(
+    observations: Sequence[Observation],
+    target_accuracy: float,
+    *,
+    steps: int = 101,
+) -> ThresholdPoint | None:
+    """Lowest threshold reaching ``target_accuracy``, so coverage stays highest.
+
+    Returns ``None`` when no threshold reaches the target, which is a real
+    answer: it means this question cannot be automated at that accuracy and the
+    honest move is to change the question rather than the threshold.
+    """
+    if not 0.0 <= target_accuracy <= 1.0:
+        raise ValueError("target_accuracy must be in [0, 1]")
+    candidates = [p for p in sweep(observations, steps) if p.covered and p.accuracy >= target_accuracy]
+    return min(candidates, key=lambda p: p.threshold) if candidates else None
+
+
+def recommend_for_coverage(
+    observations: Sequence[Observation],
+    min_coverage: float,
+    *,
+    steps: int = 101,
+) -> ThresholdPoint | None:
+    """Highest threshold still covering ``min_coverage``, so accuracy is best.
+
+    The mirror of ``recommend_for_accuracy``: use it when throughput is the
+    binding constraint and you want the most accurate threshold that still keeps
+    enough volume out of the review queue.
+    """
+    if not 0.0 <= min_coverage <= 1.0:
+        raise ValueError("min_coverage must be in [0, 1]")
+    candidates = [p for p in sweep(observations, steps) if p.coverage >= min_coverage]
+    return max(candidates, key=lambda p: p.threshold) if candidates else None

jevkit_calibrate-0.1.0/tests/test_calibrate.py

@@ -0,0 +1,131 @@
+import random
+import pytest
+from jevkit_core import Record
+from jevkit_calibrate import (Observation, brier_score, calibrate,
+                              expected_calibration_error, log_loss,
+                              maximum_calibration_error, observations_from_records,
+                              recommend_for_accuracy, recommend_for_coverage, sweep)
+
+
+def perfect(n=2000, seed=1):
+    rng = random.Random(seed)
+    out = []
+    for _ in range(n):
+        p = rng.uniform(0.5, 1.0)
+        out.append(Observation(probability=p, correct=rng.random() < p))
+    return out
+
+
+def test_probability_outside_the_unit_interval_is_rejected():
+    for bad in (-0.1, 1.1):
+        with pytest.raises(ValueError, match="probability must be"):
+            Observation(probability=bad, correct=True)
+
+
+def test_a_perfectly_calibrated_generator_has_near_zero_ece():
+    assert expected_calibration_error(perfect()) < 0.05
+
+
+def test_a_maximally_overconfident_model_has_ece_near_one():
+    obs = [Observation(probability=1.0, correct=False) for _ in range(100)]
+    assert expected_calibration_error(obs) == pytest.approx(1.0)
+
+
+def test_mce_catches_one_bad_region_that_ece_dilutes():
+    good = [Observation(probability=0.5, correct=i % 2 == 0) for i in range(998)]
+    bad = [Observation(probability=1.0, correct=False) for _ in range(2)]
+    obs = good + bad
+    assert expected_calibration_error(obs) < 0.01   # diluted
+    assert maximum_calibration_error(obs) == pytest.approx(1.0)  # caught
+
+
+def test_brier_rewards_being_decisive_as_well_as_calibrated():
+    always_half = [Observation(probability=0.5, correct=i % 2 == 0) for i in range(100)]
+    decisive = [Observation(probability=1.0, correct=True) for _ in range(100)]
+    assert brier_score(decisive) < brier_score(always_half)
+
+
+def test_log_loss_is_finite_on_a_confident_miss():
+    assert log_loss([Observation(probability=1.0, correct=False)]) < float("inf")
+
+
+def test_empty_input_produces_zeros_not_errors():
+    assert expected_calibration_error([]) == 0.0
+    assert brier_score([]) == 0.0
+    assert calibrate([]).summary() == "no labeled observations"
+
+
+def test_bins_cover_the_unit_interval_and_include_one():
+    report = calibrate([Observation(probability=1.0, correct=True)], n_bins=10)
+    populated = [b for b in report.bins if b.count]
+    assert len(populated) == 1 and populated[0].upper == 1.0
+
+
+def test_empty_bins_are_kept_so_coverage_is_visible():
+    report = calibrate([Observation(probability=0.95, correct=True)], n_bins=10)
+    assert len(report.bins) == 10
+    assert sum(1 for b in report.bins if b.count == 0) == 9
+
+
+def test_overconfidence_is_detected():
+    obs = [Observation(probability=0.9, correct=i < 50) for i in range(100)]
+    assert calibrate(obs).overconfident
+
+
+def test_diagram_renders_without_a_plotting_library():
+    out = calibrate(perfect(200)).diagram()
+    assert "claimed" in out and "actual" in out
+
+
+def test_sweep_is_monotone_in_coverage():
+    points = sweep(perfect(500))
+    coverages = [p.coverage for p in points]
+    assert coverages == sorted(coverages, reverse=True)
+
+
+def test_threshold_for_accuracy_picks_the_lowest_qualifying_one():
+    obs = perfect(3000)
+    point = recommend_for_accuracy(obs, 0.85)
+    assert point is not None and point.accuracy >= 0.85
+    lower = [p for p in sweep(obs) if p.threshold < point.threshold and p.covered]
+    assert all(p.accuracy < 0.85 for p in lower)
+
+
+def test_unreachable_accuracy_returns_none_rather_than_a_bad_threshold():
+    obs = [Observation(probability=0.6, correct=False) for _ in range(100)]
+    assert recommend_for_accuracy(obs, 0.99) is None
+
+
+def test_threshold_for_coverage_picks_the_highest_qualifying_one():
+    point = recommend_for_coverage(perfect(1000), 0.5)
+    assert point is not None and point.coverage >= 0.5
+
+
+def test_accuracy_is_one_when_nothing_is_covered():
+    obs = [Observation(probability=0.1, correct=False)]
+    top = sweep(obs)[-1]
+    assert top.covered == 0 and top.accuracy == 1.0
+
+
+def test_records_without_labels_are_skipped():
+    unlabeled = Record(model="m", state="s", questions={"q": {"type": "noul"}},
+                       answers={"q": {"type": "noul", "noul": 0.9}})
+    assert observations_from_records([unlabeled]) == []
+
+
+def test_observations_are_extracted_from_labeled_records():
+    labeled = Record(model="m", state="s", questions={"q": {"type": "choice"}},
+                     answers={"q": {"type": "choice", "choice": "a",
+                                    "probabilities": {"a": 0.8, "b": 0.2}, "confidence": 0.6}},
+                     label={"q": "a"})
+    (obs,) = observations_from_records([labeled])
+    assert obs.correct and obs.probability == pytest.approx(0.8)
+
+
+def test_use_confidence_selects_the_other_quantity():
+    labeled = Record(model="m", state="s", questions={"q": {"type": "choice"}},
+                     answers={"q": {"type": "choice", "choice": "a",
+                                    "probabilities": {"a": 0.8, "b": 0.2}, "confidence": 0.6}},
+                     label={"q": "a"})
+    (obs,) = observations_from_records([labeled], use_confidence=True)
+    assert obs.probability == pytest.approx(0.6)