quant-regress 0.1.0__tar.gz

quant_regress-0.1.0/PKG-INFO

@@ -0,0 +1,89 @@
+Metadata-Version: 2.4
+Name: quant-regress
+Version: 0.1.0
+Summary: Fail CI when quantization costs you task accuracy. Task-level regression testing for quantized models.
+Author: DawnofGenX
+License: MIT
+Project-URL: Homepage, https://github.com/DawnofGenX/quant-regress
+Project-URL: Issues, https://github.com/DawnofGenX/quant-regress/issues
+Keywords: quantization,regression-testing,evaluation,ci,llm
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: torch>=2.0
+Requires-Dist: transformers>=4.40
+Requires-Dist: numpy>=1.24
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+
+# quant-regress
+
+Fail CI when quantization costs you task accuracy.
+
+Token-level proxies — perplexity, KL divergence — routinely report that a
+quantized model is fine. Task accuracy does not, provided you actually measure
+it. This action measures a task eval at each precision and fails the build when
+accuracy drops past a threshold you set.
+
+## Why
+
+<!-- claim:int8-collapse -->
+On a cross-encoder claim verifier, dynamic INT8 quantization left aggregate
+accuracy looking survivable while destroying the capability that mattered:
+
+<!-- claim:int8-collapse -->
+| precision | accuracy | correct |
+|---|---|---|
+| fp32 (baseline) | 100.0% <!-- claim:int8-collapse --> | 20/20 |
+| int8 | 0.0% <!-- claim:int8-collapse --> | 0/20 |
+
+Regenerate with `python scripts/regen_int8_collapse.py` — every figure above
+comes from that script, and `tests/test_claims_are_reproducible.py` fails the
+build if a number in this file stops matching it.
+
+The pattern that matters: the quantized model still runs, still returns a valid
+answer, and still looks like a working model. It is simply wrong every time.
+That is why a token-level proxy can clear it and a task eval cannot.
+
+## Usage
+
+```yaml
+- uses: DawnofGenX/quant-regress@v1
+  with:
+    eval: evals/tasks.jsonl
+    model: meta-llama/Llama-3.2-1B-Instruct
+    max-drop-points: 2
+```
+
+Exit codes: `0` within tolerance, `1` accuracy regressed, `2` bad configuration.
+
+## Constraints worth knowing
+
+- **CPU only.** `quantize_dynamic` emits CPU-only quantized kernels; moving the
+  model to CUDA raises at forward time. GitHub-hosted runners have no GPU, so
+  this is not a limitation in CI — but it does mean GPU-only quantizers
+  (AWQ/GPTQ/GGUF) are out of scope for v1.
+- **The threshold unit is accuracy points**, not a ratio. `--max-drop-points: 2`
+  means "fail if any candidate is more than 2 points below the baseline".
+- Comparison is exact-after-normalisation by default. Pass a scorer if you need
+  semantic equivalence.
+- Models are downloaded at run time. A 1B model is ~2 GB; check runner disk.
+
+## Honest limits
+
+- Only `fp32` and dynamic `int8` are supported in v1.
+- The library runs generation, so it needs a task whose success is checkable in
+  code. It is not an LLM-judge harness.
+- **The bundled `evals/fixtures/collapse.jsonl` is model-specific.** Its cases
+  expect the answer `yes`, so it only demonstrates a collapse for a model that
+  answers `yes` when unquantized. Pointed at an untuned tiny LM it scores 0/N at
+  both precisions, produces no drop, and the gate correctly reports PASS. Use it
+  to see the output format; bring your own eval for a real check.
+- What the repo's own CI pins is the **exit-code contract** (0 holds, 1 regressed,
+  2 misconfigured), not any particular model's accuracy — see
+  `tests/test_selftest_gate.py`.
+- No published adoption yet. If you use it, an issue saying what you quantized
+  and what broke is genuinely useful.

quant_regress-0.1.0/README.md

@@ -0,0 +1,68 @@
+# quant-regress
+
+Fail CI when quantization costs you task accuracy.
+
+Token-level proxies — perplexity, KL divergence — routinely report that a
+quantized model is fine. Task accuracy does not, provided you actually measure
+it. This action measures a task eval at each precision and fails the build when
+accuracy drops past a threshold you set.
+
+## Why
+
+<!-- claim:int8-collapse -->
+On a cross-encoder claim verifier, dynamic INT8 quantization left aggregate
+accuracy looking survivable while destroying the capability that mattered:
+
+<!-- claim:int8-collapse -->
+| precision | accuracy | correct |
+|---|---|---|
+| fp32 (baseline) | 100.0% <!-- claim:int8-collapse --> | 20/20 |
+| int8 | 0.0% <!-- claim:int8-collapse --> | 0/20 |
+
+Regenerate with `python scripts/regen_int8_collapse.py` — every figure above
+comes from that script, and `tests/test_claims_are_reproducible.py` fails the
+build if a number in this file stops matching it.
+
+The pattern that matters: the quantized model still runs, still returns a valid
+answer, and still looks like a working model. It is simply wrong every time.
+That is why a token-level proxy can clear it and a task eval cannot.
+
+## Usage
+
+```yaml
+- uses: DawnofGenX/quant-regress@v1
+  with:
+    eval: evals/tasks.jsonl
+    model: meta-llama/Llama-3.2-1B-Instruct
+    max-drop-points: 2
+```
+
+Exit codes: `0` within tolerance, `1` accuracy regressed, `2` bad configuration.
+
+## Constraints worth knowing
+
+- **CPU only.** `quantize_dynamic` emits CPU-only quantized kernels; moving the
+  model to CUDA raises at forward time. GitHub-hosted runners have no GPU, so
+  this is not a limitation in CI — but it does mean GPU-only quantizers
+  (AWQ/GPTQ/GGUF) are out of scope for v1.
+- **The threshold unit is accuracy points**, not a ratio. `--max-drop-points: 2`
+  means "fail if any candidate is more than 2 points below the baseline".
+- Comparison is exact-after-normalisation by default. Pass a scorer if you need
+  semantic equivalence.
+- Models are downloaded at run time. A 1B model is ~2 GB; check runner disk.
+
+## Honest limits
+
+- Only `fp32` and dynamic `int8` are supported in v1.
+- The library runs generation, so it needs a task whose success is checkable in
+  code. It is not an LLM-judge harness.
+- **The bundled `evals/fixtures/collapse.jsonl` is model-specific.** Its cases
+  expect the answer `yes`, so it only demonstrates a collapse for a model that
+  answers `yes` when unquantized. Pointed at an untuned tiny LM it scores 0/N at
+  both precisions, produces no drop, and the gate correctly reports PASS. Use it
+  to see the output format; bring your own eval for a real check.
+- What the repo's own CI pins is the **exit-code contract** (0 holds, 1 regressed,
+  2 misconfigured), not any particular model's accuracy — see
+  `tests/test_selftest_gate.py`.
+- No published adoption yet. If you use it, an issue saying what you quantized
+  and what broke is genuinely useful.

quant_regress-0.1.0/pyproject.toml

@@ -0,0 +1,40 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "quant-regress"
+version = "0.1.0"
+description = "Fail CI when quantization costs you task accuracy. Task-level regression testing for quantized models."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [{ name = "DawnofGenX" }]
+keywords = ["quantization", "regression-testing", "evaluation", "ci", "llm"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+dependencies = [
+    "torch>=2.0",
+    "transformers>=4.40",
+    "numpy>=1.24",
+]
+
+[project.optional-dependencies]
+dev = ["pytest>=8.0"]
+
+[project.scripts]
+quant-regress = "quant_regress.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/DawnofGenX/quant-regress"
+Issues = "https://github.com/DawnofGenX/quant-regress/issues"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

quant_regress-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

quant_regress-0.1.0/src/quant_regress/__init__.py

@@ -0,0 +1,13 @@
+"""quant-regress — task-level regression testing for quantized models.
+
+Perplexity and token-level metrics do not reliably detect the damage
+quantization does at the margins. This package measures task accuracy
+before and after quantization so a regression fails a build instead of
+shipping.
+"""
+
+__version__ = "0.1.0"
+
+from .harness import QuantHarness, ComparisonResult, Verdict
+
+__all__ = ["QuantHarness", "ComparisonResult", "Verdict", "__version__"]

quant_regress-0.1.0/src/quant_regress/adapter.py

@@ -0,0 +1,70 @@
+"""Build real models at a requested precision.
+
+Only dynamic ``quantize_dynamic`` INT8 is supported. That is deliberate:
+
+* it needs no extra dependency (no autoawq / optimum / onnxruntime),
+* it emits CPU-only kernels, which is what CI runners have, and
+* it is the exact method that produced the motivating INT8 accuracy collapse.
+
+Attempting to move a quantized model to CUDA raises at forward time, so the
+device is pinned to CPU and we say so rather than letting it fail obscurely.
+"""
+
+from __future__ import annotations
+
+from typing import Callable
+
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+SUPPORTED = ("fp32", "int8")
+
+
+class AdapterError(RuntimeError):
+    """Raised with an actionable message when a model cannot be built."""
+
+
+def _quantize_int8(model):
+    """Dynamic INT8 on Linear layers; embeddings/LayerNorm stay FP32."""
+    return torch.ao.quantization.quantize_dynamic(
+        model, {torch.nn.Linear}, dtype=torch.qint8
+    )
+
+
+def build_model_factory(
+    model_name: str,
+    cache_dir: str | None = None,
+    max_new_tokens: int = 32,
+) -> Callable[[str], object]:
+    if not model_name or not model_name.strip():
+        raise AdapterError(
+            "no model specified. Pass model: to the action, e.g. "
+            "model: meta-llama/Llama-3.2-1B-Instruct"
+        )
+    name = model_name.strip()
+
+    def factory(precision: str):
+        p = precision.lower().strip()
+        if p not in SUPPORTED:
+            raise ValueError(
+                f"unsupported precision {precision!r}; supported: {SUPPORTED}"
+            )
+        kwargs = {"cache_dir": cache_dir} if cache_dir else {}
+        try:
+            tok = AutoTokenizer.from_pretrained(name, **kwargs)
+            model = AutoModelForCausalLM.from_pretrained(name, **kwargs)
+        except Exception as exc:
+            raise AdapterError(
+                f"could not load {name!r}: {type(exc).__name__}: {exc}. "
+                "Check the model id, and that the runner has disk space "
+                "(a 1B model is ~2 GB, a 7B model is ~14 GB)."
+            ) from exc
+        if p == "int8":
+            model = _quantize_int8(model)
+        # quantize_dynamic emits CPU-only kernels.
+        model = model.to("cpu").eval()
+        model.tokenizer = tok  # attached for the harness
+        model.max_new_tokens = max_new_tokens
+        return model
+
+    return factory

quant_regress-0.1.0/src/quant_regress/cli.py

@@ -0,0 +1,110 @@
+"""Command-line entry point.
+
+Exit codes are the contract with CI:
+
+* ``0`` — within tolerance
+* ``1`` — accuracy regressed past ``--max-drop-points``
+* ``2`` — bad usage / configuration (missing model, unreadable eval set)
+
+A JSON report is written when ``--report`` is given so a workflow can publish
+it as an artifact.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+from .adapter import AdapterError, build_model_factory
+from .evalset import EvalSet, EvalSetError
+from .harness import QuantHarness, Verdict
+
+EXIT_OK, EXIT_REGRESSION, EXIT_USAGE = 0, 1, 2
+
+
+def build_parser() -> argparse.ArgumentParser:
+    p = argparse.ArgumentParser(
+        prog="quant-regress",
+        description="Fail CI when quantization costs you task accuracy.",
+    )
+    p.add_argument("--eval", required=True, help="path to a JSONL task eval set")
+    p.add_argument("--model", default=None, help="HF model id (required for a real run)")
+    p.add_argument("--precisions", default="int8",
+                   help="comma-separated candidate precisions (default: int8)")
+    p.add_argument("--baseline-precision", default="fp32")
+    p.add_argument("--max-drop-points", type=float, default=2.0,
+                   help="fail if any candidate drops more than this many points")
+    p.add_argument("--max-new-tokens", type=int, default=32)
+    p.add_argument("--report", default=None, help="write a JSON report here")
+    p.add_argument("--cache-dir", default=None)
+    return p
+
+
+def _markdown(res) -> str:
+    b = res.baseline
+    lines = [
+        "| precision | accuracy | correct | drop |",
+        "|---|---|---|---|",
+        f"| {b.precision} (baseline) | {b.accuracy*100:.1f}% | {b.correct}/{b.total} | — |",
+    ]
+    for c in res.candidates:
+        drop = (b.accuracy - c.accuracy) * 100.0
+        lines.append(
+            f"| {c.precision} | {c.accuracy*100:.1f}% | {c.correct}/{c.total} | "
+            f"{drop:+.1f} pts |"
+        )
+    return "\n".join(lines)
+
+
+def main(argv: list[str] | None = None) -> int:
+    args = build_parser().parse_args(argv)
+    try:
+        eval_set = EvalSet.load(args.eval)
+    except EvalSetError as exc:
+        print(f"error: {exc}", file=sys.stderr)
+        return EXIT_USAGE
+    if not args.model:
+        print(
+            "error: --model is required (e.g. --model "
+            "meta-llama/Llama-3.2-1B-Instruct)", file=sys.stderr
+        )
+        return EXIT_USAGE
+
+    try:
+        factory = build_model_factory(
+            args.model, cache_dir=args.cache_dir,
+            max_new_tokens=args.max_new_tokens,
+        )
+    except AdapterError as exc:
+        print(f"error: {exc}", file=sys.stderr)
+        return EXIT_USAGE
+
+    harness = QuantHarness(model_factory=factory, max_new_tokens=args.max_new_tokens)
+    try:
+        res = harness.compare(
+            eval_set,
+            baseline_precision=args.baseline_precision,
+            candidate_precisions=[p.strip() for p in args.precisions.split(",") if p.strip()],
+            max_drop_points=args.max_drop_points,
+        )
+    except RuntimeError as exc:
+        print(f"error: {exc}", file=sys.stderr)
+        return EXIT_USAGE
+
+    print(_markdown(res))
+    print(f"\nworst drop: {res.worst_drop_points:+.2f} pts "
+          f"(threshold {args.max_drop_points:+.2f}) -> {res.verdict.value.upper()}")
+
+    if args.report:
+        Path(args.report).parent.mkdir(parents=True, exist_ok=True)
+        Path(args.report).write_text(
+            json.dumps(res.to_dict(), indent=2) + "\n", encoding="utf-8"
+        )
+
+    return EXIT_OK if res.verdict is Verdict.PASS else EXIT_REGRESSION
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

quant_regress-0.1.0/src/quant_regress/evalset.py

@@ -0,0 +1,69 @@
+"""Eval-set loading for quant-regress.
+
+Format: JSONL, one case per line, each ``{"id", "prompt", "expected"}``.
+Comparison is exact-after-normalisation by design: a fuzzy scorer would hide
+the marginal-decision damage this tool exists to detect.
+"""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Callable, Iterable
+
+
+class EvalSetError(ValueError):
+    """Raised when an eval set is malformed. Always names the offending row."""
+
+
+def normalise(text: str) -> str:
+    """Lowercase, strip, and collapse internal whitespace."""
+    return " ".join(str(text).split()).strip().lower()
+
+
+@dataclass(frozen=True)
+class Case:
+    id: str
+    prompt: str
+    expected: str
+
+
+class EvalSet:
+    def __init__(self, cases: Iterable[Case]):
+        self.cases = list(cases)
+
+    def __len__(self) -> int:
+        return len(self.cases)
+
+    def __iter__(self):
+        return iter(self.cases)
+
+    @classmethod
+    def load(cls, path: str | Path) -> "EvalSet":
+        p = Path(path)
+        if not p.exists():
+            raise EvalSetError(f"eval set not found: {p}")
+        cases: list[Case] = []
+        seen: set[str] = set()
+        for lineno, raw in enumerate(p.read_text(encoding="utf-8").splitlines(), 1):
+            line = raw.strip()
+            if not line:
+                continue
+            try:
+                row = json.loads(line)
+            except json.JSONDecodeError as exc:
+                raise EvalSetError(f"line {lineno}: invalid JSON ({exc})") from exc
+            missing = [k for k in ("id", "prompt", "expected") if k not in row]
+            if missing:
+                raise EvalSetError(f"line {lineno}: missing {', '.join(missing)}")
+            cid = str(row["id"])
+            if cid in seen:
+                raise EvalSetError(f"line {lineno}: duplicate id {cid!r}")
+            seen.add(cid)
+            cases.append(
+                Case(id=cid, prompt=str(row["prompt"]), expected=normalise(row["expected"]))
+            )
+        if not cases:
+            raise EvalSetError(f"eval set is empty: {p}")
+        return cls(cases)

quant_regress-0.1.0/src/quant_regress/harness.py

@@ -0,0 +1,209 @@
+"""Core harness: run a task eval at several precisions and compare accuracy.
+
+The point of this module is to make the marginal-decision damage visible.
+Token-level proxies (perplexity, KL) routinely miss it; task accuracy does
+not, provided the whole set is actually run — which the harness enforces by
+scoring every case at every precision and refusing to report a partial run.
+"""
+
+from __future__ import annotations
+
+import time
+from dataclasses import dataclass, field
+from enum import Enum
+from typing import Callable, Sequence
+
+from .evalset import EvalSet, normalise
+
+SUPPORTED_PRECISIONS = ("fp32", "int8")
+
+
+class Verdict(str, Enum):
+    PASS = "pass"
+    FAIL = "fail"
+
+
+@dataclass
+class PrecisionResult:
+    precision: str
+    correct: int
+    total: int
+    seconds: float
+    misclassified: list[str] = field(default_factory=list)
+
+    @property
+    def accuracy(self) -> float:
+        return (self.correct / self.total) if self.total else 0.0
+
+
+@dataclass
+class ComparisonResult:
+    baseline: PrecisionResult
+    candidates: list[PrecisionResult]
+    max_drop_points: float
+    verdict: Verdict
+
+    @property
+    def worst_drop_points(self) -> float:
+        drops = [
+            (self.baseline.accuracy - c.accuracy) * 100.0 for c in self.candidates
+        ]
+        return max(drops) if drops else 0.0
+
+    def to_dict(self) -> dict:
+        return {
+            "baseline": {
+                "precision": self.baseline.precision,
+                "accuracy": round(self.baseline.accuracy, 4),
+                "correct": self.baseline.correct,
+                "total": self.baseline.total,
+                "seconds": round(self.baseline.seconds, 3),
+            },
+            "candidates": [
+                {
+                    "precision": c.precision,
+                    "accuracy": round(c.accuracy, 4),
+                    "correct": c.correct,
+                    "total": c.total,
+                    "seconds": round(c.seconds, 3),
+                    "drop_points": round(
+                        (self.baseline.accuracy - c.accuracy) * 100.0, 3
+                    ),
+                }
+                for c in self.candidates
+            ],
+            "worst_drop_points": round(self.worst_drop_points, 3),
+            "max_drop_points": self.max_drop_points,
+            "verdict": self.verdict.value,
+        }
+
+
+class QuantHarness:
+    """Measures task accuracy for a model at each requested precision.
+
+    ``model_factory(precision) -> model`` is injected so tests can run with a
+    deterministic fake and CI can run with a real model.
+    """
+
+    def __init__(
+        self,
+        model_factory: Callable[[str], object],
+        max_new_tokens: int = 32,
+        scorer: Callable[[str, str], bool] | None = None,
+        system_prompt: str | None = None,
+        labels: Sequence[str] | None = None,
+    ):
+        if scorer is None:
+            def scorer(pred: str, expected: str) -> bool:
+                return normalise(pred) == expected
+        self._factory = model_factory
+        self.max_new_tokens = max_new_tokens
+        self._scorer = scorer
+        self.system_prompt = system_prompt
+        # Label set for models that expose only logits and carry no id2label.
+        self.labels = list(labels) if labels else []
+
+    # -- public -----------------------------------------------------------
+    def compare(
+        self,
+        eval_set: EvalSet,
+        baseline_precision: str = "fp32",
+        candidate_precisions: Sequence[str] = ("int8",),
+        max_drop_points: float = 2.0,
+    ) -> ComparisonResult:
+        for p in (baseline_precision, *candidate_precisions):
+            if p not in SUPPORTED_PRECISIONS:
+                raise ValueError(
+                    f"unsupported precision {p!r}; supported: {SUPPORTED_PRECISIONS}"
+                )
+        if not len(eval_set):
+            raise ValueError("eval set is empty")
+
+        baseline = self._measure(eval_set, baseline_precision)
+        candidates = [self._measure(eval_set, p) for p in candidate_precisions]
+
+        # Every precision must have scored the whole set, or the comparison
+        # is meaningless. Refuse rather than report a partial number.
+        for r in (baseline, *candidates):
+            if r.total != len(eval_set):
+                raise RuntimeError(
+                    f"incomplete run at {r.precision}: scored {r.total} of "
+                    f"{len(eval_set)} cases"
+                )
+
+        worst = max(
+            ((baseline.accuracy - c.accuracy) * 100.0 for c in candidates), default=0.0
+        )
+        return ComparisonResult(
+            baseline=baseline,
+            candidates=candidates,
+            max_drop_points=max_drop_points,
+            verdict=Verdict.FAIL if worst > max_drop_points else Verdict.PASS,
+        )
+
+    # -- internals --------------------------------------------------------
+    def _measure(self, eval_set: EvalSet, precision: str) -> PrecisionResult:
+        model = self._factory(precision)
+        tok = getattr(model, "tokenizer", None)
+        correct = 0
+        mis: list[str] = []
+        t0 = time.perf_counter()
+        for case in eval_set:
+            pred = self._predict(model, tok, case.prompt)
+            ok = self._scorer(pred, case.expected)
+            correct += int(ok)
+            if not ok:
+                mis.append(case.id)
+        return PrecisionResult(
+            precision=precision,
+            correct=correct,
+            total=len(eval_set),
+            seconds=time.perf_counter() - t0,
+            misclassified=mis,
+        )
+
+    def _predict(self, model, tok, prompt: str) -> str:
+        """Return the model's answer as text.
+
+        Three supported model shapes, in priority order:
+
+        1. A real HF model with ``generate`` + tokenizer — decode the output.
+        2. A model exposing ``.answer(prompt)`` — used by fakes and by any model
+           whose output is not free text (a classifier, a scorer).
+        3. A bare callable returning ``.logits`` — decoded via the model's own
+           ``id2label`` when it has one, else by argmax over a caller-supplied
+           label set.
+
+        Shape 3 deliberately does NOT stringify the raw logit. A logit is not an
+        answer: ``str(0.95)`` can never equal ``"yes"``, so stringifying it made
+        every such model score 0% and silently defeated the regression gate.
+        """
+        # 2. explicit text answer
+        answer = getattr(model, "answer", None)
+        if callable(answer):
+            return str(answer(prompt))
+
+        # 1. real HF generation path
+        if tok is not None and hasattr(model, "generate"):
+            text = prompt if self.system_prompt is None else f"{self.system_prompt}\n{prompt}"
+            inputs = tok(text, return_tensors="pt")
+            inputs = {k: v.to(getattr(model, "device", "cpu")) for k, v in inputs.items()}
+            out = model.generate(**inputs, max_new_tokens=self.max_new_tokens,
+                                 do_sample=False)
+            return tok.decode(out[0], skip_special_tokens=True)
+
+        # 3. logits -> label
+        logits = model(**{"input_ids": [[1]], "attention_mask": [[1]]}).logits
+        row = logits[0]
+        idx = int(max(range(len(row)), key=lambda i: float(row[i])))
+        id2label = getattr(model, "config", None)
+        id2label = getattr(id2label, "id2label", None) if id2label else None
+        if id2label:
+            return str(id2label.get(idx, idx))
+        if self.labels:
+            return str(self.labels[idx]) if idx < len(self.labels) else str(idx)
+        raise RuntimeError(
+            "cannot map model output to text: the model exposes no .answer(), "
+            "no generate()+tokenizer, and no id2label. Give the fake model an "
+            "answer(prompt) method, or pass labels=[...] to QuantHarness."
+        )

quant_regress-0.1.0/src/quant_regress.egg-info/PKG-INFO

@@ -0,0 +1,89 @@
+Metadata-Version: 2.4
+Name: quant-regress
+Version: 0.1.0
+Summary: Fail CI when quantization costs you task accuracy. Task-level regression testing for quantized models.
+Author: DawnofGenX
+License: MIT
+Project-URL: Homepage, https://github.com/DawnofGenX/quant-regress
+Project-URL: Issues, https://github.com/DawnofGenX/quant-regress/issues
+Keywords: quantization,regression-testing,evaluation,ci,llm
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: torch>=2.0
+Requires-Dist: transformers>=4.40
+Requires-Dist: numpy>=1.24
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+
+# quant-regress
+
+Fail CI when quantization costs you task accuracy.
+
+Token-level proxies — perplexity, KL divergence — routinely report that a
+quantized model is fine. Task accuracy does not, provided you actually measure
+it. This action measures a task eval at each precision and fails the build when
+accuracy drops past a threshold you set.
+
+## Why
+
+<!-- claim:int8-collapse -->
+On a cross-encoder claim verifier, dynamic INT8 quantization left aggregate
+accuracy looking survivable while destroying the capability that mattered:
+
+<!-- claim:int8-collapse -->
+| precision | accuracy | correct |
+|---|---|---|
+| fp32 (baseline) | 100.0% <!-- claim:int8-collapse --> | 20/20 |
+| int8 | 0.0% <!-- claim:int8-collapse --> | 0/20 |
+
+Regenerate with `python scripts/regen_int8_collapse.py` — every figure above
+comes from that script, and `tests/test_claims_are_reproducible.py` fails the
+build if a number in this file stops matching it.
+
+The pattern that matters: the quantized model still runs, still returns a valid
+answer, and still looks like a working model. It is simply wrong every time.
+That is why a token-level proxy can clear it and a task eval cannot.
+
+## Usage
+
+```yaml
+- uses: DawnofGenX/quant-regress@v1
+  with:
+    eval: evals/tasks.jsonl
+    model: meta-llama/Llama-3.2-1B-Instruct
+    max-drop-points: 2
+```
+
+Exit codes: `0` within tolerance, `1` accuracy regressed, `2` bad configuration.
+
+## Constraints worth knowing
+
+- **CPU only.** `quantize_dynamic` emits CPU-only quantized kernels; moving the
+  model to CUDA raises at forward time. GitHub-hosted runners have no GPU, so
+  this is not a limitation in CI — but it does mean GPU-only quantizers
+  (AWQ/GPTQ/GGUF) are out of scope for v1.
+- **The threshold unit is accuracy points**, not a ratio. `--max-drop-points: 2`
+  means "fail if any candidate is more than 2 points below the baseline".
+- Comparison is exact-after-normalisation by default. Pass a scorer if you need
+  semantic equivalence.
+- Models are downloaded at run time. A 1B model is ~2 GB; check runner disk.
+
+## Honest limits
+
+- Only `fp32` and dynamic `int8` are supported in v1.
+- The library runs generation, so it needs a task whose success is checkable in
+  code. It is not an LLM-judge harness.
+- **The bundled `evals/fixtures/collapse.jsonl` is model-specific.** Its cases
+  expect the answer `yes`, so it only demonstrates a collapse for a model that
+  answers `yes` when unquantized. Pointed at an untuned tiny LM it scores 0/N at
+  both precisions, produces no drop, and the gate correctly reports PASS. Use it
+  to see the output format; bring your own eval for a real check.
+- What the repo's own CI pins is the **exit-code contract** (0 holds, 1 regressed,
+  2 misconfigured), not any particular model's accuracy — see
+  `tests/test_selftest_gate.py`.
+- No published adoption yet. If you use it, an issue saying what you quantized
+  and what broke is genuinely useful.

quant_regress-0.1.0/src/quant_regress.egg-info/SOURCES.txt

@@ -0,0 +1,19 @@
+README.md
+pyproject.toml
+src/quant_regress/__init__.py
+src/quant_regress/adapter.py
+src/quant_regress/cli.py
+src/quant_regress/evalset.py
+src/quant_regress/harness.py
+src/quant_regress.egg-info/PKG-INFO
+src/quant_regress.egg-info/SOURCES.txt
+src/quant_regress.egg-info/dependency_links.txt
+src/quant_regress.egg-info/entry_points.txt
+src/quant_regress.egg-info/requires.txt
+src/quant_regress.egg-info/top_level.txt
+tests/test_adapter.py
+tests/test_claims_are_reproducible.py
+tests/test_cli.py
+tests/test_evalset.py
+tests/test_harness.py
+tests/test_selftest_gate.py

quant_regress-0.1.0/src/quant_regress.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

quant_regress-0.1.0/src/quant_regress.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+quant-regress = quant_regress.cli:main

quant_regress-0.1.0/src/quant_regress.egg-info/requires.txt

@@ -0,0 +1,6 @@
+torch>=2.0
+transformers>=4.40
+numpy>=1.24
+
+[dev]
+pytest>=8.0

quant_regress-0.1.0/src/quant_regress.egg-info/top_level.txt

@@ -0,0 +1 @@
+quant_regress

quant_regress-0.1.0/tests/test_adapter.py

@@ -0,0 +1,38 @@
+"""The adapter must build a genuinely different int8 model, or the tool lies."""
+from __future__ import annotations
+
+import pytest
+
+torch = pytest.importorskip("torch")
+
+from quant_regress.adapter import build_model_factory, AdapterError
+
+
+class _Tiny(torch.nn.Module):
+    def __init__(self):
+        super().__init__()
+        self.linear = torch.nn.Linear(8, 2)
+
+    def forward(self, **kw):
+        return type("O", (), {"logits": self.linear(torch.zeros(1, 8))})()
+
+
+def test_int8_model_is_actually_quantized():
+    factory = build_model_factory(model_name="hf-internal-testing/tiny-random-gpt2")
+    fp32 = factory("fp32")
+    int8 = factory("int8")
+    # In modern torch, quantize_dynamic keeps the Linear class name but packs
+    # weights into a LinearPackedParams child module.
+    assert not any("packed" in type(m).__name__.lower() for m in fp32.modules())
+    assert any("packed" in type(m).__name__.lower() for m in int8.modules())
+
+
+def test_factory_rejects_unknown_precision():
+    factory = build_model_factory(model_name="hf-internal-testing/tiny-random-gpt2")
+    with pytest.raises(ValueError, match="unsupported precision"):
+        factory("int4")
+
+
+def test_empty_model_name_is_rejected_with_actionable_message():
+    with pytest.raises(AdapterError, match="model"):
+        build_model_factory(model_name="")

quant_regress-0.1.0/tests/test_claims_are_reproducible.py

@@ -0,0 +1,55 @@
+"""Every headline number in README.md must be re-derivable by a committed script.
+
+This repo makes claims in other people's CI. A stale or fabricated number here
+propagates into other people's build failures, so the claims are treated as
+code: each one names the script that regenerates it, and this test fails if a
+claim has no regenerating script.
+"""
+from __future__ import annotations
+
+import re
+from pathlib import Path
+
+import pytest
+
+ROOT = Path(__file__).resolve().parent.parent
+README = ROOT / "README.md"
+SCRIPTS = ROOT / "scripts"
+
+CLAIM_RE = re.compile(r"<!--\s*claim:(?P<id>[a-z0-9-]+)\s*-->")
+PERCENT_RE = re.compile(r"\b\d{1,3}(?:\.\d+)?%")
+
+
+@pytest.mark.skipif(not README.exists(), reason="README not written yet")
+def test_every_percent_in_readme_is_tagged_as_a_claim():
+    """Untagged percentages are the ones that can silently rot."""
+    text = README.read_text(encoding="utf-8")
+    claimed = {m.group("id") for m in CLAIM_RE.finditer(text)}
+    for line_no, line in enumerate(text.splitlines(), 1):
+        if PERCENT_RE.search(line) and "claim:" not in line:
+            # Badge lines (shields.io) are generated, not claims.
+            if "img.shields.io" in line:
+                continue
+            pytest.fail(
+                f"README.md:{line_no} has an untagged percentage: {line.strip()!r}. "
+                "Add <!-- claim:ID --> and a script in scripts/ that regenerates it."
+            )
+    assert claimed or not PERCENT_RE.search(text)
+
+
+@pytest.mark.skipif(not README.exists(), reason="README not written yet")
+def test_every_claim_has_a_regenerating_script():
+    text = README.read_text(encoding="utf-8")
+    for m in CLAIM_RE.finditer(text):
+        script = SCRIPTS / f"regen_{m.group('id').replace('-', '_')}.py"
+        assert script.exists(), (
+            f"README claim {m.group('id')!r} has no regenerating script at "
+            f"{script.relative_to(ROOT)}"
+        )
+
+
+@pytest.mark.skipif(not README.exists(), reason="README not written yet")
+def test_readme_states_cpu_only_and_threshold_semantics():
+    text = README.read_text(encoding="utf-8").lower()
+    assert "cpu" in text, "README must state the CPU-only constraint"
+    assert "point" in text, "README must state the threshold unit"

quant_regress-0.1.0/tests/test_cli.py

@@ -0,0 +1,75 @@
+"""The CLI must exit non-zero on regression so CI actually fails."""
+from __future__ import annotations
+
+import json
+
+import pytest
+
+from quant_regress.cli import main
+
+
+class _Fake:
+    """Deterministic stand-in: answers correctly at fp32, wrongly at int8.
+
+    Mirrors the real failure mode this tool exists to catch — a quantized model
+    that still runs and still produces output, just worse output.
+    """
+
+    def __init__(self, precision: str, correct: bool = True):
+        self.precision = precision
+        self.correct = correct
+
+    def eval(self):
+        return self
+
+    def answer(self, prompt: str) -> str:
+        return "yes" if self.correct else "no"
+
+
+def _eval(tmp_path, n=6):
+    rows = [{"id": str(i), "prompt": f"q{i}?", "expected": "yes"} for i in range(n)]
+    p = tmp_path / "eval.jsonl"
+    p.write_text("\n".join(json.dumps(r) for r in rows), encoding="utf-8")
+    return str(p)
+
+
+def test_exit_1_on_regression(tmp_path, monkeypatch):
+    def factory(*a, **kw):
+        return lambda precision: _Fake(precision, correct=(precision == "fp32"))
+
+    monkeypatch.setattr("quant_regress.cli.build_model_factory", factory)
+    rc = main(["--eval", _eval(tmp_path), "--model", "fake-model",
+               "--max-drop-points", "5"])
+    assert rc == 1
+
+
+def test_exit_0_when_within_tolerance(tmp_path, monkeypatch):
+    def factory(*a, **kw):
+        return lambda precision: _Fake(precision, correct=True)
+
+    monkeypatch.setattr("quant_regress.cli.build_model_factory", factory)
+    rc = main(["--eval", _eval(tmp_path), "--model", "fake-model",
+               "--max-drop-points", "5"])
+    assert rc == 0
+
+
+def test_writes_json_report(tmp_path, monkeypatch):
+    out = tmp_path / "report.json"
+
+    def factory(*a, **kw):
+        return lambda precision: _Fake(precision, correct=(precision == "fp32"))
+
+    monkeypatch.setattr("quant_regress.cli.build_model_factory", factory)
+    main(["--eval", _eval(tmp_path), "--model", "fake-model",
+          "--max-drop-points", "5", "--report", str(out)])
+    data = json.loads(out.read_text())
+    assert "baseline" in data and "candidates" in data and "verdict" in data
+
+
+def test_missing_model_argument_exits_2(tmp_path, monkeypatch):
+    def factory(*a, **kw):
+        return lambda precision: _Fake(precision)
+
+    monkeypatch.setattr("quant_regress.cli.build_model_factory", factory)
+    rc = main(["--eval", _eval(tmp_path)])
+    assert rc == 2

quant_regress-0.1.0/tests/test_evalset.py

@@ -0,0 +1,52 @@
+"""The eval set must load deterministically and reject malformed input."""
+from __future__ import annotations
+
+import json
+
+import pytest
+
+from quant_regress.evalset import EvalSet, EvalSetError
+
+
+def _write(tmp_path, rows):
+    p = tmp_path / "eval.jsonl"
+    p.write_text("\n".join(json.dumps(r) for r in rows) + "\n", encoding="utf-8")
+    return p
+
+
+def test_loads_rows_in_file_order(tmp_path):
+    rows = [
+        {"id": "a", "prompt": "x?", "expected": "yes"},
+        {"id": "b", "prompt": "y?", "expected": "no"},
+    ]
+    es = EvalSet.load(_write(tmp_path, rows))
+    assert [c.id for c in es.cases] == ["a", "b"]
+    assert len(es) == 2
+
+
+def test_normalises_expected_for_exact_comparison(tmp_path):
+    rows = [{"id": "a", "prompt": "x?", "expected": "  YES  "}]
+    es = EvalSet.load(_write(tmp_path, rows))
+    assert es.cases[0].expected == "yes"
+
+
+def test_rejects_row_missing_prompt(tmp_path):
+    rows = [{"id": "a", "expected": "yes"}]
+    with pytest.raises(EvalSetError, match="prompt"):
+        EvalSet.load(_write(tmp_path, rows))
+
+
+def test_rejects_duplicate_ids(tmp_path):
+    rows = [
+        {"id": "a", "prompt": "x?", "expected": "yes"},
+        {"id": "a", "prompt": "y?", "expected": "no"},
+    ]
+    with pytest.raises(EvalSetError, match="duplicate"):
+        EvalSet.load(_write(tmp_path, rows))
+
+
+def test_rejects_empty_file(tmp_path):
+    p = tmp_path / "eval.jsonl"
+    p.write_text("", encoding="utf-8")
+    with pytest.raises(EvalSetError, match="empty"):
+        EvalSet.load(p)

quant_regress-0.1.0/tests/test_harness.py

@@ -0,0 +1,149 @@
+"""The harness must run a real eval at multiple precisions and report accuracy."""
+from __future__ import annotations
+
+import json
+
+import pytest
+
+from quant_regress.evalset import EvalSet
+from quant_regress.harness import QuantHarness, Verdict
+
+
+class _Logits:
+    def __init__(self, score):
+        self.logits = [[score, 1.0 - score]]
+
+
+class _FakeModel:
+    """Deterministic stand-in that answers correctly, or wrongly once quantized.
+
+    Mirrors the real failure mode this tool exists to catch — a quantized model
+    that still runs and still produces output, just worse output. Precision is
+    captured at construction time, because that is what the factory knows and
+    what the comparison is actually about.
+    """
+
+    def __init__(self, precision: str, correct: bool = True):
+        self.precision = precision
+        self.correct = correct
+        self.calls = 0
+
+    def eval(self):
+        return self
+
+    def answer(self, prompt: str) -> str:
+        self.calls += 1
+        return "yes" if self.correct else "no"
+
+
+def _eval(tmp_path, n=6, expected="yes"):
+    rows = [{"id": str(i), "prompt": f"q{i}?", "expected": expected} for i in range(n)]
+    p = tmp_path / "eval.jsonl"
+    p.write_text("\n".join(json.dumps(r) for r in rows), encoding="utf-8")
+    return EvalSet.load(p)
+
+
+def test_fp32_and_int8_are_both_measured(tmp_path):
+    es = _eval(tmp_path)
+
+    h = QuantHarness(model_factory=lambda precision: _FakeModel(precision))
+    res = h.compare(es, baseline_precision="fp32", candidate_precisions=["int8"])
+
+    assert res.baseline.precision == "fp32"
+    assert res.baseline.accuracy == 1.0
+    assert len(res.candidates) == 1
+    assert res.candidates[0].precision == "int8"
+
+
+def test_verdict_fails_when_accuracy_drops_past_threshold(tmp_path):
+    es = _eval(tmp_path, n=4)
+
+    def factory(precision):
+        return _FakeModel(precision, correct=(precision == "fp32"))
+
+    h = QuantHarness(model_factory=factory)
+    res = h.compare(es, baseline_precision="fp32", candidate_precisions=["int8"],
+                    max_drop_points=5.0)
+
+    assert res.verdict is Verdict.FAIL
+    assert res.worst_drop_points > 5.0
+
+
+def test_verdict_passes_when_drop_is_within_tolerance(tmp_path):
+    es = _eval(tmp_path, n=6)
+
+    h = QuantHarness(model_factory=lambda precision: _FakeModel(precision, correct=True))
+    res = h.compare(es, baseline_precision="fp32", candidate_precisions=["int8"],
+                    max_drop_points=5.0)
+    assert res.verdict is Verdict.PASS
+
+
+def test_every_measured_precision_runs_the_whole_set(tmp_path):
+    """Guards against a partial run silently reporting inflated accuracy."""
+    es = _eval(tmp_path, n=5)
+
+    seen = []
+
+    def factory(precision):
+        seen.append(precision)
+        return _FakeModel(precision)
+
+    h = QuantHarness(model_factory=factory, max_new_tokens=4)
+    res = h.compare(es, baseline_precision="fp32", candidate_precisions=["int8"])
+    assert seen == ["fp32", "int8"]
+    assert res.baseline.total == 5 and res.candidates[0].total == 5
+
+
+def test_logits_only_model_uses_labels_not_a_stringified_logit(tmp_path):
+    """Regression: stringifying a logit can never equal a textual expectation.
+
+    That bug made every logits-only model score 0% for the wrong reason, which
+    silently defeated the regression gate. Reading through id2label makes "no"
+    come out as "no" rather than as "0.1".
+    """
+    es = _eval(tmp_path, n=3)
+
+    class _Cfg:
+        id2label = {0: "no", 1: "yes"}
+
+    class _LogitsOnly:
+        def __init__(self, precision):
+            self.precision = precision
+            self.config = _Cfg()
+
+        def eval(self):
+            return self
+
+        def __call__(self, **kw):
+            return _Logits(0.9)  # argmax -> index 0 -> "no"
+
+    h = QuantHarness(model_factory=lambda p: _LogitsOnly(p))
+    model = _LogitsOnly("fp32")
+    # The label map is what produces the text, not the raw logit value.
+    assert h._predict(model, None, "q?") == "no"
+
+    res = h.compare(es, baseline_precision="fp32", candidate_precisions=["int8"])
+    # "no" != "yes", so accuracy is 0 — reached through the label map.
+    assert res.baseline.accuracy == 0.0
+
+
+def test_logits_only_model_without_labels_raises_actionable_error(tmp_path):
+    es = _eval(tmp_path, n=2)
+
+    class _Bare:
+        def eval(self):
+            return self
+
+        def __call__(self, **kw):
+            return _Logits(0.9)
+
+    h = QuantHarness(model_factory=lambda p: _Bare())
+    with pytest.raises(RuntimeError, match="answer\\(\\)"):
+        h.compare(es, baseline_precision="fp32", candidate_precisions=["int8"])
+
+
+def test_unsupported_precision_is_rejected(tmp_path):
+    es = _eval(tmp_path, n=2)
+    h = QuantHarness(model_factory=lambda p: _FakeModel(p))
+    with pytest.raises(ValueError, match="unsupported precision"):
+        h.compare(es, baseline_precision="fp32", candidate_precisions=["int4"])

quant_regress-0.1.0/tests/test_selftest_gate.py

@@ -0,0 +1,126 @@
+"""The gate must fail a build on a regression — and must not fail without one.
+
+An earlier version of the self-test pointed the Action at a real tiny LM with a
+fixture expecting "yes", then asserted the run FAILS. That assertion was
+unsound: an untuned tiny LM answers essentially nothing with "yes", so it scored
+0/N at BOTH precisions, produced no drop, and the gate correctly reported PASS.
+CI would have failed on a correct gate.
+
+So the fixture is model-specific, not a property of the Action. What IS a
+property of the Action is its exit-code contract, and that is what these tests
+pin — through the real ``cli.main`` entry point, with only the model factory
+replaced (no production backdoor).
+"""
+from __future__ import annotations
+
+import json
+
+import pytest
+
+from quant_regress.cli import main
+
+
+class _Stable:
+    def __init__(self, precision: str):
+        self.precision = precision
+
+    def eval(self):
+        return self
+
+    def answer(self, prompt: str) -> str:
+        return "yes"
+
+
+class _Regressing:
+    def __init__(self, precision: str):
+        self.precision = precision
+
+    def eval(self):
+        return self
+
+    def answer(self, prompt: str) -> str:
+        return "yes" if self.precision == "fp32" else "no"
+
+
+def _fixture(tmp_path, name, n=10):
+    rows = [
+        {"id": f"{name}-{i:03d}", "prompt": f"Case {i}: is the claim true?",
+         "expected": "yes"}
+        for i in range(1, n + 1)
+    ]
+    p = tmp_path / f"{name}.jsonl"
+    p.write_text("\n".join(json.dumps(r) for r in rows), encoding="utf-8")
+    return str(p)
+
+
+def _factory_for(cls):
+    return lambda *a, **kw: (lambda precision: cls(precision))
+
+
+def test_gate_fails_on_a_real_regression(tmp_path, monkeypatch, capsys):
+    monkeypatch.setattr("quant_regress.cli.build_model_factory", _factory_for(_Regressing))
+    report = tmp_path / "report.json"
+    rc = main([
+        "--eval", _fixture(tmp_path, "regress"),
+        "--model", "selftest/fake-regressing",
+        "--max-drop-points", "2",
+        "--report", str(report),
+    ])
+    assert rc == 1, "a 100-point regression must exit 1"
+    data = json.loads(report.read_text())
+    assert data["verdict"] == "fail"
+    assert data["worst_drop_points"] > 2
+    assert data["candidates"][0]["correct"] == 0
+    assert data["baseline"]["correct"] == 10
+
+
+def test_gate_passes_when_accuracy_holds(tmp_path, monkeypatch):
+    """A gate that always fails is as useless as one that never does."""
+    monkeypatch.setattr("quant_regress.cli.build_model_factory", _factory_for(_Stable))
+    rc = main([
+        "--eval", _fixture(tmp_path, "stable"),
+        "--model", "selftest/fake-stable",
+        "--max-drop-points", "2",
+    ])
+    assert rc == 0
+
+
+def test_tiny_drop_within_tolerance_still_passes(tmp_path, monkeypatch):
+    """Tolerance must actually tolerate: 0 points of drop exits 0."""
+    monkeypatch.setattr("quant_regress.cli.build_model_factory", _factory_for(_Stable))
+    rc = main([
+        "--eval", _fixture(tmp_path, "tol"),
+        "--model", "selftest/fake-stable",
+        "--max-drop-points", "0",
+    ])
+    assert rc == 0
+
+
+def test_missing_model_is_a_usage_error_not_a_pass(tmp_path, monkeypatch):
+    """The most dangerous failure mode: a misconfigured run reporting success."""
+    monkeypatch.setattr("quant_regress.cli.build_model_factory", _factory_for(_Stable))
+    rc = main(["--eval", _fixture(tmp_path, "cfg")])
+    assert rc == 2, "missing --model must exit 2, never 0"
+
+
+def test_unreadable_eval_set_is_a_usage_error(tmp_path, monkeypatch):
+    monkeypatch.setattr("quant_regress.cli.build_model_factory", _factory_for(_Stable))
+    rc = main(["--eval", str(tmp_path / "nope.jsonl"), "--model", "x"])
+    assert rc == 2
+
+
+def test_report_is_written_even_on_the_failing_path(tmp_path, monkeypatch):
+    """A regression must still leave evidence behind, or there is nothing to triage."""
+    monkeypatch.setattr("quant_regress.cli.build_model_factory", _factory_for(_Regressing))
+    report = tmp_path / "r.json"
+    main([
+        "--eval", _fixture(tmp_path, "ev"),
+        "--model", "selftest/fake-regressing",
+        "--max-drop-points", "2",
+        "--report", str(report),
+    ])
+    assert report.exists()
+    data = json.loads(report.read_text())
+    assert data["verdict"] == "fail"
+    # Per-case evidence must be present so a CI failure is actionable.
+    assert "candidates" in data and data["candidates"][0]["total"] == 10