gonogo-eval 0.1.0__tar.gz

gonogo_eval-0.1.0/.gitignore

@@ -0,0 +1,11 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+build/
+dist/
+.pytest_cache/
+.venv/
+venv/
+.env
+examples/data/
+examples/banking77_report.html

gonogo_eval-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 James Dominguez
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

gonogo_eval-0.1.0/PKG-INFO

@@ -0,0 +1,179 @@
+Metadata-Version: 2.4
+Name: gonogo-eval
+Version: 0.1.0
+Summary: Decide whether an agent is good enough to ship. Honest statistics for pilot-scale case sets.
+Project-URL: Homepage, https://github.com/keppy/gonogo
+Project-URL: Source, https://github.com/keppy/gonogo
+Author: James Dominguez
+License: MIT
+License-File: LICENSE
+Keywords: agents,calibration,evaluation,llm,selective-prediction
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# gonogo
+
+**47 out of 50 is not 94%.** It lands somewhere between 84% and 98%, so if you were aiming at 90%, you can't yet say you got there.
+
+Feed `gonogo` your agent and your real cases. Back comes a decision: ship it, ship it behind a human-review threshold, or walk away. When the honest answer is "you don't have enough cases to know," it says that instead of guessing.
+
+This is deliberately **not** another eval framework — several good ones already exist. What none of them do is convert a score into a deployment decision you can defend at the sample sizes pilots actually run: forty to a hundred cases, not ten thousand.
+
+```bash
+pip install gonogo-eval
+```
+
+Installs as `gonogo-eval` (the plain `gonogo` name on PyPI belongs to an unrelated project); imports as `gonogo`.
+
+## The idea
+
+Evaluate an agent on a small set of real cases and three things go wrong.
+
+**The point estimate flatters you.** 47/50 reads as 94%. The 95% interval is [83.8%, 97.9%]. Most tools print the 94% and stop there.
+
+**Accuracy isn't what anyone's asking.** The real question is how much work you can hand over at 98% precision, and how many cases end up on someone's desk. Answering it means a risk–coverage curve over an abstention threshold. Almost nothing computes one.
+
+**A dashboard isn't a decision.** Somebody still has to say ship or don't ship, and that call ought to fall out of the numbers rather than out of a meeting.
+
+## Usage
+
+```python
+from gonogo import Case, evaluate
+from gonogo.scoring import fields
+
+cases = Case.from_jsonl("invoices.jsonl")     # your real cases
+
+def agent(case):
+    result = my_pipeline(case.input)
+    return result.data, result.confidence      # confidence is optional but unlocks a lot
+
+report = evaluate(agent, cases, scorer=fields(["vendor", "total"]), target=0.95)
+print(report.markdown())
+```
+
+Your agent is any callable. No base class, no decorator, no framework to adopt. Return a bare value, a `(value, confidence)` tuple, or a `Prediction` when you want to signal abstention.
+
+## What it tells you
+
+Five verdicts, and only two of them mean ship:
+
+| Verdict | Meaning |
+| --- | --- |
+| `AUTOMATE` | The pass rate's lower bound clears your target. Ship it. |
+| `AUTOMATE WITH REVIEW` | Not good enough overall, but a confident subset is. Ship behind a threshold, route the rest to a person. |
+| `ASSIST ONLY` | Useful as a draft generator, not as an unattended step. |
+| `DO NOT AUTOMATE` | Not a fit for this workflow as scoped. |
+| `INSUFFICIENT EVIDENCE` | The estimate looks good but your sample can't support the claim. Here's roughly how many cases you'd need. |
+
+That last one is the whole reason this exists. It's the verdict an honest consultant gives and a dashboard never does.
+
+## Why confidence matters
+
+If your agent reports a per-case confidence, `gonogo` finds the abstention threshold that maximizes how much you can automate while keeping precision's *lower bound* above your target, and reports the whole curve so you can see the tradeoff:
+
+| Confidence floor | Handled | Precision | To review |
+| --- | --- | --- | --- |
+| 0.36 | 100% | 88.3% [77.8%, 94.2%] | 0 |
+| 0.70 | 87% | 94.2% [84.4%, 98.0%] | 8 |
+| 0.85 | 82% | 98.0% [89.3%, 99.6%] | 11 |
+| 0.89 | 57% | 100.0% [89.8%, 100.0%] | 26 |
+| 0.99 | 2% | 100.0% [20.7%, 100.0%] | 59 |
+
+That last row is why it uses the lower bound and not the point estimate. 100% precision on a single case is not an operating point, and the interval says so.
+
+Note what the table above actually proves: against a 95% target, *no* threshold works here. The 0.85 row looks great at 98.0% until you read its lower bound of 89.3%. So the verdict is `ASSIST ONLY`, not a ship — which is the answer you want before you wire it into production, not after.
+
+It also checks whether your confidence means anything. Expected calibration error above 0.15 and the harness refuses to recommend a threshold at all — thresholding on a number that doesn't track correctness is theater.
+
+## A real measurement
+
+`examples/banking77_routing.py` runs against [Banking77](https://github.com/PolyAI-LDN/task-specific-datasets) — 13,083 genuine retail-bank customer messages across 77 intents. The agent is an ordinary TF-IDF nearest-centroid baseline, stdlib only, no model API.
+
+On a pilot-sized sample of 250 cases against a 95% target:
+
+```
+ASSIST ONLY: Use it to draft, keep a human on every case.
+Pass rate 77.2% [71.6%, 82.0%]   (full 3,080-case split: 80.7% [79.3%, 82.1%])
+Calibration error 0.42 (ranks cases, scale unreliable)
+```
+
+The calibration table is the interesting part:
+
+| Stated confidence | Cases | Mean confidence | Actual accuracy |
+| --- | --- | --- | --- |
+| 0.0–0.2 | 83 | 0.11 | 61% |
+| 0.2–0.4 | 74 | 0.30 | 78% |
+| 0.4–0.6 | 50 | 0.51 | 84% |
+| 0.6–0.8 | 30 | 0.70 | 97% |
+| 0.8–1.0 | 13 | 0.87 | 100% |
+
+That model is *badly* miscalibrated — it says 0.11 and is right 61% of the time — while still ranking cases almost perfectly. Those are two different properties, and conflating them is a common way to throw away a usable signal. `gonogo` reports the calibration error, says the number isn't a probability, and still measures precision at each cut point empirically, because that measurement doesn't depend on the scale being meaningful.
+
+## On LLM-as-judge
+
+`scoring.judge` takes any `complete(prompt) -> str` callable, so there's no provider SDK in the dependency tree.
+
+An unvalidated judge is the most common silent failure in agent evaluation. Hand-label a subset and run `validate_judge(judge_labels, human_labels)`:
+
+```
+judge NOT USABLE: 90% agreement, kappa 0.00 on 30 hand-labelled cases
+(kappa 0.00 is below 0.60 and it passes cases you failed; fix the rubric)
+```
+
+That's the trap. A judge that rubber-stamps everything scores 90% agreement on a set that's 90% passes, and carries no information whatsoever. Kappa is the gate, not agreement.
+
+## Reports
+
+`report.markdown()` for a terminal or a PR comment; `report.html()` for a self-contained styled page with no external assets, which you can hand to whoever signs off; `report.to_dict()` for JSON.
+
+## Non-goals
+
+This is a reference implementation, around 800 lines, readable in one sitting. It will not grow into:
+
+- a hosted service or dashboard
+- tracing / observability
+- prompt management or versioning
+- an agent framework
+- a public leaderboard
+
+If you want those, use one of the platforms. This does one thing.
+
+## Run the example
+
+```bash
+python examples/invoice_extraction.py
+```
+
+Sixty simulated invoices, an agent that's good but not perfect, no API key required. Real output:
+
+```
+# Score report: Extract fields from invoice
+
+**ASSIST ONLY**: Use it to draft, keep a human on every case.
+
+Pass rate 88.3% [77.8%, 94.2%] is well short of the 95% target and no
+confident subset reaches it; useful as a draft-generator, not as an
+unattended step.
+
+| Cases evaluated   | 60                   |
+| Passed            | 53                   |
+| Pass rate         | 88.3% [77.8%, 94.2%] |
+| Target            | 95%                  |
+| Calibration error | 0.11 (usable)        |
+```
+
+The calibration table in the full report is worth a look too: this agent is
+well calibrated above 0.8 (stated 0.92, actual 98%) and badly calibrated in
+the 0.6–0.8 band (stated 0.68, actual 33%). That's the kind of thing you want
+to know before you pick a threshold.
+
+## License
+
+MIT

gonogo_eval-0.1.0/README.md

@@ -0,0 +1,159 @@
+# gonogo
+
+**47 out of 50 is not 94%.** It lands somewhere between 84% and 98%, so if you were aiming at 90%, you can't yet say you got there.
+
+Feed `gonogo` your agent and your real cases. Back comes a decision: ship it, ship it behind a human-review threshold, or walk away. When the honest answer is "you don't have enough cases to know," it says that instead of guessing.
+
+This is deliberately **not** another eval framework — several good ones already exist. What none of them do is convert a score into a deployment decision you can defend at the sample sizes pilots actually run: forty to a hundred cases, not ten thousand.
+
+```bash
+pip install gonogo-eval
+```
+
+Installs as `gonogo-eval` (the plain `gonogo` name on PyPI belongs to an unrelated project); imports as `gonogo`.
+
+## The idea
+
+Evaluate an agent on a small set of real cases and three things go wrong.
+
+**The point estimate flatters you.** 47/50 reads as 94%. The 95% interval is [83.8%, 97.9%]. Most tools print the 94% and stop there.
+
+**Accuracy isn't what anyone's asking.** The real question is how much work you can hand over at 98% precision, and how many cases end up on someone's desk. Answering it means a risk–coverage curve over an abstention threshold. Almost nothing computes one.
+
+**A dashboard isn't a decision.** Somebody still has to say ship or don't ship, and that call ought to fall out of the numbers rather than out of a meeting.
+
+## Usage
+
+```python
+from gonogo import Case, evaluate
+from gonogo.scoring import fields
+
+cases = Case.from_jsonl("invoices.jsonl")     # your real cases
+
+def agent(case):
+    result = my_pipeline(case.input)
+    return result.data, result.confidence      # confidence is optional but unlocks a lot
+
+report = evaluate(agent, cases, scorer=fields(["vendor", "total"]), target=0.95)
+print(report.markdown())
+```
+
+Your agent is any callable. No base class, no decorator, no framework to adopt. Return a bare value, a `(value, confidence)` tuple, or a `Prediction` when you want to signal abstention.
+
+## What it tells you
+
+Five verdicts, and only two of them mean ship:
+
+| Verdict | Meaning |
+| --- | --- |
+| `AUTOMATE` | The pass rate's lower bound clears your target. Ship it. |
+| `AUTOMATE WITH REVIEW` | Not good enough overall, but a confident subset is. Ship behind a threshold, route the rest to a person. |
+| `ASSIST ONLY` | Useful as a draft generator, not as an unattended step. |
+| `DO NOT AUTOMATE` | Not a fit for this workflow as scoped. |
+| `INSUFFICIENT EVIDENCE` | The estimate looks good but your sample can't support the claim. Here's roughly how many cases you'd need. |
+
+That last one is the whole reason this exists. It's the verdict an honest consultant gives and a dashboard never does.
+
+## Why confidence matters
+
+If your agent reports a per-case confidence, `gonogo` finds the abstention threshold that maximizes how much you can automate while keeping precision's *lower bound* above your target, and reports the whole curve so you can see the tradeoff:
+
+| Confidence floor | Handled | Precision | To review |
+| --- | --- | --- | --- |
+| 0.36 | 100% | 88.3% [77.8%, 94.2%] | 0 |
+| 0.70 | 87% | 94.2% [84.4%, 98.0%] | 8 |
+| 0.85 | 82% | 98.0% [89.3%, 99.6%] | 11 |
+| 0.89 | 57% | 100.0% [89.8%, 100.0%] | 26 |
+| 0.99 | 2% | 100.0% [20.7%, 100.0%] | 59 |
+
+That last row is why it uses the lower bound and not the point estimate. 100% precision on a single case is not an operating point, and the interval says so.
+
+Note what the table above actually proves: against a 95% target, *no* threshold works here. The 0.85 row looks great at 98.0% until you read its lower bound of 89.3%. So the verdict is `ASSIST ONLY`, not a ship — which is the answer you want before you wire it into production, not after.
+
+It also checks whether your confidence means anything. Expected calibration error above 0.15 and the harness refuses to recommend a threshold at all — thresholding on a number that doesn't track correctness is theater.
+
+## A real measurement
+
+`examples/banking77_routing.py` runs against [Banking77](https://github.com/PolyAI-LDN/task-specific-datasets) — 13,083 genuine retail-bank customer messages across 77 intents. The agent is an ordinary TF-IDF nearest-centroid baseline, stdlib only, no model API.
+
+On a pilot-sized sample of 250 cases against a 95% target:
+
+```
+ASSIST ONLY: Use it to draft, keep a human on every case.
+Pass rate 77.2% [71.6%, 82.0%]   (full 3,080-case split: 80.7% [79.3%, 82.1%])
+Calibration error 0.42 (ranks cases, scale unreliable)
+```
+
+The calibration table is the interesting part:
+
+| Stated confidence | Cases | Mean confidence | Actual accuracy |
+| --- | --- | --- | --- |
+| 0.0–0.2 | 83 | 0.11 | 61% |
+| 0.2–0.4 | 74 | 0.30 | 78% |
+| 0.4–0.6 | 50 | 0.51 | 84% |
+| 0.6–0.8 | 30 | 0.70 | 97% |
+| 0.8–1.0 | 13 | 0.87 | 100% |
+
+That model is *badly* miscalibrated — it says 0.11 and is right 61% of the time — while still ranking cases almost perfectly. Those are two different properties, and conflating them is a common way to throw away a usable signal. `gonogo` reports the calibration error, says the number isn't a probability, and still measures precision at each cut point empirically, because that measurement doesn't depend on the scale being meaningful.
+
+## On LLM-as-judge
+
+`scoring.judge` takes any `complete(prompt) -> str` callable, so there's no provider SDK in the dependency tree.
+
+An unvalidated judge is the most common silent failure in agent evaluation. Hand-label a subset and run `validate_judge(judge_labels, human_labels)`:
+
+```
+judge NOT USABLE: 90% agreement, kappa 0.00 on 30 hand-labelled cases
+(kappa 0.00 is below 0.60 and it passes cases you failed; fix the rubric)
+```
+
+That's the trap. A judge that rubber-stamps everything scores 90% agreement on a set that's 90% passes, and carries no information whatsoever. Kappa is the gate, not agreement.
+
+## Reports
+
+`report.markdown()` for a terminal or a PR comment; `report.html()` for a self-contained styled page with no external assets, which you can hand to whoever signs off; `report.to_dict()` for JSON.
+
+## Non-goals
+
+This is a reference implementation, around 800 lines, readable in one sitting. It will not grow into:
+
+- a hosted service or dashboard
+- tracing / observability
+- prompt management or versioning
+- an agent framework
+- a public leaderboard
+
+If you want those, use one of the platforms. This does one thing.
+
+## Run the example
+
+```bash
+python examples/invoice_extraction.py
+```
+
+Sixty simulated invoices, an agent that's good but not perfect, no API key required. Real output:
+
+```
+# Score report: Extract fields from invoice
+
+**ASSIST ONLY**: Use it to draft, keep a human on every case.
+
+Pass rate 88.3% [77.8%, 94.2%] is well short of the 95% target and no
+confident subset reaches it; useful as a draft-generator, not as an
+unattended step.
+
+| Cases evaluated   | 60                   |
+| Passed            | 53                   |
+| Pass rate         | 88.3% [77.8%, 94.2%] |
+| Target            | 95%                  |
+| Calibration error | 0.11 (usable)        |
+```
+
+The calibration table in the full report is worth a look too: this agent is
+well calibrated above 0.8 (stated 0.92, actual 98%) and badly calibrated in
+the 0.6–0.8 band (stated 0.68, actual 33%). That's the kind of thing you want
+to know before you pick a threshold.
+
+## License
+
+MIT

gonogo_eval-0.1.0/examples/banking77_routing.py

@@ -0,0 +1,147 @@
+"""A real measurement on a real public dataset.
+
+Banking77 is 13,083 genuine customer-service messages from a retail bank,
+labelled into 77 intents. Routing a message to the right queue is exactly the
+kind of workflow a pilot gets scoped around, so it makes an honest stand-in for
+"can we automate our support triage?"
+
+The agent here is a deliberately ordinary baseline: TF-IDF nearest-centroid,
+pure stdlib, no model API. That is the point. A plausible-looking baseline is
+what most demos are built on, and the question is whether the harness tells you
+the truth about it.
+
+    python examples/banking77_routing.py
+
+Data is fetched once from the PolyAI repository and cached next to this file.
+"""
+
+from __future__ import annotations
+
+import csv
+import math
+import random
+import re
+import urllib.request
+from collections import Counter, defaultdict
+from pathlib import Path
+
+from gonogo import Case, Prediction, evaluate
+from gonogo.scoring import exact
+
+BASE = "https://raw.githubusercontent.com/PolyAI-LDN/task-specific-datasets/master/banking_data"
+CACHE = Path(__file__).parent / "data"
+TOKEN = re.compile(r"[a-z0-9']+")
+
+# A pilot ships with a case set a human actually labelled, not a full benchmark
+# split. 250 is a realistic size and it keeps the small-n lesson visible.
+PILOT_SIZE = 250
+SEED = 11
+
+
+def fetch(name: str) -> list[tuple[str, str]]:
+    CACHE.mkdir(exist_ok=True)
+    path = CACHE / f"banking77_{name}.csv"
+    if not path.exists():
+        print(f"downloading banking77 {name} split...")
+        urllib.request.urlretrieve(f"{BASE}/{name}.csv", path)
+    with path.open(encoding="utf-8", newline="") as fh:
+        return [(row["text"], row["category"]) for row in csv.DictReader(fh)]
+
+
+def tokenize(text: str) -> list[str]:
+    return TOKEN.findall(text.lower())
+
+
+class NearestCentroid:
+    """TF-IDF nearest-centroid classifier with a softmax confidence.
+
+    Confidence is the softmax over centroid similarities. It is a real signal --
+    ambiguous messages do score lower -- but it was never calibrated against
+    anything, which is exactly the situation most agents are in.
+    """
+
+    def __init__(self, temperature: float = 12.0):
+        self.temperature = temperature
+        self.idf: dict[str, float] = {}
+        self.centroids: dict[str, dict[str, float]] = {}
+
+    def fit(self, rows: list[tuple[str, str]]) -> "NearestCentroid":
+        docs = [tokenize(t) for t, _ in rows]
+        df = Counter()
+        for tokens in docs:
+            df.update(set(tokens))
+        n = len(docs)
+        self.idf = {term: math.log(n / (1 + count)) + 1.0 for term, count in df.items()}
+
+        sums: dict[str, dict[str, float]] = defaultdict(lambda: defaultdict(float))
+        counts: Counter = Counter()
+        for tokens, (_, label) in zip(docs, rows):
+            for term, weight in self._vector(tokens).items():
+                sums[label][term] += weight
+            counts[label] += 1
+
+        for label, vec in sums.items():
+            centroid = {t: w / counts[label] for t, w in vec.items()}
+            self.centroids[label] = self._normalize(centroid)
+        return self
+
+    def _vector(self, tokens: list[str]) -> dict[str, float]:
+        tf = Counter(tokens)
+        vec = {t: (1 + math.log(c)) * self.idf.get(t, 1.0) for t, c in tf.items()}
+        return self._normalize(vec)
+
+    @staticmethod
+    def _normalize(vec: dict[str, float]) -> dict[str, float]:
+        norm = math.sqrt(sum(w * w for w in vec.values()))
+        return {t: w / norm for t, w in vec.items()} if norm else vec
+
+    def predict(self, text: str) -> tuple[str, float]:
+        vec = self._vector(tokenize(text))
+        scores = {
+            label: sum(w * centroid.get(t, 0.0) for t, w in vec.items())
+            for label, centroid in self.centroids.items()
+        }
+        if not scores:
+            return "", 0.0
+        best = max(scores, key=scores.get)
+        top = max(scores.values())
+        total = sum(math.exp(self.temperature * (s - top)) for s in scores.values())
+        return best, 1.0 / total
+
+
+def main() -> None:
+    train = fetch("train")
+    test = fetch("test")
+    model = NearestCentroid().fit(train)
+
+    pilot = random.Random(SEED).sample(test, PILOT_SIZE)
+    cases = [
+        Case(input=text, expected=label, id=f"msg-{i:04d}")
+        for i, (text, label) in enumerate(pilot)
+    ]
+
+    def agent(case: Case) -> Prediction:
+        label, confidence = model.predict(case.input)
+        return Prediction(output=label, confidence=round(confidence, 3))
+
+    report = evaluate(
+        agent,
+        cases,
+        scorer=exact(),
+        task="Route customer message to the right queue",
+        target=0.95,
+    )
+    print(report.markdown())
+
+    out = Path(__file__).parent / "banking77_report.html"
+    out.write_text(report.html(), encoding="utf-8")
+    print(f"\n[html report written to {out}]")
+
+    # For context: the same agent measured on the full held-out split.
+    full = [Case(input=t, expected=l, id=str(i)) for i, (t, l) in enumerate(test)]
+    full_report = evaluate(agent, full, scorer=exact(), task="full test split", target=0.95)
+    print(f"[full split, n={full_report.n}: {full_report.decision.pass_rate}]")
+
+
+if __name__ == "__main__":
+    main()

gonogo_eval-0.1.0/examples/invoice_extraction.py

@@ -0,0 +1,81 @@
+"""A worked example you can run with no API key.
+
+Simulates a document-extraction pilot: 60 real-ish invoices, an agent that is
+good but not perfect and reports a confidence, and the resulting decision.
+
+    python examples/invoice_extraction.py
+
+The point of the example is the *shape of the answer*. The agent here gets about
+88% of cases right, which looks shippable until you see the interval -- and then
+you see that abstaining on its low-confidence cases buys you a defensible
+operating point instead.
+"""
+
+from __future__ import annotations
+
+import random
+
+from gonogo import Case, Prediction, evaluate
+from gonogo.scoring import fields
+
+VENDORS = ["Acme Supply", "Northwind Freight", "Globex Paper", "Initech Legal", "Umbrella Labs"]
+
+
+def build_cases(n: int = 60, seed: int = 7) -> list[Case]:
+    """Stand-in for a real case set. In a pilot this is a JSONL file of your invoices."""
+    rng = random.Random(seed)
+    cases = []
+    for i in range(n):
+        vendor = rng.choice(VENDORS)
+        total = round(rng.uniform(40, 9000), 2)
+        # Some invoices are genuinely hard: handwritten, multi-page, or foreign currency.
+        hard = rng.random() < 0.25
+        cases.append(Case(
+            id=f"inv-{i:03d}",
+            input={"scan": f"invoice_{i:03d}.pdf", "pages": 3 if hard else 1},
+            expected={"vendor": vendor, "total": total, "currency": "USD"},
+            metadata={"hard": hard},
+        ))
+    return cases
+
+
+def agent(case: Case) -> Prediction:
+    """A plausible extraction agent: usually right, shakier on hard documents.
+
+    Crucially it reports a calibrated-ish confidence, which is what lets the
+    harness find an abstention threshold. An agent that always says 0.99 gives
+    you nothing to threshold on.
+    """
+    rng = random.Random(hash(case.id) & 0xFFFF)
+    hard = case.metadata.get("hard", False)
+    correct = rng.random() > (0.35 if hard else 0.04)
+
+    expected = case.expected
+    if correct:
+        output = dict(expected)
+    else:
+        # Realistic failure modes: wrong total, or a truncated vendor name.
+        output = dict(expected)
+        if rng.random() < 0.6:
+            output["total"] = round(expected["total"] * rng.uniform(1.05, 1.4), 2)
+        else:
+            output["vendor"] = expected["vendor"].split()[0]
+
+    confidence = rng.uniform(0.35, 0.75) if hard else rng.uniform(0.85, 0.99)
+    return Prediction(output=output, confidence=round(confidence, 2))
+
+
+def main() -> None:
+    cases = build_cases()
+    report = evaluate(
+        agent,
+        cases,
+        scorer=fields(required=["vendor", "total", "currency"]),
+        task="Extract fields from invoice",
+        target=0.95,
+    )
+    print(report.markdown())
+
+
+if __name__ == "__main__":
+    main()

gonogo_eval-0.1.0/gonogo/__init__.py

@@ -0,0 +1,44 @@
+"""gonogo -- decide whether an agent is good enough to ship.
+
+Not an eval framework. A decision harness for pilot-scale case sets, where the
+honest answer is often "not enough evidence yet" and sometimes "don't automate
+this at all".
+"""
+
+from .cases import Case, CaseResult, Prediction
+from .decide import Decision, Verdict, decide
+from .evaluate import evaluate
+from .report import Report
+from .scoring import (
+    JudgeValidation,
+    exact,
+    fields,
+    judge,
+    judge_agreement,
+    numeric,
+    set_f1,
+    validate_judge,
+)
+from .stats import (
+    Interval,
+    OperatingPoint,
+    best_operating_point,
+    expected_calibration_error,
+    reliability_table,
+    required_n,
+    risk_coverage,
+    wilson,
+)
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "Case", "CaseResult", "Prediction",
+    "Decision", "Verdict", "decide",
+    "evaluate", "Report",
+    "exact", "fields", "judge", "judge_agreement", "numeric", "set_f1",
+    "JudgeValidation", "validate_judge",
+    "Interval", "OperatingPoint", "best_operating_point",
+    "expected_calibration_error", "reliability_table", "required_n",
+    "risk_coverage", "wilson",
+]