eyewright 0.3.0__tar.gz

eyewright-0.3.0/.gitignore

@@ -0,0 +1,7 @@
+__pycache__/
+*.pyc
+.venv/
+*api-key*
+*.key
+.env
+*.log

eyewright-0.3.0/PKG-INFO

@@ -0,0 +1,7 @@
+Metadata-Version: 2.5
+Name: eyewright
+Version: 0.3.0
+Summary: eyewright - local decision core: three-eye gate + one-call typed decisions
+Requires-Python: >=3.12
+Provides-Extra: mcp
+Requires-Dist: mcp>=2; extra == 'mcp'

eyewright-0.3.0/README.md

@@ -0,0 +1,145 @@
+# eyewright
+
+**eyewright** — the decision core of the local-agent stack. A committee of local
+decision models behind one callable: used at the *decide* step of
+[Orbit](../orbit), and available to any program or AI harness.
+
+> **Name:** `eyewright` throughout — distribution, import, CLI, and MCP tools.
+> Decision + registry status: **ADR-0005** (`local-ai-system/docs/decisions/`).
+
+- **Third eye (main):** `laya-typed-decisions` — local specialist, 0.766 acc /
+  Brier 0.019 on the typed-decisions bench.
+- **Two sensory eyes (consults):** `tev1:4b` + `laya-multilingual`. In the
+  0.25–0.75 uncertain band both consult; `>= 2` confident (`>= 0.9`) and
+  agreeing → consensus; one confident + another within 0.05 of main →
+  review-conflict → the caller gathers more evidence.
+
+## Local-first, with an optional upgrade
+
+The default gate is **fully local** — Ollama (`gemma4:12b`, `tev1:4b`) plus a local
+Laya checkpoint. No prompts leave the machine; the numbers below are from the same
+600-noul gate proxy used to pick the local committee.
+
+Individual eyes can optionally be upgraded to a **cloud model with ZDR routing**:
+`typesafe/jev-1.13` (Typesafe's System One decision model) is reachable through
+OpenRouter's native decisions API and plugs in as a drop-in consult
+(`"type": "openrouter-decisions"`, `"zdr": true`, key from `OPENROUTER_API_KEY`) —
+see `examples/gate-jev.json`.
+
+Screening numbers (main = `laya-typed-decisions`; `bench/SCREENING-2026-10-09.md`;
+screening, not confirmatory — see `bench/RIGOR.md`):
+
+| gate | acted % | acted accuracy | network |
+|---|---|---|---|
+| local default: `tev1:4b + laya-multilingual` | 23.7% | 90.8% | none |
+| local System-2 add: `tev1:4b + gemma4:12b` | 34.3% | 94.7% | none |
+| cloud consult (precision): `tev1:4b + jev-1.13` | 10.0% | 100% | ZDR |
+| cloud main (automation): `jev-1.13 + tev1:4b + laya-multilingual` | 32.5% | 91.3% | ZDR |
+
+ZDR here is **request-level routing** (`provider.zdr`), not a cryptographic
+guarantee: it keeps prompts away from providers that retain data, but only
+self-hosting is independently verifiable. That trade — more automation at the cost
+of "nothing leaves the box" — is the point of making it a switch.
+
+```powershell
+$env:OPENROUTER_API_KEY = "<key>"
+uv run eyewright gate --config examples/gate-jev.json --state "..." --question "Is the task complete?"
+```
+
+## Four ways to call it
+
+**Python**
+```python
+from eyewright import ask
+
+result = ask(cfg, "a = 1, b = 2", "Which relation holds?",
+             choices={"a_bigger": "a is bigger", "b_bigger": "b is bigger"})
+# -> {"kind": "choice", "answer": "b_bigger", "confidence": 0.0038, ...}
+```
+
+**CLI**
+```powershell
+uv run eyewright ask  --state "The sky is blue." --question "Is the statement true?"
+uv run eyewright ask  --state "a = 1, b = 2" --question "Which relation holds?" --choices '{"a_bigger":"a is bigger","b_bigger":"b is bigger"}'
+uv run eyewright gate --state "..." --question "Is the task complete?"
+```
+
+**HTTP** (for any language)
+```powershell
+uv run eyewright serve --port 8090
+# GET  /health
+# POST /v1/gate  {"state": "...", "question": "..."}                  -> {p_done, decision, trace}
+# POST /v1/ask   {"state": "...", "question": "...", "choices"?|"levels"?, "engine"?: "generative"|"auto"}  -> {kind, answer, confidence, ...}
+```
+
+**MCP** (for any AI harness; needs the `mcp` extra)
+```powershell
+uv run --extra mcp eyewright mcp
+# tools: eyewright_gate(state, question), eyewright_ask(state, question, choices?, engine?)
+```
+Smoke test: `uv run --extra mcp python tests/mcp_smoke.py`.
+
+Wired into OpenCode + Cursor on this machine. Use the **venv exe directly** so a
+harness reload never triggers a `uv sync`/lock cycle:
+
+```jsonc
+"eyewright": {
+  "type": "local",
+  "command": ["C:/Users/kcpat/projects/oracle/.venv/Scripts/eyewright.exe", "mcp",
+              "--config", "C:/Users/kcpat/projects/oracle/examples/gate.json"],
+  "cwd": "C:/Users/kcpat/projects/oracle"
+}
+```
+
+Setup: `uv sync --extra mcp` once so the venv has the `mcp` extra.
+
+Config for all four faces: `examples/gate.json` (primary + consults +
+thresholds). Pass `--config <path>` elsewhere.
+
+## Engines behind `ask`
+
+| engine | what runs | notes |
+|---|---|---|
+| `typed` (default) | yes/no: the three-eye committee; choose/rate: the primary decision model | specialist domain; committee trace |
+| `generative` | a local LLM (`gemma4:12b` via the config's `generative` block) with a strict JSON contract | returns answer + reason; confidence is the model's own estimate — **not calibrated yet** (`"calibrated": false`) |
+| `auto` | typed first; generative fallback when the typed path errors or has no signal | needs a `generative` block |
+
+```powershell
+uv run eyewright ask --engine generative --state "a = 1, b = 2" --question "Is b greater than a?"
+# -> {"kind": "noul", "answer": "yes", "confidence": 1.0, "reason": "...", "calibrated": false}
+uv run eyewright ask --engine auto --state "..." --question "..."
+```
+
+Where a deterministic check exists (`a > b`), call the check — eyewright is for
+judgment, not arithmetic.
+
+Gate confidence on any von map must read the pre-band posterior (`noul_raw`);
+von is retired from the consult roster (2026-10-06). See `docs/THREE-EYE.md`.
+
+## Layout
+
+- `eyewright/gate.py` — three-eye policy (`decide_gate`) + systemone clients.
+- `eyewright/decide.py` — `ask()`: one call, any typed decision.
+- `eyewright/server.py` — `eyewright serve` HTTP service.
+- `eyewright/mcp_server.py` — `eyewright mcp` stdio MCP tools.
+- `eyewright/cli.py` — `eyewright gate|ask|serve|mcp`.
+- `bench/` — validation bench + saved responses; `docs/` — THREE-EYE,
+  GATE-ANALYSIS, RESULTS, ENSEMBLE.
+
+## Services it talks to
+
+| service | address | notes |
+|---|---|---|
+| laya-serve | `127.0.0.1:8000` | key file in `model-lab/decision-bench/`; `"model":"multilingual"` forces that checkpoint |
+| Ollama | `127.0.0.1:11434` | `tev1:4b` via `/v1/systemone` |
+| von | `127.0.0.1:8001` | `von-sdk`; retired as a consult |
+
+## Roadmap
+
+1. ✅ `eyewright ask` (package + CLI) and `eyewright serve` (HTTP).
+2. ✅ `eyewright mcp` (stdio MCP: `eyewright_gate`, `eyewright_ask`) — wired into the
+   OpenCode and Cursor configs.
+3. ✅ Generative engine behind `ask` (`--engine generative` / `auto`).
+4. Next: post-hoc calibration of the generative engine (ticket path D); wire as
+   a verifier provider into ctxd's Audit plane / control-plane's verifier
+   family (advisory, never the truth).

eyewright-0.3.0/bench/README.md

@@ -0,0 +1,34 @@
+# bench
+
+Validation bench for the Oracle's gate policy. Scripts run **from this
+directory** (they read the saved response files by name).
+
+- `three_eye_analysis.py` — consult-pair selection on the 600-noul gate proxy
+  (writes nothing; prints the THREE-EYE table).
+- `gate_analysis.py` — committee vs laya-TD-alone at matched coverage
+  (2,000-decision replay; GATE-ANALYSIS.md).
+- `ensemble.py` / `ensemble_bins.py` — main-vs-trio committee analysis.
+- `von_calibration_check.py` — von shipped vs calibrated on the gate proxy
+  (reads the calibrated run from `C:\Users\kcpat\ai-cache\von-calibrated\` when
+  present; gates on `noul_raw`).
+- `run_von_noul.py` — re-asks von only the noul questions (writes to the
+  ai-cache dir; unique temp + atomic replace).
+- `build_von_labels.py` — rebuild `von-labels-train.jsonl` from the
+  typed-decisions train parquet (`D:\datasets\typed-decisions`).
+- `score.py`, `run_laya.py`, `run_systemone.py`, `export_cases.py` — the
+  original decision-model bench harness.
+
+Data: `cases.jsonl` (400 test cases), `responses-*.jsonl` (saved per model),
+`von-labels-train.jsonl`, `von-calibration.json`.
+
+Reproduce (from `bench/`):
+
+```powershell
+uv run python three_eye_analysis.py
+uv run python gate_analysis.py
+uv run python von_calibration_check.py
+```
+
+Keys live outside the repo (secrets): `model-lab/decision-bench/.laya-api-key`,
+`.von-api-key`. The calibrated von run is written to
+`C:\Users\kcpat\ai-cache\von-calibrated\` (outside git).

eyewright-0.3.0/bench/RIGOR.md

@@ -0,0 +1,72 @@
+# Rigor notes: where the 600-noul bench stands (2026-10-09)
+
+**Verdict: this is a screening bench, not a confirmatory benchmark.** It is good
+for ranking candidates cheaply under a fixed protocol; it should not carry
+adoption claims ("X is better") without the upgrades below. Keep using it for
+fast comparisons — label results *exploratory*.
+
+## What is already right
+
+- Decision models are called through their **native decision APIs** (`/v1/systemone`),
+  not chat-prompted.
+- **Raw responses are committed** per candidate (`responses-*.jsonl`) — re-scorable.
+- Calibration discipline: refits are fitted on the **train** split, scored on test.
+- Local models are versioned by **digest** (model-lab snapshots); protocol written
+  down (THREE-EYE.md).
+- The bench targets the gate's exact shape (noul confidence), with cost/latency
+  considered in GATE-ANALYSIS.
+
+## Known weaknesses
+
+1. **Test-set reuse (selection leakage).** The 600 rows selected the consult pair,
+   the thresholds, the von verdict, and now the Jev/Gemma roster. Every round turns
+   test into dev. No frozen holdout remains.
+2. **No uncertainty reported.** Example bootstrap on the incumbent row
+   (`stats_ci.py`): acted coverage 23.7% [20.3, 27.2]; acted accuracy 90.8%
+   [85.8, 95.3]; consensus 87 firings [70, 104] at 85.1% [77.2, 92.2]. The
+   "+0.5 pt committee vs single gate" (GATE-ANALYSIS) is ~7 decisions out of
+   1,478 — noise.
+3. **Proxy labels, unaudited.** Gold is teacher-generated (3 samples); no human
+   recheck, no inter-rater agreement. The ticket's own "loop-artifact holdout"
+   is the missing external-validity piece.
+4. **Single operating point.** No coverage-risk curves, no reliability diagrams;
+   Brier/ECE reported ad hoc.
+5. **Mixed confidence kinds.** Decision models emit posteriors; generative
+   candidates (gemma, Jev-over-API, Jevify) emit **self-reported** confidence.
+   Thresholding both at 0.75/0.9 compares different objects until the generative
+   side is calibrated (tagged `"calibrated": false` in `eyewright.ask`).
+6. **Prompt / API drift.** LLM candidates are prompt-sensitive; OpenRouter models
+   change under the same id (`typesafe/jev-router` even routes to varying upstream
+   models, dynamic pricing). No paraphrase ablation, seeds, or version pinning.
+7. **Field/scope traps already hit once.**
+   - Gate on `noul_raw` (pre-band), not band `noul` (clamped to [0.80,0.85] /
+     [0.15,0.20] — cannot pass 0.9).
+   - Scope rows explicitly: the "0.413" von row is the full 2,000-decision average;
+     the gate table is noul-only (0.613).
+
+## Upgrade path (ordered by ROI)
+
+1. **Protocol freeze** — `BENCH.md`: field definitions, prompts, decode params,
+   thresholds, primary metric, decision rule; roles (screening vs confirmatory).
+2. **Role separation** — keep the 600 as screening/dev; iterate on train-derived
+   rows; reserve a frozen holdout; build the **loop-artifact holdout** (real
+   artifacts, done/not-done + defects).
+3. **Stats as standard** — bootstrap CIs; paired McNemar vs incumbent; Holm
+   correction across candidates; report n and intervals everywhere.
+4. **Label audit** — human re-adjudicate ~100 rows; report agreement; bound
+   conclusions by label noise.
+5. **Curves, not cells** — coverage-risk (accuracy vs acted %) as the headline;
+   reliability diagrams; cost/latency columns; adopt on dominance, not one cell.
+6. **Robustness** — 3 prompt paraphrases x 2 seeds for generative candidates;
+   option-order shuffle; pin API model versions + dates per run.
+
+## Candidate-specific notes (2026-10-09)
+
+- `clef-flash` is blocked twice: "non-finite logit" on Ollama 0.35.1 **and** 0.40.0,
+  and the `9b-mxfp8` tag requires an MLX runtime (unavailable on Windows). Excluded.
+- `typesafe/jev-router` (OpenRouter) is a **router**, not a fixed model: it picks
+  an upstream model per request (a ping returned `openai/gpt-6-luna`) with dynamic
+  pricing. Results measure the router product; upstream hits are tallied in the
+  screening output.
+- `jevify-gemma4-26b-a4b` is a local Gemma-4 MoE (4B active) fine-tune; via chat,
+  JSON-contract prompt, `think=false`.

eyewright-0.3.0/bench/SCREENING-2026-10-09.md

@@ -0,0 +1,52 @@
+# Screening run: Jev / Gemma / Jevify vs the Oracle on the 600-noul gate proxy (2026-10-09)
+
+Same rows and policy as the original Oracle-vs-laya comparison (THREE-EYE.md):
+main accept >= 0.75, reject <= 0.25, uncertain band consults; consensus = >= 2
+confident consults (>= 0.9) agreeing (majority for triples); conflict = one confident
++ another consult within 0.05 of main.
+
+**Screening, not confirmatory** — see `RIGOR.md`. Generative candidates' confidences
+are self-reported (uncalibrated); `jev-router` answers are routed to upstream models;
+`jev-1.13` is called through the native OpenRouter decisions API (`--zdr`).
+
+## Solo stats (600 noul decisions)
+
+| model | n | acc | >=0.9 | Brier | ECE |
+|---|---|---|---|---|---|
+| laya-typed-decisions | 600 | 0.8567 | 0.0 | 0.1436 | 0.1921 |
+| tev1:4b | 600 | 0.7883 | 0.45 | 0.1528 | 0.0495 |
+| laya-multilingual | 600 | 0.4967 | 0.598 | 0.4117 | 0.3757 |
+| von (retired) | 600 | 0.6133 | 0.0 | 0.2473 | 0.1246 |
+| jev-1.13 * | 600 | 0.7833 | 0.122 | 0.139 | 0.0805 |
+| jev-router * | 598 | 0.8194 | 0.554 | 0.1353 | 0.0703 |
+| jevify-gemma * | 598 | 0.8177 | 0.908 | 0.1687 | 0.1506 |
+| gemma4:12b * | 600 | 0.8233 | 0.858 | 0.1615 | 0.132 |
+
+\* generative candidate (or native decisions API): confidence is the model's own, not a calibrated posterior.
+
+## Gate table (same policy)
+
+| config | acted n | acted % (95% CI) | acted acc (95% CI) | consensus n/acc | conflict | not-acted\* |
+|---|---|---|---|---|---|---|
+| incumbent: laya / tev1 + laya-ml | 142 | 23.7% [20.3–27.2] | 90.8% [85.7–95.1] | 87/85.1% | 24 | 434 |
+| laya / tev1 + jev-1.13 | 60 | 10.0% [7.7–12.5] | 100.0% [100.0–100.0] | 5/100.0% | 47 | 493 |
+| laya / tev1 + jev-router | 142 | 23.7% [20.2–27.2] | 99.3% [97.7–100.0] | 87/98.9% | 4 | 454 |
+| laya / tev1 + jevify-gemma | 211 | 35.2% [31.3–39.0] | 93.8% [90.5–97.0] | 156/91.7% | 23 | 366 |
+| laya / tev1 + gemma4:12b | 206 | 34.3% [30.3–38.2] | 94.7% [91.4–97.7] | 151/92.7% | 20 | 374 |
+| TRIPLE laya / tev1 + jev-1.13 + gemma4:12b | 207 | 34.5% [30.5–38.3] | 94.7% [91.4–97.7] | 152/92.8% | 55 | 338 |
+| TRIPLE laya / tev1 + jev-router + gemma4:12b | 245 | 40.8% [36.8–44.8] | 93.5% [90.5–96.4] | 190/91.6% | 45 | 310 |
+| laya / jev-1.13 + gemma4:12b (no tev1) | 61 | 10.2% [7.8–12.7] | 100.0% [100.0–100.0] | 6/100.0% | 59 | 480 |
+| jev-1.13 main / tev1 + laya-ml | 195 | 32.5% [28.7–36.2] | 91.3% [87.1–94.9] | 61/78.7% | 15 | 390 |
+| jev-router main / tev1 + laya-ml | 364 | 60.9% [56.9–64.7] | 78.0% [73.8–82.3] | 6/50.0% | 2 | 232 |
+| jevify-gemma main / tev1 + laya-ml | 326 | 54.5% [50.5–58.4] | 80.1% [75.8–84.3] | 0/- | 0 | 272 |
+| gemma4:12b main / tev1 + laya-ml | 323 | 53.8% [49.7–57.8] | 80.8% [76.4–85.0] | 0/- | 0 | 277 |
+| reference: laya / tev1 + von | 55 | 9.2% [7.0–11.7] | 100.0% [100.0–100.0] | 0/- | 19 | 526 |
+
+\* not-acted = main-rejects (p <= 0.25) + review paths (conflicts counted separately).
+
+## jev-router upstream models (what it routed to)
+
+- deepseek/deepseek-v4.1-flash: 520 calls
+- google/gemini-3.8-flash: 100 calls
+- openai/gpt-6.1-sol: 2 calls
+

eyewright-0.3.0/bench/build_train_cases.py

@@ -0,0 +1,45 @@
+"""Build cases-train.jsonl (1,800 noul) from von-labels-train.jsonl and a
+deterministic fit/holdout split (seed 0).
+
+  cases-train.jsonl           all 1,800
+  cases-train-fit.jsonl       1,200  (calibration fitting only)
+  cases-train-holdout.jsonl     600  (frozen for evaluation; never fit on)
+"""
+
+import json
+import random
+from pathlib import Path
+
+SRC = Path("von-labels-train.jsonl")
+HOLDOUT_N = 600
+
+rows = []
+for i, line in enumerate(open(SRC, encoding="utf-8"), 1):
+    r = json.loads(line)
+    q = r["question"]
+    gold = str(r["gold"]).strip().lower()
+    label = "true" if gold in ("yes", "true", "1") else "false"
+    rows.append({
+        "id": f"train_{i:06d}",
+        "workflow": "train",
+        "state": r["state"],
+        "questions": {"q": {"type": "noul",
+                            "instructions": q.get("instructions", ""),
+                            "criteria": q.get("criteria") or {}}},
+        "gold": {"q": {"type": "noul", "label": label}},
+    })
+
+idx = list(range(len(rows)))
+random.Random(0).shuffle(idx)
+hold = set(idx[:HOLDOUT_N])
+
+
+def dump(path, items):
+    Path(path).write_text(
+        "\n".join(json.dumps(r, ensure_ascii=False) for r in items) + "\n", encoding="utf-8")
+
+
+dump("cases-train.jsonl", rows)
+dump("cases-train-fit.jsonl", [r for i, r in enumerate(rows) if i not in hold])
+dump("cases-train-holdout.jsonl", [r for i, r in enumerate(rows) if i in hold])
+print(f"train={len(rows)} fit={len(rows) - len(hold)} holdout={len(hold)}")

eyewright-0.3.0/bench/build_von_labels.py

@@ -0,0 +1,43 @@
+"""Build von calibration labels from the typed-decisions TRAIN split (noul questions only).
+
+Output: von-labels-train.jsonl, one wire question per line:
+  {"state": <state>, "instructions": <noul instructions>, "choices": {"true":..., "false":...}, "gold": "true"|"false"}
+"""
+import json
+from pathlib import Path
+
+import pandas as pd
+
+SRC = Path(r"D:\datasets\typed-decisions\all\train-00000-of-00001.parquet")
+OUT = Path(__file__).parent / "von-labels-train.jsonl"
+
+df = pd.read_parquet(SRC)
+n = 0
+with OUT.open("w", encoding="utf-8") as f:
+    for _, row in df.iterrows():
+        state = json.loads(row["state"])
+        questions = json.loads(row["questions"])
+        gold = json.loads(row["gold"])
+        for qname, q in questions.items():
+            if q.get("type") != "noul":
+                continue
+            g = gold.get(qname) or {}
+            label = g.get("label")
+            if label is None:
+                continue
+            choices = q.get("criteria") or {
+                "true": "The statement holds.",
+                "false": "The statement is false.",
+            }
+            f.write(json.dumps({
+                "state": state,
+                "question": {
+                    "type": "noul",
+                    "instructions": q.get("instructions", ""),
+                    "criteria": choices,
+                },
+                "gold": "yes" if str(label).lower() == "true" else "no",
+            }, ensure_ascii=False) + "\n")
+            n += 1
+
+print(f"wrote {n} noul calibration rows -> {OUT}")

eyewright-0.3.0/bench/calibrate_consults.py

@@ -0,0 +1,162 @@
+"""Fit a confidence calibration for a consult on the train split, apply to the
+frozen holdout + test, and report ECE/Brier before/after.
+
+Fit split: cases-train-fit.jsonl (1,200 rows). Evaluation: cases-train-holdout.jsonl
+(600, never fit on) and cases.jsonl (the original test 600). Two maps are fit and
+compared: isotonic (PAV) and a Platt-style scalar on the logit.
+
+    uv run python calibrate_consults.py --name gemma4:12b \
+        --raw-train responses-train-gemma12b.jsonl --raw-test responses-gemma12b.jsonl \
+        --out-train-cal responses-train-gemma12b-cal.jsonl \
+        --out-test-cal responses-gemma12b-cal.jsonl
+"""
+
+import argparse
+import json
+import math
+
+ap = argparse.ArgumentParser()
+ap.add_argument("--name", required=True)
+ap.add_argument("--raw-train", required=True)
+ap.add_argument("--raw-test", required=True)
+ap.add_argument("--out-train-cal", required=True)
+ap.add_argument("--out-test-cal", required=True)
+args = ap.parse_args()
+
+
+def load_gold(path):
+    g = {}
+    for line in open(path, encoding="utf-8"):
+        c = json.loads(line)
+        for q, gold in c["gold"].items():
+            if gold.get("type") == "noul":
+                g[(c["id"], q)] = 1.0 if str(gold.get("label")).lower() == "true" else 0.0
+    return g
+
+
+def load_p(path):
+    out = {}
+    for line in open(path, encoding="utf-8"):
+        r = json.loads(line)
+        if not r.get("ok"):
+            continue
+        for q, a in (r.get("resp") or {}).get("answers", {}).items():
+            p = a.get("noul_raw", a.get("noul"))
+            if p is not None:
+                out[(r["id"], q)] = float(p)
+    return out
+
+
+def fit_isotonic(points):
+    blocks = []
+    for x, y in sorted(points):
+        blocks.append([1.0, y, x])
+        while len(blocks) >= 2 and blocks[-2][1] / blocks[-2][0] > blocks[-1][1] / blocks[-1][0]:
+            w2, wy2, _ = blocks.pop()
+            w1, wy1, _ = blocks.pop()
+            blocks.append([w1 + w2, wy1 + wy2, x])
+    xs = [b[2] for b in blocks]
+    ys = [b[1] / b[0] for b in blocks]
+    return xs, ys
+
+
+def apply_isotonic(x, xs, ys):
+    if x <= xs[0]:
+        return ys[0]
+    if x >= xs[-1]:
+        return ys[-1]
+    for i in range(1, len(xs)):
+        if x <= xs[i]:
+            x0, x1, y0, y1 = xs[i - 1], xs[i], ys[i - 1], ys[i]
+            return y1 if x1 == x0 else y0 + (y1 - y0) * (x - x0) / (x1 - x0)
+    return ys[-1]
+
+
+def _logit(p, eps=1e-6):
+    p = min(1 - eps, max(eps, p))
+    return math.log(p / (1 - p))
+
+
+def fit_platt(points):
+    a, b = 1.0, 0.0
+    for _ in range(500):
+        ga = gb = 0.0
+        for x, y in points:
+            z = a * _logit(x) + b
+            q = 1 / (1 + math.exp(-z))
+            ga += (q - y) * _logit(x)
+            gb += (q - y)
+        a -= 0.1 * ga / len(points)
+        b -= 0.1 * gb / len(points)
+    return a, b
+
+
+def apply_platt(x, a, b):
+    return 1 / (1 + math.exp(-(a * _logit(x) + b)))
+
+
+def ece_brier(points, fn):
+    xs = [(fn(x), y) for x, y in points]
+    brier = sum((q - y) ** 2 for q, y in xs) / len(xs)
+    buckets = {}
+    for q, y in xs:
+        conf = q if q >= 0.5 else 1 - q
+        hit = (q >= 0.5) == (y == 1.0)
+        b = min(int(conf * 10), 9)
+        d = buckets.setdefault(b, [0.0, 0, 0])
+        d[0] += conf
+        d[1] += hit
+        d[2] += 1
+    ece = sum(v[2] / len(xs) * abs(v[0] / v[2] - v[1] / v[2]) for v in buckets.values())
+    return round(ece, 4), round(brier, 4)
+
+
+gold_train = load_gold("cases-train.jsonl")
+gold_test = load_gold("cases.jsonl")
+fit_ids = {json.loads(l)["id"] for l in open("cases-train-fit.jsonl", encoding="utf-8")}
+hold_ids = {json.loads(l)["id"] for l in open("cases-train-holdout.jsonl", encoding="utf-8")}
+raw_train = load_p(args.raw_train)
+raw_test = load_p(args.raw_test)
+
+fit_pts = [(p, gold_train[k]) for k, p in raw_train.items() if k in gold_train and k[0] in fit_ids]
+hold_pts = [(p, gold_train[k]) for k, p in raw_train.items() if k in gold_train and k[0] in hold_ids]
+test_pts = [(p, gold_test[k]) for k, p in raw_test.items() if k in gold_test]
+
+xs, ys = fit_isotonic(fit_pts)
+a, b = fit_platt(fit_pts)
+
+
+def m_iso(p):
+    return apply_isotonic(p, xs, ys)
+
+
+def m_platt(p):
+    return apply_platt(p, a, b)
+
+
+print(f"# Calibration for {args.name}")
+print(f"fit={len(fit_pts)}  holdout={len(hold_pts)}  test={len(test_pts)}   platt a={a:.3f} b={b:.3f}")
+print(f"{'split':<10} {'raw ECE/Brier':<20} {'isotonic':<20} {'platt':<20}")
+for label, pts in (("fit", fit_pts), ("holdout", hold_pts), ("test", test_pts)):
+    print(f"{label:<10} {str(ece_brier(pts, lambda x: x)):<20} "
+          f"{str(ece_brier(pts, m_iso)):<20} {str(ece_brier(pts, m_platt)):<20}")
+
+
+def write_cal(src, dst, fn):
+    with open(dst, "w", encoding="utf-8") as fo:
+        for line in open(src, encoding="utf-8"):
+            r = json.loads(line)
+            for q, a in (r.get("resp") or {}).get("answers", {}).items():
+                p = a.get("noul_raw", a.get("noul"))
+                if p is not None:
+                    c = round(fn(float(p)), 4)
+                    a["noul"] = c
+                    a["noul_raw"] = c
+                    a["calibrated_from"] = round(float(p), 4)
+                    a["calibration"] = "isotonic"
+            fo.write(json.dumps(r, ensure_ascii=False) + "\n")
+
+
+write_cal(args.raw_train, args.out_train_cal, m_iso)
+write_cal(args.raw_test, args.out_test_cal, m_iso)
+print(f"\nwrote {args.out_train_cal} and {args.out_test_cal}")