proofpath 0.2.1__tar.gz → 0.3.0__tar.gz

{proofpath-0.2.1 → proofpath-0.3.0}/CHANGELOG.md

@@ -6,6 +6,52 @@ All notable changes to this project are documented here. The format follows
 
 ## [Unreleased]
 
+## [0.3.0] - 2026-09-15
+
+The judge layer: an opt-in LLM second opinion and an opt-in model-written summary.
+The default run still makes zero LLM calls, and nothing the model says can change a
+verdict. Live run on Groq in `docs/eval/2026-09-15-judge-live.md`.
+
+### Added
+- **`check --judge`** (`judge.py`, `verify.py`). After the local verdicts are final, the
+  `low`-tier ones — never a numeric mismatch, never a claim without a quoted passage —
+  go to the model in batches of up to 20 items (about 7k tokens), each with its claim and
+  passage. The opinion (`SUPPORTED | REFUTED | NEI` plus a one-sentence rationale) is
+  attached beside the verdict: a `= judge (groq openai/gpt-oss-120b): …` line under a
+  finding it disagrees with, a `judge` column in the markdown `## Checked` table, and
+  `judge` fields in the JSON. The local `Verdict`, the finding kind and every state are
+  untouched (spec §11.1). Opinions are cached in the new `judgements` table (schema v4,
+  additive; wiped with the verdicts when a source's text changes), so a re-run asks
+  nothing, and `model_id` is untouched, so toggling `--judge` never invalidates a verdict.
+- **`check --summarize`** and the TUI's **`/summarize`**: one extra call over the finished
+  markdown report, run after the report is final, off by default in both front-ends,
+  printed as `summary    (model-written, groq openai/gpt-oss-120b) …` and as
+  `## Summary (model-written, …)` in the file. `--summarize` alone is exactly one call.
+- **`JudgeClient`**: one adapter for Groq (default `openai/gpt-oss-120b`), Gemini and
+  Ollama over the OpenAI `chat/completions` shape; strict JSON-schema output with a
+  `json_object` fallback, `reasoning_effort=low` with a fallback for providers that
+  reject it, `Retry-After` on 429 (capped, accounted), exponential backoff on 5xx, then
+  `JudgeUnavailable`. The key comes from the environment or `.env`, never from config,
+  never appears in `repr`, errors or logs; provider bodies are never echoed. Prompts are
+  packaged template files (`proofpath/prompts/review.md`, `summarize.md`).
+- **Cost on every surface**: the `Judging` and `Summarising` stage lines carry calls and
+  prompt/completion tokens; the footer counts the calls; `Report.judge_cost` and
+  `models["judge"]` land in the JSON.
+- **An unanswered judge is reported, not hidden**: `judge unavailable after N calls
+  (HTTP 401 from …); local verdicts stand` in the stage line, the report header
+  (`judge status:` / `summary status:`), the JSON and an unsuppressed terminal line — a
+  `-q` or piped run cannot look like a judged-clean one. The SARIF log does not carry it.
+  The run never fails because of the judge.
+- Gemini prints its data-use warning once per run (spec §11).
+
+### Changed
+- `check --judge` / `--summarize` no longer exit with `arrives in v0.3`.
+- Cache schema **v4** (`judgements`); a v1 file still migrates through the whole chain.
+- The markdown `## Checked` table always carries a `judge` column; without `--judge` every
+  cell is `—`, so a v0.2 report and a v0.3 one differ by that column alone.
+- `--format json` gained `summary`, `summary_model`, `judge_cost` and per-result `judge`
+  fields; every one of them is `null` on a default run.
+
 ## [0.2.1] - 2026-09-15
 
 The TUI's second look. No behaviour change: every state word, every honesty sentence,

{proofpath-0.2.1 → proofpath-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: proofpath
-Version: 0.2.1
+Version: 0.3.0
 Summary: Check whether the sources behind a claim actually say what the claim says.
 Project-URL: Homepage, https://github.com/Yigtwxx/proofpath
 Project-URL: Repository, https://github.com/Yigtwxx/proofpath
@@ -239,7 +239,12 @@ runtime teardown); fixed in v0.1 — 20 of 20 piped runs exit `1`
   counted as unreachable.
 - **Coverage is not perfectly reproducible**: two runs minutes apart can read a
   different number of sources, depending on which providers answered.
-- The LLM judge and `--summarize` arrive in **v0.3**.
+- **The judge is a second opinion, not a second verdict.** `--judge` asks the model only
+  about the low-tier verdicts (1 of 10 on the live draft), and its answer is printed
+  beside the local verdict, never in place of it. `--summarize` is one extra call over
+  the finished report, labelled model-written; if the provider does not answer, the
+  markdown report, the JSON and the terminal say so (`judge status` / `summary status`
+  in the header). The SARIF log does not carry it — it is a findings document.
 
 ## Speed
 
@@ -273,13 +278,53 @@ TablePlus or DBeaver — plain tables, no extension. Raw publisher text expires
 7 days; verdicts keep the passage they quote; a provider outage is never stored.
 `proofpath resolve REF` and `proofpath fetch URL|DOI` run either half on its own.
 
-## Optional LLM judge — arrives in v0.3
+## Optional LLM judge (v0.3)
 
-Everything above runs locally. An LLM is used only at the end, as an opt-in second
-opinion on low-confidence verdicts — it never sees a source document and cannot
-change a verdict. The settings exist today (`proofpath config check` proves a key
-works, default Groq); `check --judge` and `--summarize` are wired in v0.3. The key
-comes from the environment or a `.env` file, never from config, and is never printed.
+Everything above runs locally, and the default run makes **zero** LLM calls. Two flags
+add an LLM at the end, and only there:
+
+```bash
+proofpath check paper.pdf --judge        # a second opinion on the low-confidence verdicts
+proofpath check paper.pdf --summarize    # one model-written paragraph over the finished report
+proofpath config check                   # proves the key works before you spend a run on it
+```
+
+**What `--judge` does.** After the local verdicts are final, the verdicts the models
+were least sure about — the `low` tier, never a numeric mismatch and never a claim
+without a quoted passage — go to the model in batches of up to 20 (about 7k tokens),
+each with its claim and the passage it was checked against. The model answers from the
+passage alone, and its label and one-sentence rationale are printed **beside** the local
+verdict: `= judge (groq openai/gpt-oss-120b): NEI — …`. The local verdict, the finding
+kind and the report's states never change. Opinions are cached with the verdict, so a
+re-run asks nothing.
+
+**What it cannot do.** It never sees a source document, so it cannot introduce a claim
+or an evidence passage of its own; it cannot turn `NEI` into `SUPPORTED`; it cannot
+hide a source that could not be read. If the provider is down, rate-limited or the key
+is wrong, the run finishes on the local verdicts and says so in the stage line, the
+report header (`judge status:`) and the JSON — `-q` cannot hide it.
+
+**What `--summarize` does.** One final call turns the finished markdown report into 3–5
+plain sentences a reader can act on. It runs after the report is complete, its only
+input is that report, it is off by default in the CLI and the TUI (`/summarize` there),
+and the output is labelled `(model-written, <provider> <model>)`. `--summarize` alone
+is exactly one call; with `--judge` the escalation runs first.
+
+**Cost.** The footer counts the calls and the stage line the tokens:
+`Judging … 1 of 10 verdicts reviewed, 1 call, 613 prompt · 193 completion tokens` and
+`Summarising … 98 words, 1 call, 1,608 prompt · 343 completion tokens`
+on the live draft ([details](docs/eval/2026-09-15-judge-live.md)). Groq's free tier
+allows roughly one call a minute. Only the low-tier verdicts are sent — 1 of 10 on that
+draft — and up to 20 go in one call, so a long bibliography costs a handful of calls, not
+one per citation.
+
+**Providers.** Default is Groq `openai/gpt-oss-120b` (free without a card, no training
+on submitted data). `proofpath config set judge.provider gemini` switches to Gemini —
+note that Google trains on free-tier prompts outside the EEA/UK/CH, and proofpath prints
+that warning once per run. `judge.provider ollama` runs fully offline. Gemini and Ollama are
+fixture-tested and were not exercised live in v0.3.0. All three speak
+the OpenAI `chat/completions` shape. The key comes from `GROQ_API_KEY` / `GEMINI_API_KEY`
+in the environment or a `.env` file, never from config, and is never printed.
 
 ## Measured
 

{proofpath-0.2.1 → proofpath-0.3.0}/README.md

@@ -183,7 +183,12 @@ runtime teardown); fixed in v0.1 — 20 of 20 piped runs exit `1`
   counted as unreachable.
 - **Coverage is not perfectly reproducible**: two runs minutes apart can read a
   different number of sources, depending on which providers answered.
-- The LLM judge and `--summarize` arrive in **v0.3**.
+- **The judge is a second opinion, not a second verdict.** `--judge` asks the model only
+  about the low-tier verdicts (1 of 10 on the live draft), and its answer is printed
+  beside the local verdict, never in place of it. `--summarize` is one extra call over
+  the finished report, labelled model-written; if the provider does not answer, the
+  markdown report, the JSON and the terminal say so (`judge status` / `summary status`
+  in the header). The SARIF log does not carry it — it is a findings document.
 
 ## Speed
 
@@ -217,13 +222,53 @@ TablePlus or DBeaver — plain tables, no extension. Raw publisher text expires
 7 days; verdicts keep the passage they quote; a provider outage is never stored.
 `proofpath resolve REF` and `proofpath fetch URL|DOI` run either half on its own.
 
-## Optional LLM judge — arrives in v0.3
+## Optional LLM judge (v0.3)
 
-Everything above runs locally. An LLM is used only at the end, as an opt-in second
-opinion on low-confidence verdicts — it never sees a source document and cannot
-change a verdict. The settings exist today (`proofpath config check` proves a key
-works, default Groq); `check --judge` and `--summarize` are wired in v0.3. The key
-comes from the environment or a `.env` file, never from config, and is never printed.
+Everything above runs locally, and the default run makes **zero** LLM calls. Two flags
+add an LLM at the end, and only there:
+
+```bash
+proofpath check paper.pdf --judge        # a second opinion on the low-confidence verdicts
+proofpath check paper.pdf --summarize    # one model-written paragraph over the finished report
+proofpath config check                   # proves the key works before you spend a run on it
+```
+
+**What `--judge` does.** After the local verdicts are final, the verdicts the models
+were least sure about — the `low` tier, never a numeric mismatch and never a claim
+without a quoted passage — go to the model in batches of up to 20 (about 7k tokens),
+each with its claim and the passage it was checked against. The model answers from the
+passage alone, and its label and one-sentence rationale are printed **beside** the local
+verdict: `= judge (groq openai/gpt-oss-120b): NEI — …`. The local verdict, the finding
+kind and the report's states never change. Opinions are cached with the verdict, so a
+re-run asks nothing.
+
+**What it cannot do.** It never sees a source document, so it cannot introduce a claim
+or an evidence passage of its own; it cannot turn `NEI` into `SUPPORTED`; it cannot
+hide a source that could not be read. If the provider is down, rate-limited or the key
+is wrong, the run finishes on the local verdicts and says so in the stage line, the
+report header (`judge status:`) and the JSON — `-q` cannot hide it.
+
+**What `--summarize` does.** One final call turns the finished markdown report into 3–5
+plain sentences a reader can act on. It runs after the report is complete, its only
+input is that report, it is off by default in the CLI and the TUI (`/summarize` there),
+and the output is labelled `(model-written, <provider> <model>)`. `--summarize` alone
+is exactly one call; with `--judge` the escalation runs first.
+
+**Cost.** The footer counts the calls and the stage line the tokens:
+`Judging … 1 of 10 verdicts reviewed, 1 call, 613 prompt · 193 completion tokens` and
+`Summarising … 98 words, 1 call, 1,608 prompt · 343 completion tokens`
+on the live draft ([details](docs/eval/2026-09-15-judge-live.md)). Groq's free tier
+allows roughly one call a minute. Only the low-tier verdicts are sent — 1 of 10 on that
+draft — and up to 20 go in one call, so a long bibliography costs a handful of calls, not
+one per citation.
+
+**Providers.** Default is Groq `openai/gpt-oss-120b` (free without a card, no training
+on submitted data). `proofpath config set judge.provider gemini` switches to Gemini —
+note that Google trains on free-tier prompts outside the EEA/UK/CH, and proofpath prints
+that warning once per run. `judge.provider ollama` runs fully offline. Gemini and Ollama are
+fixture-tested and were not exercised live in v0.3.0. All three speak
+the OpenAI `chat/completions` shape. The key comes from `GROQ_API_KEY` / `GEMINI_API_KEY`
+in the environment or a `.env` file, never from config, and is never printed.
 
 ## Measured
 

proofpath-0.3.0/docs/eval/2026-09-15-judge-live.md

@@ -0,0 +1,95 @@
+# Judge layer live runs — 2026-09-15 (v0.3.0)
+
+`proofpath check tests/data/draft-live.md --judge --summarize` on Groq
+(`openai/gpt-oss-120b`, key from `.env`), Apple Silicon, models already downloaded.
+Three runs: the first against the pre-fix build (it found the defect), then a cold run
+(`--no-cache`) and a warm run against the shipped build. Stage lines are copied from
+the terminal; the key never appears in any output.
+
+## Run 1 — pre-fix build: the summary came back empty
+
+```
+Judging      groq openai/gpt-oss-120b   1 of 10 verdicts reviewed, 1 call, 661 prompt · 211 completion tokens 0.9s
+Summarising  groq openai/gpt-oss-120b   summary unavailable after 0 calls (no completion in the 200 response from https://api.groq.com/openai/v1/chat/completions (finish_reason=length)); local verdicts stand 1.0s
+```
+
+`openai/gpt-oss-120b` is a reasoning model: with the planned `max_tokens=400` the
+completion budget was spent on reasoning and the visible content was empty, and the
+written `report.md` said nothing about the summary at all (the 9.3 reviewer had found
+the same gap from the code). Both are fixed in the shipped build: requests carry
+`reasoning_effort=low` (dropped on a 400 from providers that reject it), the review
+budget is 4096 tokens and the summary budget 1500, and an unanswered summary is
+reported on the stage line, the `summary` CLI line and a `- summary status:` header
+line in the file. The 1,888-test suite was green before this run; the live run found it.
+
+## Run 2 — cold (`--no-cache`), shipped build
+
+```
+Parsing      text                              1 pages, 7 refs        0.0s
+Claims       rules                             7 citations, 0 unresolved 0.0s
+Resolving    Crossref, Semantic Scholar        6 ok, 0 amb, 1 ghost   9.3s
+Retractions  Retraction Watch                  none                   2.5s
+Fetching     arXiv, Semantic Scholar           5 full text, 1 abstract, 0 unverified 23.8s
+Verifying    coreml                            10 claims: 1 supported, 2 not supported, 7 NEI 44.5s
+Judging      groq openai/gpt-oss-120b          1 of 10 verdicts reviewed, 1 call, 613 prompt · 193 completion tokens 1.1s
+Summarising  groq openai/gpt-oss-120b          98 words, 1 call, 1,608 prompt · 343 completion tokens 1.0s
+… written  ·  2 API calls  ·  82.1s
+judge      2,221 prompt · 536 completion tokens
+```
+
+The escalation set was one verdict — the `low`-tier `NOT SUPPORTED` on the SciPy
+sentence — and the judge's opinion was printed beside it, the local verdict untouched:
+
+```
+error[not-supported]: claim is not supported by the cited source  (confidence: low)
+   = judge (groq openai/gpt-oss-120b): NEI — Passage states "SciPy provides fundamental
+     algorithms for scientific computing" but does not mention SciPy building on anything.
+```
+
+The numeric mismatch (`high`, rule-decided) and the ghost reference were not sent, as
+specified. The summary, verbatim:
+
+> The report checked seven cited references, and it found two claims not supported and
+> one ghost reference; it does not give a total number of claims that were examined. The
+> most serious problem is the high‑confidence "NOT SUPPORTED" error on line 6, where the
+> claim about training time for the Transformer contradicts the source. The coverage
+> analysis shows that only 72 % of the material was verified against full text, 14 % was
+> limited to abstracts, and the remaining 14 % could not be read, so the counts above
+> apply only to the portion that was actually examined.
+
+It repeats the coverage caveat as instructed and invents no finding. Two things it
+gets slightly wrong, which is why it is labelled model-written and sits under the
+computed report: it calls the line-6 numeric mismatch a "NOT SUPPORTED" error (the
+report says `numeric-mismatch: claim contradicts the cited source`), and "it does not
+give a total number of claims" is true of the markdown file (the stage table is not
+printed there) but not of the run (`10 claims`).
+
+## Run 3 — warm, shipped build
+
+```
+Resolving    cache                             6 ok, 0 amb, 1 ghost   0.0s
+Fetching     cache                             5 full text, 1 abstract, 0 unverified 0.0s
+Verifying    coreml                            10 claims: 1 supported, 2 not supported, 7 NEI, 10 cached 0.8s
+Judging      groq openai/gpt-oss-120b          1 of 10 verdicts reviewed, 0 calls, 0 prompt · 0 completion tokens 0.0s
+Summarising  groq openai/gpt-oss-120b          106 words, 1 call, 1,615 prompt · 263 completion tokens 0.9s
+… written  ·  1 API calls  ·  1.8s
+judge      1,615 prompt · 263 completion tokens
+```
+
+The judgement came back from the schema-v4 `judgements` table (0 calls); the summary
+is always one fresh call. A plain `check` of the same file (no flags) reports
+`api_calls: 0` and `summary: null`.
+
+## What this does and does not show
+
+- The judge layer works end to end on Groq's free tier: batching, strict-JSON output,
+  cost accounting, caching, attribution, and the report unchanged by the model.
+- One document, one low-tier verdict, one provider. Nothing here measures the judge's
+  accuracy; the escalation band is narrow by design (OPEN-ITEMS 14.1), so a paper with
+  118 citations would send a handful of verdicts, not dozens.
+- Gemini and Ollama were not exercised live; they share the adapter and are covered by
+  the `respx` tests only.
+- These runs were made with the batch cap at 7,000 prompt tokens. The whole-phase review
+  lowered it to 3,500 so that a full batch plus its 4,096-token answer budget stays under
+  Groq's 8K-per-minute tier; with one escalated item the runs above never approached
+  either figure, so nothing here measures the cap.

{proofpath-0.2.1 → proofpath-0.3.0}/docs/superpowers/OPEN-ITEMS.md

@@ -390,18 +390,15 @@ Findings from inspecting the installed base environment, and the decisions they
 | 12.6 | ~~TUI finding rows print the entry's marker twice~~ **Closed 2026-09-15 (final review fix: `strip_marker` on the label)** | `[7] [7] Marchetti, …` in the pty session: `Reference.raw` keeps its printed marker (Phase 5) and `FindingLine` prefixes the number again. The CLI diagnostic prints the entry once. Cosmetic; `strip_marker` on the label is the fix |
 | 12.7 | `loading models …` is drawn under the `Verifying` row in the TUI, above it on the CLI | the true order: `verify` emits the note inside the stage, after `StageStart`, and the TUI draws the row on `StageStart` while the CLI prints only `StageEnd`. Moving the note before `StageStart` in `verify.py` would put it outside the stage it belongs to; a `StageLine` that shows notes as its own children is the tidier fix. Cosmetic, not a `report`/`ui` one-liner |
 | 12.8 | The SARIF artifact for `check -` is `-` | `to_sarif(report, artifact=target)` takes the target as typed; a stdin run has no file, and `-` under `%SRCROOT%` is what a viewer gets. `report.document.name` (`stdin`) is no better an address; a `--artifact` override is the only honest option |
-| 12.9 | `api_calls` in the footer counts LLM calls only | a cold resolution of five references (33.5 s against Crossref and Semantic Scholar) still prints `0 API calls`; the counter is the judge's (Phase 9) and the wording says nothing about provider lookups. Rename or count — one line either way |
+| 12.9 | `api_calls` in the footer counts LLM calls only — **restated 2026-09-15 (v0.3.0)**: it is now the judge's real counter (answers, not attempts); provider lookups are still uncounted | a cold resolution of five references (33.5 s against Crossref and Semantic Scholar) still prints `0 API calls`; the counter is the judge's (Phase 9) and the wording says nothing about provider lookups. Rename or count — one line either way |
 | 12.10 | A cached resolution reprints its notes as if current | the live draft's ghost still carries `openalex unavailable (HTTP 429, retry after 65567s)` from the day the resolution was stored. Correct (nothing was re-checked) but it reads like today's outage; a `cached:` prefix on stored notes would say so |
 | 12.11 | A provider body that is valid JSON but not an object escapes `resolve` as an error | `_json` guards against a non-JSON body (`ProviderError`), but a bare list or string parses and then fails the `.get(...)` that follows with an `AttributeError`: exit 2 instead of `UNVERIFIED (provider unavailable)`. A type check in `_json` is the fix |
 | 12.12 | `/quit` waits for an in-flight mirrored `/fetch` | a `/check` run is cancelled through its scheduler; a mirrored verb runs in `asyncio.to_thread` with no cancel hook, and the loop's shutdown joins that executor, so the app exits only when the ladder returns. Bounded by the fetch timeouts, but a browser step can take a while |
-| 12.13 | Phase-9 briefs are untracked | they live under the git-ignored `.superpowers/sdd/phase9/`; the tracked plan is `docs/superpowers/plans/2026-09-12-phases-9-10-plan.md` |
+| 12.13 | ~~Phase-9 briefs are untracked~~ **Closed 2026-09-15**: Phase 9 shipped from the tracked `docs/superpowers/plans/2026-09-12-phases-9-10-plan.md`; the drift amendments live in the same git-ignored briefs and are summarised in §14 | — |
 
-### Next session
+### Next session (as of v0.2.0 — superseded by §14)
 
-1. **Phases 9–10 are deferred.** Development stops after v0.2.0 for now; when it
-   resumes, the judge layer (v0.3) and the social provider (v0.4) are planned in
-   `docs/superpowers/plans/2026-09-12-phases-9-10-plan.md` (briefs under
-   `.superpowers/sdd/phase9/`), with 12.1–12.10 above as the backlog beside them.
+1. Phases 9–10 were deferred after v0.2.0 and resumed 2026-09-15; Phase 9 is §14.
 
 ## 13. TUI v2 — v0.2.1, 2026-09-15
 
@@ -427,3 +424,45 @@ Findings from inspecting the installed base environment, and the decisions they
 | 13.3 | `PLAIN` keeps the flat v0.2.0 layout the author rated 2/10 (now pure ASCII, with `= note:` rows) | By design (v2 §2, "nothing regresses where borders cannot draw"): the flat rows, the three-line pet and the `kv` footer are v0.2.0's layout with ASCII glyphs, so a legacy-conhost, `NO_COLOR` or `-q` user sees the prototype. A `PLAIN` pass — the fixed-column stage table needs no border and would fit — is the one improvement that does not touch the ASCII rule |
 | 13.4 | The exported SVGs reference a webfont | `App.save_screenshot` (Rich's exporter) emits an `@font-face` for Fira Code with `local()` first and a `cdnjs` URL second; the files are 96 KB and 66 KB, no network is needed to read them, but an offline viewer without Fira Code falls back to its own monospace and the box drawing may not join |
 | 13.5 | 12.7 still shows in the live session | `loading models …` is drawn under the `Verifying` row in both themes (true event order); the tidier fix — a `StageLine` that owns its notes — was out of T1–T5's scope |
+
+## 14. Phase 9 — v0.3.0, 2026-09-15
+
+- Shipped: `JudgeClient` (9.1; Groq / Gemini / Ollama over `chat/completions`, packaged
+  prompts, retry/backoff, cost), `Judge` escalation with the `judgements` cache (9.2;
+  schema v4, `check --judge`), `--summarize` and the TUI `/summarize` (9.3). Briefs under
+  `.superpowers/sdd/phase9/` with their 2026-09-15 drift amendments; plan
+  `plans/2026-09-12-phases-9-10-plan.md`.
+- Live (`docs/eval/2026-09-15-judge-live.md`): on `tests/data/draft-live.md` the judge
+  escalated 1 of 10 verdicts (1 call, 613 prompt · 193 completion tokens) and answered
+  `NEI` beside a low-tier `NOT SUPPORTED`; the summary cost 1 call (1,608 · 343); the first `--summarize` attempt came back
+  empty with `finish_reason=length` — `openai/gpt-oss-120b` spends its completion budget
+  on reasoning — which is why requests now carry `reasoning_effort=low` and larger
+  budgets. The v0.1 lesson held again: the suite was green, the live run found the defect.
+- Reviewer-found before the tag: an empty 200 was counted as an answer (9.1); an
+  unavailable judge reached only the terminal `Note` (9.2); an unanswered summary was
+  invisible in the markdown file (9.3). All three were rule-2/6 failures and were fixed.
+
+### New open items
+
+| # | Item | Note |
+|---|---|---|
+| 14.1 | The `NEI`-with-a-passage escalation clause is unreachable | `pipeline.aggregate` and `decide_indexed` return `Verdict(NEI, …, passage=None)`, so the escalation set is the `low` tier only (a ~0.008-wide band, 1 of 10 verdicts on the live draft). Widening it is a pipeline decision: attach the best passage to an `NEI` and let the judge see it, or narrow the clause in the spec |
+| 14.2 | An agreeing opinion leaves no mark on a rendered finding | by design only a disagreement adds the `= judge:` line; agreement shows in the markdown `judge` column and in JSON. A markdown reader of a low-tier `NOT SUPPORTED` cannot tell "judge confirmed" from "judge never saw it" without the table |
+| 14.3 | Judgements are matched by `(claim_hash, source_id)` | two results with identical claim text against the same source share one opinion and are asked twice in one batch. Rare; carrying the result index would settle it |
+| 14.4 | `Progress` on the Judging stage counts asked items, the summary counts all results | a fully warm run emits no `Progress` and reports `1 of 10 verdicts reviewed` |
+| 14.5 | `report.py` and `cache.py` import `judge` | no cycle, but `httpx` now loads behind the output and storage layers; `JudgeOpinion`/`JudgeCost` could move to `models.py` |
+| 14.6 | `Cache.detail()` / `Cleared` do not count judgements | they are removed by cascade and on `clear`, just not reported |
+| 14.7 | A wrong or revoked key (401/403) arrives as `JudgeUnavailable` | same type as an outage; the detail names the HTTP status, so it is reportable, but a config error and a provider outage are one exception type |
+| 14.8 | ~~`--format json` carries `summary` with no inline model-written label~~ **Closed by the whole-phase review: `summary_model` sits beside it** | attribution is `models["judge"]` plus the `Summarising` stage; an inline label would change the schema |
+| 14.9 | An empty 200 is not retried while a 5xx is | right for `content_filter`, pessimistic for a flaky truncation |
+| 14.10 | Small test gaps | the bare-429 ladder (no `Retry-After`) is untested; `finish_reason` is echoed unbounded in the unavailable detail; the packaged-prompt test cannot fail on an editable install (the wheel listing is the evidence) |
+| 14.13 | The CLI's unavailable-judge line stutters | it prints `judge      judge unavailable after 0 calls (…)` because the line returns the one shared sentence verbatim while the markdown header and the sibling `summary` line strip the leading subject. Kept for v0.3.0: byte-identical wording on every surface was the point, and the same string is also printed as a `Note` where the subject is needed |
+| 14.14 | The 3,500-token batch cap rests on a rough estimator | `estimate_tokens` is `len(text) // 4 + 8` and says so; 3,500 + 4,096 leaves ~400 tokens under Groq's 8K tier, so a badly estimated batch can still meet a 429 (waited out, then reported). Calibrating the estimator against the `usage` block the provider already returns would settle it |
+| 14.11 | The unanswered judge is absent from the SARIF log | `_run_properties` carries `models` and `apiCalls` but no `judge status` / `summary status`, so a `--format sarif` log of a run whose provider was down reads like one with nothing to escalate. SARIF is a findings document, and the markdown, JSON and terminal all say it; still, a viewer-only workflow does not see it |
+| 14.12 | Groq's free tier is metered on prompt **plus** requested answer | the whole-phase review caught `TOKEN_CAP = 7000` left over from the 1,024-token answer budget: with the 4,096-token budget of 9.3 a full batch would ask for ~11.1k against an 8K/minute tier. The cap is now 3,500. Nothing measured this live — the live escalation set was one item |
+
+### Next session
+
+1. **Phase 10** (`providers/` refactor, Bluesky/HN, Reddit/Mastodon/X, AVeriTeC) runs
+   from `plans/2026-09-12-phases-9-10-plan.md`; Task 10.4a (AVeriTeC loader and scorer)
+   is already implemented and reviewed, unstaged behind the v0.3.0 tag.

{proofpath-0.2.1 → proofpath-0.3.0}/docs/superpowers/plans/2026-09-10-proofpath-implementation-plan.md

@@ -299,6 +299,19 @@ below are the original outline and stay as the scope statement.
 limit, and disabling the judge changes cost to zero without changing the report
 format. `--summarize` adds exactly one call and its absence changes nothing else.
 
+**Status (2026-09-15): released as v0.3.0.** Tasks 9.1–9.3 (`.superpowers/sdd/phase9/`,
+briefs in `2026-09-12-phases-9-10-plan.md`): `JudgeClient` with packaged prompts and cost
+accounting; `Judge` escalation (low tier with a passage; ≤ 20 items / ~7k tokens per
+call), the `judgements` cache (schema v4) and `check --judge`; `--summarize` and the TUI
+`/summarize`. Gates: 118 items pack into ≤ 6 calls at the 7k cap (unit test); a judge
+disabled run is byte-identical minus the judge fields with `api_calls == 0`;
+`--summarize` alone is exactly one call and leaves `results`/`findings` identical
+(tests). Live on Groq (`docs/eval/2026-09-15-judge-live.md`): 1 of 10 verdicts
+escalated, 1 call; the first summary attempt came back empty (`finish_reason=length`)
+and led to `reasoning_effort=low` plus larger budgets. The adapter table above (Ollama
+default, OpenRouter) was superseded by the 2026-09-11 decision: Groq default, Gemini and
+Ollama selectable.
+
 ---
 
 ## Phase 10 — Social provider (v0.4)

{proofpath-0.2.1 → proofpath-0.3.0}/docs/superpowers/specs/2026-09-10-proofpath-design.md

@@ -469,14 +469,28 @@ the judge enabled. With Ollama there is no wait at all.
 
 **Provider (decided 2026-09-11).** Default is Groq `openai/gpt-oss-120b`: free
 without a card, no training on submitted data, strict JSON schema output, but an
-8K tokens-per-minute cap that forces batches under ~7k tokens and roughly one call
-per minute. Gemini 3.8 Flash is selectable but Google trains on free-tier prompts
+8K tokens-per-minute cap that forces small batches (as built: 3,500 prompt tokens plus
+a 4,096-token answer budget) and roughly one call per minute. Gemini 3.8 Flash is selectable but Google trains on free-tier prompts
 outside the EEA/UK/CH, so choosing it prints a data-use warning. Ollama is the
 offline option. All three speak the OpenAI ``chat/completions`` shape, so one
 adapter covers them. The API key is read from an environment variable or a
 ``.env`` file, never written to config and never printed. Survey:
 ``docs/research/2026-09-11-free-llm-api-tiers.md``.
 
+**As built (v0.3.0).** The escalation set is the `low` tier plus `NEI` verdicts that
+carry a passage; a numeric mismatch (rule-decided) and any result without a passage are
+never sent (rule 1 binds the judge too). `openai/gpt-oss-120b` is a reasoning model, so
+requests carry `reasoning_effort=low` (dropped on a 400 from providers that reject it)
+and budgets of 4096 tokens per review batch and 1500 for the summary; the first live run
+with a 400-token summary budget came back empty with `finish_reason=length`. The batch
+cap counts the prompt only, so it is set to 3,500 tokens: a full batch plus its answer
+budget stays under the 8K-per-minute tier. The `NEI`-with-a-passage half of the
+escalation rule is currently unreachable — the pipeline never attaches a passage to an
+`NEI` (OPEN-ITEMS 14.1) — so in practice the judge sees the `low` tier only. A judge
+that does not answer is not a §15 state — the verdicts stand — but it is reported on the
+stage line, the `judge status:` / `summary status:` header lines, the JSON and an
+unsuppressed terminal line; `-q` cannot hide it. The SARIF log carries findings only.
+
 ## 12. Cross-platform constraints
 
 | Component | Risk | Decision |
@@ -929,8 +943,12 @@ pairing at 0.940 on its hand set (`docs/eval/2026-09-12-pairing-author-year.md`)
 beyond the plan, a resolution and retraction cache (warm re-run 1.35 s,
 `docs/eval/2026-09-15-v0.2-live.md`).
 
-**v0.3 — judge layer.** Opt-in LLM second opinion, Ollama default, OpenRouter and
-Gemini adapters, batching and cost reporting.
+**v0.3 — judge layer.** Opt-in LLM second opinion (Groq `openai/gpt-oss-120b` default,
+Gemini and Ollama selectable), batching and cost reporting, `--summarize`.
+*Shipped 2026-09-15 as v0.3.0:* `check --judge` escalates the `low`-tier verdicts with a
+passage in batches of ≤ 20 / ~7k tokens and attaches the opinion beside the verdict;
+judgements cached in schema v4; `--summarize` and the TUI `/summarize` as one extra call;
+an unanswered judge reported on every surface but SARIF. Live run: `docs/eval/2026-09-15-judge-live.md`.
 
 **v0.4 — social provider.** Bluesky and Hacker News first, Reddit via user-supplied
 OAuth app (missing credentials are reported, never silently skipped), Mastodon

{proofpath-0.2.1 → proofpath-0.3.0}/src/proofpath/__init__.py

@@ -1,5 +1,5 @@
 """proofpath — check whether the sources behind a claim support the claim."""
 
-__version__ = "0.2.1"
+__version__ = "0.3.0"
 
 __all__ = ["__version__"]

{proofpath-0.2.1 → proofpath-0.3.0}/src/proofpath/cache.py

@@ -1,7 +1,8 @@
 """The persistent cache: one plain SQLite file under the user cache dir.
 
 Holds fetched source text (with a 7-day TTL), sentence chunks with their
-embeddings, verdicts keyed ``(claim_hash, source_id, model_id)`` and the two
+embeddings, verdicts keyed ``(claim_hash, source_id, model_id)``, the optional
+judge's opinions beside them, and the two
 network lookups a run makes before it fetches anything -- reference resolution and
 the retraction check -- so a re-run of the same document costs nothing. No
 extension, no server: any SQLite GUI can open the file (spec sections 5.1, 12, 16).
@@ -21,12 +22,13 @@ from typing import Any
 
 import numpy as np
 
+from proofpath.judge import JudgeOpinion
 from proofpath.models import Label, Passage, Tier, Verdict
 from proofpath.paths import cache_dir
 from proofpath.pipeline import CUT_DECIMALS, Thresholds
 from proofpath.resolve import Candidate, FieldMatch, ResolveResult, Retraction, State, strip_marker
 
-SCHEMA_VERSION = "3"
+SCHEMA_VERSION = "4"
 RAW_TEXT_TTL_DAYS = 7
 # A resolution is a statement about a published record, which does not change; the
 # month is there so a reference an index had not yet ingested is looked at again.
@@ -130,7 +132,28 @@ _LOOKUPS_DDL: tuple[str, ...] = (
 )
 _LOOKUPS_SCHEMA = ";\n".join(_LOOKUPS_DDL) + ";\n"
 
-_SCHEMA = _BASE_SCHEMA + _CHUNKS_SCHEMA + _LOOKUPS_SCHEMA
+# The optional judge's opinions, beside the verdicts and never instead of them (spec
+# section 11.1). Keyed by the judge model as well, so a run with another judge asks
+# again rather than reading back an opinion that model never gave. ``model_id`` is
+# deliberately absent: an opinion is about the claim and the passage, not about which
+# local NLI happened to escalate it, so toggling ``--judge`` cannot invalidate a
+# cached verdict and re-tuning the thresholds cannot invalidate a cached opinion.
+_JUDGEMENTS_DDL: tuple[str, ...] = (
+    """
+    CREATE TABLE IF NOT EXISTS judgements (
+        claim_hash  TEXT NOT NULL,
+        source_id   TEXT NOT NULL REFERENCES sources(source_id) ON DELETE CASCADE,
+        judge_model TEXT NOT NULL,       -- "groq openai/gpt-oss-120b"
+        label       TEXT NOT NULL,
+        rationale   TEXT NOT NULL,
+        created_at  TEXT NOT NULL,
+        PRIMARY KEY (claim_hash, source_id, judge_model)
+    )
+    """,
+)
+_JUDGEMENTS_SCHEMA = ";\n".join(_JUDGEMENTS_DDL) + ";\n"
+
+_SCHEMA = _BASE_SCHEMA + _CHUNKS_SCHEMA + _LOOKUPS_SCHEMA + _JUDGEMENTS_SCHEMA
 
 
 def _migrate_to_v2(conn: sqlite3.Connection) -> None:
@@ -156,7 +179,19 @@ def _migrate_to_v3(conn: sqlite3.Connection) -> None:
         conn.execute(statement)
 
 
-# One step per schema version, oldest first; Phase 9 appends its own.
+def _migrate_to_v4(conn: sqlite3.Connection) -> None:
+    """v3 -> v4: the ``judgements`` table.
+
+    Additive, like v3: nothing stored before v4 holds a second opinion, and the
+    verdicts are untouched on purpose -- their ``model_id`` says nothing about the
+    judge, so a file that gains this table hands back exactly the verdicts it held
+    before and a run with ``--judge`` costs no re-verification (spec section 11.1).
+    """
+    for statement in _JUDGEMENTS_DDL:
+        conn.execute(statement)
+
+
+# One step per schema version, oldest first.
 #
 # Invariant, on which the self-healing in ``Cache.__init__`` rests: every step runs
 # inside the one transaction that also records the new version, so a step must issue
@@ -166,6 +201,7 @@ def _migrate_to_v3(conn: sqlite3.Connection) -> None:
 _MIGRATIONS: tuple[tuple[str, Callable[[sqlite3.Connection], None]], ...] = (
     ("2", _migrate_to_v2),
     ("3", _migrate_to_v3),
+    ("4", _migrate_to_v4),
 )
 
 
@@ -473,6 +509,10 @@ class Cache:
             }
             if digests - {text_sha256}:
                 self._conn.execute("DELETE FROM verdicts WHERE source_id = ?", (source_id,))
+                # The judge quoted the same passages, so its opinions go the same way:
+                # a second opinion about text the source no longer serves is evidence
+                # nobody can check (product rule 1).
+                self._conn.execute("DELETE FROM judgements WHERE source_id = ?", (source_id,))
             self._conn.execute(
                 "DELETE FROM chunks WHERE source_id = ? AND embed_model = ?",
                 (source_id, embed_model),
@@ -553,6 +593,53 @@ class Cache:
         )
         return Verdict(Label(label), float(score), _tier(tier), passage, reason=reason)
 
+    # --- judgements ---------------------------------------------------------------
+
+    def put_judgement(
+        self,
+        claim_hash_: str,
+        source_id: str,
+        judge_model: str,
+        opinion: JudgeOpinion,
+        *,
+        now: datetime | None = None,
+    ) -> None:
+        """Store one judge opinion. Re-asking the same judge replaces what it said."""
+        with self._conn:
+            self._conn.execute(
+                "INSERT INTO judgements(claim_hash, source_id, judge_model, label, rationale, "
+                "created_at) VALUES (?, ?, ?, ?, ?, ?) "
+                "ON CONFLICT(claim_hash, source_id, judge_model) DO UPDATE SET "
+                "label = excluded.label, rationale = excluded.rationale, "
+                "created_at = excluded.created_at",
+                (
+                    claim_hash_,
+                    source_id,
+                    judge_model,
+                    opinion.label.value,
+                    opinion.rationale,
+                    _iso(now or _now()),
+                ),
+            )
+
+    def get_judgement(
+        self, claim_hash_: str, source_id: str, judge_model: str
+    ) -> JudgeOpinion | None:
+        """What this judge said about this claim and source, or ``None`` if unasked.
+
+        No TTL: an opinion is about a claim and a passage, both of which are fixed.
+        The passage changing is what retires it, and ``put_chunks`` does that.
+        """
+        row = self._conn.execute(
+            "SELECT label, rationale FROM judgements "
+            "WHERE claim_hash = ? AND source_id = ? AND judge_model = ?",
+            (claim_hash_, source_id, judge_model),
+        ).fetchone()
+        if row is None:
+            return None
+        label, rationale = row
+        return JudgeOpinion(label=Label(label), rationale=str(rationale), model=judge_model)
+
     # --- resolutions and retractions ---------------------------------------------
 
     def get_resolution(self, raw: str, *, now: datetime | None = None) -> ResolveResult | None: