proofpath 0.2.1__tar.gz → 0.4.0__tar.gz

proofpath-0.4.0/.env.example

@@ -0,0 +1,9 @@
+# Copy to .env (gitignored) and fill in. Only the optional LLM judge needs a key;
+# everything else runs offline. See `proofpath judge` for where this file is read.
+GROQ_API_KEY=
+# GEMINI_API_KEY=
+# A free Reddit app (https://www.reddit.com/prefs/apps, type "script"): its id.
+REDDIT_CLIENT_ID=
+# The same app's secret. Without both, a cited Reddit post is reported as
+# UNVERIFIED (credentials missing) rather than silently skipped.
+REDDIT_CLIENT_SECRET=

{proofpath-0.2.1 → proofpath-0.4.0}/CHANGELOG.md

@@ -6,6 +6,99 @@ All notable changes to this project are documented here. The format follows
 
 ## [Unreleased]
 
+## [0.4.0] - 2026-09-16
+
+Sources that are not papers: a post's links, and the coverage block that finally says why
+a source could not be read. Also the first end-to-end measurement on real web claims, which
+says plainly that this is not yet the thing to point at a news story
+(`docs/eval/2026-09-16-averitec.md`).
+
+### Added
+- **`check --url`** reads a post and verifies **the links inside it**, never the post's own
+  words (spec §6.2). Bluesky (`public.api.bsky.app`) and Hacker News (the Firebase API) are
+  first-class and need no account; Reddit reads with a free app you register yourself
+  (`REDDIT_CLIENT_ID` / `REDDIT_CLIENT_SECRET` in `.env`); Mastodon is best effort per
+  instance; X cannot be read and says so, asking you to paste the text. Pasted text with
+  links in it is verified the same way.
+- **`UNVERIFIED (credentials missing)`** (spec §15): a platform that reads only with a
+  credential this machine does not have is its own state, never "unreachable". No request is
+  made, both variable names are printed, and the secret itself never reaches a log, an error
+  or a `repr`.
+- **The coverage block now prints a line per reason** on the terminal and in the markdown
+  report, not only in SARIF. `blocked: 3`, `credentials missing: 1`, `unreachable: 2` — a
+  source with no recorded reason is listed rather than dropped, so the lines account for
+  every unverified source. Product rule 6 on its main surface. The TUI footer does not carry
+  them yet.
+- **A `providers/` package** (`academic`, `web`, `social`) behind the `EvidenceProvider`
+  protocol of spec §5.2, so a new source family no longer means touching the core. The
+  academic and web paths came through byte-identical, pinned by a golden report.
+- **`docs/eval/2026-09-16-averitec.md`**: 100 AVeriTeC dev claims through the whole product.
+
+### Fixed
+- A bare DOI or arXiv **URL** is resolved as a record again instead of being fetched as a
+  web page, so it keeps its retraction check and its open-access full text.
+- A `doi:` source is read through the open-access chain and retraction-checked by the
+  academic provider whoever resolved the entry — a warm cache could previously route both
+  to the web ladder, where a missing retraction notice was reported as "no notice".
+- The `Resolving` stage no longer names Crossref and Semantic Scholar on a run that asked
+  neither.
+- A document that cites by linking is never paired as if it printed a numbered bibliography,
+  and vice versa: a PDF with a bibliography and no detected marker could previously pair a
+  body sentence to a reference it never cited.
+- `check --url` consults the network permission before reading anything; a denied run makes
+  no request at all.
+
+### Measured
+- AVeriTeC dev, 100 claims: **0.270 3-way accuracy against a 0.708 majority baseline**
+  (4-way 0.240). A third of the claims had no readable source; on the rest the score is
+  0.361. Every `Supported` claim was missed. Nothing was tuned after the measurement.
+
+## [0.3.0] - 2026-09-15
+
+The judge layer: an opt-in LLM second opinion and an opt-in model-written summary.
+The default run still makes zero LLM calls, and nothing the model says can change a
+verdict. Live run on Groq in `docs/eval/2026-09-15-judge-live.md`.
+
+### Added
+- **`check --judge`** (`judge.py`, `verify.py`). After the local verdicts are final, the
+  `low`-tier ones — never a numeric mismatch, never a claim without a quoted passage —
+  go to the model in batches of up to 20 items (about 7k tokens), each with its claim and
+  passage. The opinion (`SUPPORTED | REFUTED | NEI` plus a one-sentence rationale) is
+  attached beside the verdict: a `= judge (groq openai/gpt-oss-120b): …` line under a
+  finding it disagrees with, a `judge` column in the markdown `## Checked` table, and
+  `judge` fields in the JSON. The local `Verdict`, the finding kind and every state are
+  untouched (spec §11.1). Opinions are cached in the new `judgements` table (schema v4,
+  additive; wiped with the verdicts when a source's text changes), so a re-run asks
+  nothing, and `model_id` is untouched, so toggling `--judge` never invalidates a verdict.
+- **`check --summarize`** and the TUI's **`/summarize`**: one extra call over the finished
+  markdown report, run after the report is final, off by default in both front-ends,
+  printed as `summary    (model-written, groq openai/gpt-oss-120b) …` and as
+  `## Summary (model-written, …)` in the file. `--summarize` alone is exactly one call.
+- **`JudgeClient`**: one adapter for Groq (default `openai/gpt-oss-120b`), Gemini and
+  Ollama over the OpenAI `chat/completions` shape; strict JSON-schema output with a
+  `json_object` fallback, `reasoning_effort=low` with a fallback for providers that
+  reject it, `Retry-After` on 429 (capped, accounted), exponential backoff on 5xx, then
+  `JudgeUnavailable`. The key comes from the environment or `.env`, never from config,
+  never appears in `repr`, errors or logs; provider bodies are never echoed. Prompts are
+  packaged template files (`proofpath/prompts/review.md`, `summarize.md`).
+- **Cost on every surface**: the `Judging` and `Summarising` stage lines carry calls and
+  prompt/completion tokens; the footer counts the calls; `Report.judge_cost` and
+  `models["judge"]` land in the JSON.
+- **An unanswered judge is reported, not hidden**: `judge unavailable after N calls
+  (HTTP 401 from …); local verdicts stand` in the stage line, the report header
+  (`judge status:` / `summary status:`), the JSON and an unsuppressed terminal line — a
+  `-q` or piped run cannot look like a judged-clean one. The SARIF log does not carry it.
+  The run never fails because of the judge.
+- Gemini prints its data-use warning once per run (spec §11).
+
+### Changed
+- `check --judge` / `--summarize` no longer exit with `arrives in v0.3`.
+- Cache schema **v4** (`judgements`); a v1 file still migrates through the whole chain.
+- The markdown `## Checked` table always carries a `judge` column; without `--judge` every
+  cell is `—`, so a v0.2 report and a v0.3 one differ by that column alone.
+- `--format json` gained `summary`, `summary_model`, `judge_cost` and per-result `judge`
+  fields; every one of them is `null` on a default run.
+
 ## [0.2.1] - 2026-09-15
 
 The TUI's second look. No behaviour change: every state word, every honesty sentence,

{proofpath-0.2.1 → proofpath-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: proofpath
-Version: 0.2.1
+Version: 0.4.0
 Summary: Check whether the sources behind a claim actually say what the claim says.
 Project-URL: Homepage, https://github.com/Yigtwxx/proofpath
 Project-URL: Repository, https://github.com/Yigtwxx/proofpath
@@ -82,7 +82,7 @@ proofpath
   ╸┤ o o                                                                     ╰~~~~~~~~~~~~~[PROOF]
    ╰─┬─┬────────────────────────────────────────────────────────────────┬─┬──╯
      ˘ ˘                                                                ˘ ˘
-      proofpath v0.2.1                                                  academic . online . coreml
+      proofpath v0.4.0                                                  academic . online . coreml
       paste a file path, a URL, or a claim.                                  /help  /config  /quit
 ```
 
@@ -115,7 +115,7 @@ either, for screenshots and bug reports.
 ```
    ,_,
   (o.o)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~[PROOF]
-   " "    proofpath v0.2.1                                              academic . online . coreml
+   " "    proofpath v0.4.0                                              academic . online . coreml
           paste a file path, a URL, or a claim.            /help  /config  /quit
 ```
 
@@ -202,6 +202,7 @@ distinct, printed state (spec §15), never collapsed into a verdict:
 | `UNVERIFIED (reached, no text extracted)` | 200 answered, nothing readable came back |
 | `UNVERIFIED (network not permitted)` | `permissions.network = deny` |
 | `UNVERIFIED (provider unavailable)` | API down, rate limited after backoff, or answering with a page instead of a record |
+| `UNVERIFIED (credentials missing)` | the platform reads only with a credential this machine has none of — Reddit's free app, `REDDIT_CLIENT_ID` and `REDDIT_CLIENT_SECRET`. Nobody was asked, so it is not "unreachable"; the coverage block names both variables |
 | `UNVERIFIED (not in bibliographic indexes)` | web page, blog, report — indexes do not cover it, so absence proves nothing |
 | `AMBIGUOUS` | several plausible records, all listed |
 | `NEI` | the source was read and neither supports nor contradicts |
@@ -239,7 +240,12 @@ runtime teardown); fixed in v0.1 — 20 of 20 piped runs exit `1`
   counted as unreachable.
 - **Coverage is not perfectly reproducible**: two runs minutes apart can read a
   different number of sources, depending on which providers answered.
-- The LLM judge and `--summarize` arrive in **v0.3**.
+- **The judge is a second opinion, not a second verdict.** `--judge` asks the model only
+  about the low-tier verdicts (1 of 10 on the live draft), and its answer is printed
+  beside the local verdict, never in place of it. `--summarize` is one extra call over
+  the finished report, labelled model-written; if the provider does not answer, the
+  markdown report, the JSON and the terminal say so (`judge status` / `summary status`
+  in the header). The SARIF log does not carry it — it is a findings document.
 
 ## Speed
 
@@ -273,13 +279,79 @@ TablePlus or DBeaver — plain tables, no extension. Raw publisher text expires
 7 days; verdicts keep the passage they quote; a provider outage is never stored.
 `proofpath resolve REF` and `proofpath fetch URL|DOI` run either half on its own.
 
-## Optional LLM judge — arrives in v0.3
+## Posts and the links inside them (v0.4)
 
-Everything above runs locally. An LLM is used only at the end, as an opt-in second
-opinion on low-confidence verdicts — it never sees a source document and cannot
-change a verdict. The settings exist today (`proofpath config check` proves a key
-works, default Groq); `check --judge` and `--summarize` are wired in v0.3. The key
-comes from the environment or a `.env` file, never from config, and is never printed.
+A social post is not a source. What proofpath checks is whether the **links inside it**
+back what it says:
+
+```bash
+proofpath check --url https://bsky.app/profile/bsky.app/post/3movpwtbjgs2d
+proofpath check --url https://news.ycombinator.com/item?id=8863
+proofpath check -                 # paste the text of a post that cannot be read
+```
+
+Every sentence of the post is checked against the pages its links point to, and the post's
+own words are never allowed to stand as their own evidence.
+
+| Platform | How it is read |
+|---|---|
+| Bluesky | `public.api.bsky.app`, no account, first-class. Link cards, rich-text links and one level of quoted post |
+| Hacker News | the official Firebase API, no account, first-class. A story's URL and the links in a comment |
+| Reddit | with a **free app you register yourself**: put `REDDIT_CLIENT_ID` and `REDDIT_CLIENT_SECRET` in `.env`. Without them the run says `UNVERIFIED (credentials missing)` and names both variables — it never quietly skips the post |
+| Mastodon | best effort, per instance. Many instances now require a login for the public API, and that answer is reported as `UNVERIFIED (blocked)`, not as a missing post |
+| X / Twitter | cannot be read at all. `check --url` says so and asks you to paste the text; the links inside it are then verified normally |
+
+The Reddit path is built against Reddit's documented shapes and covered by fixtures, but it
+has never run against Reddit on this machine — nobody here has an app to register. Bluesky,
+Hacker News and Mastodon were each read live before release.
+
+## Optional LLM judge (v0.3)
+
+Everything above runs locally, and the default run makes **zero** LLM calls. Two flags
+add an LLM at the end, and only there:
+
+```bash
+proofpath check paper.pdf --judge        # a second opinion on the low-confidence verdicts
+proofpath check paper.pdf --summarize    # one model-written paragraph over the finished report
+proofpath config check                   # proves the key works before you spend a run on it
+```
+
+**What `--judge` does.** After the local verdicts are final, the verdicts the models
+were least sure about — the `low` tier, never a numeric mismatch and never a claim
+without a quoted passage — go to the model in batches of up to 20 (about 7k tokens),
+each with its claim and the passage it was checked against. The model answers from the
+passage alone, and its label and one-sentence rationale are printed **beside** the local
+verdict: `= judge (groq openai/gpt-oss-120b): NEI — …`. The local verdict, the finding
+kind and the report's states never change. Opinions are cached with the verdict, so a
+re-run asks nothing.
+
+**What it cannot do.** It never sees a source document, so it cannot introduce a claim
+or an evidence passage of its own; it cannot turn `NEI` into `SUPPORTED`; it cannot
+hide a source that could not be read. If the provider is down, rate-limited or the key
+is wrong, the run finishes on the local verdicts and says so in the stage line, the
+report header (`judge status:`) and the JSON — `-q` cannot hide it.
+
+**What `--summarize` does.** One final call turns the finished markdown report into 3–5
+plain sentences a reader can act on. It runs after the report is complete, its only
+input is that report, it is off by default in the CLI and the TUI (`/summarize` there),
+and the output is labelled `(model-written, <provider> <model>)`. `--summarize` alone
+is exactly one call; with `--judge` the escalation runs first.
+
+**Cost.** The footer counts the calls and the stage line the tokens:
+`Judging … 1 of 10 verdicts reviewed, 1 call, 613 prompt · 193 completion tokens` and
+`Summarising … 98 words, 1 call, 1,608 prompt · 343 completion tokens`
+on the live draft ([details](docs/eval/2026-09-15-judge-live.md)). Groq's free tier
+allows roughly one call a minute. Only the low-tier verdicts are sent — 1 of 10 on that
+draft — and up to 20 go in one call, so a long bibliography costs a handful of calls, not
+one per citation.
+
+**Providers.** Default is Groq `openai/gpt-oss-120b` (free without a card, no training
+on submitted data). `proofpath config set judge.provider gemini` switches to Gemini —
+note that Google trains on free-tier prompts outside the EEA/UK/CH, and proofpath prints
+that warning once per run. `judge.provider ollama` runs fully offline. Gemini and Ollama are
+fixture-tested and were not exercised live in v0.3.0. All three speak
+the OpenAI `chat/completions` shape. The key comes from `GROQ_API_KEY` / `GEMINI_API_KEY`
+in the environment or a `.env` file, never from config, and is never printed.
 
 ## Measured
 
@@ -290,6 +362,21 @@ comes from the environment or a `.env` file, never from config, and is never pri
 | Source access | 50 DOIs | 72 % full text, 18 % abstract only, 10 % nothing ([details](docs/eval/2026-09-11-coverage.md)) — a real biomedical paper in the live runs reached 33 % full text |
 | Citation pairing, numeric | 61 hand-built passages | 0.99 ([details](docs/eval/2026-09-11-pairing.md)) |
 | Citation pairing, author-year | 55 hand-built passages, 83 expectations | 0.940 ([details](docs/eval/2026-09-12-pairing-author-year.md)) |
+| **End to end on real web claims** | AVeriTeC dev, 100 claims | **0.270 3-way accuracy against a 0.708 majority baseline — worse than always guessing "refuted"** ([details](docs/eval/2026-09-16-averitec.md)) |
+
+**The AVeriTeC row is the one to read before trusting this tool on a news claim.** A third
+of those claims had no readable source at all: 32 of the source URLs needed the browser
+step, 29 were unreachable, 14 were refused by `robots.txt`. On the claims that *did* have a
+readable source the score is 0.361 — still below the baseline, and every one of the 19
+`Supported` claims was missed. The retrieval and entailment models were calibrated on
+scientific abstracts, and a fact-check page is a different object: long, discursive, and
+usually quoting the claim it debunks. Nothing was tuned after that measurement, and no
+blocked URL was dropped from it.
+
+What proofpath is good at is the academic path the other rows measure: finding out whether a
+cited paper exists, whether it was retracted, and whether its text says what the sentence
+citing it claims. Pointed at a news claim on the open web, it is currently a coverage
+report with a weak verdict attached.
 
 Published SciFact results sit around 70–75 F1, not 95. Nothing is tuned on a test
 split, and no number is quoted without the run that produced it.

{proofpath-0.2.1 → proofpath-0.4.0}/README.md

@@ -26,7 +26,7 @@ proofpath
   ╸┤ o o                                                                     ╰~~~~~~~~~~~~~[PROOF]
    ╰─┬─┬────────────────────────────────────────────────────────────────┬─┬──╯
      ˘ ˘                                                                ˘ ˘
-      proofpath v0.2.1                                                  academic . online . coreml
+      proofpath v0.4.0                                                  academic . online . coreml
       paste a file path, a URL, or a claim.                                  /help  /config  /quit
 ```
 
@@ -59,7 +59,7 @@ either, for screenshots and bug reports.
 ```
    ,_,
   (o.o)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~[PROOF]
-   " "    proofpath v0.2.1                                              academic . online . coreml
+   " "    proofpath v0.4.0                                              academic . online . coreml
           paste a file path, a URL, or a claim.            /help  /config  /quit
 ```
 
@@ -146,6 +146,7 @@ distinct, printed state (spec §15), never collapsed into a verdict:
 | `UNVERIFIED (reached, no text extracted)` | 200 answered, nothing readable came back |
 | `UNVERIFIED (network not permitted)` | `permissions.network = deny` |
 | `UNVERIFIED (provider unavailable)` | API down, rate limited after backoff, or answering with a page instead of a record |
+| `UNVERIFIED (credentials missing)` | the platform reads only with a credential this machine has none of — Reddit's free app, `REDDIT_CLIENT_ID` and `REDDIT_CLIENT_SECRET`. Nobody was asked, so it is not "unreachable"; the coverage block names both variables |
 | `UNVERIFIED (not in bibliographic indexes)` | web page, blog, report — indexes do not cover it, so absence proves nothing |
 | `AMBIGUOUS` | several plausible records, all listed |
 | `NEI` | the source was read and neither supports nor contradicts |
@@ -183,7 +184,12 @@ runtime teardown); fixed in v0.1 — 20 of 20 piped runs exit `1`
   counted as unreachable.
 - **Coverage is not perfectly reproducible**: two runs minutes apart can read a
   different number of sources, depending on which providers answered.
-- The LLM judge and `--summarize` arrive in **v0.3**.
+- **The judge is a second opinion, not a second verdict.** `--judge` asks the model only
+  about the low-tier verdicts (1 of 10 on the live draft), and its answer is printed
+  beside the local verdict, never in place of it. `--summarize` is one extra call over
+  the finished report, labelled model-written; if the provider does not answer, the
+  markdown report, the JSON and the terminal say so (`judge status` / `summary status`
+  in the header). The SARIF log does not carry it — it is a findings document.
 
 ## Speed
 
@@ -217,13 +223,79 @@ TablePlus or DBeaver — plain tables, no extension. Raw publisher text expires
 7 days; verdicts keep the passage they quote; a provider outage is never stored.
 `proofpath resolve REF` and `proofpath fetch URL|DOI` run either half on its own.
 
-## Optional LLM judge — arrives in v0.3
+## Posts and the links inside them (v0.4)
 
-Everything above runs locally. An LLM is used only at the end, as an opt-in second
-opinion on low-confidence verdicts — it never sees a source document and cannot
-change a verdict. The settings exist today (`proofpath config check` proves a key
-works, default Groq); `check --judge` and `--summarize` are wired in v0.3. The key
-comes from the environment or a `.env` file, never from config, and is never printed.
+A social post is not a source. What proofpath checks is whether the **links inside it**
+back what it says:
+
+```bash
+proofpath check --url https://bsky.app/profile/bsky.app/post/3movpwtbjgs2d
+proofpath check --url https://news.ycombinator.com/item?id=8863
+proofpath check -                 # paste the text of a post that cannot be read
+```
+
+Every sentence of the post is checked against the pages its links point to, and the post's
+own words are never allowed to stand as their own evidence.
+
+| Platform | How it is read |
+|---|---|
+| Bluesky | `public.api.bsky.app`, no account, first-class. Link cards, rich-text links and one level of quoted post |
+| Hacker News | the official Firebase API, no account, first-class. A story's URL and the links in a comment |
+| Reddit | with a **free app you register yourself**: put `REDDIT_CLIENT_ID` and `REDDIT_CLIENT_SECRET` in `.env`. Without them the run says `UNVERIFIED (credentials missing)` and names both variables — it never quietly skips the post |
+| Mastodon | best effort, per instance. Many instances now require a login for the public API, and that answer is reported as `UNVERIFIED (blocked)`, not as a missing post |
+| X / Twitter | cannot be read at all. `check --url` says so and asks you to paste the text; the links inside it are then verified normally |
+
+The Reddit path is built against Reddit's documented shapes and covered by fixtures, but it
+has never run against Reddit on this machine — nobody here has an app to register. Bluesky,
+Hacker News and Mastodon were each read live before release.
+
+## Optional LLM judge (v0.3)
+
+Everything above runs locally, and the default run makes **zero** LLM calls. Two flags
+add an LLM at the end, and only there:
+
+```bash
+proofpath check paper.pdf --judge        # a second opinion on the low-confidence verdicts
+proofpath check paper.pdf --summarize    # one model-written paragraph over the finished report
+proofpath config check                   # proves the key works before you spend a run on it
+```
+
+**What `--judge` does.** After the local verdicts are final, the verdicts the models
+were least sure about — the `low` tier, never a numeric mismatch and never a claim
+without a quoted passage — go to the model in batches of up to 20 (about 7k tokens),
+each with its claim and the passage it was checked against. The model answers from the
+passage alone, and its label and one-sentence rationale are printed **beside** the local
+verdict: `= judge (groq openai/gpt-oss-120b): NEI — …`. The local verdict, the finding
+kind and the report's states never change. Opinions are cached with the verdict, so a
+re-run asks nothing.
+
+**What it cannot do.** It never sees a source document, so it cannot introduce a claim
+or an evidence passage of its own; it cannot turn `NEI` into `SUPPORTED`; it cannot
+hide a source that could not be read. If the provider is down, rate-limited or the key
+is wrong, the run finishes on the local verdicts and says so in the stage line, the
+report header (`judge status:`) and the JSON — `-q` cannot hide it.
+
+**What `--summarize` does.** One final call turns the finished markdown report into 3–5
+plain sentences a reader can act on. It runs after the report is complete, its only
+input is that report, it is off by default in the CLI and the TUI (`/summarize` there),
+and the output is labelled `(model-written, <provider> <model>)`. `--summarize` alone
+is exactly one call; with `--judge` the escalation runs first.
+
+**Cost.** The footer counts the calls and the stage line the tokens:
+`Judging … 1 of 10 verdicts reviewed, 1 call, 613 prompt · 193 completion tokens` and
+`Summarising … 98 words, 1 call, 1,608 prompt · 343 completion tokens`
+on the live draft ([details](docs/eval/2026-09-15-judge-live.md)). Groq's free tier
+allows roughly one call a minute. Only the low-tier verdicts are sent — 1 of 10 on that
+draft — and up to 20 go in one call, so a long bibliography costs a handful of calls, not
+one per citation.
+
+**Providers.** Default is Groq `openai/gpt-oss-120b` (free without a card, no training
+on submitted data). `proofpath config set judge.provider gemini` switches to Gemini —
+note that Google trains on free-tier prompts outside the EEA/UK/CH, and proofpath prints
+that warning once per run. `judge.provider ollama` runs fully offline. Gemini and Ollama are
+fixture-tested and were not exercised live in v0.3.0. All three speak
+the OpenAI `chat/completions` shape. The key comes from `GROQ_API_KEY` / `GEMINI_API_KEY`
+in the environment or a `.env` file, never from config, and is never printed.
 
 ## Measured
 
@@ -234,6 +306,21 @@ comes from the environment or a `.env` file, never from config, and is never pri
 | Source access | 50 DOIs | 72 % full text, 18 % abstract only, 10 % nothing ([details](docs/eval/2026-09-11-coverage.md)) — a real biomedical paper in the live runs reached 33 % full text |
 | Citation pairing, numeric | 61 hand-built passages | 0.99 ([details](docs/eval/2026-09-11-pairing.md)) |
 | Citation pairing, author-year | 55 hand-built passages, 83 expectations | 0.940 ([details](docs/eval/2026-09-12-pairing-author-year.md)) |
+| **End to end on real web claims** | AVeriTeC dev, 100 claims | **0.270 3-way accuracy against a 0.708 majority baseline — worse than always guessing "refuted"** ([details](docs/eval/2026-09-16-averitec.md)) |
+
+**The AVeriTeC row is the one to read before trusting this tool on a news claim.** A third
+of those claims had no readable source at all: 32 of the source URLs needed the browser
+step, 29 were unreachable, 14 were refused by `robots.txt`. On the claims that *did* have a
+readable source the score is 0.361 — still below the baseline, and every one of the 19
+`Supported` claims was missed. The retrieval and entailment models were calibrated on
+scientific abstracts, and a fact-check page is a different object: long, discursive, and
+usually quoting the claim it debunks. Nothing was tuned after that measurement, and no
+blocked URL was dropped from it.
+
+What proofpath is good at is the academic path the other rows measure: finding out whether a
+cited paper exists, whether it was retracted, and whether its text says what the sentence
+citing it claims. Pointed at a news claim on the open web, it is currently a coverage
+report with a weak verdict attached.
 
 Published SciFact results sit around 70–75 F1, not 95. Nothing is tuned on a test
 split, and no number is quoted without the run that produced it.

proofpath-0.4.0/docs/eval/2026-09-15-judge-live.md

@@ -0,0 +1,95 @@
+# Judge layer live runs — 2026-09-15 (v0.3.0)
+
+`proofpath check tests/data/draft-live.md --judge --summarize` on Groq
+(`openai/gpt-oss-120b`, key from `.env`), Apple Silicon, models already downloaded.
+Three runs: the first against the pre-fix build (it found the defect), then a cold run
+(`--no-cache`) and a warm run against the shipped build. Stage lines are copied from
+the terminal; the key never appears in any output.
+
+## Run 1 — pre-fix build: the summary came back empty
+
+```
+Judging      groq openai/gpt-oss-120b   1 of 10 verdicts reviewed, 1 call, 661 prompt · 211 completion tokens 0.9s
+Summarising  groq openai/gpt-oss-120b   summary unavailable after 0 calls (no completion in the 200 response from https://api.groq.com/openai/v1/chat/completions (finish_reason=length)); local verdicts stand 1.0s
+```
+
+`openai/gpt-oss-120b` is a reasoning model: with the planned `max_tokens=400` the
+completion budget was spent on reasoning and the visible content was empty, and the
+written `report.md` said nothing about the summary at all (the 9.3 reviewer had found
+the same gap from the code). Both are fixed in the shipped build: requests carry
+`reasoning_effort=low` (dropped on a 400 from providers that reject it), the review
+budget is 4096 tokens and the summary budget 1500, and an unanswered summary is
+reported on the stage line, the `summary` CLI line and a `- summary status:` header
+line in the file. The 1,888-test suite was green before this run; the live run found it.
+
+## Run 2 — cold (`--no-cache`), shipped build
+
+```
+Parsing      text                              1 pages, 7 refs        0.0s
+Claims       rules                             7 citations, 0 unresolved 0.0s
+Resolving    Crossref, Semantic Scholar        6 ok, 0 amb, 1 ghost   9.3s
+Retractions  Retraction Watch                  none                   2.5s
+Fetching     arXiv, Semantic Scholar           5 full text, 1 abstract, 0 unverified 23.8s
+Verifying    coreml                            10 claims: 1 supported, 2 not supported, 7 NEI 44.5s
+Judging      groq openai/gpt-oss-120b          1 of 10 verdicts reviewed, 1 call, 613 prompt · 193 completion tokens 1.1s
+Summarising  groq openai/gpt-oss-120b          98 words, 1 call, 1,608 prompt · 343 completion tokens 1.0s
+… written  ·  2 API calls  ·  82.1s
+judge      2,221 prompt · 536 completion tokens
+```
+
+The escalation set was one verdict — the `low`-tier `NOT SUPPORTED` on the SciPy
+sentence — and the judge's opinion was printed beside it, the local verdict untouched:
+
+```
+error[not-supported]: claim is not supported by the cited source  (confidence: low)
+   = judge (groq openai/gpt-oss-120b): NEI — Passage states "SciPy provides fundamental
+     algorithms for scientific computing" but does not mention SciPy building on anything.
+```
+
+The numeric mismatch (`high`, rule-decided) and the ghost reference were not sent, as
+specified. The summary, verbatim:
+
+> The report checked seven cited references, and it found two claims not supported and
+> one ghost reference; it does not give a total number of claims that were examined. The
+> most serious problem is the high‑confidence "NOT SUPPORTED" error on line 6, where the
+> claim about training time for the Transformer contradicts the source. The coverage
+> analysis shows that only 72 % of the material was verified against full text, 14 % was
+> limited to abstracts, and the remaining 14 % could not be read, so the counts above
+> apply only to the portion that was actually examined.
+
+It repeats the coverage caveat as instructed and invents no finding. Two things it
+gets slightly wrong, which is why it is labelled model-written and sits under the
+computed report: it calls the line-6 numeric mismatch a "NOT SUPPORTED" error (the
+report says `numeric-mismatch: claim contradicts the cited source`), and "it does not
+give a total number of claims" is true of the markdown file (the stage table is not
+printed there) but not of the run (`10 claims`).
+
+## Run 3 — warm, shipped build
+
+```
+Resolving    cache                             6 ok, 0 amb, 1 ghost   0.0s
+Fetching     cache                             5 full text, 1 abstract, 0 unverified 0.0s
+Verifying    coreml                            10 claims: 1 supported, 2 not supported, 7 NEI, 10 cached 0.8s
+Judging      groq openai/gpt-oss-120b          1 of 10 verdicts reviewed, 0 calls, 0 prompt · 0 completion tokens 0.0s
+Summarising  groq openai/gpt-oss-120b          106 words, 1 call, 1,615 prompt · 263 completion tokens 0.9s
+… written  ·  1 API calls  ·  1.8s
+judge      1,615 prompt · 263 completion tokens
+```
+
+The judgement came back from the schema-v4 `judgements` table (0 calls); the summary
+is always one fresh call. A plain `check` of the same file (no flags) reports
+`api_calls: 0` and `summary: null`.
+
+## What this does and does not show
+
+- The judge layer works end to end on Groq's free tier: batching, strict-JSON output,
+  cost accounting, caching, attribution, and the report unchanged by the model.
+- One document, one low-tier verdict, one provider. Nothing here measures the judge's
+  accuracy; the escalation band is narrow by design (OPEN-ITEMS 14.1), so a paper with
+  118 citations would send a handful of verdicts, not dozens.
+- Gemini and Ollama were not exercised live; they share the adapter and are covered by
+  the `respx` tests only.
+- These runs were made with the batch cap at 7,000 prompt tokens. The whole-phase review
+  lowered it to 3,500 so that a full batch plus its 4,096-token answer budget stays under
+  Groq's 8K-per-minute tier; with one escalated item the runs above never approached
+  either figure, so nothing here measures the cap.

proofpath-0.4.0/docs/eval/2026-09-16-averitec.md

@@ -0,0 +1,95 @@
+# AVeriTeC dev — 2026-09-16
+
+- claims: 100  (limit=100)
+- dataset: `https://raw.githubusercontent.com/MichSchli/AVeriTeC/main/data/dev.json`  (sha256 499793726b4a…)
+- one run of the whole product: real claims, real source pages, real fetch ladder.
+
+## Headline
+
+3-way accuracy **0.270** vs majority baseline 0.708, over 89 of 100 claims.
+
+The 3-way number excludes the Conflicting Evidence/Cherrypicking rows, which proofpath has no verdict for; counting them as wrong gives a 4-way accuracy of 0.240.
+
+## Per label
+
+| label | n | correct | accuracy |
+|---|---|---|---|
+| Supported | 19 | 0 | 0.000 |
+| Refuted | 63 | 20 | 0.317 |
+| Not Enough Evidence | 7 | 4 | 0.571 |
+| Conflicting Evidence/Cherrypicking | 11 | 0 | 0.000 |
+
+## Source coverage
+
+| state | count |
+|---|---|
+| ok | 120 |
+| UNVERIFIED (blocked, browser not permitted) | 32 |
+| UNVERIFIED (unreachable) | 29 |
+| UNVERIFIED (blocked, robots.txt) | 14 |
+| not a url | 9 |
+| UNVERIFIED (reached, no text extracted) | 3 |
+| UNVERIFIED (provider unavailable) | 2 |
+
+76 of 200 source URLs are web.archive.org snapshots (38.0 %).
+
+## What the numbers say
+
+**proofpath is worse than always guessing "Refuted" on this set.** The majority baseline is
+0.708 because 63 of the 89 answerable dev claims are refutations; proofpath scores 0.270.
+That is the headline and it is not softened anywhere else in this repository.
+
+Where the loss comes from, from the per-claim rows:
+
+| | count |
+|---|---|
+| claims where **no** source could be read | 33 of 100 |
+| claims with at least one readable source | 67 |
+| 3-way accuracy on those 67 (61 answerable) | **0.361** |
+| claims answered `NEI` | 31 |
+| claims answered nothing at all (no source read) | 33 |
+| claims answered `SUPPORTED` | 7 |
+| claims answered `REFUTED` | 29 |
+
+Two separate problems, and the coverage one is the larger:
+
+1. **A third of the claims had nothing to read.** 32 source URLs needed the browser step
+   (this run passed `--no-browser`), 29 were unreachable, 14 were refused by `robots.txt`.
+   AVeriTeC's evidence is news sites, fact-check pages and archived snapshots — the exact
+   corner of the web that a polite, browserless fetcher is worst at. A claim with no
+   readable source is reported as such and counts as `NEI`, which is honest and still wrong
+   against the gold label 26 of the 28 times such a claim was scorable at all. Counting every
+   claim the run answered `NEI` — the 31 it decided and the 33 it could not read — 56 are
+   scorable and 52 of those are wrong.
+2. **Even with a source in hand, entailment over a news page is much harder than over an
+   abstract.** On the claims that did have a readable source the score is 0.361, still below
+   the baseline. Every one of the 19 `Supported` claims was missed: seven came back `NEI`,
+   seven had nothing to read, five were called `REFUTED`. The retrieval and NLI models were
+   calibrated on SciFact — scientific abstracts, one claim per abstract — and a fact-check
+   page is a different object: long, discursive, and often quoting the claim it debunks.
+
+The `Refuted` column is the only one that works (20 of 63), and some of that is the prior
+doing the work rather than the evidence.
+
+## What this does not say
+
+- It is not a statement about proofpath's academic path. SciFact dev (abstracts, 340 pairs)
+  scores 0.609 with a 0.406 baseline; the ghost set scores 0.0 % false ghosts. Those measure
+  different things and are measured separately.
+- It is not tuned away. No threshold was moved after seeing this, no URL was dropped for
+  being blocked, and the coverage table above prints every state rather than the ones that
+  flatter the run.
+- 38 % of the source URLs are `web.archive.org` snapshots, so a good part of what was read
+  is an archived copy of the page the dataset's annotators saw, not today's page.
+
+## How it was run
+
+`uv run python scripts/eval_averitec.py --limit 100 --no-browser --sleep 1.0 --resume`
+on the v0.4.0 development build, 2026-09-16, Apple Silicon, cache warm from nothing.
+Per-claim rows are in `~/Library/Caches/proofpath/datasets/averitec_results.json`; the run
+is resumable, so the same command reproduces it without re-fetching. The dataset is pinned
+by sha256 and the loader refuses a mismatch.
+
+Allowing the browser step would raise coverage and probably the score; it was left off so
+the number describes the default install, which is what a reader gets.
+