proofpath 0.3.0__tar.gz → 0.4.0__tar.gz

proofpath-0.4.0/.env.example

@@ -0,0 +1,9 @@
+# Copy to .env (gitignored) and fill in. Only the optional LLM judge needs a key;
+# everything else runs offline. See `proofpath judge` for where this file is read.
+GROQ_API_KEY=
+# GEMINI_API_KEY=
+# A free Reddit app (https://www.reddit.com/prefs/apps, type "script"): its id.
+REDDIT_CLIENT_ID=
+# The same app's secret. Without both, a cited Reddit post is reported as
+# UNVERIFIED (credentials missing) rather than silently skipped.
+REDDIT_CLIENT_SECRET=

{proofpath-0.3.0 → proofpath-0.4.0}/CHANGELOG.md

@@ -6,6 +6,53 @@ All notable changes to this project are documented here. The format follows
 
 ## [Unreleased]
 
+## [0.4.0] - 2026-09-16
+
+Sources that are not papers: a post's links, and the coverage block that finally says why
+a source could not be read. Also the first end-to-end measurement on real web claims, which
+says plainly that this is not yet the thing to point at a news story
+(`docs/eval/2026-09-16-averitec.md`).
+
+### Added
+- **`check --url`** reads a post and verifies **the links inside it**, never the post's own
+  words (spec §6.2). Bluesky (`public.api.bsky.app`) and Hacker News (the Firebase API) are
+  first-class and need no account; Reddit reads with a free app you register yourself
+  (`REDDIT_CLIENT_ID` / `REDDIT_CLIENT_SECRET` in `.env`); Mastodon is best effort per
+  instance; X cannot be read and says so, asking you to paste the text. Pasted text with
+  links in it is verified the same way.
+- **`UNVERIFIED (credentials missing)`** (spec §15): a platform that reads only with a
+  credential this machine does not have is its own state, never "unreachable". No request is
+  made, both variable names are printed, and the secret itself never reaches a log, an error
+  or a `repr`.
+- **The coverage block now prints a line per reason** on the terminal and in the markdown
+  report, not only in SARIF. `blocked: 3`, `credentials missing: 1`, `unreachable: 2` — a
+  source with no recorded reason is listed rather than dropped, so the lines account for
+  every unverified source. Product rule 6 on its main surface. The TUI footer does not carry
+  them yet.
+- **A `providers/` package** (`academic`, `web`, `social`) behind the `EvidenceProvider`
+  protocol of spec §5.2, so a new source family no longer means touching the core. The
+  academic and web paths came through byte-identical, pinned by a golden report.
+- **`docs/eval/2026-09-16-averitec.md`**: 100 AVeriTeC dev claims through the whole product.
+
+### Fixed
+- A bare DOI or arXiv **URL** is resolved as a record again instead of being fetched as a
+  web page, so it keeps its retraction check and its open-access full text.
+- A `doi:` source is read through the open-access chain and retraction-checked by the
+  academic provider whoever resolved the entry — a warm cache could previously route both
+  to the web ladder, where a missing retraction notice was reported as "no notice".
+- The `Resolving` stage no longer names Crossref and Semantic Scholar on a run that asked
+  neither.
+- A document that cites by linking is never paired as if it printed a numbered bibliography,
+  and vice versa: a PDF with a bibliography and no detected marker could previously pair a
+  body sentence to a reference it never cited.
+- `check --url` consults the network permission before reading anything; a denied run makes
+  no request at all.
+
+### Measured
+- AVeriTeC dev, 100 claims: **0.270 3-way accuracy against a 0.708 majority baseline**
+  (4-way 0.240). A third of the claims had no readable source; on the rest the score is
+  0.361. Every `Supported` claim was missed. Nothing was tuned after the measurement.
+
 ## [0.3.0] - 2026-09-15
 
 The judge layer: an opt-in LLM second opinion and an opt-in model-written summary.

{proofpath-0.3.0 → proofpath-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: proofpath
-Version: 0.3.0
+Version: 0.4.0
 Summary: Check whether the sources behind a claim actually say what the claim says.
 Project-URL: Homepage, https://github.com/Yigtwxx/proofpath
 Project-URL: Repository, https://github.com/Yigtwxx/proofpath
@@ -82,7 +82,7 @@ proofpath
   ╸┤ o o                                                                     ╰~~~~~~~~~~~~~[PROOF]
    ╰─┬─┬────────────────────────────────────────────────────────────────┬─┬──╯
      ˘ ˘                                                                ˘ ˘
-      proofpath v0.2.1                                                  academic . online . coreml
+      proofpath v0.4.0                                                  academic . online . coreml
       paste a file path, a URL, or a claim.                                  /help  /config  /quit
 ```
 
@@ -115,7 +115,7 @@ either, for screenshots and bug reports.
 ```
    ,_,
   (o.o)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~[PROOF]
-   " "    proofpath v0.2.1                                              academic . online . coreml
+   " "    proofpath v0.4.0                                              academic . online . coreml
           paste a file path, a URL, or a claim.            /help  /config  /quit
 ```
 
@@ -202,6 +202,7 @@ distinct, printed state (spec §15), never collapsed into a verdict:
 | `UNVERIFIED (reached, no text extracted)` | 200 answered, nothing readable came back |
 | `UNVERIFIED (network not permitted)` | `permissions.network = deny` |
 | `UNVERIFIED (provider unavailable)` | API down, rate limited after backoff, or answering with a page instead of a record |
+| `UNVERIFIED (credentials missing)` | the platform reads only with a credential this machine has none of — Reddit's free app, `REDDIT_CLIENT_ID` and `REDDIT_CLIENT_SECRET`. Nobody was asked, so it is not "unreachable"; the coverage block names both variables |
 | `UNVERIFIED (not in bibliographic indexes)` | web page, blog, report — indexes do not cover it, so absence proves nothing |
 | `AMBIGUOUS` | several plausible records, all listed |
 | `NEI` | the source was read and neither supports nor contradicts |
@@ -278,6 +279,32 @@ TablePlus or DBeaver — plain tables, no extension. Raw publisher text expires
 7 days; verdicts keep the passage they quote; a provider outage is never stored.
 `proofpath resolve REF` and `proofpath fetch URL|DOI` run either half on its own.
 
+## Posts and the links inside them (v0.4)
+
+A social post is not a source. What proofpath checks is whether the **links inside it**
+back what it says:
+
+```bash
+proofpath check --url https://bsky.app/profile/bsky.app/post/3movpwtbjgs2d
+proofpath check --url https://news.ycombinator.com/item?id=8863
+proofpath check -                 # paste the text of a post that cannot be read
+```
+
+Every sentence of the post is checked against the pages its links point to, and the post's
+own words are never allowed to stand as their own evidence.
+
+| Platform | How it is read |
+|---|---|
+| Bluesky | `public.api.bsky.app`, no account, first-class. Link cards, rich-text links and one level of quoted post |
+| Hacker News | the official Firebase API, no account, first-class. A story's URL and the links in a comment |
+| Reddit | with a **free app you register yourself**: put `REDDIT_CLIENT_ID` and `REDDIT_CLIENT_SECRET` in `.env`. Without them the run says `UNVERIFIED (credentials missing)` and names both variables — it never quietly skips the post |
+| Mastodon | best effort, per instance. Many instances now require a login for the public API, and that answer is reported as `UNVERIFIED (blocked)`, not as a missing post |
+| X / Twitter | cannot be read at all. `check --url` says so and asks you to paste the text; the links inside it are then verified normally |
+
+The Reddit path is built against Reddit's documented shapes and covered by fixtures, but it
+has never run against Reddit on this machine — nobody here has an app to register. Bluesky,
+Hacker News and Mastodon were each read live before release.
+
 ## Optional LLM judge (v0.3)
 
 Everything above runs locally, and the default run makes **zero** LLM calls. Two flags
@@ -335,6 +362,21 @@ in the environment or a `.env` file, never from config, and is never printed.
 | Source access | 50 DOIs | 72 % full text, 18 % abstract only, 10 % nothing ([details](docs/eval/2026-09-11-coverage.md)) — a real biomedical paper in the live runs reached 33 % full text |
 | Citation pairing, numeric | 61 hand-built passages | 0.99 ([details](docs/eval/2026-09-11-pairing.md)) |
 | Citation pairing, author-year | 55 hand-built passages, 83 expectations | 0.940 ([details](docs/eval/2026-09-12-pairing-author-year.md)) |
+| **End to end on real web claims** | AVeriTeC dev, 100 claims | **0.270 3-way accuracy against a 0.708 majority baseline — worse than always guessing "refuted"** ([details](docs/eval/2026-09-16-averitec.md)) |
+
+**The AVeriTeC row is the one to read before trusting this tool on a news claim.** A third
+of those claims had no readable source at all: 32 of the source URLs needed the browser
+step, 29 were unreachable, 14 were refused by `robots.txt`. On the claims that *did* have a
+readable source the score is 0.361 — still below the baseline, and every one of the 19
+`Supported` claims was missed. The retrieval and entailment models were calibrated on
+scientific abstracts, and a fact-check page is a different object: long, discursive, and
+usually quoting the claim it debunks. Nothing was tuned after that measurement, and no
+blocked URL was dropped from it.
+
+What proofpath is good at is the academic path the other rows measure: finding out whether a
+cited paper exists, whether it was retracted, and whether its text says what the sentence
+citing it claims. Pointed at a news claim on the open web, it is currently a coverage
+report with a weak verdict attached.
 
 Published SciFact results sit around 70–75 F1, not 95. Nothing is tuned on a test
 split, and no number is quoted without the run that produced it.

{proofpath-0.3.0 → proofpath-0.4.0}/README.md

@@ -26,7 +26,7 @@ proofpath
   ╸┤ o o                                                                     ╰~~~~~~~~~~~~~[PROOF]
    ╰─┬─┬────────────────────────────────────────────────────────────────┬─┬──╯
      ˘ ˘                                                                ˘ ˘
-      proofpath v0.2.1                                                  academic . online . coreml
+      proofpath v0.4.0                                                  academic . online . coreml
       paste a file path, a URL, or a claim.                                  /help  /config  /quit
 ```
 
@@ -59,7 +59,7 @@ either, for screenshots and bug reports.
 ```
    ,_,
   (o.o)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~[PROOF]
-   " "    proofpath v0.2.1                                              academic . online . coreml
+   " "    proofpath v0.4.0                                              academic . online . coreml
           paste a file path, a URL, or a claim.            /help  /config  /quit
 ```
 
@@ -146,6 +146,7 @@ distinct, printed state (spec §15), never collapsed into a verdict:
 | `UNVERIFIED (reached, no text extracted)` | 200 answered, nothing readable came back |
 | `UNVERIFIED (network not permitted)` | `permissions.network = deny` |
 | `UNVERIFIED (provider unavailable)` | API down, rate limited after backoff, or answering with a page instead of a record |
+| `UNVERIFIED (credentials missing)` | the platform reads only with a credential this machine has none of — Reddit's free app, `REDDIT_CLIENT_ID` and `REDDIT_CLIENT_SECRET`. Nobody was asked, so it is not "unreachable"; the coverage block names both variables |
 | `UNVERIFIED (not in bibliographic indexes)` | web page, blog, report — indexes do not cover it, so absence proves nothing |
 | `AMBIGUOUS` | several plausible records, all listed |
 | `NEI` | the source was read and neither supports nor contradicts |
@@ -222,6 +223,32 @@ TablePlus or DBeaver — plain tables, no extension. Raw publisher text expires
 7 days; verdicts keep the passage they quote; a provider outage is never stored.
 `proofpath resolve REF` and `proofpath fetch URL|DOI` run either half on its own.
 
+## Posts and the links inside them (v0.4)
+
+A social post is not a source. What proofpath checks is whether the **links inside it**
+back what it says:
+
+```bash
+proofpath check --url https://bsky.app/profile/bsky.app/post/3movpwtbjgs2d
+proofpath check --url https://news.ycombinator.com/item?id=8863
+proofpath check -                 # paste the text of a post that cannot be read
+```
+
+Every sentence of the post is checked against the pages its links point to, and the post's
+own words are never allowed to stand as their own evidence.
+
+| Platform | How it is read |
+|---|---|
+| Bluesky | `public.api.bsky.app`, no account, first-class. Link cards, rich-text links and one level of quoted post |
+| Hacker News | the official Firebase API, no account, first-class. A story's URL and the links in a comment |
+| Reddit | with a **free app you register yourself**: put `REDDIT_CLIENT_ID` and `REDDIT_CLIENT_SECRET` in `.env`. Without them the run says `UNVERIFIED (credentials missing)` and names both variables — it never quietly skips the post |
+| Mastodon | best effort, per instance. Many instances now require a login for the public API, and that answer is reported as `UNVERIFIED (blocked)`, not as a missing post |
+| X / Twitter | cannot be read at all. `check --url` says so and asks you to paste the text; the links inside it are then verified normally |
+
+The Reddit path is built against Reddit's documented shapes and covered by fixtures, but it
+has never run against Reddit on this machine — nobody here has an app to register. Bluesky,
+Hacker News and Mastodon were each read live before release.
+
 ## Optional LLM judge (v0.3)
 
 Everything above runs locally, and the default run makes **zero** LLM calls. Two flags
@@ -279,6 +306,21 @@ in the environment or a `.env` file, never from config, and is never printed.
 | Source access | 50 DOIs | 72 % full text, 18 % abstract only, 10 % nothing ([details](docs/eval/2026-09-11-coverage.md)) — a real biomedical paper in the live runs reached 33 % full text |
 | Citation pairing, numeric | 61 hand-built passages | 0.99 ([details](docs/eval/2026-09-11-pairing.md)) |
 | Citation pairing, author-year | 55 hand-built passages, 83 expectations | 0.940 ([details](docs/eval/2026-09-12-pairing-author-year.md)) |
+| **End to end on real web claims** | AVeriTeC dev, 100 claims | **0.270 3-way accuracy against a 0.708 majority baseline — worse than always guessing "refuted"** ([details](docs/eval/2026-09-16-averitec.md)) |
+
+**The AVeriTeC row is the one to read before trusting this tool on a news claim.** A third
+of those claims had no readable source at all: 32 of the source URLs needed the browser
+step, 29 were unreachable, 14 were refused by `robots.txt`. On the claims that *did* have a
+readable source the score is 0.361 — still below the baseline, and every one of the 19
+`Supported` claims was missed. The retrieval and entailment models were calibrated on
+scientific abstracts, and a fact-check page is a different object: long, discursive, and
+usually quoting the claim it debunks. Nothing was tuned after that measurement, and no
+blocked URL was dropped from it.
+
+What proofpath is good at is the academic path the other rows measure: finding out whether a
+cited paper exists, whether it was retracted, and whether its text says what the sentence
+citing it claims. Pointed at a news claim on the open web, it is currently a coverage
+report with a weak verdict attached.
 
 Published SciFact results sit around 70–75 F1, not 95. Nothing is tuned on a test
 split, and no number is quoted without the run that produced it.

proofpath-0.4.0/docs/eval/2026-09-16-averitec.md

@@ -0,0 +1,95 @@
+# AVeriTeC dev — 2026-09-16
+
+- claims: 100  (limit=100)
+- dataset: `https://raw.githubusercontent.com/MichSchli/AVeriTeC/main/data/dev.json`  (sha256 499793726b4a…)
+- one run of the whole product: real claims, real source pages, real fetch ladder.
+
+## Headline
+
+3-way accuracy **0.270** vs majority baseline 0.708, over 89 of 100 claims.
+
+The 3-way number excludes the Conflicting Evidence/Cherrypicking rows, which proofpath has no verdict for; counting them as wrong gives a 4-way accuracy of 0.240.
+
+## Per label
+
+| label | n | correct | accuracy |
+|---|---|---|---|
+| Supported | 19 | 0 | 0.000 |
+| Refuted | 63 | 20 | 0.317 |
+| Not Enough Evidence | 7 | 4 | 0.571 |
+| Conflicting Evidence/Cherrypicking | 11 | 0 | 0.000 |
+
+## Source coverage
+
+| state | count |
+|---|---|
+| ok | 120 |
+| UNVERIFIED (blocked, browser not permitted) | 32 |
+| UNVERIFIED (unreachable) | 29 |
+| UNVERIFIED (blocked, robots.txt) | 14 |
+| not a url | 9 |
+| UNVERIFIED (reached, no text extracted) | 3 |
+| UNVERIFIED (provider unavailable) | 2 |
+
+76 of 200 source URLs are web.archive.org snapshots (38.0 %).
+
+## What the numbers say
+
+**proofpath is worse than always guessing "Refuted" on this set.** The majority baseline is
+0.708 because 63 of the 89 answerable dev claims are refutations; proofpath scores 0.270.
+That is the headline and it is not softened anywhere else in this repository.
+
+Where the loss comes from, from the per-claim rows:
+
+| | count |
+|---|---|
+| claims where **no** source could be read | 33 of 100 |
+| claims with at least one readable source | 67 |
+| 3-way accuracy on those 67 (61 answerable) | **0.361** |
+| claims answered `NEI` | 31 |
+| claims answered nothing at all (no source read) | 33 |
+| claims answered `SUPPORTED` | 7 |
+| claims answered `REFUTED` | 29 |
+
+Two separate problems, and the coverage one is the larger:
+
+1. **A third of the claims had nothing to read.** 32 source URLs needed the browser step
+   (this run passed `--no-browser`), 29 were unreachable, 14 were refused by `robots.txt`.
+   AVeriTeC's evidence is news sites, fact-check pages and archived snapshots — the exact
+   corner of the web that a polite, browserless fetcher is worst at. A claim with no
+   readable source is reported as such and counts as `NEI`, which is honest and still wrong
+   against the gold label 26 of the 28 times such a claim was scorable at all. Counting every
+   claim the run answered `NEI` — the 31 it decided and the 33 it could not read — 56 are
+   scorable and 52 of those are wrong.
+2. **Even with a source in hand, entailment over a news page is much harder than over an
+   abstract.** On the claims that did have a readable source the score is 0.361, still below
+   the baseline. Every one of the 19 `Supported` claims was missed: seven came back `NEI`,
+   seven had nothing to read, five were called `REFUTED`. The retrieval and NLI models were
+   calibrated on SciFact — scientific abstracts, one claim per abstract — and a fact-check
+   page is a different object: long, discursive, and often quoting the claim it debunks.
+
+The `Refuted` column is the only one that works (20 of 63), and some of that is the prior
+doing the work rather than the evidence.
+
+## What this does not say
+
+- It is not a statement about proofpath's academic path. SciFact dev (abstracts, 340 pairs)
+  scores 0.609 with a 0.406 baseline; the ghost set scores 0.0 % false ghosts. Those measure
+  different things and are measured separately.
+- It is not tuned away. No threshold was moved after seeing this, no URL was dropped for
+  being blocked, and the coverage table above prints every state rather than the ones that
+  flatter the run.
+- 38 % of the source URLs are `web.archive.org` snapshots, so a good part of what was read
+  is an archived copy of the page the dataset's annotators saw, not today's page.
+
+## How it was run
+
+`uv run python scripts/eval_averitec.py --limit 100 --no-browser --sleep 1.0 --resume`
+on the v0.4.0 development build, 2026-09-16, Apple Silicon, cache warm from nothing.
+Per-claim rows are in `~/Library/Caches/proofpath/datasets/averitec_results.json`; the run
+is resumable, so the same command reproduces it without re-fetching. The dataset is pinned
+by sha256 and the loader refuses a mismatch.
+
+Allowing the browser step would raise coverage and probably the score; it was left off so
+the number describes the default install, which is what a reader gets.
+

{proofpath-0.3.0 → proofpath-0.4.0}/docs/superpowers/OPEN-ITEMS.md

@@ -461,8 +461,52 @@ Findings from inspecting the installed base environment, and the decisions they
 | 14.11 | The unanswered judge is absent from the SARIF log | `_run_properties` carries `models` and `apiCalls` but no `judge status` / `summary status`, so a `--format sarif` log of a run whose provider was down reads like one with nothing to escalate. SARIF is a findings document, and the markdown, JSON and terminal all say it; still, a viewer-only workflow does not see it |
 | 14.12 | Groq's free tier is metered on prompt **plus** requested answer | the whole-phase review caught `TOKEN_CAP = 7000` left over from the 1,024-token answer budget: with the 4,096-token budget of 9.3 a full batch would ask for ~11.1k against an 8K/minute tier. The cap is now 3,500. Nothing measured this live — the live escalation set was one item |
 
+### Carried into Phase 10
+
+| # | Item | Note |
+|---|---|---|
+| 14.21 | `Document.kind` now mixes origin with citation style | the five original values say where a document came from (`pdf`, `docx`, `markdown`, `text`, `post`); `linked`, added so `claims.extract` could stay a pure function of the document, says how it cites — a paste is `text` in origin and the field no longer records that. The clean shape is a second field (`cites: "marker" | "link"`), blocked today only because `Report.document` is serialised field by field into the frozen golden payload, so any new field rewrites it. Visible to a user only as `document.kind` in `--format json` |
+| 14.16 | A post is never cached | the fetch cache lives inside `Fetcher`, which the social provider bypasses, so a post is re-read on every run and `--no-cache` changes nothing for one. Nothing stale is served (the chunk cache is pinned to the text's digest), but the `Fetching` line can never attribute a post to `cache` |
+| 14.17 | A post's text is graded `fulltext` at any length | `text_kind` answers "how much of the source was read", and an API-returned post is complete, so `fulltext` is the honest word — but a three-word post cited by a bibliography then counts as a full-text source in the coverage block. Distinguishing "short source, read whole" from "long document read whole" is a `Coverage` change, not a grading one |
+| 14.18 | A directly constructed `Engine` builds a `PoliteClient` it never closes | `Engine.default` is clean (it passes the client already on `_closers`); a hand-built one allocates an `httpx` transport per engine. No socket opens until a post is read, so nothing leaks functionally |
+| 14.19 | Pasted text with no links gets no "nothing to verify" note | the stage-1 note is gated on the post path, and a paste carrying no addresses never becomes `kind="linked"`, so it prints zero findings — which is what a clean run looks like. Spec §6.2 makes this the designated path for an unreadable post, so it is the one place the gap matters |
+| 14.20 | `check --url` flattens two §15 states into one exit code | an unreachable post and an unavailable provider both exit 2; the hint distinguishes them, the exit code does not |
+| 14.15 | The resolution cache is shared across routing changes | `cache.put_resolution` stores whatever the entry's provider answered under the entry's raw string, so a row written by one release is read back by another whose routing differs. Both of Task 10.1's warm-cache defects (a `doi:` source read by the web provider; the same source skipping the retraction check) reached the code through exactly this door, and it will open again whenever a provider's routing moves |
+
 ### Next session
 
 1. **Phase 10** (`providers/` refactor, Bluesky/HN, Reddit/Mastodon/X, AVeriTeC) runs
    from `plans/2026-09-12-phases-9-10-plan.md`; Task 10.4a (AVeriTeC loader and scorer)
    is already implemented and reviewed, unstaged behind the v0.3.0 tag.
+
+## 15. Phase 10 — v0.4.0, 2026-09-16
+
+- Shipped: the `providers/` package (10.1; academic and web moved behind the §5.2 protocol
+  with the report byte-identical, pinned by a golden payload written before the refactor),
+  Bluesky and Hacker News with `check --url` and pasted-text links (10.2), Reddit/Mastodon/X
+  and the per-reason coverage block (10.3), and the AVeriTeC harness (10.4).
+- **The headline measurement is below baseline and is reported as such everywhere**:
+  0.270 3-way against a 0.708 majority baseline on 100 AVeriTeC dev claims
+  (`docs/eval/2026-09-16-averitec.md`). A third of the claims had no readable source; on the
+  rest the score is 0.361; every `Supported` claim was missed. Nothing was tuned after the
+  run and no blocked URL was dropped from it.
+- Reviewers caught, before the tag: a bare DOI/arXiv URL losing its record and its retraction
+  check; a warm-cache `doi:` source read up the web ladder with the open-access chain never
+  asked, and retraction-checked by a provider that always answers "no notice"; the `Resolving`
+  line naming providers nobody called; a PDF's body sentences paired to a bibliography entry
+  they never cited; `check --url` fetching before the network permission was read. Five
+  product-rule failures the 1,900-test suite did not catch on its own.
+
+### New open items
+
+| # | Item | Note |
+|---|---|---|
+| 15.1 | proofpath is below baseline on real-world web claims | the two causes are separable: a third of the claims had nothing readable (the browser step is off by default, `robots.txt` is honoured, news sites block), and the entailment models were calibrated on scientific abstracts. Neither is fixed by tuning a threshold. A browser-enabled run was not measured |
+| 15.2 | The Reddit path has never run against Reddit | fixtures reproduce the documented shapes; nobody on this machine has an app to register. The first real-credential run should check the comment-listing shape first |
+| 15.3 | `/r/<sub>/s/<hash>` share links are unrecognised | they degrade to the web ladder, so nothing is invented; the shape is simply not parsed |
+| 15.4 | The TUI footer ignores `Footer.reasons` | the per-reason coverage lines reach the terminal and the markdown report but not the TUI, so a TUI run whose sources were half blocked still shows three numbers and no reason |
+| 15.7 | A Mastodon status is read outside the fetch ladder | `check --url https://<any host>/@a/123` issues a request to that host's `/api/v1/statuses/123` directly: no `robots.txt` check and no entry in `polite.MIN_INTERVAL`. The user typed the address, so it is not a rule breach, but it is the one read path that skips the politeness machinery |
+| 15.8 | `target_document`'s `network_allowed` defaults to `True` | every in-tree caller passes the resolved permission, so the default is unreachable today; it is a footgun for the next caller and should be a required keyword |
+| 15.9 | The Reddit token exchange follows redirects with basic auth attached | httpx re-applies the credential along a redirect chain. The URL is a fixed Reddit constant, so it needs Reddit itself to be the attacker, but `follow_redirects=False` on that one call costs nothing |
+| 15.5 | `NO_REASON_RECORDED` is a user-visible state outside §15 | it is a guard that should never print; if it can print, it belongs in the table |
+| 15.6 | `NOT_READ_HERE` is provably unreachable through `verify` | kept deliberately for a sixth platform, like `checker_for`'s `arxiv:` row |

{proofpath-0.3.0 → proofpath-0.4.0}/docs/superpowers/plans/2026-09-10-proofpath-implementation-plan.md

@@ -331,6 +331,17 @@ For any unreadable post, the user pastes the text and `proofpath` verifies the
 
 **Done when:** measured on AVeriTeC, and no platform integration requires a paid API.
 
+**Status (2026-09-16): released as v0.4.0.** Tasks 10.1–10.4 (`.superpowers/sdd/phase10/`,
+briefs in `2026-09-12-phases-9-10-plan.md`): the `providers/` package with the academic and
+web families refactored behind it byte-identically (golden report pinned before the change
+and unchanged after); Bluesky and Hacker News with `check --url` and pasted-text links;
+Reddit (user's own OAuth app, fixtures only — nobody here has an app), Mastodon (live read)
+and X (paste only); the coverage block printing a line per reason; and the AVeriTeC harness.
+No platform integration requires a paid API. **The measurement is below baseline:** 0.270
+3-way against 0.708 on 100 dev claims (`docs/eval/2026-09-16-averitec.md`). It is reported
+in the README's own Measured table, in the spec §14 row and in the changelog, unsoftened,
+because the alternative is a tool that looks better than it is.
+
 ---
 
 ## What is deliberately not planned yet

{proofpath-0.3.0 → proofpath-0.4.0}/docs/superpowers/specs/2026-09-10-proofpath-design.md

@@ -821,7 +821,7 @@ stage so a regression can be located.
 |---|---|---|
 | SciFact (AI2 tarball, sha256 pinned; not the HF loader) | retrieval + entailment on scientific claims | label accuracy, macro-F1, rationale F1 |
 | FEVER | retrieval + entailment baseline | label accuracy |
-| AVeriTeC | real-world web claim verification | AVeriTeC score |
+| AVeriTeC | real-world web claim verification | AVeriTeC score. **Measured 2026-09-16 on 100 dev claims: 0.270 3-way against a 0.708 majority baseline** (4-way 0.240) — below the baseline. A third of the claims had no readable source (browser step off, unreachable, `robots.txt`); on the rest, 0.361, and every `Supported` claim was missed. `docs/eval/2026-09-16-averitec.md` |
 | PubHealth | high-harm domain behaviour | label accuracy |
 | X Community Notes | social provider sanity check | agreement with human notes |
 | Hand-built ghost set | reference resolution (§8) | precision/recall on ghosts, **false-ghost rate** |
@@ -885,11 +885,13 @@ presented as evidence of absence.
 | `LOW CONFIDENCE (abstract only)` | full text unavailable, abstract used |
 | `UNVERIFIED (blocked)` | 403/bot protection, Scrapling absent or defeated |
 | `UNVERIFIED (unreachable)` | dead link, Wayback miss |
+| `UNVERIFIED (no identifier to fetch)` | the reference resolved to a record that carries no DOI, arXiv id or address, so there is nothing to fetch |
 | `UNVERIFIED (reached, no text extracted)` | the page answered 200 but no text could be extracted (bot wall, image-only page) |
 | `UNVERIFIED (blocked, browser not permitted)` | steps 1–2 blocked and the §7.1 consent was denied, absent, or impossible without a TTY |
 | `UNVERIFIED (blocked, robots.txt)` | the site's `robots.txt` disallows the fetch; steps 3–4 are not attempted |
 | `UNVERIFIED (network not permitted)` | `permissions.network = deny`; nothing was fetched |
 | `UNVERIFIED (provider unavailable)` | API down or rate limited after backoff |
+| `UNVERIFIED (credentials missing)` | the source is on a platform that reads only with a credential this machine does not have — today Reddit, whose free app is `REDDIT_CLIENT_ID` and `REDDIT_CLIENT_SECRET`. Nobody was asked, so it is neither unreachable nor blocked; the coverage block names the two variables (§6.2) |
 | `AMBIGUOUS` | multiple plausible reference candidates — all listed |
 | `NEI` | source read, but it neither supports nor contradicts |
 | `UNVERIFIED (not in bibliographic indexes)` | web page, blog, report, manual or organisation-authored document; indexes do not cover it, so absence proves nothing (§8.1) |
@@ -905,8 +907,16 @@ Every report ends with a coverage summary:
 verified against full text   62%
 abstract only                21%
 unverified                   17%
+  blocked: 3
+  credentials missing: 1
 ```
 
+Whenever anything is unverified, every reason is listed under the share it
+explains — most common first, then by name — in the terminal and in the markdown
+report alike. Three percentages with no reason beside them let a run whose sources
+were half blocked read exactly like a thin one. `UNVERIFIED (credentials missing)`
+also puts the names of the variables that would fix it under the block.
+
 A run where coverage is low is a run whose conclusions are weak, and the user is
 told so directly.
 
@@ -953,6 +963,14 @@ an unanswered judge reported on every surface but SARIF. Live run: `docs/eval/20
 **v0.4 — social provider.** Bluesky and Hacker News first, Reddit via user-supplied
 OAuth app (missing credentials are reported, never silently skipped), Mastodon
 best-effort, Community Notes dumps for X. Measured on AVeriTeC.
+*Shipped 2026-09-16 as v0.4.0:* the `providers/` package (§5.2) with the academic and web
+families moved behind it byte-identically; `check --url` reading a post and verifying the
+links inside it; Bluesky, Hacker News, Reddit (user's own app), Mastodon best effort, X by
+pasting; `UNVERIFIED (credentials missing)` as its own state; the coverage block printing a
+line per reason on the terminal and in the markdown report (the TUI footer does not yet
+carry them — OPEN-ITEMS 15.4). X Community Notes dumps were **not** built — the paste
+path replaced them (OPEN-ITEMS 3.5). Measured: `docs/eval/2026-09-16-averitec.md`, and the
+number is bad — see §14.
 
 **Later.** Turkish sources as a separate provider (TR Dizin / DergiPark class),
 `spiyweb` graph retrieval as an alternative backend, GROBID parser.