backdraft 0.7.0__tar.gz → 0.8.0__tar.gz

{backdraft-0.7.0 → backdraft-0.8.0}/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "backdraft",
   "displayName": "Backdraft",
-  "version": "0.7.0",
+  "version": "0.8.0",
   "description": "Drop-in provenance for factual claims: one click from any claim to the evidence behind it. Ships the backdraft writing, backfill, and artifact-reading skills; the CLI installs separately (uv tool install backdraft).",
   "author": {
     "name": "Spencer Braun",

{backdraft-0.7.0 → backdraft-0.8.0}/DESIGN.md

@@ -143,6 +143,10 @@ Self-contained HTML (plus the machine-readable sidecar): **quotes + receipts + e
 | 2026-08-11 | **An extractor declares the `--config` keys it reads, and an undeclared key is an error.** `--config dpy=300` used to exit 0, ingest, and change nothing: the config boundary was the one place this repo's "failures are data, never warn-and-drop" rule did not hold, and nothing — help, docs, `llms.txt` — said which keys existed, so learning them meant reading `extract/vlm.py` and `extract/snapshots.py`. The `Extractor` protocol gains an optional `config_keys: Mapping[str, str]` (key → one-line meaning) and `check_config` validates against it. Validation happens **after selection and inside `Registry.ingest`**, not in the CLI: `auto` picks per file, so only the chosen extractor can say whether `dpi` is a render setting or a typo, and putting it at the registry means every substrate is covered rather than only the one with a `--config` flag. It raises `ExtractionError` rather than the `UsageError` a CLI-side check would have used — `cli_context` imports the registry, so the registry cannot import `UsageError` back — and both are `BackdraftError`, so the guard maps either to exit 1 and the user sees the same line. Each key is declared once by the module that already owns its default and reads it: the page-render budget (`dpi`, `snapshot_quality`, `snapshot_max_height`) in `extract/snapshots.py`, the provider surface (`api_key`, `base_url`, `model`, `timeout`, `retries`) in `extract/vlm_settings.py`, and the extractors compose those by reference. `pdf-text` declares the page-render keys despite reading none of them itself — the snapshot capture that runs beside it at ingest does, and a user passing `dpi` to a text-layer ingest means the images; declaring by *what one flag on one ingest affects* beats declaring by which function happens to call `config.get`. Tradeoff accepted: declaring nothing means accepting nothing, so an out-of-tree extractor that reads config without declaring it starts rejecting keys it used to receive. That is the intended direction — a silently ignored key is the failure being fixed — and the fix is one attribute. The help text that lists the keys is pinned by a test against the declarations, so a new key cannot be added in code and left undocumented. |
 | 2026-08-12 | **`show` is a gate verb, not a lookup tool.** `backdraft show <token>` answers what a token says — the inverse of minting, for a token that arrived out of an artifact, a draft, or somebody's message, which nothing in the tool could answer before without rebuilding the slug and selector by hand and re-reading a whole page. It therefore *mints what it shows*, exactly as `read` and `search` do: an agent that has seen a snippet through the gate has genuinely read it, so recording it is honest rather than a loophole, and `not_shown` stays true by construction — a status `show` can never produce. That makes a token laundering into a citable one by being looked at, which is the intended behavior and the reason this is in `gate/` and not somewhere neutral. Resolution is bind's path rather than a second reading of what a token means: `kernel.claims.parse_citation` decides `malformed` (so the reserved `bd:calc(...)` form lands identically in both) and `Registry.resolve` decides the rest, which is what makes a status printed here the status bind would print. Two smaller calls. `drifted` prints both snippets *and* mints the anchor standing at the locator now, since that anchor's token is the one worth citing and printing it unrecorded would be the under-minting the gate must never do. And drift does **not** fail the exit code: exit 1 covers `unresolved` and `malformed`, the two statuses where there was nothing to show, because whether a changed source still supports a sentence is bind's question about a document, not a lookup's about a token. Tradeoff accepted: auditing someone else's artifact against a registry now writes ledger rows into the auditor's session, so `skills/backdraft-artifact/SKILL.md` says so and keeps that check reported separately from the artifact's self-checks. |
 | 2026-08-13 | **`ingest` finishes its list; an unreadable source is data, not the end of the run.** `backdraft ingest a.md broken.pdf c.md` used to ingest `a.md`, raise on `broken.pdf`, and never attempt `c.md` — leaving a half-built registry, a message naming only the first failure, and no way to tell which sources landed from which were never tried, so ingesting a folder meant re-running and inferring the state. The loop now catches `BackdraftError` per source and records `(source, reason)`, so every source is attempted, and prints one `!` line per failure under a count (`2 of 3 sources ingested; 1 failed`) whose two numbers add to the whole list — which is precisely the fact the old behavior could not state. This is principle 4 ("failures are data, never warn-and-drop") applied at the boundary it had not reached, and the same shape the page-snapshot note already uses one level down. Grouping differs deliberately: the snapshot note groups by *reason* because one missing poppler explains every document at once, while this lists by *source*, because the source is what the caller has to go fix. The exit code is unchanged at 1 and `guard` remains the only place a `BackdraftError` becomes one — nothing caught per source is re-raised, and the report exits through `fail` like any other error line, on stderr, so a clean run's stdout is byte-identical to before and that is pinned by a test. Tradeoff accepted: a run that fails partway now leaves *more* in the registry than it used to, so "the command failed" no longer implies "nothing happened". The closing line of the report answers that directly — re-running the same list is safe, because a source already ingested with unchanged bytes, extractor and config is a no-op — which is cheaper for a caller than reconstructing the failed half by diffing `ls` against its own arguments. |
+| 2026-08-17 | **The demo cites a web page at a fixed revision, so the capability is visible and stays `resolved`.** URL sources shipped whole — fetch, snapshot, `url`/`fetched_at`, the artifact's live dated link — and none of it appeared at backdraft.dev/demo.html, because both demo sources were local files and the byte-identity test that proved the artifact change safe was the same fact that kept it invisible. The demo gains a third source: the Wikipedia article for Franklin County, Ohio — the county the fictional property sits in and the one its tax appeal is already pending before — cited for two market figures a T12 cannot supply, the county's population and its renter-occupied share. It is ingested at Wikipedia's **permanent link** (`?oldid=<rev>`) rather than the bare article URL. What that gives up is named: the demo no longer shows a live page changing under a citation, which is the better story and belongs in a `drifted` walkthrough rather than the shop window. What it buys is that the front page does not permanently display `drifted` within days of the next edit — a demo that always shows drift teaches that the tool is broken rather than that the source moved — and an immutable revision plus the rendered "as of" date is what a careful analyst cites anyway. The same reasoning is now advice rather than a one-off: the docs, `llms.txt` and the writing skill tell an agent to prefer a fixed-revision address (`?oldid=`, a DOI, an archived snapshot) and to pass `--slug` with it. `--slug` is load-bearing here, not decoration: a permanent link is served from `/w/index.php`, so the default stem would be `index` and every token in the memo would carry it. That is the queued slug-assignment item's problem and is fixed at the fetch layer there, not worked around here. One renderer change came with it. `source_title` took a source's display name from its filename's stem, which for a fetched page is the staging name `fetch.filename_for` invented — so the article titled itself "Index". Where a document carries a URL the slug is the name instead, which is the 2026-08-06 row's rule ("showing both would let the fictional one look authoritative") applied to the one surface that had kept the invented name. It changes nothing for a page ingested without `--slug`, since the slug is derived from that same stem; it defers to whoever named the document when someone did. |
+| 2026-08-18 | **One rule names a source, and every surface that names one uses it.** A fetched page's `filename` is the temporary file `fetch.filename_for` staged the bytes in — `index.html` for a permanent link, a name on nobody's disk — and the 2026-08-06 row settled what to do about it ("showing both would give a reader two names for one thing and let the fictional one look authoritative") for the artifact alone. The four CLI surfaces that name a document kept the fiction: `ingest`'s line, the gate's document list, its table-of-contents headline, and `ls` — which showed *both*, filename in column two and URL in a fifth field, the exact shape that row rejected. `gate.source_name` is now the single owner, exported the way `gate.unit` is and imported downward by `cli.py`, so a writing agent learns a source's provenance from the surface it already lives on instead of being sent to `ls` by the skill. Display only: the slug, the token and everything a citation resolves through are untouched, which is the 2026-08-05 "provenance, never identity" rule. Two consequences named rather than discovered. `ls` loses a field — a fetched row is four columns, not five — so anything parsing column five stops seeing one; that is the point, since column two was a lie and one true name beats two names of which one is fictional. And in the gate's aligned list a URL *overflows* the name column rather than widening it: the column is sized on filenames alone, so one 80-character address pushes its own row's later columns right instead of pushing every row's past column 100. The cost is a ragged right on fetched rows, accepted because the file rows a mostly-file registry is made of stay byte-identical to what they printed before — pinned by a test, the way the artifact's byte-identity is. `bind`'s unmatched-claim report still names a source by filename: the fix cannot be shared into it without a sideways `bind` → `gate` import, which SPEC § Dependency rule forbids, and duplicating the rule there is the reinvention this row exists to end. |
+| 2026-08-19 | **A slug is assigned by a rule, and the rule is written down.** The Open list carried "slug assignment/collision rules" from the first field trial, and URL sources sharpened it into a live failure: `fetch.filename_for` took the stem from a URL's last path segment, so `https://a.example/reports/2025/index.html` and `https://b.example/docs/index.html` both staged as `index.html` and became `index` and `index-2` — two handles that say which is which to nobody, carried by every token in every document written against them, permanently, since a slug is stable once assigned. The rule now has two clauses and they are stated together for files and URLs. A **file** is named by its filename's stem, kebabed and cut to 32 characters, which is unchanged and was never the problem. A **URL** is named by its last path segment, *except* when that segment names the site's plumbing rather than its page — a directory index or generic handler (`_GENERIC_STEMS`: `index`, `view`, `page`, `show`, and a dozen more), an all-digit id, or no segment at all — in which case the host goes in front of it, minus `www.` and minus the port, so the two pages above become `a-example-index` and `b-example-index`. The common case is deliberately untouched: `https://example.com/reports/q4-2025` is still `q4-2025`, because a rule that made every good name uglier to fix the bad ones would be the wrong trade. `_dedupe`'s `-2` stays, demoted from the normal outcome to the backstop it was meant to be — the host is a *better* stem, not a unique one, and two `index.html` pages on one site still collide. Truncation is trimmed on both cuts now (`slug_for` and `_dedupe`), which is the other half of the Open line: a 32-character cut lands on a hyphen often enough that `q4-report-` was a slug this repo could produce, legal in the grammar and reading like a bug. The compatibility statement, said out loud because nothing else will say it: two registries built on either side of this change disagree about a fetched page's slug, and an artifact's tokens are not portable between them. Existing registries are unaffected, and that is not luck — `_find_document` matches a re-fetch on the URL, never the slug, so re-ingesting a page keeps the slug it already has. `--slug` remains the way to overrule all of this and is still the advice for a fetched page worth citing, because `a-example-index` names a site rather than nothing, which is better and is not good. |
+| 2026-08-20 | **`ingest` says which of its three outcomes happened, and how much text came out.** One line served three different events — a document created, a *new generation* of one whose bytes moved, and a no-op that wrote nothing — so an agent re-ingesting after a fix could not tell whether anything had changed, at exactly the moment it most needed to: a new generation is when citations into the previous snapshot begin resolving to a superseded one and `bind` starts calling them `drifted`. The registry already distinguished all three internally (`_find_document`, `_is_noop`, `_upsert_document`) and threw the answer away, so `ingest` now returns an `Ingested` — a `Document` subclass carrying `outcome`, the shape `SearchResults` established for `search`: Addendum A's pinned `-> Document` still holds, every `.slug` caller and every fake is untouched, and the one consumer that cares reads the extra field. Re-deriving the outcome in the CLI was the alternative and would have meant a second implementation of `_is_noop` living away from the first. The wart is named rather than papered over: a dataclass compares by class, so an `Ingested` never equals the plain `Document` the read side returns for the same row, and the two tests that compared them now compare what persists. The second half is a number: the extracted character count, beside the page count on every line. A scanned PDF with no text layer and a page behind a login both ingest cleanly, exit 0 and print a page count like any success, which is how a shell of a source gets cited — and the writing skill's instruction to notice "thin or empty" gave the agent nothing to notice it with. Under `THIN_SOURCE_CHARS` (200) a note names the likely cause by media type and what to do about it. That threshold is a heuristic, which this repo refuses elsewhere on purpose — the `html` extractor is a parse and not a readability guess, because a heuristic that changes its mind moves anchors. This one is admissible for the reason those are not: it is display only, at exit 0, and no token, anchor, status or byte of the artifact derives from it, so being wrong costs one line of advice and never a citation. `chars` counts sheets too, where `read`'s window unit is rows: this is the extraction's volume, not a window into it. Grouping follows the 2026-08-13 rule — thin sources group by cause, since one missing text layer explains every scan at once and each document's own count is already on its own line, while the failure report still lists by source. |
 
 ## Open
 
@@ -150,8 +154,6 @@ Queued and parked work lives in ROADMAP.md; this list is questions still
 awaiting a decision:
 
 - Re-bind/orphan pass on re-ingest of changed docs (chunk ordinal drift)
-- Slug assignment/collision rules (the trailing-hyphen truncation from the
-  first field trial lives here)
 - Distribution: published to PyPI as `backdraft` 0.2.0 (2026-07-28); domain
   backdraft.dev acquired; still open: skill registration, deploying site/
 - First external user (design-partner validation layer vs eval substrate vs

{backdraft-0.7.0 → backdraft-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: backdraft
-Version: 0.7.0
+Version: 0.8.0
 Summary: Drop-in provenance for factual claims: one click from any claim to the evidence behind it.
 Project-URL: Homepage, https://backdraft.dev
 Project-URL: Documentation, https://backdraft.dev/docs.html
@@ -67,11 +67,27 @@ documents: 0
 … plus where settings live and what to do next
 
 $ backdraft ingest sources/t12-summary.pdf sources/underwriting-model.xlsx
-t12-summary  t12-summary.pdf  pdf  3 pages
-underwriting-model  underwriting-model.xlsx  xlsx  2 sheets
+t12-summary  t12-summary.pdf  pdf  3 pages  8865 chars
+underwriting-model  underwriting-model.xlsx  xlsx  2 sheets  2357 chars
 note: extracted with pdf-text (the embedded text layer). Glossy or scanned PDFs extract better through a vision model: set BACKDRAFT_VLM_API_KEY in .backdraft/env.
+
+$ backdraft ingest "https://en.wikipedia.org/w/index.php?title=Franklin_County,_Ohio&oldid=1367935775" --slug franklin-county
+franklin-county  https://en.wikipedia.org/w/index.php?title=Franklin_County,_Ohio&oldid=1367935775  html  1 page  34141 chars
 ```
 
+The demo cites that page for market context. It is Wikipedia's *permanent
+link* — `?oldid=` serves one revision's bytes forever — because a citation into
+a live article reports `drifted` the moment somebody edits it, and the quoted
+sentence would no longer be the sentence there.
+
+Every ingest line ends with how much text came out, and says which of three
+things happened: a document created, a **new generation** of one whose bytes
+moved — which is when citations into the previous snapshot start reporting
+`drifted` — or `unchanged`, a no-op because re-running would reproduce what is
+already there. When almost no text came out, a note names the likely cause (a
+scan with no text layer, a page behind a login) and what to do; it is a note at
+exit 0, because a thin snapshot is still a real one.
+
 **Read.** The page arrives with a citable name over each chunk. This is the whole
 mechanism: what you can cite is exactly what you were shown.
 
@@ -150,12 +166,12 @@ Multiple citations are `;`-separated in one href.
 
 ```console
 $ backdraft bind memo.md --session s-bridgeview --check value-trace,overlap
-bound 15 claim(s), 16 citation(s) [frontwalk]
-  resolved: 15
+bound 17 claim(s), 18 citation(s) [frontwalk]
+  resolved: 17
   unresolved: 1
-  overlap: pass 11, skip 4
-  value-trace: pass 15
-  ! unresolved: bd:t12-summary:p4.c1:1a2b — replacement reserve of $250 per unit per year @2900
+  overlap: pass 13, skip 4
+  value-trace: pass 17
+  ! unresolved: bd:t12-summary:p4.c1:1a2b — replacement reserve of $250 per unit per year @3629
 wrote .backdraft/records/memo.backdraft.json
 
 $ echo $?
@@ -244,18 +260,23 @@ address.
 
 ```console
 $ backdraft ingest https://example.com/reports/q4-2025
-q4-2025  q4-2025.html  html  1 page
+q4-2025  https://example.com/reports/q4-2025  html  1 page  18402 chars
 
 $ backdraft ls
-q4-2025	q4-2025.html	html	1 page	https://example.com/reports/q4-2025
+q4-2025	https://example.com/reports/q4-2025	html	1 page
 ```
 
+Every surface names a fetched source by its page. The filename you never see —
+`q4-2025.html` — is the temporary file the fetch staged the bytes in, and it
+exists on nobody's disk, so `ingest`, `ls`, `backdraft read` and the artifact
+all show the URL in its place rather than beside it: two names for one thing
+would let the invented one look authoritative.
+
 The origin travels into the artifact too. A claim citing a fetched page shows
-the URL as a link on its receipt, with the date the bytes were taken, and the
-source list names the page rather than the staged filename. That pointer is the
-half of citing a web page a frozen receipt cannot answer on its own: the
-snippet says what the page said, the link is how a reader asks whether it still
-says it.
+the URL as a link on its receipt, with the date the bytes were taken. That
+pointer is the half of citing a web page a frozen receipt cannot answer on its
+own: the snippet says what the page said, the link is how a reader asks whether
+it still says it.
 
 What it does not do, stated rather than worked around: JavaScript-rendered
 pages give you whatever the server sends to a plain GET, pages behind a login

{backdraft-0.7.0 → backdraft-0.8.0}/README.md

@@ -42,11 +42,27 @@ documents: 0
 … plus where settings live and what to do next
 
 $ backdraft ingest sources/t12-summary.pdf sources/underwriting-model.xlsx
-t12-summary  t12-summary.pdf  pdf  3 pages
-underwriting-model  underwriting-model.xlsx  xlsx  2 sheets
+t12-summary  t12-summary.pdf  pdf  3 pages  8865 chars
+underwriting-model  underwriting-model.xlsx  xlsx  2 sheets  2357 chars
 note: extracted with pdf-text (the embedded text layer). Glossy or scanned PDFs extract better through a vision model: set BACKDRAFT_VLM_API_KEY in .backdraft/env.
+
+$ backdraft ingest "https://en.wikipedia.org/w/index.php?title=Franklin_County,_Ohio&oldid=1367935775" --slug franklin-county
+franklin-county  https://en.wikipedia.org/w/index.php?title=Franklin_County,_Ohio&oldid=1367935775  html  1 page  34141 chars
 ```
 
+The demo cites that page for market context. It is Wikipedia's *permanent
+link* — `?oldid=` serves one revision's bytes forever — because a citation into
+a live article reports `drifted` the moment somebody edits it, and the quoted
+sentence would no longer be the sentence there.
+
+Every ingest line ends with how much text came out, and says which of three
+things happened: a document created, a **new generation** of one whose bytes
+moved — which is when citations into the previous snapshot start reporting
+`drifted` — or `unchanged`, a no-op because re-running would reproduce what is
+already there. When almost no text came out, a note names the likely cause (a
+scan with no text layer, a page behind a login) and what to do; it is a note at
+exit 0, because a thin snapshot is still a real one.
+
 **Read.** The page arrives with a citable name over each chunk. This is the whole
 mechanism: what you can cite is exactly what you were shown.
 
@@ -125,12 +141,12 @@ Multiple citations are `;`-separated in one href.
 
 ```console
 $ backdraft bind memo.md --session s-bridgeview --check value-trace,overlap
-bound 15 claim(s), 16 citation(s) [frontwalk]
-  resolved: 15
+bound 17 claim(s), 18 citation(s) [frontwalk]
+  resolved: 17
   unresolved: 1
-  overlap: pass 11, skip 4
-  value-trace: pass 15
-  ! unresolved: bd:t12-summary:p4.c1:1a2b — replacement reserve of $250 per unit per year @2900
+  overlap: pass 13, skip 4
+  value-trace: pass 17
+  ! unresolved: bd:t12-summary:p4.c1:1a2b — replacement reserve of $250 per unit per year @3629
 wrote .backdraft/records/memo.backdraft.json
 
 $ echo $?
@@ -219,18 +235,23 @@ address.
 
 ```console
 $ backdraft ingest https://example.com/reports/q4-2025
-q4-2025  q4-2025.html  html  1 page
+q4-2025  https://example.com/reports/q4-2025  html  1 page  18402 chars
 
 $ backdraft ls
-q4-2025	q4-2025.html	html	1 page	https://example.com/reports/q4-2025
+q4-2025	https://example.com/reports/q4-2025	html	1 page
 ```
 
+Every surface names a fetched source by its page. The filename you never see —
+`q4-2025.html` — is the temporary file the fetch staged the bytes in, and it
+exists on nobody's disk, so `ingest`, `ls`, `backdraft read` and the artifact
+all show the URL in its place rather than beside it: two names for one thing
+would let the invented one look authoritative.
+
 The origin travels into the artifact too. A claim citing a fetched page shows
-the URL as a link on its receipt, with the date the bytes were taken, and the
-source list names the page rather than the staged filename. That pointer is the
-half of citing a web page a frozen receipt cannot answer on its own: the
-snippet says what the page said, the link is how a reader asks whether it still
-says it.
+the URL as a link on its receipt, with the date the bytes were taken. That
+pointer is the half of citing a web page a frozen receipt cannot answer on its
+own: the snippet says what the page said, the link is how a reader asks whether
+it still says it.
 
 What it does not do, stated rather than worked around: JavaScript-rendered
 pages give you whatever the server sends to a plain GET, pages behind a login

backdraft-0.8.0/ROADMAP.md

@@ -0,0 +1,340 @@
+# Roadmap
+
+Sequenced intent. DESIGN.md records what was *decided*; this file records what
+is *queued*, so tabling something is an act of writing it down here rather than
+losing it. Items graduate by getting built and deleted.
+
+## How the queue runs
+
+The Now section is an ordered queue written for an implementing agent with no
+session context: each item carries its intent, its shape, and its acceptance
+test. The working agreement is one item at a time, top first, shipped complete
+— code, tests, docs, and a DESIGN.md decision row where the item calls for one
+— then deleted from this file in the same commit. Releases cut Friday, so an
+item that lands mid-week waits on PyPI until the release; nothing ships
+half-done to make a release. An item that turns out bigger than its sizing
+gets split here, not stretched silently.
+
+The cadence is daily and the runner is an agent: one item each morning Monday
+through Thursday, landing as a direct commit to `main` — this repo does not
+use pull requests, so a week of commits is the only review surface there is.
+
+Friday builds nothing new. It reads that week's diff instead — duplication,
+drift from the repo's own idioms, branches that landed untested, help text and
+error messages that confuse, commands the docs describe wrongly — and lands
+the fixes. Then the release, and only if something landed since the last tag:
+the version moves in both `pyproject.toml` and `.claude-plugin/plugin.json`,
+and publishing the GitHub release is what ships to PyPI. Then this section is
+refilled with five items for the week ahead, drawn against the core — the four
+verbs, the registry, the grammar, the artifact — and against where an agent
+using backdraft actually stumbles. What the maintenance pass just found is the
+best evidence available for what those five should be.
+
+## Now
+
+### 1. `backdraft verify <artifact>`: checking a receipt without the registry
+
+**Intent.** `skills/backdraft-artifact/SKILL.md` exists because there is no
+command: it walks an agent through opening a `.backdraft.json` or
+`.backdraft.html`, recomputing each `snippet_sha256`, and reporting claim
+statuses by hand. That is a deterministic check over a self-describing file,
+which is precisely the kind of thing that should not be a prose procedure an
+agent re-implements every time — and doing it by hand is how a recipient's
+check ends up weaker than the producer's. The artifact is the product; nothing in the
+tool reads one back.
+
+**Shape.** A new command in `render/cli.py` (which already owns the artifact's
+reader half through `sidecar.read`), taking the artifact or its sidecar.
+Two tiers, and the output must be explicit about which ran. Tier one needs
+nothing but the file: parse it against `kernel/artifact.py`'s format, recompute
+every `snippet_sha256` from the snippet the file carries, and confirm the
+`$format` and the claim/citation counts — this is the check a recipient with
+only the file can make, and it catches an edited artifact. Tier two runs only
+when a registry is discoverable: re-resolve each token through the same path
+`bind` uses and report the closed status set. Exit 0 when everything checked
+passed, 1 on a usage error, and — matching `bind` — 2 when something did not
+verify, so a hook can gate on it. Read-only: it opens no session and mints
+nothing, which is what distinguishes it from `read` and from `backdraft show`
+(landed 2026-08-12), whose whole point is that showing mints — an audit must not
+make its subject citable.
+
+**Acceptance.** `backdraft verify demo/memo.backdraft.html` from outside
+`demo/` reports tier one passing and says plainly that no registry was found,
+exiting 0. From inside `demo/` it also re-resolves and reports the one
+unresolved citation, exiting 2. A file with one snippet byte changed fails by
+naming the claim and the hash that did not match. A file that is not an
+artifact exits 1 saying so. `skills/backdraft-artifact/SKILL.md` is rewritten
+around the command, keeping the prose procedure only as the fallback for an
+agent that has no backdraft install.
+
+**Size.** Three days.
+
+### 2. `spec/registry.md`: the export format is normative, or it is not a format
+
+**Intent.** `backdraft export` writes `"$format": "backdraft/registry-v1"`, a
+version string that promises a specification and does not have one. `spec/`
+holds `artifact.md`, `tokens.md` and `chunking.md`, and the repo's own rule is
+that another implementation reads those and nothing else — the export is the
+only declared format outside that set. It is also the only complete, portable
+representation of a registry, so it is what a second implementation, a
+migration, or an audit would read. The gap is not theoretical: the 2026-08-05
+work added a conditional `meta` key to every exported document, a format change
+to a format with nothing to change.
+
+**Shape.** A new `spec/registry.md` in the voice of `spec/artifact.md`:
+normative prose, MUST/OPTIONAL where it means them, and written to be readable
+without the code. Cover the top-level shape, the document entry including the
+`meta` key and its `url`/`fetched_at` (OPTIONAL, present only for a fetched
+source — the same conditional the artifact spec already states, for the same
+compatibility reason), every extraction generation with `is_current`, pages
+and cells, and anchors with their receipts. State what identity is and what it
+is not, matching the concept table in `SPEC.md`. Say what a conforming reader
+must do with a key it does not recognize. Pin it the way the artifact format is
+pinned: a golden-file test over the demo registry's export, so the next
+conditional key is a test failure rather than an undocumented change.
+
+**Acceptance.** `spec/registry.md` describes every key `Registry.export`
+actually emits — check that mechanically, with a test that walks the exported
+JSON and fails on a key the spec does not name. `SPEC.md`'s file map and
+`README.md`'s spec list gain it. Exporting the demo registry and reading only
+the spec is enough to say what each field means.
+
+**Size.** Two days.
+
+### 3. `search` says when it stopped short
+
+**Intent.** `backdraft search "the"` prints `20 results for "the"` whether there
+are twenty or two hundred: `--limit` defaults to 20, and the count line reports
+the size of the *page*, not the size of the answer. Search is the discovery
+surface an agent leans on hardest — it is how a claim finds its evidence — and
+an agent that reads "20 results" reasonably concludes it has seen all of them,
+picks the best of twenty, and never learns that the sentence it actually needed
+was twenty-first. `--limit 2` says `2 results` with no hint anything was cut.
+This is the silent-cap failure the repo rejects everywhere else: `read`
+paginates and closes with `[Showing 0-464 of 3115 chars. Continue with: ...]`,
+which is exactly the shape missing here.
+
+**Shape.** `registry.store.Registry.search` and `gate/searcher.py`. The registry
+needs to know the total before it truncates — count the FTS matches and carry
+the number alongside the hits, on `SearchResults`, which is already a list
+subclass carrying `phrase_fallback` for precisely this kind of out-of-band fact
+(SPEC Addendum A pins that shape, so extending it is a spec change and Addendum
+A must say so). Then `render_search` reports honestly: an uncapped run keeps the
+line it prints today, byte for byte, because most runs are uncapped and their
+output is a contract; a capped run says how many matched and how to see the
+rest, in `read`'s continuation-hint voice and naming the real flag. Do not add
+an offset to `search` for this — `--limit` already exists, the hint can name it,
+and paging a relevance-ordered list is a different design question. Keep the
+count cheap: a `COUNT(*)` over the same FTS query, not a fetch-everything-and-
+slice, since a registry of a hundred documents is the case that matters.
+
+**Acceptance.** In a registry with more matches than the limit, `backdraft
+search <q> --limit 2` names the true total and tells the reader how to widen it;
+with fewer matches than the limit, the output is byte-identical to what it
+prints today — pin that as a test, the way the artifact's byte-identity is
+pinned. `--in <slug>` scoping counts within the scope, not across the registry.
+`site/llms.txt` and `skills/backdraft/SKILL.md` tell the agent to widen rather
+than settle when the line says it was capped.
+
+**Size.** One day.
+
+### 4. The bind report says why a verifier skipped
+
+**Intent.** A bind report prints `overlap: pass 11, skip 4` and stops. Four
+citations were not checked and the report does not say why, so a reader cannot
+tell a benign skip from a hole in the verification — and the reason exists
+already: the record's verdicts carry `"detail": "wording overlap does not apply
+to a single cell"` for every one of those four. This is the same gap the
+2026-08-13 line-item work closed one level up, where the token was printed and
+the claim it sat on was not: the record says it, nested, and the line a person
+reads does not. `DESIGN.md` holds that verifiers are evidence and never gate,
+which makes the reason *more* important, not less — a skip nobody can interpret
+is the one way a non-gating verifier can still mislead.
+
+**Shape.** `bind/cli.py`'s `_print_report`, output only; no verifier, record or
+artifact change. Skips of one method cluster on one reason, so group by reason
+the way `ingest`'s snapshot note groups (`cli.py`'s `unsnapshot` dict) rather
+than listing one line per citation the way its unread-source report does — the
+distinction is written down in the 2026-08-13 decision row and this is the
+grouping case. Keep it under the method's existing summary line and keep it one
+line per distinct reason. `fail` is not involved: a skip is not a failure, the
+exit code does not move, and a run whose verifiers all passed prints exactly
+what it prints today.
+
+**Acceptance.** Binding the demo (`bind memo.md --session s-bridgeview --check
+value-trace,overlap`) prints the four skips' reason under the `overlap` line;
+exit code is still 2 for the unresolved citation and would be 0 without it. A
+run with no skips is byte-identical to today's, pinned as a test. A method
+skipping for two different reasons prints two lines. `demo/walkthrough.md`'s
+bind blocks, `README.md`'s, and `skills/backdraft/SKILL.md`'s show the real new
+output, and the skill says a skip with a reason is not a hole to fix.
+
+**Size.** One day.
+
+### 5. `session show` says what the ledger holds
+
+**Intent.** The ledger is the mechanism the design rests on — the set of citable
+tokens is exactly the set the gate emitted — and nothing reads it back.
+`backdraft session show` prints `session default  (from default)` and stops, so
+an agent cannot answer "have I read enough to write this yet?" except by binding
+and reading the `not_shown` count, which means finding out after the draft
+exists. Worse, the default session is documented as stable across invocations
+*so reads accumulate*, which means an agent that never exported
+`BACKDRAFT_SESSION` has its `not_shown` judged against everything ever read in
+that registry by any previous run on any document — a real weakening of the
+system's strongest check, currently invisible at every surface. `skills/
+backdraft/SKILL.md` tells the agent to start a named session and calls it
+optional; nothing shows it what optional costs.
+
+**Shape.** `gate/cli.py`'s `session show` and `gate/reader.py` (the ledger is
+the gate's, and `session start` already lives there). Report what the session
+holds: how many anchors, across how many documents, and per document a slug and
+a count — read off the `shown` rows the registry already keeps, which may need
+one accessor on `Registry` beside `was_shown` (SPEC Addendum A, so state it
+there). When the session is the default one, say plainly that it accumulates
+across runs and that a named session is what makes `not_shown` mean "this
+draft's author saw it" — the same voice the `--config` and poppler notes use, a
+note at exit 0 and never a failure. An empty session says so and names
+`backdraft read` rather than printing a bare zero.
+
+**Acceptance.** After reading two pages of one document and searching another,
+`backdraft session show` names both documents with their counts and a total; on
+a fresh registry it says the session is empty and what to run. Running under
+`BACKDRAFT_SESSION=s-x` shows `s-x` and no accumulation note; running with
+nothing set shows the default and the note. `README.md`, `site/docs.html`,
+`site/llms.txt` and `skills/backdraft/SKILL.md` point at it as the way to check
+coverage before binding, and the skill's "optional" for `session start` becomes
+a stated tradeoff.
+
+**Size.** Two days.
+
+### 6. An unreadable source says what to do, not what errno it was
+
+**Intent.** `ingest` now finishes its list and names each failure (2026-08-13),
+which turned the reason string into something a calling agent actually reads —
+and the reason is Python. A directory says `cannot read adir: [Errno 21] Is a
+directory: 'adir'`; a locked file says `[Errno 13] Permission denied`; a missing
+one says `[Errno 2] No such file or directory`. Each names the source three
+times, leaks an errno an agent must not have to know, and — the real cost —
+says nothing about the fix, though the fix differs completely between the three
+and is obvious in each case. A directory means "name the files inside it, or
+glob"; the repo's own rule is that an error says what went wrong *and* what to
+do next, and this is the boundary where an unattended agent meets it most.
+
+**Shape.** Wherever the read failure is raised on the ingest path — `cli.py`'s
+`_staged`, `registry/store.py`'s ingest read, and `fetch.py` for the URL half —
+turn the `OSError` into a `BackdraftError` whose message is written for the
+caller: what the source is, why it could not be read, what to do. Map the causes
+worth distinguishing (missing, a directory, unreadable, empty, and the fetch
+side's own) and let anything unmapped keep the underlying text rather than
+guessing — an unknown cause reported plainly beats a wrong suggestion. Say the
+source once: the `!` line already leads with it, so the reason should not repeat
+it. This is *not* the queued "Ingest says what it did and what it got" item, which is about what a successful ingest says
+(no-op, new generation, character count); this is the failure half, and the two
+should not be merged.
+
+**Acceptance.** `backdraft ingest <a-directory>` names the directory once and
+says to pass the files inside it or a glob; a missing file, an unreadable file
+and an empty file each get their own reason and their own next step; none of the
+four prints `[Errno`. A URL that 404s keeps saying so. The count line and the
+closing re-run line are unchanged, and the clean-run stdout byte-identity test
+from 2026-08-13 still passes. `site/llms.txt` says the failure reasons are
+actionable so an agent surfaces them to the user rather than paraphrasing.
+
+**Size.** One day.
+
+### 7. A registry can forget a source
+
+**Intent.** Ingest is one-way. An agent working unattended over a folder — which
+is now the normal case, since `ingest` finishes its list — will ingest something
+it should not have: a scratch copy, a duplicate under two names, a file the user
+did not mean to include. Nothing removes it. It stays in `backdraft read`'s list
+forever, its anchors stay in `search` competing with the real source, and an
+agent picking evidence can cite the wrong copy without anything looking wrong.
+The only recovery is deleting `.backdraft/` and starting over, which throws away
+every other document and the whole ledger.
+
+**Shape.** Withdraw, do not delete — a new `backdraft forget <slug>` in `cli.py`
+beside `ingest`, and the storage half in `registry/store.py`. A withdrawn
+document disappears from `documents()`, from the gate's list and table of
+contents, and from `search`; its anchors keep resolving, because a token already
+written into somebody's draft or artifact must not silently become `unresolved`
+— that is the failure mode this whole system exists to prevent, and the
+2026-08-05 "provenance, never identity" row governs. So `Registry.resolve` still
+finds it and `show` and `bind` report it, with the closed status set gaining
+nothing: reuse `unresolved` and let the *reason* say withdrawn, since adding a
+status is an artifact-format change and this does not need one. Storage is a
+flag on `documents`, not a DELETE, for the same reason. Require confirmation or
+an explicit flag, since this is the one command that takes something away.
+Interacts with the queued `spec/registry.md` item: whichever lands second
+documents the flag in the export.
+
+**Acceptance.** Ingest two documents, `forget` one: `read`, `ls` and `search`
+show only the other; a token minted from the forgotten one still resolves
+through `backdraft show`, saying it was withdrawn and when; `bind` on a document
+citing it reports it and does not crash. `forget` on an unknown slug exits 1
+naming the known ones. Re-ingesting the same file after forgetting it brings it
+back as a new generation of the same document rather than a second slug — pin
+that, it is the case that decides whether the flag was the right storage.
+`README.md`, `site/docs.html`, `site/llms.txt` and `SPEC.md`'s command list gain
+it, and it earns a DESIGN.md row: withdraw-not-delete, with the tradeoff that a
+registry never actually shrinks.
+
+**Size.** Three days.
+
+## Parked
+
+Deliberately not queued, each with the reason, so picking one up starts from
+the objection rather than rediscovering it.
+
+- **Exhibits view** — the evidence-first inversion. Risk: it grows the
+  artifact and pollutes a design whose strength is restraint. Needs a
+  deliberate design pass and a real audience before any code; not a
+  weekend-feature shape.
+- **Excel region maps and range reads** — the research notes exist, but no
+  confidence the deterministic detection cascade lands well on the first
+  try. Revisit when large-sheet navigation hurts in a real corpus, with
+  test workbooks in hand.
+- **`bd:calc` derivations** — reserved in the grammar and parsed as
+  malformed-on-purpose, which is enough for now. Full support complicates
+  binding and verification for one field-trial case.
+- **Merged-cell and frozen-pane rendering** — ingest captures `merged` and
+  `frozen` in sheet meta and keeps them (cheap, and capture is the part you
+  cannot backfill). Frozen-pane rendering is cut; merged-cell rendering
+  waits for a real sheet that renders wrong without it.
+
+## Later
+
+- **Living documents** — re-ingest, re-bind, and a drift-first report: "these
+  claims cite figures that changed; cited-then vs. now." The primitives
+  (generations, `drifted`, the word-diff) all exist; the missing piece is a
+  diff-shaped report and a demo.
+- **Substrates beyond the CLI** — SDK middleware for pipelines (the exit-code
+  and `--to json` contracts are already shaped for it), and a client-side
+  drag-drop viewer page (no upload — the file never leaves the reader's
+  machine).
+- **Email (.eml)** — arrives in diligence folders alongside everything else;
+  identity questions (headers vs. body, attachments as child sources) deserve
+  a decision row before code. The web-page half of this item shipped
+  2026-08-05; `document_meta` is where an email's provenance would land.
+- **Anthropic-API provider for the VLM extractor** — Cowork's sandbox
+  egress allowlist includes api.anthropic.com but not the OpenAI-compatible
+  providers, so an Anthropic-native option would let sandboxed sessions
+  ingest at full fidelity. Needs a client abstraction the vlm module
+  currently doesn't have; the documented workaround (ingest locally, the
+  registry travels with the folder) covers it meanwhile.
+- **Distribution** — published on PyPI; the repo ships
+  `.claude-plugin/plugin.json` and `marketplace.json`, so it installs as a
+  Claude Code plugin and self-hosts its marketplace, and the AGENTS.md
+  snippet is published in the README and site docs. The community-directory
+  submission went in 2026-07-29 and is awaiting review — approval is what
+  surfaces the skills in Cowork's directory.
+
+## Someday
+
+- **Hosted team features** — upload, access control, shared registries.
+  Deliberately parked: the product's trust story is the self-contained file,
+  and a hosted viewer inverts it. Revisit only as a distinct teams product.
+- **Entail at scale** — the model-judge verifier is wired (`[entail]` extra)
+  but has never been field-calibrated; needs a corpus and a rubric before it
+  is recommended anywhere.

{backdraft-0.7.0 → backdraft-0.8.0}/SPEC.md

@@ -241,7 +241,7 @@ Registered in a plain dict; `ingest --extractor auto` picks the first `can_handl
 ## Gate (gate/)
 
 CLI-facing behavior contract:
-- `read` with no args → document list (slug, filename, media, pages). `read <slug>` → TOC (page/sheet, name, summary-or-first-120-chars). `read <slug> p3` / `p3-5` / `rent-roll` → token-marked content.
+- `read` with no args → document list (slug, name, media, pages), where the name is the filename or — for a source fetched from the web — the origin URL standing in its place (`gate.source_name`, shared with the CLI's `ingest` and `ls`). `read <slug>` → TOC (page/sheet, name, summary-or-first-120-chars). `read <slug> p3` / `p3-5` / `rent-roll` → token-marked content.
 - PDF pages render as chunks: `[bd:slug:p3.c1:a7f3]` on its own line above each chunk's text. Sheets render the markdown table as-is with a header line carrying the page token; cell tokens are NOT inlined per-cell (the `[B10]` refs are already in-band; bind composes `bd:slug:sheet!B10:hash` from the registry).
 - `search "<query>"` → FTS5 over anchors; each result: token, slug, page, snippet. Results are minted (ledger-recorded) — a searched snippet is citable without a page read.
 - `show <token>...` → the inverse of minting: per token, its bind status, its slug and locator, and the verbatim snippet, in argument order. `resolved` prints the snippet; `drifted` prints both the cited snippet and what stands at the locator now (and mints that anchor too, since it is the one worth citing); `unresolved` says whether the slug or the locator is what named nothing; `malformed` carries the kernel's reason and the grammar. Statuses and resolution are bind's, reached through `kernel.claims.parse_citation` and `Registry.resolve` rather than a second reading of what a token means; `not_shown` cannot occur, because showing mints. Exit 1 if any token was `unresolved` or `malformed`, with the block still on stdout.
@@ -342,6 +342,10 @@ class Registry:
     def ingest(self, path: Path, *, extractor: str | None = None,
                slug: str | None = None, config: dict | None = None,
                url: str | None = None, fetched_at: str | None = None) -> Document: ...
+        # Concretely an `Ingested` (a `Document` subclass) carrying `outcome: str` — one of
+        # CREATED, GENERATION, UNCHANGED, the three things ingest can do. Consumers may
+        # ignore it; the CLI prints it, since only the registry can tell a no-op from a
+        # new generation and re-deriving it outside means re-implementing `_is_noop`.
         # runs extractor → new extraction generation, pages, eager anchors, FTS rows.
         # Token carry-over per re-ingest semantics; hash-collision extension steps TOKEN_HASH_LENGTHS.
         # Sheet names sanitized to the sheetref charset here, at ingest.
@@ -390,8 +394,8 @@ class Registry:
     def export_json(self) -> dict: ...
 ```
 
-`Resolution` and `SearchHit` are small frozen dataclasses in `registry/store.py`; `SearchResults` is the `list[SearchHit]` subclass described above.
+`Resolution` and `SearchHit` are small frozen dataclasses in `registry/store.py`; `SearchResults` is the `list[SearchHit]` subclass described above, and `Ingested` the `Document` subclass. Both subclasses exist so a pinned return type can carry an out-of-band fact without every consumer and every fake changing shape.
 
 ## Addendum B — CLI assembly
 
-Top-level `cli.py` (W1) owns: typer app, registry discovery (nearest `.backdraft/` walking up from cwd; `BACKDRAFT_HOME` override), session resolution (`--session` flag > `BACKDRAFT_SESSION` env > default session), and the `init` / `ingest` / `ls` / `export` / `snapshot-pages` commands. `ingest` runs its whole source list: a `BackdraftError` from one source is caught, recorded against that source, and the loop continues, so a folder ingest is never half-attempted. Every failure is printed at the end as one `!` line naming the source and the reason, under a count of what landed, and the command exits 1 — the guard in `cli_context` stays the only place a `BackdraftError` becomes an exit code, since nothing caught per source is re-raised. After each ingest, `cli.py` captures page snapshots for any PDF whose extraction carries none — the text-layer path, since `vlm` stores its own — via `extract/snapshots.py` (poppler through pdf2image, `snapshot-pages`' internals). It sits outside the extractor so `pdf-text` stays deterministic regardless of whether poppler is installed, and it is best-effort: a `SnapshotError` leaves ingest at exit 0 with a one-line note naming `snapshot-pages` as the backfill. `cli.py` also owns the network for the same reason: an `ingest` argument with a scheme is fetched through `fetch.py` (stdlib `urllib`, http and https only, redirects followed, 32 MiB cap) and staged in a temporary file named for the content type the server declared — which is what selects the extractor — so extractors stay pure functions of bytes and the registry opens no sockets. It mounts sub-apps `gate/cli.py` (W2: `read`, `search`, `session`), `bind/cli.py` (W3: `bind`), `render/cli.py` (W4: `render`) — each exposes `app = typer.Typer()`; the top level mounts each inside a try/except ImportError so partial merges still run. The `backdraft` console script is declared by W1.
+Top-level `cli.py` (W1) owns: typer app, registry discovery (nearest `.backdraft/` walking up from cwd; `BACKDRAFT_HOME` override), session resolution (`--session` flag > `BACKDRAFT_SESSION` env > default session), and the `init` / `ingest` / `ls` / `export` / `snapshot-pages` commands. `ingest` runs its whole source list: a `BackdraftError` from one source is caught, recorded against that source, and the loop continues, so a folder ingest is never half-attempted. Every failure is printed at the end as one `!` line naming the source and the reason, under a count of what landed, and the command exits 1 — the guard in `cli_context` stays the only place a `BackdraftError` becomes an exit code, since nothing caught per source is re-raised. Each source that landed prints its extracted character count and, where it is not a fresh document, which of `Ingested.outcome`'s other two values it was: `unchanged` for a no-op, `new generation` for bytes that moved, the latter followed by one grouped note naming the documents and the `drifted` consequence. A document whose extraction came in under `cli.THIN_SOURCE_CHARS` adds a note grouped by cause (media-type-specific: a PDF's missing text layer, a page behind a login, an image-only deck; anything unmapped says plainly that it has nothing specific to offer), at exit 0 — a thin snapshot is a real snapshot, and the note is display only. After each ingest, `cli.py` captures page snapshots for any PDF whose extraction carries none — the text-layer path, since `vlm` stores its own — via `extract/snapshots.py` (poppler through pdf2image, `snapshot-pages`' internals). It sits outside the extractor so `pdf-text` stays deterministic regardless of whether poppler is installed, and it is best-effort: a `SnapshotError` leaves ingest at exit 0 with a one-line note naming `snapshot-pages` as the backfill. `cli.py` also owns the network for the same reason: an `ingest` argument with a scheme is fetched through `fetch.py` (stdlib `urllib`, http and https only, redirects followed, 32 MiB cap) and staged in a temporary file named for the content type the server declared — which is what selects the extractor — so extractors stay pure functions of bytes and the registry opens no sockets. It mounts sub-apps `gate/cli.py` (W2: `read`, `search`, `session`), `bind/cli.py` (W3: `bind`), `render/cli.py` (W4: `render`) — each exposes `app = typer.Typer()`; the top level mounts each inside a try/except ImportError so partial merges still run. The `backdraft` console script is declared by W1.