backdraft 0.8.0__tar.gz → 0.10.0__tar.gz

{backdraft-0.8.0 → backdraft-0.10.0}/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "backdraft",
   "displayName": "Backdraft",
-  "version": "0.8.0",
+  "version": "0.10.0",
   "description": "Drop-in provenance for factual claims: one click from any claim to the evidence behind it. Ships the backdraft writing, backfill, and artifact-reading skills; the CLI installs separately (uv tool install backdraft).",
   "author": {
     "name": "Spencer Braun",

{backdraft-0.8.0 → backdraft-0.10.0}/.gitignore

@@ -23,3 +23,6 @@ uv.lock.bak
 # Claude Code session state and agent worktrees
 .claude/
 .private/
+
+# coverage runs (uv run --with pytest-cov)
+.coverage

{backdraft-0.8.0 → backdraft-0.10.0}/DESIGN.md

@@ -29,7 +29,7 @@ Four verbs:
 - **`ingest`** — deterministically anchor source documents (PDF/xlsx/md/…) into the registry: extract, locate, snapshot snippets, hash.
 - **`read` / `search`** — the gate. Emit token-marked context for a model. Search results are themselves mintable evidence (search that returns nothing citable forces a page read purely to obtain an anchor).
 - **`bind`** — postprocess a written document: resolve tokens, run enabled verification switches, backfill surface forms (doc names, pages, quotes), emit sidecar claims file + binding report.
-- **`render`** — produce the clickable artifact.
+- **`render`** — produce the clickable artifact. Its reader half is `verify`: an artifact carries the checks it wants run in its own legend, so the tool that writes them ships the command that runs them.
 
 ### 3. Substrates (sequenced)
 1. **Skill + CLI, together** (first release). The CLI is the system; the skill is ~a page instructing an agent to use it. Ships both the agent-native wedge and the zero-integration backfill demo.
@@ -144,9 +144,17 @@ Self-contained HTML (plus the machine-readable sidecar): **quotes + receipts + e
 | 2026-08-12 | **`show` is a gate verb, not a lookup tool.** `backdraft show <token>` answers what a token says — the inverse of minting, for a token that arrived out of an artifact, a draft, or somebody's message, which nothing in the tool could answer before without rebuilding the slug and selector by hand and re-reading a whole page. It therefore *mints what it shows*, exactly as `read` and `search` do: an agent that has seen a snippet through the gate has genuinely read it, so recording it is honest rather than a loophole, and `not_shown` stays true by construction — a status `show` can never produce. That makes a token laundering into a citable one by being looked at, which is the intended behavior and the reason this is in `gate/` and not somewhere neutral. Resolution is bind's path rather than a second reading of what a token means: `kernel.claims.parse_citation` decides `malformed` (so the reserved `bd:calc(...)` form lands identically in both) and `Registry.resolve` decides the rest, which is what makes a status printed here the status bind would print. Two smaller calls. `drifted` prints both snippets *and* mints the anchor standing at the locator now, since that anchor's token is the one worth citing and printing it unrecorded would be the under-minting the gate must never do. And drift does **not** fail the exit code: exit 1 covers `unresolved` and `malformed`, the two statuses where there was nothing to show, because whether a changed source still supports a sentence is bind's question about a document, not a lookup's about a token. Tradeoff accepted: auditing someone else's artifact against a registry now writes ledger rows into the auditor's session, so `skills/backdraft-artifact/SKILL.md` says so and keeps that check reported separately from the artifact's self-checks. |
 | 2026-08-13 | **`ingest` finishes its list; an unreadable source is data, not the end of the run.** `backdraft ingest a.md broken.pdf c.md` used to ingest `a.md`, raise on `broken.pdf`, and never attempt `c.md` — leaving a half-built registry, a message naming only the first failure, and no way to tell which sources landed from which were never tried, so ingesting a folder meant re-running and inferring the state. The loop now catches `BackdraftError` per source and records `(source, reason)`, so every source is attempted, and prints one `!` line per failure under a count (`2 of 3 sources ingested; 1 failed`) whose two numbers add to the whole list — which is precisely the fact the old behavior could not state. This is principle 4 ("failures are data, never warn-and-drop") applied at the boundary it had not reached, and the same shape the page-snapshot note already uses one level down. Grouping differs deliberately: the snapshot note groups by *reason* because one missing poppler explains every document at once, while this lists by *source*, because the source is what the caller has to go fix. The exit code is unchanged at 1 and `guard` remains the only place a `BackdraftError` becomes one — nothing caught per source is re-raised, and the report exits through `fail` like any other error line, on stderr, so a clean run's stdout is byte-identical to before and that is pinned by a test. Tradeoff accepted: a run that fails partway now leaves *more* in the registry than it used to, so "the command failed" no longer implies "nothing happened". The closing line of the report answers that directly — re-running the same list is safe, because a source already ingested with unchanged bytes, extractor and config is a no-op — which is cheaper for a caller than reconstructing the failed half by diffing `ls` against its own arguments. |
 | 2026-08-17 | **The demo cites a web page at a fixed revision, so the capability is visible and stays `resolved`.** URL sources shipped whole — fetch, snapshot, `url`/`fetched_at`, the artifact's live dated link — and none of it appeared at backdraft.dev/demo.html, because both demo sources were local files and the byte-identity test that proved the artifact change safe was the same fact that kept it invisible. The demo gains a third source: the Wikipedia article for Franklin County, Ohio — the county the fictional property sits in and the one its tax appeal is already pending before — cited for two market figures a T12 cannot supply, the county's population and its renter-occupied share. It is ingested at Wikipedia's **permanent link** (`?oldid=<rev>`) rather than the bare article URL. What that gives up is named: the demo no longer shows a live page changing under a citation, which is the better story and belongs in a `drifted` walkthrough rather than the shop window. What it buys is that the front page does not permanently display `drifted` within days of the next edit — a demo that always shows drift teaches that the tool is broken rather than that the source moved — and an immutable revision plus the rendered "as of" date is what a careful analyst cites anyway. The same reasoning is now advice rather than a one-off: the docs, `llms.txt` and the writing skill tell an agent to prefer a fixed-revision address (`?oldid=`, a DOI, an archived snapshot) and to pass `--slug` with it. `--slug` is load-bearing here, not decoration: a permanent link is served from `/w/index.php`, so the default stem would be `index` and every token in the memo would carry it. That is the queued slug-assignment item's problem and is fixed at the fetch layer there, not worked around here. One renderer change came with it. `source_title` took a source's display name from its filename's stem, which for a fetched page is the staging name `fetch.filename_for` invented — so the article titled itself "Index". Where a document carries a URL the slug is the name instead, which is the 2026-08-06 row's rule ("showing both would let the fictional one look authoritative") applied to the one surface that had kept the invented name. It changes nothing for a page ingested without `--slug`, since the slug is derived from that same stem; it defers to whoever named the document when someone did. |
-| 2026-08-18 | **One rule names a source, and every surface that names one uses it.** A fetched page's `filename` is the temporary file `fetch.filename_for` staged the bytes in — `index.html` for a permanent link, a name on nobody's disk — and the 2026-08-06 row settled what to do about it ("showing both would give a reader two names for one thing and let the fictional one look authoritative") for the artifact alone. The four CLI surfaces that name a document kept the fiction: `ingest`'s line, the gate's document list, its table-of-contents headline, and `ls` — which showed *both*, filename in column two and URL in a fifth field, the exact shape that row rejected. `gate.source_name` is now the single owner, exported the way `gate.unit` is and imported downward by `cli.py`, so a writing agent learns a source's provenance from the surface it already lives on instead of being sent to `ls` by the skill. Display only: the slug, the token and everything a citation resolves through are untouched, which is the 2026-08-05 "provenance, never identity" rule. Two consequences named rather than discovered. `ls` loses a field — a fetched row is four columns, not five — so anything parsing column five stops seeing one; that is the point, since column two was a lie and one true name beats two names of which one is fictional. And in the gate's aligned list a URL *overflows* the name column rather than widening it: the column is sized on filenames alone, so one 80-character address pushes its own row's later columns right instead of pushing every row's past column 100. The cost is a ragged right on fetched rows, accepted because the file rows a mostly-file registry is made of stay byte-identical to what they printed before — pinned by a test, the way the artifact's byte-identity is. `bind`'s unmatched-claim report still names a source by filename: the fix cannot be shared into it without a sideways `bind` → `gate` import, which SPEC § Dependency rule forbids, and duplicating the rule there is the reinvention this row exists to end. |
+| 2026-08-18 | **One rule names a source, and every surface that names one uses it.** A fetched page's `filename` is the temporary file `fetch.filename_for` staged the bytes in — `index.html` for a permanent link, a name on nobody's disk — and the 2026-08-06 row settled what to do about it ("showing both would give a reader two names for one thing and let the fictional one look authoritative") for the artifact alone. The four CLI surfaces that name a document kept the fiction: `ingest`'s line, the gate's document list, its table-of-contents headline, and `ls` — which showed *both*, filename in column two and URL in a fifth field, the exact shape that row rejected. `gate.source_name` is now the single owner, exported the way `gate.unit` is and imported downward by `cli.py`, so a writing agent learns a source's provenance from the surface it already lives on instead of being sent to `ls` by the skill. Display only: the slug, the token and everything a citation resolves through are untouched, which is the 2026-08-05 "provenance, never identity" rule. Two consequences named rather than discovered. `ls` loses a field — a fetched row is four columns, not five — so anything parsing column five stops seeing one; that is the point, since column two was a lie and one true name beats two names of which one is fictional. And in the gate's aligned list a URL *overflows* the name column rather than widening it: the column is sized on filenames alone, so one 80-character address pushes its own row's later columns right instead of pushing every row's past column 100. The cost is a ragged right on fetched rows, accepted because the file rows a mostly-file registry is made of stay byte-identical to what they printed before — pinned by a test, the way the artifact's byte-identity is. `bind`'s unmatched-claim report still names a source by filename: the fix cannot be shared into it without a sideways `bind` → `gate` import, which SPEC § Dependency rule forbids, and duplicating the rule there is the reinvention this row exists to end. **Corrected 2026-09-03:** the exception was wrong twice over — it was larger than stated, covering bind's generated `## References` section as well as its unmatched proposals, and it was avoidable, because the rule is a pure function of a `Document` and could move down to the kernel where both packages already import from. `source_name` lives in `kernel.model`; there is no exception. |
 | 2026-08-19 | **A slug is assigned by a rule, and the rule is written down.** The Open list carried "slug assignment/collision rules" from the first field trial, and URL sources sharpened it into a live failure: `fetch.filename_for` took the stem from a URL's last path segment, so `https://a.example/reports/2025/index.html` and `https://b.example/docs/index.html` both staged as `index.html` and became `index` and `index-2` — two handles that say which is which to nobody, carried by every token in every document written against them, permanently, since a slug is stable once assigned. The rule now has two clauses and they are stated together for files and URLs. A **file** is named by its filename's stem, kebabed and cut to 32 characters, which is unchanged and was never the problem. A **URL** is named by its last path segment, *except* when that segment names the site's plumbing rather than its page — a directory index or generic handler (`_GENERIC_STEMS`: `index`, `view`, `page`, `show`, and a dozen more), an all-digit id, or no segment at all — in which case the host goes in front of it, minus `www.` and minus the port, so the two pages above become `a-example-index` and `b-example-index`. The common case is deliberately untouched: `https://example.com/reports/q4-2025` is still `q4-2025`, because a rule that made every good name uglier to fix the bad ones would be the wrong trade. `_dedupe`'s `-2` stays, demoted from the normal outcome to the backstop it was meant to be — the host is a *better* stem, not a unique one, and two `index.html` pages on one site still collide. Truncation is trimmed on both cuts now (`slug_for` and `_dedupe`), which is the other half of the Open line: a 32-character cut lands on a hyphen often enough that `q4-report-` was a slug this repo could produce, legal in the grammar and reading like a bug. The compatibility statement, said out loud because nothing else will say it: two registries built on either side of this change disagree about a fetched page's slug, and an artifact's tokens are not portable between them. Existing registries are unaffected, and that is not luck — `_find_document` matches a re-fetch on the URL, never the slug, so re-ingesting a page keeps the slug it already has. `--slug` remains the way to overrule all of this and is still the advice for a fetched page worth citing, because `a-example-index` names a site rather than nothing, which is better and is not good. |
 | 2026-08-20 | **`ingest` says which of its three outcomes happened, and how much text came out.** One line served three different events — a document created, a *new generation* of one whose bytes moved, and a no-op that wrote nothing — so an agent re-ingesting after a fix could not tell whether anything had changed, at exactly the moment it most needed to: a new generation is when citations into the previous snapshot begin resolving to a superseded one and `bind` starts calling them `drifted`. The registry already distinguished all three internally (`_find_document`, `_is_noop`, `_upsert_document`) and threw the answer away, so `ingest` now returns an `Ingested` — a `Document` subclass carrying `outcome`, the shape `SearchResults` established for `search`: Addendum A's pinned `-> Document` still holds, every `.slug` caller and every fake is untouched, and the one consumer that cares reads the extra field. Re-deriving the outcome in the CLI was the alternative and would have meant a second implementation of `_is_noop` living away from the first. The wart is named rather than papered over: a dataclass compares by class, so an `Ingested` never equals the plain `Document` the read side returns for the same row, and the two tests that compared them now compare what persists. The second half is a number: the extracted character count, beside the page count on every line. A scanned PDF with no text layer and a page behind a login both ingest cleanly, exit 0 and print a page count like any success, which is how a shell of a source gets cited — and the writing skill's instruction to notice "thin or empty" gave the agent nothing to notice it with. Under `THIN_SOURCE_CHARS` (200) a note names the likely cause by media type and what to do about it. That threshold is a heuristic, which this repo refuses elsewhere on purpose — the `html` extractor is a parse and not a readability guess, because a heuristic that changes its mind moves anchors. This one is admissible for the reason those are not: it is display only, at exit 0, and no token, anchor, status or byte of the artifact derives from it, so being wrong costs one line of advice and never a citation. `chars` counts sheets too, where `read`'s window unit is rows: this is the extraction's volume, not a window into it. Grouping follows the 2026-08-13 rule — thin sources group by cause, since one missing text layer explains every scan at once and each document's own count is already on its own line, while the failure report still lists by source. |
+| 2026-08-24 | **`backdraft verify` — the recipient's check, and the drift bug it found in the spec.** The artifact's whole promise is that someone holding only the file can check it, and until now the instructions for doing so were prose: `skills/backdraft-artifact/SKILL.md` walked an agent through recomputing hashes by hand, which is a deterministic procedure re-implemented per reading — the way a recipient's audit ends up weaker than the producer's. It is now a command, in `render/cli.py` because that module already holds the format's reader half (`render.sidecar`), taking either the `.backdraft.html` or the `.backdraft.json`. Two tiers, and the output names which ran: the record against itself, which needs nothing but the file, and — only where a `.backdraft/` is discoverable — every token re-resolved against the sources. Discovery is from **cwd**, not from the artifact's own directory as `bind` does it from the document's: an artifact is a file people forward, so the folder it landed in is evidence about the recipient's machine and not about which registry produced it. Exit 2 joins `bind`'s, which was documented as "`bind`'s alone"; the shared meaning is "the run completed and the document is not clean", which is exactly what a Stop hook gates on, so a hook written for one now catches the other — the gate's `show` stays at 1, because a lookup answering "that token names nothing" is not a document failing. Read-only and minting nothing, which is the line between it and `show`: showing is minting, and an audit that made its own subject citable would manufacture the evidence it was sent to inspect. A record faithfully carrying an `unresolved` citation still passes tier one — a kept failure is the record working, so the same bytes exit 0 in an inbox and 2 in the project, answering different questions. Two consolidations came with it rather than a third copy of each: `registry.citation_for` is the token-to-status walk `bind` and `verify` both need and the dependency rule forbids either importing from the other, so it sits beside `current_at` for the same reason, with `not_shown` deliberately left out as a fact about a session the caller holds; and `cli_context.claim_words` is the line-item shape, since two commands reporting on the same claims must not print two layouts. The real find was in the spec. `spec/artifact.md` § Checking an artifact and the legend's `verify_this_record` both said the token's hash must prefix `anchor.snippet_sha256` — false for every `drifted` citation by construction, since the token names `drifted_from` (what the author cited) while `anchor` carries what stands there now, and the two differing *is* what drift means. Implemented as written, `verify` would have called every drifted artifact forged: the one status the format exists to carry honestly. Both statements are corrected, which moves the legend and therefore every artifact's bytes — goldens and the demo regenerated — and is worth the churn, because a known-wrong normative instruction shipped inside every artifact is a worse cost than a diff. |
+| 2026-08-24 | **Math is rendered, not forbidden, and MathML is why that is affordable.** An agent writing a memo about coverage writes `$\mathrm{DSCR} = \frac{NOI}{D}$`, and the renderer handed the reader raw TeX — or worse, corrupted TeX, since `$x_1$ and $x_2$` paired its subscripts into emphasis and `\(a^2\)` lost its delimiters to the backslash-escape pass. The alternative on the table was to forbid math in the writing skill, and it was rejected twice over: this product's documents are exactly the finance and science memos that contain formulas, and a skill instruction cannot be enforced against an agent that did not read it carefully, so the corruption would have arrived anyway. Shipping *unrendered* TeX was rejected too — nobody wants to read `\frac{NOI}{D}`, and a document a reader cannot read is not a lesser artifact but a broken one. The target is MathML via `latex2mathml`, behind a `[math]` extra: it is static markup the browser lays out itself, so rules 1 and 2 of § The HTML artifact hold *by construction* rather than by care — no script, no font file, no request — where inlining KaTeX would have cost roughly a megabyte of fonts as `data:` URIs on every artifact, math or not. The tradeoff, stated because it is the one place the format's fixed-at-render promise is soft: MathML leans on whatever math font the reader's system has, so two readers of one artifact may see different glyph quality. That is a smaller and more honest cost than the alternative, and it is bounded — the markup, the spacing, and every character are identical; only the typeface is not. Dollars were the real hazard and the guard is Pandoc's: a memo saying `$250 per unit` and `$1.2M` twice in a paragraph would pair them into nonsense under a naive rule, so an opening `$` must be followed by a non-space and a closing `$` must be preceded by one and not followed by a digit — a currency amount is preceded by a space and therefore can never close a span. Code is skipped rather than scanned, so a shell snippet's `$PATH` and `$HOME` are safe, and the demo regenerates with zero math elements, which is the check that matters. Math is opt-in per call and **off for evidence**: a receipt's snippet must read as it was extracted, not as this renderer would prefer it. Conversion failure is data, at two different reliabilities — `latex2mathml` raises for some malformed input, which surfaces as a wavy mark at the formula and a note giving the reason, and silently degrades for the rest (`\notacommand{x}` becomes an identifier), which cannot be caught and is recorded here as a known limit rather than papered over. |
+| 2026-08-25 | **The registry export is a specified format, or it is not a format.** `backdraft export` has written `"$format": "backdraft/registry-v1"` since v0 — a version string promising a specification that did not exist — while being the only complete, portable form a registry has, and therefore what a second implementation, a migration or an audit actually reads. The gap stopped being theoretical when URL sources added a conditional `meta` key to every exported document: a change to a format with nothing to change against. So spec/registry.md joins tokens, chunking and artifact as normative prose, and the pin is the spec itself rather than a golden file beside it — tests/test_spec_registry.py parses the spec's key tables, exports a registry built to exercise every branch, and fails in both directions, so an undocumented key and a documented-but-absent one are equally a test failure. Two boundaries were drawn rather than blurred. `bindings[].report` stays spec/artifact.md's, quoted opaquely here, because one format with two specifications is one too many and they would drift. And the export is content, not a backup: page images, sheet styling and the FTS index are deliberately outside it, so an export round-trips every citation and every receipt and does not round-trip a database — stated in the spec as a limit rather than left for a reader to discover by restoring one. |
+| 2026-08-26 | **`search` reports the size of the answer, not the size of the page.** `--limit` defaults to 20 and the count line said `20 results` whether twenty matched or two hundred did, which is the silent cap this repo rejects everywhere else — `read` closes with `[Showing 0-464 of 3115 chars. Continue with: …]` for exactly this reason. Search is the surface an agent leans on hardest, since it is how a claim finds its evidence, and relevance order is not the same as the sentence the writer needed: a reader told it has twenty results picks the best of twenty and never learns the twenty-first existed. So `SearchResults` gains `total` beside `phrase_fallback` — the same shape for the same reason, an out-of-band fact riding on Addendum A's pinned `list[SearchHit]` so every consumer and every fake is untouched — and a capped run renders `2 of 56 results` plus a closing line naming the exact widening command, shell-quoted, because a hint carrying `$` or a space that has to be repaired before it runs is not a hint. The count is a `COUNT(*)` over the *same* joins the fetch runs, shared through `_match_sql` rather than written twice: a count that joined differently would report a total the caller can never reach. It is asked for only when the page came back full, since a page the limit did not fill is already the whole answer — so the common uncapped run still runs one query, and the output it prints is byte-identical to what it printed before, pinned as a test the way the artifact's bytes are. Two things were deliberately not built. No `--offset`: paging a relevance-ordered list is a different design question, and `--limit` already exists for the hint to name. And the total is not carried into anything durable — no token, anchor, receipt or artifact byte moves, because how many other sentences also matched is a fact about a query, not about a citation. |
+| 2026-08-31 | **`session show` reads the ledger back, and says what the default session costs.** The ledger is the mechanism the whole design rests on — the set of citable tokens is exactly the set the gate emitted — and nothing read it. `session show` printed the id and stopped, so the one question an agent has before writing, *have I read enough to write this yet?*, was answerable only by binding a draft and counting `not_shown`: finding out after the draft exists, which is the wrong end of the loop. It now prints a total and one line per document, off a new `Registry.shown_by_document` beside `was_shown` (Addendum A) — `was_shown` answers one token at a time, which is what `bind` needs and what leaves nobody able to ask what a session covers. The count is by **distinct token, not ledger row**, for the reason `was_shown` matches by token: a span that survived a re-ingest is a second anchor row with the same name, and the writer saw one thing, not two — a session would otherwise appear to have read twice as much the moment a source was re-ingested. A document nothing was shown from is absent rather than zero, since `documents()` already answers what exists. The larger find is the one the command now states out loud. `DEFAULT_SESSION` is documented as stable across invocations *so reads accumulate*, which is a convenience for one agent in one directory and a real weakening of the system's strongest check for everybody else: an agent that never exported `BACKDRAFT_SESSION` has its `not_shown` judged against everything ever read in that registry by any previous run on any document, so a citation it invented can come back `resolved` because some earlier run happened to read that page. That was true before this item and visible at no surface — this repo's own demo registry turned out to carry a default session fuller than the named one the walkthrough uses, accumulated by development runs nothing in the walkthrough performs. The fix is a note rather than a behaviour change — at exit 0, in the shape `ingest`'s poppler note uses, naming `session start` — because making the default session ephemeral would break the case it exists for, and refusing to mint without one would put a setup step in front of `backdraft read`. The skill's "optional" for `session start` becomes a stated tradeoff for the same reason: the cost is now written down where the agent making the choice reads it. Keyed on the session *id*, not on which rule supplied it, so an explicit `--session default` is warned about too — the note is about the shared ledger, not about how you got there. The summary mints nothing: counting what was shown is not being shown it, and a coverage check that minted would answer its own question. The note's wording lives in `gate/reader.py` with the block it closes and takes the environment variable's name from the CLI, which owns it — `cli_context` imports typer and the gate's library half may not. |
+| 2026-09-01 | **An unreadable source says what to do about it, and an empty one is a failure.** `ingest` finishing its list (2026-08-13) turned the reason string into something a calling agent actually reads, and the reason was Python: `cannot read adir: [Errno 21] Is a directory: 'adir'` named the source three times, leaked an errno nobody should have to know, and said nothing about a fix that differs completely between the causes and is obvious in each one. The read now goes through `store._read_source`, which maps the four causes worth distinguishing — a directory (name the files inside it, or pass a glob), a path with nothing at it (check the spelling), a file the process may not open (fix the permissions, or ingest a copy), and zero bytes — and `fetch.py`'s messages follow the same rule for the URL half. Anything outside those four keeps the operating system's own `strerror` rather than being guessed at: an unknown cause reported plainly beats a wrong next step, and that is the one place an errno-free message loses information, which is the trade taken. The reason never names the source, because every caller already leads with it — which means the exception read on its own, by a library caller rather than through `ingest`'s `!` line, is now less self-contained than it was; the CLI is the only caller and the report is the surface that matters. Empty is the substantive change rather than a rewording: zero bytes read cleanly, so an empty file used to become a document — listed by `ls`, offered by `read`, competing in `search`, citable in name only — and a fetched page whose server sent an empty body did the same, which is the shape a JavaScript-rendered page takes here. That is not the thin-source case the 2026-08-20 row admitted a heuristic for: thin is a judgement about how much text came out of real bytes and stays display-only at exit 0, while zero bytes is a fact and there is no snapshot to make. Staging a fetched page also became a per-source failure rather than a traceback, so a full temporary directory no longer ends the run the rest of the list was promised. |
+| 2026-09-02 | **A registry can forget a source, and forgetting removes nothing.** Ingest was one-way, and now that `ingest` finishes its list the unattended folder run is the normal case — so a scratch copy, the same report under two names, or a file nobody meant to include lands and stays: listed by `read`, competing in `search`, and citable by an agent choosing evidence without anything looking wrong. The only recovery was deleting `.backdraft/`, which throws away every other document and the whole ledger. `backdraft forget <slug>` **withdraws** rather than deletes, and the whole design is in that word: the document, its generations, its anchors and its receipts all stay, so `resolve` still finds every anchor and a token already written into somebody's draft or artifact still shows its snippet under `show`. What a withdrawal changes is that the registry stops *offering* the source — `documents()` excludes it, and with it `ls`, the gate's list, its table of contents, page reads and `search`, whose filter sits in the shared `_match_sql` so the pre-limit `total` a caller is offered to widen to drops too. A deletion would have stranded those citations with nothing to say about them, which is the failure this system exists to prevent. The status is the second half of the decision and the temptation was a sixth one: `citation_for` reports a withdrawn source's token `unresolved` carrying `withdrawn_reason` as its `error`, because the closed status set is the artifact format's and a new value would break every reader written against it, while `resolved` would let a document keep citing a source the registry no longer offers and report nothing. That widens `error` from `malformed`'s explanation to any status the status alone does not explain — an optional key appearing where it applies, not a new format version — and it made the legend's own `unresolved` line wrong, since a withdrawn citation *does* carry an anchor; corrected in `LEGEND` and `spec/artifact.md` at the cost of regenerated goldens and a re-rendered demo, the 2026-08-24 trade. Storage is a `withdrawals` table rather than a column on `documents`, which is the 2026-08-05 reason `document_meta` is one: an existing registry needs no migration, and this repo has no migration machinery to introduce for one flag. Ingesting the source again is the undo and deletes the row, bringing back the *same* document under the same slug — `_find_document` matches a withdrawn document like any other, which is exactly what a delete could not do, since a second slug would leave every token carrying the first one stranded. That undo is silent-by-default territory, so it is not: `Ingested.restored` is a fact beside `outcome` rather than a fourth value of it (a forgotten file that was also edited is both restored and a new generation, and one field could only say one), the line says `restored`, and a grouped note names what came back. Two costs stated rather than discovered. A registry only ever grows: nothing is reclaimed, and a corpus you must be able to shrink is a new registry, not a `forget`. And the rule is drawn at *offering a source to read*, so `snapshot-pages` still backfills page images for a withdrawn document — it serves no text and cannot reach a writer's context, and widening the rule to every command that touches a document would make it a rule nobody can state. `forget` asks before withdrawing and, where nothing can answer, names `--yes` instead of hanging on a prompt or aborting without a reason, since the usual caller here is an agent with no terminal. |
+| 2026-09-03 | **The rule moved down to the type it is about, and the last surface naming a source wrongly stopped.** The 2026-08-18 row gave `gate.source_name` one owner and accepted one exception on a dependency argument, and the argument was sound about imports and wrong about where the rule belonged. Bind's `_doc_name` returned `Document.filename`, and it feeds *both* halves of the markdown projection — the generated `## References` section and backfill's proposed anchors — so `bind --bound` called the demo's Wikipedia article `index.html` while `render --to footnotes` named its URL and the artifact named it *Franklin County*. Three of backdraft's own outputs, two answers, and the one that disagreed was the one a reader gets without a browser: the bound markdown is the form that travels into a pull request or an email, where the HTML artifact cannot follow. The fix is to stop working around the dependency rule and read it properly. `source_name` is a pure function of a `Document`, `Document` is defined in `kernel/model.py`, and gate, bind and the CLI all import that module *downward* — so the kernel was the owner the rule was looking for all along, and it stays stdlib-pure there. SPEC § Dependency rule now says so in general, because the reasoning that produced the exception ("we cannot share it sideways, so we duplicate or we live with it") will recur for the next shared rule and the escape hatch should not have to be rediscovered. `gate.source_name` is **not** left as a redirect: callers import `kernel.model.source_name` and the gate's re-export is gone, since two import paths for one function is the same shape as two names for one source — the thing the 2026-08-06 row rejected — and the gate's stated export style is names it owns, not names it passes through. The cost is that anything importing `from backdraft.gate import source_name` breaks; it is one internal caller and a library surface nobody outside this repo is known to use, and leaving both paths alive is what makes an owner stop being one. `_origin` moved with it as public `source_origin`, because the gate's document list needs the *question* ("is this name a URL, and therefore must not size the name column?") and not just the answer, and two readings of one `meta` key drift apart the moment the key does. Display only, unchanged: the slug, the token and everything a citation resolves through are untouched, and a References entry for a file source is byte-identical to what it printed before — pinned as a whole line, the way `ls`'s file rows are. The render side was deliberately not touched and was never wrong: `render/html/text.source_title` and `render/footnotes._origin` read a `url` off the *artifact's* `docs` dict, which is the artifact format's shape rather than a `Document`, and they already prefer the slug and the URL over the invented filename — they look like misses and are not. One thing this closes that was already promised in prose: `demo/walkthrough.md` told readers that a staging filename appears at "no surface that names a source", which was false for as long as this row's exception stood. |
 
 ## Open
 

{backdraft-0.8.0 → backdraft-0.10.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: backdraft
-Version: 0.8.0
+Version: 0.10.0
 Summary: Drop-in provenance for factual claims: one click from any claim to the evidence behind it.
 Project-URL: Homepage, https://backdraft.dev
 Project-URL: Documentation, https://backdraft.dev/docs.html
@@ -18,6 +18,8 @@ Requires-Dist: python-pptx>=1.0
 Requires-Dist: typer>=0.12
 Provides-Extra: entail
 Requires-Dist: anthropic>=0.40; extra == 'entail'
+Provides-Extra: math
+Requires-Dist: latex2mathml>=3.77; extra == 'math'
 Provides-Extra: vlm
 Provides-Extra: xls
 Requires-Dist: python-calamine>=0.3; extra == 'xls'
@@ -58,7 +60,11 @@ PPTX, HTML, images (png, jpeg, tiff, through the vision model), plain text and
 Markdown. A source can also be an `http(s)` URL — the page is fetched once and
 snapshotted like any other source, and the URL travels with it. A source that
 cannot be read does not end the run: the rest of the list is ingested anyway,
-and the command exits 1 naming each source that failed and why.
+and the command exits 1 naming each source that failed and why. The why is
+written to be acted on rather than relayed — a directory says to name the files
+inside it or pass a glob, a missing path says to check the spelling, an
+unreadable file says to fix its permissions or ingest a copy, and a source with
+no bytes in it is a failure rather than a document with nothing to cite.
 
 ```console
 $ backdraft init
@@ -128,6 +134,11 @@ $ backdraft search "24850000"
 [Read the page: backdraft read underwriting-model p2]
 ```
 
+Search shows twenty results by default. A run that hit the limit says so —
+`20 of 56 results` — and closes with the command that shows the rest, because
+"20 results" over an answer of 56 is how a reader concludes it has seen
+everything when it has seen a third.
+
 A cell you can already see mints directly:
 
 ```console
@@ -154,6 +165,31 @@ source has changed since prints both snippets, the one that was cited and the
 one standing there now; a token that names nothing exits 1 and says whether the
 slug or the locator is the wrong half.
 
+**Check coverage before writing.** The ledger records every token the gate put
+in front of you, and `session show` reads it back — the question "have I read
+enough to write this yet?", asked before the draft exists rather than after
+`bind` reports what it never saw.
+
+```console
+$ backdraft session show
+session s-bridgeview  (from BACKDRAFT_SESSION)
+
+77 anchors shown across 3 documents
+
+  t12-summary          7
+  underwriting-model  68
+  franklin-county      2
+
+[Read more: backdraft read <slug> <page>]
+```
+
+Anything counted there binds `resolved`; anything else in the registry binds
+`not_shown`. Without an exported session, reads land in one default ledger every
+run in the project shares and nothing ever resets — so `not_shown` weakens from
+"this writer never saw it" to "nothing here ever has". `session show` says so at
+exit 0 when you are in it, and `backdraft session start --id s-<name>` is the
+one command that fixes it.
+
 **Cite** by writing the token as the href of a markdown link on the claim span.
 Multiple citations are `;`-separated in one href.
 
@@ -170,6 +206,7 @@ bound 17 claim(s), 18 citation(s) [frontwalk]
   resolved: 17
   unresolved: 1
   overlap: pass 13, skip 4
+    skip 4 — wording overlap does not apply to a single cell
   value-trace: pass 17
   ! unresolved: bd:t12-summary:p4.c1:1a2b — replacement reserve of $250 per unit per year @3629
 wrote .backdraft/records/memo.backdraft.json
@@ -193,6 +230,29 @@ memo.backdraft.html
 That file is the whole deliverable: send it over email or Slack and the
 recipient gets the full experience by double-clicking it.
 
+**The recipient can check it.** `backdraft verify` takes the artifact or its
+sidecar and does the checking that the file is built to make possible: every
+snippet rehashed against the sha256 recorded beside it, every token checked
+against the anchor it names, `summary` recounted from the claims.
+
+```console
+$ backdraft verify memo.backdraft.html
+checked memo.backdraft.html [backdraft/artifact-v1]
+  receipts: 17 of 17 hold
+  record: 17 claim(s), 18 citation(s); the summary recount agrees
+  recorded: resolved 17, unresolved 1
+  sources: no .backdraft/ found from here — not re-checked
+[Re-check against the sources: run this inside the project it was bound in.]
+```
+
+That first tier needs nothing but the file, which is the point — it is the check
+someone who was emailed the artifact can still make. Run it inside the project
+the document was bound in and it also re-resolves every citation against the
+registry, so a source that has changed since shows up as `drifted` rather than
+going unnoticed. Exit 0 when everything it checked passed, 2 when something did
+not, so a hook can gate on it. It opens no session and mints nothing: an audit
+must not make its subject citable.
+
 **Theming.** The artifact ships with a default look and two alternates,
 `press` and `slate`:
 
@@ -268,9 +328,10 @@ q4-2025	https://example.com/reports/q4-2025	html	1 page
 
 Every surface names a fetched source by its page. The filename you never see —
 `q4-2025.html` — is the temporary file the fetch staged the bytes in, and it
-exists on nobody's disk, so `ingest`, `ls`, `backdraft read` and the artifact
-all show the URL in its place rather than beside it: two names for one thing
-would let the invented one look authoritative.
+exists on nobody's disk, so `ingest`, `ls`, `backdraft read`, the artifact and
+the `## References` section `bind --bound` writes all show the URL in its place
+rather than beside it: two names for one thing would let the invented one look
+authoritative.
 
 The origin travels into the artifact too. A claim citing a fetched page shows
 the URL as a link on its receipt, with the date the bytes were taken. That
@@ -285,6 +346,35 @@ are out of reach, and the extractor is a parse rather than a readability guess
 its mind between two versions of a site would move anchors. Responses are
 capped at 32 MiB.
 
+## Forgetting a source
+
+Ingest is otherwise one-way, and an unattended folder ingest picks up what it
+should not: a scratch copy, the same report under two names, a file nobody meant
+to include. Left in the registry it competes in `search` forever, and an agent
+choosing evidence can cite the wrong copy without anything looking wrong.
+
+```console
+$ backdraft forget scratch-copy --yes
+forgot scratch-copy (scratch-copy.md, text, 1 page)
+it is out of `backdraft read`, `search` and `ls`. Its anchors are untouched: a token already written into a draft or an artifact still shows its receipt under `backdraft show`, and `bind` reports it `unresolved` naming this withdrawal rather than passing it silently.
+[Re-ingest it to bring it back: backdraft ingest scratch-copy.md]
+```
+
+This **withdraws** rather than deletes, and the difference is the whole point.
+Nothing is removed — the document, its generations, its anchors and its
+receipts all stay — so a token already written into somebody's draft or artifact
+still resolves to the snippet it always named. What changes is that the registry
+stops offering the source: it leaves the document list, the table of contents,
+page reads and search, and `bind` reports its citations `unresolved` saying the
+source was withdrawn and when, instead of quietly passing them. A deletion would
+strand those citations with nothing to say about them, which is the failure this
+whole system exists to prevent.
+
+Ingesting the source again brings it back — as the same document under the same
+slug, not a second one beside it, so every token it ever minted resolves again.
+The price of never actually removing anything is that a registry only grows; a
+corpus you must be able to shrink is a new registry, not a `forget`.
+
 ## What lives where
 
 A working directory stays clean: the authored document and its artifact are
@@ -320,12 +410,13 @@ the keyless text layer and says so — still storing each page's image, which it
 renders locally. PDF page rendering needs poppler, installed separately; without
 it ingest succeeds without the images and names `snapshot-pages` as the backfill.
 
-Two extras:
+Three extras:
 
 | Extra | Adds | For |
 |---|---|---|
 | `[entail]` | `anthropic` | `bind --check entail`, the model-judge verifier |
 | `[xls]` | `python-calamine` | legacy `.xls` workbooks, values only, rendered the same shape as xlsx |
+| `[math]` | `latex2mathml` | LaTeX in a document becomes MathML in the artifact; without it, formulas render verbatim and `render` says so |
 
 `backdraft[vlm]` still installs: it is an empty compat alias from when the
 vision deps were an extra.
@@ -346,7 +437,7 @@ Three agent skills ship inside the package and in this repo:
 |---|---|
 | [`backdraft`](skills/backdraft/SKILL.md) | writing a new document from sources, citing as it goes |
 | [`backdraft-backfill`](skills/backdraft-backfill/SKILL.md) | attributing a document that already exists |
-| [`backdraft-artifact`](skills/backdraft-artifact/SKILL.md) | reading someone else's artifact cold |
+| [`backdraft-artifact`](skills/backdraft-artifact/SKILL.md) | checking and reading someone else's artifact cold |
 
 The CLI is the system; a skill is one page telling an agent to use it. Nothing in
 the skills is required to use backdraft by hand. How the skills reach your agent
@@ -404,9 +495,10 @@ spans are citable. In a sandboxed session run every command as
 | [spec/tokens.md](spec/tokens.md) | the citation token grammar, normatively |
 | [spec/chunking.md](spec/chunking.md) | the deterministic chunker |
 | [spec/artifact.md](spec/artifact.md) | the artifact format, sidecar payload, legend, HTML rules |
+| [spec/registry.md](spec/registry.md) | the registry export format `backdraft export` writes |
 | [demo/walkthrough.md](demo/walkthrough.md) | the whole thing end to end, with real output |
 
-The three files under `spec/` are the portable specification: another
+The four files under `spec/` are the portable specification: another
 implementation reads them and nothing else.
 
 ## Status

{backdraft-0.8.0 → backdraft-0.10.0}/README.md

@@ -33,7 +33,11 @@ PPTX, HTML, images (png, jpeg, tiff, through the vision model), plain text and
 Markdown. A source can also be an `http(s)` URL — the page is fetched once and
 snapshotted like any other source, and the URL travels with it. A source that
 cannot be read does not end the run: the rest of the list is ingested anyway,
-and the command exits 1 naming each source that failed and why.
+and the command exits 1 naming each source that failed and why. The why is
+written to be acted on rather than relayed — a directory says to name the files
+inside it or pass a glob, a missing path says to check the spelling, an
+unreadable file says to fix its permissions or ingest a copy, and a source with
+no bytes in it is a failure rather than a document with nothing to cite.
 
 ```console
 $ backdraft init
@@ -103,6 +107,11 @@ $ backdraft search "24850000"
 [Read the page: backdraft read underwriting-model p2]
 ```
 
+Search shows twenty results by default. A run that hit the limit says so —
+`20 of 56 results` — and closes with the command that shows the rest, because
+"20 results" over an answer of 56 is how a reader concludes it has seen
+everything when it has seen a third.
+
 A cell you can already see mints directly:
 
 ```console
@@ -129,6 +138,31 @@ source has changed since prints both snippets, the one that was cited and the
 one standing there now; a token that names nothing exits 1 and says whether the
 slug or the locator is the wrong half.
 
+**Check coverage before writing.** The ledger records every token the gate put
+in front of you, and `session show` reads it back — the question "have I read
+enough to write this yet?", asked before the draft exists rather than after
+`bind` reports what it never saw.
+
+```console
+$ backdraft session show
+session s-bridgeview  (from BACKDRAFT_SESSION)
+
+77 anchors shown across 3 documents
+
+  t12-summary          7
+  underwriting-model  68
+  franklin-county      2
+
+[Read more: backdraft read <slug> <page>]
+```
+
+Anything counted there binds `resolved`; anything else in the registry binds
+`not_shown`. Without an exported session, reads land in one default ledger every
+run in the project shares and nothing ever resets — so `not_shown` weakens from
+"this writer never saw it" to "nothing here ever has". `session show` says so at
+exit 0 when you are in it, and `backdraft session start --id s-<name>` is the
+one command that fixes it.
+
 **Cite** by writing the token as the href of a markdown link on the claim span.
 Multiple citations are `;`-separated in one href.
 
@@ -145,6 +179,7 @@ bound 17 claim(s), 18 citation(s) [frontwalk]
   resolved: 17
   unresolved: 1
   overlap: pass 13, skip 4
+    skip 4 — wording overlap does not apply to a single cell
   value-trace: pass 17
   ! unresolved: bd:t12-summary:p4.c1:1a2b — replacement reserve of $250 per unit per year @3629
 wrote .backdraft/records/memo.backdraft.json
@@ -168,6 +203,29 @@ memo.backdraft.html
 That file is the whole deliverable: send it over email or Slack and the
 recipient gets the full experience by double-clicking it.
 
+**The recipient can check it.** `backdraft verify` takes the artifact or its
+sidecar and does the checking that the file is built to make possible: every
+snippet rehashed against the sha256 recorded beside it, every token checked
+against the anchor it names, `summary` recounted from the claims.
+
+```console
+$ backdraft verify memo.backdraft.html
+checked memo.backdraft.html [backdraft/artifact-v1]
+  receipts: 17 of 17 hold
+  record: 17 claim(s), 18 citation(s); the summary recount agrees
+  recorded: resolved 17, unresolved 1
+  sources: no .backdraft/ found from here — not re-checked
+[Re-check against the sources: run this inside the project it was bound in.]
+```
+
+That first tier needs nothing but the file, which is the point — it is the check
+someone who was emailed the artifact can still make. Run it inside the project
+the document was bound in and it also re-resolves every citation against the
+registry, so a source that has changed since shows up as `drifted` rather than
+going unnoticed. Exit 0 when everything it checked passed, 2 when something did
+not, so a hook can gate on it. It opens no session and mints nothing: an audit
+must not make its subject citable.
+
 **Theming.** The artifact ships with a default look and two alternates,
 `press` and `slate`:
 
@@ -243,9 +301,10 @@ q4-2025	https://example.com/reports/q4-2025	html	1 page
 
 Every surface names a fetched source by its page. The filename you never see —
 `q4-2025.html` — is the temporary file the fetch staged the bytes in, and it
-exists on nobody's disk, so `ingest`, `ls`, `backdraft read` and the artifact
-all show the URL in its place rather than beside it: two names for one thing
-would let the invented one look authoritative.
+exists on nobody's disk, so `ingest`, `ls`, `backdraft read`, the artifact and
+the `## References` section `bind --bound` writes all show the URL in its place
+rather than beside it: two names for one thing would let the invented one look
+authoritative.
 
 The origin travels into the artifact too. A claim citing a fetched page shows
 the URL as a link on its receipt, with the date the bytes were taken. That
@@ -260,6 +319,35 @@ are out of reach, and the extractor is a parse rather than a readability guess
 its mind between two versions of a site would move anchors. Responses are
 capped at 32 MiB.
 
+## Forgetting a source
+
+Ingest is otherwise one-way, and an unattended folder ingest picks up what it
+should not: a scratch copy, the same report under two names, a file nobody meant
+to include. Left in the registry it competes in `search` forever, and an agent
+choosing evidence can cite the wrong copy without anything looking wrong.
+
+```console
+$ backdraft forget scratch-copy --yes
+forgot scratch-copy (scratch-copy.md, text, 1 page)
+it is out of `backdraft read`, `search` and `ls`. Its anchors are untouched: a token already written into a draft or an artifact still shows its receipt under `backdraft show`, and `bind` reports it `unresolved` naming this withdrawal rather than passing it silently.
+[Re-ingest it to bring it back: backdraft ingest scratch-copy.md]
+```
+
+This **withdraws** rather than deletes, and the difference is the whole point.
+Nothing is removed — the document, its generations, its anchors and its
+receipts all stay — so a token already written into somebody's draft or artifact
+still resolves to the snippet it always named. What changes is that the registry
+stops offering the source: it leaves the document list, the table of contents,
+page reads and search, and `bind` reports its citations `unresolved` saying the
+source was withdrawn and when, instead of quietly passing them. A deletion would
+strand those citations with nothing to say about them, which is the failure this
+whole system exists to prevent.
+
+Ingesting the source again brings it back — as the same document under the same
+slug, not a second one beside it, so every token it ever minted resolves again.
+The price of never actually removing anything is that a registry only grows; a
+corpus you must be able to shrink is a new registry, not a `forget`.
+
 ## What lives where
 
 A working directory stays clean: the authored document and its artifact are
@@ -295,12 +383,13 @@ the keyless text layer and says so — still storing each page's image, which it
 renders locally. PDF page rendering needs poppler, installed separately; without
 it ingest succeeds without the images and names `snapshot-pages` as the backfill.
 
-Two extras:
+Three extras:
 
 | Extra | Adds | For |
 |---|---|---|
 | `[entail]` | `anthropic` | `bind --check entail`, the model-judge verifier |
 | `[xls]` | `python-calamine` | legacy `.xls` workbooks, values only, rendered the same shape as xlsx |
+| `[math]` | `latex2mathml` | LaTeX in a document becomes MathML in the artifact; without it, formulas render verbatim and `render` says so |
 
 `backdraft[vlm]` still installs: it is an empty compat alias from when the
 vision deps were an extra.
@@ -321,7 +410,7 @@ Three agent skills ship inside the package and in this repo:
 |---|---|
 | [`backdraft`](skills/backdraft/SKILL.md) | writing a new document from sources, citing as it goes |
 | [`backdraft-backfill`](skills/backdraft-backfill/SKILL.md) | attributing a document that already exists |
-| [`backdraft-artifact`](skills/backdraft-artifact/SKILL.md) | reading someone else's artifact cold |
+| [`backdraft-artifact`](skills/backdraft-artifact/SKILL.md) | checking and reading someone else's artifact cold |
 
 The CLI is the system; a skill is one page telling an agent to use it. Nothing in
 the skills is required to use backdraft by hand. How the skills reach your agent
@@ -379,9 +468,10 @@ spans are citable. In a sandboxed session run every command as
 | [spec/tokens.md](spec/tokens.md) | the citation token grammar, normatively |
 | [spec/chunking.md](spec/chunking.md) | the deterministic chunker |
 | [spec/artifact.md](spec/artifact.md) | the artifact format, sidecar payload, legend, HTML rules |
+| [spec/registry.md](spec/registry.md) | the registry export format `backdraft export` writes |
 | [demo/walkthrough.md](demo/walkthrough.md) | the whole thing end to end, with real output |
 
-The three files under `spec/` are the portable specification: another
+The four files under `spec/` are the portable specification: another
 implementation reads them and nothing else.
 
 ## Status