backdraft 0.5.0__tar.gz → 0.6.0__tar.gz

{backdraft-0.5.0 → backdraft-0.6.0}/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "backdraft",
   "displayName": "Backdraft",
-  "version": "0.5.0",
+  "version": "0.6.0",
   "description": "Drop-in provenance for factual claims: one click from any claim to the evidence behind it. Ships the backdraft writing, backfill, and artifact-reading skills; the CLI installs separately (uv tool install backdraft).",
   "author": {
     "name": "Spencer Braun",

backdraft-0.6.0/.github/workflows/site.yml

@@ -0,0 +1,36 @@
+name: site
+on:
+  push:
+    branches: [main]
+    paths: ["site/**"]
+  workflow_dispatch:
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+# One deployment at a time, and a publish in flight is never cancelled: a
+# cancelled Pages deploy can leave the backend with a deployment that never
+# completes, which is a stuck site rather than a lost run.
+concurrency:
+  group: pages
+  cancel-in-progress: false
+jobs:
+  # Build and deploy are separate so that re-running the failed job re-runs
+  # only the deploy. Bundled, a retry uploaded a second `github-pages`
+  # artifact and `deploy-pages` refused the run for finding two of them.
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/upload-pages-artifact@v3
+        with:
+          path: site/
+  deploy:
+    needs: build
+    runs-on: ubuntu-latest
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - id: deployment
+        uses: actions/deploy-pages@v4

{backdraft-0.5.0 → backdraft-0.6.0}/DESIGN.md

@@ -135,6 +135,11 @@ Self-contained HTML (plus the machine-readable sidecar): **quotes + receipts + e
 | 2026-07-29 | **The vision extractor ships by default (supersedes "[vlm] as extra").** `openai` + `pdf2image` moved into core dependencies; `[vlm]` remains as an empty compat alias. Rationale: the VLM path is the recommended path for real PDFs, and an extra made the recommended path opt-in. The consent boundary does not move — installing deps was never consent; the backdraft-scoped `BACKDRAFT_VLM_API_KEY` is, and ambient provider keys stay unread. There is no slim install: pip extras can only add, never subtract; a separate core distribution is possible later if anyone asks. poppler stays a runtime capability check. |
 | 2026-07-29 | **Renderer decomposed; Python is the formatting authority.** `render/html.py` (1,300 lines) became the `render/html/` package (fmt, text, components, assets, page), byte-identical at the seam. The deliberate `fmt_cell` (Python) / `fmtCell` (JS) duplication is now pinned by one shared vector table driving both tests, with the JS run under node when available. The JS was found diverging on exact rounding ties (`toFixed` rounds half-up; Python rounds half-to-even on the double's value) and was rewritten to round exactly — Python defines correct, JS matches it. The artifact emits minified CSS/JS and unindented markup; the `$legend` and sidecar JSON stay readable in view-source (spec constraint). Snapshot budget knobs `BACKDRAFT_SNAPSHOT_QUALITY`/`BACKDRAFT_SNAPSHOT_MAX_HEIGHT` tune page-image weight at ingest — display only, tokens never derive from pixels. |
 | 2026-07-30 | **Every PDF ingest captures page images, not just the VLM path's.** The text-layer route stored none, so the keyless path produced visually poorer artifacts for no principled reason — the pages render locally through poppler, no model calls, and the capability already existed as the `snapshot-pages` backfill. The capture is a step *after* extraction, owned by `cli.py` over `extract/snapshots.py`, deliberately **not** inside `pdf-text`: an extractor whose output varied with whether poppler happened to be installed would not be `deterministic = True`, and the same bytes must extract identically on every machine. Because snapshots only ever write `page_images` — a table no token, chunk, or receipt path reads — capture is safe to make best-effort: no poppler means ingest still exits 0, with a one-line note naming `snapshot-pages` as the backfill, rather than failing over a display concern. Tradeoff accepted: PDF ingest is now slower by a local render per page (one poppler call per page rather than one per document, to bound peak memory on long PDFs), and a library caller using `Registry.ingest` directly gets no snapshots — the policy lives at the CLI, which is the substrate that has a user to notify. |
+| 2026-08-03 | **The card divides its own height; nothing inside it is pinned.** The receipt card is user-resizable, so it is a flex column: header, source selector, source line and tabs are fixed rows that never scroll away, and the evidence region takes whatever is left — dragging the card taller grows the evidence rather than the whitespace under it. The caps that survive are relative (the card at `82vh` of the window; the quote, drift diff and record at `40%` of the card), and a `5rem` floor keeps the evidence from vanishing at the card's minimum, with the citation scrolling instead. Resizing means *more*, not *smaller*: the page image keeps its scale and the pane scrolls over it, because a page shrunk to fit a short card is a page you cannot read, while the sheet window — which has no scale to lose — grows to fill, matching the full-sheet overlay it opens into. Both sheet views now stick their headers from one rule. Display only, as ever: no token, receipt or record moves, and the phone card keeps `resize:none` and the fixed caps it always had, since there is no drag there for the evidence to follow. |
+| 2026-08-04 | **A theme is an override block, not a stylesheet.** The artifact's look is themeable, and a theme is a small TOML file naming values for the CSS custom properties the stylesheet already exposes — the sixteen colors and the three font roles — plus four bounded heading knobs (family, case, weight, tracking) that the theme module compiles into one rule over the document's own headings (`HEADING_SELECTOR`) that it owns. The built-in stylesheet is not edited and not replaced: the resolved theme is emitted as a block *after* it, so with nothing configured the artifact is byte-identical to an untheme'd build and theming is provably inert by default. A theme may not touch layout, structure, or any verification affordance, and never touches a token, receipt or record — `--rail-w` and `color-scheme` are deliberately outside the allowlist for that reason, and so is the artifact's refusal to follow the reader's system theme. Keys and values are both validated against declared allowlists, so `--colour` or `#GGG` fails by name before anything is written rather than producing a half-styled file; `url(...)` is rejected with the reason, since a font the file would have to fetch is exactly what the CSP forbids. Precedence is `--theme` > project `.backdraft/theme.toml` > XDG `~/.config/backdraft/theme.toml` > built-in, the user-wide file being what makes a preference stick across projects. The bundled `default.toml` restates today's values and is pinned by a test against the stylesheet's own `:root`, which makes it the audit as well as the reference sample. Tradeoff accepted: `--serif`/`--sans`/`--mono` name roles (body, UI, code) rather than classifications, so a sans-bodied theme sets `serif`. *(2026-08-04, follow-up: `--hl` was declared in `:root` and read by no rule, so it is deleted rather than left as a themeable key that paints nothing, and the `#fff` behind page images now tracks `--paper` — white there was coincidence, and every stored image is opaque RGB, so it was dead either way. The six constants that remain hardcoded — the card and rail shadows, the overlay scrim, the PDF tab accent, the `partial` verdict amber, sheet-header grey — were considered and deliberately left: each is near-neutral at any sane palette distance, and promoting them is a wider audit for churn rather than effect. Separately, `.doc h3` through `h6` had no rule at all — the markdown renderer emits every level and only two were styled, so a memo with `### ` fell to the browser's bold-Times default mid-page — and they are now a two-step scale that *brackets* the body rather than continuing down from `h2`, since the sheet sizes in `rem` while `body` sets 17px and a plain `1rem` heading lands under the text it heads. The theme's heading selector covers the whole scale for the same reason it exists: small-caps section heads above a body-face subsection read as a mistake, not a choice.)* |
+| 2026-08-04 | **Failure speaks at the claim, not in the masthead (amends the 2026-07-28 reader doctrine).** The artifact no longer prints a failure summary under the title. "Success is silent; only failure speaks" stands, and so does the principle under it — failures are data, never warn-and-drop — but *where* it speaks moves: the wavy alarm-coloured mark on the affected claim, and that claim's note with the reason, both of which a reader meets in context. The line that went was a bare count, and a count is the one thing about a failed citation a reader cannot act on: whether `1 of 16 citations could not be traced` matters depends entirely on which claim it was, which the headline does not say and the mark and the note do. It was also aimed at the wrong reader — the writer verifying the work learns about failures from `bind`, which exits 2 and lists them, while the recipient got a number out of context before the first sentence. Nothing loses its last surface: an unmatched claim with no citations ranks `unresolved` and is marked in the body like any other failure, and a claim that is not in the document at all was already the Notes' job under artifact rule 4. This is a spec change — `spec/artifact.md` rule 6 is rewritten, and the summary is now explicitly *not* required of a conforming producer rather than merely unmentioned, so another implementation reads the same intent. |
+| 2026-08-05 | **A web page's identity is the sha256 of the snapshot fetched at ingest time; the URL is provenance, not identity.** `ingest` accepts an http(s) URL alongside files (graduating the "HTML pages" question from Later). The fetched bytes are hashed and snapshotted exactly as a file's are, so everything downstream — anchors, the ledger, `bind`, generations — is unchanged, and a page that changed since the last fetch is a *new generation of the same document*, which is the machinery drift already describes. Continuity across a re-fetch follows the URL rather than the path, because the path is a temporary staging file; `documents.sha256` is UNIQUE, so bytes still win over origin when the same content arrives twice under two URLs. The URL and `fetched_at` live in a new `document_meta` table — a JSON sibling of `page_meta`, chosen over columns on `documents` so an existing registry needs no migration and so the next provenance question (`.eml` headers) needs no second one; `fetched_at` moves on a re-fetch that changed nothing, since "when was this last confirmed to say this" is the question a reader of an old citation actually has. Fetching is stdlib `urllib` and lives in `fetch.py`, *outside* `extract/`: an extractor stays a pure function of a file's bytes, which is what keeps `deterministic` honest and lets a saved `.html` file and a fetched page extract identically. The CLI owns the network the way it already owns page-snapshot capture. The new `html` extractor is a parse, not a readability heuristic — no boilerplate stripping, no main-content guess, because a heuristic that changes its mind moves anchors, which is the failure mode this system exists to prevent; navigation and footers are part of the page. Its representation follows `docx`: tables as markdown pipe tables, lists as `- `/`1. ` lines kept in one block, `<pre>` verbatim. Decoding is BOM, then the document's own `<meta charset>`, then UTF-8 with replacement — the server's `Content-Type` charset is deliberately *not* consulted, so the bytes alone determine the text. Known limitations, accepted and documented rather than worked around: JavaScript-rendered pages yield whatever the server sends, pages behind a login are out of reach, and a non-UTF-8 page that declares its encoding only in the HTTP header will render as mojibake in the snapshot — visibly, rather than silently. |
+| 2026-08-06 | **A fetched source's origin travels into the artifact as a live link, dated.** The capture half of the row above stored `url` and `fetched_at` and stopped at the registry, so an artifact citing a web page named `q4-2025.html` — a staging filename that never existed on anyone's disk — and a reader holding the file had the frozen receipt with no way back to the page. `bind`'s evidence `documents` map now carries `url` and `fetched_at` through to the record, and both renderers show them: the receipt card, the script-free note, the source list and the rail all print the URL as a real `<a href>` with the fetch date beside it, and the markdown projection carries it as an autolink. This is an artifact-format change, so `spec/artifact.md` and the `$legend` name the two keys together, both OPTIONAL. The keys are emitted *only* where a document has a URL, which is what keeps an artifact built from files byte-identical to one built before this existed — the conditional is the compatibility guarantee, and it is a test rather than an intention. Linking does not weaken the self-containment rule: the CSP (`default-src 'none'`) forbids *fetching*, and an anchor fetches nothing until a reader chooses to follow it, so the file still renders identically offline. The href is allowlisted to `http`/`https` rather than merely screened for `javascript:`, since artifact rule 3 is a conformance requirement and an allowlist cannot be outflanked by a scheme nobody thought of; a URL under any other scheme is printed as plain text rather than dropped, because failures are data and provenance the reader must paste by hand still beats provenance withheld. In the source list the URL *replaces* the filename rather than joining it — showing both would give a reader two names for one thing and let the fictional one look authoritative. The date is the day, not the timestamp: "as of" is what makes a frozen quote from a live page defensible, and no reader needs the seconds. |
 
 ## Open
 

{backdraft-0.5.0 → backdraft-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: backdraft
-Version: 0.5.0
+Version: 0.6.0
 Summary: Drop-in provenance for factual claims: one click from any claim to the evidence behind it.
 Project-URL: Homepage, https://backdraft.dev
 Project-URL: Documentation, https://backdraft.dev/docs.html
@@ -54,8 +54,9 @@ cd demo
 
 **Ingest** the sources. Every anchor is minted here, chunks for PDF pages,
 cells for spreadsheet sheets. Formats: PDF, XLSX/XLSM, XLS, CSV/TSV, DOCX,
-PPTX, images (png, jpeg, tiff, through the vision model), plain text and
-Markdown.
+PPTX, HTML, images (png, jpeg, tiff, through the vision model), plain text and
+Markdown. A source can also be an `http(s)` URL — the page is fetched once and
+snapshotted like any other source, and the URL travels with it.
 
 ```console
 $ backdraft init
@@ -65,7 +66,7 @@ documents: 0
 
 $ backdraft ingest sources/t12-summary.pdf sources/underwriting-model.xlsx
 t12-summary  t12-summary.pdf  pdf  3 pages
-underwriting-model  underwriting-model.xlsx  xlsx  2 pages
+underwriting-model  underwriting-model.xlsx  xlsx  2 sheets
 note: extracted with pdf-text (the embedded text layer). Glossy or scanned PDFs extract better through a vision model: set BACKDRAFT_VLM_API_KEY in .backdraft/env.
 ```
 
@@ -155,6 +156,33 @@ memo.backdraft.html
 That file is the whole deliverable: send it over email or Slack and the
 recipient gets the full experience by double-clicking it.
 
+**Theming.** The artifact ships with a default look and two alternates,
+`press` and `slate`:
+
+```console
+$ backdraft render memo.md --theme slate
+```
+
+A theme is a small TOML file naming colors, font stacks and heading treatment.
+Drop one at `~/.config/backdraft/theme.toml` and every artifact you render, in
+any project, honors it without a flag; `.backdraft/theme.toml` overrides that
+for one project, and `--theme` overrides both. Themes are display only — they
+never touch a token, a receipt or the record.
+
+```console
+$ backdraft theme list
+default
+press
+slate
+
+in effect here: the built-in look
+[Start your own: backdraft theme show default > /Users/you/.config/backdraft/theme.toml]
+```
+
+`theme show` prints a validated file, so redirecting it writes a fully
+commented starting point with every key and what it paints — and pointing it
+at a file of your own checks that file without rendering anything.
+
 The full version of this, with every command and every output:
 [`demo/walkthrough.md`](demo/walkthrough.md). The artifact it produces is checked
 in at [`demo/memo.backdraft.html`](demo/memo.backdraft.html).
@@ -183,6 +211,38 @@ Failures are data throughout: `unresolved`, `not_shown`, `drifted`, `malformed`
 and `unmatched` are first-class records in the report and visible sections in the
 artifact. Nothing is ever warned about and dropped.
 
+## Web sources
+
+`backdraft ingest https://example.com/q4-results` fetches the page once and
+snapshots it exactly as a file: the bytes at fetch time are the document's
+identity, and the URL and fetch timestamp ride along as provenance. So a page
+that has changed since you cited it comes back as a new generation of the same
+document and the old citations report `drifted` — the same machinery as an
+edited PDF, which is the point of hashing the snapshot rather than trusting the
+address.
+
+```console
+$ backdraft ingest https://example.com/reports/q4-2025
+q4-2025  q4-2025.html  html  1 page
+
+$ backdraft ls
+q4-2025	q4-2025.html	html	1 page	https://example.com/reports/q4-2025
+```
+
+The origin travels into the artifact too. A claim citing a fetched page shows
+the URL as a link on its receipt, with the date the bytes were taken, and the
+source list names the page rather than the staged filename. That pointer is the
+half of citing a web page a frozen receipt cannot answer on its own: the
+snippet says what the page said, the link is how a reader asks whether it still
+says it.
+
+What it does not do, stated rather than worked around: JavaScript-rendered
+pages give you whatever the server sends to a plain GET, pages behind a login
+are out of reach, and the extractor is a parse rather than a readability guess
+— navigation and footers are part of the page, because a heuristic that changes
+its mind between two versions of a site would move anchors. Responses are
+capped at 32 MiB.
+
 ## What lives where
 
 A working directory stays clean: the authored document and its artifact are

{backdraft-0.5.0 → backdraft-0.6.0}/README.md

@@ -29,8 +29,9 @@ cd demo
 
 **Ingest** the sources. Every anchor is minted here, chunks for PDF pages,
 cells for spreadsheet sheets. Formats: PDF, XLSX/XLSM, XLS, CSV/TSV, DOCX,
-PPTX, images (png, jpeg, tiff, through the vision model), plain text and
-Markdown.
+PPTX, HTML, images (png, jpeg, tiff, through the vision model), plain text and
+Markdown. A source can also be an `http(s)` URL — the page is fetched once and
+snapshotted like any other source, and the URL travels with it.
 
 ```console
 $ backdraft init
@@ -40,7 +41,7 @@ documents: 0
 
 $ backdraft ingest sources/t12-summary.pdf sources/underwriting-model.xlsx
 t12-summary  t12-summary.pdf  pdf  3 pages
-underwriting-model  underwriting-model.xlsx  xlsx  2 pages
+underwriting-model  underwriting-model.xlsx  xlsx  2 sheets
 note: extracted with pdf-text (the embedded text layer). Glossy or scanned PDFs extract better through a vision model: set BACKDRAFT_VLM_API_KEY in .backdraft/env.
 ```
 
@@ -130,6 +131,33 @@ memo.backdraft.html
 That file is the whole deliverable: send it over email or Slack and the
 recipient gets the full experience by double-clicking it.
 
+**Theming.** The artifact ships with a default look and two alternates,
+`press` and `slate`:
+
+```console
+$ backdraft render memo.md --theme slate
+```
+
+A theme is a small TOML file naming colors, font stacks and heading treatment.
+Drop one at `~/.config/backdraft/theme.toml` and every artifact you render, in
+any project, honors it without a flag; `.backdraft/theme.toml` overrides that
+for one project, and `--theme` overrides both. Themes are display only — they
+never touch a token, a receipt or the record.
+
+```console
+$ backdraft theme list
+default
+press
+slate
+
+in effect here: the built-in look
+[Start your own: backdraft theme show default > /Users/you/.config/backdraft/theme.toml]
+```
+
+`theme show` prints a validated file, so redirecting it writes a fully
+commented starting point with every key and what it paints — and pointing it
+at a file of your own checks that file without rendering anything.
+
 The full version of this, with every command and every output:
 [`demo/walkthrough.md`](demo/walkthrough.md). The artifact it produces is checked
 in at [`demo/memo.backdraft.html`](demo/memo.backdraft.html).
@@ -158,6 +186,38 @@ Failures are data throughout: `unresolved`, `not_shown`, `drifted`, `malformed`
 and `unmatched` are first-class records in the report and visible sections in the
 artifact. Nothing is ever warned about and dropped.
 
+## Web sources
+
+`backdraft ingest https://example.com/q4-results` fetches the page once and
+snapshots it exactly as a file: the bytes at fetch time are the document's
+identity, and the URL and fetch timestamp ride along as provenance. So a page
+that has changed since you cited it comes back as a new generation of the same
+document and the old citations report `drifted` — the same machinery as an
+edited PDF, which is the point of hashing the snapshot rather than trusting the
+address.
+
+```console
+$ backdraft ingest https://example.com/reports/q4-2025
+q4-2025  q4-2025.html  html  1 page
+
+$ backdraft ls
+q4-2025	q4-2025.html	html	1 page	https://example.com/reports/q4-2025
+```
+
+The origin travels into the artifact too. A claim citing a fetched page shows
+the URL as a link on its receipt, with the date the bytes were taken, and the
+source list names the page rather than the staged filename. That pointer is the
+half of citing a web page a frozen receipt cannot answer on its own: the
+snippet says what the page said, the link is how a reader asks whether it still
+says it.
+
+What it does not do, stated rather than worked around: JavaScript-rendered
+pages give you whatever the server sends to a plain GET, pages behind a login
+are out of reach, and the extractor is a parse rather than a readability guess
+— navigation and footers are part of the page, because a heuristic that changes
+its mind between two versions of a site would move anchors. Responses are
+capped at 32 MiB.
+
 ## What lives where
 
 A working directory stays clean: the authored document and its artifact are

backdraft-0.6.0/ROADMAP.md

@@ -0,0 +1,258 @@
+# Roadmap
+
+Sequenced intent. DESIGN.md records what was *decided*; this file records what
+is *queued*, so tabling something is an act of writing it down here rather than
+losing it. Items graduate by getting built and deleted.
+
+## How the queue runs
+
+The Now section is an ordered queue written for an implementing agent with no
+session context: each item carries its intent, its shape, and its acceptance
+test. The working agreement is one item at a time, top first, shipped complete
+— code, tests, docs, and a DESIGN.md decision row where the item calls for one
+— then deleted from this file in the same commit. Releases cut Friday, so an
+item that lands mid-week waits on PyPI until the release; nothing ships
+half-done to make a release. An item that turns out bigger than its sizing
+gets split here, not stretched silently.
+
+The cadence is daily and the runner is an agent: one item each morning Monday
+through Thursday, landing as a direct commit to `main` — this repo does not
+use pull requests, so a week of commits is the only review surface there is.
+
+Friday builds nothing new. It reads that week's diff instead — duplication,
+drift from the repo's own idioms, branches that landed untested, help text and
+error messages that confuse, commands the docs describe wrongly — and lands
+the fixes. Then the release, and only if something landed since the last tag:
+the version moves in both `pyproject.toml` and `.claude-plugin/plugin.json`,
+and publishing the GitHub release is what ships to PyPI. Then this section is
+refilled with five items for the week ahead, drawn against the core — the four
+verbs, the registry, the grammar, the artifact — and against where an agent
+using backdraft actually stumbles. What the maintenance pass just found is the
+best evidence available for what those five should be.
+
+## Now
+
+### 1. Bind's failure lines name the claim, not just the token
+
+**Intent.** Exit 2 tells a calling agent that a citation did not resolve and
+prints the token: `! unresolved: bd:t12-summary:p4.c1:1a2b`. It does not say
+which sentence that token was on. `skills/backdraft/SKILL.md` then instructs
+the agent to "tell the user which claim it belongs to" — the one fact the
+report withheld — so the agent's next move is to grep its own document for a
+token string. Meanwhile the `! unmatched:` line, three lines below, already
+prints its claim text. The two line-items should carry the same weight.
+
+**Shape.** `_print_report` in `bind/cli.py`, CLI output only — the record JSON
+is a pinned format and already carries the claim/citation nesting, so nothing
+in `spec/artifact.md` moves. `BindReport.claims` holds each claim's `text`,
+`start` and `end` alongside its citations, so the mapping from a reported
+citation back to its claim exists in memory; report the citation line as the
+token, then the claim text truncated the way the `unmatched` line already
+truncates it (80 chars), then the offset. Repeats matter here: the same token
+cited on four claims currently prints four identical lines, which reads as four
+problems — print one line per (token, claim) pair, which is what the offset
+makes legible. Applies to every non-resolved status, `not_shown` and `drifted`
+included, not only `unresolved`.
+
+**Acceptance.** In `demo/`, `backdraft bind memo.md --session s-bridgeview
+--check value-trace,overlap` prints the unresolved line with the replacement-
+reserve claim text and its offset on it. A bind whose session was never read
+into prints one line per distinct claim rather than one per citation
+occurrence. Tests cover `unresolved`, `not_shown`, `drifted` and `malformed`.
+Every doc that shows a bind report — `README.md`, `demo/walkthrough.md`,
+`site/llms.txt`, `skills/backdraft/SKILL.md`,
+`skills/backdraft-backfill/SKILL.md` — is updated to the real new output.
+
+**Size.** One day.
+
+### 2. `--config` keys are declared, validated, and listed
+
+**Intent.** `backdraft ingest x.pdf --config dpy=300` exits 0, ingests, and
+does nothing about the typo. No output names the mistake, and nothing anywhere
+— help, docs, `llms.txt` — lists what keys exist; today the only way to learn
+`dpi`, `model`, `base_url`, `api_key`, `snapshot_quality` or
+`snapshot_max_height` is to read `extract/vlm.py` and `extract/snapshots.py`.
+This repo's rule is that failures are data and never warn-and-drop, and the
+config boundary is where that rule does not hold.
+
+**Shape.** Give the `Extractor` protocol in `extract/base.py` a
+`config_keys: Mapping[str, str]` (key → one-line meaning), defaulting to empty
+so no extractor is forced to declare. `vlm` declares its provider and retry
+keys; the page-render keys (`dpi`, `snapshot_quality`, `snapshot_max_height`)
+are declared once by `extract/snapshots.py`, which already owns them, and are
+accepted for every PDF path since both rasterize. `ingest` validates the parsed
+config against the selected extractor's declared set *after* selection, since
+`auto` picks per file, and raises `UsageError` in the shape the unknown-name
+errors already use: `unknown config key 'dpy' for pdf-text; known: dpi,
+snapshot_max_height, snapshot_quality`. A key that is valid for one extractor
+and not another must fail only where it does not apply.
+
+**Acceptance.** The typo above exits 1 and names both the key and the valid
+ones; `--config dpi=300` still works on both the `vlm` and the text-layer path;
+`--config model=...` fails on `pdf-text` and passes on `vlm`. Tests for each
+branch, including a declaration-free extractor rejecting any key. The docs
+page's `ingest` row and `site/llms.txt` list the keys.
+
+**Size.** Two days.
+
+### 3. `backdraft show <token>`: the inverse of minting
+
+**Intent.** An agent handed a token — out of someone's artifact, a half-written
+draft, a colleague's message — cannot ask what it says. The gate mints tokens
+and `bind` resolves them, but only as part of binding a whole document, so
+there is no way to answer "what is `bd:t12-summary:p1.c4:410d`?" without
+reconstructing the slug and selector by hand and re-reading a whole page.
+`skills/backdraft-artifact/SKILL.md` names exactly this gap when it says only
+one check needs the outside world.
+
+**Shape.** A new command in `gate/cli.py` — `show <token> [<token> ...]` —
+because this is the gate showing source text and must mint what it shows,
+exactly as `read` and `search` do; an anchor shown here is an anchor the writer
+may cite, and `not_shown` stays true by construction. Parse with
+`kernel.tokens`, resolve through the same path `bind` uses rather than a second
+implementation of what a token means, and print the closed status set bind
+reports: `resolved` (locator + verbatim snippet), `drifted` (the snippet then
+and the snippet now), `unresolved`, `malformed`. Output follows the gate's
+line-oriented shape and ends with the usual bracketed next-step hint.
+
+**Acceptance.** In `demo/`, `backdraft show bd:t12-summary:p1.c1:c2e8` prints
+the snippet, and a subsequent `bind` of a document citing it reports `resolved`
+rather than `not_shown` — that is the test that proves it minted. A
+well-formed token naming nothing exits 1 saying so; a malformed one names the
+grammar. Several tokens in one invocation print in argument order. `README.md`,
+the docs page's command table, `site/llms.txt` and
+`skills/backdraft-artifact/SKILL.md` gain it.
+
+**Size.** Two days.
+
+### 4. Ingest finishes the list, then reports what it could not read
+
+**Intent.** `backdraft ingest a.md broken.pdf c.md` ingests `a.md`, fails on
+`broken.pdf`, and never attempts `c.md`. The registry is left half-built, the
+message names only the first failure, and nothing says which files landed and
+which were never tried — so an agent ingesting a folder must re-run and infer
+the state. This is the same warn-and-drop shape the page-snapshot capture
+already avoids: it collects failures, names them, and lets the rest of the run
+stand.
+
+**Shape.** `ingest` in `cli.py`. Move the per-file work inside a try that
+catches `BackdraftError` and records `(path, reason)` instead of propagating,
+so the loop always reaches the end of the list. After the loop, print one line
+per failure naming the file and the reason, grouped the way the snapshot note
+already groups by reason, and exit 1 if anything failed — the exit code is
+unchanged, only the coverage and the reporting improve. Re-ingest is idempotent
+for deterministic extractors, so re-running after a fix costs nothing, and the
+message should say that rather than leaving the agent to guess. Keep the guard
+in `cli_context` as the only place `BackdraftError` becomes an exit code:
+ingest catches per file and re-raises nothing, printing failures as data.
+
+**Acceptance.** The three-file example above ingests `a.md` and `c.md`, prints
+one failure line for `broken.pdf` naming why, and exits 1; `backdraft ls`
+afterwards shows both good documents. A run where every file fails still exits
+1 and names each. A run where none fail is byte-identical to today's output.
+Tests for all three.
+
+**Size.** One day.
+
+### 5. The demo cites a web page, so the feature is visible
+
+**Intent.** URL sources ship complete — `ingest` fetches and snapshots, the
+registry keeps `url` and `fetched_at`, and as of 2026-08-06 the artifact links
+back to the page with its fetch date. None of it appears at
+`backdraft.dev/demo.html`, because both demo sources are local files
+(`t12-summary.pdf`, `underwriting-model.xlsx`). The one artifact anybody
+actually looks at does not show the capability, and the byte-identity test that
+proved the change was safe is the same fact that keeps it invisible. A reader
+evaluating backdraft cannot see that a web citation is a citation like any
+other.
+
+**Shape.** Add a third source to `demo/`: the Wikipedia article for the county
+the property sits in, cited for market context — population, median household
+income, the kind of external figure a screening memo genuinely leans on and
+cannot get from a T12. Bridgeview Commons is fictional and says so in its own
+first line, so give it a real county and let the memo name it; one new
+paragraph under "The asset" or a short "Market" section, with one or two
+claims. Cite Wikipedia's **permanent link** (`?oldid=<rev>`), not the bare
+article URL: an article is edited constantly, a citation into the live URL
+would report `drifted` within days, and a front-page demo that permanently
+shows drift teaches that the tool is broken rather than that the source moved.
+An immutable revision plus the rendered "as of" date is also what a careful
+analyst cites. Write the tradeoff as a DESIGN row, naming the thing given up —
+the demo no longer shows a live page changing under a citation, which is the
+better story and belongs in a `drifted` walkthrough, not the shop window.
+
+Practical notes for whoever builds this: `demo/.backdraft/` is gitignored, so
+the fetch happens once on the implementer's machine and the snapshot then
+travels in the local registry; `demo/generate_sources.py` builds the two
+fixture files and is *not* where this goes, since the page is fetched rather
+than generated. Re-bind and re-render (`bind memo.md --session s-bridgeview
+--check value-trace,overlap`, then `render memo.md --to html`), and copy the
+result over `site/demo.html` — the two are byte-identical by convention.
+`demo/walkthrough.md` shows real CLI output and its `ls` and `bind` blocks both
+move.
+
+**Acceptance.** `backdraft.dev/demo.html` shows three sources, one of them a
+clickable Wikipedia link with a fetch date, on both the resting rail and the
+end-matter list; the claim citing it opens a card whose source line carries the
+same link. `demo/walkthrough.md` matches what the commands now print. A re-bind
+some weeks later still reports `resolved`, not `drifted` — that is the check
+that the permanent link did its job.
+
+**Size.** One day.
+
+## Parked
+
+Deliberately not queued, each with the reason, so picking one up starts from
+the objection rather than rediscovering it.
+
+- **Exhibits view** — the evidence-first inversion. Risk: it grows the
+  artifact and pollutes a design whose strength is restraint. Needs a
+  deliberate design pass and a real audience before any code; not a
+  weekend-feature shape.
+- **Excel region maps and range reads** — the research notes exist, but no
+  confidence the deterministic detection cascade lands well on the first
+  try. Revisit when large-sheet navigation hurts in a real corpus, with
+  test workbooks in hand.
+- **`bd:calc` derivations** — reserved in the grammar and parsed as
+  malformed-on-purpose, which is enough for now. Full support complicates
+  binding and verification for one field-trial case.
+- **Merged-cell and frozen-pane rendering** — ingest captures `merged` and
+  `frozen` in sheet meta and keeps them (cheap, and capture is the part you
+  cannot backfill). Frozen-pane rendering is cut; merged-cell rendering
+  waits for a real sheet that renders wrong without it.
+
+## Later
+
+- **Living documents** — re-ingest, re-bind, and a drift-first report: "these
+  claims cite figures that changed; cited-then vs. now." The primitives
+  (generations, `drifted`, the word-diff) all exist; the missing piece is a
+  diff-shaped report and a demo.
+- **Substrates beyond the CLI** — SDK middleware for pipelines (the exit-code
+  and `--to json` contracts are already shaped for it), and a client-side
+  drag-drop viewer page (no upload — the file never leaves the reader's
+  machine).
+- **Email (.eml)** — arrives in diligence folders alongside everything else;
+  identity questions (headers vs. body, attachments as child sources) deserve
+  a decision row before code. The web-page half of this item shipped
+  2026-08-05; `document_meta` is where an email's provenance would land.
+- **Anthropic-API provider for the VLM extractor** — Cowork's sandbox
+  egress allowlist includes api.anthropic.com but not the OpenAI-compatible
+  providers, so an Anthropic-native option would let sandboxed sessions
+  ingest at full fidelity. Needs a client abstraction the vlm module
+  currently doesn't have; the documented workaround (ingest locally, the
+  registry travels with the folder) covers it meanwhile.
+- **Distribution** — published on PyPI; the repo ships
+  `.claude-plugin/plugin.json` and `marketplace.json`, so it installs as a
+  Claude Code plugin and self-hosts its marketplace, and the AGENTS.md
+  snippet is published in the README and site docs. The community-directory
+  submission went in 2026-07-29 and is awaiting review — approval is what
+  surfaces the skills in Cowork's directory.
+
+## Someday
+
+- **Hosted team features** — upload, access control, shared registries.
+  Deliberately parked: the product's trust story is the self-contained file,
+  and a hosted viewer inverts it. Revisit only as a distinct teams product.
+- **Entail at scale** — the model-judge verifier is wired (`[entail]` extra)
+  but has never been field-calibrated; needs a corpus and a rubric before it
+  is recommended anywhere.

{backdraft-0.5.0 → backdraft-0.6.0}/SPEC.md

@@ -8,7 +8,7 @@ Design stance: this is a small codebase built as a base platform. A pure kernel
 
 | Concept | Is | Identity |
 |---|---|---|
-| **Document** | An ingested file | sha256 of bytes; human handle is a `slug` |
+| **Document** | An ingested file, or a web page snapshotted at ingest | sha256 of bytes; human handle is a `slug`. A fetched page's origin URL is provenance metadata, never identity |
 | **Extraction** | A snapshot of a document's content produced by one extractor run | (document, extractor, version, config); generations are kept, one is `current` |
 | **Page** | Ordered unit within an extraction — a PDF page or a sheet | (extraction, number) |
 | **Anchor** | An addressable location in an extraction, carrying its **receipt**: verbatim snippet + snippet hash | locator within extraction; named by its token |
@@ -52,6 +52,7 @@ backdraft/
       pptx.py               # python-pptx; one page per slide, text floor
       text.py               # md/txt passthrough
       image.py              # an image is a one-page document; VLM transcription
+      html.py               # html/htm → one page of readable text; a parse, not a readability guess
       vlm.py                # pdf→images→VLM per page (deps ship by default; key gates use)
       vlm_settings.py       # provider/model/base-url resolution; stdlib-only
     gate/
@@ -72,9 +73,12 @@ backdraft/
       placement.py          # locating each claim's span in the document render was handed
       footnotes.py          # markdown projection
       sidecar.py            # sidecar reader/writer over kernel/artifact.py's format
+      theme.py              # the artifact's look: TOML in, one CSS override block out
+      themes/               # the bundled themes, as the same files a user would write
       _text.py              # presentation helpers both renderers share (status prose, elision)
       cli.py                # `render` — mounted by the top-level cli
-    cli.py                  # typer entry; owns discovery, sessions, and the mounts
+    fetch.py                # the whole network surface: one bounded GET over http(s), stdlib only
+    cli.py                  # typer entry; owns discovery, sessions, the network, and the mounts
   skills/
     backdraft/SKILL.md            # front-walk
     backdraft-backfill/SKILL.md
@@ -134,10 +138,15 @@ CREATE TABLE documents (
   sha256 TEXT NOT NULL UNIQUE,
   path TEXT NOT NULL,            -- as given at ingest; informational
   filename TEXT NOT NULL,
-  media_type TEXT NOT NULL,      -- 'pdf' | 'xlsx' | 'xls' | 'csv' | 'docx' | 'pptx' | 'image' | 'text'
+  media_type TEXT NOT NULL,      -- 'pdf' | 'xlsx' | 'xls' | 'csv' | 'docx' | 'pptx' | 'image' | 'html' | 'text'
   created_at TEXT NOT NULL       -- ISO-8601 UTC, everywhere
 );
 
+CREATE TABLE document_meta (
+  document_id INTEGER PRIMARY KEY REFERENCES documents(id),
+  meta TEXT NOT NULL             -- JSON provenance; a fetched page carries {url, fetched_at}. Never citation identity.
+);
+
 CREATE TABLE extractions (
   id INTEGER PRIMARY KEY,
   document_id INTEGER NOT NULL REFERENCES documents(id),
@@ -226,7 +235,7 @@ class Extractor(Protocol):
     def extract(self, path: Path, config: dict) -> Iterator[ExtractedPage]: ...
 ```
 
-Registered in a plain dict; `ingest --extractor auto` picks the first `can_handle`, built-ins ordered `xlsx, xls, csv, docx, pptx, pdf-text, image, text` (an extractor whose optional extra is missing is skipped, not fatal). `vlm` is pdf→images→VLM per page, OpenAI-compatible client, model configurable; its deps ship by default and the backdraft-scoped key gates use. **For PDFs, `auto` prefers `vlm` when it is ready (importable, key configured)** (glossy layouts and info boxes extract badly from the text layer, and the snapshot is the receipt); otherwise it falls back to `pdf-text` and prints a one-line nudge naming the VLM option. An explicit `--extractor` always wins. The sheet representation (shared by xlsx, xls, csv): `Row | A | B...` header, `[B10] value` prefixes, dimension caps, inflated-sheet placeholder pages; cell values are never rounded, since the snapshot is the receipt a claim is traced against. `docx` synthesizes pages from heading sections (the document's smallest present outline level among Heading 1–2 splits; no headings → one page), so locators stay `pN.cM`. `pptx` is one page per slide, text floor only; ingest notes that slide visuals need the PDF-export path. `pdf-text` reconstructs paragraph breaks from line geometry before handing the page over, so the chunker's blank-line rule has something to fire on.
+Registered in a plain dict; `ingest --extractor auto` picks the first `can_handle`, built-ins ordered `xlsx, xls, csv, docx, pptx, pdf-text, image, html, text` (an extractor whose optional extra is missing is skipped, not fatal). `vlm` is pdf→images→VLM per page, OpenAI-compatible client, model configurable; its deps ship by default and the backdraft-scoped key gates use. **For PDFs, `auto` prefers `vlm` when it is ready (importable, key configured)** (glossy layouts and info boxes extract badly from the text layer, and the snapshot is the receipt); otherwise it falls back to `pdf-text` and prints a one-line nudge naming the VLM option. An explicit `--extractor` always wins. The sheet representation (shared by xlsx, xls, csv): `Row | A | B...` header, `[B10] value` prefixes, dimension caps, inflated-sheet placeholder pages; cell values are never rounded, since the snapshot is the receipt a claim is traced against. `docx` synthesizes pages from heading sections (the document's smallest present outline level among Heading 1–2 splits; no headings → one page), so locators stay `pN.cM`. `pptx` is one page per slide, text floor only; ingest notes that slide visuals need the PDF-export path. `pdf-text` reconstructs paragraph breaks from line geometry before handing the page over, so the chunker's blank-line rule has something to fire on. `html` is one page for the whole document, named by its `<title>`: block elements end a block so the chunker's blank-line rule fires, `script`/`style`/`noscript`/`template`/`svg` are dropped whole, lists become `- `/`1. ` lines in a single block, and tables become pipe tables as `docx` renders them. It is a parse and not a readability heuristic — no boilerplate stripping, since a guess that changes between two versions of a site moves anchors. Decoding is a pure function of the bytes (BOM, then `<meta charset>`, then UTF-8 with replacement), so a saved page and a fetched one extract identically.
 
 ## Gate (gate/)
 
@@ -273,17 +282,20 @@ All switches **default off** (`--check` opts in). Verdicts are recorded evidence
 
 - `render <authored.md> --to html` → single self-contained file: the bound doc, click/hover per claim → receipt card (quote, doc, locator, hashes, verdicts, drift diff if any), unresolved list rendered as a visible section, embedded JSON island with the full sidecar + `$format` string + legend (subtext pattern: the artifact teaches its own decoding). No external requests; no page images in v0.
 - `--to footnotes` → plain markdown projection. `--to json` → sidecar alone.
+- `--theme <name|file>` restyles the html artifact — a TOML file of colors, font stacks and heading treatment, resolved by `render/theme.py` into one CSS block emitted *after* the stylesheet (so an unthemed render is byte-identical to one from before themes existed). Precedence: `--theme` > project `.backdraft/theme.toml` > XDG `~/.config/backdraft/theme.toml` > built-in. `backdraft theme list` names the bundled themes and which file is in effect; `theme show <name|file>` prints one, validated, so redirecting `show default` bootstraps a commented starting file and `show ./mine.toml` lints one. Display only: no token, receipt or record moves, and layout is outside the allowlist.
 - Artifact format string: `backdraft/artifact-v1` (prose spec in spec/artifact.md).
 
 ## CLI (cli.py — typer)
 
 ```
 backdraft init                      # create .backdraft/, print status
-backdraft ingest <files...> [--extractor auto] [--slug S] [--config k=v]
+backdraft ingest <sources...> [--extractor auto] [--slug S] [--config k=v]
+                                    # a source is a path or an http(s) URL
 backdraft ls | backdraft read ...   # gate, above
 backdraft search "<query>" [--in slug]
 backdraft bind <doc.md> [--session S] [--check ...] [--mode ...]
-backdraft render <doc.md> [--to html|footnotes|json] [-o out]
+backdraft render <doc.md> [--to html|footnotes|json] [-o out] [--theme name|file]
+backdraft theme [list|show <name|file>]
 backdraft export [--out registry.json]
 backdraft session [start|show] 
 ```
@@ -323,10 +335,15 @@ class Registry:
 
     # write side (W1)
     def ingest(self, path: Path, *, extractor: str | None = None,
-               slug: str | None = None, config: dict | None = None) -> Document: ...
+               slug: str | None = None, config: dict | None = None,
+               url: str | None = None, fetched_at: str | None = None) -> Document: ...
         # runs extractor → new extraction generation, pages, eager anchors, FTS rows.
         # Token carry-over per re-ingest semantics; hash-collision extension steps TOKEN_HASH_LENGTHS.
         # Sheet names sanitized to the sheetref charset here, at ingest.
+        # `url` marks `path` as a snapshot staged from the web: the document records the
+        # URL as its path, carries {url, fetched_at} as meta, and matches an earlier
+        # fetch by URL rather than by the temporary file. Identity stays the bytes.
+        # The registry never fetches; `fetch.py` does, and the CLI calls it.
 
     # read side (W1 implements; W2/W3 consume)
     def documents(self) -> list[Document]: ...
@@ -372,4 +389,4 @@ class Registry:
 
 ## Addendum B — CLI assembly
 
-Top-level `cli.py` (W1) owns: typer app, registry discovery (nearest `.backdraft/` walking up from cwd; `BACKDRAFT_HOME` override), session resolution (`--session` flag > `BACKDRAFT_SESSION` env > default session), and the `init` / `ingest` / `ls` / `export` / `snapshot-pages` commands. After each ingest, `cli.py` captures page snapshots for any PDF whose extraction carries none — the text-layer path, since `vlm` stores its own — via `extract/snapshots.py` (poppler through pdf2image, `snapshot-pages`' internals). It sits outside the extractor so `pdf-text` stays deterministic regardless of whether poppler is installed, and it is best-effort: a `SnapshotError` leaves ingest at exit 0 with a one-line note naming `snapshot-pages` as the backfill. It mounts sub-apps `gate/cli.py` (W2: `read`, `search`, `session`), `bind/cli.py` (W3: `bind`), `render/cli.py` (W4: `render`) — each exposes `app = typer.Typer()`; the top level mounts each inside a try/except ImportError so partial merges still run. The `backdraft` console script is declared by W1.
+Top-level `cli.py` (W1) owns: typer app, registry discovery (nearest `.backdraft/` walking up from cwd; `BACKDRAFT_HOME` override), session resolution (`--session` flag > `BACKDRAFT_SESSION` env > default session), and the `init` / `ingest` / `ls` / `export` / `snapshot-pages` commands. After each ingest, `cli.py` captures page snapshots for any PDF whose extraction carries none — the text-layer path, since `vlm` stores its own — via `extract/snapshots.py` (poppler through pdf2image, `snapshot-pages`' internals). It sits outside the extractor so `pdf-text` stays deterministic regardless of whether poppler is installed, and it is best-effort: a `SnapshotError` leaves ingest at exit 0 with a one-line note naming `snapshot-pages` as the backfill. `cli.py` also owns the network for the same reason: an `ingest` argument with a scheme is fetched through `fetch.py` (stdlib `urllib`, http and https only, redirects followed, 32 MiB cap) and staged in a temporary file named for the content type the server declared — which is what selects the extractor — so extractors stay pure functions of bytes and the registry opens no sockets. It mounts sub-apps `gate/cli.py` (W2: `read`, `search`, `session`), `bind/cli.py` (W3: `bind`), `render/cli.py` (W4: `render`) — each exposes `app = typer.Typer()`; the top level mounts each inside a try/except ImportError so partial merges still run. The `backdraft` console script is declared by W1.