backdraft 0.2.1__tar.gz → 0.4.0__tar.gz

backdraft-0.4.0/.github/workflows/ci.yml

@@ -0,0 +1,14 @@
+name: ci
+on:
+  push:
+    branches: [main]
+  pull_request:
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: uv sync --all-extras
+      - run: uv run pytest
+      - run: uv build

backdraft-0.4.0/.github/workflows/release.yml

@@ -0,0 +1,23 @@
+name: release
+on:
+  release:
+    types: [published]
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: uv sync --all-extras
+      - run: uv run pytest
+  publish:
+    needs: test
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write  # trusted publishing: OIDC, no stored token
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+      - run: uv build
+      - uses: pypa/gh-action-pypi-publish@release/v1

backdraft-0.4.0/.github/workflows/site.yml

@@ -0,0 +1,26 @@
+name: site
+on:
+  push:
+    branches: [main]
+    paths: ["site/**"]
+  workflow_dispatch:
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+concurrency:
+  group: pages
+  cancel-in-progress: true
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/upload-pages-artifact@v3
+        with:
+          path: site/
+      - id: deployment
+        uses: actions/deploy-pages@v4

{backdraft-0.2.1 → backdraft-0.4.0}/.gitignore

@@ -22,3 +22,4 @@ uv.lock.bak
 
 # Claude Code session state and agent worktrees
 .claude/
+.private/

{backdraft-0.2.1 → backdraft-0.4.0}/DESIGN.md

@@ -1,12 +1,12 @@
 # Backdraft — Design
 
-Status: v0 design, 2026-07-27. Decisions below are made unless marked open. Research backing this is in [research/20260727-research-findings.md](research/20260727-research-findings.md).
+Status: v0 design, 2026-07-27. Decisions below are made unless marked open.
 
 ## The point
 
-Drop-in provenance for factual claims: one click from any claim to the evidence behind it. Extracted from what was built inside the HighRoad app, redesigned from scratch, released standalone. The working problem statement is **verification cost** — an analyst who can't show where a number came from either re-reads the document or doesn't use the output, and both outcomes destroy the value of extraction.
+Drop-in provenance for factual claims: one click from any claim to the evidence behind it. The working problem statement is **verification cost** — an analyst who can't show where a number came from either re-reads the document or doesn't use the output, and both outcomes destroy the value of extraction.
 
-Not a port. The app proved the product (citations worked great from the frontend) but its mechanism was path-dependent: random UUIDs, four citation grammars, four resolver copies, silent failure everywhere, anchors that die on reprocessing. We port the ideas, not the code.
+Designed from first principles against the failure modes that kill citation systems in practice: random anchor ids, competing citation grammars, duplicated resolvers, silent failure, anchors that die on reprocessing.
 
 ## Principles
 
@@ -14,7 +14,7 @@ Not a port. The app proved the product (citations worked great from the frontend
 2. **Content-addressed everything.** Anchors derive their identity from document content + location, not from a run. Reprocessing the same bytes yields the same anchors; citations survive.
 3. **The receipt travels with the claim.** An anchor is not a pointer — it is a pointer plus captured evidence: verbatim snippet, content hash, location. Defensible without reopening the source.
 4. **Failures are data.** Unresolvable tokens, drifted sources, unmatched claims are first-class records in reports and artifacts — never warn-and-drop.
-5. **Coordinates live in-band.** The app's only working sub-page attribution was Excel, because `[B10]` markers sat in the text the model read. Human-readable location in the token; side-channel geometry rots (bbox was plumbed through six layers and read by nothing).
+5. **Coordinates live in-band.** Sub-page attribution works when markers like `[B10]` sit in the text the model reads. Human-readable location in the token; side-channel geometry rots (a bbox plumbed through six layers ends up read by nothing).
 6. **The format is the product.** Substrates (skill, CLI, SDK, viewer) are thin shells over one spec + one library. Artifacts are self-describing and outlive any substrate (subtext's `$format` + embedded legend pattern).
 
 ## Architecture: three layers
@@ -27,13 +27,13 @@ Not a port. The app proved the product (citations worked great from the frontend
 ### 2. The library / CLI (the system)
 Four verbs:
 - **`ingest`** — deterministically anchor source documents (PDF/xlsx/md/…) into the registry: extract, locate, snapshot snippets, hash.
-- **`read` / `search`** — the gate. Emit token-marked context for a model. Search results are themselves mintable evidence (fixing pipeline-v2's gap where search returned nothing citable and forced a page read).
+- **`read` / `search`** — the gate. Emit token-marked context for a model. Search results are themselves mintable evidence (search that returns nothing citable forces a page read purely to obtain an anchor).
 - **`bind`** — postprocess a written document: resolve tokens, run enabled verification switches, backfill surface forms (doc names, pages, quotes), emit sidecar claims file + binding report.
 - **`render`** — produce the clickable artifact.
 
 ### 3. Substrates (sequenced)
 1. **Skill + CLI, together** (first release). The CLI is the system; the skill is ~a page instructing an agent to use it. Ships both the agent-native wedge and the zero-integration backfill demo.
-2. **SDK middleware** — wrap a model call: minted context in, bound citations out. The app-developer drop-in (LoanCore validation-layer shape).
+2. **SDK middleware** — wrap a model call: minted context in, bound citations out. The drop-in for application developers.
 3. **Viewer** — the artifact carries its own single-file viewer; a component version for embedding comes later.
 
 ## The registry
@@ -43,7 +43,7 @@ Four verbs:
 Tables (shape, not final DDL):
 - `documents` — path, display name, **slug**, content sha256, media type, ingest metadata.
 - `anchors` — token, document, locator, verbatim snippet, snippet sha256, created-by (ingest run).
-- `ledger` — session id, token, when shown. Records every token minted into context. Bind distinguishes "cited what you saw" from "cited a valid token you were never shown" — a caught hallucination class the app couldn't express.
+- `ledger` — session id, token, when shown. Records every token minted into context. Bind distinguishes "cited what you saw" from "cited a valid token you were never shown" — a caught hallucination class most systems cannot express.
 - `claims` (written at bind) — claim span text, source doc, citation tokens, verification records.
 
 FTS5 backs `search`. No server, no tenancy — single-analyst/single-project scope; multi-user is a substrate concern, not a registry concern.
@@ -54,7 +54,7 @@ FTS5 backs `search`. No server, no tenancy — single-analyst/single-project sco
 
 - `slug` — short kebab doc identifier assigned at ingest, unique per registry (`t12-audit`, `rent-roll-2025`).
 - `locator` — media-native:
-  - PDF/text: `p8` (page), `p8.c3` (chunk ordinal within page; chunking is deterministic with an explicit within-page ordinal — the column the app never had)
+  - PDF/text: `p8` (page), `p8.c3` (chunk ordinal within page; chunking is deterministic with an explicit within-page ordinal)
   - Excel: `Rent!B10`, `Rent!B10:C12` (sheet + cell/range, spreadsheet-native)
   - Prose files: heading-path or line-range forms (open detail)
 - `hash` — first 4+ hex of sha256 of the normalized snippet. Catches transcription typos, detects drift after re-ingest, and makes the token content-addressed. Registry extends length on collision.
@@ -65,11 +65,11 @@ Reserved extension — **declared derivations**: a claim whose value appears in
 
 ## The gate (context construction)
 
-Ported ideas from pipeline-v2's read tool, rebuilt local:
+The read tool's shape:
 - Unified dispatch: list docs → per-doc TOC of page/sheet summaries → page/range read → batch read. Short slugs everywhere.
 - Progressive disclosure with continuation, plus search hints.
-- **Excel**: sheet-per-page markdown table with `Row | A | B | …` headers and `[B10] value` cell prefixes — the representation that made Excel the app's only working sub-page attribution. Hardening kept: dimension caps, inflated-sheet placeholders, trimming.
-- Fixes over the app: search mints citable snippet tokens; Excel long-context gets structural navigation (detected regions/ranges) instead of char-offset slicing (open design area).
+- **Excel**: sheet-per-page markdown table with `Row | A | B | …` headers and `[B10] value` cell prefixes — the representation that makes cell-level attribution work. Hardening: dimension caps, inflated-sheet placeholders, trimming.
+- Search mints citable snippet tokens; Excel long-context gets structural navigation (detected regions/ranges) rather than char-offset slicing (open design area).
 
 The skill's core instruction is a substitution: for source documents, use `backdraft read`/`search`, not raw Read/Grep.
 
@@ -97,11 +97,11 @@ Independent checks per citation, recorded as graded evidence, never gating gener
 | 3 | `entail` | Model judge: does snippet support claim — tiny closed question, batchable, async | qualitative |
 | 4 | `recompute` | Deterministic re-execution of declared derivations | derived values |
 
-Rationale for default-off: out of the box this is pure provenance (one click to the receipt); verification is opt-in per claim class. Rung-pass rates aggregated per run/model double as an eval substrate (the Clarion use).
+Rationale for default-off: out of the box this is pure provenance (one click to the receipt); verification is opt-in per claim class. Rung-pass rates aggregated per run/model double as an eval substrate.
 
 ## Modes
 
-- **Front-walk** — citations minted during generation; bind resolves against the session ledger. The high-quality integrated path (what the app did).
+- **Front-walk** — citations minted during generation; bind resolves against the session ledger. The high-quality integrated path.
 - **Backfill** — an existing document + ingested sources; for each claim: search → propose anchors → bind via the fuzzier strategies (overlap, value-trace, structured lookup). Unmatched claims land in an explicit unresolved list, not silent non-attribution. Zero integration required — the demo wedge.
 
 Same anchors, same bind, same artifact out.
@@ -127,20 +127,23 @@ Self-contained HTML (plus the machine-readable sidecar): **quotes + receipts + e
 | 2026-07-27 | CLI unified on the gate's patterns: shared `cli_context` (constants, discovery, sessions, one error guard mapping `BackdraftError` → exit 1), `Annotated` params, relative imports. Test fakes renamed so the real and fake registries cannot be confused. |
 | 2026-07-27 | File naming is part of the artifact format, owned by `kernel/artifact.py`: `<stem>.bound.md`, `<stem>.backdraft.json`, `<stem>.backdraft.html`, `<stem>.footnotes.md`. |
 | 2026-07-27 | `bind --check entail` without the extra records `skip` verdicts carrying the reason — verifiers never gate, including at the CLI layer. |
-| 2026-07-27 | **Supersedes "VLM when key present":** credentials are explicit. Ambient provider keys (`OPENAI_API_KEY`, `OPENROUTER_API_KEY`, `ANTHROPIC_API_KEY`) are never read; a credential reaches backdraft only via `--config`, a `BACKDRAFT_*` variable, or `.backdraft/env` (`init` writes a template). `vlm_ready` — the `auto` gate — is therefore equivalent to consent. Decided after a real unapproved-spend incident during the Darnell trial: presence of a generic key in the environment is not consent to spend it or send documents to its provider. |
+| 2026-07-27 | **Supersedes "VLM when key present":** credentials are explicit. Ambient provider keys (`OPENAI_API_KEY`, `OPENROUTER_API_KEY`, `ANTHROPIC_API_KEY`) are never read; a credential reaches backdraft only via `--config`, a `BACKDRAFT_*` variable, or `.backdraft/env` (`init` writes a template). `vlm_ready` — the `auto` gate — is therefore equivalent to consent. Decided after a real unapproved-spend incident during the first field trial: presence of a generic key in the environment is not consent to spend it or send documents to its provider. |
 | 2026-07-28 | **Reader doctrine (supersedes "no script"):** the artifact is designed for two readers — the writer verifying the AI's work, and a normal person the document is sent to — in three disclosure layers: the document (editorial typography, quiet marks), the footnote (click a claim → the source's own words, named and paged — no tokens, no hashes, no jargon), and the record (one deliberate step to the full machine-readable layer). **Success is silent; only failure speaks**: a fully-resolved artifact looks like a well-made document, while unresolved citations announce themselves before the reader invests belief. The load-bearing constraint is *no network, single file* — enforced by a CSP `meta` tag (`default-src 'none'`) — not *no script*: inline JS is allowed as progressive enhancement over a CSS-only baseline that survives script-stripping. Decided after field use: the v1 artifact rendered success and failure at equal volume and read as machinery, not a document. |
-| 2026-07-28 | **Evidence embeds in the artifact (supersedes "no page images in v0"):** the footnote layer carries the evidence itself, not just the quote — the cited page as an image, the cited cell in a mini-grid window (neighbors, row labels, column headers). Bounded by *what is cited*, never the corpus, so a memo citing ten pages costs ~2–3 MB; `--lean` skips images. Evidence is assembled at **bind** (the step that has the registry) and travels in the sidecar; render stays registry-blind and the artifact stays reproducible from the two files a reader was handed. For the VLM path, the page image the model was shown is stored at **ingest** — for a non-deterministic extractor the input pixels are the only reproducible snapshot, so the image completes the receipt chain (claim → quote → page). Reference shapes ported from embed-ses `generate-preview`: WebP quality 85 for pages; sheet windows as compact JSON with typed cell values and column-letter headers. |
-| 2026-07-28 | **Exhibits: the evidence-first inversion.** The artifact carries a second reading mode: every source document with the data points cited from it, each with its evidence context and backlinks to the claims that lean on it — "this memo rests on N data points across M documents." Ranking is deterministic in v1 (an anchor cited by three claims outranks one cited once); model-judged importance is a later switch. This is the skeptical reader's entry point and the writer's spot-check checklist. |
+| 2026-07-28 | **Evidence embeds in the artifact (supersedes "no page images in v0"):** the footnote layer carries the evidence itself, not just the quote — the cited page as an image, the cited cell in a mini-grid window (neighbors, row labels, column headers). Bounded by *what is cited*, never the corpus, so a memo citing ten pages costs ~2–3 MB; `--lean` skips images. Evidence is assembled at **bind** (the step that has the registry) and travels in the sidecar; render stays registry-blind and the artifact stays reproducible from the two files a reader was handed. For the VLM path, the page image the model was shown is stored at **ingest** — for a non-deterministic extractor the input pixels are the only reproducible snapshot, so the image completes the receipt chain (claim → quote → page). Reference shapes: WebP quality 85 for pages; sheet windows as compact JSON with typed cell values and column-letter headers. |
+| 2026-07-28 | **Exhibits: the evidence-first inversion.** The artifact carries a second reading mode: every source document with the data points cited from it, each with its evidence context and backlinks to the claims that lean on it — "this memo rests on N data points across M documents." Ranking is deterministic in v1 (an anchor cited by three claims outranks one cited once); model-judged importance is a later switch. This is the skeptical reader's entry point and the writer's spot-check checklist. *(2026-07-29: parked in ROADMAP — file-size and design-restraint concerns; needs a deliberate design pass and a real audience first.)* |
+| 2026-07-29 | **Office formats extract natively, never through conversion.** DOCX and PPTX are read straight from their XML (python-docx / python-pptx, core deps). Converting to PDF was rejected on principle: conversion is rendering, so it needs a layout engine (LibreOffice, MS Office, or a cloud API — a 700MB dependency, a platform lock, or documents leaving the machine), and converter-dependent pagination would break content-addressed locator stability. DOCX pages are heading sections: the split level is the smallest outline level present among {Heading 1, Heading 2}; preamble is section 1; no headings → one section — locators stay `pN.cM`, no grammar change. PPTX pages are slides (slide N is slide N in every renderer). The text floor deliberately misses slide visuals; the ingest note says so and names the path that captures them — the user exporting the deck to PDF with the real renderer they already have, ingested through the vision extractor — so a calling agent can relay it. |
+| 2026-07-29 | **The vision extractor ships by default (supersedes "[vlm] as extra").** `openai` + `pdf2image` moved into core dependencies; `[vlm]` remains as an empty compat alias. Rationale: the VLM path is the recommended path for real PDFs, and an extra made the recommended path opt-in. The consent boundary does not move — installing deps was never consent; the backdraft-scoped `BACKDRAFT_VLM_API_KEY` is, and ambient provider keys stay unread. There is no slim install: pip extras can only add, never subtract; a separate core distribution is possible later if anyone asks. poppler stays a runtime capability check. |
+| 2026-07-29 | **Renderer decomposed; Python is the formatting authority.** `render/html.py` (1,300 lines) became the `render/html/` package (fmt, text, components, assets, page), byte-identical at the seam. The deliberate `fmt_cell` (Python) / `fmtCell` (JS) duplication is now pinned by one shared vector table driving both tests, with the JS run under node when available. The JS was found diverging on exact rounding ties (`toFixed` rounds half-up; Python rounds half-to-even on the double's value) and was rewritten to round exactly — Python defines correct, JS matches it. The artifact emits minified CSS/JS and unindented markup; the `$legend` and sidecar JSON stay readable in view-source (spec constraint). Snapshot budget knobs `BACKDRAFT_SNAPSHOT_QUALITY`/`BACKDRAFT_SNAPSHOT_MAX_HEIGHT` tune page-image weight at ingest — display only, tokens never derive from pixels. |
 
 ## Open
 
-Queued work lives in ROADMAP.md (exhibits, region maps, formats, `bd:calc`,
-substrates); this list is questions still awaiting a decision:
+Queued and parked work lives in ROADMAP.md; this list is questions still
+awaiting a decision:
 
 - Re-bind/orphan pass on re-ingest of changed docs (chunk ordinal drift)
-- Prose-file locator forms; slug assignment/collision rules (the trailing-hyphen
-  truncation from the Darnell trial lives here)
+- Slug assignment/collision rules (the trailing-hyphen truncation from the
+  first field trial lives here)
 - Distribution: published to PyPI as `backdraft` 0.2.0 (2026-07-28); domain
   backdraft.dev acquired; still open: skill registration, deploying site/
-- First external user (LoanCore validation layer vs Clarion eval substrate vs
+- First external user (design-partner validation layer vs eval substrate vs
   open release feedback)

{backdraft-0.2.1 → backdraft-0.4.0}/PKG-INFO

@@ -1,18 +1,26 @@
 Metadata-Version: 2.4
 Name: backdraft
-Version: 0.2.1
+Version: 0.4.0
 Summary: Drop-in provenance for factual claims: one click from any claim to the evidence behind it.
+Project-URL: Homepage, https://backdraft.dev
+Project-URL: Documentation, https://backdraft.dev/docs.html
+Project-URL: Repository, https://github.com/spencerbraun/backdraft
+Project-URL: Roadmap, https://github.com/spencerbraun/backdraft/blob/main/ROADMAP.md
 License: MIT
 License-File: LICENSE
 Requires-Python: >=3.13
+Requires-Dist: openai>=1.0
 Requires-Dist: openpyxl>=3.1
+Requires-Dist: pdf2image>=1.17
 Requires-Dist: pdfplumber>=0.11
+Requires-Dist: python-docx>=1.1
+Requires-Dist: python-pptx>=1.0
 Requires-Dist: typer>=0.12
 Provides-Extra: entail
 Requires-Dist: anthropic>=0.40; extra == 'entail'
 Provides-Extra: vlm
-Requires-Dist: openai>=1.0; extra == 'vlm'
-Requires-Dist: pdf2image>=1.17; extra == 'vlm'
+Provides-Extra: xls
+Requires-Dist: python-calamine>=0.3; extra == 'xls'
 Description-Content-Type: text/markdown
 
 <img src="assets/backdraft-mark.svg" width="56" alt="">
@@ -40,12 +48,14 @@ Docs live at [backdraft.dev](https://backdraft.dev), one page for people, and
 ## 60 seconds
 
 ```bash
-uv tool install "backdraft[vlm]"
+uv tool install backdraft
 cd demo
 ```
 
 **Ingest** the sources. Every anchor is minted here, chunks for PDF pages,
-cells for spreadsheet sheets.
+cells for spreadsheet sheets. Formats: PDF, XLSX/XLSM, XLS, CSV/TSV, DOCX,
+PPTX, images (png, jpeg, tiff, through the vision model), plain text and
+Markdown.
 
 ```console
 $ backdraft init
@@ -55,7 +65,7 @@ documents: 0
 $ backdraft ingest sources/t12-summary.pdf sources/underwriting-model.xlsx
 t12-summary  t12-summary.pdf  pdf  3 pages
 underwriting-model  underwriting-model.xlsx  xlsx  2 pages
-note: extracted with pdf-text (the embedded text layer). Glossy or scanned PDFs extract better through a vision model: install backdraft[vlm] and set BACKDRAFT_VLM_API_KEY in .backdraft/env.
+note: extracted with pdf-text (the embedded text layer). Glossy or scanned PDFs extract better through a vision model: set BACKDRAFT_VLM_API_KEY in .backdraft/env.
 ```
 
 **Read.** The page arrives with a citable name over each chunk. This is the whole
@@ -184,21 +194,33 @@ confidential corpora**.
 ## Install
 
 ```bash
-uv tool install "backdraft[vlm]"   # CLI, recommended form
-uv add backdraft                   # as a dependency
-pip install backdraft              # or plain pip
+uv tool install backdraft   # CLI, recommended form
+uv add backdraft            # as a dependency
+pip install backdraft       # or plain pip
 ```
 
-Python 3.13+. Two extras:
+Python 3.13+. The vision extractor ships by default, and it is the
+recommended path for real PDFs (glossy layouts, info boxes, scans): a vision
+model reads each page and its clean representation becomes the receipt.
+Installing it is not consent to use it. `--extractor auto` prefers it only on
+explicit, backdraft-scoped consent: `BACKDRAFT_VLM_API_KEY` (env or
+`.backdraft/env`, `backdraft init` writes a template). Ambient
+`OPENAI_API_KEY`/`OPENROUTER_API_KEY` are deliberately never read; a generic
+key in the environment is not consent to send documents to its provider. The
+default provider is OpenRouter running Gemini 3.1 Flash Lite; direct OpenAI is
+base_url + model, set explicitly. Without a scoped key, `auto` falls back to
+the keyless text layer and says so. PDF page rendering needs poppler,
+installed separately.
+
+Two extras:
 
 | Extra | Adds | For |
 |---|---|---|
-| `[vlm]` | `openai`, `pdf2image` | the VLM extractor, the recommended path for real PDFs (glossy layouts, info boxes, scans): a vision model reads each page and its clean representation becomes the receipt. `--extractor auto` prefers it only on explicit, backdraft-scoped consent: `BACKDRAFT_VLM_API_KEY` (env or `.backdraft/env`, `backdraft init` writes a template). Ambient `OPENAI_API_KEY`/`OPENROUTER_API_KEY` are deliberately never read; a generic key in the environment is not consent to send documents to its provider. The default provider is OpenRouter running Gemini 3.1 Flash Lite; direct OpenAI is base_url + model, set explicitly. Without a scoped key, `auto` falls back to the keyless text layer and says so. Needs poppler installed separately |
 | `[entail]` | `anthropic` | `bind --check entail`, the model-judge verifier |
+| `[xls]` | `python-calamine` | legacy `.xls` workbooks, values only, rendered the same shape as xlsx |
 
-```bash
-pip install "backdraft[vlm,entail]"
-```
+`backdraft[vlm]` still installs: it is an empty compat alias from when the
+vision deps were an extra.
 
 From a checkout:
 
@@ -210,12 +232,12 @@ uv run pytest
 
 ## Skills
 
-Three agent skills ship in [`skills/`](skills). Copy the ones you want into your
-agent's skills directory, for Claude Code, `~/.claude/skills/` (personal) or
-`.claude/skills/` (per project):
+Three agent skills ship inside the package, and the CLI installs them:
 
 ```bash
-cp -r skills/backdraft skills/backdraft-backfill skills/backdraft-artifact ~/.claude/skills/
+backdraft skill install          # the writing skill, into ~/.claude/skills/
+backdraft skill install --all    # plus backfill and artifact-reading
+backdraft skill install --project  # into this repo's .claude/skills/
 ```
 
 | Skill | For |
@@ -244,8 +266,7 @@ implementation reads them and nothing else.
 ## Status
 
 v1. The registry format, the token grammar and the artifact format are pinned
-by prose specs and golden-file tests. What is deliberately queued, exhibits,
-Excel region maps, more formats, `bd:calc` derivations, is written down in
-[ROADMAP.md](ROADMAP.md).
+by prose specs and golden-file tests. What is queued, and what is deliberately
+parked with the objection written down, lives in [ROADMAP.md](ROADMAP.md).
 
 MIT licensed.

{backdraft-0.2.1 → backdraft-0.4.0}/README.md

@@ -23,12 +23,14 @@ Docs live at [backdraft.dev](https://backdraft.dev), one page for people, and
 ## 60 seconds
 
 ```bash
-uv tool install "backdraft[vlm]"
+uv tool install backdraft
 cd demo
 ```
 
 **Ingest** the sources. Every anchor is minted here, chunks for PDF pages,
-cells for spreadsheet sheets.
+cells for spreadsheet sheets. Formats: PDF, XLSX/XLSM, XLS, CSV/TSV, DOCX,
+PPTX, images (png, jpeg, tiff, through the vision model), plain text and
+Markdown.
 
 ```console
 $ backdraft init
@@ -38,7 +40,7 @@ documents: 0
 $ backdraft ingest sources/t12-summary.pdf sources/underwriting-model.xlsx
 t12-summary  t12-summary.pdf  pdf  3 pages
 underwriting-model  underwriting-model.xlsx  xlsx  2 pages
-note: extracted with pdf-text (the embedded text layer). Glossy or scanned PDFs extract better through a vision model: install backdraft[vlm] and set BACKDRAFT_VLM_API_KEY in .backdraft/env.
+note: extracted with pdf-text (the embedded text layer). Glossy or scanned PDFs extract better through a vision model: set BACKDRAFT_VLM_API_KEY in .backdraft/env.
 ```
 
 **Read.** The page arrives with a citable name over each chunk. This is the whole
@@ -167,21 +169,33 @@ confidential corpora**.
 ## Install
 
 ```bash
-uv tool install "backdraft[vlm]"   # CLI, recommended form
-uv add backdraft                   # as a dependency
-pip install backdraft              # or plain pip
+uv tool install backdraft   # CLI, recommended form
+uv add backdraft            # as a dependency
+pip install backdraft       # or plain pip
 ```
 
-Python 3.13+. Two extras:
+Python 3.13+. The vision extractor ships by default, and it is the
+recommended path for real PDFs (glossy layouts, info boxes, scans): a vision
+model reads each page and its clean representation becomes the receipt.
+Installing it is not consent to use it. `--extractor auto` prefers it only on
+explicit, backdraft-scoped consent: `BACKDRAFT_VLM_API_KEY` (env or
+`.backdraft/env`, `backdraft init` writes a template). Ambient
+`OPENAI_API_KEY`/`OPENROUTER_API_KEY` are deliberately never read; a generic
+key in the environment is not consent to send documents to its provider. The
+default provider is OpenRouter running Gemini 3.1 Flash Lite; direct OpenAI is
+base_url + model, set explicitly. Without a scoped key, `auto` falls back to
+the keyless text layer and says so. PDF page rendering needs poppler,
+installed separately.
+
+Two extras:
 
 | Extra | Adds | For |
 |---|---|---|
-| `[vlm]` | `openai`, `pdf2image` | the VLM extractor, the recommended path for real PDFs (glossy layouts, info boxes, scans): a vision model reads each page and its clean representation becomes the receipt. `--extractor auto` prefers it only on explicit, backdraft-scoped consent: `BACKDRAFT_VLM_API_KEY` (env or `.backdraft/env`, `backdraft init` writes a template). Ambient `OPENAI_API_KEY`/`OPENROUTER_API_KEY` are deliberately never read; a generic key in the environment is not consent to send documents to its provider. The default provider is OpenRouter running Gemini 3.1 Flash Lite; direct OpenAI is base_url + model, set explicitly. Without a scoped key, `auto` falls back to the keyless text layer and says so. Needs poppler installed separately |
 | `[entail]` | `anthropic` | `bind --check entail`, the model-judge verifier |
+| `[xls]` | `python-calamine` | legacy `.xls` workbooks, values only, rendered the same shape as xlsx |
 
-```bash
-pip install "backdraft[vlm,entail]"
-```
+`backdraft[vlm]` still installs: it is an empty compat alias from when the
+vision deps were an extra.
 
 From a checkout:
 
@@ -193,12 +207,12 @@ uv run pytest
 
 ## Skills
 
-Three agent skills ship in [`skills/`](skills). Copy the ones you want into your
-agent's skills directory, for Claude Code, `~/.claude/skills/` (personal) or
-`.claude/skills/` (per project):
+Three agent skills ship inside the package, and the CLI installs them:
 
 ```bash
-cp -r skills/backdraft skills/backdraft-backfill skills/backdraft-artifact ~/.claude/skills/
+backdraft skill install          # the writing skill, into ~/.claude/skills/
+backdraft skill install --all    # plus backfill and artifact-reading
+backdraft skill install --project  # into this repo's .claude/skills/
 ```
 
 | Skill | For |
@@ -227,8 +241,7 @@ implementation reads them and nothing else.
 ## Status
 
 v1. The registry format, the token grammar and the artifact format are pinned
-by prose specs and golden-file tests. What is deliberately queued, exhibits,
-Excel region maps, more formats, `bd:calc` derivations, is written down in
-[ROADMAP.md](ROADMAP.md).
+by prose specs and golden-file tests. What is queued, and what is deliberately
+parked with the objection written down, lives in [ROADMAP.md](ROADMAP.md).
 
 MIT licensed.

backdraft-0.4.0/ROADMAP.md

@@ -0,0 +1,59 @@
+# Roadmap
+
+Sequenced intent. DESIGN.md records what was *decided*; this file records what
+is *queued*, so tabling something is an act of writing it down here rather than
+losing it. Items graduate by getting built and deleted.
+
+## Now
+
+Empty, deliberately. The 0.4.0 wave graduated the whole section: DOCX, PPTX,
+and XLS extractors, the shared sheet-extraction module, the renderer
+decomposition with pinned Python/JS format parity, minified artifact
+emission, and the snapshot budget knobs. The next item gets promoted out of
+Parked or Later on purpose, not by momentum.
+
+## Parked
+
+Deliberately not queued, each with the reason, so picking one up starts from
+the objection rather than rediscovering it.
+
+- **Exhibits view** — the evidence-first inversion. Risk: it grows the
+  artifact and pollutes a design whose strength is restraint. Needs a
+  deliberate design pass and a real audience before any code; not a
+  weekend-feature shape.
+- **Excel region maps and range reads** — the research notes exist, but no
+  confidence the deterministic detection cascade lands well on the first
+  try. Revisit when large-sheet navigation hurts in a real corpus, with
+  test workbooks in hand.
+- **`bd:calc` derivations** — reserved in the grammar and parsed as
+  malformed-on-purpose, which is enough for now. Full support complicates
+  binding and verification for one field-trial case.
+- **Merged-cell and frozen-pane rendering** — ingest captures `merged` and
+  `frozen` in sheet meta and keeps them (cheap, and capture is the part you
+  cannot backfill). Frozen-pane rendering is cut; merged-cell rendering
+  waits for a real sheet that renders wrong without it.
+
+## Later
+
+- **Living documents** — re-ingest, re-bind, and a drift-first report: "these
+  claims cite figures that changed; cited-then vs. now." The primitives
+  (generations, `drifted`, the word-diff) all exist; the missing piece is a
+  diff-shaped report and a demo.
+- **Substrates beyond the CLI** — SDK middleware for pipelines (the exit-code
+  and `--to json` contracts are already shaped for it), and a client-side
+  drag-drop viewer page (no upload — the file never leaves the reader's
+  machine).
+- **HTML pages and email (.eml)** — both arrive in diligence folders, both
+  have messy identity questions (what is "the source" of a web page?) that
+  deserve a decision row before code.
+- **Distribution** — published on PyPI; still queued: a skills-registry
+  listing when one becomes the standard channel.
+
+## Someday
+
+- **Hosted team features** — upload, access control, shared registries.
+  Deliberately parked: the product's trust story is the self-contained file,
+  and a hosted viewer inverts it. Revisit only as a distinct teams product.
+- **Entail at scale** — the model-judge verifier is wired (`[entail]` extra)
+  but has never been field-calibrated; needs a corpus and a rubric before it
+  is recommended anywhere.

{backdraft-0.2.1 → backdraft-0.4.0}/SPEC.md

@@ -44,10 +44,16 @@ backdraft/
     extract/
       base.py               # Extractor protocol + registry of extractors
       pdf_text.py           # pdfplumber text-layer; paragraph breaks rebuilt from line geometry
-      xlsx.py               # calamine/openpyxl → markdown table with [B10] refs (port the app's representation)
+      sheet.py              # the shared tabular representation: bounds caps, [B10] row rendering
+      xlsx.py               # openpyxl → sheet representation + styling meta
+      xls.py                # [xls] extra: calamine → sheet representation, values only
+      csv.py                # sniffed-dialect csv/tsv → sheet representation
+      docx.py               # python-docx; heading sections become pages
+      pptx.py               # python-pptx; one page per slide, text floor
       text.py               # md/txt passthrough
-      vlm.py                # [vlm] extra: zerox-minimal pattern (pdf→images→VLM per page)
-      vlm_settings.py       # provider/model/base-url resolution; stdlib-only, imports without the extra
+      image.py              # an image is a one-page document; VLM transcription
+      vlm.py                # pdf→images→VLM per page (deps ship by default; key gates use)
+      vlm_settings.py       # provider/model/base-url resolution; stdlib-only
     gate/
       reader.py             # ls / toc / read page-range → token-marked context; records ledger
       searcher.py           # FTS5 → minted snippet results; records ledger
@@ -59,7 +65,9 @@ backdraft/
         base.py             # Verifier protocol
         value_trace.py  overlap.py  recompute.py  entail.py   # entail behind [entail]
     render/
-      html.py               # single-file artifact, embedded viewer + legend
+      html/                 # single-file artifact, embedded viewer + legend
+                            #   fmt.py text.py components.py assets.py page.py
+                            #   fmt_vectors.py: one vector table drives fmt_cell (py) and fmtCell (js)
       markdown.py           # the authored document's markdown → HTML, stdlib only
       placement.py          # locating each claim's span in the document render was handed
       footnotes.py          # markdown projection
@@ -113,7 +121,7 @@ Deterministic, pure: `chunk(page_text) -> [Chunk(ordinal, text, start, end)]`.
 1. Split on blank lines: regex `\n\s*\n`.
 2. Merge forward: any segment < 200 chars merges into the following segment (the last merges backward).
 3. Split long: any segment > 2400 chars splits at the sentence boundary nearest each 1200-char multiple (sentence = terminal `.!?` + space + uppercase/digit start; no abbreviation table in v0 — splitting slightly wrong is fine, it only shifts chunk boundaries deterministically). A final piece under 200 chars merges backward, since merging ran before splitting and could not see it; a chunk is therefore under 200 chars only when it is the only chunk on its page.
-4. Ordinals `c1..cN` in order. No per-page rebalancing, no chunk-count caps (the app's balancing caused tail-loss bugs; deliberately dropped).
+4. Ordinals `c1..cN` in order. No per-page rebalancing, no chunk-count caps (rebalancing causes tail-loss bugs; deliberately avoided).
 
 Anchor identity for a chunk: (extraction, page number, ordinal). `start`/`end` are char offsets into the page text — stored, enabling future span/region features.
 
@@ -126,7 +134,7 @@ CREATE TABLE documents (
   sha256 TEXT NOT NULL UNIQUE,
   path TEXT NOT NULL,            -- as given at ingest; informational
   filename TEXT NOT NULL,
-  media_type TEXT NOT NULL,      -- 'pdf' | 'xlsx' | 'text'
+  media_type TEXT NOT NULL,      -- 'pdf' | 'xlsx' | 'xls' | 'csv' | 'docx' | 'pptx' | 'image' | 'text'
   created_at TEXT NOT NULL       -- ISO-8601 UTC, everywhere
 );
 
@@ -218,7 +226,7 @@ class Extractor(Protocol):
     def extract(self, path: Path, config: dict) -> Iterator[ExtractedPage]: ...
 ```
 
-Registered in a plain dict; `ingest --extractor auto` picks the first `can_handle`, built-ins ordered `xlsx, pdf-text, text`. `vlm` (extra) follows the zerox-minimal pattern — pdf→images→VLM per page, OpenAI-compatible client, model configurable. **For PDFs, `auto` prefers `vlm` when it is installed and an API key is configured** (glossy layouts and info boxes extract badly from the text layer, and the snapshot is the receipt); otherwise it falls back to `pdf-text` and prints a one-line nudge naming the VLM option. An explicit `--extractor` always wins. xlsx ports the app's representation: `Row | A | B...` header, `[B10] value` prefixes, dimension caps, inflated-sheet placeholder pages; cell values are never rounded, since the snapshot is the receipt a claim is traced against. `pdf-text` reconstructs paragraph breaks from line geometry before handing the page over, so the chunker's blank-line rule has something to fire on.
+Registered in a plain dict; `ingest --extractor auto` picks the first `can_handle`, built-ins ordered `xlsx, xls, csv, docx, pptx, pdf-text, image, text` (an extractor whose optional extra is missing is skipped, not fatal). `vlm` is pdf→images→VLM per page, OpenAI-compatible client, model configurable; its deps ship by default and the backdraft-scoped key gates use. **For PDFs, `auto` prefers `vlm` when it is ready (importable, key configured)** (glossy layouts and info boxes extract badly from the text layer, and the snapshot is the receipt); otherwise it falls back to `pdf-text` and prints a one-line nudge naming the VLM option. An explicit `--extractor` always wins. The sheet representation (shared by xlsx, xls, csv): `Row | A | B...` header, `[B10] value` prefixes, dimension caps, inflated-sheet placeholder pages; cell values are never rounded, since the snapshot is the receipt a claim is traced against. `docx` synthesizes pages from heading sections (the document's smallest present outline level among Heading 1–2 splits; no headings → one page), so locators stay `pN.cM`. `pptx` is one page per slide, text floor only; ingest notes that slide visuals need the PDF-export path. `pdf-text` reconstructs paragraph breaks from line geometry before handing the page over, so the chunker's blank-line rule has something to fire on.
 
 ## Gate (gate/)
 

{backdraft-0.2.1 → backdraft-0.4.0}/demo/generate_sources.py

@@ -17,7 +17,7 @@ from __future__ import annotations
 from pathlib import Path
 
 from openpyxl import Workbook
-from openpyxl.styles import Alignment, Font
+from openpyxl.styles import Alignment, Font, PatternFill
 from reportlab.lib.enums import TA_JUSTIFY
 from reportlab.lib.pagesizes import LETTER
 from reportlab.lib.styles import ParagraphStyle, getSampleStyleSheet
@@ -306,13 +306,42 @@ def build_xlsx(path: Path) -> None:
     for row in ASSUMPTIONS:
         assumptions.append(list(row))
 
+    # Styling, the way a sponsor's model actually looks: a dark header band,
+    # inputs in modeling blue, currency and percent number formats, a frozen
+    # header row. Values are untouched — the snapshot text and every token
+    # are identical with or without this block.
+    header_fill = PatternFill("solid", fgColor="FF1F4E79")
     for sheet in (rent, assumptions):
         for cell in sheet[1]:
-            cell.font = Font(bold=True)
+            cell.font = Font(bold=True, color="FFFFFFFF")
+            cell.fill = header_fill
             cell.alignment = Alignment(horizontal="left")
         widths = {"A": 30, "B": 14, "C": 14, "D": 16, "E": 16, "F": 28}
         for column, width in widths.items():
             sheet.column_dimensions[column].width = width
+        sheet.freeze_panes = "A2"
+
+    for row in range(2, 6):  # unit-mix money and occupancy columns
+        for column, fmt in (("C", "#,##0"), ("D", '"$"#,##0'), ("E", '"$"#,##0')):
+            rent[f"{column}{row}"].number_format = fmt
+        rent[f"F{row}"].number_format = "0.0%"
+    for cell in rent[7]:
+        cell.font = Font(bold=True)
+    rent["F7"].number_format = "0.0%"
+    rent["B9"].number_format = '"$"#,##0'
+    rent["B10"].number_format = "0.0%"
+    rent["B11"].number_format = '"$"#,##0'
+
+    input_font = Font(color="FF0000CC")  # modeling convention: inputs in blue
+    percent_rows = (2, 3, 5, 6, 7, 8, 12, 16)
+    money_rows = (9, 10, 11, 14)
+    for row in range(2, 17):
+        cell = assumptions[f"B{row}"]
+        cell.font = input_font
+        if row in percent_rows:
+            cell.number_format = "0.00%"
+        elif row in money_rows:
+            cell.number_format = '"$"#,##0'
 
     book.save(path)