backdraft 0.3.0__tar.gz → 0.4.0__tar.gz

{backdraft-0.3.0 → backdraft-0.4.0}/DESIGN.md

@@ -130,16 +130,19 @@ Self-contained HTML (plus the machine-readable sidecar): **quotes + receipts + e
 | 2026-07-27 | **Supersedes "VLM when key present":** credentials are explicit. Ambient provider keys (`OPENAI_API_KEY`, `OPENROUTER_API_KEY`, `ANTHROPIC_API_KEY`) are never read; a credential reaches backdraft only via `--config`, a `BACKDRAFT_*` variable, or `.backdraft/env` (`init` writes a template). `vlm_ready` — the `auto` gate — is therefore equivalent to consent. Decided after a real unapproved-spend incident during the first field trial: presence of a generic key in the environment is not consent to spend it or send documents to its provider. |
 | 2026-07-28 | **Reader doctrine (supersedes "no script"):** the artifact is designed for two readers — the writer verifying the AI's work, and a normal person the document is sent to — in three disclosure layers: the document (editorial typography, quiet marks), the footnote (click a claim → the source's own words, named and paged — no tokens, no hashes, no jargon), and the record (one deliberate step to the full machine-readable layer). **Success is silent; only failure speaks**: a fully-resolved artifact looks like a well-made document, while unresolved citations announce themselves before the reader invests belief. The load-bearing constraint is *no network, single file* — enforced by a CSP `meta` tag (`default-src 'none'`) — not *no script*: inline JS is allowed as progressive enhancement over a CSS-only baseline that survives script-stripping. Decided after field use: the v1 artifact rendered success and failure at equal volume and read as machinery, not a document. |
 | 2026-07-28 | **Evidence embeds in the artifact (supersedes "no page images in v0"):** the footnote layer carries the evidence itself, not just the quote — the cited page as an image, the cited cell in a mini-grid window (neighbors, row labels, column headers). Bounded by *what is cited*, never the corpus, so a memo citing ten pages costs ~2–3 MB; `--lean` skips images. Evidence is assembled at **bind** (the step that has the registry) and travels in the sidecar; render stays registry-blind and the artifact stays reproducible from the two files a reader was handed. For the VLM path, the page image the model was shown is stored at **ingest** — for a non-deterministic extractor the input pixels are the only reproducible snapshot, so the image completes the receipt chain (claim → quote → page). Reference shapes: WebP quality 85 for pages; sheet windows as compact JSON with typed cell values and column-letter headers. |
-| 2026-07-28 | **Exhibits: the evidence-first inversion.** The artifact carries a second reading mode: every source document with the data points cited from it, each with its evidence context and backlinks to the claims that lean on it — "this memo rests on N data points across M documents." Ranking is deterministic in v1 (an anchor cited by three claims outranks one cited once); model-judged importance is a later switch. This is the skeptical reader's entry point and the writer's spot-check checklist. |
+| 2026-07-28 | **Exhibits: the evidence-first inversion.** The artifact carries a second reading mode: every source document with the data points cited from it, each with its evidence context and backlinks to the claims that lean on it — "this memo rests on N data points across M documents." Ranking is deterministic in v1 (an anchor cited by three claims outranks one cited once); model-judged importance is a later switch. This is the skeptical reader's entry point and the writer's spot-check checklist. *(2026-07-29: parked in ROADMAP — file-size and design-restraint concerns; needs a deliberate design pass and a real audience first.)* |
+| 2026-07-29 | **Office formats extract natively, never through conversion.** DOCX and PPTX are read straight from their XML (python-docx / python-pptx, core deps). Converting to PDF was rejected on principle: conversion is rendering, so it needs a layout engine (LibreOffice, MS Office, or a cloud API — a 700MB dependency, a platform lock, or documents leaving the machine), and converter-dependent pagination would break content-addressed locator stability. DOCX pages are heading sections: the split level is the smallest outline level present among {Heading 1, Heading 2}; preamble is section 1; no headings → one section — locators stay `pN.cM`, no grammar change. PPTX pages are slides (slide N is slide N in every renderer). The text floor deliberately misses slide visuals; the ingest note says so and names the path that captures them — the user exporting the deck to PDF with the real renderer they already have, ingested through the vision extractor — so a calling agent can relay it. |
+| 2026-07-29 | **The vision extractor ships by default (supersedes "[vlm] as extra").** `openai` + `pdf2image` moved into core dependencies; `[vlm]` remains as an empty compat alias. Rationale: the VLM path is the recommended path for real PDFs, and an extra made the recommended path opt-in. The consent boundary does not move — installing deps was never consent; the backdraft-scoped `BACKDRAFT_VLM_API_KEY` is, and ambient provider keys stay unread. There is no slim install: pip extras can only add, never subtract; a separate core distribution is possible later if anyone asks. poppler stays a runtime capability check. |
+| 2026-07-29 | **Renderer decomposed; Python is the formatting authority.** `render/html.py` (1,300 lines) became the `render/html/` package (fmt, text, components, assets, page), byte-identical at the seam. The deliberate `fmt_cell` (Python) / `fmtCell` (JS) duplication is now pinned by one shared vector table driving both tests, with the JS run under node when available. The JS was found diverging on exact rounding ties (`toFixed` rounds half-up; Python rounds half-to-even on the double's value) and was rewritten to round exactly — Python defines correct, JS matches it. The artifact emits minified CSS/JS and unindented markup; the `$legend` and sidecar JSON stay readable in view-source (spec constraint). Snapshot budget knobs `BACKDRAFT_SNAPSHOT_QUALITY`/`BACKDRAFT_SNAPSHOT_MAX_HEIGHT` tune page-image weight at ingest — display only, tokens never derive from pixels. |
 
 ## Open
 
-Queued work lives in ROADMAP.md (exhibits, region maps, formats, `bd:calc`,
-substrates); this list is questions still awaiting a decision:
+Queued and parked work lives in ROADMAP.md; this list is questions still
+awaiting a decision:
 
 - Re-bind/orphan pass on re-ingest of changed docs (chunk ordinal drift)
-- Prose-file locator forms; slug assignment/collision rules (the trailing-hyphen
-  truncation from the first field trial lives here)
+- Slug assignment/collision rules (the trailing-hyphen truncation from the
+  first field trial lives here)
 - Distribution: published to PyPI as `backdraft` 0.2.0 (2026-07-28); domain
   backdraft.dev acquired; still open: skill registration, deploying site/
 - First external user (design-partner validation layer vs eval substrate vs

{backdraft-0.3.0 → backdraft-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: backdraft
-Version: 0.3.0
+Version: 0.4.0
 Summary: Drop-in provenance for factual claims: one click from any claim to the evidence behind it.
 Project-URL: Homepage, https://backdraft.dev
 Project-URL: Documentation, https://backdraft.dev/docs.html
@@ -9,14 +9,18 @@ Project-URL: Roadmap, https://github.com/spencerbraun/backdraft/blob/main/ROADMA
 License: MIT
 License-File: LICENSE
 Requires-Python: >=3.13
+Requires-Dist: openai>=1.0
 Requires-Dist: openpyxl>=3.1
+Requires-Dist: pdf2image>=1.17
 Requires-Dist: pdfplumber>=0.11
+Requires-Dist: python-docx>=1.1
+Requires-Dist: python-pptx>=1.0
 Requires-Dist: typer>=0.12
 Provides-Extra: entail
 Requires-Dist: anthropic>=0.40; extra == 'entail'
 Provides-Extra: vlm
-Requires-Dist: openai>=1.0; extra == 'vlm'
-Requires-Dist: pdf2image>=1.17; extra == 'vlm'
+Provides-Extra: xls
+Requires-Dist: python-calamine>=0.3; extra == 'xls'
 Description-Content-Type: text/markdown
 
 <img src="assets/backdraft-mark.svg" width="56" alt="">
@@ -44,13 +48,14 @@ Docs live at [backdraft.dev](https://backdraft.dev), one page for people, and
 ## 60 seconds
 
 ```bash
-uv tool install "backdraft[vlm]"
+uv tool install backdraft
 cd demo
 ```
 
 **Ingest** the sources. Every anchor is minted here, chunks for PDF pages,
-cells for spreadsheet sheets. Formats: PDF, XLSX/XLSM, CSV/TSV, images
-(png, jpeg, tiff, through the vision model), plain text and Markdown.
+cells for spreadsheet sheets. Formats: PDF, XLSX/XLSM, XLS, CSV/TSV, DOCX,
+PPTX, images (png, jpeg, tiff, through the vision model), plain text and
+Markdown.
 
 ```console
 $ backdraft init
@@ -60,7 +65,7 @@ documents: 0
 $ backdraft ingest sources/t12-summary.pdf sources/underwriting-model.xlsx
 t12-summary  t12-summary.pdf  pdf  3 pages
 underwriting-model  underwriting-model.xlsx  xlsx  2 pages
-note: extracted with pdf-text (the embedded text layer). Glossy or scanned PDFs extract better through a vision model: install backdraft[vlm] and set BACKDRAFT_VLM_API_KEY in .backdraft/env.
+note: extracted with pdf-text (the embedded text layer). Glossy or scanned PDFs extract better through a vision model: set BACKDRAFT_VLM_API_KEY in .backdraft/env.
 ```
 
 **Read.** The page arrives with a citable name over each chunk. This is the whole
@@ -189,21 +194,33 @@ confidential corpora**.
 ## Install
 
 ```bash
-uv tool install "backdraft[vlm]"   # CLI, recommended form
-uv add backdraft                   # as a dependency
-pip install backdraft              # or plain pip
+uv tool install backdraft   # CLI, recommended form
+uv add backdraft            # as a dependency
+pip install backdraft       # or plain pip
 ```
 
-Python 3.13+. Two extras:
+Python 3.13+. The vision extractor ships by default, and it is the
+recommended path for real PDFs (glossy layouts, info boxes, scans): a vision
+model reads each page and its clean representation becomes the receipt.
+Installing it is not consent to use it. `--extractor auto` prefers it only on
+explicit, backdraft-scoped consent: `BACKDRAFT_VLM_API_KEY` (env or
+`.backdraft/env`, `backdraft init` writes a template). Ambient
+`OPENAI_API_KEY`/`OPENROUTER_API_KEY` are deliberately never read; a generic
+key in the environment is not consent to send documents to its provider. The
+default provider is OpenRouter running Gemini 3.1 Flash Lite; direct OpenAI is
+base_url + model, set explicitly. Without a scoped key, `auto` falls back to
+the keyless text layer and says so. PDF page rendering needs poppler,
+installed separately.
+
+Two extras:
 
 | Extra | Adds | For |
 |---|---|---|
-| `[vlm]` | `openai`, `pdf2image` | the VLM extractor, the recommended path for real PDFs (glossy layouts, info boxes, scans): a vision model reads each page and its clean representation becomes the receipt. `--extractor auto` prefers it only on explicit, backdraft-scoped consent: `BACKDRAFT_VLM_API_KEY` (env or `.backdraft/env`, `backdraft init` writes a template). Ambient `OPENAI_API_KEY`/`OPENROUTER_API_KEY` are deliberately never read; a generic key in the environment is not consent to send documents to its provider. The default provider is OpenRouter running Gemini 3.1 Flash Lite; direct OpenAI is base_url + model, set explicitly. Without a scoped key, `auto` falls back to the keyless text layer and says so. Needs poppler installed separately |
 | `[entail]` | `anthropic` | `bind --check entail`, the model-judge verifier |
+| `[xls]` | `python-calamine` | legacy `.xls` workbooks, values only, rendered the same shape as xlsx |
 
-```bash
-pip install "backdraft[vlm,entail]"
-```
+`backdraft[vlm]` still installs: it is an empty compat alias from when the
+vision deps were an extra.
 
 From a checkout:
 
@@ -249,8 +266,7 @@ implementation reads them and nothing else.
 ## Status
 
 v1. The registry format, the token grammar and the artifact format are pinned
-by prose specs and golden-file tests. What is deliberately queued, exhibits,
-Excel region maps, more formats, `bd:calc` derivations, is written down in
-[ROADMAP.md](ROADMAP.md).
+by prose specs and golden-file tests. What is queued, and what is deliberately
+parked with the objection written down, lives in [ROADMAP.md](ROADMAP.md).
 
 MIT licensed.

{backdraft-0.3.0 → backdraft-0.4.0}/README.md

@@ -23,13 +23,14 @@ Docs live at [backdraft.dev](https://backdraft.dev), one page for people, and
 ## 60 seconds
 
 ```bash
-uv tool install "backdraft[vlm]"
+uv tool install backdraft
 cd demo
 ```
 
 **Ingest** the sources. Every anchor is minted here, chunks for PDF pages,
-cells for spreadsheet sheets. Formats: PDF, XLSX/XLSM, CSV/TSV, images
-(png, jpeg, tiff, through the vision model), plain text and Markdown.
+cells for spreadsheet sheets. Formats: PDF, XLSX/XLSM, XLS, CSV/TSV, DOCX,
+PPTX, images (png, jpeg, tiff, through the vision model), plain text and
+Markdown.
 
 ```console
 $ backdraft init
@@ -39,7 +40,7 @@ documents: 0
 $ backdraft ingest sources/t12-summary.pdf sources/underwriting-model.xlsx
 t12-summary  t12-summary.pdf  pdf  3 pages
 underwriting-model  underwriting-model.xlsx  xlsx  2 pages
-note: extracted with pdf-text (the embedded text layer). Glossy or scanned PDFs extract better through a vision model: install backdraft[vlm] and set BACKDRAFT_VLM_API_KEY in .backdraft/env.
+note: extracted with pdf-text (the embedded text layer). Glossy or scanned PDFs extract better through a vision model: set BACKDRAFT_VLM_API_KEY in .backdraft/env.
 ```
 
 **Read.** The page arrives with a citable name over each chunk. This is the whole
@@ -168,21 +169,33 @@ confidential corpora**.
 ## Install
 
 ```bash
-uv tool install "backdraft[vlm]"   # CLI, recommended form
-uv add backdraft                   # as a dependency
-pip install backdraft              # or plain pip
+uv tool install backdraft   # CLI, recommended form
+uv add backdraft            # as a dependency
+pip install backdraft       # or plain pip
 ```
 
-Python 3.13+. Two extras:
+Python 3.13+. The vision extractor ships by default, and it is the
+recommended path for real PDFs (glossy layouts, info boxes, scans): a vision
+model reads each page and its clean representation becomes the receipt.
+Installing it is not consent to use it. `--extractor auto` prefers it only on
+explicit, backdraft-scoped consent: `BACKDRAFT_VLM_API_KEY` (env or
+`.backdraft/env`, `backdraft init` writes a template). Ambient
+`OPENAI_API_KEY`/`OPENROUTER_API_KEY` are deliberately never read; a generic
+key in the environment is not consent to send documents to its provider. The
+default provider is OpenRouter running Gemini 3.1 Flash Lite; direct OpenAI is
+base_url + model, set explicitly. Without a scoped key, `auto` falls back to
+the keyless text layer and says so. PDF page rendering needs poppler,
+installed separately.
+
+Two extras:
 
 | Extra | Adds | For |
 |---|---|---|
-| `[vlm]` | `openai`, `pdf2image` | the VLM extractor, the recommended path for real PDFs (glossy layouts, info boxes, scans): a vision model reads each page and its clean representation becomes the receipt. `--extractor auto` prefers it only on explicit, backdraft-scoped consent: `BACKDRAFT_VLM_API_KEY` (env or `.backdraft/env`, `backdraft init` writes a template). Ambient `OPENAI_API_KEY`/`OPENROUTER_API_KEY` are deliberately never read; a generic key in the environment is not consent to send documents to its provider. The default provider is OpenRouter running Gemini 3.1 Flash Lite; direct OpenAI is base_url + model, set explicitly. Without a scoped key, `auto` falls back to the keyless text layer and says so. Needs poppler installed separately |
 | `[entail]` | `anthropic` | `bind --check entail`, the model-judge verifier |
+| `[xls]` | `python-calamine` | legacy `.xls` workbooks, values only, rendered the same shape as xlsx |
 
-```bash
-pip install "backdraft[vlm,entail]"
-```
+`backdraft[vlm]` still installs: it is an empty compat alias from when the
+vision deps were an extra.
 
 From a checkout:
 
@@ -228,8 +241,7 @@ implementation reads them and nothing else.
 ## Status
 
 v1. The registry format, the token grammar and the artifact format are pinned
-by prose specs and golden-file tests. What is deliberately queued, exhibits,
-Excel region maps, more formats, `bd:calc` derivations, is written down in
-[ROADMAP.md](ROADMAP.md).
+by prose specs and golden-file tests. What is queued, and what is deliberately
+parked with the objection written down, lives in [ROADMAP.md](ROADMAP.md).
 
 MIT licensed.

backdraft-0.4.0/ROADMAP.md

@@ -0,0 +1,59 @@
+# Roadmap
+
+Sequenced intent. DESIGN.md records what was *decided*; this file records what
+is *queued*, so tabling something is an act of writing it down here rather than
+losing it. Items graduate by getting built and deleted.
+
+## Now
+
+Empty, deliberately. The 0.4.0 wave graduated the whole section: DOCX, PPTX,
+and XLS extractors, the shared sheet-extraction module, the renderer
+decomposition with pinned Python/JS format parity, minified artifact
+emission, and the snapshot budget knobs. The next item gets promoted out of
+Parked or Later on purpose, not by momentum.
+
+## Parked
+
+Deliberately not queued, each with the reason, so picking one up starts from
+the objection rather than rediscovering it.
+
+- **Exhibits view** — the evidence-first inversion. Risk: it grows the
+  artifact and pollutes a design whose strength is restraint. Needs a
+  deliberate design pass and a real audience before any code; not a
+  weekend-feature shape.
+- **Excel region maps and range reads** — the research notes exist, but no
+  confidence the deterministic detection cascade lands well on the first
+  try. Revisit when large-sheet navigation hurts in a real corpus, with
+  test workbooks in hand.
+- **`bd:calc` derivations** — reserved in the grammar and parsed as
+  malformed-on-purpose, which is enough for now. Full support complicates
+  binding and verification for one field-trial case.
+- **Merged-cell and frozen-pane rendering** — ingest captures `merged` and
+  `frozen` in sheet meta and keeps them (cheap, and capture is the part you
+  cannot backfill). Frozen-pane rendering is cut; merged-cell rendering
+  waits for a real sheet that renders wrong without it.
+
+## Later
+
+- **Living documents** — re-ingest, re-bind, and a drift-first report: "these
+  claims cite figures that changed; cited-then vs. now." The primitives
+  (generations, `drifted`, the word-diff) all exist; the missing piece is a
+  diff-shaped report and a demo.
+- **Substrates beyond the CLI** — SDK middleware for pipelines (the exit-code
+  and `--to json` contracts are already shaped for it), and a client-side
+  drag-drop viewer page (no upload — the file never leaves the reader's
+  machine).
+- **HTML pages and email (.eml)** — both arrive in diligence folders, both
+  have messy identity questions (what is "the source" of a web page?) that
+  deserve a decision row before code.
+- **Distribution** — published on PyPI; still queued: a skills-registry
+  listing when one becomes the standard channel.
+
+## Someday
+
+- **Hosted team features** — upload, access control, shared registries.
+  Deliberately parked: the product's trust story is the self-contained file,
+  and a hosted viewer inverts it. Revisit only as a distinct teams product.
+- **Entail at scale** — the model-judge verifier is wired (`[entail]` extra)
+  but has never been field-calibrated; needs a corpus and a rubric before it
+  is recommended anywhere.

{backdraft-0.3.0 → backdraft-0.4.0}/SPEC.md

@@ -44,10 +44,16 @@ backdraft/
     extract/
       base.py               # Extractor protocol + registry of extractors
       pdf_text.py           # pdfplumber text-layer; paragraph breaks rebuilt from line geometry
-      xlsx.py               # openpyxl → markdown table with [B10] refs
+      sheet.py              # the shared tabular representation: bounds caps, [B10] row rendering
+      xlsx.py               # openpyxl → sheet representation + styling meta
+      xls.py                # [xls] extra: calamine → sheet representation, values only
+      csv.py                # sniffed-dialect csv/tsv → sheet representation
+      docx.py               # python-docx; heading sections become pages
+      pptx.py               # python-pptx; one page per slide, text floor
       text.py               # md/txt passthrough
-      vlm.py                # [vlm] extra: pdf→images→VLM per page
-      vlm_settings.py       # provider/model/base-url resolution; stdlib-only, imports without the extra
+      image.py              # an image is a one-page document; VLM transcription
+      vlm.py                # pdf→images→VLM per page (deps ship by default; key gates use)
+      vlm_settings.py       # provider/model/base-url resolution; stdlib-only
     gate/
       reader.py             # ls / toc / read page-range → token-marked context; records ledger
       searcher.py           # FTS5 → minted snippet results; records ledger
@@ -59,7 +65,9 @@ backdraft/
         base.py             # Verifier protocol
         value_trace.py  overlap.py  recompute.py  entail.py   # entail behind [entail]
     render/
-      html.py               # single-file artifact, embedded viewer + legend
+      html/                 # single-file artifact, embedded viewer + legend
+                            #   fmt.py text.py components.py assets.py page.py
+                            #   fmt_vectors.py: one vector table drives fmt_cell (py) and fmtCell (js)
       markdown.py           # the authored document's markdown → HTML, stdlib only
       placement.py          # locating each claim's span in the document render was handed
       footnotes.py          # markdown projection
@@ -126,7 +134,7 @@ CREATE TABLE documents (
   sha256 TEXT NOT NULL UNIQUE,
   path TEXT NOT NULL,            -- as given at ingest; informational
   filename TEXT NOT NULL,
-  media_type TEXT NOT NULL,      -- 'pdf' | 'xlsx' | 'text'
+  media_type TEXT NOT NULL,      -- 'pdf' | 'xlsx' | 'xls' | 'csv' | 'docx' | 'pptx' | 'image' | 'text'
   created_at TEXT NOT NULL       -- ISO-8601 UTC, everywhere
 );
 
@@ -218,7 +226,7 @@ class Extractor(Protocol):
     def extract(self, path: Path, config: dict) -> Iterator[ExtractedPage]: ...
 ```
 
-Registered in a plain dict; `ingest --extractor auto` picks the first `can_handle`, built-ins ordered `xlsx, pdf-text, text`. `vlm` (extra) is pdf→images→VLM per page, OpenAI-compatible client, model configurable. **For PDFs, `auto` prefers `vlm` when it is installed and an API key is configured** (glossy layouts and info boxes extract badly from the text layer, and the snapshot is the receipt); otherwise it falls back to `pdf-text` and prints a one-line nudge naming the VLM option. An explicit `--extractor` always wins. xlsx's representation: `Row | A | B...` header, `[B10] value` prefixes, dimension caps, inflated-sheet placeholder pages; cell values are never rounded, since the snapshot is the receipt a claim is traced against. `pdf-text` reconstructs paragraph breaks from line geometry before handing the page over, so the chunker's blank-line rule has something to fire on.
+Registered in a plain dict; `ingest --extractor auto` picks the first `can_handle`, built-ins ordered `xlsx, xls, csv, docx, pptx, pdf-text, image, text` (an extractor whose optional extra is missing is skipped, not fatal). `vlm` is pdf→images→VLM per page, OpenAI-compatible client, model configurable; its deps ship by default and the backdraft-scoped key gates use. **For PDFs, `auto` prefers `vlm` when it is ready (importable, key configured)** (glossy layouts and info boxes extract badly from the text layer, and the snapshot is the receipt); otherwise it falls back to `pdf-text` and prints a one-line nudge naming the VLM option. An explicit `--extractor` always wins. The sheet representation (shared by xlsx, xls, csv): `Row | A | B...` header, `[B10] value` prefixes, dimension caps, inflated-sheet placeholder pages; cell values are never rounded, since the snapshot is the receipt a claim is traced against. `docx` synthesizes pages from heading sections (the document's smallest present outline level among Heading 1–2 splits; no headings → one page), so locators stay `pN.cM`. `pptx` is one page per slide, text floor only; ingest notes that slide visuals need the PDF-export path. `pdf-text` reconstructs paragraph breaks from line geometry before handing the page over, so the chunker's blank-line rule has something to fire on.
 
 ## Gate (gate/)