pageledger 0.1.0__tar.gz

pageledger-0.1.0/CHANGELOG.md

@@ -0,0 +1,93 @@
+# Changelog
+
+All notable changes to PageLedger will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to the artifact compatibility policy documented in
+`docs/run-manifest-spec.md` → Compatibility Policy.
+
+## 0.1.0 — 2026-07-06
+
+### Added
+
+- `pageledger run` — extraction command supporting dry-run artifact
+  generation and built-in `text` and `pdf_text` adapter execution.
+- `pageledger rerun` — re-extracts exactly the pages listed in a previous
+  run's `rerun-manifest.yml`, preserving page ids, recording parent lineage,
+  enforcing `run.max_rerun_depth`, and warning when a source checksum no
+  longer matches the parent manifest.
+- `pageledger compare-runs` — page-by-page diff of two run directories:
+  character/word deltas, warnings resolved/introduced, adapters, cost.
+- `pageledger init-config` — generates a minimal valid config to stdout or file.
+- `pageledger inspect-run` — summarizes a completed or failed run directory.
+- `pageledger doctor` — reports optional dependencies, external tool versions,
+  and redacted cloud environment status.
+- Filesystem-native run artifacts: `manifest.json`, `route-map.yml`,
+  `config-snapshot.yml`, `audit.json`, `audit.md`, `provenance.jsonl`,
+  `quality.jsonl`, `cost.json`, `run.log`, `rerun-manifest.yml`, and per-page
+  output under `raw/`.
+- Page-denominated budget enforcement — caps on pages, tokens, and dollars
+  with configurable warning thresholds, enforced preflight and mid-run.
+- Cost provenance: `cost.json` reports `cost_basis` (`adapter_reported`,
+  `configured_rate`, `mixed`, `none`) and runner-measured
+  `extraction_seconds`; provenance lines carry per-page `extraction_seconds`.
+- Retry with configurable `max_retries` and optional exponential backoff
+  (`retry.backoff: exponential`, 0.5 s base doubling to an 8 s cap).
+- Quality signal diagnostics with six-item warning taxonomy (`empty_text`,
+  `short_text`, `replacement_characters`, `control_characters`,
+  `suspicious_symbol_density`, `suspicious_embedded_text_delta`).
+- Quality-warning pages automatically routed to audit `review_queue` with
+  `reason: "quality_warning"`.
+- `quality_warning_pages` rollup count in `manifest.summary`.
+- Custom adapters via `module.path:object` import strings.
+- Adapter conformance checker: `pageledger.adapters.adapter_conformance_check()`.
+- Adapter metadata validation at load time (name, version, deterministic,
+  input_types, output_types, capabilities, supports, extract).
+- `usage.pages == 1` enforcement — each `extract()` call handles exactly one page.
+- Executable rerun manifests: `rerun_status`
+  (`executable`/`empty_queue`/`no_further_generations`), `rerun_depth`
+  generation tracking, and the `max_rerun_depth == 0` guard producing empty
+  items.
+- Config validation with key-path error messages and suspicious-config warnings
+  (empty taxonomy, unknown top-level keys, impossible budget thresholds).
+- JSON Schema files for all JSON/JSONL artifacts under `schemas/`.
+- Schema validation tests covering dry-run, execute, budget failure, adapter
+  failure, and empty-review-queue scenarios.
+- Failure recovery: partial-run guarantees documented with scenario table,
+  write-order, and common error/user-action mappings.
+- Comprehensive test suite: 197 tests covering all CLI commands, adapters,
+  quality signals, rerun execution, cross-run comparison, cost provenance,
+  schemas, failure paths, and edge-case inputs.
+- CI workflow with test matrix (with/without PDF extra, Python 3.10–3.13)
+  and wheel/sdist smoke tests.
+- Release checklist under `.planning/release-checklist.md`.
+
+### Changed
+
+- "Core Modules" → "Design Architecture" with explicit implementation-status
+  labels in README.
+- Added "Current Runtime Capabilities" (three tiers) and "Known Limits" sections.
+- Claim audit: removed optimistic score language, constrained the public promise
+  to implemented behavior only.
+- `adapter-protocol.md` rewritten with frozen contract, `usage.pages == 1` rule,
+  and subprocess/timeout guidance.
+
+### Removed
+
+- Python 3.14 classifier (not yet released).
+
+### Performance
+
+- Tested scale: 5,000 text pages in 2.4s (~2,100 pages/sec) locally.
+  Artifact counts verified: raw files == provenance lines == quality lines
+  == pages extracted. Stress tests reproducible without credentials or
+  private data.
+
+### Fixed
+
+- `--json` now emits parseable error JSON (`{"status": "error", "error": "..."}`)
+  on failure instead of unstructured stderr-only errors.
+- Unreadable input files now produce clean `RuntimeError` instead of raw
+  `PermissionError` tracebacks.
+- Rerun manifest `max_rerun_depth == 0` now correctly produces empty `items: []`
+  with `rerun_status: "no_further_generations"`.

pageledger-0.1.0/CITATION.cff

@@ -0,0 +1,19 @@
+cff-version: 1.2.0
+title: "PageLedger: Auditable OCR/VLM Extraction Workflows"
+message: "If you use PageLedger in research, please cite the software and separately cite the source dataset or extraction run you created with it."
+type: software
+authors:
+  - family-names: Busscher
+    given-names: Peter
+abstract: "PageLedger is an alpha toolkit for routing, auditing, budgeting, and preserving filesystem-native provenance around OCR/VLM document extraction workflows."
+keywords:
+  - document extraction
+  - OCR
+  - vision-language models
+  - digital humanities
+  - provenance
+  - corpus infrastructure
+license: MIT
+repository-code: "https://github.com/peterbussch/pageledger"
+version: "0.1.0"
+date-released: "2026-07-06"

pageledger-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Peter Busscher
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pageledger-0.1.0/MANIFEST.in

@@ -0,0 +1,19 @@
+include README.md
+include LICENSE
+include CHANGELOG.md
+include CITATION.cff
+recursive-include docs *.md *.yml
+recursive-include examples *.py *.sh
+recursive-include schemas *.json
+recursive-include skills *.md
+
+# Local-only material that may exist in a development working tree.
+# Never let it into the sdist even when present on disk.
+exclude docs/autoresearch.md
+exclude _session_research.md
+recursive-exclude prompts *
+recursive-exclude assets *
+prune .planning
+prune .stress
+prune .hermes
+prune runs

pageledger-0.1.0/PKG-INFO

@@ -0,0 +1,500 @@
+Metadata-Version: 2.4
+Name: pageledger
+Version: 0.1.0
+Summary: Page-denominated document extraction run ledger
+Author: Peter Busscher
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/peterbussch/pageledger
+Project-URL: Documentation, https://github.com/peterbussch/pageledger#readme
+Project-URL: PDF/OCR First Run, https://github.com/peterbussch/pageledger/blob/main/docs/pdf-ocr-first-run.md
+Project-URL: OCR Options, https://github.com/peterbussch/pageledger/blob/main/docs/ocr-options.md
+Project-URL: Issues, https://github.com/peterbussch/pageledger/issues
+Project-URL: Source, https://github.com/peterbussch/pageledger
+Keywords: document-ai,ocr,vlm,digital-humanities,provenance
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Text Processing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: PyYAML>=6.0.2
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == "dev"
+Requires-Dist: jsonschema>=4.0; extra == "dev"
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: twine>=6.0; extra == "dev"
+Provides-Extra: pdf
+Requires-Dist: pypdf>=6; extra == "pdf"
+Dynamic: license-file
+
+<p align="center">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/peterbussch/pageledger/main/assets/pageledger-lockup-horizontal-reversed.png">
+    <img src="https://raw.githubusercontent.com/peterbussch/pageledger/main/assets/pageledger-lockup-horizontal.png" alt="PageLedger — the tallied page" width="440">
+  </picture>
+</p>
+
+<p align="center"><em>Record, route, and review document extraction — one page at a time.</em></p>
+
+**PageLedger is for researchers who extract text from archives at scale** —
+digital humanities labs, libraries, historians, and social scientists running
+OCR/VLM over thousands of scanned pages who must be able to answer, months
+later: which pages went through which engine and prompt, what it cost, which
+pages are suspect, and what happened when they were re-extracted.
+
+It is a page-denominated **run ledger** — not an OCR model, not a
+PDF-to-Markdown converter. You bring the extraction engine (Tesseract,
+Docling, Marker, a cloud VLM); PageLedger routes pages to it, enforces
+page/token/dollar budgets, and writes the evidence as plain files:
+provenance, quality diagnostics, cost with its basis, audit queues, and
+executable rerun plans. No service, no database — a run directory you can
+grep, cite, and reproduce.
+
+PageLedger grew out of
+[soviet-corpus](https://github.com/peterbussch/soviet-corpus), a Soviet census
+digitization and corpus-linguistics project where those questions had to be
+answerable for source citation and methodological review. The run-ledger
+layer was extracted from that working pipeline and rebuilt as a standalone,
+domain-neutral tool.
+
+`0.1.0` is an early alpha focused on the run-ledger core: built-in `text` and
+`pdf_text` (born-digital) adapters, custom adapters via import strings,
+dry runs, budgets, quality signals, page-scoped reruns
+(`pageledger rerun`), and cross-run diffs (`pageledger compare-runs`). It
+does not yet OCR images itself, call network VLMs out of the box,
+automatically classify pages, align records to schemas, or compute audit
+grades — see [Current Runtime Capabilities](#current-runtime-capabilities).
+
+## Install
+
+```bash
+pip install pageledger
+```
+
+Optional born-digital PDF text extraction:
+
+```bash
+pip install "pageledger[pdf]"
+```
+
+## Quickstart
+
+Create a text fixture and a starter config:
+
+```bash
+printf 'first page\fsecond page\n' > sample.txt
+pageledger init-config --out pageledger.yml
+```
+
+Inspect the run without extraction:
+
+```bash
+pageledger run sample.txt --config pageledger.yml --out runs/dry --dry-run
+```
+
+Run the built-in text adapter, then summarize the run directory:
+
+```bash
+pageledger run sample.txt --config pageledger.yml --out runs/text --json
+pageledger inspect-run runs/text
+```
+
+For a born-digital PDF, install the optional PDF extra and use `pdf_text`:
+
+```bash
+pip install "pageledger[pdf]"
+cat > pageledger-pdf.yml <<'YAML'
+schema_version: "0.1"
+taxonomy:
+  page_types:
+    prose:
+      default_action: transcribe_text
+run:
+  adapter: pdf_text
+YAML
+pageledger run document.pdf --config pageledger-pdf.yml --out runs/pdf-dry --dry-run
+pageledger run document.pdf --config pageledger-pdf.yml --out runs/pdf-text --json
+```
+
+`--out` must point to a new empty directory. For dry runs over PDFs, PageLedger
+uses `pypdf` page counts when the PDF extra is installed; execution with
+`pdf_text` extracts an existing text layer and does not OCR scanned images.
+
+The run directory contains `manifest.json`, `route-map.yml`, `audit.json`,
+`audit.md`, `provenance.jsonl`, `quality.jsonl`, `cost.json`, `run.log`,
+`rerun-manifest.yml`, and per-page raw output under `raw/`.
+
+For a real PDF/OCR walkthrough, see
+[`docs/pdf-ocr-first-run.md`](docs/pdf-ocr-first-run.md). For local, free,
+open-source, cloud, and hybrid extraction choices, see
+[`docs/ocr-options.md`](docs/ocr-options.md). Custom adapter examples live
+under [`examples/`](examples/), including Tesseract via `pdftoppm`, OCRmyPDF
+preprocessing, and a redacted cloud/VLM skeleton.
+
+## How a Run Works
+
+```mermaid
+flowchart TD
+    A["inputs (text / PDF)"] --> B["paginate<br/>(form-feed or PDF pages)"]
+    B --> C["route pages<br/>route-map.yml"]
+    C --> D{"budget preflight<br/>max_pages"}
+    D -- over cap --> X["refuse: nothing written"]
+    D -- ok --> E["extract page via adapter<br/>(retry + backoff)"]
+    E --> F["quality signals<br/>quality.jsonl"]
+    E --> G["provenance.jsonl<br/>+ cost.json (cost_basis)"]
+    E --> H{"budget mid-run<br/>pages / tokens / USD"}
+    H -- over cap --> Y["halt: manifest status=failed,<br/>partial artifacts consistent"]
+    H -- ok --> E
+    F -- warnings --> I["review queue<br/>audit.json / audit.md"]
+    I --> J["rerun-manifest.yml"]
+    J -- "pageledger rerun<br/>(stronger adapter)" --> E
+    G --> K["pageledger compare-runs<br/>(parent vs rerun)"]
+```
+
+Every box on the right is a plain file in the run directory; nothing needs a
+service or database.
+
+## Current Runtime Capabilities
+
+**Built-in and tested:**
+
+- `pageledger run` for text fixtures (form-feed pagination) and born-digital PDF
+  text layers (via `pageledger[pdf]`).
+- Dry-run mode that generates full run artifacts without calling extractors.
+- Per-page provenance (`provenance.jsonl`), quality diagnostics
+  (`quality.jsonl`), cost rollups (`cost.json`), structured run logs
+  (`run.log`), and audit/review queues (`audit.json`, `audit.md`).
+- Page-denominated budget enforcement (pages, tokens, dollars) with preflight
+  and per-page caps.
+- Retry with configurable `max_retries` and optional exponential backoff
+  (`retry.backoff: exponential`).
+- `pageledger rerun` — re-extracts exactly the pages listed in a previous
+  run's rerun manifest (typically with a stronger adapter), preserving page
+  ids, recording parent lineage, enforcing `max_rerun_depth`, and warning if
+  a source file changed since the parent run.
+- `pageledger compare-runs` — page-by-page diff of two runs: character/word
+  deltas, warnings resolved or introduced, adapters, and cost.
+- Cost provenance: `cost.json` records `cost_basis` (`adapter_reported`,
+  `configured_rate`, `mixed`, or `none`) so derived accounting rates are
+  never mistaken for provider-billed spend, plus runner-measured
+  `extraction_seconds` per page and in total.
+- `pageledger doctor` diagnostics for optional PDF/OCR/cloud tooling.
+- Configurable custom adapters via `module.path:object` import strings.
+
+**Adapter-supported but user-supplied:**
+
+- OCR via Tesseract (external `pdftoppm` + `tesseract` commands), OCRmyPDF
+  preprocessing, or any external engine wrapped as a custom adapter.
+- Cloud OCR/VLM adapters (user provides API keys, adapter code, and pricing).
+- Local document-conversion engines (Docling, Marker, Surya) through custom
+  adapters.
+- PDF page counting for custom adapters that expose `page_count(source)`.
+
+**Documented future work (not yet implemented):**
+
+- Automatic page classification (the alpha routes every page to `review` in
+  dry-run mode; no classifier ships).
+- Schema alignment execution (the schema config section is parsed but the
+  aligner does not yet produce normalized records).
+- Audit grading (review queues are populated but no grades are computed).
+- Multi-adapter routing chains, `rerun_if`/`quarantine_if` policy enforcement,
+  and staged CLI commands (`classify`, `extract`, `align`, `audit`).
+
+## Known Limits
+
+- **`pdf_text` reads existing text layers.** It does not perform OCR. Scanned
+  PDFs, image-heavy PDFs, or PDFs with noisy/absent text layers need an external
+  OCR step (OCRmyPDF, Tesseract, Docling, Marker, Surya, or cloud OCR) before
+  PageLedger can record useful text.
+- **Quality signals are diagnostic, not calibrated.** `quality.jsonl` records
+  per-page evidence: character/word counts, replacement characters, control
+  characters, suspicious symbol density, and embedded-text deltas. A six-item
+  warning taxonomy (empty_text, short_text, replacement_characters,
+  control_characters, suspicious_symbol_density, suspicious_embedded_text_delta)
+  flags pages for review. These signals are NOT OCR accuracy scores — they are
+  evidence a human should weigh.
+- **Quality-warning pages appear in the audit review queue.** For execution
+  runs, every page with one or more quality warnings is added to
+  `audit.json` → `review_queue` with reason `"quality_warning"`. Dry-run
+  review entries use route-based reasons (`"no_classifier_available"`).
+- **No automatic page routing.** Every page routes to the configured
+  `default_action` (or `review` in dry-run mode). Projects that need
+  page-type-aware routing must classify pages outside PageLedger or wait for
+  a future classifier.
+- **Schema alignment does not execute.** The schema config section is validated
+  and preserved in the config snapshot, but the runner does not yet align
+  extractor output to declared columns, types, or checks.
+- **Reruns re-extract listed pages; they do not merge results.**
+  `rerun-manifest.yml` lists pages from `audit.json` → `review_queue`;
+  `pageledger rerun` re-extracts exactly those pages into a new run directory
+  with parent lineage. `rerun_status` is `"executable"`, `"empty_queue"`, or
+  `"no_further_generations"` (depth cap reached). Combining parent and rerun
+  outputs into one corpus remains the project's decision —
+  `pageledger compare-runs` shows the per-page evidence for making it.
+
+## Tested Scale
+
+The 0.1.0 alpha has been tested locally on:
+
+- **5,000 synthetic text pages** (2.4s, ~2,100 pages/sec, artifact counts
+  verified — raw files = provenance lines = quality lines = pages extracted).
+- **1,000 synthetic text pages** (0.25s, ~4,000 pages/sec).
+- **72-page born-digital PDF** (via `pageledger[pdf]`).
+- **5 small checked-in text fixtures** covering clean, multipage, OCR-noisy,
+  blank, and short text.
+
+This is evidence of the envelope, not a universal performance guarantee.
+Stress tests are marked `@pytest.mark.stress` and skipped in default CI.
+Run them with:
+
+```bash
+python -m pytest tests/pageledger/ -m stress
+```
+
+## The Canonical Unit: Pages
+
+PageLedger's defining decision is that **the page is the canonical unit of
+work.** It is the only unit every extraction backend shares:
+
+- Cloud OCR (Textract, Azure Document Intelligence, Google Document AI, Mistral
+  OCR) bills and reports **per page**.
+- VLM/LLM extractors expose **tokens** — but only on model-backed paths; tokens
+  are meaningless for classical OCR.
+- Self-hosted engines (e.g. Docling) have **no dollar cost at all**, only compute
+  time.
+
+Because pages are the common denominator, denominating routing, budgeting, and
+audit in pages is what lets you **compare, budget, and reproduce a run across
+heterogeneous providers** — a Textract run against a Mistral run against a local
+model — with one number. Tokens, compute-seconds, and dollars are all carried as
+*optional, provider-conditional* signals on top.
+
+Every adapter reports a usage record where **`pages` is required** and
+everything else is optional:
+
+```python
+usage = {
+    "pages": 1,             # REQUIRED — the portable unit
+    "tokens": None,         # VLM/LLM paths only
+    "compute_seconds": None, # self-hosted engines
+    "cost_usd": None,       # optional adapter-reported passthrough
+}
+```
+
+Dollar cost is **derived by PageLedger, never required of the adapter**, in
+priority order: (1) adapter-reported `cost_usd`, (2) configured unit rates
+(`cost_per_page` / `cost_per_1k_tokens`), (3) otherwise `null` — the run still
+reports raw page counts. Budgets cap on **pages, tokens, or dollars**, whichever
+the config sets, because the page count is the only value always present.
+
+## Why Not Just Use OCR?
+
+OCR and document-AI tools are improving quickly. Mistral OCR, Google Document
+AI, Azure Document Intelligence, AWS Bedrock Data Automation, Docling, Marker,
+Surya, olmOCR, OCR-D, and Unstructured all cover parts of extraction,
+conversion, layout analysis, validation, or workflow orchestration.
+
+That makes PageLedger's scope deliberately narrow: it is useful when you need a
+local, provider-neutral ledger around those tools, not when you only need one
+tool to convert a document once.
+
+| Tool | Strong At | PageLedger Difference |
+|---|---|---|
+| Mistral OCR | Hosted OCR/document understanding with Markdown and table reconstruction. | PageLedger records why pages went to Mistral, what it cost, and which pages need review or rerun. |
+| Google Document AI / Gemini layout parser | Managed OCR, layout parsing, table structure, and RAG-oriented chunking. | PageLedger keeps provider-neutral run artifacts and project-local audit queues outside a cloud workflow. |
+| Azure Document Intelligence | Managed text, key-value, table, and field extraction. | PageLedger can track Azure runs beside local or other-provider runs with the same page-denominated manifest. |
+| AWS Bedrock Data Automation | Managed classification, extraction, validation, HITL review, and business-rule workflows. | PageLedger stays lighter and filesystem-native for research projects that do not want a cloud stack. |
+| Docling | Converting PDFs and documents into structured output. | PageLedger records which pages went to Docling, the usage and cost evidence, and which pages need review or rerun. |
+| Marker | High-quality PDF-to-Markdown conversion. | PageLedger handles routing, run manifests, cost controls, and review queues around conversion. |
+| Surya | OCR, layout analysis, table recognition. | PageLedger can call Surya as an extractor, then audit the result across a full archive run. |
+| olmOCR | LLM-based PDF extraction with document-oriented output. | PageLedger adds page routing, quality diagnostics, cost evidence, and rerun manifests. |
+| OCR-D | Mature OCR workflow model with METS/PAGE/ALTO conventions. | PageLedger aims to be lighter, VLM-aware, and friendlier to local research workflows that need JSON/YAML artifacts before full library infrastructure. |
+| Unstructured | Document partitioning and preprocessing for downstream use. | PageLedger focuses on extraction run control: what ran, what passed, what failed, what cost money, and what should be rerun. |
+
+Non-goal: PageLedger is not an OCR engine, PDF converter, or layout detector.
+If a project already has clean structured text from a single tool and does not
+need routing, schema checks, review queues, cost tracking, or reruns, it likely
+does not need PageLedger.
+
+For practical provider-agnostic choices, see
+[`docs/ocr-options.md`](docs/ocr-options.md). The recommended workflow: run cheap
+local extraction first, inspect `quality.jsonl` warnings to identify weak pages,
+re-extract just those pages with a stronger adapter via
+`pageledger rerun runs/first/ --config stronger.yml --out runs/second/`, then
+`pageledger compare-runs runs/first/ runs/second/` to see what improved.
+
+## What PageLedger Cannot Do
+
+PageLedger does not make OCR or VLM output correct by itself. It cannot
+calibrate confidence scores across unrelated extractors, guarantee text
+accuracy, infer a project's source citation, or make a right-to-left,
+mixed-script, tabular, or handwritten collection work without explicit adapter
+and schema configuration.
+
+Its job is narrower: preserve enough evidence that a researcher can see what
+ran, what failed, what is uncertain, and what should be reviewed or rerun.
+
+## Why This Exists
+
+Modern OCR and document-AI tools can produce impressive text, tables, and
+Markdown, but real archive work fails in smaller and more frustrating ways:
+
+- Some pages are title pages, indexes, maps, appendices, blanks, or prose,
+  not data pages.
+- Expensive VLM calls should be routed deliberately instead of sprayed across
+  every page.
+- Model-emitted tables often have variant headers, shifted columns, missing
+  rows, merged cells, or inconsistent numeric formats.
+- Confidence scores are weak unless they are checked against schema,
+  arithmetic, page totals, and model agreement.
+- Long extraction runs need cost tracking, retry logic, checkpoints, and
+  provenance that survives reruns.
+
+PageLedger is for projects where "the model returned JSON" is only the
+beginning of the work. The current alpha is for proving local run evidence and
+adapter contracts before trusting larger extraction workflows.
+
+## Audience
+
+- Digital humanities labs processing archival scans.
+- Libraries and archives building reproducible extraction pipelines.
+- Historians and social scientists extracting structured data from source
+  books, statistical tables, reports, and registers.
+- Civic, legal, nonprofit, and local-government document projects.
+- Document-AI engineers who need a lightweight local control plane around
+  Docling, Marker, Surya, olmOCR, Tesseract, or API VLMs.
+
+## Architecture and Roadmap
+
+The alpha implements the **run controller** (budgets, retry, provenance,
+quality signals, audit queues, rerun execution, cross-run comparison) and
+the **adapter protocol** (thin wrappers that accept a routed page and return
+serializable content, warnings, confidence hints, and usage metadata — see
+[`docs/adapter-protocol.md`](docs/adapter-protocol.md)).
+
+Automatic page routing, schema alignment, audit grading, rerun execution, and
+staged CLI commands are design targets, documented with examples in
+[`docs/design.md`](docs/design.md).
+
+## CLI
+
+Six commands ship in the alpha; `run` and `rerun` are the ones that extract:
+
+```bash
+pageledger run scans/ --config pageledger.yml --out runs/run-001/
+pageledger run scans/ --config pageledger.yml --out runs/run-001/ --dry-run
+pageledger run scans/ --config pageledger.yml --out runs/run-001/ --json --log-level INFO
+pageledger rerun runs/run-001/ --config stronger.yml --out runs/run-002/
+                                              # re-extract only the flagged pages
+pageledger compare-runs runs/run-001/ runs/run-002/
+                                              # page-by-page diff of two runs
+pageledger init-config --out pageledger.yml   # write a minimal valid config
+pageledger inspect-run runs/run-001/          # summarize a run directory
+pageledger doctor --json                      # check optional dependencies
+```
+
+`--dry-run` generates the route map and planning artifacts without calling
+extractors, so you can inspect routing before spending money or trusting
+output. `--json` makes command status scriptable (parseable stdout, errors on
+stderr), and `--log-level` controls the detail written to the run log.
+
+Current non-dry-run support is deliberately tiny. Set `run.adapter: text` to
+write each page of a UTF-8 text input to `raw/doc_0001_page_0001.txt` (text
+sources are split into pages on the form-feed character), or install
+`pageledger[pdf]` and set `run.adapter: pdf_text` for born-digital PDF text
+extraction through `pypdf`. Both paths emit one `provenance.jsonl` line per
+page. This proves the preflight paginate → route → extract → provenance path
+before OCR/VLM integrations. For scanned PDFs or richer layout extraction, run
+an OCR/conversion tool first or point `run.adapter` at a project adapter such as
+`my_project.adapters:TesseractCliAdapter`; PageLedger will keep the same
+manifest, cost, log, and provenance envelope around that external engine.
+
+The doctor output reports optional packages, common OCR commands, and whether
+cloud OCR/VLM keys are present without printing their values.
+
+Starter config examples live in `docs/examples/`:
+
+- `pageledger.yml` (recommended first-use combined config)
+- `page-taxonomy.yml`
+- `table-schema.yml`
+- `run-policy.yml`
+
+The split files are useful for documentation and larger projects. The
+recommended v0.1 user experience should start with one `pageledger.yml` that
+contains `taxonomy`, `schema`, and `run` sections.
+
+## Run Artifacts
+
+A run should be inspectable without a database or service:
+
+```text
+runs/run-001/
+├── manifest.json
+├── config-snapshot.yml
+├── route-map.yml
+├── raw/
+│   └── doc_0001_page_0002.txt
+├── normalized/
+├── audit.json
+├── audit.md
+├── provenance.jsonl
+├── quality.jsonl
+├── cost.json
+├── run.log
+└── rerun-manifest.yml
+```
+
+The important artifact is not only the extracted data, but the evidence around
+it: which pages were skipped, which model or engine was used, what usage and
+cost evidence was observed, which pages failed or need review, and what should
+be rerun or reviewed.
+
+`manifest.json` is the canonical run artifact. `audit.md` is a human rendering
+of `audit.json`, not a second independent source of truth. Quarantined pages
+live in `audit.json` under `quarantine_queue`. `config-snapshot.yml` preserves
+the exact user config that produced the run. `run.log` is JSONL — one line per
+extractor call with timestamp, `page_id`, adapter, status, and any error — so
+partial or failed runs stay greppable. Every artifact validates against the
+JSON Schemas in [`schemas/`](schemas/). The artifact directory keeps the name
+`normalized/` for stability, but the current alpha does not write normalized
+records yet; schema alignment remains a documented design target.
+
+## Design Principles
+
+- Record uncertainty; do not silently fix it.
+- Treat heuristic confidence as evidence, not probability. Uncalibrated
+  extractors should not imply certainty.
+- Every run produces inspectable artifacts on disk.
+- Adapters are thin: PageLedger does not own extraction, it owns the process
+  around extraction.
+- The manifest, route map, and provenance files should be useful without a
+  running service or database.
+- Preserve separate citations for software and source data.
+
+## Related Documents
+
+- [`schemas/`](schemas/) — JSON Schema files, the machine-readable authority
+  for artifact contracts
+- [`docs/design.md`](docs/design.md) — design targets: router, schema aligner,
+  rerun policies, staged CLI, open research questions
+- [`docs/comparison.md`](docs/comparison.md) — positioning against the 2026
+  extraction ecosystem
+- [`docs/run-manifest-spec.md`](docs/run-manifest-spec.md),
+  [`docs/route-map-spec.md`](docs/route-map-spec.md),
+  [`docs/audit-spec.md`](docs/audit-spec.md),
+  [`docs/provenance-spec.md`](docs/provenance-spec.md),
+  [`docs/rerun-manifest-spec.md`](docs/rerun-manifest-spec.md) — per-artifact
+  field specs
+- [`docs/adapter-protocol.md`](docs/adapter-protocol.md) — how to wrap an
+  OCR/VLM tool as an adapter
+- [`AGENTS.md`](AGENTS.md) — orientation for AI coding agents working in this
+  repository
+- [`CITATION.cff`](CITATION.cff) — how to cite PageLedger in research output