pi-quiver 6.8.0 → 6.10.0

package/CHANGELOG.md

@@ -8,6 +8,17 @@ Published to npm as `pi-quiver` (`pi install npm:pi-quiver`). Pushing a
 via OIDC trusted publishing. The release helper at
 `.agents/skills/release/scripts/release.sh` cuts the tag; CI publishes.
 
+## v6.10.0 - 2026-10-02
+
+- `doc_to_md`: new per-call `words` option (`--words`; not settable) writes `<stem>.words.json` - per selected PDF/image page, every text-layer word and every word inline OCR recognized in the same run, with display-space bbox (points; source pixels for images) and `source` `text`/`ocr`. Under `--ocr-mode all` each sidecar gets `ocr/<stem>-pNNN.words.json`. Never triggers OCR; Markdown, page stats and OCR outcome are unchanged. Handle gains `Words:`, `--json` gains `wordsPath`, `wordsReason`, `wordsErrors`, `ocr.wordSidecars`; `--info --words` is a usage error (#28).
+- `doc_to_md`: `--help` and the generated skill list every bundle artifact under `Bundle layout` (#28).
+
+## v6.9.0 - 2026-10-01
+
+- `doc_to_md`: every conversion that goes through the Python PDF tiers (PDF, PPTX, `.doc`, DOCX via LibreOffice) writes `<stem>.pages.json` (per-page `chars`, `images`, `imageCoverage`) and the handle prints `Page-Stats:`; CLI `--json` gains `pageStats`, `pageStatsPath`, `ocrDir`. The unpdf tier writes no stats (#26).
+- `doc_to_md`: new per-call `ocrMode` (`--ocr-mode textless|all`). `all` forces OCR on an explicit `pages` selection in a second, kill-capped Python child and writes `ocr/<stem>-pNNN.md` sidecars; the Markdown stays byte-identical to the same selected-page call without OCR. Refused without `--ocr`, without explicit pages, or on non-PDF/PPTX/DOC inputs (exit 2); missing Tesseract data is a hard error. The `OCR:` line names written, no-text, failed, budget-stopped, killed and not-attempted pages, with the exact `--pages` to re-run for budget-stopped and not-attempted pages (#26).
+- `doc_to_md`: `--help` and the generated `skills/doc-to-md/SKILL.md` end with the two-pass OCR usage block (#26).
+
 ## v6.8.0 - 2026-10-01
 
 - `session-name`: opt-in automatic naming starts in the background after three completed model/tool rounds. Shorter runs start a non-blocking attempt at run end; initial generation is best-effort with a 30-second local deadline and no automatic retry per session activation.

package/README.md

@@ -65,7 +65,7 @@ A 300 KB changelog page never touches your context window - you get a preview an
 | Extension | Tool | What it does |
 | --- | --- | --- |
 | `extensions/fetch.ts` | `fetch` | Retrieve URLs over HTTP(S). HTML -> Markdown (Readability extraction, Turndown conversion). Binary saved untouched to a temp file. GitHub issue/PR/repo/actions-run/actions-job URLs auto-route through `gh` (falls back to HTTP); failed runs/jobs include failed-step logs (best-effort, summary-only otherwise). Same size gate as `fetch`. Behavior lives in `lib/fetch-core.ts`; also exposed as the `pi-quiver fetch` CLI (see [Claude Code support](#claude-code-support)). |
-| `extensions/doc_to_md.ts` | `doc_to_md` | Convert a local PDF/DOCX/DOC/PPTX/XLSX/XLSM/XLS, MSG/EML, HTML, or image file to a Markdown bundle on disk (`<stem>.md` + `images/`, spreadsheet `sheets/`, optional `pageImages` in `pages/`, and email `attachments/`) and return a handle (paths, page count, outline, diagnostics) - never inline Markdown. `info` inspects PDF/Office inputs; `pages` selects 1-based PDF/Office pages (DOCX: explicit-page-break segments); HTML, image, and email inputs reject both; every selected PDF/PPTX page or DOCX segment when `pageCount > 1` ends with `--- end of page.page_number=N ---`. Tiers: pymupdf4llm -> PyMuPDF text (degraded) -> unpdf worker (no Python only). Excel -> sheet inventory, full CSVs, bounded previews, and optional rendered views. HTML keeps local and `data:` images in the bundle and remote images as links; image inputs keep the original image. Scanned PDF pages keep a page picture on both Python tiers; opt-in OCR needs Tesseract language data. DOCX converts directly (mammoth, python-docx fallback); LibreOffice pagination marks the degraded route. The Outline lists `L<line>` and `p<page>` per heading. Settings under `quiver.docToMd`. |
+| `extensions/doc_to_md.ts` | `doc_to_md` | Convert a local PDF/DOCX/DOC/PPTX/XLSX/XLSM/XLS, MSG/EML, HTML, or image file to a Markdown bundle on disk (`<stem>.md` + `images/`, spreadsheet `sheets/`, optional `pageImages` in `pages/`, and email `attachments/`) and return a handle (paths, page count, outline, diagnostics) - never inline Markdown. `info` inspects PDF/Office inputs; `pages` selects 1-based PDF/Office pages (DOCX: explicit-page-break segments); HTML, image, and email inputs reject both; every selected PDF/PPTX page or DOCX segment when `pageCount > 1` ends with `--- end of page.page_number=N ---`. Tiers: pymupdf4llm -> PyMuPDF text (degraded) -> unpdf worker (no Python only). Excel -> sheet inventory, full CSVs, bounded previews, and optional rendered views. HTML keeps local and `data:` images in the bundle and remote images as links; image inputs keep the original image. Scanned PDF pages keep a page picture on both Python tiers; opt-in OCR needs Tesseract language data. DOCX converts directly (mammoth, python-docx fallback); LibreOffice pagination marks the degraded route. The Outline lists `L<line>` and `p<page>` per heading. Settings under `quiver.docToMd`. Every conversion that goes through the Python PDF tiers (PDF, PPTX, `.doc`, DOCX via LibreOffice) writes `<stem>.pages.json` (per-page chars, image count, image coverage; handle line `Page-Stats:`); unpdf writes no stats. `ocrMode: "all"` with `ocr: true` and an explicit `pages` forces OCR on those pages into `ocr/<stem>-pNNN.md` sidecars, leaving the Markdown byte-identical to the same selected-page call without OCR. `words: true` (CLI `--words`, per-call only - no settings key) writes `<stem>.words.json` with every word's bbox per selected PDF/image page (`source` `text` or `ocr`), plus `ocr/<stem>-pNNN.words.json` beside each forced-OCR sidecar; it never triggers OCR. |
 | `extensions/session-name.ts` | `/session-name` | Manual + opt-in automatic session naming, naming rules and deny list, long-session revisits, and Ghostty/Herdr tab rename. OFF by default. |
 | `extensions/sword-header.ts` | `/builtin-header` | Themed ASCII startup header replacing pi's default logo. OFF by default. |
 | `extensions/fast-mode.ts` | `/fast` | Inject Anthropic fast-mode (`speed: "fast"` + `anthropic-beta: fast-mode-2026-02-01`) into every Claude Opus 4.8 / Opus 5 request, any thinking level. `--fast` flag + `/fast [on\|off\|status]`. OFF by default. |
@@ -310,7 +310,9 @@ Each setting can also be overridden per-process via `PI_QUIVER_SLACK_ENABLED`, `
 | `ocr` | `false` | Opt-in OCR for scanned pages and images when Tesseract language data is installed. |
 | `ocrLanguage` | `eng` | Plain `+`-joined Tesseract language codes for OCR. |
 
-A bundle is `<outputDir>/<stem>.md` plus `<outputDir>/images/` and, for spreadsheets with data, `<outputDir>/sheets/`; without `outputDir`, the tool creates a per-call temp root. A conversion owns `<stem>.md.lock` until it atomically publishes the Markdown. An existing `<stem>.md` fails the call unless `overwrite` is set; `overwrite` replaces that Markdown and the files it owns (`images/<stem>-p<N>-<n>.*`, `images/<stem>-s<idx>[-<n>].*`, `sheets/<stem>-s<idx>-<slug>.csv`), nothing else. Temp bundles are caller-owned - the tool never deletes a bundle it produced.
+`ocrMode` (`textless` | `all`) is a per-call parameter and `--ocr-mode` flag only; a `quiver.docToMd.ocrMode` key is reported as unknown and ignored, so persisted settings cannot enable forced OCR or bypass its explicit-pages guard.
+
+A bundle is `<outputDir>/<stem>.md` plus `<outputDir>/<stem>.pages.json` on Python PDF tiers, optional OCR sidecars under `<outputDir>/ocr/`, and `<outputDir>/images/` and, for spreadsheets with data, `<outputDir>/sheets/`; without `outputDir`, the tool creates a per-call temp root. A conversion owns `<stem>.md.lock` until it atomically publishes the Markdown. An existing `<stem>.md` fails the call unless `overwrite` is set; `overwrite` replaces that Markdown and the files it owns (`images/<stem>-p<N>-<n>.*`, `images/<stem>-s<idx>[-<n>].*`, `sheets/<stem>-s<idx>-<slug>.csv`, `<stem>.pages.json`, `ocr/<stem>-pNNN.md`), nothing else. Temp bundles are caller-owned - the tool never deletes a bundle it produced.
 
 Excel needs a Python backend with openpyxl, xlrd and pillow - otherwise the call fails with `Remedy: install uv, or pip install openpyxl xlrd pillow`. The Markdown opens with a `## Sheets` table listing every sheet in workbook order (0-based `#`, `worksheet`/`chartsheet`, size, hidden, chart and image counts, rendered view, CSV link for non-empty worksheets), then one section per sheet: a `Data:` line linking the full-content CSV under `sheets/` for non-empty worksheets, chart metadata from the workbook model, embedded images, an optional rendered view, a preview of at most the first 100 rows x 50 columns, and - only when the preview is truncated - a `Columns:` profile (type, non-empty count, min/max, distinct up to 50). Sizes are the extent of non-empty cells (the `info` handle reports the raw worksheet dimensions instead, which may be larger). Rendered views (`images/<stem>-s<idx>.<fmt>`) are produced for sheets carrying charts or images when LibreOffice is on `PATH`: the workbook is exported one PDF page per sheet and rasterized under a 16 Mpx budget. Any LibreOffice or rasterization failure degrades to `Rendered view: unavailable (<reason>)` and a handle note; it never fails the conversion. Workbooks whose chartsheet drawings carry a zero-size anchor (openpyxl-authored files; Excel-authored files are unaffected) render as a degenerate page and are reported as such. `.xls` gets the inventory, CSVs and previews but no visual detection or rendering.
 

package/dist/bin/pi-quiver.js

@@ -581,6 +581,9 @@ function ownedCsvPattern(stem) {
 function ownedPagePattern(stem) {
   return new RegExp(`^${escRe(stem)}-p\\d+\\.[a-z0-9]+$`);
 }
+function ownedOcrPattern(stem) {
+  return new RegExp(`^${escRe(stem)}-p\\d+(?:\\.words\\.json|\\.md)$`);
+}
 var FILE_LINK_RE = /\[[^\]]*\]\(\s*((?:sheets|attachments)\/[^)\s]+)\s*\)/g;
 var IMG_LINK_RE = /!\[[^\]]*\]\(\s*(?:<([^>]*)>|([^)]*?))\s*\)|<img\b[^>]*\bsrc\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s>"']+))/gi;
 function imageTarget(m) {
@@ -627,6 +630,7 @@ function openBundle(root, requested, overwrite) {
   const sheetsDir = join2(root, "sheets");
   const pagesDir = join2(root, "pages");
   const attachmentsDir = join2(root, "attachments");
+  const ocrDir = join2(root, "ocr");
   try {
     if (existsSync(mdPath) && overwrite) {
       const owned = ownedPattern(stem), ownedCsv = ownedCsvPattern(stem), ownedPage = ownedPagePattern(stem);
@@ -644,6 +648,12 @@ function openBundle(root, requested, overwrite) {
       if (existsSync(attachmentsDir)) {
         for (const f of readdirSync(attachmentsDir)) if (attachments.has(f)) rmSync(join2(attachmentsDir, f), { force: true });
       }
+      rmSync(join2(root, `${stem}.pages.json`), { force: true });
+      rmSync(join2(root, `${stem}.words.json`), { force: true });
+      const ownedOcr = ownedOcrPattern(stem);
+      if (existsSync(ocrDir)) {
+        for (const f of readdirSync(ocrDir)) if (ownedOcr.test(f)) rmSync(join2(ocrDir, f), { force: true });
+      }
     }
     const lockId = randomBytes(6).toString("hex");
     const stagingDir = join2(imagesDir, `.stage-${lockId}`);
@@ -651,7 +661,8 @@ function openBundle(root, requested, overwrite) {
     mkdirSync2(stagingDir, { recursive: true });
     const pagesStagingDir = join2(pagesDir, `.stage-${lockId}`);
     const attachmentsStagingDir = join2(attachmentsDir, `.stage-${lockId}`);
-    return { root, stem, renamedFrom, renameReason, mdPath, lockPath, imagesDir, stagingDir, lockId, sheetsDir, sheetsStagingDir, pagesDir, pagesStagingDir, attachmentsDir, attachmentsStagingDir, manifest: /* @__PURE__ */ new Set(), csvManifest: /* @__PURE__ */ new Set(), pageManifest: /* @__PURE__ */ new Set(), attachmentManifest: /* @__PURE__ */ new Set(), sourceMap: /* @__PURE__ */ new Map() };
+    const ocrStagingDir = join2(ocrDir, `.stage-${lockId}`);
+    return { root, stem, renamedFrom, renameReason, mdPath, lockPath, imagesDir, stagingDir, lockId, sheetsDir, sheetsStagingDir, pagesDir, pagesStagingDir, attachmentsDir, attachmentsStagingDir, ocrDir, ocrStagingDir, pageStatsPath: join2(root, `${stem}.pages.json`), wordsPath: join2(root, `${stem}.words.json`), ocrManifest: /* @__PURE__ */ new Set(), manifest: /* @__PURE__ */ new Set(), csvManifest: /* @__PURE__ */ new Set(), pageManifest: /* @__PURE__ */ new Set(), attachmentManifest: /* @__PURE__ */ new Set(), sourceMap: /* @__PURE__ */ new Map() };
   } catch (e) {
     rmSync(lockPath, { force: true });
     throw e;
@@ -715,6 +726,45 @@ function publishPageImages(b, pageCount) {
   }
   rmSync(b.pagesStagingDir, { recursive: true, force: true });
 }
+function writePageStats(b, stats) {
+  writeFileSync2(b.pageStatsPath, `${JSON.stringify(stats, null, 2)}
+`, "utf8");
+}
+function publishWords(b) {
+  const staged = join2(b.stagingDir, "words.json");
+  try {
+    if (!existsSync(staged)) throw new Error("child staged no words.json");
+    renameSync(staged, b.wordsPath);
+    return null;
+  } catch (e) {
+    rmSync(b.wordsPath, { force: true });
+    return `write failed - ${e.message}`;
+  }
+}
+function publishSidecars(b) {
+  const sidecars = /* @__PURE__ */ new Map(), wordSidecars = /* @__PURE__ */ new Map();
+  if (!existsSync(b.ocrStagingDir)) return { sidecars, wordSidecars };
+  for (const dir of readdirSync(b.ocrStagingDir).sort()) {
+    const m = dir.match(/^p(\d+)$/);
+    if (!m) continue;
+    const pageDir = join2(b.ocrStagingDir, dir);
+    if (!existsSync(join2(pageDir, ".done"))) continue;
+    const file = `${b.stem}-${dir}.md`;
+    if (!existsSync(join2(pageDir, file))) continue;
+    mkdirSync2(b.ocrDir, { recursive: true });
+    renameSync(join2(pageDir, file), join2(b.ocrDir, file));
+    b.ocrManifest.add(file);
+    sidecars.set(Number(m[1]), join2(b.ocrDir, file));
+    const wfile = `${b.stem}-${dir}.words.json`;
+    if (existsSync(join2(pageDir, wfile))) {
+      renameSync(join2(pageDir, wfile), join2(b.ocrDir, wfile));
+      b.ocrManifest.add(wfile);
+      wordSidecars.set(Number(m[1]), join2(b.ocrDir, wfile));
+    }
+  }
+  rmSync(b.ocrStagingDir, { recursive: true, force: true });
+  return { sidecars, wordSidecars };
+}
 function publishAttachments(b) {
   if (!existsSync(b.attachmentsStagingDir)) return;
   for (const f of readdirSync(b.attachmentsStagingDir).sort()) {
@@ -772,6 +822,10 @@ function commitBundle(b, markdown) {
     fs.rmSync(b.attachmentsStagingDir, { recursive: true, force: true });
   } catch {
   }
+  try {
+    fs.rmSync(b.ocrStagingDir, { recursive: true, force: true });
+  } catch {
+  }
   try {
     fs.rmSync(b.lockPath, { force: true });
   } catch {
@@ -782,6 +836,10 @@ function abortBundle(b) {
   for (const f of b.csvManifest) rmSync(join2(b.sheetsDir, f), { force: true });
   for (const f of b.pageManifest) rmSync(join2(b.pagesDir, f), { force: true });
   for (const f of b.attachmentManifest) rmSync(join2(b.attachmentsDir, f), { force: true });
+  for (const f of b.ocrManifest) rmSync(join2(b.ocrDir, f), { force: true });
+  rmSync(b.ocrStagingDir, { recursive: true, force: true });
+  rmSync(b.pageStatsPath, { force: true });
+  rmSync(b.wordsPath, { force: true });
   rmSync(`${b.mdPath}.tmp`, { force: true });
   rmSync(b.stagingDir, { recursive: true, force: true });
   rmSync(b.sheetsStagingDir, { recursive: true, force: true });
@@ -820,7 +878,21 @@ function compactRanges(nums, maxEntries = 20) {
 }
 var INSTALL_HINT = "install Tesseract (see doc/doc-to-md.md), then rerun with ocr=true";
 var BARE_REASONS = ["fallback tier", "no Python backend"];
+var pageList = (pages) => `page${pages.length === 1 ? "" : "s"} ${pages.join(", ")}`;
+function forcedOcrLine(ocr) {
+  const clauses = [];
+  if (ocr.pages.length) clauses.push(`sidecars for ${pageList(ocr.pages)}`);
+  if (ocr.noText.length) clauses.push(`no text on ${pageList(ocr.noText)}`);
+  for (const page of ocr.ocrFailed) clauses.push(`failed on page ${page} (${ocr.ocrErrors[page] ?? "unknown error"})`);
+  if (ocr.budgetStopped.length) clauses.push(`budget-stopped ${pageList(ocr.budgetStopped)}`);
+  if (ocr.killed !== null) clauses.push(`page ${ocr.killed} killed the OCR child (timeout or crash - likely a compression bomb)`);
+  if (ocr.childError !== null) clauses.push(`OCR child failed before processing pages: ${ocr.childError}`);
+  if (ocr.notAttempted.length) clauses.push(`${pageList(ocr.notAttempted)} not attempted`);
+  const rerun = [...ocr.budgetStopped, ...ocr.notAttempted].sort((a, b) => a - b);
+  return `OCR: forced (${ocr.lang}) - ${clauses.join("; ")}${rerun.length ? ` - re-run with --pages ${rerun.join(",")}` : ""}`;
+}
 function ocrLine(ocr, type) {
+  if (ocr.mode === "all") return forcedOcrLine(ocr);
   const image = type === "image";
   const rerun = ocr.tesseract ? "rerun with ocr=true" : INSTALL_HINT;
   switch (ocr.status) {
@@ -896,6 +968,12 @@ function formatHandle(h) {
   if (h.sheetsDir) lines.push(`Sheets-Dir: ${h.sheetsDir}`);
   if (h.pagesDir && h.pageImageCount > 0) lines.push(`Pages-Dir: ${h.pagesDir} (${h.pageImageCount} pages)`);
   else if (h.pageImagesReason) lines.push(`Pages-Dir: none - ${h.pageImagesReason}`);
+  if (h.pageStatsPath) lines.push(`Page-Stats: ${h.pageStatsPath}`);
+  if (h.wordsPath) {
+    const bad = Object.keys(h.wordsErrors).map(Number).sort((a, b) => a - b);
+    lines.push(`Words: ${h.wordsPath}${bad.length ? ` (extraction failed for pages ${compactRanges(bad)}: ${h.wordsErrors[bad[0]]})` : ""}`);
+  } else if (h.wordsReason) lines.push(`Words: ${h.wordsReason}`);
+  if (h.ocrDir) lines.push(`OCR-Dir: ${h.ocrDir}`);
   lines.push(`Type: ${h.type}   Engine: ${h.engine}   Tier: ${h.tier}`);
   lines.push(`Page-Count: ${pageCountLabel(h)}   Pages: ${h.pages ? compactRanges(h.pages) : "all"}   Images: ${h.imageCount}   Size: ${formatSize2(h.bytes)} / ${h.lines} lines`);
   if (h.degraded) lines.push(`Degraded: ${h.degraded}`);
@@ -945,6 +1023,8 @@ var DOC_TO_MD_OPTIONS = [
   { key: "outputDir", flag: "--output-dir", type: "string", default: null, settable: false, help: "Bundle root for <stem>.md + images/; default a per-call temp dir. <stem> = basename without extension with [^A-Za-z0-9._-]+ -> _ (empty -> document); a second call on the same stem writes <stem>-2.md" },
   { key: "overwrite", flag: "--overwrite", type: "bool", default: false, settable: false, help: "Replace an existing completed <stem>.md bundle" },
   { key: "pageImages", flag: "--page-images", type: "bool", default: false, settable: false, help: "Also render every selected page to pages/<stem>-pNNN.<imageFormat> at imageDpi (PDF, PPTX, .doc, DOCX via LibreOffice); off by default" },
+  { key: "words", flag: "--words", type: "bool", default: false, settable: false, help: 'Write word positions: <stem>.words.json beside the Markdown lists every text-layer word of each selected page with its bbox (PDF points, top-left origin, display orientation; image inputs in source pixels) and the words inline OCR recognized, tagged source "text" or "ocr"; under --ocr-mode all the OCR words go to ocr/<stem>-pNNN.words.json beside each sidecar. Never triggers OCR. PDF and image inputs only.' },
+  { key: "ocrMode", flag: "--ocr-mode", type: "enum", default: "textless", settable: false, enumValues: ["textless", "all"], help: "OCR policy: textless (default) OCRs only pages with an empty text layer, inline; all OCRs every selected page and writes the recognized text to ocr/<stem>-pNNN.md sidecars, leaving the Markdown untouched. all requires --ocr and an explicit --pages selection (PDF, PPTX, DOC)." },
   { key: "primaryTimeoutMs", flag: "--primary-timeout", type: "int", default: 6e4, settable: true, env: "PI_DOC_TO_MD_CONVERT_TIMEOUT_MS", help: "pymupdf4llm tier and DOCX child (docx mode); also the unpdf tier" },
   { key: "fallbackTimeoutMs", flag: "--fallback-timeout", type: "int", default: 3e4, settable: true, help: "PyMuPDF get_text tier (including DOCX LibreOffice fallback); also PDF and DOCX info and Excel rendered views" },
   { key: "sofficeTimeoutMs", flag: "--soffice-timeout", type: "int", default: 12e4, settable: true, env: "PI_DOC_TO_MD_SOFFICE_TIMEOUT_MS", help: "DOCX/PPTX -> PDF via LibreOffice; also Excel rendered views" },
@@ -1051,11 +1131,40 @@ function resolveOptions(perCall, settings, env) {
     out[d.key] = value;
   }
   const o = out;
-  if (o.info && (o.pages !== null || o.outputDir !== null || o.overwrite || o.pageImages)) {
-    throw new UsageError("--info cannot be combined with --pages, --output-dir, --overwrite or --page-images");
+  if (o.info && (o.pages !== null || o.outputDir !== null || o.overwrite || o.pageImages || o.words || o.ocrMode === "all")) {
+    throw new UsageError("--info cannot be combined with --pages, --output-dir, --overwrite, --page-images, --words or --ocr-mode all");
   }
   return o;
 }
+var USAGE_PATTERNS = [
+  "Two-pass OCR (PDF, PPTX, DOC):",
+  "  1. <cmd> report.pdf --output-dir out --json",
+  '       -> "pageStatsPath" points at out/report.pages.json; pages with few',
+  '          chars and high imageCoverage are scans. "savedTo" is the Markdown.',
+  "  2. <cmd> report.pdf --output-dir out --ocr --ocr-mode all --pages 2,7 --json",
+  '       -> "ocr"."sidecars" maps 2 and 7 to out/ocr/report-2-p002.md and',
+  "          ...-p007.md (a second run in the same dir gets stem report-2);",
+  "          the Markdown of this run holds pages 2 and 7 only and equals what",
+  "          --pages 2,7 without --ocr would produce. Read sidecars and Markdown",
+  "          by the returned paths, never by guessing names.",
+  "  3. --ocr-mode all refuses to run without --ocr and an explicit --pages.",
+  '     The "ocr" object (OCR: line) names failed, budget-stopped, killed and',
+  "     not-attempted pages and the exact --pages to re-run.",
+  "  Details: doc/doc-to-md.md (bundle contract, failure buckets)."
+].join("\n");
+var usagePatterns = (cmd) => USAGE_PATTERNS.replaceAll("<cmd>", cmd);
+var BUNDLE_LAYOUT = [
+  { artifact: "<stem>.md", trigger: "always", content: "the Markdown", namedBy: "Saved-To: / savedTo" },
+  { artifact: "images/", trigger: "embedded or extracted figures", content: "image files linked from the Markdown", namedBy: "Images-Dir: / imagesDir" },
+  { artifact: "pages/<stem>-pNNN.<fmt>", trigger: "--page-images", content: "page renders at --image-dpi", namedBy: "Pages-Dir: / pagesDir" },
+  { artifact: "sheets/", trigger: "Excel input", content: "one CSV per non-empty worksheet", namedBy: "Sheets-Dir: / sheetsDir" },
+  { artifact: "attachments/", trigger: "email input", content: "saved attachments", namedBy: "Markdown attachment list" },
+  { artifact: "<stem>.pages.json", trigger: "Python PDF tiers (PDF, PPTX, DOC, DOCX via LibreOffice; not unpdf)", content: "per-page chars, image count, image coverage", namedBy: "Page-Stats: / pageStatsPath" },
+  { artifact: "<stem>.words.json", trigger: "--words", content: "per-page word boxes, source text/ocr", namedBy: "Words: / wordsPath" },
+  { artifact: "ocr/<stem>-pNNN.md", trigger: "--ocr --ocr-mode all", content: "recognized text of a forced page", namedBy: "OCR-Dir: / ocr.sidecars" },
+  { artifact: "ocr/<stem>-pNNN.words.json", trigger: "--ocr --ocr-mode all --words", content: "word boxes of that OCR", namedBy: "ocr.wordSidecars" }
+];
+var bundleLayoutText = () => BUNDLE_LAYOUT.map((r) => `  ${r.artifact.padEnd(28)} ${r.trigger}; ${r.content}; named by ${r.namedBy}`).join("\n");
 function renderHelp() {
   const row = (d) => `  ${(d.flag ?? "<path>").padEnd(26)} ${d.help}${d.default !== null && d.key !== "info" && d.key !== "overwrite" ? ` (default ${d.default})` : ""}`;
   return [
@@ -1067,8 +1176,13 @@ function renderHelp() {
     "Tunables (also settable under quiver.docToMd in settings.json; per-call > settings > default):",
     ...DOC_TO_MD_OPTIONS.filter((d) => d.settable).map(row),
     "",
-    "Result: a handle (Saved-To, Images-Dir, Page-Count, Outline ...). Read the Saved-To file for the Markdown.",
-    "Exit codes: 0 success, 1 runtime error, 2 usage error."
+    "Result: a handle (Saved-To, Images-Dir, Page-Stats, Page-Count, Outline ...). Read the Saved-To file for the Markdown.",
+    "Exit codes: 0 success, 1 runtime error, 2 usage error.",
+    "",
+    "Bundle layout:",
+    bundleLayoutText(),
+    "",
+    usagePatterns("pi-quiver doc-to-md")
   ].join("\n");
 }
 
@@ -1547,7 +1661,38 @@ function reconcileRenderMarkers(md, renderPages, fmt, sourceMap, reason) {
   if (/<!--rvs?:\d+-->/.test(md)) throw new Error("internal: unresolved render marker");
   return md;
 }
-var emptyOcr = (lang) => ({ status: "off", lang, textless: [], pages: [], noText: [], ocrFailed: [], budgetStopped: [], reason: null, tesseract: null });
+var emptyOcr = (lang) => ({ status: "off", lang, textless: [], pages: [], noText: [], ocrFailed: [], budgetStopped: [], reason: null, tesseract: null, mode: "textless", sidecars: {}, wordSidecars: {}, ocrErrors: {}, killed: null, notAttempted: [], childError: null });
+var emptyOutcome = () => ({ written: [], noText: [], ocrFailed: [], ocrErrors: {}, budgetStopped: [], killed: null, notAttempted: [], childError: null });
+var ocrPagesTag = (n) => `p${String(n).padStart(3, "0")}`;
+var SIDE_MARKER_RE = /^--- end of page\.page_number=\d+ ---$/;
+function sidecarHasText(dir) {
+  const file = readdirSync2(dir).find((f) => f.endsWith(".md"));
+  if (!file) return false;
+  return readFileSync2(join3(dir, file), "utf8").split("\n").slice(1).some((l) => l.trim() && !SIDE_MARKER_RE.test(l));
+}
+function recoverOcrPages(stagingDir, pages, detail) {
+  const out = emptyOutcome();
+  const activePath = join3(stagingDir, "active");
+  const active = existsSync2(activePath) ? Number(readFileSync2(activePath, "utf8").trim()) || null : null;
+  let sawPage = false;
+  for (const n of pages) {
+    const dir = join3(stagingDir, ocrPagesTag(n));
+    if (existsSync2(join3(dir, ".done"))) {
+      sawPage = true;
+      (sidecarHasText(dir) ? out.written : out.noText).push(n);
+    } else if (existsSync2(join3(dir, ".failed"))) {
+      sawPage = true;
+      out.ocrFailed.push(n);
+      out.ocrErrors[n] = readFileSync2(join3(dir, ".failed"), "utf8").trim() || "unknown error";
+    } else if (n === active) out.killed = n;
+    else out.notAttempted.push(n);
+  }
+  if (active === null && !sawPage) out.childError = detail;
+  return out;
+}
+function outcomeFromChild(j) {
+  return { ...emptyOutcome(), written: j.written ?? [], noText: j.noText ?? [], ocrFailed: j.ocrFailed ?? [], ocrErrors: j.ocrErrors ?? {}, budgetStopped: j.budgetStopped ?? [] };
+}
 var OCR_SENTINEL_RE = /\x00OCR ([^\x00]*)\x00/g;
 function resolveOcrLabels(md, sourceMap) {
   return md.replace(OCR_SENTINEL_RE, (_, key) => {
@@ -1559,7 +1704,7 @@ function handleOcr(tier, type, o, json) {
   if (tier === "unpdf") return o.ocr ? { ...emptyOcr(o.ocrLanguage), status: "unavailable", reason: "no Python backend" } : null;
   const x = json.ocr;
   if (!x || type !== "image" && !x.textless.length && !x.pages.length && !x.ocrFailed.length) return null;
-  return x;
+  return { ...emptyOcr(o.ocrLanguage), ...x };
 }
 async function convertDocument(o, signal, seams) {
   const s = { backend: (c) => getBackend(c, void 0, signal), runTier: runTierReal, office: tryConvertOffice, ...seams };
@@ -1568,10 +1713,17 @@ async function convertDocument(o, signal, seams) {
   if (!st || !st.isFile()) throw new Error(`Not a readable file: ${o.path}`);
   if (st.size === 0) throw new Error(`empty file: ${o.path}`);
   const type = classifyInput(inputPath);
+  const forced = o.ocrMode === "all";
+  if (forced) {
+    if (!o.ocr) throw new UsageError("--ocr-mode all requires --ocr");
+    if (o.pages === null) throw new UsageError('--ocr-mode all requires an explicit --pages selection (e.g. --pages 2,7); omitted pages and --pages "" mean all pages and are refused to keep OCR cost bounded');
+    if (type !== "pdf" && type !== "pptx" && type !== "doc") throw new UsageError("--ocr-mode all applies to PDF, PPTX and DOC inputs only (DOCX pages are page-break segments, not PDF pages; convert the DOCX to PDF first)");
+  }
   if (o.pages && (type === "html" || type === "image" || type === "email")) throw new Error(`--pages does not apply to ${type === "html" ? "HTML files" : type === "image" ? "images" : "email"}`);
   const isExcel = type === "xlsx" || type === "xlsm" || type === "xls";
   if (isExcel && o.pages) throw new Error("--pages does not apply to spreadsheets: worksheets have no stable page numbering");
   const backend = await s.backend({ pymupdfVersion: o.pymupdfVersion, warmTimeoutMs: o.warmTimeoutMs });
+  if (forced && backend.kind === "none") throw new Error(`--ocr-mode all cannot run: no Python backend (${backend.reason})`);
   if (isExcel && (backend.kind === "none" || !backend.xlsx)) throw new Error(`Excel conversion needs a Python backend with openpyxl, xlrd and pillow (${backend.kind === "none" ? backend.reason : `${backend.kind} lacks the Excel packages`}). ${EXCEL_REMEDY}`);
   if (type === "email" && extname3(inputPath).toLowerCase() === ".msg" && (backend.kind === "none" || !backend.email)) throw new Error(`MSG conversion needs the extract-msg package. Python backend: ${backendState(backend, "found without extract-msg")}. Remedy: install uv, or pip install extract-msg markdownify into that Python`);
   if (type === "email" && extname3(inputPath).toLowerCase() !== ".msg" && lacksDocx(backend)) throw new Error(`EML conversion needs the Python DOCX/HTML packages (mammoth, markdownify, python-docx). Python backend: ${backendState(backend, "found without mammoth/markdownify/python-docx")}. ${docxRemedy}`);
@@ -1580,11 +1732,12 @@ async function convertDocument(o, signal, seams) {
   let office = null;
   try {
     let pdfPath = inputPath;
-    const base = { path: inputPath, pages: o.pages, stagingDir: b.stagingDir, sheetsStagingDir: b.sheetsStagingDir, pageImages: o.pageImages, pagesStagingDir: b.pagesStagingDir, imageDpi: o.imageDpi, imageFormat: o.imageFormat, maxOutputBytes: o.maxOutputBytes, pymupdfVersion: o.pymupdfVersion, ocr: o.ocr, ocrLanguage: o.ocrLanguage, ocrBudgetMs: o.primaryTimeoutMs };
+    const base = { path: inputPath, pages: o.pages, ...o.words && (type === "pdf" || type === "image") ? { words: true } : {}, stagingDir: b.stagingDir, sheetsStagingDir: b.sheetsStagingDir, pageImages: o.pageImages, pagesStagingDir: b.pagesStagingDir, imageDpi: o.imageDpi, imageFormat: o.imageFormat, maxOutputBytes: o.maxOutputBytes, pymupdfVersion: o.pymupdfVersion, ocr: o.ocr && !forced, ocrLanguage: o.ocrLanguage, ocrBudgetMs: o.primaryTimeoutMs };
     let tier, engine, json, degraded = null, fallbackReason = null;
     let explicitBreaks = null;
     let notes = [];
     let officeRoute = null;
+    let copyReason = null;
     if (type === "html") {
       const prepared = await prepareHtml(inputPath, b.stagingDir);
       if (signal?.aborted) throw new Error("aborted");
@@ -1624,6 +1777,7 @@ async function convertDocument(o, signal, seams) {
       }
       if (signal?.aborted) throw new Error("aborted");
       if (json === void 0) {
+        copyReason = reason;
         clearStaging(b);
         const file = `original${extname3(inputPath).toLowerCase()}`;
         const dir = join3(b.stagingDir, "p1");
@@ -1757,6 +1911,36 @@ async function convertDocument(o, signal, seams) {
     if (type === "doc" || officeRoute !== null) degraded = DEGRADED_DOCX_OFFICE;
     if (officeRoute !== null) fallbackReason = fallbackReason ? `${officeRoute}; ${fallbackReason}` : officeRoute;
     if (tier === void 0 || engine === void 0 || json === void 0) throw new Error("internal: no tier produced output");
+    const pageStats = json.pageStats ?? null;
+    if (pageStats) writePageStats(b, pageStats);
+    let wordsPath = null, wordsReason = null;
+    const wordsErrors = {};
+    const takeWordsErrors = (j) => {
+      for (const [k, v] of Object.entries(j?.wordsErrors ?? {})) {
+        const page = Number(k);
+        if (Number.isFinite(page)) wordsErrors[page] = v;
+      }
+    };
+    if (o.words) {
+      if (type !== "pdf" && type !== "image") wordsReason = `none - word positions apply to PDF and image inputs only (${type})`;
+      else if (tier === "unpdf") wordsReason = "none - unpdf tier has no page geometry";
+      else if (engine === "copy") wordsReason = `none - image copied without conversion (${copyReason})`;
+      else if (json?.words === true) {
+        wordsReason = publishWords(b);
+        if (wordsReason === null) wordsPath = b.wordsPath;
+      } else wordsReason = `write failed - ${json?.wordsErrors?.file ?? "child reported no words document"}`;
+      takeWordsErrors(json);
+    }
+    let ocr = handleOcr(tier, type, o, json);
+    if (forced) {
+      const r = await s.runTier("ocr-pages", { path: pdfPath, pages: o.pages, ...o.words && type === "pdf" ? { words: true } : {}, stem: b.stem, ocrLanguage: o.ocrLanguage, ocrBudgetMs: o.primaryTimeoutMs, stagingDir: b.ocrStagingDir, dpi: o.imageDpi, maxOutputBytes: o.maxOutputBytes, pymupdfVersion: o.pymupdfVersion }, b, signal, o.primaryTimeoutMs, backend);
+      if (signal?.aborted || !r.ok && "reason" in r && r.reason === "aborted") throw new Error("aborted");
+      if (r.ok && r.json.status === "unavailable") throw new Error(`OCR unavailable: ${r.json.reason} (install Tesseract; see doc/doc-to-md.md)`);
+      const outcome = r.ok && r.json.status === "ran" ? outcomeFromChild(r.json) : recoverOcrPages(b.ocrStagingDir, o.pages, !r.ok ? "userError" in r ? r.userError : `${r.reason}${detailSuffix(r)}` : "malformed child output");
+      const { sidecars, wordSidecars } = publishSidecars(b);
+      ocr = { ...emptyOcr(o.ocrLanguage), ...json.ocr, status: "ran", reason: null, mode: "all", pages: outcome.written, noText: outcome.noText, ocrFailed: outcome.ocrFailed, ocrErrors: outcome.ocrErrors, budgetStopped: outcome.budgetStopped, killed: outcome.killed, notAttempted: outcome.notAttempted, childError: outcome.childError, sidecars: Object.fromEntries(sidecars), wordSidecars: Object.fromEntries(wordSidecars) };
+      if (o.words && r.ok) takeWordsErrors(r.json);
+    }
     if (!isExcel) notes = [...notes, ...json.notes ?? []];
     if (b.renamedFrom) notes.splice(notes[0]?.startsWith("preview truncated:") ? 1 : 0, 0, `renamed to ${b.stem} (${b.renameReason})`);
     const pageImagesReason = !o.pageImages || b.pageManifest.size || tier === "primary" || tier === "fallback" ? null : tier === "unpdf" ? "page images need the Python backend" : `${type} has no page geometry`;
@@ -1773,7 +1957,7 @@ async function convertDocument(o, signal, seams) {
 ` : "") + body;
     commitBundle(b, markdown);
     const outline = scanOutline(markdown, o.outlineMaxEntries);
-    const details = { path: inputPath, backend: backend.kind, pymupdfVersion: o.pymupdfVersion, inputType: type, file: b.mdPath, outputDir: b.root, savedTo: b.mdPath, imagesDir: b.imagesDir, sheetsDir: b.csvManifest.size ? b.sheetsDir : null, pagesDir: b.pageManifest.size ? b.pagesDir : null, pageImageCount: b.pageManifest.size, pageImagesReason, type, engine, tier, pageCount: json.pageCount ?? null, pages: o.pages, explicitBreaks, imageCount: b.manifest.size, bytes: Buffer.byteLength(markdown, "utf8"), lines: markdown.split("\n").length, degraded, fallbackReason, failedPages: (json.failedPages ?? []).map((f) => f.page), emptyPages: json.emptyPages ?? [], notes, outline: outline.entries, outlineTotal: outline.total, ocr: handleOcr(tier, type, o, json) };
+    const details = { path: inputPath, backend: backend.kind, pymupdfVersion: o.pymupdfVersion, inputType: type, file: b.mdPath, outputDir: b.root, savedTo: b.mdPath, imagesDir: b.imagesDir, sheetsDir: b.csvManifest.size ? b.sheetsDir : null, pagesDir: b.pageManifest.size ? b.pagesDir : null, pageImageCount: b.pageManifest.size, pageImagesReason, type, engine, tier, pageCount: json.pageCount ?? null, pages: o.pages, explicitBreaks, imageCount: b.manifest.size, bytes: Buffer.byteLength(markdown, "utf8"), lines: markdown.split("\n").length, degraded, fallbackReason, failedPages: (json.failedPages ?? []).map((f) => f.page), emptyPages: json.emptyPages ?? [], notes, outline: outline.entries, outlineTotal: outline.total, ocr, pageStats, pageStatsPath: pageStats ? b.pageStatsPath : null, ocrDir: b.ocrManifest.size ? b.ocrDir : null, wordsPath, wordsReason, wordsErrors };
     return { output: formatHandle(details), details };
   } catch (e) {
     abortBundle(b);
@@ -1819,7 +2003,7 @@ async function inspectDocument(o, signal, seams) {
 }
 
 // bin/pi-quiver.ts
-var USAGE = 'Usage: pi-quiver fetch <url> [--method GET|HEAD|POST] [--header "K: V"]... [--body <str>] [--raw] [--timeout-ms <n>]\n       pi-quiver doc-to-md [--json] [--info] [--page-images] [--pages <spec>] [--output-dir <dir>] [--overwrite] [tunable flags] <path>   (--help for all flags)';
+var USAGE = 'Usage: pi-quiver fetch <url> [--method GET|HEAD|POST] [--header "K: V"]... [--body <str>] [--raw] [--timeout-ms <n>]\n       pi-quiver doc-to-md [--json] [--info] [--page-images] [--words] [--pages <spec>] [--output-dir <dir>] [--overwrite] [--ocr] [--ocr-mode textless|all] [tunable flags] <path>   (--help for all flags)';
 function parseDocToMd(rest) {
   if (rest.includes("--help") || rest.includes("-h")) return { ok: true, cmd: "doc-to-md-help" };
   const json = rest.includes("--json");
@@ -1974,6 +2158,12 @@ ${USAGE}
 `);
       return 0;
     } catch (err) {
+      if (err instanceof UsageError) {
+        process.stderr.write(`${err.message}
+${USAGE}
+`);
+        return 2;
+      }
       process.stderr.write(`doc-to-md failed: ${err instanceof Error ? err.message : String(err)}
 `);
       return 1;

package/extensions/doc_to_md.ts

@@ -35,7 +35,7 @@ export default function docToMdExtension(pi: ExtensionAPI) {
 		name: "doc_to_md",
 		label: "Convert doc to Markdown bundle",
 		description:
-			"Convert a local PDF/DOCX/DOC/PPTX/XLSX/XLSM/XLS, email (.msg/.eml), HTML (.html/.htm) or image (.png .jpg .jpeg .tif .tiff .bmp .gif) to a Markdown bundle on disk and return a handle (Saved-To, Images-Dir, Page-Count, Outline, diagnostics) - the Markdown itself is never inlined; read the Saved-To file (offset/limit) for content. `info: true` returns page count, metadata and TOC (or the sheet inventory) without converting - use it to pick `pages`. `pages` selects inclusive 1-based pages (PDF/PPTX) or explicit-page-break segments (DOCX; rejected when the file has none); every selected PDF/PPTX page or DOCX segment when `pageCount > 1` ends with `--- end of page.page_number=N ---`. Images are always extracted into images/ with relative links. Primary engine pymupdf4llm, fallback PyMuPDF text (degraded, marked), pure-JS unpdf only when no Python backend exists. Excel yields a sheet inventory (worksheets + chartsheets), a full CSV per non-empty worksheet under sheets/, a bounded preview with formulas and cached values, merged/hidden disclosure, and rendered chart views when LibreOffice is available. DOCX converts directly (mammoth; python-docx text fallback) and keeps heading styles, so the handle Outline lists `L<line>` and `p<segment>` per heading; LibreOffice (soffice) is optional for DOCX (fallback route, degraded) and Excel rendered views, and required for PPTX. Use `outputDir` for a durable bundle; without it the bundle lands in a per-call temp dir. Input must be a local file path (use fetch first for URLs). HTML converts without Readability (markdownify; Turndown fallback): local and data: images are copied into images/, remote images stay links. Pages without a text layer keep their picture in images/, or pages/ with `pageImages: true`; image inputs keep their picture in images/. `pageImages: true` writes page renders to pages/<stem>-pNNN, and email attachments are saved under attachments/. OCR is off by default: `ocr: true` adds recognized text (labeled as OCR) when Tesseract language data is installed; the handle's OCR: line says what ran.",
+			"Convert a local PDF/DOCX/DOC/PPTX/XLSX/XLSM/XLS, email (.msg/.eml), HTML (.html/.htm) or image (.png .jpg .jpeg .tif .tiff .bmp .gif) to a Markdown bundle on disk and return a handle (Saved-To, Images-Dir, Page-Count, Outline, diagnostics) - the Markdown itself is never inlined; read the Saved-To file (offset/limit) for content. `info: true` returns page count, metadata and TOC (or the sheet inventory) without converting - use it to pick `pages`. `pages` selects inclusive 1-based pages (PDF/PPTX) or explicit-page-break segments (DOCX; rejected when the file has none); every selected PDF/PPTX page or DOCX segment when `pageCount > 1` ends with `--- end of page.page_number=N ---`. Images are always extracted into images/ with relative links. Primary engine pymupdf4llm, fallback PyMuPDF text (degraded, marked), pure-JS unpdf only when no Python backend exists. Excel yields a sheet inventory (worksheets + chartsheets), a full CSV per non-empty worksheet under sheets/, a bounded preview with formulas and cached values, merged/hidden disclosure, and rendered chart views when LibreOffice is available. DOCX converts directly (mammoth; python-docx text fallback) and keeps heading styles, so the handle Outline lists `L<line>` and `p<segment>` per heading; LibreOffice (soffice) is optional for DOCX (fallback route, degraded) and Excel rendered views, and required for PPTX. Use `outputDir` for a durable bundle; without it the bundle lands in a per-call temp dir. Input must be a local file path (use fetch first for URLs). HTML converts without Readability (markdownify; Turndown fallback): local and data: images are copied into images/, remote images stay links. Pages without a text layer keep their picture in images/, or pages/ with `pageImages: true`; image inputs keep their picture in images/. `pageImages: true` writes page renders to pages/<stem>-pNNN, and email attachments are saved under attachments/. OCR is off by default: `ocr: true` adds recognized text (labeled as OCR) when Tesseract language data is installed; the handle's OCR: line says what ran. Two-pass OCR: read Page-Stats first, then re-run with ocr: true, ocrMode: \"all\" and an explicit pages selection to get ocr/ sidecars for the pages you name.",
 		promptSnippet: "Convert a local PDF/DOCX/DOC/PPTX/XLSX/XLSM/XLS/MSG/EML/HTML/image to a Markdown bundle (handle returned; read Saved-To)",
 		parameters: buildSchema(),
 

package/lib/doc-to-md-bundle.ts

@@ -1,23 +1,26 @@
 /**
  * Bundle protocol: a call owns `<stem>` for its whole duration via `<stem>.md.lock`;
- * children stage assets under `images/`, `sheets/`, `pages/`, and `attachments/` staging dirs;
- * Node publishes them to stem-prefixed files in those four dirs and records every file it
+ * children stage assets under `images/`, `sheets/`, `pages/`, `attachments/`, and `ocr/` staging dirs;
+ * Node publishes them to stem-prefixed files in those asset dirs and records every file it
  * wrote in a manifest, and commits `<stem>.md` atomically (tmp + rename).
  */
 import fs, { closeSync, existsSync, mkdirSync, openSync, readdirSync, readFileSync, renameSync, rmSync, statSync, unlinkSync, writeFileSync } from "node:fs";
 import { tmpdir } from "node:os";
 import { extname, join, resolve } from "node:path";
 import { randomBytes } from "node:crypto";
+import type { PageStat } from "./doc-to-md-handle.ts";
 
 export interface Bundle {
 	root: string; stem: string; renamedFrom: string | null; renameReason: string | null; mdPath: string; lockPath: string; imagesDir: string; stagingDir: string; lockId: string;
 	sheetsDir: string; sheetsStagingDir: string;
 	pagesDir: string; pagesStagingDir: string;
 	attachmentsDir: string; attachmentsStagingDir: string;
+	ocrDir: string; ocrStagingDir: string; pageStatsPath: string; wordsPath: string;
 	manifest: Set<string>;
 	csvManifest: Set<string>;
 	pageManifest: Set<string>;
 	attachmentManifest: Set<string>;
+	ocrManifest: Set<string>;
 	sourceMap: Map<string, string>;
 }
 
@@ -33,6 +36,8 @@ export function ownedCsvPattern(stem: string): RegExp {
 
 export function ownedPagePattern(stem: string): RegExp { return new RegExp(`^${escRe(stem)}-p\\d+\\.[a-z0-9]+$`); }
 
+export function ownedOcrPattern(stem: string): RegExp { return new RegExp(`^${escRe(stem)}-p\\d+(?:\\.words\\.json|\\.md)$`); }
+
 const FILE_LINK_RE = /\[[^\]]*\]\(\s*((?:sheets|attachments)\/[^)\s]+)\s*\)/g;
 const IMG_LINK_RE = /!\[[^\]]*\]\(\s*(?:<([^>]*)>|([^)]*?))\s*\)|<img\b[^>]*\bsrc\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s>"']+))/gi;
 
@@ -75,6 +80,7 @@ export function openBundle(root: string, requested: string, overwrite: boolean):
 	const sheetsDir = join(root, "sheets");
 	const pagesDir = join(root, "pages");
 	const attachmentsDir = join(root, "attachments");
+	const ocrDir = join(root, "ocr");
 	try {
 		if (existsSync(mdPath) && overwrite) {
 			const owned = ownedPattern(stem), ownedCsv = ownedCsvPattern(stem), ownedPage = ownedPagePattern(stem);
@@ -87,6 +93,10 @@ export function openBundle(root: string, requested: string, overwrite: boolean):
 			if (existsSync(sheetsDir)) for (const f of readdirSync(sheetsDir)) if (ownedCsv.test(f)) rmSync(join(sheetsDir, f), { force: true });
 			if (existsSync(pagesDir)) for (const f of readdirSync(pagesDir)) if (ownedPage.test(f)) rmSync(join(pagesDir, f), { force: true });
 			if (existsSync(attachmentsDir)) for (const f of readdirSync(attachmentsDir)) if (attachments.has(f)) rmSync(join(attachmentsDir, f), { force: true });
+			rmSync(join(root, `${stem}.pages.json`), { force: true });
+			rmSync(join(root, `${stem}.words.json`), { force: true });
+			const ownedOcr = ownedOcrPattern(stem);
+			if (existsSync(ocrDir)) for (const f of readdirSync(ocrDir)) if (ownedOcr.test(f)) rmSync(join(ocrDir, f), { force: true });
 		}
 		const lockId = randomBytes(6).toString("hex");
 		const stagingDir = join(imagesDir, `.stage-${lockId}`);
@@ -94,7 +104,8 @@ export function openBundle(root: string, requested: string, overwrite: boolean):
 		mkdirSync(stagingDir, { recursive: true });
 		const pagesStagingDir = join(pagesDir, `.stage-${lockId}`);
 		const attachmentsStagingDir = join(attachmentsDir, `.stage-${lockId}`);
-		return { root, stem, renamedFrom, renameReason, mdPath, lockPath, imagesDir, stagingDir, lockId, sheetsDir, sheetsStagingDir, pagesDir, pagesStagingDir, attachmentsDir, attachmentsStagingDir, manifest: new Set(), csvManifest: new Set(), pageManifest: new Set(), attachmentManifest: new Set(), sourceMap: new Map() };
+		const ocrStagingDir = join(ocrDir, `.stage-${lockId}`);
+		return { root, stem, renamedFrom, renameReason, mdPath, lockPath, imagesDir, stagingDir, lockId, sheetsDir, sheetsStagingDir, pagesDir, pagesStagingDir, attachmentsDir, attachmentsStagingDir, ocrDir, ocrStagingDir, pageStatsPath: join(root, `${stem}.pages.json`), wordsPath: join(root, `${stem}.words.json`), ocrManifest: new Set(), manifest: new Set(), csvManifest: new Set(), pageManifest: new Set(), attachmentManifest: new Set(), sourceMap: new Map() };
 	} catch (e) { rmSync(lockPath, { force: true }); throw e; }
 }
 
@@ -160,6 +171,49 @@ export function publishPageImages(b: Bundle, pageCount: number): void {
 	rmSync(b.pagesStagingDir, { recursive: true, force: true });
 }
 
+export function writePageStats(b: Bundle, stats: PageStat[]): void {
+	writeFileSync(b.pageStatsPath, `${JSON.stringify(stats, null, 2)}\n`, "utf8");
+}
+
+/** Rename within the bundle root keeps the complete words document atomic. */
+export function publishWords(b: Bundle): string | null {
+	const staged = join(b.stagingDir, "words.json");
+	try {
+		if (!existsSync(staged)) throw new Error("child staged no words.json");
+		renameSync(staged, b.wordsPath);
+		return null;
+	} catch (e) {
+		rmSync(b.wordsPath, { force: true });
+		return `write failed - ${(e as Error).message}`;
+	}
+}
+
+/** Move `.done`-gated Markdown and words sidecars into `ocr/`; drop partial dirs and the checkpoint. Returns page -> absolute paths for each kind. */
+export function publishSidecars(b: Bundle): { sidecars: Map<number, string>; wordSidecars: Map<number, string> } {
+	const sidecars = new Map<number, string>(), wordSidecars = new Map<number, string>();
+	if (!existsSync(b.ocrStagingDir)) return { sidecars, wordSidecars };
+	for (const dir of readdirSync(b.ocrStagingDir).sort()) {
+		const m = dir.match(/^p(\d+)$/);
+		if (!m) continue;
+		const pageDir = join(b.ocrStagingDir, dir);
+		if (!existsSync(join(pageDir, ".done"))) continue;
+		const file = `${b.stem}-${dir}.md`;
+		if (!existsSync(join(pageDir, file))) continue;
+		mkdirSync(b.ocrDir, { recursive: true });
+		renameSync(join(pageDir, file), join(b.ocrDir, file));
+		b.ocrManifest.add(file);
+		sidecars.set(Number(m[1]), join(b.ocrDir, file));
+		const wfile = `${b.stem}-${dir}.words.json`;
+		if (existsSync(join(pageDir, wfile))) {
+			renameSync(join(pageDir, wfile), join(b.ocrDir, wfile));
+			b.ocrManifest.add(wfile);
+			wordSidecars.set(Number(m[1]), join(b.ocrDir, wfile));
+		}
+	}
+	rmSync(b.ocrStagingDir, { recursive: true, force: true });
+	return { sidecars, wordSidecars };
+}
+
 export function publishAttachments(b: Bundle): void {
 	if (!existsSync(b.attachmentsStagingDir)) return;
 	for (const f of readdirSync(b.attachmentsStagingDir).sort()) {
@@ -208,6 +262,7 @@ export function commitBundle(b: Bundle, markdown: string): void {
 	try { fs.rmSync(b.sheetsStagingDir, { recursive: true, force: true }); } catch { /* best-effort */ }
 	try { fs.rmSync(b.pagesStagingDir, { recursive: true, force: true }); } catch { /* best-effort */ }
 	try { fs.rmSync(b.attachmentsStagingDir, { recursive: true, force: true }); } catch { /* best-effort */ }
+	try { fs.rmSync(b.ocrStagingDir, { recursive: true, force: true }); } catch { /* best-effort */ }
 	try { fs.rmSync(b.lockPath, { force: true }); } catch { /* Markdown is published; cleanup is best-effort. */ }
 }
 
@@ -216,6 +271,10 @@ export function abortBundle(b: Bundle): void {
 	for (const f of b.csvManifest) rmSync(join(b.sheetsDir, f), { force: true });
 	for (const f of b.pageManifest) rmSync(join(b.pagesDir, f), { force: true });
 	for (const f of b.attachmentManifest) rmSync(join(b.attachmentsDir, f), { force: true });
+	for (const f of b.ocrManifest) rmSync(join(b.ocrDir, f), { force: true });
+	rmSync(b.ocrStagingDir, { recursive: true, force: true });
+	rmSync(b.pageStatsPath, { force: true });
+	rmSync(b.wordsPath, { force: true });
 	rmSync(`${b.mdPath}.tmp`, { force: true });
 	rmSync(b.stagingDir, { recursive: true, force: true });
 	rmSync(b.sheetsStagingDir, { recursive: true, force: true });

package/lib/doc-to-md-core.ts

@@ -12,9 +12,9 @@ import { type ChildProcess, spawn } from "node:child_process";
 import { homedir, tmpdir } from "node:os";
 import { basename, dirname, extname, isAbsolute, join, resolve } from "node:path";
 import { fileURLToPath, pathToFileURL } from "node:url";
-import { type Bundle, abortBundle, commitBundle, openBundle, publishAttachments, publishPageImages, publishSheetCsvs, publishSheetImages, publishStaged, rewriteLinks, tempBundleRoot, validateImageLinks } from "./doc-to-md-bundle.ts";
-import { type Engine, type HandleData, type InfoData, type OcrInfo, type Tier, formatHandle, formatInfoHandle, scanOutline, type SheetInfo, type TocEntry } from "./doc-to-md-handle.ts";
-import { type DocToMdOptions, type InputType, IMAGE_EXTS, TUNABLE_DEFAULTS, classifyInput, sanitizeStem } from "./doc-to-md-options.ts";
+import { type Bundle, abortBundle, commitBundle, openBundle, publishAttachments, publishSidecars, publishWords, writePageStats, publishPageImages, publishSheetCsvs, publishSheetImages, publishStaged, rewriteLinks, tempBundleRoot, validateImageLinks } from "./doc-to-md-bundle.ts";
+import { type Engine, type HandleData, type InfoData, type OcrInfo, type PageStat, type Tier, formatHandle, formatInfoHandle, scanOutline, type SheetInfo, type TocEntry } from "./doc-to-md-handle.ts";
+import { type DocToMdOptions, type InputType, IMAGE_EXTS, TUNABLE_DEFAULTS, UsageError, classifyInput, sanitizeStem } from "./doc-to-md-options.ts";
 
 export * from "./doc-to-md-options.ts";
 export { compactRanges, formatHandle, formatInfoHandle, formatSize, scanOutline } from "./doc-to-md-handle.ts";
@@ -466,8 +466,8 @@ const lacksDocx = (b: Backend) => b.kind === "none" || !b.docx;
 const clearStaging = (b: Pick<Bundle, "stagingDir">) => { for (const f of readdirSync(b.stagingDir)) rmSync(join(b.stagingDir, f), { recursive: true, force: true }); };
 export const EXCEL_REMEDY = "Remedy: install uv, or pip install openpyxl xlrd pillow";
 
-export type Mode = "html" | "image" | "info" | "pdf-primary" | "pdf-fallback" | "xlsx" | "pdf-text" | "render-pages" | "docx" | "email";
-export interface TierJson { pageImages?: { page: number; file: string }[]; ocr?: OcrInfo; markdown?: string; pages?: number[]; pageCount?: number; emptyPages?: number[]; failedPages?: { page: number; error: string }[]; notes?: string[]; images?: { sheetIndex: number; file: string }[]; metadata?: Record<string, string>; toc?: [number, string, number | null][]; explicitBreaks?: number; engine?: string; degraded?: boolean; fallbackReason?: string | null; sheets?: SheetInfo[]; renderPages?: number[]; sheetCount?: number; ok?: boolean; reason?: string; rendered?: { idx: number; file: string; dpi: number }[]; failed?: { idx: number; reason: string }[]; }
+export type Mode = "html" | "image" | "info" | "pdf-primary" | "pdf-fallback" | "xlsx" | "pdf-text" | "render-pages" | "docx" | "email" | "ocr-pages";
+export interface TierJson { words?: boolean; wordsErrors?: Record<string, string>; pageStats?: PageStat[]; status?: string; written?: number[]; noText?: number[]; ocrFailed?: number[]; ocrErrors?: Record<string, string>; budgetStopped?: number[]; pageImages?: { page: number; file: string }[]; ocr?: OcrInfo; markdown?: string; pages?: number[]; pageCount?: number; emptyPages?: number[]; failedPages?: { page: number; error: string }[]; notes?: string[]; images?: { sheetIndex: number; file: string }[]; metadata?: Record<string, string>; toc?: [number, string, number | null][]; explicitBreaks?: number; engine?: string; degraded?: boolean; fallbackReason?: string | null; sheets?: SheetInfo[]; renderPages?: number[]; sheetCount?: number; ok?: boolean; reason?: string; rendered?: { idx: number; file: string; dpi: number }[]; failed?: { idx: number; reason: string }[]; }
 export type TierResult = { ok: true; json: TierJson } | { ok: false; reason: string; detail?: string } | { ok: false; userError: string; pageCount?: number };
 
 export interface PipelineSeams {
@@ -490,7 +490,7 @@ function unpdfWorkerPath(): string {
 	], existsSync);
 }
 
-async function runTierReal(mode: Mode, childOptions: Record<string, unknown>, _b: Pick<Bundle, "stagingDir">, signal: AbortSignal | undefined, timeoutMs: number, backend: Backend): Promise<TierResult> {
+export async function runTierReal(mode: Mode, childOptions: Record<string, unknown>, _b: Pick<Bundle, "stagingDir">, signal: AbortSignal | undefined, timeoutMs: number, backend: Backend): Promise<TierResult> {
 	const cfg = { pymupdfVersion: String(childOptions.pymupdfVersion), warmTimeoutMs: 0 };
 	let cmd: string, args: string[];
 	if (mode === "pdf-text" || (mode === "info" && backend.kind === "none")) { cmd = process.execPath; args = [unpdfWorkerPath(), mode]; }
@@ -536,7 +536,39 @@ export function reconcileRenderMarkers(md: string, renderPages: number[], fmt: s
 	return md;
 }
 
-export const emptyOcr = (lang: string): OcrInfo => ({ status: "off", lang, textless: [], pages: [], noText: [], ocrFailed: [], budgetStopped: [], reason: null, tesseract: null });
+export const emptyOcr = (lang: string): OcrInfo => ({ status: "off", lang, textless: [], pages: [], noText: [], ocrFailed: [], budgetStopped: [], reason: null, tesseract: null, mode: "textless", sidecars: {}, wordSidecars: {}, ocrErrors: {}, killed: null, notAttempted: [], childError: null });
+
+export interface OcrPagesOutcome { written: number[]; noText: number[]; ocrFailed: number[]; ocrErrors: Record<number, string>; budgetStopped: number[]; killed: number | null; notAttempted: number[]; childError: string | null; }
+const emptyOutcome = (): OcrPagesOutcome => ({ written: [], noText: [], ocrFailed: [], ocrErrors: {}, budgetStopped: [], killed: null, notAttempted: [], childError: null });
+const ocrPagesTag = (n: number) => `p${String(n).padStart(3, "0")}`;
+const SIDE_MARKER_RE = /^--- end of page\.page_number=\d+ ---$/;
+
+function sidecarHasText(dir: string): boolean {
+	const file = readdirSync(dir).find((f) => f.endsWith(".md"));
+	if (!file) return false;
+	return readFileSync(join(dir, file), "utf8").split("\n").slice(1).some((l) => l.trim() && !SIDE_MARKER_RE.test(l));
+}
+
+/** Rebuild the outcome from staging markers after the child died or returned garbage. */
+export function recoverOcrPages(stagingDir: string, pages: number[], detail: string): OcrPagesOutcome {
+	const out = emptyOutcome();
+	const activePath = join(stagingDir, "active");
+	const active = existsSync(activePath) ? Number(readFileSync(activePath, "utf8").trim()) || null : null;
+	let sawPage = false;
+	for (const n of pages) {
+		const dir = join(stagingDir, ocrPagesTag(n));
+		if (existsSync(join(dir, ".done"))) { sawPage = true; (sidecarHasText(dir) ? out.written : out.noText).push(n); }
+		else if (existsSync(join(dir, ".failed"))) { sawPage = true; out.ocrFailed.push(n); out.ocrErrors[n] = readFileSync(join(dir, ".failed"), "utf8").trim() || "unknown error"; }
+		else if (n === active) out.killed = n;
+		else out.notAttempted.push(n);
+	}
+	if (active === null && !sawPage) out.childError = detail;
+	return out;
+}
+
+function outcomeFromChild(j: TierJson): OcrPagesOutcome {
+	return { ...emptyOutcome(), written: j.written ?? [], noText: j.noText ?? [], ocrFailed: j.ocrFailed ?? [], ocrErrors: j.ocrErrors ?? {}, budgetStopped: j.budgetStopped ?? [] };
+}
 const OCR_SENTINEL_RE = /\x00OCR ([^\x00]*)\x00/g;
 
 /** Child OCR labels name staged files; the published name exists only after publishStaged. */
@@ -551,7 +583,7 @@ function handleOcr(tier: Tier, type: InputType, o: DocToMdOptions, json: TierJso
 	if (tier === "unpdf") return o.ocr ? { ...emptyOcr(o.ocrLanguage), status: "unavailable", reason: "no Python backend" } : null;
 	const x = json.ocr;
 	if (!x || (type !== "image" && !x.textless.length && !x.pages.length && !x.ocrFailed.length)) return null;
-	return x;
+	return { ...emptyOcr(o.ocrLanguage), ...x };
 }
 
 export async function convertDocument(o: DocToMdOptions, signal?: AbortSignal, seams?: Partial<PipelineSeams>): Promise<ConvertOutcome> {
@@ -561,10 +593,17 @@ export async function convertDocument(o: DocToMdOptions, signal?: AbortSignal, s
 	if (!st || !st.isFile()) throw new Error(`Not a readable file: ${o.path}`);
 	if (st.size === 0) throw new Error(`empty file: ${o.path}`);
 	const type = classifyInput(inputPath);
+	const forced = o.ocrMode === "all";
+	if (forced) {
+		if (!o.ocr) throw new UsageError("--ocr-mode all requires --ocr");
+		if (o.pages === null) throw new UsageError('--ocr-mode all requires an explicit --pages selection (e.g. --pages 2,7); omitted pages and --pages "" mean all pages and are refused to keep OCR cost bounded');
+		if (type !== "pdf" && type !== "pptx" && type !== "doc") throw new UsageError("--ocr-mode all applies to PDF, PPTX and DOC inputs only (DOCX pages are page-break segments, not PDF pages; convert the DOCX to PDF first)");
+	}
 	if (o.pages && (type === "html" || type === "image" || type === "email")) throw new Error(`--pages does not apply to ${type === "html" ? "HTML files" : type === "image" ? "images" : "email"}`);
 	const isExcel = type === "xlsx" || type === "xlsm" || type === "xls";
 	if (isExcel && o.pages) throw new Error("--pages does not apply to spreadsheets: worksheets have no stable page numbering");
 	const backend = await s.backend({ pymupdfVersion: o.pymupdfVersion, warmTimeoutMs: o.warmTimeoutMs });
+	if (forced && backend.kind === "none") throw new Error(`--ocr-mode all cannot run: no Python backend (${backend.reason})`);
 	if (isExcel && (backend.kind === "none" || !backend.xlsx)) throw new Error(`Excel conversion needs a Python backend with openpyxl, xlrd and pillow (${backend.kind === "none" ? backend.reason : `${backend.kind} lacks the Excel packages`}). ${EXCEL_REMEDY}`);
 	if (type === "email" && extname(inputPath).toLowerCase() === ".msg" && (backend.kind === "none" || !backend.email)) throw new Error(`MSG conversion needs the extract-msg package. Python backend: ${backendState(backend, "found without extract-msg")}. Remedy: install uv, or pip install extract-msg markdownify into that Python`);
 	if (type === "email" && extname(inputPath).toLowerCase() !== ".msg" && lacksDocx(backend)) throw new Error(`EML conversion needs the Python DOCX/HTML packages (mammoth, markdownify, python-docx). Python backend: ${backendState(backend, "found without mammoth/markdownify/python-docx")}. ${docxRemedy}`);
@@ -573,11 +612,12 @@ export async function convertDocument(o: DocToMdOptions, signal?: AbortSignal, s
 	let office: { pdfPath: string; cleanup: () => void } | null = null;
 	try {
 		let pdfPath = inputPath;
-		const base = { path: inputPath, pages: o.pages, stagingDir: b.stagingDir, sheetsStagingDir: b.sheetsStagingDir, pageImages: o.pageImages, pagesStagingDir: b.pagesStagingDir, imageDpi: o.imageDpi, imageFormat: o.imageFormat, maxOutputBytes: o.maxOutputBytes, pymupdfVersion: o.pymupdfVersion, ocr: o.ocr, ocrLanguage: o.ocrLanguage, ocrBudgetMs: o.primaryTimeoutMs };
+		const base = { path: inputPath, pages: o.pages, ...(o.words && (type === "pdf" || type === "image") ? { words: true } : {}), stagingDir: b.stagingDir, sheetsStagingDir: b.sheetsStagingDir, pageImages: o.pageImages, pagesStagingDir: b.pagesStagingDir, imageDpi: o.imageDpi, imageFormat: o.imageFormat, maxOutputBytes: o.maxOutputBytes, pymupdfVersion: o.pymupdfVersion, ocr: o.ocr && !forced, ocrLanguage: o.ocrLanguage, ocrBudgetMs: o.primaryTimeoutMs };
 		let tier: Tier | undefined, engine: Engine | undefined, json: TierJson | undefined, degraded: string | null = null, fallbackReason: string | null = null;
 		let explicitBreaks: number | null = null;
 		let notes: string[] = [];
 		let officeRoute: string | null = null;
+		let copyReason: string | null = null;
 		if (type === "html") {
 			const prepared = await prepareHtml(inputPath, b.stagingDir);
 			if (signal?.aborted) throw new Error("aborted");
@@ -603,6 +643,7 @@ export async function convertDocument(o: DocToMdOptions, signal?: AbortSignal, s
 			}
 			if (signal?.aborted) throw new Error("aborted");
 			if (json === undefined) {
+				copyReason = reason;
 				clearStaging(b);
 				const file = `original${extname(inputPath).toLowerCase()}`;
 				const dir = join(b.stagingDir, "p1");
@@ -704,6 +745,35 @@ export async function convertDocument(o: DocToMdOptions, signal?: AbortSignal, s
 		if (type === "doc" || officeRoute !== null) degraded = DEGRADED_DOCX_OFFICE;
 		if (officeRoute !== null) fallbackReason = fallbackReason ? `${officeRoute}; ${fallbackReason}` : officeRoute;
 		if (tier === undefined || engine === undefined || json === undefined) throw new Error("internal: no tier produced output");
+		const pageStats = json.pageStats ?? null;
+		if (pageStats) writePageStats(b, pageStats);
+		let wordsPath: string | null = null, wordsReason: string | null = null;
+		const wordsErrors: Record<number, string> = {};
+		const takeWordsErrors = (j: TierJson | undefined) => {
+			for (const [k, v] of Object.entries(j?.wordsErrors ?? {})) {
+				const page = Number(k);
+				if (Number.isFinite(page)) wordsErrors[page] = v;
+			}
+		};
+		if (o.words) {
+			if (type !== "pdf" && type !== "image") wordsReason = `none - word positions apply to PDF and image inputs only (${type})`;
+			else if (tier === "unpdf") wordsReason = "none - unpdf tier has no page geometry";
+			else if (engine === "copy") wordsReason = `none - image copied without conversion (${copyReason})`;
+			else if (json?.words === true) { wordsReason = publishWords(b); if (wordsReason === null) wordsPath = b.wordsPath; }
+			else wordsReason = `write failed - ${json?.wordsErrors?.file ?? "child reported no words document"}`;
+			takeWordsErrors(json);
+		}
+		let ocr = handleOcr(tier, type, o, json);
+		if (forced) {
+			const r = await s.runTier("ocr-pages", { path: pdfPath, pages: o.pages, ...(o.words && type === "pdf" ? { words: true } : {}), stem: b.stem, ocrLanguage: o.ocrLanguage, ocrBudgetMs: o.primaryTimeoutMs, stagingDir: b.ocrStagingDir, dpi: o.imageDpi, maxOutputBytes: o.maxOutputBytes, pymupdfVersion: o.pymupdfVersion }, b, signal, o.primaryTimeoutMs, backend);
+			if (signal?.aborted || (!r.ok && "reason" in r && r.reason === "aborted")) throw new Error("aborted");
+			if (r.ok && r.json.status === "unavailable") throw new Error(`OCR unavailable: ${r.json.reason} (install Tesseract; see doc/doc-to-md.md)`);
+			const outcome = r.ok && r.json.status === "ran" ? outcomeFromChild(r.json)
+				: recoverOcrPages(b.ocrStagingDir, o.pages!, !r.ok ? ("userError" in r ? r.userError : `${r.reason}${detailSuffix(r)}`) : "malformed child output");
+			const { sidecars, wordSidecars } = publishSidecars(b);
+			ocr = { ...emptyOcr(o.ocrLanguage), ...json.ocr, status: "ran", reason: null, mode: "all", pages: outcome.written, noText: outcome.noText, ocrFailed: outcome.ocrFailed, ocrErrors: outcome.ocrErrors, budgetStopped: outcome.budgetStopped, killed: outcome.killed, notAttempted: outcome.notAttempted, childError: outcome.childError, sidecars: Object.fromEntries(sidecars), wordSidecars: Object.fromEntries(wordSidecars) };
+			if (o.words && r.ok) takeWordsErrors(r.json);
+		}
 		if (!isExcel) notes = [...notes, ...(json.notes ?? [])];
 		if (b.renamedFrom) notes.splice(notes[0]?.startsWith("preview truncated:") ? 1 : 0, 0, `renamed to ${b.stem} (${b.renameReason})`);
 		const pageImagesReason = !o.pageImages || b.pageManifest.size || tier === "primary" || tier === "fallback" ? null
@@ -719,7 +789,7 @@ export async function convertDocument(o: DocToMdOptions, signal?: AbortSignal, s
 		const markdown = (head.length ? `${head.join("\n")}\n\n` : "") + body;
 		commitBundle(b, markdown);
 		const outline = scanOutline(markdown, o.outlineMaxEntries);
-		const details: DocToMdDetails = { path: inputPath, backend: backend.kind, pymupdfVersion: o.pymupdfVersion, inputType: type, file: b.mdPath, outputDir: b.root, savedTo: b.mdPath, imagesDir: b.imagesDir, sheetsDir: b.csvManifest.size ? b.sheetsDir : null, pagesDir: b.pageManifest.size ? b.pagesDir : null, pageImageCount: b.pageManifest.size, pageImagesReason, type, engine, tier, pageCount: json.pageCount ?? null, pages: o.pages, explicitBreaks, imageCount: b.manifest.size, bytes: Buffer.byteLength(markdown, "utf8"), lines: markdown.split("\n").length, degraded, fallbackReason, failedPages: (json.failedPages ?? []).map((f) => f.page), emptyPages: json.emptyPages ?? [], notes, outline: outline.entries, outlineTotal: outline.total, ocr: handleOcr(tier, type, o, json) };
+		const details: DocToMdDetails = { path: inputPath, backend: backend.kind, pymupdfVersion: o.pymupdfVersion, inputType: type, file: b.mdPath, outputDir: b.root, savedTo: b.mdPath, imagesDir: b.imagesDir, sheetsDir: b.csvManifest.size ? b.sheetsDir : null, pagesDir: b.pageManifest.size ? b.pagesDir : null, pageImageCount: b.pageManifest.size, pageImagesReason, type, engine, tier, pageCount: json.pageCount ?? null, pages: o.pages, explicitBreaks, imageCount: b.manifest.size, bytes: Buffer.byteLength(markdown, "utf8"), lines: markdown.split("\n").length, degraded, fallbackReason, failedPages: (json.failedPages ?? []).map((f) => f.page), emptyPages: json.emptyPages ?? [], notes, outline: outline.entries, outlineTotal: outline.total, ocr, pageStats, pageStatsPath: pageStats ? b.pageStatsPath : null, ocrDir: b.ocrManifest.size ? b.ocrDir : null, wordsPath, wordsReason, wordsErrors };
 		return { output: formatHandle(details), details };
 	} catch (e) { abortBundle(b); throw e; }
 	finally { office?.cleanup(); }

package/lib/doc-to-md-handle.ts

@@ -1,4 +1,4 @@
-import type { InputType } from "./doc-to-md-options.ts";
+import type { InputType, OcrMode } from "./doc-to-md-options.ts";
 
 export type Tier = "primary" | "fallback" | "unpdf" | "excel" | "docx" | "html" | "image" | "email";
 export type Engine = "pymupdf4llm" | "pymupdf-text" | "unpdf" | "openpyxl" | "xlrd" | "mammoth" | "python-docx" | "markdownify" | "turndown" | "copy" | "extract-msg" | "email";
@@ -18,6 +18,8 @@ export interface SheetInfo {
 	csv: string | null;
 }
 
+export type PageStat = { page: number; chars: number; images: number; imageCoverage: number } | { page: number; error: string };
+
 export interface OcrInfo {
 	status: "off" | "unavailable" | "skipped" | "ran";
 	lang: string;
@@ -28,6 +30,13 @@ export interface OcrInfo {
 	budgetStopped: number[];
 	reason: string | null;
 	tesseract: boolean | null;
+	mode: OcrMode;
+	sidecars: Record<number, string>;
+	wordSidecars: Record<number, string>;
+	ocrErrors: Record<number, string>;
+	killed: number | null;
+	notAttempted: number[];
+	childError: string | null;
 }
 
 export interface HandleData {
@@ -35,6 +44,8 @@ export interface HandleData {
 	pageCount: number | null; pages: number[] | null; explicitBreaks: number | null; imageCount: number; pageImageCount: number; pageImagesReason: string | null; bytes: number; lines: number;
 	degraded: string | null; fallbackReason: string | null; failedPages: number[]; emptyPages: number[];
 	notes: string[]; outline: OutlineEntry[]; outlineTotal: number; ocr: OcrInfo | null;
+	pageStats: PageStat[] | null; pageStatsPath: string | null; ocrDir: string | null;
+	wordsPath: string | null; wordsReason: string | null; wordsErrors: Record<number, string>;
 }
 
 export interface InfoData {
@@ -72,7 +83,23 @@ export function compactRanges(nums: number[], maxEntries = 20): string {
 const INSTALL_HINT = "install Tesseract (see doc/doc-to-md.md), then rerun with ocr=true";
 const BARE_REASONS = ["fallback tier", "no Python backend"];
 
+const pageList = (pages: number[]) => `page${pages.length === 1 ? "" : "s"} ${pages.join(", ")}`;
+
+function forcedOcrLine(ocr: OcrInfo): string {
+	const clauses: string[] = [];
+	if (ocr.pages.length) clauses.push(`sidecars for ${pageList(ocr.pages)}`);
+	if (ocr.noText.length) clauses.push(`no text on ${pageList(ocr.noText)}`);
+	for (const page of ocr.ocrFailed) clauses.push(`failed on page ${page} (${ocr.ocrErrors[page] ?? "unknown error"})`);
+	if (ocr.budgetStopped.length) clauses.push(`budget-stopped ${pageList(ocr.budgetStopped)}`);
+	if (ocr.killed !== null) clauses.push(`page ${ocr.killed} killed the OCR child (timeout or crash - likely a compression bomb)`);
+	if (ocr.childError !== null) clauses.push(`OCR child failed before processing pages: ${ocr.childError}`);
+	if (ocr.notAttempted.length) clauses.push(`${pageList(ocr.notAttempted)} not attempted`);
+	const rerun = [...ocr.budgetStopped, ...ocr.notAttempted].sort((a, b) => a - b);
+	return `OCR: forced (${ocr.lang}) - ${clauses.join("; ")}${rerun.length ? ` - re-run with --pages ${rerun.join(",")}` : ""}`;
+}
+
 export function ocrLine(ocr: OcrInfo, type: InputType): string {
+	if (ocr.mode === "all") return forcedOcrLine(ocr);
 	const image = type === "image";
 	const rerun = ocr.tesseract ? "rerun with ocr=true" : INSTALL_HINT;
 	switch (ocr.status) {
@@ -143,6 +170,12 @@ export function formatHandle(h: HandleData): string {
 	if (h.sheetsDir) lines.push(`Sheets-Dir: ${h.sheetsDir}`);
 	if (h.pagesDir && h.pageImageCount > 0) lines.push(`Pages-Dir: ${h.pagesDir} (${h.pageImageCount} pages)`);
 	else if (h.pageImagesReason) lines.push(`Pages-Dir: none - ${h.pageImagesReason}`);
+	if (h.pageStatsPath) lines.push(`Page-Stats: ${h.pageStatsPath}`);
+	if (h.wordsPath) {
+		const bad = Object.keys(h.wordsErrors).map(Number).sort((a, b) => a - b);
+		lines.push(`Words: ${h.wordsPath}${bad.length ? ` (extraction failed for pages ${compactRanges(bad)}: ${h.wordsErrors[bad[0]]})` : ""}`);
+	} else if (h.wordsReason) lines.push(`Words: ${h.wordsReason}`);
+	if (h.ocrDir) lines.push(`OCR-Dir: ${h.ocrDir}`);
 	lines.push(`Type: ${h.type}   Engine: ${h.engine}   Tier: ${h.tier}`);
 	lines.push(`Page-Count: ${pageCountLabel(h)}   Pages: ${h.pages ? compactRanges(h.pages) : "all"}   Images: ${h.imageCount}   Size: ${formatSize(h.bytes)} / ${h.lines} lines`);
 	if (h.degraded) lines.push(`Degraded: ${h.degraded}`);

package/lib/doc-to-md-options.ts

@@ -6,6 +6,7 @@ import { extname } from "node:path";
 
 export type InputType = "pdf" | "docx" | "doc" | "pptx" | "xlsx" | "xlsm" | "xls" | "html" | "image" | "email";
 export type ImageFormat = "png" | "jpg";
+export type OcrMode = "textless" | "all";
 
 export interface Tunables {
 	primaryTimeoutMs: number;
@@ -29,6 +30,8 @@ export interface DocToMdOptions extends Tunables {
 	outputDir: string | null;
 	overwrite: boolean;
 	pageImages: boolean;
+	words: boolean;
+	ocrMode: OcrMode;
 }
 
 /** What adapters pass in: intents as raw strings/booleans, tunables optional. */
@@ -39,6 +42,8 @@ export interface PerCallInput extends Partial<Tunables> {
 	outputDir?: string | null;
 	overwrite?: boolean;
 	pageImages?: boolean;
+	words?: boolean;
+	ocrMode?: OcrMode;
 }
 
 export type DescriptorType = "string" | "int" | "bool" | "pages" | "enum" | "version" | "lang";
@@ -68,6 +73,8 @@ export const DOC_TO_MD_OPTIONS: readonly OptionDescriptor[] = [
 	{ key: "outputDir", flag: "--output-dir", type: "string", default: null, settable: false, help: "Bundle root for <stem>.md + images/; default a per-call temp dir. <stem> = basename without extension with [^A-Za-z0-9._-]+ -> _ (empty -> document); a second call on the same stem writes <stem>-2.md" },
 	{ key: "overwrite", flag: "--overwrite", type: "bool", default: false, settable: false, help: "Replace an existing completed <stem>.md bundle" },
 	{ key: "pageImages", flag: "--page-images", type: "bool", default: false, settable: false, help: "Also render every selected page to pages/<stem>-pNNN.<imageFormat> at imageDpi (PDF, PPTX, .doc, DOCX via LibreOffice); off by default" },
+	{ key: "words", flag: "--words", type: "bool", default: false, settable: false, help: "Write word positions: <stem>.words.json beside the Markdown lists every text-layer word of each selected page with its bbox (PDF points, top-left origin, display orientation; image inputs in source pixels) and the words inline OCR recognized, tagged source \"text\" or \"ocr\"; under --ocr-mode all the OCR words go to ocr/<stem>-pNNN.words.json beside each sidecar. Never triggers OCR. PDF and image inputs only." },
+	{ key: "ocrMode", flag: "--ocr-mode", type: "enum", default: "textless", settable: false, enumValues: ["textless", "all"], help: "OCR policy: textless (default) OCRs only pages with an empty text layer, inline; all OCRs every selected page and writes the recognized text to ocr/<stem>-pNNN.md sidecars, leaving the Markdown untouched. all requires --ocr and an explicit --pages selection (PDF, PPTX, DOC)." },
 	{ key: "primaryTimeoutMs", flag: "--primary-timeout", type: "int", default: 60000, settable: true, env: "PI_DOC_TO_MD_CONVERT_TIMEOUT_MS", help: "pymupdf4llm tier and DOCX child (docx mode); also the unpdf tier" },
 	{ key: "fallbackTimeoutMs", flag: "--fallback-timeout", type: "int", default: 30000, settable: true, help: "PyMuPDF get_text tier (including DOCX LibreOffice fallback); also PDF and DOCX info and Excel rendered views" },
 	{ key: "sofficeTimeoutMs", flag: "--soffice-timeout", type: "int", default: 120000, settable: true, env: "PI_DOC_TO_MD_SOFFICE_TIMEOUT_MS", help: "DOCX/PPTX -> PDF via LibreOffice; also Excel rendered views" },
@@ -177,12 +184,47 @@ export function resolveOptions(perCall: PerCallInput, settings: Partial<Tunables
 		out[d.key] = value;
 	}
 	const o = out as unknown as DocToMdOptions;
-	if (o.info && (o.pages !== null || o.outputDir !== null || o.overwrite || o.pageImages)) {
-		throw new UsageError("--info cannot be combined with --pages, --output-dir, --overwrite or --page-images");
+	if (o.info && (o.pages !== null || o.outputDir !== null || o.overwrite || o.pageImages || o.words || o.ocrMode === "all")) {
+		throw new UsageError("--info cannot be combined with --pages, --output-dir, --overwrite, --page-images, --words or --ocr-mode all");
 	}
 	return o;
 }
 
+/** Two-pass OCR recipe, rendered into --help and the generated skill; `<cmd>` is the command prefix. */
+export const USAGE_PATTERNS = [
+	"Two-pass OCR (PDF, PPTX, DOC):",
+	"  1. <cmd> report.pdf --output-dir out --json",
+	'       -> "pageStatsPath" points at out/report.pages.json; pages with few',
+	'          chars and high imageCoverage are scans. "savedTo" is the Markdown.',
+	"  2. <cmd> report.pdf --output-dir out --ocr --ocr-mode all --pages 2,7 --json",
+	'       -> "ocr"."sidecars" maps 2 and 7 to out/ocr/report-2-p002.md and',
+	"          ...-p007.md (a second run in the same dir gets stem report-2);",
+	"          the Markdown of this run holds pages 2 and 7 only and equals what",
+	"          --pages 2,7 without --ocr would produce. Read sidecars and Markdown",
+	"          by the returned paths, never by guessing names.",
+	"  3. --ocr-mode all refuses to run without --ocr and an explicit --pages.",
+	'     The "ocr" object (OCR: line) names failed, budget-stopped, killed and',
+	"     not-attempted pages and the exact --pages to re-run.",
+	"  Details: doc/doc-to-md.md (bundle contract, failure buckets).",
+].join("\n");
+
+export const usagePatterns = (cmd: string): string => USAGE_PATTERNS.replaceAll("<cmd>", cmd);
+
+/** Every artifact a bundle can contain; shared by --help and the generated skill. */
+export const BUNDLE_LAYOUT: readonly { artifact: string; trigger: string; content: string; namedBy: string }[] = [
+	{ artifact: "<stem>.md", trigger: "always", content: "the Markdown", namedBy: "Saved-To: / savedTo" },
+	{ artifact: "images/", trigger: "embedded or extracted figures", content: "image files linked from the Markdown", namedBy: "Images-Dir: / imagesDir" },
+	{ artifact: "pages/<stem>-pNNN.<fmt>", trigger: "--page-images", content: "page renders at --image-dpi", namedBy: "Pages-Dir: / pagesDir" },
+	{ artifact: "sheets/", trigger: "Excel input", content: "one CSV per non-empty worksheet", namedBy: "Sheets-Dir: / sheetsDir" },
+	{ artifact: "attachments/", trigger: "email input", content: "saved attachments", namedBy: "Markdown attachment list" },
+	{ artifact: "<stem>.pages.json", trigger: "Python PDF tiers (PDF, PPTX, DOC, DOCX via LibreOffice; not unpdf)", content: "per-page chars, image count, image coverage", namedBy: "Page-Stats: / pageStatsPath" },
+	{ artifact: "<stem>.words.json", trigger: "--words", content: "per-page word boxes, source text/ocr", namedBy: "Words: / wordsPath" },
+	{ artifact: "ocr/<stem>-pNNN.md", trigger: "--ocr --ocr-mode all", content: "recognized text of a forced page", namedBy: "OCR-Dir: / ocr.sidecars" },
+	{ artifact: "ocr/<stem>-pNNN.words.json", trigger: "--ocr --ocr-mode all --words", content: "word boxes of that OCR", namedBy: "ocr.wordSidecars" },
+];
+
+const bundleLayoutText = (): string => BUNDLE_LAYOUT.map((r) => `  ${r.artifact.padEnd(28)} ${r.trigger}; ${r.content}; named by ${r.namedBy}`).join("\n");
+
 export function renderHelp(): string {
 	const row = (d: OptionDescriptor) => `  ${(d.flag ?? "<path>").padEnd(26)} ${d.help}${d.default !== null && d.key !== "info" && d.key !== "overwrite" ? ` (default ${d.default})` : ""}`;
 	return [
@@ -190,7 +232,8 @@ export function renderHelp(): string {
 		"", "Per-call:", ...DOC_TO_MD_OPTIONS.filter((d) => !d.settable).map(row),
 		"", "Tunables (also settable under quiver.docToMd in settings.json; per-call > settings > default):",
 		...DOC_TO_MD_OPTIONS.filter((d) => d.settable).map(row),
-		"", "Result: a handle (Saved-To, Images-Dir, Page-Count, Outline ...). Read the Saved-To file for the Markdown.",
+		"", "Result: a handle (Saved-To, Images-Dir, Page-Stats, Page-Count, Outline ...). Read the Saved-To file for the Markdown.",
 		"Exit codes: 0 success, 1 runtime error, 2 usage error.",
+		"", "Bundle layout:", bundleLayoutText(), "", usagePatterns("pi-quiver doc-to-md"),
 	].join("\n");
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-quiver",
-  "version": "6.8.0",
+  "version": "6.10.0",
   "description": "Personal pack of Pi coding-agent extensions: context-safe fetch, doc_to_md PDF/DOCX/PPTX-to-Markdown conversion, session naming, a themed ASCII startup header, Opus 4.8 fast mode, and a provider-stall watchdog.",
   "author": "Jacek Juraszek",
   "license": "MIT",

package/scripts/doc_to_md.py

@@ -205,11 +205,22 @@ def mode_image(o):
     info = new_ocr(lang)
     status = ocr_status(bool(o.get("ocr")), lang)
     apply_status(info, status)
-    if status["status"] == "ready":
+    words_on = bool(o.get("words"))
+    word_pages, words_errors = [], {}
+    w_px = h_px = None
+    if words_on or status["status"] == "ready":
         try:
             pix = pymupdf.Pixmap(path)
             w_px, h_px = pix.width, pix.height
             del pix
+        except Exception as exc:  # noqa: BLE001
+            if words_on:
+                words_errors["1"] = words_error(exc)
+            if status["status"] == "ready":
+                info["ocrFailed"].append(1)
+                status = {**status, "status": "failed"}
+    if status["status"] == "ready":
+        try:
             if min(w_px, h_px) < MIN_OCR_SIDE_PX:
                 info["status"], info["reason"] = "skipped", "image too small"
             else:
@@ -218,6 +229,19 @@ def mode_image(o):
                     text = pymupdf4llm.to_markdown(pdf, pages=[0], write_images=False, use_ocr=True, force_ocr=True,
                                                    ocr_language=lang, ocr_dpi=image_ocr_dpi(w_px, r.width, r.height),
                                                    page_separators=False).strip()
+                    if words_on:
+                        try:
+                            pg = pdf[0]
+                            entry = words_page(pg, 1, 0, page_words(pg, True))
+                            sx, sy = w_px / r.width, h_px / r.height
+                            for word in entry["words"]:
+                                b = word["bbox"]
+                                word["bbox"] = [round(b[0] * sx, 1), round(b[1] * sy, 1),
+                                                round(b[2] * sx, 1), round(b[3] * sy, 1)]
+                            entry["width"], entry["height"] = w_px, h_px
+                            word_pages.append(entry)
+                        except Exception as exc:  # noqa: BLE001
+                            words_errors["1"] = words_error(exc)
                 if text:
                     info["pages"].append(1)
                     md += "\n\n" + ocr_block(f"p1/{name}", text)
@@ -225,7 +249,17 @@ def mode_image(o):
                     info["noText"].append(1)
         except Exception:  # noqa: BLE001 - OCR never fails the conversion
             info["ocrFailed"].append(1)
-    return {"markdown": md + "\n", "pageCount": 1, "emptyPages": [], "failedPages": [], "notes": [], "ocr": info}
+    if words_on and not o.get("ocr") and w_px is not None and "1" not in words_errors:
+        try:
+            if os.environ.get("DOC_TO_MD_WORDS_FAIL") == "1":  # tests only
+                raise RuntimeError("words injected failure")
+            word_pages.append({"page": 1, "width": w_px, "height": h_px, "rotation": 0, "words": []})
+        except Exception as exc:  # noqa: BLE001
+            words_errors["1"] = words_error(exc)
+    result = {"markdown": md + "\n", "pageCount": 1, "emptyPages": [], "failedPages": [], "notes": [], "ocr": info}
+    if words_on:
+        result.update(write_words(o["stagingDir"], "px", word_pages, words_errors))
+    return result
 
 
 def mode_info(o):
@@ -253,6 +287,71 @@ def primary_page_markdown(doc, n, d, o, kw, write_images):
         return rewrite_image_destinations(md, sources)
 
 
+def page_stats(doc, n):
+    try:
+        page = doc[n - 1]
+        chars = len(page.get_text("text").strip())
+        infos = page.get_image_info()
+        area = page.rect.width * page.rect.height
+        covered = sum(max(0.0, (b[2] - b[0]) * (b[3] - b[1])) for b in (i["bbox"] for i in infos))
+        coverage = round(min(1.0, covered / area), 2) if area > 0 else 0.0
+        return {"page": n, "chars": chars, "images": len(infos), "imageCoverage": coverage}
+    except Exception as exc:  # noqa: BLE001 - stats never cost a page its Markdown
+        return {"page": n, "error": f"{type(exc).__name__}: {exc}"[:300]}
+
+
+GLYPHLESS_FONT = "GlyphLessFont"
+
+
+def word_entries(page, words, ocr_lines):
+    import pymupdf
+    matrix = page.rotation_matrix
+    out = []
+    for x0, y0, x1, y1, text, bno, lno, _ in words:
+        r = pymupdf.Rect(x0, y0, x1, y1) * matrix
+        out.append({"text": text, "bbox": [round(r.x0, 1), round(r.y0, 1), round(r.x1, 1), round(r.y1, 1)],
+                    "source": "ocr" if ocr_lines is None or (bno, lno) in ocr_lines else "text"})
+    return out
+
+
+def page_words(page, inline_ocr_ran):
+    if not inline_ocr_ran:
+        return word_entries(page, page.get_text("words"), set())
+    # Shared textpage keeps block/line indexes aligned despite differing default image flags.
+    tp = page.get_textpage()
+    ocr_lines = {(bi, li) for bi, b in enumerate(page.get_text("dict", textpage=tp)["blocks"])
+                 for li, line in enumerate(b.get("lines", []))
+                 if line["spans"] and all(s["font"] == GLYPHLESS_FONT for s in line["spans"])}
+    return word_entries(page, page.get_text("words", textpage=tp), ocr_lines)
+
+
+def ocr_words(page, tp):
+    return word_entries(page, page.get_text("words", textpage=tp), None)
+
+
+def words_page(page, n, rotation, words):
+    if os.environ.get("DOC_TO_MD_WORDS_FAIL") == str(n):  # tests only
+        raise RuntimeError("words injected failure")
+    return {"page": n, "width": round(page.rect.width, 1), "height": round(page.rect.height, 1),
+            "rotation": rotation, "words": words}
+
+
+def words_error(exc):
+    return f"{type(exc).__name__}: {exc}"[:300]
+
+
+def write_words(staging, unit, pages, errors):
+    try:
+        os.makedirs(staging, exist_ok=True)
+        with open(os.path.join(staging, "words.json"), "w", encoding="utf-8") as fh:
+            json.dump({"unit": unit, "pages": pages}, fh, indent=2)
+            fh.write("\n")
+        return {"words": True, "wordsErrors": errors}
+    except Exception as exc:  # noqa: BLE001 - geometry never fails the conversion
+        errors["file"] = words_error(exc)
+        return {"words": False, "wordsErrors": errors}
+
+
 def mode_pdf_primary(o):
     import time
     start = time.monotonic()
@@ -260,14 +359,18 @@ def mode_pdf_primary(o):
     pages = check_pages(o.get("pages"), doc.page_count)
     staging, out, empty, failed, notes = o["stagingDir"], [], [], [], []
     page_images = []
+    stats = []
     lang, budget = o.get("ocrLanguage", "eng"), o.get("ocrBudgetMs", 60000)
     info = new_ocr(lang)
     status = ocr_status(True, lang) if o.get("ocr") else None
     ocr_ms, plain_ms = [], []
+    word_pages, words_errors = [], {}
     for i, n in enumerate(pages):
+        stats.append(page_stats(doc, n))
         d = page_dir(staging, n)
         try:
             page = doc[n - 1]
+            rotation = page.rotation
             textless = not page.get_text("text").strip()
             if textless:
                 info["textless"].append(n)
@@ -311,6 +414,11 @@ def mode_pdf_primary(o):
             if page_pic:
                 md = "\n\n".join(x for x in [md.rstrip(), f"![page {n}](pages/{page_pic})"] if x)
             out.append(md.rstrip())
+            if o.get("words") and not (textless and o.get("ocr") and not kw["use_ocr"]):
+                try:
+                    word_pages.append(words_page(page, n, rotation, page_words(page, kw["use_ocr"])))
+                except Exception as exc:  # noqa: BLE001
+                    words_errors[str(n)] = words_error(exc)
         except Exception as exc:  # noqa: BLE001
             shutil.rmtree(d, ignore_errors=True)
             failed.append({"page": n, "error": f"{type(exc).__name__}: {exc}"[:300]})
@@ -324,8 +432,11 @@ def mode_pdf_primary(o):
     missing = len(pages) - len(page_images) if o.get("pageImages") else 0
     if missing:
         notes.append(f"Page images: {missing} of {len(pages)} unavailable")
-    return {"markdown": "\n\n".join(out) + "\n", "pages": pages, "pageCount": doc.page_count,
-            "emptyPages": empty, "failedPages": failed, "notes": notes, "ocr": info, "pageImages": page_images}
+    result = {"markdown": "\n\n".join(out) + "\n", "pages": pages, "pageCount": doc.page_count,
+              "emptyPages": empty, "failedPages": failed, "notes": notes, "ocr": info, "pageImages": page_images, "pageStats": stats}
+    if o.get("words"):
+        result.update(write_words(staging, "pt", word_pages, words_errors))
+    return result
 
 
 def mode_pdf_fallback(o):
@@ -335,15 +446,20 @@ def mode_pdf_fallback(o):
     keep = {int(k): v for k, v in (o.get("keepPages") or {}).items()}
     staging, out, empty, failed = o["stagingDir"], [], [], []
     page_images = []
+    stats = []
     lang = o.get("ocrLanguage", "eng")
     ocr_info = new_ocr(lang)
+    word_pages, words_errors = [], {}
     status = {"status": "unavailable", "reason": "fallback tier", "tesseract": None} if o.get("ocr") else None
     for n in pages:
+        stats.append(page_stats(doc, n))
         links = [f"![](images/{f})" for f in keep.get(n, [])]
         text = ""
         page_pic = None
+        page_ok = False
         try:
             page = doc[n - 1]
+            rotation = page.rotation
             text = page.get_text("text").strip()
             if not text:
                 ocr_info["textless"].append(n)
@@ -371,11 +487,17 @@ def mode_pdf_fallback(o):
                         links.append(f"![](p{n}/{name})")
                 mark_done(d)
             page_pic = render_page_image(page, n, o, page_images) if o.get("pageImages") else None
+            page_ok = True
         except Exception as exc:  # noqa: BLE001
             shutil.rmtree(os.path.join(staging, f"p{n}"), ignore_errors=True)
             text = ""
             links = [f"![](images/{f})" for f in keep.get(n, [])]
             failed.append({"page": n, "error": f"{type(exc).__name__}: {exc}"[:300]})
+        if o.get("words") and page_ok and not (not text and o.get("ocr")):
+            try:
+                word_pages.append(words_page(page, n, rotation, page_words(page, False)))
+            except Exception as exc:  # noqa: BLE001
+                words_errors[str(n)] = words_error(exc)
         if not text:
             empty.append(n)
         out.append("\n\n".join(x for x in [text, "\n".join(links), f"![page {n}](pages/{page_pic})" if page_pic else ""] if x))
@@ -388,8 +510,88 @@ def mode_pdf_fallback(o):
     missing = len(pages) - len(page_images) if o.get("pageImages") else 0
     if missing:
         notes.append(f"Page images: {missing} of {len(pages)} unavailable")
-    return {"markdown": "\n\n".join(out) + "\n", "pages": pages, "pageCount": doc.page_count,
-            "emptyPages": empty, "failedPages": failed, "notes": notes, "ocr": ocr_info, "pageImages": page_images}
+    result = {"markdown": "\n\n".join(out) + "\n", "pages": pages, "pageCount": doc.page_count,
+              "emptyPages": empty, "failedPages": failed, "notes": notes, "ocr": ocr_info, "pageImages": page_images, "pageStats": stats}
+    if o.get("words"):
+        result.update(write_words(staging, "pt", word_pages, words_errors))
+    return result
+
+
+def mode_ocr_pages(o):
+    import time
+    start = time.monotonic()
+    lang, budget = o.get("ocrLanguage", "eng"), o.get("ocrBudgetMs", 60000)
+    status = ocr_status(True, lang)
+    if status["status"] != "ready":
+        return {"status": "unavailable", "reason": status["reason"]}
+    doc = open_pdf(o["path"])
+    pages = check_pages(o.get("pages"), doc.page_count)
+    staging, stem, dpi = o["stagingDir"], o["stem"], o.get("dpi", 150)
+    os.makedirs(staging, exist_ok=True)
+    active = os.path.join(staging, "active")
+    out = {"status": "ran", "written": [], "noText": [], "ocrFailed": [], "ocrErrors": {}, "budgetStopped": []}
+    if o.get("words"):
+        out["wordsErrors"] = {}
+    ocr_ms = []
+    for i, n in enumerate(pages):
+        with open(active, "w") as fh:
+            fh.write(str(n))
+        if os.environ.get("DOC_TO_MD_OCR_STALL_PAGE") == str(n):  # tests only: simulate a wedged page
+            time.sleep(3600)
+        elapsed = (time.monotonic() - start) * 1000
+        est = max(ocr_ms) if ocr_ms else OCR_EST_INITIAL_MS
+        if not ocr_admit(elapsed, est, 0, 0, budget):
+            out["budgetStopped"].extend(pages[i:])
+            os.remove(active)
+            break
+        tag = f"p{n:03d}"
+        d = os.path.join(staging, tag)
+        os.makedirs(d, exist_ok=True)
+        sidecar = os.path.join(d, f"{stem}-{tag}.md")
+        t0 = time.monotonic()
+        try:
+            page = doc[n - 1]
+            eff = clamped_dpi(page.rect.width, page.rect.height, dpi)
+            if eff is None:
+                raise RuntimeError(f"page cannot be rendered at a usable DPI ({page.rect.width:.0f} x {page.rect.height:.0f} pt)")
+            tp = page.get_textpage_ocr(full=True, language=lang, dpi=eff)
+            text = page.get_text("text", textpage=tp).strip()
+            header = f"<!-- OCR of page {n} (tesseract {lang}); recognized text, not the text layer -->"
+            with open(sidecar, "w", encoding="utf-8") as fh:
+                fh.write(header + "\n\n" + (text + "\n\n" if text else "") + SEP.format(n=n).strip("\n") + "\n")
+            if o.get("words"):
+                wpath = os.path.join(d, f"{stem}-{tag}.words.json")
+                try:
+                    entry = words_page(page, n, page.rotation, ocr_words(page, tp))
+                    entry["unit"] = "pt"
+                    with open(wpath, "w", encoding="utf-8") as fh:
+                        json.dump(entry, fh, indent=2)
+                        fh.write("\n")
+                except Exception as exc:  # noqa: BLE001 - geometry never changes the OCR outcome
+                    try:
+                        os.remove(wpath)
+                    except OSError:
+                        pass
+                    out["wordsErrors"][str(n)] = words_error(exc)
+            mark_done(d)
+            (out["written"] if text else out["noText"]).append(n)
+        except Exception as exc:  # noqa: BLE001 - one page never stops the pass
+            msg = f"{type(exc).__name__}: {exc}"
+            with open(os.path.join(d, ".failed"), "w", encoding="utf-8") as fh:
+                fh.write(msg)
+            try:
+                os.remove(sidecar)
+            except OSError:
+                pass
+            try:
+                os.remove(os.path.join(d, f"{stem}-{tag}.words.json"))
+            except OSError:
+                pass
+            out["ocrFailed"].append(n)
+            out["ocrErrors"][str(n)] = msg
+        ocr_ms.append((time.monotonic() - t0) * 1000)
+        os.remove(active)
+    return out
 
 
 def esc(v):
@@ -1249,12 +1451,12 @@ def mode_render_pages(o):
     return {"ok": True, "rendered": rendered, "failed": failed}
 
 
-MODES = {"info": None, "pdf-primary": mode_pdf_primary, "pdf-fallback": mode_pdf_fallback, "xlsx": mode_xlsx, "render-pages": mode_render_pages, "docx": mode_docx, "html": mode_html, "image": mode_image, "email": mode_email}
+MODES = {"info": None, "pdf-primary": mode_pdf_primary, "pdf-fallback": mode_pdf_fallback, "xlsx": mode_xlsx, "render-pages": mode_render_pages, "docx": mode_docx, "html": mode_html, "image": mode_image, "email": mode_email, "ocr-pages": mode_ocr_pages}
 
 
 def main():
     if len(sys.argv) != 2 or sys.argv[1] not in MODES:
-        print("usage: doc_to_md.py <info|pdf-primary|pdf-fallback|xlsx|render-pages|docx|html|image|email>  (options JSON on stdin)", file=sys.stderr)
+        print("usage: doc_to_md.py <info|pdf-primary|pdf-fallback|xlsx|render-pages|docx|html|image|email|ocr-pages>  (options JSON on stdin)", file=sys.stderr)
         return 1
     mode = sys.argv[1]
     o = json.loads(sys.stdin.read() or "{}")