pi-quiver 6.9.0 → 6.10.0

package/CHANGELOG.md

@@ -8,6 +8,11 @@ Published to npm as `pi-quiver` (`pi install npm:pi-quiver`). Pushing a
 via OIDC trusted publishing. The release helper at
 `.agents/skills/release/scripts/release.sh` cuts the tag; CI publishes.
 
+## v6.10.0 - 2026-10-02
+
+- `doc_to_md`: new per-call `words` option (`--words`; not settable) writes `<stem>.words.json` - per selected PDF/image page, every text-layer word and every word inline OCR recognized in the same run, with display-space bbox (points; source pixels for images) and `source` `text`/`ocr`. Under `--ocr-mode all` each sidecar gets `ocr/<stem>-pNNN.words.json`. Never triggers OCR; Markdown, page stats and OCR outcome are unchanged. Handle gains `Words:`, `--json` gains `wordsPath`, `wordsReason`, `wordsErrors`, `ocr.wordSidecars`; `--info --words` is a usage error (#28).
+- `doc_to_md`: `--help` and the generated skill list every bundle artifact under `Bundle layout` (#28).
+
 ## v6.9.0 - 2026-10-01
 
 - `doc_to_md`: every conversion that goes through the Python PDF tiers (PDF, PPTX, `.doc`, DOCX via LibreOffice) writes `<stem>.pages.json` (per-page `chars`, `images`, `imageCoverage`) and the handle prints `Page-Stats:`; CLI `--json` gains `pageStats`, `pageStatsPath`, `ocrDir`. The unpdf tier writes no stats (#26).

package/README.md

@@ -65,7 +65,7 @@ A 300 KB changelog page never touches your context window - you get a preview an
 | Extension | Tool | What it does |
 | --- | --- | --- |
 | `extensions/fetch.ts` | `fetch` | Retrieve URLs over HTTP(S). HTML -> Markdown (Readability extraction, Turndown conversion). Binary saved untouched to a temp file. GitHub issue/PR/repo/actions-run/actions-job URLs auto-route through `gh` (falls back to HTTP); failed runs/jobs include failed-step logs (best-effort, summary-only otherwise). Same size gate as `fetch`. Behavior lives in `lib/fetch-core.ts`; also exposed as the `pi-quiver fetch` CLI (see [Claude Code support](#claude-code-support)). |
-| `extensions/doc_to_md.ts` | `doc_to_md` | Convert a local PDF/DOCX/DOC/PPTX/XLSX/XLSM/XLS, MSG/EML, HTML, or image file to a Markdown bundle on disk (`<stem>.md` + `images/`, spreadsheet `sheets/`, optional `pageImages` in `pages/`, and email `attachments/`) and return a handle (paths, page count, outline, diagnostics) - never inline Markdown. `info` inspects PDF/Office inputs; `pages` selects 1-based PDF/Office pages (DOCX: explicit-page-break segments); HTML, image, and email inputs reject both; every selected PDF/PPTX page or DOCX segment when `pageCount > 1` ends with `--- end of page.page_number=N ---`. Tiers: pymupdf4llm -> PyMuPDF text (degraded) -> unpdf worker (no Python only). Excel -> sheet inventory, full CSVs, bounded previews, and optional rendered views. HTML keeps local and `data:` images in the bundle and remote images as links; image inputs keep the original image. Scanned PDF pages keep a page picture on both Python tiers; opt-in OCR needs Tesseract language data. DOCX converts directly (mammoth, python-docx fallback); LibreOffice pagination marks the degraded route. The Outline lists `L<line>` and `p<page>` per heading. Settings under `quiver.docToMd`. Every conversion that goes through the Python PDF tiers (PDF, PPTX, `.doc`, DOCX via LibreOffice) writes `<stem>.pages.json` (per-page chars, image count, image coverage; handle line `Page-Stats:`); unpdf writes no stats. `ocrMode: "all"` with `ocr: true` and an explicit `pages` forces OCR on those pages into `ocr/<stem>-pNNN.md` sidecars, leaving the Markdown byte-identical to the same selected-page call without OCR. |
+| `extensions/doc_to_md.ts` | `doc_to_md` | Convert a local PDF/DOCX/DOC/PPTX/XLSX/XLSM/XLS, MSG/EML, HTML, or image file to a Markdown bundle on disk (`<stem>.md` + `images/`, spreadsheet `sheets/`, optional `pageImages` in `pages/`, and email `attachments/`) and return a handle (paths, page count, outline, diagnostics) - never inline Markdown. `info` inspects PDF/Office inputs; `pages` selects 1-based PDF/Office pages (DOCX: explicit-page-break segments); HTML, image, and email inputs reject both; every selected PDF/PPTX page or DOCX segment when `pageCount > 1` ends with `--- end of page.page_number=N ---`. Tiers: pymupdf4llm -> PyMuPDF text (degraded) -> unpdf worker (no Python only). Excel -> sheet inventory, full CSVs, bounded previews, and optional rendered views. HTML keeps local and `data:` images in the bundle and remote images as links; image inputs keep the original image. Scanned PDF pages keep a page picture on both Python tiers; opt-in OCR needs Tesseract language data. DOCX converts directly (mammoth, python-docx fallback); LibreOffice pagination marks the degraded route. The Outline lists `L<line>` and `p<page>` per heading. Settings under `quiver.docToMd`. Every conversion that goes through the Python PDF tiers (PDF, PPTX, `.doc`, DOCX via LibreOffice) writes `<stem>.pages.json` (per-page chars, image count, image coverage; handle line `Page-Stats:`); unpdf writes no stats. `ocrMode: "all"` with `ocr: true` and an explicit `pages` forces OCR on those pages into `ocr/<stem>-pNNN.md` sidecars, leaving the Markdown byte-identical to the same selected-page call without OCR. `words: true` (CLI `--words`, per-call only - no settings key) writes `<stem>.words.json` with every word's bbox per selected PDF/image page (`source` `text` or `ocr`), plus `ocr/<stem>-pNNN.words.json` beside each forced-OCR sidecar; it never triggers OCR. |
 | `extensions/session-name.ts` | `/session-name` | Manual + opt-in automatic session naming, naming rules and deny list, long-session revisits, and Ghostty/Herdr tab rename. OFF by default. |
 | `extensions/sword-header.ts` | `/builtin-header` | Themed ASCII startup header replacing pi's default logo. OFF by default. |
 | `extensions/fast-mode.ts` | `/fast` | Inject Anthropic fast-mode (`speed: "fast"` + `anthropic-beta: fast-mode-2026-02-01`) into every Claude Opus 4.8 / Opus 5 request, any thinking level. `--fast` flag + `/fast [on\|off\|status]`. OFF by default. |

package/dist/bin/pi-quiver.js

@@ -582,7 +582,7 @@ function ownedPagePattern(stem) {
   return new RegExp(`^${escRe(stem)}-p\\d+\\.[a-z0-9]+$`);
 }
 function ownedOcrPattern(stem) {
-  return new RegExp(`^${escRe(stem)}-p\\d+\\.md$`);
+  return new RegExp(`^${escRe(stem)}-p\\d+(?:\\.words\\.json|\\.md)$`);
 }
 var FILE_LINK_RE = /\[[^\]]*\]\(\s*((?:sheets|attachments)\/[^)\s]+)\s*\)/g;
 var IMG_LINK_RE = /!\[[^\]]*\]\(\s*(?:<([^>]*)>|([^)]*?))\s*\)|<img\b[^>]*\bsrc\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s>"']+))/gi;
@@ -649,6 +649,7 @@ function openBundle(root, requested, overwrite) {
         for (const f of readdirSync(attachmentsDir)) if (attachments.has(f)) rmSync(join2(attachmentsDir, f), { force: true });
       }
       rmSync(join2(root, `${stem}.pages.json`), { force: true });
+      rmSync(join2(root, `${stem}.words.json`), { force: true });
       const ownedOcr = ownedOcrPattern(stem);
       if (existsSync(ocrDir)) {
         for (const f of readdirSync(ocrDir)) if (ownedOcr.test(f)) rmSync(join2(ocrDir, f), { force: true });
@@ -661,7 +662,7 @@ function openBundle(root, requested, overwrite) {
     const pagesStagingDir = join2(pagesDir, `.stage-${lockId}`);
     const attachmentsStagingDir = join2(attachmentsDir, `.stage-${lockId}`);
     const ocrStagingDir = join2(ocrDir, `.stage-${lockId}`);
-    return { root, stem, renamedFrom, renameReason, mdPath, lockPath, imagesDir, stagingDir, lockId, sheetsDir, sheetsStagingDir, pagesDir, pagesStagingDir, attachmentsDir, attachmentsStagingDir, ocrDir, ocrStagingDir, pageStatsPath: join2(root, `${stem}.pages.json`), ocrManifest: /* @__PURE__ */ new Set(), manifest: /* @__PURE__ */ new Set(), csvManifest: /* @__PURE__ */ new Set(), pageManifest: /* @__PURE__ */ new Set(), attachmentManifest: /* @__PURE__ */ new Set(), sourceMap: /* @__PURE__ */ new Map() };
+    return { root, stem, renamedFrom, renameReason, mdPath, lockPath, imagesDir, stagingDir, lockId, sheetsDir, sheetsStagingDir, pagesDir, pagesStagingDir, attachmentsDir, attachmentsStagingDir, ocrDir, ocrStagingDir, pageStatsPath: join2(root, `${stem}.pages.json`), wordsPath: join2(root, `${stem}.words.json`), ocrManifest: /* @__PURE__ */ new Set(), manifest: /* @__PURE__ */ new Set(), csvManifest: /* @__PURE__ */ new Set(), pageManifest: /* @__PURE__ */ new Set(), attachmentManifest: /* @__PURE__ */ new Set(), sourceMap: /* @__PURE__ */ new Map() };
   } catch (e) {
     rmSync(lockPath, { force: true });
     throw e;
@@ -729,9 +730,20 @@ function writePageStats(b, stats) {
   writeFileSync2(b.pageStatsPath, `${JSON.stringify(stats, null, 2)}
 `, "utf8");
 }
+function publishWords(b) {
+  const staged = join2(b.stagingDir, "words.json");
+  try {
+    if (!existsSync(staged)) throw new Error("child staged no words.json");
+    renameSync(staged, b.wordsPath);
+    return null;
+  } catch (e) {
+    rmSync(b.wordsPath, { force: true });
+    return `write failed - ${e.message}`;
+  }
+}
 function publishSidecars(b) {
-  const out = /* @__PURE__ */ new Map();
-  if (!existsSync(b.ocrStagingDir)) return out;
+  const sidecars = /* @__PURE__ */ new Map(), wordSidecars = /* @__PURE__ */ new Map();
+  if (!existsSync(b.ocrStagingDir)) return { sidecars, wordSidecars };
   for (const dir of readdirSync(b.ocrStagingDir).sort()) {
     const m = dir.match(/^p(\d+)$/);
     if (!m) continue;
@@ -742,10 +754,16 @@ function publishSidecars(b) {
     mkdirSync2(b.ocrDir, { recursive: true });
     renameSync(join2(pageDir, file), join2(b.ocrDir, file));
     b.ocrManifest.add(file);
-    out.set(Number(m[1]), join2(b.ocrDir, file));
+    sidecars.set(Number(m[1]), join2(b.ocrDir, file));
+    const wfile = `${b.stem}-${dir}.words.json`;
+    if (existsSync(join2(pageDir, wfile))) {
+      renameSync(join2(pageDir, wfile), join2(b.ocrDir, wfile));
+      b.ocrManifest.add(wfile);
+      wordSidecars.set(Number(m[1]), join2(b.ocrDir, wfile));
+    }
   }
   rmSync(b.ocrStagingDir, { recursive: true, force: true });
-  return out;
+  return { sidecars, wordSidecars };
 }
 function publishAttachments(b) {
   if (!existsSync(b.attachmentsStagingDir)) return;
@@ -821,6 +839,7 @@ function abortBundle(b) {
   for (const f of b.ocrManifest) rmSync(join2(b.ocrDir, f), { force: true });
   rmSync(b.ocrStagingDir, { recursive: true, force: true });
   rmSync(b.pageStatsPath, { force: true });
+  rmSync(b.wordsPath, { force: true });
   rmSync(`${b.mdPath}.tmp`, { force: true });
   rmSync(b.stagingDir, { recursive: true, force: true });
   rmSync(b.sheetsStagingDir, { recursive: true, force: true });
@@ -950,6 +969,10 @@ function formatHandle(h) {
   if (h.pagesDir && h.pageImageCount > 0) lines.push(`Pages-Dir: ${h.pagesDir} (${h.pageImageCount} pages)`);
   else if (h.pageImagesReason) lines.push(`Pages-Dir: none - ${h.pageImagesReason}`);
   if (h.pageStatsPath) lines.push(`Page-Stats: ${h.pageStatsPath}`);
+  if (h.wordsPath) {
+    const bad = Object.keys(h.wordsErrors).map(Number).sort((a, b) => a - b);
+    lines.push(`Words: ${h.wordsPath}${bad.length ? ` (extraction failed for pages ${compactRanges(bad)}: ${h.wordsErrors[bad[0]]})` : ""}`);
+  } else if (h.wordsReason) lines.push(`Words: ${h.wordsReason}`);
   if (h.ocrDir) lines.push(`OCR-Dir: ${h.ocrDir}`);
   lines.push(`Type: ${h.type}   Engine: ${h.engine}   Tier: ${h.tier}`);
   lines.push(`Page-Count: ${pageCountLabel(h)}   Pages: ${h.pages ? compactRanges(h.pages) : "all"}   Images: ${h.imageCount}   Size: ${formatSize2(h.bytes)} / ${h.lines} lines`);
@@ -1000,6 +1023,7 @@ var DOC_TO_MD_OPTIONS = [
   { key: "outputDir", flag: "--output-dir", type: "string", default: null, settable: false, help: "Bundle root for <stem>.md + images/; default a per-call temp dir. <stem> = basename without extension with [^A-Za-z0-9._-]+ -> _ (empty -> document); a second call on the same stem writes <stem>-2.md" },
   { key: "overwrite", flag: "--overwrite", type: "bool", default: false, settable: false, help: "Replace an existing completed <stem>.md bundle" },
   { key: "pageImages", flag: "--page-images", type: "bool", default: false, settable: false, help: "Also render every selected page to pages/<stem>-pNNN.<imageFormat> at imageDpi (PDF, PPTX, .doc, DOCX via LibreOffice); off by default" },
+  { key: "words", flag: "--words", type: "bool", default: false, settable: false, help: 'Write word positions: <stem>.words.json beside the Markdown lists every text-layer word of each selected page with its bbox (PDF points, top-left origin, display orientation; image inputs in source pixels) and the words inline OCR recognized, tagged source "text" or "ocr"; under --ocr-mode all the OCR words go to ocr/<stem>-pNNN.words.json beside each sidecar. Never triggers OCR. PDF and image inputs only.' },
   { key: "ocrMode", flag: "--ocr-mode", type: "enum", default: "textless", settable: false, enumValues: ["textless", "all"], help: "OCR policy: textless (default) OCRs only pages with an empty text layer, inline; all OCRs every selected page and writes the recognized text to ocr/<stem>-pNNN.md sidecars, leaving the Markdown untouched. all requires --ocr and an explicit --pages selection (PDF, PPTX, DOC)." },
   { key: "primaryTimeoutMs", flag: "--primary-timeout", type: "int", default: 6e4, settable: true, env: "PI_DOC_TO_MD_CONVERT_TIMEOUT_MS", help: "pymupdf4llm tier and DOCX child (docx mode); also the unpdf tier" },
   { key: "fallbackTimeoutMs", flag: "--fallback-timeout", type: "int", default: 3e4, settable: true, help: "PyMuPDF get_text tier (including DOCX LibreOffice fallback); also PDF and DOCX info and Excel rendered views" },
@@ -1107,8 +1131,8 @@ function resolveOptions(perCall, settings, env) {
     out[d.key] = value;
   }
   const o = out;
-  if (o.info && (o.pages !== null || o.outputDir !== null || o.overwrite || o.pageImages || o.ocrMode === "all")) {
-    throw new UsageError("--info cannot be combined with --pages, --output-dir, --overwrite, --page-images or --ocr-mode all");
+  if (o.info && (o.pages !== null || o.outputDir !== null || o.overwrite || o.pageImages || o.words || o.ocrMode === "all")) {
+    throw new UsageError("--info cannot be combined with --pages, --output-dir, --overwrite, --page-images, --words or --ocr-mode all");
   }
   return o;
 }
@@ -1129,6 +1153,18 @@ var USAGE_PATTERNS = [
   "  Details: doc/doc-to-md.md (bundle contract, failure buckets)."
 ].join("\n");
 var usagePatterns = (cmd) => USAGE_PATTERNS.replaceAll("<cmd>", cmd);
+var BUNDLE_LAYOUT = [
+  { artifact: "<stem>.md", trigger: "always", content: "the Markdown", namedBy: "Saved-To: / savedTo" },
+  { artifact: "images/", trigger: "embedded or extracted figures", content: "image files linked from the Markdown", namedBy: "Images-Dir: / imagesDir" },
+  { artifact: "pages/<stem>-pNNN.<fmt>", trigger: "--page-images", content: "page renders at --image-dpi", namedBy: "Pages-Dir: / pagesDir" },
+  { artifact: "sheets/", trigger: "Excel input", content: "one CSV per non-empty worksheet", namedBy: "Sheets-Dir: / sheetsDir" },
+  { artifact: "attachments/", trigger: "email input", content: "saved attachments", namedBy: "Markdown attachment list" },
+  { artifact: "<stem>.pages.json", trigger: "Python PDF tiers (PDF, PPTX, DOC, DOCX via LibreOffice; not unpdf)", content: "per-page chars, image count, image coverage", namedBy: "Page-Stats: / pageStatsPath" },
+  { artifact: "<stem>.words.json", trigger: "--words", content: "per-page word boxes, source text/ocr", namedBy: "Words: / wordsPath" },
+  { artifact: "ocr/<stem>-pNNN.md", trigger: "--ocr --ocr-mode all", content: "recognized text of a forced page", namedBy: "OCR-Dir: / ocr.sidecars" },
+  { artifact: "ocr/<stem>-pNNN.words.json", trigger: "--ocr --ocr-mode all --words", content: "word boxes of that OCR", namedBy: "ocr.wordSidecars" }
+];
+var bundleLayoutText = () => BUNDLE_LAYOUT.map((r) => `  ${r.artifact.padEnd(28)} ${r.trigger}; ${r.content}; named by ${r.namedBy}`).join("\n");
 function renderHelp() {
   const row = (d) => `  ${(d.flag ?? "<path>").padEnd(26)} ${d.help}${d.default !== null && d.key !== "info" && d.key !== "overwrite" ? ` (default ${d.default})` : ""}`;
   return [
@@ -1143,6 +1179,9 @@ function renderHelp() {
     "Result: a handle (Saved-To, Images-Dir, Page-Stats, Page-Count, Outline ...). Read the Saved-To file for the Markdown.",
     "Exit codes: 0 success, 1 runtime error, 2 usage error.",
     "",
+    "Bundle layout:",
+    bundleLayoutText(),
+    "",
     usagePatterns("pi-quiver doc-to-md")
   ].join("\n");
 }
@@ -1622,7 +1661,7 @@ function reconcileRenderMarkers(md, renderPages, fmt, sourceMap, reason) {
   if (/<!--rvs?:\d+-->/.test(md)) throw new Error("internal: unresolved render marker");
   return md;
 }
-var emptyOcr = (lang) => ({ status: "off", lang, textless: [], pages: [], noText: [], ocrFailed: [], budgetStopped: [], reason: null, tesseract: null, mode: "textless", sidecars: {}, ocrErrors: {}, killed: null, notAttempted: [], childError: null });
+var emptyOcr = (lang) => ({ status: "off", lang, textless: [], pages: [], noText: [], ocrFailed: [], budgetStopped: [], reason: null, tesseract: null, mode: "textless", sidecars: {}, wordSidecars: {}, ocrErrors: {}, killed: null, notAttempted: [], childError: null });
 var emptyOutcome = () => ({ written: [], noText: [], ocrFailed: [], ocrErrors: {}, budgetStopped: [], killed: null, notAttempted: [], childError: null });
 var ocrPagesTag = (n) => `p${String(n).padStart(3, "0")}`;
 var SIDE_MARKER_RE = /^--- end of page\.page_number=\d+ ---$/;
@@ -1693,11 +1732,12 @@ async function convertDocument(o, signal, seams) {
   let office = null;
   try {
     let pdfPath = inputPath;
-    const base = { path: inputPath, pages: o.pages, stagingDir: b.stagingDir, sheetsStagingDir: b.sheetsStagingDir, pageImages: o.pageImages, pagesStagingDir: b.pagesStagingDir, imageDpi: o.imageDpi, imageFormat: o.imageFormat, maxOutputBytes: o.maxOutputBytes, pymupdfVersion: o.pymupdfVersion, ocr: o.ocr && !forced, ocrLanguage: o.ocrLanguage, ocrBudgetMs: o.primaryTimeoutMs };
+    const base = { path: inputPath, pages: o.pages, ...o.words && (type === "pdf" || type === "image") ? { words: true } : {}, stagingDir: b.stagingDir, sheetsStagingDir: b.sheetsStagingDir, pageImages: o.pageImages, pagesStagingDir: b.pagesStagingDir, imageDpi: o.imageDpi, imageFormat: o.imageFormat, maxOutputBytes: o.maxOutputBytes, pymupdfVersion: o.pymupdfVersion, ocr: o.ocr && !forced, ocrLanguage: o.ocrLanguage, ocrBudgetMs: o.primaryTimeoutMs };
     let tier, engine, json, degraded = null, fallbackReason = null;
     let explicitBreaks = null;
     let notes = [];
     let officeRoute = null;
+    let copyReason = null;
     if (type === "html") {
       const prepared = await prepareHtml(inputPath, b.stagingDir);
       if (signal?.aborted) throw new Error("aborted");
@@ -1737,6 +1777,7 @@ async function convertDocument(o, signal, seams) {
       }
       if (signal?.aborted) throw new Error("aborted");
       if (json === void 0) {
+        copyReason = reason;
         clearStaging(b);
         const file = `original${extname3(inputPath).toLowerCase()}`;
         const dir = join3(b.stagingDir, "p1");
@@ -1872,14 +1913,33 @@ async function convertDocument(o, signal, seams) {
     if (tier === void 0 || engine === void 0 || json === void 0) throw new Error("internal: no tier produced output");
     const pageStats = json.pageStats ?? null;
     if (pageStats) writePageStats(b, pageStats);
+    let wordsPath = null, wordsReason = null;
+    const wordsErrors = {};
+    const takeWordsErrors = (j) => {
+      for (const [k, v] of Object.entries(j?.wordsErrors ?? {})) {
+        const page = Number(k);
+        if (Number.isFinite(page)) wordsErrors[page] = v;
+      }
+    };
+    if (o.words) {
+      if (type !== "pdf" && type !== "image") wordsReason = `none - word positions apply to PDF and image inputs only (${type})`;
+      else if (tier === "unpdf") wordsReason = "none - unpdf tier has no page geometry";
+      else if (engine === "copy") wordsReason = `none - image copied without conversion (${copyReason})`;
+      else if (json?.words === true) {
+        wordsReason = publishWords(b);
+        if (wordsReason === null) wordsPath = b.wordsPath;
+      } else wordsReason = `write failed - ${json?.wordsErrors?.file ?? "child reported no words document"}`;
+      takeWordsErrors(json);
+    }
     let ocr = handleOcr(tier, type, o, json);
     if (forced) {
-      const r = await s.runTier("ocr-pages", { path: pdfPath, pages: o.pages, stem: b.stem, ocrLanguage: o.ocrLanguage, ocrBudgetMs: o.primaryTimeoutMs, stagingDir: b.ocrStagingDir, dpi: o.imageDpi, maxOutputBytes: o.maxOutputBytes, pymupdfVersion: o.pymupdfVersion }, b, signal, o.primaryTimeoutMs, backend);
+      const r = await s.runTier("ocr-pages", { path: pdfPath, pages: o.pages, ...o.words && type === "pdf" ? { words: true } : {}, stem: b.stem, ocrLanguage: o.ocrLanguage, ocrBudgetMs: o.primaryTimeoutMs, stagingDir: b.ocrStagingDir, dpi: o.imageDpi, maxOutputBytes: o.maxOutputBytes, pymupdfVersion: o.pymupdfVersion }, b, signal, o.primaryTimeoutMs, backend);
       if (signal?.aborted || !r.ok && "reason" in r && r.reason === "aborted") throw new Error("aborted");
       if (r.ok && r.json.status === "unavailable") throw new Error(`OCR unavailable: ${r.json.reason} (install Tesseract; see doc/doc-to-md.md)`);
       const outcome = r.ok && r.json.status === "ran" ? outcomeFromChild(r.json) : recoverOcrPages(b.ocrStagingDir, o.pages, !r.ok ? "userError" in r ? r.userError : `${r.reason}${detailSuffix(r)}` : "malformed child output");
-      const sidecars = publishSidecars(b);
-      ocr = { ...emptyOcr(o.ocrLanguage), ...json.ocr, status: "ran", reason: null, mode: "all", pages: outcome.written, noText: outcome.noText, ocrFailed: outcome.ocrFailed, ocrErrors: outcome.ocrErrors, budgetStopped: outcome.budgetStopped, killed: outcome.killed, notAttempted: outcome.notAttempted, childError: outcome.childError, sidecars: Object.fromEntries(sidecars) };
+      const { sidecars, wordSidecars } = publishSidecars(b);
+      ocr = { ...emptyOcr(o.ocrLanguage), ...json.ocr, status: "ran", reason: null, mode: "all", pages: outcome.written, noText: outcome.noText, ocrFailed: outcome.ocrFailed, ocrErrors: outcome.ocrErrors, budgetStopped: outcome.budgetStopped, killed: outcome.killed, notAttempted: outcome.notAttempted, childError: outcome.childError, sidecars: Object.fromEntries(sidecars), wordSidecars: Object.fromEntries(wordSidecars) };
+      if (o.words && r.ok) takeWordsErrors(r.json);
     }
     if (!isExcel) notes = [...notes, ...json.notes ?? []];
     if (b.renamedFrom) notes.splice(notes[0]?.startsWith("preview truncated:") ? 1 : 0, 0, `renamed to ${b.stem} (${b.renameReason})`);
@@ -1897,7 +1957,7 @@ async function convertDocument(o, signal, seams) {
 ` : "") + body;
     commitBundle(b, markdown);
     const outline = scanOutline(markdown, o.outlineMaxEntries);
-    const details = { path: inputPath, backend: backend.kind, pymupdfVersion: o.pymupdfVersion, inputType: type, file: b.mdPath, outputDir: b.root, savedTo: b.mdPath, imagesDir: b.imagesDir, sheetsDir: b.csvManifest.size ? b.sheetsDir : null, pagesDir: b.pageManifest.size ? b.pagesDir : null, pageImageCount: b.pageManifest.size, pageImagesReason, type, engine, tier, pageCount: json.pageCount ?? null, pages: o.pages, explicitBreaks, imageCount: b.manifest.size, bytes: Buffer.byteLength(markdown, "utf8"), lines: markdown.split("\n").length, degraded, fallbackReason, failedPages: (json.failedPages ?? []).map((f) => f.page), emptyPages: json.emptyPages ?? [], notes, outline: outline.entries, outlineTotal: outline.total, ocr, pageStats, pageStatsPath: pageStats ? b.pageStatsPath : null, ocrDir: b.ocrManifest.size ? b.ocrDir : null };
+    const details = { path: inputPath, backend: backend.kind, pymupdfVersion: o.pymupdfVersion, inputType: type, file: b.mdPath, outputDir: b.root, savedTo: b.mdPath, imagesDir: b.imagesDir, sheetsDir: b.csvManifest.size ? b.sheetsDir : null, pagesDir: b.pageManifest.size ? b.pagesDir : null, pageImageCount: b.pageManifest.size, pageImagesReason, type, engine, tier, pageCount: json.pageCount ?? null, pages: o.pages, explicitBreaks, imageCount: b.manifest.size, bytes: Buffer.byteLength(markdown, "utf8"), lines: markdown.split("\n").length, degraded, fallbackReason, failedPages: (json.failedPages ?? []).map((f) => f.page), emptyPages: json.emptyPages ?? [], notes, outline: outline.entries, outlineTotal: outline.total, ocr, pageStats, pageStatsPath: pageStats ? b.pageStatsPath : null, ocrDir: b.ocrManifest.size ? b.ocrDir : null, wordsPath, wordsReason, wordsErrors };
     return { output: formatHandle(details), details };
   } catch (e) {
     abortBundle(b);
@@ -1943,7 +2003,7 @@ async function inspectDocument(o, signal, seams) {
 }
 
 // bin/pi-quiver.ts
-var USAGE = 'Usage: pi-quiver fetch <url> [--method GET|HEAD|POST] [--header "K: V"]... [--body <str>] [--raw] [--timeout-ms <n>]\n       pi-quiver doc-to-md [--json] [--info] [--page-images] [--pages <spec>] [--output-dir <dir>] [--overwrite] [--ocr] [--ocr-mode textless|all] [tunable flags] <path>   (--help for all flags)';
+var USAGE = 'Usage: pi-quiver fetch <url> [--method GET|HEAD|POST] [--header "K: V"]... [--body <str>] [--raw] [--timeout-ms <n>]\n       pi-quiver doc-to-md [--json] [--info] [--page-images] [--words] [--pages <spec>] [--output-dir <dir>] [--overwrite] [--ocr] [--ocr-mode textless|all] [tunable flags] <path>   (--help for all flags)';
 function parseDocToMd(rest) {
   if (rest.includes("--help") || rest.includes("-h")) return { ok: true, cmd: "doc-to-md-help" };
   const json = rest.includes("--json");

package/lib/doc-to-md-bundle.ts

@@ -15,7 +15,7 @@ export interface Bundle {
 	sheetsDir: string; sheetsStagingDir: string;
 	pagesDir: string; pagesStagingDir: string;
 	attachmentsDir: string; attachmentsStagingDir: string;
-	ocrDir: string; ocrStagingDir: string; pageStatsPath: string;
+	ocrDir: string; ocrStagingDir: string; pageStatsPath: string; wordsPath: string;
 	manifest: Set<string>;
 	csvManifest: Set<string>;
 	pageManifest: Set<string>;
@@ -36,7 +36,7 @@ export function ownedCsvPattern(stem: string): RegExp {
 
 export function ownedPagePattern(stem: string): RegExp { return new RegExp(`^${escRe(stem)}-p\\d+\\.[a-z0-9]+$`); }
 
-export function ownedOcrPattern(stem: string): RegExp { return new RegExp(`^${escRe(stem)}-p\\d+\\.md$`); }
+export function ownedOcrPattern(stem: string): RegExp { return new RegExp(`^${escRe(stem)}-p\\d+(?:\\.words\\.json|\\.md)$`); }
 
 const FILE_LINK_RE = /\[[^\]]*\]\(\s*((?:sheets|attachments)\/[^)\s]+)\s*\)/g;
 const IMG_LINK_RE = /!\[[^\]]*\]\(\s*(?:<([^>]*)>|([^)]*?))\s*\)|<img\b[^>]*\bsrc\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s>"']+))/gi;
@@ -94,6 +94,7 @@ export function openBundle(root: string, requested: string, overwrite: boolean):
 			if (existsSync(pagesDir)) for (const f of readdirSync(pagesDir)) if (ownedPage.test(f)) rmSync(join(pagesDir, f), { force: true });
 			if (existsSync(attachmentsDir)) for (const f of readdirSync(attachmentsDir)) if (attachments.has(f)) rmSync(join(attachmentsDir, f), { force: true });
 			rmSync(join(root, `${stem}.pages.json`), { force: true });
+			rmSync(join(root, `${stem}.words.json`), { force: true });
 			const ownedOcr = ownedOcrPattern(stem);
 			if (existsSync(ocrDir)) for (const f of readdirSync(ocrDir)) if (ownedOcr.test(f)) rmSync(join(ocrDir, f), { force: true });
 		}
@@ -104,7 +105,7 @@ export function openBundle(root: string, requested: string, overwrite: boolean):
 		const pagesStagingDir = join(pagesDir, `.stage-${lockId}`);
 		const attachmentsStagingDir = join(attachmentsDir, `.stage-${lockId}`);
 		const ocrStagingDir = join(ocrDir, `.stage-${lockId}`);
-		return { root, stem, renamedFrom, renameReason, mdPath, lockPath, imagesDir, stagingDir, lockId, sheetsDir, sheetsStagingDir, pagesDir, pagesStagingDir, attachmentsDir, attachmentsStagingDir, ocrDir, ocrStagingDir, pageStatsPath: join(root, `${stem}.pages.json`), ocrManifest: new Set(), manifest: new Set(), csvManifest: new Set(), pageManifest: new Set(), attachmentManifest: new Set(), sourceMap: new Map() };
+		return { root, stem, renamedFrom, renameReason, mdPath, lockPath, imagesDir, stagingDir, lockId, sheetsDir, sheetsStagingDir, pagesDir, pagesStagingDir, attachmentsDir, attachmentsStagingDir, ocrDir, ocrStagingDir, pageStatsPath: join(root, `${stem}.pages.json`), wordsPath: join(root, `${stem}.words.json`), ocrManifest: new Set(), manifest: new Set(), csvManifest: new Set(), pageManifest: new Set(), attachmentManifest: new Set(), sourceMap: new Map() };
 	} catch (e) { rmSync(lockPath, { force: true }); throw e; }
 }
 
@@ -174,10 +175,23 @@ export function writePageStats(b: Bundle, stats: PageStat[]): void {
 	writeFileSync(b.pageStatsPath, `${JSON.stringify(stats, null, 2)}\n`, "utf8");
 }
 
-/** Move every `.done`-gated `pNNN/<stem>-pNNN.md` into `ocr/`; partial page dirs and the checkpoint are dropped with the staging dir. Returns page -> absolute sidecar path. */
-export function publishSidecars(b: Bundle): Map<number, string> {
-	const out = new Map<number, string>();
-	if (!existsSync(b.ocrStagingDir)) return out;
+/** Rename within the bundle root keeps the complete words document atomic. */
+export function publishWords(b: Bundle): string | null {
+	const staged = join(b.stagingDir, "words.json");
+	try {
+		if (!existsSync(staged)) throw new Error("child staged no words.json");
+		renameSync(staged, b.wordsPath);
+		return null;
+	} catch (e) {
+		rmSync(b.wordsPath, { force: true });
+		return `write failed - ${(e as Error).message}`;
+	}
+}
+
+/** Move `.done`-gated Markdown and words sidecars into `ocr/`; drop partial dirs and the checkpoint. Returns page -> absolute paths for each kind. */
+export function publishSidecars(b: Bundle): { sidecars: Map<number, string>; wordSidecars: Map<number, string> } {
+	const sidecars = new Map<number, string>(), wordSidecars = new Map<number, string>();
+	if (!existsSync(b.ocrStagingDir)) return { sidecars, wordSidecars };
 	for (const dir of readdirSync(b.ocrStagingDir).sort()) {
 		const m = dir.match(/^p(\d+)$/);
 		if (!m) continue;
@@ -188,10 +202,16 @@ export function publishSidecars(b: Bundle): Map<number, string> {
 		mkdirSync(b.ocrDir, { recursive: true });
 		renameSync(join(pageDir, file), join(b.ocrDir, file));
 		b.ocrManifest.add(file);
-		out.set(Number(m[1]), join(b.ocrDir, file));
+		sidecars.set(Number(m[1]), join(b.ocrDir, file));
+		const wfile = `${b.stem}-${dir}.words.json`;
+		if (existsSync(join(pageDir, wfile))) {
+			renameSync(join(pageDir, wfile), join(b.ocrDir, wfile));
+			b.ocrManifest.add(wfile);
+			wordSidecars.set(Number(m[1]), join(b.ocrDir, wfile));
+		}
 	}
 	rmSync(b.ocrStagingDir, { recursive: true, force: true });
-	return out;
+	return { sidecars, wordSidecars };
 }
 
 export function publishAttachments(b: Bundle): void {
@@ -254,6 +274,7 @@ export function abortBundle(b: Bundle): void {
 	for (const f of b.ocrManifest) rmSync(join(b.ocrDir, f), { force: true });
 	rmSync(b.ocrStagingDir, { recursive: true, force: true });
 	rmSync(b.pageStatsPath, { force: true });
+	rmSync(b.wordsPath, { force: true });
 	rmSync(`${b.mdPath}.tmp`, { force: true });
 	rmSync(b.stagingDir, { recursive: true, force: true });
 	rmSync(b.sheetsStagingDir, { recursive: true, force: true });

package/lib/doc-to-md-core.ts

@@ -12,7 +12,7 @@ import { type ChildProcess, spawn } from "node:child_process";
 import { homedir, tmpdir } from "node:os";
 import { basename, dirname, extname, isAbsolute, join, resolve } from "node:path";
 import { fileURLToPath, pathToFileURL } from "node:url";
-import { type Bundle, abortBundle, commitBundle, openBundle, publishAttachments, publishSidecars, writePageStats, publishPageImages, publishSheetCsvs, publishSheetImages, publishStaged, rewriteLinks, tempBundleRoot, validateImageLinks } from "./doc-to-md-bundle.ts";
+import { type Bundle, abortBundle, commitBundle, openBundle, publishAttachments, publishSidecars, publishWords, writePageStats, publishPageImages, publishSheetCsvs, publishSheetImages, publishStaged, rewriteLinks, tempBundleRoot, validateImageLinks } from "./doc-to-md-bundle.ts";
 import { type Engine, type HandleData, type InfoData, type OcrInfo, type PageStat, type Tier, formatHandle, formatInfoHandle, scanOutline, type SheetInfo, type TocEntry } from "./doc-to-md-handle.ts";
 import { type DocToMdOptions, type InputType, IMAGE_EXTS, TUNABLE_DEFAULTS, UsageError, classifyInput, sanitizeStem } from "./doc-to-md-options.ts";
 
@@ -467,7 +467,7 @@ const clearStaging = (b: Pick<Bundle, "stagingDir">) => { for (const f of readdi
 export const EXCEL_REMEDY = "Remedy: install uv, or pip install openpyxl xlrd pillow";
 
 export type Mode = "html" | "image" | "info" | "pdf-primary" | "pdf-fallback" | "xlsx" | "pdf-text" | "render-pages" | "docx" | "email" | "ocr-pages";
-export interface TierJson { pageStats?: PageStat[]; status?: string; written?: number[]; noText?: number[]; ocrFailed?: number[]; ocrErrors?: Record<string, string>; budgetStopped?: number[]; pageImages?: { page: number; file: string }[]; ocr?: OcrInfo; markdown?: string; pages?: number[]; pageCount?: number; emptyPages?: number[]; failedPages?: { page: number; error: string }[]; notes?: string[]; images?: { sheetIndex: number; file: string }[]; metadata?: Record<string, string>; toc?: [number, string, number | null][]; explicitBreaks?: number; engine?: string; degraded?: boolean; fallbackReason?: string | null; sheets?: SheetInfo[]; renderPages?: number[]; sheetCount?: number; ok?: boolean; reason?: string; rendered?: { idx: number; file: string; dpi: number }[]; failed?: { idx: number; reason: string }[]; }
+export interface TierJson { words?: boolean; wordsErrors?: Record<string, string>; pageStats?: PageStat[]; status?: string; written?: number[]; noText?: number[]; ocrFailed?: number[]; ocrErrors?: Record<string, string>; budgetStopped?: number[]; pageImages?: { page: number; file: string }[]; ocr?: OcrInfo; markdown?: string; pages?: number[]; pageCount?: number; emptyPages?: number[]; failedPages?: { page: number; error: string }[]; notes?: string[]; images?: { sheetIndex: number; file: string }[]; metadata?: Record<string, string>; toc?: [number, string, number | null][]; explicitBreaks?: number; engine?: string; degraded?: boolean; fallbackReason?: string | null; sheets?: SheetInfo[]; renderPages?: number[]; sheetCount?: number; ok?: boolean; reason?: string; rendered?: { idx: number; file: string; dpi: number }[]; failed?: { idx: number; reason: string }[]; }
 export type TierResult = { ok: true; json: TierJson } | { ok: false; reason: string; detail?: string } | { ok: false; userError: string; pageCount?: number };
 
 export interface PipelineSeams {
@@ -536,7 +536,7 @@ export function reconcileRenderMarkers(md: string, renderPages: number[], fmt: s
 	return md;
 }
 
-export const emptyOcr = (lang: string): OcrInfo => ({ status: "off", lang, textless: [], pages: [], noText: [], ocrFailed: [], budgetStopped: [], reason: null, tesseract: null, mode: "textless", sidecars: {}, ocrErrors: {}, killed: null, notAttempted: [], childError: null });
+export const emptyOcr = (lang: string): OcrInfo => ({ status: "off", lang, textless: [], pages: [], noText: [], ocrFailed: [], budgetStopped: [], reason: null, tesseract: null, mode: "textless", sidecars: {}, wordSidecars: {}, ocrErrors: {}, killed: null, notAttempted: [], childError: null });
 
 export interface OcrPagesOutcome { written: number[]; noText: number[]; ocrFailed: number[]; ocrErrors: Record<number, string>; budgetStopped: number[]; killed: number | null; notAttempted: number[]; childError: string | null; }
 const emptyOutcome = (): OcrPagesOutcome => ({ written: [], noText: [], ocrFailed: [], ocrErrors: {}, budgetStopped: [], killed: null, notAttempted: [], childError: null });
@@ -612,11 +612,12 @@ export async function convertDocument(o: DocToMdOptions, signal?: AbortSignal, s
 	let office: { pdfPath: string; cleanup: () => void } | null = null;
 	try {
 		let pdfPath = inputPath;
-		const base = { path: inputPath, pages: o.pages, stagingDir: b.stagingDir, sheetsStagingDir: b.sheetsStagingDir, pageImages: o.pageImages, pagesStagingDir: b.pagesStagingDir, imageDpi: o.imageDpi, imageFormat: o.imageFormat, maxOutputBytes: o.maxOutputBytes, pymupdfVersion: o.pymupdfVersion, ocr: o.ocr && !forced, ocrLanguage: o.ocrLanguage, ocrBudgetMs: o.primaryTimeoutMs };
+		const base = { path: inputPath, pages: o.pages, ...(o.words && (type === "pdf" || type === "image") ? { words: true } : {}), stagingDir: b.stagingDir, sheetsStagingDir: b.sheetsStagingDir, pageImages: o.pageImages, pagesStagingDir: b.pagesStagingDir, imageDpi: o.imageDpi, imageFormat: o.imageFormat, maxOutputBytes: o.maxOutputBytes, pymupdfVersion: o.pymupdfVersion, ocr: o.ocr && !forced, ocrLanguage: o.ocrLanguage, ocrBudgetMs: o.primaryTimeoutMs };
 		let tier: Tier | undefined, engine: Engine | undefined, json: TierJson | undefined, degraded: string | null = null, fallbackReason: string | null = null;
 		let explicitBreaks: number | null = null;
 		let notes: string[] = [];
 		let officeRoute: string | null = null;
+		let copyReason: string | null = null;
 		if (type === "html") {
 			const prepared = await prepareHtml(inputPath, b.stagingDir);
 			if (signal?.aborted) throw new Error("aborted");
@@ -642,6 +643,7 @@ export async function convertDocument(o: DocToMdOptions, signal?: AbortSignal, s
 			}
 			if (signal?.aborted) throw new Error("aborted");
 			if (json === undefined) {
+				copyReason = reason;
 				clearStaging(b);
 				const file = `original${extname(inputPath).toLowerCase()}`;
 				const dir = join(b.stagingDir, "p1");
@@ -745,15 +747,32 @@ export async function convertDocument(o: DocToMdOptions, signal?: AbortSignal, s
 		if (tier === undefined || engine === undefined || json === undefined) throw new Error("internal: no tier produced output");
 		const pageStats = json.pageStats ?? null;
 		if (pageStats) writePageStats(b, pageStats);
+		let wordsPath: string | null = null, wordsReason: string | null = null;
+		const wordsErrors: Record<number, string> = {};
+		const takeWordsErrors = (j: TierJson | undefined) => {
+			for (const [k, v] of Object.entries(j?.wordsErrors ?? {})) {
+				const page = Number(k);
+				if (Number.isFinite(page)) wordsErrors[page] = v;
+			}
+		};
+		if (o.words) {
+			if (type !== "pdf" && type !== "image") wordsReason = `none - word positions apply to PDF and image inputs only (${type})`;
+			else if (tier === "unpdf") wordsReason = "none - unpdf tier has no page geometry";
+			else if (engine === "copy") wordsReason = `none - image copied without conversion (${copyReason})`;
+			else if (json?.words === true) { wordsReason = publishWords(b); if (wordsReason === null) wordsPath = b.wordsPath; }
+			else wordsReason = `write failed - ${json?.wordsErrors?.file ?? "child reported no words document"}`;
+			takeWordsErrors(json);
+		}
 		let ocr = handleOcr(tier, type, o, json);
 		if (forced) {
-			const r = await s.runTier("ocr-pages", { path: pdfPath, pages: o.pages, stem: b.stem, ocrLanguage: o.ocrLanguage, ocrBudgetMs: o.primaryTimeoutMs, stagingDir: b.ocrStagingDir, dpi: o.imageDpi, maxOutputBytes: o.maxOutputBytes, pymupdfVersion: o.pymupdfVersion }, b, signal, o.primaryTimeoutMs, backend);
+			const r = await s.runTier("ocr-pages", { path: pdfPath, pages: o.pages, ...(o.words && type === "pdf" ? { words: true } : {}), stem: b.stem, ocrLanguage: o.ocrLanguage, ocrBudgetMs: o.primaryTimeoutMs, stagingDir: b.ocrStagingDir, dpi: o.imageDpi, maxOutputBytes: o.maxOutputBytes, pymupdfVersion: o.pymupdfVersion }, b, signal, o.primaryTimeoutMs, backend);
 			if (signal?.aborted || (!r.ok && "reason" in r && r.reason === "aborted")) throw new Error("aborted");
 			if (r.ok && r.json.status === "unavailable") throw new Error(`OCR unavailable: ${r.json.reason} (install Tesseract; see doc/doc-to-md.md)`);
 			const outcome = r.ok && r.json.status === "ran" ? outcomeFromChild(r.json)
 				: recoverOcrPages(b.ocrStagingDir, o.pages!, !r.ok ? ("userError" in r ? r.userError : `${r.reason}${detailSuffix(r)}`) : "malformed child output");
-			const sidecars = publishSidecars(b);
-			ocr = { ...emptyOcr(o.ocrLanguage), ...json.ocr, status: "ran", reason: null, mode: "all", pages: outcome.written, noText: outcome.noText, ocrFailed: outcome.ocrFailed, ocrErrors: outcome.ocrErrors, budgetStopped: outcome.budgetStopped, killed: outcome.killed, notAttempted: outcome.notAttempted, childError: outcome.childError, sidecars: Object.fromEntries(sidecars) };
+			const { sidecars, wordSidecars } = publishSidecars(b);
+			ocr = { ...emptyOcr(o.ocrLanguage), ...json.ocr, status: "ran", reason: null, mode: "all", pages: outcome.written, noText: outcome.noText, ocrFailed: outcome.ocrFailed, ocrErrors: outcome.ocrErrors, budgetStopped: outcome.budgetStopped, killed: outcome.killed, notAttempted: outcome.notAttempted, childError: outcome.childError, sidecars: Object.fromEntries(sidecars), wordSidecars: Object.fromEntries(wordSidecars) };
+			if (o.words && r.ok) takeWordsErrors(r.json);
 		}
 		if (!isExcel) notes = [...notes, ...(json.notes ?? [])];
 		if (b.renamedFrom) notes.splice(notes[0]?.startsWith("preview truncated:") ? 1 : 0, 0, `renamed to ${b.stem} (${b.renameReason})`);
@@ -770,7 +789,7 @@ export async function convertDocument(o: DocToMdOptions, signal?: AbortSignal, s
 		const markdown = (head.length ? `${head.join("\n")}\n\n` : "") + body;
 		commitBundle(b, markdown);
 		const outline = scanOutline(markdown, o.outlineMaxEntries);
-		const details: DocToMdDetails = { path: inputPath, backend: backend.kind, pymupdfVersion: o.pymupdfVersion, inputType: type, file: b.mdPath, outputDir: b.root, savedTo: b.mdPath, imagesDir: b.imagesDir, sheetsDir: b.csvManifest.size ? b.sheetsDir : null, pagesDir: b.pageManifest.size ? b.pagesDir : null, pageImageCount: b.pageManifest.size, pageImagesReason, type, engine, tier, pageCount: json.pageCount ?? null, pages: o.pages, explicitBreaks, imageCount: b.manifest.size, bytes: Buffer.byteLength(markdown, "utf8"), lines: markdown.split("\n").length, degraded, fallbackReason, failedPages: (json.failedPages ?? []).map((f) => f.page), emptyPages: json.emptyPages ?? [], notes, outline: outline.entries, outlineTotal: outline.total, ocr, pageStats, pageStatsPath: pageStats ? b.pageStatsPath : null, ocrDir: b.ocrManifest.size ? b.ocrDir : null };
+		const details: DocToMdDetails = { path: inputPath, backend: backend.kind, pymupdfVersion: o.pymupdfVersion, inputType: type, file: b.mdPath, outputDir: b.root, savedTo: b.mdPath, imagesDir: b.imagesDir, sheetsDir: b.csvManifest.size ? b.sheetsDir : null, pagesDir: b.pageManifest.size ? b.pagesDir : null, pageImageCount: b.pageManifest.size, pageImagesReason, type, engine, tier, pageCount: json.pageCount ?? null, pages: o.pages, explicitBreaks, imageCount: b.manifest.size, bytes: Buffer.byteLength(markdown, "utf8"), lines: markdown.split("\n").length, degraded, fallbackReason, failedPages: (json.failedPages ?? []).map((f) => f.page), emptyPages: json.emptyPages ?? [], notes, outline: outline.entries, outlineTotal: outline.total, ocr, pageStats, pageStatsPath: pageStats ? b.pageStatsPath : null, ocrDir: b.ocrManifest.size ? b.ocrDir : null, wordsPath, wordsReason, wordsErrors };
 		return { output: formatHandle(details), details };
 	} catch (e) { abortBundle(b); throw e; }
 	finally { office?.cleanup(); }

package/lib/doc-to-md-handle.ts

@@ -32,6 +32,7 @@ export interface OcrInfo {
 	tesseract: boolean | null;
 	mode: OcrMode;
 	sidecars: Record<number, string>;
+	wordSidecars: Record<number, string>;
 	ocrErrors: Record<number, string>;
 	killed: number | null;
 	notAttempted: number[];
@@ -44,6 +45,7 @@ export interface HandleData {
 	degraded: string | null; fallbackReason: string | null; failedPages: number[]; emptyPages: number[];
 	notes: string[]; outline: OutlineEntry[]; outlineTotal: number; ocr: OcrInfo | null;
 	pageStats: PageStat[] | null; pageStatsPath: string | null; ocrDir: string | null;
+	wordsPath: string | null; wordsReason: string | null; wordsErrors: Record<number, string>;
 }
 
 export interface InfoData {
@@ -169,6 +171,10 @@ export function formatHandle(h: HandleData): string {
 	if (h.pagesDir && h.pageImageCount > 0) lines.push(`Pages-Dir: ${h.pagesDir} (${h.pageImageCount} pages)`);
 	else if (h.pageImagesReason) lines.push(`Pages-Dir: none - ${h.pageImagesReason}`);
 	if (h.pageStatsPath) lines.push(`Page-Stats: ${h.pageStatsPath}`);
+	if (h.wordsPath) {
+		const bad = Object.keys(h.wordsErrors).map(Number).sort((a, b) => a - b);
+		lines.push(`Words: ${h.wordsPath}${bad.length ? ` (extraction failed for pages ${compactRanges(bad)}: ${h.wordsErrors[bad[0]]})` : ""}`);
+	} else if (h.wordsReason) lines.push(`Words: ${h.wordsReason}`);
 	if (h.ocrDir) lines.push(`OCR-Dir: ${h.ocrDir}`);
 	lines.push(`Type: ${h.type}   Engine: ${h.engine}   Tier: ${h.tier}`);
 	lines.push(`Page-Count: ${pageCountLabel(h)}   Pages: ${h.pages ? compactRanges(h.pages) : "all"}   Images: ${h.imageCount}   Size: ${formatSize(h.bytes)} / ${h.lines} lines`);

package/lib/doc-to-md-options.ts

@@ -30,6 +30,7 @@ export interface DocToMdOptions extends Tunables {
 	outputDir: string | null;
 	overwrite: boolean;
 	pageImages: boolean;
+	words: boolean;
 	ocrMode: OcrMode;
 }
 
@@ -41,6 +42,7 @@ export interface PerCallInput extends Partial<Tunables> {
 	outputDir?: string | null;
 	overwrite?: boolean;
 	pageImages?: boolean;
+	words?: boolean;
 	ocrMode?: OcrMode;
 }
 
@@ -71,6 +73,7 @@ export const DOC_TO_MD_OPTIONS: readonly OptionDescriptor[] = [
 	{ key: "outputDir", flag: "--output-dir", type: "string", default: null, settable: false, help: "Bundle root for <stem>.md + images/; default a per-call temp dir. <stem> = basename without extension with [^A-Za-z0-9._-]+ -> _ (empty -> document); a second call on the same stem writes <stem>-2.md" },
 	{ key: "overwrite", flag: "--overwrite", type: "bool", default: false, settable: false, help: "Replace an existing completed <stem>.md bundle" },
 	{ key: "pageImages", flag: "--page-images", type: "bool", default: false, settable: false, help: "Also render every selected page to pages/<stem>-pNNN.<imageFormat> at imageDpi (PDF, PPTX, .doc, DOCX via LibreOffice); off by default" },
+	{ key: "words", flag: "--words", type: "bool", default: false, settable: false, help: "Write word positions: <stem>.words.json beside the Markdown lists every text-layer word of each selected page with its bbox (PDF points, top-left origin, display orientation; image inputs in source pixels) and the words inline OCR recognized, tagged source \"text\" or \"ocr\"; under --ocr-mode all the OCR words go to ocr/<stem>-pNNN.words.json beside each sidecar. Never triggers OCR. PDF and image inputs only." },
 	{ key: "ocrMode", flag: "--ocr-mode", type: "enum", default: "textless", settable: false, enumValues: ["textless", "all"], help: "OCR policy: textless (default) OCRs only pages with an empty text layer, inline; all OCRs every selected page and writes the recognized text to ocr/<stem>-pNNN.md sidecars, leaving the Markdown untouched. all requires --ocr and an explicit --pages selection (PDF, PPTX, DOC)." },
 	{ key: "primaryTimeoutMs", flag: "--primary-timeout", type: "int", default: 60000, settable: true, env: "PI_DOC_TO_MD_CONVERT_TIMEOUT_MS", help: "pymupdf4llm tier and DOCX child (docx mode); also the unpdf tier" },
 	{ key: "fallbackTimeoutMs", flag: "--fallback-timeout", type: "int", default: 30000, settable: true, help: "PyMuPDF get_text tier (including DOCX LibreOffice fallback); also PDF and DOCX info and Excel rendered views" },
@@ -181,8 +184,8 @@ export function resolveOptions(perCall: PerCallInput, settings: Partial<Tunables
 		out[d.key] = value;
 	}
 	const o = out as unknown as DocToMdOptions;
-	if (o.info && (o.pages !== null || o.outputDir !== null || o.overwrite || o.pageImages || o.ocrMode === "all")) {
-		throw new UsageError("--info cannot be combined with --pages, --output-dir, --overwrite, --page-images or --ocr-mode all");
+	if (o.info && (o.pages !== null || o.outputDir !== null || o.overwrite || o.pageImages || o.words || o.ocrMode === "all")) {
+		throw new UsageError("--info cannot be combined with --pages, --output-dir, --overwrite, --page-images, --words or --ocr-mode all");
 	}
 	return o;
 }
@@ -207,6 +210,21 @@ export const USAGE_PATTERNS = [
 
 export const usagePatterns = (cmd: string): string => USAGE_PATTERNS.replaceAll("<cmd>", cmd);
 
+/** Every artifact a bundle can contain; shared by --help and the generated skill. */
+export const BUNDLE_LAYOUT: readonly { artifact: string; trigger: string; content: string; namedBy: string }[] = [
+	{ artifact: "<stem>.md", trigger: "always", content: "the Markdown", namedBy: "Saved-To: / savedTo" },
+	{ artifact: "images/", trigger: "embedded or extracted figures", content: "image files linked from the Markdown", namedBy: "Images-Dir: / imagesDir" },
+	{ artifact: "pages/<stem>-pNNN.<fmt>", trigger: "--page-images", content: "page renders at --image-dpi", namedBy: "Pages-Dir: / pagesDir" },
+	{ artifact: "sheets/", trigger: "Excel input", content: "one CSV per non-empty worksheet", namedBy: "Sheets-Dir: / sheetsDir" },
+	{ artifact: "attachments/", trigger: "email input", content: "saved attachments", namedBy: "Markdown attachment list" },
+	{ artifact: "<stem>.pages.json", trigger: "Python PDF tiers (PDF, PPTX, DOC, DOCX via LibreOffice; not unpdf)", content: "per-page chars, image count, image coverage", namedBy: "Page-Stats: / pageStatsPath" },
+	{ artifact: "<stem>.words.json", trigger: "--words", content: "per-page word boxes, source text/ocr", namedBy: "Words: / wordsPath" },
+	{ artifact: "ocr/<stem>-pNNN.md", trigger: "--ocr --ocr-mode all", content: "recognized text of a forced page", namedBy: "OCR-Dir: / ocr.sidecars" },
+	{ artifact: "ocr/<stem>-pNNN.words.json", trigger: "--ocr --ocr-mode all --words", content: "word boxes of that OCR", namedBy: "ocr.wordSidecars" },
+];
+
+const bundleLayoutText = (): string => BUNDLE_LAYOUT.map((r) => `  ${r.artifact.padEnd(28)} ${r.trigger}; ${r.content}; named by ${r.namedBy}`).join("\n");
+
 export function renderHelp(): string {
 	const row = (d: OptionDescriptor) => `  ${(d.flag ?? "<path>").padEnd(26)} ${d.help}${d.default !== null && d.key !== "info" && d.key !== "overwrite" ? ` (default ${d.default})` : ""}`;
 	return [
@@ -216,6 +234,6 @@ export function renderHelp(): string {
 		...DOC_TO_MD_OPTIONS.filter((d) => d.settable).map(row),
 		"", "Result: a handle (Saved-To, Images-Dir, Page-Stats, Page-Count, Outline ...). Read the Saved-To file for the Markdown.",
 		"Exit codes: 0 success, 1 runtime error, 2 usage error.",
-		"", usagePatterns("pi-quiver doc-to-md"),
+		"", "Bundle layout:", bundleLayoutText(), "", usagePatterns("pi-quiver doc-to-md"),
 	].join("\n");
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-quiver",
-  "version": "6.9.0",
+  "version": "6.10.0",
   "description": "Personal pack of Pi coding-agent extensions: context-safe fetch, doc_to_md PDF/DOCX/PPTX-to-Markdown conversion, session naming, a themed ASCII startup header, Opus 4.8 fast mode, and a provider-stall watchdog.",
   "author": "Jacek Juraszek",
   "license": "MIT",

package/scripts/doc_to_md.py

@@ -205,11 +205,22 @@ def mode_image(o):
     info = new_ocr(lang)
     status = ocr_status(bool(o.get("ocr")), lang)
     apply_status(info, status)
-    if status["status"] == "ready":
+    words_on = bool(o.get("words"))
+    word_pages, words_errors = [], {}
+    w_px = h_px = None
+    if words_on or status["status"] == "ready":
         try:
             pix = pymupdf.Pixmap(path)
             w_px, h_px = pix.width, pix.height
             del pix
+        except Exception as exc:  # noqa: BLE001
+            if words_on:
+                words_errors["1"] = words_error(exc)
+            if status["status"] == "ready":
+                info["ocrFailed"].append(1)
+                status = {**status, "status": "failed"}
+    if status["status"] == "ready":
+        try:
             if min(w_px, h_px) < MIN_OCR_SIDE_PX:
                 info["status"], info["reason"] = "skipped", "image too small"
             else:
@@ -218,6 +229,19 @@ def mode_image(o):
                     text = pymupdf4llm.to_markdown(pdf, pages=[0], write_images=False, use_ocr=True, force_ocr=True,
                                                    ocr_language=lang, ocr_dpi=image_ocr_dpi(w_px, r.width, r.height),
                                                    page_separators=False).strip()
+                    if words_on:
+                        try:
+                            pg = pdf[0]
+                            entry = words_page(pg, 1, 0, page_words(pg, True))
+                            sx, sy = w_px / r.width, h_px / r.height
+                            for word in entry["words"]:
+                                b = word["bbox"]
+                                word["bbox"] = [round(b[0] * sx, 1), round(b[1] * sy, 1),
+                                                round(b[2] * sx, 1), round(b[3] * sy, 1)]
+                            entry["width"], entry["height"] = w_px, h_px
+                            word_pages.append(entry)
+                        except Exception as exc:  # noqa: BLE001
+                            words_errors["1"] = words_error(exc)
                 if text:
                     info["pages"].append(1)
                     md += "\n\n" + ocr_block(f"p1/{name}", text)
@@ -225,7 +249,17 @@ def mode_image(o):
                     info["noText"].append(1)
         except Exception:  # noqa: BLE001 - OCR never fails the conversion
             info["ocrFailed"].append(1)
-    return {"markdown": md + "\n", "pageCount": 1, "emptyPages": [], "failedPages": [], "notes": [], "ocr": info}
+    if words_on and not o.get("ocr") and w_px is not None and "1" not in words_errors:
+        try:
+            if os.environ.get("DOC_TO_MD_WORDS_FAIL") == "1":  # tests only
+                raise RuntimeError("words injected failure")
+            word_pages.append({"page": 1, "width": w_px, "height": h_px, "rotation": 0, "words": []})
+        except Exception as exc:  # noqa: BLE001
+            words_errors["1"] = words_error(exc)
+    result = {"markdown": md + "\n", "pageCount": 1, "emptyPages": [], "failedPages": [], "notes": [], "ocr": info}
+    if words_on:
+        result.update(write_words(o["stagingDir"], "px", word_pages, words_errors))
+    return result
 
 
 def mode_info(o):
@@ -266,6 +300,58 @@ def page_stats(doc, n):
         return {"page": n, "error": f"{type(exc).__name__}: {exc}"[:300]}
 
 
+GLYPHLESS_FONT = "GlyphLessFont"
+
+
+def word_entries(page, words, ocr_lines):
+    import pymupdf
+    matrix = page.rotation_matrix
+    out = []
+    for x0, y0, x1, y1, text, bno, lno, _ in words:
+        r = pymupdf.Rect(x0, y0, x1, y1) * matrix
+        out.append({"text": text, "bbox": [round(r.x0, 1), round(r.y0, 1), round(r.x1, 1), round(r.y1, 1)],
+                    "source": "ocr" if ocr_lines is None or (bno, lno) in ocr_lines else "text"})
+    return out
+
+
+def page_words(page, inline_ocr_ran):
+    if not inline_ocr_ran:
+        return word_entries(page, page.get_text("words"), set())
+    # Shared textpage keeps block/line indexes aligned despite differing default image flags.
+    tp = page.get_textpage()
+    ocr_lines = {(bi, li) for bi, b in enumerate(page.get_text("dict", textpage=tp)["blocks"])
+                 for li, line in enumerate(b.get("lines", []))
+                 if line["spans"] and all(s["font"] == GLYPHLESS_FONT for s in line["spans"])}
+    return word_entries(page, page.get_text("words", textpage=tp), ocr_lines)
+
+
+def ocr_words(page, tp):
+    return word_entries(page, page.get_text("words", textpage=tp), None)
+
+
+def words_page(page, n, rotation, words):
+    if os.environ.get("DOC_TO_MD_WORDS_FAIL") == str(n):  # tests only
+        raise RuntimeError("words injected failure")
+    return {"page": n, "width": round(page.rect.width, 1), "height": round(page.rect.height, 1),
+            "rotation": rotation, "words": words}
+
+
+def words_error(exc):
+    return f"{type(exc).__name__}: {exc}"[:300]
+
+
+def write_words(staging, unit, pages, errors):
+    try:
+        os.makedirs(staging, exist_ok=True)
+        with open(os.path.join(staging, "words.json"), "w", encoding="utf-8") as fh:
+            json.dump({"unit": unit, "pages": pages}, fh, indent=2)
+            fh.write("\n")
+        return {"words": True, "wordsErrors": errors}
+    except Exception as exc:  # noqa: BLE001 - geometry never fails the conversion
+        errors["file"] = words_error(exc)
+        return {"words": False, "wordsErrors": errors}
+
+
 def mode_pdf_primary(o):
     import time
     start = time.monotonic()
@@ -278,11 +364,13 @@ def mode_pdf_primary(o):
     info = new_ocr(lang)
     status = ocr_status(True, lang) if o.get("ocr") else None
     ocr_ms, plain_ms = [], []
+    word_pages, words_errors = [], {}
     for i, n in enumerate(pages):
         stats.append(page_stats(doc, n))
         d = page_dir(staging, n)
         try:
             page = doc[n - 1]
+            rotation = page.rotation
             textless = not page.get_text("text").strip()
             if textless:
                 info["textless"].append(n)
@@ -326,6 +414,11 @@ def mode_pdf_primary(o):
             if page_pic:
                 md = "\n\n".join(x for x in [md.rstrip(), f"![page {n}](pages/{page_pic})"] if x)
             out.append(md.rstrip())
+            if o.get("words") and not (textless and o.get("ocr") and not kw["use_ocr"]):
+                try:
+                    word_pages.append(words_page(page, n, rotation, page_words(page, kw["use_ocr"])))
+                except Exception as exc:  # noqa: BLE001
+                    words_errors[str(n)] = words_error(exc)
         except Exception as exc:  # noqa: BLE001
             shutil.rmtree(d, ignore_errors=True)
             failed.append({"page": n, "error": f"{type(exc).__name__}: {exc}"[:300]})
@@ -339,8 +432,11 @@ def mode_pdf_primary(o):
     missing = len(pages) - len(page_images) if o.get("pageImages") else 0
     if missing:
         notes.append(f"Page images: {missing} of {len(pages)} unavailable")
-    return {"markdown": "\n\n".join(out) + "\n", "pages": pages, "pageCount": doc.page_count,
-            "emptyPages": empty, "failedPages": failed, "notes": notes, "ocr": info, "pageImages": page_images, "pageStats": stats}
+    result = {"markdown": "\n\n".join(out) + "\n", "pages": pages, "pageCount": doc.page_count,
+              "emptyPages": empty, "failedPages": failed, "notes": notes, "ocr": info, "pageImages": page_images, "pageStats": stats}
+    if o.get("words"):
+        result.update(write_words(staging, "pt", word_pages, words_errors))
+    return result
 
 
 def mode_pdf_fallback(o):
@@ -353,14 +449,17 @@ def mode_pdf_fallback(o):
     stats = []
     lang = o.get("ocrLanguage", "eng")
     ocr_info = new_ocr(lang)
+    word_pages, words_errors = [], {}
     status = {"status": "unavailable", "reason": "fallback tier", "tesseract": None} if o.get("ocr") else None
     for n in pages:
         stats.append(page_stats(doc, n))
         links = [f"![](images/{f})" for f in keep.get(n, [])]
         text = ""
         page_pic = None
+        page_ok = False
         try:
             page = doc[n - 1]
+            rotation = page.rotation
             text = page.get_text("text").strip()
             if not text:
                 ocr_info["textless"].append(n)
@@ -388,11 +487,17 @@ def mode_pdf_fallback(o):
                         links.append(f"![](p{n}/{name})")
                 mark_done(d)
             page_pic = render_page_image(page, n, o, page_images) if o.get("pageImages") else None
+            page_ok = True
         except Exception as exc:  # noqa: BLE001
             shutil.rmtree(os.path.join(staging, f"p{n}"), ignore_errors=True)
             text = ""
             links = [f"![](images/{f})" for f in keep.get(n, [])]
             failed.append({"page": n, "error": f"{type(exc).__name__}: {exc}"[:300]})
+        if o.get("words") and page_ok and not (not text and o.get("ocr")):
+            try:
+                word_pages.append(words_page(page, n, rotation, page_words(page, False)))
+            except Exception as exc:  # noqa: BLE001
+                words_errors[str(n)] = words_error(exc)
         if not text:
             empty.append(n)
         out.append("\n\n".join(x for x in [text, "\n".join(links), f"![page {n}](pages/{page_pic})" if page_pic else ""] if x))
@@ -405,8 +510,11 @@ def mode_pdf_fallback(o):
     missing = len(pages) - len(page_images) if o.get("pageImages") else 0
     if missing:
         notes.append(f"Page images: {missing} of {len(pages)} unavailable")
-    return {"markdown": "\n\n".join(out) + "\n", "pages": pages, "pageCount": doc.page_count,
-            "emptyPages": empty, "failedPages": failed, "notes": notes, "ocr": ocr_info, "pageImages": page_images, "pageStats": stats}
+    result = {"markdown": "\n\n".join(out) + "\n", "pages": pages, "pageCount": doc.page_count,
+              "emptyPages": empty, "failedPages": failed, "notes": notes, "ocr": ocr_info, "pageImages": page_images, "pageStats": stats}
+    if o.get("words"):
+        result.update(write_words(staging, "pt", word_pages, words_errors))
+    return result
 
 
 def mode_ocr_pages(o):
@@ -422,6 +530,8 @@ def mode_ocr_pages(o):
     os.makedirs(staging, exist_ok=True)
     active = os.path.join(staging, "active")
     out = {"status": "ran", "written": [], "noText": [], "ocrFailed": [], "ocrErrors": {}, "budgetStopped": []}
+    if o.get("words"):
+        out["wordsErrors"] = {}
     ocr_ms = []
     for i, n in enumerate(pages):
         with open(active, "w") as fh:
@@ -449,6 +559,20 @@ def mode_ocr_pages(o):
             header = f"<!-- OCR of page {n} (tesseract {lang}); recognized text, not the text layer -->"
             with open(sidecar, "w", encoding="utf-8") as fh:
                 fh.write(header + "\n\n" + (text + "\n\n" if text else "") + SEP.format(n=n).strip("\n") + "\n")
+            if o.get("words"):
+                wpath = os.path.join(d, f"{stem}-{tag}.words.json")
+                try:
+                    entry = words_page(page, n, page.rotation, ocr_words(page, tp))
+                    entry["unit"] = "pt"
+                    with open(wpath, "w", encoding="utf-8") as fh:
+                        json.dump(entry, fh, indent=2)
+                        fh.write("\n")
+                except Exception as exc:  # noqa: BLE001 - geometry never changes the OCR outcome
+                    try:
+                        os.remove(wpath)
+                    except OSError:
+                        pass
+                    out["wordsErrors"][str(n)] = words_error(exc)
             mark_done(d)
             (out["written"] if text else out["noText"]).append(n)
         except Exception as exc:  # noqa: BLE001 - one page never stops the pass
@@ -459,6 +583,10 @@ def mode_ocr_pages(o):
                 os.remove(sidecar)
             except OSError:
                 pass
+            try:
+                os.remove(os.path.join(d, f"{stem}-{tag}.words.json"))
+            except OSError:
+                pass
             out["ocrFailed"].append(n)
             out["ocrErrors"][str(n)] = msg
         ocr_ms.append((time.monotonic() - t0) * 1000)