pi-quiver 6.5.0 → 6.7.0

package/CHANGELOG.md

@@ -8,6 +8,26 @@ Published to npm as `pi-quiver` (`pi install npm:pi-quiver`). Pushing a
 via OIDC trusted publishing. The release helper at
 `.agents/skills/release/scripts/release.sh` cuts the tag; CI publishes.
 
+## v6.7.0 - 2026-09-30
+
+- README: `swordHeader` note on why pi's built-in logo flashes before the sword appears and how `quietStartup` removes it.
+- `doc_to_md` keeps embedded images on text-bearing PDF pages when OCR is enabled; only textless pages run OCR.
+- Changed: the OCR handle says `no text on pages <ranges>` rather than counting pages without recognized text.
+- `.xlsm` uses the Excel route (macros ignored) with an aggregated `preview truncated: ...` note; `.doc` uses LibreOffice -> PDF and reports `Degraded:`; `.msg`/`.eml` convert email bodies and stage attachments.
+- `pageImages` / `--page-images` writes selected page renders to `pages/` and adds a `Pages-Dir:` handle line; Word auto-numbering labels are computed on the mammoth path with a non-blocking `Numbering:` fallback.
+- Zero-byte input fails at the boundary with `empty file: <path>`. `--json` prints structured conversion or info data; `--pages ""` selects all pages.
+- Changed: same-stem collisions write `<stem>-2.md` instead of failing with `Output exists`, with a `renamed to <stem>-2 (<stem>.md exists)` note (or a held-lock note when only the lock exists).
+- Changed: the managed Python venv is `doc-to-md-venv-v4` with pinned `extract-msg==0.56.1`.
+- Changed: the Claude Code `skills/doc-to-md/SKILL.md` is generated from the option schema and pins `pi-quiver@<release version>` instead of `@latest`; the marketplace `version` follows the npm version. DOCX without explicit breaks reports `Page-Count: 1 (no explicit page breaks) - no page markers; cite by Outline line`.
+
+## v6.6.0 - 2026-09-29
+
+- `doc_to_md` converts local `.html`/`.htm` (markdownify; Readability-free Turndown fallback) with local and `data:` images copied into the bundle and remote images kept as links.
+- `doc_to_md` accepts `.png .jpg .jpeg .tif .tiff .bmp .gif`; the image is copied into the bundle.
+- Pages without a text layer now keep a page picture on both Python tiers (a scanned PDF used to convert to nothing).
+- Opt-in OCR: `ocr` (default `false`) and `ocrLanguage` (default `eng`) tunables, `--ocr`/`--no-ocr`; needs Tesseract language data (optional OS dependency). A small-image gate and a time budget keep OCR from eating the primary timeout; the handle's `OCR:` line reports what ran.
+- DOCX tables with `|` in a cell now render as `\|` instead of breaking the table; code blocks with a `language-*` class keep their fence language.
+
 ## v6.5.0 - 2026-09-29
 
 - `doc_to_md` converts DOCX directly in the Python child (mammoth -> markdownify, python-docx text fallback) instead of LibreOffice -> PDF: heading styles survive as `#` headings, hyperlinks and footnotes are kept, pictures land under `images/` (#24). DOCX page semantics change: only author-inserted page breaks become `--- end of page.page_number=N ---` markers, `Page-Count` is suffixed `(explicit page breaks, not printed pages)` / `(no explicit page breaks)`, `pages` selects those segments and is rejected on a break-less file, and DOCX that falls back to LibreOffice (no DOCX-capable Python, or both DOCX engines failed) is marked degraded with `(LibreOffice pagination)` and rejects `pages`. LibreOffice is now optional for DOCX and still required for PPTX. Backend probe gains a `DOCX` line and pins `mammoth==1.13.0`, `markdownify==1.2.3`, `python-docx==1.2.0`; the managed venv moves to `doc-to-md-venv-v3` and the older venvs are removed after the first successful build.

package/README.md

@@ -16,7 +16,7 @@ But the moment an agent does that, one `fetch` or PDF read can dump hundreds of
 
 ## Why pi-quiver exists
 
-`fetch` brings real web pages and GitHub issues/PRs into context and is size-gated by construction: over 32 KB or 1000 lines spills to a temp file with a preview and a grep/read hint. `doc_to_md` converts local PDF/DOCX/PPTX/XLSX/XLS files into a Markdown bundle on disk and returns only a concise handle. Ingestion is what makes data-driven work possible; bounded tool results keep it safe.
+`fetch` brings real web pages and GitHub issues/PRs into context and is size-gated by construction: over 32 KB or 1000 lines spills to a temp file with a preview and a grep/read hint. `doc_to_md` converts local PDF/DOCX/DOC/PPTX/XLSX/XLSM/XLS, MSG/EML, HTML, and image files into a Markdown bundle on disk and returns only a concise handle; OCR is opt-in and uses optional Tesseract language data. Ingestion is what makes data-driven work possible; bounded tool results keep it safe.
 
 `session-name`, `sword-header`, `fast-mode`, `provider-stall-watchdog`, and `slack` are opt-in ergonomics, recovery, and integration controls: session labeling, a themed startup header, Anthropic fast mode, semantic-stall recovery, and context-safe Slack search/threads/posting with repo-policy injection, `@name` mention resolution, cached emails, and per-call unfurl control.
 
@@ -65,7 +65,7 @@ A 300 KB changelog page never touches your context window - you get a preview an
 | Extension | Tool | What it does |
 | --- | --- | --- |
 | `extensions/fetch.ts` | `fetch` | Retrieve URLs over HTTP(S). HTML -> Markdown (Readability extraction, Turndown conversion). Binary saved untouched to a temp file. GitHub issue/PR/repo/actions-run/actions-job URLs auto-route through `gh` (falls back to HTTP); failed runs/jobs include failed-step logs (best-effort, summary-only otherwise). Same size gate as `fetch`. Behavior lives in `lib/fetch-core.ts`; also exposed as the `pi-quiver fetch` CLI (see [Claude Code support](#claude-code-support)). |
-| `extensions/doc_to_md.ts` | `doc_to_md` | Convert a local PDF/DOCX/PPTX/XLSX/XLS to a Markdown bundle on disk (`<stem>.md` + `images/` and spreadsheet `sheets/`) and return a handle (paths, page count, outline, diagnostics) - never inline Markdown. `info` mode inspects first; `pages` selects 1-based pages (DOCX: explicit-page-break segments); every selected PDF/PPTX page or DOCX segment when `pageCount > 1` ends with `--- end of page.page_number=N ---`. Tiers: pymupdf4llm -> PyMuPDF text (degraded) -> unpdf worker (no Python only). Excel -> sheet inventory, full CSVs, bounded previews, and optional rendered views. DOCX converts directly (mammoth, python-docx fallback); LibreOffice pagination marks the degraded route. The Outline lists `L<line>` and `p<page>` per heading. Settings under `quiver.docToMd`. |
+| `extensions/doc_to_md.ts` | `doc_to_md` | Convert a local PDF/DOCX/DOC/PPTX/XLSX/XLSM/XLS, MSG/EML, HTML, or image file to a Markdown bundle on disk (`<stem>.md` + `images/`, spreadsheet `sheets/`, optional `pageImages` in `pages/`, and email `attachments/`) and return a handle (paths, page count, outline, diagnostics) - never inline Markdown. `info` inspects PDF/Office inputs; `pages` selects 1-based PDF/Office pages (DOCX: explicit-page-break segments); HTML, image, and email inputs reject both; every selected PDF/PPTX page or DOCX segment when `pageCount > 1` ends with `--- end of page.page_number=N ---`. Tiers: pymupdf4llm -> PyMuPDF text (degraded) -> unpdf worker (no Python only). Excel -> sheet inventory, full CSVs, bounded previews, and optional rendered views. HTML keeps local and `data:` images in the bundle and remote images as links; image inputs keep the original image. Scanned PDF pages keep a page picture on both Python tiers; opt-in OCR needs Tesseract language data. DOCX converts directly (mammoth, python-docx fallback); LibreOffice pagination marks the degraded route. The Outline lists `L<line>` and `p<page>` per heading. Settings under `quiver.docToMd`. |
 | `extensions/session-name.ts` | `/session-name` | Manual + opt-in automatic session naming, naming rules and deny list, long-session revisits, and Ghostty/Herdr tab rename. OFF by default. |
 | `extensions/sword-header.ts` | `/builtin-header` | Themed ASCII startup header replacing pi's default logo. OFF by default. |
 | `extensions/fast-mode.ts` | `/fast` | Inject Anthropic fast-mode (`speed: "fast"` + `anthropic-beta: fast-mode-2026-02-01`) into every Claude Opus 4.8 / Opus 5 request, any thinking level. `--fast` flag + `/fast [on\|off\|status]`. OFF by default. |
@@ -93,7 +93,7 @@ Full routing rules, size-gate mechanics, and config: [doc/fetch.md](doc/fetch.md
 ## When NOT to use
 
 - You need a general-purpose web scraper (JS-rendered pages, pagination, auth flows) - `fetch` does plain HTTP + Readability extraction, nothing more.
-- You need in-grid chart/image placement, `.xls` visual detection, `.xlsm`, or range selection - out of scope; chart/image association is per sheet only.
+- You need in-grid chart/image placement, `.xls` visual detection or range selection - out of scope; chart/image association is per sheet only.
 - You want automatic session naming, a custom header, fast mode, or stall recovery without opting in - all stay off until you flip the config.
 - You need *mid-stream* stall recovery in JSON, RPC, or print runs - only the pre-first-event tier arms there; mid-stream silence falls through to pi's transport timeout.
 
@@ -133,7 +133,8 @@ The npm package's bundled JS deps install automatically on `pi install`. A few *
 | Prerequisite | Needed by | If absent |
 | --- | --- | --- |
 | `gh` (GitHub CLI, installed + `gh auth login`) | `fetch` GitHub issue/PR/repo/actions-run/actions-job routing | Falls back to an HTTP fetch of the rendered page (private repos hit a login wall). |
-| `uv` (+ managed Python 3.14, fetched on first use) | `doc_to_md` high-fidelity PDF and Excel conversion and DOCX (preferred route), with `pymupdf4llm`, `openpyxl`, `xlrd`, `pillow`, `mammoth`, `markdownify`, and `python-docx` | Falls back to a system Python >= 3.12 or one-time managed venv; PDF degrades to `unpdf` only when no capable Python exists. Excel requires the Python backend (no JS fallback). |
+| `uv` (+ managed Python 3.14, fetched on first use) | `doc_to_md` high-fidelity PDF and Excel conversion and DOCX (preferred route), with `pymupdf4llm`, `openpyxl`, `xlrd`, `pillow`, `mammoth`, `markdownify`, `python-docx`, and `extract-msg` | Falls back to a system Python >= 3.12 or one-time managed venv; PDF degrades to `unpdf` only when no capable Python exists. Excel requires the Python backend (no JS fallback). |
+| Tesseract language data (`tesseract` on `PATH` or `TESSDATA_PREFIX`) | `doc_to_md` OCR with `ocr: true` for scanned pages and images | OCR is skipped and the handle reports `OCR: unavailable` (or an install hint when OCR is off); pages keep their picture. |
 | LibreOffice (`soffice` on `PATH`) | `doc_to_md` PPTX conversion, the DOCX fallback route, and Excel rendered views | PPTX errors with a remedy; DOCX converts directly via the Python backend (LibreOffice fills in when that backend lacks the DOCX packages or its DOCX child exits 1); Excel omits rendered views. |
 
 None is a hard install-time dependency of the package; they are tools you provide in the environment where pi runs.
@@ -202,6 +203,8 @@ survive untouched.
 
 `herdrTab` (default `true`) mirrors the same curated label to the Herdr tab bar over Herdr's unix socket, independent of `ghosttyTab` - either sink can be toggled off without affecting the other. A tab is claimable while its label is a bare number (`1`, `4`, ...) at any position - Herdr's vocabulary for "unnamed" - so reordering tabs before or after naming never matters; a tab you deliberately name `3` is taken over too. A non-numeric label is treated as a human's and is never overwritten; the check repeats every turn, so renumbering the tab by hand hands it back. Exactly one leading `* ` (herdr-ntfy-notify's armed marker) is not a human rename - it is preserved across renames and the shutdown restore, and removing it keeps the claim. Every session claims afresh: on each shutdown (quit, reload, `/new`, resume, fork) a claimed tab is restored to its then-current position number, and the successor session in the pane claims it again on its first name. The label the extension writes is never digits-only - a bare `1234` becomes `#1234`, and the naming prompt asks the model for `PR 1234` / `issue 123` / `ticket ABC-123` instead of a bare ID. It only ever runs in TUI mode, on an attached TTY, under Herdr (`HERDR_ENV`/`HERDR_TAB_ID`/`HERDR_SOCKET_PATH` all set) - `pi -p`/json/rpc runs and background subagents never touch the tab. Caveat: the restored number is internally a custom name (Herdr has no clear-to-auto API), so that tab keeps a number-looking label but stops renumbering on later tab closes/reorders.
 
+`swordHeader` swaps the header at `session_start`, which runs after pi has already drawn its built-in pixel logo, so the logo shows for a moment before the sword replaces it. To skip that flash set pi's own `"quietStartup": true` in `~/.pi/agent/settings.json`: pi then draws an empty header slot at startup and the sword fills it. Quiet startup also drops pi's key-hint lines, which the sword header does not reproduce.
+
 `fastMode` only affects `claude-opus-4-8` and `claude-opus-5` requests on Anthropic's `anthropic-messages` API; enabling it opts into premium fast-mode pricing. `--fast` forces it on for one launch; `/fast on|off` toggles live. Proxy providers (opencode, cloudflare-ai-gateway) are excluded. `fastMode`'s header injection needs the `before_provider_headers` hook (pi bundling `@earendil-works/pi-coding-agent` >= 0.80.5); on older pi the beta header is silently not sent. The `anthropic-beta` header is discovered at request time by probing pi's own request assembly (no network), so conditional betas pi adds - e.g. `server-side-fallback-2026-07-01` for models with server-side fallback - are preserved alongside `fast-mode-2026-02-01`. If the probe fails, the header falls back to the static OAuth-identity + fast-mode list. See [doc/fetch.md](doc/fetch.md) and [doc/doc-to-md.md](doc/doc-to-md.md) for the ingestion tools' full reference; session-name/sword-header behavior above is complete.
 
 `pi-ai` prices every fast request at standard rates - it has no `usage.speed` support and no request-level pricing modifier - so `fastMode` corrects the reported cost itself: a `message_end` handler scales all four `usage.cost` components by `FAST_MODE_COST_MULTIPLIER` (2x) and returns the corrected message. Persisted session JSONL and pi's own native cost display are always exact, since they're written from this corrected message. pi-cohort's live `Σ$` reflects the correction only when pi-quiver's `message_end` handler runs before pi-cohort's - best-effort, depending on extension load order - and is reconciled on pi-cohort's next `session_start` regardless. The upstream fix (teaching `pi-ai`'s `Usage`/`calculateCost` about `usage.speed`) is the better long-term path and is tracked separately.
@@ -304,6 +307,8 @@ Each setting can also be overridden per-process via `PI_QUIVER_SLACK_ENABLED`, `
 | `imageFormat` | `png` | Rendered image format: `png` or `jpg`; embedded images retain their extension. |
 | `maxOutputBytes` | `20000000` | Child stdout cap in bytes. |
 | `outlineMaxEntries` | `40` | Heading outline, TOC, or sheet inventory entries in the handle. |
+| `ocr` | `false` | Opt-in OCR for scanned pages and images when Tesseract language data is installed. |
+| `ocrLanguage` | `eng` | Plain `+`-joined Tesseract language codes for OCR. |
 
 A bundle is `<outputDir>/<stem>.md` plus `<outputDir>/images/` and, for spreadsheets with data, `<outputDir>/sheets/`; without `outputDir`, the tool creates a per-call temp root. A conversion owns `<stem>.md.lock` until it atomically publishes the Markdown. An existing `<stem>.md` fails the call unless `overwrite` is set; `overwrite` replaces that Markdown and the files it owns (`images/<stem>-p<N>-<n>.*`, `images/<stem>-s<idx>[-<n>].*`, `sheets/<stem>-s<idx>-<slug>.csv`), nothing else. Temp bundles are caller-owned - the tool never deletes a bundle it produced.
 
@@ -338,7 +343,7 @@ to, and a flat `slack` block is ignored without a warning.
 
 `fetch` and `doc_to_md` cores are also published through the CLI, so Claude Code can use the same routing or bundle-and-handle behavior as pi's native tools - without pi ever seeing Claude-only files.
 
-**Exposed:** the `quiver` plugin, served from this repo's `.claude-plugin/marketplace.json`, with two skills: `fetch` (invoked as `quiver:fetch` / `/quiver:fetch`) and `doc-to-md` (invoked as `quiver:doc-to-md` / `/quiver:doc-to-md`). The `fetch` skill runs `npx -y pi-quiver@latest fetch <url> [flags]` via Bash - full parameter parity with the pi tool (`--method`, `--header`, `--body`, `--raw`, `--timeout-ms`), same GitHub `gh` routing (including failed-step logs on failed runs/jobs), same size gate, same binary-to-temp-file handling. See [doc/fetch.md](doc/fetch.md#claude-code-cli-pi-quiver-fetch) for exit codes and flags. The `doc-to-md` skill runs `npx -y pi-quiver@latest doc-to-md [flags] <path>` with full flag parity and the same handle output. See [doc/doc-to-md.md](doc/doc-to-md.md#cli-pi-quiver-doc-to-md) for exit codes and flags.
+**Exposed:** the `quiver` plugin, served from this repo's `.claude-plugin/marketplace.json`, with two skills: `fetch` (invoked as `quiver:fetch` / `/quiver:fetch`) and `doc-to-md` (invoked as `quiver:doc-to-md` / `/quiver:doc-to-md`). The `fetch` skill runs `npx -y pi-quiver@latest fetch <url> [flags]` via Bash - full parameter parity with the pi tool (`--method`, `--header`, `--body`, `--raw`, `--timeout-ms`), same GitHub `gh` routing (including failed-step logs on failed runs/jobs), same size gate, same binary-to-temp-file handling. See [doc/fetch.md](doc/fetch.md#claude-code-cli-pi-quiver-fetch) for exit codes and flags. The `doc-to-md` skill is generated at release from its hand-written intro and option schema; it runs `npx -y pi-quiver@<release version> doc-to-md [flags] <path>` with full flag parity and the same handle output. See [doc/doc-to-md.md](doc/doc-to-md.md#cli-pi-quiver-doc-to-md) for exit codes and flags, and [install channels](doc/doc-to-md.md#install-channels) for npm versus marketplace updates.
 
 **Not exposed:** the other pi extensions in this package (`session-name`, `sword-header`, `fast-mode`, `provider-stall-watchdog`, `slack`) - the marketplace allowlists only `./skills/fetch` and `./skills/doc-to-md`, and the npm tarball never ships `skills/` or `.claude-plugin/` (pi's own `files` allowlist excludes them, and pi's explicit `pi.extensions` manifest makes them invisible to pi's convention-directory auto-discovery either way).
 
@@ -355,7 +360,7 @@ Add the marketplace and enable the plugin in `.claude/settings.json`:
 
 Activates on folder trust.
 
-**Release sequencing:** the skill goes live only with (or after) the npm release that ships the `pi-quiver` bin - until that tag is on npm, `npx -y pi-quiver@latest fetch` resolves a bin-less package and fails.
+The generated doc-to-md skill uses the package version at release; the marketplace entry uses that same version so plugin updates track npm releases.
 
 ## Development