ask-marcel-office-cli 1.4.0 → 1.5.0

package/CHANGELOG.md

@@ -2,6 +2,71 @@
 
 All notable changes to `ask-marcel-office-cli` are documented here.
 
+## 1.5.0
+
+Agent-ergonomics, a faster cold-start, a new command, and an LLM-safety pass.
+Mostly additive; the one rename keeps its old name as an alias. **One behaviour
+change to note**: byte commands now refuse to inline a payload over ~1 MB without
+`--output-path` (see _Changed_).
+
+### Added
+
+- **`read-mail-attachment`** — a polymorphic "read any attachment" command that
+  auto-routes by content-type: a `.zip` is unpacked and every entry converted
+  (the `{ count, files }` envelope), everything else (docx/xlsx/pptx/odf/csv/pdf/
+  `.msg`/legacy/text, reference + embedded items) runs through the markdown
+  dispatch. Images / scanned PDFs / legacy `.ppt` return the actionable
+  vision-model hint. Use the explicit `convert-mail-attachment-*` siblings only
+  to force a specific output format. (Surface: 176 → 177 commands.)
+- **`pageCount`** on the born-digital PDF text envelope (every PDF→markdown entry
+  point) so an LLM can chunk its reads without a second parse.
+- **`--id`** is now accepted by *every* command with a single required `*-id`
+  flag (was mail-message-only) — e.g. `get-calendar-event --id`, `get-team --id`.
+- **`--start` / `--end`** aliases for `--start-date-time` / `--end-date-time`
+  across the calendar-view family.
+- **Token-tier flags in the manifest** — `needsElevatedToken` (now serialized to
+  `commands.json` too, not only `help-json`) and a new `needsSubstrateToken` mark
+  the Teams elevated / chat-substrate commands, so an agent can warm up an
+  interactive login before calling them instead of dead-ending on a timeout.
+
+### Changed
+
+- **`download-onedrive-file-content` → `download-drive-item-content`** (it works
+  on any driveItem, not just OneDrive). The old name keeps working as a
+  back-compat alias.
+- **Friendlier error for a mistyped flag**: a bare word like `item--id` (instead
+  of `--item-id`) now explains flags need their leading `--`, rather than
+  commander's opaque "too many arguments".
+- **Byte commands refuse to inline a multi-MB payload** without `--output-path`:
+  `get-mail-attachment`, `get-mail-message-mime`, the `download-*` family, etc.
+  now return an actionable `inline_too_large` error above ~1 MB instead of
+  flooding an LLM's context with a base64 blob. Small payloads still inline.
+- **`read-mail-attachment` prefers content-type when the filename misleads** — a
+  real spreadsheet saved as `report.jpg` now converts as a table instead of
+  returning an image hint. The explicit `convert-*` siblings stay
+  extension-deterministic by design.
+- **Unified `--drive-id` guidance** — the 11 terse drive-item commands now point
+  at `list-drives` / `list-sharepoint-site-drives` like the rest.
+
+### Fixed
+
+- **Auth no longer pops a browser per command on re-auth.** A command whose token
+  needed refreshing used to launch the Chrome extension-capture window — once per
+  process, with no cross-process throttle, so a batch/agent run stacked up
+  windows. Command re-auth now uses a headed Edge sign-in (one login re-caches all
+  tokens); the extension capture is reserved for explicit `login --use-extension`.
+- **Stale doc numbers** — the `help-json` size hints (terse-category ~16 → ~6 KB
+  after the trim) and the README command list now match reality.
+
+### Performance
+
+- **Cold-start**: the heavy conversion deps are `--external` to the npm bundle
+  (`dist/cli.js` 7.4 MB → 1.2 MB; `node --version` ~1.0 s → 0.58 s). The compiled
+  standalone binaries (`build:bin`) stay self-contained.
+- **`help-json --terse`** summaries are compacted to their first sentence, so a
+  per-category discovery fetch drops well under budget (drive 17.9 KB → 6.5 KB,
+  mail 16.8 KB → 5.2 KB; full terse 83 KB → 31 KB).
+
 ## 1.0.0
 
 The first stable release. Two breaking changes consolidate the public output

package/README.md

@@ -1,6 +1,6 @@
 # ask-marcel-office-cli
 
-**A Microsoft Graph CLI built for LLMs.** 165 read-only commands across Mail, Calendar, OneDrive, SharePoint, Excel, Teams chats, Planner / To-Do, OneNote, and directory. Sign in once with your Microsoft 365 account — no Azure app registration, no admin consent, no client secrets.
+**A Microsoft Graph CLI built for LLMs.** 177 commands across Mail, Calendar, OneDrive, SharePoint, Excel, Teams chats, Planner / To-Do, OneNote, and directory — plus local-file tools (markdown conversion, image extraction) that need no sign-in at all. Sign in once with your Microsoft 365 account — no Azure app registration, no admin consent, no client secrets.
 
 ```bash
 npm i -g ask-marcel-office-cli
@@ -25,7 +25,7 @@ LLM tool-loops keep hitting the same three walls with Microsoft Graph:
 
 ### Read-only by design
 
-**This is the most important property.** 164 GET endpoints + 1 POST (search). No `send-mail`, no `create-event`, no `upload-file`, no `delete-anything`. A hallucinated command can't break anything — the worst case is a 404. Safe default for autonomous agents, MCP servers, and "let Claude poke around my mailbox" sessions where you can't fully review every tool call.
+**This is the most important property.** 173 GET endpoints + 2 POST (searches) + 1 POST (create draft) + 1 PATCH (update draft) = 177 commands. No `send-mail`, no `create-event`, no `upload-file`, no `delete-anything`. The only write operations are draft creation and update — a hallucinated command can at most create an unsent draft in your Drafts folder. Safe default for autonomous agents, MCP servers, and "let Claude poke around my mailbox" sessions where you can't fully review every tool call.
 
 ### One call gets the full email context
 
@@ -34,6 +34,7 @@ A typical "read this email" loop in raw Graph: GET the message → GET the attac
 `convert-mail-to-markdown` collapses that into one call:
 
 - Body rendered as markdown (turndown pipeline)
+- Quoted reply chains / forwarded-message blocks stripped by default so a long thread doesn't duplicate earlier messages into the model's context (the cut is replaced with a visible marker; opt out with `--keep-quoted true` to keep the full body)
 - Inline images embedded as base64 `data:` URIs (size-capped per image — opt out with `--inline-images false` to keep raw `cid:` refs)
 - File attachments listed below the body with id + name + size, ready for follow-up calls
 - Pair with `extract-sharepoint-links-in-mail` to resolve every SharePoint URL in the body to its driveItem in parallel (capped at 25 unique URLs per call)
@@ -42,19 +43,49 @@ A typical "read this email" loop in raw Graph: GET the message → GET the attac
 
 Feed any Office-shaped file (docx, xlsx, pptx, csv, rtf, odt, …) into the local conversion pipeline OR through Graph's `?format=pdf` when slide layout and images matter:
 
-- `download-drive-item-as-markdown` — docx via mammoth (with inline images as data URIs), xlsx as one markdown table per sheet, csv as a table, plain-text passthrough
+- `download-drive-item-as-markdown` — docx via mammoth (embedded images become `[image]` placeholders by default — `--inline-images true` to embed them as base64, or pull the full-resolution originals with `extract-drive-item-images`), xlsx as one markdown table per sheet (a sheet whose used range exceeds the `--max-cells` cap, default 50 000, becomes a band-by-band read hint instead of a multi-hundred-MB table that would OOM), csv as a table, odt/ods/odp via content.xml (headings, lists, tables, named sheets, per-slide text, with `office:annotation` comments folded inline), **pptx** flattened to per-slide text (titles + bullets + text boxes + table cells, speaker notes inline, as `## Slide N` sections — `download-drive-item-as-pdf` + a vision model when layout / images matter), **pdf** via text-layer extraction ([unpdf](https://github.com/unjs/unpdf) → `text/plain`; a scanned / image-only PDF with no text layer points you at `download-drive-item-as-pdf` + a vision model), **legacy OLE Office** (`.xls` read by sheetjs like `.xlsx`; `.doc` extracted by [word-extractor](https://www.npmjs.com/package/word-extractor) as plain text; `.ppt` has no pure-JS path → convert to PDF first), plain-text passthrough
 - `download-drive-item-as-pdf` — Graph PDF conversion for anything it supports (preserves slide layout, images, charts — the right call for pptx and image-heavy docs)
 - `convert-mail-attachment-to-markdown` / `convert-mail-attachment-to-pdf` — same pipelines but starting from an email attachment
+- `read-mail-attachment` — one call that reads any mail attachment, auto-routing by file type (a `.zip` is unpacked and every entry converted; docx/xlsx/pptx/odf/csv/PDF/`.msg`/legacy/text → markdown; images, scanned PDFs, and legacy `.ppt` return an actionable 415 pointing at the raw-bytes / vision route) so an agent never has to choose between the `convert-mail-attachment-*` siblings
+- `convert-local-file` — same pipelines but starting from a file **on disk** (`--path ./report.docx`); never calls Graph (works offline, no login). A `.zip` is unpacked with every contained file converted in one call. The two things it can't do locally — convert **to** PDF and Loop/Fluid/Whiteboard sources — need Graph's server-side renderer (upload to OneDrive and use the drive-item siblings)
+- `convert-drive-item-zip` / `convert-mail-attachment-zip` — unzip an archive (OneDrive/SharePoint item, or an Outlook attachment) and convert **every** contained file in one call; legacy GBK / CP437 entry names (Chinese vendor archives from WinRAR / Windows Explorer) are decoded correctly, never mojibaked; unsupported entries are listed with a note instead of failing the archive
+- **Outlook `.msg` files** (saved/forwarded emails) convert to markdown through every entry point above — H1 subject, From/To/Cc/Date header block, the body, and an `## Attachments` section where each attachment is itself converted recursively (depth-capped)
+- `extract-sharepoint-links-in-documents` — the doc-side sibling of `extract-sharepoint-links-in-mail`: resolve every `*.sharepoint.com` URL embedded in a docx/xlsx/pptx (read from the package's relationship parts) or an odt/ods/odp (read from the inline `xlink:href` links in content.xml) to its driveItem, so an agent can follow references out of a document the same way it follows them out of an email
+
+Pass `--include-metadata true` on any `*-as-markdown` (or `convert-mail-attachment-to-markdown`) command to surface the side-channel content the rendered body hides. For **docx** (`## DOCX metadata`): core/app/custom doc properties, people registry, external hyperlinks, comments (each quoting the document text span it annotates), tracked changes, hidden text (`w:vanish`), MERGEFIELD / HYPERLINK / DOCVARIABLE instructions, bookmarks. For **xlsx** (`## Workbook metadata`): properties, external relationships, defined names, hidden / very-hidden sheets, legacy + threaded cell comments (each tagged with its cell), the persons registry. For **pptx** (`## PPTX metadata`): properties, external relationships, slide tags, comment authors + comments (legacy + modern, each anchored to its slide), and per-slide title / speaker notes / hidden flag — appended after the per-slide text body (use `download-drive-item-as-pdf` + a vision model for slide visuals / layout). Each family also covers its macro-enabled (`.docm` / `.xlsm` / `.pptm`) and template (`.dotx` / `.xltx` / `.potx`, etc.) variants, and surfaces a `### Macros (VBA)` section flagging an embedded `vbaProject.bin` (the file can execute code on open). For **OpenDocument** (`.odt` / `.ods` / `.odp`) the flag appends a `## OpenDocument metadata` block (Dublin Core + ODF properties, keywords, user-defined custom fields) after the converted body. No-op on other sources.
+
+### Extract embedded images from documents
+
+`extract-drive-item-images` (OneDrive / SharePoint), `extract-mail-attachment-images` (Outlook attachments), and `extract-local-file-images` (a file **on disk** — no Graph, no login) pull the embedded images out of a **docx, xlsx, pptx, or pdf**. For Office files it reads the OOXML media parts (png/jpg/gif/bmp/tiff/webp/svg) — including original full-resolution / un-cropped originals and images on hidden slides that the rendered view never shows. SVG rides back as its XML source (which carries the diagram's own text labels); legacy vector (emf/wmf) and audio/video are skipped. For a PDF it walks every page via [unpdf](https://github.com/unjs/unpdf) (a pure-JS, no-native-deps pdf.js build) and re-encodes each painted image as PNG — page-oriented, so it captures images as drawn on each page (it does not reach layer-hidden/unpainted XObjects or the full uncropped original behind a clipped image). Pair with the global `--output-dir <dir>` to write every image to a folder (the directory is auto-created and each `base64` becomes a `savedTo` path); without it the bytes ride back base64-encoded so a vision model can read them directly.
 
 The CLI follows any SharePoint media-transform redirect internally, so the LLM never has to fetch an external URL.
 
+### Find every drive you can reach
+
+`list-drives` only returns your personal OneDrive(s). `list-accessible-drives` unions every discovery vector the delegated token can hit — `/me/drives` (personal), `/me/joinedTeams` (Teams libraries), `/me/memberOf` Unified groups → each group's drive (SharePoint M365-group sites), `/me/drive/sharedWithMe` (drives behind files shared with you), per-team `/teams/{id}/channels` → `filesFolder` for **private/shared channels** (which live in their own sites, not the team default drive), activity signals (`/me/drive/recent`, `/me/drive/following`, `/me/insights/{trending,used,shared}`), and every **non-default document library** of each discovered site via a path-addressed `/sites/{host}:/sites/{name}:/drives` (catches secondary libraries the default-drive vectors skip) — deduped by drive id and tagged with the `sources[]` that surfaced each one (`channel` = private/shared channel drive, `activity` = a recently-used/followed/trending item drive, `siteLibrary` = a non-default site library). These vectors catch OneDrives, channel sites, and direct-link sites the tenant search index (`search-sharepoint-sites-by-name`) never returns; the index in turn returns sites you can open but aren't a member of. For that index half, `search-all-accessible-sites` deep-pages the Microsoft Search API (`POST /search/query`, `entityTypes: ['site']`) past the single-page cap of `search-sharepoint-sites-by-name`, returning the *full* security-trimmed site index (on one tenant: ~154 sites vs 80). So **the union of `search-all-accessible-sites` + `list-accessible-drives` is the practical maximum on a delegated token** (truly enumerating *every* site in the tenant needs tenant-admin app-only `/sites/getAllSites`). Both site-search commands **exclude archived sites**: each result is probed (`GET /sites/{id}?$select=…,siteCollection`) and dropped when Graph reports it archived or fails with `423 resourceLocked` — the signal a departed/unlicensed user's auto-archived OneDrive returns (no more `sharepointerror.aspx?scenario=SiteArchived` dead links in the output); the count surfaces as `archivedExcluded`. `--max-groups` caps every fan-out, and `partialErrors[]` stays signal-only: benign "can't reach this one" results (404 no-drive, 403 access-denied / non-member channel, 423 admin-locked site, 400 stale id) are dropped silently — only actionable failures (auth, throttling, 5xx, network) are listed. Both commands also surface a best-effort `fileEstimate` — the Microsoft Search index's security-trimmed `driveItem` count, i.e. roughly how many files you can access across all of SharePoint/OneDrive (index-wide, not limited to the listed drives).
+
 ### Browser-OAuth at first launch
 
 No Azure app, no tenant admin. The CLI captures the same token the Teams web client uses — works for any Microsoft 365 account, personal or enterprise.
 
+**Login flow:** By default, the CLI uses Playwright to open a browser for authentication. Alternatively, you can use the [Ask Marcel Companion](./browser-extension/) browser extension for faster authentication.
+
+```bash
+ask-marcel login              # default: Playwright browser
+ask-marcel login --use-extension  # use browser extension (requires setup)
+```
+
+**Browser extension setup (optional, one-time):**
+
+1. Open `chrome://extensions/` (Chrome) or `edge://extensions/` (Edge)
+2. Enable "Developer mode" (top-right toggle)
+3. Click "Load unpacked" → select the `browser-extension/` folder from this repo
+4. **Enable in incognito/inprivate mode:** Click the extension's "Details" button, then enable "Allow in incognito" (Chrome) or "Allow in InPrivate" (Edge). **This is required** — the CLI opens an incognito/inprivate window, and extensions are disabled there by default.
+5. Done — use `ask-marcel login --use-extension` to authenticate via the extension
+
 ### Stable error envelope with actionable hints
 
-Every failure — Graph, CLI parser, Zod validation, substrate — comes back as `{ok: false, error, errorCode?, hint?, source}`. The `hint` field tells the model *what to do next* (e.g. "string literals MUST use single quotes; embed one by doubling it") and `source` tells it where the failure came from. Curated rules for 20+ recurring Graph errors plus cross-resolver pointers (passed a Teams URL to `resolve-mail-link`? Hint says "re-run with `resolve-teams-link`").
+Every failure — Graph, CLI parser, Zod validation, substrate — comes back as `{ok: false, error, errorCode?, hint?, source, retryAfterSeconds?}`. The `hint` field tells the model *what to do next* (e.g. "string literals MUST use single quotes; embed one by doubling it") and `source` tells it where the failure came from. Curated rules for 20+ recurring Graph errors plus cross-resolver pointers (passed a Teams URL to `resolve-mail-link`? Hint says "re-run with `resolve-teams-link`"). When Graph throttles (HTTP 429, sometimes 503) the response's `Retry-After` is surfaced as `retryAfterSeconds` — the integer seconds to wait — so a caller running tenant-scale crawls can honor the server's backoff instead of guessing.
 
 ### Lean responses
 
@@ -66,7 +97,7 @@ A typical conversion command returns multi-MB PDF bytes. Sending 5 MB of base64
 
 Two flag patterns avoid the round-trip:
 
-- **`--output-path /path/to/file.pdf`** — the CLI decodes the bytes, writes them to disk, and replaces `base64: "..."` in the envelope with `savedTo: "/path/to/file.pdf"`. The LLM sees a 3-line confirmation instead of a 7-million-character payload. Works on every command that returns binary or text content; rejected with a clear error on plain-JSON commands so a misapplied flag is never silent.
+- **`--output-path /path/to/file.pdf`** — the CLI decodes the bytes, writes them to disk, and replaces `base64: "..."` in the envelope with `savedTo: "/path/to/file.pdf"`. The LLM sees a 3-line confirmation instead of a 7-million-character payload. Works on every command that returns binary or text content; rejected with a clear error on plain-JSON commands so a misapplied flag is never silent. A binary payload over ~1 MB is **refused** without this flag (an `inline_too_large` error pointing you here), so a multi-MB base64 string can never flood the context by accident.
 - **No flag, text mode** — binary commands print a one-line summary (`binary: application/pdf, 4837291 bytes — use --output-path to save`) instead of spilling base64 to stdout. The LLM sees a hint without ever pulling the bytes.
 
 ### Relative dates on calendar windows
@@ -82,7 +113,7 @@ npm i -g ask-marcel-office-cli
 # authenticate (cached → refresh → browser fallback)
 ask-marcel login
 
-# everything else is read-only and discoverable from --help
+# the rest is read-only (the only writes are mail drafts) and discoverable from --help
 ask-marcel list-drives
 ask-marcel search-onedrive-files --drive-id "b!abc..." --query "Q3 budget"
 ask-marcel convert-mail-to-markdown --message-id "AAMkAD..."
@@ -154,29 +185,42 @@ The `AuthManager` interface is two async methods that return `Result<T, AuthErro
 
 ## Deep docs
 
-- **[All 165 commands](docs/COMMANDS.md)** — per-category tables with required params + Graph endpoint
+- **[All 177 commands](docs/COMMANDS.md)** — per-category tables with required params + Graph endpoint
 - **[Usage guide](docs/USAGE.md)** — output formats, OData passthrough, `--output-path`, pagination, library API, architecture, configuration, quality gates
 - **[Machine-readable manifest](docs/commands.json)** — JSON for programmatic discovery (LLM tool-loops, IDE plugins, MCP servers); also importable via `import manifest from 'ask-marcel-office-cli/commands.json'`
+- **[QA playbook](docs/QA-PLAYBOOK.md)** — the repeatable full-surface health-check procedure (offline gates, parameter matrix, conversion contracts, live Graph drift probes) used to audit each release
+
+
+## Agent skill (progressive disclosure)
+
+A [Codex skill](https://docs.anthropic.com/en/docs/agents-and-tools/codex) lives at `.agents/skills/ask-marcel-office/` and teaches agents how to use the CLI without loading all 177 commands into context at once.
+
+**Structure**
+
+```
+.agents/skills/ask-marcel-office/
+├── SKILL.md                          # core workflow + category index
+└── references/                       # per-domain command details, loaded on demand
+    ├── marcel-mail.md        (32 commands)
+    ├── marcel-drive.md       (30 commands)
+    ├── marcel-calendar.md    (23 commands)
+    ├── marcel-sharepoint.md  (18 commands)
+    ├── marcel-user.md        (15 commands)
+    ├── marcel-tasks.md       (15 commands)
+    ├── marcel-excel.md       (11 commands)
+    ├── marcel-notes.md       (11 commands)
+    ├── marcel-chats.md        (9 commands)
+    ├── marcel-teams.md        (7 commands)
+    └── marcel-meta.md         (6 commands)
+```
+
+**How it works**
+
+`SKILL.md` contains the authentication flow, the discovery loop (`help-json --terse --category` → `docs <cmd>` → execute), key patterns (OData passthrough, relative dates, document conversion, pagination), and a category index. The full command reference for each domain lives in `references/marcel-<category>.md` and is loaded only when the agent needs that domain — keeping the context window lean.
 
 ## Roadmap
 
-Read-only stays the default forever. The list below is additive coverage and convenience — concrete next steps, ordered by how often they come up in real LLM workflows:
-
-- **Zip archive support** — extract a `.zip` from an email attachment or OneDrive item and run every contained file through the right conversion pipeline (markdown or PDF), so an agent reading "the project handover archive" doesn't have to shell out to `unzip`.
-- **SharePoint links inside documents** — today `extract-sharepoint-links-in-mail` finds and resolves every `sharepoint.com` URL in an email body; the same pass for inline links in docx/xlsx/pptx would close the loop on "follow every reference in this document".
-- **Bulk folder conversion** — `convert-folder --drive-id … --item-id <folder> --format markdown` walks a folder and converts every Office file in one call, returning the combined markdown (or writing per-file outputs to a directory via `--output-path`).
-- **OneNote end-to-end markdown** — OneNote page reads currently return raw HTML; chaining the turndown pipeline (same one `convert-mail-to-markdown` uses) would give parity with the docx route.
-- **PowerPoint as slide images** — for vision-capable LLMs, render each pptx slide as a PNG alongside the existing PDF route so the model can reason about layout + diagrams without going through PDF parsing.
-- **Calendar meeting attachments** — `/events/{id}/attachments` isn't exposed yet; would pair with the existing `convert-mail-attachment-to-*` shape so "summarize the deck attached to my 3pm" is a single call.
-- **Multi-tenant auth profiles** — `ask-marcel login --profile work` / `--profile personal` with separate token caches at `~/.ask-marcel/<profile>/`, for consultants and contractors who routinely switch tenants.
-- **Streaming pagination output** — write each page to stdout as it arrives instead of accumulating, so long delta walks don't hold gigabytes in memory and an agent can start processing page 1 while page 2 is in flight.
-- **Excel charts as PNG** — `list-excel-worksheet-charts` already returns chart metadata; add `get-excel-chart-image` that renders the chart as a base64 PNG via Graph's `.../charts/{id}/image()` endpoint so vision-capable models can read the chart, not just its title.
-- **Yammer / Viva Engage** — community posts, threads, and replies. Currently unscoped on the Teams web-client token, so this would need a scope investigation first.
-- **Federated tenant search expansion** — `microsoft-search-query` exists but covers a narrow entity set; expanding to people, sites, lists, and chat messages would unify discovery across all of M365 behind one search call.
-- **Quoted-text stripping in `convert-mail-to-markdown`** — collapse "On Tuesday Alice wrote..." reply chains so long threads stop blowing the context budget on duplicated quoted content.
-- **`--diff` flag on `download-drive-item-version` markdown** — render the markdown diff between two historical versions of the same file (e.g. "what changed in the spec last week"), so an agent can answer revision questions without manually comparing two long markdown blobs.
-- **Full-fidelity document context** — extract every piece of written information in a document: reviewer comments, threaded comments, slide notes, revision marks, descriptions, custom metadata, and sensitivity labels — folded into the markdown output so an LLM reads the whole story of a document, not just its body text.
-- **OCR for images and image-only PDFs** — pipe images (jpg/png attachments, inline email images) and scanned PDFs through a local OCR pass, returning extracted text as markdown. Closes the loop on screenshots, scanned receipts, and image-only attachments that today come back as opaque bytes.
+Read-only stays the default forever. There's no fixed feature backlog — coverage grows out of real LLM workflows as they come up.
 
 Suggestions, requests, and pull requests welcome — see the [issues page](https://github.com/vdelacou/ask-marcel-office-cli/issues).