kushi-agents 3.4.2 → 3.13.0

package/plugin/learnings/onenote.md

@@ -0,0 +1,215 @@
+# Learnings — OneNote (`pull-onenote`)
+
+Newest on top. Format defined in [`README.md`](./README.md).
+
+_(no entries yet — append the moment a fix lands during a `pull-onenote` run)_
+
+
+## 2026-05-14 — Pre-flight gate: distinguish notebook-unavailable from auth-required
+
+**Trigger:** AGCO refresh, immediately after v3.10.2 (URL synthesis killed). Even with a freshly user-pasted `one_sectionWebUrl` and a valid Edge profile, the runner returned `auth-required`. Manual sanity-check: the user opened `https://onenote.cloud.microsoft/` directly in their browser and got the **"Sorry, we ran into a problem"** dialog at the notebook-list level — before they could even click into the AGCO section. So the runner couldn't possibly succeed; the failure was OneNote-for-Web side, not auth.
+
+**Root cause classification was overloaded.** The runner only knew two end-states: "canvas frame attached → success" and "didn't attach → auth-required". But OneNote-for-Web has at least three failure surfaces:
+1. **Login redirect** (`login.microsoftonline.com`) — genuine auth-required.
+2. **Service/notebook error dialog** ("Sorry, we ran into a problem", "We couldn't open", "This notebook can't be opened", "There was a problem") — service- or notebook-side. Auth is fine.
+3. **Silent timeout** (no chrome, no error, no redirect) — usually network or extreme service degradation.
+
+Conflating #2 with #1 sends the operator down the wrong recovery path: re-bootstrap auth when the real fix is to recover the notebook (open in OneNote desktop, force sync, wait, or re-capture the section URL).
+
+**Fix (v3.11.0):**
+
+- New `preflightOneNoteWeb()` in `runner.mjs` runs BEFORE navigating to any section URL. Probes `https://onenote.cloud.microsoft/` and classifies the end-state into `ok` / `auth-required` / `onenote-web-unavailable`. The pre-flight is also exposed as a standalone CLI mode (`--preflight`) for gate drivers.
+- The canvas-attach wait loop now ALSO scans every frame's body for the same error-dialog patterns each tick, so a section-load error (e.g. moved section, sync drift) is caught ~500ms after the dialog appears, not after `TIMEOUT_MS`.
+- New `runStatus: "notebook-unavailable"` is emitted for #2 — distinct from `auth-required` — and the runner's error message includes a verbatim recovery checklist (hard-refresh; open in OneNote desktop; wait 10–15min; re-capture URL).
+
+**Doctrinal lesson codified into `pull-onenote/SKILL.md` Pre-flight A.4:** the three-way classification IS the contract; the runner MUST distinguish all three end-states; auto-retry is allowed for `auth-required` (next refresh) but FORBIDDEN for `notebook-unavailable` — that one needs human/notebook-side recovery first.
+
+**Validation marker:** when AGCO's notebook becomes available again in OneNote-for-Web, re-run kushi pull and confirm the new runStatus values round-trip through to refresh-reports. (Pending — notebook still showing the dialog as of write time.)
+
+
+## 2026-05-14 — Conditional Access requires Edge; cookie domains don't transfer; URL must be canonical
+
+**Trigger:** Pulling AGCO OneNote evidence for the first time in this session. Bootstrap appeared to succeed (no errors), but every subsequent headless run returned `auth-required`. Re-bootstrapping multiple times did not help.
+
+**Three independent root causes had stacked, each masking the next:**
+
+1. **Wrong URL formula (fixed in v3.10.0).** AGCO's pre-doctrine registry entry had a synthesized URL using `wd=target(/<name>/)` (no fileId, no pipe). The canonical formula is `wd=target(<sectionName>|<sectionFileId>/)`. Wrong formula → OneNote-for-Web silently shows "Sorry, we ran into a problem" — indistinguishable from auth failure to a headless runner. Fixed by `recapture-section-url.mjs` auto-heal from sibling registry entries (HCA shares the same notebook, so its `notebookSourceDoc` + `spoBaseUrl` were inherited).
+
+2. **Cookie-domain isolation (fixed in v3.10.1).** Bootstrap was navigating only to `https://onenote.cloud.microsoft/`. But the canonical Doc.aspx URLs live on `https://microsoft-my.sharepoint-df.com/personal/<upn>/...`. These are SEPARATE cookie domains — signing into one does NOT authenticate the other. After `--bootstrap` succeeded, headless runs against the SPO URL still hit `login.microsoftonline.com` because SPO had no cookies. Fixed by making `--bootstrap` walk: `onenote.cloud.microsoft` → `microsoft-my.sharepoint.com` → `microsoft-my.sharepoint-df.com` in the same session, so cookies plant on all three.
+
+3. **Conditional Access blocks vanilla Chromium (fixed in v3.10.1).** Even after fixing the cookie-domain issue, the SPO sign-in surface returned a hard block: **"You can't get there from here — this application contains sensitive information and can only be accessed from devices or client applications that meet Microsoft management compliance policy. Since you're using Chrome... Alternatively, you can use Microsoft Edge or Internet Explorer."** Playwright's bundled Chromium is not Intune-managed. Fixed by switching `runner.mjs` to `chromium.launchPersistentContext(profile, { channel: 'msedge', ... })` — Playwright drives the user's installed Edge, which IS Intune-trusted.
+
+**Doctrinal lessons codified into `pull-onenote/SKILL.md` Pre-flight A.1/A.2/A.3:**
+
+- **A.1** — `channel: 'msedge'` is HARD-required. Vanilla Chromium WILL be CA-blocked in Microsoft tenants. Do not fall back to Chromium.
+- **A.2** — Bootstrap MUST visit both `onenote.cloud.microsoft` AND both `sharepoint.com` + `sharepoint-df.com` to plant cookies in all required domains. Single-host bootstrap is broken by design.
+- **A.3** — When switching browser channel (Chromium ↔ Edge), DELETE the existing profile first; cookie/cache formats are not compatible.
+
+**The "auth-required" signal is overloaded.** It can mean: (a) genuine cookie expiry, (b) wrong URL formula returning OneNote's "Sorry" dialog, (c) cookies present for one domain but not the navigation target's domain, (d) Conditional Access blocking the browser channel. The runner cannot distinguish these — bootstrap-then-retest is the only protocol. v3.10.1 closes (b)/(c)/(d) as systemic failure modes; only (a) should remain after this release.
+
+**Validation:** AGCO bootstrap completed cleanly with Edge + dual-surface walk on 2026-05-14. Profile size shrank from 27MB (Chromium) to ~8MB (Edge, just cookies after fresh start). Sign-in walked through MFA on `onenote.cloud.microsoft` and silently SSO'd on the SPO hosts.
+
+
+## 2026-05-13 — WorkIQ has TWO answer modes; index field names are the contract
+
+**Trigger:** After shipping v3.7.7 (no-fabrication rule), user disputed the claim that OneNote bodies cannot be retrieved and pasted a historical `m365-mutable.json` shape that used `one_sectionFileId`, `one_sectionPath`, `one_sectionOneNoteGuid`, citing a prior project-evidence flow that successfully pulled OneNote pages.
+
+**Root cause:** WorkIQ has two answer paths:
+1. **Search-index extractor** — triggers when the query body contains the literal M365 search index field names (`wdsectionfileid`, `wdsectiongroupid`, `wdpartid`, `wdsectiononenoteguid`). Returns the indexed body fragment verbatim.
+2. **LLM-summary path** — triggers when the query uses prose like "the page titled X" or "page id Y" or "the OneNote section called Z". Synthesizes a summary from titles + adjacent evidence. This is what v2.2.0 and v2.3.0 hit.
+
+The first eight `pull-onenote` versions used prose phrasing (`"sectionFileId <id>"`, `"pageId <pageId>"`) — this is mechanically different from `wdsectionfileid = <id>` / `wdpartid = <id>`. The previous project-evidence flow used the index field names verbatim, which is why it worked.
+
+**Fix shipped (v2.4.0 of pull-onenote):**
+1. New "Canonical lookup keys" subsection in Step A enumerates the index field name → m365-mutable.json key mapping (`wdsectionfileid` ↔ `one_sectionFileId`, etc.) with explicit "DO NOT GUESS, DO NOT PARAPHRASE" directive.
+2. Step A.1 (by `wdsectionfileid`) and A.2 (by `wdsectiongroupid`) WorkIQ queries rewritten to use the index field names verbatim with `=` syntax.
+3. Step B verbatim probe rewritten: `wdpartid = <wdpartid> AND wdsectionfileid = <sectionFileId>` lookup, asks for the **indexed page body content** (not "the body of the page").
+4. Step B explanatory paragraph added: "WorkIQ has two answer modes... the field names are the contract."
+5. Step B retry phrasing now explicitly tells WorkIQ "Your previous response was generated by the LLM-summary path, not the search-index extractor. Re-run the query against the search index using wdpartid = <wdpartid> AND wdsectionfileid = <sectionFileId>."
+6. Step C stream pass also rewritten to use `wdsectionfileid` / `wdsectiongroupid` / `lastModifiedDateTime` field names.
+7. Failure-handling note for Graph `/me/onenote/*` updated with empirical 401 evidence (az CLI app id `04b07795-…` lacks Notes.Read; tenant denies admin consent).
+
+**Doctrinal lesson:** when an enterprise tool exposes index-aware query semantics, **document the literal field names in the skill** instead of paraphrasing. Paraphrasing routes to the wrong answer path. This applies to all pull-* skills, not just OneNote — verify the same pattern for SharePoint (`SiteId`, `WebId`, `ListItemId`, `Path`), Teams (`channelIdentity`, `chatId`, `messageId`), Email (`internetMessageId`, `conversationId`).
+
+
+## 2026-05-13 — WorkIQ summarization masquerades as captured bodies; never fabricate narrative
+
+**Trigger:** User said "onenote is still sparse, did you rerun or tighten" after the HCA refresh shipped 5 OneNote page snapshots that were each ~1.5K bytes. Inspection showed every page file had: header + `❌ Partial via WorkIQ — body not extractable` marker + a 3-paragraph **AI Narrative Summary inferred from adjacent emails and chat traffic** ("plausibly the engagement-level rollup page", "if Usha's backfill includes verbatim chat-summary…"). The narrative was speculation, not capture.
+
+**Root cause (skill-level):**
+- `pull-onenote/SKILL.md` v2.2.0 allowed snapshot files with a `page-body-unavailable` marker AND an AI Narrative Summary in the same file. The depth-bar said "AI Narrative Summary REQUIRED FIRST" without an exception for the unavailable case, so the producer satisfied the contract by inferring narrative from adjacent evidence.
+- Graph `/me/onenote/*` is not a viable fallback in this tenant — Notes.Read scope requires admin consent that is denied. WorkIQ is the only path, and WorkIQ summarizes by default.
+
+**Fix shipped (v2.3.0 of pull-onenote, plus verbatim-by-default v3.7.6.1):**
+1. `pull-onenote` Step B now uses a strict verbatim-or-marker probe (the WorkIQ prompt forces one of two outcomes only — verbatim body OR the literal `page-body-unavailable: <reason>` marker, no third option).
+2. Verbatim acceptance check (HARD): rejects responses containing `"plausibly"`, `"likely"`, `"appears to"`, `"inferred from"`, `"based on adjacent evidence"`, `"this page is about"`, `"key topics include"`.
+3. No-fabrication rule (HARD): if the body is unavailable, the snapshot file MUST contain ONLY the header + the marker + a `next_step` asking the user to paste. AI Narrative Summary is forbidden in this case. Empty is correct.
+4. `items_verbatim` added to run-log alongside `items_pulled` and `items_enumerated`. Run is classified `partial-bodies` when the verbatim ratio is < 0.5 — stops "5 page files written" from masquerading as "5 pages captured" in the per-user refresh report.
+5. `verbatim-by-default.instructions.md` adds anti-pattern #8: "Inferred narrative as substitute for body".
+
+**Recovery action for HCA (this turn):** the 5 existing HCA OneNote page files at `HCA\Evidence\ushak\onenote\snapshot\pages\` will be rewritten to the v2.3.0 shape (header + unavailable marker + paste-ask, no inferred narrative) and the user will be asked once at the end of the next refresh to paste the page bodies.
+
+
+## 2026-05-14 — v3.7.8 retraction + v3.7.9 corrected doctrine
+
+**What v3.7.8 claimed (WRONG):** WorkIQ has a "search-index extractor" mode triggered by literal field names (`wdsectionfileid`, `wdpartid`) in the query body. Using these field names would return verbatim indexed bodies; using natural language would route to summary mode.
+
+**What was empirically proven against HCA on 2026-05-13/14:**
+
+1. WorkIQ does NOT honor `wdsectionfileid = <id>` as filter syntax — it routes to summary mode AND returns "OneNote internal properties not exposed as searchable fields" refusal text.
+2. The wdpartid GUIDs we observed in earlier runs were **URL fragments inside SharePoint Doc.aspx hyperlinks** that WorkIQ rendered as response footnotes — not search-index extractor outputs.
+3. The Nova-pattern (natural-language query naming the section + notebook by display name and the page by quoted title) is the actual working pattern. It returned a real verbatim body for the HCA `4/3 - HCA with Jay and Martin` page.
+4. **Body retrieval is non-deterministic** — the same 4/3 page returned a verbatim body at 19:42 PDT and `BODY-NOT-EXPOSED` at 19:48 PDT, same query, no edits. The M365 search index's exposure of OneNote bodies oscillates over time.
+5. **The blocker for months was the WorkIQ EULA.** Without `workiq accept-eula`, every OneNote query silently returns nothing useful. This is a one-time setup step, not a per-call gate.
+
+**v3.7.9 corrected doctrine (now in pull-onenote SKILL.md v2.5.0):**
+
+- Pre-flight: probe WorkIQ; if EULA prompt returned, run `workiq accept-eula` and retry.
+- Step A enumerate: natural-language query naming section + notebook by display name (NOT field-name filter syntax). Returns a markdown table with one row per page; wdpartid GUIDs extracted from response URL fragments per row.
+- Step B per-page: natural-language query naming page title + section + notebook by display name. Asks for verbatim body or the literal string `BODY-NOT-EXPOSED`.
+- Per-page retry registry: every page lives in `m365-mutable.json#knownSections.<projectKey>.one_pages` with `last_status` and `attempts`. Pages stuck at `BODY-NOT-EXPOSED` are retried on every refresh until they succeed or the user pastes.
+- Snapshot files carry yaml front-matter with the same fields, so refresh runs can read state from disk if the registry is unavailable.
+
+**HCA result (2026-05-14):** 18 pages enumerated. 1 captured verbatim (4/3). 15 pending retry (BODY-NOT-EXPOSED). 2 enumeration-only (will be probed in Step B on next refresh).
+
+**Key lesson:** when a doctrine is grounded in pattern-matching against tool responses (e.g. "field names route to extractor"), validate it empirically against the live tool BEFORE shipping. The v3.7.8 doctrine was internally consistent and self-citing but never actually tested end-to-end — the 4/3 success that motivated v3.7.9 happened only after honestly retracting v3.7.8 and replicating the Nova workflow step-by-step.
+
+
+## 2026-05-14 — v3.7.9 retraction + v3.8.0 architectural pivot
+
+**What v3.7.9 (yesterday) shipped:** WorkIQ natural-language by display name + per-page retry registry. Validated 1-page capture and codified as primary path.
+
+**What we proved against HCA on 2026-05-14:** the v3.7.9 capture-rate is structurally too low for a Mon-9am scheduled refresh. WorkIQ body retrieval is non-deterministic (same page flips exposed/not exposed across queries minutes apart), and across 18 enumerated HCA pages WorkIQ returned exactly 1 verbatim body. A Monday-9am run that captures 1 page out of 16-18 is not a refresh, it's a coincidence.
+
+**The pivot — browser-scrape via OneNote-for-Web with persisted Playwright profile:**
+
+- Constructed the OneNote-for-Web deep-link URL using values already in `m365-mutable.json`: `<spoBaseUrl>/_layouts/15/Doc.aspx?sourcedoc={<notebookSourceDoc>}&action=edit&wd=target(<sectionName>|<sectionFileId>/)`.
+- Picked the Microsoft work account at the consent prompt (one-time).
+- Found the OneNote canvas frame at `ffc-onenote.officeapps.live.com/onenoteframe.aspx` (nested 2 frames deep).
+- Enumerated pages from the accessibility tree: every page has `aria-label="<title>, page X of N, Page. Select to open page contents."` — gives ordered, complete, authoritative page list.
+- Clicked each page in the rail, waited 2.5s for canvas to settle, read `document.querySelector('#PageContentWrapper').innerText` — got full verbatim body.
+
+**Result:** 16/16 HCA pages captured (~120KB total) in ~50 seconds. Compare to WorkIQ's 1/18 in 30+ minutes of probing. Includes pages WorkIQ flagged BODY-NOT-EXPOSED on previous attempts, proving the bodies were always retrievable — WorkIQ just couldn't reach them.
+
+**Architectural decision:**
+
+- Browser-scrape is the PRIMARY path in pull-onenote v2.6.0.
+- WorkIQ is the FALLBACK (when Playwright profile auth-expires) AND the source of stream events (page-edit signals via search index — those ARE deterministic).
+- Per-page registry stores BOTH `webPageId` (browser navigation GUID) AND `wdpartid` (WorkIQ correlation GUID). New `last_status` value `auth-required` for unattended-MFA-blocked runs.
+- New runner: `plugin/skills/pull-onenote/runner.mjs` (Playwright + `launchPersistentContext` for unattended refreshes).
+
+**Known gap (documented, accepted):** Conditional Access / MFA challenges cannot be satisfied unattended. Roughly every 1-4 weeks the runner will hit a sign-in redirect, exit with `runStatus: "auth-required"`, mark all queued pages `auth-required`, and surface in the run report. The user does ONE interactive `node runner.mjs --bootstrap` and the next scheduled run resumes silently.
+
+**Lessons compounded across v3.7.x → v3.8.0:**
+
+1. When a doctrine is grounded in pattern-matching against tool responses, validate it empirically end-to-end BEFORE shipping. v3.7.8 and v3.7.9 both shipped with one-page-of-evidence and were retracted within 24 hours.
+2. When a primary path's capture-rate drops below ~80%, treat it as architecturally inadequate, not "needs more retries". v3.7.9's retry registry was the correct durability layer for a HIGH-capture-rate path; on a 5%-capture-rate path it just made the bad output more visible.
+3. Browser automation is acceptable infrastructure for evidence pulls when the tool surface is deterministic and the auth model can be persisted across runs. The Loop and expense-report skills already proved this; pull-onenote v2.6.0 follows their lead.
+4. Always store identifiers in EVERY form a tool surface uses. `wdpartid` (WorkIQ) and `webPageId` (browser) are both small strings; storing both costs nothing and keeps both retrieval paths viable.
+
+
+## 2026-05-18 — Bare runner JSON + hand-rolled file writes = layout violation + UTF-8 corruption (v3.11.5)
+
+**Trigger (John Deere):** After fixing the bootstrap sign-in and single-page regex bugs, the JD scrape succeeded — but the agent wrote a single `section.md` at `Evidence/ushak/onenote/snapshot/section.md` instead of the doctrine-mandated `snapshot/pages/<safe-title>.md`. The PowerShell-piped UTF-8 also corrupted every NBSP to `┬á` (1825 occurrences in a 13 KB body). Neither defect was caught at the runner level — both were the driver's responsibility, and the driver was the agent improvising.
+
+**Root cause: the runner JSON contract has no automated writer.** `runner.mjs` is silent about the canonical snapshot layout from `snapshot-vs-stream.instructions.md` (`snapshot/pages/<safe-title>.md`, one file per page, with the full front-matter schema). Any driver — PowerShell, Clawpilot, future automation — has to re-derive: which directory, what filename, what front-matter keys, what registry shape, what run-report format. That's where things go wrong, every time.
+
+Additionally, PowerShell's default `Out-File -Encoding utf8` writes UTF-8-BOM and re-encodes non-ASCII via the system code page — NBSP (U+00A0) becomes `┬á` (the CP1252 bytes for the UTF-8 encoding of NBSP). Even `Tee-Object` does this. The only safe way to round-trip OneNote body bytes through PowerShell is `[IO.File]::WriteAllText($path, $text, [Text.UTF8Encoding]::new($false))` — or to avoid PowerShell entirely.
+
+**Fix (v3.11.5):**
+
+- New `plugin/skills/pull-onenote/write-snapshot.mjs` is the single supported driver. It:
+  - invokes `runner.mjs` via `child_process.spawnSync` (no shell, UTF-8 preserved end-to-end);
+  - writes one `snapshot/pages/<safe-title>.md` per captured page, with the full front-matter schema mandated by `pull-onenote/SKILL.md` §"Snapshot file shape";
+  - upserts `m365Mutable.knownSections.<project>.one_pages[]` with attempts counter, dual-ID schema, snapshot_path, captured_at;
+  - emits a per-run report at `Evidence/<alias>/onenote/refresh-reports/<YYYYMMDD-HHMM>-onenote.md` per `run-reports.instructions.md`.
+- `pull-onenote/SKILL.md` "Canonical CLI invocations" section now mandates `write-snapshot.mjs` as the production path; the bare `runner.mjs` is documented only as a diagnostic tool.
+
+**Doctrinal lessons:**
+
+1. Any tool whose output requires structured file-layout work MUST ship a writer of its own, not delegate to a shell script. Hand-rolled wiring is where doctrine gets lost.
+2. PowerShell `Out-File`, `Set-Content`, `Tee-Object`, and stdout `>` redirection ALL re-encode non-ASCII when capturing piped data. If a tool's output contains non-ASCII (and OneNote bodies always do — NBSP, smart quotes, em-dashes, bullet chars), the driver MUST use `child_process` or write through Node, not PowerShell pipes.
+
+**Validation:** JD `3/13 - account team` re-pulled via `write-snapshot.mjs` on 2026-05-18 — clean layout (`Evidence/ushak/onenote/snapshot/pages/3-13-account-team.md`), clean UTF-8 (`-match '┬á'` returned False), registry upserted with attempts=1, run report written.
+
+
+## 2026-05-18 — Bootstrap sign-in skipped + single-page section regex (v3.11.2 / v3.11.3)
+
+**Triggers (John Deere refresh):**
+
+1. Re-bootstrap appeared to succeed but every subsequent scrape returned `auth-required`.
+2. After fixing #1, scrape preflight passed and the canvas frame attached, but page enumeration timed out — even with a 3-minute window — for a section with only one page.
+
+**Root causes:**
+
+1. `--bootstrap` used `page.waitForURL(/onenote\.cloud\.microsoft|m365\.cloud\.microsoft/)` which **matches instantly** because we just navigated to `https://onenote.cloud.microsoft/`. The 5-min wait collapsed to ~0 and we seeded SharePoint cookies before the user could even type their email. Logs showed `Step 1/2: Sign in…` immediately followed by `Step 2/2: Seeding SharePoint cookies` with no real wait between them. Subsequent preflight failed because OneNote-for-Web cookies (different from the SPO cookies) were never minted.
+2. The page-rail enumerator regex only accepted multi-page aria labels (`, page N of M, Page.`). **Single-page sections** (e.g. John Deere's section with just `3/13 - account team`) render as `<title>, Page. Selected.` — the regex never matched and `waitForFunction` hung until timeout, producing `pages: []` with error `frame.waitForFunction: Timeout`.
+
+**Fixes:**
+
+- **v3.11.2** — `--bootstrap` now waits for a real OneNote post-auth UI indicator (`[aria-label*="Account manager" i], [data-automationid="NotebookList"], button[aria-label*="notebook" i], iframe[src*="onenoteframe.aspx"]`) — same selectors `preflightOneNoteWeb` uses. Logs `Sign-in detected (OneNote chrome rendered).` when satisfied, or a warning if 5 min elapses without sign-in.
+- **v3.11.3** — page-rail `waitForFunction` and the `pages` enumerator both accept either format:
+  - multi-page: `<title>, page N of M, Page.`
+  - single-page: `<title>, Page. Selected.` → emitted as `{ pos: 1, total: 1 }`
+  - dedup via `seen` set so a page matched by both rules isn't double-counted.
+
+**Validation:** JD `John Deere.one` captured end-to-end on 2026-05-18 — 1/1 pages, 13 KB verbatim body for `3/13 - account team` (Black Box → Dual Write FDE intake meeting). Snapshot at `John Deere/Evidence/ushak/onenote/snapshot/section.md`.
+
+**Doctrinal lesson:** any `waitForURL` against a URL we just navigated to is a no-op — must wait for a **post-auth UI signal** (chrome that only renders after token exchange). Same anti-pattern would apply to any future bootstrap (SharePoint, Loop, M365 admin, etc.). And: any aria-label-driven enumerator must handle the **N=1 special-case format** for that UI surface, not assume the multi-element format applies universally.
+
+
+## 2026-05-18 — WorkIQ OneNote three-tier output codified (kushi v3.11.1)
+
+**What happened.** User pointed out that `hca-snip.txt` and `hca-onenote-raw.txt` (Scratchpad, 2026-05-13) exist as WorkIQ-derived OneNote evidence, contradicting the agent's earlier claim that "OneNote was unavailable" during JD bootstrap. Root cause: I had asked WorkIQ for "all 21 pages' full verbatim bodies in one call" — WorkIQ silently degraded to tier A (page enumeration) and tier B (search snippets) and returned an explicit note `combined content size … exceeds the maximum response payload that Copilot can safely render in-chat`. I read that as "OneNote unavailable" instead of "Tier C must be per-page."
+
+**Empirical contract (now codified in `workiq-only.instructions.md`):**
+
+WorkIQ returns OneNote in THREE tiers:
+
+- **Tier A — Enumeration**: titles + wdpartid + wdsectionfileid + last-modified + author + deep-link. Bulk OK. (HCA evidence: request-ids `bc780473-66d7-4809-89dc-e910e4b8ced8`, `fdbf3290-649d-4066-84ea-ceab678697ed`.)
+- **Tier B — Search snippets**: ~500-char Graph search snippet per page, verbatim from search index. Bulk OK. (HCA evidence: request-ids `f3744946-5b22-4c09-96ac-714c8508d5f6`, `d93deb47-147b-4d19-8cd7-c0a0e2f76f30`.)
+- **Tier C — Full verbatim body**: ONE page per call. Bulk refused.
+
+**Defect signature:** "give me full bodies of all pages in section X" prompt → response includes the phrase `combined content size … exceeds the maximum response payload` plus a tier-A inventory table. Misreading that as "OneNote unavailable" is the defect. Correct response: re-issue the per-page tier-C prompt (one wdpartid at a time).
+
+**Result.** `workiq-only.instructions.md` OneNote section now lists all four prompts (Tier A discovery, Tier A page index, Tier B snippets, Tier C single-page body) with the exact WorkIQ wording. `pull-onenote` v2.9.0 Playwright-primary doctrine is unchanged — Playwright is not a Graph call, and it remains the verbatim-bulk path. WorkIQ tier C is the fallback when Playwright auth expires (do per-page tier-C calls in that mode, not bulk).

package/plugin/learnings/sharepoint.md

@@ -0,0 +1,5 @@
+# Learnings — SharePoint / OneDrive (`pull-sharepoint`)
+
+Newest on top. Format defined in [`README.md`](./README.md).
+
+_(no entries yet — append the moment a fix lands during a `pull-sharepoint` run)_

package/plugin/learnings/teams.md

@@ -0,0 +1,5 @@
+# Learnings — Teams (`pull-teams`)
+
+Newest on top. Format defined in [`README.md`](./README.md).
+
+_(no entries yet — append the moment a fix lands during a `pull-teams` run)_

package/plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "kushi",
-  "description": "Multi-source project evidence + Q&A agent. Snapshot + stream capture across Email, Teams, OneNote, SharePoint, Meetings, CRM, ADO; plus read-only natural-language Q&A over the captured evidence. WorkIQ-first. Host-agnostic. Three install profiles: core (aggregator only), standard (default — adds bootstrap/refresh + FDE authoring), full (adds State/ rollup).",
-  "version": "3.3.0",
+  "description": "Multi-source project evidence + Q&A agent. Snapshot + stream capture across Email, Teams, OneNote, SharePoint, Meetings, CRM, ADO; plus read-only natural-language Q&A over the captured evidence. WorkIQ-only for M365 sources (Graph / m365_* FORBIDDEN as fallbacks; user-paste is first-class). Host-agnostic. Three install profiles: core (aggregator only), standard (default — adds bootstrap/refresh + FDE authoring), full (adds State/ rollup).",
+  "version": "3.12.1",
   "author": "ushakrishnan",
   "repository": "https://github.com/gim-home/kushi",
   "default_profile": "standard",
@@ -91,6 +91,26 @@
       "verbs": [
         "state"
       ]
+    },
+    "preview": {
+      "extends": "standard",
+      "description": "PREVIEW: opt-in two-way sync skills. Adds propose-ado-update (read-only proposal generator) and apply-ado-update (gated; v0.1.0-preview is dry-mode only — produces planned.jsonl, no real ADO writes). Governed by update-ledger.instructions.md. See docs/concepts/roadmap.md and docs/how-to/two-way-ado-update.md.",
+      "skills": [
+        "propose-ado-update",
+        "apply-ado-update"
+      ],
+      "prompts": [
+        "propose-ado",
+        "apply-ado"
+      ],
+      "templates": [
+        "ado-update"
+      ],
+      "reference_packs": [],
+      "verbs": [
+        "propose-ado",
+        "apply-ado"
+      ]
     }
   }
 }

package/plugin/prompts/apply-ado.prompt.md

@@ -0,0 +1,14 @@
+---
+name: apply-ado
+description: Apply approved ADO updates from a proposed.md (gated). Currently a preview-stub — writes a planned.jsonl, no real ADO calls yet.
+---
+
+# /apply-ado
+
+Route to `@Kushi apply ado <project>`.
+
+Delegates to the `apply-ado-update` skill. **Gated** — every write is reviewed (or auto-allowlisted via `<project>/.kushi/ado-update.yml`), every applied write appended to `ledger.jsonl` with a reverse op.
+
+In v0.1.0-preview this runs in **dry-mode only** — it produces `planned.jsonl` instead of calling ADO. The real write path lands in a follow-up release once the proposal format has been validated against real projects.
+
+Profile: **`preview`** only.

package/plugin/prompts/propose-ado.prompt.md

@@ -0,0 +1,12 @@
+---
+name: propose-ado
+description: Read-only ADO update proposal — generate proposed.md from the latest consolidated evidence. NO writes to ADO.
+---
+
+# /propose-ado
+
+Route to `@Kushi propose ado <project>`.
+
+Delegates to the `propose-ado-update` skill. Read-only — produces a Markdown preview at `<engagement-root>/ado-updates/<YYYY-MM-DD>/proposed.md`. No ADO writes.
+
+Profile: **`preview`** only.

package/plugin/reference-packs/fde/crm-field-manifest.md

@@ -0,0 +1,165 @@
+# FDE CRM (Dataverse) Field Manifest
+
+> Canonical mapping of FDE engagement-submission long-text fields + annotation pull pattern. Consumed by `pull-crm` whenever the configured entity set is an FDE engagement-submission table.
+
+Cite as: `[source: reference-packs/fde/crm-field-manifest.md · packaged]`.
+
+---
+
+## Why this manifest exists
+
+The FDE engagement-submission record in Dataverse holds the strategic answers an FDE crew needs (business scenario, why current offers don't fit, success criteria, technology workloads, work done so far). These live in **long-text fields** (`Memo` / multi-line text) that must be captured **verbatim** — paraphrasing or summarizing them at pull time strips the context Kushi is supposed to preserve.
+
+Pre-v3.7.2, `pull-crm` retrieved a CRM record with whatever fields the API returned by default, which often dropped or truncated the long-text answers. This manifest fixes that by giving `pull-crm` an explicit `$select` list and an explicit annotation fetch loop.
+
+---
+
+## Display-name → logical-name mapping
+
+The user-facing labels (left) map to Dataverse logical names (right). Logical names vary per tenant; treat the right column as the **probable** logical name and do the discovery probe described below if a `$select` returns null.
+
+| Display label (form question) | Probable logical name | Type |
+|---|---|---|
+| Engaged With | `msfre_engagedwith` | Memo (multi-line) |
+| What is the business scenario or technical blocker that FDE would solve? | `msfre_businessscenario` | Memo |
+| Why are the current offers or solutions not suitable for this customer's needs? | `msfre_currentofferinggap` | Memo |
+| What does success look like for the customer? | `msfre_successcriteria` | Memo |
+| Please list the relevant technology (1st party / 3rd party) workloads being used | `msfre_technologyworkloads` | Memo |
+| Give a description of the work completed so far (if any) | `msfre_workcompletedsofar` | Memo |
+| MSX Opportunity ID | `msfre_msxopportunityid` | Single line |
+| Customer name | `msfre_customername` (or via lookup `_msfre_account_value`) | Lookup |
+| Engagement title | `msfre_engagementtitle` (or `name`) | Single line |
+| FDE crew / division | `msfre_division`, `msfre_crew` | OptionSet / Single line |
+| Status / state | `statuscode`, `statecode` | OptionSet |
+| Created on / by | `createdon`, `_createdby_value` | Datetime / Lookup |
+| Modified on / by | `modifiedon`, `_modifiedby_value` | Datetime / Lookup |
+
+### Discovery probe (run once per project, persist to mutable)
+
+If the configured `entitySetName` is `msfre_*` and a `$select` for any of the probable logical names above returns `null`, run a **metadata probe** to confirm the actual logical names:
+
+```http
+GET /api/data/v9.2/EntityDefinitions(LogicalName='msfre_engagementsubmission')/Attributes?$select=LogicalName,DisplayName,AttributeType&$filter=AttributeType eq 'Memo'
+```
+
+Persist the resolved mapping to `m365Mutable.knownSections.<project>.crm.fieldMap` so subsequent pulls skip the probe.
+
+---
+
+## REQUIRED $select for FDE engagement-submission records
+
+Every snapshot pull MUST request these fields explicitly (no relying on default projection):
+
+```
+$select=
+  msfre_engagementsubmissionid,
+  name,
+  msfre_engagedwith,
+  msfre_businessscenario,
+  msfre_currentofferinggap,
+  msfre_successcriteria,
+  msfre_technologyworkloads,
+  msfre_workcompletedsofar,
+  msfre_msxopportunityid,
+  statuscode,
+  statecode,
+  createdon,
+  createdby,
+  modifiedon,
+  modifiedby,
+  _ownerid_value,
+  _msfre_account_value
+```
+
+Then `$expand=Annotations($select=annotationid,subject,notetext,createdon,_createdby_value,filename,mimetype)` to pull every note attached to the record.
+
+If the discovery probe returned a different logical name, substitute it before issuing the request.
+
+---
+
+## Annotation (notes) pull pattern
+
+CRM Notes are stored as `Annotation` records related to the parent record via `objectid`. They are NOT included by default — they must be expanded.
+
+For each annotation returned:
+
+1. Capture **verbatim** the full `notetext` (do not summarize, do not truncate, do not strip newlines).
+2. Capture metadata: `subject`, `createdon`, `createdby` (resolved name, not GUID), `filename` + `mimetype` if attached.
+3. Write each annotation as a `### Note — <subject> (<createdon> by <createdby>)` block under a `## Notes (verbatim)` H2 section in the snapshot file.
+4. If the annotation has an attached file (`filename` non-null), record its filename + mimetype + size; do NOT download the binary unless the user explicitly asks.
+
+---
+
+## Snapshot file shape (FDE CRM record)
+
+When the entity is FDE engagement-submission, the snapshot file MUST follow this shape:
+
+```markdown
+# CRM — <customer-name> · <engagement-title>
+
+**Record ID:** `<id>` · **MSX Opportunity:** `<msx-id>` · **Last fetched:** `<iso-ts>`
+**Status:** `<statuscode label>` · **Owner:** `<owner display name>`
+
+## Source Basis
+
+- Tool: <workiq | dataverse-rest | graph>
+- Boundary: `boundaries.crm.record_ids = [<id>]`
+- Field map: `m365Mutable.knownSections.<project>.crm.fieldMap` (probed <date>) | packaged (default)
+- Annotations fetched: <N>
+
+## AI Narrative Summary
+
+(3+ paragraphs — engagement story so far, current stage, what's at stake, latest direction. No paraphrase of the long-text answers below; summarize what they say collectively.)
+
+## Engagement context
+
+### Engaged With
+> (verbatim msfre_engagedwith)
+
+### Business scenario / technical blocker
+> (verbatim msfre_businessscenario)
+
+### Why current offers / solutions not suitable
+> (verbatim msfre_currentofferinggap)
+
+### What success looks like
+> (verbatim msfre_successcriteria)
+
+### Technology workloads (1st / 3rd party)
+> (verbatim msfre_technologyworkloads)
+
+### Work completed so far
+> (verbatim msfre_workcompletedsofar)
+
+## All other fields
+
+(Every other field returned by the $select, in a 2-column table: field | value. Empty fields shown as `_(empty)_` so absence is visible.)
+
+## Notes (verbatim)
+
+### Note — <subject> (<createdon> by <createdby>)
+> (verbatim notetext, no truncation)
+
+### Note — <subject> (<createdon> by <createdby>)
+> (verbatim notetext)
+
+(... one block per annotation, oldest first ...)
+```
+
+---
+
+## Anti-patterns (defects)
+
+- ❌ Paraphrasing the long-text fields — they MUST be verbatim.
+- ❌ Omitting empty fields — write `_(empty)_` so the reader can see we asked.
+- ❌ Skipping annotations because they look like noise — every annotation goes in.
+- ❌ Truncating long annotations with `...` — record the full text even if it's pages.
+- ❌ Falling back to "narrate from email" when CRM is unreachable — see `pull-crm/SKILL.md` Hard prerequisites; refuse instead.
+
+---
+
+## See also
+
+- `intake-questions.md` — the FDE Intake question set this manifest backs.
+- `pull-crm/SKILL.md` — the skill that must apply this manifest.
+- `core-fde-reference.md` — broader FDE operating model.

package/plugin/skills/apply-ado-update/SKILL.md

@@ -0,0 +1,125 @@
+---
+name: "apply-ado-update"
+version: "0.1.0-preview"
+status: "preview-stub"
+description: "Gated write skill: reads <engagement-root>/<project>/ado-updates/<date>/proposed.md, presents the diff for approval, applies approved items to ADO (PATCH field + POST comment), and appends to the per-project update ledger. v0.1.0-preview is a STUB — runs in dry-mode only and writes a planned-writes log; no real ADO PATCH/POST yet."
+---
+
+# Skill: apply-ado-update
+
+Run this when the user says any of: "apply ado update for `<X>`", "@Kushi apply ado `<X>`", "write to ado for `<X>`".
+
+This skill is the **only** code path in Kushi authorized to write to ADO. It implements the doctrine in `update-ledger.instructions.md`.
+
+## Status: preview-stub (v0.1.0-preview)
+
+In v0.1.0-preview this skill **runs in dry-mode only**. It:
+
+- Reads `proposed.md`.
+- Validates the proposal shape and citations.
+- Re-fetches the current ADO Initiative state and re-computes the diff (in case it drifted since `propose-ado-update` ran).
+- Asks the user to approve all / select / none.
+- Writes a **planned-writes log** to `<project>/ado-updates/<YYYY-MM-DD>/planned.jsonl` recording what *would* have been written.
+- Does **not** call any ADO `PATCH` or `POST` endpoint yet.
+
+The real write path (PATCH field + POST comment + ledger append) lands in v0.1.x once the proposed.md format, the apply UX, and the ledger schema have been validated against real projects for at least one full week. See `docs/concepts/roadmap.md`.
+
+## Profile
+
+Belongs to the **`preview`** profile. Opt in via `npx kushi-agents --clawpilot --profile preview`.
+
+## Deterministic config — do not invent paths
+
+Same as `propose-ado-update`. Reads ADO connection from `<engagement-root>/.project-evidence/ado/config.yml`, per-project `engagement_id` and `writes:` block from `<engagement-root>/<project>/integrations.yml ado:`. Never asks for paths the bootstrap layer already resolved.
+
+## Pre-flight (HARD — do not bypass)
+
+1. **Resolve engagement root + project** per `engagement-root-resolution.instructions.md`.
+2. Confirm `<project>/integrations.yml ado.engagement_id > 0`. If 0 → abort with the same message `propose-ado-update` uses ("ADO Initiative not yet linked").
+3. Confirm `<project>/ado-updates/<YYYY-MM-DD>/proposed.md` exists. If not → tell user to run `@Kushi propose ado <project>` first. Never fabricate a proposal.
+4. Per `azure-auth-patterns.instructions.md` — Section 1 (session pre-check) + Section 3 (ADO tenant validation, using `<engagement-root>/.project-evidence/ado/config.yml`) **must** complete green before any further step.
+
+## Steps (current preview-stub behavior)
+
+1. **Load proposed.md** and parse the field-update + comment-update sections (Markdown headings + fenced blocks).
+2. **Re-fetch current ADO state** (read-only GET) and compare against the `current` value embedded in `proposed.md`. If they differ → warn `proposal-drift`, show the live current value, and require explicit re-approval.
+3. **Duplicate detection** — list the latest 5 Discussion comments via ADO API; if any contain the Kushi fingerprint line for this same week → mark `duplicate-detected`, skip the comment write, still proceed with field update if not duplicate.
+4. **Allowlist gate** — every item in `proposed.md` whose `fieldRefName` is NOT in `integrations.yml ado.writes.allowlist.fields` is rejected before approval is even offered. This is the last-line guard against a malformed proposal.
+5. **Approval gate**:
+   - Default: prompt user with the diff and `[a]ll · [s]elect · [n]one`.
+   - Auto-apply: if `ado.writes.statusSummary.autoApply: true` AND the field-update item has `confidence: high` AND the new line matches the configured `<MMM YYYY>: ...` pattern → mark approver `auto:append-month-high-conf`. Same for `discussionComment.autoApply` (rare; off by default).
+   - All other items → require interactive approval.
+6. **(Stub)** Instead of calling ADO writes, append to `<project>/ado-updates/<YYYY-MM-DD>/planned.jsonl` one record per approved item:
+   ```json
+   {
+     "timestamp": "2026-05-18T16:30:00Z",
+     "engagementId": 12345,
+     "kind": "field|comment",
+     "fieldRefName": "Custom.FDEStatusSummary",
+     "currentValue": "...",
+     "proposedValue": "...",
+     "approver": "ushak | auto:append-month-high-conf",
+     "confidence": "high",
+     "evidenceCitations": ["ushak/onenote/snapshot/architecture-decisions.md · 2026-05-12"],
+     "fingerprint": "kushi-weekly-2026-05-18-12345-field",
+     "executed": false,
+     "stubReason": "v0.1.0-preview: write path not yet enabled"
+   }
+   ```
+7. **Echo summary** to user: how many items would be written, path to `planned.jsonl`, and the line "No ADO calls were made (preview-stub)."
+
+## Steps (planned for v0.1.x — when write path lands)
+
+After step 5 (approval), instead of step 6:
+
+6. **Apply field update** (one PATCH per approved field):
+   ```http
+   PATCH https://dev.azure.com/{org}/{project}/_apis/wit/workitems/{engagement_id}?api-version=7.1
+   Content-Type: application/json-patch+json
+
+   [
+     { "op": "test", "path": "/rev", "value": <currentRev> },
+     { "op": "add", "path": "/fields/Custom.FDEStatusSummary", "value": "<new value>" }
+   ]
+   ```
+   The `test` on `/rev` is the optimistic-lock guard. If it fails → ledger entry `error: rev-mismatch`, do NOT retry silently; surface to user.
+
+7. **Apply Discussion comment** (one POST per approved comment):
+   ```http
+   POST https://dev.azure.com/{org}/{project}/_apis/wit/workItems/{engagement_id}/comments?api-version=7.1-preview.4
+   Content-Type: application/json
+   { "text": "<rendered HTML>" }
+   ```
+
+8. **Append to ledger** at `<project>/ado-updates/<YYYY-MM-DD>/ledger.jsonl` per `update-ledger.instructions.md` §Schema, including the **reverse op** (so a later `revert` skill can roll back).
+
+9. **Optional notify** — if `ado.writes.approvals.notifyOnApply: teams`, post a one-line summary via `m_send_teams_message` to the user (never to others).
+
+## Failure modes (preview-stub)
+
+| Symptom | Recovery |
+|---|---|
+| `proposed.md` missing | Stop. Tell user to run `@Kushi propose ado <project>` first. |
+| `integrations.yml` missing or `ado.engagement_id == 0` | Stop with same message `propose-ado-update` uses; do not prompt for an ID. |
+| `ado.writes:` block missing | Stop. Tell user to run `@Kushi propose ado <project>` first (it scaffolds the block). |
+| ADO 401 on the read-only re-fetch in step 2 | Re-acquire token once per `azure-auth-patterns.instructions.md` §4; if still 401, abort `auth-failed`. |
+| Tenant mismatch | Per `azure-auth-patterns.instructions.md` §3 — abort with the exact `az login --tenant <id>` command. |
+| `proposal-drift` (current ADO value differs from `proposed.md` snapshot) | Show diff between live and proposed; require user to re-confirm or re-run `propose-ado-update`. |
+| `duplicate-detected` (comment fingerprint already on the work item) | Skip comment portion; continue with field if not also duplicate. |
+| Allowlist rejection | Refuse the item before approval. Tell user the field reference name is not in `integrations.yml ado.writes.allowlist.fields`. |
+
+## What this skill does NOT do (ever — not even in v0.1.x)
+
+- Does NOT auto-apply any item with `confidence < high`.
+- Does NOT auto-apply any item not on the per-project allowlist (`ado.writes.<x>.autoApply: true`).
+- Does NOT write to fields not listed in `ado.writes.allowlist.fields` of `<project>/integrations.yml`.
+- Does NOT post comments other than the single rendered Discussion comment from `proposed.md`.
+- Does NOT bulk-apply across projects in a single call (one project per invocation).
+- Does NOT delete or close work items, change parent links, change area path, or touch any field outside the configured allowlist.
+
+## References
+
+- `update-ledger.instructions.md` — ledger schema, reverse-op format, write-path safety doctrine.
+- `azure-auth-patterns.instructions.md` — pre-flight, tenant validation, token reuse, 401 recovery.
+- `propose-ado-update` SKILL.md — produces the `proposed.md` consumed by this skill.
+- `engagement-root-resolution.instructions.md` — resolving `<engagement-root>` and `<project>`.

package/plugin/skills/ask-project/SKILL.md

@@ -20,6 +20,8 @@ The user does NOT need to type `/ask-project` or `@Kushi ask`. If the message:
 - **Read-only.** No `pull-*`, no Graph writes, no WorkIQ calls during answering. Source pulls are a separate user-initiated step.
 - **No cross-project bleed.** Answer only from the resolved project's own folders. If the question requires comparing two projects, ask the user to confirm and run `ask-project` per project, then merge in the answer.
 - **Citation Ledger applies.** Every fact, decision, person, date, $ figure carries inline `[source: <alias>/<folder>/<file> · YYYY-MM-DD]` per `instructions/citation-ledger.instructions.md` and `instructions/answer-from-evidence.instructions.md`. If the corpus does not support an answer, say so explicitly — never invent.
+- **Full-view gate applies.** Walk the standard 7-source set per `instructions/full-view-gate.instructions.md` before answering and render the Source Basis disposition table. Do not call the answer a "full view" if any applicable source is `cached-stale`, `attempted-but-blocked`, or `unknown`.
+- **Confidence ladder applies.** Tag every claim about decisions/status/funding/scope/owner/commitment with `internal-only` / `communicated` / `confirmed` per `instructions/evidence-confidence-ladder.instructions.md`. Never paraphrase a CRM/ADO field flip as a settled fact without a customer-facing confirmation dated AFTER the flip.
 - **Privacy posture.** Same as Kushi global: no outbound messages, no leaking attendee emails into anything visible to others. Answer goes to chat only.
 - **Freshness gate, not freshness auto-fix.** If the freshest source relevant to the question is older than `chat.freshness_warn_days` (default 14), warn the user and offer `@Kushi refresh <project>` — but NEVER auto-refresh.
 - **Reference packs may be consulted for domain doctrine.** When the question maps to a known reference-pack domain (currently only `fde/` — FDE stages, fitness, CRM status meanings, intake gates, risk categories, "MACC", "is this FDE-fit"), ALSO load the matching reference pack as additional grounding using the 3-layer override order (project → user → packaged). Cite reference-pack assertions with `[source: reference-packs/<pack>/<file>.md · <layer>]` where layer is `packaged` / `user-override` / `project-override`. Reference-pack content NEVER overrides project Evidence/ for facts about *this* project; it only provides definitions, gates, and rubrics.