kushi-agents 3.4.2 → 3.12.1

package/plugin/skills/pull-ado/SKILL.md

@@ -1,23 +1,34 @@
 ---
 name: "pull-ado"
-version: "2.0.0"
-description: "Pull ADO evidence (snapshot: work-item fields; stream: comments + state changes + history). Conditional az login per az-auth-conditional rule."
+version: "2.3.1"
+description: "Pull ADO evidence as an engagement TREE (parent Engagement + every child WI). Per-item discussion comments + revision history (verbatim). Deterministic resolution; no parent-only summaries. Per ado-engagement-tree.instructions.md."
 ---
 
 # Skill: pull-ado
 
+
+> **v3.7.6 contracts** — This skill operates under four HARD-rule doctrines:
+> - `verbatim-by-default.instructions.md` — full bodies/notetext/fields by default; no preview-grade pulls accepted.
+> - `capture-learnings.instructions.md` — every fix/discovery is logged to `plugin/learnings/<source>.md` immediately.
+> - `cleanup-on-resolution.instructions.md` — when a value resolves, all stale `no-match` / `not yet` notes referencing the prior unresolved state must be rewritten in the same turn.
+> - `run-reports.instructions.md` — every refresh writes a per-user report under `Evidence/<alias>/refresh-reports/YYYY-MM-DD-HHMM_refresh.md`.
+
+> **Canonical evidence layout** (HARD, kushi v3.12.1+): all artifacts produced by this skill MUST be written under `<project>/Evidence/<alias>/<source>/{snapshot,stream,...}/` — sibling folders under `<project>/` (e.g. `<project>/<source>-context/`, `<project>/<source>/`, `<project>/_Weekly Summaries/`) are FORBIDDEN. See `evidence-layout-canonical.instructions.md`.
+
 Pulls **ado** evidence in two shapes per `snapshot-vs-stream.instructions.md`:
 
-- **snapshot/** — work item with all fields (title, type, area, iteration, parent, state, assignee, custom fields)
-- **stream/** — comments verbatim + state changes (old → new) + field changes + attachments — per work item
+- **snapshot/** — engagement TREE: parent Engagement + every child WI; each item gets full fields + every discussion comment verbatim + full revision history
+- **stream/** — new comments + state changes + field changes + attachments across the entire tree, weekly bucket
 
-WorkIQ-first per `workiq-first.instructions.md`. Thoroughness per `evidence-thoroughness.instructions.md`; runtime detector + auto-retry + paste-prompt per `thoroughness-detector.instructions.md`. Citations per `citation-ledger.instructions.md`. Side-by-side mutable hints written immediately on discovery per `side-by-side-config.instructions.md`.
+WorkIQ-first per `workiq-first.instructions.md` — but **for ADO, REST is preferred for snapshot** because WorkIQ summarizes discussion threads. Thoroughness per `evidence-thoroughness.instructions.md`; runtime detector + auto-retry + paste-prompt per `thoroughness-detector.instructions.md`. Citations per `citation-ledger.instructions.md`. Side-by-side mutable hints written immediately on discovery per `side-by-side-config.instructions.md`. Tree expansion + per-item discussion + revision pulls per `ado-engagement-tree.instructions.md`.
 
 ## Inputs
 
 - `<project>` — already-resolved project name.
 - `<alias>` — current contributor.
 - `<window>` — date range. For snapshot: ignored (always full re-fetch). For stream: `(from, to)`.
+- (read) `<engagement-root>/.project-evidence/ado/config.yml` — connection.
+- (read) `<engagement-root>/<project>/integrations.yml#ado` — per-project pinned IDs.
 - (read) `<engagement-root>/.project-evidence/m365/m365-mutable.json m365Mutable.knownSections.<project>` — pinned hints.
 - (read) `<engagement-root>/<project>/Evidence/<alias>/.settings.yml` — per-(project x user) overrides.
 
@@ -27,30 +38,162 @@ WorkIQ-first per `workiq-first.instructions.md`. Thoroughness per `evidence-thor
 - `ado.queryId` — saved query ID for the project
 - `ado.areaPath` / `ado.iterationPath` — for filtering
 
-## Snapshot pass
+## Hard prerequisites (REQUIRED — see `scope-boundaries.instructions.md` Rule 3)
+
+This skill is **HARD-fail** without both:
+
+1. Global config: `<engagement-root>/.project-evidence/ado/config.yml` exists with non-placeholder `organization` AND `defaultProject`.
+2. Per-project boundary: `<engagement-root>/<project>/integrations.yml#boundaries.ado.area_paths` is non-empty (OR `boundaries.ado.work_item_ids` is pinned).
+
+If either is missing, refuse with the exact message:
+
+```
+ado-config-missing — drop a filled config.yml at
+<engagement-root>/.project-evidence/ado/config.yml. See template at
+plugin/templates/init/ado-config.template.yml. Then add boundaries.ado.area_paths
+to <project>/integrations.yml.
+```
+
+Discovery loops (`title_contains`, `tags_contains`, `custom_iscrm_id`) are still
+permitted — but ONLY inside `boundaries.ado.area_paths`. There is no
+tenant-wide title-fuzzy scan in v3.7.0+.
+
+## Auth (deterministic, no improvisation)
+
+```powershell
+$cfg = Get-Content "<engagement-root>/.project-evidence/ado/config.yml" | ConvertFrom-Yaml
+$tok = az account get-access-token `
+  --resource $cfg.azDevOpsResourceId `
+  --tenant $cfg.tenantId `
+  --query accessToken -o tsv
+$h = @{ Authorization = "Bearer $tok"; 'Content-Type' = 'application/json' }
+$org  = $cfg.organization               # e.g. https://dev.azure.com/IndustrySolutions
+$proj = $cfg.defaultProject             # e.g. "IS Engagements"
+$projUrl = "$org/$([uri]::EscapeDataString($proj))"
+```
+
+If `az account get-access-token` returns nothing → ask user to `az login --tenant <tenantId>` and stop. Do NOT silently proceed.
+
+## Step 1 — Resolve the Engagement WI ID (deterministic order)
+
+Per `ado-engagement-tree.instructions.md` § Resolution order:
+
+### 1a — Pinned ID
+
+If `<project>/integrations.yml#ado.engagement_id` is non-null, use it. Skip to Step 2.
+
+### 1b — Pinned query
+
+If `ado.queryId` is set:
+
+```powershell
+$q = Invoke-RestMethod -Headers $h -Uri "$projUrl/_apis/wit/wiql/$($cfg.ado.queryId)?api-version=7.1"
+```
+
+Take the single `Engagement`-typed row. If multiple → ask user, persist choice.
+
+### 1c — Bounded WIQL
+
+Build the WIQL ONLY inside `boundaries.ado.area_paths`:
+
+```sql
+SELECT [System.Id],[System.Title],[System.WorkItemType],[System.State],
+       [System.AreaPath],[System.IterationPath],[System.Tags],[System.AssignedTo],
+       [System.Parent],[System.CreatedDate],[System.ChangedDate]
+FROM workitems
+WHERE [System.TeamProject] = '<defaultProject>'
+  AND [System.WorkItemType] = 'Engagement'
+  AND [System.AreaPath] UNDER '<area_path>'
+  AND ([System.Title] CONTAINS '<project>' OR
+       [Custom.ISCRMRequestId] = '<crm.request_id>' OR
+       [System.Tags] CONTAINS '<project>')
+  AND [System.CreatedDate] > '<2-years-ago>'
+  AND [System.State] <> 'Removed'
+ORDER BY [System.ChangedDate] DESC
+```
+
+POST to `$projUrl/_apis/wit/wiql?api-version=7.1` with body `{ "query": "<wiql>" }`.
+
+- 0 rows → write `last_discovery_result: no-match` + the WIQL used + `next_step` and STOP.
+- 1 row → use it. Persist `engagement_id` to `integrations.yml#ado` AND to `m365Mutable.knownSections.<project>.ado.engagementId`.
+- N rows → render the candidates table to the user (id, title, state, areaPath, createdDate). Ask which one. **Never auto-pick.** Persist on confirmation.
+
+## Step 2 — Expand the engagement tree
+
+```powershell
+$wi = Invoke-RestMethod -Headers $h -Uri "$projUrl/_apis/wit/workitems/$engagementId?`$expand=relations&api-version=7.1"
+$childIds = $wi.relations |
+  Where-Object { $_.rel -eq 'System.LinkTypes.Hierarchy-Forward' } |
+  ForEach-Object { ($_.url -split '/')[-1] }
+```
+
+Recurse — expand each child's `relations` the same way until the tree closes. Persist the resulting tree (parent → children IDs) to `Evidence/<alias>/ado/snapshot/tree.md`.
+
+## Step 3 — Per-item fetch (parent + every child, no batching)
+
+For each WI id in the tree:
+
+### 3a — Core fields
+
+```powershell
+$item = Invoke-RestMethod -Headers $h `
+  -Uri "$projUrl/_apis/wit/workitems/$id?`$expand=all&api-version=7.1"
+```
+
+`$expand=all` returns every populated field including custom fields. Don't pre-filter.
+
+### 3b — Discussion comments (verbatim, paginated until exhausted)
+
+```powershell
+$cmts = @()
+$ct = $null
+do {
+  $url = "$projUrl/_apis/wit/workItems/$id/comments?api-version=7.1-preview.4&`$top=200"
+  if ($ct) { $url += "&continuationToken=$ct" }
+  $page = Invoke-RestMethod -Headers $h -Uri $url
+  $cmts += $page.comments
+  $ct = $page.continuationToken
+} while ($ct)
+```
+
+Capture EVERY comment: `id`, `text` (verbatim, no truncation), `createdBy.displayName`, `createdDate`, `modifiedDate`. Render newest-first under `## Discussion (verbatim)`.
+
+### 3c — Revision history (every state + assignee + field change)
+
+```powershell
+$updates = Invoke-RestMethod -Headers $h `
+  -Uri "$projUrl/_apis/wit/workItems/$id/updates?api-version=7.1&`$top=500"
+```
+
+For each update, render `revisedBy.displayName`, `revisedDate`, and the `fields.<name>.{oldValue → newValue}` deltas. Group by date.
 
-Write `snapshot/items/<id>.md` per work item with every field. Top header: id, title, type, state, parent, area, iteration, assignee, last fetched.
+### 3d — Attachments (metadata only)
 
-Write to: `<engagement-root>/<project>/Evidence/<alias>/ado/snapshot/<entity>.md`
+From `$item.relations` where `rel = 'AttachedFile'` — record `name`, `resourceSize`, `attributes`. Do not download binary unless user asks.
 
-Use template: `templates/snapshot/ado-<kind>.template.md`
+### 3e — Write per-item snapshot file
 
-## Stream pass
+`<engagement-root>/<project>/Evidence/<alias>/ado/snapshot/items/<id>.md` (parent gets `engagement-<id>.md` at the snapshot root):
 
-Per `evidence-thoroughness.instructions.md`: every comment verbatim with author + timestamp; every state change; every field change. Bucketed by ISO week into `stream/items/<id>/<YYYY-MM-DD>_ado-stream.md`.
+- Header: id, type, title, state, areaPath, parent, assignee, last fetched.
+- `## Source Basis` — tool used (rest), boundary applied, comments fetched, revisions fetched.
+- `## AI Narrative Summary` (3+ paragraphs — REQUIRED FIRST).
+- `## All fields` — 2-column table; empty as `_(empty)_`.
+- `## Discussion (verbatim)` — every comment as `### <author> · <date>` then verbatim blockquote.
+- `## Revision history` — every update row.
+- `## Attachments` — metadata table.
 
-Write to: `<engagement-root>/<project>/Evidence/<alias>/ado/stream/<YYYY-MM-DD>_ado-stream.md` (date = Monday of the ISO week the events fall in).
+## Step 4 — Stream pass
 
-Use template: `templates/weekly/ado-summary.template.md`
+After all snapshots written, build the weekly stream from comments + updates whose `createdDate` / `revisedDate` fall in `<window>`. Bucket by Monday-of-ISO-week into `stream/<YYYY-MM-DD>_ado-stream.md` with AI Narrative Summary first then the verbatim entries grouped by WI.
 
-If a week file already exists, MERGE (dedupe by event ID, append new events, keep existing).
+If a week file exists, MERGE (dedupe by `comment.id` and `update.rev`).
 
 ## Tools (in order)
 
-1. **WorkIQ** — ````$WorkIQQuery`````
-2. **Host fallback** — ADO REST API via `az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798` (read connection/queries from `<engagement-root>/.project-evidence/ado/config.yml`)
-3. **Graph REST** — last resort, soft-fail per `az-auth-conditional.instructions.md`.
-4. **Ask user** — paste verbatim source content if all above fail.
+1. **WorkIQ** — discovery only (asking "is there an Engagement WI for HCA?" before doing the WIQL). Never used for discussion-comment fetch — it summarizes.
+2. **ADO REST** — REQUIRED for fields + comments + updates + tree expansion (preferred path). Auth block above.
+3. **Ask user** — paste verbatim source content if REST returns 401 / 404 / persistent 5xx after 2 retries (3s/6s).
 
 Document which path succeeded under `## Source Basis` in each output file.
 
@@ -61,17 +204,24 @@ If discovered, immediately write to `m365Mutable.knownSections.<project>.<source
 - `ado.engagementId` — pinned engagement work-item ID
 - `ado.queryId` — saved query ID for the project
 - `ado.areaPath` / `ado.iterationPath` — for filtering
+- `ado.tree.lastBuiltOn` + `ado.tree.itemCount` — so consolidate can detect drift between runs.
 
 ## Update run-log
 
 After successful pass:
 - `sources.ado.last_pulled = now`
-- `sources.ado.watermark = now` (stream only; snapshot has no watermark)
-- `sources.ado.item_count = <N>`
+- `sources.ado.engagement_id = <id>`
+- `sources.ado.tree_size = <N>` — total WIs in the tree
+- `sources.ado.comments_fetched = <N>` — across the tree
+- `sources.ado.updates_fetched = <N>` — across the tree
+- `sources.ado.items_failed = <N>`
+- `sources.ado.watermark = now` (stream only)
 - For each week touched, add to `weekly_files` index.
 
 ## Stop conditions
 
-- Hint missing AND fuzzy resolution returns 0 candidates → ask user once, persist answer to mutable, continue.
-- Multiple plausible candidates → ask user to pick, persist answer.
-- WorkIQ fails AND host fallback fails AND Graph fails → write evidence file with `❌ all paths failed` marker, log to run-log errors, continue with rest of run.
+- Hard-prereq missing → refuse (see Hard prerequisites above).
+- Step 1 returns multiple Engagement candidates → ask user, never auto-pick.
+- Step 1 returns zero candidates → record `no-match` + WIQL + `next_step`, stop.
+- A child WI fetch fails → mark `❌ item-fetch-failed` in `tree.md`, continue tree.
+- Discussion paging fails mid-stream → record `❌ comments-partial — N/M fetched` and continue.

package/plugin/skills/pull-crm/SKILL.md

@@ -1,14 +1,23 @@
 ---
 name: "pull-crm"
-version: "2.0.0"
-description: "Pull CRM (Dataverse) evidence (snapshot: engagement record fields; stream: notes/activities/status changes). Conditional az login per az-auth-conditional rule."
+version: "2.3.1"
+description: "Pull CRM (Dataverse) evidence (snapshot: engagement record VERBATIM long-text fields + ALL annotations with formatted values; stream: notes/activities/status changes). Per-record explicit $select + formatted-value annotations. FDE entity sets use crm-field-manifest.md. v2.3.0: exhaustive 4-step resolution sequence (title → account[s] → wide-text → recent-slice → ask). Anti-patterns codified: no statecode filter, new_companyname is NOT a field, never disable from one shallow probe."
 ---
 
 # Skill: pull-crm
 
+
+> **v3.7.6 contracts** — This skill operates under four HARD-rule doctrines:
+> - `verbatim-by-default.instructions.md` — full bodies/notetext/fields by default; no preview-grade pulls accepted.
+> - `capture-learnings.instructions.md` — every fix/discovery is logged to `plugin/learnings/<source>.md` immediately.
+> - `cleanup-on-resolution.instructions.md` — when a value resolves, all stale `no-match` / `not yet` notes referencing the prior unresolved state must be rewritten in the same turn.
+> - `run-reports.instructions.md` — every refresh writes a per-user report under `Evidence/<alias>/refresh-reports/YYYY-MM-DD-HHMM_refresh.md`.
+
+> **Canonical evidence layout** (HARD, kushi v3.12.1+): all artifacts produced by this skill MUST be written under `<project>/Evidence/<alias>/<source>/{snapshot,stream,...}/` — sibling folders under `<project>/` (e.g. `<project>/<source>-context/`, `<project>/<source>/`, `<project>/_Weekly Summaries/`) are FORBIDDEN. See `evidence-layout-canonical.instructions.md`.
+
 Pulls **crm** evidence in two shapes per `snapshot-vs-stream.instructions.md`:
 
-- **snapshot/** — engagement record with every field the API returns (owner, status, account, dates, custom fields)
+- **snapshot/** — engagement record with EVERY field the API returns (owner, status, account, dates, custom fields), all long-text fields VERBATIM, plus every related annotation (note) verbatim
 - **stream/** — notes added, activities logged, field-level changes (old → new + actor + timestamp)
 
 WorkIQ-first per `workiq-first.instructions.md`. Thoroughness per `evidence-thoroughness.instructions.md`; runtime detector + auto-retry + paste-prompt per `thoroughness-detector.instructions.md`. Citations per `citation-ledger.instructions.md`. Side-by-side mutable hints written immediately on discovery per `side-by-side-config.instructions.md`.
@@ -24,19 +33,159 @@ WorkIQ-first per `workiq-first.instructions.md`. Thoroughness per `evidence-thor
 ## Resolution hints (pinned in mutable)
 
 - `crm.recordId` — pinned engagement record GUID (FE-NNNNN-style)
-- `crm.entitySetName` — Dataverse entity set name (e.g. `msdyn_engagements`)
+- `crm.entitySetName` — Dataverse entity set name (e.g. `msfre_engagementsubmissions`, `msdyn_engagements`)
+- `crm.fieldMap` — resolved logical-name map per entity (filled by discovery probe; cached)
+
+## Hard prerequisites (REQUIRED — see `scope-boundaries.instructions.md` Rule 3)
+
+This skill is **HARD-fail** without both:
+
+1. Global config: `<engagement-root>/.project-evidence/crm/config.yml` exists with non-placeholder `environmentUrl`.
+2. Per-project boundary: `<engagement-root>/<project>/integrations.yml#boundaries.crm.record_ids` OR `boundaries.crm.request_ids` is non-empty.
+
+If either is missing, refuse with the exact message:
+
+```
+crm-config-missing — drop a filled config.yml at
+<engagement-root>/.project-evidence/crm/config.yml. See template at
+plugin/templates/init/crm-config.template.yml. Then add boundaries.crm.record_ids
+(or request_ids) to <project>/integrations.yml.
+```
+
+**No "narrate from email" fallback exists in v3.7.0+.** CRM evidence comes from CRM
+or it is absent — Coverage Notes will say so explicitly. Synthesis across sources
+happens at consolidation, not at pull.
+
+## Field manifest dispatch
+
+Before issuing the snapshot fetch, inspect `crm.entitySetName`:
+
+- If it starts with `msfre_` (FDE engagement-submission family) → use `reference-packs/fde/crm-field-manifest.md` for the explicit `$select` list, the annotation `$expand`, and the snapshot file shape. The manifest IS the contract.
+- For any other entity → still use the per-record loop below, but discover the column list from EntityDefinitions metadata once (cache to `crm.fieldMap`) and `$select` every column found (no relying on default projection).
+
+## Dataverse retrieval doctrine (REQUIRED — applies to every REST call below)
+
+Borrowed from production CRM-sync hardening. Every Dataverse REST call in this skill MUST follow these rules, or option-set / lookup fields will render as raw GUIDs and integer codes instead of human labels:
+
+### Rule 1 — Always request formatted values
+
+```powershell
+$headers = @{
+  Authorization = "Bearer $token"
+  Accept = 'application/json'
+  'OData-Version' = '4.0'
+  'OData-MaxVersion' = '4.0'
+  Prefer = 'odata.include-annotations="OData.Community.Display.V1.FormattedValue"'
+}
+```
+
+This makes Dataverse return paired properties like `statuscode@OData.Community.Display.V1.FormattedValue` and `_ownerid_value@OData.Community.Display.V1.FormattedValue` alongside the raw codes/GUIDs.
+
+### Rule 2 — Use `PSObject.Properties[...]` accessor for `@`-named annotation properties
+
+Dotted access fails for property names containing `@`. Always use the indexer:
+
+```powershell
+$ownerDisplay = $row.PSObject.Properties['_ownerid_value@OData.Community.Display.V1.FormattedValue'].Value
+$status       = $row.PSObject.Properties['statuscode@OData.Community.Display.V1.FormattedValue'].Value
+```
+
+### Rule 3 — Inspect one live row before widening
+
+If field names are uncertain (new entity, schema drift, 400 error), fetch a single row first and dump property names. Do NOT keep iterating filters blindly:
+
+```powershell
+$url = "$baseUrl/api/data/$api/${entitySet}?`$orderby=modifiedon desc&`$top=1"
+$row = (Invoke-RestMethod -Uri $url -Headers $headers -Method Get).value | Select-Object -First 1
+$row.PSObject.Properties.Name | Sort-Object
+```
+
+Cache the discovered shape to `crm.fieldMap`.
+
+### Rule 4 — Add one field/filter at a time
+
+If a query returns `400 Could not find a property named ...`, REVERT to the last known-good query shape, then add ONE field/filter and rerun. Do not widen scope past the failure.
+
+### Rule 5 — Always include `statecode` and `statuscode` in candidate searches
 
-## Snapshot pass
+Otherwise inactive/closed records become invisible during ranking.
 
-Write `snapshot/<record-id>.md` per CRM record with EVERY field the API returns (don't editorialize). Last fetched timestamp at top.
+### Rule 6 — Prefer formatted values in rendered output
 
-Write to: `<engagement-root>/<project>/Evidence/<alias>/crm/snapshot/<entity>.md`
+Snapshot files render the formatted display value in the main body. Keep raw GUID / integer code only in `## All other fields` for diagnostic value.
 
-Use template: `templates/snapshot/crm-<kind>.template.md`
+## Resolution order (when `crm.recordId` is unset)
+
+If `boundaries.crm.record_ids` is empty AND a project token is provided, resolve in this order. Persist whichever step resolves to `m365Mutable.knownSections.<project>.crm.recordId` immediately.
+
+**CRITICAL anti-patterns — read before writing any filter:**
+
+- **NEVER add a `statecode` filter** to any candidate-resolution query. The default CRM list view may only show active records, hiding inactive / withdrawn / closed / deferred records. Always query all states and report `statecode`+`statuscode` formatted values per Rule 5 + Rule 6.
+- **`new_companyname` is NOT a valid attribute on `new_frontierengineeringtriage`.** The customer is a navigation lookup (`_new_customer_value`). Filtering on `new_companyname` will always return 0 and the 400 will look like an auth/scope failure. Always resolve customer via the `accounts` entity then `_new_customer_value eq <accountid>`.
+- **Never give up after one shallow probe.** This 4-step sequence is exhaustive — only step 4's user-ask is a stop. If WorkIQ is the only path tried, that is a defect; the Dataverse REST sequence below MUST be attempted before declaring no-match. See `plugin/instructions/crm-bootstrap-discovery.instructions.md`.
+
+1. **Title-first**: filter the entity by its title-equivalent field (FDE: `new_title` or `msfre_name`) `contains '<token>'`, ordered by `modifiedon desc`, top 20.
+2. **Account fallback**: if step 1 returns 0, resolve customer in `accounts` by `name contains '<token>'`. For EACH matching account, query the engagement entity by `_<customerLookup>_value eq <accountId>` (not just the first match — token like "Deere" can resolve to multiple accounts e.g. `JOHN DEERE`, `DEERE COMPANY`, `JOHN DEERE COASTRUCTION`).
+3. **Wide text fallback**: if both above fail, retry contains on the long-text fields most likely to mention the customer by name: `new_businessscenariotechnicalblocker`, `new_engagedwith`, `new_engagementobjectives` (FDE entity). One field per query; combine results.
+4. **Recent-slice client-rank**: if all above fail, pull the most recent 200 records (`$select` minimal: id, title, requestId, customer-formatted, statecode-formatted, statuscode-formatted, modifiedon) and rank client-side by needle match against title / request id / customer display.
+5. **Ask user**: present top 5 candidates (id, title, customer display, status, modifiedon). Never auto-pick on multi-match.
+
+After steps 1–4 all return 0, write the candidate-search trail (queries attempted + result counts) to `Evidence/<alias>/refresh-reports/<ts>_refresh.md#crm-resolution-attempts` so the next refresh has audit. Do NOT silently set `boundaries.crm.disabled: true` — only step 5 (user declines or confirms none) writes `disabled` with `reason: 'no-match-after-full-4-step-resolution-<date>'`.
+
+
+
+For EACH record id in `boundaries.crm.record_ids`:
+
+### Step A — Resolve field map (once per project, cached)
+
+If `m365Mutable.knownSections.<project>.crm.fieldMap` is empty, probe:
+
+```http
+GET /api/data/v9.2/EntityDefinitions(LogicalName='<entity-logical-name>')/Attributes?$select=LogicalName,DisplayName,AttributeType
+```
+
+Persist the resolved attribute list to `crm.fieldMap`. For FDE entities, cross-check against `reference-packs/fde/crm-field-manifest.md`.
+
+### Step B — Fetch the record with explicit $select + annotation $expand
+
+Headers per **Dataverse retrieval doctrine Rule 1** (formatted values).
+
+```http
+GET /api/data/v9.2/<entitySet>(<recordId>)?$select=<every-attribute-from-fieldMap>&$expand=Annotations($select=annotationid,subject,notetext,createdon,_createdby_value,filename,mimetype)
+Prefer: odata.include-annotations="OData.Community.Display.V1.FormattedValue"
+```
+
+WorkIQ-first equivalent (when host fallback to REST isn't available):
+
+```
+workiq ask -q "Fetch the FULL Dataverse record from entity set <entitySet> with id <recordId>. Return EVERY field VERBATIM (do not summarize, do not truncate any long-text/Memo field). Then fetch ALL Annotations (notes) related to this record (objectid eq <recordId>). For each annotation return: subject, full notetext verbatim, createdon, createdby name. Output as JSON or markdown table — but do not paraphrase any long-text content."
+```
+
+If the response is summarized (heuristics: any long-text field appears trimmed, an annotation count exceeds the rendered count, phrases like `"and N more notes"` appear): **retry once** with the wording `Return raw field values verbatim, every annotation in full. Do not summarize, do not paginate, do not omit.`
+
+### Step C — Write snapshot file (one per record)
+
+Write `snapshot/<entitySet>/<record-id>.md` with:
+
+- File header (record id, MSX opp, status, owner, last-fetched timestamp).
+- `## Source Basis` — tool used, boundary applied, fieldMap source, annotation count.
+- `## AI Narrative Summary` (3+ paragraphs — REQUIRED FIRST per `evidence-thoroughness.instructions.md`).
+- `## Engagement context` — every long-text field as a verbatim blockquote (see crm-field-manifest.md for the FDE shape).
+- `## All other fields` — 2-column table; empty fields shown as `_(empty)_` so absence is visible.
+- `## Notes (verbatim)` — one `### Note — <subject> (<createdon> by <createdby>)` block per annotation, full `notetext` as a verbatim blockquote, oldest first. NO truncation regardless of length.
+
+Write to: `<engagement-root>/<project>/Evidence/<alias>/crm/snapshot/<entitySet>/<record-id>.md`
+
+Use template: `templates/snapshot/crm-<kind>.template.md` (FDE entities: shape comes from `reference-packs/fde/crm-field-manifest.md`).
+
+### Step D — Failure handling
+
+- Per-record fetch failed → write the snapshot file with a `❌ record-fetch-failed` marker + `next_step: ask user to paste record fields` and continue to next record.
+- Annotation fetch failed but record fetched → write the record fields, mark `## Notes (verbatim)` with `❌ annotation-fetch-failed — N annotations expected` and continue.
 
 ## Stream pass
 
-Per `evidence-thoroughness.instructions.md`: every annotation verbatim with author + timestamp; every field-level change with old → new + actor + timestamp. Bucketed by ISO week into `stream/<record-id>/<YYYY-MM-DD>_crm-stream.md`.
+Per `evidence-thoroughness.instructions.md`: stream files (per-week) start with an **AI Narrative Summary** covering what changed and what it means, then every annotation verbatim with author + timestamp + every field-level change with old → new + actor + timestamp. Bucketed by ISO week into `stream/<record-id>/<YYYY-MM-DD>_crm-stream.md`.
 
 Write to: `<engagement-root>/<project>/Evidence/<alias>/crm/stream/<YYYY-MM-DD>_crm-stream.md` (date = Monday of the ISO week the events fall in).
 
@@ -46,8 +195,8 @@ If a week file already exists, MERGE (dedupe by event ID, append new events, kee
 
 ## Tools (in order)
 
-1. **WorkIQ** — ````$WorkIQQuery`````
-2. **Host fallback** — Dataverse REST Web API via `az account get-access-token` against the configured tenant (read connection from `<engagement-root>/.project-evidence/crm/config.yml`)
+1. **Dataverse REST Web API** (preferred for snapshot — gives explicit `$select` + `$expand` control) via `az account get-access-token` against the configured tenant. Read connection from `<engagement-root>/.project-evidence/crm/config.yml`.
+2. **WorkIQ** — only when REST is unavailable; phrase queries to demand verbatim long-text + every annotation (see Step B template above). WorkIQ summarizes by default, so use this path only as fallback.
 3. **Graph REST** — last resort, soft-fail per `az-auth-conditional.instructions.md`.
 4. **Ask user** — paste verbatim source content if all above fail.
 
@@ -57,19 +206,23 @@ Document which path succeeded under `## Source Basis` in each output file.
 
 If discovered, immediately write to `m365Mutable.knownSections.<project>.<source>` with `discoveredOn` + `confidence`:
 
-- `crm.recordId` — pinned engagement record GUID (FE-NNNNN-style)
-- `crm.entitySetName` — Dataverse entity set name (e.g. `msdyn_engagements`)
+- `crm.recordId` — pinned engagement record GUID
+- `crm.entitySetName` — Dataverse entity set name
+- `crm.fieldMap` — resolved attribute list (from EntityDefinitions probe)
 
 ## Update run-log
 
 After successful pass:
 - `sources.crm.last_pulled = now`
 - `sources.crm.watermark = now` (stream only; snapshot has no watermark)
-- `sources.crm.item_count = <N>`
+- `sources.crm.records_fetched = <N>`
+- `sources.crm.annotations_fetched = <N>` — total notes captured across all records
+- `sources.crm.records_failed = <N>`
 - For each week touched, add to `weekly_files` index.
 
 ## Stop conditions
 
+- Hard-prereq missing → refuse (see Hard prerequisites above).
 - Hint missing AND fuzzy resolution returns 0 candidates → ask user once, persist answer to mutable, continue.
 - Multiple plausible candidates → ask user to pick, persist answer.
-- WorkIQ fails AND host fallback fails AND Graph fails → write evidence file with `❌ all paths failed` marker, log to run-log errors, continue with rest of run.
+- All paths failed for a given record → write evidence file with `❌ all paths failed` marker, log to run-log errors, continue with rest of run.