@fluentcommerce/ai-skills 0.1.0

package/content/dev/skills/fluent-trace/SKILL.md

@@ -0,0 +1,1143 @@
+---
+name: fluent-trace
+description: Trace and debug Fluent Commerce event processing. Analyze failed events, correlate with workflows and rules, identify root causes. Triggers on "trace event", "debug order", "why did event fail", "event trace", "debug fulfilment".
+user-invocable: true
+allowed-tools: Bash, Read, Write, Edit, Glob, Grep
+argument-hint: <entity-ref> [--entity-type ORDER|FULFILMENT] [--status FAILED]
+---
+
+# Event Trace & Debug
+
+Trace Fluent Commerce event processing to diagnose failures, identify root causes, and correlate events with workflow rulesets and rules.
+
+## Ownership Boundary
+
+This skill owns diagnostic workflow, correlation logic, and decision trees.
+
+Canonical MCP extension tool syntax/limits are owned by `/fluent-mcp-tools`. Reuse that skill for exact request payload patterns.
+
+## Pre-Check: Load Source Analysis
+
+Before tracing, check if `/fluent-custom-code` analysis artifacts exist:
+
+```
+accounts/<PROFILE>/analysis/custom-code/workflow-rule-map.json
+accounts/<PROFILE>/analysis/custom-code/source-map.json
+```
+
+If found, use them to:
+- Map rule names in event traces to actual source classes and file paths
+- Understand rule parameters and expected behavior
+- Identify confidence level of rule mappings (exact vs inferred)
+- Jump directly to source when a rule fails
+
+If not found, fall back to workflow JSON analysis and runtime evidence only. Treat artifact-gate failures as "not found" (for example missing required files, missing hashes, or blocking `missingSources` in `constraints.json`).
+
+## When to Use
+
+- An event was sent but the entity didn't change state
+- Event processing returned FAILED status
+- Need to understand why an order/fulfilment is stuck
+- Investigating which ruleset processed (or didn't process) an event
+- Debugging rule execution errors
+
+### Handoff from `/fluent-e2e-test`
+
+When invoked after a failed E2E step, accept and reuse the handoff payload when provided:
+- `entityRef`, `entityType`
+- `eventId` (optional)
+- expected vs actual status
+
+If `eventId` is already known, go straight to `event.get` and avoid redundant listing.
+
+### API Constraints to Remember
+
+- **Archival window:** Events are retained for ~4 months. Queries beyond this return empty results.
+- **Rate limiting:** ~100 requests/minute per user. When paginating large audit trails, add brief pauses between calls.
+- **Max count:** MCP `event.list` caps at 500 per request; REST API supports up to 5,000 but may timeout.
+- **Recommended time range:** 5 minutes per query for high-volume entities. Use narrow `from`/`to` windows.
+
+## Quick Start: One-Call Forensics
+
+For most debugging scenarios, start with `event.flowInspect`:
+
+```
+event.flowInspect({
+  rootEntityRef: "<ORDER_REF>",
+  rootEntityType: "ORDER"
+})
+```
+
+**Compact mode (default):** Returns a pre-analyzed summary (~2-3k tokens) with an `analysis` section containing `statusFlow`, `findings` (anomaly detection), `failedWebhookEndpoints`, and `slowestRulesets`. Audit arrays are stripped to essentials — only failed webhooks, top 5 mutations by name (count only), and top 5 slowest rulesets. Diagnostics show summary inspected events (id, name, status, closeMatch count).
+
+**IMPORTANT — Context budget discipline:**
+- ALWAYS start with compact mode (the default). The ~2-3k token summary is sufficient for 90% of debugging.
+- NEVER call `compact: false` as a first step — it returns ~24-30k tokens and fills context.
+- If compact findings point to a specific issue, drill down with ONE targeted flag at a time (keep `compact: true`):
+  - Rule timing issue → add `includeRuleDetails: true`
+  - Need custom log messages → add `includeCustomLogs: true`
+  - Need entity state snapshots → add `includeSnapshots: true`
+  - Need child entity events → add `includeCrossEntity: true`
+- Only use `compact: false` as a last resort when you need raw arrays and have already identified what to look for.
+
+Example (targeted drill-down for rule timing — keeps compact summary, adds rule details):
+```
+event.flowInspect({
+  rootEntityRef: "<ORDER_REF>",
+  rootEntityType: "ORDER",
+  includeRuleDetails: true
+})
+```
+
+**Response sections (compact mode):**
+- `totals` — raw event counts (always present)
+- `analysis` — local findings: `statusFlow`, `entityTypes`, `timespan`, `failedWebhookEndpoints`, `slowestRulesets`, `findings[]`
+- `orchestration` — statusCounts, entityTypeCounts, topNames (top 10, no events array)
+- `audit.exceptions` — rule exceptions (critical, always included)
+- `audit.webhookActions` — only failures (responseCode >= 400)
+- `audit.mutationActions` — top 5 by queryName (name + count only)
+- `audit.sendEventActions` — count + scheduledCount only
+- `diagnostics` — summary inspected events (no full event payload)
+
+**Additional sections (available via targeted flags, all work with compact: true):**
+- `audit.ruleEvents` — per-rule execution details (add `includeRuleDetails: true`)
+- `audit.customLogs` — custom plugin logs (add `includeCustomLogs: true`)
+- `audit.snapshots` — entity snapshots (add `includeSnapshots: true`)
+- `diagnostics.noMatchAuditDetails` — closeMatches from ruleSet audit (add `includeNoMatchDetails: true`, on by default)
+- `crossEntity` — child entity events (add `includeCrossEntity: true`)
+
+See `/fluent-mcp-tools` for full parameter reference.
+
+Use the manual step-by-step workflow below when you need finer control over individual queries or when investigating a specific event ID.
+
+## Diagnostic Workflow (Manual)
+
+### Step 1: Get Entity Current State
+
+Use MCP `graphql.query` to fetch the entity's current status and key attributes:
+
+**For an ORDER:**
+```graphql
+{
+  orders(ref: ["<ORDER_REF>"], first: 1) {
+    edges {
+      node {
+        id
+        ref
+        type
+        status
+        createdOn
+        updatedOn
+        attributes {
+          name
+          type
+          value
+        }
+        fulfilmentChoice {
+          edges {
+            node {
+              id
+              ref
+              type
+              status
+              deliveryType
+            }
+          }
+        }
+        items {
+          edges {
+            node {
+              ref
+              quantity
+              status
+              fulfilmentChoiceRef
+            }
+          }
+        }
+      }
+    }
+  }
+}
+```
+
+**For a FULFILMENT:**
+```graphql
+{
+  fulfilments(ref: ["<FULFILMENT_REF>"], first: 1) {
+    edges {
+      node {
+        id
+        ref
+        type
+        status
+        createdOn
+        updatedOn
+        attributes {
+          name
+          type
+          value
+        }
+        items {
+          edges {
+            node {
+              ref
+              status
+              requestedQuantity
+              filledQuantity
+            }
+          }
+        }
+        order {
+          id
+          ref
+          status
+        }
+        fromLocation {
+          ref
+          name
+        }
+      }
+    }
+  }
+}
+```
+
+### Step 2: Fetch Event History
+
+Use MCP `event.list` to get all events for this entity:
+
+```
+event.list({
+  "context.entityRef": "<ENTITY_REF>",
+  "context.entityType": "ORDER",
+  "count": 50
+})
+```
+
+**Filter by failed events:**
+```
+event.list({
+  "context.entityRef": "<ENTITY_REF>",
+  "context.entityType": "ORDER",
+  "eventStatus": "FAILED",
+  "count": 50
+})
+```
+
+**Filter by time window:**
+```
+event.list({
+  "context.entityRef": "<ENTITY_REF>",
+  "from": "2026-02-20T00:00:00Z",
+  "to": "2026-02-20T23:59:59Z",
+  "count": 100
+})
+```
+
+For full event-model/filter semantics and causality patterns (`context.sourceEvents`, root-entity tracing), use `/fluent-event-api`.
+
+### Step 3: Inspect Failed Event
+
+Use MCP `event.get` with the event ID from step 2:
+
+```
+event.get({ "eventId": "<EVENT_ID>" })
+```
+
+Look for:
+- `status` — FAILED, PENDING, SUCCESS
+- `response.errors` — Error messages from rule execution
+- `context.entityType` + `context.entityRef` — Which entity was targeted
+- `context.rootEntityType` + `context.rootEntityRef` — Parent entity context
+- `attributes` — What data was sent with the event
+
+### Step 4: Correlate with Workflow
+
+Given the event name and entity status at time of event, identify which ruleset should have processed it:
+
+1. **Download the workflow** via MCP `workflow.download` or read local copy
+2. **Find the matching ruleset:**
+   - If the event name matches a ruleset name → that ruleset should fire
+   - If the entity just entered a new status → find ruleset with `triggers: [{ "status": "<STATUS>" }]`
+3. **Check the rules** in that ruleset — identify which rule likely failed
+
+### Step 5: Identify Root Cause
+
+Common failure patterns:
+
+| Symptom | Likely Cause | How to Verify |
+|---------|-------------|---------------|
+| Event FAILED, no state change | Rule threw exception | Check event response errors |
+| Event SUCCESS but no state change | Rule ran but SetState was conditional | Check gate conditions, verify child entity statuses |
+| Event PENDING for a long time | Async processing queue backlog | Wait and re-check, or send again |
+| Event NO_MATCH | No ruleset matched event name or status trigger | Verify event name matches a ruleset name in the workflow; check entity subtype matches workflow flexType |
+| Event SUCCESS but wrong state | Ruleset executed unexpected rules | Check trigger status matches expected |
+| Entity stuck in intermediate state | Gate condition not met | Query all child entities (FCs, fulfilments) and check their statuses |
+| Event not found at all | Event sent to wrong entity/ref | Verify entityRef, entityType, retailerId |
+| retailerId mismatch | Event sent with wrong retailerId | Check user scope — account-level users get retailerId=0 |
+
+### Step 5B: Causal Chain Reconstruction
+
+> **Shortcut:** `event.flowInspect` with `includeRuleDetails: true` returns the full audit chain (ruleSet durations, rule-level execution with props/timing, ACTION payloads) in one call. Use the manual approach below only when you need per-event-ID granularity.
+
+When `sourceEvents` is sparse or missing, reconstruct the causal chain using correlation:
+
+```
+1. Get the ORCHESTRATION event that was sent:
+   event.get({ eventId: "<EVENT_ID>" })
+
+2. Find all audit events it spawned (time-window correlation):
+   event.list({
+     eventType: "ORCHESTRATION_AUDIT",
+     "context.entityRef": "<ENTITY_REF>",
+     from: "<event_generatedOn_minus_1s>",
+     to: "<event_generatedOn_plus_30s>",
+     count: 100
+   })
+
+3. Group audit events by category:
+   - category=ruleSet → ruleset start/stop (check startTimer/stopTimer)
+   - category=rule → individual rule execution
+   - category=ACTION → rule actions (SendEvent, SetState, Send Webhook)
+   - category=exception → failure point
+
+4. Build execution timeline:
+   ORCHESTRATION event → ruleSet audit → rule audits → ACTION audits → next ORCHESTRATION (if SendEvent)
+
+5. Identify the break point:
+   - Last successful rule before exception
+   - Missing expected ACTION (e.g., no SetState audit = state didn't change)
+   - ORCHESTRATION_AUDIT with category=exception = the failure
+```
+
+**When sourceEvents IS available:** Follow `sourceEvents[0]` upward to find the parent event. Repeat until you reach an event with empty `sourceEvents` (the root cause event).
+
+### Step 5C: Correlation Confidence Ladder
+
+When multiple candidate links exist, rank correlation confidence in this order:
+
+1. **Direct reference (highest confidence, score ~0.80)**
+   Child event `context.sourceEvents[]` contains the parent event ID.
+2. **Entity + timing match (medium confidence, score ~0.15)**
+   Same `entityType` + same entity identifier (`entityId`/`entityRef`) within **1ms timing gap**.
+3. **Root-context + sequence match (medium-low confidence, score ~0.05)**
+   Same `rootEntityType/rootEntityRef` and expected sequence (`ruleSet` -> `rule` -> `ACTION`).
+4. **Name-only inference (low confidence, score 0)**
+   Event/ruleset names suggest linkage but no causal pointer.
+
+**Scoring:** When scoring multiple candidates, sum applicable factors (max 1.0). Direct reference (0.80) + entity match (0.15) + timing proximity within 100μs (0.05) = 1.0 (perfect). Use this to pick the best match when multiple candidates exist.
+
+**Timing threshold:** Use **1 millisecond** as the maximum gap for timing-based correlation. Events further apart than 1ms on the same entity are unlikely to be causally linked via the same rule execution.
+
+Always tag conclusions as **confirmed** (1) or **inferred** (2-4), and include the evidence used.
+
+For full event model and filter semantics → `/fluent-event-api`.
+
+### Step 5D: Payload Evidence Bundle (Mutations, Webhooks, Scheduled)
+
+> **Shortcut:** `event.flowInspect` extracts all three categories below automatically: `audit.mutationActions`, `audit.webhookActions`, `audit.sendEventActions` / `audit.scheduledActions`. Use the manual approach below only for targeted queries outside the root-entity scope.
+
+After reconstructing the chain, extract payload evidence from `category=ACTION` events:
+
+1. Query:
+   ```
+   event.list({
+     "eventType": "ORCHESTRATION_AUDIT",
+     "category": "ACTION",
+     "context.rootEntityRef": "<ENTITY_REF>",
+     "count": 200
+   })
+   ```
+2. Mutation extraction (often from ACTION events with missing `name`):
+   - `attributes.request.queryType` (look for `DynamicUpdateMutation`)
+   - `attributes.request.queryName`
+   - `attributes.request.queryString`
+   - `attributes.request.variables.values.input` (actual mutation payload)
+   - `attributes.response.inner`
+3. Webhook extraction:
+   - `Request Endpoint`, `Request Headers`
+   - `Response code`, `Response reason`, `Response Body`, `Response Headers`
+4. Scheduled/future-dated extraction:
+   - `attributes["Future Dated"]`
+   - `attributes.Event.scheduledOn`
+   - `event.get(eventId).event.meta.scheduledOn`
+5. Diagnose event-status mismatches:
+   - If `ORCHESTRATION` shows `PENDING` but downstream ACTION/ruleSet audit completed, treat audit chain as runtime truth and flag status-lag behavior.
+
+### Step 6: Deep Dive — Rule-Level Analysis
+
+### Step 6A: Evidence Escalation When Logic Is Vague
+
+Do not rely on ruleset descriptions alone when rule intent is unclear.
+Escalate evidence in this order:
+
+1. **Ask focused clarification questions** (event name, expected state, actual state, and where it diverged)
+2. **Request source first** (`module.json` + Java rule classes)
+3. **If source is unavailable, request compiled artifacts** (module ZIP/JAR)
+4. **Decompile candidate rule classes** and map behavior back to workflow props and event errors
+5. **Correlate with runtime evidence** (`event.get`, `event.list`, entity GraphQL state)
+6. **Report confidence explicitly** (confirmed vs inferred behavior)
+
+Use this escalation whenever:
+- rule/ruleset descriptions are generic ("validation", "process", "update")
+- multiple rules can match the same symptom
+- event error output does not directly identify the failing line of logic
+
+### Step 6B: Rule Source Analysis
+
+If you identified the failing rule, check its source code:
+
+1. **Find the rule class** — Map rule name to Java file:
+   - `ACCT.MODULE.RuleName` → `rule/<package>/RuleName.java`
+2. **Check required props** — Does the workflow supply all props the rule expects?
+   - `@ParamString(name = "xxx")` annotations define expected props
+   - Props are CASE-SENSITIVE
+3. **Check dynamic query paths** — If rule uses `DynamicUtils.query()`, are the paths valid?
+4. **Check mutation types** — If rule uses `DynamicUtils.mutate()`, is the input type correct?
+5. **Check setting references** — If rule reads a Setting, does the Setting exist with correct data?
+   - Query via MCP: `graphql.query` with `settings(first: 1, context: "RETAILER", contextId: <RETAILER_ID>, name: ["<setting-name>"])`
+
+### Step 6C: Decompiled Analysis (No Source Case)
+
+**Prerequisite:** Check if decompiled output exists at `accounts/<PROFILE>/SOURCE/.decompiled/`. If empty or missing:
+- Run `/fluent-custom-code <PROFILE>` — it will decompile any JARs found under `SOURCE/`
+- If no JARs are available either, ask the user:
+  > "To analyze this rule's implementation, I need the module JAR. You can:
+  > 1. Copy the JAR to `accounts/<PROFILE>/SOURCE/` and re-run `/fluent-custom-code <PROFILE>`
+  > 2. Run `/fluent-connect` which will guide you through full workspace preparation including JAR decompilation
+  > 3. Continue with `plugin.list` data only (rule description and parameters, no implementation detail)"
+- For reference modules (core, order, fulfilment, commonv2, globalinventory, reporting), the JAR can typically be obtained from the Fluent module distribution or build artifacts
+
+When only JAR/ZIP artifacts are available:
+
+1. Extract class names and identify likely rule classes from `module.json` rule mappings.
+2. Decompile only the suspected classes first (avoid whole-jar noise).
+3. Confirm:
+   - which props are read
+   - what GraphQL mutations/queries/events are invoked
+   - what status transitions or guards are enforced
+4. Reconcile decompiled findings with workflow ruleset props and observed event failures.
+5. Mark any uncertain interpretation (obfuscation/inlined code) before recommending fixes.
+6. Use `accounts/<PROFILE>/SOURCE/.decompiled/<artifact-name>/` and write `DECOMPILED.md` with source artifact path + decompiler used.
+7. Treat JAR/ZIP artifacts and `.decompiled/` output as read-only evidence. Do not recommend editing decompiled code; recommend source onboarding for implementation changes.
+
+### Step 6D: Performance Bottleneck Detection
+
+When investigating slow processing (not failures), calculate execution duration per step from event timestamps:
+
+1. **Collect timing data** — For each event in the causal chain, extract `eventTime` (ISO-8601 with microsecond precision when available).
+2. **Calculate step durations** — Duration between consecutive events in the chain (parent → child).
+3. **Identify bottlenecks** — Flag any step whose duration exceeds **2× the average** step duration.
+4. **Classify severity**:
+   - Processing time <1s = normal
+   - Processing time 1–5s = slow (investigate rule complexity, external calls)
+   - Processing time >5s = critical bottleneck (likely external system wait: webhook, inventory query, payment gateway)
+5. **Report** — List steps ranked by duration, flag bottlenecks, and correlate with rule source if available.
+
+### Step 7: Workflow Coverage Analysis (Static vs Dynamic)
+
+After tracing an entity's events (Steps 1–6), overlay the trace against the workflow definition to find divergence — rulesets that should have fired but didn't, or unexpected rulesets that appeared.
+
+#### When to Use
+
+- After tracing an entity that's stuck or behaved unexpectedly
+- After E2E test pass/fail to verify the full expected path was executed
+- During go-live readiness to confirm workflow coverage on real orders
+
+#### Coverage Algorithm
+
+1. **Reconstruct the entity's status path** from the event trace (Step 2 data):
+   ```
+   Extract all status transitions from events:
+     CREATED → ON_VALIDATION → BOOKED → PICK_PACK → COMPLETE
+   Record which event name triggered each transition.
+   ```
+
+2. **Load the workflow definition** (local file or `workflow.download`):
+   ```
+   Parse all rulesets for this entity type.
+   For each status in the entity's actual path, collect:
+     a. Status-triggered rulesets (triggers[].status matches the status)
+     b. Event-name-triggered rulesets (name matches a SendEvent from the previous ruleset's rules)
+   ```
+
+3. **Build expected ruleset sequence** from static analysis:
+   ```
+   Status: CREATED
+     Expected: CREATE ruleset → rules: [CreateVariants, UpdateStatusHistory, SendWebhook, SendEvent(ValidateOrder)]
+     Expected downstream: ValidateOrder ruleset
+
+   Status: ON_VALIDATION
+     Expected: status-triggered rulesets + event-triggered rulesets from previous step's SendEvents
+   ```
+
+4. **Compare expected vs actual**:
+   ```
+   For each expected ruleset:
+     Search event trace for ORCHESTRATION_AUDIT with category=ruleSet matching this name
+     If found → MATCHED
+     If not found → MISSED
+
+   For each ruleset in the event trace:
+     Check if it appears in the expected sequence
+     If not → UNEXPECTED
+   ```
+
+5. **Analyze missed rulesets** — for each MISSED ruleset, determine why:
+   - Triggering event never sent → upstream rule didn't fire `SendEvent`
+   - Status trigger never reached → entity took a different branch
+   - Conditional rule (ForwardIf*, gate) evaluated to false → check condition props
+   - Setting missing caused the rule to skip → cross-reference with `/fluent-settings` audit
+   - Entity subtype mismatch → ruleset scoped to a different flexType
+
+#### Coverage Report Format
+
+```
+=== Workflow Coverage: ORDER ref=HD-20260222-001 ===
+
+Status Path: CREATED → ON_VALIDATION → BOOKED → PICK_PACK → COMPLETE
+
+Expected Rulesets     Actual (from trace)    Status
+──────────────────────────────────────────────────────────────
+CREATE                CREATE                 MATCHED
+ValidateOrder         ValidateOrder          MATCHED
+ConfirmValidation     ConfirmValidation      MATCHED
+ProcessOrder          ProcessOrder           MATCHED
+BookOrder             BookOrder              MATCHED
+NotifyOrderBooked     (not found)            MISSED → setting "webhook.order.booked" not found
+ConfirmPick           ConfirmPick            MATCHED
+ConfirmShipment       ConfirmShipment        MATCHED
+
+Unexpected:
+  RetryValidation  → fired at ON_VALIDATION (ScheduleEvent timeout handler)
+
+Coverage: 7/8 expected rulesets matched (87.5%)
+Missed: 1 (setting dependency — fixable via /fluent-settings)
+Unexpected: 1 (timeout handler — normal for slow processing)
+```
+
+#### Interpreting Results
+
+| Result | Meaning | Action |
+|--------|---------|--------|
+| **High coverage (>90%)** | Workflow executing as designed | No action needed |
+| **Missed rulesets with setting cause** | Settings gap — rule skipped silently | Run `/fluent-settings --audit` and create missing settings |
+| **Missed rulesets with condition cause** | Conditional branch not taken | Verify entity data meets the condition (attributes, child statuses) |
+| **Unexpected rulesets** | Timer/retry handlers or misconfigured triggers | Check if ruleset is a ScheduleEvent target (normal) or a trigger conflict (bug) |
+| **Low coverage (<70%)** | Workflow path fundamentally different from expected | Re-examine entity subtype, workflow version, and event sequence |
+
+#### Integration
+
+- Use `/fluent-workflow-analyzer` output for the expected ruleset sequence (static analysis)
+- Use Steps 1–5 event trace data for actual execution evidence (dynamic analysis)
+- Use `/fluent-settings --audit` to explain missing-setting causes for MISSED rulesets
+- Hand off MISSED rulesets to `/fluent-settings` for remediation or `/fluent-workflow-builder` for workflow fixes
+
+## Detailed Event Data Extraction
+
+After diagnosing an issue or completing an E2E test, extract specific data types from the event audit trail for comprehensive analysis. These sections go beyond root-cause diagnosis — they produce structured inventories of what the orchestration engine actually did.
+
+> **One-call alternative:** Use `event.flowInspect` to extract all sections below (8A–8F) in a single API call. It returns mutations, webhooks, scheduled dispatches, NO_MATCH diagnostics with closeMatches, custom rule logs, entity snapshots, rule execution props, and exception details — all keyed by root entity. The manual patterns below remain useful when you need to customize extraction or filter beyond what flowInspect provides.
+
+### 8A: Mutation Audit Trail
+
+Extract every GraphQL mutation the workflow executed, with the exact query string and input variables. Mutation ACTION events have a **null name** and contain the full mutation payload.
+
+#### Query
+
+```
+event.list({
+  "eventType": "ORCHESTRATION_AUDIT",
+  "context.rootEntityRef": "<ENTITY_REF>",
+  "count": 200
+})
+```
+
+Filter results where `name` is null and `attributes` contains `request.queryString`.
+
+#### Data Fields
+
+| Attribute | Description |
+|-----------|-------------|
+| `request.queryString` | Full GraphQL mutation string (e.g., `mutation updateFulfilment($input: UpdateFulfilmentInput!) { ... }`) |
+| `request.inputName` | Input type name (e.g., `UpdateFulfilmentInput`) |
+| `request.queryType` | Mutation classification (e.g., `DynamicUpdateMutation`) |
+| `request.variables.values.input` | Exact input payload — entity ID, status changes, attribute updates |
+
+#### Report Format
+
+```
+=== Mutation Audit Trail: ORDER ref=HD-20260222-001 ===
+
+#   Entity          Mutation Type           Target Status    Attributes Changed
+─────────────────────────────────────────────────────────────────────────────────
+1   ORDER:143       updateOrder             ORDER_SPLIT      status
+2   FC:89           updateFulfilmentChoice  AWAITING_FC      status
+3   FULFILMENT:201  updateFulfilment        CREATED          status, fromAddress, toAddress, deliveryType
+4   FULFILMENT:201  updateFulfilment        RECEIVED         status, attributes[allocatedAt]
+5   ORDER:143       updateOrder             IN_PROGRESS      status
+6   FULFILMENT:201  updateFulfilment        PICKPACK         status, attributes[pickedAt]
+7   FULFILMENT:201  updateFulfilment        SHIPPED          status, attributes[trackingNumber, carrierRef]
+
+Total: 7 mutations across 3 entity types
+```
+
+#### Use Cases
+
+- **Audit compliance**: Verify every state change has a corresponding mutation record
+- **Debugging stuck entities**: Identify if a SetState mutation was attempted but failed
+- **Change impact analysis**: See exactly which attributes were modified and in what order
+- **Rule behavior verification**: Confirm a rule is constructing the correct mutation input
+
+### 8B: Webhook Delivery Report
+
+Extract full webhook request/response round-trips from ACTION events named "Send Webhook". This supplements the existing Webhook Failure Triage section with a comprehensive delivery inventory.
+
+#### Query
+
+```
+event.list({
+  "eventType": "ORCHESTRATION_AUDIT",
+  "context.rootEntityRef": "<ENTITY_REF>",
+  "count": 200
+})
+```
+
+Filter results where `name == "Send Webhook"`.
+
+#### Data Fields
+
+| Attribute | Description |
+|-----------|-------------|
+| `Request Endpoint` | Target URL the webhook was sent to |
+| `Request Headers` | HTTP headers sent (may include auth tokens) |
+| `Response code` | HTTP response status (200=OK, 404=not found, 0=timeout) |
+| `Response Body` | Full response body from the target server |
+| `Response Headers` | Response headers (rate-limit, content-type) |
+| `Response reason` | HTTP status text (e.g., "Not Found", "OK") |
+| Dynamic attributes | Rule-injected attributes (e.g., `orderType`, `orderId`, `orderStatus`, `eventName`, `fulfilmentId`) |
+
+#### Report Format
+
+```
+=== Webhook Delivery Report: ORDER ref=HD-20260222-001 ===
+
+#   Ruleset                      Endpoint                                        Status   Duration
+────────────────────────────────────────────────────────────────────────────────────────────────────
+1   OrderCreatedWebhook          https://api.example.com/order/created           200 OK   120ms
+2   FulfilmentCreatedWebhook     https://api.example.com/fulfilment/created      200 OK   95ms
+3   FulfilmentShippedWebhook     https://api.example.com/fulfilment/shipped      404 NF   45ms  ← FAIL
+4   OrderCompletedWebhook        https://api.example.com/order/completed         200 OK   110ms
+
+Summary: 3/4 delivered successfully, 1 FAILED (404)
+Failed webhooks: FulfilmentShippedWebhook → check setting "webhook.fulfilment.shipped"
+```
+
+#### Dynamic Attributes Extraction
+
+Webhook ACTION events also contain dynamic attributes injected by the rule at execution time. These represent the payload data sent alongside the webhook:
+
+```
+Dynamic attributes for webhook #1 (OrderCreatedWebhook):
+  orderType: HD
+  orderId: 143
+  orderStatus: CREATED
+  eventName: CREATE
+  fulfilmentChoiceRef: FC-HD-20260222-001
+  sourceSystem: E2E_TEST
+```
+
+These dynamic attributes are valuable for:
+- Reconstructing what the external system received
+- Verifying the correct data was included in the webhook payload
+- Debugging webhook receiver failures when the response body is unhelpful
+
+#### Cross-Reference with Settings
+
+Compare each webhook's `Request Endpoint` with the configured setting value:
+
+```
+1. Extract setting name from workflow rule props (e.g., "webhook.order.created")
+2. Query: settings(name: ["webhook.order.created"], context: "RETAILER")
+3. Parse lobValue.url from the setting
+4. Compare with Request Endpoint from the ACTION event
+5. If different → CONTRACT_MISMATCH (setting was changed after the event was processed, or wrong setting was loaded)
+```
+
+### 8C: Scheduled Event Inventory
+
+Extract all scheduled (future-dated) events from ACTION events named "Send Event". These are events dispatched by `ScheduleEvent` rules with a time delay.
+
+#### Query
+
+```
+event.list({
+  "eventType": "ORCHESTRATION_AUDIT",
+  "context.rootEntityRef": "<ENTITY_REF>",
+  "count": 200
+})
+```
+
+Filter results where `name == "Send Event"`.
+
+#### Data Fields
+
+| Attribute | Description |
+|-----------|-------------|
+| `Event Name` | Name of the scheduled event (e.g., `ValidationGraceExpired`) |
+| `Event` | Full event object: ID, meta, attributes, entityType, rootEntity info |
+| `Future Dated` | `true` if the event is scheduled for the future, `false` if immediate |
+| `scheduledOn` | Epoch milliseconds when the event will fire (only present when Future Dated is true) |
+
+#### Report Format
+
+```
+=== Scheduled Event Inventory: ORDER ref=HD-20260222-001 ===
+
+#   Event Name                  Entity Type    Future Dated   Scheduled For            Delay
+───────────────────────────────────────────────────────────────────────────────────────────────
+1   ValidateOrder               ORDER          false          (immediate)              0s
+2   ValidationGraceExpired      ORDER          true           2026-02-22T13:43:30Z     30s
+3   ProcessOrder                ORDER          false          (immediate)              0s
+4   NotifyFCComplete            FC             false          (immediate)              0s
+5   SourceOrder                 ORDER          false          (immediate)              0s
+
+Future-dated events: 1 of 5
+Scheduled timers: ValidationGraceExpired (30s delay)
+```
+
+#### Scheduled Event Analysis
+
+For each future-dated event:
+1. **Calculate actual delay** — `scheduledOn` minus the parent event's `generatedOn` timestamp
+2. **Check if it fired** — Search for the matching ORCHESTRATION event after `scheduledOn`
+3. **Check if it was pre-empted** — If a status change occurred before `scheduledOn`, the scheduled event may arrive at a status where no ruleset matches (NO_MATCH is expected)
+4. **Timer health** — Summarize which scheduled events fired vs were pre-empted
+
+### 8D: NO_MATCH Deep Diagnostics
+
+When a ruleset evaluates but no rule matches (NO_MATCH), the ruleSet audit event contains structured diagnostic data explaining why. This goes beyond the simple "NO_MATCH" detection in Step 5.
+
+#### Query
+
+```
+event.list({
+  "eventType": "ORCHESTRATION_AUDIT",
+  "context.rootEntityRef": "<ENTITY_REF>",
+  "count": 200
+})
+```
+
+Filter results where `category == "ruleSet"` and attributes contain `message` with "no match" or where `closeMatches` is present.
+
+#### Data Fields
+
+| Attribute | Description |
+|-----------|-------------|
+| `message` | Summary: "No rules matched for the event and its entity" |
+| `entityStatus` | Entity status when the ruleset was evaluated |
+| `closeMatches` | Array of rulesets that almost matched, with structured mismatch reasons |
+| `closeMatches[].name` | Ruleset name that almost matched |
+| `closeMatches[].mismatchReasons` | Array of specific reasons: subtype mismatch, status mismatch, etc. |
+| `lastRule` | Last rule that was evaluated before NO_MATCH |
+| `lastRuleSet` | Last ruleset that was evaluated |
+
+#### Report Format
+
+```
+=== NO_MATCH Diagnostics: ORDER ref=HD-20260222-001 ===
+
+Event: ValidationGraceExpired → NO_MATCH
+
+Entity status at evaluation: ON_VALIDATION
+Close matches:
+  1. ProcessOrder
+     Mismatch: subtype=HD required, entity subtype=CC
+  2. CancelValidation
+     Mismatch: status=CANCELLED required, entity status=ON_VALIDATION
+
+Diagnosis: ValidationGraceExpired fired after the entity already advanced past ON_VALIDATION
+           (pre-empted by ConfirmValidation). This is EXPECTED for timer events.
+
+---
+Event: RetryFulfilment → NO_MATCH
+
+Entity status at evaluation: SHIPPED
+Close matches:
+  1. ProcessFulfilment
+     Mismatch: status=CREATED required, entity status=SHIPPED
+
+Diagnosis: Retry event arrived too late — fulfilment already shipped.
+           Review: is the ScheduleEvent delay too long?
+```
+
+#### NO_MATCH Classification
+
+| Pattern | Severity | Meaning |
+|---------|----------|---------|
+| Timer event + entity already advanced | INFO | Normal — scheduled event pre-empted by faster progression |
+| Event name has no matching ruleset | HIGH | Misconfigured workflow — event fired but no ruleset exists to handle it |
+| Subtype mismatch in closeMatches | MEDIUM | Event sent to wrong subtype — check SendEvent props |
+| Status mismatch in closeMatches | LOW | Race condition or ordering issue — usually self-resolving |
+
+### 8E: Custom Rule Log Extraction
+
+Custom rules can emit structured diagnostic logs during execution. These appear as CUSTOM category events with `LogCollection` and `EventAttribute` data.
+
+#### Query
+
+```
+event.list({
+  "eventType": "ORCHESTRATION_AUDIT",
+  "context.rootEntityRef": "<ENTITY_REF>",
+  "count": 200
+})
+```
+
+Filter results where `category == "CUSTOM"`.
+
+#### Data Fields
+
+| Attribute | Description |
+|-----------|-------------|
+| `LogCollection` | JSON string containing `logs[]` array: `{ level, source, message, timestamp }` |
+| `EventAttribute` | JSON string containing the triggering event's full attributes |
+
+#### Log Structure
+
+Each log entry in `LogCollection.logs[]`:
+
+```json
+{
+  "level": "INFO",
+  "source": "ACCT.ext.CreateFulfilmentFromSourcingLocation",
+  "message": "Creating fulfilment at location WH-MAIN for order HD-20260222-001",
+  "timestamp": "2026-02-22T13:43:05.123Z"
+}
+```
+
+#### Report Format
+
+```
+=== Custom Rule Logs: ORDER ref=HD-20260222-001 ===
+
+Timestamp                  Level   Source Rule                                    Message
+──────────────────────────────────────────────────────────────────────────────────────────────────
+2026-02-22T13:43:05.123Z   INFO   CreateFulfilmentFromSourcingLocation           Creating fulfilment at WH-MAIN
+2026-02-22T13:43:05.456Z   INFO   CreateFulfilmentFromSourcingLocation           Fulfilment ref: FUL-HD-001
+2026-02-22T13:43:06.789Z   WARN   ValidateInventoryAvailability                  Low stock: SKU-001 has 2 units
+2026-02-22T13:43:07.012Z   INFO   SendWebhookWithDynamicAttributes               Webhook sent to order.created endpoint
+
+Total: 4 log entries (3 INFO, 1 WARN, 0 ERROR)
+```
+
+#### Use Cases
+
+- **Rule execution visibility**: See exactly what a custom rule decided and why
+- **Debugging silent failures**: When a rule "succeeds" but does something unexpected, logs reveal the internal logic path
+- **Performance profiling**: Log timestamps show how long each rule's internal processing took
+- **Regression detection**: Compare log output between test runs to spot behavioral changes
+
+### 8F: Entity Snapshot Extraction
+
+Snapshot events capture the full entity state at the moment orchestration begins. These are emitted as `snapshot` category audit events and contain the complete entity (all fields, attributes, items, relationships).
+
+#### Query
+
+```
+event.list({
+  "eventType": "ORCHESTRATION_AUDIT",
+  "context.rootEntityRef": "<ENTITY_REF>",
+  "count": 200
+})
+```
+
+Filter results where `category == "snapshot"`.
+
+#### Data Content by Entity Type
+
+**ORDER snapshot includes:**
+- `id`, `ref`, `type`, `subtype`, `status`
+- `customer` — full customer object (firstName, lastName, username)
+- `items[]` — order items with ref, quantity, status, productRef, fulfilmentChoiceRef
+- `fulfilmentChoice` — FC details (ref, deliveryType, deliveryAddress)
+- `attributes[]` — all entity attributes at that moment
+- `createdOn`, `updatedOn`
+
+**FULFILMENT snapshot includes:**
+- `id`, `ref`, `type`, `subtype`, `status`
+- `items[]` — fulfilment items with ref, requestedQuantity, filledQuantity
+- `toAddress`, `fromAddress` — shipping addresses
+- `attributes[]` — including trackingNumber, carrierRef, pickedAt, etc.
+- `consignment` — consignment details if created
+- `deliveryType`
+
+#### Report Format
+
+```
+=== Entity Snapshots: ORDER ref=HD-20260222-001 ===
+
+Snapshot #1 — ORDER at CREATED (2026-02-22T13:43:01Z)
+  Status: CREATED
+  Items: 2 (SKU-001 x1, SKU-002 x2)
+  Customer: john.doe
+  Delivery: HD to 123 Main St
+  Attributes: sourceSystem=E2E_TEST, testRunId=E2E_MULTI_202602221343
+
+Snapshot #2 — FULFILMENT at CREATED (2026-02-22T13:43:06Z)
+  Status: CREATED
+  Items: 2 (SKU-001 x1, SKU-002 x2)
+  From: WH-MAIN (Warehouse Main)
+  To: 123 Main St
+  Delivery Type: HD
+  Attributes: (none yet)
+
+Snapshot #3 — FULFILMENT at PICKPACK (2026-02-22T13:43:15Z)
+  Status: PICKPACK
+  Items: 2 (SKU-001 filled:1, SKU-002 filled:2)
+  Attributes: pickedAt=2026-02-22T13:43:14Z
+```
+
+#### Use Cases
+
+- **State verification in E2E tests**: Compare expected entity state at each step against the snapshot — no need to query entity separately
+- **Before/after comparison**: Diff consecutive snapshots to see exactly what changed during a status transition
+- **Attribute tracking**: Verify attributes like `trackingNumber`, `pickedAt`, `deliveredAt` were set at the right time
+- **Customer data audit**: Confirm customer information was correctly attached at order creation
+
+#### Integration with `/fluent-e2e-test`
+
+After an E2E test completes, snapshots can be used as assertion evidence:
+
+```
+For each test step:
+  1. Extract snapshot for the entity at the expected status
+  2. Verify expected fields are present and correct
+  3. Compare with previous snapshot to confirm the transition changed the right fields
+  4. Include snapshot diffs in the test report for documentation
+```
+
+## Quick Diagnostic Queries
+
+### Order lifecycle summary (all events for an order)
+```
+event.list({
+  "context.entityRef": "<ORDER_REF>",
+  "context.entityType": "ORDER",
+  "count": 100
+})
+```
+
+### All fulfilments for an order
+```graphql
+{
+  fulfilments(orderId: [<ORDER_ID>], first: 50) {
+    edges {
+      node { id ref type status fromLocation { ref } }
+    }
+  }
+}
+```
+
+### All fulfilment choices for an order
+```graphql
+{
+  fulfilmentChoices(orderId: [<ORDER_ID>], first: 50) {
+    edges {
+      node { id ref type status deliveryType }
+    }
+  }
+}
+```
+
+### Check a setting value
+```graphql
+{
+  settings(first: 1, context: "RETAILER", contextId: <RETAILER_ID>, name: ["<SETTING_NAME>"]) {
+    edges {
+      node { id name value lobValue }
+    }
+  }
+}
+```
+
+### Check workflow version deployed
+Use MCP `workflow.list` to verify the expected version is active.
+
+## Entity State Stuck — Decision Tree
+
+```
+Entity not advancing?
+│
+├─ Check event.list for the entity
+│  ├─ No events found → Event was never sent, or wrong entityRef/entityType
+│  ├─ Events all SUCCESS → State changed but maybe to unexpected status
+│  │   └─ Query entity status → Check workflow state machine
+│  └─ Event FAILED → Inspect event.get for error details
+│     ├─ "Rule exception" → Check rule source code + props
+│     ├─ "No matching ruleset" / NO_MATCH → Event name doesn't match any ruleset; check flexType
+│     ├─ "Entity not found" → Wrong entityRef or entity deleted
+│     └─ "Permission denied" → Check user/retailer scope
+│
+├─ Entity in gate state (waiting for children)?
+│  └─ Query all child entities
+│     ├─ All children in terminal state → Gate ruleset bug (check statuses prop)
+│     └─ Some children not terminal → Wait for children to complete
+│
+└─ Event sent but still PENDING?
+   └─ Async queue delay — wait 15-30s and re-check
+```
+
+## Manual Intervention Options
+
+After identifying the root cause, choose the appropriate intervention:
+
+### Option 1: Re-send the Event
+
+If the root cause is fixed (e.g., workflow deployed, setting created, data corrected), re-send the original event:
+
+```
+event.send({
+  "name": "<EVENT_NAME>",
+  "entityType": "<ENTITY_TYPE>",
+  "entityRef": "<ENTITY_REF>",
+  "retailerId": "<RETAILER_ID>"
+})
+```
+
+### Option 2: Run-Once Workflow (Targeted Rule Execution)
+
+Execute specific rules without triggering the full workflow. Useful for re-triggering a failed webhook, running a data correction rule, or executing a rule that isn't in the current deployed workflow.
+
+Send to `POST /api/v4.1/event/sync` with the workflow attached in `meta`:
+
+```json
+{
+  "name": "FixEvent",
+  "retailerId": "1",
+  "entityType": "FULFILMENT",
+  "entityId": "143",
+  "meta": {
+    "workflow": {
+      "retailerId": "1",
+      "version": "1.0",
+      "rulesets": [{
+        "name": "FixEvent",
+        "type": "FULFILMENT",
+        "eventType": "NORMAL",
+        "rules": [{ "name": "ACCOUNT.module.RuleName", "props": { "key": "value" } }],
+        "triggers": [],
+        "userActions": []
+      }]
+    }
+  }
+}
+```
+
+**Key:** The event `name` must match the ruleset `name`. Any rule in the account can be used — it doesn't need to exist in the deployed workflow. Webhooks and mutations execute; inline events are ignored.
+
+For full run-once workflow reference → `/fluent-event-api`.
+
+### Option 3: Direct GraphQL Mutation
+
+For data corrections that don't require workflow logic, use `graphql.query` with a mutation directly.
+
+## Causal Chain Playbook
+
+Compact playbook for tracing event causality when diagnosing failures.
+
+### Principle: One ORCHESTRATION Event = One Execution Journey
+
+Every ORCHESTRATION event triggers a chain of ORCHESTRATION_AUDIT events. The audit events contain the detailed execution (rulesets, rules, actions) linked via `sourceEvents`.
+
+### Step-by-Step Chain Reconstruction
+
+1. **Find the ORCHESTRATION event** that triggered execution:
+   ```
+   event.list({ "context.entityRef": "<REF>", "eventType": "ORCHESTRATION", "count": 50 })
+   ```
+
+2. **Find all audit events it spawned** (linked by sourceEvents):
+   ```
+   event.list({
+     "eventType": "ORCHESTRATION_AUDIT",
+     "context.entityRef": "<REF>",
+     "from": "<orchestration_event_timestamp_minus_1s>",
+     "to": "<orchestration_event_timestamp_plus_60s>",
+     "count": 200
+   })
+   ```
+   Match: audit events whose `context.sourceEvents` array contains the ORCHESTRATION event ID.
+
+3. **Classify audit events by category:**
+   - `snapshot` — entity state before execution
+   - `ruleSet` — ruleset-level timing (`startTimer`/`stopTimer`)
+   - `rule` — individual rule execution
+   - `ACTION` — actions performed (SetState, SendEvent, Send Webhook)
+   - `exception` — error with stack trace
+   - `CUSTOM` — custom log output from rules
+
+4. **Trace the failure point:**
+   - Find first `exception` category event → check `attributes.message` and `attributes.exception.stackTrace`
+   - Check the `ruleSet` event → its `name` is the ruleset that was executing
+   - Check the `ACTION` events → see what actions completed before the failure
+
+5. **If sourceEvents is sparse:** fall back to entity context + time window correlation (see `/fluent-event-api` → Causality Correlation Workflow)
+
+### Webhook Failure Triage
+
+When a webhook fails during event processing:
+
+1. **Find the webhook ACTION event:**
+   ```
+   event.list({
+     "category": "ACTION",
+     "name": "Send Webhook",
+     "context.rootEntityRef": "<REF>",
+     "count": 50
+   })
+   ```
+
+2. **Inspect webhook attributes:**
+   - `Request Endpoint` — the target URL
+   - `Response code` — HTTP status (4xx = client error, 5xx = server error, 0 = connection timeout)
+   - `Response Body` — error details from the remote server
+   - `Response Headers` — may reveal rate limiting or auth issues
+
+3. **Check the webhook setting:**
+   ```graphql
+   {
+     settings(first: 1, context: "RETAILER", contextId: <ID>, name: ["<webhook-setting-name>"]) {
+       edges { node { id name value lobValue } }
+     }
+   }
+   ```
+
+4. **Common webhook failures:**
+
+   | Response Code | Likely Cause | Action |
+   |--------------|--------------|--------|
+   | 0 | Connection timeout / DNS failure | Check endpoint URL and network |
+   | 401/403 | Auth credentials invalid or expired | Update webhook setting credentials |
+   | 404 | Endpoint URL changed | Update webhook setting URL |
+   | 429 | Rate limited | Implement retry/backoff or reduce event frequency |
+   | 500+ | Remote server error | Check target system logs |
+
+## Integration with Other Skills
+
+| Task | Skill |
+|------|-------|
+| Event model, filter parameters, causality chains | `/fluent-event-api` |
+| MCP payload syntax and tool limits | `/fluent-mcp-tools` |
+| Rule/source correlation | `/fluent-custom-code` |
+| Workflow topology and trigger intent | `/fluent-workflow-analyzer` |
+
+## Reporting
+
+After diagnosis, report:
+
+1. **Entity:** ref, type, current status, expected status
+2. **Failing event:** name, ID, status, error message
+3. **Root cause:** which ruleset/rule failed and why
+4. **Fix:** what needs to change (code, workflow, setting, data)
+5. **Recommendation:** specific file/line/config to modify