@fluentcommerce/ai-skills 0.1.0

package/content/dev/skills/fluent-e2e-test/SKILL.md

@@ -0,0 +1,394 @@
+---
+name: fluent-e2e-test
+description: Run end-to-end test sequences for Fluent Commerce workflows. Discover environment, create entities dynamically, fire events, poll statuses, assert state transitions, and report results. Triggers on "run e2e test", "test order flow", "test workflow", "e2e test", "integration test".
+user-invocable: true
+allowed-tools: Bash, Read, Write, Edit, Glob, Grep
+argument-hint: [scenario-name] [--order-ref <ref>] [--entity-type ORDER|FULFILMENT]
+---
+
+# End-to-End Test Runner
+
+Run automated test sequences against Fluent Commerce workflows. Discover environment configuration, create entities with real refs, fire events in sequence, poll for state transitions, and report pass/fail results.
+
+## Ownership Boundary
+
+This skill owns test scenario choreography, polling strategy, and pass/fail reporting.
+
+Environment discovery and entity creation payloads are owned by `/fluent-test-data`.
+Canonical MCP extension tool syntax/limits are owned by `/fluent-mcp-tools`.
+Event model/filter semantics and causality patterns are owned by `/fluent-event-api`.
+
+## Pre-Check: Load Source Analysis
+
+Before testing, check if `/fluent-custom-code` analysis artifacts exist:
+
+```
+accounts/<PROFILE>/analysis/custom-code/source-map.json
+accounts/<PROFILE>/analysis/custom-code/constraints.json
+```
+
+If found, use them to:
+- `source-map.json` → `modules[].rules[].parameters` — know what attributes events need
+- `source-map.json` → `modules[].rules[].entityTypes` — validate entity type coverage
+- `constraints.json` → `risks[]` — focus tests on high-risk areas
+
+If not found, rely on workflow JSON and runtime discovery only. Treat artifact-gate failures as "not found" (for example missing required files, missing hashes, or blocking `missingSources` in `constraints.json`).
+
+## When to Use
+
+- Verifying a workflow works end-to-end after changes
+- Testing a specific path (HD delivery, CC pickup, cancellation, short pick)
+- Regression testing after deploying a new module version
+- Validating event processing and state transitions
+- Confirming external system integration points (webhooks fire correctly)
+
+## Test Execution Framework
+
+Every E2E test follows this pattern:
+
+```
+0. DISCOVER environment (via /fluent-test-data)
+1. CREATE entity using discovered refs (via /fluent-test-data)
+2. For each step:
+   a. SEND event (via MCP event.send)
+   b. WAIT for async processing (poll with timeout)
+   c. VERIFY entity state (via MCP graphql.query)
+   d. Record PASS or FAIL
+3. REPORT results
+```
+
+### Key MCP Tools Used
+
+| Operation | MCP Tool | Server |
+|-----------|----------|--------|
+| Discover environment | `graphql.query` | fluent-mcp-extn |
+| Create entity | `graphql.query` (mutation) | fluent-mcp-extn |
+| Discover available actions | `workflow.transitions` | fluent-mcp-extn |
+| Send event | `event.send` | fluent-mcp-extn |
+| Check event status | `event.list` | fluent-mcp-extn |
+| Query entity state | `graphql.query` | fluent-mcp-extn |
+| Introspect schema | `graphql.introspect` | fluent-mcp-extn |
+
+## Phase 0: Environment Discovery (mandatory)
+
+**Before creating any entity**, run the `/fluent-test-data` discovery sequence to obtain:
+
+- Retailer ID and ref
+- Active warehouse ref (for HD) and store ref (for CC)
+- Product catalogue ref and at least one product ref with inventory
+- Customer ID (create if none exist)
+- Consignment prefix (from settings, or default `A_`)
+
+**Never hardcode refs.** Different retailers have different catalogues, locations, and products. Always query the live environment.
+
+## Dynamic Transition Discovery
+
+Instead of hardcoding event sequences, use `workflow.transitions` to dynamically discover available actions at each state:
+
+```
+1. Create entity using discovered refs (via /fluent-test-data)
+2. workflow.transitions → get available actions at current status
+3. For each action:
+   a. event.send with eventName and required attributes from response
+   b. Poll entity status until transition completes
+   c. workflow.transitions → get next available actions
+4. Repeat until no more userActions
+
+If `userActions` is empty, do **not** assume terminal immediately:
+- It can also mean a system-managed/gated status.
+- Verify entity status and recent events before concluding terminal.
+- If status is non-terminal, continue via workflow/event-driven progression and trace checks.
+```
+
+**Benefits:**
+- Test sequences adapt automatically when workflows change
+- Discovers required attributes for each event (avoids missing-attribute errors)
+- Validates that expected user actions are available after deployment
+
+**Example — discover actions at current status:**
+```
+workflow.transitions({
+  triggers: [{
+    type: "ORDER",
+    subtype: "HD",
+    status: "<current_status>",
+    retailerId: "<discovered.retailerId>",
+    flexType: "ORDER::HD"
+  }]
+})
+```
+
+The response `userActions[].eventName` maps directly to `event.send`'s `name` parameter.
+The response `userActions[].attributes` tells you what to include in `event.send`'s `attributes`.
+
+## Poll-and-Assert Pattern
+
+After sending an event, the entity status changes asynchronously. Use this pattern to wait:
+
+### Poll entity status until expected state (or timeout)
+
+```
+1. Send event via event.send (mode: "async")
+2. Wait 3 seconds (initial processing time)
+3. Query entity status via graphql.query
+4. If status matches expected → PASS
+5. If status doesn't match → wait 5 more seconds, retry (up to 12 retries = 60s total)
+6. If timeout → FAIL with "Expected <EXPECTED>, got <ACTUAL> after 60s"
+```
+
+**Composite workflow note (ORDER::MULTI and similar):**
+- After `ConfirmValidation`, use a longer wait window (20-30s) before first assertion.
+- Fulfilment creation and aggregate gates can complete asynchronously in several hops.
+
+### Poll event status until processed
+
+```
+1. After event.send, note the event details
+2. event.list filtered by entityRef + name + recent time window
+3. Check eventStatus: SUCCESS = processed, FAILED = error, PENDING = still processing
+4. If PENDING → wait 5s and retry
+5. If FAILED → report error details from event.get
+```
+
+## Test Scenarios
+
+### Scenario: ORDER HD (Home Delivery) — Full Lifecycle
+
+**Pre-requisite:** Run `/fluent-test-data` discovery. Need: retailerId, warehouseRef, productRef, catalogueRef, customerId, consignmentPrefix.
+
+**Entities created:** 1 ORDER with 1 HD fulfilment choice (all refs from discovery)
+
+**Steps:**
+
+| # | Action | Event | Expected Order Status | Expected Fulfilment Status |
+|---|--------|-------|----------------------|---------------------------|
+| 1 | Create order | *(via /fluent-test-data)* | CREATED → ON_VALIDATION | — |
+| 2 | Confirm validation | `ConfirmValidation` on ORDER | ORDER_SPLIT → IN_PROGRESS | CREATED |
+| 3 | Confirm allocation | `ConfirmAllocation` on FULFILMENT | IN_PROGRESS | RECEIVED |
+| 4 | Create invoice | `CreateInvoice` on FULFILMENT | IN_PROGRESS | INVOICED |
+| 5 | Confirm pick | `ConfirmPick` on FULFILMENT | IN_PROGRESS | PICKPACK → *(auto)* |
+| 6 | Confirm shipment | `ConfirmShipment` on FULFILMENT | SHIPPED | SHIPPED |
+| 7 | Confirm delivery | `ConfirmDelivery` on FULFILMENT | COMPLETED | DELIVERED |
+
+**Step 1 — Create Order:**
+
+Run `/fluent-test-data` to generate the `createOrder` mutation with discovered refs. The skill queries locations, products, catalogues, and customers — then builds the mutation payload dynamically. See `/fluent-test-data` for the full discovery and creation flow.
+
+**Step 2 — ConfirmValidation (triggers full cascade):**
+```
+event.send({
+  name: "ConfirmValidation",
+  entityRef: "<order_ref>",
+  entityType: "ORDER",
+  retailerId: "<discovered.retailerId>"
+})
+```
+
+After this event, the async cascade runs automatically:
+- ProcessOrder → ORDER_SPLIT
+- RouteFulfilmentChoice → ProcessHDFulfilmentChoice
+- CreateFulfilmentFromSourcingLocation → creates FULFILMENT
+- NotifyFCComplete → SourceOrder → IN_PROGRESS
+
+**Wait 15 seconds**, then verify:
+- Order status = `IN_PROGRESS`
+- Fulfilment exists with status = `CREATED`
+- Query fulfilment ref for subsequent events
+
+If fulfilment is not found, run fallback lookup sequence before failing:
+1. Query fulfilment from the order edge (`order.fulfilments`).
+2. Query fulfilment by order ID (`fulfilments(orderId: [...])`) if supported.
+3. Check event history for `FAILED` or `PENDING` events.
+4. Query `ORCHESTRATION_AUDIT` (`category=exception`) for rule-level errors.
+
+**Steps 3-7 — Fulfilment events:**
+
+Each fulfilment event follows the same pattern:
+```
+event.send({
+  name: "<EVENT_NAME>",
+  entityRef: "<fulfilment_ref>",
+  entityType: "FULFILMENT",
+  retailerId: "<discovered.retailerId>",
+  attributes: { ... }
+})
+```
+
+Use `workflow.transitions` at each step to confirm what attributes are required. Common patterns:
+
+| Event | Typical Attributes |
+|-------|-------------------|
+| `ConfirmAllocation` | *(check transitions response — often none)* |
+| `CreateInvoice` | *(check transitions response — often none)* |
+| `ConfirmPick` | `pickedAt` (ISO datetime), `pickedItems` (array of `{ ref, filledQuantity }` from discovered product refs) |
+| `ConfirmShipment` | `trackingNumber` (generate: `E2E-TRACK-<TIMESTAMP>`), `carrierRef` (generate: `E2E-CARRIER`), `shippedAt` (ISO now) |
+| `ConfirmDelivery` | `deliveredAt` (ISO now) |
+
+### Scenario: ORDER CC (Click & Collect)
+
+**Pre-requisite:** Same as HD, but also need a store location (type=STORE) from discovery.
+
+Same as HD through step 5 (ConfirmPick), then diverges:
+
+| # | Event | Expected Fulfilment Status |
+|---|-------|---------------------------|
+| 6 | *(auto after pick)* | READY_FOR_PICKUP |
+| 7 | `ConfirmCollection` on FULFILMENT | COLLECTED |
+| — | *(auto)* | ORDER → COMPLETED |
+
+### Scenario: Cancel Order
+
+| # | Action | Event | Expected Status |
+|---|--------|-------|----------------|
+| 1 | Create order | *(via /fluent-test-data)* | ON_VALIDATION |
+| 2 | Cancel | `CancelOrder` on ORDER | CANCELLED |
+
+### Scenario: Short Pick
+
+| # | Action | Expected Fulfilment Status |
+|---|--------|---------------------------|
+| 1-4 | Normal flow through INVOICED | INVOICED |
+| 5 | `ConfirmPick` with `pickedItems` where `filledQuantity < requestedQuantity` | PARTIAL |
+
+### Scenario: Dynamic (Workflow-Driven)
+
+For custom or unknown workflows, use fully dynamic discovery:
+
+```
+1. Run /fluent-test-data discovery
+2. Create entity using discovered refs
+3. Loop:
+   a. workflow.transitions for current entity state
+   b. If no userActions:
+      - verify status category and recent events first
+      - stop only when entity is confirmed terminal
+   c. Pick first userAction (or specific one by label)
+   d. event.send with eventName + required attributes from response
+   e. Poll until entity transitions
+   f. Go to (a)
+4. Report all transitions walked
+```
+
+This approach works for ANY workflow without knowing the state machine in advance.
+
+## Test Report Format
+
+After running a scenario, output a structured report:
+
+```
+=== E2E Test Report ===
+Scenario: ORDER HD Full Lifecycle
+Order Ref: <generated_ref>
+Retailer: <discovered.retailerRef> (ID: <discovered.retailerId>)
+Start Time: <ISO timestamp>
+
+Discovery:
+  Warehouse: <discovered.warehouseRef>
+  Product: <discovered.productRef> (catalogue: <discovered.catalogueRef>)
+  Customer: <discovered.customerId>
+
+Step 1: Create Order
+  Action: createOrder mutation (refs from live discovery)
+  Result: PASS — Order created, ID=<id>, status=CREATED
+  Duration: <N>s
+
+Step 2: ConfirmValidation → IN_PROGRESS
+  Action: event.send ConfirmValidation
+  Expected: Order=IN_PROGRESS, Fulfilment=CREATED
+  Actual: Order=IN_PROGRESS, Fulfilment=CREATED (ref: <fulfilment_ref>)
+  Result: PASS
+  Duration: <N>s (async cascade)
+
+...
+
+=== Summary ===
+Steps: N/N passed
+Total Duration: <N>s
+Result: ALL PASSED
+```
+
+## Post-Test Event Analysis
+
+After a test scenario completes (pass or fail), extract detailed execution data from the event audit trail for comprehensive validation. This supplements the poll-and-assert pattern with full execution evidence.
+
+> **Preferred: compact flowInspect first.** After test completion, call `event.flowInspect` with the order ref as `rootEntityRef` using default compact mode (~2-3k tokens). The compact summary includes status flow, anomaly findings, failed webhooks, and slowest rulesets — sufficient for most test reports. Only add targeted flags (`includeRuleDetails`, `includeCustomLogs`) if the compact summary reveals issues that need drill-down. **NEVER use `compact: false`** — it returns ~30k tokens and fills context. Use the manual patterns below only when you need extraction beyond what flowInspect provides.
+
+### Snapshot Verification
+
+Use entity snapshots from ORCHESTRATION_AUDIT events to verify entity state at each transition without additional GraphQL queries:
+
+```
+event.list({
+  "eventType": "ORCHESTRATION_AUDIT",
+  "context.rootEntityRef": "<ORDER_REF>",
+  "count": 200
+})
+```
+
+Filter by `category == "snapshot"` to get the full entity state at each orchestration step. Each snapshot contains the complete entity (status, items, attributes, relationships) at the moment before rules executed.
+
+**Verification pattern:**
+```
+For each test step that passed:
+  1. Find the snapshot event for that entity at the expected status
+  2. Verify key fields:
+     - Items count and quantities match expected
+     - Attributes set by previous steps are present (e.g., trackingNumber after ConfirmShipment)
+     - Customer/address data is correctly attached
+     - Child entity refs (fulfilmentChoice, fulfilment) are populated
+  3. If a step failed, compare the snapshot against expected to identify what was different
+```
+
+### Mutation Verification
+
+Extract all mutations executed during the test to verify the workflow made the correct GraphQL calls:
+
+Filter ORCHESTRATION_AUDIT events where `name` is null and `attributes` contains `request.queryString`. Each mutation event shows the exact GraphQL mutation and input variables.
+
+This catches issues where:
+- A rule set the wrong status (mutation input has unexpected status value)
+- A rule updated the wrong entity (mutation targets a different entity ID)
+- Expected attribute updates were not made (mutation input missing expected fields)
+
+### Webhook Delivery Verification
+
+After test completion, verify all expected webhooks were delivered:
+
+Filter ORCHESTRATION_AUDIT events where `name == "Send Webhook"`. Check `Response code` for each:
+- `200` = delivered successfully
+- `404` = endpoint not found (check setting)
+- `0` = connection timeout (check endpoint availability)
+
+Include webhook delivery results in the test report as an additional validation layer.
+
+### Enhanced Test Report (with event analysis)
+
+After the standard test report, append an event analysis section:
+
+```
+=== Event Analysis ===
+
+Mutations: 7 total (all successful)
+Webhooks: 4 sent (3 delivered, 1 failed — webhook.fulfilment.shipped → 404)
+Snapshots: 3 captured (ORDER@CREATED, FULFILMENT@CREATED, FULFILMENT@PICKPACK)
+Scheduled Events: 1 (ValidationGraceExpired, 30s delay, pre-empted by ConfirmValidation)
+NO_MATCH Events: 1 (ValidationGraceExpired → expected, timer pre-empted)
+Custom Logs: 4 entries (3 INFO, 1 WARN)
+
+Recommendation: Fix webhook setting "webhook.fulfilment.shipped" before go-live
+```
+
+## Tips
+
+- **Always run /fluent-test-data discovery first** — Never assume refs exist
+- **Use unique refs per test run** — Include timestamp or counter to avoid collisions
+- **Wait 15s after ConfirmValidation** — The async cascade creates FC → fulfilment → gates, all in sequence
+- **For composite workflows, poll up to 60s** — CREATE/validation cascades can take longer than single-entity flows
+- **Wait 5-10s between fulfilment events** — Async processing needs time
+- **Always query fulfilment ref** after step 2 — The ref is generated by the workflow (pattern varies by implementation)
+- **Check event.list if a step fails** — Filter by entityRef + eventStatus=FAILED to see error details
+- **Query ORCHESTRATION_AUDIT events on failure** — When a step fails, query audit events for the failing entity to see exactly what happened: `event.list({ eventType: "ORCHESTRATION_AUDIT", "context.entityRef": "<REF>", from: "<recent>", count: 50 })`. Look for `category=exception` events for stack traces, `category=ACTION` for what rules executed, and `category=ruleSet` for timing.
+- **If `workflow.transitions` returns empty `userActions` on non-terminal statuses** — continue with workflow/event-driven progression and trace checks; do not stop early
+- **On FAIL, hand off to /fluent-trace** — Pass entityRef, entityType, eventId (if known), expected status, and actual status; avoid duplicating trace decision-tree logic in this skill
+- **Use event.send with mode: "async"** — Synchronous mode is not recommended for workflow events
+- **Use `workflow.transitions`** to discover required attributes at each step instead of guessing