@adia-ai/a2ui-mcp 0.0.5 → 0.1.1

package/CHANGELOG.md

@@ -11,6 +11,172 @@ zettel strategies.
 
 ---
 
+## [0.1.1] - 2026-05-01
+
+Phase 2 of [`docs/specs/semantic-validator.md`](../../../docs/specs/semantic-validator.md)
+— opt-in combined-gating in `eval-diff.mjs` + new
+`semantic-stats.mjs` companion script. **No breaking changes.**
+Default `eval-diff` behavior unchanged — Phase 1 shadow-mode is
+still the default; combined gating is opt-in via flags.
+
+### Added (`scripts/eval-diff.mjs` — Phase 2 gating flags)
+
+- **`--gate-mode {structural|combined}`** — `structural` (default)
+  preserves Phase 1 shadow behavior: `row.pass` gates on
+  `validationScore` alone; semantic verdicts are annotation-only.
+  `combined` flips `row.pass` to gate on the combined score
+  (`round(0.6 × validationScore + 0.4 × semanticScore)`); preserves
+  the pre-flip pass as `row.passStructural`; recomputes
+  `runObj.passRate` + carries `runObj.passRateStructural` (baseline)
+  alongside; records `runObj.gateMode` + `runObj.gateThreshold`;
+  `diff.md` gains structural-baseline + avgSemantic + avgCombined
+  rows.
+- **`--gate-threshold N`** — combined-mode threshold; default 70 to
+  match the existing structural threshold. Override per-run for
+  sweep-style tuning.
+- **Validation gate** — combined-mode requires `--semantic`; the
+  script rejects the flag combination at startup so the operator
+  never silently ships the gating change without the scores it
+  needs.
+
+### Added (`scripts/semantic-stats.mjs` — companion stats script)
+
+- **New** — read-only; takes two run JSON paths
+  (`evals/mcp/runs/<stamp>/{mcp,zettel}.json`); emits markdown to
+  stdout with **verdict-distribution deltas + per-intent pass-flip
+  diagnostics** (which intents flipped pass→fail or fail→pass
+  between baseline and candidate). The tooling that satisfies the
+  "no unexplained regressions" exit criterion of Phase 2 §
+  Rollout before promoting combined gating to default.
+
+### Procedure for promotion (deferred)
+
+Promotion to default is deferred until two full eval-diff runs
+(structural-only baseline + combined-gating candidate) have been
+compared via `semantic-stats.mjs` and the regression count
+justifies it. Procedure:
+
+```bash
+# 1. Capture structural-only baseline (default behavior)
+node packages/a2ui/mcp/scripts/eval-diff.mjs --engine zettel --semantic
+
+# 2. Run the candidate (combined gating)
+node packages/a2ui/mcp/scripts/eval-diff.mjs --engine zettel --semantic --gate-mode combined
+
+# 3. Compare
+node packages/a2ui/mcp/scripts/semantic-stats.mjs \
+  evals/mcp/runs/<baseline-stamp>/zettel.json \
+  evals/mcp/runs/<candidate-stamp>/zettel.json > /tmp/semantic-stats.md
+```
+
+### Implementation references
+
+- [`scripts/eval-diff.mjs`](scripts/eval-diff.mjs)
+- [`scripts/semantic-stats.mjs`](scripts/semantic-stats.mjs)
+
+### Commits
+
+- `8415ff9e` — `feat(validator): semantic Phase 2 — opt-in combined-gating + drift cleanup`
+
+## [0.1.0] - 2026-04-28
+
+**Multi-turn gen-UI tool surface (Phase A code-complete).** Adds three new
+MCP tools that turn the chunk-composition pipeline from single-shot into a
+multi-turn surface, plus extends `compose_from_chunks` to mint a `state_id`
+for refinement chains.
+
+Spec: [`docs/specs/genui-multiturn-architecture.md`](../../../docs/specs/genui-multiturn-architecture.md) (Active v0.1.0).
+Plan: [`docs/plans/genui-multiturn-rollout-2026-04-28.md`](../../../docs/plans/genui-multiturn-rollout-2026-04-28.md) (Phase A scoped).
+ADR: [`0008-multiturn-genui-architecture.md`](../../../.brain/adrs/0008-multiturn-genui-architecture.md).
+
+### Added (MCP tools)
+
+- **`refine_composition(state_id, intent | ops, max_attempts?)`** — takes a
+  `state_id` from a prior `compose_from_chunks` (or `refine_composition`)
+  call plus either a natural-language intent OR an explicit op-list, runs
+  the chunk-refiner's two-pass synthesis (locator → modifier; validator-
+  driven retry on op-validation failure), applies the resulting chunk-plan
+  ops, re-materializes HTML, mints a child `state_id` chained back to the
+  parent, and returns A2UI `updateComponents` messages (the wire format).
+  Failed ops surface in `ops_failed` with reasons; the new state is cached
+  for further refinement.
+- **`get_state(state_id)`** — read-only inspection of a cached composition
+  state. Returns the chunk binding plan, materialized HTML, ops history
+  (chronological list of every refinement applied to this state's lineage),
+  and `parent_state_id` (chain-back). Auto-fires `cache-miss-on-known-state`
+  (severity `nit`) when the id is absent.
+- **`report_issue(type, severity, title, body, state_id?, trace?, …)`** —
+  first-class telemetry / dev-process feedback tool. Writes a structured
+  JSON ticket to `.brain/audit-history/issues/<issue_id>.json`. Three
+  reporter kinds: LLM self-fire (this tool with `reporter: 'llm'`),
+  consumer-fire (passed through directly), engine auto-fire (internal,
+  per `AUTO_FIRE_POLICY` in the issue-reporter module). Severity vocabulary
+  `blocker | drift | nit` matches the existing `coherence-audit`
+  discipline. Trace levels: `'full' | 'summary' | 'none'`; oversized
+  traces (> 200 KB) spill to a sidecar `.trace.json` file. Tool count
+  goes from 25 → 28.
+
+### Changed
+
+- **`compose_from_chunks`** now mints a `state_id` and caches the result
+  before returning. The response shape gains a `state_id` field; existing
+  fields (`html`, `plan`, `source`, `score`, `warnings`, `synthesis`)
+  are unchanged. Backward-compatible — consumers ignoring `state_id` see
+  no behavior change.
+- **MCP server boot** instantiates a `getStateCache()` singleton and an
+  `ENGINE_VERSION_INFO` block (mcp 0.1.0, corpus 0.0.6, engine zettel,
+  llm_adapter anthropic) that's threaded through every issue-reporter
+  call so written tickets carry environment metadata.
+
+### Auto-fire policy (engine-driven)
+
+`refine_composition` and `get_state` auto-fire `report_issue` on these
+failure paths via the per-tool-call `IssueAccumulator`:
+
+| Path | Type | Severity |
+|---|---|---|
+| Synthesizer exhausts retries | bug | drift |
+| Validator exhausts retries on refinement | bug | blocker |
+| Locator pass returns empty for targeted intent | bug | drift |
+| Retrieval 0 + synthesis fallback fails | training-gap | drift |
+| `get_state` called with absent `state_id` | bug | nit |
+| `refine_composition` ops_failed list non-empty | bug | drift |
+
+Multiple auto-fires within one tool call coalesce into a single issue
+(highest severity wins; reasons listed in body + tags).
+
+### Smoke + eval
+
+- `smoke:state-cache` — 34/34.
+- `smoke:issues` — 62/62.
+- `smoke:refine` — 51/51 (stub LLM).
+- `test:a2ui` — 25/25 + 1 skipped (was 19/19 + 1; +6 multi-turn assertions).
+- `mcp:smoke` — server boots clean with 28 tools registered.
+- **`eval:refine-synthesis`** — 15/15 PASS. Ops 100%, validate 100%,
+  0 auto-fires, 67 s.
+- **No regression:** `eval:chunk-synthesis` 10/10, `eval:diff zettel`
+  coverage 83 / score 89 / MRR 0.986.
+
+### Dependencies
+
+- Bumps `@adia-ai/a2ui-compose` requirement from `^0.0.1` to `^0.1.0`.
+
+### Migration
+
+Additive surface; no breaking changes. The existing 25 tools are
+unchanged behaviorally; `compose_from_chunks` adds a `state_id` field to
+its response that ignoring consumers can safely drop.
+
+### Phase A simplification (documented)
+
+Refinement ops internally use a chunk-plan vocabulary
+(`rebindSlot | appendToSlot | removeFromSlot | replacePage`), wrapped
+on output as standard `updateComponents` A2UI messages with
+`components[].html` carrying the materialized payload. Strict
+component-tree shape upgrade is queued for Phase B.
+
+---
+
 ## [0.0.5] - 2026-04-28
 
 **Retires the legacy exemplar auto-ingest.** Server boot no longer pulls

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@adia-ai/a2ui-mcp",
-  "version": "0.0.5",
+  "version": "0.1.1",
   "description": "AdiaUI A2UI MCP server. Exposes the compose engine over MCP with an engine selector for monolithic + zettel strategies.",
   "type": "module",
   "bin": {
@@ -26,7 +26,7 @@
   },
   "dependencies": {
     "@modelcontextprotocol/sdk": "^1.29.0",
-    "@adia-ai/a2ui-compose": "^0.0.1",
+    "@adia-ai/a2ui-compose": "^0.1.0",
     "@adia-ai/a2ui-retrieval": "^0.0.1",
     "@adia-ai/a2ui-validator": "^0.0.1",
     "@adia-ai/a2ui-corpus": "^0.0.6",

package/scripts/eval-diff.mjs

@@ -18,6 +18,9 @@
  *   node packages/a2ui/mcp/scripts/eval-diff.mjs --engine zettel  # fragment-graph only
  *   node packages/a2ui/mcp/scripts/eval-diff.mjs --limit 20
  *   node packages/a2ui/mcp/scripts/eval-diff.mjs --domain forms
+ *   node packages/a2ui/mcp/scripts/eval-diff.mjs --semantic       # Phase 1: shadow-mode semantic annotations
+ *   node packages/a2ui/mcp/scripts/eval-diff.mjs --semantic --gate-mode combined  # Phase 2: gate row.pass on combined score
+ *   node packages/a2ui/mcp/scripts/eval-diff.mjs --semantic --gate-mode combined --gate-threshold 75
  */
 import '../../../../scripts/load-env.mjs';
 
@@ -42,8 +45,24 @@ const opt = (k) => {
 const engine = opt('engine') || 'all';
 const limit = opt('limit') ? Number(opt('limit')) : undefined;
 const domain = opt('domain');
-// Shadow-mode semantic validator (Phase 1). Opt-in; zero effect on gating.
+// Shadow-mode semantic validator (Phase 1). Opt-in; zero effect on gating
+// when --gate-mode=structural (default).
 const semanticEnabled = args.includes('--semantic');
+// Phase 2 (gating mode):
+//   structural (default) — `row.pass` gated on validationScore alone (Phase 1 behavior)
+//   combined             — `row.pass` gated on (0.6 * validationScore + 0.4 * semanticScore)
+// Combined mode requires --semantic. Threshold defaults to 70 to match the
+// existing structural threshold; override with --gate-threshold N.
+const gateMode = opt('gate-mode') || 'structural';
+const gateThreshold = opt('gate-threshold') ? Number(opt('gate-threshold')) : 70;
+if (!['structural', 'combined'].includes(gateMode)) {
+  console.error(`[eval-diff] --gate-mode must be one of: structural | combined  (got: ${gateMode})`);
+  process.exit(2);
+}
+if (gateMode === 'combined' && !semanticEnabled) {
+  console.error(`[eval-diff] --gate-mode=combined requires --semantic (semantic scores must be computed before gating on them)`);
+  process.exit(2);
+}
 
 if (!['mcp', 'zettel', 'all'].includes(engine)) {
   console.error(`[eval-diff] --engine must be one of: mcp | zettel | all  (got: ${engine})`);
@@ -144,7 +163,12 @@ async function annotateSemantic(runObj, label) {
       row.rubricVersion = v.rubricVersion;
       const structural = row.validationScore ?? 0;
       row.combinedScore = Math.round(0.6 * structural + 0.4 * v.score);
-      // NOTE: row.pass intentionally NOT updated — shadow mode only.
+      // Phase 2: when gateMode === 'combined', flip row.pass to gate on the
+      // combined score. Preserves the structural pass for diagnostic purposes.
+      if (gateMode === 'combined') {
+        row.passStructural = row.pass;
+        row.pass = row.combinedScore >= gateThreshold;
+      }
       if (!v.error) {
         semSum += v.score;
         semN += 1;
@@ -164,7 +188,8 @@ async function annotateSemantic(runObj, label) {
   }
   runObj.semantic = {
     enabled: true,
-    mode: 'shadow',
+    mode: gateMode === 'combined' ? 'gating' : 'shadow',
+    gateThreshold: gateMode === 'combined' ? gateThreshold : null,
     judged: semN,
     errors,
     cached,
@@ -174,14 +199,31 @@ async function annotateSemantic(runObj, label) {
     tokens: { input: tokensIn, output: tokensOut },
     rubricVersion: 'v1',
   };
-  console.error(`[semantic:${label}] judged=${semN} avgSem=${runObj.semantic.avgSemanticScore} avgCombined=${runObj.semantic.avgCombinedScore} cached=${cached} errors=${errors} tokens=${tokensIn}+${tokensOut}`);
+  // Phase 2: when gateMode === 'combined', recompute pass aggregates so
+  // runObj.passRate / runObj.pass reflect the new gate. Capture the
+  // structural-only pass count alongside for diagnostic comparison.
+  if (gateMode === 'combined') {
+    const structuralPassCount = runObj.results.filter((r) => r.passStructural).length;
+    const combinedPassCount = runObj.results.filter((r) => r.pass).length;
+    runObj.passStructural = structuralPassCount;
+    runObj.passRateStructural = Math.round((structuralPassCount / (runObj.results.length || 1)) * 100);
+    runObj.pass = combinedPassCount;
+    runObj.passRate = Math.round((combinedPassCount / (runObj.results.length || 1)) * 100);
+    runObj.gateMode = 'combined';
+    runObj.gateThreshold = gateThreshold;
+  }
+  const modeLabel = gateMode === 'combined' ? `gating(>=${gateThreshold})` : 'shadow';
+  console.error(`[semantic:${label}] mode=${modeLabel} judged=${semN} avgSem=${runObj.semantic.avgSemanticScore} avgCombined=${runObj.semantic.avgCombinedScore} cached=${cached} errors=${errors} tokens=${tokensIn}+${tokensOut}`);
 }
 
 if (semanticEnabled) {
   if (!process.env.ANTHROPIC_API_KEY) {
     console.error('[eval-diff] --semantic requested but ANTHROPIC_API_KEY missing; skipping.');
   } else {
-    console.error(`[eval-diff] running semantic validator (shadow mode)…`);
+    const modeNote = gateMode === 'combined'
+      ? `gating mode (combined threshold=${gateThreshold})`
+      : 'shadow mode';
+    console.error(`[eval-diff] running semantic validator (${modeNote})…`);
     if (mcp) await annotateSemantic(mcp, 'mcp');
     if (zettel) await annotateSemantic(zettel, 'zettel');
   }
@@ -209,7 +251,11 @@ md += `# Engine Eval ${mcp && zettel ? 'Diff' : 'Report'}\n\n`;
 md += `- Run: \`${stamp}\`\n`;
 md += `- Engine(s): ${engine}\n`;
 md += `- Intents: ${(mcp || zettel).total}${domain ? ` (domain: ${domain})` : ''}${limit ? ` (limit: ${limit})` : ''}\n`;
-md += `- Mode: instant\n\n`;
+md += `- Mode: instant\n`;
+if (semanticEnabled) {
+  md += `- Semantic: ${gateMode === 'combined' ? `gating (threshold=${gateThreshold})` : 'shadow'}\n`;
+}
+md += `\n`;
 
 md += `## Aggregates\n\n`;
 if (mcp && zettel) {
@@ -219,6 +265,11 @@ if (mcp && zettel) {
   md += `| avgScore (emitted only) | ${mcp.avgScoreWhenEmitted} | ${zettel.avgScoreWhenEmitted} |\n`;
   md += `| avgF1 (emitted only) | ${mcp.avgF1WhenEmitted} | ${zettel.avgF1WhenEmitted} |\n`;
   md += `| pass rate % | ${mcp.passRate} | ${zettel.passRate} |\n`;
+  if (mcp.gateMode === 'combined' || zettel.gateMode === 'combined') {
+    md += `| pass rate % (structural-only baseline) | ${fmt(mcp.passRateStructural)} | ${fmt(zettel.passRateStructural)} |\n`;
+    md += `| avgSemanticScore | ${fmt(mcp.semantic?.avgSemanticScore)} | ${fmt(zettel.semantic?.avgSemanticScore)} |\n`;
+    md += `| avgCombinedScore | ${fmt(mcp.semantic?.avgCombinedScore)} | ${fmt(zettel.semantic?.avgCombinedScore)} |\n`;
+  }
   md += `| retrieval MRR | ${fmt(mcp.retrievalMRR)} | ${fmt(zettel.retrievalMRR)} |\n\n`;
 } else {
   const e = mcp || zettel;
@@ -229,6 +280,11 @@ if (mcp && zettel) {
   md += `| avgScore (emitted only) | ${e.avgScoreWhenEmitted} |\n`;
   md += `| avgF1 (emitted only) | ${e.avgF1WhenEmitted} |\n`;
   md += `| pass rate % | ${e.passRate} |\n`;
+  if (e.gateMode === 'combined') {
+    md += `| pass rate % (structural-only baseline) | ${fmt(e.passRateStructural)} |\n`;
+    md += `| avgSemanticScore | ${fmt(e.semantic?.avgSemanticScore)} |\n`;
+    md += `| avgCombinedScore | ${fmt(e.semantic?.avgCombinedScore)} |\n`;
+  }
   md += `| retrieval MRR | ${fmt(e.retrievalMRR)} |\n\n`;
 }
 

package/scripts/eval-refine-synthesis.mjs

@@ -0,0 +1,270 @@
+#!/usr/bin/env node
+/**
+ * Real-LLM eval set for the chunk-refiner — multi-turn refinement engine.
+ *
+ * Walks 5 seed compositions × 3 refinement intents = 15 total. Seeds are
+ * deterministic chunk-binding plans (no LLM cost on the create side); the
+ * refiner exercises the full two-pass synthesis (locator → modifier),
+ * validator-driven retry, and op-application path.
+ *
+ * Pass criteria (per spec §6.2 + plan §1.7):
+ *   - ≥ 80% of refinements produce ops (no all-fail outcome).
+ *   - ≥ 90% of returned ops apply cleanly (validator + applyOps).
+ *   - ≤  5 auto-fired issues across the full run (plan §1.8 #3).
+ *
+ * Spec: docs/specs/genui-multiturn-architecture.md (Active v0.1.0).
+ * Plan: docs/plans/genui-multiturn-rollout-2026-04-28.md (Phase A).
+ *
+ * Usage:
+ *   ANTHROPIC_API_KEY=… node packages/a2ui/mcp/scripts/eval-refine-synthesis.mjs
+ */
+
+import '../../../../scripts/load-env.mjs';
+import {
+  refineFromIntent,
+  applyOps,
+} from '../../compose/engines/zettel/chunk-refiner.js';
+import { mintStateId } from '../../compose/engines/zettel/state-cache.js';
+import { createIssueAccumulator } from '../../compose/engines/zettel/issue-reporter.js';
+import { composeFromPlan } from '../../compose/engines/zettel/chunk-composer.js';
+import { listChunksByKind, getChunk } from '../../corpus/scripts/chunk-library.js';
+import { createAdapter } from '../../compose/llm/llm-bridge.js';
+
+// ── Discover corpus shape ────────────────────────────────────────────
+// Pick a page with ≥ 2 slots so refinements have room to target.
+
+const pages = listChunksByKind('page');
+const panels = listChunksByKind('panel');
+const blocks = listChunksByKind('block');
+
+const slotsOf = (c) => (c.slots || c.instances?.[0]?.slots || []).map((s) => s.name);
+
+const samplePage =
+  pages.find((p) => slotsOf(p).length >= 2)
+  || panels.find((p) => slotsOf(p).length >= 2)
+  || pages[0]
+  || panels[0];
+
+if (!samplePage || slotsOf(samplePage).length === 0) {
+  console.error('Corpus has no page/panel chunks with declared slots — aborting eval');
+  process.exit(2);
+}
+
+const pageSlots = slotsOf(samplePage);
+const slotA = pageSlots[0];                  // typically the header slot
+const slotB = pageSlots[1] || pageSlots[0];  // typically the content slot
+
+// Pick at least 4 distinct block chunks. Filter out anything missing HTML.
+const usableBlocks = blocks
+  .filter((b) => (b.html || b.instances?.[0]?.html))
+  .slice(0, 8)
+  .map((b) => b.name);
+
+if (usableBlocks.length < 4) {
+  console.error(`Corpus has only ${usableBlocks.length} usable block chunks (need ≥ 4) — aborting eval`);
+  process.exit(2);
+}
+
+const [B0, B1, B2, B3, B4 = B0, B5 = B1] = usableBlocks;
+
+console.log(`▶ refine-synthesis eval`);
+console.log(`  page: ${samplePage.name} (${pageSlots.join(', ')})`);
+console.log(`  blocks: ${usableBlocks.slice(0, 6).join(', ')}`);
+console.log('');
+
+// ── Seeds + refinements ──────────────────────────────────────────────
+// 5 seeds × 3 refinements = 15 hold-out intents.
+
+const SEEDS = [
+  {
+    label: 'two-block-content',
+    plan: {
+      page: samplePage.name,
+      slot_bindings: { [slotA]: [B0], [slotB]: [B1, B2] },
+    },
+    refinements: [
+      `add another block to ${slotB}`,
+      `remove one block from ${slotB}`,
+      `swap the ${slotA} for a different option`,
+    ],
+  },
+  {
+    label: 'single-block-content',
+    plan: {
+      page: samplePage.name,
+      slot_bindings: { [slotA]: [B0], [slotB]: [B1] },
+    },
+    refinements: [
+      `add a second block alongside the existing one in ${slotB}`,
+      `replace the ${slotA} with a more concise header`,
+      `preserve the existing block and append another to ${slotB}`,
+    ],
+  },
+  {
+    label: 'three-block-stack',
+    plan: {
+      page: samplePage.name,
+      slot_bindings: { [slotA]: [B0], [slotB]: [B1, B2, B3] },
+    },
+    refinements: [
+      `remove the middle block from ${slotB}`,
+      `drop the last block from ${slotB}`,
+      `make the layout more compact`,
+    ],
+  },
+  {
+    label: 'header-only',
+    plan: {
+      page: samplePage.name,
+      slot_bindings: { [slotA]: [B0], [slotB]: [B1] },
+    },
+    refinements: [
+      `add an additional block to ${slotB}`,
+      `change the ${slotA}`,
+      `preserve everything and add a new block to ${slotB}`,
+    ],
+  },
+  {
+    label: 'mixed-stack',
+    plan: {
+      page: samplePage.name,
+      slot_bindings: { [slotA]: [B0], [slotB]: [B2, B3] },
+    },
+    refinements: [
+      `swap the first block in ${slotB} for a different one`,
+      `add another block at the end of ${slotB}`,
+      `drop the last block from ${slotB}`,
+    ],
+  },
+];
+
+// ── Run ──────────────────────────────────────────────────────────────
+
+const llmAdapter = await createAdapter();
+const startedAt = Date.now();
+const results = [];
+const autoFires = { total: 0, byReason: {} };
+
+for (const seed of SEEDS) {
+  // Materialize the seed plan (no LLM call; pure compose-from-plan).
+  const composed = composeFromPlan(seed.plan);
+  if (!composed.html) {
+    console.log(`✗ seed [${seed.label}] failed to materialize — skipping refinements`);
+    for (const intent of seed.refinements) {
+      results.push({
+        seed: seed.label, intent, ms: 0, ops_count: 0, attempts: 0,
+        targeted: null, ops_applied: 0, ops_failed: 0,
+        auto_fires: [], error: 'seed-materialize-failed',
+      });
+    }
+    continue;
+  }
+
+  const priorState = {
+    state_id: mintStateId(seed.label, 1),
+    intent: `[seed] ${seed.label}`,
+    plan: seed.plan,
+    html: composed.html,
+    version: 1,
+  };
+
+  console.log(`── seed [${seed.label}] · ${seed.plan.slot_bindings[slotB].length} block(s) in ${slotB}`);
+
+  for (const intent of seed.refinements) {
+    const acc = createIssueAccumulator();
+    const t0 = Date.now();
+    const row = {
+      seed: seed.label, intent, ms: 0, ops_count: 0, attempts: 0,
+      targeted: null, ops_applied: 0, ops_failed: 0,
+      auto_fires: [], error: null,
+    };
+
+    try {
+      const refined = await refineFromIntent({
+        priorState,
+        intent,
+        llmAdapter,
+        maxAttempts: 2,
+        issueAccumulator: acc,
+      });
+      row.ms = Date.now() - t0;
+      row.ops_count = refined.ops.length;
+      row.attempts = refined.synthesis?.attempts ?? 0;
+      row.targeted = refined.synthesis?.targeted ?? null;
+
+      if (refined.ops.length > 0) {
+        const applied = await applyOps({ priorState, ops: refined.ops });
+        row.ops_applied = applied.ops_applied.length;
+        row.ops_failed = applied.ops_failed.length;
+      }
+    } catch (e) {
+      row.ms = Date.now() - t0;
+      row.error = e.message;
+    }
+
+    row.auto_fires = acc.reasons();
+    autoFires.total += row.auto_fires.length;
+    for (const r of row.auto_fires) {
+      autoFires.byReason[r] = (autoFires.byReason[r] || 0) + 1;
+    }
+
+    results.push(row);
+
+    const flag = row.ops_count > 0 && row.ops_failed === 0 ? '✓' : (row.ops_count > 0 ? '~' : '✗');
+    const tgtTag = row.targeted === true ? 'tgt' : row.targeted === false ? 'unt' : '???';
+    const padMs = row.ms.toString().padStart(5);
+    console.log(`  ${flag} [${tgtTag}] ${padMs}ms ops=${row.ops_count} att=${row.attempts} ${intent}`);
+    if (row.error) console.log(`        error: ${row.error}`);
+    if (row.ops_failed > 0) console.log(`        ops_failed: ${row.ops_failed}`);
+    if (row.auto_fires.length) console.log(`        auto-fires: ${row.auto_fires.join(', ')}`);
+  }
+}
+
+// ── Summary ──────────────────────────────────────────────────────────
+
+const total = results.length;
+const producedOps = results.filter((r) => r.ops_count > 0).length;
+const totalOpsReturned = results.reduce((s, r) => s + r.ops_count, 0);
+const totalOpsApplied = results.reduce((s, r) => s + r.ops_applied, 0);
+const totalOpsFailed = results.reduce((s, r) => s + r.ops_failed, 0);
+
+const opsRate = total ? producedOps / total : 0;
+const validateRate = totalOpsReturned ? totalOpsApplied / totalOpsReturned : 0;
+
+console.log(`\n── Summary ──`);
+console.log(`  Refinements: ${total}`);
+console.log(`  Produced ops: ${producedOps}/${total} (${(opsRate * 100).toFixed(0)}%)`);
+console.log(`  Ops returned: ${totalOpsReturned}; applied: ${totalOpsApplied}; failed: ${totalOpsFailed} (validate ${(validateRate * 100).toFixed(0)}%)`);
+console.log(`  Auto-fires: ${autoFires.total}`);
+if (autoFires.total > 0) {
+  for (const [reason, n] of Object.entries(autoFires.byReason)) {
+    console.log(`    ${reason}: ${n}`);
+  }
+}
+const targeted = results.filter((r) => r.targeted === true).length;
+const untargeted = results.filter((r) => r.targeted === false).length;
+console.log(`  Targeted vs untargeted: ${targeted} / ${untargeted}`);
+console.log(`  Total time: ${((Date.now() - startedAt) / 1000).toFixed(1)}s`);
+
+const opsThreshold = 0.8;
+const validateThreshold = 0.9;
+const autoFireCeiling = 5;
+
+const opsPass = opsRate >= opsThreshold;
+const validatePass = totalOpsReturned === 0 || validateRate >= validateThreshold;
+const autoFirePass = autoFires.total <= autoFireCeiling;
+
+const allPass = opsPass && validatePass && autoFirePass;
+
+console.log('');
+console.log(`  ops rate ≥ ${opsThreshold * 100}%:   ${opsPass ? '✓' : '✗'} (${(opsRate * 100).toFixed(0)}%)`);
+console.log(`  validate ≥ ${validateThreshold * 100}%:    ${validatePass ? '✓' : '✗'} (${(validateRate * 100).toFixed(0)}%)`);
+console.log(`  auto-fires ≤ ${autoFireCeiling}:    ${autoFirePass ? '✓' : '✗'} (${autoFires.total})`);
+
+if (allPass) {
+  console.log(`\n✓ PASS`);
+  process.exit(0);
+} else {
+  console.log(`\n✗ FAIL`);
+  process.exit(1);
+}