@hyperfixi/testing-framework 2.10.0 → 2.11.0

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hyperfixi/testing-framework",
-  "version": "2.10.0",
+  "version": "2.11.0",
   "description": "Cross-platform behavior testing suite for LokaScript applications",
   "main": "dist/index.js",
   "module": "dist/index.mjs",
@@ -29,7 +29,7 @@
   "scripts": {
     "build": "tsup",
     "dev": "tsup --watch",
-    "pretest": "../../scripts/ensure-fresh.sh ../intent ../framework ../semantic ../patterns-reference ../core",
+    "pretest": "../../scripts/ensure-fresh.sh ../intent ../framework ../semantic ../patterns-reference ../core ../aot-compiler ../compilation-service",
     "test": "vitest run",
     "test:watch": "vitest",
     "test:coverage": "vitest run --coverage",
@@ -57,10 +57,11 @@
   "author": "LokaScript Contributors",
   "license": "MIT",
   "dependencies": {
-    "@hyperfixi/core": "^2.10.0",
-    "@hyperfixi/patterns-reference": "^2.10.0",
-    "@lokascript/i18n": "^2.10.0",
-    "@lokascript/semantic": "^2.10.0",
+    "@hyperfixi/core": "^2.11.0",
+    "@hyperfixi/patterns-reference": "^2.11.0",
+    "@lokascript/compilation-service": "^2.11.0",
+    "@lokascript/i18n": "^2.11.0",
+    "@lokascript/semantic": "^2.11.0",
     "diff": "^8.0.3",
     "esbuild": "^0.28.0",
     "happy-dom": "^20.10.6",

package/src/agent-bench/README.md

@@ -0,0 +1,162 @@
+# Agent-loop benchmark
+
+Measures what an LLM agent actually gets from emitting hyperscript through the
+MCP validate/repair/compile loop, against twenty natural-language UI tasks.
+
+Arc 3 of [AGENT_ERA_ROADMAP.md](../../../../docs-internal/AGENT_ERA_ROADMAP.md).
+
+```bash
+cd packages/testing-framework
+npx tsx src/agent-bench/cli.ts verify-references   # every reference still usable
+npx tsx src/agent-bench/cli.ts probe-variants      # the deterministic finding
+npx tsx src/agent-bench/cli.ts list                # prompts, for a generator
+npx tsx src/agent-bench/cli.ts score --run runs/my-run.json
+```
+
+## What it measures, and why in two halves
+
+Two questions are scored separately, and keeping them apart is the point:
+
+- **Does it parse?** `CompilationService.validate()` — the exact call the
+  `validate_and_compile` MCP tool makes.
+- **Does it do the right thing?** The candidate is executed in jsdom against the
+  task's fixture and its DOM effect signature compared byte-for-byte to the
+  reference's. (Effect-signature primitives are shared with the R2 ratchet, so
+  the two cannot disagree about what a DOM effect is.)
+
+A single blended "success rate" would hide the failure mode that matters most:
+the parser **degrades rather than failing**, so a candidate can come back
+`ok: true`, confidence 1.0, zero diagnostics — and target the wrong element, or
+do nothing at all.
+
+### Half 1 — the plausible-phrasing probe (no generator needed)
+
+`probe-variants` scores a fixed catalogue of phrasings a competent generator
+plausibly reaches for ([variants.ts](./variants.ts), each annotated with why).
+No LLM runs, so the result is a reproducible property of the parser — measured
+byte-identical across runs — and the claim stays narrow and checkable: _these
+phrasings behave thus_, not _a model emits them at rate R_.
+
+Recorded in [`baselines/agent-bench-phrasings.json`](../../baselines/agent-bench-phrasings.json)
+and ratcheted both directions at tolerance 0 by
+[`agent-bench.test.ts`](./agent-bench.test.ts) — a regression fails, and so does
+an unrecorded improvement (the numbers below are quoted in the docs, so a stale
+baseline makes them lie).
+
+### Half 2 — the A/B run (needs a generator)
+
+The loop being measured is the one a real integration runs, so simulating it
+in-process would measure a simulation. Instead the harness is agent-driven:
+
+1. `cli.ts list --json` — the prompts.
+2. **One-shot condition.** The agent answers every prompt from the prompt alone,
+   with no validation. Record them; do not revise after seeing any score.
+3. **Loop condition.** The agent answers again, this time iterating through
+   `cli.ts feedback --task <id> --code "<src>"` until it parses or it gives up.
+   `feedback` returns **only** what the MCP loop returns — diagnostics and the
+   parsed IR. It never reveals the reference or the behavior verdict; leaking
+   either would turn the loop into an oracle and the delta into fiction.
+4. `cli.ts score --run <file>` — per-condition parse rate, behavior rate, and
+   the delta.
+
+Run file:
+
+```json
+{
+  "generator": { "model": "…", "date": "…", "context": "what the generator could see" },
+  "conditions": {
+    "one-shot": { "toggle-self-class": "on click toggle .active on me" },
+    "loop": { "toggle-self-class": "on click toggle .active on me" }
+  }
+}
+```
+
+**No A/B run is committed here, deliberately.** The tasks and their reference
+implementations were authored in the same session that would have produced the
+candidates, so any one-shot number from that session measures recall of
+just-written answers, not generation. A meaningful run needs a generator that
+has not seen this directory — until one exists, the honest position is a harness
+with no number attached, not a flattering number with a caveat. `score` is fully
+implemented and ready for that run.
+
+Not a CI gate: LLM-in-the-loop is nondeterministic and this repo's gates stay
+deterministic. Half 1 **is** gated, because it has no generator in it.
+
+## Findings
+
+### First probe, 2026-08-24 (pre-3b)
+
+37 plausible phrasings, 20 tasks: **97% parse, 49% behave correctly, 49% parse
+clean but misbehave** — and exactly one of the failures produced a diagnostic.
+Iterating `validate → fix → re-validate` could not move a single ☠ row, because
+the loop was never told anything was wrong. The actionable conclusion was not
+"polish the loop" but **make these failures loud** — which became Arc 3b.
+
+### Arc 3b, first diagnostic (unconsumed-input propagation)
+
+The parser had been flagging dropped tokens all along — a `warning`-severity
+`unconsumed-input` diagnostic on the node, hoisted from any depth, with a
+confidence dock — and `CompilationService.normalize()` simply never read node
+diagnostics, so `validate()` reported `ok` with an empty diagnostics array. The
+fix is pure plumbing (no parser change, so the multilingual ratchets are
+untouched): lift warning/error-severity node diagnostics into the response, as
+`UNCONSUMED_INPUT` with a repair suggestion.
+
+|                                       | pre-3b      | post-3b                |
+| ------------------------------------- | ----------- | ---------------------- |
+| parse                                 | 36/37 (97%) | 36/37 (97%)            |
+| behave correctly                      | 18/37 (49%) | 18/37 (49%)            |
+| wrong but **warned** (loop can react) | 1/37        | **12/37** (11 ⚠ + 1 ✗) |
+| wrong and **silent**                  | 18/37 (49%) | **7/37 (19%)**         |
+
+One plumbing fix moved 11 of the 18 silent rows into the visible band: the
+omitted-marker family, the whole attribute-write family, `to every .y`,
+`the X of Y` properties, and `remove element`. Behavior is unchanged — these
+phrasings are still wrong — but the loop can now see and repair them.
+
+The remaining ☠ 7 split honestly in two:
+
+- **Real diagnostic gaps (5)** — parses that consume everything yet provably
+  do nothing or bind the wrong target with no trace: `add .x to all .y` /
+  `remove .x from all .y` (note the asymmetry: `every` warns, `all` doesn't),
+  `set the text of #el`, `if #el has class .x`, `add .x to <body/>`. These are
+  the next 3b targets (a no-op-command diagnostic covers most).
+- **Valid code, different intent (2)** — `add .hidden to #menu` (adds a class
+  named "hidden"; only wrong versus the _hide_ reference) and `on mouseover`
+  (a real handler for a neighbouring event). No parser diagnostic can catch
+  these; they are exactly what IR-vs-intent review (and Arc 4's equivalence
+  checking) is for.
+
+### Arc 3b, second diagnostic (inert-shape gate)
+
+The five remaining real gaps all parsed fully-consumed at confidence 1.0 but
+left distinctive fingerprints in the IR: `all .todo` becomes a property access
+on the undefined identifier `all`; `has class` mis-tokenizes the predicate as a
+class selector (`#box .has class .danger`) so the condition is always falsy;
+`<body/>` survives as a raw non-CSS selector that `querySelector` rejects; and
+`the text of #el` becomes an invisible expando write. Gate 4 of the
+compilation-service validation pipeline (`validation/inert-shapes.ts`) matches
+those fingerprints and warns (`INERT_QUANTIFIER_TARGET`,
+`HALF_PARSED_CONDITION`, `UNSUPPORTED_QUERY_LITERAL`, `INERT_PROPERTY_WRITE`)
+— warnings only, still no parser change.
+
+|                                       | pre-3b      | after slice 1 | after slice 2          |
+| ------------------------------------- | ----------- | ------------- | ---------------------- |
+| wrong but **warned** (loop can react) | 1/37        | 12/37         | **17/37** (16 ⚠ + 1 ✗) |
+| wrong and **silent**                  | 18/37 (49%) | 7/37 (19%)    | **2/37 (5%)**          |
+
+The two remaining ☠ rows are the valid-code-different-intent pair
+(`add .hidden to #menu`, `on mouseover`) — by design not diagnosable, and the
+standing case for IR-vs-intent review and Arc 4's behavioral equivalence. **The
+parser-gap silent band is now zero.**
+
+Bands are computed by `harness.bandOf` — one function shared by the probe, the
+committed baseline, and the ratchet test, so they cannot drift apart.
+
+## Adding a task
+
+Append to [`tasks.ts`](./tasks.ts): a prompt with no hyperscript in it, a
+reference, and the fixture markup it needs. Then run `verify-references` — a
+reference that does not parse, or produces no DOM effect, is rejected (same
+eligibility bar as R2's execution subset), because scoring against an empty
+signature would make wrong answers look right.

package/src/agent-bench/agent-bench.test.ts

@@ -0,0 +1,112 @@
+/**
+ * Guards for the agent-loop benchmark.
+ *
+ * Two things rot silently and would make the benchmark lie:
+ *
+ *   1. **A reference that stops working.** Behavior correctness is defined as
+ *      "same effect signature as the reference", so a reference that stops
+ *      parsing — or starts producing NO effects — makes every candidate for
+ *      that task score wrong (or, worse, makes an empty-effect candidate score
+ *      right). Same eligibility bar as R2's execution subset.
+ *
+ *   2. **The silent band drifting.** `baselines/agent-bench-phrasings.json`
+ *      records, per plausible phrasing, whether it is correct / rejected /
+ *      silently wrong. This ratchets BOTH directions at tolerance 0: a
+ *      regression (correct → silent) fails, and so does an improvement that
+ *      wasn't re-baselined. The improvement direction matters as much here as
+ *      in the R4 allowlist — a fixed parser gap that nobody re-records leaves
+ *      the docs quoting a stale number, and the "half of plausible phrasings
+ *      misbehave" claim is load-bearing for the roadmap.
+ *
+ * Deterministic and generator-free: no LLM runs here, so this IS a legitimate
+ * CI gate (the A/B run in README.md, which needs a generator, deliberately is
+ * not). Full sweep measures ~6s.
+ */
+
+import { describe, it, expect, beforeAll } from 'vitest';
+import { readFileSync } from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { TASKS, taskById } from './tasks.js';
+import { VARIANTS } from './variants.js';
+import {
+  bandOf,
+  executeCandidate,
+  initialize,
+  scoreCandidate,
+  validateCandidate,
+} from './harness.js';
+
+const BASELINE_PATH = path.resolve(
+  path.dirname(fileURLToPath(import.meta.url)),
+  '../../baselines/agent-bench-phrasings.json'
+);
+
+interface Baseline {
+  totals: Record<string, number>;
+  phrasings: Array<{ taskId: string; code: string; band: string }>;
+}
+
+describe('agent-bench: task references', () => {
+  beforeAll(async () => {
+    await initialize();
+  }, 60_000);
+
+  it('every task has a unique id', () => {
+    const ids = TASKS.map(t => t.id);
+    expect(new Set(ids).size).toBe(ids.length);
+  });
+
+  it('no prompt leaks hyperscript syntax to the generator', () => {
+    // A prompt containing the answer measures nothing. Selector/sigil
+    // characters and command keywords in the imperative position are the tells.
+    const leaks = TASKS.filter(t => /\bon click\b|=>|\s_=|@[a-z-]+\s|\*[a-z-]+\s/i.test(t.prompt));
+    expect(leaks.map(t => t.id)).toEqual([]);
+  });
+
+  it.each(TASKS.map(t => [t.id, t] as const))(
+    'reference for %s parses and produces a usable effect signature',
+    async (_id, task) => {
+      const validation = await validateCandidate(task.reference);
+      expect(validation.ok, `reference does not parse: ${task.reference}`).toBe(true);
+      const { effects, error } = await executeCandidate(task, task.reference);
+      expect(error, `reference errored: ${error}`).toBeUndefined();
+      expect(effects.length, `reference has no DOM effect: ${task.reference}`).toBeGreaterThan(0);
+    },
+    30_000
+  );
+});
+
+describe('agent-bench: plausible-phrasing ratchet', () => {
+  let baseline: Baseline;
+
+  beforeAll(async () => {
+    await initialize();
+    baseline = JSON.parse(readFileSync(BASELINE_PATH, 'utf8')) as Baseline;
+  }, 60_000);
+
+  it('baseline covers exactly the current variant set', () => {
+    const recorded = baseline.phrasings.map(p => p.code).sort();
+    const current = VARIANTS.map(v => v.code).sort();
+    expect(
+      recorded,
+      'variants changed without regenerating: tsx src/agent-bench/cli.ts probe-variants --json > baselines/agent-bench-phrasings.json'
+    ).toEqual(current);
+  });
+
+  it('every phrasing still lands in its recorded band (both directions, tolerance 0)', async () => {
+    const drift: string[] = [];
+    for (const entry of baseline.phrasings) {
+      const task = taskById(entry.taskId);
+      expect(task, `baseline names unknown task ${entry.taskId}`).toBeDefined();
+      const s = await scoreCandidate(task!, entry.code);
+      const band = bandOf(s);
+      if (band !== entry.band) drift.push(`${entry.code}\n    ${entry.band} → ${band}`);
+    }
+    expect(
+      drift,
+      'phrasing behavior drifted. If this is an intentional improvement, regenerate:\n' +
+        '  tsx src/agent-bench/cli.ts probe-variants --json > baselines/agent-bench-phrasings.json'
+    ).toEqual([]);
+  }, 120_000);
+});

package/src/agent-bench/cli.ts

@@ -0,0 +1,315 @@
+#!/usr/bin/env tsx
+/**
+ * Agent-loop benchmark CLI.
+ *
+ *   verify-references            every reference parses and has a usable signature
+ *   list [--json]                the task prompts, for a generator to answer
+ *   feedback --task <id> --code <src>   what the loop hands back for one candidate
+ *   score --run <file>           score a run file; prints per-condition rates + delta
+ *
+ * The generator is an AGENT, not this script: the loop being measured is the one
+ * a real integration runs, so simulating it in-process would measure a
+ * simulation. `list` emits the prompts, the agent answers them, `score` grades
+ * the answers. See README.md for the protocol.
+ *
+ * `feedback` deliberately returns ONLY what the MCP loop returns — diagnostics
+ * and the parsed IR. It never reveals the reference or whether the candidate
+ * behaves correctly. An agent iterating with `feedback` therefore has exactly
+ * the information the real loop gives it; leaking the behavior verdict would
+ * turn the loop condition into an oracle and the headline delta into fiction.
+ */
+
+import { readFileSync } from 'node:fs';
+import { TASKS, taskById } from './tasks.js';
+import { VARIANTS } from './variants.js';
+import {
+  bandOf,
+  executeCandidate,
+  scoreCandidate,
+  scoreCondition,
+  validateCandidate,
+  type Band,
+  type ConditionScore,
+} from './harness.js';
+
+interface RunFile {
+  generator?: Record<string, unknown>;
+  conditions: Record<string, Record<string, string>>;
+}
+
+function arg(name: string): string | undefined {
+  const i = process.argv.indexOf(`--${name}`);
+  return i === -1 ? undefined : process.argv[i + 1];
+}
+
+const pct = (n: number): string => `${(n * 100).toFixed(0)}%`;
+
+// =============================================================================
+// verify-references
+// =============================================================================
+
+async function verifyReferences(): Promise<number> {
+  let bad = 0;
+  const seen = new Map<string, string>();
+  for (const task of TASKS) {
+    const validation = await validateCandidate(task.reference);
+    const { effects, error } = await executeCandidate(task, task.reference);
+    const problems: string[] = [];
+    if (!validation.ok) problems.push('reference does not parse');
+    if (effects.length === 0) problems.push('reference produces NO effect signature');
+    if (error) problems.push(`error: ${error}`);
+    // Two tasks with identical signatures would let a candidate score right on
+    // the wrong task; not fatal (fixtures legitimately overlap) but worth saying.
+    const key = JSON.stringify(effects);
+    const twin = seen.get(key);
+    if (effects.length > 0 && twin) problems.push(`signature identical to ${twin}`);
+    if (effects.length > 0) seen.set(key, task.id);
+
+    if (problems.length > 0) {
+      bad++;
+      console.log(`✗ ${task.id}: ${problems.join('; ')}`);
+      console.log(`    ${task.reference}`);
+    } else {
+      console.log(`✓ ${task.id}  ${effects.length} effect(s)`);
+    }
+  }
+  console.log(
+    bad === 0
+      ? `\nAll ${TASKS.length} references usable.`
+      : `\n${bad}/${TASKS.length} references UNUSABLE — fix before scoring.`
+  );
+  return bad === 0 ? 0 : 1;
+}
+
+// =============================================================================
+// list / feedback
+// =============================================================================
+
+function list(): number {
+  if (process.argv.includes('--json')) {
+    console.log(
+      JSON.stringify(
+        TASKS.map(t => ({ id: t.id, prompt: t.prompt })),
+        null,
+        2
+      )
+    );
+    return 0;
+  }
+  for (const t of TASKS) console.log(`${t.id}\n  ${t.prompt}\n`);
+  return 0;
+}
+
+async function feedback(): Promise<number> {
+  const id = arg('task');
+  const code = arg('code');
+  if (!id || code === undefined) {
+    console.error('usage: feedback --task <id> --code "<hyperscript>"');
+    return 2;
+  }
+  if (!taskById(id)) {
+    console.error(`unknown task: ${id}`);
+    return 2;
+  }
+  const v = await validateCandidate(code);
+  console.log(
+    JSON.stringify(
+      { ok: v.ok, confidence: v.confidence, parsed: v.summary, diagnostics: v.diagnostics },
+      null,
+      2
+    )
+  );
+  if (!v.ok) {
+    console.log(
+      '\nNext step: apply the diagnostics above and re-run. get_code_fixes maps ' +
+        'error codes to concrete fixes; get_command_docs lists per-command roles.'
+    );
+  }
+  return 0;
+}
+
+// =============================================================================
+// score
+// =============================================================================
+
+function reportCondition(c: ConditionScore): void {
+  console.log(`\n── ${c.condition} ──`);
+  for (const s of c.scores) {
+    const mark = s.behaviorMatch ? '✓' : s.parsed ? '~' : '✗';
+    console.log(`${mark} ${s.taskId.padEnd(22)} ${s.code}`);
+    if (!s.behaviorMatch) {
+      if (!s.parsed) {
+        const first = s.validation.diagnostics.find(d => d.severity === 'error');
+        console.log(`    did not parse: ${first?.message ?? 'unknown'}`);
+      } else {
+        console.log(`    PARSED BUT WRONG — ${s.validation.summary ?? '?'}`);
+        console.log(`      got      ${JSON.stringify(s.execution.effects)}`);
+        console.log(`      expected ${JSON.stringify(s.referenceEffects)}`);
+        if (s.execution.error) console.log(`      error: ${s.execution.error}`);
+      }
+    }
+  }
+  for (const m of c.missing) console.log(`✗ ${m.padEnd(22)} (no candidate submitted)`);
+  console.log(
+    `\n  parse rate     ${pct(c.parseRate)}` +
+      `\n  behavior rate  ${pct(c.behaviorRate)}` +
+      `\n  parsed-but-wrong ${c.silentlyWrongCount}`
+  );
+}
+
+async function score(): Promise<number> {
+  const file = arg('run');
+  if (!file) {
+    console.error('usage: score --run <file.json>');
+    return 2;
+  }
+  const run = JSON.parse(readFileSync(file, 'utf8')) as RunFile;
+  if (run.generator) console.log(`generator: ${JSON.stringify(run.generator)}`);
+
+  const results: ConditionScore[] = [];
+  for (const [condition, candidates] of Object.entries(run.conditions)) {
+    const c = await scoreCondition(condition, candidates, TASKS);
+    results.push(c);
+    reportCondition(c);
+  }
+
+  if (results.length >= 2) {
+    const [first, last] = [results[0]!, results[results.length - 1]!];
+    console.log(`\n══ ${first.condition} → ${last.condition} ══`);
+    console.log(`  parse rate     ${pct(first.parseRate)} → ${pct(last.parseRate)}`);
+    console.log(`  behavior rate  ${pct(first.behaviorRate)} → ${pct(last.behaviorRate)}`);
+    console.log(
+      `  parsed-but-wrong ${first.silentlyWrongCount} → ${last.silentlyWrongCount}   (over ${TASKS.length} tasks)`
+    );
+  }
+  return 0;
+}
+
+// =============================================================================
+// probe-variants
+// =============================================================================
+
+/**
+ * Score the plausible-phrasing catalogue. Needs no generator: each row is a
+ * fixed string, so the result is a reproducible property of the parser.
+ *
+ * The headline is the SILENT band — phrasings that parse clean, carry no
+ * warning, and misbehave. Those are invisible to the validate/repair loop by
+ * construction, so they bound how much the loop can ever deliver. Arc 3b work
+ * moves rows out of it into `warned-wrong` (wrong but visible), which the loop
+ * handles.
+ */
+async function probeVariants(): Promise<number> {
+  const rows: Array<{
+    v: (typeof VARIANTS)[number];
+    parsed: boolean;
+    behaved: boolean;
+    band: Band;
+    detail: string;
+  }> = [];
+
+  for (const v of VARIANTS) {
+    const task = taskById(v.taskId);
+    if (!task) {
+      console.error(`unknown task in variants: ${v.taskId}`);
+      return 2;
+    }
+    const s = await scoreCandidate(task, v.code);
+    const band = bandOf(s);
+    const detail =
+      band === 'correct'
+        ? 'ok'
+        : band === 'rejected'
+          ? 'rejected'
+          : band === 'warned-wrong'
+            ? `WARNED (${s.validation.diagnostics.find(d => d.severity !== 'info')?.code ?? '?'}) — wrong, but the loop can see it`
+            : band === 'silent-noop'
+              ? 'SILENT NO-OP'
+              : `WRONG EFFECT ${JSON.stringify(s.execution.effects)}`;
+    rows.push({ v, parsed: s.parsed, behaved: s.behaviorMatch, band, detail });
+  }
+
+  const n = rows.length;
+
+  if (process.argv.includes('--json')) {
+    // Baseline shape: sorted by code so the file is diff-stable, and carrying
+    // only the deterministic verdict (never timings) so regeneration on another
+    // machine produces a byte-identical file. Bands come from harness.bandOf —
+    // the same function the ratchet test recomputes with.
+    console.log(
+      JSON.stringify(
+        {
+          note:
+            'Deterministic parse-vs-behavior verdicts for plausible phrasings. ' +
+            'Regenerate with: tsx src/agent-bench/cli.ts probe-variants --json',
+          totals: {
+            phrasings: n,
+            parse: rows.filter(r => r.parsed).length,
+            behaveCorrectly: rows.filter(r => r.behaved).length,
+            warnedWrong: rows.filter(r => r.band === 'warned-wrong').length,
+            silentBand: rows.filter(r => r.band.startsWith('silent')).length,
+            silentNoop: rows.filter(r => r.band === 'silent-noop').length,
+          },
+          phrasings: rows
+            .map(r => ({ taskId: r.v.taskId, code: r.v.code, band: r.band }))
+            .sort((a, b) => (a.code < b.code ? -1 : a.code > b.code ? 1 : 0)),
+        },
+        null,
+        2
+      )
+    );
+    return 0;
+  }
+
+  for (const r of rows) {
+    const mark = r.behaved
+      ? '✓'
+      : r.band === 'warned-wrong'
+        ? '⚠'
+        : r.band === 'rejected'
+          ? '✗'
+          : '☠';
+    console.log(`${mark} ${r.v.taskId.padEnd(20)} ${r.v.code}`);
+    if (!r.behaved) console.log(`    ${r.detail}  — ${r.v.rationale}`);
+  }
+
+  const parsed = rows.filter(r => r.parsed).length;
+  const behaved = rows.filter(r => r.behaved).length;
+  const warned = rows.filter(r => r.band === 'warned-wrong').length;
+  const silent = rows.filter(r => r.band.startsWith('silent')).length;
+  const noop = rows.filter(r => r.band === 'silent-noop').length;
+  console.log(
+    `\n  ${n} plausible phrasings` +
+      `\n  parse            ${parsed}/${n} (${pct(parsed / n)})` +
+      `\n  behave correctly ${behaved}/${n} (${pct(behaved / n)})` +
+      `\n  ⚠ wrong but WARNED (loop can react): ${warned}/${n}` +
+      `\n  ☠ wrong and SILENT: ${silent}/${n} (${pct(silent / n)}) — of which ${noop} do NOTHING at all` +
+      `\n\n  The ☠ band is what the validate/repair loop cannot see: no diagnostic,` +
+      `\n  no error, nothing for an agent to react to.`
+  );
+  return 0;
+}
+
+// =============================================================================
+
+const commands: Record<string, () => Promise<number> | number> = {
+  'verify-references': verifyReferences,
+  'probe-variants': probeVariants,
+  list,
+  feedback,
+  score,
+};
+
+async function main(): Promise<void> {
+  const cmd = process.argv[2] ?? '';
+  const run = commands[cmd];
+  if (!run) {
+    console.error(`usage: cli.ts <${Object.keys(commands).join('|')}> [options]`);
+    process.exit(2);
+  }
+  // Explicit exit: esbuild's daemon keeps the event loop alive (see CLAUDE.md),
+  // so a natural return would hang the process after the report is printed.
+  process.exit(await run());
+}
+
+void main();