@mutagent/evaluator 0.1.0-alpha.6 → 0.2.0-alpha.1

package/.claude/skills/mutagent-evaluator/SKILL.md

@@ -1,24 +1,19 @@
 ---
 name: mutagent-evaluator
 description: |
-  Evaluation-development engine for AI agents and skills. Turns a subject (a skill/agent +
-  its traces) into a TRUSTWORTHY eval suite: deep-reads traces to determine success/failure
-  (even when nothing is user-marked), mines emergent BINARY ACTIONABLE criteria from the
-  ✓/✗ split, builds one critique-before-verdict LLM-judge (or code-check) per criterion,
-  validates each judge against human labels (TPR/TNR · Rogan-Gladen · bootstrap CI), then
-  runs the suite against a target → per-criterion binary+confidence → severity-gated GATE
-  verdict + agent-variance view. A judge is ONLY a judge: failures are flagged and ROUTED to
-  diagnostics, never fixed here (EV-051). First invocation auto-detects the subject + the
-  framework-substrate choice (DEFAULT agent-dispatch — host-runtime subagents, no provider ·
-  in-house AI-SDK judge optional · code-based · user's framework export).
-  Subsequent invocations dispatch the parent session + MASS-PARALLEL evaluator
-  sub-agents (one cell, discover / judge modes; host runtime, harness-capped) against the subject's traces.
-  DEFAULT INTENT: any "evaluate this skill/agent" request runs *evaluate — the JUDGE — which is what an
-  end user wants. *audit is a SEPARATE, explicit-only 4-tab static-auditor (eval-matrix · data-leak ·
-  variance · methodology) used mainly for complex agents/skills that emit HTML artifacts; it is NEVER the
-  fallback for a plain evaluate. Pinned judge model + temp=0
-  for byte-identical reruns (C-PIN). Usable standalone OR borrowed by mutagent-skill-builder's
-  SIMULATE phase as the SimVerifier.
+  The EVALUATE stage of the agentic development lifecycle (ADL ③). Builds and runs
+  trustworthy evaluations for AI agents and skills: from a subject's real traces it
+  learns what success and failure look like, turns that into binary eval criteria,
+  and scores the subject to a clear gate verdict. It only judges — failures are
+  flagged and routed to mutagent-diagnostics, never fixed here. Pick it whenever the
+  ask is to evaluate, judge, score, or build and validate evals for an agent or
+  skill: any plain "evaluate this" runs *evaluate — one judge per trajectory walks
+  the evidence layers (code checks → outcome → trajectory → tool outputs → context),
+  criteria binding evidence across all of them; *evaluate-<layer> scopes the same
+  pipeline to one layer; *discover mines new criteria, never re-recommending one it
+  already has; *audit is a separate, explicit-only static auditor and never the
+  fallback for a plain evaluate. Also serves as the verifier inside
+  mutagent-skill-builder's SIMULATE.
 license: Proprietary. LICENSE.txt has complete terms.
 compatibility: Designed for Claude Code, Codex, Cursor, OpenCode and similar coding-agent runtimes; works with git, gh CLI, jq, curl, and Bun/pnpm/npm runtimes.
 metadata:
@@ -102,11 +97,12 @@ reason for the rest. NEVER improvise.
 |---------|------|-------------------|---------|
 | `*discover-evals` | hybrid | `references/workflows/orchestrator-protocol.md` + `scripts/prep-tasks.ts` + `scripts/determine-outcome.ts` + `scripts/discover-criteria.ts` + `assets/agents/evaluator.md` (`#mode-discover`) | PREP determiner tasks → fan out `evaluator` (`#mode-discover`, mass-parallel) → AGGREGATE ✓/✗ → mine emergent BINARY ACTIONABLE criteria (EV-041/042/052) **+ T6 failure/uncertain DATASET CANDIDATES** (`collectDatasetCandidates`, reuses the derive-dataset selectors → `*build-dataset`). **GA split** — **code:** sample (broken+healthy) · aggregate · `diff-discriminate` · ground-gate · dataset-candidates; **LLM leaf:** determine + 3 detectors + cite refs + typed assumptions (root-not-symptom). |
 | `*build-evals` | hybrid | `references/workflows/orchestrator-protocol.md` + `scripts/eval-engine.ts` + `scripts/codegen-evals.ts` + `scripts/prep-tasks.ts` + `scripts/build-evals.ts` + `assets/agents/evaluator.md` (`#mode-judge-criterion`) + `scripts/render-build-cards.ts` | **ENGINE-FORK FIRST (ADL F7/F9/F14).** ASKS the eval implementation mode via `chooseEvalEngineOptions(target)` — Path A `native-matrix` (eval-matrix + LLM-judge SUB-AGENTS; SURFACES the Claude-Code + log-sink dependency up front) vs Path B `code-written` (`codegen-evals.ts` emits a portable bun/TS suite — runs WITHOUT Claude Code, F14). Target-conditional: a code framework offers BOTH, a `harness:*` target is native-only. **Path A** then PREPs judge tasks (criterion × trace-slice) → fan out `evaluator` (`#mode-judge-criterion`, mass-parallel) → one binary+confidence judge per criterion (EV-043). **GA split** — **code:** engine-resolve · spec render · `lint-grounding` · `resolve-ref` (BIND) · codegen; **LLM leaf:** judge + VERIFY (cite refs · note assumptions · abstain). Streams progress + a verbose evals entity card (F13/F16/F22). |
-| `*evaluate` | hybrid | `references/workflows/orchestrator-protocol.md` + `scripts/matrix-judge.ts` + `scripts/contracts/eval-matrix.ts` + `assets/agents/evaluator.md` (`#mode-judge-trajectory`) + `scripts/evaluate.ts` | **DEFAULT (headline):** **T1 TIER-0 deterministic pre-pass** (code-method rows run first, zero judge tokens) → PREP one matrix packet per RESIDUAL judge trajectory (**T5 adaptive-K guard, default 1:1**) → fan out `evaluator` (`#mode-judge-trajectory` — one judge/trajectory scoring the WHOLE matrix, emitting the **T2 Judge DAG v2 walk** = `judge_steps[]` + dense na-explicit map + confidence band + early-INCOMPLETE + node-2.5 candidates) → AGGREGATE: fold code+judge verdicts → **T3 independent verifier** refutes GATING fails (downgrade-only) → **T4 consolidate-by-locus + walk-derived health** → GATE verdict + variance view (EV-048). **GA split** — **code:** tier-0 · prep · `lint-grounding` · `resolve-ref` · `assemble-scorecard` (gate `fail ▸ incomplete ▸ pass`) · consolidate-by-locus · route-failures; **LLM leaf:** trajectory judge (DAG v2 walk: BIND · GROUND[absence-split] · cite · abstain) + independent VERIFY. **MAY now return `incomplete`** (indeterminate → calibrate). **ADL F20:** the rollup ALSO renders a SCORECARD DASHBOARD wireframe (`renderScorecardDashboard` — per-criterion pass/fail bar + variance + samples), not a flat terminal dump. Under a **Path B** engine, `*evaluate` instead runs the portable `codegen-evals.ts` suite (no Claude Code) and reads back its scorecard JSON from the discoverable sink. |
+| `*evaluate` | hybrid | `references/workflows/orchestrator-protocol.md` + `scripts/matrix-judge.ts` + `scripts/contracts/eval-matrix.ts` + `assets/agents/evaluator.md` (`#mode-judge-trajectory`) + `scripts/evaluate.ts` | **DEFAULT (headline):** **T1 TIER-0 deterministic pre-pass** (code-method rows run first, zero judge tokens) → PREP one matrix packet per RESIDUAL judge trajectory (**T5 adaptive-K guard, default 1:1**) → fan out `evaluator` (`#mode-judge-trajectory` — one judge/trajectory scoring the WHOLE matrix, emitting the **T2 Judge DAG v2 walk** = `judge_steps[]` + dense na-explicit map + confidence band + early-INCOMPLETE + node-2.5 candidates) → AGGREGATE: fold code+judge verdicts → **T3 independent verifier** refutes GATING fails (downgrade-only) → **T4 consolidate-by-locus + walk-derived health** → GATE verdict + variance view (EV-048). **GA split** — **code:** tier-0 · prep · `lint-grounding` · `resolve-ref` · `assemble-scorecard` (gate `fail ▸ incomplete ▸ pass`) · consolidate-by-locus · route-failures; **LLM leaf:** trajectory judge (DAG v2 walk: BIND · GROUND[absence-split] · cite · abstain) + independent VERIFY. **MAY now return `incomplete`** (indeterminate → calibrate). **ADL F20:** the rollup ALSO renders a SCORECARD DASHBOARD wireframe (`renderScorecardDashboard` — per-criterion pass/fail bar + variance + samples), not a flat terminal dump. Under a **Path B** engine, `*evaluate` instead runs the portable `codegen-evals.ts` suite (no Claude Code) and reads back its scorecard JSON from the discoverable sink. **v3 LAYERED WALK (D1=B+):** the trajectory judge performs the whole evidence-layer sequence INSIDE its one dispatch — L0 code checks → L1 outcome (scenario + expectedExit, cheap, first) → per stopping rules L2 trajectory → L3 tool outputs → L4 context — emitting `layerVerdicts[]` + `earlyExit` + `codeEvalHits[]` + an emissions self-manifest (E-NEW contract, `contracts/eval-matrix.ts`); AGGREGATE folds layers (`foldLayerVerdicts`), DETECTS cross-layer conflicts (`detectLayerConflicts` — surfaced for the human calibration queue, NEVER auto-resolved, D2-A/OT-1), enforces the COMPLETENESS LAW (`assertNoUnverdictedCriterion` THROWS on a silent skip), and folds MR-5 data-completeness (`foldEmissionsScorecard`). |
+| `*evaluate-outcome` · `*evaluate-trajectory` · `*evaluate-outputs` · `*evaluate-context` · `*evaluate-code` · `*evaluate-criteria` | hybrid | the SAME binds as `*evaluate` | **v3 PIPEABLE LAYER FILTERS (D3=A):** each runs the identical `*evaluate` pipeline with `layerScope` set in every packet (`outcome`→`[L1]` · `trajectory`→`[L2]` · `outputs`→`[L3]` · `context`→`[L4]` · `code`→`[L0]`); the judge runs ONLY those walk phases — **criteria are the SUPER-LAYER and are still verdicted over whatever those phases gather (never left unverdicted — the completeness law holds per scope)**. `*evaluate-criteria` = the criterion-axis pass (`#mode-judge-criterion`, single criterion × slice — the suite-build/validation path). Standalone-testable AND composable: a standalone layer verdict must equal the same layer's verdict inside the full walk (V9). |
 | `*optimize` | hybrid | `references/edd-loop.md` + `scripts/edd/variance-gate.ts` + `scripts/edd/change-request.ts` + `scripts/edd/edd-types.ts` + `schemas/edd-change-request.schema.yaml` + `assets/agents/evaluator.md` (`#mode-optimize`) | **ADL ③ OPTIMIZE / EDD loop (F18+F19).** **F19 VARIANCE-FIRST:** repeat-N (default 5) the SAME cases → `evaluateVarianceGate` → accuracy is entered ONLY when the variance gate passes (`assertVarianceStableBeforeAccuracy` THROWS otherwise — "accuracy over big samples is wasted on a flapping verdict"). **F18 CLOSURE:** still JUDGE-ONLY (EV-051), the evaluator emits a grounded `EddChangeRequest` (failing cases + `ref{obs,path,value}` + remedy target `agentspec`\|`impl`) to the `mutagent-builder ai-engineer` over **SendMessage**, consumes the `ChangeRequestResponse`, and re-evals on `amended`. **Bounded terminator** (`decideEddLoop`, afkloop-legal): `full-green` ▸ DONE \| `max-swings`\|`max-wallclock`\|`no-improvement-streak` ▸ STOPPED + convergence delta. **GA split** — **code:** variance gate · request/response validate · loop terminator (all PURE, injected wall-clock); **LLM leaf:** localize the flap/fail to its root + author the grounded request + decide remedy target. |
 | `*validate` | hybrid | `scripts/validate-judge.ts` + `references/validate-evaluator.md` + `assets/agents/evaluator.md` (`#mode-judge-criterion`) | **ENGINED (EV-044)** — calibrate a judge vs `*review` labels: confusion matrix → TPR/TNR · split-disjointness + test-once · Rogan-Gladen θ · deterministic bootstrap CI; `<MIN_LABELS` stays `unvalidated`+bias-corrected. **GA split** — **code:** entire path is deterministic (filter EXCLUDES indeterminate · confusion · RG · assumption-agreement). |
-| `*review` | hybrid | `scripts/build-review-ui.ts` + `references/build-review-interface.md` | **ENGINED (EV-045)** — CODE renders a browser annotation UI (one trace/screen · Pass/Fail/Defer · keyboard · auto-save · labels export); **HITL**: a human labels → `mergeLabels` persists → feeds `*validate`. |
-| `*eval` | agent-chain | `references/eval-stage.md` + `scripts/eval-engine.ts` + `scripts/materialize-dataset.ts` + `scripts/render-build-cards.ts` | **THE ADL EVAL-STAGE ENTRY (F15).** Called after `*build` hands a freshly-built agent + its agentspec. INTERACTIVELY offers `*build-dataset` / `*build-evals` derived from `agentspec.definition.evals`, and lets the user PICK THE EVAL ENGINE (mutagent native eval-matrix [Path A] vs code-written-in-target-lang [Path B]) via the target-conditional menu (`chooseEvalEngineOptions`). Surfaces Path A's Claude-Code + log-sink dependency UP FRONT (F7/F9). Streams wireframe progress + entity cards (F13/F16/F22). Parent-session only (AskUserQuestion). |
+| `*review` | hybrid | `scripts/run-review.ts` (run composer) + `scripts/render-review-report-v3.ts` (surface) + `scripts/build-review-ui.ts` (v2 fallback) + `references/build-review-interface.md` | **ENGINED (EV-045)** — CODE renders a browser review surface; **HITL**: a human labels → `mergeLabels` persists → feeds `*validate`. **Entry point:** `bun scripts/run-review.ts <runId> [--cwd <repoRoot>]` — reads an EXISTING `*evaluate` run dir (`run-input.full.json` + `verdicts/*.verdict.json`; joins on the parsed `trajectoryId`, never the packet filename) and writes `review-report.html` beside `evaluation-report.html`, plus `review-report.v2.html` from the previous annotation UI (module untouched). Dispatches NO judge — it costs nothing and invents no artifacts. **Surface:** navigator matrix (criteria rows × trace columns) → single-trace drill, right-rail verdict cards with Agree/Revise/Refute + why-note, keys `1`/`2`/`D`, >48 traces → window-cluster, submit → **calibration handoff markdown** (diagnostics remedy-bundle shape) server-generated from the run's own artifacts. Ingested-but-unjudged trajectories are NAMED in a scope note, never dropped. **Does NOT do W4:** no feedback store, no ruling persistence, no propagation into `*validate` — producing the artifact is the scope; consuming a reviewer's answer is task #9. |
+| `*eval` | agent-chain | `references/eval-stage.md` + `scripts/eval-engine.ts` + `scripts/materialize-dataset.ts` + `scripts/render-build-cards.ts` | **THE ADL EVAL-STAGE ENTRY (F15).** Called after `*build` hands a freshly-built agent + its agentspec. INTERACTIVELY offers `*build-dataset` / `*build-evals` derived from `agentspec.spec.evaluation`, and lets the user PICK THE EVAL ENGINE (mutagent native eval-matrix [Path A] vs code-written-in-target-lang [Path B]) via the target-conditional menu (`chooseEvalEngineOptions`). Surfaces Path A's Claude-Code + log-sink dependency UP FRONT (F7/F9). Streams wireframe progress + entity cards (F13/F16/F22). Parent-session only (AskUserQuestion). |
 | `*build-dataset` | hybrid | `scripts/materialize-dataset.ts` + `scripts/build-dataset.ts` + `assets/agents/dataset-builder.md` + `schemas/dataset.schema.yaml` + `references/generate-synthetic-data.md` + `scripts/render-build-cards.ts` | **ENGINED (EV-046) + ADL F8.** Now MATERIALIZES real items first: `materializeFromAgentspec` seeds ≥1 REAL DatasetCase per `dataset_category` + one per `edge_case` (seed → actual items, not definitions) → then the 3-way: HITL seed interview (~10) → `dataset-builder` agent (tuples → NL queries → realism filter) → CODE cartesian-expand + near-dup drop + monotonic merge. Streams wireframe progress cards (F13/F16) + a verbose dataset entity card (F22). |
 | `*discover-dataset` | script | `scripts/derive-dataset.ts` | **ENGINED (EV-047) · re-fronts the old `*derive-dataset`** — distill a living regression set from labeled ✓/✗ traces (reuses `sample-traces.ts` EV-052 selectors + `build-dataset.ts` merge). Code-only. **P-B:** consumes the `discovery` agent's curated `SelectionManifest` handoff — **scenario-balanced** (`ext.classification.scenario`), **edge-cases first** (`ext.classification.edgeCase.is`), worthiness-prioritized (`ext.signals.worthinessScore`). |
 | `*audit` | agent-chain | `assets/agents/audit-executor.md` + `workflows/{audit,data-leak,variance}.workflow.js` + `scripts/{flow-graph,ui-slots}.ts` + `lenses/context-flow-lens.md` | SURFACES the v1 4-tab static-auditor (EV-001..027, KEEP) **+ now the agent context-flow audit**: tool-result threading + sub-agent handoff completeness (EV-028/029, over the `flow-graph` EV-032 + expected-flow EV-037) and the first-class **HTML-artifact missing-data** dimension (computed-but-not-rendered / orphan / faithfulness, EV-039/040, subject-agnostic). |
@@ -164,8 +160,8 @@ evaluate spine and never an automatic fallback.
 
 **ADL EVAL-stage entry (`*eval`).** When `*build` hands over a freshly-built agent +
 its agentspec, `*eval` is the entry: it interactively offers `*build-dataset` /
-`*build-evals` derived from `agentspec.definition.evals` (F15), MATERIALIZES real
-dataset items per category + edge-case (F8), lets the user PICK THE EVAL ENGINE
+`*build-evals` derived from `agentspec.spec.evaluation` (F15), MATERIALIZES real
+dataset cases by projecting the 0.3.0 `datasets[].items[]` (F8), lets the user PICK THE EVAL ENGINE
 (Path A native eval-matrix vs Path B code-written — surfacing Path A's Claude-Code +
 log-sink dependency up front, F7/F9/F14), streams wireframe progress + entity cards
 (F13/F16/F22), and renders the scorecard as a dashboard (F20). See
@@ -287,7 +283,7 @@ applies, EV-053/EQ5):
 |--------|---------|------|---------|
 | `scripts/contracts/eval-engine.ts` | F7/F9/F14 | code | The ENGINE FORK contract — `EvalEngine` (`native-matrix` Path A · `code-written` Path B) + `EngineTargetInput` (the minimal agentspec.build slice) + `EvalEngineOption` (the menu, with `requiresClaudeCode`/`requiresLogSink`/`portable` SURFACED) + `EvalEnginePlan` (the resolved choice incl. discoverable `outputSink`). |
 | `scripts/eval-engine.ts` | F7/F9/F14 | code | The target-conditional resolver — `isFrameworkTarget` · `chooseEvalEngineOptions` (framework→BOTH · harness→native-only) · `resolveEvalEngine` (surfaces CC + log-sink deps) · `assertEngineMatchesTarget` (code-written⊥harness, fail-loud) · `defaultOutputSink`. PURE. |
-| `scripts/contracts/agentspec-evals.ts` | F8 | code | The MINIMAL `agentspec.definition.evals` slice (`success_criteria[]` · `scenarios[]` · `dataset_categories[]`) — local re-declaration (standalone: NEVER imports the agentspec skill). `parseAgentspecEvals` guarded. |
+| `scripts/contracts/agentspec-evals.ts` | F8 | code | The MINIMAL `agentspec.spec.evaluation` slice (0.3.0 D18/D19 — `criteria[]` · `scenarios[]` · `datasets[]` with dataset-local `categories[]` + `caseDimensions` + real `items[]`) — local re-declaration (standalone: NEVER imports the agentspec skill). `parseSpecEvaluation` guarded. |
 | `scripts/materialize-dataset.ts` | F8 | code | MATERIALIZE real dataset items from the agentspec — `materializeFromAgentspec` (≥1 real item/category + one/edge_case, seed→actual) · `dimensionsFromAgentspec` · `materializeToDataset` (monotonic). Reuses `build-dataset.ts` ids/dedup/merge. PURE. |
 | `scripts/codegen-evals.ts` | F14 (Path B) | code | Emit a PORTABLE code-written eval suite source (bun/TS) — `codegenEvalSuite` (code-checks + SDK judge + criteria → gate-bearing scorecard to a discoverable sink). NO CC dispatch. Fail-loud, deterministic. PURE. |
 | `scripts/render-build-cards.ts` | F13/F16/F20/F22 | code | The wireframe terminal surface — progress cards (`renderBuildDatasetProgressCard`/`renderBuildEvalsProgressCard`, F13/F16) · entity cards (`renderDatasetEntityCard`/`renderEvalsEntityCard`, F22) · the scorecard DASHBOARD (`renderScorecardDashboard` — pass/fail bar + variance + samples, F20, not a flat dump). PURE. |
@@ -296,7 +292,9 @@ applies, EV-053/EQ5):
 
 | Script | Req | Kind | Purpose |
 |--------|-----|------|---------|
-| `scripts/build-review-ui.ts` | EV-045 | code | The `*review` Code half — `renderReviewUi` emits a DETERMINISTIC browser annotation UI (one trace/screen, native render, Pass/Fail/Defer, notes, keyboard, localStorage auto-save, labels export) + `mergeLabels` (dedup by traceId). Holds no judge prompt; the HUMAN decides. |
+| `scripts/build-review-ui.ts` | EV-045 | code | The `*review` Code half — `renderReviewUi` emits a DETERMINISTIC browser annotation UI (one trace/screen, native render, Pass/Fail/Defer, notes, keyboard, localStorage auto-save, labels export) + `mergeLabels` (dedup by traceId). Holds no judge prompt; the HUMAN decides. Retained as the `review-report.v2.html` transition fallback under the W4 surface. |
+| `scripts/run-review.ts` | EV-045 | code | **The `*review` RUN COMPOSER** — the analogue of run-evaluate's `writeRunReport`. Reads an existing run dir (run-input + `verdicts/*.verdict.json`, excluding the `*.verify.json` ledger) and writes `review-report.html` + the v2 fallback. Joins verdicts to traces on the parsed `trajectoryId`, NOT the packet filename; treats the lean run-input's `"<N EvalTraces>"` placeholder as ABSENT rather than data; NAMES ingested-but-unjudged trajectories in a scope note. Fail-loud on a missing run dir / run-input / verdicts / criteria, and a malformed verdict names its FILE. Dispatches no judge. |
+| `scripts/render-review-report-v3.ts` | EV-045 | code | The W4 frozen-contract review surface (navigator + drill · rail verdict cards · rulings · calibration handoff MD) + `REVIEW_REPORT_SPEC`, the surface's verification spec, exported here so the unit test and the composer assert the same shape. |
 | `scripts/build-dataset.ts` | EV-046 | code | The `*build-dataset` Code half — cartesian dimension×value expand · deterministic token-Jaccard near-dup removal · content-derived id · `mergeCases` MONOTONIC merge. Agent proposes realism; script enforces non-redundancy. |
 | `scripts/derive-dataset.ts` | EV-047 | code | `*discover-dataset` — distill a living regression set from past ✓/✗ (reuses `sample-traces.ts` selectors + `build-dataset.ts` merge); trace→`DatasetCase`. Append-only, no new decision. |
 | `scripts/living-suite.ts` | EV-053 | code | Generic append-only writer (`appendOnly` + `assertMonotonicGrowth`) shared by datasets + criteria — a living artifact NEVER shrinks. Pure-counter provenance (no clock → C-PIN byte-identity). |

package/.claude/skills/mutagent-evaluator/assets/agents/dataset-builder.md

@@ -131,11 +131,12 @@ not run the subject and does not decide pass/fail.
 1. **Pre-read** `references/generate-synthetic-data.md` — the dimension → tuple → query → filter
    process is the lens. Read the passed `dimensions` + `seed_tuples`.
    - **ADL F8 — agentspec-materialized seeds.** In the ADL EVAL stage the parent ALSO PREPs real
-     items MATERIALIZED from `agentspec.definition.evals` (`scripts/materialize-dataset.ts`): ≥1 real
-     `DatasetCase` per `dataset_category` + one per `edge_case`. When present, treat these
-     materialized cases as ADDITIONAL realism anchors (alongside the HITL `seed_tuples`) — they are
-     real queries from the spec, NOT definitions. EXPAND around them; do not re-derive them. The
-     deterministic dedup/merge (`build-dataset.ts`) drops any overlap.
+     items PROJECTED from `agentspec.spec.evaluation` (0.3.0, `scripts/materialize-dataset.ts`): one
+     real `DatasetCase` per `datasets[].items[]` entry — tuple = the item's `category` + `case` axis
+     assignment, query = the item's opaque `input` payload. When present, treat these materialized
+     cases as ADDITIONAL realism anchors (alongside the HITL `seed_tuples`) — they are real queries
+     from the spec, NOT definitions. EXPAND around them; do not re-derive them. The deterministic
+     dedup/merge (`build-dataset.ts`) drops any overlap.
 2. **Generate more tuples (Step 3).** Produce `target_count` new `(dim1, dim2, dim3, …)` combinations
    for the subject. Avoid duplicates of the seed + each other; vary values across dimensions. This is
    ONE step — tuples only, no queries yet.

package/.claude/skills/mutagent-evaluator/assets/agents/evaluator.md

@@ -224,11 +224,15 @@ optional_verdict_fields:
 # structured refs + typed assumptions). A LIGHTWEIGHT / FLATTENED emit that drops discovery.evidence.refs
 # or discovery.assumptions is a DEFECT — the gate (lint-grounding) + diff-discriminate operate on those
 # fields; stripping them silently downgrades every criterion to inferred.
-required_label_fields: [traceId, verdict, rootCause, refs, evidencePointer]         # discover — `rootCause` REPLACES firstThingWrong (root-not-symptom); `refs` = structured grounding
+required_label_fields: [traceId, verdict, firstThingWrong, evidencePointer, refs]   # discover — the EXACT keys DiscoverLabelSchema accepts (closed object); `firstThingWrong` HOLDS THE ROOT, not the first visible symptom (root_not_symptom); `refs` = structured grounding
 required_category_fields: [name, definition, class, fixOrEval, exampleTraceIds]     # discover
 # optional_category_fields (P2/P2b, ADDITIVE — the leaf MAY emit these §5b metric-metadata
-# PROPOSALS; AGGREGATE (scripts/aggregate-discover.ts) applies GENERIC defaults when absent,
-# so emitting them is OPTIONAL and backward-compatible): [dimension, level, generality, severity, judgeInputs, codeEval]
+# PROPOSALS; AGGREGATE (scripts/aggregate-discover.ts) HONORS what is present and applies GENERIC
+# defaults ONLY where absent, so emitting them is OPTIONAL and backward-compatible):
+# [dimension, level, generality, severity, judgeInputs, codeEval]
+# Each is a CLOSED union (below) — a value outside it fails LOUD at parse time rather than being
+# silently ignored. (Until the P2 passthrough fix the JOIN dropped all four, so the defaults applied
+# unconditionally and every mined criterion read `level: output` no matter what the leaf proposed.)
 #   dimension  ∈ operation-correctness | data-correctness | operational-deviation
 #   level      ∈ context | output | cross-stage
 #   generality ∈ general-structural | specific-semantic    (else derived from class: code→general-structural)
@@ -310,6 +314,7 @@ commands:
         - "references/grounded-adjudication.md"  # GA doctrine: bind · gather refs · typed assumptions · abstain
         - "subjects/<name>/"                    # the auto-generated subject profile / event taxonomy
         - "references/workflows/orchestrator-protocol.md"  # Step 1 dispatch FSM
+        - "<run_dir>/existing-criteria.json"    # SUITE-AWARENESS — the criteria that ALREADY exist (see suite_awareness)
       workflow:
         - "Pre-read references/error-analysis.md + references/grounded-adjudication.md + the subjects/<name>/ profile + your assigned <key>.task.json determiner specs"
         - "Determine outcome per trace (EV-042) — event→intended goal, trajectory→what happened, terminal→verdict; a guard-hold is a PASS"
@@ -319,14 +324,35 @@ commands:
         - "Trace each Fail to its ROOT (root-not-symptom): one criterion per root; multiple INDEPENDENT roots ⇒ multiple criteria; a causal edge must be GROUNDED (cite it) or surfaced as a typed assumption; deep recursive-why → route to diagnostics"
         - "Surface TYPED assumptions (factual-intent · normative · scope) for any premise the trace did not establish"
         - "Cluster notes into 5-10 emergent categories; tag class (objective→code/subjective→judge/hybrid) + fixOrEval"
+        - "NAME each cluster suite-aware (AFTER clustering, never before — see suite_awareness): if an EXISTING criterion in existing-criteria.json already states this behaviour, REUSE its exact id as the category name and attach your traces as REINFORCING evidence instead of minting a restatement under a new name"
+        - "Emit the §5b PROPOSALS you actually judged — level · severity · dimension · generality. AGGREGATE now HONORS them (they are no longer silently replaced by generic defaults), so an omitted field means 'derive it', not 'it does not matter'"
         - "Flag fixable-vs-eval-worthy (EV-051) — route fixable + infra-class to diagnostics; keep behavioral criteria"
         - "Write <key>.verdict.json (critique-before-verdict) + discover/<batch_id>.json (labels + FULL MinedCriterion per category, incl. discovery.evidence.refs + discovery.assumptions)"
+      # suite_awareness — the leaf SEES the criteria that already exist, so a behaviour
+      # already covered gains EVIDENCE instead of a second name for the same check.
+      # The parent writes `existing-criteria.json` (scripts/aggregate-discover.ts
+      # writeExistingSuiteBrief) into the run dir before dispatch: {id, statement,
+      # severity, level, checkedBy} per existing criterion — NO evidence, deliberately —
+      # PLUS `openQuestions[]`: pairs already awaiting a ruling on whether they are one
+      # check, each with the guard reason. Those carry only the pair + reason; both
+      # sides' statements are already in `criteria[]`, which is what keeps the whole
+      # brief evidence-free by construction.
+      suite_awareness:
+        - "WHEN: consult it AFTER you have clustered, to NAME a cluster. NEVER before."
+        - "WHY THAT ORDER IS LOAD-BEARING: reading the existing suite BEFORE the traces would seed a pre-defined failure list and re-introduce exactly the confirmation bias `emergent_categories_only` forbids — you would go hunting for the criteria you were shown. Clustering FIRST keeps the categories emergent; the suite is consulted only to decide what an already-emerged cluster should be CALLED. Emergence is preserved because the brief cannot influence what you found, only what you name it."
+        - "IT CARRIES NO EVIDENCE ON PURPOSE: the brief ships statements, not the traces or refs behind them, so it cannot tempt you into re-citing a failure you did not read this batch."
+        - "REINFORCE: if an existing criterion already states the behaviour your cluster found, emit the cluster under THAT criterion's exact id. Seeing the behaviour again is legitimate REINFORCING evidence — the parent unions your traces onto the existing criterion and the report marks it REINFORCED."
+        - "OPEN QUESTIONS: the brief's `openQuestions[]` lists pairs of EXISTING criteria already awaiting a human ruling on whether they are one check, each with the reason they were kept apart (e.g. `marker parity: [BEFORE] vs [AFTER]`). BOTH sides are live criteria and both statements are in `criteria[]`. If your cluster matches one side, emit it under THAT side's id — do NOT mint a third name for a behaviour that is already an open question, and do NOT try to settle the question yourself (you are not the ruling authority; that is the calibration loop's). If your cluster genuinely matches NEITHER side, mint it as new."
+        - "INFORMS, NEVER BLOCKS: a full-looking suite is not a reason to suppress a real finding. If the traces show a behaviour no existing criterion states, MINT IT — a genuinely new criterion is always correct, and the parent's deterministic merge is the safety net for a restatement you mint anyway."
+        - "NEVER FABRICATE: do not carry an existing criterion into your mining report because it is in the brief. You may only emit clusters backed by traces YOU actually read this batch — an id reused with no trace of your own behind it is an invented finding."
+        - "ABSENT/EMPTY brief ⇒ proceed exactly as before (a first run has no suite; that is normal, not a gap)."
       compresses:
         - "deep-read trace → goal-attainment Pass/Fail"
         - "DETECT across 3 lenses (drift · tool-output-failure · missing-context)"
         - "GATHER structured refs (claim + absence) · BIND criterion terms"
         - "root-not-symptom localization + typed-assumption surfacing"
         - "emergent category clustering"
+        - "suite-aware NAMING (reuse an existing id ⇒ REINFORCE, never restate)"
         - "fixable-vs-eval-worthy flag"
         - "emit FULL-MinedCriterion verdict + mining files"
       preserves: "the error-analyst discipline — folded INLINE as 'Mode: discover' (below); the standalone assets/agents/error-analyst.md was RETIRED in the 5→3 consolidation (Phase 3a, df6a6e8c8). This file is its canonical home."
@@ -403,13 +429,15 @@ commands:
         - "Pre-read references/write-judge-prompt.md (the 4-component judging contract — your lens) + references/grounded-adjudication.md"
         - "Read your assigned <trajectory_key>.packet.json (MatrixPacket: subject · trajectoryId · WHOLE matrix · trajectory · transcript · pinned envelope) — judge exactly this, never re-derive the data"
         - "Frame the trajectory in its ROUTE/intended-outcome (CONTEXT) before scoring"
-        - "Score EVERY criterion in the matrix for THIS trajectory: read only what the row needs (judgeInputs); compare against statement + passCondition"
+        - "v3 LAYERED WALK (D1=B+, operator-frozen): the walk IS a fixed layer sequence you run INSIDE this one dispatch. Honor `packet.layerScope` when present (absent/empty = FULL walk). The phases: (L0 code checks) if the code-eval library is present at scripts/code-eval-library.ts, RUN it yourself via Bash over the packet's trajectory JSON and record each hit as codeEvalHits[{pattern,anchor,detail}] — a fired pattern is evidence, not a verdict; (L1 outcome) classify the SCENARIO from the inbound intent + tool inventory, derive its expectedExit[] terminal actions, glance ONLY at end-state evidence and emit layerVerdicts[{layer:'L1',verdict,scenario,expectedExit,refs,note}]; (STOPPING RULES) if L1 passes AND every criterion is answerable from evidence gathered so far → you may set earlyExit:'L1' and mark L2-L4 as {verdict:'skipped'} — but a criterion may NEVER go unverdicted because the agent did the right thing; if ANY criterion is unanswerable OR L1 fails/undecidable → descend; (L2 trajectory) actual-vs-expected decision path, read from the END, emit layerVerdicts L2 (+ divergence in localize); (L3 tool outputs) per-observation structural validity + completeness — external-data correctness is UNVERIFIABLE (fence), a failed tool makes everything downstream suspect; emit layerVerdicts L3 (+firstProblemOrigin); (L4 context) for each critical operational send: was required context retrieved AND present in history AT THAT TIME (chronology-sorted!) — distinguish gapKind retrieval (never fetched) vs attention (present, ignored); emit layerVerdicts L4. Record layersEngaged. CONFLICT DISCIPLINE (OT-1): when your own layers disagree (e.g. L1 pass vs L2 fail) emit BOTH verdicts honestly with a note — NEVER reconcile them yourself; the aggregate surfaces the conflict for a human."
+        - "Score EVERY criterion in the matrix for THIS trajectory: read only what the row needs (judgeInputs); compare against statement + passCondition — criteria are the SUPER-LAYER: they may bind evidence from ANY walk phase"
         - "BIND (L1) per row: each criterion TERM must resolve in THIS trajectory; an unbound term ⇒ uncertain + blockedBy:{kind:factual-intent} (INDETERMINATE), ABSTAIN — never fail"
-        - "GATHER: cite a structured ref {obs,path,value} for the claim AND any absence; surface TYPED assumptions for any ungroundable premise → uncertain(blockedBy)"
+        - "GATHER: cite a structured ref {obs,path,value} for the claim AND any absence (ref.obs = the packet's trajectoryId, NEVER an observation name; ref.path = packet-shaped trajectory.N/transcript.N OR trace-shaped observations.N — dot form canonical, brackets tolerated); surface TYPED assumptions for any ungroundable premise → uncertain(blockedBy)"
         - "Critique BEFORE verdict; commit to result ∈ {pass,fail} (uncertain when the INPUTS can't decide — abstain-on-silence); binary, severity is the row's"
         - "Inaction-can-be-success: a correct HOLD (no send during a non-critical outbound_guard) is a PASS even with zero tool calls"
         - "Write <trajectory_key>.verdict.json (a MatrixVerdictFile {trajectoryId, judgeModel, temperature:0, verdicts[]} — each verdict may carry refs?/assumptions?/blockedBy?)"
         - "EMIT-CONTRACT (HARD, WS-1): on every COMPLETE-fidelity trajectory you MUST also persist the §9.4 walk — `understanding` (M2), `expectedTrajectory` (M3), `agentSteps` (the target step lane), and `judgeSteps` (your anchored reasoning). These are NOT optional: the parent runs a machine `assessEmitCompleteness` gate (scripts/emit-completeness.ts) that LOUDLY flags any wholly-dropped field, and a verdict missing them STARVES the report's Trajectory (§2) + Self-Eval (§5) tabs. `agentSteps` is factual trace data (reconstruct it from the ordered tool steps you were given — order by startTime, the observations array is reverse-chronological); M2/M3/judgeSteps are YOUR reasoning. Only an INCOMPLETE (node-1) trajectory is exempt (it emits verdicts:[] and no walk)."
+        - "v3 EMIT-CONTRACT ADDITIONS (E-NEW): alongside the §9.4 walk, persist layerVerdicts[] (one per engaged layer, skipped ones marked), earlyExit?, layersEngaged, codeEvalHits[] (when the library ran), and the emissions SELF-MANIFEST {emitted[], missing[{key,reason}]} — list honestly what you gathered vs what this contract expects; a missing emission with a stated reason is a NAMED degrade, a silent drop is a defect (MR-5 catches it same-run)."
         - "Verdicts are independently VERIFIED (#mode-verify · ≠ judge · downgrade-only); a CRIT/HIGH uncertain rolls the run up to INCOMPLETE at the gate (no false-green)"
       compresses:
         - "read MatrixPacket DATA; frame in ROUTE (CONTEXT)"
@@ -615,8 +643,13 @@ a new eval mid-judging. Your focus is judging what IS, not pondering what could
    - Compare the trajectory/transcript against the row's `statement` + `passCondition`.
    - **GROUND (node 4 — ABSENCE-SPLIT) — REFS ARE MANDATORY, NOT DECORATIVE:** every **decided**
      verdict (`result ∈ {pass, fail}`) MUST carry **≥1 structured `ref {obs, path, value}`** that
-     grounds it — `obs` = the trace/observation id you read, `path` = the field path within it
-     (e.g. `output.response`, `observations[3].output.success`), `value` = the EXACT cited string
+     grounds it — **`obs` = the packet's `trajectoryId` (the trace id), NEVER an observation
+     name** (in a multi-trajectory batch an observation-name `obs` can locate the WRONG
+     trajectory — the resolver takes the FIRST match in batch order), `path` = the field path
+     within the situation you were handed — packet-shaped (`trajectory.0.name`,
+     `transcript.0.content`) and trace-shaped (`observations.3.output.success`) BOTH re-resolve
+     (the verifier reconstructs the packet views deterministically); dot form is canonical,
+     bracket indices (`observations[3]`) are tolerated — `value` = the EXACT cited string
      (re-resolved by whitespace-normalized exact match by `#mode-verify`). **The string your prose
      critique already quotes IS the ref's `value`** — you are not inventing evidence, you are making
      the evidence the critique cites MACHINE-CHECKABLE. A decided verdict with a critique that cites
@@ -690,10 +723,12 @@ a new eval mid-judging. Your focus is judging what IS, not pondering what could
      "refs": [
        { "obs": "ef30a271", "path": "input.prompt", "value": "<outbound_guard consecutive_outbound=\"7\">" },
        { "obs": "ef30a271", "path": "output.response", "value": "held" },
-       { "obs": "ef30a271", "path": "observations[0].name", "value": "ai.generateText" }
+       { "obs": "ef30a271", "path": "observations.0.name", "value": "ai.generateText" }
      ]
    }
    ```
+   (`"ef30a271"` is the packet's `trajectoryId`. Note the dot-form index `observations.0.name` —
+   canonical; `observations[0].name` also resolves.)
    Each `ref.value` is a VERBATIM string from the trace the critique quotes — `#mode-verify`
    re-resolves it by exact match; a value the trace doesn't contain is a dead ref → downgrade.
    **Counter-example (DEFECT — do NOT emit):** the same verdict with `"refs": []` (or `refs`
@@ -860,6 +895,29 @@ this inline section is now its canonical home.
    pick a primitive + field/params); else keep `class: judge` and emit NO codeEval. A code/hybrid
    category WITHOUT a `codeEval` is a HARD error (tier-0 inert). The `statement` stays the human
    "Pass = …"; the `codeEval` is its runnable twin (run by the tier-0 pre-pass, zero judge tokens).
+   **§5b PROPOSALS ARE NOW HONORED:** the `level` / `severity` / `dimension` / `generality` you emit
+   are carried onto the mined criterion and the generic heuristics are the FALLBACK, not the
+   override (they used to be dropped in the JOIN, which pinned every criterion to `level: output`).
+   Emit what you actually judged; omit a field only when you want it derived.
+4b. **NAME the cluster suite-aware — `reinforce_not_restate`.** Only now (AFTER clustering, never
+   before) open `existing-criteria.json`: the criteria the suite ALREADY holds, as
+   `{id, statement, severity, level, checkedBy}`. If one of them already states the behaviour your
+   cluster found, emit your cluster under **that criterion's exact id** — your traces then attach to
+   it as REINFORCING evidence rather than standing up a second name for one check. The ORDER is
+   load-bearing: consulting the suite before reading would hand you a pre-defined failure list and
+   re-create the confirmation bias `emergent_categories_only` exists to prevent. Clustering first
+   keeps what you FIND emergent; the brief only informs what you CALL it. It ships no evidence, so
+   it cannot tempt you into re-citing a failure you did not read. It **informs, it never blocks** —
+   if the traces show something no existing criterion states, MINT IT; and never emit an existing id
+   with no trace of your own behind it (that is an invented finding). An absent or empty brief is
+   normal on a first run — proceed unchanged.
+   The brief also lists **`openQuestions[]`** — pairs of EXISTING criteria already awaiting a human
+   ruling on whether they are really one check, each with the reason they were kept apart (e.g.
+   `marker parity: [BEFORE] vs [AFTER]`). Both sides are live and both statements appear in
+   `criteria[]`. If your cluster matches one side, emit it under **that side's id**: minting a third
+   name for a behaviour that is already an open question makes the question harder to answer, not
+   easier. Do **not** attempt to settle the question yourself — you are not the ruling authority.
+   If your cluster genuinely matches neither side, it is new; mint it.
 5. **Flag fixable-vs-eval-worthy (S4 / EV-051).** Route the fixable + infra-class (e.g. sample C4
    dead-channel `account_number_unavailable`) to diagnostics; keep the genuinely behavioral criteria
    for `*build-evals`. NEVER fix.
@@ -871,14 +929,18 @@ this inline section is now its canonical home.
    the FAILURE + UNCERTAIN traces into DATASET CANDIDATES (`collectDatasetCandidates`, reusing the
    derive-dataset selectors) and folds in the `*evaluate` judge's node-2.5 unmatched-detection
    handoff — both consumable by `*build-dataset`. You mine + flag; the parent emits the candidates.
-7. **Render the discover `discovery-report.html` via the SHIPPED `writeDiscoverRunReport` composer**
-   (`render-discover-report.ts`) — pass the run dir; it builds `triage-summary.json` from
-   `triage.json` + wires every companion (verdicts → Proof-of-work, dataset-candidates → Dataset,
-   triage census → coverage funnel, profile → entity hero) so the report is COMPLETE BY DEFAULT.
+7. **Render the discover `discovery-report.html` via the SHIPPED `writeDiscoverRunReportV3` composer**
+   (`render-discover-report-v3.ts`) — pass the run dir PLUS the AGGREGATE result you already hold in
+   memory (`criteria` · `annotations` · the leaf `miningReport` · the ingested `traces` ·
+   `datasetCandidates` · the grown `suite` provenance). It writes BOTH surfaces: the PRODUCTION
+   W3 frozen-contract report at `discovery-report.html` and the previous renderer's output at
+   `discovery-report.v2.html` (transition fallback) — mirroring eval's `writeRunReport`. Anything you
+   do not pass is read from the run dir; a genuinely absent companion renders as a **NAMED absence**
+   (e.g. "sampling strategy census: not recorded by this run"), NEVER a fabricated number.
    **Do NOT render via bare `writeDiscoverReport(criteria, grounding)`** — without companions it
    degrades the funnel to em-dash and leaves Proof-of-work + Dataset empty (the A4 thin-report gap;
-   the data exists, it just isn't forwarded). Eval's `writeRunReport` is already complete-by-default;
-   discover MUST use its composer to match.
+   the data exists, it just isn't forwarded). The v2 composer `writeDiscoverRunReport`
+   (`render-discover-report.ts`) stays available unchanged for the legacy surface.
 
 ### Governing invariants (verbatim — must survive the merge)
 
@@ -890,6 +952,7 @@ this inline section is now its canonical home.
 - `gather_structured_refs`: "Every observed claim AND every absence claim cites a structured ref {obs,path,value} (ground-absence = a positive field check, never inferred from silence)."
 - `typed_assumptions`: "Every surfaced assumption is TYPED — factual-intent · normative · scope — so its lifecycle/blocking routes correctly (GA-3)."
 - `emergent_categories_only`: "Categories EMERGE from what the traces show. Never start from a pre-defined failure list (confirmation bias). No generic scores as categories."
+- `reinforce_not_restate`: "The existing suite (existing-criteria.json) is consulted AFTER clustering, to NAME an already-emerged cluster — never before, which would seed a pre-defined list and violate emergent_categories_only. A cluster whose behaviour an existing criterion already states is emitted under THAT id (reinforcing evidence), not as a second name for one check. INFORMS, never blocks: a genuinely new behaviour is always minted. An existing id emitted with no trace of your own behind it is a fabricated finding."
 - `judge_only_never_fix`: "EV-051 — FLAGS fixable-vs-eval-worthy and routes the fixable + infra-class to diagnostics. NEVER fixes the subject."
 - `binary_actionable`: "Each emergent category is one binary criterion whose verdict points at a concrete fix locus."
 - `full_mined_criterion_emit`: "Emit the FULL MinedCriterion (base + §5b metadata + §5c discovery incl. structured refs + typed assumptions). A lightweight/flattened emit that drops refs/assumptions is a DEFECT."