@mutagent/evaluator 0.1.0-alpha.5 → 0.2.0-alpha.1

package/.claude/skills/mutagent-evaluator/SKILL.md

@@ -1,24 +1,19 @@
 ---
 name: mutagent-evaluator
 description: |
-  Evaluation-development engine for AI agents and skills. Turns a subject (a skill/agent +
-  its traces) into a TRUSTWORTHY eval suite: deep-reads traces to determine success/failure
-  (even when nothing is user-marked), mines emergent BINARY ACTIONABLE criteria from the
-  ✓/✗ split, builds one critique-before-verdict LLM-judge (or code-check) per criterion,
-  validates each judge against human labels (TPR/TNR · Rogan-Gladen · bootstrap CI), then
-  runs the suite against a target → per-criterion binary+confidence → severity-gated GATE
-  verdict + agent-variance view. A judge is ONLY a judge: failures are flagged and ROUTED to
-  diagnostics, never fixed here (EV-051). First invocation auto-detects the subject + the
-  framework-substrate choice (DEFAULT agent-dispatch — host-runtime subagents, no provider ·
-  in-house AI-SDK judge optional · code-based · user's framework export).
-  Subsequent invocations dispatch the parent session + MASS-PARALLEL evaluator
-  sub-agents (one cell, discover / judge modes; host runtime, harness-capped) against the subject's traces.
-  DEFAULT INTENT: any "evaluate this skill/agent" request runs *evaluate — the JUDGE — which is what an
-  end user wants. *audit is a SEPARATE, explicit-only 4-tab static-auditor (eval-matrix · data-leak ·
-  variance · methodology) used mainly for complex agents/skills that emit HTML artifacts; it is NEVER the
-  fallback for a plain evaluate. Pinned judge model + temp=0
-  for byte-identical reruns (C-PIN). Usable standalone OR borrowed by mutagent-skill-builder's
-  SIMULATE phase as the SimVerifier.
+  The EVALUATE stage of the agentic development lifecycle (ADL ③). Builds and runs
+  trustworthy evaluations for AI agents and skills: from a subject's real traces it
+  learns what success and failure look like, turns that into binary eval criteria,
+  and scores the subject to a clear gate verdict. It only judges — failures are
+  flagged and routed to mutagent-diagnostics, never fixed here. Pick it whenever the
+  ask is to evaluate, judge, score, or build and validate evals for an agent or
+  skill: any plain "evaluate this" runs *evaluate — one judge per trajectory walks
+  the evidence layers (code checks → outcome → trajectory → tool outputs → context),
+  criteria binding evidence across all of them; *evaluate-<layer> scopes the same
+  pipeline to one layer; *discover mines new criteria, never re-recommending one it
+  already has; *audit is a separate, explicit-only static auditor and never the
+  fallback for a plain evaluate. Also serves as the verifier inside
+  mutagent-skill-builder's SIMULATE.
 license: Proprietary. LICENSE.txt has complete terms.
 compatibility: Designed for Claude Code, Codex, Cursor, OpenCode and similar coding-agent runtimes; works with git, gh CLI, jq, curl, and Bun/pnpm/npm runtimes.
 metadata:
@@ -102,11 +97,12 @@ reason for the rest. NEVER improvise.
 |---------|------|-------------------|---------|
 | `*discover-evals` | hybrid | `references/workflows/orchestrator-protocol.md` + `scripts/prep-tasks.ts` + `scripts/determine-outcome.ts` + `scripts/discover-criteria.ts` + `assets/agents/evaluator.md` (`#mode-discover`) | PREP determiner tasks → fan out `evaluator` (`#mode-discover`, mass-parallel) → AGGREGATE ✓/✗ → mine emergent BINARY ACTIONABLE criteria (EV-041/042/052) **+ T6 failure/uncertain DATASET CANDIDATES** (`collectDatasetCandidates`, reuses the derive-dataset selectors → `*build-dataset`). **GA split** — **code:** sample (broken+healthy) · aggregate · `diff-discriminate` · ground-gate · dataset-candidates; **LLM leaf:** determine + 3 detectors + cite refs + typed assumptions (root-not-symptom). |
 | `*build-evals` | hybrid | `references/workflows/orchestrator-protocol.md` + `scripts/eval-engine.ts` + `scripts/codegen-evals.ts` + `scripts/prep-tasks.ts` + `scripts/build-evals.ts` + `assets/agents/evaluator.md` (`#mode-judge-criterion`) + `scripts/render-build-cards.ts` | **ENGINE-FORK FIRST (ADL F7/F9/F14).** ASKS the eval implementation mode via `chooseEvalEngineOptions(target)` — Path A `native-matrix` (eval-matrix + LLM-judge SUB-AGENTS; SURFACES the Claude-Code + log-sink dependency up front) vs Path B `code-written` (`codegen-evals.ts` emits a portable bun/TS suite — runs WITHOUT Claude Code, F14). Target-conditional: a code framework offers BOTH, a `harness:*` target is native-only. **Path A** then PREPs judge tasks (criterion × trace-slice) → fan out `evaluator` (`#mode-judge-criterion`, mass-parallel) → one binary+confidence judge per criterion (EV-043). **GA split** — **code:** engine-resolve · spec render · `lint-grounding` · `resolve-ref` (BIND) · codegen; **LLM leaf:** judge + VERIFY (cite refs · note assumptions · abstain). Streams progress + a verbose evals entity card (F13/F16/F22). |
-| `*evaluate` | hybrid | `references/workflows/orchestrator-protocol.md` + `scripts/matrix-judge.ts` + `scripts/contracts/eval-matrix.ts` + `assets/agents/evaluator.md` (`#mode-judge-trajectory`) + `scripts/evaluate.ts` | **DEFAULT (headline):** **T1 TIER-0 deterministic pre-pass** (code-method rows run first, zero judge tokens) → PREP one matrix packet per RESIDUAL judge trajectory (**T5 adaptive-K guard, default 1:1**) → fan out `evaluator` (`#mode-judge-trajectory` — one judge/trajectory scoring the WHOLE matrix, emitting the **T2 Judge DAG v2 walk** = `judge_steps[]` + dense na-explicit map + confidence band + early-INCOMPLETE + node-2.5 candidates) → AGGREGATE: fold code+judge verdicts → **T3 independent verifier** refutes GATING fails (downgrade-only) → **T4 consolidate-by-locus + walk-derived health** → GATE verdict + variance view (EV-048). **GA split** — **code:** tier-0 · prep · `lint-grounding` · `resolve-ref` · `assemble-scorecard` (gate `fail ▸ incomplete ▸ pass`) · consolidate-by-locus · route-failures; **LLM leaf:** trajectory judge (DAG v2 walk: BIND · GROUND[absence-split] · cite · abstain) + independent VERIFY. **MAY now return `incomplete`** (indeterminate → calibrate). **ADL F20:** the rollup ALSO renders a SCORECARD DASHBOARD wireframe (`renderScorecardDashboard` — per-criterion pass/fail bar + variance + samples), not a flat terminal dump. Under a **Path B** engine, `*evaluate` instead runs the portable `codegen-evals.ts` suite (no Claude Code) and reads back its scorecard JSON from the discoverable sink. |
-| `*improve` | hybrid | `references/edd-loop.md` + `scripts/edd/variance-gate.ts` + `scripts/edd/change-request.ts` + `scripts/edd/edd-types.ts` + `schemas/edd-change-request.schema.yaml` + `assets/agents/evaluator.md` (`#mode-improve`) | **ADL ③ IMPROVE / EDD loop (F18+F19).** **F19 VARIANCE-FIRST:** repeat-N (default 5) the SAME cases → `evaluateVarianceGate` → accuracy is entered ONLY when the variance gate passes (`assertVarianceStableBeforeAccuracy` THROWS otherwise — "accuracy over big samples is wasted on a flapping verdict"). **F18 CLOSURE:** still JUDGE-ONLY (EV-051), the evaluator emits a grounded `EddChangeRequest` (failing cases + `ref{obs,path,value}` + remedy target `agentspec`\|`impl`) to the `mutagent-builder ai-engineer` over **SendMessage**, consumes the `ChangeRequestResponse`, and re-evals on `amended`. **Bounded terminator** (`decideEddLoop`, afkloop-legal): `full-green` ▸ DONE \| `max-swings`\|`max-wallclock`\|`no-improvement-streak` ▸ STOPPED + convergence delta. **GA split** — **code:** variance gate · request/response validate · loop terminator (all PURE, injected wall-clock); **LLM leaf:** localize the flap/fail to its root + author the grounded request + decide remedy target. |
+| `*evaluate` | hybrid | `references/workflows/orchestrator-protocol.md` + `scripts/matrix-judge.ts` + `scripts/contracts/eval-matrix.ts` + `assets/agents/evaluator.md` (`#mode-judge-trajectory`) + `scripts/evaluate.ts` | **DEFAULT (headline):** **T1 TIER-0 deterministic pre-pass** (code-method rows run first, zero judge tokens) → PREP one matrix packet per RESIDUAL judge trajectory (**T5 adaptive-K guard, default 1:1**) → fan out `evaluator` (`#mode-judge-trajectory` — one judge/trajectory scoring the WHOLE matrix, emitting the **T2 Judge DAG v2 walk** = `judge_steps[]` + dense na-explicit map + confidence band + early-INCOMPLETE + node-2.5 candidates) → AGGREGATE: fold code+judge verdicts → **T3 independent verifier** refutes GATING fails (downgrade-only) → **T4 consolidate-by-locus + walk-derived health** → GATE verdict + variance view (EV-048). **GA split** — **code:** tier-0 · prep · `lint-grounding` · `resolve-ref` · `assemble-scorecard` (gate `fail ▸ incomplete ▸ pass`) · consolidate-by-locus · route-failures; **LLM leaf:** trajectory judge (DAG v2 walk: BIND · GROUND[absence-split] · cite · abstain) + independent VERIFY. **MAY now return `incomplete`** (indeterminate → calibrate). **ADL F20:** the rollup ALSO renders a SCORECARD DASHBOARD wireframe (`renderScorecardDashboard` — per-criterion pass/fail bar + variance + samples), not a flat terminal dump. Under a **Path B** engine, `*evaluate` instead runs the portable `codegen-evals.ts` suite (no Claude Code) and reads back its scorecard JSON from the discoverable sink. **v3 LAYERED WALK (D1=B+):** the trajectory judge performs the whole evidence-layer sequence INSIDE its one dispatch — L0 code checks → L1 outcome (scenario + expectedExit, cheap, first) → per stopping rules L2 trajectory → L3 tool outputs → L4 context — emitting `layerVerdicts[]` + `earlyExit` + `codeEvalHits[]` + an emissions self-manifest (E-NEW contract, `contracts/eval-matrix.ts`); AGGREGATE folds layers (`foldLayerVerdicts`), DETECTS cross-layer conflicts (`detectLayerConflicts` — surfaced for the human calibration queue, NEVER auto-resolved, D2-A/OT-1), enforces the COMPLETENESS LAW (`assertNoUnverdictedCriterion` THROWS on a silent skip), and folds MR-5 data-completeness (`foldEmissionsScorecard`). |
+| `*evaluate-outcome` · `*evaluate-trajectory` · `*evaluate-outputs` · `*evaluate-context` · `*evaluate-code` · `*evaluate-criteria` | hybrid | the SAME binds as `*evaluate` | **v3 PIPEABLE LAYER FILTERS (D3=A):** each runs the identical `*evaluate` pipeline with `layerScope` set in every packet (`outcome`→`[L1]` · `trajectory`→`[L2]` · `outputs`→`[L3]` · `context`→`[L4]` · `code`→`[L0]`); the judge runs ONLY those walk phases — **criteria are the SUPER-LAYER and are still verdicted over whatever those phases gather (never left unverdicted — the completeness law holds per scope)**. `*evaluate-criteria` = the criterion-axis pass (`#mode-judge-criterion`, single criterion × slice — the suite-build/validation path). Standalone-testable AND composable: a standalone layer verdict must equal the same layer's verdict inside the full walk (V9). |
+| `*optimize` | hybrid | `references/edd-loop.md` + `scripts/edd/variance-gate.ts` + `scripts/edd/change-request.ts` + `scripts/edd/edd-types.ts` + `schemas/edd-change-request.schema.yaml` + `assets/agents/evaluator.md` (`#mode-optimize`) | **ADL ③ OPTIMIZE / EDD loop (F18+F19).** **F19 VARIANCE-FIRST:** repeat-N (default 5) the SAME cases → `evaluateVarianceGate` → accuracy is entered ONLY when the variance gate passes (`assertVarianceStableBeforeAccuracy` THROWS otherwise — "accuracy over big samples is wasted on a flapping verdict"). **F18 CLOSURE:** still JUDGE-ONLY (EV-051), the evaluator emits a grounded `EddChangeRequest` (failing cases + `ref{obs,path,value}` + remedy target `agentspec`\|`impl`) to the `mutagent-builder ai-engineer` over **SendMessage**, consumes the `ChangeRequestResponse`, and re-evals on `amended`. **Bounded terminator** (`decideEddLoop`, afkloop-legal): `full-green` ▸ DONE \| `max-swings`\|`max-wallclock`\|`no-improvement-streak` ▸ STOPPED + convergence delta. **GA split** — **code:** variance gate · request/response validate · loop terminator (all PURE, injected wall-clock); **LLM leaf:** localize the flap/fail to its root + author the grounded request + decide remedy target. |
 | `*validate` | hybrid | `scripts/validate-judge.ts` + `references/validate-evaluator.md` + `assets/agents/evaluator.md` (`#mode-judge-criterion`) | **ENGINED (EV-044)** — calibrate a judge vs `*review` labels: confusion matrix → TPR/TNR · split-disjointness + test-once · Rogan-Gladen θ · deterministic bootstrap CI; `<MIN_LABELS` stays `unvalidated`+bias-corrected. **GA split** — **code:** entire path is deterministic (filter EXCLUDES indeterminate · confusion · RG · assumption-agreement). |
-| `*review` | hybrid | `scripts/build-review-ui.ts` + `references/build-review-interface.md` | **ENGINED (EV-045)** — CODE renders a browser annotation UI (one trace/screen · Pass/Fail/Defer · keyboard · auto-save · labels export); **HITL**: a human labels → `mergeLabels` persists → feeds `*validate`. |
-| `*eval` | agent-chain | `references/eval-stage.md` + `scripts/eval-engine.ts` + `scripts/materialize-dataset.ts` + `scripts/render-build-cards.ts` | **THE ADL EVAL-STAGE ENTRY (F15).** Called after `*build` hands a freshly-built agent + its agentspec. INTERACTIVELY offers `*build-dataset` / `*build-evals` derived from `agentspec.definition.evals`, and lets the user PICK THE EVAL ENGINE (mutagent native eval-matrix [Path A] vs code-written-in-target-lang [Path B]) via the target-conditional menu (`chooseEvalEngineOptions`). Surfaces Path A's Claude-Code + log-sink dependency UP FRONT (F7/F9). Streams wireframe progress + entity cards (F13/F16/F22). Parent-session only (AskUserQuestion). |
+| `*review` | hybrid | `scripts/run-review.ts` (run composer) + `scripts/render-review-report-v3.ts` (surface) + `scripts/build-review-ui.ts` (v2 fallback) + `references/build-review-interface.md` | **ENGINED (EV-045)** — CODE renders a browser review surface; **HITL**: a human labels → `mergeLabels` persists → feeds `*validate`. **Entry point:** `bun scripts/run-review.ts <runId> [--cwd <repoRoot>]` — reads an EXISTING `*evaluate` run dir (`run-input.full.json` + `verdicts/*.verdict.json`; joins on the parsed `trajectoryId`, never the packet filename) and writes `review-report.html` beside `evaluation-report.html`, plus `review-report.v2.html` from the previous annotation UI (module untouched). Dispatches NO judge — it costs nothing and invents no artifacts. **Surface:** navigator matrix (criteria rows × trace columns) → single-trace drill, right-rail verdict cards with Agree/Revise/Refute + why-note, keys `1`/`2`/`D`, >48 traces → window-cluster, submit → **calibration handoff markdown** (diagnostics remedy-bundle shape) server-generated from the run's own artifacts. Ingested-but-unjudged trajectories are NAMED in a scope note, never dropped. **Does NOT do W4:** no feedback store, no ruling persistence, no propagation into `*validate` — producing the artifact is the scope; consuming a reviewer's answer is task #9. |
+| `*eval` | agent-chain | `references/eval-stage.md` + `scripts/eval-engine.ts` + `scripts/materialize-dataset.ts` + `scripts/render-build-cards.ts` | **THE ADL EVAL-STAGE ENTRY (F15).** Called after `*build` hands a freshly-built agent + its agentspec. INTERACTIVELY offers `*build-dataset` / `*build-evals` derived from `agentspec.spec.evaluation`, and lets the user PICK THE EVAL ENGINE (mutagent native eval-matrix [Path A] vs code-written-in-target-lang [Path B]) via the target-conditional menu (`chooseEvalEngineOptions`). Surfaces Path A's Claude-Code + log-sink dependency UP FRONT (F7/F9). Streams wireframe progress + entity cards (F13/F16/F22). Parent-session only (AskUserQuestion). |
 | `*build-dataset` | hybrid | `scripts/materialize-dataset.ts` + `scripts/build-dataset.ts` + `assets/agents/dataset-builder.md` + `schemas/dataset.schema.yaml` + `references/generate-synthetic-data.md` + `scripts/render-build-cards.ts` | **ENGINED (EV-046) + ADL F8.** Now MATERIALIZES real items first: `materializeFromAgentspec` seeds ≥1 REAL DatasetCase per `dataset_category` + one per `edge_case` (seed → actual items, not definitions) → then the 3-way: HITL seed interview (~10) → `dataset-builder` agent (tuples → NL queries → realism filter) → CODE cartesian-expand + near-dup drop + monotonic merge. Streams wireframe progress cards (F13/F16) + a verbose dataset entity card (F22). |
 | `*discover-dataset` | script | `scripts/derive-dataset.ts` | **ENGINED (EV-047) · re-fronts the old `*derive-dataset`** — distill a living regression set from labeled ✓/✗ traces (reuses `sample-traces.ts` EV-052 selectors + `build-dataset.ts` merge). Code-only. **P-B:** consumes the `discovery` agent's curated `SelectionManifest` handoff — **scenario-balanced** (`ext.classification.scenario`), **edge-cases first** (`ext.classification.edgeCase.is`), worthiness-prioritized (`ext.signals.worthinessScore`). |
 | `*audit` | agent-chain | `assets/agents/audit-executor.md` + `workflows/{audit,data-leak,variance}.workflow.js` + `scripts/{flow-graph,ui-slots}.ts` + `lenses/context-flow-lens.md` | SURFACES the v1 4-tab static-auditor (EV-001..027, KEEP) **+ now the agent context-flow audit**: tool-result threading + sub-agent handoff completeness (EV-028/029, over the `flow-graph` EV-032 + expected-flow EV-037) and the first-class **HTML-artifact missing-data** dimension (computed-but-not-rendered / orphan / faithfulness, EV-039/040, subject-agnostic). |
@@ -138,7 +134,7 @@ When you encounter a *<name> token:
 Invoke me with:
 - `evaluate skill <name>` / `evaluate agent <name>` / `/mutagent-evaluator`  → **`*evaluate`** (default)
 - `*discover-evals` (mine criteria) · `build evals for <x>` · `validate judge <x>` · `*audit <subject>` (explicit only)
-- `*improve` (the ADL ③ EDD loop — variance-first then request-amend-reeval to full green, bounded)
+- `*optimize` (the ADL ③ EDD loop — variance-first then request-amend-reeval to full green, bounded)
 - `*self-audit` (the eval-of-the-eval — audit my OWN eval-dev; on-demand only)
 - borrowed by `mutagent-skill-builder` SIMULATE (SimVerifier role)
 - `--reconfigure` to re-enter onboarding (subject + substrate)
@@ -151,7 +147,7 @@ Minimal eval-dev spine (build order — do NOT reorder):
 *discover-evals     → deep-read traces, determine ✓/✗, mine 5-10 binary actionable criteria
 *build-evals  → one critique-before-verdict judge (or code-check) per criterion
 *evaluate     → run vs target → per-criterion binary+confidence → GATE verdict + variance
-*improve      → ADL ③ EDD loop: F19 variance-first (stabilize per-case spread, repeat-N)
+*optimize      → ADL ③ EDD loop: F19 variance-first (stabilize per-case spread, repeat-N)
                 BEFORE accuracy, then F18 closure (judge-only REQUESTS the ai-engineer to
                 amend agentspec|impl over SendMessage → re-eval) → full green OR bounded STOP
 ```
@@ -159,13 +155,13 @@ Minimal eval-dev spine (build order — do NOT reorder):
 `*validate` (calibrate the judge) layers on top. `*audit` (the v1 4-tab master-audit) is an
 **explicit-only** specialized surface (complex / HTML-artifact subjects) — NOT part of the default
 evaluate spine and never an automatic fallback.
-`*improve` is the IMPROVE stage — it runs AFTER an initial `*build` + `*evaluate` and is **bounded**
+`*optimize` is the OPTIMIZE stage — it runs AFTER an initial `*build` + `*evaluate` and is **bounded**
 (never infinite): full-green ⇒ DONE; else max-swings / max-wallclock / no-improvement ⇒ STOP + delta.
 
 **ADL EVAL-stage entry (`*eval`).** When `*build` hands over a freshly-built agent +
 its agentspec, `*eval` is the entry: it interactively offers `*build-dataset` /
-`*build-evals` derived from `agentspec.definition.evals` (F15), MATERIALIZES real
-dataset items per category + edge-case (F8), lets the user PICK THE EVAL ENGINE
+`*build-evals` derived from `agentspec.spec.evaluation` (F15), MATERIALIZES real
+dataset cases by projecting the 0.3.0 `datasets[].items[]` (F8), lets the user PICK THE EVAL ENGINE
 (Path A native eval-matrix vs Path B code-written — surfacing Path A's Claude-Code +
 log-sink dependency up front, F7/F9/F14), streams wireframe progress + entity cards
 (F13/F16/F22), and renders the scorecard as a dashboard (F20). See
@@ -276,7 +272,7 @@ applies, EV-053/EQ5):
 | `scripts/aggregate-discover.ts` | P2 (§5a) + §9.4 (T6) | code | The real `*discover-evals` AGGREGATE half: parse the leaf's per-trace verdict files + per-batch mining report → `TraceAnnotation[]` (FAIL-LOUD via `missingVerdictKeys`; THROW on an undispatched-trace category ref) → `deriveMinedCriteria` (§5b metadata + §5c DR-2) → `growLivingSuite` (append-only, monotonic, saturation-stop). **T6:** `collectDatasetCandidates` (failure/uncertain → `DatasetCase[]` via the derive-dataset selectors) + `unmatchedDetectionCandidates` (the `*evaluate` judge's node-2.5 handoff) → consumable by `*build-dataset`. `DiscoverMiningReport` contract. No hand-rolled annotations. |
 | `scripts/code-eval.ts` | P3 (EX-2) | code | The v2 CODE-track primitive library — generic PURE `(trace) → {result, detail}` checks: presence · string-equality · format-validity · schema-conformance · ref-integrity (cross-stage). SV-1 dotted-path read incl. `obs:<name>.<path>`. Exhaustive `runCodeEval` (unknown primitive → THROW). NO LLM, byte-identical. Disjoint from the v1 `run-deterministic.ts` audit world. |
 | `scripts/check-method-router.ts` | P3 (EX-2) | hybrid | THE load-bearing §5b `check_method` router: `deterministic`→CODE-EXEC (judge NEVER called) · `llm-judge`→JUDGE seam · `hybrid`→code pre-filter that GATES the judge. Unknown → THROW (exhaustive). `RoutedCriterionVerdict` = `CriterionVerdict` + provenance `{producedBy, codeResult, judgeReached}` (structural superset → `evaluate.ts` rollup unchanged). `extractCodeEval(registry)` fail-loud (a code-class criterion with no spec THROWS — never silently judge). "Code before judge" lives INSIDE the agent. |
-| `scripts/code-quality-verdict.ts` | W2I1 (⑤ IMPROVE) | hybrid | The **code-quality JUDGE mode** (`#mode-judge-code-quality`) — the (b) half of the ⑤ IMPROVE code-target BOTH-gate. Scores a `code` subject's QUALITY (spec-faithfulness · addresses-diagnosis · no-regression · error-handling · maintainability) BEYOND tests-green → severity-gated binary PASS/FAIL. `buildCodeQualityPrompt` (pure renderer, critique-before-verdict) + `codeQualityGatePass` (pure aggregate) + `runCodeQualityJudge` (C-PIN run wrapper on the injected JudgeInvoke seam). Judge-only (EV-051), host-runtime, pinned model + temp 0. **DISTINCT from `code-eval.ts`** (deterministic primitives — no LLM). `DEFAULT_CODE_QUALITY_CRITERIA` + its aggregation rule are FLAGGED-for-operator-sign-off. |
+| `scripts/code-quality-verdict.ts` | W2I1 (⑤ OPTIMIZE) | hybrid | The **code-quality JUDGE mode** (`#mode-judge-code-quality`) — the (b) half of the ⑤ OPTIMIZE code-target BOTH-gate. Scores a `code` subject's QUALITY (spec-faithfulness · addresses-diagnosis · no-regression · error-handling · maintainability) BEYOND tests-green → severity-gated binary PASS/FAIL. `buildCodeQualityPrompt` (pure renderer, critique-before-verdict) + `codeQualityGatePass` (pure aggregate) + `runCodeQualityJudge` (C-PIN run wrapper on the injected JudgeInvoke seam). Judge-only (EV-051), host-runtime, pinned model + temp 0. **DISTINCT from `code-eval.ts`** (deterministic primitives — no LLM). `DEFAULT_CODE_QUALITY_CRITERIA` + its aggregation rule are FLAGGED-for-operator-sign-off. |
 | `scripts/run-evaluate.ts` | P5 + §9.4 (T1·T3·T4·T5) | hybrid | The real-engine `*evaluate` end-to-end COMPOSER: `tier0Plan` (T1 — code rows decided in the pre-pass, residual judge rows only in packets) → `prepMatrixPackets` (residual-only) → `adaptivePacketPlan` (T5 overload guard, default 1:1) → [dispatch #mode-judge-trajectory] → `aggregateEvaluate` (FAIL-LOUD readiness gate over DISPATCHED ids → `aggregateMatrixScorecard` folding code+judge + T3 `independentVerify` downgrade-only on gating fails → GATE + variance → T4 `consolidateByLocus` lociClusters → EV-051 `routeFailures`→validated HandoverBundle). `maskedScorecard` = the C-PIN byte-identity artifact. |
 | `scripts/render-eval-report.ts` | P4 (D-1/2/3) | code | The v2 reporting (v1 `render-report.ts` untouched): `renderEvalCards` (D-1 terminal per-criterion cards) + `buildEvalReportInput` (derives the rich tabs from real run data — ledger · coverage · cohorts · gating rolls · top-findings · judge-health) + `renderEvalReport` (the operator-APPROVED 5 tabs: ① Overview KPIs+coverage+gating-table+top-findings · ② Trajectory‖Judge ledger + click-row Target-Agent‖Judge side-by-side · ③ Eval Scorecard cohort heatmap + nested subcards + inline calibration · ④ Findings verbatim-evidence + judge chain + agree/revise/refute · ⑤ Self-Eval [INTERNAL, strip-marked]). Consumes the §9.4 judge-walk (`judge_steps[]` + dense map) when present, degrades to the per-trajectory scorecard otherwise. Tabs are `<button>` (no `<s>` strikethrough). Unified design-system tokens (`assets/brand/theme.css`). byte-identical after the `generatedAt` mask. `autoOpenCommand` cross-platform helper (orchestrator fires it). |
 | `scripts/contracts/eval-types.ts` *(extended)* | P2/P2b | code | + `MinedCriterion` (additive subtype of `DiscoveredCriterion`) carrying the §5b `MetricMetadata` (generality·dimension·level·check_method [3-value SF-2 router]·substrate·applies_to·severity·judge_inputs·flag) + the §5c `DiscoveryRationale` (DR-2: targets·why·evidence{grounding observed\|inferred·prevalence k/n·refs}·assumptions·expected_impact). `parseMinedCriterion` enforces the evidence-first gate (OBSERVED ⇒ refs + k>0). |
@@ -287,7 +283,7 @@ applies, EV-053/EQ5):
 |--------|---------|------|---------|
 | `scripts/contracts/eval-engine.ts` | F7/F9/F14 | code | The ENGINE FORK contract — `EvalEngine` (`native-matrix` Path A · `code-written` Path B) + `EngineTargetInput` (the minimal agentspec.build slice) + `EvalEngineOption` (the menu, with `requiresClaudeCode`/`requiresLogSink`/`portable` SURFACED) + `EvalEnginePlan` (the resolved choice incl. discoverable `outputSink`). |
 | `scripts/eval-engine.ts` | F7/F9/F14 | code | The target-conditional resolver — `isFrameworkTarget` · `chooseEvalEngineOptions` (framework→BOTH · harness→native-only) · `resolveEvalEngine` (surfaces CC + log-sink deps) · `assertEngineMatchesTarget` (code-written⊥harness, fail-loud) · `defaultOutputSink`. PURE. |
-| `scripts/contracts/agentspec-evals.ts` | F8 | code | The MINIMAL `agentspec.definition.evals` slice (`success_criteria[]` · `scenarios[]` · `dataset_categories[]`) — local re-declaration (standalone: NEVER imports the agentspec skill). `parseAgentspecEvals` guarded. |
+| `scripts/contracts/agentspec-evals.ts` | F8 | code | The MINIMAL `agentspec.spec.evaluation` slice (0.3.0 D18/D19 — `criteria[]` · `scenarios[]` · `datasets[]` with dataset-local `categories[]` + `caseDimensions` + real `items[]`) — local re-declaration (standalone: NEVER imports the agentspec skill). `parseSpecEvaluation` guarded. |
 | `scripts/materialize-dataset.ts` | F8 | code | MATERIALIZE real dataset items from the agentspec — `materializeFromAgentspec` (≥1 real item/category + one/edge_case, seed→actual) · `dimensionsFromAgentspec` · `materializeToDataset` (monotonic). Reuses `build-dataset.ts` ids/dedup/merge. PURE. |
 | `scripts/codegen-evals.ts` | F14 (Path B) | code | Emit a PORTABLE code-written eval suite source (bun/TS) — `codegenEvalSuite` (code-checks + SDK judge + criteria → gate-bearing scorecard to a discoverable sink). NO CC dispatch. Fail-loud, deterministic. PURE. |
 | `scripts/render-build-cards.ts` | F13/F16/F20/F22 | code | The wireframe terminal surface — progress cards (`renderBuildDatasetProgressCard`/`renderBuildEvalsProgressCard`, F13/F16) · entity cards (`renderDatasetEntityCard`/`renderEvalsEntityCard`, F22) · the scorecard DASHBOARD (`renderScorecardDashboard` — pass/fail bar + variance + samples, F20, not a flat dump). PURE. |
@@ -296,7 +292,9 @@ applies, EV-053/EQ5):
 
 | Script | Req | Kind | Purpose |
 |--------|-----|------|---------|
-| `scripts/build-review-ui.ts` | EV-045 | code | The `*review` Code half — `renderReviewUi` emits a DETERMINISTIC browser annotation UI (one trace/screen, native render, Pass/Fail/Defer, notes, keyboard, localStorage auto-save, labels export) + `mergeLabels` (dedup by traceId). Holds no judge prompt; the HUMAN decides. |
+| `scripts/build-review-ui.ts` | EV-045 | code | The `*review` Code half — `renderReviewUi` emits a DETERMINISTIC browser annotation UI (one trace/screen, native render, Pass/Fail/Defer, notes, keyboard, localStorage auto-save, labels export) + `mergeLabels` (dedup by traceId). Holds no judge prompt; the HUMAN decides. Retained as the `review-report.v2.html` transition fallback under the W4 surface. |
+| `scripts/run-review.ts` | EV-045 | code | **The `*review` RUN COMPOSER** — the analogue of run-evaluate's `writeRunReport`. Reads an existing run dir (run-input + `verdicts/*.verdict.json`, excluding the `*.verify.json` ledger) and writes `review-report.html` + the v2 fallback. Joins verdicts to traces on the parsed `trajectoryId`, NOT the packet filename; treats the lean run-input's `"<N EvalTraces>"` placeholder as ABSENT rather than data; NAMES ingested-but-unjudged trajectories in a scope note. Fail-loud on a missing run dir / run-input / verdicts / criteria, and a malformed verdict names its FILE. Dispatches no judge. |
+| `scripts/render-review-report-v3.ts` | EV-045 | code | The W4 frozen-contract review surface (navigator + drill · rail verdict cards · rulings · calibration handoff MD) + `REVIEW_REPORT_SPEC`, the surface's verification spec, exported here so the unit test and the composer assert the same shape. |
 | `scripts/build-dataset.ts` | EV-046 | code | The `*build-dataset` Code half — cartesian dimension×value expand · deterministic token-Jaccard near-dup removal · content-derived id · `mergeCases` MONOTONIC merge. Agent proposes realism; script enforces non-redundancy. |
 | `scripts/derive-dataset.ts` | EV-047 | code | `*discover-dataset` — distill a living regression set from past ✓/✗ (reuses `sample-traces.ts` selectors + `build-dataset.ts` merge); trace→`DatasetCase`. Append-only, no new decision. |
 | `scripts/living-suite.ts` | EV-053 | code | Generic append-only writer (`appendOnly` + `assertMonotonicGrowth`) shared by datasets + criteria — a living artifact NEVER shrinks. Pure-counter provenance (no clock → C-PIN byte-identity). |

package/.claude/skills/mutagent-evaluator/assets/agents/dataset-builder.md

@@ -131,11 +131,12 @@ not run the subject and does not decide pass/fail.
 1. **Pre-read** `references/generate-synthetic-data.md` — the dimension → tuple → query → filter
    process is the lens. Read the passed `dimensions` + `seed_tuples`.
    - **ADL F8 — agentspec-materialized seeds.** In the ADL EVAL stage the parent ALSO PREPs real
-     items MATERIALIZED from `agentspec.definition.evals` (`scripts/materialize-dataset.ts`): ≥1 real
-     `DatasetCase` per `dataset_category` + one per `edge_case`. When present, treat these
-     materialized cases as ADDITIONAL realism anchors (alongside the HITL `seed_tuples`) — they are
-     real queries from the spec, NOT definitions. EXPAND around them; do not re-derive them. The
-     deterministic dedup/merge (`build-dataset.ts`) drops any overlap.
+     items PROJECTED from `agentspec.spec.evaluation` (0.3.0, `scripts/materialize-dataset.ts`): one
+     real `DatasetCase` per `datasets[].items[]` entry — tuple = the item's `category` + `case` axis
+     assignment, query = the item's opaque `input` payload. When present, treat these materialized
+     cases as ADDITIONAL realism anchors (alongside the HITL `seed_tuples`) — they are real queries
+     from the spec, NOT definitions. EXPAND around them; do not re-derive them. The deterministic
+     dedup/merge (`build-dataset.ts`) drops any overlap.
 2. **Generate more tuples (Step 3).** Produce `target_count` new `(dim1, dim2, dim3, …)` combinations
    for the subject. Avoid duplicates of the seed + each other; vary values across dimensions. This is
    ONE step — tuples only, no queries yet.

package/.claude/skills/mutagent-evaluator/assets/agents/evaluator.md

@@ -20,9 +20,9 @@ ACTIVATION-NOTICE: This file contains your full agent operating guidelines. Read
 class: pure_subagent_executor
 isolation: none
 
-modes: [discover, judge, verify, improve]  # discover · judge · verify (GA-5) · improve (EDD ③ — F18/F19)
+modes: [discover, judge, verify, optimize]  # discover · judge · verify (GA-5) · optimize (EDD ③ — F18/F19)
 judge_axes: [trajectory, criterion] # judge-mode fan-out axes
-# `improve` is the ADL ③ IMPROVE / Eval-Driven-Development (EDD) LOOP mode (F18+F19). It is a MODE of
+# `optimize` is the ADL ③ OPTIMIZE / Eval-Driven-Development (EDD) LOOP mode (F18+F19). It is a MODE of
 # this ONE agent (the roster stays 3) — it drives the variance-first gate (F19) then, while still
 # JUDGE-ONLY (EV-051), REQUESTS the mutagent-builder ai-engineer to amend the Agent/AgentSpec (F18) over
 # SendMessage and re-evals, bounded by an afkloop-legal terminator. It NEVER fixes the subject itself.
@@ -224,11 +224,15 @@ optional_verdict_fields:
 # structured refs + typed assumptions). A LIGHTWEIGHT / FLATTENED emit that drops discovery.evidence.refs
 # or discovery.assumptions is a DEFECT — the gate (lint-grounding) + diff-discriminate operate on those
 # fields; stripping them silently downgrades every criterion to inferred.
-required_label_fields: [traceId, verdict, rootCause, refs, evidencePointer]         # discover — `rootCause` REPLACES firstThingWrong (root-not-symptom); `refs` = structured grounding
+required_label_fields: [traceId, verdict, firstThingWrong, evidencePointer, refs]   # discover — the EXACT keys DiscoverLabelSchema accepts (closed object); `firstThingWrong` HOLDS THE ROOT, not the first visible symptom (root_not_symptom); `refs` = structured grounding
 required_category_fields: [name, definition, class, fixOrEval, exampleTraceIds]     # discover
 # optional_category_fields (P2/P2b, ADDITIVE — the leaf MAY emit these §5b metric-metadata
-# PROPOSALS; AGGREGATE (scripts/aggregate-discover.ts) applies GENERIC defaults when absent,
-# so emitting them is OPTIONAL and backward-compatible): [dimension, level, generality, severity, judgeInputs, codeEval]
+# PROPOSALS; AGGREGATE (scripts/aggregate-discover.ts) HONORS what is present and applies GENERIC
+# defaults ONLY where absent, so emitting them is OPTIONAL and backward-compatible):
+# [dimension, level, generality, severity, judgeInputs, codeEval]
+# Each is a CLOSED union (below) — a value outside it fails LOUD at parse time rather than being
+# silently ignored. (Until the P2 passthrough fix the JOIN dropped all four, so the defaults applied
+# unconditionally and every mined criterion read `level: output` no matter what the leaf proposed.)
 #   dimension  ∈ operation-correctness | data-correctness | operational-deviation
 #   level      ∈ context | output | cross-stage
 #   generality ∈ general-structural | specific-semantic    (else derived from class: code→general-structural)
@@ -310,6 +314,7 @@ commands:
         - "references/grounded-adjudication.md"  # GA doctrine: bind · gather refs · typed assumptions · abstain
         - "subjects/<name>/"                    # the auto-generated subject profile / event taxonomy
         - "references/workflows/orchestrator-protocol.md"  # Step 1 dispatch FSM
+        - "<run_dir>/existing-criteria.json"    # SUITE-AWARENESS — the criteria that ALREADY exist (see suite_awareness)
       workflow:
         - "Pre-read references/error-analysis.md + references/grounded-adjudication.md + the subjects/<name>/ profile + your assigned <key>.task.json determiner specs"
         - "Determine outcome per trace (EV-042) — event→intended goal, trajectory→what happened, terminal→verdict; a guard-hold is a PASS"
@@ -319,14 +324,35 @@ commands:
         - "Trace each Fail to its ROOT (root-not-symptom): one criterion per root; multiple INDEPENDENT roots ⇒ multiple criteria; a causal edge must be GROUNDED (cite it) or surfaced as a typed assumption; deep recursive-why → route to diagnostics"
         - "Surface TYPED assumptions (factual-intent · normative · scope) for any premise the trace did not establish"
         - "Cluster notes into 5-10 emergent categories; tag class (objective→code/subjective→judge/hybrid) + fixOrEval"
+        - "NAME each cluster suite-aware (AFTER clustering, never before — see suite_awareness): if an EXISTING criterion in existing-criteria.json already states this behaviour, REUSE its exact id as the category name and attach your traces as REINFORCING evidence instead of minting a restatement under a new name"
+        - "Emit the §5b PROPOSALS you actually judged — level · severity · dimension · generality. AGGREGATE now HONORS them (they are no longer silently replaced by generic defaults), so an omitted field means 'derive it', not 'it does not matter'"
         - "Flag fixable-vs-eval-worthy (EV-051) — route fixable + infra-class to diagnostics; keep behavioral criteria"
         - "Write <key>.verdict.json (critique-before-verdict) + discover/<batch_id>.json (labels + FULL MinedCriterion per category, incl. discovery.evidence.refs + discovery.assumptions)"
+      # suite_awareness — the leaf SEES the criteria that already exist, so a behaviour
+      # already covered gains EVIDENCE instead of a second name for the same check.
+      # The parent writes `existing-criteria.json` (scripts/aggregate-discover.ts
+      # writeExistingSuiteBrief) into the run dir before dispatch: {id, statement,
+      # severity, level, checkedBy} per existing criterion — NO evidence, deliberately —
+      # PLUS `openQuestions[]`: pairs already awaiting a ruling on whether they are one
+      # check, each with the guard reason. Those carry only the pair + reason; both
+      # sides' statements are already in `criteria[]`, which is what keeps the whole
+      # brief evidence-free by construction.
+      suite_awareness:
+        - "WHEN: consult it AFTER you have clustered, to NAME a cluster. NEVER before."
+        - "WHY THAT ORDER IS LOAD-BEARING: reading the existing suite BEFORE the traces would seed a pre-defined failure list and re-introduce exactly the confirmation bias `emergent_categories_only` forbids — you would go hunting for the criteria you were shown. Clustering FIRST keeps the categories emergent; the suite is consulted only to decide what an already-emerged cluster should be CALLED. Emergence is preserved because the brief cannot influence what you found, only what you name it."
+        - "IT CARRIES NO EVIDENCE ON PURPOSE: the brief ships statements, not the traces or refs behind them, so it cannot tempt you into re-citing a failure you did not read this batch."
+        - "REINFORCE: if an existing criterion already states the behaviour your cluster found, emit the cluster under THAT criterion's exact id. Seeing the behaviour again is legitimate REINFORCING evidence — the parent unions your traces onto the existing criterion and the report marks it REINFORCED."
+        - "OPEN QUESTIONS: the brief's `openQuestions[]` lists pairs of EXISTING criteria already awaiting a human ruling on whether they are one check, each with the reason they were kept apart (e.g. `marker parity: [BEFORE] vs [AFTER]`). BOTH sides are live criteria and both statements are in `criteria[]`. If your cluster matches one side, emit it under THAT side's id — do NOT mint a third name for a behaviour that is already an open question, and do NOT try to settle the question yourself (you are not the ruling authority; that is the calibration loop's). If your cluster genuinely matches NEITHER side, mint it as new."
+        - "INFORMS, NEVER BLOCKS: a full-looking suite is not a reason to suppress a real finding. If the traces show a behaviour no existing criterion states, MINT IT — a genuinely new criterion is always correct, and the parent's deterministic merge is the safety net for a restatement you mint anyway."
+        - "NEVER FABRICATE: do not carry an existing criterion into your mining report because it is in the brief. You may only emit clusters backed by traces YOU actually read this batch — an id reused with no trace of your own behind it is an invented finding."
+        - "ABSENT/EMPTY brief ⇒ proceed exactly as before (a first run has no suite; that is normal, not a gap)."
       compresses:
         - "deep-read trace → goal-attainment Pass/Fail"
         - "DETECT across 3 lenses (drift · tool-output-failure · missing-context)"
         - "GATHER structured refs (claim + absence) · BIND criterion terms"
         - "root-not-symptom localization + typed-assumption surfacing"
         - "emergent category clustering"
+        - "suite-aware NAMING (reuse an existing id ⇒ REINFORCE, never restate)"
         - "fixable-vs-eval-worthy flag"
         - "emit FULL-MinedCriterion verdict + mining files"
       preserves: "the error-analyst discipline — folded INLINE as 'Mode: discover' (below); the standalone assets/agents/error-analyst.md was RETIRED in the 5→3 consolidation (Phase 3a, df6a6e8c8). This file is its canonical home."
@@ -403,13 +429,15 @@ commands:
         - "Pre-read references/write-judge-prompt.md (the 4-component judging contract — your lens) + references/grounded-adjudication.md"
         - "Read your assigned <trajectory_key>.packet.json (MatrixPacket: subject · trajectoryId · WHOLE matrix · trajectory · transcript · pinned envelope) — judge exactly this, never re-derive the data"
         - "Frame the trajectory in its ROUTE/intended-outcome (CONTEXT) before scoring"
-        - "Score EVERY criterion in the matrix for THIS trajectory: read only what the row needs (judgeInputs); compare against statement + passCondition"
+        - "v3 LAYERED WALK (D1=B+, operator-frozen): the walk IS a fixed layer sequence you run INSIDE this one dispatch. Honor `packet.layerScope` when present (absent/empty = FULL walk). The phases: (L0 code checks) if the code-eval library is present at scripts/code-eval-library.ts, RUN it yourself via Bash over the packet's trajectory JSON and record each hit as codeEvalHits[{pattern,anchor,detail}] — a fired pattern is evidence, not a verdict; (L1 outcome) classify the SCENARIO from the inbound intent + tool inventory, derive its expectedExit[] terminal actions, glance ONLY at end-state evidence and emit layerVerdicts[{layer:'L1',verdict,scenario,expectedExit,refs,note}]; (STOPPING RULES) if L1 passes AND every criterion is answerable from evidence gathered so far → you may set earlyExit:'L1' and mark L2-L4 as {verdict:'skipped'} — but a criterion may NEVER go unverdicted because the agent did the right thing; if ANY criterion is unanswerable OR L1 fails/undecidable → descend; (L2 trajectory) actual-vs-expected decision path, read from the END, emit layerVerdicts L2 (+ divergence in localize); (L3 tool outputs) per-observation structural validity + completeness — external-data correctness is UNVERIFIABLE (fence), a failed tool makes everything downstream suspect; emit layerVerdicts L3 (+firstProblemOrigin); (L4 context) for each critical operational send: was required context retrieved AND present in history AT THAT TIME (chronology-sorted!) — distinguish gapKind retrieval (never fetched) vs attention (present, ignored); emit layerVerdicts L4. Record layersEngaged. CONFLICT DISCIPLINE (OT-1): when your own layers disagree (e.g. L1 pass vs L2 fail) emit BOTH verdicts honestly with a note — NEVER reconcile them yourself; the aggregate surfaces the conflict for a human."
+        - "Score EVERY criterion in the matrix for THIS trajectory: read only what the row needs (judgeInputs); compare against statement + passCondition — criteria are the SUPER-LAYER: they may bind evidence from ANY walk phase"
         - "BIND (L1) per row: each criterion TERM must resolve in THIS trajectory; an unbound term ⇒ uncertain + blockedBy:{kind:factual-intent} (INDETERMINATE), ABSTAIN — never fail"
-        - "GATHER: cite a structured ref {obs,path,value} for the claim AND any absence; surface TYPED assumptions for any ungroundable premise → uncertain(blockedBy)"
+        - "GATHER: cite a structured ref {obs,path,value} for the claim AND any absence (ref.obs = the packet's trajectoryId, NEVER an observation name; ref.path = packet-shaped trajectory.N/transcript.N OR trace-shaped observations.N — dot form canonical, brackets tolerated); surface TYPED assumptions for any ungroundable premise → uncertain(blockedBy)"
         - "Critique BEFORE verdict; commit to result ∈ {pass,fail} (uncertain when the INPUTS can't decide — abstain-on-silence); binary, severity is the row's"
         - "Inaction-can-be-success: a correct HOLD (no send during a non-critical outbound_guard) is a PASS even with zero tool calls"
         - "Write <trajectory_key>.verdict.json (a MatrixVerdictFile {trajectoryId, judgeModel, temperature:0, verdicts[]} — each verdict may carry refs?/assumptions?/blockedBy?)"
         - "EMIT-CONTRACT (HARD, WS-1): on every COMPLETE-fidelity trajectory you MUST also persist the §9.4 walk — `understanding` (M2), `expectedTrajectory` (M3), `agentSteps` (the target step lane), and `judgeSteps` (your anchored reasoning). These are NOT optional: the parent runs a machine `assessEmitCompleteness` gate (scripts/emit-completeness.ts) that LOUDLY flags any wholly-dropped field, and a verdict missing them STARVES the report's Trajectory (§2) + Self-Eval (§5) tabs. `agentSteps` is factual trace data (reconstruct it from the ordered tool steps you were given — order by startTime, the observations array is reverse-chronological); M2/M3/judgeSteps are YOUR reasoning. Only an INCOMPLETE (node-1) trajectory is exempt (it emits verdicts:[] and no walk)."
+        - "v3 EMIT-CONTRACT ADDITIONS (E-NEW): alongside the §9.4 walk, persist layerVerdicts[] (one per engaged layer, skipped ones marked), earlyExit?, layersEngaged, codeEvalHits[] (when the library ran), and the emissions SELF-MANIFEST {emitted[], missing[{key,reason}]} — list honestly what you gathered vs what this contract expects; a missing emission with a stated reason is a NAMED degrade, a silent drop is a defect (MR-5 catches it same-run)."
         - "Verdicts are independently VERIFIED (#mode-verify · ≠ judge · downgrade-only); a CRIT/HIGH uncertain rolls the run up to INCOMPLETE at the gate (no false-green)"
       compresses:
         - "read MatrixPacket DATA; frame in ROUTE (CONTEXT)"
@@ -456,9 +484,9 @@ commands:
         - "emit verified verdict (never flip, never fix)"
       preserves: "the GA-5 result-verifier contract (scripts/result-verify.ts verifyVerdict / verifyVerdicts) — see '## Mode: verify' below. A MODE of evaluator, not a 4th registered agent (grounded-adjudication.md sign-off c4)."
 
-  # F18+F19: the ADL ③ IMPROVE / Eval-Driven-Development loop. A MODE of evaluator (roster stays 3),
+  # F18+F19: the ADL ③ OPTIMIZE / Eval-Driven-Development loop. A MODE of evaluator (roster stays 3),
   # not a new subagent. JUDGE-ONLY held: this mode REQUESTS the engineer to fix; it never patches.
-  - improve:
+  - optimize:
       meta:
         what: "Drive the EDD loop: variance-first (F19) → REQUEST the builder ai-engineer to amend (F18) → re-eval → bounded terminate."
         does: "After the initial *build, FIRST stabilizes per-case VARIANCE (repeat-N the SAME cases, default 5; gate the spread) BEFORE measuring accuracy. While JUDGE-ONLY (EV-051), emits a structured EddChangeRequest (failing cases + grounded refs + remedy target agentspec|impl) to the mutagent-builder ai-engineer over SendMessage; consumes the ChangeRequestResponse; re-evals; loops to full green OR a bounded STOP."
@@ -466,7 +494,7 @@ commands:
         how:  "scripts/edd/variance-gate.ts gates variance (evaluateVarianceGate · nextPhaseAfterVariance · assertVarianceStableBeforeAccuracy); scripts/edd/change-request.ts is the seam + the bounded terminator (buildChangeRequest · validate{Change,Response} · reEvalWarranted · decideEddLoop). The eval-runner (sibling) is consumed via the clean interface (references/edd-loop.md §Assumed eval-runner interface) — this mode REBUILDS no runner."
       display: "The EDD loop — variance-first, then request-amend-reeval to full green (bounded)"
       description: |
-        PURPOSE: the ADL ③ IMPROVE stage. Drive the subject to full green via the
+        PURPOSE: the ADL ③ OPTIMIZE stage. Drive the subject to full green via the
         Eval-Driven-Development loop — F19 VARIANCE-FIRST (stabilize per-case spread
         BEFORE accuracy) then F18 EDD CLOSURE (REQUEST the builder ai-engineer to amend the
         Agent/AgentSpec, re-eval, repeat) — bounded by an afkloop-legal terminator.
@@ -475,11 +503,11 @@ commands:
         JUDGE-ONLY (EV-051): this mode NEVER edits the subject — it emits a grounded
         EddChangeRequest over SendMessage and re-evals what the engineer amends.
         NEVER self-dispatches the engineer's fix; NEVER skips re-eval on an amend.
-      dispatch: { mode: improve }
+      dispatch: { mode: optimize }
       pre_gate.loads:
         - "references/edd-loop.md"               # the F18/F19 loop doctrine + assumed eval-runner interface
         - "references/grounded-adjudication.md"  # GA doctrine: every failing-case ask is grounded (≥1 ref)
-        - "references/workflows/orchestrator-protocol.md"  # the improve dispatch step
+        - "references/workflows/orchestrator-protocol.md"  # the optimize dispatch step
       workflow:
         - "Pre-read references/edd-loop.md (the EDD doctrine + the assumed eval-runner interface) + references/grounded-adjudication.md"
         - "PHASE = variance (F19): for each case run repeat-N (default 5) via the eval-runner; collect per-case CaseVarianceObservation {verdicts[], trajectories?}"
@@ -496,7 +524,7 @@ commands:
         - "build GROUNDED change-request (failing cases + remedy target) — judge-only, request-not-patch"
         - "SendMessage to ai-engineer · consume response · re-eval on amend"
         - "bounded terminator (decideEddLoop): full-green ▸ DONE | swings/wallclock/no-improve ▸ STOPPED+delta"
-      preserves: "JUDGE-ONLY (EV-051) — the improve mode REQUESTS a fix and re-evals; it never edits the subject. The amend is the mutagent-builder ai-engineer's job (the ONE agent allowed to touch the Agent/AgentSpec). See '## Mode: improve' below."
+      preserves: "JUDGE-ONLY (EV-051) — the optimize mode REQUESTS a fix and re-evals; it never edits the subject. The amend is the mutagent-builder ai-engineer's job (the ONE agent allowed to touch the Agent/AgentSpec). See '## Mode: optimize' below."
 
 resolution_contract: |
   When you encounter a *<name> token:
@@ -507,8 +535,8 @@ resolution_contract: |
         mode: judge, axis: trajectory   => run #mode-judge-trajectory (preserves eval-matrix-judge); input is a .packet.json.
         mode: judge, axis: criterion    => run #mode-judge-criterion (preserves eval-judge);   input is a .task.json.
         mode: verify                    => run #mode-verify (GA-5 result-verifier; reviewer≠judge, downgrade-only); input is a decided verdict + its situation.
-        mode: improve                   => run #mode-improve (EDD ③ loop — F19 variance-first then F18 request-amend-reeval, bounded). JUDGE-ONLY: REQUEST, never patch.
-        mode: judge, axis: code-quality => run #mode-judge-code-quality (Wave-2 W2I1 — score a CODE subject's QUALITY for the ⑤ IMPROVE code-target BOTH-gate). JUDGE-ONLY; binary quality PASS/FAIL. Rubric mirrored in scripts/code-quality-verdict.ts.
+        mode: optimize                   => run #mode-optimize (EDD ③ loop — F19 variance-first then F18 request-amend-reeval, bounded). JUDGE-ONLY: REQUEST, never patch.
+        mode: judge, axis: code-quality => run #mode-judge-code-quality (Wave-2 W2I1 — score a CODE subject's QUALITY for the ⑤ OPTIMIZE code-target BOTH-gate). JUDGE-ONLY; binary quality PASS/FAIL. Rubric mirrored in scripts/code-quality-verdict.ts.
    4. DISPATCH-INPUT DISAMBIGUATION — the parent hands you files:
         a *.packet.json (MatrixPacket)  => judge/trajectory, ALWAYS (matrix-judge path; no unit field).
         a *.task.json  (JudgeTaskSpec)  => unit.kind: "discover" => discover; else unit.axis: "criterion" => judge/criterion.
@@ -615,8 +643,13 @@ a new eval mid-judging. Your focus is judging what IS, not pondering what could
    - Compare the trajectory/transcript against the row's `statement` + `passCondition`.
    - **GROUND (node 4 — ABSENCE-SPLIT) — REFS ARE MANDATORY, NOT DECORATIVE:** every **decided**
      verdict (`result ∈ {pass, fail}`) MUST carry **≥1 structured `ref {obs, path, value}`** that
-     grounds it — `obs` = the trace/observation id you read, `path` = the field path within it
-     (e.g. `output.response`, `observations[3].output.success`), `value` = the EXACT cited string
+     grounds it — **`obs` = the packet's `trajectoryId` (the trace id), NEVER an observation
+     name** (in a multi-trajectory batch an observation-name `obs` can locate the WRONG
+     trajectory — the resolver takes the FIRST match in batch order), `path` = the field path
+     within the situation you were handed — packet-shaped (`trajectory.0.name`,
+     `transcript.0.content`) and trace-shaped (`observations.3.output.success`) BOTH re-resolve
+     (the verifier reconstructs the packet views deterministically); dot form is canonical,
+     bracket indices (`observations[3]`) are tolerated — `value` = the EXACT cited string
      (re-resolved by whitespace-normalized exact match by `#mode-verify`). **The string your prose
      critique already quotes IS the ref's `value`** — you are not inventing evidence, you are making
      the evidence the critique cites MACHINE-CHECKABLE. A decided verdict with a critique that cites
@@ -690,10 +723,12 @@ a new eval mid-judging. Your focus is judging what IS, not pondering what could
      "refs": [
        { "obs": "ef30a271", "path": "input.prompt", "value": "<outbound_guard consecutive_outbound=\"7\">" },
        { "obs": "ef30a271", "path": "output.response", "value": "held" },
-       { "obs": "ef30a271", "path": "observations[0].name", "value": "ai.generateText" }
+       { "obs": "ef30a271", "path": "observations.0.name", "value": "ai.generateText" }
      ]
    }
    ```
+   (`"ef30a271"` is the packet's `trajectoryId`. Note the dot-form index `observations.0.name` —
+   canonical; `observations[0].name` also resolves.)
    Each `ref.value` is a VERBATIM string from the trace the critique quotes — `#mode-verify`
    re-resolves it by exact match; a value the trace doesn't contain is a dead ref → downgrade.
    **Counter-example (DEFECT — do NOT emit):** the same verdict with `"refs": []` (or `refs`
@@ -860,6 +895,29 @@ this inline section is now its canonical home.
    pick a primitive + field/params); else keep `class: judge` and emit NO codeEval. A code/hybrid
    category WITHOUT a `codeEval` is a HARD error (tier-0 inert). The `statement` stays the human
    "Pass = …"; the `codeEval` is its runnable twin (run by the tier-0 pre-pass, zero judge tokens).
+   **§5b PROPOSALS ARE NOW HONORED:** the `level` / `severity` / `dimension` / `generality` you emit
+   are carried onto the mined criterion and the generic heuristics are the FALLBACK, not the
+   override (they used to be dropped in the JOIN, which pinned every criterion to `level: output`).
+   Emit what you actually judged; omit a field only when you want it derived.
+4b. **NAME the cluster suite-aware — `reinforce_not_restate`.** Only now (AFTER clustering, never
+   before) open `existing-criteria.json`: the criteria the suite ALREADY holds, as
+   `{id, statement, severity, level, checkedBy}`. If one of them already states the behaviour your
+   cluster found, emit your cluster under **that criterion's exact id** — your traces then attach to
+   it as REINFORCING evidence rather than standing up a second name for one check. The ORDER is
+   load-bearing: consulting the suite before reading would hand you a pre-defined failure list and
+   re-create the confirmation bias `emergent_categories_only` exists to prevent. Clustering first
+   keeps what you FIND emergent; the brief only informs what you CALL it. It ships no evidence, so
+   it cannot tempt you into re-citing a failure you did not read. It **informs, it never blocks** —
+   if the traces show something no existing criterion states, MINT IT; and never emit an existing id
+   with no trace of your own behind it (that is an invented finding). An absent or empty brief is
+   normal on a first run — proceed unchanged.
+   The brief also lists **`openQuestions[]`** — pairs of EXISTING criteria already awaiting a human
+   ruling on whether they are really one check, each with the reason they were kept apart (e.g.
+   `marker parity: [BEFORE] vs [AFTER]`). Both sides are live and both statements appear in
+   `criteria[]`. If your cluster matches one side, emit it under **that side's id**: minting a third
+   name for a behaviour that is already an open question makes the question harder to answer, not
+   easier. Do **not** attempt to settle the question yourself — you are not the ruling authority.
+   If your cluster genuinely matches neither side, it is new; mint it.
 5. **Flag fixable-vs-eval-worthy (S4 / EV-051).** Route the fixable + infra-class (e.g. sample C4
    dead-channel `account_number_unavailable`) to diagnostics; keep the genuinely behavioral criteria
    for `*build-evals`. NEVER fix.
@@ -871,14 +929,18 @@ this inline section is now its canonical home.
    the FAILURE + UNCERTAIN traces into DATASET CANDIDATES (`collectDatasetCandidates`, reusing the
    derive-dataset selectors) and folds in the `*evaluate` judge's node-2.5 unmatched-detection
    handoff — both consumable by `*build-dataset`. You mine + flag; the parent emits the candidates.
-7. **Render the discover `discovery-report.html` via the SHIPPED `writeDiscoverRunReport` composer**
-   (`render-discover-report.ts`) — pass the run dir; it builds `triage-summary.json` from
-   `triage.json` + wires every companion (verdicts → Proof-of-work, dataset-candidates → Dataset,
-   triage census → coverage funnel, profile → entity hero) so the report is COMPLETE BY DEFAULT.
+7. **Render the discover `discovery-report.html` via the SHIPPED `writeDiscoverRunReportV3` composer**
+   (`render-discover-report-v3.ts`) — pass the run dir PLUS the AGGREGATE result you already hold in
+   memory (`criteria` · `annotations` · the leaf `miningReport` · the ingested `traces` ·
+   `datasetCandidates` · the grown `suite` provenance). It writes BOTH surfaces: the PRODUCTION
+   W3 frozen-contract report at `discovery-report.html` and the previous renderer's output at
+   `discovery-report.v2.html` (transition fallback) — mirroring eval's `writeRunReport`. Anything you
+   do not pass is read from the run dir; a genuinely absent companion renders as a **NAMED absence**
+   (e.g. "sampling strategy census: not recorded by this run"), NEVER a fabricated number.
    **Do NOT render via bare `writeDiscoverReport(criteria, grounding)`** — without companions it
    degrades the funnel to em-dash and leaves Proof-of-work + Dataset empty (the A4 thin-report gap;
-   the data exists, it just isn't forwarded). Eval's `writeRunReport` is already complete-by-default;
-   discover MUST use its composer to match.
+   the data exists, it just isn't forwarded). The v2 composer `writeDiscoverRunReport`
+   (`render-discover-report.ts`) stays available unchanged for the legacy surface.
 
 ### Governing invariants (verbatim — must survive the merge)
 
@@ -890,6 +952,7 @@ this inline section is now its canonical home.
 - `gather_structured_refs`: "Every observed claim AND every absence claim cites a structured ref {obs,path,value} (ground-absence = a positive field check, never inferred from silence)."
 - `typed_assumptions`: "Every surfaced assumption is TYPED — factual-intent · normative · scope — so its lifecycle/blocking routes correctly (GA-3)."
 - `emergent_categories_only`: "Categories EMERGE from what the traces show. Never start from a pre-defined failure list (confirmation bias). No generic scores as categories."
+- `reinforce_not_restate`: "The existing suite (existing-criteria.json) is consulted AFTER clustering, to NAME an already-emerged cluster — never before, which would seed a pre-defined list and violate emergent_categories_only. A cluster whose behaviour an existing criterion already states is emitted under THAT id (reinforcing evidence), not as a second name for one check. INFORMS, never blocks: a genuinely new behaviour is always minted. An existing id emitted with no trace of your own behind it is a fabricated finding."
 - `judge_only_never_fix`: "EV-051 — FLAGS fixable-vs-eval-worthy and routes the fixable + infra-class to diagnostics. NEVER fixes the subject."
 - `binary_actionable`: "Each emergent category is one binary criterion whose verdict points at a concrete fix locus."
 - `full_mined_criterion_emit`: "Emit the FULL MinedCriterion (base + §5b metadata + §5c discovery incl. structured refs + typed assumptions). A lightweight/flattened emit that drops refs/assumptions is a DEFECT."
@@ -949,9 +1012,9 @@ registered subagent** (the roster stays evaluator · dataset-builder · audit-ex
 - **Feeds the gate:** a downgraded CRIT/HIGH verdict makes the run **INCOMPLETE** (the ternary gate
   `fail ▸ incomplete ▸ pass`) — the latent false-green killer.
 
-## Mode: improve    ← EDD ③ IMPROVE loop (F18 closure + F19 variance-first)
+## Mode: optimize    ← EDD ③ OPTIMIZE loop (F18 closure + F19 variance-first)
 
-THE **Eval-Driven-Development loop** — the ADL ③ IMPROVE stage. After the initial `*build` +
+THE **Eval-Driven-Development loop** — the ADL ③ OPTIMIZE stage. After the initial `*build` +
 `*evaluate`, this mode drives the subject to **full green**. It is a **MODE of `evaluator`, not a new
 registered subagent** (the roster stays evaluator · dataset-builder · audit-executor). It backs
 `scripts/edd/variance-gate.ts` (F19) + `scripts/edd/change-request.ts` (F18). It stays **JUDGE-ONLY
@@ -1013,7 +1076,7 @@ never re-implements, the runner. The interface this mode ASSUMES (documented ver
 ### Governing invariants (must survive any merge)
 
 - `variance_before_accuracy` (F19): "Stabilize per-case variance (repeat-N, default 5) BEFORE measuring accuracy. The accuracy phase is entered ONLY when the variance gate passes (`assertVarianceStableBeforeAccuracy` THROWS otherwise). Accuracy over big samples is wasted on a flapping verdict."
-- `judge_only_requests_never_patches` (F18 · EV-051): "The improve mode REQUESTS the builder ai-engineer to amend (over SendMessage) and re-evals what is amended. It NEVER edits the subject/spec/source itself — the mutagent-builder ai-engineer is the ONE agent allowed to touch the Agent/AgentSpec."
+- `judge_only_requests_never_patches` (F18 · EV-051): "The optimize mode REQUESTS the builder ai-engineer to amend (over SendMessage) and re-evals what is amended. It NEVER edits the subject/spec/source itself — the mutagent-builder ai-engineer is the ONE agent allowed to touch the Agent/AgentSpec."
 - `grounded_change_request` (F18 · GA-1): "Every failing case in a change-request carries the verbatim critique + ≥1 structured ref{obs,path,value}. An ungrounded ask fails loud (`validateChangeRequest`)."
 - `remedy_target_def_vs_impl` (F18): "The evaluator PROPOSES where the fix belongs — `agentspec` (the DEFINITION is wrong → def→impl cascade via *build) vs `impl` (a wiring/faithfulness defect). It proposes; the engineer decides + may reject with a reason."
 - `reeval_always_follows_amend` (F18): "An `amended` response ALWAYS triggers a fresh evaluation swing (from the variance phase). There is no 'fixed it, trust me, skip re-eval' path. A `rejected` response never re-evals an unchanged subject."
@@ -1030,9 +1093,9 @@ never re-implements, the runner. The interface this mode ASSUMES (documented ver
   C-PIN (pinned model + temperature 0).
 - **Additive to the runner:** consumes the assumed eval-runner interface; rebuilds nothing.
 
-## Mode: judge — axis code-quality   ← the ⑤ IMPROVE code-target quality verdict (Wave-2 W2I1)
+## Mode: judge — axis code-quality   ← the ⑤ OPTIMIZE code-target quality verdict (Wave-2 W2I1)
 
-The **(b) half of the ⑤ IMPROVE loop's code-target BOTH-gate.** For a `code`-kind subject, "converged"
+The **(b) half of the ⑤ OPTIMIZE loop's code-target BOTH-gate.** For a `code`-kind subject, "converged"
 requires BOTH (a) the subject's OWN test suite green — a hard DETERMINISTIC gate the ai-engineer
 reports and the optimize oracle records as `testsGreen` — AND (b) THIS code-quality verdict passing.
 This mode is (b): score the QUALITY of the applied code change, BEYOND tests-green (a deterministic
@@ -1064,7 +1127,7 @@ mirroring how the other judge prompts mirror their agent modes).
 ## Eval-leg reconcile — criteria MAINTENANCE, not judging (Wave-2 W2I5 · KP-003)
 
 The evaluator owns the **eval leg** of the def → impl → eval triad — the third leg of `#sync-spec`. When
-an impl amends (the ⑤ IMPROVE loop's ai-engineer, or brownfield drift), the eval criteria that ground the
+an impl amends (the ⑤ OPTIMIZE loop's ai-engineer, or brownfield drift), the eval criteria that ground the
 subject's evaluation can go stale. The reconcile REASONING — which criteria a changed impl makes stale,
 what new criterion it needs — is the **ai-architect's** (`#sync-spec`, session-dispatched, Model-B). The
 evaluator exposes only the deterministic CONTRACT the reconcile reads/writes, in

package/.claude/skills/mutagent-evaluator/assets/code-quality-criteria.yaml

@@ -0,0 +1,75 @@
+# code-quality-criteria.yaml — the CODE-QUALITY rubric (⑤ OPTIMIZE code-target BOTH-gate, W2I1)
+# ═══════════════════════════════════════════════════════════════════════════════
+# THIS FILE IS THE RUBRIC. Edit it to tune what "code quality-pass" means — no code
+# change, no release. `scripts/code-quality-verdict.ts` loads it at import time and
+# exports it as DEFAULT_CODE_QUALITY_CRITERIA.
+#
+# WHAT IT GATES: for a subject whose medium is `code`, "converged" = the subject's OWN
+# test suite green (deterministic) AND this rubric PASSes. Neither alone converges.
+# Prompt/markdown subjects never touch this file — they converge on evals alone.
+#
+# AGGREGATION (severity-gated): a `fail` OR `uncertain` on any `critical`/`high`
+# criterion BLOCKS the quality-pass. `medium` is advisory (recorded, never blocking).
+#
+# EDITING RULES:
+#   - `id` is a byte-identity key + report anchor. Changing an id breaks historical
+#     comparison — add a new criterion instead of renaming an old one.
+#   - Every criterion must be BINARY (decidable pass/fail), never a Likert score.
+#   - `severity` must be exactly one of: critical | high | medium.
+#   - Keep passDefinition/failDefinition CONCRETE — the judge must not have to guess.
+#   - Bump `version` on any change to this file.
+#
+# STATUS: the picks below are the conservative, subject-agnostic starting set
+# (FLAGGED-for-operator-sign-off, Wave-2 W2I1) — a default, not a final contract.
+
+version: 1
+
+criteria:
+  - id: Q1-spec-faithful
+    statement: The implementation realizes the agentspec definition the amend targeted.
+    passDefinition: >-
+      PASS: every spec-declared tool / job-to-be-done is present and the operative
+      system_prompt is unchanged except where the amend intended; the def→impl cascade held.
+    failDefinition: >-
+      FAIL: a spec-declared tool or JTBD is missing, or the system_prompt was altered
+      beyond the amend intent (the impl forked from the SSoT).
+    severity: critical
+
+  - id: Q2-addresses-diagnosis
+    statement: The change targets the diagnosed root-cause locus, not an adjacent symptom.
+    passDefinition: >-
+      PASS: the edit is at (or directly causes a fix at) the RCA-named locus / the
+      failing-case root.
+    failDefinition: >-
+      FAIL: the change edits an unrelated place or patches a symptom while the root
+      cause remains.
+    severity: high
+
+  - id: Q3-no-regression
+    statement: The change introduces no unrelated regression and stays in scope.
+    passDefinition: >-
+      PASS: only the remedy-named surface changed; no unrelated behavior was removed
+      or broken.
+    failDefinition: >-
+      FAIL: unrelated code was changed/removed, or a previously-working path was broken.
+    severity: high
+
+  - id: Q4-error-handling
+    statement: Failure paths are handled, not silently dropped.
+    passDefinition: >-
+      PASS: error/failure branches surface, recover, or log per the subject's contract;
+      no new silent drop.
+    failDefinition: >-
+      FAIL: a failure is swallowed with no recovery/notify/observability (a silent-drop
+      regression).
+    severity: high
+
+  - id: Q5-maintainable
+    statement: The code is readable, framework-idiomatic, and free of obvious footguns.
+    passDefinition: >-
+      PASS: clear structure, idiomatic to the target framework, no dead code /
+      copy-paste / obvious footgun introduced.
+    failDefinition: >-
+      FAIL: unreadable/duplicated/dead code or an obvious maintainability footgun was
+      introduced.
+    severity: medium