@mutagent/evaluator 0.1.0-alpha.4 → 0.1.0-alpha.6

package/.claude/skills/mutagent-evaluator/SKILL.md

@@ -23,7 +23,7 @@ license: Proprietary. LICENSE.txt has complete terms.
 compatibility: Designed for Claude Code, Codex, Cursor, OpenCode and similar coding-agent runtimes; works with git, gh CLI, jq, curl, and Bun/pnpm/npm runtimes.
 metadata:
   author: mutagent
-  version: "0.1.0-alpha.2"
+  version: "0.1.0-alpha.4"
 # allowed-tools: OMITTED — agent uses all native tools per host runtime
 ---
 
@@ -103,7 +103,7 @@ reason for the rest. NEVER improvise.
 | `*discover-evals` | hybrid | `references/workflows/orchestrator-protocol.md` + `scripts/prep-tasks.ts` + `scripts/determine-outcome.ts` + `scripts/discover-criteria.ts` + `assets/agents/evaluator.md` (`#mode-discover`) | PREP determiner tasks → fan out `evaluator` (`#mode-discover`, mass-parallel) → AGGREGATE ✓/✗ → mine emergent BINARY ACTIONABLE criteria (EV-041/042/052) **+ T6 failure/uncertain DATASET CANDIDATES** (`collectDatasetCandidates`, reuses the derive-dataset selectors → `*build-dataset`). **GA split** — **code:** sample (broken+healthy) · aggregate · `diff-discriminate` · ground-gate · dataset-candidates; **LLM leaf:** determine + 3 detectors + cite refs + typed assumptions (root-not-symptom). |
 | `*build-evals` | hybrid | `references/workflows/orchestrator-protocol.md` + `scripts/eval-engine.ts` + `scripts/codegen-evals.ts` + `scripts/prep-tasks.ts` + `scripts/build-evals.ts` + `assets/agents/evaluator.md` (`#mode-judge-criterion`) + `scripts/render-build-cards.ts` | **ENGINE-FORK FIRST (ADL F7/F9/F14).** ASKS the eval implementation mode via `chooseEvalEngineOptions(target)` — Path A `native-matrix` (eval-matrix + LLM-judge SUB-AGENTS; SURFACES the Claude-Code + log-sink dependency up front) vs Path B `code-written` (`codegen-evals.ts` emits a portable bun/TS suite — runs WITHOUT Claude Code, F14). Target-conditional: a code framework offers BOTH, a `harness:*` target is native-only. **Path A** then PREPs judge tasks (criterion × trace-slice) → fan out `evaluator` (`#mode-judge-criterion`, mass-parallel) → one binary+confidence judge per criterion (EV-043). **GA split** — **code:** engine-resolve · spec render · `lint-grounding` · `resolve-ref` (BIND) · codegen; **LLM leaf:** judge + VERIFY (cite refs · note assumptions · abstain). Streams progress + a verbose evals entity card (F13/F16/F22). |
 | `*evaluate` | hybrid | `references/workflows/orchestrator-protocol.md` + `scripts/matrix-judge.ts` + `scripts/contracts/eval-matrix.ts` + `assets/agents/evaluator.md` (`#mode-judge-trajectory`) + `scripts/evaluate.ts` | **DEFAULT (headline):** **T1 TIER-0 deterministic pre-pass** (code-method rows run first, zero judge tokens) → PREP one matrix packet per RESIDUAL judge trajectory (**T5 adaptive-K guard, default 1:1**) → fan out `evaluator` (`#mode-judge-trajectory` — one judge/trajectory scoring the WHOLE matrix, emitting the **T2 Judge DAG v2 walk** = `judge_steps[]` + dense na-explicit map + confidence band + early-INCOMPLETE + node-2.5 candidates) → AGGREGATE: fold code+judge verdicts → **T3 independent verifier** refutes GATING fails (downgrade-only) → **T4 consolidate-by-locus + walk-derived health** → GATE verdict + variance view (EV-048). **GA split** — **code:** tier-0 · prep · `lint-grounding` · `resolve-ref` · `assemble-scorecard` (gate `fail ▸ incomplete ▸ pass`) · consolidate-by-locus · route-failures; **LLM leaf:** trajectory judge (DAG v2 walk: BIND · GROUND[absence-split] · cite · abstain) + independent VERIFY. **MAY now return `incomplete`** (indeterminate → calibrate). **ADL F20:** the rollup ALSO renders a SCORECARD DASHBOARD wireframe (`renderScorecardDashboard` — per-criterion pass/fail bar + variance + samples), not a flat terminal dump. Under a **Path B** engine, `*evaluate` instead runs the portable `codegen-evals.ts` suite (no Claude Code) and reads back its scorecard JSON from the discoverable sink. |
-| `*improve` | hybrid | `references/edd-loop.md` + `scripts/edd/variance-gate.ts` + `scripts/edd/change-request.ts` + `scripts/edd/edd-types.ts` + `schemas/edd-change-request.schema.yaml` + `assets/agents/evaluator.md` (`#mode-improve`) | **ADL ③ IMPROVE / EDD loop (F18+F19).** **F19 VARIANCE-FIRST:** repeat-N (default 5) the SAME cases → `evaluateVarianceGate` → accuracy is entered ONLY when the variance gate passes (`assertVarianceStableBeforeAccuracy` THROWS otherwise — "accuracy over big samples is wasted on a flapping verdict"). **F18 CLOSURE:** still JUDGE-ONLY (EV-051), the evaluator emits a grounded `EddChangeRequest` (failing cases + `ref{obs,path,value}` + remedy target `agentspec`\|`impl`) to the `mutagent-builder ai-engineer` over **SendMessage**, consumes the `ChangeRequestResponse`, and re-evals on `amended`. **Bounded terminator** (`decideEddLoop`, afkloop-legal): `full-green` ▸ DONE \| `max-swings`\|`max-wallclock`\|`no-improvement-streak` ▸ STOPPED + convergence delta. **GA split** — **code:** variance gate · request/response validate · loop terminator (all PURE, injected wall-clock); **LLM leaf:** localize the flap/fail to its root + author the grounded request + decide remedy target. |
+| `*optimize` | hybrid | `references/edd-loop.md` + `scripts/edd/variance-gate.ts` + `scripts/edd/change-request.ts` + `scripts/edd/edd-types.ts` + `schemas/edd-change-request.schema.yaml` + `assets/agents/evaluator.md` (`#mode-optimize`) | **ADL ③ OPTIMIZE / EDD loop (F18+F19).** **F19 VARIANCE-FIRST:** repeat-N (default 5) the SAME cases → `evaluateVarianceGate` → accuracy is entered ONLY when the variance gate passes (`assertVarianceStableBeforeAccuracy` THROWS otherwise — "accuracy over big samples is wasted on a flapping verdict"). **F18 CLOSURE:** still JUDGE-ONLY (EV-051), the evaluator emits a grounded `EddChangeRequest` (failing cases + `ref{obs,path,value}` + remedy target `agentspec`\|`impl`) to the `mutagent-builder ai-engineer` over **SendMessage**, consumes the `ChangeRequestResponse`, and re-evals on `amended`. **Bounded terminator** (`decideEddLoop`, afkloop-legal): `full-green` ▸ DONE \| `max-swings`\|`max-wallclock`\|`no-improvement-streak` ▸ STOPPED + convergence delta. **GA split** — **code:** variance gate · request/response validate · loop terminator (all PURE, injected wall-clock); **LLM leaf:** localize the flap/fail to its root + author the grounded request + decide remedy target. |
 | `*validate` | hybrid | `scripts/validate-judge.ts` + `references/validate-evaluator.md` + `assets/agents/evaluator.md` (`#mode-judge-criterion`) | **ENGINED (EV-044)** — calibrate a judge vs `*review` labels: confusion matrix → TPR/TNR · split-disjointness + test-once · Rogan-Gladen θ · deterministic bootstrap CI; `<MIN_LABELS` stays `unvalidated`+bias-corrected. **GA split** — **code:** entire path is deterministic (filter EXCLUDES indeterminate · confusion · RG · assumption-agreement). |
 | `*review` | hybrid | `scripts/build-review-ui.ts` + `references/build-review-interface.md` | **ENGINED (EV-045)** — CODE renders a browser annotation UI (one trace/screen · Pass/Fail/Defer · keyboard · auto-save · labels export); **HITL**: a human labels → `mergeLabels` persists → feeds `*validate`. |
 | `*eval` | agent-chain | `references/eval-stage.md` + `scripts/eval-engine.ts` + `scripts/materialize-dataset.ts` + `scripts/render-build-cards.ts` | **THE ADL EVAL-STAGE ENTRY (F15).** Called after `*build` hands a freshly-built agent + its agentspec. INTERACTIVELY offers `*build-dataset` / `*build-evals` derived from `agentspec.definition.evals`, and lets the user PICK THE EVAL ENGINE (mutagent native eval-matrix [Path A] vs code-written-in-target-lang [Path B]) via the target-conditional menu (`chooseEvalEngineOptions`). Surfaces Path A's Claude-Code + log-sink dependency UP FRONT (F7/F9). Streams wireframe progress + entity cards (F13/F16/F22). Parent-session only (AskUserQuestion). |
@@ -138,7 +138,7 @@ When you encounter a *<name> token:
 Invoke me with:
 - `evaluate skill <name>` / `evaluate agent <name>` / `/mutagent-evaluator`  → **`*evaluate`** (default)
 - `*discover-evals` (mine criteria) · `build evals for <x>` · `validate judge <x>` · `*audit <subject>` (explicit only)
-- `*improve` (the ADL ③ EDD loop — variance-first then request-amend-reeval to full green, bounded)
+- `*optimize` (the ADL ③ EDD loop — variance-first then request-amend-reeval to full green, bounded)
 - `*self-audit` (the eval-of-the-eval — audit my OWN eval-dev; on-demand only)
 - borrowed by `mutagent-skill-builder` SIMULATE (SimVerifier role)
 - `--reconfigure` to re-enter onboarding (subject + substrate)
@@ -151,7 +151,7 @@ Minimal eval-dev spine (build order — do NOT reorder):
 *discover-evals     → deep-read traces, determine ✓/✗, mine 5-10 binary actionable criteria
 *build-evals  → one critique-before-verdict judge (or code-check) per criterion
 *evaluate     → run vs target → per-criterion binary+confidence → GATE verdict + variance
-*improve      → ADL ③ EDD loop: F19 variance-first (stabilize per-case spread, repeat-N)
+*optimize      → ADL ③ EDD loop: F19 variance-first (stabilize per-case spread, repeat-N)
                 BEFORE accuracy, then F18 closure (judge-only REQUESTS the ai-engineer to
                 amend agentspec|impl over SendMessage → re-eval) → full green OR bounded STOP
 ```
@@ -159,7 +159,7 @@ Minimal eval-dev spine (build order — do NOT reorder):
 `*validate` (calibrate the judge) layers on top. `*audit` (the v1 4-tab master-audit) is an
 **explicit-only** specialized surface (complex / HTML-artifact subjects) — NOT part of the default
 evaluate spine and never an automatic fallback.
-`*improve` is the IMPROVE stage — it runs AFTER an initial `*build` + `*evaluate` and is **bounded**
+`*optimize` is the OPTIMIZE stage — it runs AFTER an initial `*build` + `*evaluate` and is **bounded**
 (never infinite): full-green ⇒ DONE; else max-swings / max-wallclock / no-improvement ⇒ STOP + delta.
 
 **ADL EVAL-stage entry (`*eval`).** When `*build` hands over a freshly-built agent +
@@ -276,6 +276,7 @@ applies, EV-053/EQ5):
 | `scripts/aggregate-discover.ts` | P2 (§5a) + §9.4 (T6) | code | The real `*discover-evals` AGGREGATE half: parse the leaf's per-trace verdict files + per-batch mining report → `TraceAnnotation[]` (FAIL-LOUD via `missingVerdictKeys`; THROW on an undispatched-trace category ref) → `deriveMinedCriteria` (§5b metadata + §5c DR-2) → `growLivingSuite` (append-only, monotonic, saturation-stop). **T6:** `collectDatasetCandidates` (failure/uncertain → `DatasetCase[]` via the derive-dataset selectors) + `unmatchedDetectionCandidates` (the `*evaluate` judge's node-2.5 handoff) → consumable by `*build-dataset`. `DiscoverMiningReport` contract. No hand-rolled annotations. |
 | `scripts/code-eval.ts` | P3 (EX-2) | code | The v2 CODE-track primitive library — generic PURE `(trace) → {result, detail}` checks: presence · string-equality · format-validity · schema-conformance · ref-integrity (cross-stage). SV-1 dotted-path read incl. `obs:<name>.<path>`. Exhaustive `runCodeEval` (unknown primitive → THROW). NO LLM, byte-identical. Disjoint from the v1 `run-deterministic.ts` audit world. |
 | `scripts/check-method-router.ts` | P3 (EX-2) | hybrid | THE load-bearing §5b `check_method` router: `deterministic`→CODE-EXEC (judge NEVER called) · `llm-judge`→JUDGE seam · `hybrid`→code pre-filter that GATES the judge. Unknown → THROW (exhaustive). `RoutedCriterionVerdict` = `CriterionVerdict` + provenance `{producedBy, codeResult, judgeReached}` (structural superset → `evaluate.ts` rollup unchanged). `extractCodeEval(registry)` fail-loud (a code-class criterion with no spec THROWS — never silently judge). "Code before judge" lives INSIDE the agent. |
+| `scripts/code-quality-verdict.ts` | W2I1 (⑤ OPTIMIZE) | hybrid | The **code-quality JUDGE mode** (`#mode-judge-code-quality`) — the (b) half of the ⑤ OPTIMIZE code-target BOTH-gate. Scores a `code` subject's QUALITY (spec-faithfulness · addresses-diagnosis · no-regression · error-handling · maintainability) BEYOND tests-green → severity-gated binary PASS/FAIL. `buildCodeQualityPrompt` (pure renderer, critique-before-verdict) + `codeQualityGatePass` (pure aggregate) + `runCodeQualityJudge` (C-PIN run wrapper on the injected JudgeInvoke seam). Judge-only (EV-051), host-runtime, pinned model + temp 0. **DISTINCT from `code-eval.ts`** (deterministic primitives — no LLM). `DEFAULT_CODE_QUALITY_CRITERIA` + its aggregation rule are FLAGGED-for-operator-sign-off. |
 | `scripts/run-evaluate.ts` | P5 + §9.4 (T1·T3·T4·T5) | hybrid | The real-engine `*evaluate` end-to-end COMPOSER: `tier0Plan` (T1 — code rows decided in the pre-pass, residual judge rows only in packets) → `prepMatrixPackets` (residual-only) → `adaptivePacketPlan` (T5 overload guard, default 1:1) → [dispatch #mode-judge-trajectory] → `aggregateEvaluate` (FAIL-LOUD readiness gate over DISPATCHED ids → `aggregateMatrixScorecard` folding code+judge + T3 `independentVerify` downgrade-only on gating fails → GATE + variance → T4 `consolidateByLocus` lociClusters → EV-051 `routeFailures`→validated HandoverBundle). `maskedScorecard` = the C-PIN byte-identity artifact. |
 | `scripts/render-eval-report.ts` | P4 (D-1/2/3) | code | The v2 reporting (v1 `render-report.ts` untouched): `renderEvalCards` (D-1 terminal per-criterion cards) + `buildEvalReportInput` (derives the rich tabs from real run data — ledger · coverage · cohorts · gating rolls · top-findings · judge-health) + `renderEvalReport` (the operator-APPROVED 5 tabs: ① Overview KPIs+coverage+gating-table+top-findings · ② Trajectory‖Judge ledger + click-row Target-Agent‖Judge side-by-side · ③ Eval Scorecard cohort heatmap + nested subcards + inline calibration · ④ Findings verbatim-evidence + judge chain + agree/revise/refute · ⑤ Self-Eval [INTERNAL, strip-marked]). Consumes the §9.4 judge-walk (`judge_steps[]` + dense map) when present, degrades to the per-trajectory scorecard otherwise. Tabs are `<button>` (no `<s>` strikethrough). Unified design-system tokens (`assets/brand/theme.css`). byte-identical after the `generatedAt` mask. `autoOpenCommand` cross-platform helper (orchestrator fires it). |
 | `scripts/contracts/eval-types.ts` *(extended)* | P2/P2b | code | + `MinedCriterion` (additive subtype of `DiscoveredCriterion`) carrying the §5b `MetricMetadata` (generality·dimension·level·check_method [3-value SF-2 router]·substrate·applies_to·severity·judge_inputs·flag) + the §5c `DiscoveryRationale` (DR-2: targets·why·evidence{grounding observed\|inferred·prevalence k/n·refs}·assumptions·expected_impact). `parseMinedCriterion` enforces the evidence-first gate (OBSERVED ⇒ refs + k>0). |

package/.claude/skills/mutagent-evaluator/assets/agents/evaluator.md

@@ -20,9 +20,9 @@ ACTIVATION-NOTICE: This file contains your full agent operating guidelines. Read
 class: pure_subagent_executor
 isolation: none
 
-modes: [discover, judge, verify, improve]  # discover · judge · verify (GA-5) · improve (EDD ③ — F18/F19)
+modes: [discover, judge, verify, optimize]  # discover · judge · verify (GA-5) · optimize (EDD ③ — F18/F19)
 judge_axes: [trajectory, criterion] # judge-mode fan-out axes
-# `improve` is the ADL ③ IMPROVE / Eval-Driven-Development (EDD) LOOP mode (F18+F19). It is a MODE of
+# `optimize` is the ADL ③ OPTIMIZE / Eval-Driven-Development (EDD) LOOP mode (F18+F19). It is a MODE of
 # this ONE agent (the roster stays 3) — it drives the variance-first gate (F19) then, while still
 # JUDGE-ONLY (EV-051), REQUESTS the mutagent-builder ai-engineer to amend the Agent/AgentSpec (F18) over
 # SendMessage and re-evals, bounded by an afkloop-legal terminator. It NEVER fixes the subject itself.
@@ -456,9 +456,9 @@ commands:
         - "emit verified verdict (never flip, never fix)"
       preserves: "the GA-5 result-verifier contract (scripts/result-verify.ts verifyVerdict / verifyVerdicts) — see '## Mode: verify' below. A MODE of evaluator, not a 4th registered agent (grounded-adjudication.md sign-off c4)."
 
-  # F18+F19: the ADL ③ IMPROVE / Eval-Driven-Development loop. A MODE of evaluator (roster stays 3),
+  # F18+F19: the ADL ③ OPTIMIZE / Eval-Driven-Development loop. A MODE of evaluator (roster stays 3),
   # not a new subagent. JUDGE-ONLY held: this mode REQUESTS the engineer to fix; it never patches.
-  - improve:
+  - optimize:
       meta:
         what: "Drive the EDD loop: variance-first (F19) → REQUEST the builder ai-engineer to amend (F18) → re-eval → bounded terminate."
         does: "After the initial *build, FIRST stabilizes per-case VARIANCE (repeat-N the SAME cases, default 5; gate the spread) BEFORE measuring accuracy. While JUDGE-ONLY (EV-051), emits a structured EddChangeRequest (failing cases + grounded refs + remedy target agentspec|impl) to the mutagent-builder ai-engineer over SendMessage; consumes the ChangeRequestResponse; re-evals; loops to full green OR a bounded STOP."
@@ -466,7 +466,7 @@ commands:
         how:  "scripts/edd/variance-gate.ts gates variance (evaluateVarianceGate · nextPhaseAfterVariance · assertVarianceStableBeforeAccuracy); scripts/edd/change-request.ts is the seam + the bounded terminator (buildChangeRequest · validate{Change,Response} · reEvalWarranted · decideEddLoop). The eval-runner (sibling) is consumed via the clean interface (references/edd-loop.md §Assumed eval-runner interface) — this mode REBUILDS no runner."
       display: "The EDD loop — variance-first, then request-amend-reeval to full green (bounded)"
       description: |
-        PURPOSE: the ADL ③ IMPROVE stage. Drive the subject to full green via the
+        PURPOSE: the ADL ③ OPTIMIZE stage. Drive the subject to full green via the
         Eval-Driven-Development loop — F19 VARIANCE-FIRST (stabilize per-case spread
         BEFORE accuracy) then F18 EDD CLOSURE (REQUEST the builder ai-engineer to amend the
         Agent/AgentSpec, re-eval, repeat) — bounded by an afkloop-legal terminator.
@@ -475,11 +475,11 @@ commands:
         JUDGE-ONLY (EV-051): this mode NEVER edits the subject — it emits a grounded
         EddChangeRequest over SendMessage and re-evals what the engineer amends.
         NEVER self-dispatches the engineer's fix; NEVER skips re-eval on an amend.
-      dispatch: { mode: improve }
+      dispatch: { mode: optimize }
       pre_gate.loads:
         - "references/edd-loop.md"               # the F18/F19 loop doctrine + assumed eval-runner interface
         - "references/grounded-adjudication.md"  # GA doctrine: every failing-case ask is grounded (≥1 ref)
-        - "references/workflows/orchestrator-protocol.md"  # the improve dispatch step
+        - "references/workflows/orchestrator-protocol.md"  # the optimize dispatch step
       workflow:
         - "Pre-read references/edd-loop.md (the EDD doctrine + the assumed eval-runner interface) + references/grounded-adjudication.md"
         - "PHASE = variance (F19): for each case run repeat-N (default 5) via the eval-runner; collect per-case CaseVarianceObservation {verdicts[], trajectories?}"
@@ -496,7 +496,7 @@ commands:
         - "build GROUNDED change-request (failing cases + remedy target) — judge-only, request-not-patch"
         - "SendMessage to ai-engineer · consume response · re-eval on amend"
         - "bounded terminator (decideEddLoop): full-green ▸ DONE | swings/wallclock/no-improve ▸ STOPPED+delta"
-      preserves: "JUDGE-ONLY (EV-051) — the improve mode REQUESTS a fix and re-evals; it never edits the subject. The amend is the mutagent-builder ai-engineer's job (the ONE agent allowed to touch the Agent/AgentSpec). See '## Mode: improve' below."
+      preserves: "JUDGE-ONLY (EV-051) — the optimize mode REQUESTS a fix and re-evals; it never edits the subject. The amend is the mutagent-builder ai-engineer's job (the ONE agent allowed to touch the Agent/AgentSpec). See '## Mode: optimize' below."
 
 resolution_contract: |
   When you encounter a *<name> token:
@@ -507,7 +507,8 @@ resolution_contract: |
         mode: judge, axis: trajectory   => run #mode-judge-trajectory (preserves eval-matrix-judge); input is a .packet.json.
         mode: judge, axis: criterion    => run #mode-judge-criterion (preserves eval-judge);   input is a .task.json.
         mode: verify                    => run #mode-verify (GA-5 result-verifier; reviewer≠judge, downgrade-only); input is a decided verdict + its situation.
-        mode: improve                   => run #mode-improve (EDD ③ loop — F19 variance-first then F18 request-amend-reeval, bounded). JUDGE-ONLY: REQUEST, never patch.
+        mode: optimize                   => run #mode-optimize (EDD ③ loop — F19 variance-first then F18 request-amend-reeval, bounded). JUDGE-ONLY: REQUEST, never patch.
+        mode: judge, axis: code-quality => run #mode-judge-code-quality (Wave-2 W2I1 — score a CODE subject's QUALITY for the ⑤ OPTIMIZE code-target BOTH-gate). JUDGE-ONLY; binary quality PASS/FAIL. Rubric mirrored in scripts/code-quality-verdict.ts.
    4. DISPATCH-INPUT DISAMBIGUATION — the parent hands you files:
         a *.packet.json (MatrixPacket)  => judge/trajectory, ALWAYS (matrix-judge path; no unit field).
         a *.task.json  (JudgeTaskSpec)  => unit.kind: "discover" => discover; else unit.axis: "criterion" => judge/criterion.
@@ -948,9 +949,9 @@ registered subagent** (the roster stays evaluator · dataset-builder · audit-ex
 - **Feeds the gate:** a downgraded CRIT/HIGH verdict makes the run **INCOMPLETE** (the ternary gate
   `fail ▸ incomplete ▸ pass`) — the latent false-green killer.
 
-## Mode: improve    ← EDD ③ IMPROVE loop (F18 closure + F19 variance-first)
+## Mode: optimize    ← EDD ③ OPTIMIZE loop (F18 closure + F19 variance-first)
 
-THE **Eval-Driven-Development loop** — the ADL ③ IMPROVE stage. After the initial `*build` +
+THE **Eval-Driven-Development loop** — the ADL ③ OPTIMIZE stage. After the initial `*build` +
 `*evaluate`, this mode drives the subject to **full green**. It is a **MODE of `evaluator`, not a new
 registered subagent** (the roster stays evaluator · dataset-builder · audit-executor). It backs
 `scripts/edd/variance-gate.ts` (F19) + `scripts/edd/change-request.ts` (F18). It stays **JUDGE-ONLY
@@ -1012,7 +1013,7 @@ never re-implements, the runner. The interface this mode ASSUMES (documented ver
 ### Governing invariants (must survive any merge)
 
 - `variance_before_accuracy` (F19): "Stabilize per-case variance (repeat-N, default 5) BEFORE measuring accuracy. The accuracy phase is entered ONLY when the variance gate passes (`assertVarianceStableBeforeAccuracy` THROWS otherwise). Accuracy over big samples is wasted on a flapping verdict."
-- `judge_only_requests_never_patches` (F18 · EV-051): "The improve mode REQUESTS the builder ai-engineer to amend (over SendMessage) and re-evals what is amended. It NEVER edits the subject/spec/source itself — the mutagent-builder ai-engineer is the ONE agent allowed to touch the Agent/AgentSpec."
+- `judge_only_requests_never_patches` (F18 · EV-051): "The optimize mode REQUESTS the builder ai-engineer to amend (over SendMessage) and re-evals what is amended. It NEVER edits the subject/spec/source itself — the mutagent-builder ai-engineer is the ONE agent allowed to touch the Agent/AgentSpec."
 - `grounded_change_request` (F18 · GA-1): "Every failing case in a change-request carries the verbatim critique + ≥1 structured ref{obs,path,value}. An ungrounded ask fails loud (`validateChangeRequest`)."
 - `remedy_target_def_vs_impl` (F18): "The evaluator PROPOSES where the fix belongs — `agentspec` (the DEFINITION is wrong → def→impl cascade via *build) vs `impl` (a wiring/faithfulness defect). It proposes; the engineer decides + may reject with a reason."
 - `reeval_always_follows_amend` (F18): "An `amended` response ALWAYS triggers a fresh evaluation swing (from the variance phase). There is no 'fixed it, trust me, skip re-eval' path. A `rejected` response never re-evals an unchanged subject."
@@ -1029,6 +1030,69 @@ never re-implements, the runner. The interface this mode ASSUMES (documented ver
   C-PIN (pinned model + temperature 0).
 - **Additive to the runner:** consumes the assumed eval-runner interface; rebuilds nothing.
 
+## Mode: judge — axis code-quality   ← the ⑤ OPTIMIZE code-target quality verdict (Wave-2 W2I1)
+
+The **(b) half of the ⑤ OPTIMIZE loop's code-target BOTH-gate.** For a `code`-kind subject, "converged"
+requires BOTH (a) the subject's OWN test suite green — a hard DETERMINISTIC gate the ai-engineer
+reports and the optimize oracle records as `testsGreen` — AND (b) THIS code-quality verdict passing.
+This mode is (b): score the QUALITY of the applied code change, BEYOND tests-green (a deterministic
+primitive cannot decide spec-faithfulness, scope, error-handling, maintainability — a judge must).
+
+### What it does (per dispatch)
+
+Judge the CODE subject (the amend diff / scaffold against the agentspec definition it must realize)
+on the DRAFT criteria set — **critique-BEFORE-verdict, strictly BINARY** per criterion — and emit a
+severity-gated binary quality **PASS/FAIL**. The authoritative rubric + the drafted default criteria
+live in `scripts/code-quality-verdict.ts` (`DEFAULT_CODE_QUALITY_CRITERIA` — Q1 spec-faithful [crit] ·
+Q2 addresses-diagnosis [high] · Q3 no-regression [high] · Q4 error-handling [high] · Q5 maintainable
+[med]); this mode is that rubric's dispatched form (the provider-callable mirror is the same file,
+mirroring how the other judge prompts mirror their agent modes).
+
+> **⚠️ FLAGGED-FOR-OPERATOR-SIGN-OFF.** The exact acceptance criteria + the severity-gated aggregation
+> rule (CRIT/HIGH `fail`|`uncertain` blocks; MEDIUM advisory) are a DRAFT default, implemented behind
+> a clearly-named constant so the operator can confirm/adjust WITHOUT a loop code change. Do not treat
+> the current set as final.
+
+### Boundaries
+
+- **Judge-only (EV-051):** score the quality; NEVER propose or write a fix (the ⑤ loop's ai-engineer
+  amends). A FAIL just keeps the loop iterating.
+- **Host-runtime, no provider key; C-PIN:** pinned model + temperature 0 — byte-identical reruns.
+- **Separate from tests-green:** you judge quality, NOT whether tests pass; the deterministic test
+  gate is ai-engineer's report, ANDed by the oracle — never fold it into your quality verdict.
+
+## Eval-leg reconcile — criteria MAINTENANCE, not judging (Wave-2 W2I5 · KP-003)
+
+The evaluator owns the **eval leg** of the def → impl → eval triad — the third leg of `#sync-spec`. When
+an impl amends (the ⑤ OPTIMIZE loop's ai-engineer, or brownfield drift), the eval criteria that ground the
+subject's evaluation can go stale. The reconcile REASONING — which criteria a changed impl makes stale,
+what new criterion it needs — is the **ai-architect's** (`#sync-spec`, session-dispatched, Model-B). The
+evaluator exposes only the deterministic CONTRACT the reconcile reads/writes, in
+`scripts/sync-eval-criteria.ts`:
+
+- `evalLegForSubjectKind(kind)` — the shape resolver: agent/skill/composite → the **eval-suite criteria**
+  (this cell's discovered/maintained criteria); code → the **code-quality criteria**
+  (`DEFAULT_CODE_QUALITY_CRITERIA` / `#mode-judge-code-quality`, Wave-2 W2I1).
+- `flagEvalLegDrift(...)` / `assessEvalLeg(...)` — a pure freshness FLAG (mirrors the builder probe).
+- `reconcileEvalCriteria(request)` — the COMPUTE: upsert the ai-architect-proposed criteria delta by id,
+  append the novel, and NEVER drop a criterion (EV-053, via `assertMonotonicGrowth`). It returns the grown
+  set as DATA (`result.criteria`) — it does **not** touch disk; the reconcile hook COMPUTES, it is not the write.
+- `persistEvalCriteria(result, { cwd, updatedAt? })` (`scripts/persist-eval-criteria.ts`) — the WRITE: the
+  evaluator session PERSISTS `result.criteria` back to the located criteria artifact
+  (`.mutagent/evaluator/living-suite/<leg>.criteria.json`, per-leg — eval-suite for an agent/skill/composite
+  subject, code-quality for a code subject). That write is what BUMPS the artifact's freshness (its mtime — and
+  the optional `updatedAt` marker the builder probe reads), which returns the eval leg to `in-sync`: the loop
+  only closes once the reconciled criteria are on disk, not when they are merely computed.
+
+> **Judge-only PRESERVED (EV-051).** This is criteria MAINTENANCE — growing/revising the criteria SET to
+> match a changed impl — NOT judging a subject. `sync-eval-criteria.ts` scores nothing and decides no
+> pass/fail; it is pure set algebra + a freshness flag (no clock/random/network — C-PIN byte-identical). The
+> `persist-eval-criteria.ts` WRITER is the IO layer (it is the one place that touches disk, mirroring how
+> `artifact-paths.ts` does IO while `living-suite.ts` stays pure); its serialization is still deterministic
+> (stable key order, no clock unless the caller passes `updatedAt`), so a persisted artifact round-trips
+> byte-faithful. The evaluator never mutates the impl and never fixes the subject; it only maintains — and now
+> persists — its own criteria.
+
 ## Boundaries (shared)
 
 judge-only (EV-051) · model-intent-sacred · critique-before-verdict · binary-not-Likert ·

package/.claude/skills/mutagent-evaluator/assets/code-quality-criteria.yaml

@@ -0,0 +1,75 @@
+# code-quality-criteria.yaml — the CODE-QUALITY rubric (⑤ OPTIMIZE code-target BOTH-gate, W2I1)
+# ═══════════════════════════════════════════════════════════════════════════════
+# THIS FILE IS THE RUBRIC. Edit it to tune what "code quality-pass" means — no code
+# change, no release. `scripts/code-quality-verdict.ts` loads it at import time and
+# exports it as DEFAULT_CODE_QUALITY_CRITERIA.
+#
+# WHAT IT GATES: for a subject whose medium is `code`, "converged" = the subject's OWN
+# test suite green (deterministic) AND this rubric PASSes. Neither alone converges.
+# Prompt/markdown subjects never touch this file — they converge on evals alone.
+#
+# AGGREGATION (severity-gated): a `fail` OR `uncertain` on any `critical`/`high`
+# criterion BLOCKS the quality-pass. `medium` is advisory (recorded, never blocking).
+#
+# EDITING RULES:
+#   - `id` is a byte-identity key + report anchor. Changing an id breaks historical
+#     comparison — add a new criterion instead of renaming an old one.
+#   - Every criterion must be BINARY (decidable pass/fail), never a Likert score.
+#   - `severity` must be exactly one of: critical | high | medium.
+#   - Keep passDefinition/failDefinition CONCRETE — the judge must not have to guess.
+#   - Bump `version` on any change to this file.
+#
+# STATUS: the picks below are the conservative, subject-agnostic starting set
+# (FLAGGED-for-operator-sign-off, Wave-2 W2I1) — a default, not a final contract.
+
+version: 1
+
+criteria:
+  - id: Q1-spec-faithful
+    statement: The implementation realizes the agentspec definition the amend targeted.
+    passDefinition: >-
+      PASS: every spec-declared tool / job-to-be-done is present and the operative
+      system_prompt is unchanged except where the amend intended; the def→impl cascade held.
+    failDefinition: >-
+      FAIL: a spec-declared tool or JTBD is missing, or the system_prompt was altered
+      beyond the amend intent (the impl forked from the SSoT).
+    severity: critical
+
+  - id: Q2-addresses-diagnosis
+    statement: The change targets the diagnosed root-cause locus, not an adjacent symptom.
+    passDefinition: >-
+      PASS: the edit is at (or directly causes a fix at) the RCA-named locus / the
+      failing-case root.
+    failDefinition: >-
+      FAIL: the change edits an unrelated place or patches a symptom while the root
+      cause remains.
+    severity: high
+
+  - id: Q3-no-regression
+    statement: The change introduces no unrelated regression and stays in scope.
+    passDefinition: >-
+      PASS: only the remedy-named surface changed; no unrelated behavior was removed
+      or broken.
+    failDefinition: >-
+      FAIL: unrelated code was changed/removed, or a previously-working path was broken.
+    severity: high
+
+  - id: Q4-error-handling
+    statement: Failure paths are handled, not silently dropped.
+    passDefinition: >-
+      PASS: error/failure branches surface, recover, or log per the subject's contract;
+      no new silent drop.
+    failDefinition: >-
+      FAIL: a failure is swallowed with no recovery/notify/observability (a silent-drop
+      regression).
+    severity: high
+
+  - id: Q5-maintainable
+    statement: The code is readable, framework-idiomatic, and free of obvious footguns.
+    passDefinition: >-
+      PASS: clear structure, idiomatic to the target framework, no dead code /
+      copy-paste / obvious footgun introduced.
+    failDefinition: >-
+      FAIL: unreadable/duplicated/dead code or an obvious maintainability footgun was
+      introduced.
+    severity: medium

package/.claude/skills/mutagent-evaluator/lenses/methodology-critic-lens.md

@@ -44,7 +44,7 @@ findings:
   - mr: MR-7
     verdict: <fit | mis-fit | inefficient>
     observation: <what the run actually did at this methodology decision>
-    recommendation: <concrete rearrange/improve proposal>
+    recommendation: <concrete rearrange/optimize proposal>
     impactRank: <1=highest>          # ranked self-feedback (MR-5)
     advisory: true                    # NEVER gates the run
 ```

package/.claude/skills/mutagent-evaluator/references/edd-loop.md

@@ -1,11 +1,11 @@
-# edd-loop — the ADL ③ IMPROVE / Eval-Driven-Development loop (F18 + F19)
+# edd-loop — the ADL ③ OPTIMIZE / Eval-Driven-Development loop (F18 + F19)
 
-> **Loaded by:** the evaluator `improve` mode (`assets/agents/evaluator.md#mode-improve`). Load on demand.
+> **Loaded by:** the evaluator `optimize` mode (`assets/agents/evaluator.md#mode-optimize`). Load on demand.
 > **Backed by:** `scripts/edd/variance-gate.ts` (F19) · `scripts/edd/change-request.ts` (F18) ·
 > `scripts/edd/edd-types.ts` + `schemas/edd-change-request.schema.yaml` (the contract).
 > **Sibling refs:** `grounded-adjudication.md` (every ask is grounded) · `validate-evaluator.md` (trust the judge first).
 
-The IMPROVE stage closes the **PR-011 spec↔impl↔eval triad**. A judge that only reports is half a
+The OPTIMIZE stage closes the **PR-011 spec↔impl↔eval triad**. A judge that only reports is half a
 loop; EDD drives the subject to **full green** while the evaluator stays **judge-only (EV-051)** — it
 **REQUESTS** the `ai-engineer` to amend the Agent/AgentSpec and **re-evals**, but never
 patches the subject itself.
@@ -100,11 +100,11 @@ controller never reads a clock) → the decision is PURE + deterministic (C-PIN)
 ## Assumed eval-runner interface (the clean seam — REBUILD nothing)
 
 The EVAL engine (dataset build · the eval runner · Path A/B) is built **in parallel** (sibling
-worktree). The `improve` mode is **additive**: it orchestrates the LOOP around the runner and assumes
+worktree). The `optimize` mode is **additive**: it orchestrates the LOOP around the runner and assumes
 this interface — it does not re-implement it.
 
 ```
-// the runner the improve mode CONSUMES (owned by the eval engine, e.g. run-evaluate.ts):
+// the runner the optimize mode CONSUMES (owned by the eval engine, e.g. run-evaluate.ts):
 runOnce(caseIds: string[]) -> Array<{
   caseId: string;
   criterionId: string;
@@ -116,7 +116,7 @@ runOnce(caseIds: string[]) -> Array<{
 - **repeat-N** = `runOnce` called N times over the SAME `caseIds`; the per-rerun verdicts feed
   `CaseVarianceObservation.verdicts[]` (+ `trajectories[]`) → `evaluateVarianceGate`.
 - **accuracy** = `runOnce` over the FULL dataset → per-case verdicts → an accuracy ratio vs target.
-- the runner is **C-PIN** (pinned model + temperature 0). The improve mode adds **no** provider call.
+- the runner is **C-PIN** (pinned model + temperature 0). The optimize mode adds **no** provider call.
 
 If the sibling runner exposes a different shape, adapt at the seam (map its result into
 `CaseVarianceObservation` / the accuracy ratio) — keep the gate + loop modules untouched. The gate

package/.claude/skills/mutagent-evaluator/references/eval-stage.md

@@ -38,8 +38,37 @@ After `*build`, OFFER the two derivations (AskUserQuestion, parent session only)
 1. **`*build-dataset`** — materialize + grow the golden dataset from the spec.
 2. **`*build-evals`** — build the eval suite (and pick the engine).
 
+**State what the stage will produce, up front (EVAL-1).** Before running either build,
+EMIT the "what this stage will produce" notice (`renderStageProducesNotice`, `scripts/
+render-build-cards.ts`) naming the deliverables — a **dataset** (golden cases + the sink
+file) and/or the **eval criteria** (the judges/code-checks + their sink) — so the operator
+knows what's coming. Notification only — **no approval gate** (operator constraint): once a
+build is triggered, run it; don't ask for edit-approval, just present clearly.
+
 Let the user run either/both. They are independent; `*evaluate` consumes both.
 
+**Either order — no hard sequence (EVAL-4).** There is NO build-time ordering rule between
+the dataset and the criteria: you may build the **criteria first** and derive a dataset to
+exercise them (when you can define "good" up front), or build a **dataset first** and mine
+criteria from it (when you bring examples of good/bad and fit criteria to them). Start from
+either end. The ONLY directional constraint is a RUN-TIME prerequisite, not a build order:
+`*evaluate` cannot run against no dataset, so it auto-routes to `*build-dataset` first (below).
+Do not mistake that run-time prerequisite for a build-time sequence.
+
+**The three dataset pipelines — kept SEPARATE (EVAL-2).** Do NOT fuse the source paths into
+one step (on a cold start there are no traces to ground against). They stay distinct:
+1. **cold-from-definition** — materialize the dataset from the agentspec (`materialize-dataset.ts`
+   `materializeFromAgentspec`). The only path with no traces.
+2. **from-labeled-traces** — grow the set from already-labeled traces
+   (`derive-dataset.ts` `deriveRegressionCases` / `growRegressionSet`).
+3. **seed-then-ground** — seed from the definition, THEN ground from traces (definition seeds →
+   traces ground). Reuses the shared monotonic merge floor (`build-dataset.ts` — nothing is ever
+   dropped), so the paths can be combined later without losing cases.
+
+**Precedence — scored traces outrank a stale spec (EVAL-2).** When BOTH a spec and traces
+carrying feedback/scores exist for the same case, the **scored traces WIN**: real observed
+outcomes outrank a possibly-stale definition. The spec seeds; the scored traces correct.
+
 **Dataset gate (dogfood H5).** `*evaluate` REQUIRES a materialized dataset. If `*build-evals` runs
 (or `*evaluate` is invoked) with no dataset present, do NOT dead-end — auto-route to `*build-dataset`
 (interactive) first, then resume. Never leave the user with an eval suite and nothing to run it on.
@@ -126,9 +155,13 @@ Stream PROGRESS as wireframe cards for BOTH `*build-dataset` and `*build-evals`
 
 - `renderBuildDatasetProgressCard` / `renderBuildEvalsProgressCard` — phase +
   progress bar + counters, emitted per phase (F13/F16).
-- After each, emit the VERBOSE entity card (F22): `renderDatasetEntityCard`
-  (per-category breakdown · version · sink) · `renderEvalsEntityCard` (engine ·
-  SURFACED CC/log-sink dependency · criteria).
+- **REQUIRED emission (EVAL-1).** After each build, you MUST emit the VERBOSE entity
+  card (F22) — this is not optional: `renderDatasetEntityCard` (per-category breakdown ·
+  version · sink) · `renderEvalsEntityCard` (engine · SURFACED CC/log-sink dependency ·
+  criteria). A build that reports and a build that stays silent must NOT look identical
+  (the dogfood symptom). The renderers WARN loudly (`⚠ EMPTY`) when a build produced no
+  items/criteria, so a silent no-op build can never render as a success. No approval
+  branch — notification only.
 
 ---
 

package/.claude/skills/mutagent-evaluator/references/memory-format.md

@@ -21,7 +21,7 @@ name: <short-kebab-case-slug>
 description: <one-line summary — used to decide relevance during recall>
 metadata:
   type: user | feedback | project | reference    # Claude Code's four types
-  lifecycle: spec|build|evaluate|diagnose|improve|general   # ADL extension
+  lifecycle: spec|build|evaluate|diagnose|optimize|general   # ADL extension
   created: YYYY-MM-DD                             # REQUIRED · every memory is DATED (stale-pruning)
 ---
 

package/.claude/skills/mutagent-evaluator/references/operation-inventory.md

@@ -76,6 +76,7 @@ makes NO pass/fail decision; the LLM/HITL halves are Type B (below).
 | `*build-dataset` expand (EV-046) | `scripts/build-dataset.ts` | `buildCase` · cartesian dimension×value expand · deterministic token-Jaccard near-dup removal · content-derived id · `mergeCases` MONOTONIC merge. The agent PROPOSES realism; the script ENFORCES non-redundancy. Code half of the 3-way `*build-dataset` Hybrid. |
 | `*discover-dataset` distill (EV-047) | `scripts/derive-dataset.ts` | Distill a living regression set from past ✓/✗ — reuses EV-052 `sample-traces.ts` selectors (failure/outlier/uncertainty) + EV-046 monotonic merge; trace→`DatasetCase` whose query IS the real input. Labels already on traces (from `*discover-evals`); makes NO new decision. Type A Code-only. |
 | Living-suite writer (EV-053) | `scripts/living-suite.ts` | `appendOnly` + `assertMonotonicGrowth` — generic-over-`T` append-only writer (a living artifact NEVER shrinks) shared by datasets + criteria. Pure-counter provenance (no clock → byte-identity, C-PIN). Pure set algebra w/ fail-loud monotonicity guard. |
+| Eval-leg reconcile contract (W2I5 · KP-003) | `scripts/sync-eval-criteria.ts` | The eval leg of `#sync-spec` (def → impl → eval triad). `evalLegForSubjectKind` (agent/skill → eval-suite · code → code-quality) · `flagEvalLegDrift`/`assessEvalLeg` (pure freshness flag) · `reconcileEvalCriteria` (upsert-by-id criteria MAINTENANCE via `assertMonotonicGrowth` — never drops, EV-053). Criteria maintenance, NOT judging (EV-051): scores nothing, decides no pass/fail. The reconcile REASONING is `ai-architect #sync-spec`'s (Model-B); this holds only the deterministic contract. Pure (no clock/random/network, C-PIN). |
 | W2 dataset contract | `scripts/contracts/dataset.ts` | TypeBox `Dimension`/`DatasetTuple`/`DatasetCase`/`Dataset` (companion to `schemas/dataset.schema.yaml`) + `tupleKey`. NEW W2-OWN file — does NOT touch shared `contracts/eval-types.ts`. Data contract only. |
 | W2 trust contract | `scripts/contracts/validation.ts` | TypeBox `HumanLabel` (produced by `*review`, consumed by `*validate`) + the confusion-matrix / validation-result shapes. NEW W2-OWN file; disjoint from shared types. Data contract only. |
 

package/.claude/skills/mutagent-evaluator/schemas/methodology-review.schema.yaml

@@ -5,7 +5,7 @@
 # This is the SECOND requirement family: FITNESS, not conformance. It does not
 # pass/fail a component; it interrogates the methodology's own decision-tree,
 # workflow structure, and data-flow, and emits ranked recommendations to
-# rearrange / improve the process. MR-1..9 (Tab-4 lens reads these).
+# rearrange / optimize the process. MR-1..9 (Tab-4 lens reads these).
 # ============================================================================
 $id: mutagent-evaluator/methodology-review
 title: Subject Methodology-Review Rubric