@mutagent/evaluator 0.1.0-alpha.3 → 0.1.0-alpha.5

package/.claude/skills/mutagent-evaluator/SKILL.md

@@ -12,15 +12,18 @@ description: |
   framework-substrate choice (DEFAULT agent-dispatch — host-runtime subagents, no provider ·
   in-house AI-SDK judge optional · code-based · user's framework export).
   Subsequent invocations dispatch the parent session + MASS-PARALLEL evaluator
-  sub-agents (one cell, discover / judge modes; host runtime, harness-capped) against the subject's traces. ALSO surfaces the existing v1 4-tab static-auditor
-  (eval-matrix · data-leak · variance · methodology) via *audit. Pinned judge model + temp=0
+  sub-agents (one cell, discover / judge modes; host runtime, harness-capped) against the subject's traces.
+  DEFAULT INTENT: any "evaluate this skill/agent" request runs *evaluate — the JUDGE — which is what an
+  end user wants. *audit is a SEPARATE, explicit-only 4-tab static-auditor (eval-matrix · data-leak ·
+  variance · methodology) used mainly for complex agents/skills that emit HTML artifacts; it is NEVER the
+  fallback for a plain evaluate. Pinned judge model + temp=0
   for byte-identical reruns (C-PIN). Usable standalone OR borrowed by mutagent-skill-builder's
   SIMULATE phase as the SimVerifier.
 license: Proprietary. LICENSE.txt has complete terms.
 compatibility: Designed for Claude Code, Codex, Cursor, OpenCode and similar coding-agent runtimes; works with git, gh CLI, jq, curl, and Bun/pnpm/npm runtimes.
 metadata:
   author: mutagent
-  version: "0.1.0-alpha.2"
+  version: "0.1.0-alpha.4"
 # allowed-tools: OMITTED — agent uses all native tools per host runtime
 ---
 
@@ -126,9 +129,15 @@ When you encounter a *<name> token:
 
 ## §1 — Triggers
 
+> **DEFAULT INTENT = `*evaluate` (the Judge).** "evaluate this skill/agent", "score/judge this",
+> a bare `/mutagent-evaluator`, or any ambiguous evaluate ask → **`*evaluate`**. **Never** fall back to
+> `*audit`. `*audit` (the 4-tab master-audit) is a SPECIALIZED, **explicit-only** surface — invoked by
+> name (`*audit <subject>`), typically for complex agents/skills that emit HTML artifacts. If unsure
+> between the two, choose `*evaluate`.
+
 Invoke me with:
-- `evaluate skill <name>` / `evaluate agent <name>` / `/mutagent-evaluator`
-- `*discover-evals` (mine criteria) · `build evals for <x>` · `validate judge <x>` · `*audit <subject>`
+- `evaluate skill <name>` / `evaluate agent <name>` / `/mutagent-evaluator`  → **`*evaluate`** (default)
+- `*discover-evals` (mine criteria) · `build evals for <x>` · `validate judge <x>` · `*audit <subject>` (explicit only)
 - `*improve` (the ADL ③ EDD loop — variance-first then request-amend-reeval to full green, bounded)
 - `*self-audit` (the eval-of-the-eval — audit my OWN eval-dev; on-demand only)
 - borrowed by `mutagent-skill-builder` SIMULATE (SimVerifier role)
@@ -147,7 +156,9 @@ Minimal eval-dev spine (build order — do NOT reorder):
                 amend agentspec|impl over SendMessage → re-eval) → full green OR bounded STOP
 ```
 
-`*validate` (calibrate the judge) and `*audit` (the v1 4-tab static report) layer on top.
+`*validate` (calibrate the judge) layers on top. `*audit` (the v1 4-tab master-audit) is an
+**explicit-only** specialized surface (complex / HTML-artifact subjects) — NOT part of the default
+evaluate spine and never an automatic fallback.
 `*improve` is the IMPROVE stage — it runs AFTER an initial `*build` + `*evaluate` and is **bounded**
 (never infinite): full-green ⇒ DONE; else max-swings / max-wallclock / no-improvement ⇒ STOP + delta.
 
@@ -167,7 +178,7 @@ log-sink dependency up front, F7/F9/F14), streams wireframe progress + entity ca
 - **Curated Discovery handoff (P-B)** — when the traces arrive from the `discovery` SYSTEM AGENT under a PURPOSED intent (`*discover-evals` → `operationIntent: evals` · `*discover-dataset` → `operationIntent: dataset`), the handover is a **curated, classified** set: a `SelectionManifest` (`mutagent-tools` `selection.ts` — `byScenario` · `byWorthiness` · labeled/unlabeled counts) alongside `traces.jsonl` where each kept record carries `ext.signals` (deterministic worthiness from `trace select`) + `ext.classification` (`scenario` · `edgeCase` · `outcome` · `worthiness`, from the agent's LLM classify). BOTH evaluator commands READ this handoff (skills never fetch) and consume the labels: **`*discover-evals`** STRATIFIES criteria-mining by `ext.classification.scenario` and PRIORITIZES by `ext.signals.worthinessScore` (unlabeled/low-worthiness traces are kept for coverage but down-weighted); **`*discover-dataset`** distills a **scenario-balanced** regression set, **edge-cases first** (`ext.classification.edgeCase.is`), reusing the `derive-dataset.ts` selectors + `build-dataset.ts` merge. A generic `fetch` handoff (no `ext.signals`/`ext.classification`) degrades gracefully to the pre-P-B behavior. The judge/mining internals are unchanged — only the sampling/stratification READS the new curated fields.
 - **Code-before-judge (EX-2)** — the §5b `check_method` routes each metric: `deterministic`→a runnable code-eval the agent executes via Bash (zero judge tokens, byte-identical), `llm-judge`→the host judge, `hybrid`→code pre-filter that gates the judge (`check-method-router.ts` + `code-eval.ts`).
 - **`*evaluate`** — the real spine (`run-evaluate.ts`): **T1 tier-0** code-method pre-pass (`tier0Plan` — code rows decided deterministically, only residual judge rows dispatched) → PREP packet (**T5 `adaptivePacketPlan`** overload guard, default 1:1) → dispatch `#mode-judge-trajectory` (emits the **T2 DAG v2 walk**) → fail-loud readiness gate → fold code+judge → **T3 independent verifier** (downgrade-only on gating fails) → **T4 `consolidateByLocus` + `deriveWalkHealth`** → GATE + variance → EV-051 route-to-diagnostics; the masked scorecard is byte-identical across reruns (C-PIN).
-- **Report** — terminal eval-cards + the operator-APPROVED 5-tab HTML eval-report (`render-eval-report.ts`): §1 Overview (KPIs · coverage-contract · gating table · top-findings) · §2 Trajectory‖Judge per-trace ledger + click-row Target-Agent‖Judge side-by-side · §3 Eval Scorecard cohort heatmap + nested subcards + inline calibration · §4 Findings verbatim-evidence + judge chain + agree/revise/refute · §5 Self-Eval [INTERNAL, stripped on publish]. `run-evaluate.ts` renders `report.html` post-aggregate; consumes the §9.4 judge-walk (dense na-explicit map + `judge_steps[]`) when present, degrades to the per-trajectory scorecard otherwise.
+- **Report** — terminal eval-cards + the operator-APPROVED 5-tab HTML eval-report (`render-eval-report.ts`): §1 Overview (KPIs · coverage-contract · gating table · top-findings) · §2 Trajectory‖Judge per-trace ledger + click-row Target-Agent‖Judge side-by-side · §3 Eval Scorecard cohort heatmap + nested subcards + inline calibration · §4 Findings verbatim-evidence + judge chain + agree/revise/refute · §5 Self-Eval [INTERNAL, stripped on publish]. `run-evaluate.ts` renders `evaluation-report.html` post-aggregate (client-audience by default; §5 shown only for `--audience internal`); consumes the §9.4 judge-walk (dense na-explicit map + `judge_steps[]`) when present, degrades to the per-trajectory scorecard otherwise.
 
 ## §3 — Architecture Overview
 
@@ -265,6 +276,7 @@ applies, EV-053/EQ5):
 | `scripts/aggregate-discover.ts` | P2 (§5a) + §9.4 (T6) | code | The real `*discover-evals` AGGREGATE half: parse the leaf's per-trace verdict files + per-batch mining report → `TraceAnnotation[]` (FAIL-LOUD via `missingVerdictKeys`; THROW on an undispatched-trace category ref) → `deriveMinedCriteria` (§5b metadata + §5c DR-2) → `growLivingSuite` (append-only, monotonic, saturation-stop). **T6:** `collectDatasetCandidates` (failure/uncertain → `DatasetCase[]` via the derive-dataset selectors) + `unmatchedDetectionCandidates` (the `*evaluate` judge's node-2.5 handoff) → consumable by `*build-dataset`. `DiscoverMiningReport` contract. No hand-rolled annotations. |
 | `scripts/code-eval.ts` | P3 (EX-2) | code | The v2 CODE-track primitive library — generic PURE `(trace) → {result, detail}` checks: presence · string-equality · format-validity · schema-conformance · ref-integrity (cross-stage). SV-1 dotted-path read incl. `obs:<name>.<path>`. Exhaustive `runCodeEval` (unknown primitive → THROW). NO LLM, byte-identical. Disjoint from the v1 `run-deterministic.ts` audit world. |
 | `scripts/check-method-router.ts` | P3 (EX-2) | hybrid | THE load-bearing §5b `check_method` router: `deterministic`→CODE-EXEC (judge NEVER called) · `llm-judge`→JUDGE seam · `hybrid`→code pre-filter that GATES the judge. Unknown → THROW (exhaustive). `RoutedCriterionVerdict` = `CriterionVerdict` + provenance `{producedBy, codeResult, judgeReached}` (structural superset → `evaluate.ts` rollup unchanged). `extractCodeEval(registry)` fail-loud (a code-class criterion with no spec THROWS — never silently judge). "Code before judge" lives INSIDE the agent. |
+| `scripts/code-quality-verdict.ts` | W2I1 (⑤ IMPROVE) | hybrid | The **code-quality JUDGE mode** (`#mode-judge-code-quality`) — the (b) half of the ⑤ IMPROVE code-target BOTH-gate. Scores a `code` subject's QUALITY (spec-faithfulness · addresses-diagnosis · no-regression · error-handling · maintainability) BEYOND tests-green → severity-gated binary PASS/FAIL. `buildCodeQualityPrompt` (pure renderer, critique-before-verdict) + `codeQualityGatePass` (pure aggregate) + `runCodeQualityJudge` (C-PIN run wrapper on the injected JudgeInvoke seam). Judge-only (EV-051), host-runtime, pinned model + temp 0. **DISTINCT from `code-eval.ts`** (deterministic primitives — no LLM). `DEFAULT_CODE_QUALITY_CRITERIA` + its aggregation rule are FLAGGED-for-operator-sign-off. |
 | `scripts/run-evaluate.ts` | P5 + §9.4 (T1·T3·T4·T5) | hybrid | The real-engine `*evaluate` end-to-end COMPOSER: `tier0Plan` (T1 — code rows decided in the pre-pass, residual judge rows only in packets) → `prepMatrixPackets` (residual-only) → `adaptivePacketPlan` (T5 overload guard, default 1:1) → [dispatch #mode-judge-trajectory] → `aggregateEvaluate` (FAIL-LOUD readiness gate over DISPATCHED ids → `aggregateMatrixScorecard` folding code+judge + T3 `independentVerify` downgrade-only on gating fails → GATE + variance → T4 `consolidateByLocus` lociClusters → EV-051 `routeFailures`→validated HandoverBundle). `maskedScorecard` = the C-PIN byte-identity artifact. |
 | `scripts/render-eval-report.ts` | P4 (D-1/2/3) | code | The v2 reporting (v1 `render-report.ts` untouched): `renderEvalCards` (D-1 terminal per-criterion cards) + `buildEvalReportInput` (derives the rich tabs from real run data — ledger · coverage · cohorts · gating rolls · top-findings · judge-health) + `renderEvalReport` (the operator-APPROVED 5 tabs: ① Overview KPIs+coverage+gating-table+top-findings · ② Trajectory‖Judge ledger + click-row Target-Agent‖Judge side-by-side · ③ Eval Scorecard cohort heatmap + nested subcards + inline calibration · ④ Findings verbatim-evidence + judge chain + agree/revise/refute · ⑤ Self-Eval [INTERNAL, strip-marked]). Consumes the §9.4 judge-walk (`judge_steps[]` + dense map) when present, degrades to the per-trajectory scorecard otherwise. Tabs are `<button>` (no `<s>` strikethrough). Unified design-system tokens (`assets/brand/theme.css`). byte-identical after the `generatedAt` mask. `autoOpenCommand` cross-platform helper (orchestrator fires it). |
 | `scripts/contracts/eval-types.ts` *(extended)* | P2/P2b | code | + `MinedCriterion` (additive subtype of `DiscoveredCriterion`) carrying the §5b `MetricMetadata` (generality·dimension·level·check_method [3-value SF-2 router]·substrate·applies_to·severity·judge_inputs·flag) + the §5c `DiscoveryRationale` (DR-2: targets·why·evidence{grounding observed\|inferred·prevalence k/n·refs}·assumptions·expected_impact). `parseMinedCriterion` enforces the evidence-first gate (OBSERVED ⇒ refs + k>0). |

package/.claude/skills/mutagent-evaluator/assets/agents/discovery.md

@@ -18,7 +18,7 @@ ACTIVATION-NOTICE: This file contains your full agent operating guidelines. Read
 class: pure_subagent_executor
 isolation: none
 
-# STATUS: SCOPED / FUTURE — not dispatched today.
+# STATUS: DISPATCHED standalone as the vendored delegate-or-artifact trace agent (diagnostics Step 3.6 · evaluator Step 0.5). The orchestrator `*collect-traces` alias route remains future.
 # Helix currently calls `mutagent-cli trace count` + `trace fetch --export` DIRECTLY (the
 # example workflows in trace-layer-plan.md §Example Workflows). This agent FORMALIZES that
 # pattern into a delegatable cell once the CLI surface is proven in production. The contracts
@@ -359,7 +359,7 @@ You are a **discovery** agent. You collect agent traces on behalf of Helix (or a
 orchestrator) and hand the result to a downstream skill (Diagnostics or Evaluator). You do
 NOT analyze traces — you only **collect, normalize, and validate** them.
 
-> **SCOPED / FUTURE.** This agent contract is written now so the contracts are locked and
+> **STATUS.** The vendored agent IS dispatched standalone today (diagnostics Step 3.6 · evaluator Step 0.5) as the delegate-or-artifact trace agent; only the `*collect-traces` orchestrator route remains future. The contract below was written ahead so it was locked and
 > the handoff chain is fully specified. Helix currently calls `mutagent-cli` directly (the
 > example workflows in `trace-layer-plan.md`). This agent formalizes that into a delegatable
 > cell once the CLI surface is proven in production. Dispatch wiring (`*collect-traces` route

package/.claude/skills/mutagent-evaluator/assets/agents/evaluator.md

@@ -508,6 +508,7 @@ resolution_contract: |
         mode: judge, axis: criterion    => run #mode-judge-criterion (preserves eval-judge);   input is a .task.json.
         mode: verify                    => run #mode-verify (GA-5 result-verifier; reviewer≠judge, downgrade-only); input is a decided verdict + its situation.
         mode: improve                   => run #mode-improve (EDD ③ loop — F19 variance-first then F18 request-amend-reeval, bounded). JUDGE-ONLY: REQUEST, never patch.
+        mode: judge, axis: code-quality => run #mode-judge-code-quality (Wave-2 W2I1 — score a CODE subject's QUALITY for the ⑤ IMPROVE code-target BOTH-gate). JUDGE-ONLY; binary quality PASS/FAIL. Rubric mirrored in scripts/code-quality-verdict.ts.
    4. DISPATCH-INPUT DISAMBIGUATION — the parent hands you files:
         a *.packet.json (MatrixPacket)  => judge/trajectory, ALWAYS (matrix-judge path; no unit field).
         a *.task.json  (JudgeTaskSpec)  => unit.kind: "discover" => discover; else unit.axis: "criterion" => judge/criterion.
@@ -870,7 +871,7 @@ this inline section is now its canonical home.
    the FAILURE + UNCERTAIN traces into DATASET CANDIDATES (`collectDatasetCandidates`, reusing the
    derive-dataset selectors) and folds in the `*evaluate` judge's node-2.5 unmatched-detection
    handoff — both consumable by `*build-dataset`. You mine + flag; the parent emits the candidates.
-7. **Render the discover `report.html` via the SHIPPED `writeDiscoverRunReport` composer**
+7. **Render the discover `discovery-report.html` via the SHIPPED `writeDiscoverRunReport` composer**
    (`render-discover-report.ts`) — pass the run dir; it builds `triage-summary.json` from
    `triage.json` + wires every companion (verdicts → Proof-of-work, dataset-candidates → Dataset,
    triage census → coverage funnel, profile → entity hero) so the report is COMPLETE BY DEFAULT.
@@ -1029,6 +1030,69 @@ never re-implements, the runner. The interface this mode ASSUMES (documented ver
   C-PIN (pinned model + temperature 0).
 - **Additive to the runner:** consumes the assumed eval-runner interface; rebuilds nothing.
 
+## Mode: judge — axis code-quality   ← the ⑤ IMPROVE code-target quality verdict (Wave-2 W2I1)
+
+The **(b) half of the ⑤ IMPROVE loop's code-target BOTH-gate.** For a `code`-kind subject, "converged"
+requires BOTH (a) the subject's OWN test suite green — a hard DETERMINISTIC gate the ai-engineer
+reports and the optimize oracle records as `testsGreen` — AND (b) THIS code-quality verdict passing.
+This mode is (b): score the QUALITY of the applied code change, BEYOND tests-green (a deterministic
+primitive cannot decide spec-faithfulness, scope, error-handling, maintainability — a judge must).
+
+### What it does (per dispatch)
+
+Judge the CODE subject (the amend diff / scaffold against the agentspec definition it must realize)
+on the DRAFT criteria set — **critique-BEFORE-verdict, strictly BINARY** per criterion — and emit a
+severity-gated binary quality **PASS/FAIL**. The authoritative rubric + the drafted default criteria
+live in `scripts/code-quality-verdict.ts` (`DEFAULT_CODE_QUALITY_CRITERIA` — Q1 spec-faithful [crit] ·
+Q2 addresses-diagnosis [high] · Q3 no-regression [high] · Q4 error-handling [high] · Q5 maintainable
+[med]); this mode is that rubric's dispatched form (the provider-callable mirror is the same file,
+mirroring how the other judge prompts mirror their agent modes).
+
+> **⚠️ FLAGGED-FOR-OPERATOR-SIGN-OFF.** The exact acceptance criteria + the severity-gated aggregation
+> rule (CRIT/HIGH `fail`|`uncertain` blocks; MEDIUM advisory) are a DRAFT default, implemented behind
+> a clearly-named constant so the operator can confirm/adjust WITHOUT a loop code change. Do not treat
+> the current set as final.
+
+### Boundaries
+
+- **Judge-only (EV-051):** score the quality; NEVER propose or write a fix (the ⑤ loop's ai-engineer
+  amends). A FAIL just keeps the loop iterating.
+- **Host-runtime, no provider key; C-PIN:** pinned model + temperature 0 — byte-identical reruns.
+- **Separate from tests-green:** you judge quality, NOT whether tests pass; the deterministic test
+  gate is ai-engineer's report, ANDed by the oracle — never fold it into your quality verdict.
+
+## Eval-leg reconcile — criteria MAINTENANCE, not judging (Wave-2 W2I5 · KP-003)
+
+The evaluator owns the **eval leg** of the def → impl → eval triad — the third leg of `#sync-spec`. When
+an impl amends (the ⑤ IMPROVE loop's ai-engineer, or brownfield drift), the eval criteria that ground the
+subject's evaluation can go stale. The reconcile REASONING — which criteria a changed impl makes stale,
+what new criterion it needs — is the **ai-architect's** (`#sync-spec`, session-dispatched, Model-B). The
+evaluator exposes only the deterministic CONTRACT the reconcile reads/writes, in
+`scripts/sync-eval-criteria.ts`:
+
+- `evalLegForSubjectKind(kind)` — the shape resolver: agent/skill/composite → the **eval-suite criteria**
+  (this cell's discovered/maintained criteria); code → the **code-quality criteria**
+  (`DEFAULT_CODE_QUALITY_CRITERIA` / `#mode-judge-code-quality`, Wave-2 W2I1).
+- `flagEvalLegDrift(...)` / `assessEvalLeg(...)` — a pure freshness FLAG (mirrors the builder probe).
+- `reconcileEvalCriteria(request)` — the COMPUTE: upsert the ai-architect-proposed criteria delta by id,
+  append the novel, and NEVER drop a criterion (EV-053, via `assertMonotonicGrowth`). It returns the grown
+  set as DATA (`result.criteria`) — it does **not** touch disk; the reconcile hook COMPUTES, it is not the write.
+- `persistEvalCriteria(result, { cwd, updatedAt? })` (`scripts/persist-eval-criteria.ts`) — the WRITE: the
+  evaluator session PERSISTS `result.criteria` back to the located criteria artifact
+  (`.mutagent/evaluator/living-suite/<leg>.criteria.json`, per-leg — eval-suite for an agent/skill/composite
+  subject, code-quality for a code subject). That write is what BUMPS the artifact's freshness (its mtime — and
+  the optional `updatedAt` marker the builder probe reads), which returns the eval leg to `in-sync`: the loop
+  only closes once the reconciled criteria are on disk, not when they are merely computed.
+
+> **Judge-only PRESERVED (EV-051).** This is criteria MAINTENANCE — growing/revising the criteria SET to
+> match a changed impl — NOT judging a subject. `sync-eval-criteria.ts` scores nothing and decides no
+> pass/fail; it is pure set algebra + a freshness flag (no clock/random/network — C-PIN byte-identical). The
+> `persist-eval-criteria.ts` WRITER is the IO layer (it is the one place that touches disk, mirroring how
+> `artifact-paths.ts` does IO while `living-suite.ts` stays pure); its serialization is still deterministic
+> (stable key order, no clock unless the caller passes `updatedAt`), so a persisted artifact round-trips
+> byte-faithful. The evaluator never mutates the impl and never fixes the subject; it only maintains — and now
+> persists — its own criteria.
+
 ## Boundaries (shared)
 
 judge-only (EV-051) · model-intent-sacred · critique-before-verdict · binary-not-Likert ·

package/.claude/skills/mutagent-evaluator/references/operation-inventory.md

@@ -76,6 +76,7 @@ makes NO pass/fail decision; the LLM/HITL halves are Type B (below).
 | `*build-dataset` expand (EV-046) | `scripts/build-dataset.ts` | `buildCase` · cartesian dimension×value expand · deterministic token-Jaccard near-dup removal · content-derived id · `mergeCases` MONOTONIC merge. The agent PROPOSES realism; the script ENFORCES non-redundancy. Code half of the 3-way `*build-dataset` Hybrid. |
 | `*discover-dataset` distill (EV-047) | `scripts/derive-dataset.ts` | Distill a living regression set from past ✓/✗ — reuses EV-052 `sample-traces.ts` selectors (failure/outlier/uncertainty) + EV-046 monotonic merge; trace→`DatasetCase` whose query IS the real input. Labels already on traces (from `*discover-evals`); makes NO new decision. Type A Code-only. |
 | Living-suite writer (EV-053) | `scripts/living-suite.ts` | `appendOnly` + `assertMonotonicGrowth` — generic-over-`T` append-only writer (a living artifact NEVER shrinks) shared by datasets + criteria. Pure-counter provenance (no clock → byte-identity, C-PIN). Pure set algebra w/ fail-loud monotonicity guard. |
+| Eval-leg reconcile contract (W2I5 · KP-003) | `scripts/sync-eval-criteria.ts` | The eval leg of `#sync-spec` (def → impl → eval triad). `evalLegForSubjectKind` (agent/skill → eval-suite · code → code-quality) · `flagEvalLegDrift`/`assessEvalLeg` (pure freshness flag) · `reconcileEvalCriteria` (upsert-by-id criteria MAINTENANCE via `assertMonotonicGrowth` — never drops, EV-053). Criteria maintenance, NOT judging (EV-051): scores nothing, decides no pass/fail. The reconcile REASONING is `ai-architect #sync-spec`'s (Model-B); this holds only the deterministic contract. Pure (no clock/random/network, C-PIN). |
 | W2 dataset contract | `scripts/contracts/dataset.ts` | TypeBox `Dimension`/`DatasetTuple`/`DatasetCase`/`Dataset` (companion to `schemas/dataset.schema.yaml`) + `tupleKey`. NEW W2-OWN file — does NOT touch shared `contracts/eval-types.ts`. Data contract only. |
 | W2 trust contract | `scripts/contracts/validation.ts` | TypeBox `HumanLabel` (produced by `*review`, consumed by `*validate`) + the confusion-matrix / validation-result shapes. NEW W2-OWN file; disjoint from shared types. Data contract only. |
 

package/.claude/skills/mutagent-evaluator/references/workflows/orchestrator-protocol.md

@@ -94,13 +94,26 @@ Resolve the pinned envelope ONCE for the whole run (C-PIN): `model` from
 
 ---
 
-## Step 0.5 — Trace intake (UniTF handover — the skill NEVER fetches)
-
-Traces arrive as a **pre-produced UniTF JSONL handover**: one `UnifiedTrace`
-record per line, written by `mutagent-cli trace fetch --export <path>` BEFORE this
-skill is dispatched (Helix runs the fetch in its PRE-STAGE — the evaluator owns no
-fetch/normalize code). Fetch + normalize live entirely in `mutagent-cli`; run
-`mutagent-cli trace --help` to discover the surface (count · fetch · export).
+## Step 0.5 — Trace intake (delegate-or-artifact — the evaluator ACQUIRES traces · reports-PRD R-6)
+
+The evaluator owns NO fetch/normalize code (that lives in `mutagent-cli` + the `discovery`
+agent) — but **a real user will not hand you a trace file**. So, in **parity with Diagnostics
+Step 3.6 (V3-9)**, traces are acquired by EXACTLY ONE of the 3 legal paths:
+
+| # | Path | Who fetches | When |
+|---|---|---|---|
+| a | **Helix pre-fetch / handover** | Helix (pre-stage) or a pre-produced `mutagent-cli trace fetch --export`, handed in via `--traces <handover.jsonl>` | orchestrated runs |
+| b | **`*discover` dispatch** | the `discovery` system agent | orchestrated collection |
+| c | **Standalone: artifact intake OR vendored-Discovery spawn** | a pre-staged artifact, or the vendored `discovery` sub-agent | THIS step |
+
+**Standalone flow (path c), in order:**
+1. **Handover / artifact intake first.** If `--traces <handover.jsonl>` (`.gz` ok) was handed in,
+   or a pre-staged `.mutagent/evaluator/<run>/ingest/traces.jsonl` exists → consume it; do NOT spawn Discovery.
+2. **Else DELEGATE to the vendored Discovery agent.** Spawn `assets/agents/discovery.md` (installed at
+   `.claude/agents/discovery.md`) via the Task tool (`subagent_type: "discovery"`), handing it the
+   resolved TraceQuery (source + filters + window + `--export .mutagent/evaluator/<run>/ingest/traces.jsonl`).
+   Discovery fetches + curates → a `SelectionManifest` + `traces.jsonl`; the PREP/AGGREGATE CLIs then
+   read it via `--traces` (P-B curated `ext.signals`/`ext.classification` consumed when present).
 
 The PREP/AGGREGATE CLIs (`scripts/cli/{prep,aggregate,dogfood}.ts`) take the
 handover file via `--traces <handover.jsonl>` (`.gz` also accepted) and read it

package/.claude/skills/mutagent-evaluator/scripts/aggregate-discover.ts

@@ -338,7 +338,7 @@ export function writeDiscoverReport(input: WriteDiscoverReportInput): string {
     ...(input.companions ?? {}),
   });
   mkdirSync(input.reportDir, { recursive: true });
-  const outPath = join(input.reportDir, "report.html");
+  const outPath = join(input.reportDir, "discovery-report.html");
   writeFileSync(outPath, html);
   return outPath;
 }

package/.claude/skills/mutagent-evaluator/scripts/artifact-paths.ts

@@ -91,6 +91,22 @@ export function scorecardDir(runId: string, cwd?: string): string {
 export function reportDir(runId: string, cwd?: string): string {
   return underRoot(cwd, "reports", runId);
 }
+
+// ── the three named report artifacts (evaluator-reports PRD R-1/R-3) ──────────
+// Distinct names under ONE run folder — no `report.html` collision, and *audit no longer
+// writes under `.mutagent/diagnostics/`. Single resolver = single source of truth.
+/** `*evaluate` → the 5-tab judge report. */
+export function evaluationReportPath(runId: string, cwd?: string): string {
+  return underRoot(cwd, "reports", runId, "evaluation-report.html");
+}
+/** `*discover-evals` → the mined-criteria report. */
+export function discoveryReportPath(runId: string, cwd?: string): string {
+  return underRoot(cwd, "reports", runId, "discovery-report.html");
+}
+/** `*audit` → the 4-tab master-audit (now under the evaluator root, not diagnostics). */
+export function masterAuditPath(runId: string, cwd?: string): string {
+  return underRoot(cwd, "reports", runId, "master-audit.html");
+}
 export function livingSuiteDir(cwd?: string): string {
   return underRoot(cwd, "living-suite");
 }

package/.claude/skills/mutagent-evaluator/scripts/cli/audit-run.ts

@@ -26,10 +26,11 @@ import {
   resolveProfilePaths,
 } from "../load-profile.ts";
 import { loadBundle } from "../load-bundle.ts";
-import { runDeterministic } from "../run-deterministic.ts";
+import { runDeterministic, reexecDeterministicCheck } from "../run-deterministic.ts";
 import { assembleScorecard } from "../assemble-scorecard.ts";
 import { renderReport } from "../render-report.ts";
 import { maskedCanonicalJson } from "../mask.ts";
+import { reportDir } from "../artifact-paths.ts";
 
 const HERE = dirname(fileURLToPath(import.meta.url));
 const PKG_ROOT = join(HERE, "..", "..");
@@ -106,8 +107,11 @@ export function main(argvIn?: string[]): number {
   // 2. Run-bundle
   const { bundle } = loadBundle(args.bundleDir, args.runId);
 
-  // 3. Deterministic pass
-  const det = runDeterministic(matrix, bundle);
+  // 3. Deterministic pass — WIRE the real live re-exec (Item #4): every mapped
+  // deterministic row runs an actual checkMethod-keyed check over the produced
+  // artifact instead of a presence proxy. In-cell, no sub-agents (PR-ORCH-01),
+  // judge-only (EV-051), C-PIN byte-stable.
+  const det = runDeterministic(matrix, bundle, { liveReexec: reexecDeterministicCheck });
 
   // 4. Two-track scorecard
   const scorecard = assembleScorecard({
@@ -120,10 +124,13 @@ export function main(argvIn?: string[]): number {
   // 5. Render
   const html = renderReport(scorecard);
 
-  const outDir = args.outDir ?? args.bundleDir;
-  mkdirSync(join(outDir, "audit"), { recursive: true });
-  const scPath = join(outDir, "audit", "scorecard.json");
-  const htmlPath = join(outDir, "audit", "master-audit.html");
+  // Output → the UNIFIED evaluator reports root (PRD R-3): reports/<runId>/master-audit.html
+  // (was under .mutagent/diagnostics/<runId>/audit/ — the wrong stage root). bundleDir stays the
+  // INPUT (the run-bundle being audited); the report is written to the evaluator namespace.
+  const outDir = args.outDir ?? reportDir(args.runId);
+  mkdirSync(outDir, { recursive: true });
+  const scPath = join(outDir, "master-audit.scorecard.json");
+  const htmlPath = join(outDir, "master-audit.html");
   const scJson = args.emitMasked
     ? maskedCanonicalJson(scorecard)
     : JSON.stringify(scorecard, null, 2);

package/.claude/skills/mutagent-evaluator/scripts/cli/prep.ts

@@ -16,8 +16,10 @@
  * SOURCE SELECTION (--source): when `global.sources[]` has >1 entry and none is
  * `default:true`, the run-start config surfaces a `needs-selection` state; the
  * PARENT session then ASKS the operator to pick one (AskUserQuestion / chat
- * multi-choice) and re-invokes with `--source <name>`. A single source, exactly
- * one `default:true`, or an explicit `--source` skips the prompt.
+ * multi-choice) and re-invokes with `--source <name>`. When >1 entry has exactly
+ * one `default:true`, the config surfaces `confirm-default` (F2) — the PARENT
+ * CONFIRMS the preselected default (a yes/no ask; distinct from the open pick)
+ * before it is used. A single source or an explicit `--source` skips the prompt.
  *
  * MODEL INTENT SACRED: --model is the pinned host model (temperature pinned 0,
  * C-PIN); it is carried verbatim on every task spec for the subagent to honor —
@@ -58,6 +60,20 @@ function flag(argv: string[], name: string): string | undefined {
  */
 function surfaceSourceBinding(binding: SourceBinding): void {
   switch (binding.kind) {
+    case "confirm-default": {
+      // F2 — a preselected default among many: surface a CONFIRM ask (yes/no),
+      // distinct from the open needs-selection pick. NON-fatal: the default is usable
+      // as-is on accept / non-interactive / --source.
+      const names = binding.sources.map((s) => s.name).join(", ");
+      process.stdout.write(
+        `prep: source CONFIRM-DEFAULT — ${binding.sources.length} sources, default is ` +
+          `'${binding.source.name}'. PARENT: confirm the default via the platform ask ` +
+          `mechanism (AskUserQuestion / chat multi-choice; '${binding.source.name}' PRESELECTED, ` +
+          `operator may override), or accept it non-interactively (--source <name> / CI). ` +
+          `Candidates: ${names}\n`,
+      );
+      break;
+    }
     case "needs-selection": {
       const names = binding.sources.map((s) => s.name).join(", ");
       process.stdout.write(

package/.claude/skills/mutagent-evaluator/scripts/code-quality-verdict.ts

@@ -0,0 +1,285 @@
+/**
+ * scripts/code-quality-verdict.ts — the CODE-QUALITY judge mode (Wave-2 W2I1).
+ * ---------------------------------------------------------------------------
+ * A JUDGE mode that scores a CODE subject's QUALITY (the amended/scaffolded
+ * implementation of a `code`-kind subject) and emits a binary PASS/FAIL quality
+ * verdict with a critique-before-verdict per criterion. It is the (b) half of the
+ * ⑤ IMPROVE loop's code-target BOTH-gate: "converged" for a `code` subject = the
+ * subject's OWN test suite green (a deterministic gate the session records as
+ * `testsGreen`) AND this code-quality verdict PASSes. Neither alone converges.
+ *
+ * DISTINCT FROM `code-eval.ts`. That file is the DETERMINISTIC code-track primitive
+ * library (presence/string-equality/... over an EvalTrace — no LLM, byte-identical).
+ * This file is an LLM JUDGE mode: it scores the irreducibly-subjective quality of a
+ * code implementation (spec-faithfulness, maintainability, error-handling, ...) that
+ * a deterministic primitive cannot decide.
+ *
+ * INVARIANTS (consistent with the existing evaluator cell):
+ *   - JUDGE-ONLY (EV-051). It scores; it NEVER fixes. Failures route to the ⑤ loop's
+ *     ai-engineer amend (via the loop, not from here).
+ *   - HOST-RUNTIME. Under the DEFAULT agent-dispatch substrate the authoritative
+ *     rubric lives in the evaluator agent def (`#mode-judge-code-quality`), and the
+ *     verdict comes from a dispatched subagent — this file's prompt is the
+ *     provider-callable MIRROR for the OPTIONAL in-house / export substrate (same
+ *     operator-named exception as judge-prompt-template.ts). No provider key on the
+ *     default path.
+ *   - PINNED model + temperature 0 (C-PIN). `runCodeQualityJudge` THROWS if the pin
+ *     is not (modelId present AND temperature === 0) — reruns must be byte-identical.
+ *   - PURE renderer + a thin injected-seam run wrapper. No clock, no random, no
+ *     network; the LLM call is the injected `JudgeInvoke` seam so tests drive it.
+ */
+import { parseCritiqueVerdict, type JudgeInvoke } from "./determine-outcome.ts";
+import type { JudgePin } from "./build-evals.ts";
+import type { CritiqueVerdict } from "./contracts/eval-types.ts";
+
+/** Severity of a code-quality criterion — gates the aggregate (CRIT/HIGH block). */
+export const CodeQualitySeverity = {
+  Critical: "critical",
+  High: "high",
+  Medium: "medium",
+} as const;
+export type CodeQualitySeverityValue =
+  (typeof CodeQualitySeverity)[keyof typeof CodeQualitySeverity];
+
+/** One binary code-quality criterion the judge scores. */
+export interface CodeQualityCriterion {
+  /** Stable id (byte-identity key + report anchor). */
+  id: string;
+  /** The one-line binary statement the judge decides pass/fail on. */
+  statement: string;
+  /** What a PASS looks like (concrete, so the judge is not guessing). */
+  passDefinition: string;
+  /** What a FAIL looks like. */
+  failDefinition: string;
+  /** Gating weight — a `fail`/`uncertain` on a CRIT/HIGH criterion blocks the gate. */
+  severity: CodeQualitySeverityValue;
+}
+
+/**
+ * ⚠️ FLAGGED-FOR-OPERATOR-SIGN-OFF (Wave-2 W2I1). This is the DRAFT default set of
+ * acceptance criteria that define "code quality-pass" BEYOND tests-green. It is
+ * implemented behind this clearly-named default so the operator can confirm/adjust
+ * the criteria + their severities WITHOUT a code change to the loop. The picks
+ * below are a conservative, subject-agnostic starting point — NOT a final contract.
+ *
+ * The five drafted criteria (each BINARY, critique-before-verdict):
+ *   Q1 spec-faithful  (critical) — the impl realizes the agentspec DEFINITION the
+ *                                  amend targeted; no dropped tool / JTBD / silently
+ *                                  altered system_prompt (def→impl cascade held).
+ *   Q2 addresses-diagnosis (high) — the change actually targets the diagnosed
+ *                                  root-cause locus, not an adjacent symptom.
+ *   Q3 no-regression  (high)    — no unrelated behavior removed/broken; scope stayed
+ *                                  on what the remedy named (scope discipline).
+ *   Q4 error-handling (high)    — failure paths are handled, not silently dropped
+ *                                  (recovery/observability preserved).
+ *   Q5 maintainable   (medium)  — readable, framework-idiomatic, no dead code /
+ *                                  copy-paste / obvious footgun introduced.
+ *
+ * Aggregation (also FLAGGED): the gate is severity-gated — a `fail` OR `uncertain`
+ * on any CRITICAL/HIGH criterion blocks the quality-pass; MEDIUM is advisory. The
+ * operator may instead choose strict-all-pass or a different severity map.
+ */
+export const DEFAULT_CODE_QUALITY_CRITERIA: readonly CodeQualityCriterion[] = [
+  {
+    id: "Q1-spec-faithful",
+    statement: "The implementation realizes the agentspec definition the amend targeted.",
+    passDefinition:
+      "PASS: every spec-declared tool / job-to-be-done is present and the operative system_prompt is unchanged except where the amend intended; the def→impl cascade held.",
+    failDefinition:
+      "FAIL: a spec-declared tool or JTBD is missing, or the system_prompt was altered beyond the amend intent (the impl forked from the SSoT).",
+    severity: CodeQualitySeverity.Critical,
+  },
+  {
+    id: "Q2-addresses-diagnosis",
+    statement: "The change targets the diagnosed root-cause locus, not an adjacent symptom.",
+    passDefinition:
+      "PASS: the edit is at (or directly causes a fix at) the RCA-named locus / the failing-case root.",
+    failDefinition:
+      "FAIL: the change edits an unrelated place or patches a symptom while the root cause remains.",
+    severity: CodeQualitySeverity.High,
+  },
+  {
+    id: "Q3-no-regression",
+    statement: "The change introduces no unrelated regression and stays in scope.",
+    passDefinition:
+      "PASS: only the remedy-named surface changed; no unrelated behavior was removed or broken.",
+    failDefinition:
+      "FAIL: unrelated code was changed/removed, or a previously-working path was broken.",
+    severity: CodeQualitySeverity.High,
+  },
+  {
+    id: "Q4-error-handling",
+    statement: "Failure paths are handled, not silently dropped.",
+    passDefinition:
+      "PASS: error/failure branches surface, recover, or log per the subject's contract; no new silent drop.",
+    failDefinition:
+      "FAIL: a failure is swallowed with no recovery/notify/observability (a silent-drop regression).",
+    severity: CodeQualitySeverity.High,
+  },
+  {
+    id: "Q5-maintainable",
+    statement: "The code is readable, framework-idiomatic, and free of obvious footguns.",
+    passDefinition:
+      "PASS: clear structure, idiomatic to the target framework, no dead code / copy-paste / obvious footgun introduced.",
+    failDefinition:
+      "FAIL: unreadable/duplicated/dead code or an obvious maintainability footgun was introduced.",
+    severity: CodeQualitySeverity.Medium,
+  },
+] as const;
+
+/** The code subject under quality review (minimal, subject-agnostic). */
+export interface CodeQualitySubject {
+  /** Subject id (spec_id / target name) — provenance only. */
+  subjectId: string;
+  /** The target framework/harness the impl was built into (context for the judge). */
+  framework: string;
+  /** A compact summary of the agentspec DEFINITION the impl must realize (the SSoT). */
+  specSummary: string;
+  /** The applied change under review — the amend diff, or the scaffold summary. */
+  appliedChange: string;
+  /**
+   * The diagnosed root-cause / remedy locus this change was meant to address (feeds
+   * Q2). Empty for a fresh scaffold with no prior diagnosis.
+   */
+  diagnosisLocus?: string;
+}
+
+/** One criterion's judged result (critique-before-verdict, binary). */
+export interface CodeQualityCriterionVerdict {
+  criterionId: string;
+  severity: CodeQualitySeverityValue;
+  result: CritiqueVerdict["result"]; // pass | fail | uncertain
+  critique: string;
+  confidence: number;
+}
+
+/** The aggregated code-quality verdict for one subject. */
+export interface CodeQualityVerdict {
+  subjectId: string;
+  /** The binary quality gate — the (b) half of the ⑤ code-target BOTH-gate. */
+  pass: boolean;
+  perCriterion: CodeQualityCriterionVerdict[];
+  /** C-PIN provenance stamped on the verdict (byte-identity guarantee). */
+  pinned: JudgePin;
+}
+
+/**
+ * Render the code-quality judge prompt for ONE criterion — the provider-callable
+ * mirror of the `#mode-judge-code-quality` rubric. Critique-BEFORE-verdict, strictly
+ * BINARY (no Likert), judge-only. No decision is made here.
+ */
+export function buildCodeQualityPrompt(
+  criterion: CodeQualityCriterion,
+  subject: CodeQualitySubject,
+): { system: string; user: string } {
+  const system = [
+    "You are a BINARY Pass/Fail code-QUALITY judge for ONE criterion of a CODE",
+    "subject (an agent/tooling implementation). Judge exactly this criterion and",
+    "nothing else. You are a JUDGE ONLY — you NEVER propose or write a fix.",
+    "",
+    "Quality is assessed BEYOND the subject's test suite (tests-green is a separate",
+    "deterministic gate). Judge the code as written against the criterion.",
+    "",
+    `Criterion (${criterion.id}, severity ${criterion.severity}): ${criterion.statement}`,
+    criterion.passDefinition,
+    criterion.failDefinition,
+    "",
+    "Outcomes are strictly BINARY: pass or fail (use uncertain ONLY when the change",
+    "genuinely lacks the evidence to decide). NO Likert scales, NO 1-5 / letter",
+    "grades, NO partial credit.",
+    "",
+    "Output STRICT JSON with the critique BEFORE the verdict (reason first, then",
+    "commit):",
+    '{ "critique": "<evidence-citing assessment>", "result": "pass"|"fail"|"uncertain",',
+    '  "confidence": <0..1> }',
+  ].join("\n");
+
+  const user = [
+    `Subject: ${subject.subjectId}  ·  target framework: ${subject.framework}`,
+    "",
+    "Agentspec definition the implementation must realize (the SSoT):",
+    subject.specSummary,
+    "",
+    subject.diagnosisLocus !== undefined && subject.diagnosisLocus.length > 0
+      ? `Diagnosed root-cause / remedy locus (for the addresses-diagnosis check):\n${subject.diagnosisLocus}\n`
+      : "",
+    "Applied change under review (diff or scaffold summary):",
+    subject.appliedChange,
+  ]
+    .filter((l) => l !== "")
+    .join("\n");
+
+  return { system, user };
+}
+
+/**
+ * The PURE aggregation: severity-gate the per-criterion verdicts into ONE binary
+ * quality-pass. FLAGGED default rule — a `fail` OR `uncertain` on any CRITICAL/HIGH
+ * criterion blocks the pass; MEDIUM is advisory (never blocks). Deterministic.
+ */
+export function codeQualityGatePass(
+  perCriterion: readonly CodeQualityCriterionVerdict[],
+): boolean {
+  for (const v of perCriterion) {
+    const gating =
+      v.severity === CodeQualitySeverity.Critical || v.severity === CodeQualitySeverity.High;
+    if (gating && v.result !== "pass") return false;
+  }
+  return true;
+}
+
+/**
+ * Run the code-quality judge over the criteria set under a PINNED model. THROWS if
+ * the pin is not (modelId present AND temperature === 0) — C-PIN (byte-identical
+ * reruns). Renders each criterion → the injected `JudgeInvoke` seam → parses
+ * (critique-before-verdict) → severity-gated aggregate. Under the DEFAULT
+ * agent-dispatch substrate the verdicts instead come from a dispatched
+ * `#mode-judge-code-quality` subagent (verdict files) and this wrapper is not used.
+ * Deterministic given (subject, judge, pin, criteria).
+ */
+export async function runCodeQualityJudge(
+  subject: CodeQualitySubject,
+  judge: JudgeInvoke,
+  pin: JudgePin,
+  criteria: readonly CodeQualityCriterion[] = DEFAULT_CODE_QUALITY_CRITERIA,
+): Promise<CodeQualityVerdict> {
+  if (typeof pin.modelId !== "string" || pin.modelId.length === 0) {
+    throw new Error(
+      "runCodeQualityJudge: judge is not pinned (missing modelId). MODEL INTENT IS " +
+        "SACRED — a non-pinned judge can never produce a verdict (C-PIN).",
+    );
+  }
+  if (pin.temperature !== 0) {
+    throw new Error(
+      `runCodeQualityJudge: judge temperature=${pin.temperature} (!= 0) — not pinned. ` +
+        "MODEL INTENT IS SACRED: reruns must be byte-identical (C-PIN); refusing.",
+    );
+  }
+  if (criteria.length === 0) {
+    throw new Error(
+      "runCodeQualityJudge: empty criteria set — a code-quality verdict with no " +
+        "criteria would vacuously pass (a false-green). Refusing.",
+    );
+  }
+
+  const perCriterion: CodeQualityCriterionVerdict[] = [];
+  for (const criterion of criteria) {
+    const { system, user } = buildCodeQualityPrompt(criterion, subject);
+    const raw = await judge(system, user);
+    const v = parseCritiqueVerdict(raw);
+    perCriterion.push({
+      criterionId: criterion.id,
+      severity: criterion.severity,
+      result: v.result,
+      critique: v.critique,
+      confidence: v.confidence,
+    });
+  }
+
+  return {
+    subjectId: subject.subjectId,
+    pass: codeQualityGatePass(perCriterion),
+    perCriterion,
+    pinned: { modelId: pin.modelId, temperature: pin.temperature },
+  };
+}