@mutagent/helix 0.1.0-alpha.14

package/mutagent-evaluator/.claude/skills/mutagent-evaluator/references/edd-loop.md

@@ -0,0 +1,134 @@
+# edd-loop — the ADL ③ IMPROVE / Eval-Driven-Development loop (F18 + F19)
+
+> **Loaded by:** the evaluator `improve` mode (`assets/agents/evaluator.md#mode-improve`). Load on demand.
+> **Backed by:** `scripts/edd/variance-gate.ts` (F19) · `scripts/edd/change-request.ts` (F18) ·
+> `scripts/edd/edd-types.ts` + `schemas/edd-change-request.schema.yaml` (the contract).
+> **Sibling refs:** `grounded-adjudication.md` (every ask is grounded) · `validate-evaluator.md` (trust the judge first).
+
+The IMPROVE stage closes the **PR-011 spec↔impl↔eval triad**. A judge that only reports is half a
+loop; EDD drives the subject to **full green** while the evaluator stays **judge-only (EV-051)** — it
+**REQUESTS** the `ai-engineer` to amend the Agent/AgentSpec and **re-evals**, but never
+patches the subject itself.
+
+---
+
+## The two doctrine lines
+
+> **F19 — VARIANCE FIRST.** *Stabilize per-case variance (run each case ~N times, default 5, until the
+> verdict stops flapping) BEFORE measuring accuracy over the full dataset. Without stabilizing
+> variance first, accuracy over big samples is wasted.*
+
+> **F18 — EDD CLOSURE.** *The evaluator localizes WHERE a fix belongs (agentspec vs impl), REQUESTS
+> the ai-engineer to amend it (over SendMessage, grounded), and re-evals what is amended — looping to
+> full green or a bounded STOP. The evaluator never patches; the engineer never judges.*
+
+---
+
+## F19 — the variance-first gate
+
+After the initial `*build` + `*evaluate`, the loop's FIRST phase is **variance**, not accuracy:
+
+1. **repeat-N** — run each case the SAME way N times (default `DEFAULT_REPEAT_N = 5`). Collect a
+   `CaseVarianceObservation { caseId, criterionId, verdicts[], trajectories? }` per case.
+2. **gate** — `evaluateVarianceGate(observations)`:
+   - per-case spread via the already-shipped `evalScoreVariance` (EV-054) — NO new variance math.
+   - a case is **stable** iff `variance ≤ maxVariance` (default 0) **AND** it was sampled ≥ `repeatN`
+     (under-sampling never reads as stable — you cannot certify a spread you did not measure).
+   - the **gate passes** iff EVERY case is stable. An empty case set FAILS (nothing measured).
+3. **order** — `nextPhaseAfterVariance(current, gate)` advances `build`/`variance` → `accuracy` ONLY
+   when the gate passed; otherwise it stays in `variance`. `assertVarianceStableBeforeAccuracy(gate)`
+   **THROWS** if accuracy is attempted on a flapping suite — the hard guard for the doctrine line.
+
+Only once variance is stable does the **accuracy** phase run the suite over the full dataset.
+
+| Knob | Default | Meaning |
+|------|---------|---------|
+| `repeatN` | 5 | reruns per case before the spread is certifiable |
+| `maxVariance` | 0 | per-case eval-score variance ceiling (0 = byte-identical verdict) |
+
+A looser `maxVariance` (e.g. `0.05`) admits a single `uncertain` among passes without flapping the
+gate — set it per subject when an occasional principled abstain is acceptable.
+
+---
+
+## F18 — the change-request contract + the loop
+
+When a case fails (in EITHER phase), the evaluator builds a **grounded** change-request and hands it
+off — it does not patch:
+
+```
+evaluator  --(EddChangeRequest)-->  ai-engineer      # SendMessage
+ai-engineer --(ChangeRequestResponse)--> evaluator   # amended | rejected
+evaluator  re-evaluates (from the variance phase)  →  loop
+```
+
+> **User-facing terminology (dogfood H1).** `EddChangeRequest` / `ChangeRequestResponse` are INTERNAL
+> type names — NEVER surface them to the operator. In Helix's prose the loop reads as plain language:
+> "the evaluator sends the engineer **a fix request** with the failing cases; the engineer **amends
+> and replies**." Show the outcome, never the symbol (INTERNALS UNDER THE HOOD).
+
+- **`EddChangeRequest`** (`buildChangeRequest` + `validateChangeRequest`): `swing`, `subject`,
+  `remedyTarget ∈ {agentspec, impl}`, `failingCases[]` (each with the verbatim `critique` + **≥1**
+  `ref{obs,path,value}` — an ungrounded ask fails loud, GA-1), `proposedRemedy` (a HYPOTHESIS).
+  - **`agentspec`** = the DEFINITION is wrong → the engineer edits `agentspec.yaml` and **re-runs
+    `*build`** so def→impl cascades.
+  - **`impl`** = a wiring / build-faithfulness defect that does NOT change the spec.
+- **`ChangeRequestResponse`** (`validateChangeResponse`): `amended` (→ `reEvalWarranted` ⇒ re-eval
+  from the variance phase) or `rejected` (with a mandatory `note` — never re-eval an unchanged
+  subject). There is no "trust me, skip re-eval" path: an amend ALWAYS triggers a fresh swing.
+
+### The bounded terminator (afkloop-legal — NEVER infinite)
+
+`decideEddLoop(state, budget)` reads ONLY observable, injected state
+`{ phase, swing, varianceStable, accuracyMet, elapsedMs, noImprovementStreak }` and decides
+**success-first**:
+
+| Check | Condition | Result |
+|-------|-----------|--------|
+| 1 | `varianceStable && accuracyMet` | **DONE** (`full-green`) |
+| 2 | `swing ≥ maxSwings` | **STOPPED** (`max-swings`) + convergence delta |
+| 3 | `elapsedMs ≥ maxWallclockMs` | **STOPPED** (`max-wallclock`) |
+| 4 | `noImprovementStreak ≥ noImprovementStreakLimit` | **STOPPED** (`no-improvement-streak`) |
+| else | — | CONTINUE (variance phase until stable, then accuracy) |
+
+Defaults (`DEFAULT_EDD_LOOP_BUDGET`): `maxSwings 6` · `maxWallclockMs 30min` ·
+`noImprovementStreakLimit 2`. `elapsedMs` is **injected** (the caller measures wall-clock; the
+controller never reads a clock) → the decision is PURE + deterministic (C-PIN).
+
+---
+
+## Assumed eval-runner interface (the clean seam — REBUILD nothing)
+
+The EVAL engine (dataset build · the eval runner · Path A/B) is built **in parallel** (sibling
+worktree). The `improve` mode is **additive**: it orchestrates the LOOP around the runner and assumes
+this interface — it does not re-implement it.
+
+```
+// the runner the improve mode CONSUMES (owned by the eval engine, e.g. run-evaluate.ts):
+runOnce(caseIds: string[]) -> Array<{
+  caseId: string;
+  criterionId: string;
+  verdict: "pass" | "fail" | "uncertain";   // OutcomeVerdict
+  trajectory: string[];                       // ordered tool names (for trajectory spread)
+}>
+```
+
+- **repeat-N** = `runOnce` called N times over the SAME `caseIds`; the per-rerun verdicts feed
+  `CaseVarianceObservation.verdicts[]` (+ `trajectories[]`) → `evaluateVarianceGate`.
+- **accuracy** = `runOnce` over the FULL dataset → per-case verdicts → an accuracy ratio vs target.
+- the runner is **C-PIN** (pinned model + temperature 0). The improve mode adds **no** provider call.
+
+If the sibling runner exposes a different shape, adapt at the seam (map its result into
+`CaseVarianceObservation` / the accuracy ratio) — keep the gate + loop modules untouched. The gate
+(`variance-gate.ts`) + the loop (`change-request.ts`) are PURE and runner-agnostic by design.
+
+---
+
+## Boundaries
+
+- **Judge-only (EV-051):** the evaluator REQUESTS + re-evals; the `ai-engineer` is the ONE
+  agent allowed to amend the Agent/AgentSpec. Fixables + infra-class still route to diagnostics.
+- **Variance-first (F19):** accuracy is never measured before the variance gate passes (assert-guarded).
+- **Bounded (F18):** every loop path terminates within the budget; STOP reports the convergence delta.
+- **Lockstep (PR-011):** the request names the locus, the engineer cascades def→impl, the re-eval
+  re-grounds the verdict on the amended subject — spec + impl + eval move together.

package/mutagent-evaluator/.claude/skills/mutagent-evaluator/references/error-analysis.md

@@ -0,0 +1,113 @@
+# error-analysis — mine emergent eval criteria from traces
+
+> **Source:** absorbed from `mutagent-system/.memory/features/evals-skills-source/skills/error-analysis/SKILL.md`.
+> **Loaded by:** `*discover-evals` (EV-041/042/052). Load on demand.
+> **Sibling refs:** `write-judge-prompt.md` (build the judge) · `validate-evaluator.md` (calibrate it) · `grounded-adjudication.md` (the GA doctrine: bind · gather refs · typed assumptions · abstain · verify).
+
+The foundation of every eval suite: read traces, judge ✓/✗, and let failure CATEGORIES emerge —
+never start from a pre-defined list. The categories that emerge ARE the binary actionable criteria
+`*build-evals` turns into judges.
+
+## The 7-step process
+
+1. **Collect ~100 representative traces.** Capture the FULL trace: input event, every intermediate
+   LLM call, tool uses, retrieved docs, reasoning, final output. ~100 is roughly where new traces
+   stop revealing new KINDS of failure (system-complexity dependent).
+2. **Read each trace → judge Pass/Fail → trace the failure to its ROOT.** This is `*discover-evals`'s
+   determine-outcome step (EV-042). Run the **3 detection lenses** (below) to surface candidate
+   failures, then apply **root-not-symptom**: trace each Fail to its ROOT with judgement — NOT the
+   first visible symptom (the first wrong is often downstream of the real root). Write observations
+   **grounded by a structured ref** `{obs, path, value}`, not explanations ("SQL missed the budget
+   constraint @ obs `a1b2/tool.input.query`", not "the model probably didn't understand"); ground
+   every ABSENCE claim ("did not", "never", "no X") with a positive field check, never inferred from
+   silence. **Critical rule (sample):** "inaction can be success" — never use "took an action /
+   called a tool" as a Pass proxy. A guard-hold (the agent correctly does NOT send) is a Pass.
+3. **Group failures into 5-10 categories** after the first 30-50 traces (don't wait for all 100 —
+   early grouping sharpens what to look for). Split notes with different root causes; group notes
+   with the same one. Each category gets a clear name + one-sentence definition. LLM-assisted
+   clustering is allowed ONLY after 30-50 human-reviewed traces — and always re-review the groupings
+   (LLMs cluster by surface similarity).
+4. **Label every trace** against the categories — one binary column per category.
+5. **Compute failure rates** (`category_failures / total`); sort descending. The most frequent
+   category is where to focus first.
+6. **Decide what to do about each failure (S4 — fix-vs-eval-worthy):**
+   - **Can we just fix it?** Prompt never asked for the behavior → add the instruction. Tool missing
+     → add the tool. Engineering bug → fix the code. If a clear fix resolves it, do that FIRST.
+   - **Is an evaluator worth the cost?** Only for failures that PERSIST after fixing AND that the
+     user will iterate on repeatedly (frequency × business impact). Critical requirements (safety,
+     compliance) warrant a judge as a regression guard even after fixing.
+   - **In this skill (EV-051): we never fix.** The evaluator FLAGS fixable-vs-eval-worthy and ROUTES
+     the fixable + infra-class failures to `mutagent-diagnostics` (e.g. sample C4 dead-channel
+     `account_number_unavailable` → WHY=`dependency-failure`, WHERE=`provider-side`). It builds
+     judges only for the genuinely eval-worthy behavioral criteria.
+   - Prefer **code-based checks** (regex / parsing / schema / tool-output flags) for anything
+     objective; reserve LLM judges for true judgment calls.
+7. **Iterate** — expect 2-3 rounds. Merge overlapping categories, split too-broad ones, clarify
+   definitions, re-label.
+
+## The 3 detection lenses (make them explicit)
+
+Determination is not a single undifferentiated read — surface candidate failures through **three
+named lenses**, and tag which lens each candidate fired on (this is what later becomes the
+criterion's `dimension` / `class`):
+
+| Lens | What it catches | Typical disposition |
+|------|-----------------|---------------------|
+| **drift / off-path** | The agent left the intended ROUTE — wrong tool, wrong order, answered a different question, ignored a constraint. A judgement call. | → **judge**-class behavioral criterion |
+| **tool-output failure** | A tool errored, timed out, or returned unusable/empty output the agent then mis-used. | → often **code**/fixable → route to diagnostics |
+| **missing-context** | A referent the agent NEEDED was never supplied (blank brief, dropped field, unresolved variable). This is the **BIND** detector — a term with no referent. | → typed **factual-intent** assumption; situation is indeterminate, not a fail |
+
+The lenses are detectors, not categories — the same trace can fire more than one. Categories still
+EMERGE from what fires (step 3); the lenses only make sure you LOOK in all three places.
+
+## root-not-symptom (REPLACES first_thing_wrong_only)
+
+The old `first_thing_wrong_only` rule conflated **detection** with **localization** — the first
+*visible* wrong is frequently downstream of the real root, and its single-cascade assumption silently
+drops independent causes. The corrected invariant:
+
+- **KEEP:** one criterion per **ROOT** — dedup the cascade (don't mint a criterion per downstream
+  symptom of the same root).
+- **FIX:** trace to the root with **judgement**, not the first symptom.
+- **Multiple INDEPENDENT roots ⇒ multiple criteria** (the cascade assumption is dropped).
+- A **causal-link claim** (root → symptom) must be **GROUNDED** — cite the edge with a ref — OR
+  surfaced as a **typed assumption**, which makes the localization **INDETERMINATE** (not a fail).
+- **Deep recursive-why → route to `mutagent-diagnostics`.** The evaluator LOCALIZES (locus: prompt ·
+  tool · context · skill); it does not run full root-cause analysis.
+
+## Stopping criterion (saturation)
+
+Stop when new traces stop revealing new KINDS of failure — roughly ~100 reviewed with no new failure
+type in the last ~20. Exact number depends on system complexity.
+
+## Trace sampling strategies (EV-052 — the 5)
+
+When production volume is high, use a MIX — never a single angle:
+
+| Strategy | When | Method |
+|----------|------|--------|
+| **Random** | Default starting point | Sample uniformly from recent traces |
+| **Outlier** | Surface unusual behavior | Sort by length / latency / tool-call count; review extremes |
+| **Failure-driven** | After guardrail violations / complaints | Prioritize flagged traces |
+| **Uncertainty** | When automated judges exist | Focus where judges disagree or have low confidence |
+| **Stratified** | Ensure coverage across segments | Sample within each dimension (query type, segment, feature) |
+
+Aim for a balanced ✓/✗ split (~50/50) so the criteria are mined from BOTH success and failure modes.
+
+## Worked example — sample-email-agent (the dogfood)
+
+The sample export carried **0 scores / 0 tags on all 1946 traces** — no label column to shortcut to,
+so determine-outcome (EV-042) had to deep-read every trace. Emergent categories (the 8 candidate
+criteria): C1 outbound-guard compliance · C2 goal-attainment · C3 manager-override honored · C4
+send-failure recovery · C5 channel discipline · C6 draft→send integrity · C7 escalation
+appropriateness · C8 memory hygiene. C4 + dead-channel were flagged fixable/infra → routed to
+diagnostics, NOT judged. C1/C2/C3/C7 are eval-worthy behavioral judgments → judges built.
+
+## Anti-patterns
+
+- **Brainstorming categories before reading traces.** Read first; categorize what you find.
+- **Starting with pre-defined categories.** A fixed list causes confirmation bias.
+- **Generic scores as categories** ("hallucination score", "helpfulness score") — not grounded in
+  the application's actual failure modes.
+- **Building evaluators before fixing obvious problems** (prompt gaps, missing tools, bugs).
+- **Treating error analysis as one-time.** Re-run after every significant change.

package/mutagent-evaluator/.claude/skills/mutagent-evaluator/references/eval-audit.md

@@ -0,0 +1,154 @@
+# eval-audit — audit an eval pipeline (incl. the evaluator's OWN) for trustworthiness
+
+> **Source:** absorbed from `mutagent-system/.memory/features/evals-skills-source/skills/eval-audit/SKILL.md`
+> (S16 six-area diagnostic) + its meta-skill framing (S17 — *eval-of-the-eval*).
+> **Loaded by:** `*audit` / `*self-audit` (the eval-of-the-eval mode, EV-055). Load on demand.
+> **Sibling refs:** `error-analysis.md` (Area 1) · `write-judge-prompt.md` (Area 2) ·
+> `validate-evaluator.md` (Area 3) · `build-review-interface.md` (Area 4) ·
+> `generate-synthetic-data.md` (Area 5).
+> **On-demand only:** this audit NEVER auto-fires — no cron, no monitor, no cadence
+> (`feedback_self_diagnostics_on_demand_only`). The operator invokes it explicitly.
+
+Inspect an LLM eval pipeline and produce a **prioritized list of problems**, each linked to a
+concrete fix. Use it when inheriting an eval system, when unsure whether evals are trustworthy, or —
+the **meta-skill (S17)** — to point the evaluator at **its own** eval-development artifacts:
+*are my judges validated? is my dataset balanced? are my criteria binary + actionable? is my suite
+living?* That eval-of-the-eval is `*self-audit` (EV-055), built on the existing `*audit` surface +
+the `*validate` stats (it REUSES them — it does not rebuild a second auditor).
+
+## The six diagnostic areas (S16)
+
+Work each area: inspect the actual artifacts, decide whether the problem exists, record a finding if
+it does. **Order findings by impact** — most impactful first. Each area maps onto a surface this
+skill already ships, so the audit is concrete, never a generic checklist.
+
+### 1. Error Analysis — were criteria MINED from real ✓/✗ traces, or brainstormed?
+
+**Check:** Was systematic error analysis done on real (or synthetic) traces, with *observed*
+failure categories — not generic labels borrowed from research ("helpfulness", "coherence",
+"hallucination score")?
+
+**Problem if missing/brainstormed:** evaluators built without error analysis measure generic
+qualities instead of actual failure modes — they score well on paper and miss real problems.
+**Fix:** `error-analysis.md` (`*discover-evals`): mine emergent BINARY ACTIONABLE categories from the
+✓/✗ split; if no traces exist, `generate-synthetic-data.md` first.
+**Self-audit signal:** a discovered criterion with **no `sourceTraceIds`** (no trace-grounded
+evidence) is a brainstormed-not-observed smell — flag it.
+
+### 2. Evaluator Design — binary? failure-mode-specific? code where possible?
+
+**Check:** Are evaluators **binary pass/fail** (not Likert 1-5 / letter grades / un-thresholded
+scores)? Does each LLM judge target **exactly one** failure mode (not "is this helpful?")? Are
+**code-based checks** used for objectively-checkable criteria (format, schema, keyword,
+constraint)? Are similarity metrics (ROUGE / BERTScore / cosine) kept OUT of generation-quality
+scoring?
+
+**Problem if not:** Likert scales can't be calibrated; holistic judges produce unactionable
+verdicts; LLM judges on objective criteria waste tokens and add noise; similarity metrics measure
+surface overlap, not correctness.
+**Fix:** `write-judge-prompt.md` (`*build-evals`) — binary, one-failure-mode, critique-before-verdict;
+**code before judge** (SKILL §8). The criterion class (`objective→code · subjective→judge · hybrid`,
+SKILL §9) IS this routing.
+**Self-audit signal:** a criterion whose verdict space is not `{Pass, Fail}` (ordinal / >2 values),
+or an `objective`-class criterion routed to an LLM judge instead of a code-check — flag it.
+
+### 3. Judge Validation — validated vs human labels, with TPR/TNR + a clean split?
+
+**Check:** Are LLM judges validated against human labels (confusion matrix, **TPR/TNR** — not raw
+accuracy / percent-agreement / Cohen's κ)? Is there a disjoint **train/dev/test** split (few-shot
+examples NOT drawn from the measurement set)?
+
+**Problem if not:** an unvalidated judge may silently miss failures or flag passing traces; raw
+accuracy hides this under class imbalance (always-"Pass" scores 90% when 90% pass, catches zero
+fails); dev/test-as-few-shot leaks and inflates alignment.
+**Fix:** `validate-evaluator.md` (`*validate`, EV-044) — TPR/TNR, test-once, Rogan-Gladen
+correction, bootstrap CI; `<MIN_LABELS` stays `unvalidated`+bias-corrected (never blocks).
+**Self-audit signal:** a judge whose `ValidationResult` is **`unvalidated`**, or whose TPR/TNR is
+**below `TARGET_TPR`/`TARGET_TNR` (0.9)**, or a split that is not disjoint — flag it (this REUSES
+the `*validate` stats, it does not re-run validation).
+
+### 4. Human Review Process — domain experts, full traces, natural rendering?
+
+**Check:** Are domain experts (not generic annotators) labeling? Do reviewers see the **full trace**
+(input · intermediate steps · tool calls · retrieved context · output), not just the final output?
+Is data rendered **naturally** (markdown rendered, code highlighted, tables as tables) — not raw
+JSON in spreadsheet cells?
+
+**Problem if not:** general annotators catch formatting errors but miss domain failures; output-only
+review hides WHERE the pipeline broke; raw formats make reviewers parse instead of judge.
+**Fix:** `build-review-interface.md` (`*review`, EV-045) — one full trace/screen, native render,
+Pass/Fail/Defer. **Autonomy caveat:** in an afkloop run no human labels in-browser; the UI is built +
+DOM-smoke-tested and surfaced as an artifact.
+**Self-audit signal:** **zero `HumanLabel`s** feeding `*validate` (so no judge CAN be validated), or
+a review surface that renders only final outputs — flag it.
+
+### 5. Labeled Data — enough, and balanced?
+
+**Check:** Is there enough labeled data — ~100 traces for error-analysis saturation, **~50 Pass +
+~50 Fail** for reliable TPR/TNR? Are sampling strategies used to find informative traces (random +
+clustering + outlier + classification + feedback)?
+
+**Problem if insufficient/skewed:** small or class-imbalanced sets produce unreliable rates and wide
+CIs; a Fail-starved set can't measure TNR.
+**Fix:** the five sampling strategies in `error-analysis.md` + `sample-traces.ts` (EV-052) selectors;
+`generate-synthetic-data.md` (`*build-dataset`, EV-046) cartesian-expand with **deterministic
+near-dup removal** to supplement; `*discover-dataset` (EV-047) distills a living set from labeled ✓/✗.
+**Self-audit signal:** a dataset with **fewer than `MIN_PER_CLASS` Pass or Fail** cases, or with
+near-duplicate cases that `build-dataset`'s Jaccard drop should have removed — flag it.
+
+### 6. Pipeline Hygiene — re-run after change, evaluators maintained?
+
+**Check:** Is error analysis re-run after significant changes (model switch, prompt rewrite, new
+feature, incident)? Are judges periodically re-validated and datasets refreshed — a **living** suite,
+not set-and-forget?
+
+**Problem if stale:** failure modes shift after pipeline changes; evaluators built for the old
+pipeline miss new failure types; judges degrade silently.
+**Fix:** the **living-suite** invariant (EV-053) — `appendOnly` + `assertMonotonicGrowth`: a living
+artifact NEVER shrinks and grows as the subject evolves; re-validate on any model change (C-PIN
+forces it).
+**Self-audit signal:** a suite that **shrank** (monotonic-growth violation), or a pinned `judgeModel`
+that changed with **no subsequent re-validation** — flag it.
+
+## No eval infrastructure
+
+If the subject has no eval artifacts (no traces, no evaluators, no labeled data):
+1. Start with `error-analysis.md` (`*discover-evals`) on a sample of real traces.
+2. If no production data exists, `generate-synthetic-data.md` (`*build-dataset`) to create inputs,
+   run them through the subject, then apply error analysis to the resulting traces.
+3. Do **not** build evaluators / judges / dashboards before error analysis is complete.
+
+## Report format
+
+Findings ordered by impact. For each:
+
+```
+### [Problem Title]
+**Status:** Problem exists | OK | Cannot determine
+[1-2 sentences — the specific problem found in THIS pipeline, not generic advice]
+**Fix:** [concrete action → the *command / reference / skill that addresses it]
+```
+
+Group under the six areas; omit areas with no problems. For `*self-audit` the "subject" IS the
+evaluator's own eval-dev output, and the report is the eval-of-the-eval.
+
+## How `*self-audit` (EV-055) reuses the existing surfaces — austerity
+
+The eval-of-the-eval is NOT a second auditor. The **deterministic** Area checks (2 binary-shape,
+3 TPR/TNR-threshold, 5 balance/redundancy, 6 monotonicity, plus the Area-1/4 grounding/label
+counts) are pure code in `scripts/self-audit.ts` (Type A — emits **finding DATA**, no verdict prose,
+no LLM). The **nuanced** judgments that genuinely need reasoning — *is this criterion actionable or
+merely generic? does this judge prompt truly target one failure mode?* — are dispatched to the
+`audit-executor` agent (`*self-audit` mode, host-runtime, **NO provider key / NO Gemini**), exactly
+like every other Type-B judging op. The score it consumes (TPR/TNR, balance, monotonicity) comes
+straight from `*validate` + the living-suite provenance — reused, never recomputed.
+
+## Anti-patterns
+
+- Running the audit as a checklist without inspecting the actual artifacts.
+- Reporting generic advice disconnected from what was found in THIS pipeline.
+- Recommending evaluators before error analysis is complete.
+- Using an LLM judge for a failure a code-check can handle.
+- Treating the audit as one-time — re-audit after significant changes (Area 6 applies to the
+  evaluator itself).
+- Wiring `*self-audit` to a cron / monitor / cadence — it is on-demand only, ships dormant.

package/mutagent-evaluator/.claude/skills/mutagent-evaluator/references/eval-stage.md

@@ -0,0 +1,168 @@
+# `*eval` — the ADL EVAL-stage flow
+
+> Source: the ADL lifecycle (SPEC → BUILD → **EVAL**). Load on demand.
+> Drives `*eval` (the entry), and is the shared contract `*build-dataset` /
+> `*build-evals` / `*evaluate` follow. PARENT-SESSION ONLY (AskUserQuestion).
+
+The EVAL stage turns a freshly-`*build`-ed agent + its **agentspec** into a
+trustworthy eval suite and runs it to a GATE verdict. It is the evaluator's ADL
+entry-point: after `*build` hands over the built agent and the agentspec, `*eval`
+interactively derives the dataset + the eval suite FROM the spec, lets the user
+pick the eval ENGINE, and streams wireframe cards the whole way.
+
+This file is the operational glue; the deep mechanics live in the existing
+spine (`error-analysis.md` · `write-judge-prompt.md` · `validate-evaluator.md` ·
+`generate-synthetic-data.md`). It adds the ADL-specific findings F7–F22.
+
+---
+
+## Inputs (from `*build`)
+
+| Input | Shape | Used for |
+|-------|-------|----------|
+| the built agent | a target-framework impl (e.g. a Mastra/TS agent) OR a harness agent | the eval SUBJECT |
+| `agentspec.definition.evals` | `{ success_criteria[], scenarios[], dataset_categories[] }` | dataset materialization (F8) + criteria |
+| `agentspec.build` | `{ target_framework, runtime, target_eval_framework }` | the target-conditional engine fork (F9) |
+
+The skill is **standalone** — it NEVER imports the agentspec schema. It consumes
+a MINIMAL local slice (`scripts/contracts/agentspec-evals.ts` ·
+`scripts/contracts/eval-engine.ts EngineTargetInput`) the caller maps from the
+agentspec. No cross-skill import.
+
+---
+
+## Step 0 — interactive offer (F15)
+
+After `*build`, OFFER the two derivations (AskUserQuestion, parent session only):
+
+1. **`*build-dataset`** — materialize + grow the golden dataset from the spec.
+2. **`*build-evals`** — build the eval suite (and pick the engine).
+
+Let the user run either/both. They are independent; `*evaluate` consumes both.
+
+**Dataset gate (dogfood H5).** `*evaluate` REQUIRES a materialized dataset. If `*build-evals` runs
+(or `*evaluate` is invoked) with no dataset present, do NOT dead-end — auto-route to `*build-dataset`
+(interactive) first, then resume. Never leave the user with an eval suite and nothing to run it on.
+
+---
+
+## Step 0.5 — Ground in what the agent DOES, first (dogfood H2)
+
+BEFORE asking ANY eval-config question (engine, runtime, models), surface the SUBJECT: read the
+agent's `jobs_to_be_done` + `scenarios` + `success_criteria` from the spec and CONFIRM with the user
+which use-cases matter for this eval. Understand what the agent does and what "good" means for it
+BEFORE scoping the eval target — never lead with framework/runtime config. The confirmed use-cases
+scope everything downstream (dataset categories, judge criteria, the engine fork below).
+
+---
+
+## Step 1 — the ENGINE FORK (F7 / F9 / F14)
+
+`*build-evals` (and the eval-impl half of `*eval`) MUST ASK the eval
+implementation MODE before building anything. Use
+`scripts/eval-engine.ts chooseEvalEngineOptions(target)`:
+
+| Engine | What | Dependency (SURFACED up front) | Portable? |
+|--------|------|--------------------------------|-----------|
+| **Path A `native-matrix`** | mutagent native eval-matrix + LLM-judge SUB-AGENTS | **needs a Claude-Code host** to spawn judges; on a CODE-framework target ALSO needs the built agent to **emit logs/traces to a known sink** for the judge to read | NO |
+| **Path B `code-written`** | evals in the target's OWN language (bun/TS): code-checks + LLM-judge via SDK + criteria checks | none — runs WITHOUT Claude Code | YES |
+
+**Target-conditional (F9).** A code FRAMEWORK target (mastra · langgraph ·
+pydantic-ai) offers BOTH engines; a `harness:*` target (`harness:claude-code`,
+`harness:codex`) is **native-only** (no target language to write portable evals
+into) — `assertEngineMatchesTarget` THROWS on `code-written` + harness.
+
+**Surface the dependency BEFORE the choice (F7).** Each menu option carries
+`requiresClaudeCode` + `requiresLogSink` + `portable` — render them in the
+AskUserQuestion preview so a user is NEVER silently "stuck with the matrix".
+
+**F14 — don't trap non-CC users.** `code-written` is the *recommended default*
+for a code framework. It compiles via `scripts/codegen-evals.ts codegenEvalSuite`
+to a self-contained suite source the user runs directly; the only LLM path is
+THEIR provider SDK (`judgeViaSdk`), never a CC sub-agent.
+
+`resolveEvalEngine(choice, target)` → the `EvalEnginePlan` (`engine` ·
+`requiresClaudeCode` · `requiresLogSink` · `outputSink` · `rationale`). The plan's
+`outputSink` is always a discoverable path (the "outputs land in a discoverable
+sink" success-gate).
+
+**Don't over-ask the judge model (dogfood H4).** When the user has already chosen a HOST runtime
+(Path A / Claude-Code host), DO NOT ask which judge model to use — default the judge to **Opus** (the
+general judge; other models may be unavailable). Derive the judge from prior answers; only ask what
+isn't already implied by the runtime choice. Collapse the eval-config chain to the minimal
+un-inferable set — never re-prompt for something a previous answer already determined.
+
+> **Native-on-a-code-framework requires the log sink (F9).** If the user picks
+> Path A for a Mastra/TS target, the built agent MUST emit logs/traces to
+> `plan.outputSink` (CC session transcripts or a declared log file) so the judge
+> sub-agents have something to read. Confirm this is wired before dispatching
+> judges — otherwise the matrix has no input.
+
+---
+
+## Step 2 — MATERIALIZE the dataset (F8)
+
+`*build-dataset` MATERIALIZES real items, not just definitions. Use
+`scripts/materialize-dataset.ts`:
+
+- `materializeFromAgentspec(evals)` → ≥1 REAL `DatasetCase` per `dataset_category`
+  + one per declared `edge_case` (flagged `edge_case: "true"`). The base item
+  derives from a category-tagged scenario when present, else the category
+  description ("seed, don't duplicate").
+- `materializeToDataset(subject, evals, existing?)` → a schema-valid `Dataset`,
+  merged MONOTONICALLY (re-materializing adds no duplicates; version bumps).
+
+These seeds are then handed to the `dataset-builder` agent for synthetic
+expansion (`generate-synthetic-data.md`); `build-dataset.ts` dedup/merge dedups
+any overlap. **Success gate:** a dataset with ≥1 real item per category incl.
+edge-cases.
+
+---
+
+## Step 3 — STREAM wireframe cards (F13 / F16 / F22)
+
+Stream PROGRESS as wireframe cards for BOTH `*build-dataset` and `*build-evals`
+(`scripts/render-build-cards.ts`):
+
+- `renderBuildDatasetProgressCard` / `renderBuildEvalsProgressCard` — phase +
+  progress bar + counters, emitted per phase (F13/F16).
+- After each, emit the VERBOSE entity card (F22): `renderDatasetEntityCard`
+  (per-category breakdown · version · sink) · `renderEvalsEntityCard` (engine ·
+  SURFACED CC/log-sink dependency · criteria).
+
+---
+
+## Step 4 — RUN + the SCORECARD DASHBOARD (F20)
+
+`*evaluate` runs the suite vs the target:
+
+- **Path A** — the existing `run-evaluate.ts` spine (PREP → dispatch
+  `#mode-judge-trajectory` → aggregate → GATE).
+- **Path B** — run the `codegen-evals.ts` suite (no Claude Code) and read back its
+  scorecard JSON from `plan.outputSink`.
+
+Either way, render the result as a DASHBOARD wireframe, NOT a flat dump:
+`renderScorecardDashboard` — gate banner + per-criterion pass/fail BAR +
+pass/fail/indeterminate split + variance + sample count (F20). (The 5-tab HTML
+eval-report `render-eval-report.ts` is the rich artifact; the dashboard is the
+terminal at-a-glance.)
+
+---
+
+## Success gates (the EVAL-stage definition of done)
+
+- a dataset with **≥1 real item per category (incl. edge-cases)** — F8.
+- the eval suite **runs and emits a scorecard dashboard** — F20.
+- the **engine matches the target** (target-conditional gate) — F9.
+- **Path B works without Claude Code** — F14.
+- **outputs land in a discoverable sink** (`plan.outputSink`).
+
+## Boundaries (held from the spine)
+
+- **Judge-only, never fix (EV-051)** — flag + route failures to diagnostics.
+- **Reviewer ≠ executor** — never grade a run the evaluator produced.
+- **C-PIN** — pinned judge model + temp 0; masked scorecard byte-identical.
+- **Standalone** — ships its own agents; no host architect/developer dependency.
+- **Relative paths only** — every sink/path resolves relative to the installed
+  `.claude/skills/mutagent-evaluator/` layout (a fresh install lives in a
+  DIFFERENT repo); NO absolute `/Users/...` paths anywhere.

package/mutagent-evaluator/.claude/skills/mutagent-evaluator/references/generate-synthetic-data.md

@@ -0,0 +1,81 @@
+# generate-synthetic-data — bootstrap an eval dataset by dimension-based tuple generation
+
+> **Source:** absorbed from `mutagent-system/.memory/features/evals-skills-source/skills/generate-synthetic-data/SKILL.md`.
+> **Loaded by:** `*build-dataset` (EV-046), LLM steps executed by `assets/agents/dataset-builder.md`. Load on demand.
+> **Sibling refs:** `error-analysis.md` (mine the criteria) · `build-review-interface.md` (label the traces) · `validate-evaluator.md` (calibrate the judge).
+
+Create diverse, realistic test inputs that cover a subject's FAILURE space — for bootstrapping an
+eval dataset when real traces are sparse, or for stress-testing a specific failure hypothesis. Do
+NOT use when you already have 100+ representative real traces (use `sample-traces.ts` stratified
+sampling instead), or when the task is collecting production logs.
+
+## How this maps onto the evaluator (the 3-way Hybrid, EV-046)
+
+`*build-dataset` is a **Hybrid** op with three pieces — keep them separated (operation-inventory):
+
+| Piece | Type | Who | What |
+|-------|------|-----|------|
+| **Seed interview** (~10 tuples) | HITL gate | `AskUserQuestion` / chat fallback | The user confirms the seed tuples are realistic. The user's domain knowledge is what makes synthetic data trustworthy. |
+| **Tuple gen → NL query → quality filter** | LLM-only | `assets/agents/dataset-builder.md` (host-runtime, NO provider key) | Steps 1·3·4·5 below — the reasoning. Lives in the agent def, NEVER in a script. |
+| **Cartesian expand · dedup · schema · append** | Code-only | `scripts/build-dataset.ts` | The deterministic shape (Step 2 mechanics + Step 5 dedup gate). Holds NO prompt prose. |
+
+**Subject-agnostic (EV-002/EV-049):** dimensions + their values come from the subject profile
+(`subjects/<name>/`) + the seed interview, NEVER hard-coded into the engine. The same `*build-dataset`
+runs on any subject.
+
+## Core process
+
+### Step 1 — Define dimensions (target failures, not arbitrary variation)
+Dimensions are axes of variation specific to the subject. Choose them where failures are EXPECTED
+(known failure-prone areas, existing feedback, hypotheses from traces). Start with **3**; add a
+dimension only when initial traces reveal a failure pattern along a new axis.
+
+```
+Dimension 1: [Name] — [what it captures]
+  Values: [value_a, value_b, value_c, …]
+Dimension 2: …
+Dimension 3: …
+```
+
+### Step 2 — Draft ~10–20 seed tuples WITH the user (HITL gate)
+A tuple is one combination of dimension values = one specific test case. Present the seed tuples and
+iterate until the user confirms they reflect realistic scenarios. This is the human-in-the-loop seed
+gate; the user knows which combinations actually occur and which are unrealistic.
+
+```
+(Dim1: value, Dim2: value, Dim3: value)
+```
+
+### Step 3 — Generate more tuples with the LLM (dataset-builder)
+Prompt for N more `(dim1, dim2, dim3)` combinations for the subject — avoid duplicates, vary values
+across dimensions. **Tuple generation is a SEPARATE step from query generation** (combining them
+produces repetitive phrasing).
+
+### Step 4 — Convert each tuple to a natural-language query (dataset-builder)
+A separate prompt per tuple: "given these dimension values, write a realistic query a user might
+enter, reflecting the persona/scenario." Seed it with one hand-written example.
+
+### Step 5 — Filter for quality (dataset-builder proposes · build-dataset dedups)
+Discard + regenerate when phrasing is awkward/unrealistic, content doesn't match the tuple's intent,
+or queries are too similar. Optional: rate realism 1–5, discard < 3. The **near-duplicate dedup is
+deterministic** (`build-dataset.ts`); the realism judgment is the agent's.
+
+### Step 6 — Run queries through the subject (OUT OF SCOPE here)
+Running the generated queries through the subject pipeline to capture full traces is the SUBJECT's
+harness, not the evaluator's. `*build-dataset` produces the dataset (queries + tuples + provenance);
+the dataset is appended to the living suite via `living-suite.ts` (EV-053). **Target ~100
+high-quality diverse cases** — the rough saturation heuristic.
+
+## Sampling real data instead (when you have it)
+Don't sample randomly. Use stratified sampling (`sample-traces.ts`): identify high-variance
+dimensions → assign labels → sample from each group. Use synthetic data only to fill gaps in
+underrepresented query types.
+
+## Anti-patterns
+- **Unstructured generation** ("give me test queries") → generic, repetitive, happy-path examples.
+- **Single-step generation** (tuples + queries in one prompt) → less diverse than the two-step split.
+- **Arbitrary dimensions** that don't target failure-prone regions → wasted test budget.
+- **Skipping the user's tuple review** → no way to judge whether LLM tuples are realistic.
+- **Synthetic data when no one can judge realism**, or for complex domain-specific content
+  (legal/medical) or low-resource dialects → use real data instead.
+- **Hard-coding a subject's dimensions into a script** — they belong in the subject profile (EV-002).