@mutagent/evaluator 0.1.0-alpha.3 → 0.1.0-alpha.4

package/.claude/skills/mutagent-evaluator/SKILL.md

@@ -12,8 +12,11 @@ description: |
   framework-substrate choice (DEFAULT agent-dispatch — host-runtime subagents, no provider ·
   in-house AI-SDK judge optional · code-based · user's framework export).
   Subsequent invocations dispatch the parent session + MASS-PARALLEL evaluator
-  sub-agents (one cell, discover / judge modes; host runtime, harness-capped) against the subject's traces. ALSO surfaces the existing v1 4-tab static-auditor
-  (eval-matrix · data-leak · variance · methodology) via *audit. Pinned judge model + temp=0
+  sub-agents (one cell, discover / judge modes; host runtime, harness-capped) against the subject's traces.
+  DEFAULT INTENT: any "evaluate this skill/agent" request runs *evaluate — the JUDGE — which is what an
+  end user wants. *audit is a SEPARATE, explicit-only 4-tab static-auditor (eval-matrix · data-leak ·
+  variance · methodology) used mainly for complex agents/skills that emit HTML artifacts; it is NEVER the
+  fallback for a plain evaluate. Pinned judge model + temp=0
   for byte-identical reruns (C-PIN). Usable standalone OR borrowed by mutagent-skill-builder's
   SIMULATE phase as the SimVerifier.
 license: Proprietary. LICENSE.txt has complete terms.
@@ -126,9 +129,15 @@ When you encounter a *<name> token:
 
 ## §1 — Triggers
 
+> **DEFAULT INTENT = `*evaluate` (the Judge).** "evaluate this skill/agent", "score/judge this",
+> a bare `/mutagent-evaluator`, or any ambiguous evaluate ask → **`*evaluate`**. **Never** fall back to
+> `*audit`. `*audit` (the 4-tab master-audit) is a SPECIALIZED, **explicit-only** surface — invoked by
+> name (`*audit <subject>`), typically for complex agents/skills that emit HTML artifacts. If unsure
+> between the two, choose `*evaluate`.
+
 Invoke me with:
-- `evaluate skill <name>` / `evaluate agent <name>` / `/mutagent-evaluator`
-- `*discover-evals` (mine criteria) · `build evals for <x>` · `validate judge <x>` · `*audit <subject>`
+- `evaluate skill <name>` / `evaluate agent <name>` / `/mutagent-evaluator`  → **`*evaluate`** (default)
+- `*discover-evals` (mine criteria) · `build evals for <x>` · `validate judge <x>` · `*audit <subject>` (explicit only)
 - `*improve` (the ADL ③ EDD loop — variance-first then request-amend-reeval to full green, bounded)
 - `*self-audit` (the eval-of-the-eval — audit my OWN eval-dev; on-demand only)
 - borrowed by `mutagent-skill-builder` SIMULATE (SimVerifier role)
@@ -147,7 +156,9 @@ Minimal eval-dev spine (build order — do NOT reorder):
                 amend agentspec|impl over SendMessage → re-eval) → full green OR bounded STOP
 ```
 
-`*validate` (calibrate the judge) and `*audit` (the v1 4-tab static report) layer on top.
+`*validate` (calibrate the judge) layers on top. `*audit` (the v1 4-tab master-audit) is an
+**explicit-only** specialized surface (complex / HTML-artifact subjects) — NOT part of the default
+evaluate spine and never an automatic fallback.
 `*improve` is the IMPROVE stage — it runs AFTER an initial `*build` + `*evaluate` and is **bounded**
 (never infinite): full-green ⇒ DONE; else max-swings / max-wallclock / no-improvement ⇒ STOP + delta.
 
@@ -167,7 +178,7 @@ log-sink dependency up front, F7/F9/F14), streams wireframe progress + entity ca
 - **Curated Discovery handoff (P-B)** — when the traces arrive from the `discovery` SYSTEM AGENT under a PURPOSED intent (`*discover-evals` → `operationIntent: evals` · `*discover-dataset` → `operationIntent: dataset`), the handover is a **curated, classified** set: a `SelectionManifest` (`mutagent-tools` `selection.ts` — `byScenario` · `byWorthiness` · labeled/unlabeled counts) alongside `traces.jsonl` where each kept record carries `ext.signals` (deterministic worthiness from `trace select`) + `ext.classification` (`scenario` · `edgeCase` · `outcome` · `worthiness`, from the agent's LLM classify). BOTH evaluator commands READ this handoff (skills never fetch) and consume the labels: **`*discover-evals`** STRATIFIES criteria-mining by `ext.classification.scenario` and PRIORITIZES by `ext.signals.worthinessScore` (unlabeled/low-worthiness traces are kept for coverage but down-weighted); **`*discover-dataset`** distills a **scenario-balanced** regression set, **edge-cases first** (`ext.classification.edgeCase.is`), reusing the `derive-dataset.ts` selectors + `build-dataset.ts` merge. A generic `fetch` handoff (no `ext.signals`/`ext.classification`) degrades gracefully to the pre-P-B behavior. The judge/mining internals are unchanged — only the sampling/stratification READS the new curated fields.
 - **Code-before-judge (EX-2)** — the §5b `check_method` routes each metric: `deterministic`→a runnable code-eval the agent executes via Bash (zero judge tokens, byte-identical), `llm-judge`→the host judge, `hybrid`→code pre-filter that gates the judge (`check-method-router.ts` + `code-eval.ts`).
 - **`*evaluate`** — the real spine (`run-evaluate.ts`): **T1 tier-0** code-method pre-pass (`tier0Plan` — code rows decided deterministically, only residual judge rows dispatched) → PREP packet (**T5 `adaptivePacketPlan`** overload guard, default 1:1) → dispatch `#mode-judge-trajectory` (emits the **T2 DAG v2 walk**) → fail-loud readiness gate → fold code+judge → **T3 independent verifier** (downgrade-only on gating fails) → **T4 `consolidateByLocus` + `deriveWalkHealth`** → GATE + variance → EV-051 route-to-diagnostics; the masked scorecard is byte-identical across reruns (C-PIN).
-- **Report** — terminal eval-cards + the operator-APPROVED 5-tab HTML eval-report (`render-eval-report.ts`): §1 Overview (KPIs · coverage-contract · gating table · top-findings) · §2 Trajectory‖Judge per-trace ledger + click-row Target-Agent‖Judge side-by-side · §3 Eval Scorecard cohort heatmap + nested subcards + inline calibration · §4 Findings verbatim-evidence + judge chain + agree/revise/refute · §5 Self-Eval [INTERNAL, stripped on publish]. `run-evaluate.ts` renders `report.html` post-aggregate; consumes the §9.4 judge-walk (dense na-explicit map + `judge_steps[]`) when present, degrades to the per-trajectory scorecard otherwise.
+- **Report** — terminal eval-cards + the operator-APPROVED 5-tab HTML eval-report (`render-eval-report.ts`): §1 Overview (KPIs · coverage-contract · gating table · top-findings) · §2 Trajectory‖Judge per-trace ledger + click-row Target-Agent‖Judge side-by-side · §3 Eval Scorecard cohort heatmap + nested subcards + inline calibration · §4 Findings verbatim-evidence + judge chain + agree/revise/refute · §5 Self-Eval [INTERNAL, stripped on publish]. `run-evaluate.ts` renders `evaluation-report.html` post-aggregate (client-audience by default; §5 shown only for `--audience internal`); consumes the §9.4 judge-walk (dense na-explicit map + `judge_steps[]`) when present, degrades to the per-trajectory scorecard otherwise.
 
 ## §3 — Architecture Overview
 

package/.claude/skills/mutagent-evaluator/assets/agents/discovery.md

@@ -18,7 +18,7 @@ ACTIVATION-NOTICE: This file contains your full agent operating guidelines. Read
 class: pure_subagent_executor
 isolation: none
 
-# STATUS: SCOPED / FUTURE — not dispatched today.
+# STATUS: DISPATCHED standalone as the vendored delegate-or-artifact trace agent (diagnostics Step 3.6 · evaluator Step 0.5). The orchestrator `*collect-traces` alias route remains future.
 # Helix currently calls `mutagent-cli trace count` + `trace fetch --export` DIRECTLY (the
 # example workflows in trace-layer-plan.md §Example Workflows). This agent FORMALIZES that
 # pattern into a delegatable cell once the CLI surface is proven in production. The contracts
@@ -359,7 +359,7 @@ You are a **discovery** agent. You collect agent traces on behalf of Helix (or a
 orchestrator) and hand the result to a downstream skill (Diagnostics or Evaluator). You do
 NOT analyze traces — you only **collect, normalize, and validate** them.
 
-> **SCOPED / FUTURE.** This agent contract is written now so the contracts are locked and
+> **STATUS.** The vendored agent IS dispatched standalone today (diagnostics Step 3.6 · evaluator Step 0.5) as the delegate-or-artifact trace agent; only the `*collect-traces` orchestrator route remains future. The contract below was written ahead so it was locked and
 > the handoff chain is fully specified. Helix currently calls `mutagent-cli` directly (the
 > example workflows in `trace-layer-plan.md`). This agent formalizes that into a delegatable
 > cell once the CLI surface is proven in production. Dispatch wiring (`*collect-traces` route

package/.claude/skills/mutagent-evaluator/assets/agents/evaluator.md

@@ -870,7 +870,7 @@ this inline section is now its canonical home.
    the FAILURE + UNCERTAIN traces into DATASET CANDIDATES (`collectDatasetCandidates`, reusing the
    derive-dataset selectors) and folds in the `*evaluate` judge's node-2.5 unmatched-detection
    handoff — both consumable by `*build-dataset`. You mine + flag; the parent emits the candidates.
-7. **Render the discover `report.html` via the SHIPPED `writeDiscoverRunReport` composer**
+7. **Render the discover `discovery-report.html` via the SHIPPED `writeDiscoverRunReport` composer**
    (`render-discover-report.ts`) — pass the run dir; it builds `triage-summary.json` from
    `triage.json` + wires every companion (verdicts → Proof-of-work, dataset-candidates → Dataset,
    triage census → coverage funnel, profile → entity hero) so the report is COMPLETE BY DEFAULT.

package/.claude/skills/mutagent-evaluator/references/workflows/orchestrator-protocol.md

@@ -94,13 +94,26 @@ Resolve the pinned envelope ONCE for the whole run (C-PIN): `model` from
 
 ---
 
-## Step 0.5 — Trace intake (UniTF handover — the skill NEVER fetches)
-
-Traces arrive as a **pre-produced UniTF JSONL handover**: one `UnifiedTrace`
-record per line, written by `mutagent-cli trace fetch --export <path>` BEFORE this
-skill is dispatched (Helix runs the fetch in its PRE-STAGE — the evaluator owns no
-fetch/normalize code). Fetch + normalize live entirely in `mutagent-cli`; run
-`mutagent-cli trace --help` to discover the surface (count · fetch · export).
+## Step 0.5 — Trace intake (delegate-or-artifact — the evaluator ACQUIRES traces · reports-PRD R-6)
+
+The evaluator owns NO fetch/normalize code (that lives in `mutagent-cli` + the `discovery`
+agent) — but **a real user will not hand you a trace file**. So, in **parity with Diagnostics
+Step 3.6 (V3-9)**, traces are acquired by EXACTLY ONE of the 3 legal paths:
+
+| # | Path | Who fetches | When |
+|---|---|---|---|
+| a | **Helix pre-fetch / handover** | Helix (pre-stage) or a pre-produced `mutagent-cli trace fetch --export`, handed in via `--traces <handover.jsonl>` | orchestrated runs |
+| b | **`*discover` dispatch** | the `discovery` system agent | orchestrated collection |
+| c | **Standalone: artifact intake OR vendored-Discovery spawn** | a pre-staged artifact, or the vendored `discovery` sub-agent | THIS step |
+
+**Standalone flow (path c), in order:**
+1. **Handover / artifact intake first.** If `--traces <handover.jsonl>` (`.gz` ok) was handed in,
+   or a pre-staged `.mutagent/evaluator/<run>/ingest/traces.jsonl` exists → consume it; do NOT spawn Discovery.
+2. **Else DELEGATE to the vendored Discovery agent.** Spawn `assets/agents/discovery.md` (installed at
+   `.claude/agents/discovery.md`) via the Task tool (`subagent_type: "discovery"`), handing it the
+   resolved TraceQuery (source + filters + window + `--export .mutagent/evaluator/<run>/ingest/traces.jsonl`).
+   Discovery fetches + curates → a `SelectionManifest` + `traces.jsonl`; the PREP/AGGREGATE CLIs then
+   read it via `--traces` (P-B curated `ext.signals`/`ext.classification` consumed when present).
 
 The PREP/AGGREGATE CLIs (`scripts/cli/{prep,aggregate,dogfood}.ts`) take the
 handover file via `--traces <handover.jsonl>` (`.gz` also accepted) and read it

package/.claude/skills/mutagent-evaluator/scripts/aggregate-discover.ts

@@ -338,7 +338,7 @@ export function writeDiscoverReport(input: WriteDiscoverReportInput): string {
     ...(input.companions ?? {}),
   });
   mkdirSync(input.reportDir, { recursive: true });
-  const outPath = join(input.reportDir, "report.html");
+  const outPath = join(input.reportDir, "discovery-report.html");
   writeFileSync(outPath, html);
   return outPath;
 }

package/.claude/skills/mutagent-evaluator/scripts/artifact-paths.ts

@@ -91,6 +91,22 @@ export function scorecardDir(runId: string, cwd?: string): string {
 export function reportDir(runId: string, cwd?: string): string {
   return underRoot(cwd, "reports", runId);
 }
+
+// ── the three named report artifacts (evaluator-reports PRD R-1/R-3) ──────────
+// Distinct names under ONE run folder — no `report.html` collision, and *audit no longer
+// writes under `.mutagent/diagnostics/`. Single resolver = single source of truth.
+/** `*evaluate` → the 5-tab judge report. */
+export function evaluationReportPath(runId: string, cwd?: string): string {
+  return underRoot(cwd, "reports", runId, "evaluation-report.html");
+}
+/** `*discover-evals` → the mined-criteria report. */
+export function discoveryReportPath(runId: string, cwd?: string): string {
+  return underRoot(cwd, "reports", runId, "discovery-report.html");
+}
+/** `*audit` → the 4-tab master-audit (now under the evaluator root, not diagnostics). */
+export function masterAuditPath(runId: string, cwd?: string): string {
+  return underRoot(cwd, "reports", runId, "master-audit.html");
+}
 export function livingSuiteDir(cwd?: string): string {
   return underRoot(cwd, "living-suite");
 }

package/.claude/skills/mutagent-evaluator/scripts/cli/audit-run.ts

@@ -30,6 +30,7 @@ import { runDeterministic } from "../run-deterministic.ts";
 import { assembleScorecard } from "../assemble-scorecard.ts";
 import { renderReport } from "../render-report.ts";
 import { maskedCanonicalJson } from "../mask.ts";
+import { reportDir } from "../artifact-paths.ts";
 
 const HERE = dirname(fileURLToPath(import.meta.url));
 const PKG_ROOT = join(HERE, "..", "..");
@@ -120,10 +121,13 @@ export function main(argvIn?: string[]): number {
   // 5. Render
   const html = renderReport(scorecard);
 
-  const outDir = args.outDir ?? args.bundleDir;
-  mkdirSync(join(outDir, "audit"), { recursive: true });
-  const scPath = join(outDir, "audit", "scorecard.json");
-  const htmlPath = join(outDir, "audit", "master-audit.html");
+  // Output → the UNIFIED evaluator reports root (PRD R-3): reports/<runId>/master-audit.html
+  // (was under .mutagent/diagnostics/<runId>/audit/ — the wrong stage root). bundleDir stays the
+  // INPUT (the run-bundle being audited); the report is written to the evaluator namespace.
+  const outDir = args.outDir ?? reportDir(args.runId);
+  mkdirSync(outDir, { recursive: true });
+  const scPath = join(outDir, "master-audit.scorecard.json");
+  const htmlPath = join(outDir, "master-audit.html");
   const scJson = args.emitMasked
     ? maskedCanonicalJson(scorecard)
     : JSON.stringify(scorecard, null, 2);

package/.claude/skills/mutagent-evaluator/scripts/cli/prep.ts

@@ -16,8 +16,10 @@
  * SOURCE SELECTION (--source): when `global.sources[]` has >1 entry and none is
  * `default:true`, the run-start config surfaces a `needs-selection` state; the
  * PARENT session then ASKS the operator to pick one (AskUserQuestion / chat
- * multi-choice) and re-invokes with `--source <name>`. A single source, exactly
- * one `default:true`, or an explicit `--source` skips the prompt.
+ * multi-choice) and re-invokes with `--source <name>`. When >1 entry has exactly
+ * one `default:true`, the config surfaces `confirm-default` (F2) — the PARENT
+ * CONFIRMS the preselected default (a yes/no ask; distinct from the open pick)
+ * before it is used. A single source or an explicit `--source` skips the prompt.
  *
  * MODEL INTENT SACRED: --model is the pinned host model (temperature pinned 0,
  * C-PIN); it is carried verbatim on every task spec for the subagent to honor —
@@ -58,6 +60,20 @@ function flag(argv: string[], name: string): string | undefined {
  */
 function surfaceSourceBinding(binding: SourceBinding): void {
   switch (binding.kind) {
+    case "confirm-default": {
+      // F2 — a preselected default among many: surface a CONFIRM ask (yes/no),
+      // distinct from the open needs-selection pick. NON-fatal: the default is usable
+      // as-is on accept / non-interactive / --source.
+      const names = binding.sources.map((s) => s.name).join(", ");
+      process.stdout.write(
+        `prep: source CONFIRM-DEFAULT — ${binding.sources.length} sources, default is ` +
+          `'${binding.source.name}'. PARENT: confirm the default via the platform ask ` +
+          `mechanism (AskUserQuestion / chat multi-choice; '${binding.source.name}' PRESELECTED, ` +
+          `operator may override), or accept it non-interactively (--source <name> / CI). ` +
+          `Candidates: ${names}\n`,
+      );
+      break;
+    }
     case "needs-selection": {
       const names = binding.sources.map((s) => s.name).join(", ");
       process.stdout.write(

package/.claude/skills/mutagent-evaluator/scripts/config/load.ts

@@ -93,16 +93,29 @@ export function detectLegacyConfig(obj: unknown): string[] {
 
 /**
  * How the evaluator's source was bound (or why it couldn't be). The SELECTION
- * contract (v0.2.1 — parity with orchestrator + diagnostics):
- *   bound            — exactly one source resolved (single entry · `default:true` · `--source` hit).
+ * contract (v0.2.2 — parity with orchestrator + diagnostics; F2 disambiguation UX):
+ *   bound            — exactly one source resolved with NO ambiguity (single entry ·
+ *                      `--source` hit). No prompt.
+ *   confirm-default  — >1 source with exactly one `default:true` (F2). The default is
+ *                      PRESELECTED (`source`) but the caller ISSUES A CONFIRM ASK before
+ *                      using it (distinct from a silent bind — be explicit around ambiguity).
+ *                      NON-fatal + bindable: `source` is usable as-is if the operator accepts
+ *                      or the run is non-interactive (`--source` / CI). Carries the full
+ *                      candidate list so the confirm can offer an override.
  *   none             — zero sources (DISCOVER needs a pick; code/dataset runs don't need a source).
  *   needs-selection  — >1 source, none `default:true`, no `--source` ⇒ the caller must PROMPT the
- *                      operator to pick (NON-fatal — the source EXISTS, only the pick is missing).
+ *                      operator to pick with NO preselection (NON-fatal — the source EXISTS,
+ *                      only the pick is missing).
  *   multiple-defaults— >1 source with `default:true` — an AMBIGUOUS config the operator must fix.
  *   unknown-name     — an explicit `--source <name>` that matched no catalog entry.
+ *
+ * confirm-default vs needs-selection: BOTH defer a run-time ASK, but confirm-default
+ * carries a PRESELECTED default (a yes/no confirm) whereas needs-selection has none (an
+ * open pick). Both are bindable-in-principle; the distinction drives the ASK shape.
  */
 export type SourceBinding =
   | { kind: "bound"; source: GlobalSource }
+  | { kind: "confirm-default"; source: GlobalSource; sources: GlobalSource[] }
   | { kind: "none" }
   | { kind: "needs-selection"; sources: GlobalSource[] }
   | { kind: "multiple-defaults"; sources: GlobalSource[] }
@@ -128,6 +141,17 @@ export interface EvaluatorConfigResolved {
   evaluatorContext: ContextLink[];
   /** present targets (evaluator is judge-only — carried for completeness only). */
   targets: GlobalTarget[];
+  /** PRD R-4 — the admin default render audience for the evaluation report. Client-by-default
+   *  (leak-safe); an admin sets `internal` in config.yaml to show the §5 Self-Eval tab. */
+  defaultAudience: "client" | "internal";
+}
+
+/** PRD R-4 — resolve the effective render audience: explicit flag → config → "client". */
+export function resolveEvalAudience(
+  explicit: "client" | "internal" | undefined,
+  config?: { defaultAudience?: "client" | "internal" } | undefined,
+): "client" | "internal" {
+  return explicit ?? config?.defaultAudience ?? "client";
 }
 
 export type LoadEvaluatorConfigResult =
@@ -142,14 +166,16 @@ export type LoadEvaluatorConfigResult =
  *
  *   1. explicit `selectName` (`--source`) WINS — find by name; miss ⇒ `unknown-name`.
  *   2. 0 sources                  ⇒ `none`.
- *   3. 1 source                   ⇒ `bound` (auto-bind).
+ *   3. 1 source                   ⇒ `bound` (auto-bind — no ambiguity, no prompt).
  *   4. >1 sources:
- *        exactly one `default:true` ⇒ `bound` (silent default).
+ *        exactly one `default:true` ⇒ `confirm-default` (F2 — preselected default; caller
+ *                                     CONFIRMS before use, not a silent bind).
  *        >1 `default:true`          ⇒ `multiple-defaults` (ambiguous — operator fixes config).
- *        0  `default:true`          ⇒ `needs-selection` (caller PROMPTS the operator).
+ *        0  `default:true`          ⇒ `needs-selection` (caller PROMPTS the operator — open pick).
  *
- * Pure; no clock/random/network. NON-fatal on `needs-selection` — the source
- * exists, only the disambiguation pick is deferred to the run-start ASK.
+ * Pure; no clock/random/network. NON-fatal on `confirm-default`/`needs-selection` —
+ * the source exists; only the disambiguation confirm/pick is deferred to the run-start ASK.
+ * `--source` (explicit) always short-circuits both, satisfying the non-interactive/CI escape.
  */
 function bindSourceByRole(
   sources: GlobalSource[],
@@ -167,7 +193,11 @@ function bindSourceByRole(
   if (sources.length === 1) return { kind: "bound", source: sources[0] as GlobalSource };
   // (4) — multiple entries; disambiguate by `default:true`.
   const defaults = sources.filter((s) => s.default === true);
-  if (defaults.length === 1) return { kind: "bound", source: defaults[0] as GlobalSource };
+  // F2: a lone default is PRESELECTED but CONFIRMED (be explicit around ambiguity) —
+  // it is NOT silently bound. The caller (parent session) issues a yes/no confirm ask.
+  if (defaults.length === 1) {
+    return { kind: "confirm-default", source: defaults[0] as GlobalSource, sources };
+  }
   if (defaults.length > 1) return { kind: "multiple-defaults", sources };
   return { kind: "needs-selection", sources };
 }
@@ -266,6 +296,11 @@ export function loadEvaluatorConfig(
     typeof evaluatorSection.judge_runtime === "string" && evaluatorSection.judge_runtime !== ""
       ? evaluatorSection.judge_runtime
       : undefined;
+  // PRD R-4 — admin audience override. Precedence: lifecycle.evaluator.default_audience >
+  // global.default_audience > "client". Only "internal" flips it; anything else stays client.
+  const rawAudience =
+    evaluatorSection.default_audience ?? globalBlock.default_audience;
+  const defaultAudience: "client" | "internal" = rawAudience === "internal" ? "internal" : "client";
 
   return {
     status: "ok",
@@ -278,6 +313,7 @@ export function loadEvaluatorConfig(
       globalContext,
       evaluatorContext,
       targets,
+      defaultAudience,
     },
   };
 }

package/.claude/skills/mutagent-evaluator/scripts/config/schema.ts

@@ -45,7 +45,12 @@ export const LEGACY_CONFIG_VERSIONS = ["0.1.0"] as const;
 
 // ── Categorical constants (no magic strings — coding-rules) ──────────────────
 
-/** Source platforms a `global.sources[]` entry can pull traces from. */
+/**
+ * Source platforms a `global.sources[]` entry can pull traces from.
+ * CONSUMER: the trace-fetch layer (`mutagent-cli trace fetch`, run by Helix BEFORE
+ * dispatch — the evaluator never fetches) selects the adapter by this value.
+ * PURPOSE: names WHERE the subject's traces live so they can be normalized to UniTF.
+ */
 export const SourcePlatform = {
   Langfuse: "langfuse",
   Otel: "otel",
@@ -56,12 +61,21 @@ export const SourcePlatform = {
 export type SourcePlatformValue =
   (typeof SourcePlatform)[keyof typeof SourcePlatform];
 
-/** Normalizer format hint for a source (how to parse the records). */
+/**
+ * Normalizer format hint for a source (how to parse the records).
+ * CONSUMER: the trace-fetch/normalize layer (`@mutagent/tools` adapters) — picks the
+ * parser for a source's records. PURPOSE: disambiguates record shape within a platform.
+ *   `unitf` (F1) — the records are ALREADY conformant UniTF jsonl (e.g. emitted by the
+ *   code-run capture WRAP, `captureUniTF` in `@mutagent/tools`). The fetch layer then
+ *   SKIPS per-platform normalization and reads the jsonl DIRECTLY. Pairs with
+ *   `platform: local-jsonl` + `paths:[…]`. PARITY PORT of the orchestrator's SourceFormat.
+ */
 export const SourceFormat = {
   LangfuseExport: "langfuse-export",
   ClaudeCode: "claude-code",
   Codex: "codex",
   Raw: "raw",
+  Unitf: "unitf",
 } as const;
 export type SourceFormatValue = (typeof SourceFormat)[keyof typeof SourceFormat];
 
@@ -137,7 +151,16 @@ export const ContextLinkSchema = Type.Object(
 );
 export type ContextLink = Static<typeof ContextLinkSchema>;
 
-/** The default model + the pinned judge model (`judge_model` — was `pinned_judge`). */
+/**
+ * The default model + the pinned judge model (`judge_model` — was `pinned_judge`).
+ * CONSUMER: config/load.ts `resolveJudgeModel` → prep.ts `--model` gap-fill.
+ *   default     — the project-wide fallback judge model. PURPOSE: used ONLY when
+ *                 `judge_model` is absent (`from: "default"`). ASYMMETRY: diagnostics
+ *                 IGNORES `models.default` for its judge (it pins its own) — the field
+ *                 is a shared config key that the two skills consume differently.
+ *   judge_model — the C-PIN pinned judge (model-intent-sacred; temp pinned 0). PURPOSE:
+ *                 the authoritative judge model; wins over `default` (`from: "judge_model"`).
+ */
 export const ModelsSchema = Type.Object(
   {
     default: Type.Optional(Type.String({ minLength: 1 })),
@@ -154,7 +177,10 @@ export type Models = Static<typeof ModelsSchema>;
  */
 export const GlobalSourceSchema = Type.Object(
   {
+    // name — the catalog key. CONSUMER: config/load.ts `bindSourceByRole` (matched by
+    // `--source <name>`) + prep.ts surface messages. PURPOSE: stable id for selection.
     name: Type.String({ minLength: 1 }),
+    // platform — see SourcePlatform. CONSUMER: the fetch/normalize adapter selector.
     platform: Type.Union([
       Type.Literal(SourcePlatform.Langfuse),
       Type.Literal(SourcePlatform.Otel),
@@ -162,19 +188,34 @@ export const GlobalSourceSchema = Type.Object(
       Type.Literal(SourcePlatform.ClaudeCode),
       Type.Literal(SourcePlatform.Codex),
     ]),
+    // project — remote-source project/scope id. CONSUMER: the remote fetch adapter
+    // (e.g. Langfuse project). PURPOSE: scopes a pull on a multi-project backend.
     project: Type.Optional(Type.String({ minLength: 1 })),
+    // endpoint — remote base URL. CONSUMER: the remote fetch adapter. PURPOSE: where to
+    // pull from (mutually exclusive with `paths`; onboarding enforces, schema tolerates).
     endpoint: Type.Optional(Type.String({ minLength: 1 })),
+    // credential_ref — env-var NAME / {env,path} (NEVER a secret). CONSUMER: resolve-
+    // credential.ts at fetch-time. PURPOSE: names the key for a remote pull.
     credential_ref: Type.Optional(CredentialRefSchema),
+    // paths — file-source globs. CONSUMER: the local-jsonl fetch adapter. PURPOSE: the
+    // on-disk jsonl location (pairs with `platform: local-jsonl`; see `format: unitf`).
     paths: Type.Optional(Type.Array(Type.String({ minLength: 1 }))),
+    // format — see SourceFormat. CONSUMER: the normalize step (`unitf` ⇒ skip normalize,
+    // read jsonl direct). PURPOSE: record-shape hint within a platform.
     format: Type.Optional(
       Type.Union([
         Type.Literal(SourceFormat.LangfuseExport),
         Type.Literal(SourceFormat.ClaudeCode),
         Type.Literal(SourceFormat.Codex),
         Type.Literal(SourceFormat.Raw),
+        Type.Literal(SourceFormat.Unitf),
       ]),
     ),
+    // agent_field — record path naming the agent/subject. CONSUMER: the normalizer's
+    // subject-grouping. PURPOSE: which field identifies the agent when a trace is multi-agent.
     agent_field: Type.Optional(Type.String({ minLength: 1 })),
+    // latency_unit — see LatencyUnit (`auto` = infer). CONSUMER: the normalizer's
+    // latency projection. PURPOSE: overrides ambiguous ms-vs-s latency records.
     latency_unit: Type.Optional(
       Type.Union([
         Type.Literal(LatencyUnit.Auto),
@@ -182,16 +223,23 @@ export const GlobalSourceSchema = Type.Object(
         Type.Literal(LatencyUnit.S),
       ]),
     ),
-    // PARITY (orchestrator SourceSchema): the catalog's DEFAULT pick when >1 entry
-    // exists — the selection contract's silent-bind marker (exactly one `default:true`
-    // ⇒ auto-bind, skipping the disambiguation prompt). Optional: absent ⇒ no default.
+    // default — PARITY (orchestrator SourceSchema): the catalog's DEFAULT pick when >1
+    // entry exists. CONSUMER: config/load.ts `bindSourceByRole` — exactly one `default:true`
+    // ⇒ `confirm-default` (F2: preselected, operator confirms), not a silent bind. PURPOSE:
+    // marks the preferred source among many. Optional: absent ⇒ no default.
     default: Type.Optional(Type.Boolean()),
   },
   { additionalProperties: false },
 );
 export type GlobalSource = Static<typeof GlobalSourceSchema>;
 
-/** A precise link to an Agent / Tooling definition file in a code target. */
+/**
+ * A precise link to an Agent / Tooling definition file in a code target.
+ * RESERVED — consumed by Build/Improve (future). The evaluator is JUDGE-ONLY (EV-051)
+ * and never writes a target; `targets[].code_refs` is ported for shape parity only and
+ * has NO evaluator consumer today. PURPOSE (future): tells BUILD/IMPROVE which files
+ * realize the agent so a fix lands on the right def.
+ */
 export const CodeRefSchema = Type.Object(
   {
     path: Type.String({ minLength: 1 }),
@@ -201,7 +249,14 @@ export const CodeRefSchema = Type.Object(
 );
 export type CodeRef = Static<typeof CodeRefSchema>;
 
-/** HOW a fix is applied for a target. `kind` gates the report-only branch. */
+/**
+ * HOW a fix is applied for a target. `kind` gates the report-only branch.
+ * CONSUMER: the diagnostics APPLY path (not the evaluator — judge-only). PURPOSE:
+ * selects the write mechanism for a fix.
+ *   versioning / pr — RESERVED, consumed by Build/Improve (future). No evaluator
+ *                     consumer; ported for parity. PURPOSE (future): toggle spec-bump /
+ *                     open-a-PR on apply.
+ */
 export const ApplySchema = Type.Object(
   {
     kind: Type.Union([

package/.claude/skills/mutagent-evaluator/scripts/load-bundle.ts

@@ -31,7 +31,7 @@ export const WELL_KNOWN_ARTIFACTS: Readonly<Record<string, string>> = {
   tracesMetadata: "traces-metadata.json",
   entityContext: "entity-context.json",
   renderInput: "render-input.json",
-  report: "report.html",
+  report: "evaluation-report.html",
 };
 
 /** Logical names whose VALUE is a sub-directory of evidence files. */

package/.claude/skills/mutagent-evaluator/scripts/render-discover-report.ts

@@ -1566,7 +1566,7 @@ export function writeDiscoverRunReport(
   return writeDiscoverReportFromFiles(
     {
       criteriaPath: join(dir, "criteria.json"),
-      outPath: join(dir, "report.html"),
+      outPath: join(dir, "discovery-report.html"),
       groundingPath: join(dir, "grounding-check.json"),
       triagePath: triageSummaryPath,
       verdictsDir: join(dir, "verdicts"),

package/.claude/skills/mutagent-evaluator/scripts/render-eval-report.ts

@@ -358,6 +358,9 @@ export interface CriterionResolution {
 
 export interface EvalReportInput {
   subject: { name: string; org?: string; source?: string; models?: string[] };
+  /** PRD R-4 — render audience. Default "client" (leak-safe): the §5 Self-Eval [INTERNAL]
+   *  tab + nav button are NOT emitted. "internal" (admin opt-in) shows them. */
+  audience?: "client" | "internal";
   /** the v2 *evaluate Scorecard (GATE + variance). */
   scorecard: Scorecard;
   /** the folded per-criterion verdicts. */
@@ -3479,6 +3482,16 @@ export function renderEvalReport(input: EvalReportInput): string {
   const theme = readFileSync(join(BRAND_DIR, "theme.css"), "utf8");
   const wordmark = readFileSync(join(BRAND_DIR, "wordmark.html"), "utf8");
 
+  // PRD R-4 — audience gate. CLIENT is the DEFAULT (leak-safe): the §5 Self-Eval [INTERNAL]
+  // tab + nav button are NOT emitted into the HTML at all (NODE-STRIP, not CSS-hidden).
+  const internal = (input.audience ?? "client") === "internal";
+  const selfEvalNav = internal
+    ? `  <button class="tab-btn internal" data-tab="t5">⑤ Self-Eval [INTERNAL]</button>\n`
+    : "";
+  const selfEvalPanel = internal
+    ? `  <section class="panel" id="t5">${selfEvalTab(input)}</section>\n`
+    : "";
+
   const headerTitle = "evaluator · Eval Report";
   const headerMeta =
     `<span class="mk">subject</span> <span class="mv">${esc(input.subject.name)}</span>` +
@@ -3564,15 +3577,13 @@ ${header}
   <button class="tab-btn" data-tab="t2">② Trajectory · Judge Behaviour</button>
   <button class="tab-btn" data-tab="t3">③ Eval Scorecard</button>
   <button class="tab-btn" data-tab="t4">④ Findings</button>
-  <button class="tab-btn internal" data-tab="t5">⑤ Self-Eval [INTERNAL]</button>
-</nav>
+${selfEvalNav}</nav>
 <main>
   <section class="panel active" id="t1">${overviewTab(input)}</section>
   <section class="panel" id="t2">${trajectoryTab(input)}</section>
   <section class="panel" id="t3">${scorecardTab(input)}</section>
   <section class="panel" id="t4">${findingsTab(input)}</section>
-  <section class="panel" id="t5">${selfEvalTab(input)}</section>
-</main>
+${selfEvalPanel}</main>
 <script data-pii="ledger">
 ${clientScript(ledgerJson, critJson, walksJson, resMetaJson)}
 </script>
@@ -3635,6 +3646,8 @@ export function writeEvalReportFromFiles(
      *  one trace off the raw ndjson and passes it here; the system prompt is identical
      *  across the batch so one trace suffices. ABSENT ⇒ system prompt stays UNAVAILABLE. */
     traces?: EvalTrace[];
+    /** PRD R-4 — render audience (default "client", leak-safe). "internal" shows §5. */
+    audience?: "client" | "internal";
   },
   io: EvalReportFilesIO,
 ): { report: string; completeness: EvalReportCompleteness } {
@@ -3746,6 +3759,8 @@ export function writeEvalReportFromFiles(
     // WS-6 — forward the trace(s) so the system prompt is reconstructed for the hero.
     ...(params.traces !== undefined ? { traces: params.traces } : {}),
   });
+  // PRD R-4 — set audience on the built input (buildEvalReportInput does not forward it).
+  if (params.audience !== undefined) input.audience = params.audience;
   io.writeFile(params.outPath, renderEvalReport(input));
 
   // 5) the data-completeness probe over the BOUND ledger (what the UI will render).
@@ -3803,9 +3818,9 @@ async function main(): Promise<void> {
   const { join: pjoin } = await import("node:path");
   const input = JSON.parse(rf(inputPath, "utf8")) as EvalReportInput;
   // P8: a bare run DEFAULTS the report under the localized dot-root
-  // `.mutagent/evaluator/reports/<runId>/report.html` (never /tmp or .memory).
+  // `.mutagent/evaluator/reports/<runId>/evaluation-report.html` (never /tmp or .memory).
   const runId = runIdArg ?? input.subject.name;
-  const outPath = outArg ?? pjoin(reportDir(runId), "report.html");
+  const outPath = outArg ?? pjoin(reportDir(runId), "evaluation-report.html");
   mkdirSync(pjoin(outPath, ".."), { recursive: true });
   console.info(renderEvalCards(input));
   writeFileSync(outPath, renderEvalReport(input));

package/.claude/skills/mutagent-evaluator/scripts/run-evaluate.ts

@@ -542,7 +542,7 @@ export function readRunVerdictFiles(input: EvaluateRunInput): MatrixVerdictFile[
  * REPORT — build the rich `EvalReportInput` from the REAL run outputs (the folded
  * scorecard + per-criterion verdicts + the per-trajectory Judge-Agent verdict
  * files + the matrix criteria + the EV-051 handover) and render the 5-tab HTML
- * eval-report into `<reportDir>/report.html`. This is the wiring D-2/D-3 had been
+ * eval-report into `<reportDir>/evaluation-report.html`. This is the wiring D-2/D-3 had been
  * MISSING: `*evaluate` produced a scorecard but emitted no report. The report
  * consumes the §9.4 judge-walk fields when present (side-by-side) and degrades to
  * the per-trajectory scorecard otherwise. Returns the written report path.
@@ -555,6 +555,9 @@ export function writeRunReport(
   result: EvaluateRunResult,
   runId: string,
   cwd?: string,
+  // PRD R-4 — render audience (resolved by the caller: explicit flag → config.default_audience →
+  // "client"). Omitted ⇒ the renderer defaults to "client" (leak-safe). "internal" shows §5.
+  audience?: "client" | "internal",
 ): string {
   const matrixVerdictFiles = readRunVerdictFiles(input);
   const reportInput = buildEvalReportInput({
@@ -586,9 +589,10 @@ export function writeRunReport(
       cPin: true,
     },
   });
+  if (audience) reportInput.audience = audience; // else renderEvalReport defaults to "client"
   const dir = reportDir(runId, cwd);
   mkdirSync(dir, { recursive: true });
-  const outPath = join(dir, "report.html");
+  const outPath = join(dir, "evaluation-report.html");
   writeFileSync(outPath, renderEvalReport(reportInput));
   return outPath;
 }

package/.claude/skills/mutagent-evaluator/workflows/audit.workflow.js

@@ -47,7 +47,7 @@ phase('Deterministic spine')
 const spine = await agent(
   `Run the DETERMINISTIC spine for the master audit. Use Bash:\n` +
   `  bash ${PKG}/scripts/cli/run.sh ${PKG}/scripts/cli/audit-run.ts ${RUN_ID} --subject ${SUBJECT} --bundle-dir ${BUNDLE_DIR} --subjects-root ${PKG}/subjects --generated-at ${GEN}\n` +
-  `This writes <bundle>/audit/scorecard.json (two-track GATE+TREND) + <bundle>/audit/master-audit.html (the base report). Return the parsed JSON the script prints (subject, runId, criteria, deterministicRows, judgeRows, gateRunPass, scorecard, report).`,
+  `This writes the master-audit under the unified evaluator reports root: .mutagent/evaluator/reports/<runId>/master-audit.html + master-audit.scorecard.json (two-track GATE+TREND). Return the parsed JSON the script prints (subject, runId, criteria, deterministicRows, judgeRows, gateRunPass, scorecard, report).`,
   { label: 'spine:audit-run', phase: 'Deterministic spine' },
 )
 log(`spine: ${spine && spine.criteria} criteria, ${spine && spine.deterministicRows} deterministic + ${spine && spine.judgeRows} judge rows; gateRunPass=${spine && spine.gateRunPass}`)

package/bin/mutagent-cli.mjs

@@ -5170,11 +5170,15 @@ var OperationIntent = {
   Evals: "evals",
   Dataset: "dataset"
 };
+var SourceFormat = {
+  Unitf: "unitf"
+};
 var QuerySourceSchema = Type.Object({
   endpoint_ref: Type.Optional(Type.String()),
   credential_ref: Type.Optional(Type.String()),
   paths: Type.Optional(Type.Array(Type.String())),
-  root: Type.Optional(Type.String())
+  root: Type.Optional(Type.String()),
+  format: Type.Optional(Type.Literal(SourceFormat.Unitf))
 }, { additionalProperties: false });
 var QueryFiltersSchema = Type.Object({
   sessionId: Type.Optional(Type.String()),
@@ -5449,6 +5453,7 @@ function parseNdjson(text) {
 // ../mutagent-tools/src/adapters/local-ndjson.ts
 function collect(ctx) {
   const src = ctx.query.source ?? {};
+  const passthrough = src.format === "unitf";
   const files = discoverFiles({ paths: src.paths, root: src.root });
   const traces = [];
   let malformed = 0;
@@ -5465,6 +5470,13 @@ function collect(ctx) {
     const parsed = parseNdjson(text);
     malformed += parsed.malformed;
     for (const rec of parsed.records) {
+      if (passthrough) {
+        if (validateUnifiedTrace(rec).ok)
+          traces.push(rec);
+        else
+          malformed++;
+        continue;
+      }
       const t = mapGenericRecord(rec, "local-ndjson");
       if (t)
         traces.push(t);
@@ -7560,6 +7572,16 @@ function buildAttributeBag(hit) {
   if (hit.error_message !== undefined && hit.error_message !== null && hit.error_message !== "") {
     bag["error.message"] = hit.error_message;
   }
+  const CC_CONTENT_COLS = ["user_prompt", "user_prompt_length", "tool_name", "full_command", "file_path", "tool_use_id"];
+  for (const c of CC_CONTENT_COLS) {
+    const v = hit[c];
+    if (v !== undefined && v !== null && v !== "" && bag[c] === undefined)
+      bag[c] = v;
+  }
+  const userPrompt = hit["user_prompt"];
+  if (typeof userPrompt === "string" && userPrompt !== "" && bag["gen_ai.prompt"] === undefined && bag["gen_ai.input.messages"] === undefined) {
+    bag["gen_ai.prompt"] = userPrompt;
+  }
   return bag;
 }
 function toOtelAttribute(key, value) {
@@ -7689,7 +7711,9 @@ var ASSISTANT_RESPONSE_KEYS = [
   "assistant_response",
   "completion",
   "output_text",
-  "message"
+  "message",
+  "text",
+  "content"
 ];
 function eventTimeToIso(v) {
   if (typeof v === "string" && v.length > 0) {
@@ -7816,6 +7840,23 @@ function eventToSpan(hit, sessionId, index) {
         span.attributes = { model };
       break;
     }
+    case "assistant_response": {
+      span.kind = "llm";
+      span.role = "assistant";
+      const response = firstStr(hit, ASSISTANT_RESPONSE_KEYS);
+      if (response !== undefined)
+        span.output = response;
+      const tok = eventTokens(hit);
+      if (tok !== undefined)
+        span.tokens = tok;
+      const cost = num3(hit.cost_usd);
+      if (cost !== undefined)
+        span.costUsd = cost;
+      const model = str3(hit.model);
+      if (model !== undefined)
+        span.attributes = { model };
+      break;
+    }
     case "tool_result": {
       span.kind = "tool";
       span.role = "tool";
@@ -8924,12 +8965,14 @@ function buildQuery(sub, positionals, flags) {
   const root = getStr(flags, "root");
   const endpointRef = getStr(flags, "endpoint-ref");
   const credRef = getStr(flags, "credential-ref");
-  if (paths || root || endpointRef || credRef) {
+  const unitfPassthrough = getStr(flags, "format") === "unitf";
+  if (paths || root || endpointRef || credRef || unitfPassthrough) {
     q.source = {
       ...paths ? { paths } : {},
       ...root ? { root } : {},
       ...endpointRef ? { endpoint_ref: endpointRef } : {},
-      ...credRef ? { credential_ref: credRef } : {}
+      ...credRef ? { credential_ref: credRef } : {},
+      ...unitfPassthrough ? { format: "unitf" } : {}
     };
   }
   const exportPath = getStr(flags, "export");
@@ -8988,6 +9031,7 @@ var FLAGS_HELP = [
   ["--limit <n>", "cap the result count"],
   ["--page-size <n> / --cursor <c>", "pagination"],
   ["--paths <file> / --root <dir>", "local source discovery (repeat --paths; --root pins a dir)"],
+  ["--format unitf", "local files are ALREADY UniTF \u2014 skip normalize, pass through (validate only)"],
   ["--endpoint-ref / --credential-ref", "env-var NAMES for a remote source (never raw secrets)"],
   ["--export <path.jsonl>", "write UniTF JSONL + <path>.manifest.json (the handover artifacts)"],
   ["--in <unitf.jsonl>", "`select`: UniTF JSONL input (repeatable; `--in -` reads stdin)"],

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mutagent/evaluator",
-  "version": "0.1.0-alpha.3",
+  "version": "0.1.0-alpha.4",
   "description": "mutagent-evaluator: a generic, subject-agnostic AI-agent auditor — reviewer, never executor. Audits any skill/agent against a generated subject profile and emits a 4-tab master-audit report.",
   "license": "Apache-2.0",
   "type": "module",
@@ -29,7 +29,7 @@
   },
   "dependencies": {
     "@sinclair/typebox": "^0.34.0",
-    "yaml": "^2.4.0"
+    "yaml": "^2.8.0"
   },
   "optionalDependencies": {
     "@langchain/google-genai": "^2.1.27"