@tangle-network/agent-eval 0.94.0 → 0.95.0

package/docs/pilot/integration-tangle-stack.md

@@ -1,248 +0,0 @@
-# Integration — Tangle Intelligence on the Tangle stack (sandbox + tcloud)
-
-Step-by-step. This is what we run with you on the onboarding call.
-
-## Zero-setup demo first (30 seconds, no install)
-
-```sh
-npx @tangle-network/intelligence demo
-```
-
-End-to-end loop against synthetic data — agent + judge + scenarios + selfImprove. Prints the `InsightReport` shape you'll get on your real data. Useful to confirm the output is what you want before any integration. Hosted equivalent: open **[staging-intelligence.tangle.tools](https://staging-intelligence.tangle.tools)**.
-
-## Prerequisites you already have
-
-- `@tangle-network/sandbox` running your agent in a session
-- `@tangle-network/tcloud` for LLM routing (or any OpenAI-compat router)
-- Your scenarios (the inputs your agent handles) listed somewhere — even as YAML or a TS array
-- A judge function for scoring outputs — LLM-as-judge is fine for v1
-
-## Install
-
-The CLI scaffolds and runs everything; you only add the substrate package if your code calls primitives directly:
-
-```sh
-# CLI (zero-install via npx, or add to your repo as a dev-dep)
-npx @tangle-network/intelligence init
-
-# Optional — only if your code imports analyzeRuns / selfImprove directly
-pnpm add @tangle-network/agent-eval
-# or for Python customers:
-pip install agent-eval-rpc
-```
-
-`@tangle-network/intelligence` is the customer-facing CLI + hosted product (binary `tangle-intel`). `@tangle-network/agent-eval` is the substrate it wraps — install only if you want to script directly against the primitives.
-
-## Step 1 — Ingest your trace stream
-
-You already emit traces via sandbox sessions. Pull them into canonical `RunRecord[]`:
-
-```ts
-import { fromTangleSandbox } from '@tangle-network/agent-eval/adapters/sandbox'
-
-const runs = await fromTangleSandbox({
-  sessionIds: ['session_abc', 'session_def'],   // your current week
-  fromMs: lastReportTime,
-  toMs: Date.now(),
-})
-// runs is RunRecord[] — canonical wire shape, ready for any downstream substrate primitive
-```
-
-If your agent emits OTel directly instead of going through `@tangle-network/sandbox`:
-
-```ts
-import { fromOtelSpans } from '@tangle-network/agent-eval'
-
-const runs = fromOtelSpans({ spans: yourOtelSpans })
-```
-
-## Step 2 — Get the decision packet (no LLM cost)
-
-```ts
-import { analyzeRuns } from '@tangle-network/agent-eval/contract'
-
-const report = await analyzeRuns({
-  runs: thisWeek,
-  baselineRuns: lastWeek,          // optional — gives you the "did my change help?" answer
-  baselineLabel: 'vs prior 7 days',
-})
-
-console.log(report.composite.mean)                       // overall score
-console.log(report.composite.tailRuns)                   // worst 5 runs by name
-console.log(report.priorPeriodComparison?.improvedMetrics)  // ['composite'] if significantly better
-console.log(report.priorPeriodComparison?.regressedMetrics) // ['cost'] if cost went up significantly
-console.log(report.recommendations)                      // priority-ranked actions
-```
-
-That's the **full deterministic flow** — no LLM, $0 cost, runs in ms.
-
-Render in your dashboard or pipe to Slack:
-
-```ts
-for (const rec of report.recommendations) {
-  if (rec.priority === 'critical') {
-    await slack.post(`🔴 ${rec.title}\n${rec.detail}`)
-  }
-}
-```
-
-## Step 3 — Wire the closed loop (real LLM cost — opt-in)
-
-Pick the surface you want to optimize. For most customers this is the agent's system-prompt addendum:
-
-```ts
-import { selfImprove, gepaDriver } from '@tangle-network/agent-eval/contract'
-
-const result = await selfImprove({
-  scenarios: yourScenarios,         // 20-50 representative inputs
-  agent: async (surface, scenario) => {
-    // Your existing agent invocation, with the substrate-proposed surface
-    // injected as the system-prompt addendum.
-    return await runYourAgent({
-      ...scenario,
-      systemPromptAddendum: surface as string,
-    })
-  },
-  judge: yourJudge,                  // function (artifact) → { composite, dimensions }
-  baselineSurface: currentAddendum,  // the production string today
-  driver: gepaDriver({
-    llm: { apiKey: tcloudKey, baseUrl: 'https://router.tangle.tools/v1' },
-    model: 'anthropic/claude-sonnet-4.6',
-    target: 'agent system-prompt addendum',
-  }),
-  budget: {
-    generations: 3,
-    populationSize: 4,
-    holdoutFraction: 0.3,
-    maxUsd: 25,                      // hard ceiling — refuses to overspend
-  },
-})
-
-console.log(`gate: ${result.gateDecision.kind}`)
-console.log(`lift: ${result.lift.delta.toFixed(3)} CI=[${result.lift.ci95.join(', ')}]`)
-console.log(`cost spent: $${result.totalCostUsd.toFixed(2)}`)
-```
-
-`result.gateDecision` is one of:
-- `ship-substrate` — winner statistically beats baseline; safe to deploy
-- `inconclusive` — CI straddles zero; either run more rollouts or expand corpus
-- `ship-harness` / `merge` — only when `driftPolicy: 'benchmark-branches'` is on (advanced)
-
-## Step 4 — Auto-PR the winner
-
-```ts
-if (result.gateDecision.kind === 'ship-substrate') {
-  await openAutoPr({
-    title: `eval: auto-improve ${target} (composite +${result.lift.delta.toFixed(3)})`,
-    body: `${result.gateDecision.reason}\n\n${formatInsight(result.insight)}`,
-    filePath: 'src/lib/.server/production-loop/prompt-addendum.ts',
-    newContent: result.diff.kind === 'replace' ? result.diff.content : applyDiff(currentAddendum, result.diff),
-  })
-}
-```
-
-We ship `openAutoPr` from `@tangle-network/agent-eval/contract`. It wraps the GitHub PR flow with your existing token.
-
-## The full canonical flow (script you copy and run)
-
-```ts
-// scripts/weekly-improvement.ts — run from a cron / GitHub Action
-
-import { fromTangleSandbox } from '@tangle-network/agent-eval/adapters/sandbox'
-import {
-  analyzeRuns,
-  gepaDriver,
-  openAutoPr,
-  selfImprove,
-} from '@tangle-network/agent-eval/contract'
-import { scenarios } from './eval/scenarios'
-import { judge } from './eval/judges'
-import { runYourAgent } from './src/agent'
-import { PRODUCTION_ADDENDUM } from './src/lib/.server/production-loop/prompt-addendum'
-
-const lastWeek = Date.now() - 7 * 24 * 60 * 60 * 1000
-const twoWeeksAgo = lastWeek - 7 * 24 * 60 * 60 * 1000
-
-const thisWeekRuns = await fromTangleSandbox({ fromMs: lastWeek, toMs: Date.now() })
-const lastWeekRuns = await fromTangleSandbox({ fromMs: twoWeeksAgo, toMs: lastWeek })
-
-// 1. Deterministic packet — always
-const report = await analyzeRuns({
-  runs: thisWeekRuns,
-  baselineRuns: lastWeekRuns,
-  baselineLabel: 'vs prior 7 days',
-})
-
-// 2. Closed loop — only if composite regressed OR we haven't tried in a while
-const shouldRun =
-  report.priorPeriodComparison?.regressedMetrics.includes('composite') ||
-  daysSinceLastImprovement() > 7
-
-if (!shouldRun) {
-  console.log('No regression + recent run; skipping.')
-  process.exit(0)
-}
-
-const result = await selfImprove({
-  scenarios,
-  agent: (surface, scenario) =>
-    runYourAgent({ ...scenario, systemPromptAddendum: surface as string }),
-  judge,
-  baselineSurface: PRODUCTION_ADDENDUM,
-  driver: gepaDriver({
-    llm: { apiKey: process.env.TANGLE_KEY!, baseUrl: 'https://router.tangle.tools/v1' },
-    model: 'anthropic/claude-sonnet-4.6',
-    target: 'production agent system-prompt addendum',
-  }),
-  budget: { generations: 3, populationSize: 4, holdoutFraction: 0.3, maxUsd: 50 },
-})
-
-if (result.gateDecision.kind === 'ship-substrate') {
-  await openAutoPr({
-    title: `eval: auto-improve addendum (composite +${result.lift.delta.toFixed(3)})`,
-    body: renderInsightAsPrBody(result.insight),
-    filePath: 'src/lib/.server/production-loop/prompt-addendum.ts',
-    newContent: result.diff.kind === 'replace' ? result.diff.content : '...',
-  })
-}
-```
-
-## What we'll do together on the onboarding call
-
-1. **Map your existing setup** — where do your traces emit? which sandbox sessions? which scenarios exist already?
-2. **Stub the judge** — even a single dimension is enough to start
-3. **Run a deterministic `analyzeRuns()` against your live data** — first decision packet rendered live
-4. **Wire one selfImprove cycle** — small budget, single generation, see the loop fire
-5. **Schedule the cron + auto-PR target** — the loop runs autonomously thereafter
-
-Time budget: ~90 minutes. By the end you have a working pilot.
-
-## What our hosted tier adds on top
-
-- Decision packet rendered weekly in the Intelligence dashboard — no code changes needed
-- Slack / email digest on `regressedMetrics`
-- Pareto chart, judge calibration, failure-cluster drilldown in the UI
-- Multi-week trend lines
-- Stripe-billed usage tracking
-
-If you want self-hosted only, every primitive above works locally. The hosted tier is a convenience.
-
-## FAQ
-
-**Q: What's the smallest scenario corpus that gives useful results?**
-A: ~15 scenarios for the deterministic packet (you get distributional stats + recommendations). For `selfImprove`'s held-out gate you want ≥20 since `holdoutFraction: 0.3` reserves 6 for the gate. Below that, the gate often returns `inconclusive`.
-
-**Q: What if my judge isn't reliable yet?**
-A: That's normal. Use multi-rater intake (`fromFeedbackTable`) to get inter-rater agreement (κ) first, then iterate on the judge until raters agree. Substrate has an `interRater` block in InsightReport showing exactly which scenarios raters disagree on.
-
-**Q: What if a `selfImprove` campaign returns `inconclusive`?**
-A: It refused to claim improvement because the CI straddles zero. Either expand the corpus, raise `holdoutFraction`, or run more generations. Better than shipping noise.
-
-**Q: Can I use a non-tcloud LLM provider?**
-A: Yes — `gepaDriver` accepts any `LlmClientOptions` (any OpenAI-compatible endpoint). We default to tcloud because we already have your auth.
-
-**Q: How do I see what changed when the gate ships?**
-A: `result.diff` is a structured patch. We also ship `diffRuns()` separately if you want to compare two campaign outputs.
-
-**Q: What if my agent self-modifies (Hermes / Claude Code skills)?**
-A: This is the offline/online drift case. We have the architecture spec ready (`docs/specs/profile-versioning.md`) but the implementation is gated on a forcing-function experiment. For v0.5x pilots we assume the substrate is the only writer to your agent's optimizable surface.

package/docs/pilot/one-pager.md

@@ -1,161 +0,0 @@
-# Statistical self-improvement for your agent — one-pager
-
-**For:** teams running an agent on the Tangle stack (sandbox + tcloud), OR any agent emitting OTel traces, OR LangChain / LlamaIndex / Anthropic SDK / OpenAI Assistants / OpenRouter / custom — we meet you where you are.
-**The pitch:** every week, get a statistically-rigorous answer to *"did my last change help?"* + a closed loop that proposes the next improvement + a held-out gate that refuses to ship regressions.
-
-## What you get
-
-| Deliverable | Cadence | LLM cost |
-|---|---|---|
-| **Decision packet** — composite distribution, per-dimension judges, cost-quality Pareto, failure clusters, named worst-N runs, ranked recommendations | Whenever you want it. Hosted runs on a 15-min schedule by default. | $0 (deterministic) |
-| **Prior-period comparison** — Welch CI on composite / cost / duration / per-dimension deltas vs your prior week, with regressed + improved metrics named | Same cadence | $0 |
-| **Closed-loop improvement** — `selfImprove()` proposes prompt edits, runs scenarios, gates on paired-bootstrap CI, auto-PRs the winner | On-demand, opt-in | Real $; you set a `maxUsd` ceiling |
-
-Every claim is falsifiable: `n=`, `CI95=[a, b]`, `p=`, `Cohen's d=`. No vibes, no "score went up." Where the data doesn't support a section, the report says so explicitly instead of inventing signal.
-
-## Why this is different
-
-| | LangSmith / Braintrust / Phoenix | Hermes / Claude Code skills | **Tangle** |
-|---|---|---|---|
-| Trace ingest | proprietary | own runtime | universal (sandbox + tcloud + OTel + any custom) |
-| Decision packet | scorecards (no CI) | none | **paired-bootstrap CI on every claim** |
-| Closed loop | none | heuristic, no gate | **statistically-gated; refuses regressions** |
-| Prior-period delta | none | none | **Welch CI on every metric** |
-| Sample-size guidance | none | none | **MDE-aware** |
-| Auto-PR promotion | none | none | **opt-in, on green gate only** |
-
-## Integration paths — pick your stack
-
-| Your stack | Intake adapter | LLM provider for closed loop |
-|---|---|---|
-| **Tangle (sandbox + tcloud)** | `fromTangleSandbox` | tcloud (already wired) |
-| Any OTel exporter (Datadog APM, Honeycomb, NewRelic, OpenInference) | `fromOtelSpans` | any OpenAI-compat |
-| LangChain (LangSmith) | LangSmith → OTel export → `fromOtelSpans` today; `fromLangChain` queued 0.55.0 | OpenAI, Anthropic, OpenRouter, tcloud |
-| LlamaIndex | `OpenInferenceCallbackHandler` → OTel → ingest | any OpenAI-compat |
-| Anthropic SDK direct | OTel wrapping (~20 LOC) → `fromOtelSpans`; `fromAnthropicSDK` queued | Anthropic, OpenRouter |
-| OpenAI Assistants API | Custom mapper (~20 LOC) today; `fromOpenAIAssistants` queued | OpenAI, OpenRouter |
-| OpenRouter (any model on any path) | Whatever you already use for tracing | OpenRouter (OpenAI-compat baseUrl) |
-| vLLM / Ollama / LMStudio / self-hosted | OTel wrap | Your local OpenAI-compat endpoint |
-| Multi-rater human feedback (no automated judge yet) | `fromFeedbackTable` | n/a — gives you κ + disagreement triage |
-| Custom logs / DB rows | ~20-line mapper to `RunRecord` | any OpenAI-compat |
-
-Full integration walkthroughs:
-- **Tangle stack** → [`integration-tangle-stack.md`](./integration-tangle-stack.md)
-- **Everything else** → [`integration-foreign-stack.md`](./integration-foreign-stack.md)
-
-## Zero-setup demo first — 30 seconds
-
-Before any integration, run the demo against synthetic data so you see the output shape live:
-
-```sh
-npx @tangle-network/intelligence demo
-```
-
-No install, no key, no data. Synthetic agent runs through synthetic scenarios; the CLI prints a real `InsightReport` with composite distribution + Pareto + prior-period delta + ranked recommendations. Same output shape you'll get on your real data once we integrate.
-
-When you're ready to integrate, the same CLI scaffolds your repo:
-
-```sh
-npx @tangle-network/intelligence init           # creates eval/scenarios.json + judges.ts + pnpm scripts + .runs/
-npx @tangle-network/intelligence report          # renders InsightReport from your latest traces
-npx @tangle-network/intelligence improve --max-usd 25    # runs selfImprove with cost ceiling, opens auto-PR on green gate
-```
-
-Hosted equivalent: **[staging-intelligence.tangle.tools](https://staging-intelligence.tangle.tools)** — open in your browser, ingest your traces, see the dashboard render the same packet your CLI produces.
-
-## How you integrate (Tangle stack — 4 steps)
-
-```ts
-import { fromTangleSandbox } from '@tangle-network/agent-eval/adapters/sandbox'
-import { analyzeRuns, selfImprove, gepaDriver } from '@tangle-network/agent-eval/contract'
-
-// 1. You already emit traces via @tangle-network/sandbox + tcloud.
-//    Pull them into canonical RunRecord[]:
-const runs = fromTangleSandbox({ sessionId, sinceMs: lastReportTime })
-
-// 2. Get the decision packet — no LLM cost.
-const report = await analyzeRuns({ runs, baselineRuns: priorWeekRuns })
-//    → report.composite + .priorPeriodComparison + .recommendations
-
-// 3. When you want to actually improve, run the closed loop.
-const result = await selfImprove({
-  scenarios: yourScenarios,                     // we help you build these
-  agent: (surface, scenario) => runYourAgent(scenario, surface),
-  judge: yourJudge,                              // any function (artifact) → JudgeScore
-  baselineSurface: currentSystemPrompt,
-  driver: gepaDriver({ llm: tcloud, model: 'claude-sonnet-4.6', target: 'agent prompt' }),
-  budget: { generations: 3, populationSize: 4, holdoutFraction: 0.3, maxUsd: 25 },
-})
-
-// 4. Result is a verifiable diff with statistical evidence.
-//    Auto-PR if result.gateDecision === 'ship-substrate'.
-```
-
-That's it. ~30 lines of integration code; the rest is your existing agent + tcloud setup.
-
-## What we need from you
-
-- API key for tcloud (you already have this)
-- Read access to your sandbox session traces
-- A list of 20-50 representative scenarios your agent should handle
-- A judge function — even a simple LLM-as-judge gets you 80% of the value
-- An LLM-cost budget for the closed loop (default: $25/campaign)
-
-## What you ship back to your customers
-
-The substrate produces a single JSON `InsightReport` your dashboard renders. Live demo embedded in the Tangle Intelligence dashboard. Example below — every section optional based on what your data supports.
-
-```json
-{
-  "n": 36,
-  "composite": {
-    "mean": 0.823, "p50": 0.85, "p95": 0.96, "stddev": 0.11,
-    "tailRuns": [
-      { "runId": "scenario::checkout-bug", "score": 0.41 },
-      { "runId": "scenario::refund-policy", "score": 0.48 }
-    ]
-  },
-  "priorPeriodComparison": {
-    "baselineN": 34,
-    "currentN": 36,
-    "windowLabel": "vs prior 7 days",
-    "metrics": {
-      "composite": {
-        "current": 0.823, "baseline": 0.731, "delta": 0.092,
-        "ci95": [0.041, 0.143], "pValue": 0.0008,
-        "cohensD": 0.84, "significant": true
-      }
-    },
-    "improvedMetrics": ["composite"],
-    "regressedMetrics": []
-  },
-  "recommendations": [
-    {
-      "priority": "low",
-      "kind": "ship",
-      "title": "composite improved from 0.731 → 0.823 vs prior 7 days",
-      "detail": "Welch CI95=[0.041, 0.143], p=0.0008, Cohen's d=0.84 (n_current=36, n_baseline=34). Statistically significant improvement worth flagging."
-    },
-    {
-      "priority": "high",
-      "kind": "investigate",
-      "title": "Top failure cluster: refund-policy (12% of failures)",
-      "detail": "4 runs failed. Largest cluster groups by intent — agent missed compliance flag in 3 of 4."
-    }
-  ]
-}
-```
-
-## Pricing for the pilot
-
-- Free for the first 30 days
-- Hosted decision-packet generation: included
-- LLM cost on closed-loop campaigns: pass-through to your tcloud account
-- Post-pilot: per-campaign pricing tied to budget cap + per-decision-packet billed monthly
-
-## Next step
-
-Reply with: which agent + which week you want to start, and we'll set up the integration on a shared call. ~1 hour to first running report.
-
-—
-*Tangle Network · @tangle-network/agent-eval @0.53.0 · MIT · Self-hostable*

package/docs/pilot/sample-insight-report.json

@@ -1,172 +0,0 @@
-{
-  "n": 36,
-  "composite": {
-    "n": 36,
-    "mean": 0.823,
-    "p50": 0.85,
-    "p95": 0.96,
-    "stddev": 0.114,
-    "min": 0.41,
-    "max": 0.98,
-    "tailRuns": [
-      { "runId": "scenario::checkout-bug", "score": 0.41 },
-      { "runId": "scenario::refund-policy-edge", "score": 0.48 },
-      { "runId": "scenario::multi-tenant-isolation", "score": 0.52 },
-      { "runId": "scenario::stale-cache-invalidation", "score": 0.55 },
-      { "runId": "scenario::partial-payment", "score": 0.61 }
-    ],
-    "histogram": [
-      { "lo": 0.40, "hi": 0.46, "count": 1 },
-      { "lo": 0.46, "hi": 0.51, "count": 1 },
-      { "lo": 0.51, "hi": 0.57, "count": 1 },
-      { "lo": 0.57, "hi": 0.62, "count": 1 },
-      { "lo": 0.62, "hi": 0.68, "count": 2 },
-      { "lo": 0.68, "hi": 0.74, "count": 2 },
-      { "lo": 0.74, "hi": 0.80, "count": 4 },
-      { "lo": 0.80, "hi": 0.85, "count": 9 },
-      { "lo": 0.85, "hi": 0.91, "count": 8 },
-      { "lo": 0.91, "hi": 0.96, "count": 6 },
-      { "lo": 0.96, "hi": 1.0, "count": 1 }
-    ]
-  },
-  "perDimension": {
-    "intent-recognition": {
-      "n": 36, "mean": 0.89, "p50": 0.92, "p95": 0.98, "stddev": 0.08
-    },
-    "compliance-flagging": {
-      "n": 36, "mean": 0.71, "p50": 0.75, "p95": 0.92, "stddev": 0.18
-    },
-    "tone": {
-      "n": 36, "mean": 0.94, "p50": 0.95, "p95": 0.99, "stddev": 0.04
-    }
-  },
-  "costQuality": {
-    "cost": {
-      "n": 36, "mean": 0.087, "p50": 0.082, "p95": 0.124, "stddev": 0.021
-    },
-    "pareto": {
-      "kind": "pareto-cost-quality",
-      "split": "holdout",
-      "axes": { "x": "costUsd", "y": "score" },
-      "points": [
-        { "candidateId": "baseline-v3.1", "cost": 0.087, "quality": 0.823, "n": 36, "onFrontier": true }
-      ]
-    }
-  },
-  "judges": {
-    "claude-sonnet-4.6": {
-      "n": 36,
-      "meanScore": 0.831,
-      "calibration": null
-    }
-  },
-  "lift": null,
-  "failureClusters": {
-    "totalFailures": 4,
-    "clusters": [
-      {
-        "id": "cluster_refund_compliance",
-        "name": "refund-policy missed compliance flag",
-        "share": 0.75,
-        "exemplars": [
-          "scenario::refund-policy-edge",
-          "scenario::partial-payment",
-          "scenario::cross-border-refund"
-        ],
-        "suggestedFix": "Add explicit step to the addendum: when refund amount > $100 OR cross-border, surface compliance flag before responding."
-      }
-    ]
-  },
-  "priorPeriodComparison": {
-    "baselineN": 34,
-    "currentN": 36,
-    "windowLabel": "vs prior 7 days",
-    "metrics": {
-      "composite": {
-        "current": 0.823,
-        "baseline": 0.731,
-        "delta": 0.092,
-        "ci95": [0.041, 0.143],
-        "pValue": 0.0008,
-        "cohensD": 0.84,
-        "baselineN": 34,
-        "currentN": 36,
-        "significant": true
-      },
-      "cost": {
-        "current": 0.087,
-        "baseline": 0.082,
-        "delta": 0.005,
-        "ci95": [-0.003, 0.013],
-        "pValue": 0.21,
-        "cohensD": 0.23,
-        "baselineN": 34,
-        "currentN": 36,
-        "significant": false
-      },
-      "duration": {
-        "current": 4820,
-        "baseline": 5340,
-        "delta": -520,
-        "ci95": [-840, -200],
-        "pValue": 0.002,
-        "cohensD": -0.71,
-        "baselineN": 34,
-        "currentN": 36,
-        "significant": true
-      },
-      "dim.compliance-flagging": {
-        "current": 0.71,
-        "baseline": 0.58,
-        "delta": 0.13,
-        "ci95": [0.06, 0.20],
-        "pValue": 0.0004,
-        "cohensD": 0.79,
-        "baselineN": 34,
-        "currentN": 36,
-        "significant": true
-      }
-    },
-    "improvedMetrics": ["composite", "duration", "dim.compliance-flagging"],
-    "regressedMetrics": []
-  },
-  "release": {
-    "status": "pass",
-    "axes": [
-      { "name": "quality-lift", "status": "pass", "detail": "no candidate/baseline pair within campaign; relying on priorPeriodComparison" },
-      { "name": "contamination", "status": "pass", "detail": "no canaries supplied" },
-      { "name": "composite-distribution", "status": "pass", "detail": "mean=0.823, p50=0.85, p95=0.96 over n=36" }
-    ],
-    "issues": []
-  },
-  "recommendations": [
-    {
-      "priority": "low",
-      "kind": "ship",
-      "title": "composite improved from 0.731 → 0.823 vs prior 7 days",
-      "detail": "Welch CI95=[0.041, 0.143], p=0.0008, Cohen's d=0.84 (n_current=36, n_baseline=34). Statistically significant improvement worth flagging.",
-      "evidencePath": "priorPeriodComparison.metrics.composite"
-    },
-    {
-      "priority": "low",
-      "kind": "ship",
-      "title": "compliance-flagging dimension improved from 0.58 → 0.71",
-      "detail": "Welch CI95=[0.06, 0.20], p=0.0004, Cohen's d=0.79. The fix from last week's PR is statistically validated.",
-      "evidencePath": "priorPeriodComparison.metrics.dim.compliance-flagging"
-    },
-    {
-      "priority": "high",
-      "kind": "investigate",
-      "title": "Top failure cluster: refund-policy missed compliance flag (75% of failures)",
-      "detail": "3 of 4 failed runs cluster here. Suggested fix: add explicit step to addendum for refund > $100 OR cross-border → surface compliance flag.",
-      "evidencePath": "failureClusters.clusters[0]"
-    },
-    {
-      "priority": "low",
-      "kind": "ship",
-      "title": "duration improved from 5340ms → 4820ms vs prior 7 days",
-      "detail": "Welch CI95=[-840, -200]ms, p=0.002, Cohen's d=-0.71. Agent is meaningfully faster — worth keeping the optimization that drove this.",
-      "evidencePath": "priorPeriodComparison.metrics.duration"
-    }
-  ]
-}