@tangle-network/agent-app 0.47.25 → 0.48.2

package/.claude/skills/eval-architect/SKILL.md

@@ -1,44 +1,47 @@
 ---
 name: eval-architect
-description: Build a measurement that scores an agent's REAL deliverable — not a proxy — for a product you've never seen before. Use when scaffolding or repairing the eval an Improve loop optimizes against. Get this wrong and every downstream optimization perfects a fiction.
+description: Build or repair evaluations that score the agent's actual deliverable through the production path, with independent controls and visible missing evidence.
 ---
 
-# Eval Architect — measure the real deliverable
+# Eval architect
 
-You are building the measurement an improvement loop will optimize against. **The loop optimizes whatever you measure.** If you measure the wrong thing, the loop perfects the wrong thing — confidently, expensively, and invisibly. The measurement is the product. Everything else in the Improve stack is downstream of getting this right.
+Build the measurement around the product's required outcome and actual execution path.
+Reuse maintained Eval contracts and existing product checks before creating another scorer.
 
-This skill is held by the agent that *builds* the eval (often a delegated coding agent). Pair it with `measurement-validation` (the gate that proves your eval is sound before anyone spends money on it).
+## Locate the deliverable
 
-## The cardinal question
+1. Inspect real runs to find the output channel: replies, validated tool calls, persisted artifacts, application state, or rendered UI.
+   Score the channel that carries the required outcome.
+2. Define the completion boundary for the task.
+   For work that accumulates across turns, evaluate the completed artifact and retain intermediate evidence needed to explain failures.
+3. Trace every consumer of the score, including completion checks, optimization selection, and release decisions.
+   When the output channel changes, update every affected consumer.
 
-**Where does this agent's deliverable actually land?** Prose in the reply? Validated tool calls? Persisted artifacts (vault docs, DB rows)? A PR? A rendered UI? Find out by inspecting *real runs* — never by assuming it's the chat text.
+## Build the checks
 
-> Worked failure (legal-agent, this is why the skill exists): the eval scored the assistant's chat prose. A tool-migration moved the deliverable into `submit_proposal` calls + vault docs, leaving the prose empty. Every scorer reading prose silently collapsed to ~0. The loop would have optimized an empty string. The deliverable had *moved* and the measurement didn't follow it.
+1. Map each requirement to observable evidence and an explicit failure condition.
+   Use answer keys when available; otherwise use independently justified constraints, executable checks, or calibrated judgment.
+   Keep unsupported requirements and missing evidence visible.
+2. Establish a simple baseline through the same entrypoint as the candidate.
+   Investigate surprising scores instead of assuming either the scorer or the agent caused them.
+3. Separate training, candidate selection, and final comparison evidence where the improvement claim requires those partitions.
+   Preserve scenario identities and shared source-unit mappings across baseline and candidate runs.
+4. Define critical failure checks separately from aggregate quality.
+   A favorable composite must not erase a failure that violates the product's requirements.
+5. Preserve scorer identity, actual cost receipts, execution failures, and diagnostic artifacts.
+   Change `judgeVersion` when an ensemble scorer's configuration changes.
 
-## Invariant (non-negotiable — violate these and the loop is a slot machine)
+## Prove the measurement
 
-1. **Score the produced artifact, not the conversation.** Locate the real output channel and score *that*.
-2. **For accumulating-artifact agents, score the CONVERGED multi-shot artifact, not turn 1.** Most real agents build their deliverable over several turns. Define a convergence criterion (e.g. the artifact stops growing for N shots) and score the converged state.
-3. **A held-out split exists and is never trained on.** No held-out → no honest gate → no trustworthy lift.
-4. **Every requirement has gold the scorer matches against, from real records — never fabricated.** A requirement with no gold means there is nothing to verify; fail loud, do not pass-by-default. A fluent hallucination that produced nothing must score 0, not 0.9.
+Run known positive and negative examples through the complete scoring path.
+Perturb a real deliverable so required behavior improves or regresses, then check that the score detects each change.
+Confirm that absent output and evaluator failure remain distinguishable from measured low quality.
+Report case coverage, detectable failures, uncertainty, and any requirements the evaluation cannot assess.
+A training gain without a final-comparison gain needs diagnosis; it does not identify the cause by itself.
 
-## Judgment (figure this out per product — the agentic core)
+## Then consider
 
-- What *is* the deliverable here, and where does it persist? Read the runtime events / tool calls / storage, not the transcript.
-- What is the convergence criterion for this agent's artifact? When has it stopped accumulating?
-- What gold defines "correct" for each requirement, and where does it come from (real records, never invented figures)?
-- Which dimensions matter, and what are their weights? What is the one dimension that, if it regresses, kills the deal regardless of the composite (safety, hallucination, the regulated invariant)?
-
-## Self-test (prove the metric works before trusting it)
-
-- **Baseline sanity:** run it. Is the score non-zero and plausible for a competent agent? A near-zero baseline usually means you're scoring the wrong channel, not that the agent is terrible.
-- **The mutation test (the one that catches the empty-string bug):** hand-edit the produced artifact to be *obviously better* and *obviously worse*. Does the score move in the right direction and magnitude? A metric that doesn't move under obvious changes is measuring the wrong thing.
-- **Audit EVERY scoring surface together.** Completion, quality, and the optimizer's own scorer all read *something*. When the deliverable's channel moves, all of them that read the old channel silently zero. (Session: completion + quality were fixed; the optimizer's own scorer was missed and only found by tracing. Three surfaces — enumerate them, don't assume one.)
-
-## Evolves-by
-
-When a later optimization shows lift on *training* but none on *held-out*, your eval was overfittable or gameable — add the gap it missed as a new judgment rule. The architect's judgment surface is itself optimized by the meta-eval *"did evals built this way yield real held-out lift, no critical regression?"* See `skill-evolution`.
-
-## Fleet as dogfood
-
-legal / tax / gtm / creative / insurance each put their deliverable in a *different* channel — filings, forms, published copy, rendered artifacts, routed proposals. The skill is general precisely because it forces you to *locate* the channel for the product in front of you rather than hardcode "the reply text."
+| Condition | Skill |
+|---|---|
+| The evaluation path executes and needs calibration or comparison checks | `measurement-validation` with the baseline and control results |
+| The validated measurement supports a candidate search | `surface-evolution` with the target surface, evidence, and resource limits |

package/.claude/skills/improve-conductor/SKILL.md

@@ -1,45 +1,50 @@
 ---
 name: improve-conductor
-description: The user-facing controller for the Improve button. Decide whether a request is improvable, translate a dollar budget into a run, read the verdict honestly, and promote or refuse with a reason. Never promise a lift you cannot measure.
+description: Drive a requested optimization from its target and resource limits through measured candidate selection, evidence review, and the product's promotion policy.
 ---
 
-# Improve Conductor — own the user's trust
-
-You are the agent the user talks to when they click **Improve**. You do not build the eval or run the loop yourself — you decide *whether* to, *how much* to spend, and *what to tell the user about the result*. The product you are protecting is **trust**, not lift. You would rather say "I could not prove an improvement — here is what another $X buys" than ship noise and call it a win.
-
-You delegate the building to an agent holding `eval-architect` + `surface-evolution`; you both share `measurement-validation` as the honesty contract.
-
-## Invariant (non-negotiable)
-
-1. **Never promise or report a lift you cannot measure with valid paired evidence.** Surface the honest verdict: `ship` / `hold` / `need-more-data` / `invalid`. A paired, significant lift is still not shippable until a footprint-matched placebo shows the gain comes from the CONTENT, not from added prompt/mount footprint — the substrate's `neutralizationGate` / `runImprovementLoop({ neutralize })` (`@tangle-network/agent-eval/campaign`). A lift that a neutralized twin reproduces is footprint, not improvement — refuse it. Route this check through `measurement-validation`. "Invalid" (incomplete or unpaired evidence) is a first-class outcome — say it plainly, never paper over it with a survivor-mean number.
-2. **Refuse below the data threshold, and say why** — "I have N real outcomes; I won't optimize below M. Here's how to get to M." A refusal with a reason builds more trust than a fabricated win.
-3. **Route correctly.** Improvable by surface-tuning → dispatch `surface-evolution`. Needs a new capability or architecture → escalate and say so; don't pretend tuning will fix a structural gap.
-4. **No optimization spend before the target is confirmed and the measurement is real.** If there is no improvement infrastructure yet, you do NOT improvise a metric and start spending — you dispatch `eval-bootstrap` to BUILD a validated, externally-grounded harness first. The gate between "build the apparatus" and "spend optimizing" is yours to hold.
-
-## Cold start — no infrastructure yet
-
-The most dangerous request is "improve this" for a product with no eval. The wrong move is to invent a metric and start a loop — you'll perfect a proxy and report a fake win. The right move is a strict two-step you orchestrate:
-
-1. **Frame + build (no spend):** confirm with the user *what "better" means* — the thing they'd reject a draft over, tied to a product-value claim — then dispatch `eval-bootstrap` (often a delegated agent-runtime build loop) to construct a harness grounded in **external truth**, exiting only when `measurement-validation` passes. The improver is a *builder* here, not a tuner.
-2. **Then optimize (spend):** only once the harness is validated, dispatch `surface-evolution` against it.
-
-Never let the user believe step 2 happened when only a toy of step 1 did. If you can't yet build a real measurement (no grounding, target unclear), say so and ask for what you need — that's the honest move, not a loop against an invented number.
-
-## Judgment (figure this out per request)
-
-- Is this a surface-tuning problem or an architectural one? (If the agent literally cannot do the task, no prompt edit fixes it.)
-- Translate the user's dollars into a run: more spend = wider candidate search + more reps = tighter CI + higher chance of clearing the gate. $0.20 ≈ one quick generation on a couple scenarios; $50 ≈ multi-generation search with a held-out gate that can actually reach significance.
-- When to stop: threshold met, plateaued, or budget exhausted — and report which.
-
-## Self-test
-
-- **Before spending,** you can state out loud: the metric, its variance, the threshold, the held-out set, and what this budget buys. If you can't, you're not ready to charge for the click.
-- **After,** you report the gated lift with its CI and the decision's *reason*. If the run came back `invalid` (a cell errored, evidence unpaired), you tell the user that and offer the re-run — you do not quote the broken number.
-
-## Evolves-by
-
-User accept/reject of promotions; spend→lift efficiency; the rate of `invalid` runs. A rising invalid rate is a signal the measurement or the infra needs hardening — route it back to `measurement-validation` / `eval-architect`, don't absorb it silently. See `skill-evolution`.
-
-## Why this is calibrated, not timid
-
-A naive Improve button maximizes the displayed number and tells the user "improved +47%". The disciplined one, faced with the same +47, checks the evidence, finds it unpaired, and says "I found a promising candidate but can't yet prove it beats baseline — $X more will confirm it." The second one is the one people pay for twice.
+# Improve conductor
+
+Own the requested outcome, actual spend, and the decision supported by the evidence.
+Distinguish building an evaluation, searching for candidates, and proving an improvement.
+
+## Establish the work
+
+1. Recover the target, required outcome, existing evaluation, and authorization from the user's request and product context.
+   Ask only for consequential information that the available evidence cannot resolve.
+2. Inspect the current baseline and the system's failure cases.
+   Choose a surface change, capability change, or architecture experiment according to the observed limitation.
+3. Check that the evaluation can detect the required behavior through the production entrypoint.
+   If it cannot, build that measurement before making improvement claims.
+   Record measurement work as measurement work, including its actual cost.
+4. Set resource limits and a stopping rule for the selected experiment.
+   Estimate cost from the actual execution path and retain uncertainty about additional calls, retries, and candidate evaluations.
+   More spend does not guarantee a useful candidate or a conclusive result.
+
+## Run and decide
+
+1. Use the maintained optimization method and execution path already available to the product.
+   Preserve training, selection, and final-comparison boundaries along with the registered observation units.
+2. Retain candidate artifacts, scorer identity, paired raw evidence, failures, and complete attempt costs.
+   Missing usage remains an explicit accounting gap.
+3. Read the shared deciding statistic and the producer's actual verdict.
+   Distinguish missing evidence, a measured failure, an inconclusive comparison, and a result that meets the release policy.
+4. Investigate surprising gains and null results with controls that isolate the suspected mechanism.
+   Use a footprint control when the claim concerns content versus added context; it is not a universal release prerequisite.
+5. Promote only through the product's authorized decision path after its required checks pass.
+   A promising exploratory result can justify another scoped experiment without establishing an improvement.
+
+## Report
+
+State what changed, the baseline comparison, deciding interval, independent-unit count, actual costs, and the verdict's reasons.
+Explain whether the run stopped because it reached its registered criterion, exhausted its budget, or could not capture valid evidence.
+A proposed follow-up must state which uncertainty it could resolve; extra budget alone does not promise confirmation.
+
+## Then consider
+
+| Condition | Skill |
+|---|---|
+| The product has no usable evaluation path | `eval-bootstrap` with the observed deliverable and missing checks |
+| A scorer or output-channel defect prevents assessment | `eval-architect` with the failed case and execution evidence |
+| Existing measurements need calibration or comparison review | `measurement-validation` with the baseline and retained results |
+| The supported next experiment changes an existing surface | `surface-evolution` with the target, acceptance criteria, and resource limits |

package/.claude/skills/measurement-validation/SKILL.md

@@ -1,38 +1,44 @@
 ---
 name: measurement-validation
-description: Prove a measurement is sound BEFORE spending money optimizing against it. The gate that decides whether an Improve run is allowed to start, and whether its result is allowed to be believed. Refuse metrics whose noise exceeds the effect, that have no held-out split, or whose evidence is incomplete.
+description: Check whether an evaluation supports an optimization or release decision through calibrated scoring, independent comparison units, and complete paired evidence.
 ---
 
-# Measurement Validation — earn the right to optimize
-
-Optimization is only as trustworthy as the measurement under it. This skill is the gate on both ends: **before** a paid run (is this metric allowed to be optimized?) and **after** (is this result allowed to be believed?). It is the difference between an Improve button that is a product and one that is a slot machine.
-
-Held by both the orchestrator (`improve-conductor`) and the builder (`eval-architect`). It is the shared honesty contract.
-
-## Invariant (non-negotiable)
-
-1. **Refuse to optimize if CV(metric) > the target delta.** If the run-to-run noise is bigger than the effect you're paying to move, the metric *cannot* validate the change — raise reps or fix the metric first. Do not tune against noise.
-2. **Refuse to report a lift over INCOMPLETE or UNPAIRED evidence.** Every held-out scenario must have a non-errored cell on *both* the baseline and the candidate side. Below the paired-n floor (≥3), the run is **invalid**, not a verdict. A lift computed over survivors is worse than no number.
-   > **Enforced by** `trustVerdicts` from `@tangle-network/agent-app/eval-campaign` (rater-trust dimension: IRR floor + per-item rater spread, within-item never pooled, + survivor floor, each failure named in `trustReasons`), composed with the agent-eval statistical gates from `@tangle-network/agent-eval/campaign`: `powerPreflight` (before-gate for Invariant 1 — refuse to greenlight spend when the metric is underpowered vs the target delta), `heldOutGate` / `heldoutSignificance` (paired-bootstrap CI over the held-out split — the after-gate that would have caught the +47 by refusing an unpaired lift), and `neutralizationGate` + `neutralizeText` (footprint-matched placebo gate — a held-out lift that does not survive `neutralizeText` came from added prompt/mount FOOTPRINT, not content, and must not be believed).
-3. **Every metric ties to a product-value claim** — "if this number moves, *this* user-visible outcome moves with it." No claim → it's a proxy → don't optimize it.
-4. **Below the data threshold of real outcomes, refuse to optimize** — state N and say why. You cannot improve what you have not yet observed enough of.
-
-> Worked failures (this is why the skill exists):
-> - **Noise read as signal:** ~6 optimization rounds were burned chasing ±0.15 run-to-run swings as if they were real. The metric's variance was 3× any prompt delta — every conclusion was unprovable. The bug was the *measurement*, not the model.
-> - **A lift that was a lie:** a GEPA run reported `heldOutLift = +47`. Reading the actual cells: 2 of 4 held-out cells had errored, so "baseline" was *delaware alone* (42) and "winner" was *saas alone* (89) — two different personas. The +47 was differencing unlike cells. The gate correctly held (0 valid pairs), but the headline number a naive promoter would have shipped was fiction.
-
-## Judgment (figure this out per metric)
-
-- How many reps establish variance for *this* metric? (Noisy targets need 5+, converged-artifact metrics fewer.)
-- Is an observed "noisy" result model variance, or a measurement smell? **Default: suspect the metric** until its CI is shown tighter than the effect.
-- Where might this metric diverge from real value (the Goodhart risk specific to this product)?
-
-## Self-test
-
-- Report **mean ± 95% CI over K converged rollouts.** Show CI < target delta *before* greenlighting spend. If you can't, you haven't earned the right to optimize yet.
-- Confirm the held-out split is disjoint from training and large enough that the paired-n floor survives an errored cell.
-- **Verify against ground truth, never the summary.** Read the actual cells / artifacts, not the provenance headline. (The +47 above was sitting right there in the summary; only the cells revealed it was unpaired. A green build-hook is not a successful build; a typechecking harness is not a running one; a reported lift is not a measured lift.)
-
-## Evolves-by
-
-Track promotions that passed validation but regressed in production → that's a missed variance source or an unguarded dimension; strengthen the preflight. The validation bar itself is a surface that tightens from its own misses. See `skill-evolution`.
+# Measurement validation
+
+Check the measurement through the product's actual evaluation path before interpreting an optimization result.
+Separate exploratory evidence from evidence that satisfies a release policy.
+
+## Establish the measurement
+
+1. State the user outcome, target population, independent comparison unit, and smallest useful effect.
+   Record the scoring revision, selection procedure, resource limits, and stopping rule before comparing candidates.
+2. Run a simple baseline and independent positive and negative controls through the scorer.
+   Use `auditEvaluator` from `@tangle-network/agent-eval/meta-eval` when the scorer needs an accuracy audit.
+   Report false acceptances, false rejections, and missing observations against the product's requirements.
+3. Check repeatability and choose enough independent units to resolve the intended effect.
+   Use `powerPreflight` to guide sampling and budget decisions.
+   An underpowered result can guide exploration; it cannot certify an improvement.
+4. Keep final comparison cases separate from training and candidate selection.
+   Register shared source units when several scenarios come from the same source.
+   Additional repetitions do not create new independent units.
+
+## Assess the result
+
+1. Inspect raw baseline and candidate cells, judge failures, pairing, and the registered unit mapping.
+   Missing or asymmetric evidence must remain visible and must block a promotion claim.
+2. Use Eval's `heldOutGate` or `heldoutSignificance` for the shared promotion decision.
+   Report its deciding interval, independent-unit count, eligibility, and vetoes.
+   The bootstrap diagnostic may differ from the deciding interval for binary outcomes.
+3. Use App's `trustVerdicts` to check rater agreement and surviving-judge coverage when an ensemble is present.
+   Agreement alone does not establish accuracy against independent controls or authorize release.
+4. Investigate null or surprising results before assigning a cause.
+   A control intervention supports a causal explanation only when it isolates the proposed mechanism.
+5. Retain actual costs, failures, exclusions, and uncertainty with the result.
+   Apply the product's release policy to the deciding evidence and record what the evidence cannot establish.
+
+## Then consider
+
+| Condition | Skill |
+|---|---|
+| Scoring or case coverage cannot test the required outcome | `eval-architect` with the failed controls and missing coverage |
+| Measurement supports a scoped optimization experiment | `improve-conductor` with the baseline, registered comparison, and resource limits |

package/README.md

@@ -151,6 +151,13 @@ The **complete, always-current reference** — every published subpath, its expo
 - [`/runtime`](src/runtime) — the bounded tool loop; the same loop drives a sandbox agent, a Worker, or an in-browser copilot behind one `streamTurn` seam.
 - [`/trace`](src/trace) — bounded stage timing, turn waterfalls, and mission traces over an injected telemetry carrier.
 
+**Evaluate changes**
+
+- [`/eval-campaign`](docs/api/eval-campaign.md) composes Eval campaigns, optimization methods, and ensemble judges.
+  Change `judgeVersion` when the scorer configuration changes.
+  Record paid judge calls through the supplied `costLedger`, `costPhase`, and `costTags`.
+  `trustVerdicts` checks agreement and coverage; use Eval's `auditEvaluator` to test accuracy against independent controls.
+
 **The server chat vertical** ([`examples/chat-app.md`](./examples/chat-app.md))
 - [`/chat-routes`](src/chat-routes) — `createChatTurnRoutes`: auth → store → streaming turn with buffered replay → uploads → sidecar question answering, assembled. Plus `runDetachedTurn` for autonomous turns a browser can still watch live.
 - [`/chat-store`](src/chat-store) · [`/interactions`](src/interactions) · [`/plans`](src/plans) — persistence, human-in-the-loop asks, and the durable plan projection.

package/dist/eval-campaign/index.d.ts

@@ -27,6 +27,8 @@ import type { JudgeConfig, Scenario } from '@tangle-network/agent-eval/campaign'
 export interface EnsembleJudgeConfig<TArtifact, TScenario extends Scenario, D extends string> {
     /** Judge name — appears in traces and scorecards. */
     name: string;
+    /** Scoring revision for campaign caches. Change it when rubric, model, or callback settings change. */
+    judgeVersion?: JudgeConfig<TArtifact, TScenario>['judgeVersion'];
     /** Stable-ordered rubric dimensions. Drives the `JudgeDimension` list AND the
      *  reducer keys, so a judge that omits a dimension scores it 0 (never silently
      *  dropped). */
@@ -38,11 +40,9 @@ export interface EnsembleJudgeConfig<TArtifact, TScenario extends Scenario, D ex
      * `{ model, perDimension: null }` to record a judge failure WITHOUT killing
      * the ensemble; throw only on an unrecoverable error (the whole rep is then
      * treated as a failed judge).
+     * Use the supplied costLedger, costPhase, and costTags to record paid judge calls.
      */
-    scoreOne: (input: {
-        artifact: TArtifact;
-        scenario: TScenario;
-        signal: AbortSignal;
+    scoreOne: (input: Parameters<JudgeConfig<TArtifact, TScenario>['score']>[0] & {
         rep: number;
     }) => Promise<JudgeVerdict<D>>;
     /** Independent judge calls per artifact, reduced by `aggregateJudgeVerdicts`.

package/dist/eval-campaign/index.js

@@ -115,10 +115,11 @@ function buildEnsembleJudge(cfg) {
   }
   return {
     name: cfg.name,
+    ...cfg.judgeVersion === void 0 ? {} : { judgeVersion: cfg.judgeVersion },
     dimensions: cfg.rubric.map((key) => ({ key, description: cfg.describe?.(key) ?? key })),
-    async score({ artifact, scenario, signal }) {
+    async score(input) {
       const settled = await Promise.allSettled(
-        Array.from({ length: reps }, (_, rep) => cfg.scoreOne({ artifact, scenario, signal, rep }))
+        Array.from({ length: reps }, (_, rep) => cfg.scoreOne({ ...input, rep }))
       );
       const verdicts = settled.map(
         (r, rep) => r.status === "fulfilled" ? r.value : { model: `${cfg.name}-rep${rep}`, perDimension: null, rationale: String(r.reason) }

package/dist/eval-campaign/index.js.map

@@ -1 +1 @@
-{"version":3,"sources":["../../src/eval-campaign/index.ts","../../src/eval-campaign/trust-gate.ts"],"sourcesContent":["/**\n * Eval-campaign — the app-shell's curated surface for a product's\n * self-improvement loop, NOT a reimplementation.\n *\n * The loop ENGINE lives in `@tangle-network/agent-eval` (a peer dependency):\n * `selfImprove` already owns execution, scoring, data separation, release\n * decisions, durable provenance, and hosted ingest. Candidate search is always\n * explicit: pass an official optimization `method`, or pass a caller-owned\n * `SurfaceProposer`.\n *\n * This module adds the one piece `selfImprove` does not own and which every\n * multi-model product re-hand-rolls — the ensemble judge:\n *\n *   {@link buildEnsembleJudge} — turn a per-rubric `scoreOne` into a\n *   `JudgeConfig` that fans out N uncorrelated judge calls and reduces them via\n *   the substrate's `aggregateJudgeVerdicts` (survivor-mean, inter-rater spread,\n *   fail-loud on all-failed). A product writes its rubric + one judge call; the\n *   fan-out, partial-failure handling, and composite are the scaffold's.\n *\n * Everything else is a curated re-export so a product has ONE eval import:\n * `selfImprove` + release policies + optimization methods + their types. See\n * `.claude/skills/eval-campaign/SKILL.md` for the wiring contract.\n */\n\nimport {\n  aggregateJudgeVerdicts,\n  type JudgeVerdict,\n} from '@tangle-network/agent-eval'\nimport type {\n  JudgeConfig,\n  JudgeScore,\n  Scenario,\n} from '@tangle-network/agent-eval/campaign'\n\n/** Config for {@link buildEnsembleJudge}. `D` = the rubric's dimension union. */\nexport interface EnsembleJudgeConfig<TArtifact, TScenario extends Scenario, D extends string> {\n  /** Judge name — appears in traces and scorecards. */\n  name: string\n  /** Stable-ordered rubric dimensions. Drives the `JudgeDimension` list AND the\n   *  reducer keys, so a judge that omits a dimension scores it 0 (never silently\n   *  dropped). */\n  rubric: readonly D[]\n  /**\n   * Score ONE artifact on the rubric → a raw per-dimension verdict. Called\n   * `judgeReps` times per artifact; vary the model by `rep` for an uncorrelated\n   * ensemble (judges that share a base model share its bias). Return\n   * `{ model, perDimension: null }` to record a judge failure WITHOUT killing\n   * the ensemble; throw only on an unrecoverable error (the whole rep is then\n   * treated as a failed judge).\n   */\n  scoreOne: (input: {\n    artifact: TArtifact\n    scenario: TScenario\n    signal: AbortSignal\n    rep: number\n  }) => Promise<JudgeVerdict<D>>\n  /** Independent judge calls per artifact, reduced by `aggregateJudgeVerdicts`.\n   *  Default 1. Raise (with model variety in `scoreOne`) for inter-rater bands. */\n  judgeReps?: number\n  /** Per-dimension composite weights. Default: uniform over `rubric`. A partial\n   *  map selects-and-weights exactly the named dimensions. */\n  weights?: Partial<Record<D, number>>\n  /** Optional human-readable dimension descriptions. Default: the key itself. */\n  describe?: (dim: D) => string\n}\n\n/**\n * Build a `JudgeConfig` whose `score()` fans out `judgeReps` independent\n * `scoreOne` calls and reduces them with the substrate's\n * `aggregateJudgeVerdicts`. A single judge call failing does NOT fail the cell\n * (it is recorded and dropped); only ALL judges failing throws — which the\n * campaign records as a failed cell, never a silent zero.\n *\n * Pass the result straight to `selfImprove({ judge })` (or `runCampaign`).\n */\nexport function buildEnsembleJudge<TArtifact, TScenario extends Scenario, D extends string>(\n  cfg: EnsembleJudgeConfig<TArtifact, TScenario, D>,\n): JudgeConfig<TArtifact, TScenario> {\n  const reps = cfg.judgeReps ?? 1\n  if (reps < 1) {\n    throw new Error(`buildEnsembleJudge: judgeReps must be >= 1 (got ${reps})`)\n  }\n  if (cfg.rubric.length === 0) {\n    throw new Error('buildEnsembleJudge: rubric is empty')\n  }\n  return {\n    name: cfg.name,\n    dimensions: cfg.rubric.map((key) => ({ key, description: cfg.describe?.(key) ?? key })),\n    async score({ artifact, scenario, signal }): Promise<JudgeScore> {\n      const settled = await Promise.allSettled(\n        Array.from({ length: reps }, (_, rep) => cfg.scoreOne({ artifact, scenario, signal, rep })),\n      )\n      const verdicts: JudgeVerdict<D>[] = settled.map((r, rep) =>\n        r.status === 'fulfilled'\n          ? r.value\n          : { model: `${cfg.name}-rep${rep}`, perDimension: null, rationale: String(r.reason) },\n      )\n      // Throws iff EVERY rep failed → the campaign records a failed cell.\n      const agg = aggregateJudgeVerdicts(verdicts, cfg.rubric, cfg.weights)\n      return { composite: agg.composite, dimensions: agg.perDimension, notes: agg.rationale }\n    },\n  }\n}\n\n// ── Trust gate — the after-gate (\"is this result allowed to be believed\") ────\n// One level up from `aggregateJudgeVerdicts`: it audits the raters ACROSS items\n// and reports whether the composites are believable before a lift is reported.\nexport {\n  trustVerdicts,\n  type TrustItem,\n  type TrustThresholds,\n  type TrustVerdict,\n} from './trust-gate'\n\n// ── Curated re-exports — the one eval import for a product loop ──────────────\n// The loop engine + gates + drivers + the ensemble reducer, so a product wires\n// its self-improvement loop from a single module instead of reaching across\n// three agent-eval subpaths. All DOWNWARD imports (agent-app consumes the\n// substrate); the layering rule is preserved.\n\nexport { aggregateJudgeVerdicts } from '@tangle-network/agent-eval'\nexport type {\n  EnsembleAggregate,\n  JudgeVerdict,\n  RunRecord,\n} from '@tangle-network/agent-eval'\nexport {\n  compareOptimizationMethods,\n  defaultProductionGate,\n  externalTextOptimizationMethod,\n  gepaOptimizationMethod,\n  paretoSignificanceGate,\n  runCampaign,\n  skillOptOptimizationMethod,\n} from '@tangle-network/agent-eval/campaign'\nexport type {\n  CampaignResult,\n  CompareOptimizationMethodsOptions,\n  DispatchContext,\n  ExternalTextOptimizationMethodConfig,\n  Gate,\n  GepaOptimizationMethodConfig,\n  JudgeConfig,\n  JudgeDimension,\n  JudgeScore,\n  LabeledScenarioStore,\n  MutableSurface,\n  OptimizationMethod,\n  OptimizationMethodResult,\n  Scenario,\n  SkillOptOptimizationMethodConfig,\n  SurfaceProposer,\n} from '@tangle-network/agent-eval/campaign'\nexport { selfImprove } from '@tangle-network/agent-eval/contract'\nexport type {\n  SelfImproveBudget,\n  SelfImproveOptions,\n  SelfImproveResult,\n} from '@tangle-network/agent-eval/contract'\n","/**\n * Trust gate — decides whether an ensemble's scores are allowed to be BELIEVED,\n * one level up from {@link aggregateJudgeVerdicts} (which only reduces ONE\n * artifact's raters to a composite). A composite is a number; this is the check\n * that the number means anything. It is the code \"Enforced by\" for the\n * measurement-validation skill's after-gate (\"is this result allowed to be\n * believed\").\n *\n * Three checks, each fail-loud and named in `trustReasons`:\n *   (1) inter-rater reliability over the corpus ≥ `irrFloor` — raters that\n *       disagree no better than chance carry no signal to optimize against.\n *   (2) per-item rater spread ≤ `spreadCeiling` — for EACH item, raters must\n *       converge on THAT item.\n *   (3) surviving raters per item ≥ `minSurvivors` — a mean over one or two\n *       raters is an anecdote, not an ensemble.\n *\n * CRITICAL metric semantics — per-item spread is rater disagreement about the\n * SAME item: `max(score) − min(score)` across the raters that scored THAT item\n * (max over its dimensions), never pooled across different items or across the\n * baseline/candidate sides. Pooling reads a genuine quality gap BETWEEN items as\n * \"the raters split\" and so trips the gate exactly when the finding is largest —\n * the failure mode the after-gate exists to prevent. The corpus IRR (check 1)\n * leans on the substrate's `interRaterReliability`, whose expected-disagreement\n * denominator already pools across items, so genuine item-to-item variation\n * RAISES reliability rather than lowering it.\n */\n\nimport {\n  interRaterReliability,\n  type JudgeScore,\n  type JudgeVerdict,\n} from '@tangle-network/agent-eval'\n\n/** One item's raters: the per-judge verdicts {@link aggregateJudgeVerdicts}\n *  reduces, tagged with the item they scored so spread stays within-item. */\nexport interface TrustItem<D extends string = string> {\n  /** Stable item identifier — surfaces in `perItemSpread` and `trustReasons`. */\n  itemId: string\n  /** The raters' verdicts for THIS item (one per judge call). A failed judge\n   *  (`perDimension: null`) is dropped before spread/IRR, never folded as 0. */\n  verdicts: readonly JudgeVerdict<D>[]\n}\n\n/** Thresholds for {@link trustVerdicts}. All overridable; defaults are the\n *  conservative after-gate bar. */\nexport interface TrustThresholds {\n  /** Minimum corpus inter-rater reliability (Krippendorff-style α). Below this\n   *  the raters agree no better than chance. Default 0.2. */\n  irrFloor?: number\n  /** Maximum per-item rater spread (`max − min` over a single item's surviving\n   *  raters, across its dimensions). Above this the raters split ON THAT ITEM.\n   *  Default 0.5. */\n  spreadCeiling?: number\n  /** Minimum surviving (non-failed) raters required per item. Default 3. */\n  minSurvivors?: number\n}\n\n/** Result of the trust gate. `trustworthy` iff every check passed; `trustReasons`\n *  is empty iff `trustworthy`. */\nexport interface TrustVerdict {\n  /** True iff IRR ≥ floor AND every item's spread ≤ ceiling AND every item has\n   *  ≥ `minSurvivors` surviving raters. */\n  trustworthy: boolean\n  /** One entry per FAILED check, each naming its number + the offending value.\n   *  Empty iff `trustworthy`. */\n  trustReasons: string[]\n  /** Corpus inter-rater reliability actually measured (the check-1 value). */\n  interRaterReliability: number\n  /** Per-item spread (`max − min` over surviving raters, max over dimensions),\n   *  keyed by `itemId`. The check-2 input, surfaced for drill-down. */\n  perItemSpread: Record<string, number>\n}\n\nconst DEFAULT_IRR_FLOOR = 0.2\nconst DEFAULT_SPREAD_CEILING = 0.5\nconst DEFAULT_MIN_SURVIVORS = 3\n\n/** Surviving (non-failed) verdicts for an item — those with a real\n *  `perDimension` map. A failed judge carries no scores and is excluded from\n *  every statistic (it is NOT a zero rater). */\nfunction survivors<D extends string>(item: TrustItem<D>): JudgeVerdict<D>[] {\n  return item.verdicts.filter((v) => v.perDimension !== null)\n}\n\n/**\n * Within-item rater spread: for each dimension, `max − min` across the item's\n * surviving raters; the item's spread is the max over its dimensions (the worst\n * dimension the raters split on). Pooled ONLY within this one item — never\n * across items — so a quality gap between items cannot inflate it.\n */\nfunction itemSpread<D extends string>(survivorVerdicts: JudgeVerdict<D>[]): number {\n  if (survivorVerdicts.length < 2) return 0\n  const dims = new Set<string>()\n  for (const v of survivorVerdicts) {\n    for (const d of Object.keys(v.perDimension as Record<string, number>)) dims.add(d)\n  }\n  let worst = 0\n  for (const d of dims) {\n    let min = Infinity\n    let max = -Infinity\n    for (const v of survivorVerdicts) {\n      const score = (v.perDimension as Record<string, number>)[d]\n      if (score === undefined) continue\n      if (score < min) min = score\n      if (score > max) max = score\n    }\n    if (max > -Infinity && max - min > worst) worst = max - min\n  }\n  return worst\n}\n\n/**\n * Decide whether an ensemble's per-item verdicts are trustworthy enough to\n * believe a lift computed from them. Pure: no LLM, no I/O, no clock, no random —\n * the same `items` + `thresholds` always yield the same verdict.\n *\n * Sibling to {@link aggregateJudgeVerdicts}: that reduces ONE item's raters to a\n * composite; this audits the raters ACROSS items and reports whether the\n * composites are believable. Run it on the corpus of held-out items before\n * reporting any lift over their scores.\n *\n * @throws if `items` is empty — an empty corpus has no measurable trust, and a\n *   silent `trustworthy: true` over zero evidence is the exact lie the gate\n *   exists to refuse.\n */\nexport function trustVerdicts<D extends string>(\n  items: readonly TrustItem<D>[],\n  thresholds: TrustThresholds = {},\n): TrustVerdict {\n  if (items.length === 0) {\n    throw new Error('trustVerdicts: items is empty — no evidence to trust')\n  }\n  const irrFloor = thresholds.irrFloor ?? DEFAULT_IRR_FLOOR\n  const spreadCeiling = thresholds.spreadCeiling ?? DEFAULT_SPREAD_CEILING\n  const minSurvivors = thresholds.minSurvivors ?? DEFAULT_MIN_SURVIVORS\n\n  // Rater-major JudgeScore series for the substrate's IRR. Each item's surviving\n  // raters are assigned a stable column index so the same rater across items\n  // lines up; per (item, dimension) one JudgeScore per rater, in item-then-\n  // dimension order — the layout interRaterReliability chunks back into items.\n  const maxRaters = items.reduce((m, it) => Math.max(m, survivors(it).length), 0)\n  const raterSeries: JudgeScore[][] = Array.from({ length: maxRaters }, () => [])\n  const perItemSpread: Record<string, number> = {}\n  const splitItems: Array<{ itemId: string; spread: number }> = []\n  const starvedItems: Array<{ itemId: string; n: number }> = []\n\n  for (const item of items) {\n    const surv = survivors(item)\n    if (surv.length < minSurvivors) starvedItems.push({ itemId: item.itemId, n: surv.length })\n\n    const spread = itemSpread(surv)\n    perItemSpread[item.itemId] = spread\n    if (spread > spreadCeiling) splitItems.push({ itemId: item.itemId, spread })\n\n    if (surv.length >= 2) {\n      const dims = Array.from(\n        new Set(surv.flatMap((v) => Object.keys(v.perDimension as Record<string, number>))),\n      ).sort()\n      surv.forEach((v, raterIdx) => {\n        // raterIdx < surv.length ≤ maxRaters = raterSeries.length, so the column\n        // always exists; the ??= keeps the access provably defined for the type.\n        const column = (raterSeries[raterIdx] ??= [])\n        const pd = v.perDimension as Record<string, number>\n        for (const d of dims) {\n          const score = pd[d]\n          if (score === undefined) continue\n          column.push({\n            judgeName: v.model,\n            dimension: `${item.itemId}::${d}`,\n            score,\n            reasoning: v.rationale ?? '',\n          })\n        }\n      })\n    }\n  }\n\n  const irr = interRaterReliability(raterSeries)\n\n  const trustReasons: string[] = []\n  if (irr < irrFloor) {\n    trustReasons.push(`(1) IRR ${round(irr)} < ${irrFloor}`)\n  }\n  for (const { itemId, spread } of splitItems) {\n    trustReasons.push(`(2) item ${itemId} spread ${round(spread)} > ${spreadCeiling} — raters split`)\n  }\n  for (const { itemId, n } of starvedItems) {\n    trustReasons.push(`(3) item ${itemId}: ${n} surviving raters < ${minSurvivors}`)\n  }\n\n  return {\n    trustworthy: trustReasons.length === 0,\n    trustReasons,\n    interRaterReliability: irr,\n    perItemSpread,\n  }\n}\n\n/** Round to 2 decimals for stable, readable reason strings. */\nfunction round(n: number): number {\n  return Math.round(n * 100) / 100\n}\n"],"mappings":";AAwBA;AAAA,EACE;AAAA,OAEK;;;ACAP;AAAA,EACE;AAAA,OAGK;AA0CP,IAAM,oBAAoB;AAC1B,IAAM,yBAAyB;AAC/B,IAAM,wBAAwB;AAK9B,SAAS,UAA4B,MAAuC;AAC1E,SAAO,KAAK,SAAS,OAAO,CAAC,MAAM,EAAE,iBAAiB,IAAI;AAC5D;AAQA,SAAS,WAA6B,kBAA6C;AACjF,MAAI,iBAAiB,SAAS,EAAG,QAAO;AACxC,QAAM,OAAO,oBAAI,IAAY;AAC7B,aAAW,KAAK,kBAAkB;AAChC,eAAW,KAAK,OAAO,KAAK,EAAE,YAAsC,EAAG,MAAK,IAAI,CAAC;AAAA,EACnF;AACA,MAAI,QAAQ;AACZ,aAAW,KAAK,MAAM;AACpB,QAAI,MAAM;AACV,QAAI,MAAM;AACV,eAAW,KAAK,kBAAkB;AAChC,YAAM,QAAS,EAAE,aAAwC,CAAC;AAC1D,UAAI,UAAU,OAAW;AACzB,UAAI,QAAQ,IAAK,OAAM;AACvB,UAAI,QAAQ,IAAK,OAAM;AAAA,IACzB;AACA,QAAI,MAAM,aAAa,MAAM,MAAM,MAAO,SAAQ,MAAM;AAAA,EAC1D;AACA,SAAO;AACT;AAgBO,SAAS,cACd,OACA,aAA8B,CAAC,GACjB;AACd,MAAI,MAAM,WAAW,GAAG;AACtB,UAAM,IAAI,MAAM,2DAAsD;AAAA,EACxE;AACA,QAAM,WAAW,WAAW,YAAY;AACxC,QAAM,gBAAgB,WAAW,iBAAiB;AAClD,QAAM,eAAe,WAAW,gBAAgB;AAMhD,QAAM,YAAY,MAAM,OAAO,CAAC,GAAG,OAAO,KAAK,IAAI,GAAG,UAAU,EAAE,EAAE,MAAM,GAAG,CAAC;AAC9E,QAAM,cAA8B,MAAM,KAAK,EAAE,QAAQ,UAAU,GAAG,MAAM,CAAC,CAAC;AAC9E,QAAM,gBAAwC,CAAC;AAC/C,QAAM,aAAwD,CAAC;AAC/D,QAAM,eAAqD,CAAC;AAE5D,aAAW,QAAQ,OAAO;AACxB,UAAM,OAAO,UAAU,IAAI;AAC3B,QAAI,KAAK,SAAS,aAAc,cAAa,KAAK,EAAE,QAAQ,KAAK,QAAQ,GAAG,KAAK,OAAO,CAAC;AAEzF,UAAM,SAAS,WAAW,IAAI;AAC9B,kBAAc,KAAK,MAAM,IAAI;AAC7B,QAAI,SAAS,cAAe,YAAW,KAAK,EAAE,QAAQ,KAAK,QAAQ,OAAO,CAAC;AAE3E,QAAI,KAAK,UAAU,GAAG;AACpB,YAAM,OAAO,MAAM;AAAA,QACjB,IAAI,IAAI,KAAK,QAAQ,CAAC,MAAM,OAAO,KAAK,EAAE,YAAsC,CAAC,CAAC;AAAA,MACpF,EAAE,KAAK;AACP,WAAK,QAAQ,CAAC,GAAG,aAAa;AAG5B,cAAM,SAAU,YAAY,QAAQ,MAAM,CAAC;AAC3C,cAAM,KAAK,EAAE;AACb,mBAAW,KAAK,MAAM;AACpB,gBAAM,QAAQ,GAAG,CAAC;AAClB,cAAI,UAAU,OAAW;AACzB,iBAAO,KAAK;AAAA,YACV,WAAW,EAAE;AAAA,YACb,WAAW,GAAG,KAAK,MAAM,KAAK,CAAC;AAAA,YAC/B;AAAA,YACA,WAAW,EAAE,aAAa;AAAA,UAC5B,CAAC;AAAA,QACH;AAAA,MACF,CAAC;AAAA,IACH;AAAA,EACF;AAEA,QAAM,MAAM,sBAAsB,WAAW;AAE7C,QAAM,eAAyB,CAAC;AAChC,MAAI,MAAM,UAAU;AAClB,iBAAa,KAAK,WAAW,MAAM,GAAG,CAAC,MAAM,QAAQ,EAAE;AAAA,EACzD;AACA,aAAW,EAAE,QAAQ,OAAO,KAAK,YAAY;AAC3C,iBAAa,KAAK,YAAY,MAAM,WAAW,MAAM,MAAM,CAAC,MAAM,aAAa,sBAAiB;AAAA,EAClG;AACA,aAAW,EAAE,QAAQ,EAAE,KAAK,cAAc;AACxC,iBAAa,KAAK,YAAY,MAAM,KAAK,CAAC,uBAAuB,YAAY,EAAE;AAAA,EACjF;AAEA,SAAO;AAAA,IACL,aAAa,aAAa,WAAW;AAAA,IACrC;AAAA,IACA,uBAAuB;AAAA,IACvB;AAAA,EACF;AACF;AAGA,SAAS,MAAM,GAAmB;AAChC,SAAO,KAAK,MAAM,IAAI,GAAG,IAAI;AAC/B;;;ADjFA,SAAS,0BAAAA,+BAA8B;AAMvC;AAAA,EACE;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,OACK;AAmBP,SAAS,mBAAmB;AA9ErB,SAAS,mBACd,KACmC;AACnC,QAAM,OAAO,IAAI,aAAa;AAC9B,MAAI,OAAO,GAAG;AACZ,UAAM,IAAI,MAAM,mDAAmD,IAAI,GAAG;AAAA,EAC5E;AACA,MAAI,IAAI,OAAO,WAAW,GAAG;AAC3B,UAAM,IAAI,MAAM,qCAAqC;AAAA,EACvD;AACA,SAAO;AAAA,IACL,MAAM,IAAI;AAAA,IACV,YAAY,IAAI,OAAO,IAAI,CAAC,SAAS,EAAE,KAAK,aAAa,IAAI,WAAW,GAAG,KAAK,IAAI,EAAE;AAAA,IACtF,MAAM,MAAM,EAAE,UAAU,UAAU,OAAO,GAAwB;AAC/D,YAAM,UAAU,MAAM,QAAQ;AAAA,QAC5B,MAAM,KAAK,EAAE,QAAQ,KAAK,GAAG,CAAC,GAAG,QAAQ,IAAI,SAAS,EAAE,UAAU,UAAU,QAAQ,IAAI,CAAC,CAAC;AAAA,MAC5F;AACA,YAAM,WAA8B,QAAQ;AAAA,QAAI,CAAC,GAAG,QAClD,EAAE,WAAW,cACT,EAAE,QACF,EAAE,OAAO,GAAG,IAAI,IAAI,OAAO,GAAG,IAAI,cAAc,MAAM,WAAW,OAAO,EAAE,MAAM,EAAE;AAAA,MACxF;AAEA,YAAM,MAAM,uBAAuB,UAAU,IAAI,QAAQ,IAAI,OAAO;AACpE,aAAO,EAAE,WAAW,IAAI,WAAW,YAAY,IAAI,cAAc,OAAO,IAAI,UAAU;AAAA,IACxF;AAAA,EACF;AACF;","names":["aggregateJudgeVerdicts"]}
+{"version":3,"sources":["../../src/eval-campaign/index.ts","../../src/eval-campaign/trust-gate.ts"],"sourcesContent":["/**\n * Eval-campaign — the app-shell's curated surface for a product's\n * self-improvement loop, NOT a reimplementation.\n *\n * The loop ENGINE lives in `@tangle-network/agent-eval` (a peer dependency):\n * `selfImprove` already owns execution, scoring, data separation, release\n * decisions, durable provenance, and hosted ingest. Candidate search is always\n * explicit: pass an official optimization `method`, or pass a caller-owned\n * `SurfaceProposer`.\n *\n * This module adds the one piece `selfImprove` does not own and which every\n * multi-model product re-hand-rolls — the ensemble judge:\n *\n *   {@link buildEnsembleJudge} — turn a per-rubric `scoreOne` into a\n *   `JudgeConfig` that fans out N uncorrelated judge calls and reduces them via\n *   the substrate's `aggregateJudgeVerdicts` (survivor-mean, inter-rater spread,\n *   fail-loud on all-failed). A product writes its rubric + one judge call; the\n *   fan-out, partial-failure handling, and composite are the scaffold's.\n *\n * Everything else is a curated re-export so a product has ONE eval import:\n * `selfImprove` + release policies + optimization methods + their types. See\n * `.claude/skills/eval-campaign/SKILL.md` for the wiring contract.\n */\n\nimport {\n  aggregateJudgeVerdicts,\n  type JudgeVerdict,\n} from '@tangle-network/agent-eval'\nimport type {\n  JudgeConfig,\n  JudgeScore,\n  Scenario,\n} from '@tangle-network/agent-eval/campaign'\n\n/** Config for {@link buildEnsembleJudge}. `D` = the rubric's dimension union. */\nexport interface EnsembleJudgeConfig<TArtifact, TScenario extends Scenario, D extends string> {\n  /** Judge name — appears in traces and scorecards. */\n  name: string\n  /** Scoring revision for campaign caches. Change it when rubric, model, or callback settings change. */\n  judgeVersion?: JudgeConfig<TArtifact, TScenario>['judgeVersion']\n  /** Stable-ordered rubric dimensions. Drives the `JudgeDimension` list AND the\n   *  reducer keys, so a judge that omits a dimension scores it 0 (never silently\n   *  dropped). */\n  rubric: readonly D[]\n  /**\n   * Score ONE artifact on the rubric → a raw per-dimension verdict. Called\n   * `judgeReps` times per artifact; vary the model by `rep` for an uncorrelated\n   * ensemble (judges that share a base model share its bias). Return\n   * `{ model, perDimension: null }` to record a judge failure WITHOUT killing\n   * the ensemble; throw only on an unrecoverable error (the whole rep is then\n   * treated as a failed judge).\n   * Use the supplied costLedger, costPhase, and costTags to record paid judge calls.\n   */\n  scoreOne: (input: Parameters<JudgeConfig<TArtifact, TScenario>['score']>[0] & {\n    rep: number\n  }) => Promise<JudgeVerdict<D>>\n  /** Independent judge calls per artifact, reduced by `aggregateJudgeVerdicts`.\n   *  Default 1. Raise (with model variety in `scoreOne`) for inter-rater bands. */\n  judgeReps?: number\n  /** Per-dimension composite weights. Default: uniform over `rubric`. A partial\n   *  map selects-and-weights exactly the named dimensions. */\n  weights?: Partial<Record<D, number>>\n  /** Optional human-readable dimension descriptions. Default: the key itself. */\n  describe?: (dim: D) => string\n}\n\n/**\n * Build a `JudgeConfig` whose `score()` fans out `judgeReps` independent\n * `scoreOne` calls and reduces them with the substrate's\n * `aggregateJudgeVerdicts`. A single judge call failing does NOT fail the cell\n * (it is recorded and dropped); only ALL judges failing throws — which the\n * campaign records as a failed cell, never a silent zero.\n *\n * Pass the result straight to `selfImprove({ judge })` (or `runCampaign`).\n */\nexport function buildEnsembleJudge<TArtifact, TScenario extends Scenario, D extends string>(\n  cfg: EnsembleJudgeConfig<TArtifact, TScenario, D>,\n): JudgeConfig<TArtifact, TScenario> {\n  const reps = cfg.judgeReps ?? 1\n  if (reps < 1) {\n    throw new Error(`buildEnsembleJudge: judgeReps must be >= 1 (got ${reps})`)\n  }\n  if (cfg.rubric.length === 0) {\n    throw new Error('buildEnsembleJudge: rubric is empty')\n  }\n  return {\n    name: cfg.name,\n    ...(cfg.judgeVersion === undefined ? {} : { judgeVersion: cfg.judgeVersion }),\n    dimensions: cfg.rubric.map((key) => ({ key, description: cfg.describe?.(key) ?? key })),\n    async score(input): Promise<JudgeScore> {\n      const settled = await Promise.allSettled(\n        Array.from({ length: reps }, (_, rep) => cfg.scoreOne({ ...input, rep })),\n      )\n      const verdicts: JudgeVerdict<D>[] = settled.map((r, rep) =>\n        r.status === 'fulfilled'\n          ? r.value\n          : { model: `${cfg.name}-rep${rep}`, perDimension: null, rationale: String(r.reason) },\n      )\n      // Throws iff EVERY rep failed → the campaign records a failed cell.\n      const agg = aggregateJudgeVerdicts(verdicts, cfg.rubric, cfg.weights)\n      return { composite: agg.composite, dimensions: agg.perDimension, notes: agg.rationale }\n    },\n  }\n}\n\n// Agreement across items is distinct from evaluator accuracy against independent controls.\n// Consumers can opt into auditEvaluator from agent-eval/meta-eval for that separate evidence.\nexport {\n  trustVerdicts,\n  type TrustItem,\n  type TrustThresholds,\n  type TrustVerdict,\n} from './trust-gate'\n\n// ── Curated re-exports — the one eval import for a product loop ──────────────\n// The loop engine + gates + drivers + the ensemble reducer, so a product wires\n// its self-improvement loop from a single module instead of reaching across\n// three agent-eval subpaths. All DOWNWARD imports (agent-app consumes the\n// substrate); the layering rule is preserved.\n\nexport { aggregateJudgeVerdicts } from '@tangle-network/agent-eval'\nexport type {\n  EnsembleAggregate,\n  JudgeVerdict,\n  RunRecord,\n} from '@tangle-network/agent-eval'\nexport {\n  compareOptimizationMethods,\n  defaultProductionGate,\n  externalTextOptimizationMethod,\n  gepaOptimizationMethod,\n  paretoSignificanceGate,\n  runCampaign,\n  skillOptOptimizationMethod,\n} from '@tangle-network/agent-eval/campaign'\nexport type {\n  CampaignResult,\n  CompareOptimizationMethodsOptions,\n  DispatchContext,\n  ExternalTextOptimizationMethodConfig,\n  Gate,\n  GepaOptimizationMethodConfig,\n  JudgeConfig,\n  JudgeDimension,\n  JudgeScore,\n  LabeledScenarioStore,\n  MutableSurface,\n  OptimizationMethod,\n  OptimizationMethodResult,\n  Scenario,\n  SkillOptOptimizationMethodConfig,\n  SurfaceProposer,\n} from '@tangle-network/agent-eval/campaign'\nexport { selfImprove } from '@tangle-network/agent-eval/contract'\nexport type {\n  SelfImproveBudget,\n  SelfImproveOptions,\n  SelfImproveResult,\n} from '@tangle-network/agent-eval/contract'\n","/**\n * Summarize rater agreement, within-item spread, and surviving-judge coverage.\n * Consumers choose the thresholds that a trustworthy result must satisfy.\n * Agreement does not measure evaluator errors against independent controls.\n * Use agent-eval/meta-eval's auditEvaluator when the consumer needs that separate evidence.\n * Spread stays within each item so differences in task quality cannot mimic rater disagreement.\n */\n\nimport {\n  interRaterReliability,\n  type DimensionJudgeScore,\n  type JudgeVerdict,\n} from '@tangle-network/agent-eval'\n\n/** One item's raters: the per-judge verdicts {@link aggregateJudgeVerdicts}\n *  reduces, tagged with the item they scored so spread stays within-item. */\nexport interface TrustItem<D extends string = string> {\n  /** Stable item identifier — surfaces in `perItemSpread` and `trustReasons`. */\n  itemId: string\n  /** The raters' verdicts for THIS item (one per judge call). A failed judge\n   *  (`perDimension: null`) is dropped before spread/IRR, never folded as 0. */\n  verdicts: readonly JudgeVerdict<D>[]\n}\n\n/** Configurable agreement and coverage thresholds for {@link trustVerdicts}. */\nexport interface TrustThresholds {\n  /** Minimum corpus inter-rater reliability (Krippendorff-style α). Default 0.2. */\n  irrFloor?: number\n  /** Maximum per-item rater spread (`max − min` over a single item's surviving\n   *  raters, across its dimensions). Above this the raters split ON THAT ITEM.\n   *  Default 0.5. */\n  spreadCeiling?: number\n  /** Minimum surviving (non-failed) raters required per item. Default 3. */\n  minSurvivors?: number\n}\n\n/** Result of the trust gate. `trustworthy` iff every check passed; `trustReasons`\n *  is empty iff `trustworthy`. */\nexport interface TrustVerdict {\n  /** True iff IRR ≥ floor AND every item's spread ≤ ceiling AND every item has\n   *  ≥ `minSurvivors` surviving raters. */\n  trustworthy: boolean\n  /** One entry per FAILED check, each naming its number + the offending value.\n   *  Empty iff `trustworthy`. */\n  trustReasons: string[]\n  /** Corpus inter-rater reliability actually measured (the check-1 value). */\n  interRaterReliability: number\n  /** Per-item spread (`max − min` over surviving raters, max over dimensions),\n   *  keyed by `itemId`. The check-2 input, surfaced for drill-down. */\n  perItemSpread: Record<string, number>\n}\n\nconst DEFAULT_IRR_FLOOR = 0.2\nconst DEFAULT_SPREAD_CEILING = 0.5\nconst DEFAULT_MIN_SURVIVORS = 3\n\n/** Surviving (non-failed) verdicts for an item — those with a real\n *  `perDimension` map. A failed judge carries no scores and is excluded from\n *  every statistic (it is NOT a zero rater). */\nfunction survivors<D extends string>(item: TrustItem<D>): JudgeVerdict<D>[] {\n  return item.verdicts.filter((v) => v.perDimension !== null)\n}\n\n/**\n * Within-item rater spread: for each dimension, `max − min` across the item's\n * surviving raters; the item's spread is the max over its dimensions (the worst\n * dimension the raters split on). Pooled ONLY within this one item — never\n * across items — so a quality gap between items cannot inflate it.\n */\nfunction itemSpread<D extends string>(survivorVerdicts: JudgeVerdict<D>[]): number {\n  if (survivorVerdicts.length < 2) return 0\n  const dims = new Set<string>()\n  for (const v of survivorVerdicts) {\n    for (const d of Object.keys(v.perDimension as Record<string, number>)) dims.add(d)\n  }\n  let worst = 0\n  for (const d of dims) {\n    let min = Infinity\n    let max = -Infinity\n    for (const v of survivorVerdicts) {\n      const score = (v.perDimension as Record<string, number>)[d]\n      if (score === undefined) continue\n      if (score < min) min = score\n      if (score > max) max = score\n    }\n    if (max > -Infinity && max - min > worst) worst = max - min\n  }\n  return worst\n}\n\n/**\n * Check an ensemble against its configured agreement and coverage thresholds. Pure: no LLM, no I/O, no clock, no random —\n * the same `items` + `thresholds` always yield the same verdict.\n *\n * Sibling to {@link aggregateJudgeVerdicts}: that reduces ONE item's raters to a\n * composite; this summarizes agreement across the supplied items.\n * A passing result does not establish evaluator accuracy or authorize a release.\n *\n * @throws if `items` is empty — an empty corpus has no measurable trust, and a\n *   silent `trustworthy: true` over zero evidence is the exact lie the gate\n *   exists to refuse.\n */\nexport function trustVerdicts<D extends string>(\n  items: readonly TrustItem<D>[],\n  thresholds: TrustThresholds = {},\n): TrustVerdict {\n  if (items.length === 0) {\n    throw new Error('trustVerdicts: items is empty — no evidence to trust')\n  }\n  const irrFloor = thresholds.irrFloor ?? DEFAULT_IRR_FLOOR\n  const spreadCeiling = thresholds.spreadCeiling ?? DEFAULT_SPREAD_CEILING\n  const minSurvivors = thresholds.minSurvivors ?? DEFAULT_MIN_SURVIVORS\n\n  // Rater-major JudgeScore series for the substrate's IRR. Each item's surviving\n  // raters are assigned a stable column index so the same rater across items\n  // lines up; per (item, dimension) one JudgeScore per rater, in item-then-\n  // dimension order — the layout interRaterReliability chunks back into items.\n  const maxRaters = items.reduce((m, it) => Math.max(m, survivors(it).length), 0)\n  const raterSeries: DimensionJudgeScore[][] = Array.from({ length: maxRaters }, () => [])\n  const perItemSpread: Record<string, number> = {}\n  const splitItems: Array<{ itemId: string; spread: number }> = []\n  const starvedItems: Array<{ itemId: string; n: number }> = []\n\n  for (const item of items) {\n    const surv = survivors(item)\n    if (surv.length < minSurvivors) starvedItems.push({ itemId: item.itemId, n: surv.length })\n\n    const spread = itemSpread(surv)\n    perItemSpread[item.itemId] = spread\n    if (spread > spreadCeiling) splitItems.push({ itemId: item.itemId, spread })\n\n    if (surv.length >= 2) {\n      const dims = Array.from(\n        new Set(surv.flatMap((v) => Object.keys(v.perDimension as Record<string, number>))),\n      ).sort()\n      surv.forEach((v, raterIdx) => {\n        // raterIdx < surv.length ≤ maxRaters = raterSeries.length, so the column\n        // always exists; the ??= keeps the access provably defined for the type.\n        const column = (raterSeries[raterIdx] ??= [])\n        const pd = v.perDimension as Record<string, number>\n        for (const d of dims) {\n          const score = pd[d]\n          if (score === undefined) continue\n          column.push({\n            judgeName: v.model,\n            dimension: `${item.itemId}::${d}`,\n            score,\n            reasoning: v.rationale ?? '',\n          })\n        }\n      })\n    }\n  }\n\n  const irr = interRaterReliability(raterSeries)\n\n  const trustReasons: string[] = []\n  if (irr < irrFloor) {\n    trustReasons.push(`(1) IRR ${round(irr)} < ${irrFloor}`)\n  }\n  for (const { itemId, spread } of splitItems) {\n    trustReasons.push(`(2) item ${itemId} spread ${round(spread)} > ${spreadCeiling} — raters split`)\n  }\n  for (const { itemId, n } of starvedItems) {\n    trustReasons.push(`(3) item ${itemId}: ${n} surviving raters < ${minSurvivors}`)\n  }\n\n  return {\n    trustworthy: trustReasons.length === 0,\n    trustReasons,\n    interRaterReliability: irr,\n    perItemSpread,\n  }\n}\n\n/** Round to 2 decimals for stable, readable reason strings. */\nfunction round(n: number): number {\n  return Math.round(n * 100) / 100\n}\n"],"mappings":";AAwBA;AAAA,EACE;AAAA,OAEK;;;ACnBP;AAAA,EACE;AAAA,OAGK;AAwCP,IAAM,oBAAoB;AAC1B,IAAM,yBAAyB;AAC/B,IAAM,wBAAwB;AAK9B,SAAS,UAA4B,MAAuC;AAC1E,SAAO,KAAK,SAAS,OAAO,CAAC,MAAM,EAAE,iBAAiB,IAAI;AAC5D;AAQA,SAAS,WAA6B,kBAA6C;AACjF,MAAI,iBAAiB,SAAS,EAAG,QAAO;AACxC,QAAM,OAAO,oBAAI,IAAY;AAC7B,aAAW,KAAK,kBAAkB;AAChC,eAAW,KAAK,OAAO,KAAK,EAAE,YAAsC,EAAG,MAAK,IAAI,CAAC;AAAA,EACnF;AACA,MAAI,QAAQ;AACZ,aAAW,KAAK,MAAM;AACpB,QAAI,MAAM;AACV,QAAI,MAAM;AACV,eAAW,KAAK,kBAAkB;AAChC,YAAM,QAAS,EAAE,aAAwC,CAAC;AAC1D,UAAI,UAAU,OAAW;AACzB,UAAI,QAAQ,IAAK,OAAM;AACvB,UAAI,QAAQ,IAAK,OAAM;AAAA,IACzB;AACA,QAAI,MAAM,aAAa,MAAM,MAAM,MAAO,SAAQ,MAAM;AAAA,EAC1D;AACA,SAAO;AACT;AAcO,SAAS,cACd,OACA,aAA8B,CAAC,GACjB;AACd,MAAI,MAAM,WAAW,GAAG;AACtB,UAAM,IAAI,MAAM,2DAAsD;AAAA,EACxE;AACA,QAAM,WAAW,WAAW,YAAY;AACxC,QAAM,gBAAgB,WAAW,iBAAiB;AAClD,QAAM,eAAe,WAAW,gBAAgB;AAMhD,QAAM,YAAY,MAAM,OAAO,CAAC,GAAG,OAAO,KAAK,IAAI,GAAG,UAAU,EAAE,EAAE,MAAM,GAAG,CAAC;AAC9E,QAAM,cAAuC,MAAM,KAAK,EAAE,QAAQ,UAAU,GAAG,MAAM,CAAC,CAAC;AACvF,QAAM,gBAAwC,CAAC;AAC/C,QAAM,aAAwD,CAAC;AAC/D,QAAM,eAAqD,CAAC;AAE5D,aAAW,QAAQ,OAAO;AACxB,UAAM,OAAO,UAAU,IAAI;AAC3B,QAAI,KAAK,SAAS,aAAc,cAAa,KAAK,EAAE,QAAQ,KAAK,QAAQ,GAAG,KAAK,OAAO,CAAC;AAEzF,UAAM,SAAS,WAAW,IAAI;AAC9B,kBAAc,KAAK,MAAM,IAAI;AAC7B,QAAI,SAAS,cAAe,YAAW,KAAK,EAAE,QAAQ,KAAK,QAAQ,OAAO,CAAC;AAE3E,QAAI,KAAK,UAAU,GAAG;AACpB,YAAM,OAAO,MAAM;AAAA,QACjB,IAAI,IAAI,KAAK,QAAQ,CAAC,MAAM,OAAO,KAAK,EAAE,YAAsC,CAAC,CAAC;AAAA,MACpF,EAAE,KAAK;AACP,WAAK,QAAQ,CAAC,GAAG,aAAa;AAG5B,cAAM,SAAU,YAAY,QAAQ,MAAM,CAAC;AAC3C,cAAM,KAAK,EAAE;AACb,mBAAW,KAAK,MAAM;AACpB,gBAAM,QAAQ,GAAG,CAAC;AAClB,cAAI,UAAU,OAAW;AACzB,iBAAO,KAAK;AAAA,YACV,WAAW,EAAE;AAAA,YACb,WAAW,GAAG,KAAK,MAAM,KAAK,CAAC;AAAA,YAC/B;AAAA,YACA,WAAW,EAAE,aAAa;AAAA,UAC5B,CAAC;AAAA,QACH;AAAA,MACF,CAAC;AAAA,IACH;AAAA,EACF;AAEA,QAAM,MAAM,sBAAsB,WAAW;AAE7C,QAAM,eAAyB,CAAC;AAChC,MAAI,MAAM,UAAU;AAClB,iBAAa,KAAK,WAAW,MAAM,GAAG,CAAC,MAAM,QAAQ,EAAE;AAAA,EACzD;AACA,aAAW,EAAE,QAAQ,OAAO,KAAK,YAAY;AAC3C,iBAAa,KAAK,YAAY,MAAM,WAAW,MAAM,MAAM,CAAC,MAAM,aAAa,sBAAiB;AAAA,EAClG;AACA,aAAW,EAAE,QAAQ,EAAE,KAAK,cAAc;AACxC,iBAAa,KAAK,YAAY,MAAM,KAAK,CAAC,uBAAuB,YAAY,EAAE;AAAA,EACjF;AAEA,SAAO;AAAA,IACL,aAAa,aAAa,WAAW;AAAA,IACrC;AAAA,IACA,uBAAuB;AAAA,IACvB;AAAA,EACF;AACF;AAGA,SAAS,MAAM,GAAmB;AAChC,SAAO,KAAK,MAAM,IAAI,GAAG,IAAI;AAC/B;;;AD1DA,SAAS,0BAAAA,+BAA8B;AAMvC;AAAA,EACE;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,OACK;AAmBP,SAAS,mBAAmB;AA9ErB,SAAS,mBACd,KACmC;AACnC,QAAM,OAAO,IAAI,aAAa;AAC9B,MAAI,OAAO,GAAG;AACZ,UAAM,IAAI,MAAM,mDAAmD,IAAI,GAAG;AAAA,EAC5E;AACA,MAAI,IAAI,OAAO,WAAW,GAAG;AAC3B,UAAM,IAAI,MAAM,qCAAqC;AAAA,EACvD;AACA,SAAO;AAAA,IACL,MAAM,IAAI;AAAA,IACV,GAAI,IAAI,iBAAiB,SAAY,CAAC,IAAI,EAAE,cAAc,IAAI,aAAa;AAAA,IAC3E,YAAY,IAAI,OAAO,IAAI,CAAC,SAAS,EAAE,KAAK,aAAa,IAAI,WAAW,GAAG,KAAK,IAAI,EAAE;AAAA,IACtF,MAAM,MAAM,OAA4B;AACtC,YAAM,UAAU,MAAM,QAAQ;AAAA,QAC5B,MAAM,KAAK,EAAE,QAAQ,KAAK,GAAG,CAAC,GAAG,QAAQ,IAAI,SAAS,EAAE,GAAG,OAAO,IAAI,CAAC,CAAC;AAAA,MAC1E;AACA,YAAM,WAA8B,QAAQ;AAAA,QAAI,CAAC,GAAG,QAClD,EAAE,WAAW,cACT,EAAE,QACF,EAAE,OAAO,GAAG,IAAI,IAAI,OAAO,GAAG,IAAI,cAAc,MAAM,WAAW,OAAO,EAAE,MAAM,EAAE;AAAA,MACxF;AAEA,YAAM,MAAM,uBAAuB,UAAU,IAAI,QAAQ,IAAI,OAAO;AACpE,aAAO,EAAE,WAAW,IAAI,WAAW,YAAY,IAAI,cAAc,OAAO,IAAI,UAAU;AAAA,IACxF;AAAA,EACF;AACF;","names":["aggregateJudgeVerdicts"]}

package/dist/eval-campaign/trust-gate.d.ts

@@ -1,28 +1,9 @@
 /**
- * Trust gate — decides whether an ensemble's scores are allowed to be BELIEVED,
- * one level up from {@link aggregateJudgeVerdicts} (which only reduces ONE
- * artifact's raters to a composite). A composite is a number; this is the check
- * that the number means anything. It is the code "Enforced by" for the
- * measurement-validation skill's after-gate ("is this result allowed to be
- * believed").
- *
- * Three checks, each fail-loud and named in `trustReasons`:
- *   (1) inter-rater reliability over the corpus ≥ `irrFloor` — raters that
- *       disagree no better than chance carry no signal to optimize against.
- *   (2) per-item rater spread ≤ `spreadCeiling` — for EACH item, raters must
- *       converge on THAT item.
- *   (3) surviving raters per item ≥ `minSurvivors` — a mean over one or two
- *       raters is an anecdote, not an ensemble.
- *
- * CRITICAL metric semantics — per-item spread is rater disagreement about the
- * SAME item: `max(score) − min(score)` across the raters that scored THAT item
- * (max over its dimensions), never pooled across different items or across the
- * baseline/candidate sides. Pooling reads a genuine quality gap BETWEEN items as
- * "the raters split" and so trips the gate exactly when the finding is largest —
- * the failure mode the after-gate exists to prevent. The corpus IRR (check 1)
- * leans on the substrate's `interRaterReliability`, whose expected-disagreement
- * denominator already pools across items, so genuine item-to-item variation
- * RAISES reliability rather than lowering it.
+ * Summarize rater agreement, within-item spread, and surviving-judge coverage.
+ * Consumers choose the thresholds that a trustworthy result must satisfy.
+ * Agreement does not measure evaluator errors against independent controls.
+ * Use agent-eval/meta-eval's auditEvaluator when the consumer needs that separate evidence.
+ * Spread stays within each item so differences in task quality cannot mimic rater disagreement.
  */
 import { type JudgeVerdict } from '@tangle-network/agent-eval';
 /** One item's raters: the per-judge verdicts {@link aggregateJudgeVerdicts}
@@ -34,11 +15,9 @@ export interface TrustItem<D extends string = string> {
      *  (`perDimension: null`) is dropped before spread/IRR, never folded as 0. */
     verdicts: readonly JudgeVerdict<D>[];
 }
-/** Thresholds for {@link trustVerdicts}. All overridable; defaults are the
- *  conservative after-gate bar. */
+/** Configurable agreement and coverage thresholds for {@link trustVerdicts}. */
 export interface TrustThresholds {
-    /** Minimum corpus inter-rater reliability (Krippendorff-style α). Below this
-     *  the raters agree no better than chance. Default 0.2. */
+    /** Minimum corpus inter-rater reliability (Krippendorff-style α). Default 0.2. */
     irrFloor?: number;
     /** Maximum per-item rater spread (`max − min` over a single item's surviving
      *  raters, across its dimensions). Above this the raters split ON THAT ITEM.
@@ -63,14 +42,12 @@ export interface TrustVerdict {
     perItemSpread: Record<string, number>;
 }
 /**
- * Decide whether an ensemble's per-item verdicts are trustworthy enough to
- * believe a lift computed from them. Pure: no LLM, no I/O, no clock, no random —
+ * Check an ensemble against its configured agreement and coverage thresholds. Pure: no LLM, no I/O, no clock, no random —
  * the same `items` + `thresholds` always yield the same verdict.
  *
  * Sibling to {@link aggregateJudgeVerdicts}: that reduces ONE item's raters to a
- * composite; this audits the raters ACROSS items and reports whether the
- * composites are believable. Run it on the corpus of held-out items before
- * reporting any lift over their scores.
+ * composite; this summarizes agreement across the supplied items.
+ * A passing result does not establish evaluator accuracy or authorize a release.
  *
  * @throws if `items` is empty — an empty corpus has no measurable trust, and a
  *   silent `trustworthy: true` over zero evidence is the exact lie the gate

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tangle-network/agent-app",
-  "version": "0.47.25",
+  "version": "0.48.2",
   "packageManager": "pnpm@11.24.0",
   "description": "Build agent applications with typed chat, tools, sandboxes, integrations, billing, and evaluation.",
   "keywords": [
@@ -535,15 +535,15 @@
     "@storybook/react-vite": "^10.5.10",
     "@tailwindcss/postcss": "^4.3.3",
     "@tangle-network/agent-docs": "0.2.1",
-    "@tangle-network/agent-eval": "0.180.0",
+    "@tangle-network/agent-eval": "0.182.0",
     "@tangle-network/agent-gateway": "0.10.0",
-    "@tangle-network/agent-integrations": "0.53.55",
-    "@tangle-network/agent-interface": "2.6.0",
-    "@tangle-network/agent-knowledge": "15.0.3",
-    "@tangle-network/agent-profile-materialize": "0.19.0",
-    "@tangle-network/agent-runtime": "0.222.1",
+    "@tangle-network/agent-integrations": "0.54.0",
+    "@tangle-network/agent-interface": "2.6.1",
+    "@tangle-network/agent-knowledge": "17.0.2",
+    "@tangle-network/agent-profile-materialize": "0.20.1",
+    "@tangle-network/agent-runtime": "0.231.1",
     "@tangle-network/brand": "1.5.0",
-    "@tangle-network/sandbox": "0.39.4",
+    "@tangle-network/sandbox": "0.40.2",
     "@tangle-network/sandbox-ui": "0.113.3",
     "@tangle-network/ui": "^11.8.0",
     "@testing-library/dom": "^10.4.1",
@@ -596,14 +596,14 @@
   "peerDependencies": {
     "@firecrawl/pdf-inspector-wasm": ">=0.1.3",
     "@huggingface/transformers": ">=3",
-    "@tangle-network/agent-eval": ">=0.180.0 <0.181.0",
-    "@tangle-network/agent-integrations": ">=0.53.55 <0.54.0",
-    "@tangle-network/agent-interface": "^2.6.0",
-    "@tangle-network/agent-knowledge": "^15.0.3",
-    "@tangle-network/agent-profile-materialize": ">=0.19.0 <0.20.0",
-    "@tangle-network/agent-runtime": ">=0.222.1 <0.223.0",
+    "@tangle-network/agent-eval": ">=0.182.0 <0.183.0",
+    "@tangle-network/agent-integrations": ">=0.53.55 <0.55.0",
+    "@tangle-network/agent-interface": "^2.6.1",
+    "@tangle-network/agent-knowledge": "^17.0.2",
+    "@tangle-network/agent-profile-materialize": ">=0.20.1 <0.21.0",
+    "@tangle-network/agent-runtime": ">=0.231.1 <0.232.0",
     "@tangle-network/brand": ">=1.5.0",
-    "@tangle-network/sandbox": ">=0.39.4 <0.40.0",
+    "@tangle-network/sandbox": ">=0.40.2 <0.41.0",
     "@tangle-network/sandbox-ui": ">=0.113.3 <0.114.0",
     "@tangle-network/ui": ">=11.6.0 <12.0.0",
     "@tiptap/core": ">=3.28.0 <4.0.0",