@tangle-network/agent-eval 0.65.0 → 0.67.0

package/CHANGELOG.md

@@ -4,6 +4,31 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-
 
 ---
 
+## [0.67.0] — 2026-05-30 — the promotion gate is statistically trustworthy (no more shipping noise)
+
+An adversarial review of a real "ship +4.0 lift" decision found it was a **triple false positive**: the driver's candidate lost on train, so the winner was the baseline (empty diff); the loop re-scored the baseline against ITSELF on the holdout and read run-to-run model noise (91 vs 95) as a "+4 lift"; and a point-estimate gate (`delta >= 0.03` on a 0-100 scale, `reps:1`) shipped it — while the reward-hacking gate was blind to a −30 regression on a safety dimension hiding under the +4 net. The promotion gate could not tell a real improvement from noise or from a Goodhart trade.
+
+### Fixed / Added
+
+- **No-op guard** (`runImprovementLoop`) — when the winner is byte-identical to the baseline (no candidate beat the training baseline, empty diff), the loop now forces `hold` and skips the meaningless baseline-vs-itself holdout pass, instead of shipping the noise delta.
+- **Statistical held-out gate** — `defaultProductionGate`'s held-out check is now a **paired bootstrap CI**, not a point estimate. It pairs candidate vs baseline holdout cells by **full `cellId` (`scenario:rep`)** — never averaging reps away — and ships only when the CI lower bound clears `deltaThreshold` (default 0 ⇒ confidently positive). Below `minProductiveRuns` (default 3) paired observations it HOLDS with `few_runs` rather than reading a degenerate interval. (New module `src/campaign/gates/statistical-heldout.ts`; reuses `pairedBootstrap` from `src/statistics.ts`.)
+- **Per-dimension regression guard (anti-Goodhart)** — `criticalDimensions` + `regressionTolerance` on `DefaultProductionGateOptions`. The gate HOLDS if any guarded dimension's paired-delta CI lower bound falls below −tolerance, even when the net composite rose. Tolerance auto-scales (0.05 on [0,1], 5 on 0-100) so a default expressed for one scale isn't a silent no-op on the other.
+- **Exports** `pairHoldout`, `heldoutSignificance`, `dimensionRegressions`, `detectScale` from `/campaign`.
+
+This collapses the duplicated gate tech-debt (a rigorous `src/held-out-gate.ts` existed but the loop wired the weak adapter) onto the shared `pairedBootstrap` statistics. 12 new regression tests, including the exact noisy-same-mean false positive and the composite-up/dimension-down Goodhart trade. Full suite (1624) green. The remaining path to a *proven* self-improvement (headroom corpus + Goodhart-resistant measurement, driver effectiveness, inter-cycle compounding) is tracked separately.
+
+## [0.66.0] — 2026-05-30 — the improvement loop can no longer hang silently or ingest to the wrong URL
+
+### Fixed
+
+- **`runCampaign` per-cell dispatch deadline (`dispatchTimeoutMs`).** A dispatch that neither resolves nor rejects — a stalled model request, an exhausted runtime resource, a stream that never closes — used to hang the cell, and with it the lane, the campaign, `runImprovementLoop`, and the CI job above them, **forever, with no diagnostic**. The cell now races its dispatch against the deadline; on timeout it aborts the cell's `ctx.signal` and records a LOUD error (`dispatch exceeded <N>ms`) while the campaign proceeds. `undefined`/`0` = unbounded (legacy).
+- **`runImprovementLoop` fails loud on an empty holdout.** When every holdout dispatch or judge errored, the gate read both means as 0, computed delta 0, and silently **"held" on garbage** — indistinguishable from a real no-lift result, masking upstream crashes (e.g. a consumer scorer that threw on a malformed scenario). The loop now throws a diagnostic error naming the first underlying failure instead of emitting a verdict over zero scorable cells. It also applies a default per-cell deadline (`DEFAULT_DISPATCH_TIMEOUT_MS`, 10 min, overridable) to every campaign it runs.
+- **Hosted ingest URL normalization.** The client appends the versioned `/v1/ingest/...` path itself, but callers (and the client's own prior doc) routinely pass the versioned base `https://host/v1` — producing `/v1/v1/ingest/...` → **404, silently dropping every event**. `post()` now strips a trailing `/v1` (and slashes) from the endpoint so both `https://host` and `https://host/v1` resolve correctly; the doc now shows the bare host.
+
+### Why it matters
+
+These three were a single failure chain in production: a consumer's judge threw on a subset of scenarios → the holdout produced no scorable cells → the loop hung instead of failing loud → no decision, no provenance — and even when it did complete, the activated ingest env (`…/v1`) 404'd. The loop now either completes with real data or fails loud, and its provenance lands.
+
 ## [0.65.0] — 2026-05-30 — `emitLoopProvenance` ships the eval-run event too (full dashboard visibility)
 
 ### Fixed

package/dist/adapters/otel.d.ts

@@ -1,4 +1,4 @@
-import { T as TraceSpanEvent, H as HostedClient } from '../index-CzhtwYBT.js';
+import { T as TraceSpanEvent, H as HostedClient } from '../index-DSEHMwvS.js';
 import '../types-c2R2kfmv.js';
 import '../run-record-BgTFzO2r.js';
 import '../errors-Dwqw-T_m.js';

package/dist/campaign/index.d.ts

@@ -1,8 +1,10 @@
-import { a as RunCampaignOptions, C as CampaignStorage } from '../provenance-lqyLpOYR.js';
-export { B as BuildLoopProvenanceArgs, D as DefaultProductionGateOptions, m as EmitLoopProvenanceArgs, n as EmitLoopProvenanceResult, E as EvolutionaryDriverOptions, o as GepaDriverConstraints, G as GepaDriverOptions, H as HeldOutGateOptions, p as LoopProvenanceBackend, q as LoopProvenanceCandidate, L as LoopProvenanceRecord, O as OpenAutoPrOptions, s as OpenAutoPrResult, b as RunEvalOptions, c as RunImprovementLoopOptions, R as RunImprovementLoopResult, t as RunOptimizationOptions, u as RunOptimizationResult, v as buildLoopProvenanceRecord, d as composeGate, w as countSentenceEdits, e as defaultProductionGate, x as defaultRenderDiff, y as emitLoopProvenance, f as evolutionaryDriver, z as extractH2Sections, g as fsCampaignStorage, h as gepaDriver, i as heldOutGate, j as inMemoryCampaignStorage, A as loopProvenanceSpans, F as openAutoPr, I as provenanceRecordPath, J as provenanceSpansPath, r as runCampaign, k as runEval, l as runImprovementLoop, K as runOptimization, M as surfaceContentHash, N as surfaceHash } from '../provenance-lqyLpOYR.js';
+import { a as RunCampaignOptions, C as CampaignStorage } from '../run-improvement-loop-BKpM5T4t.js';
+export { d as GepaDriverConstraints, G as GepaDriverOptions, O as OpenAutoPrOptions, e as OpenAutoPrResult, b as RunImprovementLoopOptions, R as RunImprovementLoopResult, h as RunOptimizationOptions, j as RunOptimizationResult, k as countSentenceEdits, l as defaultRenderDiff, m as extractH2Sections, f as fsCampaignStorage, g as gepaDriver, i as inMemoryCampaignStorage, o as openAutoPr, r as runCampaign, c as runImprovementLoop, n as runOptimization, s as surfaceHash } from '../run-improvement-loop-BKpM5T4t.js';
+export { B as BuildLoopProvenanceArgs, D as DefaultProductionGateOptions, a as EmitLoopProvenanceArgs, b as EmitLoopProvenanceResult, E as EvolutionaryDriverOptions, H as HeldOutGateOptions, f as LoopProvenanceBackend, g as LoopProvenanceCandidate, L as LoopProvenanceRecord, R as RunEvalOptions, i as buildLoopProvenanceRecord, c as composeGate, d as defaultProductionGate, j as emitLoopProvenance, e as evolutionaryDriver, h as heldOutGate, l as loopProvenanceSpans, p as provenanceRecordPath, k as provenanceSpansPath, r as runEval, s as surfaceContentHash } from '../provenance-CChUqexv.js';
 import { L as LlmClientOptions } from '../llm-client-DbjLfz-K.js';
-import { I as ImprovementDriver, L as LabeledScenarioStore, q as LabeledScenarioWrite, r as LabeledScenarioSampleArgs, s as LabeledScenarioRecord, t as LabelTrust, S as Scenario, M as MutableSurface, b as DispatchContext, a as JudgeConfig, u as LabeledScenarioSource, f as CampaignResult, h as CodeSurface } from '../types-c2R2kfmv.js';
-export { C as CampaignAggregates, c as CampaignArtifactWriter, d as CampaignCellResult, e as CampaignCostMeter, v as CampaignTokenUsage, g as CampaignTraceWriter, D as DispatchFn, G as Gate, i as GateContext, j as GateDecision, k as GateResult, l as GenerationCandidate, m as GenerationRecord, w as JudgeAggregate, n as JudgeDimension, J as JudgeScore, o as Mutator, O as OptimizerConfig, P as ParetoParent, x as ProposeContext, y as ProposedCandidate, R as RedactionStatus, z as ScenarioAggregate, p as SessionScript, T as TraceSpan, A as isProposedCandidate, B as labelTrustRank } from '../types-c2R2kfmv.js';
+import { I as ImprovementDriver, J as JudgeScore, L as LabeledScenarioStore, q as LabeledScenarioWrite, r as LabeledScenarioSampleArgs, s as LabeledScenarioRecord, t as LabelTrust, S as Scenario, M as MutableSurface, b as DispatchContext, a as JudgeConfig, u as LabeledScenarioSource, f as CampaignResult, h as CodeSurface } from '../types-c2R2kfmv.js';
+export { C as CampaignAggregates, c as CampaignArtifactWriter, d as CampaignCellResult, e as CampaignCostMeter, v as CampaignTokenUsage, g as CampaignTraceWriter, D as DispatchFn, G as Gate, i as GateContext, j as GateDecision, k as GateResult, l as GenerationCandidate, m as GenerationRecord, w as JudgeAggregate, n as JudgeDimension, o as Mutator, O as OptimizerConfig, P as ParetoParent, x as ProposeContext, y as ProposedCandidate, R as RedactionStatus, z as ScenarioAggregate, p as SessionScript, T as TraceSpan, A as isProposedCandidate, B as labelTrustRank } from '../types-c2R2kfmv.js';
+import { a as PairedBootstrapResult } from '../statistics-B7yCbi9i.js';
 import { A as AgentProfile, B as BackendIntegrityReport } from '../agent-profile-DzcPHR1Z.js';
 import { A as AgentEvalError } from '../errors-Dwqw-T_m.js';
 import { b as RunSplitTag, R as RunRecord } from '../run-record-BgTFzO2r.js';
@@ -10,11 +12,13 @@ import '../red-team-DW9Ca_tj.js';
 import '../dataset-B2kL-fSM.js';
 import '../store-CKUAgsJz.js';
 import '../schema-m0gsnbt3.js';
-import '../index-CzhtwYBT.js';
+import '../index-DSEHMwvS.js';
 import '../summary-report-ByiOUrHj.js';
 import '../failure-cluster-CL7IVgkJ.js';
 import '../judge-calibration-DilmB3Ml.js';
 import '../raw-provider-sink-C46HDghv.js';
+import '../types-Croy5h7V.js';
+import '@tangle-network/tcloud';
 
 /**
  * @experimental
@@ -163,6 +167,106 @@ declare class SkillPatchParseError extends Error {
 }
 declare function parseSkillPatchResponse(raw: string, maxPatches: number, editBudget: number): SkillPatch[];
 
+/**
+ * @experimental
+ *
+ * Statistical held-out promotion machinery — the trustworthy core the
+ * point-estimate `heldout-delta` gate lacked.
+ *
+ * The shipped false positive it prevents: a winner re-scored against the
+ * baseline on the holdout read run-to-run model NOISE (e.g. 91 vs 95) as a
+ * "+4 lift" and shipped, because the gate compared point estimates with no
+ * confidence interval. Here we pair candidate vs baseline holdout observations
+ * and bootstrap a CI on the paired delta — a candidate ships only when the CI
+ * lower bound clears the effect-size threshold (the gain is real at the
+ * confidence level, not noise), and is blocked when a critical dimension
+ * (e.g. `hallucination_free` for a legal agent) significantly regresses even if
+ * the net composite rose (anti-Goodhart).
+ *
+ * Two traps this module is built around (both produce a NEW false positive if
+ * gotten wrong):
+ *   1. PAIRING GRANULARITY — pairs by FULL `cellId` (`scenario:rep`), never by
+ *      `scenarioId` (which averages reps away and destroys the within-pair
+ *      variance reduction that makes a paired bootstrap tighter than unpaired).
+ *      One paired observation per cell ⇒ reps multiply n.
+ *   2. SCALE — a judge may emit composites/dimensions on [0,1] or 0-100. The
+ *      threshold + tolerance are interpreted in the judge's NATIVE scale; the
+ *      per-dimension tolerance auto-scales off the observed baseline magnitudes
+ *      so `-0.10` on [0,1] doesn't silently become a no-op on a 0-100 dimension.
+ */
+
+interface PairedHoldout {
+    /** Baseline scalar per paired cell (same order as `after`/`cellIds`). */
+    before: number[];
+    /** Candidate scalar per paired cell. */
+    after: number[];
+    /** The full cellIds (`scenario:rep`) that paired, in order. */
+    cellIds: string[];
+}
+/**
+ * Pair candidate vs baseline holdout observations by FULL cellId. `select`
+ * pulls the scalar from a cell's judge reports (composite, or a named
+ * dimension); a cell contributes the mean of `select` across its judges. Cells
+ * whose scenario is not in `scenarioIds`, or where `select` is undefined for
+ * every judge on either side, are skipped on BOTH sides so the arrays stay
+ * paired. Throws when the two maps disagree on which holdout cells exist — a
+ * load-bearing invariant: the baseline + winner holdout campaigns run the same
+ * scenarios with the same seed base, so their cellIds MUST align; a mismatch
+ * means a silent pairing bug, not a soft fallback.
+ */
+declare function pairHoldout(candidate: Map<string, Record<string, JudgeScore>>, baseline: Map<string, Record<string, JudgeScore>>, scenarioIds: Set<string>, select: (s: JudgeScore) => number | undefined): PairedHoldout;
+interface HeldoutSignificance {
+    paired: PairedHoldout;
+    bootstrap: PairedBootstrapResult;
+    /** n paired observations. */
+    n: number;
+    /** True iff n >= minProductiveRuns AND the CI lower bound clears the threshold. */
+    significant: boolean;
+    /** Set when n < minProductiveRuns — too little evidence to claim significance. */
+    fewRuns: boolean;
+}
+interface HeldoutSignificanceOptions {
+    deltaThreshold?: number;
+    minProductiveRuns?: number;
+    confidence?: number;
+    resamples?: number;
+    /** Fixed by default for a deterministic, reproducible gate verdict. */
+    seed?: number;
+    statistic?: 'mean' | 'median';
+}
+/** Significance of the held-out composite lift: ship only when the paired
+ *  bootstrap CI lower bound on (candidate − baseline) exceeds `deltaThreshold`
+ *  (default 0 ⇒ "confidently positive"). Below `minProductiveRuns` paired
+ *  observations there is not enough evidence to claim significance → not
+ *  significant (`fewRuns`). Interpret `deltaThreshold` in the judge's native
+ *  composite scale. */
+declare function heldoutSignificance(paired: PairedHoldout, opts?: HeldoutSignificanceOptions): HeldoutSignificance;
+interface DimensionRegression {
+    dimension: string;
+    bootstrap: PairedBootstrapResult;
+    /** True iff the CI lower bound on (candidate − baseline) is below −tolerance:
+     *  the candidate may have regressed this dimension by more than tolerance. */
+    regressed: boolean;
+    tolerance: number;
+    n: number;
+}
+/** Detect the native scale of a set of scores: 0-100 when any magnitude clears
+ *  1.5, else [0,1]. Used to auto-scale the regression tolerance so a default
+ *  expressed for [0,1] is not silently a no-op on a 0-100 dimension. */
+declare function detectScale(values: number[]): 1 | 100;
+/** Per-critical-dimension regression guard. For each dimension, pair the
+ *  candidate vs baseline values by full cellId and bootstrap the paired delta;
+ *  a dimension is "regressed" when the CI lower bound < −tolerance (conservative
+ *  — blocks if the credible worst case exceeds tolerance, which is the right
+ *  posture for safety dimensions like `hallucination_free`). When `tolerance`
+ *  is omitted it auto-scales: 0.05 on [0,1], 5 on 0-100. */
+declare function dimensionRegressions(candidate: Map<string, Record<string, JudgeScore>>, baseline: Map<string, Record<string, JudgeScore>>, scenarioIds: Set<string>, criticalDimensions: string[], opts?: {
+    tolerance?: number;
+    confidence?: number;
+    resamples?: number;
+    seed?: number;
+}): DimensionRegression[];
+
 /**
  * @experimental
  *
@@ -647,4 +751,4 @@ declare function gitWorktreeAdapter(opts: GitWorktreeAdapterOptions): WorktreeAd
  *  as a ref under the adapter's worktree dir. */
 declare function resolveWorktreePath(surface: CodeSurface, worktreeDir?: string): string;
 
-export { type AcceptedEdit, type ApplySkillPatchResult, type CampaignBreakdown, CampaignResult, CampaignStorage, CodeSurface, type CompareDriversOptions, DispatchContext, type DriverComparison, type DriverEntry, type DriverPairwise, type DriverScore, FsLabeledScenarioStore, type FsLabeledScenarioStoreOptions, type GitWorktreeAdapterOptions, ImprovementDriver, JudgeConfig, LabelTrust, LabeledScenarioRecord, LabeledScenarioSampleArgs, LabeledScenarioSource, LabeledScenarioStore, LabeledScenarioStoreError, LabeledScenarioWrite, MutableSurface, type OptimizerEntryConfig, type ProfileDispatchFn, ProfileMatrixError, type ProfileSummary, type ProposePatchesArgs, type RejectedEdit, RunCampaignOptions, type RunProfileMatrixOptions, type RunProfileMatrixResult, type RunSkillOptOptions, type RunSkillOptResult, Scenario, type ScenarioRollup, type SkillOptDriver, type SkillOptDriverOptions, type SkillOptEpochRecord, type SkillOptEvidence, type SkillPatch, type SkillPatchOp, SkillPatchParseError, type SkillPatchRejection, type Worktree, type WorktreeAdapter, WorktreeAdapterError, applySkillPatch, campaignBreakdown, campaignMeanComposite, compareDrivers, gepaParetoEntry, gepaReflectionEntry, gitWorktreeAdapter, parseSkillPatchResponse, patchEditCount, resolveWorktreePath, runProfileMatrix, runSkillOpt, skillOptDriver, skillOptEntry };
+export { type AcceptedEdit, type ApplySkillPatchResult, type CampaignBreakdown, CampaignResult, CampaignStorage, CodeSurface, type CompareDriversOptions, type DimensionRegression, DispatchContext, type DriverComparison, type DriverEntry, type DriverPairwise, type DriverScore, FsLabeledScenarioStore, type FsLabeledScenarioStoreOptions, type GitWorktreeAdapterOptions, type HeldoutSignificance, type HeldoutSignificanceOptions, ImprovementDriver, JudgeConfig, JudgeScore, LabelTrust, LabeledScenarioRecord, LabeledScenarioSampleArgs, LabeledScenarioSource, LabeledScenarioStore, LabeledScenarioStoreError, LabeledScenarioWrite, MutableSurface, type OptimizerEntryConfig, type PairedHoldout, type ProfileDispatchFn, ProfileMatrixError, type ProfileSummary, type ProposePatchesArgs, type RejectedEdit, RunCampaignOptions, type RunProfileMatrixOptions, type RunProfileMatrixResult, type RunSkillOptOptions, type RunSkillOptResult, Scenario, type ScenarioRollup, type SkillOptDriver, type SkillOptDriverOptions, type SkillOptEpochRecord, type SkillOptEvidence, type SkillPatch, type SkillPatchOp, SkillPatchParseError, type SkillPatchRejection, type Worktree, type WorktreeAdapter, WorktreeAdapterError, applySkillPatch, campaignBreakdown, campaignMeanComposite, compareDrivers, detectScale, dimensionRegressions, gepaParetoEntry, gepaReflectionEntry, gitWorktreeAdapter, heldoutSignificance, pairHoldout, parseSkillPatchResponse, patchEditCount, resolveWorktreePath, runProfileMatrix, runSkillOpt, skillOptDriver, skillOptEntry };

package/dist/campaign/index.js

@@ -1,41 +1,44 @@
 import {
   buildLoopProvenanceRecord,
-  campaignBreakdown,
-  campaignMeanComposite,
   composeGate,
-  countSentenceEdits,
   defaultProductionGate,
-  defaultRenderDiff,
+  detectScale,
+  dimensionRegressions,
   emitLoopProvenance,
   evolutionaryDriver,
+  heldoutSignificance,
+  loopProvenanceSpans,
+  pairHoldout,
+  provenanceRecordPath,
+  provenanceSpansPath,
+  runEval,
+  surfaceContentHash
+} from "../chunk-MZ2IYGGN.js";
+import {
+  agentProfileHash
+} from "../chunk-PQV2TKC3.js";
+import {
+  campaignBreakdown,
+  campaignMeanComposite,
+  countSentenceEdits,
+  defaultRenderDiff,
   extractH2Sections,
   gepaDriver,
   heldOutGate,
   isProposedCandidate,
   labelTrustRank,
-  loopProvenanceSpans,
   openAutoPr,
-  provenanceRecordPath,
-  provenanceSpansPath,
-  runEval,
   runImprovementLoop,
   runOptimization,
-  surfaceContentHash,
   surfaceHash
-} from "../chunk-CZRKD2X2.js";
+} from "../chunk-NV2PF37Q.js";
 import {
+  assertRealBackend,
   fsCampaignStorage,
   inMemoryCampaignStorage,
-  runCampaign
-} from "../chunk-7TPYV2ER.js";
-import {
-  agentProfileHash
-} from "../chunk-PQV2TKC3.js";
-import "../chunk-4ODZXQV2.js";
-import {
-  assertRealBackend,
+  runCampaign,
   summarizeBackendIntegrity
-} from "../chunk-E22YUOAL.js";
+} from "../chunk-6XQIEUQ2.js";
 import "../chunk-YV7J7X5N.js";
 import {
   validateRunRecord
@@ -1092,6 +1095,8 @@ export {
   countSentenceEdits,
   defaultProductionGate,
   defaultRenderDiff,
+  detectScale,
+  dimensionRegressions,
   emitLoopProvenance,
   evolutionaryDriver,
   extractH2Sections,
@@ -1101,11 +1106,13 @@ export {
   gepaReflectionEntry,
   gitWorktreeAdapter,
   heldOutGate,
+  heldoutSignificance,
   inMemoryCampaignStorage,
   isProposedCandidate,
   labelTrustRank,
   loopProvenanceSpans,
   openAutoPr,
+  pairHoldout,
   parseSkillPatchResponse,
   patchEditCount,
   provenanceRecordPath,