@tangle-network/agent-eval 0.107.0 → 0.108.0

package/CHANGELOG.md

@@ -4,6 +4,27 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-
 
 ---
 
+## [0.108.0] — 2026-07-08 — placebo control reaches the facades
+
+### Added
+
+- **`neutralize` passthrough on `selfImprove`.** 0.107.0 wired the footprint-matched placebo arm at `runImprovementLoop`; the public `selfImprove` facade did not forward it, so the placebo gate was unreachable from the one-call entry point. `selfImprove({ ..., neutralize })` now scores the third placebo arm and exposes `ctx.neutralizedJudgeScores` to the gate — compose `neutralizationGate` into `gate` to act on it.
+
+Additive (one optional field); no consumer bump required.
+
+---
+
+## [0.107.0] — 2026-07-07 — footprint-matched placebo promotion gate
+
+### Added
+
+- **`neutralizationGate` + `neutralizeText` (`/campaign`).** A composable promotion gate that proves a held-out lift comes from the candidate's *content* rather than from the prompt/mount *footprint* the content added. A held-out gate proves "candidate beat baseline"; it cannot tell an informative surface from one that merely added bytes the model spends attention on. `neutralizationGate` compares the candidate's lift against a footprint-matched neutralized variant (same layout + length, zero content, via `neutralizeText`) and holds any win whose lift survives blanking (decorative) — however large or significant the raw lift. Compose after significance: `composeGate(heldOutGate({ … }), neutralizationGate({ … }))`.
+- **`runImprovementLoop({ neutralize })` + `GateContext.neutralizedJudgeScores` / `neutralizedArtifacts`.** When a `neutralize` function is supplied and the winner changed, the loop scores a third holdout arm (the blanked winner) and exposes it to the gate. Opt-in — one extra holdout campaign only when wired; existing callers are unaffected.
+
+Additive (new exports + optional fields only); no consumer bump required.
+
+---
+
 ## [0.100.3] — 2026-07-01 — product benchmark contract + eval fixture UX
 
 ### Added

package/README.md

@@ -122,7 +122,7 @@ Each example: `README.md` + a single `index.ts` runnable via `pnpm tsx`. Prints
 
 | Subpath | What it gives you |
 |---|---|
-| `…/contract` | **The headline, frozen surface — new code starts here.** `defineAgentEval`, `selfImprove`, `analyzeRuns`, `runEval`, `runCampaign`, `runImprovementLoop`, `diffRuns`; intake adapters (`fromFeedbackTable`, `fromOtelSpans`); proposers (`gepaProposer`, `evolutionaryProposer`); gates (`defaultProductionGate`, `heldOutGate`, `paretoSignificanceGate`, `composeGate`); the deployment-outcome store; storage; and the five core types `Scenario` / `Dispatch` / `JudgeConfig` / `SurfaceProposer` / `Gate`. |
+| `…/contract` | **The headline, frozen surface — new code starts here.** `defineAgentEval`, `selfImprove`, `analyzeRuns`, `runEval`, `runCampaign`, `runImprovementLoop`, `diffRuns`; intake adapters (`fromFeedbackTable`, `fromOtelSpans`); proposers (`gepaProposer`, `evolutionaryProposer`); gates (`defaultProductionGate`, `heldOutGate`, `paretoSignificanceGate`, `neutralizationGate`, `composeGate`); the deployment-outcome store; storage; and the five core types `Scenario` / `Dispatch` / `JudgeConfig` / `SurfaceProposer` / `Gate`. |
 | `…/hosted` | `createHostedClient` / `hostedClientFromEnv` + the wire types to ship eval-run events + trace spans to a hosted orchestrator (ours or your own implementation of the spec) |
 | `…/adapters/otel` | `createOtelBridge` — forwards OpenTelemetry-shape spans into the hosted-tier ingest, no `@opentelemetry/*` dependency |
 | `…/adapters/langchain` | Wrap any LangChain `Runnable` as a `Dispatch` (or `JudgeConfig`), no `@langchain/core` peer dep |
@@ -136,7 +136,7 @@ Each example: `README.md` + a single `index.ts` runnable via `pnpm tsx`. Prints
 | `…/matrix` | `runAgentMatrix` — an N-axis cartesian over caller-supplied substrate values, per-axis pass/score/cost/duration |
 | `…/multishot` | N-shot persona × shot matrix runner (`runMultishot` / `runMultishotMatrix`) |
 | `…/wire` | The cross-language HTTP/RPC server + Zod schemas (the source-of-truth protocol the Python client speaks) + the built-in rubric registry |
-| `…/benchmarks` | `BenchmarkAdapter` contract + `deterministicSplit` + the bundled `routing` reference benchmark |
+| `…/benchmarks` | `BenchmarkAdapter` contract, `runBenchmarkAdapter`, `calibrateBenchmarkMetric`, standard retrieval parsers + ranked retrieval metrics, `deterministicSplit`, and the bundled `routing` reference benchmark |
 
 **Specialized surfaces** (subpath-only): `…/prm` (process-reward grading + best-of-N), `…/meta-eval` (judge calibration + the deployment-outcome store), `…/belief-state` (decision-point extraction + selective-policy reports), `…/pipelines` (trace-diagnostic views: budget breach, failure cluster, stuck loop, …), `…/governance` (EU AI Act / NIST AI RMF / SOC2 reports), `…/knowledge` (knowledge-readiness gating before a run), `…/builder-eval` (code-generator three-layer eval), `…/storyboard` (trace → watchable replay), `…/authenticity` (anti-Goodhart "real or convincing BS" scorer over produced files), `…/workflow` (workflow-trace eval + partner export), `…/telemetry` (Workers-safe telemetry client), `…/testing` (test-only reset helpers).
 

package/dist/adapters/http.d.ts

@@ -1,4 +1,4 @@
-import { S as Scenario, D as DispatchFn, b as DispatchContext } from '../types-CNZ0tHET.js';
+import { S as Scenario, D as DispatchFn, b as DispatchContext } from '../types-BdIv5dvA.js';
 import '../run-record-I-Z3JNvO.js';
 import '@tangle-network/agent-interface';
 import '../errors-oeQrLqXC.js';

package/dist/adapters/langchain.d.ts

@@ -1,4 +1,4 @@
-import { S as Scenario, J as JudgeScore, D as DispatchFn, a as JudgeConfig } from '../types-CNZ0tHET.js';
+import { S as Scenario, J as JudgeScore, D as DispatchFn, a as JudgeConfig } from '../types-BdIv5dvA.js';
 import '../run-record-I-Z3JNvO.js';
 import '@tangle-network/agent-interface';
 import '../errors-oeQrLqXC.js';

package/dist/adapters/otel.d.ts

@@ -1,5 +1,5 @@
 import { TraceSpanEvent, HostedClient } from '../hosted/index.js';
-import '../types-CNZ0tHET.js';
+import '../types-BdIv5dvA.js';
 import '../run-record-I-Z3JNvO.js';
 import '@tangle-network/agent-interface';
 import '../errors-oeQrLqXC.js';

package/dist/benchmarks/index.d.ts

@@ -1,5 +1,7 @@
-export { B as BENCHMARK_SPLIT_SEED, a as BenchmarkAdapter, b as BenchmarkDatasetItem, c as BenchmarkEvaluation, d as deterministicSplit, e as routing } from '../index-C2CC_dry.js';
+export { B as BENCHMARK_SPLIT_SEED, a as BenchmarkAdapter, b as BenchmarkDatasetItem, k as BenchmarkDistribution, c as BenchmarkEvaluation, d as BenchmarkFamily, l as BenchmarkMetricCalibrationOptions, m as BenchmarkMetricCalibrationResult, n as BenchmarkReport, e as BenchmarkResponder, o as BenchmarkRunOptions, p as BenchmarkRunResult, f as BenchmarkScenario, q as BenchmarkSliceSummary, g as BenchmarkSource, h as BenchmarkTaskKind, r as BuildStandardRetrievalItemsOptions, R as RetrievalIdAdapterOptions, S as StandardRetrievalArtifact, s as StandardRetrievalDocument, t as StandardRetrievalEvaluationOptions, u as StandardRetrievalPayload, v as StandardRetrievalQrel, w as StandardRetrievalQuery, x as StandardRetrievalResult, y as buildStandardRetrievalItems, z as calibrateBenchmarkMetric, A as createRetrievalIdBenchmarkAdapter, i as deterministicSplit, C as evaluateStandardRetrieval, D as normalizeRetrievedDocumentIds, E as parseBeirCorpusJsonl, F as parseBeirQueriesJsonl, G as parseJsonlRows, H as parseQrels, I as parseTsvRows, J as renderBenchmarkReportMarkdown, K as retrievalMetricsAtCutoff, L as routing, M as runBenchmarkAdapter, N as summarizeBenchmarkCampaign } from '../index-pPtfoIJO.js';
+import '../types-BdIv5dvA.js';
 import '../run-record-I-Z3JNvO.js';
 import '@tangle-network/agent-interface';
 import '../errors-oeQrLqXC.js';
 import '../schema-m0gsnbt3.js';
+import '../storage-Dw_f7WMt.js';

package/dist/benchmarks/index.js

@@ -1,12 +1,58 @@
 import {
   BENCHMARK_SPLIT_SEED,
+  buildStandardRetrievalItems,
+  calibrateBenchmarkMetric,
+  createRetrievalIdBenchmarkAdapter,
   deterministicSplit,
-  routing_exports
-} from "../chunk-6QDKWHLS.js";
+  evaluateStandardRetrieval,
+  normalizeRetrievedDocumentIds,
+  parseBeirCorpusJsonl,
+  parseBeirQueriesJsonl,
+  parseJsonlRows,
+  parseQrels,
+  parseTsvRows,
+  renderBenchmarkReportMarkdown,
+  retrievalMetricsAtCutoff,
+  routing_exports,
+  runBenchmarkAdapter,
+  summarizeBenchmarkCampaign
+} from "../chunk-T6W5ADLG.js";
+import "../chunk-4VLZEPJ3.js";
+import "../chunk-OVPVM4JC.js";
+import "../chunk-3PFZBGMR.js";
+import "../chunk-VI2UW6B6.js";
+import "../chunk-LOJ2QVCE.js";
+import "../chunk-2OGPXHOB.js";
+import "../chunk-N22ZO7FV.js";
+import "../chunk-FUCQVFMU.js";
+import "../chunk-45EEMHTC.js";
+import "../chunk-XMSYF4A7.js";
+import "../chunk-RPDDVKI7.js";
+import "../chunk-GGE4NNQT.js";
+import "../chunk-LNQEP766.js";
+import "../chunk-PC4UYEBM.js";
+import "../chunk-VK6HBGAE.js";
+import "../chunk-XJYR7XFV.js";
+import "../chunk-VSMTAMNK.js";
+import "../chunk-ONWEPEDO.js";
 import "../chunk-PZ5AY32C.js";
 export {
   BENCHMARK_SPLIT_SEED,
+  buildStandardRetrievalItems,
+  calibrateBenchmarkMetric,
+  createRetrievalIdBenchmarkAdapter,
   deterministicSplit,
-  routing_exports as routing
+  evaluateStandardRetrieval,
+  normalizeRetrievedDocumentIds,
+  parseBeirCorpusJsonl,
+  parseBeirQueriesJsonl,
+  parseJsonlRows,
+  parseQrels,
+  parseTsvRows,
+  renderBenchmarkReportMarkdown,
+  retrievalMetricsAtCutoff,
+  routing_exports as routing,
+  runBenchmarkAdapter,
+  summarizeBenchmarkCampaign
 };
 //# sourceMappingURL=index.js.map

package/dist/campaign/index.d.ts

@@ -1,11 +1,13 @@
-import { S as SignedManifest, B as BackendIntegrityReport, C as CompletionRequirement, R as RuntimeEventLike, a as CompletionVerdict, P as ProducedState, b as CorrectnessChecker } from '../pre-registration-x3f0VpxT.js';
-export { L as LlmJudgeDimension, c as LlmJudgeOptions, l as llmJudge } from '../pre-registration-x3f0VpxT.js';
+import { S as SignedManifest, B as BackendIntegrityReport, C as CompletionRequirement, R as RuntimeEventLike, a as CompletionVerdict, P as ProducedState, b as CorrectnessChecker } from '../pre-registration-BUhVPzE7.js';
+export { L as LlmJudgeDimension, c as LlmJudgeOptions, l as llmJudge } from '../pre-registration-BUhVPzE7.js';
 import { A as AnalyzeTracesOptions, a as AnalyzeTracesInput, b as AnalyzeTracesResult } from '../analyst-C8HHvfJp.js';
-import { S as Scenario, M as MutableSurface, b as DispatchContext, a as JudgeConfig, g as Gate, e as GenerationRecord, J as JudgeScore, L as LabeledScenarioStore, s as LabeledScenarioWrite, t as LabeledScenarioSampleArgs, u as LabeledScenarioRecord, v as LabelTrust, f as SurfaceProposer, w as ProposedCandidate, x as ProposeContext, y as LabeledScenarioSource, C as CampaignResult, m as CodeSurface } from '../types-CNZ0tHET.js';
-export { i as CampaignAggregates, j as CampaignArtifactWriter, k as CampaignCellResult, l as CampaignCostMeter, z as CampaignTokenUsage, d as CampaignTraceWriter, D as DispatchFn, n as GateContext, h as GateDecision, G as GateResult, o as GenerationCandidate, A as JudgeAggregate, c as JudgeDimension, p as Mutator, O as OptimizationProposer, q as OptimizerConfig, P as ParetoParent, R as RedactionStatus, B as ScenarioAggregate, r as SessionScript, T as TraceSpan, E as isProposedCandidate, F as labelTrustRank } from '../types-CNZ0tHET.js';
-import { e as CampaignRunPlan, P as PlanCampaignRunOptions, C as CampaignStorage, b as RunCampaignOptions, c as RunImprovementLoopOptions } from '../gepa-BmqTrdtg.js';
-export { h as CampaignRunPlanCell, j as GepaProposerConstraints, G as GepaProposerOptions, O as OpenAutoPrOptions, k as OpenAutoPrResult, a as RunImprovementLoopResult, R as RunOptimizationOptions, l as RunOptimizationResult, m as countSentenceEdits, n as defaultRenderDiff, o as extractH2Sections, f as fsCampaignStorage, g as gepaProposer, i as inMemoryCampaignStorage, p as openAutoPr, q as planCampaignRun, r as runCampaign, d as runImprovementLoop, s as runOptimization, t as surfaceHash } from '../gepa-BmqTrdtg.js';
-export { A as AxisEvidence, a as AxisVerdict, B as BuildEvidenceVectorOptions, l as BuildLoopProvenanceArgs, D as DefaultProductionGateOptions, m as EmitLoopProvenanceArgs, n as EmitLoopProvenanceResult, E as EvidenceVector, b as EvolutionaryProposerOptions, H as HeldOutGateOptions, o as LoopProvenanceBackend, q as LoopProvenanceCandidate, L as LoopProvenanceRecord, O as ObjectiveSource, c as ParetoSignificanceGateOptions, P as PowerPreflight, s as PowerPreflightOptions, d as PromotionObjective, e as PromotionPolicy, R as RunEvalOptions, f as buildEvidenceVector, t as buildLoopProvenanceRecord, g as composeGate, h as defaultProductionGate, u as emitLoopProvenance, i as evolutionaryProposer, j as heldOutGate, v as loopProvenanceSpans, p as paretoPolicy, k as paretoSignificanceGate, w as powerPreflight, x as provenanceRecordPath, y as provenanceSpansPath, r as runEval, z as surfaceContentHash } from '../provenance-CIjRcI6m.js';
+import { S as Scenario, M as MutableSurface, b as DispatchContext, a as JudgeConfig, f as Gate, d as GenerationRecord, J as JudgeScore, L as LabeledScenarioStore, s as LabeledScenarioWrite, t as LabeledScenarioSampleArgs, u as LabeledScenarioRecord, v as LabelTrust, e as SurfaceProposer, w as ProposedCandidate, x as ProposeContext, y as LabeledScenarioSource, C as CampaignResult, m as CodeSurface } from '../types-BdIv5dvA.js';
+export { i as CampaignAggregates, j as CampaignArtifactWriter, k as CampaignCellResult, l as CampaignCostMeter, z as CampaignTokenUsage, c as CampaignTraceWriter, D as DispatchFn, n as GateContext, h as GateDecision, G as GateResult, o as GenerationCandidate, A as JudgeAggregate, g as JudgeDimension, p as Mutator, O as OptimizationProposer, q as OptimizerConfig, P as ParetoParent, R as RedactionStatus, B as ScenarioAggregate, r as SessionScript, T as TraceSpan, E as isProposedCandidate, F as labelTrustRank } from '../types-BdIv5dvA.js';
+import { C as CampaignRunPlan, P as PlanCampaignRunOptions, b as RunCampaignOptions, c as RunImprovementLoopOptions } from '../gepa-B3x5Ulcv.js';
+export { e as CampaignRunPlanCell, f as GepaProposerConstraints, G as GepaProposerOptions, O as OpenAutoPrOptions, h as OpenAutoPrResult, a as RunImprovementLoopResult, R as RunOptimizationOptions, i as RunOptimizationResult, j as countSentenceEdits, k as defaultRenderDiff, l as extractH2Sections, g as gepaProposer, o as openAutoPr, p as planCampaignRun, r as runCampaign, d as runImprovementLoop, m as runOptimization, s as surfaceHash } from '../gepa-B3x5Ulcv.js';
+import { C as CampaignStorage } from '../storage-Dw_f7WMt.js';
+export { f as fsCampaignStorage, i as inMemoryCampaignStorage } from '../storage-Dw_f7WMt.js';
+export { A as AxisEvidence, a as AxisVerdict, B as BuildEvidenceVectorOptions, l as BuildLoopProvenanceArgs, D as DefaultProductionGateOptions, m as EmitLoopProvenanceArgs, n as EmitLoopProvenanceResult, E as EvidenceVector, b as EvolutionaryProposerOptions, H as HeldOutGateOptions, o as LoopProvenanceBackend, q as LoopProvenanceCandidate, L as LoopProvenanceRecord, O as ObjectiveSource, c as ParetoSignificanceGateOptions, P as PowerPreflight, s as PowerPreflightOptions, d as PromotionObjective, e as PromotionPolicy, R as RunEvalOptions, f as buildEvidenceVector, t as buildLoopProvenanceRecord, g as composeGate, h as defaultProductionGate, u as emitLoopProvenance, i as evolutionaryProposer, j as heldOutGate, v as loopProvenanceSpans, p as paretoPolicy, k as paretoSignificanceGate, w as powerPreflight, x as provenanceRecordPath, y as provenanceSpansPath, r as runEval, z as surfaceContentHash } from '../provenance-DdDhf6cg.js';
 import { E as EProcessState, a as PairedBootstrapResult } from '../statistics-D88peojY.js';
 import { L as LlmClientOptions } from '../llm-client-DyqEH4jH.js';
 import { AgentProfile } from '@tangle-network/agent-interface';