@tangle-network/agent-eval 0.123.2 → 0.123.3

package/CHANGELOG.md

@@ -8,6 +8,7 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-
 
 ### Added
 
+- `callLlmJson()` accepts `jsonPayloadMode: 'exact'` when callers must reject fenced, prose-wrapped, or multi-root responses instead of extracting a JSON value.
 - `llmPolicyEditProposer({ redactCurrentSurfaceForModel })` can remove credentials and unrelated fields from the current surface sent to the model while applying validated edits to the complete original surface.
 - `CostLedger.listPending()` exposes immutable pending paid calls and distinguishes calls that are active, late after cancellation, or interrupted by a prior process so durable workflows can reconcile exact reservations before resuming.
 - `traceAnalystProposer()` accepts an opt-in `resolvePriorFindings` callback that forwards canonical prior findings into the existing analyst registry.
@@ -21,12 +22,35 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-
 
 ### Fixed
 
+- `llmPolicyEditProposer()` asks the model only for semantic edit choices, deterministically binds caller-owned policy fields, and requires one exact whole-response JSON value before admission.
+- `compareOptimizationMethods()` owns three non-empty, pairwise-disjoint scenario sets.
+  Methods receive independent copies of train and selection data, every method finishes before final test scoring starts, and final test data is absent from `OptimizationMethodInput`.
+  Built-in GEPA, SkillOpt, and FAPO methods read the shared baseline, runner, judges, directories, and execution settings from the comparison call instead of duplicating them in each method config.
+  Final test scoring uses one shared spend limit.
+  Reused cost ledgers isolate comparison receipts from unrelated calls, and final test scenarios are copied for each measured surface.
+  Results report optimization, test, and total cost with accounting completeness and reasons for unknown charges.
+  Per-method results include the paired scenario values used to compute lift.
+  Cost orders a tied-lift group only when every method in that group has complete accounting.
+  Bootstrap intervals adjust across all method contrasts, and the default resample count increases when needed to represent the adjusted tails.
+  `optimizationConcurrency` can run independent methods in parallel.
+- `runSkillOpt()` validates every numeric control before scoring, enforces `patchesPerEpoch` for custom proposers, and no longer mutates the caller's `runDir`.
 - Include the complete required response shape in `llmPolicyEditProposer` model instructions, and add `LlmClientOptions.jsonSchemaTransport: 'json-object'` for providers that do not implement native JSON Schema enforcement.
 - `compositeProposer` restores each member's original labels when replaying history, so stateful members do not repeat candidates whose labels were decorated for provenance.
 - `InsightReport.interRater.kappa` now reports quadratic weighted kappa instead of Pearson correlation.
   Read `interRater.pearson` for the previous correlation measure; `icc` and `spearman` are now reported separately.
 - Compute contextual-bandit doubly robust estimates with separate logged-action and target-policy value terms, expose how many rows use DR versus IPS or the deprecated scalar path, and carry both values through belief-state records.
 
+### Breaking
+
+- `compareProposers()` is replaced by `compareOptimizationMethods()`.
+  Rename `proposers` to `methods`, `ProposerEntry` to `OptimizationMethod`, and the built-in `*Entry` factories to `*Method`.
+  Pass `trainScenarios`, `selectionScenarios`, and `testScenarios`; the ambiguous `holdoutScenarios` option is rejected.
+  `OptimizerEntryConfig` is replaced by `BuiltinOptimizationMethodConfig`, which contains method-specific settings only.
+  Move shared method execution settings to `optimizationRunOptions` on the comparison call.
+  Read `optimizationCost`, `testCost`, and `totalCost`; each includes `totalCostUsd`, `accountingComplete`, and `incompleteReasons`.
+- `runSkillOpt({ holdoutScenarios })` fails closed because those rows are adaptively reused.
+  Pass `selectionScenarios`; selection result fields now use `Selection` instead of `Holdout`, and `lift` is now `selectionLift`.
+
 ## [0.122.2] — 2026-07-17 — premeasured optimization continuation
 
 ### Added

package/README.md

@@ -8,7 +8,7 @@ A TypeScript library that measures whether your AI agent got better or worse, us
 [![license: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](./LICENSE)
 
 You give it agent runs: outputs, traces, scores, and production feedback.
-It gives you numbers you can act on: did the new prompt beat the old one, is the difference statistically real, what failed and why, and whether the change should ship.
+It gives you numbers you can act on: how much the new prompt changed outcomes, how uncertain that estimate is, what failed and why, and whether the change meets your release rule.
 
 Use it when you need to:
 
@@ -132,11 +132,53 @@ const judge = llmJudge<string, SupportScenario>(
 Pass `judge` to `defineAgentEval()` in place of the offline judge.
 The model receives the scenario, artifact, scoring prompt, and dimension descriptions.
 
+### Compare Optimization Methods
+
+Use `compareOptimizationMethods()` when you need to compare complete search procedures rather than individual prompts.
+The function gives every method the same baseline, runner, judges, train data, and selection data, then ranks their selected surfaces on separate final test data.
+
+```ts
+import {
+  type BuiltinOptimizationMethodConfig,
+  compareOptimizationMethods,
+  gepaParetoMethod,
+  gepaReflectionMethod,
+  skillOptMethod,
+} from '@tangle-network/agent-eval/campaign'
+
+const methodConfig: BuiltinOptimizationMethodConfig<MyScenario, MyArtifact> = {
+  llm,
+  model,
+  target: 'the complete prompt being improved',
+}
+
+const result = await compareOptimizationMethods<MyScenario, MyArtifact>({
+  methods: [
+    gepaReflectionMethod(methodConfig),
+    gepaParetoMethod(methodConfig),
+    skillOptMethod(methodConfig),
+  ],
+  baselineSurface,
+  trainScenarios,
+  selectionScenarios,
+  testScenarios,
+  dispatchWithSurface,
+  judges,
+  runDir,
+})
+```
+
+Read `result.scores` for lift and intervals.
+Read `result.totalCost` for dollars plus whether every charge was known.
+Ranks follow estimated lift; use the intervals and pairwise results to determine whether the observed difference excludes zero.
+See the [method-comparison guide](./docs/campaign-proposers.md) and [runnable example](./examples/compare-optimization-methods/).
+
 ---
 
-## What's in the box
+## Core APIs
 
-One-line tour of the primitives. All of these are plain functions and interfaces you compose; start from `/contract` and pull in more only when you need it.
+Start from `/contract` for the common path.
+Use `/campaign` when you need direct control over runs, candidate generation, or method comparison.
 
 | Primitive | What it does |
 |---|---|
@@ -144,6 +186,7 @@ One-line tour of the primitives. All of these are plain functions and interfaces
 | **Scoring** (`JudgeConfig`, `llmJudge`, calibration) | Score one output on weighted dimensions with code or a model, then compare model scores against human ratings. |
 | **Release rules** (`heldOutGate`, `paretoSignificanceGate`, `composeGate`, …) | Decide whether a candidate ships, such as requiring an improvement on scenarios that candidate generation never saw. |
 | **Candidate generation** (`gepaProposer`, `evolutionaryProposer`, …) | Generate candidate prompts or configs from prior failures. |
+| **Method comparison** (`compareOptimizationMethods`) | Run complete optimization methods on shared train and selection data, then rank them on separate final test data. |
 | **Run analysis** (`analyzeRuns`, `diffRuns`) | Turn any set of `RunRecord`s into a report: score distributions, baseline-vs-candidate lift with confidence intervals, failure clusters, cost breakdown, recommendations. |
 | **Intake adapters** (`fromFeedbackTable`, `fromOtelSpans`) | Convert data you already have, such as human ratings tables and OpenTelemetry spans, into `RunRecord`s. |
 | **Cost tracking** | Attribute every model call's tokens and dollars to the run, phase, and judge that spent them, including interrupted calls. |
@@ -159,6 +202,7 @@ Our own experiments with these primitives live in [`examples/`](./examples/READM
 | [`examples/selfimprove-quickstart/`](./examples/selfimprove-quickstart/) | The closed improve-and-verify loop, fully offline |
 | [`examples/customer-feedback-loop/`](./examples/customer-feedback-loop/) | Multi-rater human feedback (CSV/Sheets/Obsidian) → per-rater judges → report |
 | [`examples/customer-otel-traces/`](./examples/customer-otel-traces/) | Production OpenTelemetry traces → report, no closed loop required |
+| [`examples/compare-optimization-methods/`](./examples/compare-optimization-methods/) | Compare complete optimization methods with separate train, selection, and test data |
 
 Each is a single `index.ts` you run with `pnpm tsx`.
 
@@ -192,7 +236,7 @@ The root export (`@tangle-network/agent-eval`) remains broad for compatibility;
 - [`docs/concepts.md`](./docs/concepts.md): the mental model for runs, judges, verifiers, traces, and the top-level functions (5-minute read)
 - [`docs/customer-journeys.md`](./docs/customer-journeys.md): three complete adoption paths with code
 - [`docs/insight-report.md`](./docs/insight-report.md): annotated walkthrough of every section of the `analyzeRuns()` report
-- [`docs/campaign-proposers.md`](./docs/campaign-proposers.md): which proposer to use and when
+- [`docs/campaign-proposers.md`](./docs/campaign-proposers.md): candidate generation and fair comparison of complete optimization methods
 - [`docs/adapters-observability.md`](./docs/adapters-observability.md): composing with LangSmith, Langfuse, Phoenix, and OpenLLMetry
 - [`docs/wire-protocol.md`](./docs/wire-protocol.md): the HTTP/RPC contract for other languages
 - [`docs/design.md`](./docs/design.md): how this package relates to the rest of the Tangle agent stack, and the dependency rules that keep it reusable

package/dist/analyst/index.d.ts

@@ -844,9 +844,16 @@ interface CostReceiptInput extends CostUsage {
     costUnknown?: boolean;
     usageUnknown?: boolean;
 }
+/** Per-million token rates for a model or endpoint not covered by package pricing. */
+interface CustomTokenPricing {
+    inputUsdPerMillion: number;
+    outputUsdPerMillion: number;
+}
 type MaximumCharge = {
     externallyEnforcedMaximumUsd: number;
 } | ({
+    customTokenPricing: CustomTokenPricing;
+} & Pick<CostUsage, 'inputTokens' | 'outputTokens'>) | ({
     model: string;
 } & CostUsage);
 interface RunPaidCallInput<T> {
@@ -858,7 +865,7 @@ interface RunPaidCallInput<T> {
     model?: string;
     tags?: Record<string, string>;
     signal?: AbortSignal;
-    /** Provider-enforced dollar maximum, or maximum priced token usage. Required when capped. */
+    /** Provider-enforced dollar maximum, or maximum token usage with known pricing. Required when capped. */
     maximumCharge?: MaximumCharge;
     /** `callId` can be forwarded as the provider's idempotency key. */
     execute(signal: AbortSignal, callId: string): Promise<T>;
@@ -1042,8 +1049,8 @@ interface LlmCallResult {
     content: string;
     usage: LlmUsage;
     /**
-     * Cost in USD. Pulled from proxy's `_response_cost` field when present;
-     * `null` when neither the proxy nor the caller can derive it.
+     * Cost in USD. Uses the provider's reported cost when present, otherwise
+     * caller-supplied token pricing. `null` when neither is available.
      */
     costUsd: number | null;
     /** Model name actually used (echoed from response). */
@@ -1101,6 +1108,8 @@ interface LlmClientOptions {
     deadlineMs?: number;
     /** Total provider attempts. Legacy option name; default 3 (1 initial + 2 retries). */
     maxRetries?: number;
+    /** Token rates used when the provider omits cost or package pricing does not cover the model. */
+    customTokenPricing?: CustomTokenPricing;
     /**
      * Transport for requests that declare `jsonSchema`. `native` sends
      * `response_format: json_schema`; `json-object` sends the broadly supported
@@ -1108,6 +1117,12 @@ interface LlmClientOptions {
      * instructions. Default: `native`.
      */
     jsonSchemaTransport?: 'native' | 'json-object';
+    /**
+     * JSON payload parsing policy. `extract` accepts fenced or prose-prefixed JSON.
+     * `exact` requires the complete response content to be one JSON value.
+     * Default: `extract`.
+     */
+    jsonPayloadMode?: 'extract' | 'exact';
     /** Fetch implementation — defaults to global `fetch`. Override for custom transport (e.g. tests). */
     fetch?: typeof fetch;
     /**

package/dist/analyst/index.js

@@ -9,13 +9,13 @@ import {
   diffFindings,
   emitSkillUsageFindings,
   runSemanticConceptJudge
-} from "../chunk-WMJR67FX.js";
+} from "../chunk-QNVBALEX.js";
 import {
   behavioralAnalyst,
   buildDefaultAnalystRegistry,
   createChatClient,
   deriveEfficiencyFindings
-} from "../chunk-5YMKIFYP.js";
+} from "../chunk-WP3GEXKI.js";
 import "../chunk-HHWE3POT.js";
 import {
   ANALYST_SEVERITIES,
@@ -71,11 +71,11 @@ import {
   validatePolicyEdit,
   validatePolicyEditCandidateRecord,
   validateUsageSettlementTimeout
-} from "../chunk-WXQTVEKM.js";
-import "../chunk-J7S4YM27.js";
+} from "../chunk-MHPEGJHC.js";
+import "../chunk-MK3Q6GCL.js";
 import {
   CostLedger
-} from "../chunk-BGVTIE2C.js";
+} from "../chunk-GRCDRKII.js";
 import "../chunk-VI2UW6B6.js";
 import "../chunk-IR3KBHOY.js";
 import "../chunk-PC4UYEBM.js";

package/dist/benchmarks/index.d.ts

@@ -66,9 +66,16 @@ interface CostReceiptInput extends CostUsage {
     costUnknown?: boolean;
     usageUnknown?: boolean;
 }
+/** Per-million token rates for a model or endpoint not covered by package pricing. */
+interface CustomTokenPricing {
+    inputUsdPerMillion: number;
+    outputUsdPerMillion: number;
+}
 type MaximumCharge = {
     externallyEnforcedMaximumUsd: number;
 } | ({
+    customTokenPricing: CustomTokenPricing;
+} & Pick<CostUsage, 'inputTokens' | 'outputTokens'>) | ({
     model: string;
 } & CostUsage);
 interface RunPaidCallInput<T> {
@@ -80,7 +87,7 @@ interface RunPaidCallInput<T> {
     model?: string;
     tags?: Record<string, string>;
     signal?: AbortSignal;
-    /** Provider-enforced dollar maximum, or maximum priced token usage. Required when capped. */
+    /** Provider-enforced dollar maximum, or maximum token usage with known pricing. Required when capped. */
     maximumCharge?: MaximumCharge;
     /** `callId` can be forwarded as the provider's idempotency key. */
     execute(signal: AbortSignal, callId: string): Promise<T>;
@@ -205,8 +212,8 @@ interface LlmCallResult {
     content: string;
     usage: LlmUsage;
     /**
-     * Cost in USD. Pulled from proxy's `_response_cost` field when present;
-     * `null` when neither the proxy nor the caller can derive it.
+     * Cost in USD. Uses the provider's reported cost when present, otherwise
+     * caller-supplied token pricing. `null` when neither is available.
      */
     costUsd: number | null;
     /** Model name actually used (echoed from response). */

package/dist/benchmarks/index.js

@@ -16,15 +16,15 @@ import {
   routing_exports,
   runBenchmarkAdapter,
   summarizeBenchmarkCampaign
-} from "../chunk-FHFTYX2Q.js";
-import "../chunk-RQP5UTK5.js";
-import "../chunk-HQY7LBV2.js";
-import "../chunk-3FCG7FBV.js";
-import "../chunk-WXQTVEKM.js";
+} from "../chunk-JKDNAOF5.js";
+import "../chunk-22VO7T2I.js";
+import "../chunk-SUN7QLPB.js";
+import "../chunk-D5JZ7UDZ.js";
+import "../chunk-MHPEGJHC.js";
 import "../chunk-ARU2PZFM.js";
-import "../chunk-J7S4YM27.js";
+import "../chunk-MK3Q6GCL.js";
 import "../chunk-PJQFMIOX.js";
-import "../chunk-BGVTIE2C.js";
+import "../chunk-GRCDRKII.js";
 import "../chunk-VI2UW6B6.js";
 import "../chunk-NUKSVU3W.js";
 import "../chunk-GGE4NNQT.js";