@tangle-network/agent-eval 0.123.1 → 0.123.3

package/CHANGELOG.md

@@ -8,6 +8,8 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-
 
 ### Added
 
+- `callLlmJson()` accepts `jsonPayloadMode: 'exact'` when callers must reject fenced, prose-wrapped, or multi-root responses instead of extracting a JSON value.
+- `llmPolicyEditProposer({ redactCurrentSurfaceForModel })` can remove credentials and unrelated fields from the current surface sent to the model while applying validated edits to the complete original surface.
 - `CostLedger.listPending()` exposes immutable pending paid calls and distinguishes calls that are active, late after cancellation, or interrupted by a prior process so durable workflows can reconcile exact reservations before resuming.
 - `traceAnalystProposer()` accepts an opt-in `resolvePriorFindings` callback that forwards canonical prior findings into the existing analyst registry.
 
@@ -20,8 +22,35 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-
 
 ### Fixed
 
+- `llmPolicyEditProposer()` asks the model only for semantic edit choices, deterministically binds caller-owned policy fields, and requires one exact whole-response JSON value before admission.
+- `compareOptimizationMethods()` owns three non-empty, pairwise-disjoint scenario sets.
+  Methods receive independent copies of train and selection data, every method finishes before final test scoring starts, and final test data is absent from `OptimizationMethodInput`.
+  Built-in GEPA, SkillOpt, and FAPO methods read the shared baseline, runner, judges, directories, and execution settings from the comparison call instead of duplicating them in each method config.
+  Final test scoring uses one shared spend limit.
+  Reused cost ledgers isolate comparison receipts from unrelated calls, and final test scenarios are copied for each measured surface.
+  Results report optimization, test, and total cost with accounting completeness and reasons for unknown charges.
+  Per-method results include the paired scenario values used to compute lift.
+  Cost orders a tied-lift group only when every method in that group has complete accounting.
+  Bootstrap intervals adjust across all method contrasts, and the default resample count increases when needed to represent the adjusted tails.
+  `optimizationConcurrency` can run independent methods in parallel.
+- `runSkillOpt()` validates every numeric control before scoring, enforces `patchesPerEpoch` for custom proposers, and no longer mutates the caller's `runDir`.
+- Include the complete required response shape in `llmPolicyEditProposer` model instructions, and add `LlmClientOptions.jsonSchemaTransport: 'json-object'` for providers that do not implement native JSON Schema enforcement.
+- `compositeProposer` restores each member's original labels when replaying history, so stateful members do not repeat candidates whose labels were decorated for provenance.
+- `InsightReport.interRater.kappa` now reports quadratic weighted kappa instead of Pearson correlation.
+  Read `interRater.pearson` for the previous correlation measure; `icc` and `spearman` are now reported separately.
 - Compute contextual-bandit doubly robust estimates with separate logged-action and target-policy value terms, expose how many rows use DR versus IPS or the deprecated scalar path, and carry both values through belief-state records.
 
+### Breaking
+
+- `compareProposers()` is replaced by `compareOptimizationMethods()`.
+  Rename `proposers` to `methods`, `ProposerEntry` to `OptimizationMethod`, and the built-in `*Entry` factories to `*Method`.
+  Pass `trainScenarios`, `selectionScenarios`, and `testScenarios`; the ambiguous `holdoutScenarios` option is rejected.
+  `OptimizerEntryConfig` is replaced by `BuiltinOptimizationMethodConfig`, which contains method-specific settings only.
+  Move shared method execution settings to `optimizationRunOptions` on the comparison call.
+  Read `optimizationCost`, `testCost`, and `totalCost`; each includes `totalCostUsd`, `accountingComplete`, and `incompleteReasons`.
+- `runSkillOpt({ holdoutScenarios })` fails closed because those rows are adaptively reused.
+  Pass `selectionScenarios`; selection result fields now use `Selection` instead of `Holdout`, and `lift` is now `selectionLift`.
+
 ## [0.122.2] — 2026-07-17 — premeasured optimization continuation
 
 ### Added

package/README.md

@@ -1,202 +1,255 @@
 # `@tangle-network/agent-eval`
 
-Evaluate and improve AI agents from the runs they already produce.
-
-`agent-eval` turns agent outputs, traces, judge scores, and production feedback into a decision packet: did this change help, what failed, what should ship, and what needs more data?
+A TypeScript library that measures whether your AI agent got better or worse, using the runs it already produces.
 
 [![npm](https://img.shields.io/npm/v/@tangle-network/agent-eval.svg)](https://www.npmjs.com/package/@tangle-network/agent-eval)
 [![pypi](https://img.shields.io/pypi/v/agent-eval-rpc.svg)](https://pypi.org/project/agent-eval-rpc/)
 [![tests](https://github.com/tangle-network/agent-eval/actions/workflows/ci.yml/badge.svg)](https://github.com/tangle-network/agent-eval/actions/workflows/ci.yml)
 [![license: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](./LICENSE)
 
+You give it agent runs: outputs, traces, scores, and production feedback.
+It gives you numbers you can act on: how much the new prompt changed outcomes, how uncertain that estimate is, what failed and why, and whether the change meets your release rule.
+
 Use it when you need to:
 
-- compare a candidate agent/prompt/model against a baseline,
-- turn production traces or human feedback into eval results,
-- run a gated self-improvement loop,
-- explain failures by cluster, cost, judge disagreement, and release risk.
+- compare a candidate prompt/model/config against a baseline, with confidence intervals instead of vibes,
+- turn production traces or human feedback you already collect into eval results,
+- run an automated improve-and-verify loop over a prompt, held to a promotion rule you choose,
+- explain failures by cluster, cost, and judge disagreement.
 
-It is a library, not a SaaS requirement. TypeScript is first-class; Python can call the same wire protocol through `agent-eval-rpc`.
+The deterministic evaluator runs in your process and makes no network calls.
+Features that use a model send their inputs to the model client you pass.
+Trace exporters and hosted ingestion are also opt-in.
+Python can drive the same engine over HTTP via [`agent-eval-rpc`](./clients/python/README.md).
 
 ---
 
 ## Install
 
 ```sh
-pnpm add @tangle-network/agent-eval
+pnpm add @tangle-network/agent-eval   # or npm / yarn
 ```
 
-Python clients can use the RPC package:
-
 ```sh
-pip install agent-eval-rpc
+pip install agent-eval-rpc            # optional Python client
 ```
 
 ---
 
-## Quick start
-
-### 1. Analyze runs you already have
+## Quickstart
 
-Start here if you already have production logs, benchmark rows, human ratings, or agent run records.
+Copy this into `quickstart.ts` and run `npx tsx quickstart.ts`.
+It is fully offline; the "agent" and "judge" are plain functions you replace with your own.
 
 ```ts
-import { analyzeRuns } from '@tangle-network/agent-eval/contract'
+import { defineAgentEval } from '@tangle-network/agent-eval/contract'
 
-const report = await analyzeRuns({
-  runs, // RunRecord[]
-  baselineRuns,
+interface SupportScenario {
+  id: string
+  kind: 'support'
+}
+
+async function main() {
+  const scenarios: SupportScenario[] = [
+    { id: 'refund', kind: 'support' },
+    { id: 'shipping', kind: 'support' },
+    { id: 'cancel', kind: 'support' },
+  ]
+
+  const evalKit = defineAgentEval<SupportScenario, string>({
+    scenarios,
+    // Your agent takes the prompt under test and one scenario, then returns its output.
+    agent: async (prompt, scenario) =>
+      String(prompt).includes('ticket') ? `Re ${scenario.id}: on it.` : 'On it.',
+    // Your judge scores one output from 0 to 1. Swap in an LLM judge for real work.
+    judge: {
+      name: 'cites-ticket',
+      dimensions: [{ key: 'ticket_id', description: 'The answer includes the ticket id' }],
+      score: ({ artifact, scenario }) => {
+        const ticketId = artifact.includes(scenario.id) ? 1 : 0
+        return { dimensions: { ticket_id: ticketId }, composite: ticketId, notes: '' }
+      },
+    },
+    baselineSurface: 'Answer the customer politely.',
+    expectUsage: 'off',
+  })
+
+  console.log('baseline: ', (await evalKit.evaluate()).aggregates.byJudge)
+  const candidate = await evalKit.evaluate({ surface: 'Answer politely, cite the ticket id.' })
+  console.log('candidate:', candidate.aggregates.byJudge)
+}
+
+main().catch((error: unknown) => {
+  console.error(error)
+  process.exitCode = 1
 })
+```
+
+Output:
 
-console.log(report.recommendations)
-console.log(report.lift)
-console.log(report.failureClusters)
 ```
+baseline:  { 'cites-ticket': { mean: 0, stdev: 0, ci95: [ 0, 0 ], n: 3 } }
+candidate: { 'cites-ticket': { mean: 1, stdev: 0, ci95: [ 1, 1 ], n: 3 } }
+```
+
+Each `evaluate()` call runs every scenario through the agent, scores each output with the judge, and returns per-judge score distributions.
+The "surface" is the thing you are changing: here a system-prompt string, and in general any prompt or config value.
+From the same definition, `evalKit.improve()` proposes candidate prompts, measures each one, and checks the winner against a held-back scenario set before recommending it.
+The default candidate generator calls a model, so pass `llm: { baseUrl, apiKey, model }` to `.improve()` or provide your own `proposer`.
 
-The output includes score distributions, lift confidence intervals, failure modes, cost-quality tradeoffs, judge agreement, contamination checks, and release recommendations when the input supports them.
+Already have run data and no runnable agent? Skip the loop and call [`analyzeRuns()`](./docs/concepts.md#the-top-level-functions) on your existing records instead.
 
-### 2. Define one eval, then score or improve
+### Use A Model Judge
 
-Use this when you have scenarios, a runnable agent, and judges.
+`llmJudge()` converts one model call into the same `JudgeConfig` used above:
 
 ```ts
-import { defineAgentEval } from '@tangle-network/agent-eval/contract'
+import { createChatClient, llmJudge } from '@tangle-network/agent-eval/contract'
 
-const evalKit = defineAgentEval({
-  scenarios,
-  agent: async (surface, scenario, ctx) =>
-    myAgent.run({ scenario, systemPrompt: String(surface), signal: ctx.signal }),
-  judge: myJudge,
-  baselineSurface: currentPrompt,
-})
+const apiKey = process.env.OPENAI_API_KEY
+if (!apiKey) throw new Error('OPENAI_API_KEY is required')
 
-const baseline = await evalKit.evaluate()
-const result = await evalKit.improve({ budget: { generations: 2 } })
+const chat = createChatClient({
+  transport: 'direct-provider',
+  baseUrl: process.env.OPENAI_BASE_URL ?? 'https://api.openai.com/v1',
+  apiKey,
+  defaultModel: 'gpt-4.1-mini',
+})
 
-console.log(baseline.aggregates.byJudge)
-console.log(result.gateDecision)
-console.log(result.winner.surface)
-console.log(result.insight.recommendations)
+const judge = llmJudge<string, SupportScenario>(
+  'support-quality',
+  'Score whether the response resolves the request using only supported facts.',
+  {
+    chat,
+    dimensions: [
+      { key: 'correct', description: 'The answer is factually correct' },
+      { key: 'complete', description: 'The answer addresses the whole request' },
+    ],
+  },
+)
 ```
 
-`defineAgentEval()` is a small wrapper over `runEval()` and `selfImprove()`. It lets you define scenarios, agent, judge, and baseline once, then either score one surface with `.evaluate()` or run the gated loop with `.improve()`.
+Pass `judge` to `defineAgentEval()` in place of the offline judge.
+The model receives the scenario, artifact, scoring prompt, and dimension descriptions.
 
-### 3. Adapt existing data
+### Compare Optimization Methods
 
-```ts
-import { analyzeRuns, fromFeedbackTable, fromOtelSpans } from '@tangle-network/agent-eval/contract'
+Use `compareOptimizationMethods()` when you need to compare complete search procedures rather than individual prompts.
+The function gives every method the same baseline, runner, judges, train data, and selection data, then ranks their selected surfaces on separate final test data.
 
-const { runs, raterScores } = fromFeedbackTable({
-  ratings: parseYourFeedbackTable(),
+```ts
+import {
+  type BuiltinOptimizationMethodConfig,
+  compareOptimizationMethods,
+  gepaParetoMethod,
+  gepaReflectionMethod,
+  skillOptMethod,
+} from '@tangle-network/agent-eval/campaign'
+
+const methodConfig: BuiltinOptimizationMethodConfig<MyScenario, MyArtifact> = {
+  llm,
+  model,
+  target: 'the complete prompt being improved',
+}
+
+const result = await compareOptimizationMethods<MyScenario, MyArtifact>({
+  methods: [
+    gepaReflectionMethod(methodConfig),
+    gepaParetoMethod(methodConfig),
+    skillOptMethod(methodConfig),
+  ],
+  baselineSurface,
+  trainScenarios,
+  selectionScenarios,
+  testScenarios,
+  dispatchWithSurface,
+  judges,
+  runDir,
 })
-
-const traceRuns = fromOtelSpans({ spans: yourOtelSpans })
-
-await analyzeRuns({ runs: [...runs, ...traceRuns], raterScores })
 ```
 
----
+Read `result.scores` for lift and intervals.
+Read `result.totalCost` for dollars plus whether every charge was known.
+Ranks follow estimated lift; use the intervals and pairwise results to determine whether the observed difference excludes zero.
+See the [method-comparison guide](./docs/campaign-proposers.md) and [runnable example](./examples/compare-optimization-methods/).
 
-## Core concepts
+---
 
-- **RunRecord**: the durable row for one agent run: model, prompt/config hashes, split, cost, tokens, outcome.
-- **Scenario**: one task or case the agent attempts.
-- **Judge**: a scoring function, rule-based or model-based.
-- **Surface**: the thing being changed, usually a prompt string or config object.
-- **InsightReport**: the decision packet returned by `analyzeRuns()` and embedded in `selfImprove()`.
-- **Gate**: the policy that decides `ship`, `hold`, or `need_more_work`.
+## Core APIs
 
-## Examples
+Start from `/contract` for the common path.
+Use `/campaign` when you need direct control over runs, candidate generation, or method comparison.
 
-| Journey | Example | Who it's for |
-|---|---|---|
-| **Closed loop** — improve a prompt under statistical confidence | [`examples/selfimprove-quickstart/`](./examples/selfimprove-quickstart/) | Teams with scenarios + judges + agent in hand |
-| **Multi-rater feedback corpus** — turn Obsidian/Sheets/CSV ratings into actionable insights | [`examples/customer-feedback-loop/`](./examples/customer-feedback-loop/) | Teams reviewing AI outputs by hand who want to compress that taste into per-member LLM judges + close the loop |
-| **Production OTel traces** — analyze logs you already have, no closed loop required | [`examples/customer-otel-traces/`](./examples/customer-otel-traces/) | Teams running agents in prod with observability, no eval discipline yet |
+| Primitive | What it does |
+|---|---|
+| **Evaluation** (`runEval`, `runCampaign`) | Run agent × scenarios × repetitions, score every run, and record the result. |
+| **Scoring** (`JudgeConfig`, `llmJudge`, calibration) | Score one output on weighted dimensions with code or a model, then compare model scores against human ratings. |
+| **Release rules** (`heldOutGate`, `paretoSignificanceGate`, `composeGate`, …) | Decide whether a candidate ships, such as requiring an improvement on scenarios that candidate generation never saw. |
+| **Candidate generation** (`gepaProposer`, `evolutionaryProposer`, …) | Generate candidate prompts or configs from prior failures. |
+| **Method comparison** (`compareOptimizationMethods`) | Run complete optimization methods on shared train and selection data, then rank them on separate final test data. |
+| **Run analysis** (`analyzeRuns`, `diffRuns`) | Turn any set of `RunRecord`s into a report: score distributions, baseline-vs-candidate lift with confidence intervals, failure clusters, cost breakdown, recommendations. |
+| **Intake adapters** (`fromFeedbackTable`, `fromOtelSpans`) | Convert data you already have, such as human ratings tables and OpenTelemetry spans, into `RunRecord`s. |
+| **Cost tracking** | Attribute every model call's tokens and dollars to the run, phase, and judge that spent them, including interrupted calls. |
+| **Human feedback storage** | Persist runs with approved, rejected, or edited labels so review activity becomes training and eval data. |
+| **Statistics** (`pairedBootstrap`, `benjaminiHochberg`, sequential tests) | The release-decision math, usable standalone. |
+| **Trace tools** (`/traces`, `/analyst`) | Store and replay structured run traces; cluster failures with an LLM analyst panel. |
+| **HTTP and RPC** (`/wire`) | Expose judging and ingestion to non-TypeScript stacks, including the Python client. |
+
+Our own experiments with these primitives live in [`examples/`](./examples/README.md); they are demonstrations, not part of the API.
+
+| Runnable example | Shows |
+|---|---|
+| [`examples/selfimprove-quickstart/`](./examples/selfimprove-quickstart/) | The closed improve-and-verify loop, fully offline |
+| [`examples/customer-feedback-loop/`](./examples/customer-feedback-loop/) | Multi-rater human feedback (CSV/Sheets/Obsidian) → per-rater judges → report |
+| [`examples/customer-otel-traces/`](./examples/customer-otel-traces/) | Production OpenTelemetry traces → report, no closed loop required |
+| [`examples/compare-optimization-methods/`](./examples/compare-optimization-methods/) | Compare complete optimization methods with separate train, selection, and test data |
 
-Each example: `README.md` + a single `index.ts` runnable via `pnpm tsx`. Prints the resulting `InsightReport` to stdout.
+Each is a single `index.ts` you run with `pnpm tsx`.
 
 ---
 
-## Subpath entry points
+## Entry points
+
+Import from `@tangle-network/agent-eval/<subpath>`. Every row below is verified importable from the published package.
 
 | Subpath | What it gives you |
 |---|---|
-| `…/contract` | **The headline, frozen surface — new code starts here.** `defineAgentEval`, `selfImprove`, `analyzeRuns`, `runEval`, `runCampaign`, `runImprovementLoop`, `diffRuns`; intake adapters (`fromFeedbackTable`, `fromOtelSpans`); proposers (`gepaProposer`, `evolutionaryProposer`); gates (`defaultProductionGate`, `heldOutGate`, `paretoSignificanceGate`, `neutralizationGate`, `composeGate`); the deployment-outcome store; storage; and the five core types `Scenario` / `Dispatch` / `JudgeConfig` / `SurfaceProposer` / `Gate`. |
-| `…/hosted` | `createHostedClient` / `hostedClientFromEnv` + the wire types to ship eval-run events + trace spans to a hosted orchestrator (ours or your own implementation of the spec) |
-| `…/adapters/otel` | `createOtelBridge` — forwards OpenTelemetry-shape spans into the hosted-tier ingest, no `@opentelemetry/*` dependency |
-| `…/adapters/langchain` | Wrap any LangChain `Runnable` as a `Dispatch` (or `JudgeConfig`), no `@langchain/core` peer dep |
-| `…/adapters/http` | `httpDispatch` + `runDispatchServer` — run a campaign's worker on another machine (multi-region, remote worker execution) |
-| `…/campaign` | **The measurement + improvement engine**: `runProfileMatrix`, `compareProposers`, `analyzeCrossSurfaceInteractions`, every surface proposer (`gepaProposer`, `fapoProposer`, `parameterSweepProposer`, `haloProposer`, `skillOptProposer`, `aceProposer`, `memoryCurationProposer`, …), the gates, storage backends, and loop provenance. `/contract` re-exports the app-facing subset. |
-| `…/rl` | Bridge from eval artifacts to training signal: verifiable rewards, preferences, OPE, tournaments, contamination, compute curves, trainer-format exporters, process rewards, plus the durable corpus + `buildRlDataset` / datasheet bundle |
-| `…/reporting` | Release-decision statistics: `pairedBootstrap`, `benjaminiHochberg`, anytime-valid sequential e-values, `evaluateReleaseConfidence`, and the report renderers |
-| `…/analyst` | The trace-analyst surface: `AnalystRegistry` + `buildDefaultAnalystRegistry` (run the failure-clustering panel), `FindingsStore`, and the LLM chat transports |
-| `…/traces` | Trace stores + emitters, OTLP-JSONL deterministic replay, `analyzeTraces`, and the `traceAnalystOnRunComplete` hook |
-| `…/control` | Agent control loop: `runAgentControlLoop` (observe → validate → decide → act), action policy, propose/review |
-| `…/matrix` | `runAgentMatrix` — an N-axis cartesian over caller-supplied substrate values, per-axis pass/score/cost/duration |
-| `…/multishot` | N-shot persona × shot matrix runner (`runMultishot` / `runMultishotMatrix`) |
-| `…/wire` | The cross-language HTTP/RPC server + Zod schemas (the source-of-truth protocol the Python client speaks) + the built-in rubric registry |
-| `…/benchmarks` | `BenchmarkAdapter` contract, `runBenchmarkAdapter`, `calibrateBenchmarkMetric`, standard retrieval parsers + ranked retrieval metrics, `deterministicSplit`, and the bundled `routing` reference benchmark |
-
-**Specialized surfaces** (subpath-only): `…/prm` (process-reward grading + best-of-N), `…/meta-eval` (judge calibration + the deployment-outcome store), `…/belief-state` (decision-point extraction + selective-policy reports), `…/pipelines` (trace-diagnostic views: budget breach, failure cluster, stuck loop, …), `…/governance` (EU AI Act / NIST AI RMF / SOC2 reports), `…/knowledge` (knowledge-readiness gating before a run), `…/builder-eval` (code-generator three-layer eval), `…/storyboard` (trace → watchable replay), `…/authenticity` (anti-Goodhart "real or convincing BS" scorer over produced files), `…/workflow` (workflow-trace eval + partner export), `…/telemetry` (Workers-safe telemetry client), `…/testing` (test-only reset helpers).
-
-The root export remains broad for compatibility; new code should prefer the focused subpaths above — `/contract` first.
+| `/contract` | **Start here.** Stable APIs for defining an eval, running it, improving a prompt, judging outputs, analyzing existing runs, and storing results. |
+| `/campaign` | Lower-level control over candidate generation, release rules, storage, and comparisons. |
+| `/reporting` | Statistical comparisons and report renderers. |
+| `/analyst` | Model-based failure clustering and stored findings. |
+| `/traces` | Trace stores, emitters, deterministic replay, trace analysis. |
+| `/rl` | Export eval artifacts as training signal: rewards, preferences, trainer-format datasets. |
+| `/benchmarks` | Benchmark adapter contract + retrieval metrics + a bundled reference benchmark. |
+| `/wire` | The HTTP/RPC server and Zod schemas (what the Python client speaks). |
+| `/hosted` | Client for shipping eval-run events to a remote orchestrator (see below). |
+| `/control` | A generic observe → validate → decide → act agent loop with eval-backed stopping rules. |
+| `/matrix`, `/multishot` | N-axis configuration sweeps; multi-turn persona × turn-count runners. |
+| `/meta-eval`, `/belief-state`, `/builder-eval`, `/pipelines`, `/storyboard`, `/authenticity`, `/fuzz`, `/trace-attributes` | Specialized surfaces: judge calibration, decision-point extraction, code-generator grading, trace diagnostics, run replay rendering, anti-gaming output checks, input fuzzing, trace attribute vocabulary. |
+
+The root export (`@tangle-network/agent-eval`) remains broad for compatibility; prefer the subpaths for new code.
 
 ---
 
-## Composition with the stack
-
-agent-eval is the bottom of the layering: consumers depend on it, it depends on none of them.
-
-```
-agent-runtime    Runs agents (chat turns, one-shot tasks, multi-attempt loops), captures every
-                 run as a trace, and calls optimizePrompt / runImprovementLoop. Produces the
-                 RunRecords + traces agent-eval scores. Depends on agent-eval.
+## Documentation
 
-agent-eval       selfImprove, analyzeRuns, runCampaign + surface proposers (GEPA proposer, …), the gates
-   (this repo)   (heldOutGate, defaultProductionGate, paretoSignificanceGate), the InsightReport
-                 decision packet, the RL bridge, the wire protocol. Depends on neither consumer.
-
-agent-knowledge  proposeKnowledgeWrites / applyKnowledgeWriteBlocks. agent-eval's analyst findings
-                 feed it; the knowledge gate consumes them. Depends on agent-eval.
-
-sandbox          Sandbox.create, streamPrompt. One execution surface the runtime's
-                 loops run on; agent-eval scores what comes back.
-```
-
-The rule: **agent-eval has zero upward dependencies on a consumer.** A concept that makes sense *without* a running agent loop — a verdict, a run record, a scenario, a judge score — is substrate and lives here. Runtime execution details (a validation context with an abort signal, a concrete sandbox session) live in agent-runtime or sandbox. Agent profile shape is the shared `@tangle-network/agent-interface` contract.
+- [`docs/concepts.md`](./docs/concepts.md): the mental model for runs, judges, verifiers, traces, and the top-level functions (5-minute read)
+- [`docs/customer-journeys.md`](./docs/customer-journeys.md): three complete adoption paths with code
+- [`docs/insight-report.md`](./docs/insight-report.md): annotated walkthrough of every section of the `analyzeRuns()` report
+- [`docs/campaign-proposers.md`](./docs/campaign-proposers.md): candidate generation and fair comparison of complete optimization methods
+- [`docs/adapters-observability.md`](./docs/adapters-observability.md): composing with LangSmith, Langfuse, Phoenix, and OpenLLMetry
+- [`docs/wire-protocol.md`](./docs/wire-protocol.md): the HTTP/RPC contract for other languages
+- [`docs/design.md`](./docs/design.md): how this package relates to the rest of the Tangle agent stack, and the dependency rules that keep it reusable
+- [`CHANGELOG.md`](./CHANGELOG.md): every release, with additive and breaking changes identified
 
 ---
 
-## Concepts + design
+## Optional hosted tier
 
-- [`docs/concepts.md`](./docs/concepts.md) — the top-level entry points, the layering rule, and the wire-protocol contract (the five core contract types are documented in the `/contract` barrel itself)
-- [`docs/campaign-proposers.md`](./docs/campaign-proposers.md) — ELI5 proposer inputs/outputs, when to use each proposer, and the FAPO escalation policy
-- [`docs/insight-report.md`](./docs/insight-report.md) — annotated walkthrough of every section of the decision packet
-- [`docs/customer-journeys.md`](./docs/customer-journeys.md) — three end-to-end journeys with code + expected output
-- [`docs/adapters-observability.md`](./docs/adapters-observability.md) — composing agent-eval with LangSmith, Langfuse, Phoenix, OpenLLMetry, TraceAI
-- [`docs/wire-protocol.md`](./docs/wire-protocol.md) — the HTTP/RPC contract Python (and any future language) speaks
-- [`docs/hosted-ingest-spec.md`](./docs/hosted-ingest-spec.md) — the hosted-tier wire format, frozen at `2026-05-26.v1`
-- [`docs/design/loop-taxonomy.md`](./docs/design/loop-taxonomy.md) — plain-language vocabulary for execution drivers, workers, measurements, and proposers
-
-The `.claude/skills/agent-eval/SKILL.md` skill ships embedded directives so LLM agents writing integration code don't reintroduce historical bug classes.
-
----
-
-## Hosted tier
-
-Wire your loop to a hosted orchestrator (ours, or your own implementation of the spec) with one config:
+The library is complete without it.
+If you want a dashboard over many loops, point any run at our remote orchestrator or your own implementation of the [open ingest spec](./docs/hosted-ingest-spec.md):
 
 ```ts
-import { defineAgentEval } from '@tangle-network/agent-eval/contract'
-
-const evalKit = defineAgentEval({
-  scenarios,
-  agent,
-  judge,
-  baselineSurface,
-})
-
 await evalKit.improve({
   hostedTenant: {
     endpoint: 'https://intelligence.tangle.tools',
@@ -206,42 +259,23 @@ await evalKit.improve({
 })
 ```
 
-The substrate runs the loop in your process. Only the eval-run events + (optional) trace spans go to the orchestrator. Your scenarios, your judges, your raw data — never sent. Spec at [`docs/hosted-ingest-spec.md`](./docs/hosted-ingest-spec.md); reference receiver at [`examples/hosted-ingest-server/`](./examples/hosted-ingest-server/).
+The loop still runs in your process.
+Hosted ingest sends run identifiers and paths, scenario IDs, candidate surfaces, scores, errors, costs, summaries, and trace attributes.
+Review the [wire format](./docs/hosted-ingest-spec.md) before enabling it for sensitive inputs.
+A reference receiver you can self-host is at [`examples/hosted-ingest-server/`](./examples/hosted-ingest-server/).
 
 ---
 
 ## Development
 
-Run an example:
-
-```sh
-pnpm tsx examples/selfimprove-quickstart/index.ts
-pnpm tsx examples/customer-feedback-loop/index.ts
-pnpm tsx examples/customer-otel-traces/index.ts
-```
-
-Run the test suite:
-
 ```sh
 pnpm install
 pnpm build
-pnpm test
+pnpm test        # vitest, ~3300 tests
+pnpm typecheck
 ```
 
----
-
-## Public API
-
-The `/contract` surface is the **stability contract**: its barrel freezes the API — a `0.x` minor only *adds*; nothing there changes shape or disappears. Start there for app code.
-
-| Surface | Meaning |
-|---|---|
-| `/contract` | Frozen app-facing API. Prefer this first. |
-| Named subpaths | Public capability areas such as `/campaign`, `/rl`, `/prm`, `/meta-eval`, `/belief-state`, `/wire`, `/reporting`, `/traces`, and `/analyst`. |
-| `/testing` | Test-only helpers. Do not import from production code. |
-| Unexported source paths | Not public API. Open an issue if you need one promoted. |
-
-[`CHANGELOG.md`](./CHANGELOG.md) tracks every release with what's new / additive / breaking.
+Run any example: `pnpm tsx examples/selfimprove-quickstart/index.ts`
 
 ---
 

package/dist/analyst/index.d.ts

@@ -224,9 +224,10 @@ declare class ValidationError extends AgentEvalError {
 }
 
 type AgentProfileCellSchemaVersion = 'agent-profile-cell/v1';
-type AgentProfileJson = string | number | boolean | null | AgentProfileJson[] | {
+type AgentProfileJsonObject = {
     [key: string]: AgentProfileJson;
 };
+type AgentProfileJson = string | number | boolean | null | AgentProfileJson[] | AgentProfileJsonObject;
 type AgentProfileDimensionValue = string | number | boolean | null;
 interface AgentProfileSource {
     /** Runtime/profile contract being fingerprinted, e.g. `agent-interface-profile`. */
@@ -843,9 +844,16 @@ interface CostReceiptInput extends CostUsage {
     costUnknown?: boolean;
     usageUnknown?: boolean;
 }
+/** Per-million token rates for a model or endpoint not covered by package pricing. */
+interface CustomTokenPricing {
+    inputUsdPerMillion: number;
+    outputUsdPerMillion: number;
+}
 type MaximumCharge = {
     externallyEnforcedMaximumUsd: number;
 } | ({
+    customTokenPricing: CustomTokenPricing;
+} & Pick<CostUsage, 'inputTokens' | 'outputTokens'>) | ({
     model: string;
 } & CostUsage);
 interface RunPaidCallInput<T> {
@@ -857,7 +865,7 @@ interface RunPaidCallInput<T> {
     model?: string;
     tags?: Record<string, string>;
     signal?: AbortSignal;
-    /** Provider-enforced dollar maximum, or maximum priced token usage. Required when capped. */
+    /** Provider-enforced dollar maximum, or maximum token usage with known pricing. Required when capped. */
     maximumCharge?: MaximumCharge;
     /** `callId` can be forwarded as the provider's idempotency key. */
     execute(signal: AbortSignal, callId: string): Promise<T>;
@@ -1041,8 +1049,8 @@ interface LlmCallResult {
     content: string;
     usage: LlmUsage;
     /**
-     * Cost in USD. Pulled from proxy's `_response_cost` field when present;
-     * `null` when neither the proxy nor the caller can derive it.
+     * Cost in USD. Uses the provider's reported cost when present, otherwise
+     * caller-supplied token pricing. `null` when neither is available.
      */
     costUsd: number | null;
     /** Model name actually used (echoed from response). */
@@ -1100,6 +1108,21 @@ interface LlmClientOptions {
     deadlineMs?: number;
     /** Total provider attempts. Legacy option name; default 3 (1 initial + 2 retries). */
     maxRetries?: number;
+    /** Token rates used when the provider omits cost or package pricing does not cover the model. */
+    customTokenPricing?: CustomTokenPricing;
+    /**
+     * Transport for requests that declare `jsonSchema`. `native` sends
+     * `response_format: json_schema`; `json-object` sends the broadly supported
+     * JSON mode and relies on the caller to include the schema in model-visible
+     * instructions. Default: `native`.
+     */
+    jsonSchemaTransport?: 'native' | 'json-object';
+    /**
+     * JSON payload parsing policy. `extract` accepts fenced or prose-prefixed JSON.
+     * `exact` requires the complete response content to be one JSON value.
+     * Default: `extract`.
+     */
+    jsonPayloadMode?: 'extract' | 'exact';
     /** Fetch implementation — defaults to global `fetch`. Override for custom transport (e.g. tests). */
     fetch?: typeof fetch;
     /**

package/dist/analyst/index.js

@@ -9,13 +9,13 @@ import {
   diffFindings,
   emitSkillUsageFindings,
   runSemanticConceptJudge
-} from "../chunk-G2GPNLSX.js";
+} from "../chunk-QNVBALEX.js";
 import {
   behavioralAnalyst,
   buildDefaultAnalystRegistry,
   createChatClient,
   deriveEfficiencyFindings
-} from "../chunk-VJ7T5WIO.js";
+} from "../chunk-WP3GEXKI.js";
 import "../chunk-HHWE3POT.js";
 import {
   ANALYST_SEVERITIES,
@@ -71,15 +71,15 @@ import {
   validatePolicyEdit,
   validatePolicyEditCandidateRecord,
   validateUsageSettlementTimeout
-} from "../chunk-FC5NDO3E.js";
-import "../chunk-NJC7U437.js";
+} from "../chunk-MHPEGJHC.js";
+import "../chunk-MK3Q6GCL.js";
 import {
   CostLedger
-} from "../chunk-BGVTIE2C.js";
+} from "../chunk-GRCDRKII.js";
 import "../chunk-VI2UW6B6.js";
 import "../chunk-IR3KBHOY.js";
 import "../chunk-PC4UYEBM.js";
-import "../chunk-XJYR7XFV.js";
+import "../chunk-GC4ATIKK.js";
 import "../chunk-VSMTAMNK.js";
 import "../chunk-ONWEPEDO.js";
 import "../chunk-K4DBDHLK.js";

package/dist/authenticity/index.js

@@ -11,10 +11,11 @@ function escapeRe(s) {
 function declaredNames(content) {
   const names = /* @__PURE__ */ new Set();
   const re = /\b(?:contract|library|interface|abstract\s+contract|class|enum|struct|module|package)\s+([A-Za-z_]\w*)/g;
-  let m;
-  while (m = re.exec(content)) {
+  let m = re.exec(content);
+  while (m) {
     const name = m[1];
     if (name && name.length >= 4) names.add(name);
+    m = re.exec(content);
   }
   return [...names];
 }