@tangle-network/agent-eval 0.145.13 → 0.145.15

package/CHANGELOG.md

@@ -8,6 +8,34 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-
 
 ---
 
+## [0.145.15] — 2026-08-15
+
+### Added
+
+- Add runnable entry points for evaluation, experiment planning, trace analysis, sealed experiments, verification, and optimizer adapters.
+- Add measured unconditional-continuation and gated-stop studies for trace repair.
+
+### Changed
+
+- Align the exact `@tangle-network/agent-core` dependency with `0.9.0`.
+- Size gated-stop confirmation draws from the registered power curve and reject substituted served models before spend.
+
+### Fixed
+
+- Read registered binary outcomes as `1` or `0` in paired estimates and intervals.
+- Give each continuation rollout an independent seed and clear stale assay rewards before every run.
+
+---
+
+## [0.145.14] — 2026-08-15
+
+### Changed
+
+- Align the exact `@tangle-network/agent-core` and `@tangle-network/agent-interface` dependencies with `0.8.1` and `0.53.0`.
+- Packed consumers now resolve the one-copy `0.8.1` Core and `0.53.0` Interface cohort through Eval's release verification.
+
+---
+
 ## [0.145.13] — 2026-08-15
 
 ### Added

package/README.md

@@ -1,22 +1,16 @@
 # `@tangle-network/agent-eval`
 
-Measure agent behavior, compare changes on the same cases, and improve prompts or skills without exposing final test cases to the optimizer.
+Measure agent behavior, compare changes on the same cases, and improve prompts or skills without showing the final test cases to the optimizer.
 
 [![npm](https://img.shields.io/npm/v/@tangle-network/agent-eval.svg)](https://www.npmjs.com/package/@tangle-network/agent-eval)
 [![pypi](https://img.shields.io/pypi/v/agent-eval-rpc.svg)](https://pypi.org/project/agent-eval-rpc/)
 [![tests](https://github.com/tangle-network/agent-eval/actions/workflows/ci.yml/badge.svg)](https://github.com/tangle-network/agent-eval/actions/workflows/ci.yml)
 [![license: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](./LICENSE)
 
-Use this package to:
-
-- run an agent over representative cases and score every result,
-- compare a candidate with a baseline using paired statistics,
-- analyze existing runs, traces, or human feedback,
-- optimize a prompt or skill with official GEPA or SkillOpt,
-- supply your own candidate generator for product-specific changes.
-
 The evaluation path runs in your TypeScript process.
-Model calls occur only through the clients and agents you configure.
+Model calls happen only through the clients and agents you configure.
+
+New to the package? Read [concepts](./docs/concepts.md) first — it takes five minutes and defines every word used here.
 
 ## Install
 
@@ -24,56 +18,10 @@ Model calls occur only through the clients and agents you configure.
 pnpm add @tangle-network/agent-eval
 ```
 
-## Configure Model Calls
-
-Benchmarks, user drivers, executors, built-in judges, completion checkers, and judge adapters accept the same `ChatClient`.
-
-```ts
-import { createChatClient } from '@tangle-network/agent-eval'
-
-const chat = createChatClient({
-  transport: 'router',
-  apiKey: process.env.TANGLE_API_KEY!,
-  defaultModel: 'openai/gpt-4.1',
-  maximumAttempts: 3,
-})
-```
-
-Use `direct-provider` for an OpenAI-compatible endpoint, `cli-bridge` for a local subscription, `sandbox-sdk` for Sandbox, or `custom` to adapt another SDK.
-A custom adapter must return `ChatResponse` and declare `maximumAttempts` before a capped cost ledger can dispatch it.
-
-The official optimizers use the Python bridge.
-Install only the optimizer you plan to run:
-
-```sh
-# Microsoft SkillOpt at the tested source revision
-python -m pip install agent-eval-rpc
-python -m pip install \
-  "skillopt @ git+https://github.com/microsoft/SkillOpt.git@61735e3922efc2b90c6d6cab561e62e98452ca90"
-
-# Standard GEPA engine from the published package
-python -m pip install agent-eval-rpc
-python -m pip install "gepa[full]==0.1.4"
-
-# GEPA Omni and source-only engines
-python -m pip install \
-  "gepa[full] @ git+https://github.com/gepa-ai/gepa.git@f919db0a622e2e9f9204779b81fe00cc1b2d808f"
-
-# DSPy 3.2.1 with Agent Eval metrics
-python -m pip install "agent-eval-rpc[dspy]"
-```
-
-The published GEPA package supports the standard `gepa` engine.
-Sequential, adaptive, best-of, vote, Omni, AutoResearch, Meta Harness, and Best-of-N currently require the tested official source revision.
-Move that revision only after both release and source compatibility tests pass.
-The published `skillopt==0.2.0` wheel omits the prompt files required by `ReflACTTrainer`, so the tested SkillOpt source revision is also intentional.
-DSPy 3.2.1 requires GEPA 0.0.27, while the general bridge requires GEPA 0.1.4.
-Install the DSPy adapter and the general GEPA bridge in separate Python environments.
+## Quickstart
 
-## Evaluate An Agent
-
-This example is offline.
-Replace the agent and judge functions with your product code.
+This example is offline and complete.
+Copy it, run it, then replace the agent and the judge with your product code.
 
 ```ts
 import { defineAgentEval } from '@tangle-network/agent-eval/contract'
@@ -110,371 +58,106 @@ console.log(
 )
 ```
 
-Each call runs every case, records the artifact, applies the same judge, and returns score distributions.
-The surface is the value being changed, such as a prompt, skill, or serialized configuration.
-
-### Inspect cached cells before rerunning
-
-`runCampaign()` refuses to start when an existing cache file is unreadable or lacks trustworthy cost data.
-This check covers the full schedule before concurrent work begins, so one bad cache cannot waste paid calls from earlier cells.
-Use `planCampaignRun()` to see which cells are reusable, runnable, or blocked:
-
-```ts
-const plan = planCampaignRun({
-  scenarios,
-  dispatch,
-  judges: [judge],
-  runDir: 'release-candidate',
-})
-
-console.table(plan.cells)
-```
-
-After inspecting the plan, set `rerunInvalidCachedCells: true` to rerun only blocked cells while retaining valid cached cells.
-Set `resumable: false` only when you intend to rerun every cell.
-When execution uses a shared `costLedger` or custom `costTags`, pass those same values to `planCampaignRun()` so both calls inspect the same receipts.
-
-### Stop after the first failed cell
-
-`runCampaign()` normally records a dispatch or judge error on that cell and continues the remaining cases.
-Set `abortOnCellError: true` when another failed cell would only waste time or money:
-
-```ts
-await runCampaign({
-  scenarios,
-  dispatch,
-  judges: [judge],
-  runDir: 'release-candidate',
-  abortOnCellError: true,
-})
-```
-
-The failed cell is written first to `<runDir>/<cell>/failure-receipt.json`.
-That receipt contains the original error, the cell result, exact call IDs, and settled agent-plus-judge cost and token totals.
-Active sibling cells are cancelled and allowed to finish recording their own receipts before the campaign rejects with the original cell error.
-Leaving `abortOnCellError` unset preserves continue-on-error behavior.
-
-When an external text optimizer uses that continue-on-error behavior, a failed evaluation returns a finite optimizer penalty of `0` with `info.status: 'failed'`.
-The campaign cell remains failed and unscored, so this protocol penalty is not a measured zero.
-The failed cell is not cached; a later request retries it with a fresh `ctx.runAttemptId`, while completed cells remain resumable.
-Set `abortOnCellError: true` to keep fail-fast behavior.
-
-`runProfileMatrix()` retains a failed cell when a moving model alias produced no served model snapshot.
-The row carries `model: 'unknown'`, uncaptured cost, and incomplete token usage so downstream reports can count the failure without inventing measurements.
-Successful moving-model cells still require one immutable served snapshot.
-
-## Adapt Another Text Optimizer
-
-Use `externalTextOptimizationMethod()` when an existing package owns search and selection for a text prompt or named text components.
-Its `run` callback receives the starting candidate plus serialized train and selection cases, but it never receives final test cases.
-The optimizer must score candidates through `context.evaluate()` so Agent Eval can enforce the evaluation limit and use the configured execution and judges.
-Every optimizer-owned paid call must use `context.cost.runPaidCall()`.
-Set `source` to the package version and revision, and set `evaluationId` to a commit, content hash, or other stable identity for the execution and scoring behavior.
-Agent Eval derives the run identity from those values, the exact dispatch identity, the optimizer settings, the starting surface, the described data, and the seed.
-The callback returns the selected candidate, whether compatible state was restored, and how optimizer spend was recorded.
-
-See [Adapt A Third-Party Text Optimizer](./docs/campaign-proposers.md#adapt-a-third-party-text-optimizer) for a complete minimal adapter.
-
-## Optimize With Official GEPA
-
-`gepaOptimizationMethod()` delegates candidate search and recipe composition to the installed GEPA package.
-Agent Eval supplies the train and selection cases, executes candidates, records cost, and evaluates the selected result on final cases after GEPA exits.
-
-```ts
-import { gepaOptimizationMethod } from '@tangle-network/agent-eval/campaign'
-
-// Supplied by the package that owns model execution. Discovery uses Runtime
-// with one exact AgentProfile; Agent Eval receives no provider credential.
-declare const optimizerExecution: {
-  call: import('@tangle-network/agent-eval/campaign').ExternalOptimizerModelCall
-  callRef: string
-}
-
-const optimizerPricing = {
-  inputUsdPerMillion: Number(process.env.OPTIMIZER_INPUT_USD_PER_MILLION),
-  outputUsdPerMillion: Number(process.env.OPTIMIZER_OUTPUT_USD_PER_MILLION),
-}
-
-const gepa = gepaOptimizationMethod<MyCase, MyArtifact>({
-  objective: 'Improve the instructions so the agent returns valid, complete JSON.',
-  evaluationId: 'json-agent',
-  recipe: {
-    kind: 'engine',
-    run: {
-      engine: 'gepa',
-      maxEvaluations: 40,
-      maxProposerCostUsd: 5,
-    },
-  },
-  optimizer: {
-    model: 'gpt-4.1-mini',
-    ...optimizerExecution,
-    budget: {
-      maxCostUsd: 5,
-      maxRequests: 100,
-      maxRequestBytes: 2_000_000,
-      maxResponseBytes: 2_000_000,
-      maxOutputTokensPerRequest: 32_768,
-      pricing: optimizerPricing,
-    },
-  },
-  describeScenario: (scenario) => ({ input: scenario.input }),
-  describeArtifact: (artifact) => ({ output: artifact.output }),
-})
-```
-
-GEPA also supports official sequential, adaptive, best-of, vote, and Omni recipes through the same factory.
-When every recipe stage uses the standard GEPA engine, Agent Eval gives Python only a loopback address and asks the caller-owned execution package to make every model call.
-Every call must return measured usage and finite JSON execution evidence; Agent Eval stores both and fails the attempt when either is missing.
-Other official GEPA engines can still run through `engineConfig`, but their model spend remains incomplete unless the engine reports it.
-Custom engines can register through GEPA's official registry by listing their Python modules in `engineModules`.
-
-Run the repository example:
-
-```sh
-OPTIMIZERS=gepa \
-LLM_API_KEY="$OPENAI_API_KEY" \
-OPTIMIZER_EXECUTION_OWNER_MODULE=@acme/runtime-optimizer-owner \
-GEPA_PRICE_IN_PER_M=0.4 \
-GEPA_PRICE_OUT_PER_M=1.6 \
-pnpm tsx examples/compare-optimization-methods/index.ts
-```
-
-Replace the example rates with the exact endpoint rates.
-
-## Optimize With Official SkillOpt
-
-`skillOptOptimizationMethod()` runs Microsoft's `ReflACTTrainer` against the same TypeScript execution and scoring path.
-SkillOpt receives train and selection cases but never receives final cases.
-
-```ts
-import { skillOptOptimizationMethod } from '@tangle-network/agent-eval/campaign'
-
-declare const optimizerExecution: {
-  call: import('@tangle-network/agent-eval/campaign').ExternalOptimizerModelCall
-  callRef: string
-}
-
-const optimizerPricing = {
-  inputUsdPerMillion: Number(process.env.OPTIMIZER_INPUT_USD_PER_MILLION),
-  outputUsdPerMillion: Number(process.env.OPTIMIZER_OUTPUT_USD_PER_MILLION),
-}
-
-const skillopt = skillOptOptimizationMethod<MyCase, MyArtifact>({
-  objective: 'Improve the skill so the agent returns valid, complete JSON.',
-  evaluationId: 'json-agent',
-  trainer: {
-    epochs: 2,
-    batchSize: 4,
-  },
-  optimizer: {
-    model: 'gpt-4.1-mini',
-    ...optimizerExecution,
-    budget: {
-      maxCostUsd: 5,
-      maxRequests: 100,
-      maxRequestBytes: 2_000_000,
-      maxResponseBytes: 2_000_000,
-      maxOutputTokensPerRequest: 32_768,
-      pricing: optimizerPricing,
-    },
-  },
-  maxEvaluations: 80,
-  describeScenario: (scenario) => ({ input: scenario.input }),
-  describeArtifact: (artifact) => ({ output: artifact.output }),
-})
-```
-
-Replace the example rates with the exact rates for your endpoint.
-Agent Eval places a local OpenAI-compatible proxy between each standard optimizer and the caller-owned execution package.
-The proxy enforces request, byte, and token limits before one callback invocation, records the caller's measured receipt and execution evidence, and does not accept a provider URL or credential.
-Dollar limits and pricing are optional; omit both when billed USD is unknown instead of supplying a catalog estimate as if it were observed billing.
-
-## Optimize A DSPy Program
-
-DSPy owns its program optimizers.
-`DspyJudgeMetric` lets them use the same Agent Eval rubric as TypeScript agents.
-
-```python
-import dspy
-
-from agent_eval_rpc import DspyJudgeMetric
-
-dspy.configure(lm=dspy.LM("openai/gpt-4.1-mini"))
-dspy.configure_cache(restrict_pickle=True)
-metric = DspyJudgeMetric(rubric_name="answer-quality")
-
-# GEPA needs both the numeric score and diagnostic feedback.
-optimizer = dspy.GEPA(
-    metric=metric.feedback,
-    reflection_lm=dspy.LM("openai/gpt-4.1-mini"),
-    max_metric_calls=100,
-)
-optimized_program = optimizer.compile(program, trainset=train, valset=selection)
-
-# MIPROv2, SIMBA, and few-shot optimizers use the numeric metric directly.
-mipro = dspy.MIPROv2(metric=metric, auto="light")
-```
-
-Use official DSPy directly for DSPy programs.
-Use `gepaOptimizationMethod()` for text or named component surfaces in non-DSPy agents.
-Use agent-runtime's worktree path for executable code changes.
-
-## Compare Complete Methods
-
-`compareOptimizationMethods()` gives each method the same starting surface, execution function, judges, train cases, and selection cases.
-It waits for optimization to finish before it evaluates any selected surface on the final cases.
-
-```ts
-import { compareOptimizationMethods } from '@tangle-network/agent-eval/campaign'
-
-const result = await compareOptimizationMethods({
-  methods: [gepa, skillopt],
-  baselineSurface,
-  trainScenarios,
-  selectionScenarios,
-  testScenarios,
-  dispatchWithSurface,
-  judges,
-  runDir: '.agent-eval/optimizer-comparison',
-})
-
-console.table(result.scores)
-```
-
-Read `scores` for final-case lift and intervals.
-Read `pairwise` before claiming one method beat another.
-Read `totalCost.accountingComplete` before using the reported dollars as a complete total.
-Each official method score records the optimizer and bridge package versions, source revisions and source-tree hashes, Python runtime, configured optimizer model when present, custom engine module hashes, compatible run ID, exact attempt ID, resume status, evaluation count, artifact directory, and available optimizer token usage in `provenance`.
-Call `readExternalOptimizerObservationArtifact()` with `provenance.observations` to read every distinct callback-submitted candidate.
-The reader verifies the artifact digest, canonical rows, sequence, candidate identities, and summary counts before returning candidates.
-This verification proves that the bytes match the supplied summary; use a summary from trusted method provenance when authenticity matters.
-For a direct standard GEPA run, call `readGepaCandidatePopulationArtifact()` with `provenance.gepaCandidatePopulation`.
-It returns GEPA's accepted candidates with exact parent indices, aggregate scores, per-case selection scores, and discovery evaluation counts.
-The callback artifact remains the complete source for rejected or refused proposals that GEPA did not add to its accepted population.
-
-The [optimizer guide](./docs/campaign-proposers.md) covers recipes, budgets, resuming, and data separation.
-The [runnable comparison](./examples/compare-optimization-methods/) can run GEPA, SkillOpt, or both.
+Each call runs every case, records what the agent produced, applies the same judge, and returns score distributions.
+
+Three words carry this example.
+A **case** is one task the agent must do.
+A **surface** is the value being changed: a prompt, a skill, or a serialized configuration.
+A **judge** is a function that scores one produced result.
+
+`expectUsage: 'off'` is set because this agent makes no paid calls.
+The default, `'assert'`, fails a run whose cells report no cost receipt.
+Keep the default whenever real model calls happen.
+
+Runnable copy: [`examples/evaluate-a-change`](./examples/evaluate-a-change/).
+
+## Which Front Door
+
+Every row is a function you call. Each links to a runnable example.
+
+| When to call it | What you give it | What you get back |
+|---|---|---|
+| [`defineAgentEval()`](./examples/evaluate-a-change/) — you changed a surface and must know whether it helped | cases, an agent, a judge, a starting surface | `evaluate()` for scores, `improve()` for a search plus a release decision |
+| [`selfImprove()`](./examples/selfimprove-quickstart/) — you want candidate generation, scoring, and a release decision in one call | cases, an agent, a judge, a starting surface | a report, a winner surface, and a `gateDecision` |
+| [`analyzeRuns()`](./examples/analyze-existing-runs/) — the runs already happened and no agent needs to run again | `RunRecord[]` | an `InsightReport`: distributions, paired lift, judge agreement, cost, failure clusters |
+| `fromFeedbackTable()` ([example](./examples/customer-feedback-loop/)) / `fromOtelSpans()` ([example](./examples/customer-otel-traces/)) — your data is in a table or an OTel collector, not in `RunRecord` shape | source rows or spans | `RunRecord[]` ready for `analyzeRuns()` |
+| [`planCampaignRun()` / `runCampaign()`](./examples/plan-before-you-spend/) — you need direct control of the case grid, or you must see it before paying for it | cases, a dispatch function, judges, a run directory | a per-cell schedule, then a campaign result with cached cells |
+| [`loadEvalFixtureScenarios()`](./examples/eval-fixtures-quickstart/) — agents should add cases as folders on disk | `evals/<name>/PROMPT.md` plus checks | `Scenario[]` for `runCampaign()` |
+| [`compareOptimizationMethods()`](./examples/compare-optimization-methods/) — two search methods must be compared at equal budget | methods, a starting surface, train, selection, and final cases | per-method final lift, intervals, pairwise contrasts, and cost |
+| [`gepaOptimizationMethod()` / `skillOptOptimizationMethod()`](./examples/compare-optimization-methods/) — official GEPA or Microsoft SkillOpt should own the search | an objective, a recipe or trainer, an optimizer budget | an optimization method for the comparison above |
+| [`externalTextOptimizationMethod()`](./examples/adapt-a-text-optimizer/) — another package owns text search and you keep the scoring | the package identity, limits, and a `run` callback | the same, with the final cases never exposed |
+| [`SurfaceProposer`](./examples/selfimprove-quickstart/) — candidate generation belongs to your product | a `propose()` function | candidates the campaign executes, scores, and gates |
+| [`runProfileMatrix()`](./docs/eval-surface-map.md) — the same cases must run across models or profiles | axes of models and profiles, cases | one row per cell, with an explicit `unknown` model rather than an invented one |
+| [`sealExperiment()` / `openSealedExperiment()`](./examples/sealed-experiment/) — the result must convince someone who does not trust you | arms, an admission funnel, an estimand, an interval, a decision table | a hashed rule tree, and executors that can run no other rule |
+| [`runEquivalenceCheck()` / `VERIFICATION_STRATEGIES`](./examples/verify-without-an-answer-key/) — the work has no held-out test suite | a claim, two blind arms, an injected checker | a certification that names who vouched and how it can fail |
+| [`AnalystRegistry.runExact()`](./examples/custom-trace-analyst/) — a batch of runs failed and you need cited findings | recorded evidence, a declared analyst list | findings with evidence references, an execution plan, and a receipt |
+| [`runAnalystBenchmark()`](./docs/trace-analysis.md) — an analyst's accuracy must be measured, not assumed | labeled issues and exact span locations | scored findings, trace reads, model calls, tokens, cost, and runtime |
+| [`deltaRepair()`](./docs/trace-repair-grader.md) — a finding must be graded by executing the repair it proposes | a trajectory, an analyst finding, a sandbox | the repair's measured effect against a no-fix control |
+| [`replayVerify()`](./docs/trajectory-replay.md) — you must know whether a recorded failure still reproduces | a recorded shell trajectory and its pinned image | a re-execution verdict and the divergences found |
+| [`analyzeSupervisorRun()`](./docs/adapters-observability.md) — a recursive or supervised run directory must be read | a run directory | counts that stay missing when a measurement is missing, never zero |
+| [`buildRlDataset()`](./examples/publish-rl-dataset/) — scored runs should become training data | run records and preferences | reward, preference, and supervised rows |
 
-## Supply Your Own Candidate Generator
+## Configure Model Calls
 
-Use `SurfaceProposer` when candidate creation belongs to your product or an agent runtime.
-The campaign still owns execution, scoring, history, stopping, and release decisions.
+Benchmarks, user drivers, executors, built-in judges, completion checkers, and judge adapters all take the same `ChatClient`.
 
 ```ts
-import {
-  defineAgentEval,
-  type SurfaceProposer,
-} from '@tangle-network/agent-eval/contract'
-
-const proposer: SurfaceProposer = {
-  kind: 'product-rules',
-  async propose({ currentSurface, populationSize }) {
-    const prompt = String(currentSurface)
-    return [
-      {
-        surface: `${prompt}\nReturn JSON only.`,
-        label: 'json-only',
-        rationale: 'Training failures contained prose around the JSON object.',
-      },
-    ].slice(0, populationSize)
-  },
-}
-
-const result = await defineAgentEval({
-  scenarios,
-  agent,
-  model: 'gpt-4.1-2025-04-14',
-  judge,
-  baselineSurface,
-  proposer,
-  budget: { generations: 1, populationSize: 1, holdoutFraction: 0.3 },
-}).improve()
-```
-
-`model` supplies the worker snapshot when the agent does not report paid calls through `ctx.cost.runPaidCall()`.
-When every cell reports a concrete model receipt, omit it.
-
-Run the complete offline example:
-
-```sh
-pnpm tsx examples/selfimprove-quickstart/index.ts
-```
-
-## Start From Existing Runs
-
-You do not need a runnable agent to analyze data you already captured.
-Use `analyzeRuns()` for `RunRecord[]`.
-For traces, run a registry of built-in or custom analysts, measure it on labeled issues and exact span locations, then turn only reviewed findings into eval data.
-For a public quality check, convert CodeTraceBench with `traces import-codetracebench`, then run `agent-eval analyst-benchmark` against pinned labels.
-The command compares an empty baseline with the official DSPy `RLM` trace analyst and records its trace reads, model calls, tokens, cost, runtime, and cited findings.
-
-Use `AnalystRegistry.runExact()` when the caller, rather than registry defaults, must own every execution choice.
-The ordered `analystIds` array is the execution order, and `null` explicitly disables optional budget, timeout, cancellation, cost, tag, or prior-finding channels.
-Exact runs are serial; callers that need recursive or concurrent scheduling compose them through their runtime rather than adding a second scheduler here.
+import { createChatClient } from '@tangle-network/agent-eval'
 
-```ts
-const result = await registry.runExact('analysis-1', inputs, {
-  analystIds: ['failure-mode', 'improvement'],
-  budget: { kind: 'equal', totalUsd: 2 },
-  totalTimeoutMs: 30_000,
-  signal: null,
-  costLedger: null,
-  costLedgerIdentity: null,
-  costPhase: null,
-  tags: null,
-  priorFindings: null,
-  chainFindings: true,
-  missingInputMode: 'abort',
-  applyRegistryHooks: false,
-  useRegistryChat: false,
+const chat = createChatClient({
+  transport: 'router',
+  apiKey: process.env.TANGLE_API_KEY!,
+  defaultModel: 'openai/gpt-4.1',
+  maximumAttempts: 3,
 })
 ```
 
-Custom analysts passed to `runExact()` declare canonical `executionConfig`.
-The same `defineTraceAnalyst()` helper returns an exact-capable analyst when that field is present.
-Built-in analysts already declare it.
-Trace analysts selected by `runExact()` also require `aiIdentity`, using the same non-secret `id`, `version`, and canonical `config` shape as cost ledgers, registry hooks, and registry chat clients.
-Exact lifecycle hooks receive frozen snapshots for observation; they cannot rewrite the planned context.
-Persisted results store configuration digests, not raw configuration.
-The persisted plan records the exact equal or weighted allocation for every routed analyst, and archival validates summaries against that same plan.
-Every exact receipt says whether it is `complete` or `failed`; a complete receipt must cover the full plan, while a failed receipt may contain only the executed prefix.
-Any failure after an exact run starts rejects with `ExactAnalystRunExecutionError`; its immutable failed receipt preserves valid completed summaries, findings, usage, and cost.
+Use `direct-provider` for an OpenAI-compatible endpoint, `cli-bridge` for a local subscription, `sandbox-sdk` for Sandbox, or `custom` to adapt another SDK.
+A custom adapter must return a `ChatResponse` and declare `maximumAttempts` before a capped cost account can dispatch it.
 
-See [concepts](./docs/concepts.md), [customer paths](./docs/customer-journeys.md), and [trace analysis](./docs/trace-analysis.md).
+The official GEPA and SkillOpt optimizers run through a Python bridge.
+Install commands, version pins, and the reason for each pin:
+[GEPA](./docs/campaign-proposers.md#install-official-gepa),
+[SkillOpt](./docs/campaign-proposers.md#install-official-skillopt),
+and [DSPy](./docs/campaign-proposers.md#use-official-dspy-optimizers).
 
 ## Entry Points
 
 | Import | Use |
 |---|---|
-| `@tangle-network/agent-eval/contract` | Define an evaluation, run it, improve with a custom candidate generator, and analyze runs. |
-| `@tangle-network/agent-eval/campaign` | Control campaigns, official optimization methods, comparisons, storage, and release rules. |
-| `@tangle-network/agent-eval/profile-cell` | Create and validate portable agent-profile identities. |
-| `@tangle-network/agent-eval/ledger-core` | Generic hash-chained append-only journal: idempotent append, chain verification, trusted-head pinning, replay-to-projection, cross-process locking. |
-| `@tangle-network/agent-eval/reporting` | Statistical comparisons and report rendering. |
+| `@tangle-network/agent-eval/contract` | Define an evaluation, run it, improve it, and analyze existing runs. |
+| `@tangle-network/agent-eval/campaign` | Campaigns, optimization methods, comparisons, storage, and release rules. |
+| `@tangle-network/agent-eval/experiment` | Experiments as sealed objects: registered rules, funnels, estimands, refusals. |
 | `@tangle-network/agent-eval/analyst` | Built-in and custom trace analysts, labeled comparison, costs, and reports. |
+| `@tangle-network/agent-eval/trace-repair` | Grade one analyst finding by executing the repair it proposes. |
+| `@tangle-network/agent-eval/trajectory-replay` | Re-execute a recorded shell trajectory and check whether its failure reproduces. |
 | `@tangle-network/agent-eval/traces` | Store, replay, and inspect structured traces. |
-| `@tangle-network/agent-eval/supervisor-run` | Read loops or agent-runtime recursive run directories without collapsing missing measurements to zero. |
-| `@tangle-network/agent-eval/trajectory-replay` | Re-execute a recorded shell trajectory in its own image and check whether the recorded failure reproduces. |
-| `@tangle-network/agent-eval/trace-repair` | Grade one analyst finding by executing the repair it proposes, and measure Delta-repair against a no-fix control. |
+| `@tangle-network/agent-eval/reporting` | Statistical comparisons and report rendering. |
+| `@tangle-network/agent-eval/supervisor-run` | Read recursive run directories without collapsing missing measurements to zero. |
+| `@tangle-network/agent-eval/profile-cell` | Create and validate portable agent-profile identities. |
+| `@tangle-network/agent-eval/ledger-core` | Append-only hash-chained journal with idempotent append and chain verification. |
 | `@tangle-network/agent-eval/benchmarks` | Benchmark adapters and retrieval metrics. |
 | `@tangle-network/agent-eval/rl` | Export rewards, preferences, and training rows. |
 | `@tangle-network/agent-eval/wire` | HTTP and RPC schemas for other languages. |
 
-Use subpaths when you want an explicit capability boundary.
 Use the root import for common primitives.
+Use a subpath when you want an explicit capability boundary.
 
-## Examples
+## Documentation
 
-| Goal | Example |
+| Question | Read |
 |---|---|
-| Evaluate and improve with a custom candidate generator | [`selfimprove-quickstart`](./examples/selfimprove-quickstart/) |
-| Run official GEPA or SkillOpt | [`compare-optimization-methods`](./examples/compare-optimization-methods/) |
-| Analyze human feedback | [`customer-feedback-loop`](./examples/customer-feedback-loop/) |
-| Analyze OpenTelemetry traces | [`customer-otel-traces`](./examples/customer-otel-traces/) |
-| Run public benchmark adapters | [`benchmarks`](./examples/benchmarks/) |
-
-See the [example index](./examples/README.md) for the full list.
+| What do these words mean? | [`docs/concepts.md`](./docs/concepts.md) |
+| Why does this package exist, and where is it going? | [`docs/charter.md`](./docs/charter.md) |
+| Which `run*` function do I want? | [`docs/eval-surface-map.md`](./docs/eval-surface-map.md) |
+| How do I choose a candidate-generation method? | [`docs/campaign-proposers.md`](./docs/campaign-proposers.md) |
+| What is in an `InsightReport`? | [`docs/insight-report.md`](./docs/insight-report.md) |
+| How do I register an experiment as a sealed object? | [`docs/experiment.md`](./docs/experiment.md) |
+| How is something certified without an answer key? | [`docs/verification-strategies.md`](./docs/verification-strategies.md) |
+| Where does every verifier land its result? | [`docs/verdicts.md`](./docs/verdicts.md) |
+| How do I score a string from another language? | [`docs/wire-protocol.md`](./docs/wire-protocol.md) |
+
+The [example index](./examples/README.md) lists every runnable example.
 
 ## Development
 

package/dist/analyst/index.d.ts

@@ -1299,7 +1299,7 @@ declare const ANALYST_BENCHMARK_HELP = "agent-eval analyst-benchmark\n\nRun the
 declare const ANALYST_BENCHMARK_IMPLEMENTATION_DIGEST_ALGORITHM = "sha256-canonical-source-manifest";
 declare const ANALYST_BENCHMARK_DEPENDENCY_LOCK_DIGEST_ALGORITHM = "sha256-canonical-file-manifest";
 declare const ANALYST_BENCHMARK_DEPENDENCY_LOCK_FILES: readonly string[];
-declare const ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256 = "ca56679a57495f6d4012ed2b242586da36c231195bd802507ec18cc541b89fe5";
+declare const ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256 = "ccd8fbc5b9517c5a0297020cf0d1ca09cfb5d84e393d5d461d5899b315baa2ab";
 /** The published benchmark evidence was produced at this package version, by
  * the retired one-shot direct runner, before trace analysts moved to the
  * recursive DSPy RLM engine. Both evidence digests below are historical facts

package/dist/analyst/index.js

@@ -7,7 +7,7 @@ import { a as assertExactRegistryRunOpts, c as KNOWLEDGE_GAP_KIND_SPEC, d as CON
 import { t as createDspyRlmTraceEngine } from "../dspy-rlm-engine-CqwhQQBw.js";
 import { a as diffFindings, i as defaultIsMaterial, n as runSemanticConceptJudge, r as FindingsStore, t as SEMANTIC_CONCEPT_JUDGE_VERSION } from "../semantic-concept-judge-C6M-qOeb.js";
 import { a as scoreAnalystFindings, i as summarizeAnalystBenchmarkRunner, n as runAnalystBenchmark, r as traceStoreEvidenceResolver, t as registryBenchmarkRunner } from "../benchmark-BhT16ep9.js";
-import { $ as analystBenchmarkDependencyLockDigest, A as compareAnalystRunners, B as DEFAULT_MAX_VERIFICATION_ARTIFACT_BYTES, C as analystDefinitionProtocolSha256, D as readAnalystBenchmarkArtifact, E as expandCodeTraceFailureBlocks, F as MAX_INCORRECT_BLOCKS, G as ANALYST_BENCHMARK_DEPENDENCY_LOCK_FILES, H as loadCodeTraceVerificationArtifacts, I as MAX_INCORRECT_BLOCK_STEPS, J as ANALYST_BENCHMARK_EVIDENCE_IMPLEMENTATION_SHA256, K as ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256, L as publicBenchmarkProtocolSha256, M as effectiveAnalystProtocolSha256, N as readAnalystInstructionsOverride, O as renderCodeTraceCalibrationMarkdown, P as CODE_TRACE_BENCH_ANALYST_PROMPT, Q as ANALYST_BENCHMARK_IMPLEMENTATION_SHA256, R as publicBenchmarkRlmInstructions, S as analystDefinitionAsymmetries, T as emptyPublicBenchmarkRunner, U as parseVerificationOutcome, V as appendVerificationArtifactsToOtlp, W as ANALYST_BENCHMARK_DEPENDENCY_LOCK_DIGEST_ALGORITHM, X as ANALYST_BENCHMARK_IMPLEMENTATION_DIGEST_ALGORITHM, Y as ANALYST_BENCHMARK_EVIDENCE_PACKAGE_VERSION, Z as ANALYST_BENCHMARK_IMPLEMENTATION_FILES, _ as runReplVariableAnalystDefinition, a as primeAnalystProtocolSha256, at as AGENT_RX_UPSTREAM_REVISION, b as runChunkedAnalystDefinition, c as nodeHttpPrimeBridgeTransport, ct as codeTraceBenchCase, d as publicBenchmarkDistributions, dt as agentRxPredictionsToFindings, et as analystBenchmarkImplementationDigest, f as publicBenchmarkSelectionReport, ft as normalizeAgentRxCategory, g as rlmEngineLimits, h as publicRlmAnalystDefinition, i as createPrimeBenchmarkRunner, it as ANALYST_BENCHMARK_OBSERVATIONS_FILE, j as analystInstructionsOverrideFromText, k as summarizeCodeTraceCalibration, l as loadPublicBenchmarkRows, lt as codeTracerPredictionsToFindings, m as createPublicBenchmarkRlmRunner, mt as normalizeBenchmarkLabel, n as runAnalystBenchmarkCommand, nt as ANALYST_BENCHMARK_LOCAL_RECEIPT_FILE, o as primeCodeTraceAnalystDefinition, ot as renderAgentRxCalibrationMarkdown, p as selectPublicBenchmarkRows, pt as roundAgentRxStep, q as ANALYST_BENCHMARK_EVIDENCE_DEPENDENCY_LOCK_SHA256, r as renderAnalystBenchmarkMarkdown, rt as ANALYST_BENCHMARK_MANIFEST_FILE, s as runInlineAnalystDefinition, st as summarizeAgentRxCalibration, t as ANALYST_BENCHMARK_HELP, tt as ANALYST_BENCHMARK_COST_LEDGER_FILE, u as preparePublicAnalystBenchmark, ut as agentRxBenchmarkCase, v as createPublicBenchmarkDirectRunner, w as adaptPublicBenchmarkFindings, x as AnalystExpressivenessError, y as publicDirectAnalystDefinition, z as publicBenchmarkSystemPrompt } from "../benchmark-command-RyFRoiYm.js";
+import { $ as analystBenchmarkDependencyLockDigest, A as compareAnalystRunners, B as DEFAULT_MAX_VERIFICATION_ARTIFACT_BYTES, C as analystDefinitionProtocolSha256, D as readAnalystBenchmarkArtifact, E as expandCodeTraceFailureBlocks, F as MAX_INCORRECT_BLOCKS, G as ANALYST_BENCHMARK_DEPENDENCY_LOCK_FILES, H as loadCodeTraceVerificationArtifacts, I as MAX_INCORRECT_BLOCK_STEPS, J as ANALYST_BENCHMARK_EVIDENCE_IMPLEMENTATION_SHA256, K as ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256, L as publicBenchmarkProtocolSha256, M as effectiveAnalystProtocolSha256, N as readAnalystInstructionsOverride, O as renderCodeTraceCalibrationMarkdown, P as CODE_TRACE_BENCH_ANALYST_PROMPT, Q as ANALYST_BENCHMARK_IMPLEMENTATION_SHA256, R as publicBenchmarkRlmInstructions, S as analystDefinitionAsymmetries, T as emptyPublicBenchmarkRunner, U as parseVerificationOutcome, V as appendVerificationArtifactsToOtlp, W as ANALYST_BENCHMARK_DEPENDENCY_LOCK_DIGEST_ALGORITHM, X as ANALYST_BENCHMARK_IMPLEMENTATION_DIGEST_ALGORITHM, Y as ANALYST_BENCHMARK_EVIDENCE_PACKAGE_VERSION, Z as ANALYST_BENCHMARK_IMPLEMENTATION_FILES, _ as runReplVariableAnalystDefinition, a as primeAnalystProtocolSha256, at as AGENT_RX_UPSTREAM_REVISION, b as runChunkedAnalystDefinition, c as nodeHttpPrimeBridgeTransport, ct as codeTraceBenchCase, d as publicBenchmarkDistributions, dt as agentRxPredictionsToFindings, et as analystBenchmarkImplementationDigest, f as publicBenchmarkSelectionReport, ft as normalizeAgentRxCategory, g as rlmEngineLimits, h as publicRlmAnalystDefinition, i as createPrimeBenchmarkRunner, it as ANALYST_BENCHMARK_OBSERVATIONS_FILE, j as analystInstructionsOverrideFromText, k as summarizeCodeTraceCalibration, l as loadPublicBenchmarkRows, lt as codeTracerPredictionsToFindings, m as createPublicBenchmarkRlmRunner, mt as normalizeBenchmarkLabel, n as runAnalystBenchmarkCommand, nt as ANALYST_BENCHMARK_LOCAL_RECEIPT_FILE, o as primeCodeTraceAnalystDefinition, ot as renderAgentRxCalibrationMarkdown, p as selectPublicBenchmarkRows, pt as roundAgentRxStep, q as ANALYST_BENCHMARK_EVIDENCE_DEPENDENCY_LOCK_SHA256, r as renderAnalystBenchmarkMarkdown, rt as ANALYST_BENCHMARK_MANIFEST_FILE, s as runInlineAnalystDefinition, st as summarizeAgentRxCalibration, t as ANALYST_BENCHMARK_HELP, tt as ANALYST_BENCHMARK_COST_LEDGER_FILE, u as preparePublicAnalystBenchmark, ut as agentRxBenchmarkCase, v as createPublicBenchmarkDirectRunner, w as adaptPublicBenchmarkFindings, x as AnalystExpressivenessError, y as publicDirectAnalystDefinition, z as publicBenchmarkSystemPrompt } from "../benchmark-command-CIosBh1L.js";
 import { a as extractPrimeJsonObject, c as primeProtocolSha256, d as runPrimeExchange, f as decodeReplyRows, i as emptyPrimeRawUsage, l as primeReplyDefect, n as buildPrimePrompt, o as mergePrimeRawUsage, r as buildPrimeRepairPrompt, s as normalizePrimeUsage, t as analystUsageReceiptFromPrimeUsage, u as projectPrimeTrajectory } from "../prime-protocol-6tZTVsWm.js";
 import { existsSync, readFileSync, readdirSync, statSync } from "node:fs";
 import { join } from "node:path";

package/dist/{benchmark-command-RyFRoiYm.js → benchmark-command-CIosBh1L.js}

@@ -1209,7 +1209,7 @@ const ANALYST_BENCHMARK_DEPENDENCY_LOCK_FILES = Object.freeze([
 	"package.json",
 	"pnpm-lock.yaml"
 ]);
-const ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256 = "ca56679a57495f6d4012ed2b242586da36c231195bd802507ec18cc541b89fe5";
+const ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256 = "ccd8fbc5b9517c5a0297020cf0d1ca09cfb5d84e393d5d461d5899b315baa2ab";
 /** The published benchmark evidence was produced at this package version, by
 * the retired one-shot direct runner, before trace analysts moved to the
 * recursive DSPy RLM engine. Both evidence digests below are historical facts
@@ -6369,4 +6369,4 @@ function shellQuote(value) {
 //#endregion
 export { analystBenchmarkDependencyLockDigest as $, compareAnalystRunners as A, DEFAULT_MAX_VERIFICATION_ARTIFACT_BYTES as B, analystDefinitionProtocolSha256 as C, readAnalystBenchmarkArtifact as D, expandCodeTraceFailureBlocks as E, MAX_INCORRECT_BLOCKS as F, ANALYST_BENCHMARK_DEPENDENCY_LOCK_FILES as G, loadCodeTraceVerificationArtifacts as H, MAX_INCORRECT_BLOCK_STEPS as I, ANALYST_BENCHMARK_EVIDENCE_IMPLEMENTATION_SHA256 as J, ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256 as K, publicBenchmarkProtocolSha256 as L, effectiveAnalystProtocolSha256 as M, readAnalystInstructionsOverride as N, renderCodeTraceCalibrationMarkdown as O, CODE_TRACE_BENCH_ANALYST_PROMPT as P, ANALYST_BENCHMARK_IMPLEMENTATION_SHA256 as Q, publicBenchmarkRlmInstructions as R, analystDefinitionAsymmetries as S, emptyPublicBenchmarkRunner as T, parseVerificationOutcome as U, appendVerificationArtifactsToOtlp as V, ANALYST_BENCHMARK_DEPENDENCY_LOCK_DIGEST_ALGORITHM as W, ANALYST_BENCHMARK_IMPLEMENTATION_DIGEST_ALGORITHM as X, ANALYST_BENCHMARK_EVIDENCE_PACKAGE_VERSION as Y, ANALYST_BENCHMARK_IMPLEMENTATION_FILES as Z, runReplVariableAnalystDefinition as _, primeAnalystProtocolSha256 as a, AGENT_RX_UPSTREAM_REVISION as at, runChunkedAnalystDefinition as b, nodeHttpPrimeBridgeTransport as c, codeTraceBenchCase as ct, publicBenchmarkDistributions as d, agentRxPredictionsToFindings as dt, analystBenchmarkImplementationDigest as et, publicBenchmarkSelectionReport as f, normalizeAgentRxCategory as ft, rlmEngineLimits as g, publicRlmAnalystDefinition as h, createPrimeBenchmarkRunner as i, ANALYST_BENCHMARK_OBSERVATIONS_FILE as it, analystInstructionsOverrideFromText as j, summarizeCodeTraceCalibration as k, loadPublicBenchmarkRows as l, codeTracerPredictionsToFindings as lt, createPublicBenchmarkRlmRunner as m, normalizeBenchmarkLabel as mt, runAnalystBenchmarkCommand as n, ANALYST_BENCHMARK_LOCAL_RECEIPT_FILE as nt, primeCodeTraceAnalystDefinition as o, renderAgentRxCalibrationMarkdown as ot, selectPublicBenchmarkRows as p, roundAgentRxStep as pt, ANALYST_BENCHMARK_EVIDENCE_DEPENDENCY_LOCK_SHA256 as q, renderAnalystBenchmarkMarkdown as r, ANALYST_BENCHMARK_MANIFEST_FILE as rt, runInlineAnalystDefinition as s, summarizeAgentRxCalibration as st, ANALYST_BENCHMARK_HELP as t, ANALYST_BENCHMARK_COST_LEDGER_FILE as tt, preparePublicAnalystBenchmark as u, agentRxBenchmarkCase as ut, createPublicBenchmarkDirectRunner as v, adaptPublicBenchmarkFindings as w, AnalystExpressivenessError as x, publicDirectAnalystDefinition as y, publicBenchmarkSystemPrompt as z };
 
-//# sourceMappingURL=benchmark-command-RyFRoiYm.js.map
+//# sourceMappingURL=benchmark-command-CIosBh1L.js.map