@tangle-network/agent-eval 0.137.0 → 0.139.0

package/CHANGELOG.md

@@ -4,6 +4,84 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-
 
 ---
 
+## [0.139.0] - 2026-07-31 - recursive RLM trace analysts and caller failure reasons
+
+### Added
+
+- Trace analysts run official DSPy RLMs; the Ax stack is retired (#495).
+  The published CodeTraceBench evidence remains bound to the retired direct runner via the evidence digests; a fresh certified run must replace it before any accuracy number is attributed to the new engine.
+- `CostLedger.reconcile` accepts a caller-supplied failure reason: `reconcile(callId, observed, { error })` settles a failed receipt carrying that reason, and supplying a reason implies failure.
+  0.138.0 had narrowed `CostReceipt.error` to the ledger's own `'paid-call-failed'`, silently discarding caller reasons — a crash orphan settled as a successful $0 call.
+  The receipt schema accepts any non-empty reason again, so ledgers persisted before 0.138.0 parse.
+
+## [0.138.0] - 2026-07-30 - exact analyst runs with sealed receipts
+
+### Fixed
+
+- A release version bump no longer invalidates the published analyst benchmark evidence.
+  The dependency-lock pin now tracks the current lockfiles, the evidence keeps its own creation-time digest and package version, and a test proves the two locks differ by the version stamp alone — a real dependency change still forces a new benchmark run or explicit retirement of the evidence.
+
+### Added
+
+- `AnalystRegistry.runExact()` requires ordered analyst ids and an explicit value for every run-policy field.
+  It never inherits registry insertion order or the constructor's default budget.
+  Disabled budgets, timeouts, cancellation, cost attribution, tags, and prior findings use explicit `null` values.
+  Exact runs bind analyst, ledger, hook, chat, and policy identities with non-secret configuration digests.
+  They use the registry's shared serial execution path and persist canonical equal or weighted allocations instead of adding a second scheduler or allocator.
+  Exact receipts explicitly distinguish complete execution from a failed ordered prefix.
+  `ExactAnalystRunExecutionError.result` is always a canonical immutable failed receipt with valid completed work and accounting.
+- `defineTraceAnalyst()` accepts canonical `executionConfig` and returns an exact-capable custom analyst when it is supplied, while preserving its existing minimal form.
+- **Breaking:** `AnalystBenchmarkCase` now requires `clusterId` and `labelState`.
+  A case must identify its independent source unit and state whether labels prove an issue, prove no issue, or leave the outcome unknown.
+  The benchmark no longer guesses either field from an empty issue list.
+- `agent-eval analyst-benchmark` compares an empty baseline with a real-model AgentRx or CodeTraceBench analyst through any OpenAI-compatible endpoint.
+  It requires an explicit case limit and immutable dataset revision, validates labeled spans before paid work, uses benchmark-specific output adapters, and writes complete JSON plus Markdown results.
+  CodeTraceBench cases also retain hashed final verification artifacts, parse known upstream result formats, and mark missing outcomes unavailable.
+  Limited hash samples report source-versus-selected class, agent, model, difficulty, and solved distributions without claiming representativeness.
+  The published all-row score is retained, while a calibrated view measures solved label-empty trajectories as trusted negatives and keeps failed or unknown label-empty trajectories unlabeled.
+  Interrupted runs persist hash-chained observations and resume only when public inputs, model settings, local paths, and endpoint still match.
+  Reports include pooled and per-case step-localization metrics, exact source-quote coverage, final-result availability, and imported runtime duration.
+  External runner failures retain reported token, cost, duration, and metadata instead of becoming telemetry gaps.
+  Public model runs use one structured model call over a bounded trace projection.
+  A durable run-wide cost ledger enforces `--max-cost-usd` across concurrency and resume.
+  Paid responses are cached under deterministic call ids before settlement, so resume neither loses a completed response nor creates a second reservation after interruption.
+  Completed results retain a digest of every behavior-defining source file and are read through one strict recursive schema.
+  The repository includes one pinned 32-case CodeTraceBench input, two complete 64-call GLM-5.2 Agent Eval runs, and a failure-inclusive run of pinned CodeTracer on the same trajectories.
+  Exact source, input, result, resume, usage, cost, and secret-scan checks are committed with the results.
+- The Python package gains a `dspy` extra: `agent-eval-rpc[dspy]` runs official `dspy.RLM` in a sandboxed Deno/Pyodide child process with seven allowlisted trace tools and strict JSON I/O.
+
+### Changed
+
+- **Breaking:** trace analysts are recursive research programs run through an explicit analysis engine.
+  `analyzeTraces` requires an `engine` (the DSPy RLM engine is the primary implementation) and reports engine iterations; `maxTurns`, `maxSubqueries`, `onTurn`, and `AnalyzeTracesTurnSnapshot` are gone.
+  `callLlmJson` remains only as the one-shot `direct` benchmark baseline.
+- **Breaking:** `defineTraceAnalyst()` returns an inert `TraceAnalystDefinition` for registration instead of a registrable analyst, and no longer takes a `cost` declaration — analyst cost is always metered LLM usage.
+  `createTraceAnalystKind` is now `createTraceAnalyst`; `TraceAnalystKindSpec` and `CreateTraceAnalystKindOpts` are replaced by `TraceAnalystDefinition`.
+- **Breaking:** `BuildTraceAnalystSurfaceDispatchOptions.analyze` receives `instructions` instead of `actorDescription`.
+- **Breaking:** `SteeringOptimizerBackend` narrows to `'pairwise'`.
+- Citation verification is store-backed: cited trace and span ids must resolve in the trace analysis store, and encoded or foreign ids are rejected.
+- External-optimizer subprocess calls fail on HTTP 200 responses with zero input and output usage, and on output over-reservation after recording the actual charge.
+- Relative external-optimizer runner commands resolve against the caller's working directory instead of the child's temporary directory.
+
+### Removed
+
+- **Breaking:** the Ax analyst stack and the `@ax-llm/ax` dependency: `createAnalystAi`, `CreateAnalystAiConfig`, `structureFindings`, `StructureFindingsOptions`, `StructureFindingsResult`, `AxGepaSteeringOptimizer`, and `AxSteeringOptimizerConfig`.
+- **Breaking:** `createPublicBenchmarkModelRunner`; the analyst benchmark CLI defaults to the DSPy RLM runner and keeps the one-shot runner as the explicit `direct` baseline.
+- `buildTraceAnalystTools`; trace tools are built from the transport-neutral descriptors.
+
+### Fixed
+
+- Canonical `trace://<trace>/span/<span>` evidence is classified as span evidence instead of artifact evidence.
+- Public model output selects positive integer assistant step ids.
+  The runner builds canonical trace URIs and exact action excerpts from those spans, and rejects missing, non-assistant, or empty steps.
+- Capped concurrent paid calls wait for active reservations to settle when their final spend may still fit.
+  They fail immediately only when committed spend plus the next enforced maximum exceeds the run limit.
+- Releasing a single-run lock now removes its process exit listener instead of leaking one callback per completed campaign.
+- Phoenix evaluator tests use OpenTelemetry Core 2.10 instead of the vulnerable 1.x transitive dependency.
+- CodeTracer prediction adapters accept the published schema and both flat and grouped step-label outputs emitted by CodeTracer 0.2.
+- The CodeTraceBench model prompt now matches the public incorrect-step task by scoring wrong actions that are later recovered, instead of treating final task success as proof that earlier steps were correct.
+- JSON-text finding rows reach the existing per-row schema repair instead of failing the entire trace analyst response.
+
 ## [0.137.0] - 2026-07-29 - trace analyst measurement and review integrity
 
 ### Added

package/README.md

@@ -357,6 +357,40 @@ pnpm tsx examples/selfimprove-quickstart/index.ts
 You do not need a runnable agent to analyze data you already captured.
 Use `analyzeRuns()` for `RunRecord[]`.
 For traces, run a registry of built-in or custom analysts, measure it on labeled issues and exact span locations, then turn only reviewed findings into eval data.
+For a public quality check, convert CodeTraceBench with `traces import-codetracebench`, then run `agent-eval analyst-benchmark` against pinned labels.
+The command compares an empty baseline with the official DSPy `RLM` trace analyst and records its trace reads, model calls, tokens, cost, runtime, and cited findings.
+
+Use `AnalystRegistry.runExact()` when the caller, rather than registry defaults, must own every execution choice.
+The ordered `analystIds` array is the execution order, and `null` explicitly disables optional budget, timeout, cancellation, cost, tag, or prior-finding channels.
+Exact runs are serial; callers that need recursive or concurrent scheduling compose them through their runtime rather than adding a second scheduler here.
+
+```ts
+const result = await registry.runExact('analysis-1', inputs, {
+  analystIds: ['failure-mode', 'improvement'],
+  budget: { kind: 'equal', totalUsd: 2 },
+  totalTimeoutMs: 30_000,
+  signal: null,
+  costLedger: null,
+  costLedgerIdentity: null,
+  costPhase: null,
+  tags: null,
+  priorFindings: null,
+  chainFindings: true,
+  missingInputMode: 'abort',
+  applyRegistryHooks: false,
+  useRegistryChat: false,
+})
+```
+
+Custom analysts passed to `runExact()` declare canonical `executionConfig`.
+The same `defineTraceAnalyst()` helper returns an exact-capable analyst when that field is present.
+Built-in analysts already declare it.
+Trace analysts selected by `runExact()` also require `aiIdentity`, using the same non-secret `id`, `version`, and canonical `config` shape as cost ledgers, registry hooks, and registry chat clients.
+Exact lifecycle hooks receive frozen snapshots for observation; they cannot rewrite the planned context.
+Persisted results store configuration digests, not raw configuration.
+The persisted plan records the exact equal or weighted allocation for every routed analyst, and archival validates summaries against that same plan.
+Every exact receipt says whether it is `complete` or `failed`; a complete receipt must cover the full plan, while a failed receipt may contain only the executed prefix.
+Any failure after an exact run starts rejects with `ExactAnalystRunExecutionError`; its immutable failed receipt preserves valid completed summaries, findings, usage, and cost.
 
 See [concepts](./docs/concepts.md), [customer paths](./docs/customer-journeys.md), and [trace analysis](./docs/trace-analysis.md).