@tangle-network/agent-eval 0.71.0 → 0.72.3

package/CHANGELOG.md

@@ -4,6 +4,69 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-
 
 ---
 
+## [0.72.3] — 2026-06-01 — workflow trace hardening and driver backtests
+
+### Added
+
+- **Canonical workflow branch events in `/workflow`.** Runtime traces now project branch start/end/failure counts into workflow summaries, RunRecords, and feedback trajectories so fanout topology failures are measurable instead of hidden in raw trace blobs.
+- **`workflowPhaseGraph` in `/workflow`.** Builds phase nodes and branch edges from workflow trace events with per-phase calls, branch failures, cost, and token counters. Product adopters can consume this instead of maintaining local graph mirrors.
+- **Stricter workflow event schema validation.** Workflow traces now reject unknown event kinds, malformed typed payloads, non-monotonic timestamps, missing `workflow.started`, multiple terminal events, and events after terminal completion.
+- **Driver comparison substrate proof.** `compareDrivers` now carries analyst findings through the canonical campaign path and includes GSM8K/AppWorld driver backtest examples.
+
+### Fixed
+
+- **Publish skew guard.** PyPI publishing depends on successful npm publishing, and the npm publish job now checks registry authentication and `@tangle-network` package access before building or attempting a publish.
+
+---
+
+## [0.72.2] — 2026-06-01 — workflow driver promotion gates
+
+### Added
+
+- **`decideWorkflowDriverPromotion` in `/workflow`.** Compares a dynamic workflow driver against the reviewer-loop baseline using paired heldout `RunRecord`s keyed by `scenarioId::seed`, then fails closed on missing pairs, too few pairs, insufficient lift, or candidate cost ceilings.
+- **Explicit workflow comparison axis.** `expectedScenarioIds` defines the promotion gate's comparison set so unrelated scenarios cannot skew the lift or confidence interval.
+
+### Fixed
+
+- **No seed-only workflow pairing.** Promotion records without `scenarioId` are rejected instead of being paired by seed alone.
+
+---
+
+## [0.72.1] — 2026-06-01 — workflow execution summaries for dynamic drivers
+
+### Added
+
+- **`summarizeWorkflowExecution` in `/workflow`.** Builds the canonical rich projection from a workflow trace: event-kind counts, phase order, agent and loop delegate summaries, verifier/analyst/reviewer checkpoint outputs, cost, tokens, and failure status.
+- **Checkpoint output extraction.** Verifier, analyst, and reviewer traces preserve the returned output through `trace.checkpointOutput`, with `trace.output` accepted for compatibility.
+
+### Fixed
+
+- **npm/PyPI version lock.** The Python RPC package version is bumped back into lockstep with the npm package so the publish workflow can release both artifacts from one tag.
+
+## [0.72.0] — 2026-05-31 — cost axis prices unpriced-at-source models (every run carries a real, labeled cost)
+
+A live tax-agent full-loop run (real sandbox, `deepseek-v4-pro`, real tokens) exposed the second root of the cost-ledger split: the sandbox reported `totalCostUsd: 0` despite `17537` input / `622` output tokens — not a stub, not a mis-wired ledger, but a model the **source** can't rate. The cost / Pareto / `tokens_per_dollar` axes blanked even though the substrate's pricing table prices `deepseek` correctly; the table was simply never consulted on the matrix cost projection. A $0 cost on a run that burned real tokens reads as "free," which is the more misleading state.
+
+### Fixed
+
+- **`runProfileMatrix` prices measured tokens when the source reports $0.** Cost precedence is now explicit: **source-billed > token-estimated > none**. When `cell.costUsd === 0` and real output tokens flowed and the model is priced (`isModelPriced`), `buildRunRecord` sets the cost from `estimateCost(in, out, model)` (real published rate × real tokens) and stamps `raw.cost_estimated = 1`. A billed cost is never overridden; a model the table also can't rate stays $0 (no fabrication). The estimate flows into `record.costUsd`, so `byProfile.totalCostUsd`, `integrity.totalCostUsd`, and `tokens_per_dollar` / `cost_per_quality` all populate.
+- **Every cost surface in the matrix result agrees.** The embedded `campaigns[id].aggregates.totalCostUsd` is reconciled to the priced total instead of runCampaign's raw `ctx.cost` ledger (which only sees the source's $0). No more two-`totalCostUsd`-that-disagree in one result.
+- **Honest integrity diagnosis.** `summarizeBackendIntegrity`'s uncosted-records message now names **both** roots — mis-wired ledger OR unpriced-at-source model — and points at `estimateCost` for the latter, instead of asserting the ledger is broken.
+
+Live proof: the same tax case that recorded `$0` now records **`$0.0059453`** (`17537 × 0.0003/1k + 622 × 0.0011/1k`, exact), `cost_estimated: 1`, `uncostedRecords: 0`, verdict `real`. Generalizes to every consumer of `runProfileMatrix`. New regression tests: priced-when-source-zero, billed-takes-precedence, truly-unpriced-stays-$0, campaign-aggregate-reconciled. Full suite (1663) green.
+
+## [0.71.0] — 2026-05-31 — corpus-by-default + multi-dimensional capture (datasets as eval exhaust)
+
+Every matrix run now emits a multi-dimensional, dataset-able record with no side-channel — the groundwork for "datasets gathered for free by running evals."
+
+### Added
+
+- **Multi-dim guardrail projection in `buildRunRecord`.** Each `RunRecord.outcome.raw` carries `cost_usd`, `tokens_input` / `tokens_output` (+ `tokens_cached` when present), `latency_ms`, and the guarded ratios `tokens_per_dollar` / `cost_per_quality`. RAW-ONLY — the composite stays the judge objective (anti-Goodhart); these are tracked + dashboarded + carried into datasets, never optimized.
+- **Corpus-by-default via `corpusText`.** An optional `corpusText(artifact, scenario) => {prompt, completion}` stamps the trajectory text onto each record (the `CorpusRecord` shape), so a run is dataset-able with no side-channel. Fail-soft: a throwing extractor omits the text and keeps the graded record.
+- **`appendToCorpus` / `readCorpus` / `buildDatasetFromCorpus`** (`src/rl/corpus.ts`) — append-only JSONL corpus (deduped by `runId`), with score/split filtering into a train/holdout dataset.
+
+`buildRunRecord` is generic over `<TScenario, TArtifact>`; a `scenarioById` map threads each scenario into the projection.
+
 ## [0.70.0] — 2026-05-31 — error-grounded reflection (the driver targets real failures, not blind rewrites)
 
 Adversarial verification on TWO domains (legal + tax, two worker models) found the same root cause: the gepaDriver's candidates **regressed** the baseline, so the gate correctly held — but nothing improved. The driver was reflecting on per-scenario *scores* only; the judge's `notes` (the "why it failed") were computed but **dropped** before the reflection. So it proposed generic rewrites a capable model already knows, which distract rather than help.

package/dist/adapters/http.d.ts

@@ -1,4 +1,4 @@
-import { S as Scenario, D as DispatchFn, b as DispatchContext } from '../types-CnmZ2bkP.js';
+import { S as Scenario, D as DispatchFn, b as DispatchContext } from '../types-Bba0vl1V.js';
 import '../run-record-BgTFzO2r.js';
 import '../errors-Dwqw-T_m.js';
 import '../schema-m0gsnbt3.js';

package/dist/adapters/langchain.d.ts

@@ -1,4 +1,4 @@
-import { S as Scenario, J as JudgeScore, D as DispatchFn, a as JudgeConfig } from '../types-CnmZ2bkP.js';
+import { S as Scenario, J as JudgeScore, D as DispatchFn, a as JudgeConfig } from '../types-Bba0vl1V.js';
 import '../run-record-BgTFzO2r.js';
 import '../errors-Dwqw-T_m.js';
 import '../schema-m0gsnbt3.js';

package/dist/adapters/otel.d.ts

@@ -1,8 +1,9 @@
-import { T as TraceSpanEvent, H as HostedClient } from '../index-BGBrVS24.js';
-import '../types-CnmZ2bkP.js';
+import { TraceSpanEvent, HostedClient } from '../hosted/index.js';
+import '../types-Bba0vl1V.js';
 import '../run-record-BgTFzO2r.js';
 import '../errors-Dwqw-T_m.js';
 import '../schema-m0gsnbt3.js';
+import '../insight-report-Df3lxYXM.js';
 import '../summary-report-ByiOUrHj.js';
 import '../failure-cluster-CL7IVgkJ.js';
 import '../store-CKUAgsJz.js';

package/dist/agent-profile-DYRboYWu.d.ts

@@ -0,0 +1,364 @@
+import { A as AgentEvalError } from './errors-Dwqw-T_m.js';
+import { R as RunRecord } from './run-record-BgTFzO2r.js';
+import { TCloud } from '@tangle-network/tcloud';
+
+/**
+ * Backend-integrity guard: distinguish "agent failed" from "eval ran against
+ * a stub / unconfigured backend." Without this guard a canonical eval can
+ * silently report `0/N passed` and look like an agent-quality problem when
+ * the LLM was never actually called — the failure mode we just hit running
+ * the 4-vertical parallel eval (legal-sandbox-stub returned hard-coded 33-104
+ * char strings; gtm/creative defaulted to a cli-bridge that wasn't running).
+ *
+ * The shape:
+ *
+ *   const report = summarizeBackendIntegrity(records)
+ *   assertRealBackend(records)   // throws BackendIntegrityError if 100% stub
+ *
+ * A record is "stub-mode" if its `tokenUsage.input === 0 && tokenUsage.output === 0`.
+ * (`costUsd` alone is unreliable — some backends successfully call LLMs but
+ *  don't propagate pricing, producing real tokens with $0 cost.)
+ *
+ * Verdicts:
+ *   - `real`   — at least one record has nonzero token usage
+ *   - `stub`   — every record is stub-mode (eval ran blind)
+ *   - `mixed`  — some records real, some stub (partial backend failure;
+ *                often the 429-cascade or auth-half-failed case)
+ */
+
+interface BackendIntegrityReport {
+    /** Total records inspected. */
+    totalRecords: number;
+    /** Records with input=0 AND output=0 (a stub fingerprint). */
+    stubRecords: number;
+    /** Records with nonzero token usage (real LLM activity). */
+    realRecords: number;
+    /** Records where output>0 but costUsd=0 (real LLM, broken cost ledger). */
+    uncostedRecords: number;
+    /** Sum of input tokens across all records. */
+    totalInputTokens: number;
+    /** Sum of output tokens across all records. */
+    totalOutputTokens: number;
+    /** Sum of costUsd across all records. */
+    totalCostUsd: number;
+    /** Worst-case integrity verdict. */
+    verdict: 'real' | 'mixed' | 'stub';
+    /** Human-readable diagnosis suitable for terminal output. */
+    diagnosis: string;
+}
+/**
+ * Error thrown when an integrity assertion fails. Caller can pattern-match
+ * by `code === 'AGENT_EVAL_BACKEND_STUB'` to differentiate from other
+ * errors.
+ */
+declare class BackendIntegrityError extends AgentEvalError {
+    readonly report: BackendIntegrityReport;
+    constructor(message: string, report: BackendIntegrityReport);
+}
+/**
+ * Inspect a batch of RunRecords and return an integrity report. Pure
+ * function — no I/O, no logging. The caller decides what to do with the
+ * verdict (print warning, throw, gate CI, etc.).
+ */
+declare function summarizeBackendIntegrity(records: ReadonlyArray<RunRecord>): BackendIntegrityReport;
+/**
+ * Throw BackendIntegrityError if the verdict is 'stub' — i.e. every record
+ * shows zero LLM activity. Non-strict callers can pass `{ allowMixed: false }`
+ * to also reject mixed verdicts (recommended for CI gates).
+ *
+ * Real backends pass through silently.
+ */
+declare function assertRealBackend(records: ReadonlyArray<RunRecord>, opts?: {
+    allowMixed?: boolean;
+}): BackendIntegrityReport;
+
+/**
+ * Artifact validators.
+ *
+ * Generic "score a produced artifact" primitive. Tax uses it for PDF form
+ * correctness, research for sourced briefs, browser for task assertions, coding
+ * for social posts. One interface, many validators; all plug into
+ * `BenchmarkRunner` the same way.
+ *
+ * A validator receives an `Artifact` (file on disk, JSON blob, text, binary)
+ * plus a `ValidationContext` (scenario id, the turns that produced it) and
+ * returns a `ValidationResult` with pass/fail + 0..1 score + structured
+ * issues.
+ */
+interface Artifact {
+    /** Logical kind — validators type-guard on this */
+    kind: 'file' | 'json' | 'text' | 'binary' | string;
+    /** Filesystem-style path, optional */
+    path?: string;
+    /** String content for text/json/file kinds */
+    content?: string;
+    /** Binary content (if kind === 'binary') */
+    bytes?: Uint8Array;
+    /** Caller-supplied metadata (mimeType, sha256, size, etc.) */
+    metadata?: Record<string, unknown>;
+}
+interface ValidationContext {
+    scenarioId: string;
+    turnIndex?: number;
+    /** Prior artifacts for multi-artifact scenarios */
+    priorArtifacts?: Artifact[];
+    /** Free-form hints the validator uses for domain-specific checks */
+    hints?: Record<string, unknown>;
+}
+interface ValidationIssue {
+    severity: 'error' | 'warning' | 'info';
+    message: string;
+    /** Optional path into the artifact (e.g. JSON path or byte offset) */
+    locus?: string;
+}
+interface ValidationResult {
+    pass: boolean;
+    /** 0–1 normalized score. Validators should be monotonic in pass-ness. */
+    score: number;
+    issues: ValidationIssue[];
+    /** Diagnostic payload for reporters */
+    evidence?: Record<string, unknown>;
+}
+interface ArtifactValidator {
+    /** Stable identifier for the validator; appears in reports. */
+    name: string;
+    /** Optional description for human-facing reports. */
+    description?: string;
+    /** Called once per artifact; validators are expected to be pure + idempotent. */
+    validate(artifact: Artifact, context: ValidationContext): Promise<ValidationResult>;
+}
+/**
+ * Run every validator on the same artifact; aggregate pass as AND, score as
+ * (weighted) mean, issues concatenated. Weights default to 1 each.
+ */
+declare function composeValidators(validators: ArtifactValidator[], options?: {
+    name?: string;
+    weights?: number[];
+}): ArtifactValidator;
+/** Pass if the artifact body matches a provided regex. */
+declare function regexMatch(name: string, pattern: RegExp): ArtifactValidator;
+/** Pass if JSON parses and every required key is present. */
+declare function jsonHasKeys(name: string, requiredPaths: string[]): ArtifactValidator;
+/** Pass if min ≤ byte length ≤ max. */
+declare function byteLengthRange(name: string, min: number, max: number): ArtifactValidator;
+/** Pass if the artifact contains every required substring (case-insensitive by default). */
+declare function containsAll(name: string, required: string[], options?: {
+    caseSensitive?: boolean;
+}): ArtifactValidator;
+
+/**
+ * Completion verifier — the task-completion oracle.
+ *
+ * Answers the only eval question that is not a proxy: did the agent actually
+ * COMPLETE the task — produce every required deliverable, persisted and
+ * correct — rather than describe what should be done. A fluent transcript
+ * that never produces the artifact scores zero here.
+ *
+ * Per requirement, a two-stage check:
+ *   1. Structural — a produced item (vault artifact / approved proposal /
+ *      tool call) of the right kind is matched against the requirement and
+ *      carries non-empty content. Deterministic; no LLM.
+ *   2. Correctness — only if structurally present AND the matched item
+ *      carries content, one targeted check decides whether that item
+ *      actually fulfils the requirement. A hallucinated artifact fails here;
+ *      an absent one already failed stage 1.
+ *
+ * `completionRate` is satisfied / total. Quality dimensions are meaningless
+ * on an incomplete task — callers gate on `fullyComplete` / `completionRate`
+ * before scoring quality.
+ */
+
+/** What kind of produced state can satisfy a requirement structurally. */
+type SatisfiedBy = 'artifact' | 'proposal' | 'tool-call' | 'any';
+interface CompletionRequirement {
+    /** Stable id from the task gold (e.g. a persona's `expected_requirements[].req_id`). */
+    reqId: string;
+    /** Human-readable description of the required deliverable. */
+    title: string;
+    /** Optional kind/category hint, matched against a produced item's kind. */
+    category?: string;
+    /** What produced state satisfies this requirement. Defaults to 'any'. */
+    satisfiedBy?: SatisfiedBy;
+}
+interface TaskGold {
+    taskId: string;
+    requirements: CompletionRequirement[];
+}
+interface ProducedProposal {
+    id: string;
+    title: string;
+    status: 'pending' | 'approved' | 'rejected';
+    /** Optional persisted body — when present, enables a correctness check. */
+    content?: string;
+}
+/** Everything observable about what a run actually produced. */
+interface ProducedState {
+    /** Persisted vault artifacts. Reuses the shared `Artifact` shape. */
+    artifacts: Artifact[];
+    /** Proposals / filings the agent created. */
+    proposals: ProducedProposal[];
+    /** Names of tools the agent invoked. */
+    toolCalls: string[];
+}
+interface RequirementCheck {
+    reqId: string;
+    title: string;
+    /** A produced item of the right kind matched the requirement, non-empty. */
+    structurallyPresent: boolean;
+    /**
+     * Whether the matched item actually fulfils the requirement. `null` when
+     * not structurally present, or when the matched item carries no content
+     * to assess.
+     */
+    correct: boolean | null;
+    /** structurallyPresent && correct !== false. */
+    satisfied: boolean;
+    /** Human-readable evidence for the verdict. */
+    evidence: string[];
+}
+interface CompletionVerdict {
+    taskId: string;
+    requirements: RequirementCheck[];
+    /** satisfied / total requirements. */
+    completionRate: number;
+    /** Every requirement satisfied. */
+    fullyComplete: boolean;
+}
+/**
+ * Decides whether a produced item's content actually fulfils a requirement.
+ * Injected so the structural verifier stays pure and unit-testable; the
+ * production implementation is `createLlmCorrectnessChecker`.
+ */
+type CorrectnessChecker = (requirement: CompletionRequirement, content: string) => Promise<{
+    correct: boolean;
+    reason: string;
+}>;
+/**
+ * Verify whether a run completed the task. `checkCorrectness` is injected —
+ * `createLlmCorrectnessChecker` for production, a deterministic stub in tests.
+ *
+ * Throws on a gold spec with no requirements: an eval task that requires
+ * nothing is a misconfiguration, not a vacuously-complete task.
+ */
+declare function verifyCompletion(gold: TaskGold, state: ProducedState, checkCorrectness: CorrectnessChecker): Promise<CompletionVerdict>;
+interface LlmCorrectnessCheckerOpts {
+    model?: string;
+    /** Max chars of artifact content sent to the checker. */
+    maxContentChars?: number;
+}
+/** Parse the correctness checker's model response. Fails loud on a bad shape. */
+declare function parseCorrectnessResponse(raw: string): {
+    correct: boolean;
+    reason: string;
+};
+/**
+ * Production `CorrectnessChecker` — one LLM call per matched artifact,
+ * deterministic (temperature 0), structured JSON out. Judges fulfilment
+ * only: a plan, a gesture, or a description of what should be done does not
+ * fulfil a requirement — the artifact must BE the deliverable.
+ */
+declare function createLlmCorrectnessChecker(tc: TCloud, opts?: LlmCorrectnessCheckerOpts): CorrectnessChecker;
+
+/**
+ * Produced-state extraction — normalize a run's runtime event stream into the
+ * typed `ProducedState` the completion oracle consumes.
+ *
+ * `ProducedState` answers "what did the agent actually produce" — vault
+ * artifacts, proposals, tool calls. The runtime emits these as a stream of
+ * events; this module is the single normalization point from that stream to
+ * the shape `verifyCompletion` expects.
+ *
+ * Input is structurally typed (`RuntimeEventLike`) so this module does not
+ * depend on agent-runtime — agent-runtime's `RuntimeStreamEvent` satisfies it
+ * structurally. The `content` on `ArtifactEventLike` and the whole
+ * `proposal_created` variant are the runtime-side enrichments this contract
+ * requires; the runtime emits them, this module consumes them.
+ */
+
+/** A tool the agent invoked. */
+interface ToolCallEventLike {
+    type: 'tool_call';
+    toolName: string;
+}
+/**
+ * An artifact the agent produced. `content` is the enriched field — the
+ * runtime's base `artifact` event carries only metadata; the completion
+ * oracle needs the body to verify the deliverable, so the runtime emits it.
+ */
+interface ArtifactEventLike {
+    type: 'artifact';
+    artifactId: string;
+    name?: string;
+    mimeType?: string;
+    uri?: string;
+    content?: string;
+}
+/** A proposal / filing the agent created. */
+interface ProposalEventLike {
+    type: 'proposal_created';
+    proposalId: string;
+    title: string;
+    status?: 'pending' | 'approved' | 'rejected';
+}
+/**
+ * The subset of runtime stream events `extractProducedState` consumes.
+ * agent-runtime's full `RuntimeStreamEvent` union satisfies this structurally;
+ * the `{ type: string }` catch-all keeps the input permissive so callers can
+ * pass the whole unfiltered telemetry stream — unrecognized events are skipped.
+ */
+type RuntimeEventLike = ToolCallEventLike | ArtifactEventLike | ProposalEventLike | {
+    type: string;
+};
+/**
+ * Normalize a run's runtime event stream into `ProducedState`.
+ *
+ * Pure and total — unrecognized event types are skipped. `toolCalls` is
+ * deduplicated by name in first-seen order (completion cares about a tool's
+ * presence, not its call count). An artifact with neither a name nor a uri
+ * still yields an entry keyed by its `artifactId` so it is never silently
+ * dropped; an artifact with no `content` yields empty content, which the
+ * completion oracle's structural check then rejects on its own.
+ */
+declare function extractProducedState(events: readonly RuntimeEventLike[]): ProducedState;
+
+/**
+ * @stable
+ *
+ * AgentProfile — the eval harness's unit of variation.
+ *
+ * A profile pins everything that changes agent behaviour for a benchmark
+ * cell: the model, the active skills, the prompt version, the available
+ * tools. Vary the profile — swap a model, add a skill — and re-run the suite
+ * to benchmark the change. The scorecard keys a cell on
+ * `(scenarioId, profileHash)`, so the model is not a separate axis: it lives
+ * inside the profile, and two profiles with the same model but different
+ * skills are different cells.
+ *
+ * `agentProfileHash` is the profile's behaviour identity. Two profiles that
+ * produce the same agent behaviour share a hash (and a scorecard cell);
+ * reordering `skills` or `tools` does not change it; the human-facing `id`
+ * label does not affect it.
+ */
+interface AgentProfile {
+    /** Human-facing label, e.g. `sonnet-legal-skills-v3`. Not part of the hash. */
+    id: string;
+    /** Model snapshot id this profile pins, e.g. `claude-sonnet-4-6@2025-04-15`. */
+    model: string;
+    /** Skill ids/versions active in this profile — the primary behaviour lever. */
+    skills?: string[];
+    /** Prompt version identifier. */
+    promptVersion?: string;
+    /** Tool ids available to the agent. */
+    tools?: string[];
+    /** Any other behaviour-bearing knobs that should fingerprint into the hash. */
+    metadata?: Record<string, string | number | boolean>;
+}
+/**
+ * Deterministic behaviour identity of a profile — a sha256 over the
+ * behaviour-bearing fields. `skills` and `tools` are order-insensitive; the
+ * `id` label is excluded. Throws on a profile with no `model` — an unkeyable
+ * profile must fail loud rather than collapse into a blank-model cell.
+ */
+declare function agentProfileHash(profile: AgentProfile): string;
+
+export { type AgentProfile as A, type BackendIntegrityReport as B, type CompletionRequirement as C, type LlmCorrectnessCheckerOpts as L, type ProducedState as P, type RuntimeEventLike as R, type SatisfiedBy as S, type TaskGold as T, type ValidationContext as V, type CompletionVerdict as a, type CorrectnessChecker as b, type Artifact as c, type ArtifactEventLike as d, type ArtifactValidator as e, BackendIntegrityError as f, type ProducedProposal as g, type ProposalEventLike as h, type RequirementCheck as i, type ToolCallEventLike as j, type ValidationIssue as k, type ValidationResult as l, agentProfileHash as m, assertRealBackend as n, byteLengthRange as o, composeValidators as p, containsAll as q, createLlmCorrectnessChecker as r, extractProducedState as s, jsonHasKeys as t, parseCorrectnessResponse as u, regexMatch as v, summarizeBackendIntegrity as w, verifyCompletion as x };