@tangle-network/agent-eval 0.137.0 → 0.139.0

package/docs/trace-analysis.md

@@ -1,438 +1,244 @@
 # Trace Analysis
 
-Trace analysis answers three different questions:
+A trace analyst is a recursive research program that answers a question by choosing which trace data to inspect.
+It is not a single prompt over a prebuilt trace dump.
 
-1. What happened in the run?
-2. Which exact steps support a suspected problem?
-3. Does an analyst find labeled problems reliably enough to use?
+Agent Eval separates five concerns:
 
-Keep those answers separate.
-A generated finding is a review request, not training truth.
+| Part | Responsibility |
+|---|---|
+| Analyst definition | The question, investigation policy, allowed tools, and limits |
+| Analysis engine | Runs the recursive investigation |
+| Trace store | Provides bounded reads over OTLP traces |
+| Finding | Records one structured claim with exact evidence |
+| Analysis result | Returns the prose answer, findings, investigation steps, model calls, tool calls, and runtime |
 
-## Run The Built-In Analysts
+The built-in model-backed analysts use the official DSPy `RLM`.
+DSPy runs the research loop and a sandboxed Python interpreter.
+Agent Eval owns trace access, credentials, cancellation, cost accounting, and output validation.
 
-The default registry always includes deterministic checks.
-Model-assisted analysts are added only when you provide a model client.
+`callLlmJson()` and `createPublicBenchmarkDirectRunner()` are direct-call baselines.
+They are not trace analysts.
 
-```ts
-import {
-  buildDefaultAnalystRegistry,
-} from '@tangle-network/agent-eval/analyst'
-import { OtlpFileTraceStore } from '@tangle-network/agent-eval/traces'
-
-const traceStore = new OtlpFileTraceStore({ path: 'traces.otlp.jsonl' })
-const analysts = buildDefaultAnalystRegistry()
+## Install
 
-const result = await analysts.run('release-42', { traceStore })
+Install the TypeScript package and the matching Python package with DSPy:
 
-for (const finding of result.findings) {
-  console.log(finding.claim, finding.evidence_refs)
-}
+```sh
+pnpm add @tangle-network/agent-eval
+python -m venv .venv
+.venv/bin/pip install "agent-eval-rpc[dspy]"
 ```
 
-Use `result.per_analyst` to inspect failures, latency, calls, tokens, and cost.
-An analyst failure is recorded separately from an agent failure.
+The Python extra pins the tested stable DSPy and Deno versions.
 
-Products can implement `TraceAnalysisStore`; they do not need to use the file store in production.
-Custom stores provide `hasTrace` and batched `hasSpans` alongside the seven reads, and accept a `TraceAnalysisStoreContext` so cancellation reaches storage and scans.
-The binding validates every custom-store result.
-Missing fields, undeclared fields, inconsistent counts, and false continuation flags throw `TraceAnalysisStoreContractError` with code `backend_integrity`.
-The analyst runs one Ax executor loop and accepts only an explicit structured `final(task, { report, findings })` result; max-turn fallback text fails loud.
-
-### Bind the same reads into another agent environment
-
-`buildTraceAnalysisToolDescriptors()` is the canonical definition of the analyst's seven bounded read operations and does not expose Ax types.
-Each descriptor carries the stable `traces` namespace, function name, description, JSON input schema in `parameters`, and a handler already bound to the supplied `TraceAnalysisStore`.
-`buildTraceAnalystTools()` adapts those descriptors into Ax functions; it does not define a second tool surface.
-The bound handlers wrap custom stores with `createBoundedTraceAnalysisStore()`, so page limits, byte ceilings, not-found errors, and cancellation do not depend on the transport or adapter.
+## Answer One Question
 
 ```ts
 import {
-  buildTraceAnalysisToolDescriptors,
-  type TraceAnalysisStore,
-} from '@tangle-network/agent-eval/traces'
-
-declare const store: TraceAnalysisStore
-declare function qualifyToolName(namespace: string, name: string): string
-
-const tools = buildTraceAnalysisToolDescriptors({ store }).map(
-  ({ namespace, name, description, parameters, handler }) => ({
-    name: qualifyToolName(namespace, name),
-    description,
-    inputSchema: parameters,
-    handler,
-  }),
-)
-```
-
-Map these fields into the host's existing tool transport.
-The host owns namespace encoding; use its existing convention instead of inventing one here.
-Do not copy the schemas or reimplement the handlers in an MCP, Runtime, or provider adapter.
-
-`queryTraces.limit`, `viewSpans.span_ids`, and search `max_matches` caps are present in the JSON Schemas and enforced before store calls.
-Invalid arguments throw `TraceAnalysisValidationError` with code `validation`; responses that cannot fit their byte ceiling throw `TraceAnalysisLimitError` with code `limit_exceeded`.
-Search patterns use RE2 syntax, which rejects backreferences and lookaround instead of allowing exponential-time expressions.
-Search results return `hits` and an exact `has_more` flag; they do not invent a total after a capped scan.
-`viewSpans` partitions every requested id across `spans`, `missing_span_ids`, and `omitted_span_ids`; `has_more` is true when omitted ids must be retried.
-Attribute and match text shortening includes a deterministic marker.
-Trace pages set `has_more`, and the overview returns every error cluster or fails explicitly when the configured response limit is too small.
-
-### Analyze captured tool spans in memory
-
-Use `toolSpansToTraceAnalysisStore()` when a live worker already returns canonical `ToolSpan[]` records.
-The function snapshots the records immediately, groups them by `runId`, and exposes the same bounded reads and searches as the file-backed store.
-
-```ts
+  createDspyRlmTraceEngine,
+} from '@tangle-network/agent-eval/analyst'
 import {
   analyzeTraces,
-  toolSpansToTraceAnalysisStore,
-  type ToolSpan,
-  ToolTraceMissingError,
 } from '@tangle-network/agent-eval/traces'
 
-declare const capturedToolSpans: ToolSpan[] | undefined
-
-if (!capturedToolSpans?.length) throw new ToolTraceMissingError()
-const source = toolSpansToTraceAnalysisStore(capturedToolSpans)
-const result = await analyzeTraces({ question: 'Why are tools failing?' }, { source, ai, model })
-```
-
-`undefined`, `null`, and an empty array throw `ToolTraceMissingError` with code `capture_integrity`.
-An empty tool list cannot distinguish a real tool-free run from broken capture, so the adapter never reports it as a clean trace set.
-Use a complete OTLP trace source when proving that a run executed successfully without tools.
-
-## Deterministic failure coverage (no LLM)
-
-Before (or alongside) the LLM analyst, `OtlpFileTraceStore.getOverview()` returns a
-`DatasetOverview` whose `error_clusters` are computed deterministically: error
-spans are grouped by a normalized failure signature (uuids / hex ids / numbers /
-absolute paths / durations collapsed), each cluster carrying its prevalence,
-exemplar `trace_id`/`span_id`, and a verbatim sample. This is a zero-LLM,
-reproducible failure checklist the analyst then explains and closes:
-
-```ts
-const overview = await store.getOverview()
-for (const c of overview.error_clusters) {
-  console.log(`${c.trace_count}× ${c.signature}: e.g. trace ${c.exemplar_trace_ids[0]}`)
-}
-```
-
-## Recursive control integrity (no LLM)
-
-`CONTROL_INTEGRITY_ANALYST` checks the existing `SupervisorRunSources` or `SupervisorRunTree` directly.
-It does not define another run format.
-Register it as a custom-input analyst and pass the existing value under its stable id:
-
-```ts
-import {
-  AnalystRegistry,
-  CONTROL_INTEGRITY_ANALYST,
-} from '@tangle-network/agent-eval/analyst'
-import {
-  readLoopsSupervisorRun,
-} from '@tangle-network/agent-eval/supervisor-run'
-
-const sources = await readLoopsSupervisorRun(runDir)
-const registry = new AnalystRegistry()
-registry.register(CONTROL_INTEGRITY_ANALYST)
-
-const result = await registry.run('run-123', {
-  custom: { 'control-integrity': sources },
+const engine = createDspyRlmTraceEngine({
+  baseUrl: process.env.LLM_BASE_URL!,
+  apiKey: process.env.LLM_API_KEY!,
+  model: process.env.LLM_MODEL!,
+  pricing: {
+    inputUsdPerMillion: 3,
+    outputUsdPerMillion: 15,
+  },
+  maxCostUsd: 0.50,
+  runner: { command: '.venv/bin/python' },
 })
-```
-
-Pass `SupervisorRunSources` when it is available.
-A `SupervisorRunTree` does not retain raw journal multiplicity or worker request and acknowledgement rows, so tree input explicitly reports those checks as unavailable.
 
-The deterministic pass can prove only facts represented by these two existing surfaces.
+const result = await analyzeTraces(
+  { question: 'What first caused this run to fail?' },
+  {
+    source: 'run.otlp.jsonl',
+    engine,
+    toolGroup: 'singleTrace',
+    limits: {
+      maxIterations: 8,
+      maxLlmCalls: 4,
+      maxToolCalls: 32,
+    },
+  },
+)
 
-| Question | Current evidence | What the analyst can say |
-|---|---|---|
-| Is every invocation attached to one unambiguous tree? | `rootId`, `rollout_id`, `parent_rollout_id`, `run_id` | Duplicate ids, missing parents, extra parentless roots, cross-run edges, and ancestry cycles are violations with exact field references. |
-| Did invocation roles survive capture? | Explicit journal and `RolloutLine.role` values | The root must remain `supervisor`; non-root roles are consumed as recorded, and workers may spawn workers. |
-| Is the causal order possible? | `outcome.metrics.spawned_at`, `started_at`, `settled_at`, `completed_at`, `finished_at` when present | A child before its parent, a child after its parent closed, or a close before a start is a violation; absent timestamps produce no timing claim. |
-| Did a queued steer reach the worker? | `SupervisorRunSources.workers[].inbox` and `.events` | Requests and acknowledgements are joined by request id, not compared as totals. Missing, malformed, duplicate, or uncorrelated rows make the affected count unavailable. |
-| Can behavior be attributed to an exact profile? | `policy.agent_profile_cell_id` | An absent id is reported as unavailable. |
-| Can action authorship or reasoning be inspected? | `messages[]` | Empty gap rows are reported as unavailable. |
+console.log(result.answer)
+console.log(result.findings)
+console.log(result.trajectory)
+```
 
-An empty finding list means only that no implemented rule fired on the captured fields.
-It does not certify that an agent chose the action, that the action was authorized, that a budget or depth limit was enforced, or that a finding caused a later decision.
-Those claims require upstream action-decision events carrying `action_id`, `actor_rollout_id`, `target_rollout_id`, `action_kind`, `authority_snapshot_id`, requested and granted resource/depth values, the authorization result, and any `finding_id` or evidence references that caused the action.
-Resume integrity additionally requires an explicit prior-session id and resumed-session id rather than a prose summary.
+Omit `pricing` only when Agent Eval already recognizes the exact model or model family.
+Unknown pricing fails before the first model call.
 
-Malformed source rows are excluded from structural claims.
-Their count is retained in `SupervisorRunTree.gaps`, so analyzing a projected tree later cannot turn an unreadable parent row into a missing-parent violation.
+The provider key remains in the Node process.
+The Python process receives an authenticated loopback model endpoint with an ephemeral credential.
+Each trace read also crosses an authenticated loopback callback and counts against `maxToolCalls`.
 
-## Add A Custom Analyst
+## Define A Reusable Analyst
 
-`defineTraceAnalyst()` fills the fixed registry fields.
-The custom function receives the same bounded `TraceAnalysisStore` used by the built-ins.
+An analyst definition contains no model, credentials, or execution state.
+The same definition can run with DSPy or another engine that implements `TraceAnalysisEngine`.
 
 ```ts
 import {
-  AnalystRegistry,
   defineTraceAnalyst,
-  makeFinding,
+  runTraceAnalyst,
 } from '@tangle-network/agent-eval/analyst'
-
-const analysts = new AnalystRegistry()
-
-analysts.register(defineTraceAnalyst({
-  id: 'repeated-tool-errors',
-  description: 'Reports the largest repeated tool error cluster.',
-  cost: { kind: 'deterministic' },
-  async analyze(store) {
-    const overview = await store.getOverview({ has_errors: true })
-    const cluster = overview.error_clusters[0]
-    if (!cluster) return []
-
-    return [makeFinding({
-      analyst_id: 'repeated-tool-errors',
-      area: 'tool-use',
-      subject: cluster.signature,
-      claim: `${cluster.span_count} failed spans share one error`,
-      severity: 'high',
-      confidence: 1,
-      evidence_refs: [{
-        kind: 'span',
-        uri: `trace://${encodeURIComponent(cluster.exemplar_trace_ids[0])}/span/${encodeURIComponent(cluster.exemplar_span_ids[0])}`,
-        excerpt: cluster.status_message_sample,
-      }],
-      recommended_action: 'Fix the highest-frequency tool error before changing prompts.',
-      validation_plan: 'Run fresh cases and require this error signature to disappear.',
-    })]
-  },
-}))
-```
-
-Use code for exact facts such as exit codes, missing fields, and repeated calls.
-Use model-assisted analysts for semantic questions such as whether a response ignored user intent.
-
-## Measure An Analyst
-
-Do not judge an analyst by persuasive prose.
-Label the issue identity and exact evidence locations, then run the same cases through every implementation.
-
-```ts
 import {
-  compareAnalystRunners,
-  registryBenchmarkRunner,
-  renderAnalystBenchmarkMarkdown,
-  runAnalystBenchmark,
-  traceStoreEvidenceResolver,
-} from '@tangle-network/agent-eval/analyst'
+  OtlpFileTraceStore,
+} from '@tangle-network/agent-eval/traces'
 
-const benchmark = await runAnalystBenchmark({
-  cases: [{
-    id: 'failed-command',
-    input: { traceStore },
-    expectedIssues: [{
-      id: 'repeated-command',
-      subjects: ['failure-mode:repeated-command'],
-      evidence: [{ kind: 'span', uri: 'trace://run-1/span/tool-3' }],
-      criticalEvidence: [{ kind: 'span', uri: 'trace://run-1/span/tool-1' }],
-    }],
-    labeledEvidence: [
-      { kind: 'span', uri: 'trace://run-1/span/tool-1' },
-      { kind: 'span', uri: 'trace://run-1/span/tool-3' },
-    ],
-  }],
-  runners: [registryBenchmarkRunner({ id: 'built-in', registry: analysts })],
-  repetitions: 3,
-  resolveEvidence: traceStoreEvidenceResolver((input) => input.traceStore),
-  benchmark: {
-    id: 'failure-localization',
-    dataset: {
-      id: 'my-team/trace-failures',
-      revision: 'git-sha-or-content-digest',
-      split: 'test',
-    },
+const repeatedFailure = defineTraceAnalyst({
+  id: 'repeated-tool-failure',
+  description: 'Finds the repeated tool failure with the largest impact.',
+  area: 'tool-use',
+  version: '1.0.0',
+  question: 'Which repeated tool failure should we fix first?',
+  instructions: [
+    'Compare frequency and downstream impact.',
+    'Cite the failing span and at least one affected downstream span.',
+    'Return no finding when the traces do not support a repeated failure.',
+  ].join('\n'),
+  toolGroup: 'all',
+  minimumEvidenceCitations: 2,
+  limits: {
+    maxIterations: 10,
+    maxLlmCalls: 6,
+    maxToolCalls: 40,
   },
 })
 
-console.log(renderAnalystBenchmarkMarkdown(benchmark))
-```
-
-The result reports:
-
-- issue recall and finding precision,
-- first bad step accuracy,
-- citation coverage, agreement with labeled locations, and actual location resolution,
-- false positives on clean cases,
-- repeat agreement,
-- failed runs,
-- latency, calls, every reported token counter, and known or missing cost,
-- dataset revision, case tags, case metadata, and runner metadata.
-
-Use `compareAnalystRunners()` for paired differences between two implementations.
-Repetitions are averaged within each case before comparison.
-Treat its interval as inferential only with at least 20 independent cases.
-
-## Load Public Trace Labels
-
-Use the published label adapters with `@tangle-network/traces` or your own trajectory loader.
-Agent Eval does not download datasets or own trace capture.
-Load public data at an immutable commit and record that commit in `benchmark.dataset.revision`.
-
-```ts
-import {
-  agentRxBenchmarkCase,
-  agentRxPredictionsToFindings,
-  codeTraceBenchCase,
-  codeTracerPredictionsToFindings,
-} from '@tangle-network/agent-eval/analyst'
-import { otlpTextToTraceAnalysisStore } from '@tangle-network/agent-eval/traces'
-import { chatTrajectoryToSpans, serializeSpans } from '@tangle-network/traces'
-
-const codeSpans = chatTrajectoryToSpans(codeTraceTrajectory, {
-  traceId: codeTraceRow.traj_id,
-})
-const codeCase = codeTraceBenchCase(codeTraceRow, {
-  traceStore: otlpTextToTraceAnalysisStore(serializeSpans(codeSpans)),
-})
+const store = new OtlpFileTraceStore({ path: 'runs.otlp.jsonl' })
+await store.ensureIndexed()
 
-const agentRxSpans = chatTrajectoryToSpans(agentRxMessages, {
-  traceId: String(agentRxRow.trajectory_id),
-  stepMode: 'message',
+const result = await runTraceAnalyst({
+  definition: repeatedFailure,
+  engine,
+  store,
+  context: { runId: 'release-42' },
 })
-const rootCauseCase = agentRxBenchmarkCase(agentRxRow, {
-  traceStore: otlpTextToTraceAnalysisStore(serializeSpans(agentRxSpans)),
-}, {
-  stepCount: agentRxMessages.length,
-})
-
-const codeTracerFindings = codeTracerPredictionsToFindings(
-  codeTraceRow.traj_id,
-  codetracerLabels,
-  { stepCount: codeTraceRow.step_count },
-)
-const agentRxFindings = agentRxPredictionsToFindings(
-  agentRxRow.trajectory_id,
-  agentRxJudgeOutput,
-  { stepCount: agentRxMessages.length },
-)
 ```
 
-`codeTraceBenchCase()` accepts the public [CodeTraceBench](https://huggingface.co/datasets/NJU-LINK/CodeTraceBench) JSONL format.
-It scores the published incorrect-step task by default, including clean trajectories.
-Pass `labelSet: 'incorrect-and-unuseful'` to both the case and prediction adapters only for an explicitly combined experiment.
-Every cited step is checked against `step_count`.
-
-`agentRxBenchmarkCase()` accepts the public [AgentRx](https://huggingface.co/datasets/microsoft/AgentRx) label format.
-AgentRx category quality and root-step accuracy are scored independently.
-`traceAnalystQualityJudge` averages them when a root-step label exists.
-Pass `target: 'all-failures'` only when the analyst is designed to identify every annotated failure.
+Keep product policy in the definition.
+Keep model transport, recursion mechanics, secrets, and accounting in the engine.
+Keep exact trace facts in deterministic tools or checks.
 
-Both adapters emit `trace://<id>/span/step-<n>` evidence by default.
-`@tangle-network/traces` uses the same IDs when converting chat trajectories.
-Pass `stepUri` when your trace store uses another URI scheme.
-`codeTracerPredictionsToFindings()` and `agentRxPredictionsToFindings()` translate the maintained upstream engines' native outputs into the same evidence and category shape.
-AgentRx `Report.to_dict()` judge votes reduce to the upstream majority failure type and Python-rounded mean step, and direct `failures` arrays use the same reduction.
-`failure_case: 0` produces no finding, which scores as a missed root cause on AgentRx's failed trajectories.
+## Run The Built-In Set
 
-## Use Upstream Scorers
+The default registry always includes deterministic behavior checks.
+It adds the four recursive analysts only when an engine is supplied:
 
-Agent Eval adapts upstream evaluators instead of copying them.
+- `failure-mode`
+- `knowledge-gap`
+- `knowledge-poisoning`
+- `improvement`
 
 ```ts
-import { createEvaluator } from '@arizeai/phoenix-evals'
-import { ExactMatch } from 'autoevals'
 import {
-  autoevalsScorerJudge,
-  phoenixEvaluatorJudge,
-} from '@tangle-network/agent-eval/campaign'
-
-const phoenix = createEvaluator(
-  ({ output, expected }) => output === expected ? 1 : 0,
-  { name: 'exact-match', kind: 'CODE', telemetry: { isEnabled: false } },
-)
+  buildDefaultAnalystRegistry,
+} from '@tangle-network/agent-eval/analyst'
 
-const phoenixJudge = phoenixEvaluatorJudge(phoenix, {
-  mapInput: ({ artifact, scenario }) => ({ output: artifact, expected: scenario.expected }),
-})
+const registry = buildDefaultAnalystRegistry({ engine })
+const run = await registry.run('release-42', { traceStore: store })
 
-const autoevalsJudge = autoevalsScorerJudge(ExactMatch, {
-  name: 'exact-match',
-  kind: 'CODE',
-  mapInput: ({ artifact, scenario }) => ({ output: artifact, expected: scenario.expected }),
-})
+for (const finding of run.findings) {
+  console.log(finding.analyst_id, finding.claim, finding.evidence_refs)
+}
 ```
 
-These adapters do not install either upstream package for consumers.
-Install only the scorer package you use.
-Missing or non-finite scores throw instead of becoming passes.
-Phoenix evaluators marked `MINIMIZE` or `NEUTRAL` require `toComposite` so candidate selection never assumes the wrong direction.
-Mark model-backed evaluators as `kind: 'LLM'` and provide `paidCall` with the model and a receipt mapper.
-The campaign then passes its cancellation signal and cost ledger through the adapter.
-An LLM evaluator is rejected before execution when either cost capture or the campaign ledger is missing.
-
-## Turn Reviewed Findings Into Eval Data
-
-Generated findings can populate a review queue.
-They cannot promote themselves into learning data.
-
-```ts
-import {
-  analystFindingDigest,
-  analystRunDigest,
-  analystRunToFeedbackTrajectory,
-  analystRunToReviewRequests,
-} from '@tangle-network/agent-eval'
-
-const runDigest = analystRunDigest(result)
-const requests = analystRunToReviewRequests(result)
-await reviewQueue.add(requests)
-
-declare const acceptedFindingIds: ReadonlySet<string>
-
-const trajectory = analystRunToFeedbackTrajectory(result, {
-  task: { intent: 'Find why the command failed.' },
-  reviewRequests: requests,
-  reviewDecisions: [
-    ...result.findings.map((finding) => ({
-      runDigest,
-      findingId: finding.finding_id,
-      findingDigest: analystFindingDigest(finding),
-      verdict: acceptedFindingIds.has(finding.finding_id) ? 'confirmed' as const : 'rejected' as const,
-      source: 'user' as const,
-      reviewerId: 'reviewer-42',
-      reviewId: 'trace-review-918',
-      reason: 'Reviewed against the cited span.',
-      decidedAt: new Date().toISOString(),
-    })),
-    {
-      runDigest,
-      verdict: 'completeness_assessed',
-      missedIssues: [],
-      source: 'user',
-      reviewerId: 'reviewer-42',
-      reviewId: 'trace-review-918',
-      reason: 'Reviewed the full run for omitted findings.',
-      decidedAt: new Date().toISOString(),
-    },
-  ],
-  trace: { artifactUri: 'traces.otlp.jsonl', traceIds: ['run-1'] },
-})
+`run.per_analyst` records each analyst's status, latency, calls, tokens, and cost.
+One analyst failure does not become an agent failure.
+
+Pass `definitions` to replace the four built-ins.
+Omit `engine` for deterministic-only analysis.
+
+## Trace Tools
+
+The engine receives only the tool group declared by the analyst:
+
+| Group | Use |
+|---|---|
+| `singleTrace` | Inspect one known trajectory |
+| `discoveryAndSearch` | Find relevant traces and spans across a dataset |
+| `all` | Use every bounded trace operation |
+
+The canonical operations are:
+
+- `getDatasetOverview`
+- `queryTraces`
+- `countTraces`
+- `viewTrace`
+- `viewSpans`
+- `searchTrace`
+- `searchSpan`
+
+Use `buildTraceAnalysisToolDescriptors({ store })` to bind the same operations into another runtime.
+Each descriptor includes its name, namespace, description, JSON input schema, and bounded handler.
+Do not copy the schemas or reimplement the handlers in another adapter.
+
+Custom stores implement `TraceAnalysisStore`.
+`OtlpFileTraceStore`, `otlpTextToTraceAnalysisStore()`, and `toolSpansToTraceAnalysisStore()` provide common adapters.
+Store results are checked for missing fields, undeclared fields, inconsistent counts, invalid continuation flags, oversized responses, and unsafe search patterns.
+
+## Result Contract
+
+Every recursive run returns:
+
+| Field | Meaning |
+|---|---|
+| `answer` | Direct answer to the analyst question |
+| `findings` | Valid cited claims accepted by the analyst policy |
+| `trajectory` | DSPy RLM investigation steps |
+| `modelCalls` | Successful provider completions used by the engine |
+| `toolCalls` | Trace reads admitted for execution, including a read that later fails |
+| `runtime` | Engine, package, sandbox identity, provider attempts, and successful completions |
+
+Each finding requires exact `trace://` or `finding://` evidence.
+Trace citations must resolve to an existing span.
+When a citation includes an excerpt, the exact text must occur in that span or referenced finding.
+Unknown, transformed, or fabricated identifiers are rejected.
+An empty findings array means no submitted claim passed the evidence rules.
+It does not prove the run was correct.
+
+## Measure Analyst Quality
+
+Measure the analyst on labeled traces before using its findings for automated changes.
+At minimum, report issue recall, finding precision, exact evidence accuracy, trusted-negative false positives, repeat agreement, failures, calls, tokens, cost, and latency.
+
+`runAnalystBenchmark()` compares any `AnalystBenchmarkRunner` implementations.
+`agent-eval analyst-benchmark` runs the public AgentRx or CodeTraceBench adapters with:
+
+1. an empty-finding baseline,
+2. the actual DSPy RLM trace analyst.
+
+```sh
+agent-eval analyst-benchmark \
+  --dataset codetracebench \
+  --labels .artifacts/manifest.jsonl \
+  --trace-dir .artifacts/traces \
+  --artifact-dir .artifacts/results \
+  --out .artifacts/analyst-run \
+  --revision aa213b84ffb6690fc37ca15766d6ca174ec36d4d \
+  --split verified \
+  --base-url http://127.0.0.1:3355/v1 \
+  --api-key-env CLI_BRIDGE_BEARER \
+  --model claude-code/sonnet \
+  --python .venv/bin/python \
+  --limit 20 \
+  --seed 7 \
+  --concurrency 1 \
+  --max-cost-usd 5
 ```
 
-`analystRunToFeedbackTrajectory()` stores review requests separately from labels.
-It can archive an unreviewed run.
-`feedbackTrajectoryToOptimizerRow()` requires every decision to match the complete run digest, every finding decision to match the finding digest, and one independent completeness assessment.
-Its score is F1 over confirmed findings and independently identified misses.
-Generic labels and run-level outcomes do not satisfy these requirements.
-
-## Required Trace Data
-
-Useful analysis needs:
-
-- stable run, trace, and span IDs,
-- parent-child links and ordered timestamps,
-- model, prompt, and configuration identity,
-- complete tool names, arguments, results, and error codes,
-- token, cost, and latency data when available,
-- retrieval source IDs and scores when retrieval is involved,
-- final environment outcomes such as tests, task completion, or policy blocks.
-
-Do not include secrets, raw OAuth tokens, or unredacted personal data.
+The command writes every observation before producing `result.json` and `report.md`.
+It can resume without repeating completed cases.
+Dataset revisions, selected case IDs, input hashes, trace hashes, result artifacts, usage, errors, and comparison settings remain in the output.
 
-Use [`@tangle-network/traces`](https://github.com/tangle-network/traces) to normalize coding-agent sessions, run HALO as an external report engine, or run Hodoscope as a behavior-discovery engine.
+Use fresh development cases while changing questions or instructions.
+Report final quality only on an untouched holdout.