@tangle-network/agent-eval 0.189.0 → 0.190.1

package/docs/adapters-observability.md

@@ -1,11 +1,16 @@
 # Composing agent-eval with your observability stack
 
-`@tangle-network/agent-eval` ships its own OpenTelemetry pipeline
-(`createOtelExporter`, from `@tangle-network/agent-eval/traces`) that
-emits spans for every cell, judge invocation, mutator proposal, and
-gate decision. **It's just OTel**: same protocol as Langfuse SDK,
-OpenLLMetry, Arize Phoenix, TraceAI, and the OpenTelemetry GenAI
-semantic conventions.
+`@tangle-network/agent-eval` records runs and spans in a `TraceStore` and
+writes them as OTLP/JSON: `exportRunAsOtlp(store, runId)` for one run and
+`convertTraceStoresToOtlp` for a directory of stores, both from
+`@tangle-network/agent-eval/traces`. **It's just OTel**: same protocol as
+Langfuse SDK, OpenLLMetry, Arize Phoenix, TraceAI, and the OpenTelemetry
+GenAI semantic conventions.
+
+To stream spans to a collector while a run executes, use
+`createOtelExporter` from `@tangle-network/agent-runtime`. It bounds its
+queue, checks every response, and counts written and dropped spans, so a
+missing trace is never mistaken for an empty run.
 
 That means: if you already instrument your agent with any OTel-native
 observability tool, the two compose **for free at the protocol layer**.
@@ -18,11 +23,10 @@ code required.
 2. Configure your observability tool (TraceAI / Langfuse / OpenLLMetry /
    Phoenix) to register its instrumentations against a tracer provider that
    points at that endpoint.
-3. Configure agent-eval's exporter (`createOtelExporter`, from
-   `@tangle-network/agent-eval/traces`) against the same endpoint. It has no
-   `@opentelemetry/*` SDK dependency of its own — it reads
-   `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_EXPORTER_OTLP_HEADERS` directly, so
-   pointing both sides at the same collector is usually just sharing env vars.
+3. Post agent-eval's OTLP export (`exportRunAsOtlp`) to the same endpoint's
+   `/v1/traces`, or stream with agent-runtime's `createOtelExporter`, which
+   reads `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_EXPORTER_OTLP_HEADERS`.
+   Neither needs an `@opentelemetry/*` SDK dependency.
 4. Run a campaign. Both sets of spans land at your OTel collector.
 5. Filter / route / fan-out at the collector layer: Jaeger, Tempo,
    Phoenix, Langfuse cloud, your private collector, whatever.
@@ -63,8 +67,8 @@ it*. Unified at the trace level, you see both as one timeline per cell.
 - OSS auto-instrumentation library; OTel-native by design.
 - Wide framework coverage (LangChain, LlamaIndex, Haystack, OpenAI,
   Anthropic).
-- Compose: set up Traceloop's exporter; agent-eval's exporter shares
-  the same trace context per cell.
+- Compose: set up Traceloop's exporter and post agent-eval's OTLP export
+  to the same collector.
 
 ### Arize Phoenix
 
@@ -93,18 +97,18 @@ provider.register()
 //    Example for TraceAI / OpenLLMetry / Langfuse: call their init.
 //    (See each tool's docs.)
 
-// 3. agent-eval's exporter is not built on the OTel SDK provider above —
-//    it's a minimal OTLP/JSON poster with no `@opentelemetry/*` dependency
-//    of its own. It reads the same OTEL_EXPORTER_OTLP_ENDPOINT env var, so
-//    pointing it at the same collector is usually the only step needed:
-import { createOtelExporter } from '@tangle-network/agent-eval/traces'
-const exporter = createOtelExporter()  // undefined if no endpoint is configured
-// Feed it spans as your campaign's TraceEmitter closes them; call
-// `exporter?.shutdown()` when the campaign ends to flush pending spans.
-
-// 4. Run your campaign: both sets of spans land at the collector.
-import { runEval } from '@tangle-network/agent-eval/contract'
-await runEval({ /* ... */ })
+// 3. Run your campaign, then post each run's OTLP/JSON to the same collector.
+//    exportRunAsOtlp is not built on the OTel SDK provider above and needs no
+//    `@opentelemetry/*` dependency. Check the response: a collector that
+//    refuses the batch is a lost trace, not an empty one.
+import { exportRunAsOtlp } from '@tangle-network/agent-eval/traces'
+const body = await exportRunAsOtlp(store, runId)
+const res = await fetch('http://localhost:4318/v1/traces', {
+  method: 'POST',
+  headers: { 'content-type': 'application/json' },
+  body: JSON.stringify(body),
+})
+if (!res.ok) throw new Error(`collector refused the trace: HTTP ${res.status}`)
 ```
 
 That's it. No new adapter shipping required: the libs are already
@@ -132,8 +136,8 @@ works today without them.
 
 No new dependencies. No new peer deps. No `@traceai/*`, no
 `@langfuse/*`, no `@opentelemetry/*` in our manifest. You bring the
-observability stack you want; agent-eval's exporter emits the same
-OTLP wire format independently, keyed on the endpoint you point it at.
+observability stack you want; agent-eval's OTLP export is the same wire
+format, posted to the endpoint you choose.
 
 
 ## Supervisor-run resource receipts

package/docs/charter.md

@@ -36,7 +36,7 @@ Outcome association can motivate an experiment; it cannot establish that changin
 
 **Can another reader verify the evidence?**
 [Evidence receipts](./experiment.md) bind reports to declared identities and provenance.
-[Search-history receipts](./search-history-receipts.md) account for planned and attempted search slots.
+The [search ledger](./search-ledger.md) records every node, edge and cell of a search, and its receipt proves the search closed.
 [Verdict certifications](./verdicts.md) name the checker and its unverified assumptions.
 The [evidence registry](../evidence/README.md) retains published measurements and their freshness state.
 

package/docs/design/mlbenchmarks-book-review.md

@@ -116,7 +116,7 @@ These are implementation findings, not an assessment of adoption in every consum
 | Sequential testing | [sequential gate](../../src/campaign/gates/sequential.ts), [e-process](../../src/statistics/sequential-eprocess.ts) | Present. Optional stopping within a valid stream differs from adapting hypotheses across streams or counting dependent replicas as independent evidence. |
 | Search/final separation | [selfImprove](../../src/contract/self-improve.ts), [method comparison](../../src/campaign/presets/compare-optimization-methods.ts) | Present within calls. Final cases are withheld from method inputs. A persistent cross-campaign exposure policy is missing from these paths. |
 | Dataset identity and access | [Dataset](../../src/dataset.ts), [contamination helpers](../../src/contamination-guard.ts), [labeled store](../../src/campaign/labeled-store/fs-adapter.ts) | Partial. Hashes, split labels, mutation locks, temporal sampling, and access logs exist. They do not establish secrecy, lineage independence, or one-time certification use. |
-| Complete search history | [SearchLedger](../../src/campaign/search-ledger.ts), [history receipt documentation](../search-history-receipts.md) | Present. Reuse the canonical ledger; do not create another optimizer event log. Requiring complete history remains a caller policy. |
+| Complete search history | [SearchLedger](../../src/campaign/search-ledger.ts), [search ledger documentation](../search-ledger.md) | Present. Reuse the canonical ledger; do not create another optimizer event log. Requiring complete history remains a caller policy. |
 | Evidence identity and authority | [EvidenceReceipt](../../src/experiment/evidence-receipt.ts), [campaign receipts](../../src/experiment/campaign-evidence.ts), [registry](../../src/experiment/evidence-record.ts) | Present. These bind identities and preserve declared authority. Hashes and authority labels alone do not prove independent execution or valid sampling. |
 | Judge quality and drift | [calibration](../../src/judge-calibration.ts), [sentinel](../../src/meta-eval/sentinel.ts), [plants](../../src/meta-eval/plants.ts) | Present. Per-candidate residual bias, ranking validity, and independent evaluator admission need composition. Position and self-preference helpers exist internally; only verbosity has a root public export. |
 | Multiple objectives | [promotion policy](../../src/campaign/gates/promotion-policy.ts), [production gate](../../src/campaign/gates/default-production-gate.ts) | Present. Per-dimension regression guards and evidence vectors exist. Explicit target-population weights and aggregation sensitivity remain useful additions. |

package/docs/distributed-driver.md

@@ -142,7 +142,7 @@ round-robin, region-affinity from a previous run, scheduling table).
 | **Auth** | Bearer token on `Authorization`; pluggable via `auth: string \| () => string \| Promise<string>` for rotation/refresh. |
 | **Payload size** | Server enforces `maxBodyBytes` (default 10 MB). |
 | **Traces** | Both ends emit OTel: if both point at the same OTLP collector, you get a unified trace per cell. See `docs/adapters-observability.md`. |
-| **Cost** | Worker's `ctx.cost.runPaidCall(...)` writes durable receipts in the worker process. Roll up those receipts server-side and attach them to worker telemetry; they are not forwarded to the coordinator automatically. |
+| **Cost** | Worker's `ctx.cost.runPaidCall(...)` writes durable receipts in the worker's own `CostLedger`. `runDispatchServer` returns them with the artifact; `httpDispatch` replays each one into the coordinator's `ctx.cost`, so the coordinator's `CostLedger` — its summary, its cost ceiling, `CampaignCellResult.costUsd` — sees remote spend exactly like an in-process cell's. A worker whose `contextFactory` never wires a real cost meter (the default `NOOP_COST`) sends no receipts, so a real deployment must wire one. A dispatch that fails after paying for part of its work still returns those receipts with its error, so a remote failure never drops spend on the floor. |
 
 ## Running the reference example
 
@@ -164,10 +164,6 @@ and using `cellPlacement` to fan across many of them.
 
 ## Known gaps + follow-ups
 
-- **Cost roll-up across the wire**: worker-side `ctx.cost` observations
-  stay on the worker. We need to forward them in the response body so
-  `defaultProductionGate`'s `budgetUsd` ceiling reflects total spend, not
-  coordinator-side spend. Tracked as a 0.45.x follow-up.
 - **Per-cell artifact streaming**: when the worker writes intermediate artifacts through `ctx.artifacts.write`, those files remain in worker storage.
   Multi-worker campaigns need a shared `CampaignStorage` implementation reachable from both sides.
   This package does not include an S3 or GCS implementation.

package/docs/experiment.md

@@ -273,6 +273,21 @@ The statistical machinery underneath is re-exported from its existing homes; thi
 `HypothesisManifest` stays as the lightweight single-metric registration; `sealExperiment` is the full-design registration.
 The trace-repair admission machinery (`buildDenominatorChain`, oracle determinism, control policy) keeps its repair vocabulary in `./trace-repair`; this module is the general form new experiments should register against.
 
+## Bind final measurements to execution evidence
+
+`compareOptimizationMethods` and `selfImprove({ method })` accept optional `evidence: CampaignEvidenceContext`.
+It requires the caller's pursuit, evaluator, environment, authority, and attestation provenance.
+Eval does not infer independent authority or certify those external declarations.
+
+Each measured baseline and winner receives an `EvidenceReceipt`.
+Eval derives candidate, input-set, output, and result digests from the executed surface and complete campaign.
+A missing or deferred final measurement cannot produce a receipt.
+Receipts describe measurements; they do not override the release gate or replace the method's selected winner.
+
+`createCampaignEvidenceReceipt` provides the same binding for other complete campaign consumers.
+Changing an output changes its output digest; changing a judge result changes its measurement digest.
+The final receipt retains caller authority, including `candidate-self-report`, without upgrading it.
+
 ## Where this sits
 
 The [charter](./charter.md) describes current package ownership and host responsibilities.