@tangle-network/agent-eval 0.180.0 → 0.182.0

package/docs/outcome-validity.md

@@ -0,0 +1,182 @@
+# Connect rubric scores to deployment outcomes
+
+`rubricPredictiveValidity()` measures associations between rubric scores and observations from deployment.
+Declare the desired direction for each outcome before reading the results.
+Define rubric scores so higher values mean better evaluated behavior.
+
+```ts
+import {
+  InMemoryOutcomeStore,
+  rubricPredictiveValidity,
+  type DeploymentOutcome,
+  type OutcomeMetricSpec,
+} from '@tangle-network/agent-eval/meta-eval'
+import type { RunRecord } from '@tangle-network/agent-eval'
+
+async function assessOutcomes(runs: RunRecord[], observations: DeploymentOutcome[]) {
+  const outcomes = new InMemoryOutcomeStore()
+  for (const observation of observations) await outcomes.append(observation)
+  const metrics: OutcomeMetricSpec[] = [
+    { id: 'success_rate', direction: 'higher-is-better' },
+    { id: 'failure_rate', direction: 'lower-is-better' },
+  ]
+
+  return rubricPredictiveValidity({
+    runs,
+    outcomes,
+    outcomeMetrics: metrics,
+    rubrics: ['task_quality'],
+  })
+}
+```
+
+Populate `RunRecord.outcome.raw.task_quality` with the captured rubric score.
+Append deployment observations with the same `runId` and exact outcome metric keys.
+The store accepts finite numbers, including zero.
+Omit an unmeasured metric instead of replacing it with zero.
+
+An observation records when and where its outcomes were captured:
+
+```ts
+const observation: DeploymentOutcome = {
+  runId: 'support-run-42',
+  capturedAt: Date.now(),
+  metrics: { success_rate: 1, failure_rate: 0 },
+  labels: { cohort: 'support' },
+  source: 'resolution-events-v1',
+}
+```
+
+`capturedAt` uses epoch milliseconds.
+The matching run supplies the rubric score; this observation supplies the measured deployment outcome.
+
+The default reduction selects the latest finite observation of each requested metric.
+A newer row containing another metric cannot supply its value or erase an older observation.
+The `mean` and `max` reductions operate on that metric alone.
+
+## Read the report
+
+| Field | Interpretation |
+|---|---|
+| `pearson`, `spearman` | Association with the recorded outcome, retaining its original sign. |
+| `alignedPearson`, `alignedSpearman` | Positive means higher rubric scores associate with better outcomes. |
+| `pearsonCi95`, `spearmanCi95` | Bootstrap intervals for the raw associations. |
+| `alignedSpearmanCi95` | Spearman interval after applying the declared outcome direction. |
+| `verdict` | `aligned` at aligned Spearman ≥ 0.4; `inverse` at ≤ −0.4; otherwise `weak`. |
+| `n` | Finite joined run observations for this rubric and outcome. |
+| `excludedPairs` | Unestimated pairs, their observation counts, and the reason for exclusion. |
+| `rubricsWithoutData` | Declared rubrics with no finite score in the supplied runs. |
+
+A rubric can correlate negatively with failure rate and still receive `aligned`.
+The same negative correlation with success rate receives `inverse`.
+These labels describe association strength and direction.
+They do not grant release authority or establish that changing a rubric weight will improve outcomes.
+
+Pairs require eight observations by default.
+`minSamples` can set another integer of at least three.
+Insufficient observations and constant scores or outcomes remain explicit exclusions.
+Constant observations cannot establish a perfect predictor.
+Intervals remain `null` when no bootstrap resample has an estimable correlation.
+
+`joinedSamples + skippedRuns` equals the supplied run count.
+A run is joined when at least one finite score and outcome pair exists.
+It remains joined even if that pair has too few observations for estimation.
+Duplicate run IDs are rejected.
+
+Each run is an independent bootstrap observation.
+Repeated observations from the same person or task can violate that assumption.
+Aggregate at the independent unit or use a study with an appropriate grouped estimator.
+
+`ranked` selects each rubric's highest direction-aligned association across the declared outcomes.
+This ordering is exploratory and can conceal conflicts among outcomes.
+Inspect all `pairs` and use a target chosen before analysis for an automated recommendation.
+Confirm any proposed change with fresh evidence and fixed scoring rules.
+
+## Propose an experiment against one target
+
+```ts
+import { PredictiveValidityResearcher } from '@tangle-network/agent-eval/rl'
+import { InMemoryOutcomeStore } from '@tangle-network/agent-eval/meta-eval'
+
+const outcomes = new InMemoryOutcomeStore()
+const researcher = new PredictiveValidityResearcher({
+  outcomes,
+  targetOutcome: { id: 'success_rate', direction: 'higher-is-better' },
+  rubrics: ['task_quality'],
+})
+```
+
+Supply observed outcomes before calling `runValidityCheck(runs)`.
+Pass the resulting failure groups to `proposeChange(failures)`.
+The researcher uses its declared target even if another outcome has a stronger association.
+It proposes increased-weight experiments for aligned associations and reversal or replacement experiments for inverse associations.
+Both require an aligned Spearman interval that excludes zero.
+Weak or uncertain associations produce requests for calibration evidence.
+Missing estimates produce requests for more outcome observations.
+
+The proposals contain their association, interval, sample count, and target direction.
+They contain no predicted improvement because a correlation does not identify a causal treatment effect.
+`applyChange()` appends proposals to a plan.
+`evaluateChange()` declines promotion because the caller owns experiment execution.
+
+`runRLCampaign()` accepts the same outcome specifications through `outcomeMetrics` when `outcomeStore` is supplied.
+Supply both options together; incomplete or empty declarations fail before campaign execution.
+Its summary reports direction-aligned association and preserves missing estimates.
+The target declaration remains fixed while the campaign runs.
+
+The researcher copies its target declaration, rubric list, and cached reports.
+Changing callback arguments, returned reports, or proposal payloads cannot rewrite its cached evidence.
+
+## Store observations without losing failures
+
+`InMemoryOutcomeStore` and `FileSystemOutcomeStore` copy observations at ingestion and retrieval.
+Changing a caller's metric or label object cannot rewrite stored evidence.
+The filesystem store rereads observations so another instance's later writes remain visible.
+Use one writer per directory; operations on that writer are serialized.
+
+A nonexistent directory is an empty store.
+An unreadable file, malformed JSON, or invalid outcome record throws `OutcomeStoreError`.
+The error carries its operation, path, source line when available, and original cause.
+Read errors never become empty studies or partial successful results.
+Repair the source and retry the read.
+
+For trace data, `correlationStudy()` accepts outcome names and reports descriptive associations without a desired direction.
+It shares the metric reduction, bootstrap, and exclusion behavior.
+It excludes observations captured before the run started.
+`maxCaptureLagMs` optionally bounds the elapsed time from run start through outcome capture, including both endpoints.
+
+## Inspect calibration by score range
+
+Use `calibrationFromPairs()` when scores and outcomes are already joined.
+It accepts readonly observations directly.
+
+```ts
+import { calibrationFromPairs } from '@tangle-network/agent-eval/meta-eval'
+
+const calibration = calibrationFromPairs([
+  { evalScore: 1, outcome: 0 },
+  { evalScore: 1, outcome: 0 },
+], 'predicted-success', 'observed-success')
+
+console.log(calibration?.ece) // 1: confident predictions, observed failures.
+```
+
+Direct input rejects nonfinite pairs with the offending index.
+It does not silently discard them from the denominator.
+`calibrationCurve()` performs the join for trace and outcome stores using the latest finite observation of the named outcome metric.
+It reports each bin's count, mean score, mean outcome, and absolute gap.
+`ece` weights each bin's gap by its share of the finite joined observations.
+Both quantities must use comparable numerical scales for this difference to measure calibration.
+
+The `range` option clips scores before binning and retains every joined observation.
+The reported calibration error then describes the clipped scores; retain that transformation with the result.
+Constant scores form one bin, so a consistently wrong predictor still receives a measured calibration error.
+Equal-frequency binning produces the requested number of bins, capped by the observation count.
+Bin counts differ by at most one, except when all scores share one value.
+Equal-width binning omits empty bins.
+On either path, bin counts sum to the reported `n`.
+
+Fewer than two finite pairs returns `null`.
+Both entry points validate metric identities, binning options, and range bounds.
+Invalid requests fail even when no evidence is available.
+The store entry point validates before reading evidence.

package/docs/product-eval-adoption.md

@@ -152,8 +152,7 @@ set with a signed note.
 
 ## Optimization
 
-Use `runImprovementLoop()` when the system is a multi-step agent, not a
-single prompt.
+Use `runImprovementLoop()` when a `SurfaceProposer` supplies candidates for Agent Eval's search loop.
 
 Good optimization targets:
 

package/docs/public-api.md

@@ -7,11 +7,11 @@ Generated by `pnpm api:census` on demand — this is a dated reading, not a gate
 | measure | count |
 | --- | --- |
 | export subpaths | 28 |
-| published value exports (subpath x symbol) | 1403 |
-| distinct symbols | 1231 |
+| published value exports (subpath x symbol) | 1400 |
+| distinct symbols | 1228 |
 | production | 944 |
-| planned | 218 |
-| none | 241 |
+| planned | 216 |
+| none | 240 |
 
 Type-only exports are not listed: a type binds no runtime surface, and removing one cannot break a caller at run time.
 
@@ -1289,7 +1289,7 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 
 ### `./supervisor-run`
 
-33 value exports — 23 production, 4 planned, 6 none.
+30 value exports — 23 production, 2 planned, 5 none.
 
 | symbol | consumer | evidence |
 | --- | --- | --- |
@@ -1297,17 +1297,14 @@ The `none` set was reviewed symbol by symbol on 2026-08-21. Four rules decided m
 | `analyzeSupervisorRunIntegrity` | production | discovery-lab:pursuits/final-agent-authored-campaign-20260810/materialize-candidates.mjs |
 | `analyzeSupervisorRunSources` | production | discovery-lab:experiments/0005-run-visibility/probe.mjs |
 | `claudeCodeSupervisorRunReader` | none | only this package's tests: src/supervisor-run/claude-code-reader.test.ts:6 |
-| `findSupervisorRunDirIn` | none | only this package's tests: src/supervisor-run/loops-reader.test.ts:11 |
 | `findSupervisorRunDirs` | production | traces:src/cli.ts |
 | `isRuntimeSupervisorRunDir` | production | discovery-lab:tools/disco.mjs |
 | `isUnavailable` | production | discovery-lab:tools/disco.mjs |
-| `loopsSupervisorRunReader` | planned | named in traces (bind not in the import graph) |
 | `NO_SOURCE_LIMITS` | production | traces:src/supervisor-run-context.ts |
 | `NO_TERMINAL_RECORD` | none | only this package's tests: src/supervisor-run/terminal-record.test.ts:8 |
 | `parsePatch` | planned | named in browser-agent-driver (bind not in the import graph) |
 | `parseSupervisorTree` | production | discovery-lab:experiments/0005-run-visibility/probe.mjs |
 | `readClaudeCodeSupervisorRun` | none | only this package's tests: src/supervisor-run/claude-code-reader.test.ts:6 |
-| `readLoopsSupervisorRun` | planned | named in discovery-lab (bind not in the import graph) |
 | `readRuntimeSupervisorRun` | production | discovery-lab:pursuits/final-agent-authored-campaign-20260810/materialize-candidates.mjs |
 | `readTerminalRecord` | production | this package: src/supervisor-run/analyze.ts:18 |
 | `renderSupervisorRollupMarkdown` | production | traces:src/cli.ts |

package/docs/research-report-methodology.md

@@ -60,14 +60,14 @@ In order: first match wins:
 
 | Quantity | Function | Source file |
 |---|---|---|
-| Marginal CI on score mean | `confidenceInterval` | `statistics.ts` |
-| Paired Cohen's dz vs comparator | `pairedCohensDz` | `statistics.ts` |
-| Wilcoxon signed-rank (paired), exact at n ≤ 20 | `wilcoxonSignedRank` | `statistics.ts` |
-| BH-FDR q-values | `benjaminiHochberg` | `statistics.ts` |
-| Paired bootstrap CI on median delta | `pairedBootstrap` | `statistics.ts` |
-| Smallest p a rank-test design can produce | `pFloor` on the result | `statistics.ts` |
+| Marginal CI on score mean | `confidenceInterval` | [`descriptive.ts`](../src/statistics/descriptive.ts) |
+| Paired Cohen's dz vs comparator | `pairedCohensDz` | [`effect-sizes.ts`](../src/statistics/effect-sizes.ts) |
+| Wilcoxon signed-rank (paired), exact at n ≤ 20 | `wilcoxonSignedRank` | [`rank-tests.ts`](../src/statistics/rank-tests.ts) |
+| BH-FDR q-values | `benjaminiHochberg` | [`multiplicity.ts`](../src/statistics/multiplicity.ts) |
+| Paired bootstrap CI on median delta | `pairedBootstrap` | [`paired-tests.ts`](../src/statistics/paired-tests.ts) |
+| Smallest p a rank-test design can produce | `pFloor` on the result | [`rank-tests.ts`](../src/statistics/rank-tests.ts) |
 | Bayesian-bootstrap Pr(Δ>0), Pr(Δ∈ROPE) | `bayesianBootstrapMeanSamples` | `summary-report.ts` (private) |
-| Minimum detectable paired effect | `pairedMde` | `statistics.ts` |
+| Minimum detectable paired effect | `pairedMde` | [`power-and-mde.ts`](../src/statistics/power-and-mde.ts) |
 | Run fingerprint | `hashJson(...)` | `pre-registration.ts` |
 
 The Pr(Δ>0) and Pr(Δ∈ROPE) summaries use Rubin's Bayesian bootstrap.

package/docs/search-history-receipts.md

@@ -222,3 +222,11 @@ Receipts describe measurements; they do not override the release gate or replace
 `createCampaignEvidenceReceipt` on `/experiment` provides the same binding for other complete campaign consumers.
 Changing an output changes its output digest; changing a judge result changes its measurement digest.
 The final receipt retains caller authority, including `candidate-self-report`, without upgrading it.
+
+## Implementation ownership
+
+`src/campaign/search-ledger.ts` owns validation, canonical event normalization, and composition with the shared ledger journal.
+`search-ledger-projector.ts` owns campaign replay invariants and audit projection.
+`search-ledger-types.ts` defines the shared data contracts, reexported through the existing facade.
+`search-ledger-ordering.ts` shares artifact identity and deterministic string ordering between normalization and replay.
+The journal in `src/ledger-core` owns hashing, locking, durable appends, and chain verification.

package/docs/statistical-evidence.md

@@ -0,0 +1,129 @@
+# Statistical evidence
+
+Repeated runs measure execution variation on the cases you supplied.
+A claim about new cases also needs independent observations from its target population.
+Repeating one incident 100 times supplies one incident, even when every execution has a distinct identifier.
+
+## Declare the independent unit
+
+Pass `independentUnitByScenarioId` to `defaultProductionGate`, `heldoutSignificance`, or `dimensionRegressions` when scenarios share a source.
+The map assigns each scenario to the incident, document, task family, or other unit sampled independently.
+Choose this assignment before examining scores.
+
+```ts
+import { defaultProductionGate } from '@tangle-network/agent-eval/campaign'
+
+const holdoutScenarios = [
+  { id: 'incident:17:original', kind: 'support' },
+  { id: 'incident:17:paraphrase', kind: 'support' },
+  { id: 'incident:28:original', kind: 'support' },
+]
+
+const gate = defaultProductionGate({
+  holdoutScenarios,
+  independentUnitByScenarioId: new Map([
+    ['incident:17:original', 'incident-17'],
+    ['incident:17:paraphrase', 'incident-17'],
+    ['incident:28:original', 'incident-28'],
+  ]),
+  deltaThreshold: 0.05,
+  criticalDimensions: ['factualAccuracy'],
+})
+```
+
+This example has two independent units and cannot meet its observation minimum.
+More repetitions of these scenarios will not change that count.
+The default gate copies the map when it is constructed.
+Later mutations cannot change the grouping used by that gate.
+
+Pairing precedes aggregation.
+Candidate and baseline must contain the same full cell identifiers and the same selected judges within each cell.
+The last numeric suffix identifies a repetition, so colons within scenario identifiers remain valid.
+The implementation averages matched cells within each unit, then gives every unit equal weight.
+It refuses asymmetric cells, asymmetric selected judges, duplicate cells, non-finite scores, and missing unit assignments.
+
+Reports distinguish the following counts:
+
+| Field | Meaning |
+|---|---|
+| `n` | Paired observation units used for inference |
+| `pairedCellN` | Matched execution cells before grouping |
+| `observationUnit` | `registered` when a supplied map defines units; `cell` on an ungrouped fixed-roster path |
+| `unitIds` | The units represented in a held-out significance result |
+
+Without a map, fixed-roster significance uses execution cells as observations.
+Its uncertainty concerns independently sampled execution outcomes conditional on that roster.
+It does not establish generalization to new incidents or task families.
+Identifiers and aggregation do not establish independence; the sampling design must justify it.
+
+## Read the interval that decided
+
+`heldoutSignificance().decision` contains the test, interval, observation minimum, and promotion decision.
+Pass/fail scores use the shared paired risk-difference rule.
+Continuous scores use the shared paired bootstrap or its small-sample test.
+`bootstrap` and `medianBootstrap` remain diagnostics when another test decides.
+
+Continuous mean targets require 20 observations for bootstrap eligibility.
+Below that count, a reported exact sign-test diagnostic cannot establish a mean effect.
+Binary outcomes and explicitly requested median targets use their actual confidence-dependent observation minimum.
+The observation minimum establishes estimator eligibility; it does not establish statistical power or representative sampling.
+A zero-width bootstrap interval cannot establish improvement under the shared decision rule.
+
+Required dimensions also need sufficient observations and complete coverage.
+The default gate reports `not_evaluated` when a required dimension lacks enough units or omits matched cells or scenarios.
+`fewRuns`, `missingCellIds`, and `missingScenarioIds` preserve the missing evidence.
+An observed regression can still hold the gate while evidence remains incomplete.
+Passing the regression guard does not certify every safety property of the candidate.
+
+## Sequential decisions require a conditional-mean assumption
+
+`sequentialPairedGate` consumes one observation per scenario by default in `decide()`.
+Its `independentUnitByScenarioId` option groups related scenarios before testing and copies the mapping at construction.
+`maxN` counts these independent units.
+The report preserves both the consumed count `n` and the available counts `pairedN` and `pairedCellN`.
+
+The statistical guarantee requires each next delta's conditional expectation to remain below the registered null boundary.
+Independent sampling with that bound is sufficient.
+Shuffling an exchangeable sequence does not establish the condition.
+One random sign repeated 100 times has only one independent draw.
+After its first observation, later signs reveal no new evidence.
+
+Direct `observe(delta)` callers must perform the required aggregation themselves.
+They must also justify the sampling assumption.
+`sequentialDecide` stops candidate exploration heuristically; candidate selection and reused incumbent scores prevent a general type-I error guarantee.
+The selected candidate still requires fresh held-out evidence.
+
+## Adaptation comparisons pair whole scenarios
+
+`runAdaptationCurve` and `compareAdaptationCurves` are exported from `@tangle-network/agent-eval/rl`.
+Scenarios require explicit, unique `scenarioId` values.
+The runner validates repetitions, the demonstration grid, and finite scores in `[0,1]`.
+
+Comparisons require identical scenario cohorts and identical demonstration grids.
+Missing pairs, duplicate identities, and changing cohorts are errors.
+The comparison computes one area per scenario before applying the existing paired estimators.
+This preserves dependence across demonstration counts.
+Repeated executions improve each scenario mean without increasing the number of independent scenarios.
+
+`perK` intervals describe the curve.
+The paired area decisions determine `a_better`, `b_better`, `inconclusive`, or `insufficient_evidence`.
+Continuous area outcomes require the bootstrap minimum of 20 paired scenarios.
+Binary area outcomes use the shared score-interval rule and can support a decision with fewer observations.
+The deciding interval must remain nondegenerate.
+`inconclusive` does not establish equivalence.
+`firstPassK` describes the first observed crossing and carries no reliability guarantee.
+
+## Perturbation sensitivity is a diagnostic
+
+`runContaminationProbe` reports observed score differences and one global Wilcoxon paired test.
+Use `alpha` for that test's significance threshold.
+Per-item differences carry no p-values or q-values because the probe defines no calibrated item-level sampling null.
+
+The report retains every observed pair and lists `excludedScenarioIds` when a score floor excludes evidence.
+Summaries describe the included population.
+Fewer than four included pairs produce `pairedTest: null` while preserving measured means and medians.
+Zero included pairs produce null summaries.
+
+A significant drop can reflect changed task difficulty, broken perturbations, or contamination.
+`contaminationSuspected` therefore requests investigation; it does not identify the cause.
+The global test also relies on independent pairs and the Wilcoxon assumptions for paired differences.

package/docs/verdicts.md

@@ -1,67 +1,94 @@
-# One verdict vocabulary
+# Verdicts and certifications
 
-Every verification path in this package lands in one type: `DefaultVerdict` (`src/verdict.ts`).
-`valid` answers "did it pass", `score` answers "how well" in [0, 1], `scores` carries the per-dimension breakdown, and `certification` says WHO certified.
+`DefaultVerdict` is the shared base type for validator results.
+Campaign judges return `JudgeScore`, and release gates return `GateResult`; their fields and score scales differ.
+See [release check results](./concepts.md#release-check-results) when interpreting a campaign decision.
 
-A certification is the epistemics a bare `valid` + `score` pair cannot carry.
-A kernel-checked proof and an LLM judge can produce the same `{ valid: true, score: 1 }`; the certification is what tells them apart:
+In `DefaultVerdict`, `valid` reports whether the validator's pass criteria were met, and `score` is its aggregate in [0, 1].
+Optional `scores` and `notes` carry dimensions and explanation.
+Optional `certification` records the verification strategy, checker identity, assumptions, and evidence digest.
+A certification can accompany a failed or incomplete check.
+It does not establish that the result passed or that every required measurement exists.
 
-- `strategy` — which verification-strategy member vouches (the 10-member family with per-member failure modes: [docs/verification-strategies.md](./verification-strategies.md));
-- `checker` — the exact identity that ran, with version and content pins, so the check is re-runnable;
-- `assumptions` — every step the certificate rests on that the checker did NOT verify, named one by one;
-- `evidenceDigest` — sha-256 of the evidence artifact (`certificationEvidenceDigest`).
+The certification fields are:
 
-An absent certification is itself a statement: scored, but nothing vouches.
-No producer fakes one — a closed gate, an unexecuted proof, or an unattested checker yields an uncertified verdict, never an invented certificate.
+- `strategy`: the [verification strategy](./verification-strategies.md), with its documented failure mode.
+- `checker`: a name, version, and optional dependency pins.
+- `assumptions`: the producer's list of steps the checker did not verify.
+- `evidenceDigest`: the evidence identity; `certificationEvidenceDigest()` hashes its JSON-serialized form with canonical JSON and SHA-256.
+
+These fields record the producer's claims.
+Reproduction also requires access to the evidence, checker, dependencies, and execution environment.
+An absent certification means no verification strategy is recorded for that verdict.
 
 ## Producers
 
-Every verifier below returns a `DefaultVerdict` (usually a richer extension of it) with a produced certification.
+These result types extend `DefaultVerdict`.
+Their additional fields distinguish failed checks from incomplete or unmeasured work.
 
-| Verifier | Verdict type | Strategy | Certifies | Assumptions it names |
+| Producer | Result type | Strategy | Scope | Fields to inspect |
 | --- | --- | --- | --- | --- |
-| `MultiLayerVerifier.run` (`src/multi-layer-verifier.ts`) | `VerificationReport` | `composite` | ordered layer pipeline blend | each skipped / errored / timed-out layer |
-| `verifyCompletion` (`src/completion-verifier.ts`) | `CompletionVerdict` | the checker's own (`judge` for the LLM checker, `schema` for token recall) | task completion over produced state | lexical structural stage + the checker's attestation |
-| `evaluateTraceContract` (`src/trace-contracts.ts`) | `ContractVerdict` | `invariant` | LTLf rules over a span sequence | array ordering without timestamps; custom predicate functions |
-| `evaluateOracles` (`src/oracle.ts`) | `OracleReport` | `test` | declarative expected-outcome assertions | — (the oracle set is its own answer key) |
-| `replayVerify` (`src/trajectory-replay/verify.ts`) | `ReplayVerdict` | `replication` | recorded failure reproduced (and fix vanished) under re-execution | unadjudicated prefix steps; truncated prefix; returncode-only signature |
-| `verifyFindings` (`src/trajectory-replay/findings.ts`) | `VerifyFindingsRun` | `replication` | a batch of analyst findings under executed replay | not-replayable findings leave the denominator |
-| `gradeRepairRow` (`src/trace-repair/grade.ts`) | `RepairRowResult` | `test` | a proposed repair against the row's held-out suite (pins: suite + policy digests) | vacuous reproduction gate; prefix divergence |
-| `runEquivalenceCheck` + `equivalenceVerdict` (`src/verification-strategy.ts`, `src/verdict.ts`) | `DefaultVerdict` | the spec's member (`proof-kernel` in the pilot) | two blind formal statements are equivalent | arms self-declare blindness |
-
-Two producers certify conditionally, on purpose:
-
-- `gradeRepairRow` certifies only a `measured` outcome — a funnel gate that closed before the suite ran has nothing to vouch for.
-- `verifyFindings` certifies only when at least one proof executed — a batch where nothing was replayable measured nothing.
-
-## Reading a score out of a judge
-
-A model judge emits one grade per dimension. Discrete grades tie: two candidates that both score `8` carry no ranking signal between them, and a best-of-N selection then picks arbitrarily.
-
-`llmJudge({ scoring })` chooses how the number is read:
-
-| `scoring` | What it reads | Requires |
+| `MultiLayerVerifier.run()` | `VerificationReport` | `composite` | Ordered verification layers | `layers`, `allPass`, and optional `taskScore` |
+| `verifyCompletion()` | `CompletionVerdict` | The supplied checker's strategy | Completion requirements matched against produced state | Requirement evidence, `correct`, and `unmeasured` |
+| `evaluateTraceContract()` | `ContractVerdict` | `invariant` | Temporal rules over recorded spans | Rule results and assumptions about ordering and predicates |
+| `evaluateOracles()` | `OracleReport` | `test` | Declared expected-outcome assertions | `results`, `passCount`, and `failCount` |
+| `replayVerify()` | `ReplayVerdict` | `replication` | Failure reproduction and an optional fix under re-execution | Prefix fidelity, signature matches, and both execution arms |
+| `verifyFindings()` | `VerifyFindingsRun` | `replication` | Analyst findings checked through replay | `executions`, `counts`, and individual verifications |
+| `gradeRepairRow()` | `RepairRowResult` | `test` | A repair against the admitted row's held-out suite | The grade's outcome and funnel evidence |
+| `equivalenceVerdict(record)` | `DefaultVerdict` | The record's strategy | Formal-statement equivalence checked by `runEquivalenceCheck()` | Whether the obligation was proved, refuted, or unresolved |
+
+Certification is conditional for several producers:
+
+- `verifyCompletion()` includes it when the supplied checker provides an attestation.
+  Inspect requirement evidence to see which checks were assessed.
+- `equivalenceVerdict()` includes it for a proved or refuted obligation with an evidence digest.
+  A certified refutation has `valid: false`.
+- `gradeRepairRow()` includes it only for a `measured` grade.
+- `verifyFindings()` includes it only when at least one replay execution ran.
+  Non-replayable findings remain in `counts` and are excluded from the score's denominator.
+  A score of 1 can therefore coexist with `valid: false` when some findings were not replayable.
+
+Some result shapes use `score: 0` when no task measurement is available.
+For `VerificationReport`, use the presence of `taskScore` to identify a complete task measurement; `blendedScore` can describe a partial panel.
+An empty oracle set has certification metadata but returns `valid: false` and no executed oracle results.
+Read these completeness fields before using scores as task labels or release evidence.
+
+## Reading a score from a model judge
+
+A tied dimension score supplies no ordering between candidates.
+`llmJudge({ scoring })` controls how the dimension score is read:
+
+| `scoring` | Measurement | Requirement |
 |---|---|---|
-| `{ method: 'sampled' }` (default) | the grade the model emitted | nothing |
-| `{ method: 'expectation', whenUnavailable }` | the expected grade over the integer grades the model considered at the score token | `scale: 'ten'` and a provider that returns log probabilities |
-
-Expectation scoring asks the provider for `logprobs` with `top_logprobs`, finds the token that carried each dimension's grade, and averages the integer grades in that token's probability window, weighted by probability. Two answers that both sample `8` separate by how much mass sat on `7` versus `9`.
+| `{ method: 'sampled' }` (default) | The emitted grade | A valid grade response |
+| `{ method: 'expectation', whenUnavailable }` | A probability-weighted grade from returned token alternatives | `scale: 'ten'` and provider log probabilities |
 
-It needs one integer in one token, which is why `scale: 'ten'` is required: a `[0,1]` float is several tokens, and no single position carries its distribution. A grade that did not land in exactly one token — a two-token `10` — is refused rather than approximated.
+With `scale: 'ten'`, the model emits grades from 0 to 10; `llmJudge()` divides them by 10 before returning dimensions and composite.
+Expectation scoring finds each grade's token, keeps valid integer alternatives, and renormalizes their returned probabilities before averaging.
+Its distribution is limited to those returned alternatives.
+Additional precision alone does not establish better calibration or ranking accuracy.
 
-`whenUnavailable` decides what happens when the provider returns no log probabilities, or the grade spans tokens:
+Each emitted grade must occupy one integer token for expectation scoring.
+A split `10`, missing grade token, or unavailable log probabilities invokes `whenUnavailable`:
 
-- `'fail'` throws, and the campaign records a failed cell.
-- `'sampled'` reads the emitted grade instead.
+- `'fail'` throws; the campaign records a judge failure.
+- `'sampled'` uses the emitted grades for that judge result.
 
-`JudgeScore.scoringMethod` reports what actually produced the number, so a declared expectation run that fell back reads `'sampled'` and stays auditable. `JudgeScore.distribution` carries the probability mass per grade, and is present only for an expectation score. Panels are unchanged: `ensembleJudge` consumes the composite either way.
+`JudgeScore.scoringMethod` is present when `scoring` was explicitly configured and records the method used.
+An expectation request that falls back reports `'sampled'`.
+When `scoring` is omitted, sampled scoring is the default and this metadata field is absent.
+`JudgeScore.distribution` is present only for expectation scoring and contains the normalized probabilities over returned integer alternatives.
+`ensembleJudge()` consumes the resulting composite through its usual interface.
 
-Whether a given endpoint returns `logprobs.content` is a property of that provider and model, not of this package. `LlmCallResult.logprobs` is `null` when the provider returned none — never an inferred distribution. See `evidence/records/judge-logprob-wire-support.json` for the current verification state of that wire behavior.
+Provider and model support determines whether log probabilities are available.
+`LlmCallResult.logprobs` is `null` when none were returned.
+The [recorded wire check](../evidence/records/judge-logprob-wire-support.json) documents the endpoints and conditions that were inspected.
 
-## Consuming a certification
+## Interpreting certification limits
 
-Read `certification.strategy`, then weigh the member's documented failure mode — `VERIFICATION_STRATEGIES[strategy].failureMode` carries it at runtime.
-"Certified" is never one bit: a `judge` certificate is Goodhart-gameable, a `test` certificate covers only its suite, a `composite` certificate can hide which member carried the score.
-The assumptions list is the honest remainder; an empty list is the producer's explicit claim that nothing was left unverified, not a default.
+Read `VERIFICATION_STRATEGIES[strategy].failureMode` alongside the evidence and assumptions.
+A judge can reward misleading output; tests cover their suite; a proof can establish the wrong formal statement for the intended task.
+A composite certification requires inspection of its component results.
+An empty assumptions list is the producer's declaration, not independent verification that no assumptions remain.
 
-Related docs: [verification-strategies.md](./verification-strategies.md) (the family and the equivalence protocol), [trace-repair-grader.md](./trace-repair-grader.md), [trajectory-replay.md](./trajectory-replay.md).
+See [verification strategies](./verification-strategies.md), [repair grading](./trace-repair-grader.md), and [trajectory replay](./trajectory-replay.md) for the corresponding execution contracts.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tangle-network/agent-eval",
-  "version": "0.180.0",
+  "version": "0.182.0",
   "description": "Evaluate and improve AI agents from runs, traces, judges, and feedback. Compare candidates, cluster failures, measure lift, and gate releases.",
   "homepage": "https://github.com/tangle-network/agent-eval#readme",
   "repository": {