@tangle-network/agent-eval 0.150.2 → 0.161.1

package/docs/design/statistics-decisions.md

@@ -90,7 +90,7 @@ These 17 matched their references with zero mismatches and stay as they are, pin
 | `wilson` | keep-with-CI-oracle | Matched the closed form to `< 1e-8` on all 13 `(successes, n)` pairs including `0/1`, `10/10`, `1/1000`, `0/0`. Correctly asymmetric at the boundaries: `wilson(0, 10)` returns `[0, 0.27753280]`. |
 | `passAtK` | keep-with-CI-oracle | Chen et al. 2021 unbiased estimator, exhaustively verified for every `(n, c, k)` with `n = 1..6` against exact integer `math.comb`: zero mismatches. Stable at scale (`n=1000, c=3, k=100` agreed to `1.11e-16`). `passAtK(10, 3, 5) = 0.9166666667`. |
 | `corpusInterRaterAgreement` | keep-with-CI-oracle | The ICC(2,1) it surfaces matched a hand-derived two-way random-effects ANOVA reference to `< 1e-9` on 5 matrices, including the inverted case at `−1.959459`. This is a genuinely different and correct computation from `interRaterReliability`: it pivots to a proper items × judges matrix and delegates to `continuousAgreement`. Its fail-loud contract behaves — empty input, fewer than two judges, fewer than two common items, duplicate records, and absent dimensions all throw `ValidationError`. |
-| `eProcess` | keep-with-CI-oracle | Betting test-martingale. Empirical type-I over 4,000 sequences of 200 observations at `α = 0.05`: 2.40 % at the null boundary, 0.00 % in the null interior, 1.33 % on continuous uniform — all inside Ville's bound. Power at `E[x] = 0.7` is 99.775 %. The predictability invariant holds: the first update leaves wealth at exactly 1. |
+| `eProcess` | keep-with-CI-oracle | Betting test-martingale. Empirical type-I over 4,000 sequences of 200 observations at `α = 0.05`: 2.40 % at the null boundary, 0.00 % in the null interior, 1.33 % on continuous uniform — all inside Ville's bound. Power at `E[x] = 0.7` is 99.775 %. The predictability invariant holds: the first update leaves wealth at exactly 1. Restart reconstruction: `state()` carries the running sums (`sumX`, `varSum`) next to wealth and n, and `eProcess({ resume })` rebuilds the process from that snapshot; a process interrupted at any n and resumed from a JSON round-trip of its state reproduces the uninterrupted wealth sequence and decision exactly (`sequential-eprocess.test.ts`, interruptions at n = 0, 1, 7, 30, 119 and at the crossing), and `sequentialPairedGate({ resume })` does the same for the gate's observe-stream. A snapshot recorded under other parameters, or one whose fields cannot all be true at once, is refused. |
 | `holm` | keep-with-CI-oracle | Matched `statsmodels.multipletests` to `1e-9` on a 6-value reference vector, and uses `≤` at the boundary, which is the correct rule. It also validates both `alpha` and the p range, which `bonferroni` does not. |
 | `zQuantile` | keep-with-CI-oracle | Acklam inverse-normal, structurally independent of `normalCdf`. This independence is why the sample-size functions were untouched by the CDF defect, and why `mcnemarPower`'s failure to invert `mcnemarRequiredN` was a valid detector of it. |
 | `mcnemarRequiredN` | keep-with-CI-oracle | Matched the Lachin closed form exactly on all 4 parameter sets: `234`, `77`, `155`, `Infinity`. Unchanged by the CDF fix. Round-trip against the repaired `mcnemarPower` now holds across 16 configurations: power at `requiredN` meets the target with overshoot `≤ 0.0049`, and power at `requiredN − 1` is below target in all 16. |
@@ -269,3 +269,91 @@ Any bootstrap interval recorded from a release at or before 0.133.0 through `ana
 Re-running it against the old release will not give the same interval. From this release the seed is derived from the data when the caller supplies none, so it is reproducible either way — but an interval recorded earlier cannot be reconstructed.
 
 Mirror this section into `CHANGELOG.md` at the release that carries the fix, with the affected version range and the re-check bands, so a consumer who never reads this file still gets the notice.
+
+## Random-source audit
+
+Every `Math.random` reference in `src/`, what it feeds, and whether a reported number depends on it.
+The rule this audit enforces: **no result-bearing draw may come from an unseeded source.**
+A result-bearing draw is one whose value reaches a reported statistic, an interval, a gate verdict, or an allocation a caller acts on.
+
+The canonical seeded generator is `mulberry32` in `src/statistics/random.ts`, reached through `makeRng(seed, ...series)` in `src/statistics/internal.ts`.
+`makeRng` derives the seed from the observations when the caller supplies none, so an unseeded call is still reproducible: same input, same interval.
+That is why the option stays optional — a required seed would push the caller into inventing one, and an invented seed is not more honest than a derived one.
+
+### Result-bearing draws — all seeded
+
+| Site | Formula | Consumer | Seed source |
+| --- | --- | --- | --- |
+| `statistics/internal.ts` `makeRng` | mulberry32 | every resampling path in `src/statistics` | caller seed, else FNV-1a over the IEEE-754 bytes of the observations |
+| `statistics/descriptive.ts` `confidenceInterval` | percentile bootstrap | reported interval | `makeRng(opts.seed, scores)` |
+| `statistics/paired-tests.ts` `pairedBootstrap` | paired percentile bootstrap | promotion intervals | `makeRng(opts.seed, deltas)` |
+| `statistics/rank-tests.ts` `mannWhitneyU`, `wilcoxonSignedRank` | permutation null | exact rank-test p | `makeRng(opts.seed, …)`, order-independent for the symmetric two-sample case |
+| `promotion-gate.ts` | bootstrap over the two arms | ship / hold verdict | `options.seed`, else `hashSeed(baseline, candidate)` |
+| `held-out-gate.ts` | delegates to `pairedBootstrap` | held-out gate verdict | caller seed, else the paired observations |
+| `summary-report.ts` `bayesianBootstrapMeanSamples` | Dirichlet-weight bootstrap | reported posterior and gain histogram | `makeRng(seed, deltas)` |
+| `meta-eval/rubric-predictive-validity.ts` `bootstrapCi` | percentile bootstrap of Pearson r | rubric verdict (`load_bearing` / `informative`) | `makeRng(input.seed, xs, ys)` |
+| `meta-eval/correlation-study.ts` `bootstrapPearsonCi` | percentile bootstrap of Pearson r | reported correlation CI | `makeRng(options.seed, xs, ys)` |
+| `rl/active-curriculum.ts` `thompsonCurriculum` | Beta posterior sampling | sample-budget allocation | `makeRng(opts.seed, scores)` |
+| `rl/adaptation-eval.ts` `compareAdaptationCurves` | bootstrap of per-k mean deltas | per-k CI and AUC delta | `makeRng(opts.seed, a-means, b-means)` |
+| `campaign/gates/sequential.ts` | seeded Fisher-Yates over paired deltas | exchangeability guard before the e-process | `shuffleSeed`, default 1337, data-independent by construction |
+
+### Draws that reach no result
+
+These stay on `Math.random`. Each produces an identifier or a retry delay; no reported number reads them, and seeding them would make two concurrent runs collide on the same identifier.
+
+| Site | Formula | What it feeds |
+| --- | --- | --- |
+| `llm-client.ts:1050` | `Date.now().toString(36)` + random suffix | per-call identifier in the ledger tag |
+| `trace/emitter.ts:350` | same | span identifier |
+| `builder-eval/builder-session.ts:255` | same | builder session identifier |
+| `campaign/worktree/index.ts:697` | same, 4-character suffix | worktree branch name |
+| `matrix/runner.ts:67` | 8 hex characters | matrix run identifier |
+| `adapters/http.ts:152,162` | `2^attempt * 200 + random * 200` | retry backoff jitter |
+| `hosted/client.ts:128,138` | same | retry backoff jitter |
+
+### What changed in this pass
+
+Five byte-identical private copies of mulberry32 existed alongside the canonical one, in `meta-eval/rubric-predictive-validity.ts`, `rl/active-curriculum.ts`, `rl/adaptation-eval.ts`, `summary-report.ts`, and `promotion-gate.ts`.
+Four of them fell back to `Math.random` when the caller passed no seed, so four reported statistics were silently non-reproducible.
+`meta-eval/correlation-study.ts` called `Math.random` directly inside its bootstrap and had no seed option at all.
+All six now route through `makeRng`, and `correlationStudy` gained the `seed` option its sibling already had.
+`Math.random` references in `src/` fell from 19 to 10, and none of the remaining ten is result-bearing.
+
+## Test and documentation classification
+
+The audit in #411 counted assertions that prove nothing, module mocks, and conditional skips. The disposition:
+
+**No-throw assertions.** 66 `expect(...).not.toThrow()` sites across 36 files. They split in two:
+
+- *A guard's accept case.* `assertLlmRoute`, `assertNoHiddenLeak`, `assertCapabilityHeadroom`, `assertMultishotShotResult`, `assertMatchedMethodLimits`, `assertGateReport`, `assertAnalystBenchmarkObservation`, `assertDeterministicOracle`, `assertDenominatorIntact`, and `assertJsonValue` return nothing. For these the no-throw IS the assertion: it pins the false-positive boundary of a fail-closed gate, and a gate that refuses a legal input is as broken as one that admits an illegal one. Each sits next to a `toThrow` sibling that proves the refusal. Kept.
+- *A value-returning call.* `validateRunRecord` and friends return the validated value, so the no-throw discarded the only thing worth checking. Converted to assert the returned value.
+
+**Module mocks.** 5 sites, each now carrying a one-line justification in its file header: four wrap a `node:` builtin to reproduce a race or an absent native binding that a real filesystem cannot be timed into (`trace/store.test.ts`, `analyst/benchmark-verification-artifacts.test.ts`, `analyst/benchmark-command-public-data.test.ts`, `rollout/readers/opencode-sqlite-lazy.test.ts`); one replaces the provider call in the wire handler test, which is about request validation and error mapping (`tests/wire/handlers.test.ts`).
+
+**Conditional skips.** 6 sites, all environment-gated, none silent:
+
+| Site | Gate | Why |
+| --- | --- | --- |
+| `command-runner.test.ts:148` | `process.platform === 'win32' \|\| getuid() === 0` | POSIX permission bits do not constrain root or Windows |
+| `ledger-core/trusted-head.test.ts:473` | same, as `runIf` | the same permission assumption |
+| `analyst/benchmark-command-public-data.test.ts:72` | `process.platform === 'win32'` | POSIX path semantics |
+| `analyst/kinds/skill-usage.test.ts:198` | `SKILL_USAGE_REAL` | needs a real skill corpus on disk |
+| `tests/campaign/gepa-official-integration.test.ts:22` | `AGENT_EVAL_TEST_PYTHON` | needs the official GEPA python environment; CI sets it |
+| `tests/campaign/skillopt-official-integration.test.ts:22` | `AGENT_EVAL_TEST_PYTHON` | needs the official SkillOpt python environment; CI sets it |
+
+The two python-gated files run in CI, which installs the environment; the platform gates never skip on CI's Linux runner.
+
+## Runnable-fence sweep
+
+A one-time pass, not a build gate. The 73 Markdown files under `docs/`, `examples/`, the two READMEs, `CLAUDE.md`, and the maintainer skill hold 66 fences marked `ts`.
+Each was extracted to a file and compiled with the examples compiler options, with `@tangle-network/agent-eval` and every subpath mapped to `src/`, so the check reads the surface this repository ships rather than the version installed in `node_modules`.
+Reading the installed copy is the trap: it reports every export added since the last release as missing, which is how a fence check produces false alarms and gets switched off.
+
+Result:
+
+- **3 fences were not TypeScript** and are now marked `text`: an object fragment in `campaign-proposers.md`, a call elided as `{ ... }` in `feedback-trajectories.md`, and a method sketch in `examples/benchmarks/README.md`.
+- **2 fences imported a path that cannot resolve** — `@tangle-network/agent-eval/../src/trace-repair` in `trace-repair-admission.md` and `trace-repair-continuation.md` — now `@tangle-network/agent-eval/trace-repair`.
+- **0 of the remaining 63 name an export that does not exist.** Every symbol a fence imports is on the current surface.
+- The remaining compiler complaints are inherent to an excerpt: a name declared in the prose around it, a shorthand property, top-level `await` outside a module. Three fences import a deliberate placeholder (`your-text-optimizer`, `./my-engine`), and three import `@tangle-network/agent-eval/adapters/http`, which `distributed-driver.md` already declares as source-only and unpublished.
+
+There is no fence checker in `scripts/` and no CI job. Compiling illustrative snippets fails builds for prose edits, and the failure mode it catches — a doc naming a symbol that no longer exists — is what the export census and review already cover. Repeat this pass by hand after a large surface change.

package/docs/eval-surface-map.md

@@ -43,6 +43,20 @@ is `runCampaign` with capture inverted; `runAgentMatrix` is the scheduler undern
 Merging any two of these conflates distinct mental models (measure ≠ search ≠
 release-gate). Keep them separate; pick by the table.
 
+## Failed cells: receipts and bounded retry
+
+A failed cell writes `<cell>/failure-receipt.json` before the campaign can abort.
+The receipt records the stage (`dispatch` or `judge`), the serialized error, the exact cell result, and the settled cost of that cell.
+`abortOnCellError: true` stops the campaign on the first failed cell; the default keeps the remaining schedule running and returns the failed cell.
+
+`cellRetry: { attempts, retryable }` opts in to bounded in-run retry.
+A failed attempt that `retryable` accepts is dispatched again in the same slot (same `cellId`, same seed) until it succeeds or `attempts` is exhausted.
+Use `transientDispatchFailure()` as the predicate to retry only dispatch-stage transport failures (502/503/504, dropped streams, admission rejections) and never judge-stage failures.
+Every attempt charges the shared cost ledger, so the final cell's `costUsd` and `costCallIds` cover all attempts.
+A retried attempt keeps its receipt at `<cell>/failure-receipt.attempt-<n>.json`, and the final cell records the retry count as `retryAttempts`.
+With `abortOnCellError`, the abort fires only when a cell's final attempt fails.
+Without `cellRetry`, a failed cell is final: one transient 503 leaves campaign coverage incomplete, and `runImprovementLoop` then refuses the holdout comparison.
+
 ## Produced-state grading: there is NO persona-dispatch wrapper
 
 To grade what an agent actually **produced** (filed the proposal, wrote the

package/docs/experiment.md

@@ -35,7 +35,7 @@ import {
   sealExperiment,
 } from '@tangle-network/agent-eval/experiment'
 
-const sealed = await sealExperiment(spec)      // canonicalize + sha256 over the whole tree
+const sealed = await sealExperiment(spec)      // RFC 8785 + sha256 over the whole tree
 const registered = await openSealedExperiment(sealed) // verifies the digest first
 
 const admission = registered.admit(rows)       // funnel + survivors, from the sealed rule
@@ -44,6 +44,23 @@ const halt = registered.halt([gate])           // refuse-spend fires before any
 const outcome = registered.decide(quantities)  // the sealed table; non-total tables throw
 ```
 
+#### Digest schemes and retention
+
+A seal and a signed `HypothesisManifest` are durable records: each is written
+once and verified later, possibly by a different release. Both carry an `algo`
+field that names the digest scheme, and verification selects the encoder from
+that field.
+
+| `algo` | Serialization | Status |
+|---|---|---|
+| `sha256-rfc8785` | RFC 8785 canonical JSON, from `ledger-core/canonical` | What `sealExperiment` and `signManifest` write |
+| `sha256-content` | key-sorted `JSON.stringify` | Read-only. Records written before the RFC 8785 scheme carry it, or carry no `algo` at all, and still verify |
+
+The `sha256-content` encoder is private to the module that verifies with it and
+is unreachable from any path that writes a digest. Retire it once no record
+carrying that tag needs to verify; until then, deleting it would make those
+records unverifiable rather than invalid.
+
 `openSealedExperiment` is the only execution surface.
 A rule that is not in the sealed spec cannot run; a rule that is cannot run differently.
 
@@ -93,7 +110,7 @@ The statistical machinery underneath is re-exported from its existing homes; thi
 | `heldoutSignificance`, `pairHoldout` | `src/campaign/gates/statistical-heldout.ts` |
 | `paretoSignificanceGate`, `buildEvidenceVector` | `src/campaign/gates/promotion-policy.ts` |
 | `pairArms`, `comparePairedArms`, `pairRunRecords` | `src/paired-arms.ts` |
-| `canonicalize`, `hashJson`, `signManifest`, `verifyManifest`, `HypothesisManifest` | `src/pre-registration.ts` |
+| `hashJson`, `manifestContentDigest`, `signManifest`, `verifyManifest`, `HypothesisManifest` | `src/pre-registration.ts` |
 | `ExperimentTracker` (run ledger with KEEP/ITERATE/NOISE/REGRESSION) | `src/experiment-tracker.ts` |
 
 `HypothesisManifest` stays as the lightweight single-metric registration; `sealExperiment` is the full-design registration.

package/docs/feedback-trajectories.md

@@ -205,7 +205,7 @@ of only "was the final answer good?"
 `controlRunToFeedbackTrajectory` turns a finished control-loop run into a
 trajectory:
 
-```ts
+```text
 const run = await runAgentControlLoop({ ... })
 const trajectory = controlRunToFeedbackTrajectory(run, {
   projectId: 'coding-agent',

package/docs/multishot-golden-records.md

@@ -45,13 +45,13 @@ It refuses an `only` id the catalog does not hold, so a stale id after a rename
 An external conversation engine plugs into the matrix through one seam: `RunMultishotMatrixOptions.runShot`.
 The matrix runs that engine for every cell and reads its result through `MultishotCellOutput`, so the same golden checks grade any engine that implements the seam.
 
-The matrix pair is `assertMultishotMatrixGoldenScenario` / `checkMultishotMatrixGoldenScenario`; both take a `runDir` the engine may write into, and both install a deterministic judge wire on `globalThis.fetch` for the duration of the run.
-That wire is process-wide, so run matrix checks serially within one process and keep other fetch traffic out of it.
-Both rules are enforced, not just documented: a second concurrent install throws, and the wire fails loud on any request it does not recognise rather than answering it.
+The matrix pair is `assertMultishotMatrixGoldenScenario` / `checkMultishotMatrixGoldenScenario`; both take a `runDir` the engine may write into.
+Each judge carries a scripted `MultishotTransport`, exactly like the agent and driver legs, so a matrix check owns no process-wide resource and two checks may run at once.
+The judge transport fails loud on a system prompt it does not recognise rather than answering it.
 
 ## Determinism rules
 
-Every scenario is a closed system: scripted transports, scripted tool executors, a fixed persona and profile, fixed token budgets.
+Every scenario is a closed system: scripted transports on every leg including the judges, scripted tool executors, a fixed persona and profile, fixed token budgets.
 No network, no random number, and no clock in a COMPARED field — the fixture envelope carries a `recordedAt` stamp as provenance, and nothing compares it.
 Matrix cells run one at a time, so the request ledger is a property of the conversation engine rather than of how two engines interleave their microtasks.