@tangle-network/agent-eval 0.180.0 → 0.181.0

package/docs/evaluation-integrity.md

@@ -0,0 +1,220 @@
+# Evaluation claims and automated improvement
+
+Use reusable evaluations to guide development and select candidates.
+Declare stronger evidence requirements when a result must support a broader claim.
+`selfImprove()` returns its selected surface and measured lift even when its release gate remains inconclusive.
+
+The package separates three questions:
+
+| Question | Evidence | Public entry |
+| --- | --- | --- |
+| Did this change help on these cases? | Paired scores, failures, cost, and case coverage. | `defineAgentEval()` or `selfImprove()` from `/contract`. |
+| Does the improvement extend to new tasks? | Representative independent tasks and an appropriate comparison; a useful effect when making an improvement decision. | Optional `claim` on campaign comparisons; registered rules from `/experiment`. |
+| Can fresh final evidence support this adaptive decision? | A frozen comparison, retained access boundaries, and a durable exposure record. | Optional `finalEvidence` on the same comparison. |
+
+These controls reuse the existing execution path, paired estimators, sealed experiments, and locked journal.
+They do not add another optimizer or agent runner.
+
+## Declare the comparison without consuming data
+
+```ts
+import { defineEvaluationClaim } from '@tangle-network/agent-eval/experiment'
+
+const claim = defineEvaluationClaim({
+  use: 'comparison',
+  population: {
+    id: 'support-incidents',
+    description: 'Support incidents from the deployed product',
+  },
+  samplingFrame: 'A random sample of incidents from the declared collection window',
+  independentUnit: 'source.incidentId',
+  generalization: 'new-units',
+  minimumEffect: 0.05,
+})
+```
+
+Pass `claim` to `selfImprove()`, `runImprovementLoop()`, or `compareOptimizationMethods()`.
+It remains independent of final-evidence storage.
+`minimumEffect` is optional because development reports and absolute-rate measurements need not test an improvement threshold.
+When omitted, the default `selfImprove()` gate uses a 0.05 gain threshold; method comparison uses 0.
+Custom gates choose their own thresholds.
+
+`independentUnit` names a field path in each scenario or evidence row.
+Variants from one incident must carry the same source identity.
+For new-unit claims, automatic `selfImprove()` partitions keep those variants together.
+Fixed-roster claims retain source units in reports without requiring disjoint sources between development partitions.
+Explicit partitions for new-unit claims must not share source units between development and final evaluation.
+
+The default self-improvement gate averages paired cells within registered units.
+Method comparison first averages repetitions within each scenario, then averages scenarios within each source unit.
+It weights source units equally.
+Results retain `scenarioScores`, `unitScores`, `units`, and `pairedCellN` so callers can inspect each denominator.
+Custom improvement gates receive raw cell scores and scenarios.
+Pass the claim into your gate's configuration and apply its grouping and decision rules there.
+
+`fixed-roster` describes the specified cases.
+Repeated executions can measure execution variability on that roster.
+They do not establish task diversity or performance on unseen users.
+Claim metadata records the intended scope; it does not authenticate sampling or turn an exploratory result into certification.
+
+## Interpret small and inconclusive results
+
+There is no universal task count that proves an improvement.
+The effect, outcome type, dependence, confidence level, and decision procedure determine what the evidence supports.
+
+Paired binary decisions use the shared score interval.
+At nonnegative gain thresholds, the exact discordance check can also veto promotion.
+Negative thresholds ask whether a regression stays within a tolerance and do not use that veto.
+A sufficiently large binary gain can pass with fewer than 20 independent pairs.
+Continuous mean decisions require the existing bootstrap path's 20-pair eligibility threshold.
+That implementation threshold does not establish adequate power or guarantee interval coverage for every distribution.
+The smaller-sample sign test answers a different question about directional or median change.
+
+Method rankings describe observed lift.
+`favored: null` means the evidence does not establish a favored method; it does not establish equivalence.
+Each score and pairwise contrast retains its full `decision`, including the estimator, threshold, minimum, and sufficiency.
+An inconclusive gate leaves the selected candidate available for further development or a narrower evaluation.
+
+For a design that matches its outcome model, `clusteredPower()` simulates power at the declared `minimumEffect`.
+A registered `power-floor` gate checks a supplied power curve against its minimum effect and target power.
+These optional design checks do not run automatically when you pass a `claim`.
+High power at a much larger effect cannot substitute for power at the improvement that matters.
+
+## Opt into fresh final evidence
+
+```ts
+import { openFinalEvidenceLedger } from '@tangle-network/agent-eval/experiment'
+
+const finalEvidence = {
+  ledger: openFinalEvidenceLedger({ path: '.agent-eval/final-evidence.jsonl' }),
+  requestId: 'support-comparison-2026-09-13',
+  evaluatorDigest, // Content identity of the actual evaluator and its configuration.
+}
+
+// Pass both claim and finalEvidence to the existing comparison entrypoint.
+```
+
+Ordinary regression and development evaluations remain reusable.
+Supply this policy when freshness is part of the evidence supporting a particular final comparison.
+It requires a comparison or certification claim and measured final execution.
+
+Comparison entrypoints capture judge configuration and callbacks before search or final exposure.
+The host must keep callback receiver and closed-over state stable throughout the comparison.
+
+The campaign reserves source units before candidate search.
+It records exposure before dispatching the baseline, selected candidates, or an optional neutralized control.
+Failed or interrupted final execution still consumes that evidence.
+An already exposed request cannot start another final measurement.
+
+The ledger permits exact retries of reservation and exposure writes.
+Campaign entrypoints refuse a replayed exposure so competing workers cannot each start a new measurement.
+Read retained campaign artifacts after exposure; a fresh request ID does not restore freshness.
+
+Source unit identities are unique across the shared ledger, including across population labels.
+The ledger also rejects the same dataset digest under another request.
+Use one persistent ledger for related decisions and preserve stable source identities.
+Opening another empty ledger or inventing new lineage identities cannot establish independent evidence.
+
+`FinalEvidenceLedger` returns typed outcomes.
+Inspect `succeeded` before reading `value`.
+Failures distinguish conflicting use, invalid input, and unavailable or damaged storage.
+Campaign errors preserve these categories through `FinalEvidenceError.kind`.
+
+The filesystem implementation uses the existing hash-chained journal, process locks, durable writes, and required trusted head.
+Preserve both the journal and its `.head` file.
+The head detects truncation while it remains trusted.
+An actor who can replace both files can replace the recorded history.
+
+The host owns answer-file permissions, model context, credentials, and author/evaluator separation.
+The ledger records exposure; it cannot prove that earlier undisclosed access never occurred.
+
+## Seal the rule and measured field
+
+Attach the same `claim` to `defineExperiment()` before calling `sealExperiment()`.
+Cluster intervals register both the source unit and measured field:
+
+```ts
+const interval = {
+  kind: 'cluster-bootstrap' as const,
+  clusterBy: 'source.incidentId',
+  value: 'pairedDelta',
+  resamples: 2000,
+  seed: 7,
+  level: 0.95,
+  method: 'percentile' as const,
+}
+
+// Register interval in the experiment's intervals map before sealing.
+// Then execute registered.interval('lift', { kind: 'rows', rows }).
+```
+
+For new-unit claims, each cluster interval's `clusterBy` must equal `claim.independentUnit`.
+Registered binomial intervals require one unique `unitId` per trial for new-unit claims.
+Opened seals capture validated rules before asynchronous execution.
+Caller mutation cannot change the opened experiment's rules.
+
+Only current canonical digest schemes can execute.
+Retain historical artifacts with their original identities; re-register current work under the supported format.
+See [registered experiments](./experiment.md) for the complete rule language.
+
+## Audit an evaluator's errors
+
+Use `auditEvaluator()` from `/meta-eval` when admitting a new checker or model judge.
+Provide actual judgments of independently verified good and bad controls.
+Each observation names its source unit, evidence reference, expected decision, observed decision, and development exposure.
+This audit is separate from `selfImprove()`; the caller decides whether to require admission before search or release.
+
+The audit measures false acceptance and false rejection separately.
+A source unit fails a class when any variant in that class is misjudged.
+Repeated variants increase case coverage without increasing the independent-unit count.
+If a source appeared during evaluator development, every supplied variant from that source is excluded from fresh audit evidence.
+
+The audit uses exact binomial bounds and adjusts the two intervals for simultaneous confidence.
+Unknown judgments remain visible and contribute their most adverse possible outcomes to each upper bound.
+Admission is possible when both worst-case upper bounds meet policy.
+If unresolved outcomes affect the measured error rate, that rate is `null`.
+An always-accept checker fails false acceptance; an always-reject checker fails false rejection.
+
+Reports retain inputs, source coverage, exclusions, unknowns, limits, and content digests.
+The declared audit authority must differ from the evaluator author.
+Different identifiers alone do not prove independence; the host must enforce and record separation.
+Audit cases must represent the stated population and a consistent control-generation procedure.
+Changing the number or kind of variants changes the meaning of an any-variant error rate.
+
+`auditEvaluator()` measures supplied judgments.
+It does not execute models or automatically approve a deployment.
+For outcome associations and direct score calibration, use [the outcome-validity tools](./outcome-validity.md).
+
+## Test whether self-improvement is useful
+
+These integrity checks establish execution and measurement behavior.
+They do not establish that a particular optimizer improves agents across domains.
+The [historical evidence audit](./design/self-improvement-evidence-audit.md) records prior gains, nulls, regressions, and their limits.
+
+For a benefit experiment, define the user behavior and useful effect before search.
+Compare the starting agent, a direct edit or simple search baseline, and the proposed improvement method at equal actual resources.
+Give every method the same allowed preparation, feedback, tools, and candidate surface.
+Retain all attempts, costs, failures, selected candidates, and final comparisons.
+
+Before interpreting a null result, verify that candidate generation executed and the evaluator distinguishes plausible improvements from regressions.
+Measure improvement under the conditions where the method claims an advantage.
+Use fresh tasks when the conclusion concerns unseen tasks.
+Treat a result on a fixed product workflow as evidence for that workflow.
+Repeat across distinct domains before making a broad claim.
+
+The [offline example](../examples/evaluation-integrity/) exercises these public APIs and exports its report without paid calls.
+It verifies the integration with deterministic fixtures; it is not an optimizer-benefit study.
+
+## Source and design rationale
+
+The book motivates the distinctions; the API and policies are project design choices.
+
+| Source | Applied idea |
+| --- | --- |
+| [Chapter 4: purposes of holdout](https://mlbenchmarks.org/04-holdout-method.html#whats-the-holdout-method-for) | Development feedback, selection, and capability measurement require different evidence. |
+| [Chapter 3: detecting differences](https://mlbenchmarks.org/03-detecting-differences.html#comparing-similar-models) | Pair comparisons and evaluate precision against the actual effect and independent observations. |
+| [Chapter 5: test-set reuse](https://mlbenchmarks.org/05-test-set-reuse.html) | Preserve development feedback while tracking adaptive final-data exposure. |
+| [Chapter 11: confounded evaluations](https://mlbenchmarks.org/11-evaluating-language-models.html#confounded-evaluations) | Give methods comparable preparation before judging their adaptation potential. |
+| [Chapter 14: judge agreement](https://mlbenchmarks.org/14-evaluation-frontier.html#agreement-alone-is-not-enough) | Measure consequential evaluator errors; agreement alone does not establish correct rankings. |
+
+The [complete review](./design/mlbenchmarks-book-review.md) records all available chapters, repository evidence, and remaining research questions.

package/docs/experiment.md

@@ -1,31 +1,31 @@
 # The experiment subpath
 
-`@tangle-network/agent-eval/experiment` turns an experiment's registration into the object that runs it.
+`@tangle-network/agent-eval/experiment` registers decision rules as data and provides interpreters bound to a verified seal.
 
-The registry of measured claims those experiments produce lives in [`evidence/`](../evidence/README.md); a sealed experiment's digest is the `experimentDigest` its registry record carries.
+The [`evidence/` registry](../evidence/README.md) stores published measurements and their experiment identities.
+Sealing does not execute an agent or publish an evidence record.
 
-## The covenant
+## What a seal enforces
 
-1. **The registered rule is the executed rule.**
-   Every rule — row admission, subset selection, estimand, interval, decision table, validity gate, halt, budget, matched budget, reissue — is a typed data node, never a closure or prose.
-   `sealExperiment` canonicalizes and hashes the whole tree into one digest.
-   Every interpreter takes only a sealed node plus evidence records.
-   No execution surface has a parameter for alpha, threshold, metric, or stopping rule, so registered-vs-ran drift is unrepresentable rather than checked.
-2. **Refusals live inside artifacts.**
-   An inadequate cluster count, a mismatched arm budget, a non-monotone funnel stage, a non-total decision table — each produces a typed verdict object (or a typed error), never a warning sentence beside a number.
-3. **A change is a re-seal.**
-   `amendExperiment` verifies the current seal, validates the new spec, and appends a `{at, reason, blind[], digest}` entry.
-   The digest history is the audit trail; changing what is decided without a new digest is impossible.
+1. `sealExperiment()` validates and hashes the specification, including its registered rules and optional claim.
+2. `openSealedExperiment()` verifies that digest and captures the rules before returning the execution handle.
+   Its decision, interval, gate, and budget methods read their rules from that captured specification.
+3. `amendExperiment()` verifies the current seal, validates the replacement specification, and records its digest, reason, time, and declared blindness.
+
+A seal verifies the current specification's identity.
+It does not authenticate registration time, amendment history, or the origin of supplied measurements.
+The host must retain evidence, invoke the required checks, and honor their refusal results.
+Calling `registered.decide()` does not automatically run admission, power, budget, or halt checks.
 
 ## The objects
 
-| object | entry point | what it closes |
+| Object | Entry point | Purpose |
 | --- | --- | --- |
-| Registered-rule AST | `src/experiment/ast.ts` (14 node families) | prose rules; every registered condition compiles to data the seal covers |
-| Define / seal / execute | `defineExperiment`, `sealExperiment`, `amendExperiment`, `openSealedExperiment` | hand-written PREREG.md files; the runner executes the sealed rule itself |
-| Cluster-aware power | `clusteredPower`, `assertDesignAdequate` | "4 clusters cannot certify any effect size, including 1.0" — learned by running the experiment, now refused before a dollar is spent |
-| Denominator chain | `buildFunnel`, `executeAdmissionRule`, `composeFunnels`, `renderFunnelTable` | hand-assembled `20 → 15 → 14 → admitted` chains, formatted differently each run |
-| Matched budgets | `verifyMatchedBudgets`, `assertMatchedBudgets` | "realized tokens must agree within 5%" verified by hand |
+| Registered rules | [Rule types](../src/experiment/ast.ts) and [`ExperimentSpec`](../src/experiment/define.ts) | Describe admission, estimation, intervals, and decisions as data. |
+| Define / seal / execute | `defineExperiment`, `sealExperiment`, `amendExperiment`, `openSealedExperiment` | Validate, identify, and execute the registered rules. |
+| Cluster-aware power | `clusteredPower`, `assertDesignAdequate` | Assess a declared effect under a simulated outcome model and cluster-count policy. |
+| Denominator chain | `buildFunnel`, `executeAdmissionRule`, `composeFunnels`, `renderFunnelTable` | Reconcile retained and excluded evidence. |
+| Matched budgets | `verifyMatchedBudgets`, `assertMatchedBudgets` | Check realized tokens against a declared tolerance. |
 
 ### Sealing and execution
 
@@ -35,67 +35,123 @@ import {
   sealExperiment,
 } from '@tangle-network/agent-eval/experiment'
 
-const sealed = await sealExperiment(spec)      // RFC 8785 + sha256 over the whole tree
-const registered = await openSealedExperiment(sealed) // verifies the digest first
+const sealed = await sealExperiment(spec)
+const registered = await openSealedExperiment(sealed)
 
-const admission = registered.admit(rows)       // funnel + survivors, from the sealed rule
+const admission = registered.admit(rows)
 const gate = registered.gate('power-floor', { kind: 'power-floor', curve })
-const halt = registered.halt([gate])           // refuse-spend fires before any contrast
-const outcome = registered.decide(quantities)  // the sealed table; non-total tables throw
+const halt = registered.halt([gate])
+if (halt.fired) throw new Error(`Experiment halted: ${halt.failedGates.join(', ')}`)
+// Compute quantities from the admitted evidence, then call registered.decide(quantities).
 ```
 
-#### Digest schemes and retention
+This fragment assumes `spec` registers admission, the named gate, and a halt rule.
+Malformed specifications and unusable evidence throw typed errors; decision and validity refusals remain in returned artifacts.
 
-A seal and a signed `HypothesisManifest` are durable records: each is written
-once and verified later, possibly by a different release. Both carry an `algo`
-field that names the digest scheme, and verification selects the encoder from
-that field.
+Cluster intervals register both `clusterBy` and `value` inside the sealed `IntervalSpec`.
+Call `registered.interval('gain95', { kind: 'rows', rows })` to apply those fields.
+The row evidence cannot override the registered value field.
+Changing the measured field requires a new seal.
 
-| `algo` | Serialization | Status |
-|---|---|---|
-| `sha256-rfc8785` | RFC 8785 canonical JSON, from `ledger-core/canonical` | What `sealExperiment` and `signManifest` write |
-| `sha256-content` | key-sorted `JSON.stringify` | Read-only. Records written before the RFC 8785 scheme carry it, or carry no `algo` at all, and still verify |
+For a paired contrast, prepare one difference per pair and register that difference field as `value`.
+A pooled pass rate from both arms measures a different quantity.
+The [runnable sealed experiment](../examples/sealed-experiment/index.ts) demonstrates the paired path.
+Confidence levels, field paths, seeds, and resample counts are validated before sealing and direct computation.
+
+Older cluster interval registrations omitted `value` and require their original package version for execution.
+Retain their original bytes and evidence; create a new registration for subsequent measurements.
 
-The `sha256-content` encoder is private to the module that verifies with it and
-is unreachable from any path that writes a digest. Retire it once no record
-carrying that tag needs to verify; until then, deleting it would make those
-records unverifiable rather than invalid.
+#### Canonical identities and migration
 
-`openSealedExperiment` is the only execution surface.
-A rule that is not in the sealed spec cannot run; a rule that is cannot run differently.
+Readers and writers use RFC 8785 canonical JSON from `ledger-core/canonical`.
+Verification refuses missing or unsupported digest schemes.
+
+| Artifact | Required identity | Refusal |
+|---|---|---|
+| Sealed experiment | `algo: 'sha256-rfc8785'` | `verifySealedExperiment()` returns `false`; `openSealedExperiment()` refuses execution |
+| Signed hypothesis | `algo: 'sha256-rfc8785'` | `verifyManifest()` returns `false`; synchronous digest checks and hypothesis evaluation refuse the record |
+| Agent profile cell | `agent-profile-cell:sha256-rfc8785:<digest>` | Cell validation refuses any other scheme |
+| Report attestation | Report hash and required `envelopeHash` over its provenance | `verifyAttestation()` returns an invalid result with a reason |
+
+The package no longer verifies `sha256-content` records, untagged manifests, bare `agent-profile-cell:sha256:` identifiers, or attestations without provenance envelopes.
+Keep those records unchanged as historical artifacts with their original package version.
+Create new registrations with `sealExperiment()` or `signManifest()` before collecting new decision evidence.
+Use `buildAgentProfileCell()` and `attest()` to produce current identities from independently verified source material.
+Never relabel an existing digest or reconstruct a provenance envelope from unverified metadata.
+A new digest cannot establish that a registration existed before its evidence was observed.
+
+Use the opened handle when the result must follow a particular registration.
+Direct helpers such as `computeInterval()` and `executeDecisionRule()` also accept unsealed rules for development.
+They do not establish a link to a registered experiment.
 
 ### Cluster-aware power refusal
 
-Two floors, one simulation:
+`clusteredPower()` combines a cluster-count policy with a simulated power curve:
+
+- The exact whole-cluster sign-flip test has a minimum two-sided p-value of `2^(1-C)` for `C` independent clusters.
+  At alpha 0.05, this policy requires at least six clusters; four give 0.125 and three give 0.25.
+- Seeded simulations draw paired contrasts under the configured win/loss model and apply a whole-cluster percentile bootstrap.
+  Power is the fraction of simulated intervals that exclude zero.
 
-- **Closed form, zero spend.** With `C` independent clusters, the exact whole-cluster sign-flip test can never produce a two-sided p below `2^(1-C)`.
-  Four clusters give 0.125 and three give 0.25 — both above alpha 0.05, so those designs are refused at any effect size.
-  Six clusters is the smallest certifiable count at 0.05.
-- **Seeded simulation.** Per-row paired contrasts are drawn under a registered effect model (base win/loss rates, optional noisy clusters), each trial takes a whole-cluster percentile bootstrap, and power is the fraction of trials whose interval excludes zero.
+The six-cluster floor is a policy for this helper, not a universal requirement for every estimator or fixed-roster evaluation.
+The helper computes both results; it does not skip simulation when the cluster-count policy fails.
 
 The refusal is a verdict inside the returned artifact (`result.refusal`), with `assertDesignAdequate` as the throwing form.
-The `power-floor` validity gate consumes a power curve as evidence and fails when the curve tops out under the registered target.
+Both `clusteredPower` and the registered `power-floor` gate require `minimumEffect`.
+The gate evaluates a supplied curve; it does not run the simulation itself.
+The effect must appear exactly in the supplied grid; the API does not interpolate.
+Adequacy requires target power at that effect.
+`maxPower` describes the grid and cannot establish adequacy at a smaller effect.
+
+```ts
+import { assertDesignAdequate, clusteredPower } from '@tangle-network/agent-eval/experiment'
+
+const power = clusteredPower({
+  clusterSizes: Array.from({ length: 24 }, () => 3),
+  effects: [0.05, 0.1, 0.2],
+  minimumEffect: 0.1,
+  targetPower: 0.8,
+  seed: 17,
+})
+
+assertDesignAdequate(power)
+```
+
+Simulation effects are expected paired contrasts within signal clusters.
+The zero-effect model requires equal `baseWinRate` and `baseLossRate`.
+Configured noisy clusters retain zero expected contrast, so they dilute the pooled population effect.
+Power remains conditional on this outcome model, the registered sampling structure, and the simulated test.
+The [statistical evidence guide](./statistical-evidence.md) explains unit counts, adaptation comparisons, and sequential assumptions.
 
 ### The funnel
 
 `buildFunnel` refuses a stage that gains rows, named exclusions that do not sum, and partitions that overdraw their source stage.
-`executeAdmissionRule` runs a sealed admission rule over rows and returns the funnel, the survivors, and the partition rows in one object — the chain and the rows can never disagree.
-Partitions carry `pooling: 'never'`: a secondary set is reported beside the primary chain and cannot be pooled into it.
+`registered.admit(rows)` applies the sealed admission rule and returns the funnel, survivors, and partition rows together.
+The standalone `executeAdmissionRule(rule, rows)` also accepts an unsealed rule.
+Partitions carry `pooling: 'never'`: report each secondary set separately from the primary chain.
 The object is its own JSON render; `renderFunnelTable` prints the text table with the reconciliation line (`input = surviving + excluded`).
 
 ### Matched budgets
 
 `verifyMatchedBudgets` compares realized per-arm tokens under the registered tolerance and returns a verdict whose `refusal` field carries `onFail: 'refuse-contrast'` when arms diverge.
-A contrast between arms that spent differently is not a contrast; the refusal is the artifact that says so.
+Use this check when the claim requires matched token use.
+An unequal-budget comparison answers a different question and must retain the resource difference in its interpretation.
 
 ## Acceptance: the three preregistrations
 
-The module's acceptance suite (`tests/experiment/preregistration-acceptance.test.ts`) re-derives the week's three hand-written preregistrations as sealed specs and reproduces each recorded decision by executing the sealed rules against the recorded evidence:
+The [acceptance suite](../tests/experiment/preregistration-acceptance.test.ts) encodes three historical preregistrations as sealed specifications.
+It checks their recorded decisions against fixed evidence:
 
-- **killtest-20260810** — all four validity gates fail on the recorded evidence (the rep-4 oracle flip, the 2-row population drift, the zero-call control, the 0.692 power ceiling) and the halt rule refuses the spend, matching the recorded `$0.00, contrast never run`.
+- **killtest-20260810**: all four validity gates fail on the recorded evidence.
+  The failures are the rep-4 oracle flip, 2-row population drift, zero-call control, and 0.692 power ceiling.
+  The halt rule refuses spend, matching the recorded `$0.00, contrast never run`.
   The obligation node routes a positive interval without the registered control to `blocked-pending-registered-control`, never to `thesis-survives`.
-- **freelunch-20260810** — the admission funnel reproduces the recorded `48 > 43 > 35 > 35 > 32` chain with the 3-row secondary partition; the uniform-pass budget reproduces the recorded uniform n=2; the amendment-6 ledger under the same sealed rule refuses pass 2 — the registered-vs-ran divergence the seal makes unrepresentable; the report-only decision reproduces `3/64` and `2/32`.
-- **tbench-20260808 milestone 2** — the round-robin selection reproduces the recorded 20-row subset in pick order; the m3 subset filters the SEALED m2 draw (16 rows); the decision table on the recorded interval reproduces `not-certified-at-this-n`.
+- **freelunch-20260810**: the admission funnel reproduces `48 > 43 > 35 > 35 > 32` with the 3-row secondary partition.
+  The uniform-pass budget reproduces uniform n=2; the amendment-6 ledger under the same sealed rule refuses pass 2.
+  The report-only decision reproduces `3/64` and `2/32`.
+- **tbench-20260808 milestone 2**: round-robin selection reproduces the recorded 20-row subset in pick order.
+  The m3 subset filters the sealed m2 draw to 16 rows.
+  The decision table on the recorded interval reproduces `not-certified-at-this-n`.
 
 ## What is composed, not duplicated
 
@@ -103,7 +159,7 @@ The statistical machinery underneath is re-exported from its existing homes; thi
 
 | family | home |
 | --- | --- |
-| `pairedBootstrap`, `mcnemar`/`mcnemarPower`/`mcnemarRequiredN`, `pairedRiskDifference*`, `holm`, `benjaminiHochberg`, `eProcess`, `wilson`, `mulberry32`, sample-size helpers | `src/statistics.ts` |
+| `pairedBootstrap`, `mcnemar`/`mcnemarPower`/`mcnemarRequiredN`, `pairedRiskDifference*`, `holm`, `benjaminiHochberg`, `eProcess`, `wilson`, `mulberry32`, sample-size helpers | [`src/statistics/index.ts`](../src/statistics/index.ts) |
 | `pairedEvalueSequence` (anytime-valid) | `src/sequential.ts` |
 | `powerPreflight` (variance-based MDE refusal) | `src/campaign/gates/power-preflight.ts` |
 | `sequentialPairedGate`, `sequentialDecide` (manifest-bound) | `src/campaign/gates/sequential.ts` |
@@ -118,5 +174,5 @@ The trace-repair admission machinery (`buildDenominatorChain`, oracle determinis
 
 ## Where this sits
 
-This is Wave 2 of the [charter](./charter.md): the experiment subpath, built after the kill test that re-derived the three preregistrations as decision-rule objects (verdict: extended — ten node families beyond the seed AST, no opaque node, no rule dropped).
-Wave 3 wires these objects to the live-sandbox seam; the improvement receipt (Wave 4) serializes a sealed experiment's digest, gates, and refusal outcomes into one attested file.
+The [charter](./charter.md) describes current package ownership and host responsibilities.
+Use [evaluation claims and final evidence](./evaluation-integrity.md) when connecting a registration to an automated improvement workflow.

package/docs/feature-guide.md

@@ -55,10 +55,10 @@ user intent
   -> datasets and optimizers replay the same adapter
 ```
 
-The important part is that production and eval do not use different loops. The
-adapter can swap dependencies (real user session, replay fixture, sandbox), but
-the state shape, validators, actions, budgets, and stop policies should stay
-the same. That is what makes benchmark gains transfer to real usage.
+Keep the production state, validators, actions, budgets, and stop policies in the evaluation path.
+The adapter can supply a real user session, replay fixture, or sandbox.
+This tests the behavior that production executes.
+Transfer to future tasks still requires representative evaluation data and a measured comparison.
 
 ### Agent Runtime Integration
 
@@ -82,8 +82,7 @@ Implementation ownership:
   assignment, and optimizer row conversion in `agent-eval`.
 - Put product state readers, action executors, approval policy, credentials,
   workspace paths, and UI-specific storage in the downstream repo.
-- Promote a product adapter into `agent-eval` only after at least two products
-  need the same adapter shape.
+- Keep product execution adapters in the consuming repository.
 
 ### Code Generator
 

package/docs/hosted-ingest-spec.md

@@ -178,17 +178,10 @@ It is not production storage because process restart clears its in-memory data.
 TENANT_KEY=dev-token TENANT_ID=acme pnpm tsx examples/hosted-ingest-server/server.ts
 ```
 
-In another terminal:
-
-```sh
-HOSTED_ENDPOINT=http://localhost:8080 \
-HOSTED_TENANT_KEY=dev-token \
-HOSTED_TENANT_ID=acme \
-pnpm tsx examples/foreign-agent-quickstart/index.ts
-```
-
-The quickstart's eval-run gets POSTed to the reference receiver; the
-receiver's `GET /v1/runs` lists it back.
+Send events with [`createHostedClient`](../src/hosted/client.ts) from `@tangle-network/agent-eval/hosted`.
+Call `client.ingestEvalRun(event)` with a valid `EvalRunEvent` after configuring the client's endpoint, tenant ID, and API key.
+For `selfImprove`, configure `hostedTenant` as shown in the [receiver example](../examples/hosted-ingest-server/).
+The receiver's authenticated `GET /v1/runs` endpoint lists ingested runs.
 
 ---