peeksafe 0.0.0-stage → 0.2.0

package/README.md

@@ -1,3 +1,491 @@
-# Temporary Holding Version
+# peeksafe
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Statistically valid gates for non-deterministic eval suites. Stop early without losing error control.
+
+```bash
+npm install peeksafe
+```
+
+Zero runtime dependencies. TypeScript, ESM, Node 22 or newer.
+
+## The problem
+
+You have an eval suite. Each case is non-deterministic, so you run it several times and compare a pass rate against a recorded baseline. Runs cost money, so you watch results arrive and stop a case once it looks decided. Then you correct for multiplicity with Benjamini-Hochberg, because 200 cases at 5% each is a lot of false alarms.
+
+That construction has no error control, and it fails in two independent places.
+
+**A p-value taken at a stopping boundary is not a p-value.** `P(p_n <= a) <= a` holds for each fixed `n`. A procedure that adds runs and re-tests until the result looks significant is reporting the minimum over the peeks, and that minimum is not bounded by `a`. Benjamini-Hochberg takes `P(p <= t) <= t` as its input assumption, so it inherits nothing and controls nothing.
+
+**Your baseline is not an oracle.** A 60-run baseline at a true rate of 0.85 has a standard error of 0.046, which is a third of a 15-point effect. Testing against the point estimate treats a noisy number as truth, and when the baseline happens to have come in high, every candidate looks worse than it is.
+
+Neither of these is a new observation. The first is documented: Ramdas's KDD 2019 tutorial has a section titled "Why Benjamini-Hochberg cannot be used online", and the entire e-BH line of work exists because of it. The second is standard in the external-control literature. What this library does is implement the construction that works, and measure what the broken one costs.
+
+## The part that is genuinely counterintuitive
+
+You would expect the damage to be worst on a big suite. It is worst on a small one.
+
+Benjamini-Hochberg's bar for the smallest p-value is `q/m`. At `m = 200` that is 2.5e-4, and an exact test on realistic counts rarely reaches it, so BH's own conservativeness hides the invalidity. At `m = 10` the bar is 20 times looser and the same defect produces visible false discoveries.
+
+Measured on 20 pull requests that changed nothing, peeking every 4 runs from run 8 up to 96:
+
+| suite size | false discoveries per no-op PR |
+| --- | --- |
+| `m = 10` | 0.65 |
+| `m = 200` | 0.20 |
+
+Small, curated suites are what cost economics push you toward. That is the regime where this bites.
+
+## What it costs, measured
+
+Same 20 no-op pull requests, same underlying draws, three ways of analysing them:
+
+| construction | per-case P(p <= 0.05) | discoveries over 20 no-op PRs |
+| --- | --- | --- |
+| peek every 4 runs, then BH | 16.5% | 13 |
+| the same runs, one test at a sample size fixed in advance | 9.0% | 0 |
+| two-sample e-value, then e-BH | not applicable | **0** |
+
+Every discovery in that table is false, because nothing moved in any of those pull requests.
+
+Two honest notes on those numbers. The fixed-N rate of 9.0% is high because the baseline is only 60 runs and the suite spans rates from 0.55 to 0.99, so baseline noise alone produces false positives even without peeking; that is the second defect above, visible. And these are measurements on one simulated suite, reproducible with `pnpm test`, not a general result. Run it against your own rates and see what you get.
+
+## Quick start
+
+You bring counts. peeksafe brings the statistics. It has no opinion about how you run your evals, no runner, no config file, and no CLI.
+
+```ts
+import { gate } from 'peeksafe';
+
+const result = gate([
+  { id: 'parsing/nested',   successes: 88, trials: 96, baseline: { caseId: 'parsing/nested',   successes: 51, trials: 60 } },
+  { id: 'routing/fallback', successes: 20, trials: 96, baseline: { caseId: 'routing/fallback', successes: 57, trials: 60 } },
+  { id: 'summary/new-case', successes: 30, trials: 96 },
+], { mde: 0.15, fdr: 0.05 });
+
+result.verdict;                      // 'FAIL'
+result.regressed.map((c) => c.id);   // ['routing/fallback']
+result.newCases;                     // ['summary/new-case']
+```
+
+`mde` is the drop, in rate, that counts as a regression. `fdr` is the false discovery rate you are willing to accept across the whole suite. Both default to the values above.
+
+### Reading a case verdict
+
+Every case comes back with the evidence behind its decision, not just the decision:
+
+```ts
+for (const c of result.cases) {
+  c.regressed;      // did e-BH certify it, at the suite's FDR
+  c.evalue;         // the evidence. Larger is more evidence AGAINST the null
+  c.ceiling;        // the largest e-value this case could EVER reach
+  c.undetectable;   // true when `ceiling` is below the bar it has to clear
+  c.impossible;     // true when the baseline rate is at or below `mde`, so the case is not tested
+  c.observed;       // { successes, trials }
+  c.baseline;       // { successes, trials }
+}
+
+result.threshold;      // the bar, given how many discoveries were made
+result.soloThreshold;  // the bar a case clears on its own evidence, whatever the others do
+```
+
+`undetectable` is a statement about the *baseline*, not about this run: it means a regression of exactly `mde` could never be certified against a baseline this thin. A larger regression can still be caught, so `undetectable: true` alongside `regressed: true` is consistent and means "we caught this one because it is much worse than `mde`".
+
+## Use it in CI
+
+The whole integration is an exit code and a report.
+
+```ts
+import { gate, PeeksafeError } from 'peeksafe';
+
+try {
+  const result = gate(cases, { mde: 0.15, fdr: 0.05 });
+
+  // One line that does not let a PASS hide what was never checked.
+  console.log(result.headline);
+
+  for (const c of result.regressed) {
+    console.error(
+      `REGRESSED ${c.id}: ${c.observed.successes}/${c.observed.trials} ` +
+      `vs baseline ${c.baseline.successes}/${c.baseline.trials} (e=${c.evalue.toExponential(2)})`
+    );
+  }
+  for (const c of result.cases.filter((x) => x.undetectable && !x.regressed)) {
+    console.warn(`UNDETECTABLE ${c.id}: baseline too thin to certify an ${result.options.mde} drop`);
+  }
+  for (const id of result.newCases) console.log(`NEW ${id}: no baseline, not gated`);
+
+  process.exit(result.verdict === 'FAIL' ? 1 : 0);
+} catch (e) {
+  // A refusal is a bug in the harness, not a regression. Do not let it read as a pass.
+  if (e instanceof PeeksafeError) console.error(`${e.code}: ${e.message}\n${e.hint ?? ''}`);
+  process.exit(2);
+}
+```
+
+The `newCases` and `undetectable` lists are the two that matter for keeping a suite honest over time: the first tells you what is not being gated yet, the second tells you what *cannot* be gated until you spend more on a baseline. `result.headline` names both in a sentence, so a green check cannot quietly mean "nothing was checked":
+
+```
+PASS: no case cleared the bar of 40 at 5% FDR over 2 gated case(s). Read with care:
+1 of 2 gated case(s) have a baseline too thin to certify a 15pt drop at any candidate
+budget (thin), which needs more baseline runs, not more candidate runs; 1 case(s) have no baseline
+and were not gated (new).
+```
+
+## Stopping early
+
+This is the reason the library exists. Stop each case whenever you like, for any reason, including "it already looks bad". The guarantee does not depend on the rule you used, because the statistic is an e-value rather than a p-value: a non-negative statistic with mean at most 1 under the null, so Ville's inequality bounds `P(sup_n E_n >= 1/a) <= a` over the whole trajectory. The multiplicity correction is e-BH (Wang and Ramdas), which consumes e-values directly and controls FDR under arbitrary dependence between cases.
+
+### What that guarantee rests on
+
+Read this before relying on the gate outside its defaults. The per-case statistic is a Bayes factor whose null is the baseline's posterior, and the martingale argument gives it mean exactly 1 when the shared rate is drawn from the uniform prior that posterior starts from. It does not give mean at most 1 at every fixed rate, and there are rates where it is not: at a true rate of 30% with a 240-run baseline and `mde` 0.3, the fixed-rate mean after 100 runs is above 1.15.
+
+The null is "not worse", so a candidate that improved is a null case too. The Bayes factor alone also grows for a candidate far above its baseline, because the alternative is wider than the null, so the statistic is capped at 1 whenever the candidate's observed rate is at or above the baseline's. Lowering an e-value never breaks one.
+
+So the property you actually rely on, that a case which did not move clears `1/a` with probability at most `a` however long it runs, is computed rather than inferred. `test/error-control.test.ts` computes it exactly, with no sampling error, at the default `altConcentration` over baselines of 10 to 240 runs, rates from 0.05 to 0.99, `mde` from 0.05 to 0.3, `a` of 0.05 and 1/4000, and up to 600 candidate runs. The worst cell is 0.75 of `a`. It checks improved candidates the same way, from 30% to 50% up to 90% to 99%. It also runs the loop above end to end, `shouldStop` then `gate`, on 400 simulated pull requests with nothing moved and 400 with three real regressions out of ten, and checks the false alarm rate and the mean false discovery proportion stay under `fdr`.
+
+Outside that grid it is not proven. A concentrated alternative is the one known way it fails at an ordinary rate: 1.14 times `a` with an `altConcentration` of 100 at a true rate equal to `mde`, and 1.04 times `a` at 32 with a 60-run baseline, `mde` 0.3 and a true rate of 0.26, where the few lucky baselines carry the alternative over the bar. So `gate` and `shouldStop` refuse an `altConcentration` above 16 (`MAX_ALT_CONCENTRATION`) with the default statistic. At 16 the same test checks rates from 4 points below `mde` to 10 above it, with baselines of 10 to 240 runs, and the worst cell is 0.91 of `a`. `universal` is valid at every rate by construction and takes any positive value. Two thin-baseline corners at very low rates used to fail as well (1.09 and 2.42 times `a`), because the alternative's shape parameters could fall to 0.35 and pile its mass onto a rate of zero; the shapes are now floored at 1 and those corners are at 0.09 and 0.05 of `a`. If your suite lives near a corner, run the same computation on your own rates before trusting a verdict:
+
+```ts
+import { typeOneError } from 'peeksafe';
+
+// A case at 50% with a 60-run baseline, in a 10-case suite at 5% FDR, capped at 200 runs.
+typeOneError({ rate: 0.5, baselineTrials: 60, alpha: 0.05 / 10, horizon: 200 });   // 0.0014, under 0.005
+
+// The corner above.
+typeOneError({ rate: 0.15, baselineTrials: 240, mde: 0.15, altConcentration: 100, alpha: 1 / 4000, horizon: 600 });   // 0.00028, over 0.00025
+```
+
+It is exact, not simulated, and takes milliseconds. `alpha` is `fdr / suiteSize`, the bar a case clears on its own evidence. `certifyProbability` is the same computation with separate baseline and candidate rates: the power when the candidate is lower, the type I error when it is not.
+
+### An always-valid alternative
+
+If you would rather have a proof than a computation, pass `evidence: 'universal'` to `gate` and `shouldStop` (and to `typeOneError` and `certifyProbability` when you plan). It uses `universalTwoSampleLogE`, which is valid at every pair of rates and at any stopping time by construction: universal inference (Wasserman, Ramdas and Balakrishnan 2020), a joint alternative for baseline and candidate over the largest likelihood any null pair of rates gives the data. It holds in all of the corners above.
+
+It is not the default because it costs evidence. It is the default statistic times the pooled data's uniform mixture over its maximum likelihood, so it is never larger, and the gap grows like half the log of the run count. On this page's examples, with `mde` 0.15:
+
+| baseline | candidate | default e | `universal` e |
+| --- | --- | ---: | ---: |
+| 54/60 | 72/96 | 5.91 | 0.466 |
+| 216/240 | 72/96 | 89.1 | 4.26 |
+| 216/240 | 3600/4800 | 2.25e+6 | 3.40e+4 |
+| 57/60 | 20/96 | 2.03e+17 | 2.02e+16 |
+
+As power, the exact probability of certifying a case that really dropped by `mde` within 200 candidate runs:
+
+| suite | baseline runs | rate | `mde` | default | `universal` |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| 10 cases | 60 | 0.75 | 0.15 | 0.126 | 0.030 |
+| 10 cases | 240 | 0.75 | 0.15 | 0.502 | 0.223 |
+| 10 cases | 960 | 0.75 | 0.15 | 0.771 | 0.455 |
+| 10 cases | 240 | 0.95 | 0.15 | 0.935 | 0.746 |
+| 10 cases | 240 | 0.75 | 0.25 | 0.979 | 0.903 |
+| 200 cases | 240 | 0.75 | 0.15 | 0.212 | 0.074 |
+| 200 cases | 960 | 0.95 | 0.15 | 0.977 | 0.892 |
+
+Roughly, `universal` needs four times the baseline runs to match the default's power. `test/universal.test.ts` computes the full grid and checks that. The gap is structural rather than a weakness of this construction: a statistic valid at every rate has to hold up at the worst rate the baseline still permits, where the default averages over them. Recentring and truncating the alternative per null rate recovers at most 2 to 5 points at a 60 or 240-run baseline (`test/universal.test.ts`). If you want a proof without that price, pair the runs (see "Paired cases in `gate`").
+
+Turner, Ly and Grünwald's e-values for 2x2 tables are the other always-valid construction, and they lose less. They need both arms in every block of data, though, and a stored baseline followed by candidate-only runs has one block with both and then none: a block with one arm carries no evidence against a null that leaves the baseline rate free. Measured as a single block at a fixed 200 runs, which is valid only if you never stop early, they land between the two, much closer to `universal` than to the default. Re-running the baseline alongside the candidate would let them work; that is the paired design, which `gate` takes directly.
+
+`shouldStop` is that decision for one case, from the counts you have so far:
+
+```ts
+import { shouldStop, gate } from 'peeksafe';
+
+while (ids.some((id) => !state[id].stopped)) {
+  for (const id of ids) {
+    const s = state[id];
+    if (s.stopped) continue;
+
+    for (let k = 0; k < 8; k++) {
+      s.trials++;
+      if (await runCase(id)) s.successes++;
+    }
+
+    const d = shouldStop(s, baseline[id], { suiteSize: ids.length, maxTrials: 200 });
+    if (d.stop) {
+      s.stopped = d.reason;   // 'regressed' | 'settled' | 'futile' | 'impossible' | 'budget'
+      console.log(d.detail);  // one line saying why
+    }
+  }
+}
+
+const result = gate(ids.map((id) => ({ id, ...state[id], baseline: baseline[id] })), { mde: 0.15, fdr: 0.05 });
+```
+
+On a three-case suite where one case really did drop from 90% to 60%:
+
+```
+parsing/nested     160 runs  (settled)
+routing/fallback    24 runs  (regressed)
+summary/tone       120 runs  (settled)
+spent 304 runs, cap would have been 600
+```
+
+### The five ways a case finishes
+
+| reason | what happened | what to do |
+| --- | --- | --- |
+| `regressed` | the e-value cleared `m / fdr`; certified on its own evidence | fix the regression |
+| `settled` | the counts have ruled out a certifiable drop; more runs cannot change the verdict | nothing, it passed |
+| `futile` | the **baseline** is too thin to certify an `mde`-sized drop at any candidate budget | record more baseline runs |
+| `impossible` | the baseline passes at or below `mde`, so an `mde`-sized drop cannot happen and the case is not tested | lower `mde` if smaller drops on this case matter |
+| `budget` | `maxTrials` reached with no decision either way | raise the cap, or accept the ambiguity |
+
+`settled` is the one worth understanding, because an e-value on its own cannot produce it. Evidence accumulates *for* a regression, so a broken case reaches the bar fast, but a healthy case never accumulates evidence that it is healthy, and would run to your cap forever. What stops it is the **ceiling**: once even the worst rate the counts still permit could not produce enough evidence, further runs cannot change the answer. That is why both healthy cases above stopped at 160 and 120 rather than 200.
+
+`futile` and `settled` both mean "this can never be certified", and they are deliberately not one reason, because they call for opposite actions. `futile` is a defect in your baseline. `settled` is a pass.
+
+### Why futility is hard to trigger
+
+The ceiling depends on how bad the regression actually is, and early on that is barely known: for a 60-run baseline it ranges from 2.9 at a rate of 0.75 to 62 at a rate of 0.20. Declaring futility from a noisy point estimate would abandon cases that were about to be certified: a missed regression, reported as a green check, which is the exact failure this library exists to prevent.
+
+So `futile` and `settled` are judged at the **most pessimistic rate the counts still permit**, the lower end of a Wilson interval on the candidate. The question is "even if the truth is as bad as these counts plausibly allow, could more runs ever clear the bar?" Only a no stops the case. A catastrophic regression is therefore never stopped early at any sample size, which `test/stop.test.ts` pins across two baselines and eight sample sizes.
+
+The cost is that futility fires late. If you want to know a case is undetectable *before* spending anything, that is `makePlan`'s job; see below.
+
+## When more runs will not help
+
+There is a second kind of impossible, and it is less obvious than "this case cannot lose 15 points".
+
+The unpaired two-sample e-value has a finite limit as candidate runs go to infinity. Both marginals are the same binomial likelihood under different priors, so everything that grows with `n` cancels and the Bayes factor converges to the prior density ratio at the observed rate. If that ceiling sits below the bar, **no number of candidate runs will ever certify the case.**
+
+The same case at 90% losing 15 points, holding the candidate result fixed and varying only how well the baseline was measured:
+
+| baseline runs | candidate runs | e-value | ceiling | certified |
+| ---: | ---: | ---: | ---: | --- |
+| 24 | 96 | 2.6 | 2.4 | no |
+| 24 | 4,800 | 3.3 | 2.4 | no |
+| 60 | 96 | 5.9 | 19 | no |
+| 60 | 4,800 | 18 | 19 | no |
+| 240 | 96 | 89 | 4.3e+6 | **yes** |
+| 240 | 4,800 | 2.2e+6 | 4.3e+6 | **yes** |
+
+Fifty times the candidate runs moves a baseline-60 case from 5.9 to 18, against a bar of 20. It never gets there. Two hundred and forty baseline runs certifies the same regression on the *first* 96 candidate runs. Adding candidate runs is not merely expensive here, it is futile; what lifts the ceiling is more baseline runs, and those are amortised across every pull request rather than paid per pull request.
+
+`gate` reports this per case as `ceiling` and `undetectable`. `samplesForEvidence` answers the budget question directly, and returns `Infinity` rather than a large number when the honest answer is that no candidate budget will do:
+
+```ts
+import { evidenceCeilingLogE, samplesForEvidence } from 'peeksafe';
+
+const bar = Math.log(200 / 0.05);   // log(m / fdr) for a 200-case suite
+
+evidenceCeilingLogE(0.75, 54, 60, 0.15, 8);   // 2.94  - under the bar of 8.29
+samplesForEvidence(0.75, 54, 60, 0.15, bar);  // Infinity - no candidate budget works
+
+evidenceCeilingLogE(0.75, 216, 240, 0.15, 8); // 15.28 - clears it
+samplesForEvidence(0.75, 216, 240, 0.15, bar);// 262 candidate runs
+```
+
+To ask the other question, how many *baseline* runs would make a case workable, use `baselineRunsNeeded` from the planner below.
+
+The ceiling grows linearly in baseline runs at rate `KL(baseline rate || candidate rate)`, which gives a planning rule: `log(m/q) / KL` baseline runs, and budget about 20% above that. At 85% losing 15 points the rule says **136** and the exact answer is **162**.
+
+Note the KL direction. At 85% losing 15 points the two directions are 0.0611 and 0.0720: an 18% error in the one constant that decides how long a baseline has to be.
+
+The mathematics here is a corollary of the standard Laplace expansion for Bayes factors (Kass and Raftery 1995), and the practical consequence is the cap on prior effective sample size known in the historical-borrowing literature. The closed form, the KL direction, and the planning inversion are the parts we did not find stated elsewhere. [`RESEARCH-NOTES.md`](RESEARCH-NOTES.md) records what was checked, what was found, and which of this library's original novelty claims did not survive.
+
+## Planning: how many runs do I need
+
+```ts
+import { makePlan, toBaselineMap, DEFAULT_PLAN } from 'peeksafe';
+
+const plan = makePlan(cases, toBaselineMap(baselines), { ...DEFAULT_PLAN, mde: 0.15 });
+
+for (const c of plan.cases) {
+  c.detectability;        // 'UNPAIRED' | 'PAIRED_ONLY' | 'TOO_EXPENSIVE' | 'IMPOSSIBLE'
+  c.unpaired.runs;        // what the default design costs, or Infinity
+  c.paired.pairs;         // what the paired design costs
+  c.baselineRunsNeeded;   // baseline runs that would make unpaired work, or null
+  c.design;               // 'unpaired' | 'paired' | null, the cheaper design that decides it
+}
+```
+
+Four baselines, one plan:
+
+```
+parsing/nested     UNPAIRED       unpaired=    93 paired=   59 needBaseline=-
+routing/fallback   PAIRED_ONLY    unpaired=Infinity paired=   59 needBaseline=75
+summary/tone       IMPOSSIBLE     unpaired=Infinity paired=Infinity needBaseline=-
+search/rerank      PAIRED_ONLY    unpaired=Infinity paired=   53 needBaseline=63
+```
+
+Read that as four different answers, not one number. `parsing/nested` has a 240-run baseline and needs 93 candidate runs. `routing/fallback` and `search/rerank` have thin baselines, so the unpaired design cannot do them at any budget, but 75 and 63 more baseline runs respectively would fix that, and pairing works today. `summary/tone` passes 10% of the time and cannot lose 15 points at all, so `sampleSizeTwoProportion(0.1, 0.15)` is `Infinity` rather than a plausible-looking 44.
+
+`plan.totals` carries the suite-level roll-up, including `decidableUnpaired` and `decidableBest`: how many cases each design can decide at all.
+
+### Choosing a design by cost
+
+`c.design` is the design to run a case with: of the two that decide it within `runsPerCaseCeiling`, the one that needs fewer runs, unpaired on a tie, and `null` when neither does. A paired observation is two runs, the candidate and a fresh baseline run, so pairing wins only when the coupling between the arms cuts the discordances enough to pay for the second run. `DEFAULT_PLAN.pairCoupling` assumes 0.9. Measure it instead: run a short pilot with both arms on the same revision and pass each case's pair table to `makePlan`, which prices that case at the pilot's `pairPhi` (floored at 0; an empty table is ignored).
+
+```ts
+import { makePlan, toBaselineMap, DEFAULT_PLAN } from 'peeksafe';
+
+const baselines = [
+  { caseId: 'checkout/flow', successes: 180, trials: 240 },
+  { caseId: 'parsing/nested', successes: 216, trials: 240 },
+];
+const cases = baselines.map((b) => ({ id: b.caseId }));
+const pilots = new Map([['checkout/flow', { bothPass: 27, worse: 3, better: 3, bothFail: 7 }]]);
+
+const assumed = makePlan(cases, toBaselineMap(baselines), DEFAULT_PLAN);
+const measured = makePlan(cases, toBaselineMap(baselines), DEFAULT_PLAN, pilots);
+```
+
+```
+                 unpaired   paired at 0.9   design   paired at the pilot's 0.6   design
+checkout/flow         164             118   paired                         206   unpaired
+parsing/nested         75             100   unpaired              (no pilot) 100   unpaired
+```
+
+At the assumed coupling, `checkout/flow` would pair and spend 118 runs instead of 164. The pilot measures 0.6, pairing would cost 206, and the plan keeps it on the stored baseline. A case only pairing can decide (`PAIRED_ONLY`) is paired whatever the pilot says, because independent pairs are still valid; the coupling only moves its price.
+
+The expected bill, what a pull request in which nothing moved spends, prices each case at its own `design`, and a case with none at the suite's `recommendedDesign`. A paired case is counted in the pairs `shouldStopPaired` takes to settle it at its coupling, and an unpaired one in the runs `shouldStop` takes to stop it, both checked at `minTrials` and then every `batchSize` runs. If you run them at a `futilityConfidence` other than the default, set the same one in the config, since a stricter level stops later. At the assumed coupling the bill is 138 runs: 36 pairs of `checkout/flow` at two runs each and 66 runs of `parsing/nested`. Set `design` in the config to price every case at one design instead: 127 runs unpaired, 136 paired.
+
+## The paired alternative
+
+Re-run the baseline alongside the candidate on the same seed and the null becomes an exact point mass at one half: under "nothing changed", a discordant pair points either way with probability exactly 0.5, and under "the candidate improved" it points the worse way less often than that. The alternative only bets on the worse direction, so the paired e-value is valid against both. Nothing is estimated, so nothing caps the evidence, and the paired e-value grows linearly in discordant pairs.
+
+```ts
+import { pairedLogE, discordant, mcnemarSamplesForEvidence, ebhSoloThreshold } from 'peeksafe';
+
+const counts = { bothPass: 70, worse: 22, better: 4, bothFail: 4 };
+const bar = ebhSoloThreshold(10, 0.05);            // 10 cases at 5% FDR -> 200
+
+discordant(counts);                                 // 26, the only pairs that carry information
+Math.exp(pairedLogE(counts.worse, discordant(counts)));  // 429 -> certified
+
+mcnemarSamplesForEvidence(0.90, 0.15, 0.6, Math.log(bar));  // 103 pairs, planned in advance
+```
+
+It costs two runs per observation and is routinely still cheaper, and it is the only design that works at all for a case whose ceiling is under the bar.
+
+### Paired cases in `gate`
+
+A case can carry its pair table instead of candidate counts and a stored baseline, and paired and unpaired cases share one e-BH family:
+
+```ts
+import { gate, shouldStopPaired } from 'peeksafe';
+
+const result = gate([
+  { id: 'routing/fallback', paired: { bothPass: 70, worse: 22, better: 4, bothFail: 4 } },
+  { id: 'parsing/nested', successes: 88, trials: 96, baseline: { caseId: 'parsing/nested', successes: 51, trials: 60 } },
+]);
+result.regressed.map((c) => c.id);   // ['routing/fallback']
+result.cases[0].design;              // 'paired'
+
+shouldStopPaired({ bothPass: 150, worse: 12, better: 12, bothFail: 26 }, { suiteSize: 10 }).reason;   // 'settled'
+```
+
+A paired case is never `undetectable`, `impossible` or `futile`: its null is exact at every rate, whichever `evidence` the unpaired cases use. `shouldStopPaired` has no ceiling to stop a healthy case with, so `settled` comes from a second e-value, against "the candidate dropped by at least `mde`", and a case that really dropped that far is settled with probability at most `1 - futilityConfidence` however often you ask.
+
+The seed is optional. Two independent runs paired in the order they finished are still valid, because under "not worse" a discordant pair points the worse way with probability at most one half either way. A shared seed makes discordances rarer and more one-sided, which is where the extra power comes from. `pairedCertifyProbability` computes the power exactly; `coupling` is the share of pairs whose two runs see the same draw. Within 200 pairs, so 200 candidate runs plus 200 fresh baseline runs, against the default statistic's 200 candidate runs and a 240-run stored baseline:
+
+| suite | rate | `mde` | default, 240-run baseline | paired, independent runs | paired, coupling 0.5 |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| 10 cases | 0.75 | 0.15 | 0.502 | 0.435 | 0.745 |
+| 10 cases | 0.95 | 0.15 | 0.935 | 0.917 | 0.983 |
+| 10 cases | 0.75 | 0.25 | 0.979 | 0.958 | 0.998 |
+| 200 cases | 0.75 | 0.15 | 0.212 | 0.169 | 0.427 |
+
+With no stored baseline at all, independent pairs come within seven points of the default and have about twice the power of `universal` against the same 240-run baseline, with error control that is proven rather than computed. Seeded pairs beat both. The price is the baseline runs, paid again on every pull request instead of once.
+
+## Suites are not independent draws
+
+Several cases generated from one source file share a failure mode, so a change that breaks the file moves all of them together. Treating them as independent makes the suite-level interval too narrow, which is the direction that produces confident wrong answers.
+
+```ts
+import { clusteredEffect, caseFamily } from 'peeksafe';
+
+const eff = clusteredEffect([
+  { cluster: 'parsing.yaml', value: 0.042 },
+  { cluster: 'parsing.yaml', value: 0.044 },
+  // ...one entry per case, `cluster` being the file it came from
+]);
+
+eff.cr2;                       // the honest interval: CR2 with small-cluster dof
+eff.naive;                     // what you get assuming independence
+eff.designEffect;              // (clustered SE / naive SE)^2
+eff.widthRatio;                // how much wider the honest interval is
+eff.icc;                       // intra-cluster correlation
+eff.signFlippedByClustering;   // the finding, when there is one
+eff.headline;                  // one line an operator can read
+```
+
+`caseFamily('parsing.yaml::nested-anchors')` derives the cluster key when your ids already encode the source.
+
+On twelve cases from three files, where one file's cases all moved together:
+
+```
+naive 95% CI      [0.0052, 0.0288]      ← excludes zero: "the suite improved"
+CR2   95% CI      [-0.0433, 0.0773]     ← does not
+designEffect 5.43   widthRatio 5.12   icc 0.989
+signFlippedByClustering  true
+```
+
+and `eff.headline` says it in a sentence:
+
+> suite effect +1.7pts [-4.3, +7.7] over 3 case families (12 cases); clustering widens the interval 5.12× (design effect 5.43, ICC 0.99), so the suite is worth ~2 independent cases, not 12. The naive interval excluded zero and this one does not, the apparent suite-level move does not survive the correction.
+
+`signFlippedByClustering` is the one to alert on. It means a claim that the suite moved did not survive being told the cases are not independent.
+
+## What it refuses to do
+
+A gate that fails open is worse than no gate, because it produces a green check nobody investigates. These all throw `PeeksafeError` rather than returning a plausible number:
+
+| what | code |
+| --- | --- |
+| no case in the suite has a baseline | `PEEKSAFE_E_BASELINE_MISSING` |
+| `successes > trials`, or negative counts | `PEEKSAFE_E_STAT_DOMAIN` |
+| two cases sharing an id | `PEEKSAFE_E_CASE_DUPLICATE` |
+| `fdr` or `mde` outside `(0, 1)` | `PEEKSAFE_E_STAT_DOMAIN` |
+| `altConcentration` above 16 with the default statistic | `PEEKSAFE_E_CONFIG` |
+
+The first one is the one that matters. The prototype this library came from treated a missing baseline as `0/0`, whose Beta(1,1) posterior median is 0.5, and gated the case against a null of "this case passes half the time". It reported a verdict of PASS and a suite-level improvement of +87.5 points, measured against nothing at all.
+
+A case whose baseline passes at or below `mde` is not tested either. It cannot lose `mde` points, and testing it anyway puts the alternative on top of the rate the case already runs at, which with the original prior let the false positive rate exceed `fdr`. It stays in the e-BH family with an e-value of 1, is marked `impossible`, and the headline names it. `makePlan` has always called these cases `IMPOSSIBLE`.
+
+A case with no baseline is not an error, though. It comes back in `newCases`, takes no part in the verdict or the e-BH family, and is reported so you know it is not being gated.
+
+## API
+
+Four layers. Most callers need the first.
+
+**The decision.** `gate` for a whole suite from final counts or pair tables, `shouldStop` and `shouldStopPaired` for one case mid-run, and the types around them.
+
+**The budget.** `makePlan`, `planCase`, `affordabilityGrid`, `computeFrontier`, `enumerateFrontier`.
+
+**The check.** `typeOneError`, `certifyProbability` and `pairedCertifyProbability`, the error rate and power the gate delivers at rates and a baseline size you name.
+
+**The statistics.** Everything the first two are built from, exported so you can check the arithmetic or build a different gate: special functions (`logGamma`, `ibeta`, `normalQuantile`), intervals (`wilsonInterval`, `betaCredibleInterval`, `diffInterval`), fixed-sample tests for comparison (`fisherExact2x2`, `twoProportionZTest`), sequential (`sprtDecision`, `sprtExpectedN`), multiplicity (`bhCorrect`, `ebhCorrect`, `ebhSoloThreshold`), e-values (`twoSampleLogE`, `evidenceCeilingLogE`, `samplesForEvidence`), paired designs (`pairedLogE`, `mcnemarSamplesForEvidence`), and clustering.
+
+Every export is exercised by a test, and `test/exports.test.ts` fails when one is not.
+
+## Reproducing the numbers
+
+```bash
+pnpm install
+pnpm test
+```
+
+`test/paper.test.ts` produces the measured figures in this README and prints them. It is self-contained: seeded Bernoulli draws, no runner, no fixtures.
+
+`test/readme.test.ts` executes every example on this page and asserts every number quoted as output, including the run counts in the stopping loop, the four rows of the planning table, and the two intervals in the clustering section. Documentation drifts in a way code does not (a renamed field keeps compiling everywhere except in the prose), so the prose is tested. `test/error-control.test.ts` checks the type I error and false discovery rate the gate actually delivers, as described under "What that guarantee rests on". `test/stop.test.ts` pins the stopping rules, including the property that a catastrophic regression is never stopped early. `test/gate.test.ts` pins the refusals.
+
+The generator is a fixed LCG with an integer avalanche, spelled out in `src/rand.ts` rather than imported, because replacing it would change every number above.
+
+## What this is not
+
+It does not run your evals, grade outputs, store baselines, or integrate with CI. It takes counts and returns statistics. Everything else is yours.
+
+## References
+
+- Ramdas, *Fundamentals of large-scale sequential experimentation*, KDD 2019 tutorial.
+- Wang and Ramdas, *False discovery rate control with e-values*, 2022.
+- Wang, Dandapanthula and Ramdas, *Anytime-valid FDR control with the stopped e-BH procedure*, 2025.
+- Grunwald, de Heide and Koolen, *Safe Testing*, 2019.
+- Kass and Raftery, *Bayes Factors*, JASA 1995.
+- Benjamini and Hochberg, 1995. Wald, *Sequential Analysis*, 1945. Bell and McCaffrey, CR2, 2002.
+
+## Licence
+
+Apache-2.0.