peeksafe 0.0.0-stage → 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,146 @@
+# Changelog
+
+All notable changes to this project are documented here. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project uses
+[semantic versioning](https://semver.org/spec/v2.0.0.html).
+
+## Unreleased
+
+## 0.1.0 (2026-10-20)
+
+First release. There is no earlier version to compare against, so this section
+describes what the package contains rather than what changed.
+
+### Added
+
+- `gate()`, the decision: given per-case candidate and baseline counts, which
+  cases regressed, at a false discovery rate you name. Valid at any stopping
+  time, so it does not matter when or why you stopped a case.
+- `shouldStop(observed, baseline, options)`: the per-case stopping decision, returning
+  `regressed` | `settled` | `futile` | `impossible` | `budget` | `continue` with the evidence
+  behind it. This is the loop every caller was writing by hand. `futilityConfidence` is any
+  value strictly between 0 and 1.
+- Paired cases in `gate`: a case may carry `paired: { bothPass, worse, better, bothFail }` instead
+  of candidate counts and a stored baseline, and is decided by `pairedLogE` in the same e-BH family
+  as the unpaired cases. Its null is exact, so it has no ceiling and is valid at every rate by
+  construction. The runs need not share a seed: independent runs paired in order are still valid.
+  Within 200 pairs, a 10-case suite at 75% losing 15 points is certified 43.5% of the time with
+  independent runs and 74.5% with half the pairs seeded, against 50.2% for the default statistic
+  with a 240-run stored baseline and 22.3% for `universal`.
+- `shouldStopPaired(counts, options)`: `shouldStop` for a paired case. `settled` comes from an
+  e-value against "dropped by at least `mde`", so a case that really dropped that far is settled
+  with probability at most `1 - futilityConfidence` however often you ask.
+- `pairedCertifyProbability(options)`: the exact power and type I error of a paired case, at
+  separate baseline and candidate rates and a `coupling` for shared seeds.
+- `universalTwoSampleLogE` and `evidence: 'universal'` on `gate`, `shouldStop`, `typeOneError` and
+  `certifyProbability`: a two-sample e-value valid at every rate by construction, for anyone who
+  wants a proof rather than a computation. It needs roughly four times the baseline runs to match
+  the default's power, so the default stays. `universalCeilingLogE` bounds it for `shouldStop`.
+- `certifyProbability(options)`: the exact probability a case is certified at separate baseline
+  and candidate rates, which is the power when the candidate is lower.
+- `typeOneError(options)`: the exact probability that a case which did not move is ever
+  certified, at a rate, baseline size and bar you name. The README's error-control grid is built
+  on it, and it is what to run when your suite sits outside that grid.
+- `GateResult.headline`: one line an operator can read, naming the two things a PASS can hide
+  (cases with no baseline, and cases whose baseline is too thin to certify an `mde`-sized drop).
+- Planning: `makePlan`, `planCase`, `affordabilityGrid` answer how many runs a
+  suite needs before anything is spent, including the two ways the answer can be
+  "no budget does this". An effect larger than the rate is arithmetically
+  impossible; an effect under the evidence ceiling is impossible at any
+  *candidate* budget and needs more baseline runs instead.
+- The frontier: `computeFrontier` and `enumerateFrontier`, what is certifiable
+  inside a budget.
+- The statistical core, exported so callers can check the arithmetic or build a
+  different gate: special functions, intervals, the SPRT, two-sample and paired
+  e-values, e-BH, power, and the evidence ceiling with its closed form.
+- Cluster-robust suite inference (`clusteredEffect`, CR2 with small-cluster
+  degrees of freedom, a random-effects cross-check, and a diagnostic for a
+  grouping key that collapses the suite).
+- `RESEARCH-NOTES.md`, a literature check on what in this work is novel and what
+  is not. Two of the four claims the prototype made turned out to be documented
+  elsewhere, and the prose says so.
+- `PlanCase.design`: the design to run each case with, the one of unpaired and paired that
+  decides it within `runsPerCaseCeiling` in fewer runs (unpaired on a tie), or `null` when neither
+  does. `PlanCase.pairCoupling` is the coupling its paired figures were priced at.
+- `makePlan(cases, baseline, cfg, pilots)`: an optional map of per-case pilot pair tables. A case
+  with one is priced at its measured `pairPhi` (floored at 0) instead of `cfg.pairCoupling`; an
+  empty table is ignored. In the README example a case at 75% on a 240-run baseline pairs at the
+  assumed 0.9 (118 runs against 164) and stays unpaired at a measured 0.6 (206 runs).
+
+### Notes
+
+- Zero runtime dependencies, verified in CI against a packed tarball rather than
+  against `package.json`.
+- The package ships `src/` next to `dist/`, so the source and declaration maps resolve and
+  go-to-definition lands on the TypeScript rather than on compiled output.
+- Extracted from an internal prototype and reduced to the statistical core. The
+  runner, graders, store, CI adapters and CLI were deliberately left behind:
+  this library takes counts and returns statistics, and has no opinion about how
+  you run evals.
+- The default two-sample e-value has mean exactly 1 averaged over the shared rate, not at every
+  fixed rate, so its error control is computed rather than inferred. `test/error-control.test.ts`
+  computes the type I error exactly over a grid of baselines, rates and `mde`, and runs
+  `shouldStop` and `gate` end to end on simulated suites. At the defaults the worst cell found is
+  0.75 of alpha. `gate` and `shouldStop` refuse an `altConcentration` above 16
+  (`MAX_ALT_CONCENTRATION`, exported) with the default statistic, with `PEEKSAFE_E_CONFIG`, because a
+  concentrated alternative is the one known way it exceeds alpha inside the checked range: 1.04
+  times alpha at 32. At 16 the worst checked cell is 0.91 of alpha. `universal` takes any positive
+  value, and `typeOneError` and `certifyProbability` still compute any concentration, so a corner
+  can be inspected.
+- The alternative's Beta shapes are floored at 1 rather than anywhere below it, because a shape
+  under 1 piles the alternative's mass onto a rate of zero, which is where a thin baseline at a
+  very low rate already sits. That corner is the one that costs the most error: a 30-run baseline
+  at 0.5% with `mde` 0.05, and an `altConcentration` of 2 with a 5-run baseline, are both under a
+  tenth of alpha, pinned in `test/error-control.test.ts`.
+- `settled` is the part an e-value cannot give you on its own: a healthy case never accumulates
+  evidence that it is healthy, so it would run to your cap forever. The evidence ceiling stops it
+  once the counts have ruled out a certifiable drop.
+- `futile` is kept separate from `settled` because they call for opposite actions: a thin baseline
+  is a defect to fix, a settled case is a pass.
+- Futility is judged at the most pessimistic rate the counts still permit, not at the point
+  estimate, so it fires late and cannot abandon a case that was about to be certified. A
+  catastrophic regression is never stopped early at any sample size.
+- An improvement is never certified as a regression. The Bayes factor behind `twoSampleLogE` has a
+  wider alternative than null, so on its own it also grows for a candidate far above its baseline:
+  96/96 against 30/60 scores e^19.5 uncapped. The statistic is therefore capped at 1 whenever the
+  candidate's observed rate is at or above the baseline's, and `test/error-control.test.ts` checks
+  improved candidates stay under alpha. `pairedLogE` has the same property from a proof rather than
+  a cap: its alternative is truncated to "worse more often than not", which makes it an e-value for
+  every candidate that did not get worse. `test/paired.test.ts` computes it exactly.
+- A case whose baseline passes at or below `mde` cannot drop by `mde` at all, so `gate` marks it
+  `impossible` and does not test it, `makePlan` reports it the same way, and `shouldStop` stops it
+  at once with the reason `impossible`. Testing such a case would mean testing it against an
+  alternative clamped onto a rate near zero, where the case already sits, and a type I error above
+  the nominal level.
+- `PeeksafeErrorCode` lists only the five codes the library throws, and `test/exports.test.ts`
+  fails if a code is listed that no source file throws. Refusals are consistent across entry
+  points: a duplicate case id is `PEEKSAFE_E_CASE_DUPLICATE` from `toBaselineMap` as well as from
+  `gate`, a non-positive `altConcentration` is `PEEKSAFE_E_CONFIG` from `certifyProbability` and
+  `typeOneError` as well as from `gate` and `shouldStop`, and `costShares` refuses a `basis` it
+  does not recognise rather than quoting one it does. An omitted `basis` is `typical`.
+- A reported `evalue` is clamped to `Number.MAX_VALUE`, because `exp()` of a log e-value past about
+  709 overflows and a strong regression goes well past that: 0/1000 against 990/1000 has a log
+  e-value near 1291. `gate` and `shouldStop` both report it through the same clamp, and no decision
+  depends on the number.
+- `PlanCase.baselineRunsNeeded` scans for the smallest baseline that clears the bar rather than
+  bisecting for one, because the evidence ceiling is not monotone in the baseline size: holding the
+  rate fixed still forces an integer success count, so a larger baseline can carry less evidence
+  than a smaller one and a bisection returns whichever crossing it happened to bracket.
+  `test/budget.test.ts` checks the answer against a scan over a grid of baselines and effects.
+- The expected-cost half of the planner is Wald's approximation, and it errs low. `sprtExpectedN`
+  evaluates the operating characteristic exactly, `(1 - B^h) / (A^h - B^h)` at the tilt `h` solving
+  `E_p[lambda^h] = 1`, which is `-1` at `p1` and `+1` at `p0` and so agrees with both hypotheses,
+  with the limit `-log A * log B / E_p[(log lambda)^2]` taken where the drift vanishes. The
+  expected sample number built on it still treats the test as stopping exactly on a wall where a
+  real one overshoots it, so it under-states the true expected sample number and every cost built
+  on it. No decision depends on this: it is the pricing half of the planner, not the gate.
+- `randomEffectsMean` reports the grand mean with a standard error of 0 and a zero-width interval
+  on a suite where every case moved by the same amount, which is what a pull request that moved
+  nothing produces, and what `clusterRobustMean` and `iidMean` already answer for the same data.
+  `compareEstimators` then reads a ratio of 1 and reports that the two estimators agree.
+- `makePlan`, `affordabilityGrid` and `computeFrontier` refuse error rates with `alpha + beta >= 1`
+  with `PEEKSAFE_E_CONFIG`, because the sequential test decides before it sees a run there and
+  every case would be priced at `minTrials`. `PlanCase.fixedNPerArm` prices the fixed-N comparator
+  at the plan's own `alpha` and `beta`; the defaults are 0.05 and 0.1.
+- `probabilityMoved` refuses an `mde` outside (0, 1) and a `nodes` that is not a positive integer
+  with `PEEKSAFE_E_STAT_DOMAIN`.

package/LICENSE

@@ -0,0 +1,190 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to the Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by the Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding any notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   Copyright 2026 Alexis Rico
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

package/NOTICE

@@ -0,0 +1,78 @@
+peeksafe
+Copyright 2026 Alexis Rico
+
+Licensed under the Apache License, Version 2.0. See LICENSE.
+
+--------------------------------------------------------------------------------
+Third-party code
+--------------------------------------------------------------------------------
+
+None. peeksafe has zero runtime dependencies, vendors no source, and ships no
+code it did not write. `.github/workflows/ci.yml` asserts the first of those on
+every pull request by packing the tarball, installing it into an empty project,
+and failing if `pnpm ls` resolves more than two packages.
+
+Development dependencies (the devDependencies in package.json) are not
+redistributed and are not part of the published package.
+
+--------------------------------------------------------------------------------
+Algorithms and published constants
+--------------------------------------------------------------------------------
+
+Everything statistical here is implemented from published closed forms and
+recurrences rather than ported from an existing implementation. Mathematical
+methods and tabulated constants are not anybody's intellectual property, but
+naming their sources is both good manners and the fastest way for a reader to
+check the arithmetic.
+
+  Regularized incomplete beta       Continued fraction: Abramowitz & Stegun,
+  src/stats.ts                      Handbook of Mathematical Functions, 26.5.8.
+                                    Evaluated by the modified Lentz algorithm:
+                                    Lentz (1976), amended by Thompson & Barnett
+                                    (1986).
+
+  log Γ(x)                          Lanczos approximation, g = 7, n = 9. The
+  src/stats.ts                      coefficient set is the widely reprinted one
+                                    originating with Lanczos (1964).
+
+  Φ⁻¹(p)                            Peter Acklam's algorithm for the inverse
+  src/stats.ts                      normal CDF, refined here by one Halley
+                                    step. Acklam released it for unrestricted
+                                    use.
+
+  Lower incomplete gamma            Standard series and continued-fraction
+  src/stats.ts                      expansions, Abramowitz & Stegun 6.5.
+
+  Linear congruential generator     Multiplier 1664525, increment 1013904223,
+  src/rand.ts                       modulus 2^32. Tabulated parameters from
+                                    Knuth, The Art of Computer Programming,
+                                    volume 2.
+
+  32-bit avalanche (mix32)          The `lowbias32` multipliers from Chris
+  src/rand.ts                       Wellons' hash function prospector, placed
+                                    by the author in the public domain.
+
+  hash32                            FNV-1a. Public domain by the authors'
+  src/rand.ts                       explicit statement.
+
+--------------------------------------------------------------------------------
+Statistical methods
+--------------------------------------------------------------------------------
+
+The procedures peeksafe implements are established results. README.md is
+explicit about which of this library's claims are new and which are not, and
+RESEARCH-NOTES.md records an adversarial literature check that withdrew two of
+four originally-claimed novelties.
+
+  e-values and testing by betting   Shafer; Grünwald, de Heide & Koolen (Safe
+                                    Testing); Waudby-Smith & Ramdas.
+  e-BH                              Wang & Ramdas, False discovery rate control
+                                    with e-values (2022).
+  Stopped e-BH                      Wang, Dandapanthula & Ramdas (2025).
+  Benjamini-Hochberg                Benjamini & Hochberg (1995).
+  Sequential probability ratio      Wald, Sequential Analysis (1945).
+  McNemar's test                    McNemar (1947).
+  Cluster-robust variance (CR2)     Liang & Zeger (1986); Bell & McCaffrey
+                                    (2002).
+  Bayes factor asymptotics          Kass & Raftery (1995).
+  Spectrum bias                     Ransohoff & Feinstein (1978).