peeksafe 0.0.0-stage → 0.1.0

package/RESEARCH-NOTES.md

@@ -0,0 +1,274 @@
+# What is new here, and what is not
+
+peeksafe started life with four claims about being novel. This document is the adversarial
+literature check that was run against them, and the record of what it cost: **two of the four did not
+survive, one is a corollary of a textbook result, and one was not found on a search weak enough that
+its absence proves little.**
+
+It is kept in the repository, rather than quietly deleted once the README was corrected, for two
+reasons. A reader deciding whether to trust a statistics library should be able to see where its
+author was wrong. And the searches are written down, so anyone who thinks a verdict is too harsh,
+or too generous, can check the work instead of taking it on faith.
+
+**The measurements were never in question and none of them changed.** What changed is the sentence
+above them.
+
+## The short version
+
+| Claim | Where it lives | Verdict |
+| --- | --- | --- |
+| Sequential testing plus BH is invalid, twice over | `gate`, `bhCorrect`, `ebhCorrect` | **Known.** Taught as a named tutorial section since 2019 |
+| A cheap proxy screen's accuracy does not transfer | the `proxy` screen in `computeFrontier` | **Known since 1978.** It is spectrum bias |
+| The two-sample e-value has an evidence ceiling | `evidenceCeilingLogE`, `samplesForEvidence` | **Partially known.** A corollary of Laplace asymptotics; the closed form was not found |
+| The certifiable-within-budget frontier | `computeFrontier`, `enumerateFrontier` | **Not found**, on three searches, which is weak evidence for an engineering artifact |
+
+---
+
+## 1 · Sequential testing plus Benjamini-Hochberg
+
+**Verdict: known, both halves, and the more prominent half is a titled section in a widely-taught
+tutorial. This was the headline claim and it does not survive as stated.**
+
+The construction peeksafe exists to replace is: run each case until it looks decided, take a p-value
+at the stopping boundary, then correct across cases with Benjamini-Hochberg.
+
+**Half one: peeking invalidates the p-value.** Known since Armitage, McPherson and Rowe (1969), and
+thoroughly re-established in the industrial A/B testing literature, where it is called peeking and is
+the motivating problem for always-valid inference. Ramdas's KDD 2019 tutorial states it directly:
+"repeatedly running batch tests on accumulating data invalidates the p-value and inflates type-1
+error".
+
+**Half two: BH cannot consume those p-values.** This is not merely implied in that literature. It is
+a titled section of the same tutorial: **"Why Benjamini-Hochberg cannot be used online (5 mins)"**.
+The tutorial further warns that "ignoring the interplay with the outer sequential process could
+unknowingly inflate the number of false discoveries", which was this library's headline conclusion,
+published in 2019.
+
+The entire e-BH line of work exists because of this. Wang and Ramdas (2022) introduced e-BH so FDR
+control survives arbitrary dependence and sequential evidence, and the follow-up on stopped e-BH
+studies exactly what happens when e-processes are stopped adaptively and multiplicity is corrected
+afterwards. A reader who knows that literature reads the original framing as a restatement of its
+premise.
+
+**A third part, also known.** Testing against a historical rate treated as if it were known inflates
+type I error, and the standard remedy is to carry the reference arm's uncertainty into the null
+rather than plugging in a point estimate. That is standard in the external-control and historical-
+borrowing literature, and `gate`'s use of the baseline posterior as the null is that standard remedy
+correctly applied, not a discovery.
+
+### What genuinely survives
+
+One observation, and it is worth keeping:
+
+> BH's own conservativeness at large `m` masks the per-case invalidity, so the naive construction
+> produces visible false discoveries **more often at `m = 10` than at `m = 200`**, the opposite of
+> where a practitioner would look for it, and `m = 10` is the regime cost economics push you toward.
+
+That was not found stated anywhere. It is a quantitative nuance on a known defect rather than a new
+defect, and it is genuinely counterintuitive and practically useful. The measurement itself (4.6×
+per-case inflation on an eval suite, with the same underlying draws analysed all three ways) is also
+a contribution, because measurements on a new domain have value even when the theory is old.
+
+This is what the README's section "The part that is genuinely counterintuitive" now leads with, and
+the rest is presented as implementing a known-correct construction rather than discovering a defect.
+
+### Citations
+
+- Ramdas, *Fundamentals of large-scale sequential experimentation*, KDD 2019 tutorial, including the
+  section "Why Benjamini-Hochberg cannot be used online": https://stat.cmu.edu/~aramdas/kdd19/
+- Wang and Ramdas, *False discovery rate control with e-values*, 2022.
+- Wang, Dandapanthula and Ramdas, *Anytime-valid FDR control with the stopped e-BH procedure*, 2025:
+  https://arxiv.org/abs/2502.08539
+- Grünwald, de Heide and Koolen, *Safe Testing*: https://arxiv.org/abs/1906.07801
+- Principled type I error rate inflation in two-arm designs with external control borrowing:
+  https://arxiv.org/html/2508.16348
+- Armitage, McPherson and Rowe, *Repeated significance tests on accumulating data*, JRSS-A 1969.
+
+### Searches run
+
+Five. Optional stopping invalidating p-values with BH and FDR; anytime-valid FDR and stopped e-BH;
+external control type I error inflation with a plug-in null; e-value safe testing with informative
+nulls; and a direct fetch of the tutorial page, because a search result had already named the
+section. The confirming step was the fetch, not a search.
+
+---
+
+## 2 · The proxy screen that does not transfer
+
+**Verdict: known. It has a name and a 1978 citation.**
+
+`computeFrontier` offers three screens (`none`, `expensive`, and `proxy`) and prices the `proxy`
+one by attenuating its effect with Youden's J. The claim under check was that a cheap screen's
+accuracy measured on a whole suite does not transfer to the narrow population that actually decides
+anything.
+
+The phenomenon is **spectrum bias**, first reported by Ransohoff and Feinstein in 1978. Sensitivity
+and specificity, and therefore Youden's J, differ between subgroups with different severity or
+features, so a test's measured accuracy on a broad population does not carry to a narrow one. The
+modern literature refines this into a spectrum *effect* (accuracy differs by subgroup) becoming a
+spectrum *bias* (the difference changes the likelihood ratios, and therefore the decisions).
+
+A screen with J of 0.67 over a whole suite and J of 0.00 over the handful of genuinely regressed
+cases is that, exactly. The consequence for this library is a modelling constraint rather than a
+result: a J measured suite-wide is the wrong number to attenuate the frontier with, and
+`computeFrontier` therefore takes screen recall as a caller-supplied parameter rather than inferring
+one.
+
+### What would add value, and is not in this package
+
+Two things, and both are about demonstration rather than the phenomenon: a controlled design that
+holds the expensive labels fixed and varies only the text of a failing answer, and the distinction
+between a proxy that is *noisy* and one that is *structurally blind* to a failure mode, the latter
+explaining why no number of runs helps. Neither of those experiments ships here. peeksafe models the
+screen; it does not measure one for you.
+
+### Citations
+
+- Ransohoff and Feinstein, *Problems of spectrum and bias in evaluating the efficacy of diagnostic
+  tests*, NEJM 1978: https://pubmed.ncbi.nlm.nih.gov/692598/
+- Spectrum bias, Catalog of Bias: https://catalogofbias.org/biases/spectrum-bias/
+- *Distinguishing spectrum effects from spectrum biases*, BMC Medical Research Methodology 2008:
+  https://link.springer.com/article/10.1186/1471-2288-8-7
+- *Spectrum bias: why clinicians need to be cautious when applying diagnostic test studies*, Family
+  Practice: https://academic.oup.com/fampra/article/25/5/390/443697
+
+### Searches run
+
+One, which was decisive. The phenomenon is named and the first search returned the originating
+citation, so further searching would only have confirmed it.
+
+---
+
+## 3 · The evidence ceiling
+
+**Verdict: partially known. The mathematics is a corollary of a textbook result. The closed form, the
+naming, and the planning inversion were not found.**
+
+`twoSampleLogE` has a finite limit as candidate runs go to infinity, and `evidenceCeilingLogE`
+computes it. The claim under check was that this ceiling is a new observation.
+
+### What is established
+
+The asymptotic expansion behind the derivation is the standard Laplace approximation to a marginal
+likelihood, stated in Kass and Raftery's review as
+
+```
+log m(D) = log P(D | θ̂, H) − (d/2) log n + O(1)
+```
+
+With two hypotheses that share a likelihood and a parameter dimension, the first two terms are
+identical and cancel, leaving the `O(1)` term: the ratio of prior densities at the maximum. So a
+bounded Bayes factor in this situation is not a surprise; it is what the standard expansion says must
+happen.
+
+The complementary fact is equally standard and is stated the other way round in the Bayes factor
+design literature: a Bayes factor is consistent when the alternative prior is continuous and positive
+at the true value and the hypotheses are genuinely different distributions. The remark that "one
+expects a Bayes factor to grow without bound in the data, and it does when the hypotheses are
+different distributions" is a restatement of that known consistency condition, and should be read as
+citing it rather than naming a failure of intuition nobody had noticed.
+
+The practical consequence is also established, in a different vocabulary. In clinical trials the
+information obtainable when testing against a historical or external control is capped by that
+control's size, quantified as prior effective sample size with explicit maximal borrowing caps. That
+is the same statement as "the ceiling grows linearly in baseline runs", written for a trial-design
+audience.
+
+### What was not found
+
+- The phrase "evidence ceiling" as a named concept.
+- The explicit closed form for the two-sample beta-binomial case,
+  `ceiling(n_b) ≈ n_b · KL(p̄_b ‖ p) + log f_alt(p) + ½ log(2π p̄_b(1−p̄_b)/n_b)`.
+- The KL direction being called out as load bearing, with the 18% error from reversing it.
+- The planning inversion `n_b ≳ log(m/q) / KL(p̄_b ‖ p̄_b − δ)`, and the observation that no planner
+  prints it.
+
+Those four are the contribution, and `samplesForEvidence` is the last of them made executable. They
+are engineering artifacts derived from known asymptotics, which is worth publishing, but it is a
+different claim from discovering the ceiling.
+
+### Citations
+
+- Kass and Raftery, *Bayes Factors*, JASA 1995:
+  https://sites.stat.washington.edu/raftery/Research/PDF/weakliem1999.pdf
+- Laplace approximation and Bayesian asymptotics, the expansion in standard form:
+  https://www.stats.ox.ac.uk/~steffen/teaching/bs2HT9/bayes.pdf
+- Prior effective sample size when borrowing, Zhang et al., *Statistics in Medicine* 2025:
+  https://onlinelibrary.wiley.com/doi/10.1002/sim.70235
+- Scoping review of borrowing quantification, including maximal borrowing caps:
+  https://arxiv.org/abs/2605.27184
+- Bayes factor consistency conditions, closed-form power and sample size:
+  https://arxiv.org/pdf/2406.19940
+
+### Searches run
+
+Five. Bayes factor bounded, converging to a prior density ratio under Laplace; Bayes factor with two
+priors on the same parameter; e-value safe testing with bounded evidence against a historical control
+under an informative null; Kass and Raftery same-dimension cancellation; and the phrase "evidence
+ceiling" itself, which returned nothing using that term.
+
+---
+
+## 4 · The certifiable-within-budget frontier
+
+**Verdict: not found, on three searches. That is weak evidence, and adjacent work exists.**
+
+`computeFrontier` and `enumerateFrontier` produce a Pareto region of configurations that can be
+certified within a budget, with the amortisation window as a first-class axis. Neither the frontier
+as an object, nor the finding that the amortisation window rather than variance decides between
+paired and unpaired designs, was found.
+
+Two caveats a reader should weigh:
+
+1. **This is an engineering and economics artifact, not a theorem.** Work of this kind lives in
+   company engineering blogs and internal design documents far more often than in the literature, so
+   absence from academic search is weaker evidence here than for the claims above.
+2. **Adjacent work exists.** Casting benchmarking as finite-population inference under a fixed query
+   budget is an active topic, seeking tight confidence intervals for model accuracy with valid
+   coverage under a budget constraint. That is a different object (accuracy intervals under a query
+   budget, rather than a Pareto region of certifiable configurations with an amortisation axis), but
+   it is close enough to belong in related work rather than be left out.
+
+The classical paired-versus-unpaired decision is made on correlation and variance reduction. That the
+amortisation window displaces that criterion is, as far as these searches went, this library's own.
+It is stated as "we have not found this drawn anywhere" rather than as established novelty, and the
+distinction matters.
+
+### Citations
+
+- *Efficient Evaluation of LLM Performance with Statistical Guarantees*:
+  https://arxiv.org/html/2601.20251v1
+- Matched pairs design, standard exposition of the classical criterion:
+  https://www.growthbook.io/insights/matched-pairs-design-explained-definition-benefits
+
+### Searches run
+
+Three. Paired versus unpaired decided by control-arm cost and amortisation; LLM evaluation sample
+size and budget for regression detection in CI; and the historical-control angle from claim 3's
+searches, which surfaced the trial-design trade-offs but not a frontier.
+
+---
+
+## What this changed in the README
+
+Three sentences, all narrowing:
+
+1. The opening no longer says the sequential-plus-BH result is not written down anywhere. It says
+   outright that neither defect is a new observation, names the tutorial section, and states that
+   what the library contributes is the working construction plus a measurement of what the broken one
+   costs.
+2. The evidence ceiling is introduced as a corollary of the standard Laplace expansion, with the
+   historical-borrowing cap named, before the closed form and the planning rule are claimed.
+3. "As far as we can tell" is gone from the two places it could not be supported.
+
+What did **not** change: every number. The 4.6× per-case inflation, the 13-against-0 false
+discoveries, the ceiling ladder, the `log(m/q) / KL` planning rule against its exact answer, and the
+frontier cells. Prior art does not touch a measurement, and `test/paper.test.ts` still reproduces all
+of them from seeded draws with `pnpm test`.
+
+## Standing invitation
+
+If a verdict here is wrong in either direction (a claim marked known that isn't, or one marked "not
+found" that is sitting in a paper somewhere), the citation is worth more than the claim. Open an
+issue. The searches above are written down precisely so that this is checkable rather than a matter
+of trust.

package/dist/baseline.d.ts

@@ -0,0 +1,45 @@
+/**
+ * baseline.ts: what a case's reference revision did, and the null it implies.
+ *
+ * peeksafe never runs anything. A caller brings counts, and these are the shapes
+ * those counts arrive in.
+ *
+ * The distinction that matters is between a baseline that says "this case passed
+ * 51 of 60 times" and no baseline at all. The prototype this library came from
+ * treated a missing baseline as `0/60`, whose Beta(1,1) posterior median is 0.5,
+ * and cheerfully gated against a null of "this case passes half the time". It
+ * reported a verdict of PASS and a suite improvement of +87.5 points, measured
+ * against nothing. Every entry point here therefore takes `BaselineStat |
+ * undefined` explicitly and refuses to invent one.
+ */
+/** One case's recorded reference performance. */
+export interface BaselineStat {
+    caseId: string;
+    successes: number;
+    trials: number;
+}
+/**
+ * The minimum a case has to declare to be planned or gated. `suite` groups
+ * cases into families for the cluster-robust suite summary; leave it undefined
+ * and every case is its own family.
+ */
+export interface CaseRef {
+    id: string;
+    suite?: string;
+}
+/**
+ * The rate the null is tested at.
+ *
+ * Not `successes / trials`. A 60-run baseline at a true rate of 0.85 has a
+ * standard error of 0.046, which is a third of a 15-point effect, so testing
+ * against the point estimate treats a noisy number as an oracle. When the
+ * baseline happens to have come in high, every candidate looks worse than it is.
+ * Taking a low quantile of the baseline's own posterior instead is what stops
+ * that, at the cost of some power.
+ *
+ * `q` is that quantile. Lower is more conservative.
+ */
+export declare function baselineNullRate(b: BaselineStat, q?: number): number;
+/** Index a list of baselines by case id, rejecting duplicates rather than merging them. */
+export declare function toBaselineMap(stats: readonly BaselineStat[]): Map<string, BaselineStat>;
+//# sourceMappingURL=baseline.d.ts.map

package/dist/baseline.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"baseline.d.ts","sourceRoot":"","sources":["../src/baseline.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;GAaG;AAKH,iDAAiD;AACjD,MAAM,WAAW,YAAY;IAC3B,MAAM,EAAE,MAAM,CAAC;IACf,SAAS,EAAE,MAAM,CAAC;IAClB,MAAM,EAAE,MAAM,CAAC;CAChB;AAED;;;;GAIG;AACH,MAAM,WAAW,OAAO;IACtB,EAAE,EAAE,MAAM,CAAC;IACX,KAAK,CAAC,EAAE,MAAM,CAAC;CAChB;AAED;;;;;;;;;;;GAWG;AACH,wBAAgB,gBAAgB,CAAC,CAAC,EAAE,YAAY,EAAE,CAAC,SAAO,GAAG,MAAM,CAIlE;AAED,2FAA2F;AAC3F,wBAAgB,aAAa,CAAC,KAAK,EAAE,SAAS,YAAY,EAAE,GAAG,GAAG,CAAC,MAAM,EAAE,YAAY,CAAC,CAiBvF"}

package/dist/baseline.js

@@ -0,0 +1,49 @@
+/**
+ * baseline.ts: what a case's reference revision did, and the null it implies.
+ *
+ * peeksafe never runs anything. A caller brings counts, and these are the shapes
+ * those counts arrive in.
+ *
+ * The distinction that matters is between a baseline that says "this case passed
+ * 51 of 60 times" and no baseline at all. The prototype this library came from
+ * treated a missing baseline as `0/60`, whose Beta(1,1) posterior median is 0.5,
+ * and cheerfully gated against a null of "this case passes half the time". It
+ * reported a verdict of PASS and a suite improvement of +87.5 points, measured
+ * against nothing. Every entry point here therefore takes `BaselineStat |
+ * undefined` explicitly and refuses to invent one.
+ */
+import { PeeksafeError, requireCounts, requireOpenProbability } from "./errors.js";
+import { betaQuantile } from "./stats.js";
+/**
+ * The rate the null is tested at.
+ *
+ * Not `successes / trials`. A 60-run baseline at a true rate of 0.85 has a
+ * standard error of 0.046, which is a third of a 15-point effect, so testing
+ * against the point estimate treats a noisy number as an oracle. When the
+ * baseline happens to have come in high, every candidate looks worse than it is.
+ * Taking a low quantile of the baseline's own posterior instead is what stops
+ * that, at the cost of some power.
+ *
+ * `q` is that quantile. Lower is more conservative.
+ */
+export function baselineNullRate(b, q = 0.25) {
+    requireCounts(b.successes, b.trials, "baselineNullRate");
+    requireOpenProbability(q, "q", "baselineNullRate");
+    return betaQuantile(1 + b.successes, 1 + b.trials - b.successes, q);
+}
+/** Index a list of baselines by case id, rejecting duplicates rather than merging them. */
+export function toBaselineMap(stats) {
+    const m = new Map();
+    for (const b of stats) {
+        if (m.has(b.caseId)) {
+            throw new PeeksafeError("PEEKSAFE_E_CASE_DUPLICATE", `toBaselineMap: duplicate baseline for case ${b.caseId}`, {
+                detail: { caseId: b.caseId },
+                hint: "merging two baselines for one case would count its runs twice; dedupe them before calling",
+            });
+        }
+        requireCounts(b.successes, b.trials, `toBaselineMap(${b.caseId})`);
+        m.set(b.caseId, b);
+    }
+    return m;
+}
+//# sourceMappingURL=baseline.js.map

package/dist/baseline.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"baseline.js","sourceRoot":"","sources":["../src/baseline.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;GAaG;AAEH,OAAO,EAAE,aAAa,EAAE,aAAa,EAAE,sBAAsB,EAAE,MAAM,aAAa,CAAC;AACnF,OAAO,EAAE,YAAY,EAAE,MAAM,YAAY,CAAC;AAmB1C;;;;;;;;;;;GAWG;AACH,MAAM,UAAU,gBAAgB,CAAC,CAAe,EAAE,CAAC,GAAG,IAAI;IACxD,aAAa,CAAC,CAAC,CAAC,SAAS,EAAE,CAAC,CAAC,MAAM,EAAE,kBAAkB,CAAC,CAAC;IACzD,sBAAsB,CAAC,CAAC,EAAE,GAAG,EAAE,kBAAkB,CAAC,CAAC;IACnD,OAAO,YAAY,CAAC,CAAC,GAAG,CAAC,CAAC,SAAS,EAAE,CAAC,GAAG,CAAC,CAAC,MAAM,GAAG,CAAC,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC;AACtE,CAAC;AAED,2FAA2F;AAC3F,MAAM,UAAU,aAAa,CAAC,KAA8B;IAC1D,MAAM,CAAC,GAAG,IAAI,GAAG,EAAwB,CAAC;IAC1C,KAAK,MAAM,CAAC,IAAI,KAAK,EAAE,CAAC;QACtB,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,MAAM,CAAC,EAAE,CAAC;YACpB,MAAM,IAAI,aAAa,CACrB,2BAA2B,EAC3B,8CAA8C,CAAC,CAAC,MAAM,EAAE,EACxD;gBACE,MAAM,EAAE,EAAE,MAAM,EAAE,CAAC,CAAC,MAAM,EAAE;gBAC5B,IAAI,EAAE,2FAA2F;aAClG,CACF,CAAC;QACJ,CAAC;QACD,aAAa,CAAC,CAAC,CAAC,SAAS,EAAE,CAAC,CAAC,MAAM,EAAE,iBAAiB,CAAC,CAAC,MAAM,GAAG,CAAC,CAAC;QACnE,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC;IACrB,CAAC;IACD,OAAO,CAAC,CAAC;AACX,CAAC"}

package/dist/check.d.ts

@@ -0,0 +1,64 @@
+/**
+ * check.ts: the error rate and the power the gate delivers at rates you name.
+ *
+ * The two-sample e-value is not an e-value at every fixed rate (README, "What
+ * that guarantee rests on"), so its error control is computed rather than
+ * assumed. This is that computation, for a caller whose rates, baseline sizes
+ * or options sit outside the grid the tests check. Moving the candidate below
+ * the baseline turns the same computation into the power.
+ */
+import { type Evidence } from "./evidence.js";
+export interface TypeOneErrorOptions {
+    /** Shared true pass rate of the baseline and the candidate. */
+    rate: number;
+    /** Baseline runs the case will be gated against. */
+    baselineTrials: number;
+    /** The bar is `1 / alpha`. For `gate`'s solo bar pass `fdr / suiteSize`. */
+    alpha: number;
+    /** The most candidate runs any case could reach, e.g. `shouldStop`'s `maxTrials`. */
+    horizon: number;
+    mde?: number;
+    altConcentration?: number;
+    /** Which e-value `gate` will use. */
+    evidence?: Evidence;
+}
+export interface CertifyProbabilityOptions extends Omit<TypeOneErrorOptions, "rate"> {
+    /** True pass rate of the baseline revision. */
+    baselineRate: number;
+    /** True pass rate of the candidate revision. */
+    candidateRate: number;
+}
+/**
+ * Exact probability that a case is certified as regressed within `horizon`
+ * candidate runs, however it was stopped. With the candidate below the baseline
+ * this is the power; at or above it, the type I error.
+ *
+ * Exact because the e-value depends on the candidate only through its counts,
+ * so a dynamic program over the success count replaces simulation. A 240-run
+ * baseline at a horizon of 600 takes tens of milliseconds.
+ */
+export declare function certifyProbability(options: CertifyProbabilityOptions): number;
+/**
+ * Exact probability that a case which did not move is ever certified within
+ * `horizon` candidate runs, however it was stopped. At or below `alpha` means
+ * the gate keeps its promise for this case.
+ */
+export declare function typeOneError(options: TypeOneErrorOptions): number;
+export interface PairedCertifyProbabilityOptions {
+    /** True pass rate of the baseline revision. */
+    baselineRate: number;
+    /** True pass rate of the candidate revision. */
+    candidateRate: number;
+    /** Share of pairs whose two runs see the same random draw (a shared seed); 0 for independent runs. */
+    coupling?: number;
+    /** The bar is `1 / alpha`. For `gate`'s solo bar pass `fdr / suiteSize`. */
+    alpha: number;
+    /** The most pairs any case could reach. */
+    horizon: number;
+}
+/**
+ * `certifyProbability` for a paired case: the exact probability it is certified within
+ * `horizon` pairs, however it was stopped. The power when the candidate is lower.
+ */
+export declare function pairedCertifyProbability(options: PairedCertifyProbabilityOptions): number;
+//# sourceMappingURL=check.d.ts.map

package/dist/check.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"check.d.ts","sourceRoot":"","sources":["../src/check.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAQH,OAAO,EAAE,KAAK,QAAQ,EAAmB,MAAM,eAAe,CAAC;AAI/D,MAAM,WAAW,mBAAmB;IAClC,+DAA+D;IAC/D,IAAI,EAAE,MAAM,CAAC;IACb,oDAAoD;IACpD,cAAc,EAAE,MAAM,CAAC;IACvB,4EAA4E;IAC5E,KAAK,EAAE,MAAM,CAAC;IACd,qFAAqF;IACrF,OAAO,EAAE,MAAM,CAAC;IAChB,GAAG,CAAC,EAAE,MAAM,CAAC;IACb,gBAAgB,CAAC,EAAE,MAAM,CAAC;IAC1B,qCAAqC;IACrC,QAAQ,CAAC,EAAE,QAAQ,CAAC;CACrB;AAED,MAAM,WAAW,yBAA0B,SAAQ,IAAI,CAAC,mBAAmB,EAAE,MAAM,CAAC;IAClF,+CAA+C;IAC/C,YAAY,EAAE,MAAM,CAAC;IACrB,gDAAgD;IAChD,aAAa,EAAE,MAAM,CAAC;CACvB;AAED;;;;;;;;GAQG;AACH,wBAAgB,kBAAkB,CAAC,OAAO,EAAE,yBAAyB,GAAG,MAAM,CA6E7E;AA8BD;;;;GAIG;AACH,wBAAgB,YAAY,CAAC,OAAO,EAAE,mBAAmB,GAAG,MAAM,CAIjE;AAED,MAAM,WAAW,+BAA+B;IAC9C,+CAA+C;IAC/C,YAAY,EAAE,MAAM,CAAC;IACrB,gDAAgD;IAChD,aAAa,EAAE,MAAM,CAAC;IACtB,sGAAsG;IACtG,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,4EAA4E;IAC5E,KAAK,EAAE,MAAM,CAAC;IACd,2CAA2C;IAC3C,OAAO,EAAE,MAAM,CAAC;CACjB;AAED;;;GAGG;AACH,wBAAgB,wBAAwB,CAAC,OAAO,EAAE,+BAA+B,GAAG,MAAM,CAmDzF"}

package/dist/check.js

@@ -0,0 +1,172 @@
+/**
+ * check.ts: the error rate and the power the gate delivers at rates you name.
+ *
+ * The two-sample e-value is not an e-value at every fixed rate (README, "What
+ * that guarantee rests on"), so its error control is computed rather than
+ * assumed. This is that computation, for a caller whose rates, baseline sizes
+ * or options sit outside the grid the tests check. Moving the candidate below
+ * the baseline turns the same computation into the power.
+ */
+import { PeeksafeError, requireOpenProbability, requirePositiveConfig, requireProbability, } from "./errors.js";
+import { requireEvidence } from "./evidence.js";
+import { cannotDropBy, DEFAULT_GATE_OPTIONS } from "./gate.js";
+import { logBeta, logGamma, pairedLogE, twoSamplePriors } from "./stats.js";
+/**
+ * Exact probability that a case is certified as regressed within `horizon`
+ * candidate runs, however it was stopped. With the candidate below the baseline
+ * this is the power; at or above it, the type I error.
+ *
+ * Exact because the e-value depends on the candidate only through its counts,
+ * so a dynamic program over the success count replaces simulation. A 240-run
+ * baseline at a horizon of 600 takes tens of milliseconds.
+ */
+export function certifyProbability(options) {
+    const { baselineRate, candidateRate: pc, baselineTrials: nb, alpha, horizon } = options;
+    const mde = options.mde ?? DEFAULT_GATE_OPTIONS.mde;
+    const concentration = options.altConcentration ?? DEFAULT_GATE_OPTIONS.altConcentration;
+    const evidence = options.evidence ?? DEFAULT_GATE_OPTIONS.evidence;
+    requireEvidence(evidence, "certifyProbability");
+    requireOpenProbability(baselineRate, "baselineRate", "certifyProbability");
+    requireOpenProbability(pc, "candidateRate", "certifyProbability");
+    requireOpenProbability(alpha, "alpha", "certifyProbability");
+    requireOpenProbability(mde, "mde", "certifyProbability");
+    requirePositiveConfig(concentration, "altConcentration", "certifyProbability");
+    for (const [name, v] of [
+        ["baselineTrials", nb],
+        ["horizon", horizon],
+    ]) {
+        if (!Number.isInteger(v) || v < 1) {
+            throw new PeeksafeError("PEEKSAFE_E_CONFIG", `certifyProbability: ${name} must be a positive integer, got ${v}`, {
+                detail: { [name]: v },
+            });
+        }
+    }
+    const bar = Math.log(1 / alpha);
+    const logChoose = (n, k) => logGamma(n + 1) - logGamma(k + 1) - logGamma(n - k + 1);
+    let total = 0;
+    for (let sb = 0; sb <= nb; sb++) {
+        const weight = Math.exp(logChoose(nb, sb) + sb * Math.log(baselineRate) + (nb - sb) * Math.log(1 - baselineRate));
+        if (weight < 1e-12 || cannotDropBy({ successes: sb, trials: nb }, mde))
+            continue;
+        const { nullPrior: n0, altPrior: a1 } = twoSamplePriors(sb, nb, mde, concentration);
+        let alive = new Float64Array(horizon + 1);
+        alive[0] = 1;
+        let crossed = 0;
+        // Only the band of success counts still carrying mass is stepped.
+        let lo = 0;
+        let hi = 0;
+        for (let n = 1; n <= horizon; n++) {
+            const next = new Float64Array(horizon + 1);
+            for (let s = lo; s <= hi; s++) {
+                next[s + 1] += alive[s] * pc;
+                next[s] += alive[s] * (1 - pc);
+            }
+            hi++;
+            // Every count in the band is tested: logE is not monotone in s, so the
+            // crossing set need not be a prefix. Successive counts differ by a ratio
+            // of Beta functions, which keeps this cheap.
+            let logE = rawLogE(lo, n, sb, nb, n0, a1);
+            for (let s = lo; s <= hi; s++) {
+                if (s > lo) {
+                    const f = n - s + 1;
+                    logE +=
+                        Math.log((a1.a + s - 1) / (a1.b + f - 1)) - Math.log((n0.a + s - 1) / (n0.b + f - 1));
+                }
+                const decided = evidence === "universal"
+                    ? universalFromBayes(logE, s, n, sb, nb)
+                    : s / n >= sb / nb
+                        ? Math.min(0, logE)
+                        : logE;
+                if (decided >= bar) {
+                    crossed += next[s];
+                    next[s] = 0;
+                }
+            }
+            while (lo < hi && next[lo] < 1e-18)
+                lo++;
+            while (hi > lo && next[hi] < 1e-18)
+                hi--;
+            alive = next;
+        }
+        total += weight * crossed;
+    }
+    return total;
+}
+const xlogy = (x, y) => (x === 0 ? 0 : x * Math.log(y));
+/** The uncapped Bayes factor, which both statistics are built from. */
+const rawLogE = (s, n, _sb, _nb, n0, a1) => logBeta(a1.a + s, a1.b + n - s) -
+    logBeta(a1.a, a1.b) -
+    (logBeta(n0.a + s, n0.b + n - s) - logBeta(n0.a, n0.b));
+/** `universalTwoSampleLogE`, from the uncapped Bayes factor: times the pooled mixture over the null maximum. */
+function universalFromBayes(rawLog, s, n, sb, nb) {
+    const S = sb + s;
+    const N = nb + n;
+    const pb = sb / nb;
+    const pc = s / n;
+    const sup = pc >= pb
+        ? xlogy(sb, pb) + xlogy(nb - sb, 1 - pb) + xlogy(s, pc) + xlogy(n - s, 1 - pc)
+        : xlogy(S, S / N) + xlogy(N - S, 1 - S / N);
+    return rawLog + logBeta(1 + S, 1 + N - S) - sup;
+}
+/**
+ * Exact probability that a case which did not move is ever certified within
+ * `horizon` candidate runs, however it was stopped. At or below `alpha` means
+ * the gate keeps its promise for this case.
+ */
+export function typeOneError(options) {
+    const { rate, ...rest } = options;
+    requireOpenProbability(rate, "rate", "typeOneError");
+    return certifyProbability({ ...rest, baselineRate: rate, candidateRate: rate });
+}
+/**
+ * `certifyProbability` for a paired case: the exact probability it is certified within
+ * `horizon` pairs, however it was stopped. The power when the candidate is lower.
+ */
+export function pairedCertifyProbability(options) {
+    const { baselineRate: pb, candidateRate: pc, alpha, horizon } = options;
+    const rho = options.coupling ?? 0;
+    requireOpenProbability(pb, "baselineRate", "pairedCertifyProbability");
+    requireOpenProbability(pc, "candidateRate", "pairedCertifyProbability");
+    requireOpenProbability(alpha, "alpha", "pairedCertifyProbability");
+    requireProbability(rho, "coupling", "pairedCertifyProbability");
+    if (!Number.isInteger(horizon) || horizon < 1) {
+        throw new PeeksafeError("PEEKSAFE_E_CONFIG", `pairedCertifyProbability: horizon must be a positive integer, got ${horizon}`, {
+            detail: { horizon },
+        });
+    }
+    const worse = rho * Math.max(0, pb - pc) + (1 - rho) * pb * (1 - pc);
+    const better = rho * Math.max(0, pc - pb) + (1 - rho) * pc * (1 - pb);
+    const bar = Math.log(1 / alpha);
+    // The e-value depends on (worse, discordant) only, so crossing is tabulated once per state.
+    const crosses = Array.from({ length: horizon + 1 }, (_, d) => Uint8Array.from({ length: d + 1 }, (_, w) => (pairedLogE(w, d) >= bar ? 1 : 0)));
+    // alive[d][w]: probability of d discordant pairs, w of them worse, and no crossing yet.
+    let alive = [Float64Array.of(1)];
+    let crossed = 0;
+    for (let n = 1; n <= horizon; n++) {
+        const next = Array.from({ length: n + 1 }, (_, d) => new Float64Array(d + 1));
+        for (let d = 0; d < n; d++) {
+            const row = alive[d];
+            for (let w = 0; w <= d; w++) {
+                const m = row[w];
+                if (m < 1e-300)
+                    continue;
+                next[d][w] += m * (1 - worse - better);
+                next[d + 1][w + 1] += m * worse;
+                next[d + 1][w] += m * better;
+            }
+        }
+        for (let d = 1; d <= n; d++) {
+            const row = next[d];
+            const cross = crosses[d];
+            for (let w = 0; w <= d; w++) {
+                if (cross[w] && row[w] > 0) {
+                    crossed += row[w];
+                    row[w] = 0;
+                }
+            }
+        }
+        alive = next;
+    }
+    return crossed;
+}
+//# sourceMappingURL=check.js.map

package/dist/check.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"check.js","sourceRoot":"","sources":["../src/check.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAEH,OAAO,EACL,aAAa,EACb,sBAAsB,EACtB,qBAAqB,EACrB,kBAAkB,GACnB,MAAM,aAAa,CAAC;AACrB,OAAO,EAAiB,eAAe,EAAE,MAAM,eAAe,CAAC;AAC/D,OAAO,EAAE,YAAY,EAAE,oBAAoB,EAAE,MAAM,WAAW,CAAC;AAC/D,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,YAAY,CAAC;AAwB5E;;;;;;;;GAQG;AACH,MAAM,UAAU,kBAAkB,CAAC,OAAkC;IACnE,MAAM,EAAE,YAAY,EAAE,aAAa,EAAE,EAAE,EAAE,cAAc,EAAE,EAAE,EAAE,KAAK,EAAE,OAAO,EAAE,GAAG,OAAO,CAAC;IACxF,MAAM,GAAG,GAAG,OAAO,CAAC,GAAG,IAAI,oBAAoB,CAAC,GAAG,CAAC;IACpD,MAAM,aAAa,GAAG,OAAO,CAAC,gBAAgB,IAAI,oBAAoB,CAAC,gBAAgB,CAAC;IACxF,MAAM,QAAQ,GAAG,OAAO,CAAC,QAAQ,IAAI,oBAAoB,CAAC,QAAQ,CAAC;IACnE,eAAe,CAAC,QAAQ,EAAE,oBAAoB,CAAC,CAAC;IAChD,sBAAsB,CAAC,YAAY,EAAE,cAAc,EAAE,oBAAoB,CAAC,CAAC;IAC3E,sBAAsB,CAAC,EAAE,EAAE,eAAe,EAAE,oBAAoB,CAAC,CAAC;IAClE,sBAAsB,CAAC,KAAK,EAAE,OAAO,EAAE,oBAAoB,CAAC,CAAC;IAC7D,sBAAsB,CAAC,GAAG,EAAE,KAAK,EAAE,oBAAoB,CAAC,CAAC;IACzD,qBAAqB,CAAC,aAAa,EAAE,kBAAkB,EAAE,oBAAoB,CAAC,CAAC;IAC/E,KAAK,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC,IAAI;QACtB,CAAC,gBAAgB,EAAE,EAAE,CAAC;QACtB,CAAC,SAAS,EAAE,OAAO,CAAC;KACZ,EAAE,CAAC;QACX,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;YAClC,MAAM,IAAI,aAAa,CACrB,mBAAmB,EACnB,uBAAuB,IAAI,oCAAoC,CAAC,EAAE,EAClE;gBACE,MAAM,EAAE,EAAE,CAAC,IAAI,CAAC,EAAE,CAAC,EAAE;aACtB,CACF,CAAC;QACJ,CAAC;IACH,CAAC;IAED,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,CAAC,CAAC;IAChC,MAAM,SAAS,GAAG,CAAC,CAAS,EAAE,CAAS,EAAE,EAAE,CACzC,QAAQ,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,QAAQ,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,QAAQ,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC;IAC1D,IAAI,KAAK,GAAG,CAAC,CAAC;IACd,KAAK,IAAI,EAAE,GAAG,CAAC,EAAE,EAAE,IAAI,EAAE,EAAE,EAAE,EAAE,EAAE,CAAC;QAChC,MAAM,MAAM,GAAG,IAAI,CAAC,GAAG,CACrB,SAAS,CAAC,EAAE,EAAE,EAAE,CAAC,GAAG,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,YAAY,CAAC,GAAG,CAAC,EAAE,GAAG,EAAE,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,YAAY,CAAC,CACzF,CAAC;QACF,IAAI,MAAM,GAAG,KAAK,IAAI,YAAY,CAAC,EAAE,SAAS,EAAE,EAAE,EAAE,MAAM,EAAE,EAAE,EAAE,EAAE,GAAG,CAAC;YAAE,SAAS;QACjF,MAAM,EAAE,SAAS,EAAE,EAAE,EAAE,QAAQ,EAAE,EAAE,EAAE,GAAG,eAAe,CAAC,EAAE,EAAE,EAAE,EAAE,GAAG,EAAE,aAAa,CAAC,CAAC;QACpF,IAAI,KAAK,GAAG,IAAI,YAAY,CAAC,OAAO,GAAG,CAAC,CAAC,CAAC;QAC1C,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC;QACb,IAAI,OAAO,GAAG,CAAC,CAAC;QAChB,kEAAkE;QAClE,IAAI,EAAE,GAAG,CAAC,CAAC;QACX,IAAI,EAAE,GAAG,CAAC,CAAC;QACX,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,IAAI,OAAO,EAAE,CAAC,EAAE,EAAE,CAAC;YAClC,MAAM,IAAI,GAAG,IAAI,YAAY,CAAC,OAAO,GAAG,CAAC,CAAC,CAAC;YAC3C,KAAK,IAAI,CAAC,GAAG,EAAE,EAAE,CAAC,IAAI,EAAE,EAAE,CAAC,EAAE,EAAE,CAAC;gBAC9B,IAAI,CAAC,CAAC,GAAG,CAAC,CAAE,IAAI,KAAK,CAAC,CAAC,CAAE,GAAG,EAAE,CAAC;gBAC/B,IAAI,CAAC,CAAC,CAAE,IAAI,KAAK,CAAC,CAAC,CAAE,GAAG,CAAC,CAAC,GAAG,EAAE,CAAC,CAAC;YACnC,CAAC;YACD,EAAE,EAAE,CAAC;YACL,uEAAuE;YACvE,yEAAyE;YACzE,6CAA6C;YAC7C,IAAI,IAAI,GAAG,OAAO,CAAC,EAAE,EAAE,CAAC,EAAE,EAAE,EAAE,EAAE,EAAE,EAAE,EAAE,EAAE,CAAC,CAAC;YAC1C,KAAK,IAAI,CAAC,GAAG,EAAE,EAAE,CAAC,IAAI,EAAE,EAAE,CAAC,EAAE,EAAE,CAAC;gBAC9B,IAAI,CAAC,GAAG,EAAE,EAAE,CAAC;oBACX,MAAM,CAAC,GAAG,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;oBACpB,IAAI;wBACF,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;gBAC1F,CAAC;gBACD,MAAM,OAAO,GACX,QAAQ,KAAK,WAAW;oBACtB,CAAC,CAAC,kBAAkB,CAAC,IAAI,EAAE,CAAC,EAAE,CAAC,EAAE,EAAE,EAAE,EAAE,CAAC;oBACxC,CAAC,CAAC,CAAC,GAAG,CAAC,IAAI,EAAE,GAAG,EAAE;wBAChB,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC;wBACnB,CAAC,CAAC,IAAI,CAAC;gBACb,IAAI,OAAO,IAAI,GAAG,EAAE,CAAC;oBACnB,OAAO,IAAI,IAAI,CAAC,CAAC,CAAE,CAAC;oBACpB,IAAI,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC;gBACd,CAAC;YACH,CAAC;YACD,OAAO,EAAE,GAAG,EAAE,IAAI,IAAI,CAAC,EAAE,CAAE,GAAG,KAAK;gBAAE,EAAE,EAAE,CAAC;YAC1C,OAAO,EAAE,GAAG,EAAE,IAAI,IAAI,CAAC,EAAE,CAAE,GAAG,KAAK;gBAAE,EAAE,EAAE,CAAC;YAC1C,KAAK,GAAG,IAAI,CAAC;QACf,CAAC;QACD,KAAK,IAAI,MAAM,GAAG,OAAO,CAAC;IAC5B,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,MAAM,KAAK,GAAG,CAAC,CAAS,EAAE,CAAS,EAAU,EAAE,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC;AAEhF,uEAAuE;AACvE,MAAM,OAAO,GAAG,CACd,CAAS,EACT,CAAS,EACT,GAAW,EACX,GAAW,EACX,EAA4B,EAC5B,EAA4B,EACpB,EAAE,CACV,OAAO,CAAC,EAAE,CAAC,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;IAC/B,OAAO,CAAC,EAAE,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC;IACnB,CAAC,OAAO,CAAC,EAAE,CAAC,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,GAAG,OAAO,CAAC,EAAE,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC;AAE1D,gHAAgH;AAChH,SAAS,kBAAkB,CAAC,MAAc,EAAE,CAAS,EAAE,CAAS,EAAE,EAAU,EAAE,EAAU;IACtF,MAAM,CAAC,GAAG,EAAE,GAAG,CAAC,CAAC;IACjB,MAAM,CAAC,GAAG,EAAE,GAAG,CAAC,CAAC;IACjB,MAAM,EAAE,GAAG,EAAE,GAAG,EAAE,CAAC;IACnB,MAAM,EAAE,GAAG,CAAC,GAAG,CAAC,CAAC;IACjB,MAAM,GAAG,GACP,EAAE,IAAI,EAAE;QACN,CAAC,CAAC,KAAK,CAAC,EAAE,EAAE,EAAE,CAAC,GAAG,KAAK,CAAC,EAAE,GAAG,EAAE,EAAE,CAAC,GAAG,EAAE,CAAC,GAAG,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,GAAG,KAAK,CAAC,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,EAAE,CAAC;QAC9E,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,CAAC,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC;IAChD,OAAO,MAAM,GAAG,OAAO,CAAC,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,GAAG,GAAG,CAAC;AAClD,CAAC;AAED;;;;GAIG;AACH,MAAM,UAAU,YAAY,CAAC,OAA4B;IACvD,MAAM,EAAE,IAAI,EAAE,GAAG,IAAI,EAAE,GAAG,OAAO,CAAC;IAClC,sBAAsB,CAAC,IAAI,EAAE,MAAM,EAAE,cAAc,CAAC,CAAC;IACrD,OAAO,kBAAkB,CAAC,EAAE,GAAG,IAAI,EAAE,YAAY,EAAE,IAAI,EAAE,aAAa,EAAE,IAAI,EAAE,CAAC,CAAC;AAClF,CAAC;AAeD;;;GAGG;AACH,MAAM,UAAU,wBAAwB,CAAC,OAAwC;IAC/E,MAAM,EAAE,YAAY,EAAE,EAAE,EAAE,aAAa,EAAE,EAAE,EAAE,KAAK,EAAE,OAAO,EAAE,GAAG,OAAO,CAAC;IACxE,MAAM,GAAG,GAAG,OAAO,CAAC,QAAQ,IAAI,CAAC,CAAC;IAClC,sBAAsB,CAAC,EAAE,EAAE,cAAc,EAAE,0BAA0B,CAAC,CAAC;IACvE,sBAAsB,CAAC,EAAE,EAAE,eAAe,EAAE,0BAA0B,CAAC,CAAC;IACxE,sBAAsB,CAAC,KAAK,EAAE,OAAO,EAAE,0BAA0B,CAAC,CAAC;IACnE,kBAAkB,CAAC,GAAG,EAAE,UAAU,EAAE,0BAA0B,CAAC,CAAC;IAChE,IAAI,CAAC,MAAM,CAAC,SAAS,CAAC,OAAO,CAAC,IAAI,OAAO,GAAG,CAAC,EAAE,CAAC;QAC9C,MAAM,IAAI,aAAa,CACrB,mBAAmB,EACnB,qEAAqE,OAAO,EAAE,EAC9E;YACE,MAAM,EAAE,EAAE,OAAO,EAAE;SACpB,CACF,CAAC;IACJ,CAAC;IACD,MAAM,KAAK,GAAG,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,GAAG,EAAE,CAAC,GAAG,CAAC,CAAC,GAAG,GAAG,CAAC,GAAG,EAAE,GAAG,CAAC,CAAC,GAAG,EAAE,CAAC,CAAC;IACrE,MAAM,MAAM,GAAG,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,GAAG,EAAE,CAAC,GAAG,CAAC,CAAC,GAAG,GAAG,CAAC,GAAG,EAAE,GAAG,CAAC,CAAC,GAAG,EAAE,CAAC,CAAC;IACtE,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,CAAC,CAAC;IAChC,4FAA4F;IAC5F,MAAM,OAAO,GAAG,KAAK,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,OAAO,GAAG,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAC3D,UAAU,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,UAAU,CAAC,CAAC,EAAE,CAAC,CAAC,IAAI,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAChF,CAAC;IACF,wFAAwF;IACxF,IAAI,KAAK,GAAmB,CAAC,YAAY,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC;IACjD,IAAI,OAAO,GAAG,CAAC,CAAC;IAChB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,IAAI,OAAO,EAAE,CAAC,EAAE,EAAE,CAAC;QAClC,MAAM,IAAI,GAAG,KAAK,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,IAAI,YAAY,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;QAC9E,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC;YAC3B,MAAM,GAAG,GAAG,KAAK,CAAC,CAAC,CAAE,CAAC;YACtB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,IAAI,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC;gBAC5B,MAAM,CAAC,GAAG,GAAG,CAAC,CAAC,CAAE,CAAC;gBAClB,IAAI,CAAC,GAAG,MAAM;oBAAE,SAAS;gBACzB,IAAI,CAAC,CAAC,CAAE,CAAC,CAAC,CAAE,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,GAAG,MAAM,CAAC,CAAC;gBACzC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAE,CAAC,CAAC,GAAG,CAAC,CAAE,IAAI,CAAC,GAAG,KAAK,CAAC;gBAClC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAE,CAAC,CAAC,CAAE,IAAI,CAAC,GAAG,MAAM,CAAC;YACjC,CAAC;QACH,CAAC;QACD,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,IAAI,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC;YAC5B,MAAM,GAAG,GAAG,IAAI,CAAC,CAAC,CAAE,CAAC;YACrB,MAAM,KAAK,GAAG,OAAO,CAAC,CAAC,CAAE,CAAC;YAC1B,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,IAAI,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC;gBAC5B,IAAI,KAAK,CAAC,CAAC,CAAC,IAAI,GAAG,CAAC,CAAC,CAAE,GAAG,CAAC,EAAE,CAAC;oBAC5B,OAAO,IAAI,GAAG,CAAC,CAAC,CAAE,CAAC;oBACnB,GAAG,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC;gBACb,CAAC;YACH,CAAC;QACH,CAAC;QACD,KAAK,GAAG,IAAI,CAAC;IACf,CAAC;IACD,OAAO,OAAO,CAAC;AACjB,CAAC"}