@mark1russell7/rigor-bench 0.2.13 → 0.2.15

package/README.md

@@ -42,6 +42,8 @@ v.comparison;                                     // the cases that gain and dro
 | `pooledBand(pool, level)` | The pooled band: the larger of the floor and the A/A spread of the records of earlier runs. |
 | `caseIntervals(records, level)` | The ratio of each case with a distribution-free interval of its median over the rounds. The level 0.95 needs 6 rounds or more. |
 | `limitVerdict(candidate, aa, options)` | The verdict of the claim "the ratio of each case is at most `limit`": `within`, `beyond` or `cannot decide`. |
+| `gateOf(cases)` | The gate of a list of case outcomes: `pass`, `block` or `no read`. |
+| `notSlowerVerdict(jobs, options)` | The gate of the claim "not slower by more than the margin", with looks and a family level. |
 | `budgetVerdict(rounds, aa, options)` | The verdict of the claim "the median of each case is at most `limit`", for one build: `within`, `beyond` or `cannot decide`. |
 | `budgetLooksVerdict(rounds, aa, options)` | The budget verdict of a run that adds rounds to the cases with no outcome, up to a stated budget. |
 | `budgetLookSpend(looks)` | The chance of error that a list of looks spends. |
@@ -78,6 +80,31 @@ v.cases;                                          // the interval, the bound and
 
 The verdict has no correction for many cases. Each case has its own interval at the level `caseLevel`. Thus with many cases, a verdict `beyond` can come from noise only. Set `caseLevel` higher when a bench has many cases. A correction of Holm is an option for a later version.
 
+## A gate that can fail
+
+A check that blocks a merge needs a result that can fail. `gateOf` reads the case outcomes of a verdict, and `notSlowerVerdict` gives the claim of a speed gate.
+
+```ts
+const gate = notSlowerVerdict(jobs, { margin: Math.log(1.015) });
+
+gate.outcome;                                     // "pass", "block" or "no read"
+gate.cases;                                       // "not slower", "slower" or "no read" for each case, with the facts of each job
+gate.more;                                        // the next look of each case with no outcome
+```
+
+- The claim is "not slower by more than the margin". A one-sided bound answers it, thus a case can read `not slower` when its interval holds 1. The word `same` of `timeVerdict` asks for more.
+- The outcome of each case uses the words of `LimitOutcome`: `within` is `not slower`, `beyond` is `slower`, and `cannot decide` is `no read`. Thus `gateOf` also reads `limitVerdict` and `budgetVerdict`.
+- A case needs its bound in each of `jobs` jobs, by default 2. More rounds do not remove a difference between jobs.
+- The claim "the bound holds in at least `r` of `J` jobs" is a partial conjunction. Its p is `(J - r + 1)` times the `r`-th smallest p of the jobs (Benjamini and Heller, 2008). With 2 jobs of 2, this p is the larger p.
+- The sign test compares the median difference with the margin in each job. It uses no random numbers.
+- A pair holds two records with the same first arm. A runner can give the two arms opposite orders in one round, and a difference of two orders holds the effect of the order.
+- The `level` is for the family. `holm` corrects each look over all cases. Each look has an equal share of the level.
+- The first look with an outcome decides a case. `GATE_LOOKS` is 12, 24 and 48 rounds.
+- Take 16 cases and the default looks. The first test of a family at 12 rounds needs 12 of 12 pairs below the margin in both jobs. Thus a noisy case needs more rounds, or it stays `no read`.
+- `no read` is not a pass. A caller adds the rounds of `more`, then more jobs. A job that it adds after the last look is one more look, and the level does not count it.
+
+The level needs independent rounds. Rounds with a shared cause, for example a slow computer for one job, do not have it.
+
 ## A claim with a budget
 
 `limitVerdict` judges a ratio of two arms. A budget is an absolute limit on one build. An example is "the median blocking time of this page is at most 450 ms". A budget needs `budgetVerdict`:
@@ -206,6 +233,62 @@ const shards = controlFactor(shardRows);
 - The fields `low` and `high` are the quartiles of the ratios. A wide spread means that one factor does not explain the load.
 - The function refuses a group with fewer usable keys than `minPairs`. The default is 5.
 
+## Check that a trick does not buy a measure
+
+A control is a strategy made to win one measure and nothing else. A real entry can win that measure by the same trick. So `checkMeasures` reads the controls of a bench, and `winClaim` tells a page what to show beside a win. Each project keeps its own controls.
+
+```ts
+import { checkMeasures, winClaim } from "@mark1russell7/rigor-bench";
+
+const check = checkMeasures(rows, {
+  asOf: "2026-11-05",
+  controls: ["only-fill"],
+  rules: [
+    { measure: "fill", sense: "higher", controls: ["only-fill"], partners: ["order"] },
+    { measure: "order", sense: "lower", controls: [], partners: [], noControl: { because: "The control has no code yet.", until: "2026-11-20" } },
+  ],
+});
+if (check.findings.length > 0) process.exitCode = 1;
+const claim = winClaim(check, "fill", "my-entry");
+```
+
+Each row has a `measure`, a `group`, a `key` and a `value`. The optional fields are `censored`, `method` and `stratum`. The function throws a `RangeError` for a repeat of a measure, a group and a key. The caller states how repeats reduce to one row.
+
+- **Beats:** the Holm-corrected sign test passes. With a `resolution`, the median paired difference is also over it. Without one, the output says `test only`.
+- **A control wins** when no real entry beats it. A tie counts for the control.
+- **A control loses a partner** when one real entry beats it on that partner, at the level divided by the count of partners. One passed partner clears `uncaught`, so each partner is one more chance.
+- **Families:** one Holm family holds one control on one measure, over all real entries and all strata. A second family holds the same tests in the other direction, and `beatsEntries` of `of` gives its count. One family holds the spread of one measure.
+- **Too few:** the need comes from `signTestPower` at the level of that family, over the untied keys. So 31 real entries need 10 untied keys at the level 0.05. A test under the need beats nothing. A pair with enough keys that ties reads as equal.
+- **Strata:** the keys of one stratum come from one family of inputs, so they are not independent draws. A result is about these keys.
+
+Strata make a family larger, so each stratum needs more keys:
+
+| The family | Tests | Sides | Untied keys for each test |
+| --- | --- | --- | --- |
+| 31 entries, no strata | 31 | 1 | 10 |
+| 14 entries, 15 strata | 210 | 1 | 13 |
+| 30 entries, 20 strata | 600 | 1 | 14 |
+| The spread of 31 entries, 20 strata | 9,300 | 2 | 19 |
+
+So a stratum of 10 keys reads `too-few` by construction. That result is true, and the check keeps it. A caller gets the pooled read when the rows have no `stratum`. A sign test needs independent signs and not equal distributions, so a read can pool the keys of different families. A bench reads pooled for its gate, and by stratum where the keys are sufficient.
+
+Each finding fails a gate, and each one carries its tests with `n`, `ties`, `p` and the Holm result.
+
+| Finding | Meaning |
+| --- | --- |
+| `uncaught` | A control wins the measure and loses no partner. The trick pays. |
+| `weak-control` | A control with power beats no real entry on its measure, in any stratum. The trick buys nothing. A control in second place that beats other entries is not weak. |
+| `censored-win` | A control wins with more censored keys than an entry it is read against. |
+| `no-spread` | No pair of real entries separates on the measure. |
+| `too-few` | No read of the controls or of the spread has enough keys. A control that wins and has a partner with too few keys also gives it. A small stratum alone gives only a row. |
+| `mixed-method` | The rows mix methods, or a method and no method. The check has no read of the measure. |
+| `no-rule` | The rows hold a measure with no rule. |
+| `no-rows` | A rule names a measure or a control with no rows. This does not read as "not beaten". |
+| `no-control` | A rule has no control and no `noControl`. |
+| `control-late` | The date of `noControl` is before `asOf`. |
+
+`winClaim(check, measure, entry, stratum)` gives `alone`, `show`, `likeControl`, `censored` and `open`. The fields are about this entry. The field `likeControl` is true when this entry does not beat a control on the measure and beats it on no partner. A win by another entry clears nothing. The field `alone` is true only when this entry itself beats each control on the measure, so a measure with no control is not `alone`.
+
 ## The fresh runner
 
 The entry `@mark1russell7/rigor-bench/fresh` starts child processes, so it is for Node only. The root entry stays free of `node:child_process`.