skiftet_statistical 0.1.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +346 -2
  3. data/README.md +1618 -22
  4. data/lib/skiftet_statistical/additive_contrasts.rb +894 -0
  5. data/lib/skiftet_statistical/bandit/allocator.rb +688 -0
  6. data/lib/skiftet_statistical/bandit/checks.rb +128 -0
  7. data/lib/skiftet_statistical/bandit/decay.rb +86 -0
  8. data/lib/skiftet_statistical/bandit/draw.rb +179 -0
  9. data/lib/skiftet_statistical/bandit/normal_argmax.rb +209 -0
  10. data/lib/skiftet_statistical/bandit/pass.rb +94 -0
  11. data/lib/skiftet_statistical/bandit/posteriors.rb +497 -0
  12. data/lib/skiftet_statistical/bandit/prior.rb +264 -0
  13. data/lib/skiftet_statistical/bandit/probabilities.rb +929 -0
  14. data/lib/skiftet_statistical/bandit/ranking.rb +552 -0
  15. data/lib/skiftet_statistical/bandit/summary.rb +152 -0
  16. data/lib/skiftet_statistical/bandit/tallies.rb +297 -0
  17. data/lib/skiftet_statistical/bandit/weighted_posterior.rb +65 -0
  18. data/lib/skiftet_statistical/bandit/win_hold.rb +86 -0
  19. data/lib/skiftet_statistical/bandit.rb +3 -3
  20. data/lib/skiftet_statistical/beta_argmax.rb +526 -0
  21. data/lib/skiftet_statistical/beta_function.rb +153 -0
  22. data/lib/skiftet_statistical/cholesky.rb +244 -0
  23. data/lib/skiftet_statistical/cusum.rb +183 -0
  24. data/lib/skiftet_statistical/descriptive.rb +3 -3
  25. data/lib/skiftet_statistical/gamma.rb +97 -0
  26. data/lib/skiftet_statistical/gauss_legendre.rb +73 -0
  27. data/lib/skiftet_statistical/hierarchical_rates.rb +450 -0
  28. data/lib/skiftet_statistical/inputs.rb +214 -0
  29. data/lib/skiftet_statistical/joint_beta.rb +63 -0
  30. data/lib/skiftet_statistical/linear_value.rb +364 -0
  31. data/lib/skiftet_statistical/numbers.rb +98 -0
  32. data/lib/skiftet_statistical/prequential.rb +206 -0
  33. data/lib/skiftet_statistical/shared_credit.rb +811 -0
  34. data/lib/skiftet_statistical/significance.rb +65 -5
  35. data/lib/skiftet_statistical/version.rb +1 -1
  36. data/lib/skiftet_statistical.rb +57 -5
  37. metadata +43 -11
data/README.md CHANGED
@@ -1,24 +1,64 @@
1
1
  # skiftet_statistical
2
2
 
3
- Skiftet's shared, dependency-free **statistics toolkit** — the workspace home for
4
- reusable statistical analysis code, so the same z-test, percentile, or sampler
5
- isn't re-implemented (differently) in every app.
3
+ A dependency-free **statistics toolkit** for Ruby: reusable statistical analysis
4
+ code, so the same z-test, percentile, sampler or bandit isn't re-implemented
5
+ (differently) in every app.
6
6
 
7
7
  Modules:
8
8
 
9
9
  - **`Descriptive`** — mean, variance, standard deviation, percentiles/median.
10
10
  - **`Significance`** — A/B significance testing: two-proportion z-test, Welch's
11
- t-test, exact normal CDF and two-tailed p-values.
11
+ t-test, exact normal CDF and two-tailed p-values, the normal quantile, the
12
+ chi-squared survival and the sample-ratio check.
13
+ - **`Gamma`** / **`BetaFunction`** — the regularized incomplete gamma and beta
14
+ functions, and the Beta quantile.
12
15
  - **`Sampler`** — Gamma/Beta/Gaussian random sampling, RNG-injectable.
16
+ - **`JointBeta`** — joint Monte Carlo draws over Beta posteriors (P(best),
17
+ expected loss).
13
18
  - **`Bandit`** (+ `Policies`) — multi-armed bandit (Thompson Sampling,
14
19
  epsilon-greedy, UCB1, Softmax) for online explore/exploit decisions.
20
+ - **`Bandit::Posteriors`** (with **`Bandit::Prior`**, **`Bandit::Decay`**,
21
+ **`Bandit::Draw`**, **`Bandit::NormalArgmax`** and **`Bandit::Pass`**) — new
22
+ in 0.4.0, the generic bandit: a set of items with their views and yes/no or
23
+ amount posteriors, ordered and picked by one rule from a key, given uniforms
24
+ or an RNG, with the odds of what it draws (exact for a pick among up to
25
+ `EXACT_PICK_MAX` (20) contenders and for orders and positions over up to
26
+ `EXACT_ORDER_MAX` (10) free items with none barred from leading, otherwise a
27
+ seeded Monte Carlo estimate with its standard error) and the summaries an
28
+ allocation reads.
29
+ - **`Bandit::Allocator`**, **`Bandit::WinHold`**, **`Bandit::WeightedPosterior`**
30
+ — an A/B test's traffic allocation: Thompson sampling with a floor per arm, a
31
+ step limit, an automatic win and a hold on it.
32
+ Also per-arm floors and fixed shares, and one phase rule for any posterior
33
+ (`Allocator.from_summary`).
34
+ - **`LinearValue`** (with **`Cholesky`**) — P(best), expected loss and
35
+ intervals of a linear value of several rates when both the rates (one joint
36
+ normal) and their weights are uncertain.
37
+ - **`SharedCredit`** — Hájek inverse-propensity means of several parts per arm,
38
+ with one covariance over every (arm, part) pair, when one row may credit
39
+ several arms; its sums fold once and store as JSON.
40
+ - **`HierarchicalRates`** — Gamma-Poisson rates on a tree: a node's ratio to
41
+ what its parent leads us to expect, the fit of a level's strength (and of a
42
+ two-groups hot share), the leaf fit, and checks beside them.
43
+ - **`AdditiveContrasts`** — the arms' contrasts against a reference arm pooled
44
+ across cells along a nested tree and crossed groupings, a test of no
45
+ variation, each cell's level from all its visits, and the pooled estimate a
46
+ cell is allocated on.
47
+ - **`BetaArgmax`** (with **`GaussLegendre`** and `BetaFunction.log_density`) —
48
+ the exact probability that each of several Betas is the largest, the seedless
49
+ pick it describes, and the greedy pick; since 0.4.0 also a draw's position,
50
+ the distribution function and E[the largest].
51
+ - **`Prequential`** — the prequential log score, and the bits per event a
52
+ deeper predictor saves, with cluster-robust intervals.
53
+ - **`Cusum`** — a one-sided change test: its statistic, its run length, the
54
+ threshold for a stated false-alarm rate and the smallest change it can see.
15
55
 
16
56
  ## Install
17
57
 
18
- In a Gemfile (path dependency within the Skiftet workspace):
58
+ In a Gemfile (published on rubygems.org):
19
59
 
20
60
  ```ruby
21
- gem "skiftet_statistical", path: "../skiftet_statistical"
61
+ gem "skiftet_statistical", "~> 0.4.0"
22
62
  ```
23
63
 
24
64
  Or build/install locally:
@@ -63,26 +103,92 @@ S.two_tailed_p_value(1.96) # => ~0.05
63
103
  `two_proportion_z_test` / `welch_t_test` return `nil` when the test is undefined
64
104
  (an empty group or zero variance), matching the existing analyzers' behaviour.
65
105
 
106
+ ### Sample-ratio check
107
+
108
+ Does each arm's traffic fit the split its weights promise? Pearson's chi-squared
109
+ over the arms expected to get traffic, on k - 1 degrees of freedom. A small p
110
+ means the split is not doing what it says (a broken arm, a cache, a bot).
111
+
112
+ ```ruby
113
+ S.sample_ratio_p_value(observed: { control: 700, a: 300 },
114
+ expected: { control: 500.0, a: 500.0 }) # => ~1.1e-36
115
+ S.sample_ratio_p_value(observed: ..., expected: ..., minimum_total: 500)
116
+ ```
117
+
118
+ `nil` (not tested) under `minimum_total` observations (default 200) or with
119
+ fewer than two arms expected to get traffic; `0.0` when an arm expected to get
120
+ nothing got some.
121
+
122
+ ## Distribution functions
123
+
124
+ Pure Ruby, Float in and out. Every function takes Integers too.
125
+
126
+ ```ruby
127
+ S.normal_quantile(0.975) # => 1.9599639845400532 (the probit; bisection on erfc)
128
+ S.chi_squared_survival(3.841458820694124, 1) # => 0.05000000000000006: P(X >= statistic), X ~ chi-squared(1)
129
+
130
+ G = SkiftetStatistical::Gamma
131
+ G.regularized_lower(a, x) # P(a, x), the Gamma(a, 1) distribution function
132
+ G.regularized_upper(a, x) # Q(a, x) = 1 - P(a, x)
133
+
134
+ B = SkiftetStatistical::BetaFunction
135
+ B.regularized(x, alpha, beta) # I_x(alpha, beta), the Beta(alpha, beta) distribution function
136
+ B.quantile(0.975, 41.0, 361.0) # its inverse: an exact 95 % credible bound
137
+ ```
138
+
139
+ The arithmetic and its order are part of the API: an app that pins these
140
+ outputs (at 1e-12, say) compares them with this exact sequence of
141
+ operations, so a reordered formula is a breaking change even when it is "the
142
+ same" formula.
143
+
144
+ ## Joint Beta draws
145
+
146
+ For Bayesian A/B readouts and Thompson-sampling allocators: draw every arm's
147
+ Beta posterior once per draw, jointly, so "P(arm is best)" and "expected loss"
148
+ compare the same worlds.
149
+
150
+ ```ruby
151
+ posteriors = [ [ 41.0, 361.0 ], [ 52.0, 350.0 ] ] # [alpha, beta] per arm
152
+
153
+ SkiftetStatistical::JointBeta.each_draw(posteriors, draws: 2_000, rng: Random.new(7)) do |row|
154
+ row # => one Float per arm, in order
155
+ end
156
+
157
+ summary = SkiftetStatistical::JointBeta.summary(posteriors, draws: 2_000, rng: Random.new(7))
158
+ summary[:samples] # each arm's 2 000 draws
159
+ summary[:wins] # draws each arm won (a tie split equally); / draws = P(best)
160
+ summary[:losses] # summed (best - arm) per arm; / draws = expected loss
161
+ ```
162
+
163
+ **The draw order is part of the API.** Per draw, arm by arm in the order
164
+ given, one `Sampler#beta` each. `Sampler#gamma` takes a variable number of
165
+ values from the RNG, so one arm's posterior moves every later draw: a seeded
166
+ caller gets the same rows only while this order holds.
167
+
66
168
  ## Quick start (multi-armed bandit)
67
169
 
68
170
  ```ruby
69
171
  require "skiftet_statistical"
70
172
 
71
173
  bandit = SkiftetStatistical.bandit(
72
- arms: %w[facebook whatsapp bluesky x email],
174
+ arms: %w[a b c d e],
73
175
  policy: SkiftetStatistical::Policies::ThompsonSampling.new,
74
176
  )
75
177
 
76
- choice = bandit.select # which channel to promote right now, e.g. "whatsapp"
178
+ choice = bandit.select # which option to show right now, e.g. "b"
77
179
  # ... show that option to the user ...
78
180
  bandit.record(choice, 1) # reward: 1 = it converted, 0 = it didn't
79
181
 
80
182
  bandit.best_arm # current best by empirical mean
81
- bandit.stats # { "whatsapp" => { pulls:, mean:, reward_sum: }, ... }
183
+ bandit.stats # { "b" => { pulls:, mean:, reward_sum: }, ... }
82
184
  ```
83
185
 
84
- Rewards are expected in **[0.0, 1.0]** — a binary `0`/`1` (e.g. "did this share
85
- lead to a signup?") is the common case, but any value in that range works.
186
+ Rewards are expected in **[0.0, 1.0]** — a binary `0`/`1` (e.g. "did this view
187
+ lead to an outcome?") is the common case, but any value in that range works.
188
+
189
+ To order several items, or to pick one by a sticky key with the odds of the
190
+ pick (exact up to `EXACT_PICK_MAX` contenders), use the posterior set
191
+ (`Bandit::Posteriors`, below).
86
192
 
87
193
  ## Policies
88
194
 
@@ -105,6 +211,12 @@ no tuning, and explores exactly as much as the evidence warrants. Cold start (no
105
211
  data) is `Beta(1,1)` on every arm, i.e. uniform random, so early plays are pure
106
212
  exploration.
107
213
 
214
+ The four policies are kept as 0.3.0 wrote them: in 0.4.0, `ThompsonSampling`
215
+ and `Softmax` included, each answers as in 0.3.0 for every input, the same arm
216
+ and the same use of the rng. New callers draw through `Bandit::Posteriors`
217
+ instead, the posterior set, which holds one set of items and ranks, picks and
218
+ computes odds and summaries on it.
219
+
108
220
  ## Persistence
109
221
 
110
222
  A bandit's state is just its arms' counters, so it round-trips through a `Hash`
@@ -132,24 +244,1506 @@ Every stochastic policy (and the sampler) takes an `rng:`. Inject a seeded
132
244
  policy = SkiftetStatistical::Policies::ThompsonSampling.new(rng: Random.new(42))
133
245
  ```
134
246
 
135
- ## Example: f1 share-channel bandit
247
+ ## The generic bandit (`Bandit::Posteriors`)
136
248
 
137
- The motivating use case — make the petition ShareStep's **primary** button the
138
- channel that drives the most signups, while continuously testing the others:
249
+ New in 0.4.0. One set of items, each with its posterior and its views, and every
250
+ operation a bandit needs on it, each written once: the order rule (`rank`, with
251
+ `pick` its head), the odds of what that rule draws, per-item summaries for an
252
+ allocation, and plain draws. The caller decides what counts as a view (an item
253
+ was seen) and what counts as an outcome; an outcome is yes/no (a Beta posterior
254
+ an item) or an amount (a Normal posterior for the mean per view). Every random
255
+ input is injected: a key, given uniforms or an RNG.
139
256
 
140
257
  ```ruby
141
- # Nightly (or per request) build a bandit from observed share -> signup data.
142
- bandit = SkiftetStatistical::Bandit.from_h(
143
- Rails.cache.read("share_bandit_state") || { arms: SHARE_CHANNELS.map { { name: _1 } } },
144
- policy: SkiftetStatistical::Policies::ThompsonSampling.new,
258
+ P = SkiftetStatistical::Bandit::Posteriors
259
+
260
+ set = P.yes_no({ "a" => { views: 40, outcomes: 8 },
261
+ "b" => { views: 10, outcomes: 1 },
262
+ "c" => { views: 25, outcomes: 4 } })
263
+ set.items # => ["a", "b", "c"]
264
+ set.kind # => :beta
265
+ set.parameters # => { "a" => [9.0, 33.0], "b" => [2.0, 10.0], "c" => [5.0, 22.0] } (Beta(1, 1) prior)
266
+ set.views # => { "a" => 40.0, "b" => 10.0, "c" => 25.0 }
267
+ set.greedy # => "a", the largest posterior mean
268
+
269
+ order = set.rank(key: %w[list visit-1], shuffle: 0.1)
270
+ order.items # => ["a", "c", "b"], the same for this key in every process
271
+ order.leader # => "a"
272
+ order.branch # => :thompson (:shuffled for about 10 % of keys)
273
+ order.draws # => { "a" => 0.2907..., "b" => 0.1004..., "c" => 0.1386... }
274
+
275
+ set.pick(rng: Random.new(1)).item # => "c", the head of one Thompson draw
276
+ set.values(rng: Random.new(1)) # => [0.1721..., 0.0861..., 0.3054...], one draw an item
277
+ ```
278
+
279
+ ### Building a set
280
+
281
+ - `yes_no(items, priors: nil)`: `{ item => tally }`, a tally
282
+ `{ views:, outcomes: }` (Symbol or String keys, numbers >= 0, fractional
283
+ allowed, the outcomes clamped to the views), optionally with
284
+ inverse-propensity sums `weighted_views` (the sum of 1 / p),
285
+ `weighted_outcomes` (of y / p) and `weighted_views_squared` (of 1 / p^2):
286
+ the weighted outcomes are then rescaled to the Kish size, never above the
287
+ views, so an item that was shown more while it led is not overrated. Priors:
288
+ nil (Beta(1, 1)), one pair `[alpha, beta]` for every item, or
289
+ `{ item => pair }`.
290
+ - `amounts(items, priors:)`: a tally `{ views:, outcomes:, outcome_squares: }`
291
+ (the outcomes the sum of amounts, any sign; optionally the weighted sums and
292
+ `weighted_outcome_squares`) under one `Prior::Amount` or
293
+ `{ item => Prior::Amount }`; there is no flat amounts prior, as an amount has
294
+ a scale. With k prior views at mean mu0 and per-view variance v0, and n views
295
+ of mean m and variance s^2, the mean per view is Normal with mean
296
+ (k mu0 + n m) / (k + n) and variance
297
+ [k v0 + n s^2 + k n (m - mu0)^2 / (k + n)] / (k + n)^2, formed exactly and
298
+ rounded once.
299
+ - `beta(pairs, views: nil)` and `normal(pairs, views: nil)`:
300
+ `{ item => [alpha, beta] }` or `{ item => [mean, standard_deviation] }` as
301
+ they are, with views when given (an item missing from them reads 0).
302
+
303
+ ```ruby
304
+ prior = SkiftetStatistical::Bandit::Prior.amount(mean: 4.0, variance: 2_000.0, strength: 20.0)
305
+ amounts = P.amounts({ "a" => { views: 300, outcomes: 900.0, outcome_squares: 400_000.0 },
306
+ "b" => { views: 200, outcomes: 500.0, outcome_squares: 300_000.0 } }, priors: prior)
307
+ amounts.parameters # => { "a" => [3.0625, 2.0665...], "b" => [2.6363..., 2.6457...] } ([mean, sd])
308
+
309
+ P.beta({ "a" => [3.0e6, 7.0e6] }).parameters # => { "a" => [300000.0, 700000.0] }, scaled to LARGEST_TOTAL
310
+ ```
311
+
312
+ - Items are non-blank Strings, any number of them (an empty set ranks with no
313
+ head), in input order, which is the tie order everywhere. A Symbol, a blank
314
+ or repeated name, a name holding a NUL byte (the byte a key joins its parts
315
+ with) or the same bytes as an earlier name in another encoding is an
316
+ `ArgumentError` when the set is built; so is a Hash of priors or views that
317
+ names something that is not an item, or two keys naming one item.
318
+ - One pair everywhere. A Beta pair whose total is above `LARGEST_TOTAL` (1e6)
319
+ is scaled down to it, its mean kept, in `parameters`: the one pair every draw
320
+ (`rank`, `pick` and `values`, from any source), every odds call (exact or
321
+ Monte Carlo), `summary`'s P(best) and loss and `greedy` read. The pairs as
322
+ given feed `summary`'s means and intervals alone. A scaled pair never sums
323
+ above `LARGEST_TOTAL`, so a set built from another set's `parameters` keeps
324
+ them bit for bit.
325
+ - A Beta parameter above 0 but below `BetaArgmax::SMALLEST_PARAMETER` (1e-6)
326
+ builds: the quantile draws (a key or given uniforms), every odds call and
327
+ `summary` refuse it, and the RNG draws take it.
328
+
329
+ ### The order rule (`rank`, `pick`)
330
+
331
+ ```ruby
332
+ set.rank(key: "k", may_lead: %w[b c], min_views: 20) # only b or c may lead; a leader needs 20 views
333
+ set.rank(rng: Random.new(1), pinned: { 0 => "c" }, top: 2).items # => ["c", "a"]
334
+ set.rank(uniforms: { explore: 0.5, priority: { "a" => 0.1, "b" => 0.2, "c" => 0.3 },
335
+ draw: { "a" => 0.5, "b" => 0.5, "c" => 0.5 } }).items # => ["a", "c", "b"]
336
+ set.pick(key: %w[list visit-1], explore: 0.1) # => a Choice: item, head, branch, draws
337
+ # Order#repin: rank again with new pins under the held rule (shuffle and explore 0, no may_lead,
338
+ # min_views 0, the same top), on the order's own draws and the source as rank read it
339
+ set.rank(key: "k", explore: 0.1).repin({ 0 => "c" }) # == set.rank(key: "k", pinned: { 0 => "c" })
340
+ ```
341
+
342
+ The rule: `shuffle:` and `explore:` (each in [0, 1], summing to at most 1),
343
+ `may_lead:` (the items that may lead; nil lets every free item lead),
344
+ `min_views:` (0), `pinned:` (`{ position => item }`, 0-based) and `top:` (nil:
345
+ every item). Pinned items keep their positions and draw nothing; the free items
346
+ fill the other positions in order. With the branch value c: below `shuffle` the
347
+ order is `:shuffled` (by priority); below `shuffle + explore`, when a contender
348
+ has at least `min_views` views, it is `:explored` (the eligible item of smallest
349
+ priority leads, the rest follow by draw); otherwise it is `:thompson` (the
350
+ largest draw leads, the rest follow by draw). Ties go to the earlier item.
351
+
352
+ `rank` answers an `Order`: `items` (the first `top`, pinned items at their
353
+ positions), `leader` (the free head when it has at least `min_views` views,
354
+ else nil), `branch` and `draws` (each free item's posterior value; none on the
355
+ shuffled branch). `pick` is `rank` with top 1 and no pins, answering a `Choice`:
356
+ `item` (the leader), `head`, `branch` and `draws`. Both are frozen; `==`
357
+ compares those four members.
358
+
359
+ Exactly one source of randomness:
360
+
361
+ - `key:` (a String, or an Array of parts): the branch value is
362
+ `Draw.uniform_for(*key, "explore")`, an item's priority
363
+ `uniform_for(*key, "priority", item)` and its draw
364
+ `uniform_for(*key, "draw", item)`. The same key draws the same order in every
365
+ process with nothing stored, and no item's uniforms depend on another item:
366
+ over 1 000 keys with `shuffle: 0.1` and no exclusion or pin, an added item
367
+ never changed the others' relative order (with an item pinned at position 1
368
+ it did on 352 of them, as the added item takes a free slot ahead of the pin).
369
+ - `uniforms:` (`{ explore:, priority: { item => Float }, draw: { item => Float } }`):
370
+ the same three, given.
371
+ - `rng:` (anything that answers `rand`): one branch value when
372
+ `shuffle + explore` is above 0, one priority a free item on the shuffled or
373
+ explored branch, then one `Sampler` draw a free item where the branch draws,
374
+ in item order, and nothing else from the stream.
375
+
376
+ A key or given uniforms draw by quantile: a Beta item at
377
+ `BetaArgmax.position(u, alpha, beta)`, compared by that position so that two
378
+ draws never tie by rounding, and a Normal item at mean + sd times
379
+ `Significance.normal_quantile(u)`, u on the open grid (`Draw.open_uniform`). An
380
+ RNG draws by `Sampler`. Two Normal draws are compared, from every source, by the
381
+ exact sign of (mean_i - mean_j) + (sd_i z_i - sd_j z_j), each mean + sd * z
382
+ formed exactly, never rounded, so they never tie or misorder where a standard
383
+ deviation lies below a rounding of its mean; `draws` answers the rounded values.
384
+ Each source draws exactly from the posteriors, so one set of odds (below) covers
385
+ all three.
386
+
387
+ Refused, each an `ArgumentError` before any draw and before anything is taken
388
+ from an RNG: a rule outside its bounds, a pinned position or name that is not an
389
+ item, a name pinned twice, a `may_lead` name that is not an item, `top` below 1,
390
+ no source or more than one, a malformed `uniforms` Hash, and under a key or
391
+ given uniforms a Beta parameter below 1e-6. An RNG's own answer is checked as
392
+ it comes: one that is not a Float in [0, 1), or a raise from its `rand`, is an
393
+ `ArgumentError` at that answer, after the draws taken before it.
394
+
395
+ `greedy(may_lead: nil)` is the contender with the largest posterior mean on
396
+ `parameters` (`BetaArgmax.greedy`'s rule, for a set of any size); `values(rng:)`
397
+ is one posterior draw an item, in item order.
398
+
399
+ ### Odds of a pick, an order and a position
400
+
401
+ The posterior set answers the probability of what its rule draws, under that
402
+ same rule (`shuffle`, `explore`, `may_lead`, `min_views`, and for orders
403
+ `pinned` and `top`):
404
+
405
+ ```ruby
406
+ set = SkiftetStatistical::Bandit::Posteriors.beta({ "a" => [ 1.0, 1.0 ], "b" => [ 2.0, 1.0 ] })
407
+ set.pick_probabilities(explore: 0.1)[:heads] # => { "a" => 0.35000000000000003, "b" => 0.6500000000000001 }
408
+ set.order_probability(%w[b a]) # => { probability: 0.6666..., method: :exact, standard_error: 0.0 }
409
+ set.positions[:probabilities] # => { "a" => [0.3333..., 0.6666...], "b" => [0.6666..., 0.3333...] }
410
+ set.order_summary(%w[b a]) # => { probability: 0.6666..., positions: { "a" => [...], "b" => [...] } }
411
+ set.rank(key: "k1").probability # => 0.6666..., the odds of the order that key drew
412
+
413
+ SkiftetStatistical::Bandit::Posteriors.mixture(
414
+ shares: { "x" => 0.25, "y" => 0.75 },
415
+ probabilities: { "x" => { "a" => 0.5, "b" => 0.5 }, "y" => { "a" => 0.2, "c" => 0.8 } },
416
+ ) # => { "a" => 0.275, "b" => 0.125, "c" => 0.6000000000000001 }
417
+ ```
418
+
419
+ - `pick_probabilities` gives each item's probability of heading the pick
420
+ (`heads`) and of being the pick's leader (`leaders`, nil for no leader);
421
+ `order_probability(order)` the probability that the rule's whole order
422
+ begins with `order`; `positions` each item's probability at each slot;
423
+ `order_summary(order)` the posterior alone, with no rule: the probability
424
+ that `order` is the best one and each item's probability at each position,
425
+ for a set of at most `EXACT_ORDER_MAX` (10) items. `Choice#probability` and
426
+ `Order#probability` answer the same odds for a pick or an order the set made,
427
+ computed on the first call and kept.
428
+ - `Posteriors.mixture(shares:, probabilities:)` is the probability that a
429
+ two-layer draw shows each item: the sum over the first layer's branches of a
430
+ branch's share times its odds of the item.
431
+ - They are exact, by quadrature, up to `EXACT_PICK_MAX` (20) contenders for the
432
+ pick and up to `EXACT_ORDER_MAX` (10) free items for orders and positions
433
+ when no free item is barred from leading. The pick's P(largest) is
434
+ `BetaArgmax.probabilities` or `NormalArgmax.probabilities`. The order odds
435
+ sum on every free item's quantiles at 400 levels (an internal grid, not a
436
+ public constant), each sum on points where its own item's points keep their
437
+ digits: Betas share one grid in the position on a log scale the rule
438
+ compares, and each Normal sums on its own points in its own standardised
439
+ variable, the other items read there, so a
440
+ parameter near 1e-6, a standard deviation small beside its mean, or two
441
+ narrow items beside a far one loses no point; each sum is extrapolated from
442
+ the points and their every other level. They read within 4e-6 of closed
443
+ forms, inside the library's stated 1e-4.
444
+ - Otherwise each is one of two stated Monte Carlo forms, `draws:` (20 000)
445
+ draws on `Random.new(seed:)` (seed 0), so the same call answers the same
446
+ figures, with `method: :monte_carlo` and the largest standard error: above
447
+ `EXACT_PICK_MAX` contenders the pick counts how often each contender's draw
448
+ is the largest; orders and positions (an item barred from leading, or more
449
+ than `EXACT_ORDER_MAX` free items) count the rule's own visits. Both draw on
450
+ the scaled pairs (`parameters`), the pairs the exact odds read. A Beta
451
+ parameter far below 1 draws 0.0 in a share of `Sampler`'s draws (about a
452
+ tenth at 0.003), where two such draws tie and the tie goes to the earlier
453
+ item, so below about 0.003 the Monte Carlo reads that tie rule as well.
454
+ - A Beta parameter below `BetaArgmax::SMALLEST_PARAMETER` (1e-6) is an
455
+ `ArgumentError` for every odds call at every size, exact or Monte Carlo,
456
+ over the items a keyed draw of the same set and rule refuses.
457
+
458
+ Measured against the rule's own visits in `odds_simulation_spec.rb` (below):
459
+ every head and position within 2.75 standard errors of its exact value over
460
+ 2 * 10^5 visits a fixture.
461
+
462
+ ### Summaries (`summary`)
463
+
464
+ ```ruby
465
+ set = SkiftetStatistical::Bandit::Posteriors.beta({ "a" => [ 2.0, 8.0 ], "b" => [ 3.0, 7.0 ] },
466
+ views: { "a" => 120.0, "b" => 340.0 })
467
+ summary = set.summary(credible_mass: 0.95)
468
+ # => { "a" => { views: 120.0, mean: 0.2, lower: 0.0281..., upper: 0.4825..., p_best: 0.2882...,
469
+ # expected_loss: 0.1335... },
470
+ # "b" => { views: 340.0, mean: 0.3, lower: 0.0748..., upper: 0.6000..., p_best: 0.7117...,
471
+ # expected_loss: 0.0335... } }
472
+ SkiftetStatistical::Bandit::Allocator.from_summary(arms: %w[a b], summary: summary,
473
+ exposures: { "a" => 120, "b" => 340 },
474
+ status: "running", winner_arm: nil)
475
+ set.summary(among: []) # views, mean and interval only, for a set of any size
476
+ set.means # => { "a" => 0.2, "b" => 0.3 }, the means alone: no interval, no quantile
477
+ ```
478
+
479
+ `summary(credible_mass: 0.95, among: nil)` gives every item, in item order, its
480
+ views, its posterior mean and its equal-tailed interval, read on the pair as
481
+ given, and for the items `among` names (nil: every item; at most
482
+ `EXACT_PICK_MAX`) the exact P(best) and expected loss, E[the largest] less the
483
+ item's mean, among those items, on `parameters` (`BetaArgmax.probabilities` and
484
+ `.expected_maximum`, or `NormalArgmax`'s). A loss that rounds below 0 is
485
+ answered as 0. The answer is `Allocator.from_summary`'s `summary:` as it is, so
486
+ a pass on exact figures is one call more. `among: []` computes no P(best) and no
487
+ loss; `means` answers the means alone (frozen, summary's mean bit for bit) and
488
+ computes no interval, so it calls no quantile. A Beta interval is
489
+ `BetaFunction.quantile`, accurate to about 2e-10 up to totals (alpha + beta)
490
+ near 10^8; beyond about 3 * 10^8 one bound can fall on or past the mean (at a
491
+ mean of 1/2 the lower bound lies 0.0025 standard deviations above it at
492
+ 4 * 10^8). A Normal interval is the mean less and plus its standard deviation
493
+ times the Normal quantile, the tail read on the open grid.
494
+
495
+ ### Priors (`Bandit::Prior`)
496
+
497
+ A yes/no prior is a Beta pair `[alpha, beta]`; an amounts prior a
498
+ `Prior::Amount` (a mean per view, a variance per view and a strength in views).
499
+ Three rules are written once, and every other prior is built from them: a mean
500
+ held at a strength (`given`), a mean shrunk toward a prior mean (`pooled_mean`)
501
+ and the leave-one-out rule (`borrowed`: what a group holds once the item's own
502
+ evidence is taken out, views at least 0 and outcomes within [0, views]).
503
+
504
+ ```ruby
505
+ PR = SkiftetStatistical::Bandit::Prior
506
+
507
+ PR.flat # => [1.0, 1.0]
508
+ PR.given(mean: 0.1, strength: 20.0) # => [2.0, 18.0]
509
+ PR.amount(mean: 4.0, variance: 2_000.0, strength: 20.0)
510
+ PR.borrowed(group: { views: 1_000, outcomes: 100 }, own: { views: 200, outcomes: 50 })
511
+ # => { views: 800.0, outcomes: 50.0 }
512
+ PR.pooled(group: { views: 1_000, outcomes: 100 }, own: { views: 200, outcomes: 50 }, strength: 20.0)
513
+ # => [1.399..., 20.600...]: the borrowed mean (1 + 50) / (2 + 800) held at 2 + 20 views
514
+ PR.pooled_mean(views: 40, outcomes: 8, prior_mean: 0.1, strength: 20.0) # => 0.1666...
515
+ PR.tilts({ "a" => 2.0, "b" => 0.5, "c" => 1.0 }, clip: [ 0.5, 1.5 ]) # => { "a" => 1.5, "b" => 0.5, "c" => 1.0 }
516
+ PR.chain(levels: [ { views: 1_000, outcomes: 120, strength: 20.0 },
517
+ { views: 200, outcomes: 30, strength: 30.0 } ], root_mean: 0.1)
518
+ # => [3.365..., 26.634...]: the top level's mean, the lower level taken out, held at its 30 views
519
+ ```
520
+
521
+ - `pooled(group:, own:, strength:, base: FLAT)`: the borrowed mean
522
+ (base_alpha + y) / (base_alpha + base_beta + n) held at
523
+ base_alpha + base_beta + min(strength, n), so a large group never outweighs
524
+ the item's own evidence for long.
525
+ - `pooled_new(others:, group:, own:, weight: 2.0)`: a new item's prior, the
526
+ others' borrowed evidence pooled into (1 + Y) / (2 + N) at `weight`, plus the
527
+ new item's own borrowed evidence.
528
+ - `tilts(multipliers, clip:)`: each multiplier over the geometric mean of all
529
+ of them, clamped to `clip`, so tilts reorder items without raising them all.
530
+ - `chain(levels:, root_mean:, tilt: 1.0, clip: [0.001, 0.999])`: the prior of
531
+ an item at the bottom of nested levels, top first, each level's mean shrunk
532
+ toward the one above with the level below taken out.
533
+ - `pooled_amount(group:, own:, strength:)`: the borrowed mean and variance of
534
+ amounts at min(strength, n); it refuses a group with nothing left out or a
535
+ variance of 0, and the caller then states its own scale with `amount`.
536
+
537
+ ### Decay (`Bandit::Decay`)
538
+
539
+ ```ruby
540
+ SkiftetStatistical::Bandit::Decay.weight(72, half_life: 72, cap: 336) # => 0.5
541
+ SkiftetStatistical::Bandit::Decay.weight(337, half_life: 72, cap: 336) # => 0.0
542
+ rows = [ { age: 0, views: 10, outcomes: 3 }, { age: 72, views: 20, outcomes: 4 } ]
543
+ SkiftetStatistical::Bandit::Decay.fold(rows, half_life: 72, cap: 336)
544
+ # => { views: 20.0, outcomes: 5.0, outcome_squares: 0.0 }
545
+ ```
546
+
547
+ One formula, 2 ** (-age / half_life), and 0.0 for an age above the cap; age,
548
+ half-life and cap share one unit of the caller's choosing. `fold` weighs every
549
+ sum of a row (views, outcomes and outcome squares alike), so the folded tally is
550
+ what a posterior reads. The fold is linear, so a caller that sums in a database
551
+ groups its rows by age bucket and folds the buckets.
552
+
553
+ ### Keyed uniforms and the draw by weight (`Bandit::Draw`)
554
+
555
+ ```ruby
556
+ D = SkiftetStatistical::Bandit::Draw
557
+
558
+ D.uniform("t") # => 0.8895498694938413
559
+ D.uniform_for("t", "explore") # => 0.856909028643178
560
+ D.seed_for("t", "explore") # => 15807181545831752473, for Random.new
561
+ D.weighted({ "a" => 1, "b" => 3 }, 0.5) # => ["b", 0.75]
562
+ D.from_weights({ "a" => 1, "b" => 3 }, 0.5) # => "b"
563
+ D.open_uniform(0.0) # => 1.1102230246251565e-16 (2**-53)
564
+ ```
565
+
566
+ A uniform is the first 8 bytes of SHA256 of the text, big-endian, shifted right
567
+ by 11 and divided by 2**53: the same text gives the same uniform in every
568
+ process, so a draw keyed by a visit, a pass or a test comes back without being
569
+ stored. `uniform_for` joins its parts by one NUL byte (nil reads "", a part
570
+ holding a NUL byte is refused); `seed_for` is those 8 bytes as an Integer.
571
+ `weighted` is the one draw by weight: a weight that is not a finite number above
572
+ 0 counts 0, all 0 counts every key 1, and it answers the key with its share, the
573
+ probability that a uniform draw answers it; `from_weights` is its key. Keys are
574
+ any objects, answered as given. `open_uniform` clamps a uniform to
575
+ [2**-53, 1 - 2**-53], the one reading of a quantile draw's uniform.
576
+
577
+ ### Normal argmax (`Bandit::NormalArgmax`)
578
+
579
+ ```ruby
580
+ NA = SkiftetStatistical::Bandit::NormalArgmax
581
+
582
+ NA.probabilities([ [ 0.0, 1.0 ], [ 0.0, 1.0 ] ]) # => [0.5, 0.5]
583
+ NA.probabilities([ [ 5.0, 0.4 ], [ 5.3, 0.5 ] ], epsilon: 0.1) # => [0.3377..., 0.6622...]
584
+ NA.expected_maximum([ [ 0.0, 1.0 ], [ 0.0, 1.0 ] ]) # => 0.5641... (1 / sqrt(pi))
585
+ ```
586
+
587
+ `BetaArgmax`'s rule on independent Normals `[mean, standard_deviation]`, 1 to 20
588
+ members: the probability that each is the largest, mixed with epsilon / k, and
589
+ E[the largest]. Each member's probability is its own integral in its own
590
+ standardised variable, so a narrow member far from the others is weighed on its
591
+ own scale; each answer lies within 2 k (k + 1) * 1e-10 + 2e-12 of the exact
592
+ probability for k members (1.2e-9 for two, 8.4e-8 for 20).
593
+
594
+ ### A pass with a hold on its win (`Bandit::Pass`)
595
+
596
+ ```ruby
597
+ hold = SkiftetStatistical::Bandit::WinHold.new(leading_arm: "a", leading_since: now - 90_000, now: now,
598
+ last_pass_at: now - 600)
599
+ SkiftetStatistical::Bandit::Pass.call(status: "running", hold: hold) do |hold_satisfied, rng|
600
+ SkiftetStatistical::Bandit::Allocator.call(arms: %i[control a], tallies: tallies, status: "running",
601
+ winner_arm: nil, previous_weights: weights, rng: rng,
602
+ draws: 20_000, hold_satisfied: hold_satisfied)
603
+ end
604
+ # => the crown's answer (phase: :won, winner: :a, leader: :a, ...) and ready: true
605
+ ```
606
+
607
+ The probe yields (false, `Random.new(seed)`), so a leader gets the step, never
608
+ the won split. The pass is ready when the status reads "running", the probe
609
+ names a leader and `hold.satisfied_for?(leader)` answers true; then, and only
610
+ when `crown:` is true (the default), the block runs again with (true, a fresh
611
+ `Random.new(seed)`), so the probe and the crown read the same draws and only the
612
+ decision changes. The answer is the last block's Hash plus `ready:`; with
613
+ `crown: false` a ready leader keeps the probe's answer and `ready: true` says it
614
+ would have been crowned. `seed:` defaults to `Random.new_seed`; a given seed
615
+ makes the pass repeatable.
616
+
617
+ ### Amounts: what the Normal posterior gets wrong
618
+
619
+ Under about 30 non-zero amounts the mean of amounts is right-skewed and the
620
+ Normal posterior understates its upside; the prior's k views at v0 keep it
621
+ wide, and the shuffle or exploration share keeps the item shown. One large
622
+ amount x moves a mean of n views by x / n, so cap amounts before summing them;
623
+ `min_views` holds the lead until an item has enough views. Draws and interval
624
+ bounds may fall below 0: they are answered as computed, and the order rule
625
+ compares draws only. When `Prior.pooled_amount` refuses (nothing borrowed, or a
626
+ variance of 0), state a scale with `Prior.amount`. Measured in
627
+ `order_simulation_spec.rb` (below): on amounts at 300, 1 000 and 3 000 visits a
628
+ day the best item was on top on day 28 in every run, and the rule lost 4.3, 2.6
629
+ and 1.9 % of the best order's expected outcomes.
630
+
631
+ ### Inputs
632
+
633
+ Every number the code new in 0.4.0 reads must lie in one input domain: a real
634
+ number whose Float is finite and at most `SkiftetStatistical::Numbers::LARGEST`
635
+ (1e100) either side of 0. A larger number, a NaN, an infinity or a String is an
636
+ `ArgumentError` where it is read, so no sum or product of inputs can leave a
637
+ Float on its way to an answer, and every answer is checked finite before it is
638
+ returned. A caller's object is called only where it must be (a key part's
639
+ `to_s`, a name's `hash` or `eql?`, a list's `each`, an RNG's `rand`); what it
640
+ raises there is an `ArgumentError` naming the input. A caller's String, Hash or
641
+ Array is read by its contents, never by its own methods, so a missing key reads
642
+ as absent whatever default the Hash holds. A list is any Enumerable but a Hash,
643
+ read once by its `each`.
644
+
645
+ ### Costs
646
+
647
+ CPU time a call with YJIT, Ruby 3.4.9, on one development machine on
648
+ 2026-11-02 while other work kept it busy (CPU time moves less with the load than
649
+ the wall clock does):
650
+
651
+ | Call | Time |
652
+ |---|---|
653
+ | `Posteriors.yes_no`, 8 items | 0.055 ms |
654
+ | `Posteriors.amounts`, 2 items | 0.034 ms |
655
+ | `rank(key:, explore: 0.1)`, 8 Beta items, totals up to 1e3 | 0.21 ms |
656
+ | the same, totals up to 1e5 | 0.54 ms |
657
+ | the same, 8 Normal items | 0.08 ms |
658
+ | `rank(rng:, shuffle: 0.1)`, 8 Beta or Normal items | 0.017 ms |
659
+ | `BetaArgmax.position`, Beta(3, 300) / Beta(41, 960) / Beta(4 000, 96 000) | 0.019 / 0.028 / 0.057 ms (0.3.0's bisection: 0.11 / 0.29 / 0.92 ms) |
660
+ | `pick_probabilities(explore: 0.1)`, 5 Betas / 5 Normals | 24 / 1.6 ms |
661
+ | `summary`, 5 Betas / 20 Betas / 5 Normals | 46 / 440 / 3.8 ms |
662
+ | `summary(among: [])`, 60 Betas | 26 ms |
663
+ | `NormalArgmax.probabilities`, 20 members | 91 ms |
664
+ | `order_probability` (exact), 5 Betas / 5 Normals | 145 / 14 ms |
665
+ | `positions` (exact), 5 / 10 Betas | 160 / 560 ms |
666
+ | `positions(explore: 0.1)` (exact), 10 Betas | 1.2 s |
667
+ | `order_probability` (Monte Carlo, 20 000 visits), 5 Betas | 200 ms |
668
+ | `pick_probabilities` (Monte Carlo, 20 000 draws), 21 Betas | 470 ms |
669
+
670
+ A draw is cheap enough for every visit; the odds and summaries belong off the
671
+ request path, once a pass (a set keeps the grids of its exact orders and
672
+ positions, so later odds calls on the same set cost less).
673
+
674
+ ## Example: an ordered list, one visit at a time
675
+
676
+ ```ruby
677
+ P = SkiftetStatistical::Bandit::Posteriors
678
+
679
+ # Once a pass (hourly, say): the posterior set from stored tallies, where a view
680
+ # counts only for an item the visit reached, kept as its pairs.
681
+ set = P.yes_no(tallies) # { "a" => { views:, outcomes: }, ... }
682
+ cache.write("list:pairs", { pairs: set.parameters, views: set.views })
683
+
684
+ # Each visit: the set rebuilt from its pairs (bit for bit) and ranked by the
685
+ # visit's key, so a visit sees the same order on every request and nothing is
686
+ # stored per visit.
687
+ kept = cache.read("list:pairs")
688
+ order = P.beta(kept[:pairs], views: kept[:views]).rank(key: [ "list", visit_id ], shuffle: 0.1)
689
+ order.items # show the items in this order
690
+
691
+ # Off the request path, once a pass: each item's odds at each position under
692
+ # that rule, for an estimate that weighs a view by 1 / its probability.
693
+ set.positions(shuffle: 0.1)[:probabilities]
694
+ ```
695
+
696
+ ## A/B-test allocation (`Bandit::Allocator`)
697
+
698
+ The decision an A/B test's allocation job makes on every pass: how much traffic
699
+ each arm gets next, and whether an arm has won. It is the rule an app's
700
+ allocation job ran, moved here; pure (no clock, no state), with the randomness only from the injected `rng`.
701
+
702
+ ```ruby
703
+ A = SkiftetStatistical::Bandit::Allocator
704
+
705
+ result = A.call(
706
+ arms: %i[control a],
707
+ tallies: { control: { exposures: 1_000, conversions: 500 },
708
+ a: { exposures: 1_000, conversions: 533 } },
709
+ status: "running", winner_arm: nil,
710
+ previous_weights: { control: 0.5, a: 0.5 }, # the latest allocation, or nil
711
+ rng: Random.new(7), draws: 20_000,
712
+ )
713
+ result[:weights] # => { control: 0.35, a: 0.65 } (next pass's split)
714
+ result[:phase] # => :shifting (:warmup, :shifting, :won or :paused)
715
+ result[:stats][:a] # => { exposures: 1000, conversions: 533, rate: 0.533, weighted_rate: 0.533,
716
+ # lower: 0.502..., upper: 0.563..., p_best: 0.93175, expected_loss: 0.00066...,
717
+ # guardrails: {} }
718
+ result[:winner] # => nil (the crowned arm)
719
+ result[:leader] # => nil (the arm meeting the win rule at this pass)
720
+ result[:result_changed] # => false (a won test whose winner fell below P(best) 0.5)
721
+ ```
722
+
723
+ The rule, in order:
724
+
725
+ 1. **Paused** (`status: "paused"`): the previous weights unchanged (equal
726
+ weights without them).
727
+ 2. **Won** (`status: "won"` and `winner_arm` one of the arms): the won split,
728
+ 95 % to the winner and 5 % shared by the others. `result_changed` flags a
729
+ winner whose P(best) has fallen below 0.5; a won test never un-wins.
730
+ 3. **Warm-up**, while any arm has fewer than 300 exposures: equal weights.
731
+ 4. **The automatic win**: the first arm (in the order given) with P(best)
732
+ >= 0.95 and expected loss <= 0.0025 is the `leader`. It gets the won split
733
+ when `hold_satisfied:` is true (the default); with false (the hold below is
734
+ still running) it gets step 5's weights.
735
+ 5. **Shifting**: the target is each arm's P(best), every arm at least 10 %
736
+ (the others scaled down in proportion), reached by a convex step from the
737
+ previous weights in which no arm moves more than 0.15.
738
+
739
+ P(best) and expected loss come from `draws` joint draws of the arms' Beta
740
+ posteriors (`JointBeta`); the interval is the 95 % equal-tailed one of the
741
+ arm's draws. **The draw order is part of the API**: with the same input and
742
+ seed the result is the same only while no draw is added, dropped or reordered.
743
+
744
+ A tally may carry inverse-propensity sums, `weighted_exposures` and
745
+ `weighted_conversions` (each exposure counted as 1 / the arm's share when it
746
+ was drawn) and `weighted_exposures_squared`: the posterior is then built on
747
+ the weighted conversions rescaled to the Kish effective sample size, never
748
+ above the raw count (`WeightedPosterior`), so an arm whose lead comes from a
749
+ traffic surge it was favoured in is not crowned. A tally may also carry
750
+ `guardrails: { goal => conversions }`, reported per arm with its raw rate and
751
+ never decided on. Tallies and previous weights may use Symbol or String keys
752
+ (rows read back from JSON); previous weights that do not name exactly the arms
753
+ with finite numbers read as none.
754
+
755
+ Every number above is a setting of `Allocator::Config`, with the defaults
756
+ below:
757
+
758
+ ```ruby
759
+ config = A::Config.new(warmup_exposures: 100, floor: 0.05) # the rest stay
760
+ config.max_arms # => 20 (1 / floor)
761
+ A.call(..., config: config)
762
+ A.won_weights(%i[control a b], :a, config: config) # => { control: 0.025, a: 0.95, b: 0.025 }
763
+ ```
764
+
765
+ Settings: `warmup_exposures` (300), `floor` (0.10), `max_step` (0.15),
766
+ `win_p_best` (0.95), `win_loss` (0.0025), `won_share` (0.95), `won_rest`
767
+ (0.05), `result_changed_below` (0.5), `credible_mass` (0.95), `prior_alpha`
768
+ and `prior_beta` (1.0). More arms than `max_arms` raise `ArgumentError`, and
769
+ `allocation: :fixed` (a split that is measured, not allocated) raises
770
+ `A::FixedAllocation`, an `ArgumentError`, before anything is computed.
771
+
772
+ ### Per-arm floors and fixed shares (`arm_floors:`, `fixed_shares:`)
773
+
774
+ Both keywords are nil by default, and then nothing new is checked and every
775
+ output is the rule above, bit for bit. `arm_floors: { arm => floor }` gives an
776
+ arm its own floor, a number in (0, 1], in place of `Config#floor` (an arm it
777
+ does not name keeps `Config#floor`); `fixed_shares: { arm => share }` holds an
778
+ arm at exactly its share, a number in (0, 1), in every phase.
779
+
780
+ ```ruby
781
+ warm = { control: { exposures: 120, conversions: 30 }, a: { exposures: 110, conversions: 25 },
782
+ b: { exposures: 100, conversions: 31 } } # still in warm-up
783
+ inputs = { arms: %i[control a b], tallies: warm, status: "running", winner_arm: nil,
784
+ rng: Random.new(7), draws: 2_000 }
785
+ A.call(**inputs, arm_floors: { control: 0.5 })[:weights] # => { control: 0.5, a: 0.25, b: 0.25 }
786
+ A.call(**inputs, fixed_shares: { control: 0.2 })[:weights] # => { control: 0.2, a: 0.4, b: 0.4 }
787
+ ```
788
+
789
+ - The keys are arms of the call (Symbols or Strings); no arm in both; the fixed
790
+ shares and the other arms' floors sum to at most 1; every arm fixed raises
791
+ `FixedAllocation`. Each refusal is an `ArgumentError`, raised before any draw.
792
+ - In every phase each fixed arm gets exactly its share, and the free arms (the
793
+ others, in the order given) share the rest, `1 - F` with `F` the sum of the
794
+ fixed shares, each at least its floor.
795
+ - One lift serves every phase: a target over the free arms is scaled into the
796
+ free mass, every arm under its floor is pinned there, and the others are
797
+ scaled again into what is left, until none is under. With no fixed share and
798
+ every floor `Config#floor` it is the shifting phase's floor above, public as
799
+ `Allocator.lift`.
800
+ - Paused: the previous weights while they keep every floor and share, else the
801
+ free arms' previous shares through the lift (a pause on
802
+ `{ control: 0.02, a: 0.49, b: 0.49 }` with `arm_floors: { control: 0.05 }`
803
+ answers `{ control: 0.05, a: 0.475, b: 0.475 }`). Won: the won split over the
804
+ free arms; a fixed arm is never the won arm, so a stored win of one raises.
805
+ Warm-up: equal targets over the free arms. Shifting: the free arms' P(best),
806
+ reached by the step over the free arms and clamped onto their floors.
807
+ - With fixed shares, P(best) and expected loss are taken among the free arms in
808
+ each joint draw (every arm is still drawn, in order), so a fixed arm's are
809
+ nil. A lone free arm beside fixed arms can lead and win; with no share
810
+ declared the 0.2.0 guard stays, and a single arm never leads.
811
+
812
+ Measured (`value_allocation_simulation_spec.rb`, below): no share below its
813
+ floor or off its fixed share in 2,800 passes through every phase, and nothing
814
+ crowned in 2,000 passes at `hold_satisfied: false` while the win rule named a
815
+ leader in 1,926 and 1,978 of them.
816
+
817
+ ### Any posterior (`Allocator.from_summary`)
818
+
819
+ The same paused / won / warm-up / automatic-win / shifting rule, with the same
820
+ per-arm shares and checks, on statistics computed elsewhere: a `summary`
821
+ `{ arm => { p_best:, expected_loss:, mean:, lower:, upper: } }` and `exposures`
822
+ `{ arm => Integer }` in place of tallies, draws and rng. `call` builds its Beta
823
+ statistics and hands them to that one private rule, so the two entry points
824
+ cannot disagree on a phase: fed `call`'s own statistics, `from_summary` answers
825
+ `call`'s decision.
826
+
827
+ ```ruby
828
+ A.from_summary(
829
+ arms: %i[control a],
830
+ summary: { control: { p_best: 0.3, expected_loss: 0.02, mean: 1.1, lower: 0.9, upper: 1.3 },
831
+ a: { p_best: 0.7, expected_loss: 0.005, mean: 1.2, lower: 1.0, upper: 1.4 } },
832
+ exposures: { control: 1_000, a: 1_000 }, status: "running", winner_arm: nil,
833
+ previous_weights: { control: 0.5, a: 0.5 }, hold_satisfied: false,
834
+ )
835
+ # => { weights: { control: 0.35, a: 0.6499999999999999 }, phase: :shifting, stats: { ... },
836
+ # winner: nil, result_changed: false, leader: nil }
837
+ ```
838
+
839
+ - `summary: nil` (no posterior yet) holds warm-up whatever the exposures say; a
840
+ pause and a stored win still come first.
841
+ - A test that never ends passes `hold_satisfied: false` at every pass: nothing
842
+ is crowned and the leader is still named, which a caller whose own rule shows
843
+ no leader blanks itself. A warm-up of 0 is `Config.new(warmup_exposures: 0)`.
844
+ - `expected_loss` and `Config#win_loss` are read in the caller's units (value
845
+ per visit for a value summary). The rule never reads `mean`; `stats` carry the
846
+ summary's figures as Floats. Keys may be Symbols or Strings; the answer names
847
+ every arm by its Symbol.
848
+
849
+ ### The lift (`Allocator.lift`)
850
+
851
+ ```ruby
852
+ A.lift({ control: 0.99, a: 0.01 }) # => { control: 0.9, a: 0.1 }
853
+ A.lift({ a: 0.9, b: 0.06, c: 0.04 }) # => { a: 0.8, b: 0.1, c: 0.1 }
854
+ ```
855
+
856
+ Every arm at least `Config#floor` (`config:` as for `call`), the others scaled
857
+ in proportion into what the pinned floors leave: 0.2.0's private `with_floor`,
858
+ step for step, made public (the private method stays, delegating to it).
859
+
860
+ ### The hold on a win (`Bandit::WinHold`)
861
+
862
+ An automatic win counts only after the same arm met the win rule at every pass
863
+ for 24 hours. The caller keeps the state (the leading arm, when it started
864
+ leading, the time of the previous pass) and gives the clock:
865
+
866
+ ```ruby
867
+ hold = SkiftetStatistical::Bandit::WinHold.new(
868
+ leading_arm: "a", leading_since: now - 90_000, now: now, last_pass_at: now - 900,
869
+ # hold_seconds: 86_400, max_gap_seconds: 1_200 (the defaults)
870
+ )
871
+ hold.satisfied? # => true: the stored leader held for 24 hours or more
872
+ hold.satisfied_for?(:a)
873
+ result = A.call(..., hold_satisfied: hold.satisfied?)
874
+ state = hold.after_pass(result[:leader]) # => the leading arm and since, to store
875
+ ```
876
+
877
+ A run of passes with a gap longer than `max_gap_seconds` (or no previous pass)
878
+ is interrupted: not satisfied, and a leader starts over. Call the allocator a
879
+ second time with the same seed and `hold_satisfied: false` when the arm it
880
+ crowns is not the arm that held (`satisfied_for?`); `Bandit::Pass` (above)
881
+ makes both calls in one.
882
+
883
+ ## Linear value of several rates (`LinearValue`)
884
+
885
+ When an arm's worth is a weighted sum of several rates, V_a = sum over parts p
886
+ of w_p * r_(a, p), and both the rates and the weights are uncertain,
887
+ `LinearValue` draws them together: the rates from one joint normal over every
888
+ (arm, part) pair (so arms that share evidence stay correlated), the weights from
889
+ named parameters drawn afresh in every draw. P(best) and the expected loss then
890
+ hedge every uncertain weight at once instead of judging the arms at one assumed
891
+ weight.
892
+
893
+ ```ruby
894
+ LV = SkiftetStatistical::LinearValue
895
+
896
+ inputs = {
897
+ arms: %i[a b], parts: %i[p q],
898
+ means: { a: [ 0.12, 0.031 ], b: [ 0.105, 0.044 ] },
899
+ covariance: [ [ 4e-4, 6e-5, 2e-4, 1e-5 ], [ 6e-5, 1e-4, 2e-5, 4e-5 ], # arm-major:
900
+ [ 2e-4, 2e-5, 5e-4, -5e-5 ], [ 1e-5, 4e-5, -5e-5, 1.2e-4 ] ], # (a, p), (a, q), (b, p), (b, q)
901
+ parameters: { base: { fixed: 1.5 }, worth: { uniform: [ 2.0, 6.0 ] },
902
+ lift: { normal: [ 0.5, 0.2 ], clip: [ 0.0, 1.0 ] } },
903
+ weights: ->(drawn) { { p: drawn.fetch(:base), q: drawn.fetch(:worth) * drawn.fetch(:lift) } }
904
+ }
905
+ LV.summary(**inputs, draws: 20_000, rng: Random.new(7))
906
+ # => { a: { point: 0.242, mean: 0.2423..., lower: 0.1516..., upper: 0.3587..., p_best: 0.47715,
907
+ # expected_loss: 0.0188... },
908
+ # b: { point: 0.2455, mean: 0.2461..., lower: 0.1460..., upper: 0.3736..., p_best: 0.52285,
909
+ # expected_loss: 0.0150... } }
910
+ LV.each_draw(**inputs, draws: 3, rng: Random.new(7)).to_a
911
+ # => [[0.2233..., 0.1935...], [0.2945..., 0.3738...], [0.1634..., 0.2035...]] (summary's first rows)
912
+ ```
913
+
914
+ `point` is the value at every parameter's centre (the fixed value, the
915
+ uniform's midpoint, the normal's mean, each clipped) and the part means;
916
+ `mean`, `lower` and `upper` are the drawn values' mean and equal-tailed
917
+ `credible_mass` interval (0.95 by default); a tie splits P(best) equally.
918
+ `each_draw` yields each draw's values, the rows `summary` is built on (an
919
+ Enumerator without a block).
920
+
921
+ **The draw order is part of the API.** Per draw: each parameter in the order
922
+ given (`fixed: x` takes nothing, `uniform: [lo, hi]` one `rng.rand`,
923
+ `normal: [mean, sd]` one `Sampler#gaussian`, then its `clip: [lo, hi]` when it
924
+ has one); the part weights from the `weights` callable; one joint normal draw
925
+ of every rate, means + L * n, with n the K * P `Sampler#gaussian` values in
926
+ arm-major order and L the covariance's `Cholesky` factor; each arm's value. A
927
+ part with no variance is drawn at its mean and still takes its normal, so the
928
+ numbers taken never depend on the data.
929
+
930
+ Everything is checked before the first random number (`ArgumentError`): 1 to 7
931
+ arms and 1 to 12 parts (`MAX_ARMS`, `MAX_PARTS`; sum the parts that share one
932
+ weight), a symmetric positive semidefinite covariance, finite numbers, a weight
933
+ for every part. `draws:` is an Integer >= 0 (`summary` needs at least 1).
934
+ `rounding:` (0.0 by default) is how far each covariance entry may be off: hand
935
+ it a `SharedCredit` estimate's `#rounding`, so a covariance summed from rows is
936
+ never refused for its own rounding.
937
+
938
+ Measured (`value_allocation_simulation_spec.rb`, below): the 95 % intervals of
939
+ each arm's value and of each difference covered the truth in 377 to 385 of 400
940
+ runs (bound 372), the true weights drawn inside their ranges per run;
941
+ allocating on it through `Allocator.from_summary`, values that arrived up to 23
942
+ passes late left a final bias within 1.08 standard errors, and a drift shared by
943
+ every arm moved no equal arm's mean P(best) more than 0.028 from 1 / 4.
944
+
945
+ ## Cholesky factor (`Cholesky`)
946
+
947
+ ```ruby
948
+ SkiftetStatistical::Cholesky.factor([ [ 4.0, 2.0 ], [ 2.0, 5.0 ] ]) # => [[2.0, 0.0], [1.0, 2.0]]
949
+ SkiftetStatistical::Cholesky.root([ [ 1.0, 2.0 ], [ 2.0, 4.0 ] ]) # => [[1.0, 0.0], [2.0, 0.0]]
950
+ ```
951
+
952
+ `factor` answers the lower-triangular L with L * L^T = the matrix, rows in and
953
+ rows out, the lower triangle read, with no matrix library. The matrix must be
954
+ square, finite and symmetric (entries (i, j) and (j, i) within 1e-12 of their
955
+ scale). A pivot below 1e-12 times the largest diagonal entry is no variance and
956
+ is set to 0 with its column (a residual under it beyond rounding raises: a part
957
+ with no variance cannot covary); a pivot below -1e-9 times the trace raises:
958
+ the matrix is not positive semidefinite. `root(rows, rounding: 0.0)` is the
959
+ same arithmetic taken largest pivot first, for a matrix of lower rank whose
960
+ rounding the given order would carry into every later pivot: a square root S
961
+ with S * S^T = the matrix within its tolerance, not triangular in the given
962
+ order. `LinearValue` draws through `factor`, and through `root` where `factor`
963
+ refuses.
964
+
965
+ Measured: no simulation of its own; every `LinearValue` draw in the
966
+ simulations below goes through it.
967
+
968
+ ## Shared-credit estimates (`SharedCredit`)
969
+
970
+ Each arm's mean of several parts, with one covariance over every (arm, part)
971
+ pair, when one row may count for several arms. A row is
972
+ `{ credit: { arm => w >= 0 }, values: [x_1, ..., x_P] }`: in an ordinary test
973
+ it credits only the arm drawn, with 1 / p; with shared credit it credits every
974
+ arm whose own choice is what the row shows, each with 1 over the probability
975
+ that the row showed it. One estimator serves both.
976
+
977
+ ```ruby
978
+ SC = SkiftetStatistical::SharedCredit
979
+
980
+ estimate = SC.estimate(
981
+ [ { credit: { a: 2.0 }, values: [ 1.0, 0.0 ] },
982
+ { credit: { a: 1.0, b: 1.0 }, values: [ 0.0, 2.0 ] },
983
+ { credit: { b: 4.0 }, values: [ 1.0, 1.0 ] } ],
984
+ arms: %i[a b], parts: %i[p q],
145
985
  )
146
- primary = bandit.select # the channel to feature as the primary CTA
986
+ # => { means: { a: [0.6666..., 0.6666...], b: [0.8, 1.2] },
987
+ # covariance: [[0.0987..., -0.1975..., 0.0355..., -0.0355...], ...], # arm-major: (a, p), (a, q), (b, p), (b, q)
988
+ # weight_sums: { a: 3.0, b: 5.0 }, effective: { a: 1.8, b: 1.4705... } }
989
+ estimate.parts # => [:p, :q]
990
+ estimate.rounding # => 7.78...e-15, the largest rounding bound of an entry
991
+ SC.contrast(estimate, :b, :a, weights: { p: 1.0, q: 0.5 })
992
+ # => { mean: 0.3999999999999999, variance: 0.012800000000000138 }
993
+ ```
147
994
 
148
- # When a share converts:
149
- bandit.record(channel, 1)
150
- Rails.cache.write("share_bandit_state", bandit.to_h)
995
+ Means are Hájek, r_ap = sum_i w_ai * x_ip / W_a; the covariance is the sandwich
996
+ sum_i psi_ap(i) * psi_bq(i) with psi_ap(i) = w_ai * (x_ip - r_ap) / W_a, so arms
997
+ that share rows covary. It is laid out arm-major, as `LinearValue` reads it,
998
+ computed on one triangle and mirrored, so it is exactly symmetric; Kish's
999
+ effective size stands beside it. An arm no row credits has nil means and zero
1000
+ rows. An entry within its cancellation rounding ((8 + n) * `Float::EPSILON`
1001
+ times the sum of its terms' sizes, n the larger of Kish's size and the rows
1002
+ folded) answers exactly 0.0, so an arm of constant values gives zeros that
1003
+ `LinearValue` accepts.
1004
+
1005
+ The sums are public, for a caller that keeps its rows in a store and must fold
1006
+ each one once:
1007
+
1008
+ ```ruby
1009
+ acc = SC.accumulator(arms: %i[a b], parts: %i[p q])
1010
+ acc.add({ credit: { a: 2.0 }, values: [ 1.0, 0.0 ] })
1011
+ saved = acc.to_h # JSON-safe: { "format" => 1, "arms" => ["a", "b"], "parts" => ["p", "q"], ... }
1012
+ SC::Accumulator.from_h(saved).merge(later).estimate # = one estimate over every row of both
151
1013
  ```
152
1014
 
1015
+ For an ordinary test the per-arm sums are enough, `{ arm => { w:, w2:,
1016
+ a: { part => }, b: { part => }, c: { [p, q] => } } }`, with w = sum 1/p,
1017
+ w2 = sum 1/p^2, a = sum x/p, b = sum x/p^2 and c = sum x_p * x_q / p^2 (both
1018
+ orders of each pair):
1019
+
1020
+ ```ruby
1021
+ sums = { a: { w: 4.0, w2: 8.0, a: { p: 2.0, q: 2.0 }, b: { p: 4.0, q: 4.0 },
1022
+ c: { %i[p p] => 4.0, %i[p q] => 0.0, %i[q p] => 0.0, %i[q q] => 4.0 } },
1023
+ b: { w: 8.0, w2: 32.0, a: { p: 4.0, q: 4.0 }, b: { p: 16.0, q: 16.0 },
1024
+ c: { %i[p p] => 16.0, %i[p q] => 16.0, %i[q p] => 16.0, %i[q q] => 16.0 } } }
1025
+ SC.from_sums(sums, arms: %i[a b], parts: %i[p q])
1026
+ # => { means: { a: [0.5, 0.5], b: [0.5, 0.5] },
1027
+ # covariance: [[0.125, -0.125, 0.0, 0.0], [-0.125, 0.125, 0.0, 0.0],
1028
+ # [0.0, 0.0, 0.125, 0.125], [0.0, 0.0, 0.125, 0.125]],
1029
+ # weight_sums: { a: 4.0, b: 8.0 }, effective: { a: 2.0, b: 2.0 } }
1030
+ # the estimate of four rows: a credited 2.0 with [1, 0] and [0, 1], b credited 4.0 with [1, 1] and [0, 0]
1031
+ ```
1032
+
1033
+ - `linear(estimate, weights:)`: each arm's value at fixed weights and the K x K
1034
+ covariance of the values. `contrast(estimate, a, b, weights:)`: one
1035
+ difference with its variance. `weights:` is an Array in the parts' order or a
1036
+ Hash by part name (every part once, in any order; refused on an estimate read
1037
+ back from JSON, which names no parts).
1038
+ - `contrast_for_fit(estimate, arm, reference, weights:, minimum_effective: 20)`:
1039
+ the contrast a fit across cells takes, the same difference with one per-visit
1040
+ variance pooled over the two arms, s^2 * (1 / n_a + 1 / n_r), and nil when
1041
+ either arm has nil means or a Kish's size under `minimum_effective`. An arm
1042
+ held at a small floor reads a plug-in variance that moves with its own
1043
+ estimate (a rare part it has not seen yet reads a variance near 0), and a
1044
+ test across cells would then reject too often.
1045
+
1046
+ Measured (`two_layer_simulation_spec.rb`, below): scored by greedy shared
1047
+ credit, two options of identical value read a mean P(best) of 0.58 and 0.42
1048
+ (bound 0.40 to 0.60) where the same visits scored as they ran read 0.63 and
1049
+ 0.37; in the pooled cells, `contrast_for_fit` left 1,135 of 24,000 cell
1050
+ contrasts out of the fit.
1051
+
1052
+ ## Hierarchical Poisson rates (`HierarchicalRates`)
1053
+
1054
+ How far a node's rate stands from its parent's, from its count s and the count
1055
+ E its parent leads us to expect there. The node's ratio has a Gamma prior with
1056
+ mean 1 and strength k (Clayton and Kaldor 1987), so its posterior is
1057
+ Gamma(k + s, k + E); under a two-groups prior (Efron 2008) the node is null
1058
+ with probability 1 - h (its count Poisson(E), its ratio 1) and otherwise drawn
1059
+ from that slab.
1060
+
1061
+ ```ruby
1062
+ HR = SkiftetStatistical::HierarchicalRates
1063
+
1064
+ HR.leave_one_out_share(parent_count: 40.0, parent_total: 1_000.0, node_count: 6.0, node_total: 100.0,
1065
+ prior_share: 0.04, prior_strength: 16.0) # => 0.0378... (34.64 / 916)
1066
+ HR.node(count: 6.0, expected: 2.5, strength: 8.0, threshold: 1.25, parent_tail: 0.0, parent_ratio: 1.0)
1067
+ # => { ratio: 1.3333333333333333 (14 / 10.5), hot: 1.0, tail: 0.5593048238168208 (Q(14, 13.125)) }
1068
+
1069
+ pairs = [ { count: 6.0, expected: 2.5, siblings: 3 }, { count: 0.0, expected: 2.0, siblings: 3 },
1070
+ { count: 3.0, expected: 3.5, siblings: 3 }, { count: 12.0, expected: 4.0, siblings: 2 },
1071
+ { count: 1.0, expected: 3.0, siblings: 2 }, { count: 5.0, expected: 1.0, siblings: 1 } ]
1072
+ HR.fit_strength(pairs: pairs)
1073
+ # => { strength: 1.1892... (2**0.25), hot_share: 1.0, log_likelihood: -12.57..., at_upper_bound: false,
1074
+ # interval: [0.25, 9.5136...], profile: [[0.25, ...], ...] }
1075
+ HR.marshall(pairs: pairs) # => { variance: 0.9136..., strength: 1.0945... }
1076
+ HR.boundary_test(pairs: pairs, strength: 2**0.25) # => { statistic: 7.5936..., p_value: 0.0029... }
1077
+ HR.smooth_strength(previous: 8.0, fitted: 16.0) # => 9.1895... (exp(0.8 * log 8 + 0.2 * log 16))
1078
+ ```
1079
+
1080
+ The last pair (siblings 1) is left out of the fit and of both checks: its
1081
+ parent has one child with data, which carries nothing about the level.
1082
+
1083
+ - `node(count:, expected:, strength:, hot_share: 1.0, threshold:, parent_tail:,
1084
+ parent_ratio:)`: hot = h * NB(s; k, k / (k + E)) / [h * NB + (1 - h) *
1085
+ Poisson(s; E)] (1.0 at h = 1), ratio = (1 - hot) + hot * (k + s) / (k + E),
1086
+ tail = (1 - hot) * parent_tail + hot * Q(k + s, (k + E) * L / parent_ratio).
1087
+ - `fit_strength(pairs:, hot_grid: nil)`: the marginal log-likelihood maximised
1088
+ on `GRID` (57 strengths, 2^-2 to 2^12 in quarter powers of 2) and on
1089
+ `hot_grid` (h = 1 without one); `interval` holds the strengths within 1.92
1090
+ (half the chi-squared(1) 95 % point) of the maximum, and `at_upper_bound`
1091
+ says the level's nodes inherit their parent.
1092
+ - `fit_leaves(parents:, hot_shares:, grid: GRID)`: the leaf level's fit, each
1093
+ parent's null rate estimated jointly with k and h (from its pooled rate, each
1094
+ step weighting its leaves by their posterior chance of being null, at most 50
1095
+ steps) instead of read from a leave-one-out share. With about five leaves to a
1096
+ parent, a hot leaf lifts its siblings' leave-one-out share, they read as cold,
1097
+ and a fit on those expected counts reads the hot share several times too
1098
+ high. `parents` is `[[{ count:, total: }]]`, `hot_shares` each in [0, 1); the
1099
+ answer holds `strength`, `hot_share`, `log_likelihood`, `at_upper_bound`, and
1100
+ per parent `converged`, `rates` and each leaf's `hot`.
1101
+ - `smooth_strength(previous:, fitted:, weight: 0.2)`, a moving average in
1102
+ logs; `marshall(pairs:)`, Marshall's (1991) moment estimate;
1103
+ `boundary_test(pairs:, strength:, hot_share: 1.0)`, the likelihood-ratio test
1104
+ on the boundary (Self and Liang 1987).
1105
+
1106
+ Counts are real numbers >= 0: a node that stands in its parent by a share w has
1107
+ w multiplied into its count and total by the caller. The probability mass
1108
+ functions are taken in logs, through log-gamma.
1109
+
1110
+ Measured (`tree_recovery_simulation_spec.rb`, below): on generated five-level
1111
+ trees each level's fitted strength was within a factor of 2 of the true one in
1112
+ 20 of 20 runs, the leaves' hot share within 0.05 in 20 of 20 on `fit_leaves`
1113
+ (0.05 to 0.1 for a true 0.02 on leave-one-out expected counts), and the
1114
+ shrunken ratio beat the raw one on a held-out half at every level below the
1115
+ first in every run.
1116
+
1117
+ ## Additive contrasts across cells (`AdditiveContrasts`)
1118
+
1119
+ What differs between cells is pooled on the arms' contrasts against a reference
1120
+ arm, never on the arms' own levels (Hong, Kveton, Zaheer and Ghavamzadeh 2022).
1121
+ Per arm a, delta(a | cell) = delta_root(a) + u_tree(a) + the sum over groupings
1122
+ g of u_g(a): the tree nested (each node's deviation N(0, tau^2 of its depth)),
1123
+ every other grouping crossed (N(0, tau^2_g)).
1124
+
1125
+ ```ruby
1126
+ AC = SkiftetStatistical::AdditiveContrasts
1127
+
1128
+ cells = [ { id: 1, contrasts: { a: [ 0.0, 1.0 ] }, tree: %i[root x], groups: {} }, # [estimate, sampling variance]
1129
+ { id: 2, contrasts: { a: [ 1.0, 1.0 ] }, tree: %i[root y], groups: {} },
1130
+ { id: 3, contrasts: { a: [ 5.0, 1.0 ] }, tree: %i[root z], groups: {} } ]
1131
+ fit = AC.fit(cells: cells, arms: %i[a])
1132
+ # => { variances: { "tree:1" => { a: 6.0 } },
1133
+ # effects: { "tree:1" => { x: { a: [-1.7142..., 2.5714...] }, y: { a: [-0.8571..., 2.5714...] },
1134
+ # z: { a: [2.5714..., 2.5714...] } } },
1135
+ # cells: { 1 => { a: [0.2857..., 0.9047...] }, 2 => { a: [1.1428..., 0.9047...] },
1136
+ # 3 => { a: [4.5714..., 0.9047...] } } }
1137
+ fit.variation_tests # => { "tree:1" => { a: { statistic: 14.0, degrees: 2, p_value: 0.000911... } } }
1138
+ AC.fit(cells: cells, arms: %i[a], variances: { "tree:1" => { a: 0.0 } })[:cells]
1139
+ # => { 1 => { a: [2.0, 0.3333...] }, 2 => { a: [2.0, 0.3333...] }, 3 => { a: [2.0, 0.3333...] } }
1140
+ ```
1141
+
1142
+ A tree of one level with no other grouping, as here, is plain normal shrinkage
1143
+ toward the root: the library's one home of it. Variances come by moments on the
1144
+ contrasts, per depth, grouping and arm: tau^2 = max(0, the weighted spread of
1145
+ the group estimates less their mean sampling variance), unbiased whatever the
1146
+ precisions. `variances:` holds what it names fixed (an earlier fit's
1147
+ `variances`), and the rest is fitted. Posteriors are normal-normal: the tree is
1148
+ solved exactly for given variances, and the crossed groupings are backfitted
1149
+ with it, `sweeps` times (5 by default). A cell may leave out an arm (nil); an
1150
+ arm no cell has a contrast for raises.
1151
+
1152
+ `fit` answers an `AdditiveContrasts::Fit`, the Hash above, whose
1153
+ `#variation_tests` stand beside each variance: Cochran's Q (the spread the
1154
+ moment reads), its degrees and the chi-squared chance under tau^2 = 0 of a
1155
+ spread at least that large (1.0 with no degrees). A caller that holds a
1156
+ variance at 0 unless its test rejects hands the next fit that 0 in
1157
+ `variances:`.
1158
+
1159
+ `baseline(rows:, effects:)` is each cell's level from all its visits, less each
1160
+ arm's fitted effect and adjusted on a categorical covariate, with its Hájek
1161
+ variance. A row sums one (cell, arm, covariate)'s visits: `weight` (sum w),
1162
+ `value` (sum w * x) and, beyond one visit, `weight_squared`, `value_squared` and
1163
+ `cross` (sum w^2, sum w^2 * x^2, sum w^2 * x). The reference arm is the one arm
1164
+ of the rows that no cell of `effects` names.
1165
+
1166
+ ```ruby
1167
+ AC.baseline(
1168
+ rows: [ { cell: "c1", arm: :ref, covariate: "k1", weight: 2.0, value: 2.0 },
1169
+ { cell: "c1", arm: :a, covariate: "k2", weight: 1.0, value: 2.0 },
1170
+ { cell: "c2", arm: :ref, covariate: "k2", weight: 1.0, value: 0.0 } ],
1171
+ effects: { "c1" => { a: [ 1.0, 0.25 ] }, "c2" => { a: [ 0.5, 0.25 ] } },
1172
+ )
1173
+ # => { "c1" => [0.9166... (11 / 12), 0.0246... (2 / 81)], "c2" => [0.25, 0.0] }
1174
+ ```
1175
+
1176
+ Pooling per (arm, part) is what reaches a decision: `fit`'s `arms:` name the
1177
+ pairs (`AC.pair(:a, :p) # => "a|p"`), one `baseline` runs per part, and
1178
+ `pooled_estimate(own:, effects:, baselines:, cell:)` puts them back together as
1179
+ the cell's estimate in `SharedCredit`'s shape, with `#parts` and `#rounding`,
1180
+ which `LinearValue` and the allocator read as they read the cell's own. Means
1181
+ are mu_p + delta_ap (delta 0 at the reference arm); the covariance is
1182
+ Cov(mu_p, mu_q) + [a == b] * Cov(delta_ap, delta_aq): the cell's level is one
1183
+ quantity shared by every arm, each arm's effect is its own, and each block keeps
1184
+ the cell's own correlation between parts. A contrast of two arms then holds
1185
+ their effects alone, so no arm's share follows its cell's level.
1186
+
1187
+ ```ruby
1188
+ # own: the cell's SharedCredit estimate, means ref [1.0] and a [2.0], covariance [[0.04, 0.012], [0.012, 0.09]]
1189
+ AC.pooled_estimate(own: own, effects: { "c1" => { "a|p" => [ 0.5, 0.03 ] } },
1190
+ baselines: { p: { "c1" => [ 1.0, 0.01 ] } }, cell: "c1")
1191
+ # => { means: { ref: [1.0], a: [1.5] }, covariance: [[0.01, 0.01], [0.01, 0.04]], weight_sums: ..., effective: ... }
1192
+ ```
1193
+
1194
+ Measured (`contrasts_simulation_spec.rb` and `two_layer_simulation_spec.rb`,
1195
+ below): every variance of at least 4 times the mean sampling variance came back
1196
+ within a factor of 2, and every variance of 0 under a quarter of it, in each of
1197
+ 100 runs; across 40 cells whose levels differ with no effect difference, each
1198
+ allocated on its pooled estimate, the reference arm's mean share spread 0.0066
1199
+ (the equal-level noise floor's 0.0065) at a rank correlation of -0.073 with the
1200
+ level.
1201
+
1202
+ ## Beta argmax (`BetaArgmax`, `GaussLegendre`, `BetaFunction.log_density`)
1203
+
1204
+ Which of several independent Beta variables is the largest: the exact
1205
+ probability of each, the seedless pick that probability describes, and the
1206
+ greedy pick.
1207
+
1208
+ ```ruby
1209
+ BA = SkiftetStatistical::BetaArgmax
1210
+
1211
+ BA.probabilities([ [ 1.0, 1.0 ], [ 2.0, 1.0 ] ]) # => [0.3333..., 0.6666...] (1/3, 2/3)
1212
+ BA.probabilities([ [ 1.0, 1.0 ], [ 2.0, 1.0 ] ], epsilon: 0.1) # => [0.35..., 0.65...]
1213
+ BA.pick([ [ 2.0, 8.0 ], [ 8.0, 2.0 ] ], explore: 0.5, priorities: [ 0.3, 0.6 ],
1214
+ uniforms: [ 0.5, 0.5 ], epsilon: 0.1) # => 1
1215
+ BA.greedy([ [ 1.0, 3.0 ], [ 3.0, 1.0 ] ]) # => 1
1216
+
1217
+ SkiftetStatistical::GaussLegendre.nodes(2)
1218
+ # => [[-0.5773..., 1.0000...], [0.5773..., 1.0000...]] (x = -+1 / sqrt(3), w = 1)
1219
+ SkiftetStatistical::BetaFunction.log_density(0.25, 2.0, 3.0) # => 0.5232... (log 27/16)
1220
+ ```
1221
+
1222
+ `pick` takes its uniforms from the caller, never from an RNG: below `epsilon`
1223
+ (`explore < epsilon`) the member with the smallest priority, else the member
1224
+ whose Beta quantile at its own uniform is the largest. Over uniform inputs
1225
+ member c is picked with probability epsilon / k + (1 - epsilon) * the integral
1226
+ of f_c * the product over j != c of F_j, which is what `probabilities` answers,
1227
+ so a caller can log the exact probability of the pick it made. The integral is
1228
+ composite Gauss-Legendre (`NODES`, 20 a panel), the panels broken at every
1229
+ member's quantiles at `BREAKS` (1e-10 to 1 - 1e-10, `TAIL` = 1e-10), each half
1230
+ of (0, 1) worked in the distance from its own end so that a parameter far below
1231
+ 1 does not round, and normalised to sum 1. It is exact to 1e-8 absolute for 1
1232
+ to 20 members (`MAX_MEMBERS`) with every parameter at least 1e-6
1233
+ (`SMALLEST_PARAMETER`) and alpha + beta at most 3 * 10^6 (`LARGEST_TOTAL`):
1234
+ beyond those parameter limits `probabilities` and `pick` raise, while `greedy`
1235
+ (the largest mean) takes any parameters; every call takes 1 to 20 members. Ties
1236
+ go to the lower index.
1237
+
1238
+ `GaussLegendre.nodes(n)` is the n-point rule on [-1, 1], ascending and exactly
1239
+ symmetric, computed once per n and kept, frozen. `BetaFunction.log_density` is
1240
+ -Infinity outside (0, 1); `regularized` and `quantile` are unchanged.
1241
+
1242
+ Since 0.4.0 three more answers sit beside these, and `probabilities`, `pick` and
1243
+ `greedy` answer as in 0.3.0 (`pick` still compares its own position):
1244
+
1245
+ ```ruby
1246
+ BA.position(0.5, 2.0, 2.0) # => 0.0, the draw 1/2
1247
+ BA.distribution(0.3, 2.0, 8.0) # => 0.8039967659999996
1248
+ BA.expected_maximum([ [ 1.0, 1.0 ], [ 1.0, 1.0 ] ]) # => 0.6666666666666667 (2/3)
1249
+ ```
1250
+
1251
+ - `position(uniform, alpha, beta)`: where the draw of Beta(alpha, beta) at a
1252
+ uniform lies, log(2 x) up to 1/2 and -log(2 (1 - x)) beyond, a key that
1253
+ rounds at neither end; the posterior set's quantile draws compare by it. A
1254
+ guarded Newton step on the same distribution function. Against `pick`'s own
1255
+ bisection (0.3.0's position), recorded on 30 members at 1 000 uniforms each,
1256
+ the spec bounds |position - recorded| / max(1, |recorded|), an absolute
1257
+ error where the recorded position lies within 1 of 0 and a relative one
1258
+ beyond: 1e-12 (measured at most 4.7e-13), 1e-9 where alpha + beta exceeds
1259
+ 10^4 (measured 3.0e-13) and 1e-8 at the smallest parameter (measured
1260
+ 9.5e-11). It takes 0.019 / 0.028 / 0.057 ms at Beta(3, 300) / Beta(41, 960) /
1261
+ Beta(4 000, 96 000) where the bisection took 0.11 / 0.29 / 0.92 ms.
1262
+ - `distribution(x, alpha, beta)`: that distribution function itself, the one
1263
+ `probabilities`, `pick`, `position` and `expected_maximum` read, bit for bit.
1264
+ - `expected_maximum(params)`: E[the largest member], 1 less the integral over
1265
+ (0, 1) of the product of the members' distribution functions.
1266
+
1267
+ Measured (`argmax_accuracy_simulation_spec.rb`, below): on 30 fixtures, sharp
1268
+ ones with alpha + beta up to 2 * 10^5 among them, `pick`'s frequencies over
1269
+ 10^4 draws came within 2.44 standard deviations of `probabilities`, and
1270
+ `probabilities` within 2.43 of 10^6 plain `Sampler#beta` draws (bound 5).
1271
+
1272
+ ## Prequential log score (`Prequential`)
1273
+
1274
+ ```ruby
1275
+ PQ = SkiftetStatistical::Prequential
1276
+
1277
+ rows = [ { cluster: 1, log_probs: { shallow: Math.log(0.25), deep: Math.log(0.5) } },
1278
+ { cluster: 1, log_probs: { shallow: Math.log(0.5), deep: Math.log(0.5) } },
1279
+ { cluster: 2, log_probs: { shallow: Math.log(0.5), deep: Math.log(0.25) } },
1280
+ { cluster: 3, log_probs: { shallow: Math.log(0.125), deep: Math.log(0.5) } } ]
1281
+ PQ.score(rows, depth: :deep) # => 1.25 (bits per event)
1282
+ PQ.gains(rows, depths: %i[shallow deep]) # => { deep: { bits: 0.49999999999999994, se: 0.6495..., events: 4, clusters: 3 } }
1283
+ PQ.clustered_mean(sums: { 1 => 1.0, 2 => -1.0, 3 => 2.0 }, counts: { 1 => 2, 2 => 1, 3 => 1 })
1284
+ # => { mean: 0.5, lower: -0.7730..., upper: 1.7730..., n: 4, clusters: 3 }
1285
+ ```
1286
+
1287
+ The prequential log score (Dawid 1984): how many bits an outcome cost a
1288
+ predictor that gave it probability p, -log2 p, on outcomes it had not seen.
1289
+ `gains` compares depths of one predictor, each deeper than the one before, on
1290
+ the same events: per depth after the first, the mean gain in bits per event and
1291
+ its standard error. Events of one cluster move together, so the standard error
1292
+ comes from the spread between clusters, with the factor G / (G - 1) (Liang and
1293
+ Zeger 1986); `clustered_mean(sums:, counts:, level: 0.95)` is the same interval
1294
+ for a caller that stores per-cluster totals and counts, and `gains` is built on
1295
+ it, so the two cannot differ. Under 2 clusters the standard error and the
1296
+ bounds are nil. A log probability must be finite and at most 0: smooth a
1297
+ probability of 0 first.
1298
+
1299
+ Measured: no simulation; its specs hold hand-checked values.
1300
+
1301
+ ## One-sided change test (`Cusum`)
1302
+
1303
+ ```ruby
1304
+ CU = SkiftetStatistical::Cusum
1305
+
1306
+ CU.run([ 0.0, 0.0, 5.0 ], reference: 0.5, threshold: 4.0)
1307
+ # => { statistic: 4.5, alarm_at: 3, path: [0.0, 0.0, 4.5] }
1308
+ CU.average_run_length(shift: 0.0, reference: 0.5, threshold: 4.0) # => 338.09...
1309
+ h = CU.threshold(reference: 0.5, false_alarm_steps: 365.0) # => 4.0743...
1310
+ CU.minimum_detectable(reference: 0.5, threshold: h, horizon_steps: 30) # => 0.4752...
1311
+ ```
1312
+
1313
+ Page's (1954) statistic over standardised values,
1314
+ S_t = max(0, S_(t-1) + z_t - k), alarms at the first S_t above the threshold h
1315
+ (steps count from 1; nothing resets after an alarm). Run lengths are
1316
+ Siegmund's approximation, (exp(-2 theta b) + 2 theta b - 1) / (2 theta^2) with
1317
+ theta = shift - k and b = h + 1.166 (b^2 at theta = 0, summed as its series
1318
+ near 0). `threshold` turns a stated false-alarm rate, one alarm in
1319
+ `false_alarm_steps`, into h by bisection on [0, 50]; the rate is the
1320
+ approximation's, so a caller states it as about that rate.
1321
+ `minimum_detectable` is the smallest shift, in standard units, whose run length
1322
+ is at most `horizon_steps`; the caller scales it by its standard error.
1323
+
1324
+ Measured (`change_test_simulation_spec.rb`, below): a stated one false alarm in
1325
+ 365 steps gave 0.983 per 365 null steps, and one in 1,460 gave 0.260 per 365
1326
+ (stated 0.25), each inside its 95 % binomial interval; a shift of the minimum
1327
+ detectable size was caught within the horizon in 62 to 66 % of runs (bound
1328
+ half).
1329
+
1330
+ ## Bandit simulations: measured results
1331
+
1332
+ `spec/simulations` holds ten files, tagged `simulation: true` and run only
1333
+ with `SIMULATIONS=1` (`bundle exec rspec spec/simulations`; CI's simulations
1334
+ job runs them nightly, by hand, and on a change to the code they measure). Each
1335
+ example asserts its bounds; the figures beside the bounds were measured with
1336
+ the same code and seeds, and the files new in 0.3.0 and 0.4.0 print theirs
1337
+ through RSpec's reporter (the JSON formatter keeps them under "messages"). Every
1338
+ run is seeded, so each count below is exact for this version of the code, and
1339
+ every shape is generated here from round generic values. The eight files from
1340
+ 0.2.0 and 0.3.0 were measured on 2026-10-31 with skiftet_statistical 0.3.0, and
1341
+ again on 2026-11-02 with 0.4.0, every count the same; the two files
1342
+ new in 0.4.0 were measured on 2026-11-02. Ruby 3.4.9, YJIT on, as CI runs
1343
+ them.
1344
+
1345
+ The two simulations from 0.2.0 replay A/B tests through `Bandit::Allocator` (and
1346
+ `Bandit::WinHold`) in memory: every pass, 40 visits are drawn by the current
1347
+ weights, each exposure counts 1 / the weight it was drawn under, and the
1348
+ allocator is called with the previous weights, as an allocation job does every
1349
+ 15 minutes. Their three examples assert the bounds; the other figures were measured with the
1350
+ same code and seeds. Moved from the app that first ran them, with the same
1351
+ seeds, runs, passes, draws and bounds; measured again here on 2026-10-15 with
1352
+ skiftet_statistical 0.2.0 and Ruby 3.4.9, and every count the app had recorded
1353
+ came out the same.
1354
+ Measured again on 2026-10-31 with 0.3.0, the three asserted examples counted
1355
+ exactly as below (15 / 200; 0 / 100, an equal arm in 21 / 100; worse 0, better
1356
+ 46, undecided 154 / 200). The other counts are 0.2.0's, from an allocator that
1357
+ 0.3.0 and 0.4.0 keep bit for bit for every call that declares no per-arm share.
1358
+ In 0.4.0 these simulations draw a visit's arm by `Draw.from_weights`, and over
1359
+ all 12,263,200 draws of the four files that draw by weight it answered the
1360
+ draw the files had written by hand every time.
1361
+
1362
+ ### The automatic win (`allocator_simulation_spec.rb`)
1363
+
1364
+ 400 passes of 40 visits, 400 draws.
1365
+
1366
+ | Runs | Arms (true conversion rate) | Bound (asserted) | Measured |
1367
+ |---|---|---|---|
1368
+ | 200 (seeds 20 261 003 + run) | four arms at 30 % | an arm crowned in fewer than 20 / 200 | 15 / 200 (7.5 %) |
1369
+ | 100 (seeds 30 261 003 + run) | three arms at 30 %, one at 25 % | the worse arm crowned in 0 / 100 | 0 / 100 (an equal arm in 21 / 100) |
1370
+
1371
+ With equal arms a crowned arm costs nothing (it is as good as the rest, and its
1372
+ expected loss is at most `win_loss`), so the bound caps noise wins; a truly
1373
+ worse arm is never crowned.
1374
+
1375
+ ### The hold (`win_hold_simulation_spec.rb`)
1376
+
1377
+ Two arms, 11 % against 10 % (the worse arm), 400 passes of 40 visits (100
1378
+ hours), 200 runs (seeds 40 261 003 + run). The decision step is an allocation
1379
+ job's: the allocator with `hold_satisfied: hold.satisfied?`, and the same seed again with
1380
+ `hold_satisfied: false` when the arm it crowns is not the arm that held.
1381
+
1382
+ - WITHOUT the hold (the allocator alone, `hold_satisfied: true` at every pass):
1383
+ the worse arm crowned in 26 / 200 (13.0 %) at 400 draws, 24 / 200 (12.0 %) at
1384
+ 2 000 draws.
1385
+ - The hold's code with no wait (`hold_seconds: 0`), 1 000 draws: the worse arm
1386
+ crowned in 13 / 200 (6.5 %).
1387
+ - WITH the hold (24 hours): the worse arm crowned in 0 / 200 at 400, 1 000,
1388
+ 2 000 and 20 000 draws; the better arm in 41-46 / 200 (43 at 400, 46 at
1389
+ 1 000, 41 at 2 000, 44 at 20 000), and the rest undecided after 100 hours.
1390
+ - The asserted example runs at 1 000 draws: the worse arm at most 6 / 200 and
1391
+ the better arm at least 20 / 200 (a hold that never releases would crown
1392
+ nothing and pass the first bound alone). Measured: worse 0 / 200, better
1393
+ 46 / 200, undecided 154 / 200.
1394
+
1395
+ ### A value of several parts (`value_allocation_simulation_spec.rb`)
1396
+
1397
+ Each run replays a caller's passes in memory: visits drawn by the latest
1398
+ weights, each credited 1 / the share it was drawn under; the per-arm Hájek sums
1399
+ turned into the arms' joint normal (`SharedCredit.from_sums`), the value drawn
1400
+ with its uncertain weights (`LinearValue.summary`, P(best) among the free arms)
1401
+ and handed to `Allocator.from_summary`. A pass where a free arm has no credit
1402
+ yet takes the warm-up split (`summary: nil`). Where marked, the same shape also
1403
+ runs on `Allocator.call`'s yes/no tallies of the binary part. Unless stated, a
1404
+ visit's parts are a binary part worth 1 and a count part (1, 2 or 3 when not 0)
1405
+ worth uniform [0.25, 0.75], 40 visits a pass.
1406
+
1407
+ - **(a) Floors in every phase.** A reference arm floored at 0.05 and four arms
1408
+ at 0.02 (one arm clearly best), 400 passes walking warm-up, shifting, a pause
1409
+ on a stored split that breaks every floor, the automatic win and its stored
1410
+ win, and the reopened test; the same arms with one arm fixed at 0.30 beside
1411
+ them. 7 runs: floored on the value model 3, fixed on the value model 1,
1412
+ floored on tallies 2, fixed on tallies 1. Bound (asserted): no share below its
1413
+ floor or off its fixed share in any phase; every phase and an automatic win
1414
+ in every run. Measured: 0 breaches in 2,800 passes; per run warm-up 38 to 46
1415
+ passes, shifting 204 to 212, paused 20, won 130, one automatic win each;
1416
+ arm-passes held exactly at a floor 1,411, 1,428 and 1,428 (floored, value),
1417
+ 1,054 (fixed, value), 1,364 and 1,428 (floored, tallies), 1,063 (fixed,
1418
+ tallies).
1419
+ - **(b) No end.** A reference arm (floor 0.05) and two arms (0.02), one at twice
1420
+ the others' rate, 2,000 passes at `hold_satisfied: false`, on the value model
1421
+ and on tallies. Bound (asserted): nothing crowned, no share below its floor, and
1422
+ the better arm named leader in at least 1,800 passes. Measured: crowned 0 and
1423
+ 0; leader named in 1,926 (value) and 1,978 (tallies) passes; the final split
1424
+ 0.05 / 0.02 / 0.93 in both.
1425
+ - **(c) Late values.** Three arms whose values (0.15, 0.17 and 0.20 at the
1426
+ weights' centre) arrive 0 to 23 passes after the visit, uniformly, so every
1427
+ pass allocates on sums that lack the values still on their way; 60 passes,
1428
+ then the last values arrive; 200 runs. Bound (asserted): each arm's final
1429
+ estimate (`LinearValue`'s point) within 2 standard errors of the truth.
1430
+ Measured: bias -0.00054, -0.00197 and -0.00116 (standard errors 0.00166,
1431
+ 0.00188, 0.00107; z -0.33, -1.05, -1.08). Before the last values arrived (at
1432
+ pass 60, reported): -0.02798, -0.03399 and -0.03958 (z -19.7, -22.3, -41.8).
1433
+ - **(d) A drift shared by every arm.** Four equal arms, every probability three
1434
+ times its own over passes 41 to 60; the mean P(best) over passes 41 to 120 of
1435
+ 30 runs. Bound (asserted): each within 0.05 of 1 / 4. Measured: 0.2254,
1436
+ 0.2424, 0.2779 and 0.2543 over 2,400 scored passes. Held uneven: the same
1437
+ shape paused through the drift on a stored split giving one arm 0.65, 20
1438
+ runs. Bound (asserted): that arm's mean P(best) at most 0.30, while unweighted
1439
+ sums of the same visits (scored beside, never allocating) give it more than
1440
+ 0.5. Measured: 0.2092 (the others 0.3129, 0.2569 and 0.2209) over 1,600
1441
+ passes; on unweighted sums 0.9601 (the others 0.0114, 0.0157, 0.0129). The
1442
+ reference arm reads above 1 / 4 for its spread: it had the least traffic
1443
+ through the drift.
1444
+ - **(f) Coverage with drawn weights.** Three arms on a fixed split (0.2, 0.4,
1445
+ 0.4), 6,000 visits a run, a count part that is 0 in 95 % or more of visits
1446
+ (worth uniform [0.5, 1.5]) and a binary part at 1.0 to 1.5 % (worth uniform
1447
+ [2, 6]), the true worths drawn inside those ranges per run; 4,000 joint draws;
1448
+ 400 runs. Bound (asserted): each arm's 95 % interval and each difference's
1449
+ covers the truth in at least 372 of 400 (93 %). Measured: arms 378, 377 and
1450
+ 383; differences 382, 385 and 383 (94.25 to 96.25 %).
1451
+
1452
+ ### Pooled cells and two layers (`two_layer_simulation_spec.rb`)
1453
+
1454
+ - **(e) Cells whose levels differ.** 40 cells in 8 groups of 5, crossed with 4
1455
+ bands. Each cell's level is its factor (0.5 to 2.45 in steps of 0.05, in an
1456
+ order that follows neither group nor band) times its covariate mix: each
1457
+ visit's covariate value multiplies its probabilities by 0.5, 1.0 or 1.5, in
1458
+ proportions that differ by cell. No arm's values differ from the reference
1459
+ arm's in any cell. Three arms (the reference floored at 0.05, two at 0.02),
1460
+ 30 visits a cell a pass, 50 passes, 3 runs. Each cell is allocated at every
1461
+ pass on its pooled estimate: its contrasts by `SharedCredit.contrast_for_fit`
1462
+ (Kish's minimum 20), `AdditiveContrasts.fit` over (arm, part) pairs, one
1463
+ `baseline` a part, `pooled_estimate`, `LinearValue.summary` and
1464
+ `from_summary` (warm-up 0, nothing crowned). The variances run as designed: a
1465
+ weekly fit by moments every 7 passes, each variance held at 0 unless its test
1466
+ of no variation rejects at 0.05 divided by that week's number of tests, and
1467
+ held fixed between. Bound (asserted): the reference arm's mean share across
1468
+ the 40 cells at a rank correlation with the level within plus or minus 0.3,
1469
+ and a spread (largest less smallest) at most 1.5 times the noise floor's, the
1470
+ same simulation, seeds, runs and passes with every cell's level equal.
1471
+ Measured: 0.1426 to 0.1492 (spread 0.0066, standard deviation 0.0018) over
1472
+ 5,880 cell passes allocated on a pooled estimate, rank correlation -0.073;
1473
+ noise floor 0.1225 to 0.1291 (spread 0.0065, standard deviation 0.0014), so
1474
+ 1.007 times; 21 weekly fits, each test read at 0.004167, 1 of 252 variances
1475
+ kept at its moment (0.004293) and none in the noise floor; 1,135 of 24,000
1476
+ cell contrasts left out of the fit (1,440 in the noise floor). Against a fixed
1477
+ 0.03 the spread is 0.0066 (reported). With every pass fitting the variances by
1478
+ moments instead (reported, no bound): 0.11 to 0.2625 (spread 0.1525,
1479
+ standard deviation 0.0425), rank correlation -0.047, 1,051 contrasts left
1480
+ out; at no true variation a fitted variance comes out above 0 in about half
1481
+ the fits, and a cell then keeps that share of its own noise in its effects.
1482
+ - **(j) Two layers.** Two options of identical value, each above an inner layer
1483
+ that picks one of three candidates (rates 0.30, 0.20 and 0.10). The warm
1484
+ option's candidates are held at 1,000 showings each; the cold option's start
1485
+ at Beta(1, 1) and learn from their showings, their best placed first, second
1486
+ or third by run. Each option's candidates are one posterior set
1487
+ (`Posteriors.beta`); each visit's item is drawn by the exact pick odds
1488
+ (`Posteriors#pick_probabilities`, explore 0.1) through `Draw.from_weights`,
1489
+ and each option is scored by its greedy pick (`Posteriors#greedy`) through
1490
+ `SharedCredit`: a visit credits every option whose greedy pick it showed,
1491
+ with 1 / the visit's probability of showing that item, the two layers' odds
1492
+ combined by `Posteriors.mixture`. 30 passes of 80 visits, 100 runs. Bound (asserted): both options' mean
1493
+ P(best) in 0.40 to 0.60, and the cold option at its floor at the end in at
1494
+ most 10 runs. Measured: 0.5807 and 0.4193 over 3,000 passes, the cold option
1495
+ at its floor in 2 of 100. Scored as they ran (each visit crediting its drawn
1496
+ option with 1 / its share; reported): 0.6273 and 0.3727, and an allocation on
1497
+ that score ends the cold option at its floor in 5.
1498
+
1499
+ ### Tree recovery (`tree_recovery_simulation_spec.rb`)
1500
+
1501
+ Five-level generated trees, 1, 20, 200, 800 and 4,000 nodes a level, with
1502
+ single-child parents (1, 10 and 40 at levels 1 to 3), parents that are one
1503
+ child's near-whole (95 % of their exposure, at a rate of 0.05) and 2 % of the
1504
+ leaves split between two parents, so counts above the leaves are real numbers.
1505
+ One law for every item: multipliers Gamma(k, rate k) with k = 8, 8 and 16 at
1506
+ levels 1 to 3, and at the leaves a two-groups law, hot share 0.02 and slab
1507
+ strength 0.5. Nine items, three national (200,000 counts a half), three
1508
+ regional (20,000) and three very local (2,000); each count Poisson, drawn twice:
1509
+ half A is fitted, half B held out. 20 runs (seeds 20 261 131 + run).
1510
+
1511
+ - **Level strengths** (bound: within a factor of 2 in at least 18 of 20 runs):
1512
+ level 1 (true 8) in 20 of 20, fitted 5.657 four times, 6.727 twelve times and
1513
+ 8.0 four times; level 2 (true 8) in 20 of 20, 5.657 nine times and 6.727
1514
+ eleven times; level 3 (true 16) in 20 of 20, 8.0 twice and 9.514 eighteen
1515
+ times. The upper levels read low (they absorb part of the variance below
1516
+ them), and the two level-3 runs at 8.0 sit on the bound's edge.
1517
+ - **Leaves on `fit_leaves`** (bound: the slab strength within a factor of 2 and
1518
+ the hot share within 0.05, each in at least 18 of 20 runs): slab in 20 of 20
1519
+ (0.354 ten times, 0.5 nine times, 0.707 once), hot share in 20 of 20 (0.01
1520
+ nineteen times, 0.02 once); 7,200 parents a run, 0 to 7 of them not settled
1521
+ after 50 steps. The same leaves fitted on leave-one-out expected counts
1522
+ (`fit_strength`; reported, no bound): hot share 0.05 five times and 0.1
1523
+ fifteen times, slab 0.42 once, 0.5 twice, 0.595 twice, 0.841 twice, 1.0 five
1524
+ times and 1.189 eight times (up to 2.4 times the truth).
1525
+ - **Single-child parents** (bound: every fit identical without their pairs):
1526
+ identical at level 2 (8 to 9 such pairs a run, 20 runs), level 3 (76 to 88,
1527
+ 20 runs) and level 4 (269 to 276, 2 runs); `fit_leaves` identical without the
1528
+ 360 parents of a single leaf a run (2 runs).
1529
+ - **Held-out log score of the shrunken ratio less the raw one** (bound: better
1530
+ summed over the levels, and at each level below the first, in every run):
1531
+ level 2 better in 20 of 20 (min 251.3, median 478.7, max 749.0 nats), level 3
1532
+ in 20 of 20 (4,431.6; 5,364.2; 6,051.1), level 4 in 20 of 20 (46,538.5;
1533
+ 48,844.4; 51,500.6). Level 1, where a node holds thousands of an item's counts
1534
+ and shrinking moves a share by parts in 10^5 (reported): better in 14 of 20
1535
+ (min -2.6, median 3.0, max 31.3).
1536
+
1537
+ ### Contrasts with known variances (`contrasts_simulation_spec.rb`)
1538
+
1539
+ One root, 400 nodes under it ("tree:1"), 6 leaves a node ("tree:2"), 2 cells a
1540
+ leaf (4,800 cells), crossed with grouping a (300 keys) and grouping b (150
1541
+ keys), each cell's keys drawn uniformly; each cell's sampling variance uniform
1542
+ on [0.4, 1.4] (mean 0.8999 as handed to `fit`). Three arms in one call at the
1543
+ default 5 sweeps: x with every term at tau^2 = 4, y with every term at 0, z with
1544
+ "tree:1" and a at 0 beside "tree:2" at 4 and b at 8. 100 runs (seeds
1545
+ 20 261 101 + run). Bound (asserted, in every run): a variance of at least 4
1546
+ times the mean sampling variance (and of its groups' own) within a factor of 2;
1547
+ a variance of 0 under a quarter of the mean sampling variance. Measured, the
1548
+ mean (min to max) of 100 fits:
1549
+
1550
+ | Term | x (truth 4) | y (truth 0) | z |
1551
+ |---|---|---|---|
1552
+ | tree:1 | 3.9072 (2.9473 to 4.5137) | 0.0010 (to 0.0118) | truth 0: 0.0173 (to 0.1126) |
1553
+ | tree:2 | 3.8028 (3.3739 to 4.2671) | 0.0049 (to 0.0387) | truth 4: 3.9761 (3.6161 to 4.3358) |
1554
+ | a | 4.0323 (3.3199 to 4.6050) | 0.0012 (to 0.0088) | truth 0: 0.0006 (to 0.0111) |
1555
+ | b | 3.9434 (2.8883 to 5.6817) | 0.0009 (to 0.0086) | truth 8: 7.9093 (6.0397 to 10.5169) |
1556
+
1557
+ The groups' own mean sampling variances: 0.7373 (x and z at "tree:1"), 0.0665
1558
+ (y at "tree:1"), 0.4038 ("tree:2"), 0.0499 (a) and 0.0250 (b). Two leaves a
1559
+ node, where the K - 1 in the moment matters most (bound: the mean of 100 runs
1560
+ within 5 % of the truth): "tree:1" truth 2.0, mean 2.0097 (1.6855 to 2.3061);
1561
+ "tree:2" truth 1.0, mean 1.0004 (0.7745 to 1.2209). Measured beside the shape
1562
+ and kept out of it: with one cell a leaf, a leaf's deviation and its cell's
1563
+ noise cannot be told apart, and at 5 sweeps a crossed grouping at 0 beside one
1564
+ at 8 reads about 0.6 to 1.2.
1565
+
1566
+ ### The argmax (`argmax_accuracy_simulation_spec.rb`)
1567
+
1568
+ 30 fixtures from round values (closed forms, parameters well below 1 at either
1569
+ end, U shapes, close competitors, up to 8 members, mixed widths, and sharp
1570
+ members with alpha + beta from 10^3 to 2 * 10^5), each with the epsilon `pick`
1571
+ explores at. `pick` takes its uniforms from one seeded Random a fixture, 10^4
1572
+ draws; the reference for `probabilities` (at epsilon 0) is the plain argmax of
1573
+ one `Sampler#beta` draw a member, 10^6 draws a fixture. Bound (asserted): every
1574
+ member within 5 standard deviations, sqrt(p * (1 - p) / n) with p the answer of
1575
+ `probabilities`. Measured, the largest gap of each fixture in standard
1576
+ deviations: `pick` at most 2.44, `probabilities` at most 2.43.
1577
+
1578
+ | Fixture | Members `[alpha, beta]` | `pick`'s epsilon | `pick` | `probabilities` (largest absolute gap) |
1579
+ |---|---|---|---|---|
1580
+ | 0 | `[[1, 1], [2, 1]]` | 0.0 | 0.47 | 1.41 (6.62e-04) |
1581
+ | 1 | `[[1, 1], [1, 1]]` | 0.0 | 0.80 | 0.44 (2.19e-04) |
1582
+ | 2 | `[[2, 8], [8, 2]]` | 0.1 | 0.51 | 0.18 (7.55e-06) |
1583
+ | 3 | `[[1, 0.05], [1, 1]]` | 0.0 | 0.57 | 0.78 (1.65e-04) |
1584
+ | 4 | `[[0.05, 1], [1, 1]]` | 0.0 | 1.02 | 0.51 (1.08e-04) |
1585
+ | 5 | `[[0.5, 0.5], [1, 1]]` | 0.0 | 0.74 | 0.18 (8.80e-05) |
1586
+ | 6 | `[[0.2, 0.3], [0.3, 0.2]]` | 0.05 | 0.02 | 0.87 (4.14e-04) |
1587
+ | 7 | `[[0.1, 0.1], [2, 2]]` | 0.0 | 1.44 | 0.61 (3.04e-04) |
1588
+ | 8 | `[[3, 7], [4, 6], [5, 5]]` | 0.0 | 0.83 | 1.34 (4.08e-04) |
1589
+ | 9 | `[[1, 9], [2, 8], [3, 7], [4, 6]]` | 0.05 | 1.66 | 0.35 (9.44e-05) |
1590
+ | 10 | `[[2, 3], [3, 2], [5, 5], [1, 1], [4, 6]]` | 0.1 | 2.44 | 1.65 (6.99e-04) |
1591
+ | 11 | `[[12, 88], [15, 85], [9, 91], [20, 180], [3, 17]]` | 0.1 | 1.00 | 0.66 (3.22e-04) |
1592
+ | 12 | `[[0.5, 50], [50, 0.5]]` | 0.0 | 0.00 | 0.00 (1.24e-31) |
1593
+ | 13 | `[[10, 10], [10.5, 9.5]]` | 0.0 | 1.03 | 0.65 (3.25e-04) |
1594
+ | 14 | `[[1, 3], [3, 1], [2, 2]]` | 0.2 | 0.82 | 1.15 (4.77e-04) |
1595
+ | 15 | `[[0.8, 0.8], [0.9, 0.7], [0.7, 0.9]]` | 0.0 | 1.29 | 0.89 (4.41e-04) |
1596
+ | 16 | `[[3, 7], [3.1, 6.9], [2.9, 7.1], [3.05, 6.95]]` | 0.1 | 1.90 | 2.01 (8.67e-04) |
1597
+ | 17 | `[[1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1]]` | 0.0 | 1.38 | 2.42 (9.01e-04) |
1598
+ | 18 | `[[5, 1], [1, 5]]` | 0.0 | 0.59 | 0.38 (2.37e-05) |
1599
+ | 19 | `[[100, 900], [1, 9]]` | 0.0 | 0.71 | 1.45 (7.06e-04) |
1600
+ | 20 | `[[2, 2], [2, 2], [2, 2], [3, 3]]` | 0.3 | 1.45 | 2.29 (1.00e-03) |
1601
+ | 21 | `[[1, 2], [2, 3], [3, 4], [4, 5], [5, 6], [6, 7], [7, 8], [8, 9]]` | 0.05 | 1.15 | 2.43 (8.03e-04) |
1602
+ | 22 | `[[1, 2], [2, 1], [1.5, 1.5]]` | 0.1 | 1.24 | 1.07 (4.78e-04) |
1603
+ | 23 | `[[4, 4], [4.4, 3.6]]` | 0.0 | 0.45 | 1.53 (7.54e-04) |
1604
+ | 24 | `[[0.3, 3], [3, 0.3], [1, 1]]` | 0.0 | 1.08 | 1.94 (5.38e-04) |
1605
+ | 25 | `[[700, 300], [7000, 3000]]` | 0.1 | 1.05 | 0.88 (4.39e-04) |
1606
+ | 26 | `[[20000, 80000], [20, 80]]` | 0.0 | 0.85 | 0.03 (1.52e-05) |
1607
+ | 27 | `[[50, 99950], [1, 999]]` | 0.0 | 0.55 | 1.40 (6.83e-04) |
1608
+ | 28 | `[[99950, 50], [999, 1]]` | 0.0 | 0.34 | 0.31 (1.52e-04) |
1609
+ | 29 | `[[60000, 140000], [60150, 139850]]` | 0.1 | 0.69 | 0.64 (2.94e-04) |
1610
+
1611
+ ### The change test (`change_test_simulation_spec.rb`)
1612
+
1613
+ Two settings, each a stated false-alarm rate and a horizon. False alarms: one
1614
+ null stream of 2,000 consecutive spans of 365 steps (730,000 steps, z ~ N(0, 1)
1615
+ by `Sampler#gaussian`), the statistic starting again from 0 after each alarm.
1616
+ Detection: 2,000 runs shifted by the smallest detectable change from their
1617
+ first step, and the same change after 365 null steps. Bound (asserted): the
1618
+ false alarms inside the stated rate's 95 % binomial interval, and the change
1619
+ caught within the horizon in at least 1,000 of 2,000 runs, both ways.
1620
+
1621
+ | Reference | Stated rate | Threshold | False alarms in 730,000 steps (95 % interval) | Per 365 steps | Horizon | Smallest change | Caught from the start | Caught after 365 null steps |
1622
+ |---|---|---|---|---|---|---|---|---|
1623
+ | 0.5 | 1 in 365 | 4.0744 | 1,966 (1,913 to 2,088) | 0.983 | 30 | 0.4752 | 1,239 (0.620) | 1,322 (0.661) |
1624
+ | 0.25 | 1 in 1,460 | 9.3147 | 521 (457 to 544) | 0.260 | 60 | 0.3532 | 1,264 (0.632) | 1,305 (0.652) |
1625
+
1626
+ Measured beside it: 2,000 separate spans, each starting at 0, count 1.2 % fewer
1627
+ alarms at reference 0.5 and 3.9 % fewer at 0.25, because every span's first
1628
+ steps start from 0.
1629
+
1630
+ ### The order rule at small-site traffic (`order_simulation_spec.rb`)
1631
+
1632
+ A list of K items (3, 5 or 10), each with its own true rate, fixed for a run and
1633
+ dealt to the items in a random order a run. Every visit is shown the whole list
1634
+ in one order and sees it down to a depth drawn from a reach curve, the share of
1635
+ visits that see position l: (1, 0.6, 0.35) at K = 3,
1636
+ (1, 0.70, 0.50, 0.35, 0.25) at K = 5 and 0.8^l at K = 10. Each item a visit
1637
+ sees yields an outcome at its rate: yes/no, or an amount (log-normal, mean 200,
1638
+ sigma 0.9, capped at 2 000). The caller's whole part: a view counts only where
1639
+ an item was seen; every hour the set is rebuilt from the tallies
1640
+ (`Posteriors.yes_no` on the flat prior, or `Posteriors.amounts` under
1641
+ `Prior.amount(mean: 4.0, variance: 2_000.0, strength: 20.0)`), and every visit
1642
+ of that hour is shown the set's `rank(rng:, shuffle: 0.10)`. 28 days of 24
1643
+ hours, 50 seeded runs a cell (`Random.new(20 261 102 + 50 i + r)` for run r of
1644
+ cell i).
1645
+
1646
+ Measured: lost, the expected outcomes the shown orders missed against the best
1647
+ order (the items by true rate), as a share of the best order's, and the same for
1648
+ a fixed order (one random order a run shown to every visit); the posterior-mean
1649
+ order (the items by `summary(among: [])`'s means) on days 7 and 28, whether its
1650
+ first item is the best one and whether it is the best order exactly. Mixed
1651
+ visits: half shallow (reach (1, 0.5, 0.3, 0.15, 0.1), every rate halved), half
1652
+ deep (reach (1, 0.9, 0.7, 0.55, 0.4), every rate times 1.5), the known weakness
1653
+ of counting views only where seen, as an item shown low is seen mostly by deep
1654
+ visits and reads a rate above the one it would show on top. Bound (asserted,
1655
+ every cell): lost at most half the fixed order's, and the random share within 4
1656
+ standard errors of 0.10 (measured 0.0990 to 0.1003); each cell's own bounds
1657
+ below. The reference is the same model run 100 times a cell in another language
1658
+ and generator, the figures the bounds were set from with margin (fixed lost /
1659
+ lost / best on top day 7 / day 28 / exact order day 28); 50 runs and another
1660
+ generator read near them, not equal to them.
1661
+
1662
+ | Rates (true) | Visits a day | Fixed order lost | Lost | Best on top, day 7 | Best on top, day 28 | Exact order, day 28 | Bound (asserted) | Reference (100 runs) |
1663
+ |---|---|---|---|---|---|---|---|---|
1664
+ | five, close: 5, 4.5, 4, 3.5, 3 % | 300 | 8.0 % | 2.4 % | 74 % | 88 % | 62 % | day 28 >= 75 %, exact >= 35 %, lost <= 4.5 % | 8.3 / 2.8 / 69 / 90 / 54 |
1665
+ | five, close | 1 000 | 7.4 % | 1.6 % | 82 % | 100 % | 94 % | day 28 >= 90 %, exact >= 75 %, lost <= 3.0 % | 7.6 / 1.6 / 91 / 99 / 91 |
1666
+ | five, close | 3 000 | 7.4 % | 1.1 % | 94 % | 100 % | 100 % | day 28 >= 90 %, exact >= 90 %, lost <= 2.0 % | 7.7 / 1.1 / 99 / 100 / 100 |
1667
+ | five, clear: 8, 6, 5, 4, 3 % | 300 | 14.1 % | 2.5 % | 98 % | 100 % | 86 % | day 7 >= 90 % | 12.8 / 2.4 / 100 / 100 / 97 |
1668
+ | five, clear | 1 000 | 14.3 % | 1.7 % | 100 % | 100 % | 100 % | day 7 >= 90 % | 12.8 / 1.7 / 100 / 100 / 100 |
1669
+ | five, clear | 3 000 | 13.9 % | 1.5 % | 100 % | 100 % | 100 % | day 7 >= 90 % | 13.6 / 1.5 / 100 / 100 / 100 |
1670
+ | three: 5, 4, 3 % | 1 000 | 7.7 % | 1.1 % | 98 % | 100 % | 100 % | day 28 >= 90 % | 7.0 / 1.1 / 100 / 100 / 100 |
1671
+ | ten: 6 % down to 1.5 % by 0.5 | 3 000 | 19.6 % | 2.2 % | 100 % | 100 % | 100 % | day 28 >= 90 % | 18.6 / 2.2 / 98 / 100 / 100 |
1672
+ | five amounts: 4, 3, 2.5, 2, 1.5 % | 300 | 13.8 % | 4.3 % | 82 % | 100 % | 68 % | day 28 >= 85 %, lost <= 6 % | 13.1 / 4.3 / 88 / 98 / 53 |
1673
+ | five amounts | 1 000 | 13.7 % | 2.6 % | 92 % | 100 % | 84 % | day 28 >= 95 %, lost <= 4 % | 12.4 / 2.6 / 98 / 100 / 88 |
1674
+ | five amounts | 3 000 | 12.6 % | 1.9 % | 100 % | 100 % | 100 % | day 28 >= 95 %, lost <= 3 % | 12.9 / 1.8 / 100 / 100 / 98 |
1675
+ | five, close, mixed visits | 300 | 6.9 % | 2.3 % | 78 % | 94 % | 70 % | day 28 >= 75 % | 6.4 / 2.4 / - / 90 / 58 |
1676
+ | five, close, mixed visits | 1 000 | 5.5 % | 1.5 % | 82 % | 98 % | 96 % | day 28 >= 90 % | 6.2 / 1.5 / - / 99 / 95 |
1677
+
1678
+ At 300 visits a day the order rule loses a fifth to a third of what a fixed
1679
+ order loses, and the posterior-mean order finds the best item first in
1680
+ most runs by day 28; at 1 000 and more it finds the best order exactly in most.
1681
+ The mixed cells hold their bounds, though the deep visits flatter the items
1682
+ shown low.
1683
+
1684
+ ### The odds against the rule's visits (`odds_simulation_spec.rb`)
1685
+
1686
+ Visits: 2 * 10^5 calls of `rank(rng:)` a fixture on one seeded Random (seed
1687
+ 20 261 131); every head and every position against its exact value
1688
+ (`pick_probabilities`, `positions`), and the orders by one chi-square test
1689
+ against `order_probability` (each order with n p >= 5 its own bin, the rest
1690
+ pooled into one). Bound (asserted): every head and position within 4 standard
1691
+ errors, sqrt(p (1 - p) / n), the chi-square's p-value above 0.001, and no order
1692
+ seen that the exact odds give no probability. Items a, b, c, ... in the order
1693
+ their pairs are listed; slots count from 0.
1694
+
1695
+ | Fixture | Rule | Largest gap (standard errors) | Orders' chi-square | p |
1696
+ |---|---|---|---|---|
1697
+ | five Betas: [0.02, 19.98], [3, 60], [4, 70], [6, 95], [2, 45] | shuffle 0.1, explore 0.1 | 2.75 (d at slot 2) | 137.5 on 119 degrees | 0.1186 |
1698
+ | four Betas: [0.3, 20], [1, 20], [2, 30], [5, 90]; views 50, 400, 400, 400 | explore 0.2 among items of at least 100 views | 1.12 (a at slot 1) | 15.8 on 23 | 0.8656 |
1699
+ | four Normals near 5 a view: [5.0, 0.4], [5.3, 0.5], [4.8, 0.3], [5.1, 0.8] | shuffle 0.05 | 1.71 (c at slot 1) | 23.3 on 23 | 0.4457 |
1700
+ | the four Betas above, c pinned at slot 2 | shuffle 0.1 | 1.33 (a at slot 3) | 2.3 on 5 | 0.8086 |
1701
+ | five Normals of mixed widths: [1.0, 1.0], [1.5, 0.5], [0.5, 2.0], [1.2, 0.2], [0.8, 0.7] | explore 0.1, the top 2 | 1.34 (e as head) | 12.2 on 19 | 0.8781 |
1702
+
1703
+ Joint draws: 10^6 a fixture (seed 20 261 141), one draw an item, in order; how
1704
+ often each item is the largest against `summary`'s p_best (and, for Normals,
1705
+ `NormalArgmax.probabilities`, which read the same), and the mean of max - X_i
1706
+ against its expected loss. Bound (asserted): each within 4 standard errors of
1707
+ the draws. Measured, the gap of each item in standard errors:
1708
+
1709
+ | Fixture | P(best) | Expected loss |
1710
+ |---|---|---|
1711
+ | the four Normals near 5 a view | 1.60, 0.28, 1.57, 0.82 | 0.87, 0.74, 0.46, 1.34 |
1712
+ | the five Normals of mixed widths | 0.41, 1.14, 0.79, 1.30, 0.01 | 1.20, 0.44, 0.55, 0.04, 0.39 |
1713
+ | four Betas: [12, 88], [15, 85], [9, 91], [3, 17] | 1.37, 1.07, 0.37, 0.24 | 0.26, 0.18, 0.21, 0.31 |
1714
+ | the four Betas with an alpha of 0.3 | 1.44, 1.16, 1.10, 1.61 | 1.20, 0.85, 1.44, 0.76 |
1715
+
1716
+ ### Run time
1717
+
1718
+ On one development machine, Ruby 3.4.9 with YJIT, in seconds, both columns
1719
+ measured the same way: each figure is the sum of a file's examples' own run
1720
+ times as RSpec reports them, Ruby's boot and the file's load left out. 0.3.0 ran
1721
+ on 2026-10-31 with the machine otherwise idle, 0.4.0 on 2026-11-02 while other
1722
+ work kept it busy (a load average of 31 to 46), so the two columns differ by the
1723
+ load as well as by the code.
1724
+
1725
+ | File | Examples | 0.3.0 | 0.4.0 |
1726
+ |---|---|---|---|
1727
+ | `allocator_simulation_spec.rb` | 2 | 150 | 173 |
1728
+ | `win_hold_simulation_spec.rb` | 1 | 132 | 160 |
1729
+ | `value_allocation_simulation_spec.rb` | 6 | 56 | 85 |
1730
+ | `two_layer_simulation_spec.rb` | 3 | 68 | 96 |
1731
+ | `tree_recovery_simulation_spec.rb` | 3 | 178 | 515 |
1732
+ | `contrasts_simulation_spec.rb` | 2 | 17 | 19 |
1733
+ | `argmax_accuracy_simulation_spec.rb` | 2 | 232 | 265 |
1734
+ | `change_test_simulation_spec.rb` | 4 | 1 | 1 |
1735
+ | `odds_simulation_spec.rb` | 3 | - | 33 |
1736
+ | `order_simulation_spec.rb` | 13 | - | 667 |
1737
+ | All | 39 | 833 (about 14 minutes) | 2,013 (about 34 minutes) |
1738
+
1739
+ In three other runs the order file alone took 263, 311 and 407 seconds, the
1740
+ more the busier the machine. Without YJIT the eight files of 0.3.0 took about 22
1741
+ minutes in earlier runs on the same machine. The budget is 25 minutes on CI's
1742
+ Ruby 3.4; the job's timeout is 45 minutes (30 before 0.4.0).
1743
+
1744
+ A change that moves any of these counts changes the rule's behaviour; measure
1745
+ again and update these figures in the same change.
1746
+
153
1747
  ## Development
154
1748
 
155
1749
  ```sh
@@ -157,6 +1751,8 @@ bundle install
157
1751
  bundle exec rake spec # run the specs
158
1752
  bundle exec rake rubocop # lint
159
1753
  bundle exec rake # both
1754
+ SIMULATIONS=1 bundle exec rspec spec/simulations # the bandit simulations (minutes)
1755
+ RUBY_YJIT_ENABLE=1 SIMULATIONS=1 bundle exec rspec spec/simulations # as CI runs them, in about 60 % of the time
160
1756
  ```
161
1757
 
162
1758
  ## License