skiftet_statistical 0.1.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +346 -2
- data/README.md +1618 -22
- data/lib/skiftet_statistical/additive_contrasts.rb +894 -0
- data/lib/skiftet_statistical/bandit/allocator.rb +688 -0
- data/lib/skiftet_statistical/bandit/checks.rb +128 -0
- data/lib/skiftet_statistical/bandit/decay.rb +86 -0
- data/lib/skiftet_statistical/bandit/draw.rb +179 -0
- data/lib/skiftet_statistical/bandit/normal_argmax.rb +209 -0
- data/lib/skiftet_statistical/bandit/pass.rb +94 -0
- data/lib/skiftet_statistical/bandit/posteriors.rb +497 -0
- data/lib/skiftet_statistical/bandit/prior.rb +264 -0
- data/lib/skiftet_statistical/bandit/probabilities.rb +929 -0
- data/lib/skiftet_statistical/bandit/ranking.rb +552 -0
- data/lib/skiftet_statistical/bandit/summary.rb +152 -0
- data/lib/skiftet_statistical/bandit/tallies.rb +297 -0
- data/lib/skiftet_statistical/bandit/weighted_posterior.rb +65 -0
- data/lib/skiftet_statistical/bandit/win_hold.rb +86 -0
- data/lib/skiftet_statistical/bandit.rb +3 -3
- data/lib/skiftet_statistical/beta_argmax.rb +526 -0
- data/lib/skiftet_statistical/beta_function.rb +153 -0
- data/lib/skiftet_statistical/cholesky.rb +244 -0
- data/lib/skiftet_statistical/cusum.rb +183 -0
- data/lib/skiftet_statistical/descriptive.rb +3 -3
- data/lib/skiftet_statistical/gamma.rb +97 -0
- data/lib/skiftet_statistical/gauss_legendre.rb +73 -0
- data/lib/skiftet_statistical/hierarchical_rates.rb +450 -0
- data/lib/skiftet_statistical/inputs.rb +214 -0
- data/lib/skiftet_statistical/joint_beta.rb +63 -0
- data/lib/skiftet_statistical/linear_value.rb +364 -0
- data/lib/skiftet_statistical/numbers.rb +98 -0
- data/lib/skiftet_statistical/prequential.rb +206 -0
- data/lib/skiftet_statistical/shared_credit.rb +811 -0
- data/lib/skiftet_statistical/significance.rb +65 -5
- data/lib/skiftet_statistical/version.rb +1 -1
- data/lib/skiftet_statistical.rb +57 -5
- metadata +43 -11
data/README.md
CHANGED
|
@@ -1,24 +1,64 @@
|
|
|
1
1
|
# skiftet_statistical
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
3
|
+
A dependency-free **statistics toolkit** for Ruby: reusable statistical analysis
|
|
4
|
+
code, so the same z-test, percentile, sampler or bandit isn't re-implemented
|
|
5
|
+
(differently) in every app.
|
|
6
6
|
|
|
7
7
|
Modules:
|
|
8
8
|
|
|
9
9
|
- **`Descriptive`** — mean, variance, standard deviation, percentiles/median.
|
|
10
10
|
- **`Significance`** — A/B significance testing: two-proportion z-test, Welch's
|
|
11
|
-
t-test, exact normal CDF and two-tailed p-values
|
|
11
|
+
t-test, exact normal CDF and two-tailed p-values, the normal quantile, the
|
|
12
|
+
chi-squared survival and the sample-ratio check.
|
|
13
|
+
- **`Gamma`** / **`BetaFunction`** — the regularized incomplete gamma and beta
|
|
14
|
+
functions, and the Beta quantile.
|
|
12
15
|
- **`Sampler`** — Gamma/Beta/Gaussian random sampling, RNG-injectable.
|
|
16
|
+
- **`JointBeta`** — joint Monte Carlo draws over Beta posteriors (P(best),
|
|
17
|
+
expected loss).
|
|
13
18
|
- **`Bandit`** (+ `Policies`) — multi-armed bandit (Thompson Sampling,
|
|
14
19
|
epsilon-greedy, UCB1, Softmax) for online explore/exploit decisions.
|
|
20
|
+
- **`Bandit::Posteriors`** (with **`Bandit::Prior`**, **`Bandit::Decay`**,
|
|
21
|
+
**`Bandit::Draw`**, **`Bandit::NormalArgmax`** and **`Bandit::Pass`**) — new
|
|
22
|
+
in 0.4.0, the generic bandit: a set of items with their views and yes/no or
|
|
23
|
+
amount posteriors, ordered and picked by one rule from a key, given uniforms
|
|
24
|
+
or an RNG, with the odds of what it draws (exact for a pick among up to
|
|
25
|
+
`EXACT_PICK_MAX` (20) contenders and for orders and positions over up to
|
|
26
|
+
`EXACT_ORDER_MAX` (10) free items with none barred from leading, otherwise a
|
|
27
|
+
seeded Monte Carlo estimate with its standard error) and the summaries an
|
|
28
|
+
allocation reads.
|
|
29
|
+
- **`Bandit::Allocator`**, **`Bandit::WinHold`**, **`Bandit::WeightedPosterior`**
|
|
30
|
+
— an A/B test's traffic allocation: Thompson sampling with a floor per arm, a
|
|
31
|
+
step limit, an automatic win and a hold on it.
|
|
32
|
+
Also per-arm floors and fixed shares, and one phase rule for any posterior
|
|
33
|
+
(`Allocator.from_summary`).
|
|
34
|
+
- **`LinearValue`** (with **`Cholesky`**) — P(best), expected loss and
|
|
35
|
+
intervals of a linear value of several rates when both the rates (one joint
|
|
36
|
+
normal) and their weights are uncertain.
|
|
37
|
+
- **`SharedCredit`** — Hájek inverse-propensity means of several parts per arm,
|
|
38
|
+
with one covariance over every (arm, part) pair, when one row may credit
|
|
39
|
+
several arms; its sums fold once and store as JSON.
|
|
40
|
+
- **`HierarchicalRates`** — Gamma-Poisson rates on a tree: a node's ratio to
|
|
41
|
+
what its parent leads us to expect, the fit of a level's strength (and of a
|
|
42
|
+
two-groups hot share), the leaf fit, and checks beside them.
|
|
43
|
+
- **`AdditiveContrasts`** — the arms' contrasts against a reference arm pooled
|
|
44
|
+
across cells along a nested tree and crossed groupings, a test of no
|
|
45
|
+
variation, each cell's level from all its visits, and the pooled estimate a
|
|
46
|
+
cell is allocated on.
|
|
47
|
+
- **`BetaArgmax`** (with **`GaussLegendre`** and `BetaFunction.log_density`) —
|
|
48
|
+
the exact probability that each of several Betas is the largest, the seedless
|
|
49
|
+
pick it describes, and the greedy pick; since 0.4.0 also a draw's position,
|
|
50
|
+
the distribution function and E[the largest].
|
|
51
|
+
- **`Prequential`** — the prequential log score, and the bits per event a
|
|
52
|
+
deeper predictor saves, with cluster-robust intervals.
|
|
53
|
+
- **`Cusum`** — a one-sided change test: its statistic, its run length, the
|
|
54
|
+
threshold for a stated false-alarm rate and the smallest change it can see.
|
|
15
55
|
|
|
16
56
|
## Install
|
|
17
57
|
|
|
18
|
-
In a Gemfile (
|
|
58
|
+
In a Gemfile (published on rubygems.org):
|
|
19
59
|
|
|
20
60
|
```ruby
|
|
21
|
-
gem "skiftet_statistical",
|
|
61
|
+
gem "skiftet_statistical", "~> 0.4.0"
|
|
22
62
|
```
|
|
23
63
|
|
|
24
64
|
Or build/install locally:
|
|
@@ -63,26 +103,92 @@ S.two_tailed_p_value(1.96) # => ~0.05
|
|
|
63
103
|
`two_proportion_z_test` / `welch_t_test` return `nil` when the test is undefined
|
|
64
104
|
(an empty group or zero variance), matching the existing analyzers' behaviour.
|
|
65
105
|
|
|
106
|
+
### Sample-ratio check
|
|
107
|
+
|
|
108
|
+
Does each arm's traffic fit the split its weights promise? Pearson's chi-squared
|
|
109
|
+
over the arms expected to get traffic, on k - 1 degrees of freedom. A small p
|
|
110
|
+
means the split is not doing what it says (a broken arm, a cache, a bot).
|
|
111
|
+
|
|
112
|
+
```ruby
|
|
113
|
+
S.sample_ratio_p_value(observed: { control: 700, a: 300 },
|
|
114
|
+
expected: { control: 500.0, a: 500.0 }) # => ~1.1e-36
|
|
115
|
+
S.sample_ratio_p_value(observed: ..., expected: ..., minimum_total: 500)
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
`nil` (not tested) under `minimum_total` observations (default 200) or with
|
|
119
|
+
fewer than two arms expected to get traffic; `0.0` when an arm expected to get
|
|
120
|
+
nothing got some.
|
|
121
|
+
|
|
122
|
+
## Distribution functions
|
|
123
|
+
|
|
124
|
+
Pure Ruby, Float in and out. Every function takes Integers too.
|
|
125
|
+
|
|
126
|
+
```ruby
|
|
127
|
+
S.normal_quantile(0.975) # => 1.9599639845400532 (the probit; bisection on erfc)
|
|
128
|
+
S.chi_squared_survival(3.841458820694124, 1) # => 0.05000000000000006: P(X >= statistic), X ~ chi-squared(1)
|
|
129
|
+
|
|
130
|
+
G = SkiftetStatistical::Gamma
|
|
131
|
+
G.regularized_lower(a, x) # P(a, x), the Gamma(a, 1) distribution function
|
|
132
|
+
G.regularized_upper(a, x) # Q(a, x) = 1 - P(a, x)
|
|
133
|
+
|
|
134
|
+
B = SkiftetStatistical::BetaFunction
|
|
135
|
+
B.regularized(x, alpha, beta) # I_x(alpha, beta), the Beta(alpha, beta) distribution function
|
|
136
|
+
B.quantile(0.975, 41.0, 361.0) # its inverse: an exact 95 % credible bound
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
The arithmetic and its order are part of the API: an app that pins these
|
|
140
|
+
outputs (at 1e-12, say) compares them with this exact sequence of
|
|
141
|
+
operations, so a reordered formula is a breaking change even when it is "the
|
|
142
|
+
same" formula.
|
|
143
|
+
|
|
144
|
+
## Joint Beta draws
|
|
145
|
+
|
|
146
|
+
For Bayesian A/B readouts and Thompson-sampling allocators: draw every arm's
|
|
147
|
+
Beta posterior once per draw, jointly, so "P(arm is best)" and "expected loss"
|
|
148
|
+
compare the same worlds.
|
|
149
|
+
|
|
150
|
+
```ruby
|
|
151
|
+
posteriors = [ [ 41.0, 361.0 ], [ 52.0, 350.0 ] ] # [alpha, beta] per arm
|
|
152
|
+
|
|
153
|
+
SkiftetStatistical::JointBeta.each_draw(posteriors, draws: 2_000, rng: Random.new(7)) do |row|
|
|
154
|
+
row # => one Float per arm, in order
|
|
155
|
+
end
|
|
156
|
+
|
|
157
|
+
summary = SkiftetStatistical::JointBeta.summary(posteriors, draws: 2_000, rng: Random.new(7))
|
|
158
|
+
summary[:samples] # each arm's 2 000 draws
|
|
159
|
+
summary[:wins] # draws each arm won (a tie split equally); / draws = P(best)
|
|
160
|
+
summary[:losses] # summed (best - arm) per arm; / draws = expected loss
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
**The draw order is part of the API.** Per draw, arm by arm in the order
|
|
164
|
+
given, one `Sampler#beta` each. `Sampler#gamma` takes a variable number of
|
|
165
|
+
values from the RNG, so one arm's posterior moves every later draw: a seeded
|
|
166
|
+
caller gets the same rows only while this order holds.
|
|
167
|
+
|
|
66
168
|
## Quick start (multi-armed bandit)
|
|
67
169
|
|
|
68
170
|
```ruby
|
|
69
171
|
require "skiftet_statistical"
|
|
70
172
|
|
|
71
173
|
bandit = SkiftetStatistical.bandit(
|
|
72
|
-
arms: %w[
|
|
174
|
+
arms: %w[a b c d e],
|
|
73
175
|
policy: SkiftetStatistical::Policies::ThompsonSampling.new,
|
|
74
176
|
)
|
|
75
177
|
|
|
76
|
-
choice = bandit.select # which
|
|
178
|
+
choice = bandit.select # which option to show right now, e.g. "b"
|
|
77
179
|
# ... show that option to the user ...
|
|
78
180
|
bandit.record(choice, 1) # reward: 1 = it converted, 0 = it didn't
|
|
79
181
|
|
|
80
182
|
bandit.best_arm # current best by empirical mean
|
|
81
|
-
bandit.stats # { "
|
|
183
|
+
bandit.stats # { "b" => { pulls:, mean:, reward_sum: }, ... }
|
|
82
184
|
```
|
|
83
185
|
|
|
84
|
-
Rewards are expected in **[0.0, 1.0]** — a binary `0`/`1` (e.g. "did this
|
|
85
|
-
lead to
|
|
186
|
+
Rewards are expected in **[0.0, 1.0]** — a binary `0`/`1` (e.g. "did this view
|
|
187
|
+
lead to an outcome?") is the common case, but any value in that range works.
|
|
188
|
+
|
|
189
|
+
To order several items, or to pick one by a sticky key with the odds of the
|
|
190
|
+
pick (exact up to `EXACT_PICK_MAX` contenders), use the posterior set
|
|
191
|
+
(`Bandit::Posteriors`, below).
|
|
86
192
|
|
|
87
193
|
## Policies
|
|
88
194
|
|
|
@@ -105,6 +211,12 @@ no tuning, and explores exactly as much as the evidence warrants. Cold start (no
|
|
|
105
211
|
data) is `Beta(1,1)` on every arm, i.e. uniform random, so early plays are pure
|
|
106
212
|
exploration.
|
|
107
213
|
|
|
214
|
+
The four policies are kept as 0.3.0 wrote them: in 0.4.0, `ThompsonSampling`
|
|
215
|
+
and `Softmax` included, each answers as in 0.3.0 for every input, the same arm
|
|
216
|
+
and the same use of the rng. New callers draw through `Bandit::Posteriors`
|
|
217
|
+
instead, the posterior set, which holds one set of items and ranks, picks and
|
|
218
|
+
computes odds and summaries on it.
|
|
219
|
+
|
|
108
220
|
## Persistence
|
|
109
221
|
|
|
110
222
|
A bandit's state is just its arms' counters, so it round-trips through a `Hash`
|
|
@@ -132,24 +244,1506 @@ Every stochastic policy (and the sampler) takes an `rng:`. Inject a seeded
|
|
|
132
244
|
policy = SkiftetStatistical::Policies::ThompsonSampling.new(rng: Random.new(42))
|
|
133
245
|
```
|
|
134
246
|
|
|
135
|
-
##
|
|
247
|
+
## The generic bandit (`Bandit::Posteriors`)
|
|
136
248
|
|
|
137
|
-
|
|
138
|
-
|
|
249
|
+
New in 0.4.0. One set of items, each with its posterior and its views, and every
|
|
250
|
+
operation a bandit needs on it, each written once: the order rule (`rank`, with
|
|
251
|
+
`pick` its head), the odds of what that rule draws, per-item summaries for an
|
|
252
|
+
allocation, and plain draws. The caller decides what counts as a view (an item
|
|
253
|
+
was seen) and what counts as an outcome; an outcome is yes/no (a Beta posterior
|
|
254
|
+
an item) or an amount (a Normal posterior for the mean per view). Every random
|
|
255
|
+
input is injected: a key, given uniforms or an RNG.
|
|
139
256
|
|
|
140
257
|
```ruby
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
258
|
+
P = SkiftetStatistical::Bandit::Posteriors
|
|
259
|
+
|
|
260
|
+
set = P.yes_no({ "a" => { views: 40, outcomes: 8 },
|
|
261
|
+
"b" => { views: 10, outcomes: 1 },
|
|
262
|
+
"c" => { views: 25, outcomes: 4 } })
|
|
263
|
+
set.items # => ["a", "b", "c"]
|
|
264
|
+
set.kind # => :beta
|
|
265
|
+
set.parameters # => { "a" => [9.0, 33.0], "b" => [2.0, 10.0], "c" => [5.0, 22.0] } (Beta(1, 1) prior)
|
|
266
|
+
set.views # => { "a" => 40.0, "b" => 10.0, "c" => 25.0 }
|
|
267
|
+
set.greedy # => "a", the largest posterior mean
|
|
268
|
+
|
|
269
|
+
order = set.rank(key: %w[list visit-1], shuffle: 0.1)
|
|
270
|
+
order.items # => ["a", "c", "b"], the same for this key in every process
|
|
271
|
+
order.leader # => "a"
|
|
272
|
+
order.branch # => :thompson (:shuffled for about 10 % of keys)
|
|
273
|
+
order.draws # => { "a" => 0.2907..., "b" => 0.1004..., "c" => 0.1386... }
|
|
274
|
+
|
|
275
|
+
set.pick(rng: Random.new(1)).item # => "c", the head of one Thompson draw
|
|
276
|
+
set.values(rng: Random.new(1)) # => [0.1721..., 0.0861..., 0.3054...], one draw an item
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
### Building a set
|
|
280
|
+
|
|
281
|
+
- `yes_no(items, priors: nil)`: `{ item => tally }`, a tally
|
|
282
|
+
`{ views:, outcomes: }` (Symbol or String keys, numbers >= 0, fractional
|
|
283
|
+
allowed, the outcomes clamped to the views), optionally with
|
|
284
|
+
inverse-propensity sums `weighted_views` (the sum of 1 / p),
|
|
285
|
+
`weighted_outcomes` (of y / p) and `weighted_views_squared` (of 1 / p^2):
|
|
286
|
+
the weighted outcomes are then rescaled to the Kish size, never above the
|
|
287
|
+
views, so an item that was shown more while it led is not overrated. Priors:
|
|
288
|
+
nil (Beta(1, 1)), one pair `[alpha, beta]` for every item, or
|
|
289
|
+
`{ item => pair }`.
|
|
290
|
+
- `amounts(items, priors:)`: a tally `{ views:, outcomes:, outcome_squares: }`
|
|
291
|
+
(the outcomes the sum of amounts, any sign; optionally the weighted sums and
|
|
292
|
+
`weighted_outcome_squares`) under one `Prior::Amount` or
|
|
293
|
+
`{ item => Prior::Amount }`; there is no flat amounts prior, as an amount has
|
|
294
|
+
a scale. With k prior views at mean mu0 and per-view variance v0, and n views
|
|
295
|
+
of mean m and variance s^2, the mean per view is Normal with mean
|
|
296
|
+
(k mu0 + n m) / (k + n) and variance
|
|
297
|
+
[k v0 + n s^2 + k n (m - mu0)^2 / (k + n)] / (k + n)^2, formed exactly and
|
|
298
|
+
rounded once.
|
|
299
|
+
- `beta(pairs, views: nil)` and `normal(pairs, views: nil)`:
|
|
300
|
+
`{ item => [alpha, beta] }` or `{ item => [mean, standard_deviation] }` as
|
|
301
|
+
they are, with views when given (an item missing from them reads 0).
|
|
302
|
+
|
|
303
|
+
```ruby
|
|
304
|
+
prior = SkiftetStatistical::Bandit::Prior.amount(mean: 4.0, variance: 2_000.0, strength: 20.0)
|
|
305
|
+
amounts = P.amounts({ "a" => { views: 300, outcomes: 900.0, outcome_squares: 400_000.0 },
|
|
306
|
+
"b" => { views: 200, outcomes: 500.0, outcome_squares: 300_000.0 } }, priors: prior)
|
|
307
|
+
amounts.parameters # => { "a" => [3.0625, 2.0665...], "b" => [2.6363..., 2.6457...] } ([mean, sd])
|
|
308
|
+
|
|
309
|
+
P.beta({ "a" => [3.0e6, 7.0e6] }).parameters # => { "a" => [300000.0, 700000.0] }, scaled to LARGEST_TOTAL
|
|
310
|
+
```
|
|
311
|
+
|
|
312
|
+
- Items are non-blank Strings, any number of them (an empty set ranks with no
|
|
313
|
+
head), in input order, which is the tie order everywhere. A Symbol, a blank
|
|
314
|
+
or repeated name, a name holding a NUL byte (the byte a key joins its parts
|
|
315
|
+
with) or the same bytes as an earlier name in another encoding is an
|
|
316
|
+
`ArgumentError` when the set is built; so is a Hash of priors or views that
|
|
317
|
+
names something that is not an item, or two keys naming one item.
|
|
318
|
+
- One pair everywhere. A Beta pair whose total is above `LARGEST_TOTAL` (1e6)
|
|
319
|
+
is scaled down to it, its mean kept, in `parameters`: the one pair every draw
|
|
320
|
+
(`rank`, `pick` and `values`, from any source), every odds call (exact or
|
|
321
|
+
Monte Carlo), `summary`'s P(best) and loss and `greedy` read. The pairs as
|
|
322
|
+
given feed `summary`'s means and intervals alone. A scaled pair never sums
|
|
323
|
+
above `LARGEST_TOTAL`, so a set built from another set's `parameters` keeps
|
|
324
|
+
them bit for bit.
|
|
325
|
+
- A Beta parameter above 0 but below `BetaArgmax::SMALLEST_PARAMETER` (1e-6)
|
|
326
|
+
builds: the quantile draws (a key or given uniforms), every odds call and
|
|
327
|
+
`summary` refuse it, and the RNG draws take it.
|
|
328
|
+
|
|
329
|
+
### The order rule (`rank`, `pick`)
|
|
330
|
+
|
|
331
|
+
```ruby
|
|
332
|
+
set.rank(key: "k", may_lead: %w[b c], min_views: 20) # only b or c may lead; a leader needs 20 views
|
|
333
|
+
set.rank(rng: Random.new(1), pinned: { 0 => "c" }, top: 2).items # => ["c", "a"]
|
|
334
|
+
set.rank(uniforms: { explore: 0.5, priority: { "a" => 0.1, "b" => 0.2, "c" => 0.3 },
|
|
335
|
+
draw: { "a" => 0.5, "b" => 0.5, "c" => 0.5 } }).items # => ["a", "c", "b"]
|
|
336
|
+
set.pick(key: %w[list visit-1], explore: 0.1) # => a Choice: item, head, branch, draws
|
|
337
|
+
# Order#repin: rank again with new pins under the held rule (shuffle and explore 0, no may_lead,
|
|
338
|
+
# min_views 0, the same top), on the order's own draws and the source as rank read it
|
|
339
|
+
set.rank(key: "k", explore: 0.1).repin({ 0 => "c" }) # == set.rank(key: "k", pinned: { 0 => "c" })
|
|
340
|
+
```
|
|
341
|
+
|
|
342
|
+
The rule: `shuffle:` and `explore:` (each in [0, 1], summing to at most 1),
|
|
343
|
+
`may_lead:` (the items that may lead; nil lets every free item lead),
|
|
344
|
+
`min_views:` (0), `pinned:` (`{ position => item }`, 0-based) and `top:` (nil:
|
|
345
|
+
every item). Pinned items keep their positions and draw nothing; the free items
|
|
346
|
+
fill the other positions in order. With the branch value c: below `shuffle` the
|
|
347
|
+
order is `:shuffled` (by priority); below `shuffle + explore`, when a contender
|
|
348
|
+
has at least `min_views` views, it is `:explored` (the eligible item of smallest
|
|
349
|
+
priority leads, the rest follow by draw); otherwise it is `:thompson` (the
|
|
350
|
+
largest draw leads, the rest follow by draw). Ties go to the earlier item.
|
|
351
|
+
|
|
352
|
+
`rank` answers an `Order`: `items` (the first `top`, pinned items at their
|
|
353
|
+
positions), `leader` (the free head when it has at least `min_views` views,
|
|
354
|
+
else nil), `branch` and `draws` (each free item's posterior value; none on the
|
|
355
|
+
shuffled branch). `pick` is `rank` with top 1 and no pins, answering a `Choice`:
|
|
356
|
+
`item` (the leader), `head`, `branch` and `draws`. Both are frozen; `==`
|
|
357
|
+
compares those four members.
|
|
358
|
+
|
|
359
|
+
Exactly one source of randomness:
|
|
360
|
+
|
|
361
|
+
- `key:` (a String, or an Array of parts): the branch value is
|
|
362
|
+
`Draw.uniform_for(*key, "explore")`, an item's priority
|
|
363
|
+
`uniform_for(*key, "priority", item)` and its draw
|
|
364
|
+
`uniform_for(*key, "draw", item)`. The same key draws the same order in every
|
|
365
|
+
process with nothing stored, and no item's uniforms depend on another item:
|
|
366
|
+
over 1 000 keys with `shuffle: 0.1` and no exclusion or pin, an added item
|
|
367
|
+
never changed the others' relative order (with an item pinned at position 1
|
|
368
|
+
it did on 352 of them, as the added item takes a free slot ahead of the pin).
|
|
369
|
+
- `uniforms:` (`{ explore:, priority: { item => Float }, draw: { item => Float } }`):
|
|
370
|
+
the same three, given.
|
|
371
|
+
- `rng:` (anything that answers `rand`): one branch value when
|
|
372
|
+
`shuffle + explore` is above 0, one priority a free item on the shuffled or
|
|
373
|
+
explored branch, then one `Sampler` draw a free item where the branch draws,
|
|
374
|
+
in item order, and nothing else from the stream.
|
|
375
|
+
|
|
376
|
+
A key or given uniforms draw by quantile: a Beta item at
|
|
377
|
+
`BetaArgmax.position(u, alpha, beta)`, compared by that position so that two
|
|
378
|
+
draws never tie by rounding, and a Normal item at mean + sd times
|
|
379
|
+
`Significance.normal_quantile(u)`, u on the open grid (`Draw.open_uniform`). An
|
|
380
|
+
RNG draws by `Sampler`. Two Normal draws are compared, from every source, by the
|
|
381
|
+
exact sign of (mean_i - mean_j) + (sd_i z_i - sd_j z_j), each mean + sd * z
|
|
382
|
+
formed exactly, never rounded, so they never tie or misorder where a standard
|
|
383
|
+
deviation lies below a rounding of its mean; `draws` answers the rounded values.
|
|
384
|
+
Each source draws exactly from the posteriors, so one set of odds (below) covers
|
|
385
|
+
all three.
|
|
386
|
+
|
|
387
|
+
Refused, each an `ArgumentError` before any draw and before anything is taken
|
|
388
|
+
from an RNG: a rule outside its bounds, a pinned position or name that is not an
|
|
389
|
+
item, a name pinned twice, a `may_lead` name that is not an item, `top` below 1,
|
|
390
|
+
no source or more than one, a malformed `uniforms` Hash, and under a key or
|
|
391
|
+
given uniforms a Beta parameter below 1e-6. An RNG's own answer is checked as
|
|
392
|
+
it comes: one that is not a Float in [0, 1), or a raise from its `rand`, is an
|
|
393
|
+
`ArgumentError` at that answer, after the draws taken before it.
|
|
394
|
+
|
|
395
|
+
`greedy(may_lead: nil)` is the contender with the largest posterior mean on
|
|
396
|
+
`parameters` (`BetaArgmax.greedy`'s rule, for a set of any size); `values(rng:)`
|
|
397
|
+
is one posterior draw an item, in item order.
|
|
398
|
+
|
|
399
|
+
### Odds of a pick, an order and a position
|
|
400
|
+
|
|
401
|
+
The posterior set answers the probability of what its rule draws, under that
|
|
402
|
+
same rule (`shuffle`, `explore`, `may_lead`, `min_views`, and for orders
|
|
403
|
+
`pinned` and `top`):
|
|
404
|
+
|
|
405
|
+
```ruby
|
|
406
|
+
set = SkiftetStatistical::Bandit::Posteriors.beta({ "a" => [ 1.0, 1.0 ], "b" => [ 2.0, 1.0 ] })
|
|
407
|
+
set.pick_probabilities(explore: 0.1)[:heads] # => { "a" => 0.35000000000000003, "b" => 0.6500000000000001 }
|
|
408
|
+
set.order_probability(%w[b a]) # => { probability: 0.6666..., method: :exact, standard_error: 0.0 }
|
|
409
|
+
set.positions[:probabilities] # => { "a" => [0.3333..., 0.6666...], "b" => [0.6666..., 0.3333...] }
|
|
410
|
+
set.order_summary(%w[b a]) # => { probability: 0.6666..., positions: { "a" => [...], "b" => [...] } }
|
|
411
|
+
set.rank(key: "k1").probability # => 0.6666..., the odds of the order that key drew
|
|
412
|
+
|
|
413
|
+
SkiftetStatistical::Bandit::Posteriors.mixture(
|
|
414
|
+
shares: { "x" => 0.25, "y" => 0.75 },
|
|
415
|
+
probabilities: { "x" => { "a" => 0.5, "b" => 0.5 }, "y" => { "a" => 0.2, "c" => 0.8 } },
|
|
416
|
+
) # => { "a" => 0.275, "b" => 0.125, "c" => 0.6000000000000001 }
|
|
417
|
+
```
|
|
418
|
+
|
|
419
|
+
- `pick_probabilities` gives each item's probability of heading the pick
|
|
420
|
+
(`heads`) and of being the pick's leader (`leaders`, nil for no leader);
|
|
421
|
+
`order_probability(order)` the probability that the rule's whole order
|
|
422
|
+
begins with `order`; `positions` each item's probability at each slot;
|
|
423
|
+
`order_summary(order)` the posterior alone, with no rule: the probability
|
|
424
|
+
that `order` is the best one and each item's probability at each position,
|
|
425
|
+
for a set of at most `EXACT_ORDER_MAX` (10) items. `Choice#probability` and
|
|
426
|
+
`Order#probability` answer the same odds for a pick or an order the set made,
|
|
427
|
+
computed on the first call and kept.
|
|
428
|
+
- `Posteriors.mixture(shares:, probabilities:)` is the probability that a
|
|
429
|
+
two-layer draw shows each item: the sum over the first layer's branches of a
|
|
430
|
+
branch's share times its odds of the item.
|
|
431
|
+
- They are exact, by quadrature, up to `EXACT_PICK_MAX` (20) contenders for the
|
|
432
|
+
pick and up to `EXACT_ORDER_MAX` (10) free items for orders and positions
|
|
433
|
+
when no free item is barred from leading. The pick's P(largest) is
|
|
434
|
+
`BetaArgmax.probabilities` or `NormalArgmax.probabilities`. The order odds
|
|
435
|
+
sum on every free item's quantiles at 400 levels (an internal grid, not a
|
|
436
|
+
public constant), each sum on points where its own item's points keep their
|
|
437
|
+
digits: Betas share one grid in the position on a log scale the rule
|
|
438
|
+
compares, and each Normal sums on its own points in its own standardised
|
|
439
|
+
variable, the other items read there, so a
|
|
440
|
+
parameter near 1e-6, a standard deviation small beside its mean, or two
|
|
441
|
+
narrow items beside a far one loses no point; each sum is extrapolated from
|
|
442
|
+
the points and their every other level. They read within 4e-6 of closed
|
|
443
|
+
forms, inside the library's stated 1e-4.
|
|
444
|
+
- Otherwise each is one of two stated Monte Carlo forms, `draws:` (20 000)
|
|
445
|
+
draws on `Random.new(seed:)` (seed 0), so the same call answers the same
|
|
446
|
+
figures, with `method: :monte_carlo` and the largest standard error: above
|
|
447
|
+
`EXACT_PICK_MAX` contenders the pick counts how often each contender's draw
|
|
448
|
+
is the largest; orders and positions (an item barred from leading, or more
|
|
449
|
+
than `EXACT_ORDER_MAX` free items) count the rule's own visits. Both draw on
|
|
450
|
+
the scaled pairs (`parameters`), the pairs the exact odds read. A Beta
|
|
451
|
+
parameter far below 1 draws 0.0 in a share of `Sampler`'s draws (about a
|
|
452
|
+
tenth at 0.003), where two such draws tie and the tie goes to the earlier
|
|
453
|
+
item, so below about 0.003 the Monte Carlo reads that tie rule as well.
|
|
454
|
+
- A Beta parameter below `BetaArgmax::SMALLEST_PARAMETER` (1e-6) is an
|
|
455
|
+
`ArgumentError` for every odds call at every size, exact or Monte Carlo,
|
|
456
|
+
over the items a keyed draw of the same set and rule refuses.
|
|
457
|
+
|
|
458
|
+
Measured against the rule's own visits in `odds_simulation_spec.rb` (below):
|
|
459
|
+
every head and position within 2.75 standard errors of its exact value over
|
|
460
|
+
2 * 10^5 visits a fixture.
|
|
461
|
+
|
|
462
|
+
### Summaries (`summary`)
|
|
463
|
+
|
|
464
|
+
```ruby
|
|
465
|
+
set = SkiftetStatistical::Bandit::Posteriors.beta({ "a" => [ 2.0, 8.0 ], "b" => [ 3.0, 7.0 ] },
|
|
466
|
+
views: { "a" => 120.0, "b" => 340.0 })
|
|
467
|
+
summary = set.summary(credible_mass: 0.95)
|
|
468
|
+
# => { "a" => { views: 120.0, mean: 0.2, lower: 0.0281..., upper: 0.4825..., p_best: 0.2882...,
|
|
469
|
+
# expected_loss: 0.1335... },
|
|
470
|
+
# "b" => { views: 340.0, mean: 0.3, lower: 0.0748..., upper: 0.6000..., p_best: 0.7117...,
|
|
471
|
+
# expected_loss: 0.0335... } }
|
|
472
|
+
SkiftetStatistical::Bandit::Allocator.from_summary(arms: %w[a b], summary: summary,
|
|
473
|
+
exposures: { "a" => 120, "b" => 340 },
|
|
474
|
+
status: "running", winner_arm: nil)
|
|
475
|
+
set.summary(among: []) # views, mean and interval only, for a set of any size
|
|
476
|
+
set.means # => { "a" => 0.2, "b" => 0.3 }, the means alone: no interval, no quantile
|
|
477
|
+
```
|
|
478
|
+
|
|
479
|
+
`summary(credible_mass: 0.95, among: nil)` gives every item, in item order, its
|
|
480
|
+
views, its posterior mean and its equal-tailed interval, read on the pair as
|
|
481
|
+
given, and for the items `among` names (nil: every item; at most
|
|
482
|
+
`EXACT_PICK_MAX`) the exact P(best) and expected loss, E[the largest] less the
|
|
483
|
+
item's mean, among those items, on `parameters` (`BetaArgmax.probabilities` and
|
|
484
|
+
`.expected_maximum`, or `NormalArgmax`'s). A loss that rounds below 0 is
|
|
485
|
+
answered as 0. The answer is `Allocator.from_summary`'s `summary:` as it is, so
|
|
486
|
+
a pass on exact figures is one call more. `among: []` computes no P(best) and no
|
|
487
|
+
loss; `means` answers the means alone (frozen, summary's mean bit for bit) and
|
|
488
|
+
computes no interval, so it calls no quantile. A Beta interval is
|
|
489
|
+
`BetaFunction.quantile`, accurate to about 2e-10 up to totals (alpha + beta)
|
|
490
|
+
near 10^8; beyond about 3 * 10^8 one bound can fall on or past the mean (at a
|
|
491
|
+
mean of 1/2 the lower bound lies 0.0025 standard deviations above it at
|
|
492
|
+
4 * 10^8). A Normal interval is the mean less and plus its standard deviation
|
|
493
|
+
times the Normal quantile, the tail read on the open grid.
|
|
494
|
+
|
|
495
|
+
### Priors (`Bandit::Prior`)
|
|
496
|
+
|
|
497
|
+
A yes/no prior is a Beta pair `[alpha, beta]`; an amounts prior a
|
|
498
|
+
`Prior::Amount` (a mean per view, a variance per view and a strength in views).
|
|
499
|
+
Three rules are written once, and every other prior is built from them: a mean
|
|
500
|
+
held at a strength (`given`), a mean shrunk toward a prior mean (`pooled_mean`)
|
|
501
|
+
and the leave-one-out rule (`borrowed`: what a group holds once the item's own
|
|
502
|
+
evidence is taken out, views at least 0 and outcomes within [0, views]).
|
|
503
|
+
|
|
504
|
+
```ruby
|
|
505
|
+
PR = SkiftetStatistical::Bandit::Prior
|
|
506
|
+
|
|
507
|
+
PR.flat # => [1.0, 1.0]
|
|
508
|
+
PR.given(mean: 0.1, strength: 20.0) # => [2.0, 18.0]
|
|
509
|
+
PR.amount(mean: 4.0, variance: 2_000.0, strength: 20.0)
|
|
510
|
+
PR.borrowed(group: { views: 1_000, outcomes: 100 }, own: { views: 200, outcomes: 50 })
|
|
511
|
+
# => { views: 800.0, outcomes: 50.0 }
|
|
512
|
+
PR.pooled(group: { views: 1_000, outcomes: 100 }, own: { views: 200, outcomes: 50 }, strength: 20.0)
|
|
513
|
+
# => [1.399..., 20.600...]: the borrowed mean (1 + 50) / (2 + 800) held at 2 + 20 views
|
|
514
|
+
PR.pooled_mean(views: 40, outcomes: 8, prior_mean: 0.1, strength: 20.0) # => 0.1666...
|
|
515
|
+
PR.tilts({ "a" => 2.0, "b" => 0.5, "c" => 1.0 }, clip: [ 0.5, 1.5 ]) # => { "a" => 1.5, "b" => 0.5, "c" => 1.0 }
|
|
516
|
+
PR.chain(levels: [ { views: 1_000, outcomes: 120, strength: 20.0 },
|
|
517
|
+
{ views: 200, outcomes: 30, strength: 30.0 } ], root_mean: 0.1)
|
|
518
|
+
# => [3.365..., 26.634...]: the top level's mean, the lower level taken out, held at its 30 views
|
|
519
|
+
```
|
|
520
|
+
|
|
521
|
+
- `pooled(group:, own:, strength:, base: FLAT)`: the borrowed mean
|
|
522
|
+
(base_alpha + y) / (base_alpha + base_beta + n) held at
|
|
523
|
+
base_alpha + base_beta + min(strength, n), so a large group never outweighs
|
|
524
|
+
the item's own evidence for long.
|
|
525
|
+
- `pooled_new(others:, group:, own:, weight: 2.0)`: a new item's prior, the
|
|
526
|
+
others' borrowed evidence pooled into (1 + Y) / (2 + N) at `weight`, plus the
|
|
527
|
+
new item's own borrowed evidence.
|
|
528
|
+
- `tilts(multipliers, clip:)`: each multiplier over the geometric mean of all
|
|
529
|
+
of them, clamped to `clip`, so tilts reorder items without raising them all.
|
|
530
|
+
- `chain(levels:, root_mean:, tilt: 1.0, clip: [0.001, 0.999])`: the prior of
|
|
531
|
+
an item at the bottom of nested levels, top first, each level's mean shrunk
|
|
532
|
+
toward the one above with the level below taken out.
|
|
533
|
+
- `pooled_amount(group:, own:, strength:)`: the borrowed mean and variance of
|
|
534
|
+
amounts at min(strength, n); it refuses a group with nothing left out or a
|
|
535
|
+
variance of 0, and the caller then states its own scale with `amount`.
|
|
536
|
+
|
|
537
|
+
### Decay (`Bandit::Decay`)
|
|
538
|
+
|
|
539
|
+
```ruby
|
|
540
|
+
SkiftetStatistical::Bandit::Decay.weight(72, half_life: 72, cap: 336) # => 0.5
|
|
541
|
+
SkiftetStatistical::Bandit::Decay.weight(337, half_life: 72, cap: 336) # => 0.0
|
|
542
|
+
rows = [ { age: 0, views: 10, outcomes: 3 }, { age: 72, views: 20, outcomes: 4 } ]
|
|
543
|
+
SkiftetStatistical::Bandit::Decay.fold(rows, half_life: 72, cap: 336)
|
|
544
|
+
# => { views: 20.0, outcomes: 5.0, outcome_squares: 0.0 }
|
|
545
|
+
```
|
|
546
|
+
|
|
547
|
+
One formula, 2 ** (-age / half_life), and 0.0 for an age above the cap; age,
|
|
548
|
+
half-life and cap share one unit of the caller's choosing. `fold` weighs every
|
|
549
|
+
sum of a row (views, outcomes and outcome squares alike), so the folded tally is
|
|
550
|
+
what a posterior reads. The fold is linear, so a caller that sums in a database
|
|
551
|
+
groups its rows by age bucket and folds the buckets.
|
|
552
|
+
|
|
553
|
+
### Keyed uniforms and the draw by weight (`Bandit::Draw`)
|
|
554
|
+
|
|
555
|
+
```ruby
|
|
556
|
+
D = SkiftetStatistical::Bandit::Draw
|
|
557
|
+
|
|
558
|
+
D.uniform("t") # => 0.8895498694938413
|
|
559
|
+
D.uniform_for("t", "explore") # => 0.856909028643178
|
|
560
|
+
D.seed_for("t", "explore") # => 15807181545831752473, for Random.new
|
|
561
|
+
D.weighted({ "a" => 1, "b" => 3 }, 0.5) # => ["b", 0.75]
|
|
562
|
+
D.from_weights({ "a" => 1, "b" => 3 }, 0.5) # => "b"
|
|
563
|
+
D.open_uniform(0.0) # => 1.1102230246251565e-16 (2**-53)
|
|
564
|
+
```
|
|
565
|
+
|
|
566
|
+
A uniform is the first 8 bytes of SHA256 of the text, big-endian, shifted right
|
|
567
|
+
by 11 and divided by 2**53: the same text gives the same uniform in every
|
|
568
|
+
process, so a draw keyed by a visit, a pass or a test comes back without being
|
|
569
|
+
stored. `uniform_for` joins its parts by one NUL byte (nil reads "", a part
|
|
570
|
+
holding a NUL byte is refused); `seed_for` is those 8 bytes as an Integer.
|
|
571
|
+
`weighted` is the one draw by weight: a weight that is not a finite number above
|
|
572
|
+
0 counts 0, all 0 counts every key 1, and it answers the key with its share, the
|
|
573
|
+
probability that a uniform draw answers it; `from_weights` is its key. Keys are
|
|
574
|
+
any objects, answered as given. `open_uniform` clamps a uniform to
|
|
575
|
+
[2**-53, 1 - 2**-53], the one reading of a quantile draw's uniform.
|
|
576
|
+
|
|
577
|
+
### Normal argmax (`Bandit::NormalArgmax`)
|
|
578
|
+
|
|
579
|
+
```ruby
|
|
580
|
+
NA = SkiftetStatistical::Bandit::NormalArgmax
|
|
581
|
+
|
|
582
|
+
NA.probabilities([ [ 0.0, 1.0 ], [ 0.0, 1.0 ] ]) # => [0.5, 0.5]
|
|
583
|
+
NA.probabilities([ [ 5.0, 0.4 ], [ 5.3, 0.5 ] ], epsilon: 0.1) # => [0.3377..., 0.6622...]
|
|
584
|
+
NA.expected_maximum([ [ 0.0, 1.0 ], [ 0.0, 1.0 ] ]) # => 0.5641... (1 / sqrt(pi))
|
|
585
|
+
```
|
|
586
|
+
|
|
587
|
+
`BetaArgmax`'s rule on independent Normals `[mean, standard_deviation]`, 1 to 20
|
|
588
|
+
members: the probability that each is the largest, mixed with epsilon / k, and
|
|
589
|
+
E[the largest]. Each member's probability is its own integral in its own
|
|
590
|
+
standardised variable, so a narrow member far from the others is weighed on its
|
|
591
|
+
own scale; each answer lies within 2 k (k + 1) * 1e-10 + 2e-12 of the exact
|
|
592
|
+
probability for k members (1.2e-9 for two, 8.4e-8 for 20).
|
|
593
|
+
|
|
594
|
+
### A pass with a hold on its win (`Bandit::Pass`)
|
|
595
|
+
|
|
596
|
+
```ruby
|
|
597
|
+
hold = SkiftetStatistical::Bandit::WinHold.new(leading_arm: "a", leading_since: now - 90_000, now: now,
|
|
598
|
+
last_pass_at: now - 600)
|
|
599
|
+
SkiftetStatistical::Bandit::Pass.call(status: "running", hold: hold) do |hold_satisfied, rng|
|
|
600
|
+
SkiftetStatistical::Bandit::Allocator.call(arms: %i[control a], tallies: tallies, status: "running",
|
|
601
|
+
winner_arm: nil, previous_weights: weights, rng: rng,
|
|
602
|
+
draws: 20_000, hold_satisfied: hold_satisfied)
|
|
603
|
+
end
|
|
604
|
+
# => the crown's answer (phase: :won, winner: :a, leader: :a, ...) and ready: true
|
|
605
|
+
```
|
|
606
|
+
|
|
607
|
+
The probe yields (false, `Random.new(seed)`), so a leader gets the step, never
|
|
608
|
+
the won split. The pass is ready when the status reads "running", the probe
|
|
609
|
+
names a leader and `hold.satisfied_for?(leader)` answers true; then, and only
|
|
610
|
+
when `crown:` is true (the default), the block runs again with (true, a fresh
|
|
611
|
+
`Random.new(seed)`), so the probe and the crown read the same draws and only the
|
|
612
|
+
decision changes. The answer is the last block's Hash plus `ready:`; with
|
|
613
|
+
`crown: false` a ready leader keeps the probe's answer and `ready: true` says it
|
|
614
|
+
would have been crowned. `seed:` defaults to `Random.new_seed`; a given seed
|
|
615
|
+
makes the pass repeatable.
|
|
616
|
+
|
|
617
|
+
### Amounts: what the Normal posterior gets wrong
|
|
618
|
+
|
|
619
|
+
Under about 30 non-zero amounts the mean of amounts is right-skewed and the
|
|
620
|
+
Normal posterior understates its upside; the prior's k views at v0 keep it
|
|
621
|
+
wide, and the shuffle or exploration share keeps the item shown. One large
|
|
622
|
+
amount x moves a mean of n views by x / n, so cap amounts before summing them;
|
|
623
|
+
`min_views` holds the lead until an item has enough views. Draws and interval
|
|
624
|
+
bounds may fall below 0: they are answered as computed, and the order rule
|
|
625
|
+
compares draws only. When `Prior.pooled_amount` refuses (nothing borrowed, or a
|
|
626
|
+
variance of 0), state a scale with `Prior.amount`. Measured in
|
|
627
|
+
`order_simulation_spec.rb` (below): on amounts at 300, 1 000 and 3 000 visits a
|
|
628
|
+
day the best item was on top on day 28 in every run, and the rule lost 4.3, 2.6
|
|
629
|
+
and 1.9 % of the best order's expected outcomes.
|
|
630
|
+
|
|
631
|
+
### Inputs
|
|
632
|
+
|
|
633
|
+
Every number the code new in 0.4.0 reads must lie in one input domain: a real
|
|
634
|
+
number whose Float is finite and at most `SkiftetStatistical::Numbers::LARGEST`
|
|
635
|
+
(1e100) either side of 0. A larger number, a NaN, an infinity or a String is an
|
|
636
|
+
`ArgumentError` where it is read, so no sum or product of inputs can leave a
|
|
637
|
+
Float on its way to an answer, and every answer is checked finite before it is
|
|
638
|
+
returned. A caller's object is called only where it must be (a key part's
|
|
639
|
+
`to_s`, a name's `hash` or `eql?`, a list's `each`, an RNG's `rand`); what it
|
|
640
|
+
raises there is an `ArgumentError` naming the input. A caller's String, Hash or
|
|
641
|
+
Array is read by its contents, never by its own methods, so a missing key reads
|
|
642
|
+
as absent whatever default the Hash holds. A list is any Enumerable but a Hash,
|
|
643
|
+
read once by its `each`.
|
|
644
|
+
|
|
645
|
+
### Costs
|
|
646
|
+
|
|
647
|
+
CPU time a call with YJIT, Ruby 3.4.9, on one development machine on
|
|
648
|
+
2026-11-02 while other work kept it busy (CPU time moves less with the load than
|
|
649
|
+
the wall clock does):
|
|
650
|
+
|
|
651
|
+
| Call | Time |
|
|
652
|
+
|---|---|
|
|
653
|
+
| `Posteriors.yes_no`, 8 items | 0.055 ms |
|
|
654
|
+
| `Posteriors.amounts`, 2 items | 0.034 ms |
|
|
655
|
+
| `rank(key:, explore: 0.1)`, 8 Beta items, totals up to 1e3 | 0.21 ms |
|
|
656
|
+
| the same, totals up to 1e5 | 0.54 ms |
|
|
657
|
+
| the same, 8 Normal items | 0.08 ms |
|
|
658
|
+
| `rank(rng:, shuffle: 0.1)`, 8 Beta or Normal items | 0.017 ms |
|
|
659
|
+
| `BetaArgmax.position`, Beta(3, 300) / Beta(41, 960) / Beta(4 000, 96 000) | 0.019 / 0.028 / 0.057 ms (0.3.0's bisection: 0.11 / 0.29 / 0.92 ms) |
|
|
660
|
+
| `pick_probabilities(explore: 0.1)`, 5 Betas / 5 Normals | 24 / 1.6 ms |
|
|
661
|
+
| `summary`, 5 Betas / 20 Betas / 5 Normals | 46 / 440 / 3.8 ms |
|
|
662
|
+
| `summary(among: [])`, 60 Betas | 26 ms |
|
|
663
|
+
| `NormalArgmax.probabilities`, 20 members | 91 ms |
|
|
664
|
+
| `order_probability` (exact), 5 Betas / 5 Normals | 145 / 14 ms |
|
|
665
|
+
| `positions` (exact), 5 / 10 Betas | 160 / 560 ms |
|
|
666
|
+
| `positions(explore: 0.1)` (exact), 10 Betas | 1.2 s |
|
|
667
|
+
| `order_probability` (Monte Carlo, 20 000 visits), 5 Betas | 200 ms |
|
|
668
|
+
| `pick_probabilities` (Monte Carlo, 20 000 draws), 21 Betas | 470 ms |
|
|
669
|
+
|
|
670
|
+
A draw is cheap enough for every visit; the odds and summaries belong off the
|
|
671
|
+
request path, once a pass (a set keeps the grids of its exact orders and
|
|
672
|
+
positions, so later odds calls on the same set cost less).
|
|
673
|
+
|
|
674
|
+
## Example: an ordered list, one visit at a time
|
|
675
|
+
|
|
676
|
+
```ruby
|
|
677
|
+
P = SkiftetStatistical::Bandit::Posteriors
|
|
678
|
+
|
|
679
|
+
# Once a pass (hourly, say): the posterior set from stored tallies, where a view
|
|
680
|
+
# counts only for an item the visit reached, kept as its pairs.
|
|
681
|
+
set = P.yes_no(tallies) # { "a" => { views:, outcomes: }, ... }
|
|
682
|
+
cache.write("list:pairs", { pairs: set.parameters, views: set.views })
|
|
683
|
+
|
|
684
|
+
# Each visit: the set rebuilt from its pairs (bit for bit) and ranked by the
|
|
685
|
+
# visit's key, so a visit sees the same order on every request and nothing is
|
|
686
|
+
# stored per visit.
|
|
687
|
+
kept = cache.read("list:pairs")
|
|
688
|
+
order = P.beta(kept[:pairs], views: kept[:views]).rank(key: [ "list", visit_id ], shuffle: 0.1)
|
|
689
|
+
order.items # show the items in this order
|
|
690
|
+
|
|
691
|
+
# Off the request path, once a pass: each item's odds at each position under
|
|
692
|
+
# that rule, for an estimate that weighs a view by 1 / its probability.
|
|
693
|
+
set.positions(shuffle: 0.1)[:probabilities]
|
|
694
|
+
```
|
|
695
|
+
|
|
696
|
+
## A/B-test allocation (`Bandit::Allocator`)
|
|
697
|
+
|
|
698
|
+
The decision an A/B test's allocation job makes on every pass: how much traffic
|
|
699
|
+
each arm gets next, and whether an arm has won. It is the rule an app's
|
|
700
|
+
allocation job ran, moved here; pure (no clock, no state), with the randomness only from the injected `rng`.
|
|
701
|
+
|
|
702
|
+
```ruby
|
|
703
|
+
A = SkiftetStatistical::Bandit::Allocator
|
|
704
|
+
|
|
705
|
+
result = A.call(
|
|
706
|
+
arms: %i[control a],
|
|
707
|
+
tallies: { control: { exposures: 1_000, conversions: 500 },
|
|
708
|
+
a: { exposures: 1_000, conversions: 533 } },
|
|
709
|
+
status: "running", winner_arm: nil,
|
|
710
|
+
previous_weights: { control: 0.5, a: 0.5 }, # the latest allocation, or nil
|
|
711
|
+
rng: Random.new(7), draws: 20_000,
|
|
712
|
+
)
|
|
713
|
+
result[:weights] # => { control: 0.35, a: 0.65 } (next pass's split)
|
|
714
|
+
result[:phase] # => :shifting (:warmup, :shifting, :won or :paused)
|
|
715
|
+
result[:stats][:a] # => { exposures: 1000, conversions: 533, rate: 0.533, weighted_rate: 0.533,
|
|
716
|
+
# lower: 0.502..., upper: 0.563..., p_best: 0.93175, expected_loss: 0.00066...,
|
|
717
|
+
# guardrails: {} }
|
|
718
|
+
result[:winner] # => nil (the crowned arm)
|
|
719
|
+
result[:leader] # => nil (the arm meeting the win rule at this pass)
|
|
720
|
+
result[:result_changed] # => false (a won test whose winner fell below P(best) 0.5)
|
|
721
|
+
```
|
|
722
|
+
|
|
723
|
+
The rule, in order:
|
|
724
|
+
|
|
725
|
+
1. **Paused** (`status: "paused"`): the previous weights unchanged (equal
|
|
726
|
+
weights without them).
|
|
727
|
+
2. **Won** (`status: "won"` and `winner_arm` one of the arms): the won split,
|
|
728
|
+
95 % to the winner and 5 % shared by the others. `result_changed` flags a
|
|
729
|
+
winner whose P(best) has fallen below 0.5; a won test never un-wins.
|
|
730
|
+
3. **Warm-up**, while any arm has fewer than 300 exposures: equal weights.
|
|
731
|
+
4. **The automatic win**: the first arm (in the order given) with P(best)
|
|
732
|
+
>= 0.95 and expected loss <= 0.0025 is the `leader`. It gets the won split
|
|
733
|
+
when `hold_satisfied:` is true (the default); with false (the hold below is
|
|
734
|
+
still running) it gets step 5's weights.
|
|
735
|
+
5. **Shifting**: the target is each arm's P(best), every arm at least 10 %
|
|
736
|
+
(the others scaled down in proportion), reached by a convex step from the
|
|
737
|
+
previous weights in which no arm moves more than 0.15.
|
|
738
|
+
|
|
739
|
+
P(best) and expected loss come from `draws` joint draws of the arms' Beta
|
|
740
|
+
posteriors (`JointBeta`); the interval is the 95 % equal-tailed one of the
|
|
741
|
+
arm's draws. **The draw order is part of the API**: with the same input and
|
|
742
|
+
seed the result is the same only while no draw is added, dropped or reordered.
|
|
743
|
+
|
|
744
|
+
A tally may carry inverse-propensity sums, `weighted_exposures` and
|
|
745
|
+
`weighted_conversions` (each exposure counted as 1 / the arm's share when it
|
|
746
|
+
was drawn) and `weighted_exposures_squared`: the posterior is then built on
|
|
747
|
+
the weighted conversions rescaled to the Kish effective sample size, never
|
|
748
|
+
above the raw count (`WeightedPosterior`), so an arm whose lead comes from a
|
|
749
|
+
traffic surge it was favoured in is not crowned. A tally may also carry
|
|
750
|
+
`guardrails: { goal => conversions }`, reported per arm with its raw rate and
|
|
751
|
+
never decided on. Tallies and previous weights may use Symbol or String keys
|
|
752
|
+
(rows read back from JSON); previous weights that do not name exactly the arms
|
|
753
|
+
with finite numbers read as none.
|
|
754
|
+
|
|
755
|
+
Every number above is a setting of `Allocator::Config`, with the defaults
|
|
756
|
+
below:
|
|
757
|
+
|
|
758
|
+
```ruby
|
|
759
|
+
config = A::Config.new(warmup_exposures: 100, floor: 0.05) # the rest stay
|
|
760
|
+
config.max_arms # => 20 (1 / floor)
|
|
761
|
+
A.call(..., config: config)
|
|
762
|
+
A.won_weights(%i[control a b], :a, config: config) # => { control: 0.025, a: 0.95, b: 0.025 }
|
|
763
|
+
```
|
|
764
|
+
|
|
765
|
+
Settings: `warmup_exposures` (300), `floor` (0.10), `max_step` (0.15),
|
|
766
|
+
`win_p_best` (0.95), `win_loss` (0.0025), `won_share` (0.95), `won_rest`
|
|
767
|
+
(0.05), `result_changed_below` (0.5), `credible_mass` (0.95), `prior_alpha`
|
|
768
|
+
and `prior_beta` (1.0). More arms than `max_arms` raise `ArgumentError`, and
|
|
769
|
+
`allocation: :fixed` (a split that is measured, not allocated) raises
|
|
770
|
+
`A::FixedAllocation`, an `ArgumentError`, before anything is computed.
|
|
771
|
+
|
|
772
|
+
### Per-arm floors and fixed shares (`arm_floors:`, `fixed_shares:`)
|
|
773
|
+
|
|
774
|
+
Both keywords are nil by default, and then nothing new is checked and every
|
|
775
|
+
output is the rule above, bit for bit. `arm_floors: { arm => floor }` gives an
|
|
776
|
+
arm its own floor, a number in (0, 1], in place of `Config#floor` (an arm it
|
|
777
|
+
does not name keeps `Config#floor`); `fixed_shares: { arm => share }` holds an
|
|
778
|
+
arm at exactly its share, a number in (0, 1), in every phase.
|
|
779
|
+
|
|
780
|
+
```ruby
|
|
781
|
+
warm = { control: { exposures: 120, conversions: 30 }, a: { exposures: 110, conversions: 25 },
|
|
782
|
+
b: { exposures: 100, conversions: 31 } } # still in warm-up
|
|
783
|
+
inputs = { arms: %i[control a b], tallies: warm, status: "running", winner_arm: nil,
|
|
784
|
+
rng: Random.new(7), draws: 2_000 }
|
|
785
|
+
A.call(**inputs, arm_floors: { control: 0.5 })[:weights] # => { control: 0.5, a: 0.25, b: 0.25 }
|
|
786
|
+
A.call(**inputs, fixed_shares: { control: 0.2 })[:weights] # => { control: 0.2, a: 0.4, b: 0.4 }
|
|
787
|
+
```
|
|
788
|
+
|
|
789
|
+
- The keys are arms of the call (Symbols or Strings); no arm in both; the fixed
|
|
790
|
+
shares and the other arms' floors sum to at most 1; every arm fixed raises
|
|
791
|
+
`FixedAllocation`. Each refusal is an `ArgumentError`, raised before any draw.
|
|
792
|
+
- In every phase each fixed arm gets exactly its share, and the free arms (the
|
|
793
|
+
others, in the order given) share the rest, `1 - F` with `F` the sum of the
|
|
794
|
+
fixed shares, each at least its floor.
|
|
795
|
+
- One lift serves every phase: a target over the free arms is scaled into the
|
|
796
|
+
free mass, every arm under its floor is pinned there, and the others are
|
|
797
|
+
scaled again into what is left, until none is under. With no fixed share and
|
|
798
|
+
every floor `Config#floor` it is the shifting phase's floor above, public as
|
|
799
|
+
`Allocator.lift`.
|
|
800
|
+
- Paused: the previous weights while they keep every floor and share, else the
|
|
801
|
+
free arms' previous shares through the lift (a pause on
|
|
802
|
+
`{ control: 0.02, a: 0.49, b: 0.49 }` with `arm_floors: { control: 0.05 }`
|
|
803
|
+
answers `{ control: 0.05, a: 0.475, b: 0.475 }`). Won: the won split over the
|
|
804
|
+
free arms; a fixed arm is never the won arm, so a stored win of one raises.
|
|
805
|
+
Warm-up: equal targets over the free arms. Shifting: the free arms' P(best),
|
|
806
|
+
reached by the step over the free arms and clamped onto their floors.
|
|
807
|
+
- With fixed shares, P(best) and expected loss are taken among the free arms in
|
|
808
|
+
each joint draw (every arm is still drawn, in order), so a fixed arm's are
|
|
809
|
+
nil. A lone free arm beside fixed arms can lead and win; with no share
|
|
810
|
+
declared the 0.2.0 guard stays, and a single arm never leads.
|
|
811
|
+
|
|
812
|
+
Measured (`value_allocation_simulation_spec.rb`, below): no share below its
|
|
813
|
+
floor or off its fixed share in 2,800 passes through every phase, and nothing
|
|
814
|
+
crowned in 2,000 passes at `hold_satisfied: false` while the win rule named a
|
|
815
|
+
leader in 1,926 and 1,978 of them.
|
|
816
|
+
|
|
817
|
+
### Any posterior (`Allocator.from_summary`)
|
|
818
|
+
|
|
819
|
+
The same paused / won / warm-up / automatic-win / shifting rule, with the same
|
|
820
|
+
per-arm shares and checks, on statistics computed elsewhere: a `summary`
|
|
821
|
+
`{ arm => { p_best:, expected_loss:, mean:, lower:, upper: } }` and `exposures`
|
|
822
|
+
`{ arm => Integer }` in place of tallies, draws and rng. `call` builds its Beta
|
|
823
|
+
statistics and hands them to that one private rule, so the two entry points
|
|
824
|
+
cannot disagree on a phase: fed `call`'s own statistics, `from_summary` answers
|
|
825
|
+
`call`'s decision.
|
|
826
|
+
|
|
827
|
+
```ruby
|
|
828
|
+
A.from_summary(
|
|
829
|
+
arms: %i[control a],
|
|
830
|
+
summary: { control: { p_best: 0.3, expected_loss: 0.02, mean: 1.1, lower: 0.9, upper: 1.3 },
|
|
831
|
+
a: { p_best: 0.7, expected_loss: 0.005, mean: 1.2, lower: 1.0, upper: 1.4 } },
|
|
832
|
+
exposures: { control: 1_000, a: 1_000 }, status: "running", winner_arm: nil,
|
|
833
|
+
previous_weights: { control: 0.5, a: 0.5 }, hold_satisfied: false,
|
|
834
|
+
)
|
|
835
|
+
# => { weights: { control: 0.35, a: 0.6499999999999999 }, phase: :shifting, stats: { ... },
|
|
836
|
+
# winner: nil, result_changed: false, leader: nil }
|
|
837
|
+
```
|
|
838
|
+
|
|
839
|
+
- `summary: nil` (no posterior yet) holds warm-up whatever the exposures say; a
|
|
840
|
+
pause and a stored win still come first.
|
|
841
|
+
- A test that never ends passes `hold_satisfied: false` at every pass: nothing
|
|
842
|
+
is crowned and the leader is still named, which a caller whose own rule shows
|
|
843
|
+
no leader blanks itself. A warm-up of 0 is `Config.new(warmup_exposures: 0)`.
|
|
844
|
+
- `expected_loss` and `Config#win_loss` are read in the caller's units (value
|
|
845
|
+
per visit for a value summary). The rule never reads `mean`; `stats` carry the
|
|
846
|
+
summary's figures as Floats. Keys may be Symbols or Strings; the answer names
|
|
847
|
+
every arm by its Symbol.
|
|
848
|
+
|
|
849
|
+
### The lift (`Allocator.lift`)
|
|
850
|
+
|
|
851
|
+
```ruby
|
|
852
|
+
A.lift({ control: 0.99, a: 0.01 }) # => { control: 0.9, a: 0.1 }
|
|
853
|
+
A.lift({ a: 0.9, b: 0.06, c: 0.04 }) # => { a: 0.8, b: 0.1, c: 0.1 }
|
|
854
|
+
```
|
|
855
|
+
|
|
856
|
+
Every arm at least `Config#floor` (`config:` as for `call`), the others scaled
|
|
857
|
+
in proportion into what the pinned floors leave: 0.2.0's private `with_floor`,
|
|
858
|
+
step for step, made public (the private method stays, delegating to it).
|
|
859
|
+
|
|
860
|
+
### The hold on a win (`Bandit::WinHold`)
|
|
861
|
+
|
|
862
|
+
An automatic win counts only after the same arm met the win rule at every pass
|
|
863
|
+
for 24 hours. The caller keeps the state (the leading arm, when it started
|
|
864
|
+
leading, the time of the previous pass) and gives the clock:
|
|
865
|
+
|
|
866
|
+
```ruby
|
|
867
|
+
hold = SkiftetStatistical::Bandit::WinHold.new(
|
|
868
|
+
leading_arm: "a", leading_since: now - 90_000, now: now, last_pass_at: now - 900,
|
|
869
|
+
# hold_seconds: 86_400, max_gap_seconds: 1_200 (the defaults)
|
|
870
|
+
)
|
|
871
|
+
hold.satisfied? # => true: the stored leader held for 24 hours or more
|
|
872
|
+
hold.satisfied_for?(:a)
|
|
873
|
+
result = A.call(..., hold_satisfied: hold.satisfied?)
|
|
874
|
+
state = hold.after_pass(result[:leader]) # => the leading arm and since, to store
|
|
875
|
+
```
|
|
876
|
+
|
|
877
|
+
A run of passes with a gap longer than `max_gap_seconds` (or no previous pass)
|
|
878
|
+
is interrupted: not satisfied, and a leader starts over. Call the allocator a
|
|
879
|
+
second time with the same seed and `hold_satisfied: false` when the arm it
|
|
880
|
+
crowns is not the arm that held (`satisfied_for?`); `Bandit::Pass` (above)
|
|
881
|
+
makes both calls in one.
|
|
882
|
+
|
|
883
|
+
## Linear value of several rates (`LinearValue`)
|
|
884
|
+
|
|
885
|
+
When an arm's worth is a weighted sum of several rates, V_a = sum over parts p
|
|
886
|
+
of w_p * r_(a, p), and both the rates and the weights are uncertain,
|
|
887
|
+
`LinearValue` draws them together: the rates from one joint normal over every
|
|
888
|
+
(arm, part) pair (so arms that share evidence stay correlated), the weights from
|
|
889
|
+
named parameters drawn afresh in every draw. P(best) and the expected loss then
|
|
890
|
+
hedge every uncertain weight at once instead of judging the arms at one assumed
|
|
891
|
+
weight.
|
|
892
|
+
|
|
893
|
+
```ruby
|
|
894
|
+
LV = SkiftetStatistical::LinearValue
|
|
895
|
+
|
|
896
|
+
inputs = {
|
|
897
|
+
arms: %i[a b], parts: %i[p q],
|
|
898
|
+
means: { a: [ 0.12, 0.031 ], b: [ 0.105, 0.044 ] },
|
|
899
|
+
covariance: [ [ 4e-4, 6e-5, 2e-4, 1e-5 ], [ 6e-5, 1e-4, 2e-5, 4e-5 ], # arm-major:
|
|
900
|
+
[ 2e-4, 2e-5, 5e-4, -5e-5 ], [ 1e-5, 4e-5, -5e-5, 1.2e-4 ] ], # (a, p), (a, q), (b, p), (b, q)
|
|
901
|
+
parameters: { base: { fixed: 1.5 }, worth: { uniform: [ 2.0, 6.0 ] },
|
|
902
|
+
lift: { normal: [ 0.5, 0.2 ], clip: [ 0.0, 1.0 ] } },
|
|
903
|
+
weights: ->(drawn) { { p: drawn.fetch(:base), q: drawn.fetch(:worth) * drawn.fetch(:lift) } }
|
|
904
|
+
}
|
|
905
|
+
LV.summary(**inputs, draws: 20_000, rng: Random.new(7))
|
|
906
|
+
# => { a: { point: 0.242, mean: 0.2423..., lower: 0.1516..., upper: 0.3587..., p_best: 0.47715,
|
|
907
|
+
# expected_loss: 0.0188... },
|
|
908
|
+
# b: { point: 0.2455, mean: 0.2461..., lower: 0.1460..., upper: 0.3736..., p_best: 0.52285,
|
|
909
|
+
# expected_loss: 0.0150... } }
|
|
910
|
+
LV.each_draw(**inputs, draws: 3, rng: Random.new(7)).to_a
|
|
911
|
+
# => [[0.2233..., 0.1935...], [0.2945..., 0.3738...], [0.1634..., 0.2035...]] (summary's first rows)
|
|
912
|
+
```
|
|
913
|
+
|
|
914
|
+
`point` is the value at every parameter's centre (the fixed value, the
|
|
915
|
+
uniform's midpoint, the normal's mean, each clipped) and the part means;
|
|
916
|
+
`mean`, `lower` and `upper` are the drawn values' mean and equal-tailed
|
|
917
|
+
`credible_mass` interval (0.95 by default); a tie splits P(best) equally.
|
|
918
|
+
`each_draw` yields each draw's values, the rows `summary` is built on (an
|
|
919
|
+
Enumerator without a block).
|
|
920
|
+
|
|
921
|
+
**The draw order is part of the API.** Per draw: each parameter in the order
|
|
922
|
+
given (`fixed: x` takes nothing, `uniform: [lo, hi]` one `rng.rand`,
|
|
923
|
+
`normal: [mean, sd]` one `Sampler#gaussian`, then its `clip: [lo, hi]` when it
|
|
924
|
+
has one); the part weights from the `weights` callable; one joint normal draw
|
|
925
|
+
of every rate, means + L * n, with n the K * P `Sampler#gaussian` values in
|
|
926
|
+
arm-major order and L the covariance's `Cholesky` factor; each arm's value. A
|
|
927
|
+
part with no variance is drawn at its mean and still takes its normal, so the
|
|
928
|
+
numbers taken never depend on the data.
|
|
929
|
+
|
|
930
|
+
Everything is checked before the first random number (`ArgumentError`): 1 to 7
|
|
931
|
+
arms and 1 to 12 parts (`MAX_ARMS`, `MAX_PARTS`; sum the parts that share one
|
|
932
|
+
weight), a symmetric positive semidefinite covariance, finite numbers, a weight
|
|
933
|
+
for every part. `draws:` is an Integer >= 0 (`summary` needs at least 1).
|
|
934
|
+
`rounding:` (0.0 by default) is how far each covariance entry may be off: hand
|
|
935
|
+
it a `SharedCredit` estimate's `#rounding`, so a covariance summed from rows is
|
|
936
|
+
never refused for its own rounding.
|
|
937
|
+
|
|
938
|
+
Measured (`value_allocation_simulation_spec.rb`, below): the 95 % intervals of
|
|
939
|
+
each arm's value and of each difference covered the truth in 377 to 385 of 400
|
|
940
|
+
runs (bound 372), the true weights drawn inside their ranges per run;
|
|
941
|
+
allocating on it through `Allocator.from_summary`, values that arrived up to 23
|
|
942
|
+
passes late left a final bias within 1.08 standard errors, and a drift shared by
|
|
943
|
+
every arm moved no equal arm's mean P(best) more than 0.028 from 1 / 4.
|
|
944
|
+
|
|
945
|
+
## Cholesky factor (`Cholesky`)
|
|
946
|
+
|
|
947
|
+
```ruby
|
|
948
|
+
SkiftetStatistical::Cholesky.factor([ [ 4.0, 2.0 ], [ 2.0, 5.0 ] ]) # => [[2.0, 0.0], [1.0, 2.0]]
|
|
949
|
+
SkiftetStatistical::Cholesky.root([ [ 1.0, 2.0 ], [ 2.0, 4.0 ] ]) # => [[1.0, 0.0], [2.0, 0.0]]
|
|
950
|
+
```
|
|
951
|
+
|
|
952
|
+
`factor` answers the lower-triangular L with L * L^T = the matrix, rows in and
|
|
953
|
+
rows out, the lower triangle read, with no matrix library. The matrix must be
|
|
954
|
+
square, finite and symmetric (entries (i, j) and (j, i) within 1e-12 of their
|
|
955
|
+
scale). A pivot below 1e-12 times the largest diagonal entry is no variance and
|
|
956
|
+
is set to 0 with its column (a residual under it beyond rounding raises: a part
|
|
957
|
+
with no variance cannot covary); a pivot below -1e-9 times the trace raises:
|
|
958
|
+
the matrix is not positive semidefinite. `root(rows, rounding: 0.0)` is the
|
|
959
|
+
same arithmetic taken largest pivot first, for a matrix of lower rank whose
|
|
960
|
+
rounding the given order would carry into every later pivot: a square root S
|
|
961
|
+
with S * S^T = the matrix within its tolerance, not triangular in the given
|
|
962
|
+
order. `LinearValue` draws through `factor`, and through `root` where `factor`
|
|
963
|
+
refuses.
|
|
964
|
+
|
|
965
|
+
Measured: no simulation of its own; every `LinearValue` draw in the
|
|
966
|
+
simulations below goes through it.
|
|
967
|
+
|
|
968
|
+
## Shared-credit estimates (`SharedCredit`)
|
|
969
|
+
|
|
970
|
+
Each arm's mean of several parts, with one covariance over every (arm, part)
|
|
971
|
+
pair, when one row may count for several arms. A row is
|
|
972
|
+
`{ credit: { arm => w >= 0 }, values: [x_1, ..., x_P] }`: in an ordinary test
|
|
973
|
+
it credits only the arm drawn, with 1 / p; with shared credit it credits every
|
|
974
|
+
arm whose own choice is what the row shows, each with 1 over the probability
|
|
975
|
+
that the row showed it. One estimator serves both.
|
|
976
|
+
|
|
977
|
+
```ruby
|
|
978
|
+
SC = SkiftetStatistical::SharedCredit
|
|
979
|
+
|
|
980
|
+
estimate = SC.estimate(
|
|
981
|
+
[ { credit: { a: 2.0 }, values: [ 1.0, 0.0 ] },
|
|
982
|
+
{ credit: { a: 1.0, b: 1.0 }, values: [ 0.0, 2.0 ] },
|
|
983
|
+
{ credit: { b: 4.0 }, values: [ 1.0, 1.0 ] } ],
|
|
984
|
+
arms: %i[a b], parts: %i[p q],
|
|
145
985
|
)
|
|
146
|
-
|
|
986
|
+
# => { means: { a: [0.6666..., 0.6666...], b: [0.8, 1.2] },
|
|
987
|
+
# covariance: [[0.0987..., -0.1975..., 0.0355..., -0.0355...], ...], # arm-major: (a, p), (a, q), (b, p), (b, q)
|
|
988
|
+
# weight_sums: { a: 3.0, b: 5.0 }, effective: { a: 1.8, b: 1.4705... } }
|
|
989
|
+
estimate.parts # => [:p, :q]
|
|
990
|
+
estimate.rounding # => 7.78...e-15, the largest rounding bound of an entry
|
|
991
|
+
SC.contrast(estimate, :b, :a, weights: { p: 1.0, q: 0.5 })
|
|
992
|
+
# => { mean: 0.3999999999999999, variance: 0.012800000000000138 }
|
|
993
|
+
```
|
|
147
994
|
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
995
|
+
Means are Hájek, r_ap = sum_i w_ai * x_ip / W_a; the covariance is the sandwich
|
|
996
|
+
sum_i psi_ap(i) * psi_bq(i) with psi_ap(i) = w_ai * (x_ip - r_ap) / W_a, so arms
|
|
997
|
+
that share rows covary. It is laid out arm-major, as `LinearValue` reads it,
|
|
998
|
+
computed on one triangle and mirrored, so it is exactly symmetric; Kish's
|
|
999
|
+
effective size stands beside it. An arm no row credits has nil means and zero
|
|
1000
|
+
rows. An entry within its cancellation rounding ((8 + n) * `Float::EPSILON`
|
|
1001
|
+
times the sum of its terms' sizes, n the larger of Kish's size and the rows
|
|
1002
|
+
folded) answers exactly 0.0, so an arm of constant values gives zeros that
|
|
1003
|
+
`LinearValue` accepts.
|
|
1004
|
+
|
|
1005
|
+
The sums are public, for a caller that keeps its rows in a store and must fold
|
|
1006
|
+
each one once:
|
|
1007
|
+
|
|
1008
|
+
```ruby
|
|
1009
|
+
acc = SC.accumulator(arms: %i[a b], parts: %i[p q])
|
|
1010
|
+
acc.add({ credit: { a: 2.0 }, values: [ 1.0, 0.0 ] })
|
|
1011
|
+
saved = acc.to_h # JSON-safe: { "format" => 1, "arms" => ["a", "b"], "parts" => ["p", "q"], ... }
|
|
1012
|
+
SC::Accumulator.from_h(saved).merge(later).estimate # = one estimate over every row of both
|
|
151
1013
|
```
|
|
152
1014
|
|
|
1015
|
+
For an ordinary test the per-arm sums are enough, `{ arm => { w:, w2:,
|
|
1016
|
+
a: { part => }, b: { part => }, c: { [p, q] => } } }`, with w = sum 1/p,
|
|
1017
|
+
w2 = sum 1/p^2, a = sum x/p, b = sum x/p^2 and c = sum x_p * x_q / p^2 (both
|
|
1018
|
+
orders of each pair):
|
|
1019
|
+
|
|
1020
|
+
```ruby
|
|
1021
|
+
sums = { a: { w: 4.0, w2: 8.0, a: { p: 2.0, q: 2.0 }, b: { p: 4.0, q: 4.0 },
|
|
1022
|
+
c: { %i[p p] => 4.0, %i[p q] => 0.0, %i[q p] => 0.0, %i[q q] => 4.0 } },
|
|
1023
|
+
b: { w: 8.0, w2: 32.0, a: { p: 4.0, q: 4.0 }, b: { p: 16.0, q: 16.0 },
|
|
1024
|
+
c: { %i[p p] => 16.0, %i[p q] => 16.0, %i[q p] => 16.0, %i[q q] => 16.0 } } }
|
|
1025
|
+
SC.from_sums(sums, arms: %i[a b], parts: %i[p q])
|
|
1026
|
+
# => { means: { a: [0.5, 0.5], b: [0.5, 0.5] },
|
|
1027
|
+
# covariance: [[0.125, -0.125, 0.0, 0.0], [-0.125, 0.125, 0.0, 0.0],
|
|
1028
|
+
# [0.0, 0.0, 0.125, 0.125], [0.0, 0.0, 0.125, 0.125]],
|
|
1029
|
+
# weight_sums: { a: 4.0, b: 8.0 }, effective: { a: 2.0, b: 2.0 } }
|
|
1030
|
+
# the estimate of four rows: a credited 2.0 with [1, 0] and [0, 1], b credited 4.0 with [1, 1] and [0, 0]
|
|
1031
|
+
```
|
|
1032
|
+
|
|
1033
|
+
- `linear(estimate, weights:)`: each arm's value at fixed weights and the K x K
|
|
1034
|
+
covariance of the values. `contrast(estimate, a, b, weights:)`: one
|
|
1035
|
+
difference with its variance. `weights:` is an Array in the parts' order or a
|
|
1036
|
+
Hash by part name (every part once, in any order; refused on an estimate read
|
|
1037
|
+
back from JSON, which names no parts).
|
|
1038
|
+
- `contrast_for_fit(estimate, arm, reference, weights:, minimum_effective: 20)`:
|
|
1039
|
+
the contrast a fit across cells takes, the same difference with one per-visit
|
|
1040
|
+
variance pooled over the two arms, s^2 * (1 / n_a + 1 / n_r), and nil when
|
|
1041
|
+
either arm has nil means or a Kish's size under `minimum_effective`. An arm
|
|
1042
|
+
held at a small floor reads a plug-in variance that moves with its own
|
|
1043
|
+
estimate (a rare part it has not seen yet reads a variance near 0), and a
|
|
1044
|
+
test across cells would then reject too often.
|
|
1045
|
+
|
|
1046
|
+
Measured (`two_layer_simulation_spec.rb`, below): scored by greedy shared
|
|
1047
|
+
credit, two options of identical value read a mean P(best) of 0.58 and 0.42
|
|
1048
|
+
(bound 0.40 to 0.60) where the same visits scored as they ran read 0.63 and
|
|
1049
|
+
0.37; in the pooled cells, `contrast_for_fit` left 1,135 of 24,000 cell
|
|
1050
|
+
contrasts out of the fit.
|
|
1051
|
+
|
|
1052
|
+
## Hierarchical Poisson rates (`HierarchicalRates`)
|
|
1053
|
+
|
|
1054
|
+
How far a node's rate stands from its parent's, from its count s and the count
|
|
1055
|
+
E its parent leads us to expect there. The node's ratio has a Gamma prior with
|
|
1056
|
+
mean 1 and strength k (Clayton and Kaldor 1987), so its posterior is
|
|
1057
|
+
Gamma(k + s, k + E); under a two-groups prior (Efron 2008) the node is null
|
|
1058
|
+
with probability 1 - h (its count Poisson(E), its ratio 1) and otherwise drawn
|
|
1059
|
+
from that slab.
|
|
1060
|
+
|
|
1061
|
+
```ruby
|
|
1062
|
+
HR = SkiftetStatistical::HierarchicalRates
|
|
1063
|
+
|
|
1064
|
+
HR.leave_one_out_share(parent_count: 40.0, parent_total: 1_000.0, node_count: 6.0, node_total: 100.0,
|
|
1065
|
+
prior_share: 0.04, prior_strength: 16.0) # => 0.0378... (34.64 / 916)
|
|
1066
|
+
HR.node(count: 6.0, expected: 2.5, strength: 8.0, threshold: 1.25, parent_tail: 0.0, parent_ratio: 1.0)
|
|
1067
|
+
# => { ratio: 1.3333333333333333 (14 / 10.5), hot: 1.0, tail: 0.5593048238168208 (Q(14, 13.125)) }
|
|
1068
|
+
|
|
1069
|
+
pairs = [ { count: 6.0, expected: 2.5, siblings: 3 }, { count: 0.0, expected: 2.0, siblings: 3 },
|
|
1070
|
+
{ count: 3.0, expected: 3.5, siblings: 3 }, { count: 12.0, expected: 4.0, siblings: 2 },
|
|
1071
|
+
{ count: 1.0, expected: 3.0, siblings: 2 }, { count: 5.0, expected: 1.0, siblings: 1 } ]
|
|
1072
|
+
HR.fit_strength(pairs: pairs)
|
|
1073
|
+
# => { strength: 1.1892... (2**0.25), hot_share: 1.0, log_likelihood: -12.57..., at_upper_bound: false,
|
|
1074
|
+
# interval: [0.25, 9.5136...], profile: [[0.25, ...], ...] }
|
|
1075
|
+
HR.marshall(pairs: pairs) # => { variance: 0.9136..., strength: 1.0945... }
|
|
1076
|
+
HR.boundary_test(pairs: pairs, strength: 2**0.25) # => { statistic: 7.5936..., p_value: 0.0029... }
|
|
1077
|
+
HR.smooth_strength(previous: 8.0, fitted: 16.0) # => 9.1895... (exp(0.8 * log 8 + 0.2 * log 16))
|
|
1078
|
+
```
|
|
1079
|
+
|
|
1080
|
+
The last pair (siblings 1) is left out of the fit and of both checks: its
|
|
1081
|
+
parent has one child with data, which carries nothing about the level.
|
|
1082
|
+
|
|
1083
|
+
- `node(count:, expected:, strength:, hot_share: 1.0, threshold:, parent_tail:,
|
|
1084
|
+
parent_ratio:)`: hot = h * NB(s; k, k / (k + E)) / [h * NB + (1 - h) *
|
|
1085
|
+
Poisson(s; E)] (1.0 at h = 1), ratio = (1 - hot) + hot * (k + s) / (k + E),
|
|
1086
|
+
tail = (1 - hot) * parent_tail + hot * Q(k + s, (k + E) * L / parent_ratio).
|
|
1087
|
+
- `fit_strength(pairs:, hot_grid: nil)`: the marginal log-likelihood maximised
|
|
1088
|
+
on `GRID` (57 strengths, 2^-2 to 2^12 in quarter powers of 2) and on
|
|
1089
|
+
`hot_grid` (h = 1 without one); `interval` holds the strengths within 1.92
|
|
1090
|
+
(half the chi-squared(1) 95 % point) of the maximum, and `at_upper_bound`
|
|
1091
|
+
says the level's nodes inherit their parent.
|
|
1092
|
+
- `fit_leaves(parents:, hot_shares:, grid: GRID)`: the leaf level's fit, each
|
|
1093
|
+
parent's null rate estimated jointly with k and h (from its pooled rate, each
|
|
1094
|
+
step weighting its leaves by their posterior chance of being null, at most 50
|
|
1095
|
+
steps) instead of read from a leave-one-out share. With about five leaves to a
|
|
1096
|
+
parent, a hot leaf lifts its siblings' leave-one-out share, they read as cold,
|
|
1097
|
+
and a fit on those expected counts reads the hot share several times too
|
|
1098
|
+
high. `parents` is `[[{ count:, total: }]]`, `hot_shares` each in [0, 1); the
|
|
1099
|
+
answer holds `strength`, `hot_share`, `log_likelihood`, `at_upper_bound`, and
|
|
1100
|
+
per parent `converged`, `rates` and each leaf's `hot`.
|
|
1101
|
+
- `smooth_strength(previous:, fitted:, weight: 0.2)`, a moving average in
|
|
1102
|
+
logs; `marshall(pairs:)`, Marshall's (1991) moment estimate;
|
|
1103
|
+
`boundary_test(pairs:, strength:, hot_share: 1.0)`, the likelihood-ratio test
|
|
1104
|
+
on the boundary (Self and Liang 1987).
|
|
1105
|
+
|
|
1106
|
+
Counts are real numbers >= 0: a node that stands in its parent by a share w has
|
|
1107
|
+
w multiplied into its count and total by the caller. The probability mass
|
|
1108
|
+
functions are taken in logs, through log-gamma.
|
|
1109
|
+
|
|
1110
|
+
Measured (`tree_recovery_simulation_spec.rb`, below): on generated five-level
|
|
1111
|
+
trees each level's fitted strength was within a factor of 2 of the true one in
|
|
1112
|
+
20 of 20 runs, the leaves' hot share within 0.05 in 20 of 20 on `fit_leaves`
|
|
1113
|
+
(0.05 to 0.1 for a true 0.02 on leave-one-out expected counts), and the
|
|
1114
|
+
shrunken ratio beat the raw one on a held-out half at every level below the
|
|
1115
|
+
first in every run.
|
|
1116
|
+
|
|
1117
|
+
## Additive contrasts across cells (`AdditiveContrasts`)
|
|
1118
|
+
|
|
1119
|
+
What differs between cells is pooled on the arms' contrasts against a reference
|
|
1120
|
+
arm, never on the arms' own levels (Hong, Kveton, Zaheer and Ghavamzadeh 2022).
|
|
1121
|
+
Per arm a, delta(a | cell) = delta_root(a) + u_tree(a) + the sum over groupings
|
|
1122
|
+
g of u_g(a): the tree nested (each node's deviation N(0, tau^2 of its depth)),
|
|
1123
|
+
every other grouping crossed (N(0, tau^2_g)).
|
|
1124
|
+
|
|
1125
|
+
```ruby
|
|
1126
|
+
AC = SkiftetStatistical::AdditiveContrasts
|
|
1127
|
+
|
|
1128
|
+
cells = [ { id: 1, contrasts: { a: [ 0.0, 1.0 ] }, tree: %i[root x], groups: {} }, # [estimate, sampling variance]
|
|
1129
|
+
{ id: 2, contrasts: { a: [ 1.0, 1.0 ] }, tree: %i[root y], groups: {} },
|
|
1130
|
+
{ id: 3, contrasts: { a: [ 5.0, 1.0 ] }, tree: %i[root z], groups: {} } ]
|
|
1131
|
+
fit = AC.fit(cells: cells, arms: %i[a])
|
|
1132
|
+
# => { variances: { "tree:1" => { a: 6.0 } },
|
|
1133
|
+
# effects: { "tree:1" => { x: { a: [-1.7142..., 2.5714...] }, y: { a: [-0.8571..., 2.5714...] },
|
|
1134
|
+
# z: { a: [2.5714..., 2.5714...] } } },
|
|
1135
|
+
# cells: { 1 => { a: [0.2857..., 0.9047...] }, 2 => { a: [1.1428..., 0.9047...] },
|
|
1136
|
+
# 3 => { a: [4.5714..., 0.9047...] } } }
|
|
1137
|
+
fit.variation_tests # => { "tree:1" => { a: { statistic: 14.0, degrees: 2, p_value: 0.000911... } } }
|
|
1138
|
+
AC.fit(cells: cells, arms: %i[a], variances: { "tree:1" => { a: 0.0 } })[:cells]
|
|
1139
|
+
# => { 1 => { a: [2.0, 0.3333...] }, 2 => { a: [2.0, 0.3333...] }, 3 => { a: [2.0, 0.3333...] } }
|
|
1140
|
+
```
|
|
1141
|
+
|
|
1142
|
+
A tree of one level with no other grouping, as here, is plain normal shrinkage
|
|
1143
|
+
toward the root: the library's one home of it. Variances come by moments on the
|
|
1144
|
+
contrasts, per depth, grouping and arm: tau^2 = max(0, the weighted spread of
|
|
1145
|
+
the group estimates less their mean sampling variance), unbiased whatever the
|
|
1146
|
+
precisions. `variances:` holds what it names fixed (an earlier fit's
|
|
1147
|
+
`variances`), and the rest is fitted. Posteriors are normal-normal: the tree is
|
|
1148
|
+
solved exactly for given variances, and the crossed groupings are backfitted
|
|
1149
|
+
with it, `sweeps` times (5 by default). A cell may leave out an arm (nil); an
|
|
1150
|
+
arm no cell has a contrast for raises.
|
|
1151
|
+
|
|
1152
|
+
`fit` answers an `AdditiveContrasts::Fit`, the Hash above, whose
|
|
1153
|
+
`#variation_tests` stand beside each variance: Cochran's Q (the spread the
|
|
1154
|
+
moment reads), its degrees and the chi-squared chance under tau^2 = 0 of a
|
|
1155
|
+
spread at least that large (1.0 with no degrees). A caller that holds a
|
|
1156
|
+
variance at 0 unless its test rejects hands the next fit that 0 in
|
|
1157
|
+
`variances:`.
|
|
1158
|
+
|
|
1159
|
+
`baseline(rows:, effects:)` is each cell's level from all its visits, less each
|
|
1160
|
+
arm's fitted effect and adjusted on a categorical covariate, with its Hájek
|
|
1161
|
+
variance. A row sums one (cell, arm, covariate)'s visits: `weight` (sum w),
|
|
1162
|
+
`value` (sum w * x) and, beyond one visit, `weight_squared`, `value_squared` and
|
|
1163
|
+
`cross` (sum w^2, sum w^2 * x^2, sum w^2 * x). The reference arm is the one arm
|
|
1164
|
+
of the rows that no cell of `effects` names.
|
|
1165
|
+
|
|
1166
|
+
```ruby
|
|
1167
|
+
AC.baseline(
|
|
1168
|
+
rows: [ { cell: "c1", arm: :ref, covariate: "k1", weight: 2.0, value: 2.0 },
|
|
1169
|
+
{ cell: "c1", arm: :a, covariate: "k2", weight: 1.0, value: 2.0 },
|
|
1170
|
+
{ cell: "c2", arm: :ref, covariate: "k2", weight: 1.0, value: 0.0 } ],
|
|
1171
|
+
effects: { "c1" => { a: [ 1.0, 0.25 ] }, "c2" => { a: [ 0.5, 0.25 ] } },
|
|
1172
|
+
)
|
|
1173
|
+
# => { "c1" => [0.9166... (11 / 12), 0.0246... (2 / 81)], "c2" => [0.25, 0.0] }
|
|
1174
|
+
```
|
|
1175
|
+
|
|
1176
|
+
Pooling per (arm, part) is what reaches a decision: `fit`'s `arms:` name the
|
|
1177
|
+
pairs (`AC.pair(:a, :p) # => "a|p"`), one `baseline` runs per part, and
|
|
1178
|
+
`pooled_estimate(own:, effects:, baselines:, cell:)` puts them back together as
|
|
1179
|
+
the cell's estimate in `SharedCredit`'s shape, with `#parts` and `#rounding`,
|
|
1180
|
+
which `LinearValue` and the allocator read as they read the cell's own. Means
|
|
1181
|
+
are mu_p + delta_ap (delta 0 at the reference arm); the covariance is
|
|
1182
|
+
Cov(mu_p, mu_q) + [a == b] * Cov(delta_ap, delta_aq): the cell's level is one
|
|
1183
|
+
quantity shared by every arm, each arm's effect is its own, and each block keeps
|
|
1184
|
+
the cell's own correlation between parts. A contrast of two arms then holds
|
|
1185
|
+
their effects alone, so no arm's share follows its cell's level.
|
|
1186
|
+
|
|
1187
|
+
```ruby
|
|
1188
|
+
# own: the cell's SharedCredit estimate, means ref [1.0] and a [2.0], covariance [[0.04, 0.012], [0.012, 0.09]]
|
|
1189
|
+
AC.pooled_estimate(own: own, effects: { "c1" => { "a|p" => [ 0.5, 0.03 ] } },
|
|
1190
|
+
baselines: { p: { "c1" => [ 1.0, 0.01 ] } }, cell: "c1")
|
|
1191
|
+
# => { means: { ref: [1.0], a: [1.5] }, covariance: [[0.01, 0.01], [0.01, 0.04]], weight_sums: ..., effective: ... }
|
|
1192
|
+
```
|
|
1193
|
+
|
|
1194
|
+
Measured (`contrasts_simulation_spec.rb` and `two_layer_simulation_spec.rb`,
|
|
1195
|
+
below): every variance of at least 4 times the mean sampling variance came back
|
|
1196
|
+
within a factor of 2, and every variance of 0 under a quarter of it, in each of
|
|
1197
|
+
100 runs; across 40 cells whose levels differ with no effect difference, each
|
|
1198
|
+
allocated on its pooled estimate, the reference arm's mean share spread 0.0066
|
|
1199
|
+
(the equal-level noise floor's 0.0065) at a rank correlation of -0.073 with the
|
|
1200
|
+
level.
|
|
1201
|
+
|
|
1202
|
+
## Beta argmax (`BetaArgmax`, `GaussLegendre`, `BetaFunction.log_density`)
|
|
1203
|
+
|
|
1204
|
+
Which of several independent Beta variables is the largest: the exact
|
|
1205
|
+
probability of each, the seedless pick that probability describes, and the
|
|
1206
|
+
greedy pick.
|
|
1207
|
+
|
|
1208
|
+
```ruby
|
|
1209
|
+
BA = SkiftetStatistical::BetaArgmax
|
|
1210
|
+
|
|
1211
|
+
BA.probabilities([ [ 1.0, 1.0 ], [ 2.0, 1.0 ] ]) # => [0.3333..., 0.6666...] (1/3, 2/3)
|
|
1212
|
+
BA.probabilities([ [ 1.0, 1.0 ], [ 2.0, 1.0 ] ], epsilon: 0.1) # => [0.35..., 0.65...]
|
|
1213
|
+
BA.pick([ [ 2.0, 8.0 ], [ 8.0, 2.0 ] ], explore: 0.5, priorities: [ 0.3, 0.6 ],
|
|
1214
|
+
uniforms: [ 0.5, 0.5 ], epsilon: 0.1) # => 1
|
|
1215
|
+
BA.greedy([ [ 1.0, 3.0 ], [ 3.0, 1.0 ] ]) # => 1
|
|
1216
|
+
|
|
1217
|
+
SkiftetStatistical::GaussLegendre.nodes(2)
|
|
1218
|
+
# => [[-0.5773..., 1.0000...], [0.5773..., 1.0000...]] (x = -+1 / sqrt(3), w = 1)
|
|
1219
|
+
SkiftetStatistical::BetaFunction.log_density(0.25, 2.0, 3.0) # => 0.5232... (log 27/16)
|
|
1220
|
+
```
|
|
1221
|
+
|
|
1222
|
+
`pick` takes its uniforms from the caller, never from an RNG: below `epsilon`
|
|
1223
|
+
(`explore < epsilon`) the member with the smallest priority, else the member
|
|
1224
|
+
whose Beta quantile at its own uniform is the largest. Over uniform inputs
|
|
1225
|
+
member c is picked with probability epsilon / k + (1 - epsilon) * the integral
|
|
1226
|
+
of f_c * the product over j != c of F_j, which is what `probabilities` answers,
|
|
1227
|
+
so a caller can log the exact probability of the pick it made. The integral is
|
|
1228
|
+
composite Gauss-Legendre (`NODES`, 20 a panel), the panels broken at every
|
|
1229
|
+
member's quantiles at `BREAKS` (1e-10 to 1 - 1e-10, `TAIL` = 1e-10), each half
|
|
1230
|
+
of (0, 1) worked in the distance from its own end so that a parameter far below
|
|
1231
|
+
1 does not round, and normalised to sum 1. It is exact to 1e-8 absolute for 1
|
|
1232
|
+
to 20 members (`MAX_MEMBERS`) with every parameter at least 1e-6
|
|
1233
|
+
(`SMALLEST_PARAMETER`) and alpha + beta at most 3 * 10^6 (`LARGEST_TOTAL`):
|
|
1234
|
+
beyond those parameter limits `probabilities` and `pick` raise, while `greedy`
|
|
1235
|
+
(the largest mean) takes any parameters; every call takes 1 to 20 members. Ties
|
|
1236
|
+
go to the lower index.
|
|
1237
|
+
|
|
1238
|
+
`GaussLegendre.nodes(n)` is the n-point rule on [-1, 1], ascending and exactly
|
|
1239
|
+
symmetric, computed once per n and kept, frozen. `BetaFunction.log_density` is
|
|
1240
|
+
-Infinity outside (0, 1); `regularized` and `quantile` are unchanged.
|
|
1241
|
+
|
|
1242
|
+
Since 0.4.0 three more answers sit beside these, and `probabilities`, `pick` and
|
|
1243
|
+
`greedy` answer as in 0.3.0 (`pick` still compares its own position):
|
|
1244
|
+
|
|
1245
|
+
```ruby
|
|
1246
|
+
BA.position(0.5, 2.0, 2.0) # => 0.0, the draw 1/2
|
|
1247
|
+
BA.distribution(0.3, 2.0, 8.0) # => 0.8039967659999996
|
|
1248
|
+
BA.expected_maximum([ [ 1.0, 1.0 ], [ 1.0, 1.0 ] ]) # => 0.6666666666666667 (2/3)
|
|
1249
|
+
```
|
|
1250
|
+
|
|
1251
|
+
- `position(uniform, alpha, beta)`: where the draw of Beta(alpha, beta) at a
|
|
1252
|
+
uniform lies, log(2 x) up to 1/2 and -log(2 (1 - x)) beyond, a key that
|
|
1253
|
+
rounds at neither end; the posterior set's quantile draws compare by it. A
|
|
1254
|
+
guarded Newton step on the same distribution function. Against `pick`'s own
|
|
1255
|
+
bisection (0.3.0's position), recorded on 30 members at 1 000 uniforms each,
|
|
1256
|
+
the spec bounds |position - recorded| / max(1, |recorded|), an absolute
|
|
1257
|
+
error where the recorded position lies within 1 of 0 and a relative one
|
|
1258
|
+
beyond: 1e-12 (measured at most 4.7e-13), 1e-9 where alpha + beta exceeds
|
|
1259
|
+
10^4 (measured 3.0e-13) and 1e-8 at the smallest parameter (measured
|
|
1260
|
+
9.5e-11). It takes 0.019 / 0.028 / 0.057 ms at Beta(3, 300) / Beta(41, 960) /
|
|
1261
|
+
Beta(4 000, 96 000) where the bisection took 0.11 / 0.29 / 0.92 ms.
|
|
1262
|
+
- `distribution(x, alpha, beta)`: that distribution function itself, the one
|
|
1263
|
+
`probabilities`, `pick`, `position` and `expected_maximum` read, bit for bit.
|
|
1264
|
+
- `expected_maximum(params)`: E[the largest member], 1 less the integral over
|
|
1265
|
+
(0, 1) of the product of the members' distribution functions.
|
|
1266
|
+
|
|
1267
|
+
Measured (`argmax_accuracy_simulation_spec.rb`, below): on 30 fixtures, sharp
|
|
1268
|
+
ones with alpha + beta up to 2 * 10^5 among them, `pick`'s frequencies over
|
|
1269
|
+
10^4 draws came within 2.44 standard deviations of `probabilities`, and
|
|
1270
|
+
`probabilities` within 2.43 of 10^6 plain `Sampler#beta` draws (bound 5).
|
|
1271
|
+
|
|
1272
|
+
## Prequential log score (`Prequential`)
|
|
1273
|
+
|
|
1274
|
+
```ruby
|
|
1275
|
+
PQ = SkiftetStatistical::Prequential
|
|
1276
|
+
|
|
1277
|
+
rows = [ { cluster: 1, log_probs: { shallow: Math.log(0.25), deep: Math.log(0.5) } },
|
|
1278
|
+
{ cluster: 1, log_probs: { shallow: Math.log(0.5), deep: Math.log(0.5) } },
|
|
1279
|
+
{ cluster: 2, log_probs: { shallow: Math.log(0.5), deep: Math.log(0.25) } },
|
|
1280
|
+
{ cluster: 3, log_probs: { shallow: Math.log(0.125), deep: Math.log(0.5) } } ]
|
|
1281
|
+
PQ.score(rows, depth: :deep) # => 1.25 (bits per event)
|
|
1282
|
+
PQ.gains(rows, depths: %i[shallow deep]) # => { deep: { bits: 0.49999999999999994, se: 0.6495..., events: 4, clusters: 3 } }
|
|
1283
|
+
PQ.clustered_mean(sums: { 1 => 1.0, 2 => -1.0, 3 => 2.0 }, counts: { 1 => 2, 2 => 1, 3 => 1 })
|
|
1284
|
+
# => { mean: 0.5, lower: -0.7730..., upper: 1.7730..., n: 4, clusters: 3 }
|
|
1285
|
+
```
|
|
1286
|
+
|
|
1287
|
+
The prequential log score (Dawid 1984): how many bits an outcome cost a
|
|
1288
|
+
predictor that gave it probability p, -log2 p, on outcomes it had not seen.
|
|
1289
|
+
`gains` compares depths of one predictor, each deeper than the one before, on
|
|
1290
|
+
the same events: per depth after the first, the mean gain in bits per event and
|
|
1291
|
+
its standard error. Events of one cluster move together, so the standard error
|
|
1292
|
+
comes from the spread between clusters, with the factor G / (G - 1) (Liang and
|
|
1293
|
+
Zeger 1986); `clustered_mean(sums:, counts:, level: 0.95)` is the same interval
|
|
1294
|
+
for a caller that stores per-cluster totals and counts, and `gains` is built on
|
|
1295
|
+
it, so the two cannot differ. Under 2 clusters the standard error and the
|
|
1296
|
+
bounds are nil. A log probability must be finite and at most 0: smooth a
|
|
1297
|
+
probability of 0 first.
|
|
1298
|
+
|
|
1299
|
+
Measured: no simulation; its specs hold hand-checked values.
|
|
1300
|
+
|
|
1301
|
+
## One-sided change test (`Cusum`)
|
|
1302
|
+
|
|
1303
|
+
```ruby
|
|
1304
|
+
CU = SkiftetStatistical::Cusum
|
|
1305
|
+
|
|
1306
|
+
CU.run([ 0.0, 0.0, 5.0 ], reference: 0.5, threshold: 4.0)
|
|
1307
|
+
# => { statistic: 4.5, alarm_at: 3, path: [0.0, 0.0, 4.5] }
|
|
1308
|
+
CU.average_run_length(shift: 0.0, reference: 0.5, threshold: 4.0) # => 338.09...
|
|
1309
|
+
h = CU.threshold(reference: 0.5, false_alarm_steps: 365.0) # => 4.0743...
|
|
1310
|
+
CU.minimum_detectable(reference: 0.5, threshold: h, horizon_steps: 30) # => 0.4752...
|
|
1311
|
+
```
|
|
1312
|
+
|
|
1313
|
+
Page's (1954) statistic over standardised values,
|
|
1314
|
+
S_t = max(0, S_(t-1) + z_t - k), alarms at the first S_t above the threshold h
|
|
1315
|
+
(steps count from 1; nothing resets after an alarm). Run lengths are
|
|
1316
|
+
Siegmund's approximation, (exp(-2 theta b) + 2 theta b - 1) / (2 theta^2) with
|
|
1317
|
+
theta = shift - k and b = h + 1.166 (b^2 at theta = 0, summed as its series
|
|
1318
|
+
near 0). `threshold` turns a stated false-alarm rate, one alarm in
|
|
1319
|
+
`false_alarm_steps`, into h by bisection on [0, 50]; the rate is the
|
|
1320
|
+
approximation's, so a caller states it as about that rate.
|
|
1321
|
+
`minimum_detectable` is the smallest shift, in standard units, whose run length
|
|
1322
|
+
is at most `horizon_steps`; the caller scales it by its standard error.
|
|
1323
|
+
|
|
1324
|
+
Measured (`change_test_simulation_spec.rb`, below): a stated one false alarm in
|
|
1325
|
+
365 steps gave 0.983 per 365 null steps, and one in 1,460 gave 0.260 per 365
|
|
1326
|
+
(stated 0.25), each inside its 95 % binomial interval; a shift of the minimum
|
|
1327
|
+
detectable size was caught within the horizon in 62 to 66 % of runs (bound
|
|
1328
|
+
half).
|
|
1329
|
+
|
|
1330
|
+
## Bandit simulations: measured results
|
|
1331
|
+
|
|
1332
|
+
`spec/simulations` holds ten files, tagged `simulation: true` and run only
|
|
1333
|
+
with `SIMULATIONS=1` (`bundle exec rspec spec/simulations`; CI's simulations
|
|
1334
|
+
job runs them nightly, by hand, and on a change to the code they measure). Each
|
|
1335
|
+
example asserts its bounds; the figures beside the bounds were measured with
|
|
1336
|
+
the same code and seeds, and the files new in 0.3.0 and 0.4.0 print theirs
|
|
1337
|
+
through RSpec's reporter (the JSON formatter keeps them under "messages"). Every
|
|
1338
|
+
run is seeded, so each count below is exact for this version of the code, and
|
|
1339
|
+
every shape is generated here from round generic values. The eight files from
|
|
1340
|
+
0.2.0 and 0.3.0 were measured on 2026-10-31 with skiftet_statistical 0.3.0, and
|
|
1341
|
+
again on 2026-11-02 with 0.4.0, every count the same; the two files
|
|
1342
|
+
new in 0.4.0 were measured on 2026-11-02. Ruby 3.4.9, YJIT on, as CI runs
|
|
1343
|
+
them.
|
|
1344
|
+
|
|
1345
|
+
The two simulations from 0.2.0 replay A/B tests through `Bandit::Allocator` (and
|
|
1346
|
+
`Bandit::WinHold`) in memory: every pass, 40 visits are drawn by the current
|
|
1347
|
+
weights, each exposure counts 1 / the weight it was drawn under, and the
|
|
1348
|
+
allocator is called with the previous weights, as an allocation job does every
|
|
1349
|
+
15 minutes. Their three examples assert the bounds; the other figures were measured with the
|
|
1350
|
+
same code and seeds. Moved from the app that first ran them, with the same
|
|
1351
|
+
seeds, runs, passes, draws and bounds; measured again here on 2026-10-15 with
|
|
1352
|
+
skiftet_statistical 0.2.0 and Ruby 3.4.9, and every count the app had recorded
|
|
1353
|
+
came out the same.
|
|
1354
|
+
Measured again on 2026-10-31 with 0.3.0, the three asserted examples counted
|
|
1355
|
+
exactly as below (15 / 200; 0 / 100, an equal arm in 21 / 100; worse 0, better
|
|
1356
|
+
46, undecided 154 / 200). The other counts are 0.2.0's, from an allocator that
|
|
1357
|
+
0.3.0 and 0.4.0 keep bit for bit for every call that declares no per-arm share.
|
|
1358
|
+
In 0.4.0 these simulations draw a visit's arm by `Draw.from_weights`, and over
|
|
1359
|
+
all 12,263,200 draws of the four files that draw by weight it answered the
|
|
1360
|
+
draw the files had written by hand every time.
|
|
1361
|
+
|
|
1362
|
+
### The automatic win (`allocator_simulation_spec.rb`)
|
|
1363
|
+
|
|
1364
|
+
400 passes of 40 visits, 400 draws.
|
|
1365
|
+
|
|
1366
|
+
| Runs | Arms (true conversion rate) | Bound (asserted) | Measured |
|
|
1367
|
+
|---|---|---|---|
|
|
1368
|
+
| 200 (seeds 20 261 003 + run) | four arms at 30 % | an arm crowned in fewer than 20 / 200 | 15 / 200 (7.5 %) |
|
|
1369
|
+
| 100 (seeds 30 261 003 + run) | three arms at 30 %, one at 25 % | the worse arm crowned in 0 / 100 | 0 / 100 (an equal arm in 21 / 100) |
|
|
1370
|
+
|
|
1371
|
+
With equal arms a crowned arm costs nothing (it is as good as the rest, and its
|
|
1372
|
+
expected loss is at most `win_loss`), so the bound caps noise wins; a truly
|
|
1373
|
+
worse arm is never crowned.
|
|
1374
|
+
|
|
1375
|
+
### The hold (`win_hold_simulation_spec.rb`)
|
|
1376
|
+
|
|
1377
|
+
Two arms, 11 % against 10 % (the worse arm), 400 passes of 40 visits (100
|
|
1378
|
+
hours), 200 runs (seeds 40 261 003 + run). The decision step is an allocation
|
|
1379
|
+
job's: the allocator with `hold_satisfied: hold.satisfied?`, and the same seed again with
|
|
1380
|
+
`hold_satisfied: false` when the arm it crowns is not the arm that held.
|
|
1381
|
+
|
|
1382
|
+
- WITHOUT the hold (the allocator alone, `hold_satisfied: true` at every pass):
|
|
1383
|
+
the worse arm crowned in 26 / 200 (13.0 %) at 400 draws, 24 / 200 (12.0 %) at
|
|
1384
|
+
2 000 draws.
|
|
1385
|
+
- The hold's code with no wait (`hold_seconds: 0`), 1 000 draws: the worse arm
|
|
1386
|
+
crowned in 13 / 200 (6.5 %).
|
|
1387
|
+
- WITH the hold (24 hours): the worse arm crowned in 0 / 200 at 400, 1 000,
|
|
1388
|
+
2 000 and 20 000 draws; the better arm in 41-46 / 200 (43 at 400, 46 at
|
|
1389
|
+
1 000, 41 at 2 000, 44 at 20 000), and the rest undecided after 100 hours.
|
|
1390
|
+
- The asserted example runs at 1 000 draws: the worse arm at most 6 / 200 and
|
|
1391
|
+
the better arm at least 20 / 200 (a hold that never releases would crown
|
|
1392
|
+
nothing and pass the first bound alone). Measured: worse 0 / 200, better
|
|
1393
|
+
46 / 200, undecided 154 / 200.
|
|
1394
|
+
|
|
1395
|
+
### A value of several parts (`value_allocation_simulation_spec.rb`)
|
|
1396
|
+
|
|
1397
|
+
Each run replays a caller's passes in memory: visits drawn by the latest
|
|
1398
|
+
weights, each credited 1 / the share it was drawn under; the per-arm Hájek sums
|
|
1399
|
+
turned into the arms' joint normal (`SharedCredit.from_sums`), the value drawn
|
|
1400
|
+
with its uncertain weights (`LinearValue.summary`, P(best) among the free arms)
|
|
1401
|
+
and handed to `Allocator.from_summary`. A pass where a free arm has no credit
|
|
1402
|
+
yet takes the warm-up split (`summary: nil`). Where marked, the same shape also
|
|
1403
|
+
runs on `Allocator.call`'s yes/no tallies of the binary part. Unless stated, a
|
|
1404
|
+
visit's parts are a binary part worth 1 and a count part (1, 2 or 3 when not 0)
|
|
1405
|
+
worth uniform [0.25, 0.75], 40 visits a pass.
|
|
1406
|
+
|
|
1407
|
+
- **(a) Floors in every phase.** A reference arm floored at 0.05 and four arms
|
|
1408
|
+
at 0.02 (one arm clearly best), 400 passes walking warm-up, shifting, a pause
|
|
1409
|
+
on a stored split that breaks every floor, the automatic win and its stored
|
|
1410
|
+
win, and the reopened test; the same arms with one arm fixed at 0.30 beside
|
|
1411
|
+
them. 7 runs: floored on the value model 3, fixed on the value model 1,
|
|
1412
|
+
floored on tallies 2, fixed on tallies 1. Bound (asserted): no share below its
|
|
1413
|
+
floor or off its fixed share in any phase; every phase and an automatic win
|
|
1414
|
+
in every run. Measured: 0 breaches in 2,800 passes; per run warm-up 38 to 46
|
|
1415
|
+
passes, shifting 204 to 212, paused 20, won 130, one automatic win each;
|
|
1416
|
+
arm-passes held exactly at a floor 1,411, 1,428 and 1,428 (floored, value),
|
|
1417
|
+
1,054 (fixed, value), 1,364 and 1,428 (floored, tallies), 1,063 (fixed,
|
|
1418
|
+
tallies).
|
|
1419
|
+
- **(b) No end.** A reference arm (floor 0.05) and two arms (0.02), one at twice
|
|
1420
|
+
the others' rate, 2,000 passes at `hold_satisfied: false`, on the value model
|
|
1421
|
+
and on tallies. Bound (asserted): nothing crowned, no share below its floor, and
|
|
1422
|
+
the better arm named leader in at least 1,800 passes. Measured: crowned 0 and
|
|
1423
|
+
0; leader named in 1,926 (value) and 1,978 (tallies) passes; the final split
|
|
1424
|
+
0.05 / 0.02 / 0.93 in both.
|
|
1425
|
+
- **(c) Late values.** Three arms whose values (0.15, 0.17 and 0.20 at the
|
|
1426
|
+
weights' centre) arrive 0 to 23 passes after the visit, uniformly, so every
|
|
1427
|
+
pass allocates on sums that lack the values still on their way; 60 passes,
|
|
1428
|
+
then the last values arrive; 200 runs. Bound (asserted): each arm's final
|
|
1429
|
+
estimate (`LinearValue`'s point) within 2 standard errors of the truth.
|
|
1430
|
+
Measured: bias -0.00054, -0.00197 and -0.00116 (standard errors 0.00166,
|
|
1431
|
+
0.00188, 0.00107; z -0.33, -1.05, -1.08). Before the last values arrived (at
|
|
1432
|
+
pass 60, reported): -0.02798, -0.03399 and -0.03958 (z -19.7, -22.3, -41.8).
|
|
1433
|
+
- **(d) A drift shared by every arm.** Four equal arms, every probability three
|
|
1434
|
+
times its own over passes 41 to 60; the mean P(best) over passes 41 to 120 of
|
|
1435
|
+
30 runs. Bound (asserted): each within 0.05 of 1 / 4. Measured: 0.2254,
|
|
1436
|
+
0.2424, 0.2779 and 0.2543 over 2,400 scored passes. Held uneven: the same
|
|
1437
|
+
shape paused through the drift on a stored split giving one arm 0.65, 20
|
|
1438
|
+
runs. Bound (asserted): that arm's mean P(best) at most 0.30, while unweighted
|
|
1439
|
+
sums of the same visits (scored beside, never allocating) give it more than
|
|
1440
|
+
0.5. Measured: 0.2092 (the others 0.3129, 0.2569 and 0.2209) over 1,600
|
|
1441
|
+
passes; on unweighted sums 0.9601 (the others 0.0114, 0.0157, 0.0129). The
|
|
1442
|
+
reference arm reads above 1 / 4 for its spread: it had the least traffic
|
|
1443
|
+
through the drift.
|
|
1444
|
+
- **(f) Coverage with drawn weights.** Three arms on a fixed split (0.2, 0.4,
|
|
1445
|
+
0.4), 6,000 visits a run, a count part that is 0 in 95 % or more of visits
|
|
1446
|
+
(worth uniform [0.5, 1.5]) and a binary part at 1.0 to 1.5 % (worth uniform
|
|
1447
|
+
[2, 6]), the true worths drawn inside those ranges per run; 4,000 joint draws;
|
|
1448
|
+
400 runs. Bound (asserted): each arm's 95 % interval and each difference's
|
|
1449
|
+
covers the truth in at least 372 of 400 (93 %). Measured: arms 378, 377 and
|
|
1450
|
+
383; differences 382, 385 and 383 (94.25 to 96.25 %).
|
|
1451
|
+
|
|
1452
|
+
### Pooled cells and two layers (`two_layer_simulation_spec.rb`)
|
|
1453
|
+
|
|
1454
|
+
- **(e) Cells whose levels differ.** 40 cells in 8 groups of 5, crossed with 4
|
|
1455
|
+
bands. Each cell's level is its factor (0.5 to 2.45 in steps of 0.05, in an
|
|
1456
|
+
order that follows neither group nor band) times its covariate mix: each
|
|
1457
|
+
visit's covariate value multiplies its probabilities by 0.5, 1.0 or 1.5, in
|
|
1458
|
+
proportions that differ by cell. No arm's values differ from the reference
|
|
1459
|
+
arm's in any cell. Three arms (the reference floored at 0.05, two at 0.02),
|
|
1460
|
+
30 visits a cell a pass, 50 passes, 3 runs. Each cell is allocated at every
|
|
1461
|
+
pass on its pooled estimate: its contrasts by `SharedCredit.contrast_for_fit`
|
|
1462
|
+
(Kish's minimum 20), `AdditiveContrasts.fit` over (arm, part) pairs, one
|
|
1463
|
+
`baseline` a part, `pooled_estimate`, `LinearValue.summary` and
|
|
1464
|
+
`from_summary` (warm-up 0, nothing crowned). The variances run as designed: a
|
|
1465
|
+
weekly fit by moments every 7 passes, each variance held at 0 unless its test
|
|
1466
|
+
of no variation rejects at 0.05 divided by that week's number of tests, and
|
|
1467
|
+
held fixed between. Bound (asserted): the reference arm's mean share across
|
|
1468
|
+
the 40 cells at a rank correlation with the level within plus or minus 0.3,
|
|
1469
|
+
and a spread (largest less smallest) at most 1.5 times the noise floor's, the
|
|
1470
|
+
same simulation, seeds, runs and passes with every cell's level equal.
|
|
1471
|
+
Measured: 0.1426 to 0.1492 (spread 0.0066, standard deviation 0.0018) over
|
|
1472
|
+
5,880 cell passes allocated on a pooled estimate, rank correlation -0.073;
|
|
1473
|
+
noise floor 0.1225 to 0.1291 (spread 0.0065, standard deviation 0.0014), so
|
|
1474
|
+
1.007 times; 21 weekly fits, each test read at 0.004167, 1 of 252 variances
|
|
1475
|
+
kept at its moment (0.004293) and none in the noise floor; 1,135 of 24,000
|
|
1476
|
+
cell contrasts left out of the fit (1,440 in the noise floor). Against a fixed
|
|
1477
|
+
0.03 the spread is 0.0066 (reported). With every pass fitting the variances by
|
|
1478
|
+
moments instead (reported, no bound): 0.11 to 0.2625 (spread 0.1525,
|
|
1479
|
+
standard deviation 0.0425), rank correlation -0.047, 1,051 contrasts left
|
|
1480
|
+
out; at no true variation a fitted variance comes out above 0 in about half
|
|
1481
|
+
the fits, and a cell then keeps that share of its own noise in its effects.
|
|
1482
|
+
- **(j) Two layers.** Two options of identical value, each above an inner layer
|
|
1483
|
+
that picks one of three candidates (rates 0.30, 0.20 and 0.10). The warm
|
|
1484
|
+
option's candidates are held at 1,000 showings each; the cold option's start
|
|
1485
|
+
at Beta(1, 1) and learn from their showings, their best placed first, second
|
|
1486
|
+
or third by run. Each option's candidates are one posterior set
|
|
1487
|
+
(`Posteriors.beta`); each visit's item is drawn by the exact pick odds
|
|
1488
|
+
(`Posteriors#pick_probabilities`, explore 0.1) through `Draw.from_weights`,
|
|
1489
|
+
and each option is scored by its greedy pick (`Posteriors#greedy`) through
|
|
1490
|
+
`SharedCredit`: a visit credits every option whose greedy pick it showed,
|
|
1491
|
+
with 1 / the visit's probability of showing that item, the two layers' odds
|
|
1492
|
+
combined by `Posteriors.mixture`. 30 passes of 80 visits, 100 runs. Bound (asserted): both options' mean
|
|
1493
|
+
P(best) in 0.40 to 0.60, and the cold option at its floor at the end in at
|
|
1494
|
+
most 10 runs. Measured: 0.5807 and 0.4193 over 3,000 passes, the cold option
|
|
1495
|
+
at its floor in 2 of 100. Scored as they ran (each visit crediting its drawn
|
|
1496
|
+
option with 1 / its share; reported): 0.6273 and 0.3727, and an allocation on
|
|
1497
|
+
that score ends the cold option at its floor in 5.
|
|
1498
|
+
|
|
1499
|
+
### Tree recovery (`tree_recovery_simulation_spec.rb`)
|
|
1500
|
+
|
|
1501
|
+
Five-level generated trees, 1, 20, 200, 800 and 4,000 nodes a level, with
|
|
1502
|
+
single-child parents (1, 10 and 40 at levels 1 to 3), parents that are one
|
|
1503
|
+
child's near-whole (95 % of their exposure, at a rate of 0.05) and 2 % of the
|
|
1504
|
+
leaves split between two parents, so counts above the leaves are real numbers.
|
|
1505
|
+
One law for every item: multipliers Gamma(k, rate k) with k = 8, 8 and 16 at
|
|
1506
|
+
levels 1 to 3, and at the leaves a two-groups law, hot share 0.02 and slab
|
|
1507
|
+
strength 0.5. Nine items, three national (200,000 counts a half), three
|
|
1508
|
+
regional (20,000) and three very local (2,000); each count Poisson, drawn twice:
|
|
1509
|
+
half A is fitted, half B held out. 20 runs (seeds 20 261 131 + run).
|
|
1510
|
+
|
|
1511
|
+
- **Level strengths** (bound: within a factor of 2 in at least 18 of 20 runs):
|
|
1512
|
+
level 1 (true 8) in 20 of 20, fitted 5.657 four times, 6.727 twelve times and
|
|
1513
|
+
8.0 four times; level 2 (true 8) in 20 of 20, 5.657 nine times and 6.727
|
|
1514
|
+
eleven times; level 3 (true 16) in 20 of 20, 8.0 twice and 9.514 eighteen
|
|
1515
|
+
times. The upper levels read low (they absorb part of the variance below
|
|
1516
|
+
them), and the two level-3 runs at 8.0 sit on the bound's edge.
|
|
1517
|
+
- **Leaves on `fit_leaves`** (bound: the slab strength within a factor of 2 and
|
|
1518
|
+
the hot share within 0.05, each in at least 18 of 20 runs): slab in 20 of 20
|
|
1519
|
+
(0.354 ten times, 0.5 nine times, 0.707 once), hot share in 20 of 20 (0.01
|
|
1520
|
+
nineteen times, 0.02 once); 7,200 parents a run, 0 to 7 of them not settled
|
|
1521
|
+
after 50 steps. The same leaves fitted on leave-one-out expected counts
|
|
1522
|
+
(`fit_strength`; reported, no bound): hot share 0.05 five times and 0.1
|
|
1523
|
+
fifteen times, slab 0.42 once, 0.5 twice, 0.595 twice, 0.841 twice, 1.0 five
|
|
1524
|
+
times and 1.189 eight times (up to 2.4 times the truth).
|
|
1525
|
+
- **Single-child parents** (bound: every fit identical without their pairs):
|
|
1526
|
+
identical at level 2 (8 to 9 such pairs a run, 20 runs), level 3 (76 to 88,
|
|
1527
|
+
20 runs) and level 4 (269 to 276, 2 runs); `fit_leaves` identical without the
|
|
1528
|
+
360 parents of a single leaf a run (2 runs).
|
|
1529
|
+
- **Held-out log score of the shrunken ratio less the raw one** (bound: better
|
|
1530
|
+
summed over the levels, and at each level below the first, in every run):
|
|
1531
|
+
level 2 better in 20 of 20 (min 251.3, median 478.7, max 749.0 nats), level 3
|
|
1532
|
+
in 20 of 20 (4,431.6; 5,364.2; 6,051.1), level 4 in 20 of 20 (46,538.5;
|
|
1533
|
+
48,844.4; 51,500.6). Level 1, where a node holds thousands of an item's counts
|
|
1534
|
+
and shrinking moves a share by parts in 10^5 (reported): better in 14 of 20
|
|
1535
|
+
(min -2.6, median 3.0, max 31.3).
|
|
1536
|
+
|
|
1537
|
+
### Contrasts with known variances (`contrasts_simulation_spec.rb`)
|
|
1538
|
+
|
|
1539
|
+
One root, 400 nodes under it ("tree:1"), 6 leaves a node ("tree:2"), 2 cells a
|
|
1540
|
+
leaf (4,800 cells), crossed with grouping a (300 keys) and grouping b (150
|
|
1541
|
+
keys), each cell's keys drawn uniformly; each cell's sampling variance uniform
|
|
1542
|
+
on [0.4, 1.4] (mean 0.8999 as handed to `fit`). Three arms in one call at the
|
|
1543
|
+
default 5 sweeps: x with every term at tau^2 = 4, y with every term at 0, z with
|
|
1544
|
+
"tree:1" and a at 0 beside "tree:2" at 4 and b at 8. 100 runs (seeds
|
|
1545
|
+
20 261 101 + run). Bound (asserted, in every run): a variance of at least 4
|
|
1546
|
+
times the mean sampling variance (and of its groups' own) within a factor of 2;
|
|
1547
|
+
a variance of 0 under a quarter of the mean sampling variance. Measured, the
|
|
1548
|
+
mean (min to max) of 100 fits:
|
|
1549
|
+
|
|
1550
|
+
| Term | x (truth 4) | y (truth 0) | z |
|
|
1551
|
+
|---|---|---|---|
|
|
1552
|
+
| tree:1 | 3.9072 (2.9473 to 4.5137) | 0.0010 (to 0.0118) | truth 0: 0.0173 (to 0.1126) |
|
|
1553
|
+
| tree:2 | 3.8028 (3.3739 to 4.2671) | 0.0049 (to 0.0387) | truth 4: 3.9761 (3.6161 to 4.3358) |
|
|
1554
|
+
| a | 4.0323 (3.3199 to 4.6050) | 0.0012 (to 0.0088) | truth 0: 0.0006 (to 0.0111) |
|
|
1555
|
+
| b | 3.9434 (2.8883 to 5.6817) | 0.0009 (to 0.0086) | truth 8: 7.9093 (6.0397 to 10.5169) |
|
|
1556
|
+
|
|
1557
|
+
The groups' own mean sampling variances: 0.7373 (x and z at "tree:1"), 0.0665
|
|
1558
|
+
(y at "tree:1"), 0.4038 ("tree:2"), 0.0499 (a) and 0.0250 (b). Two leaves a
|
|
1559
|
+
node, where the K - 1 in the moment matters most (bound: the mean of 100 runs
|
|
1560
|
+
within 5 % of the truth): "tree:1" truth 2.0, mean 2.0097 (1.6855 to 2.3061);
|
|
1561
|
+
"tree:2" truth 1.0, mean 1.0004 (0.7745 to 1.2209). Measured beside the shape
|
|
1562
|
+
and kept out of it: with one cell a leaf, a leaf's deviation and its cell's
|
|
1563
|
+
noise cannot be told apart, and at 5 sweeps a crossed grouping at 0 beside one
|
|
1564
|
+
at 8 reads about 0.6 to 1.2.
|
|
1565
|
+
|
|
1566
|
+
### The argmax (`argmax_accuracy_simulation_spec.rb`)
|
|
1567
|
+
|
|
1568
|
+
30 fixtures from round values (closed forms, parameters well below 1 at either
|
|
1569
|
+
end, U shapes, close competitors, up to 8 members, mixed widths, and sharp
|
|
1570
|
+
members with alpha + beta from 10^3 to 2 * 10^5), each with the epsilon `pick`
|
|
1571
|
+
explores at. `pick` takes its uniforms from one seeded Random a fixture, 10^4
|
|
1572
|
+
draws; the reference for `probabilities` (at epsilon 0) is the plain argmax of
|
|
1573
|
+
one `Sampler#beta` draw a member, 10^6 draws a fixture. Bound (asserted): every
|
|
1574
|
+
member within 5 standard deviations, sqrt(p * (1 - p) / n) with p the answer of
|
|
1575
|
+
`probabilities`. Measured, the largest gap of each fixture in standard
|
|
1576
|
+
deviations: `pick` at most 2.44, `probabilities` at most 2.43.
|
|
1577
|
+
|
|
1578
|
+
| Fixture | Members `[alpha, beta]` | `pick`'s epsilon | `pick` | `probabilities` (largest absolute gap) |
|
|
1579
|
+
|---|---|---|---|---|
|
|
1580
|
+
| 0 | `[[1, 1], [2, 1]]` | 0.0 | 0.47 | 1.41 (6.62e-04) |
|
|
1581
|
+
| 1 | `[[1, 1], [1, 1]]` | 0.0 | 0.80 | 0.44 (2.19e-04) |
|
|
1582
|
+
| 2 | `[[2, 8], [8, 2]]` | 0.1 | 0.51 | 0.18 (7.55e-06) |
|
|
1583
|
+
| 3 | `[[1, 0.05], [1, 1]]` | 0.0 | 0.57 | 0.78 (1.65e-04) |
|
|
1584
|
+
| 4 | `[[0.05, 1], [1, 1]]` | 0.0 | 1.02 | 0.51 (1.08e-04) |
|
|
1585
|
+
| 5 | `[[0.5, 0.5], [1, 1]]` | 0.0 | 0.74 | 0.18 (8.80e-05) |
|
|
1586
|
+
| 6 | `[[0.2, 0.3], [0.3, 0.2]]` | 0.05 | 0.02 | 0.87 (4.14e-04) |
|
|
1587
|
+
| 7 | `[[0.1, 0.1], [2, 2]]` | 0.0 | 1.44 | 0.61 (3.04e-04) |
|
|
1588
|
+
| 8 | `[[3, 7], [4, 6], [5, 5]]` | 0.0 | 0.83 | 1.34 (4.08e-04) |
|
|
1589
|
+
| 9 | `[[1, 9], [2, 8], [3, 7], [4, 6]]` | 0.05 | 1.66 | 0.35 (9.44e-05) |
|
|
1590
|
+
| 10 | `[[2, 3], [3, 2], [5, 5], [1, 1], [4, 6]]` | 0.1 | 2.44 | 1.65 (6.99e-04) |
|
|
1591
|
+
| 11 | `[[12, 88], [15, 85], [9, 91], [20, 180], [3, 17]]` | 0.1 | 1.00 | 0.66 (3.22e-04) |
|
|
1592
|
+
| 12 | `[[0.5, 50], [50, 0.5]]` | 0.0 | 0.00 | 0.00 (1.24e-31) |
|
|
1593
|
+
| 13 | `[[10, 10], [10.5, 9.5]]` | 0.0 | 1.03 | 0.65 (3.25e-04) |
|
|
1594
|
+
| 14 | `[[1, 3], [3, 1], [2, 2]]` | 0.2 | 0.82 | 1.15 (4.77e-04) |
|
|
1595
|
+
| 15 | `[[0.8, 0.8], [0.9, 0.7], [0.7, 0.9]]` | 0.0 | 1.29 | 0.89 (4.41e-04) |
|
|
1596
|
+
| 16 | `[[3, 7], [3.1, 6.9], [2.9, 7.1], [3.05, 6.95]]` | 0.1 | 1.90 | 2.01 (8.67e-04) |
|
|
1597
|
+
| 17 | `[[1, 1], [1, 1], [1, 1], [1, 1], [1, 1], [1, 1]]` | 0.0 | 1.38 | 2.42 (9.01e-04) |
|
|
1598
|
+
| 18 | `[[5, 1], [1, 5]]` | 0.0 | 0.59 | 0.38 (2.37e-05) |
|
|
1599
|
+
| 19 | `[[100, 900], [1, 9]]` | 0.0 | 0.71 | 1.45 (7.06e-04) |
|
|
1600
|
+
| 20 | `[[2, 2], [2, 2], [2, 2], [3, 3]]` | 0.3 | 1.45 | 2.29 (1.00e-03) |
|
|
1601
|
+
| 21 | `[[1, 2], [2, 3], [3, 4], [4, 5], [5, 6], [6, 7], [7, 8], [8, 9]]` | 0.05 | 1.15 | 2.43 (8.03e-04) |
|
|
1602
|
+
| 22 | `[[1, 2], [2, 1], [1.5, 1.5]]` | 0.1 | 1.24 | 1.07 (4.78e-04) |
|
|
1603
|
+
| 23 | `[[4, 4], [4.4, 3.6]]` | 0.0 | 0.45 | 1.53 (7.54e-04) |
|
|
1604
|
+
| 24 | `[[0.3, 3], [3, 0.3], [1, 1]]` | 0.0 | 1.08 | 1.94 (5.38e-04) |
|
|
1605
|
+
| 25 | `[[700, 300], [7000, 3000]]` | 0.1 | 1.05 | 0.88 (4.39e-04) |
|
|
1606
|
+
| 26 | `[[20000, 80000], [20, 80]]` | 0.0 | 0.85 | 0.03 (1.52e-05) |
|
|
1607
|
+
| 27 | `[[50, 99950], [1, 999]]` | 0.0 | 0.55 | 1.40 (6.83e-04) |
|
|
1608
|
+
| 28 | `[[99950, 50], [999, 1]]` | 0.0 | 0.34 | 0.31 (1.52e-04) |
|
|
1609
|
+
| 29 | `[[60000, 140000], [60150, 139850]]` | 0.1 | 0.69 | 0.64 (2.94e-04) |
|
|
1610
|
+
|
|
1611
|
+
### The change test (`change_test_simulation_spec.rb`)
|
|
1612
|
+
|
|
1613
|
+
Two settings, each a stated false-alarm rate and a horizon. False alarms: one
|
|
1614
|
+
null stream of 2,000 consecutive spans of 365 steps (730,000 steps, z ~ N(0, 1)
|
|
1615
|
+
by `Sampler#gaussian`), the statistic starting again from 0 after each alarm.
|
|
1616
|
+
Detection: 2,000 runs shifted by the smallest detectable change from their
|
|
1617
|
+
first step, and the same change after 365 null steps. Bound (asserted): the
|
|
1618
|
+
false alarms inside the stated rate's 95 % binomial interval, and the change
|
|
1619
|
+
caught within the horizon in at least 1,000 of 2,000 runs, both ways.
|
|
1620
|
+
|
|
1621
|
+
| Reference | Stated rate | Threshold | False alarms in 730,000 steps (95 % interval) | Per 365 steps | Horizon | Smallest change | Caught from the start | Caught after 365 null steps |
|
|
1622
|
+
|---|---|---|---|---|---|---|---|---|
|
|
1623
|
+
| 0.5 | 1 in 365 | 4.0744 | 1,966 (1,913 to 2,088) | 0.983 | 30 | 0.4752 | 1,239 (0.620) | 1,322 (0.661) |
|
|
1624
|
+
| 0.25 | 1 in 1,460 | 9.3147 | 521 (457 to 544) | 0.260 | 60 | 0.3532 | 1,264 (0.632) | 1,305 (0.652) |
|
|
1625
|
+
|
|
1626
|
+
Measured beside it: 2,000 separate spans, each starting at 0, count 1.2 % fewer
|
|
1627
|
+
alarms at reference 0.5 and 3.9 % fewer at 0.25, because every span's first
|
|
1628
|
+
steps start from 0.
|
|
1629
|
+
|
|
1630
|
+
### The order rule at small-site traffic (`order_simulation_spec.rb`)
|
|
1631
|
+
|
|
1632
|
+
A list of K items (3, 5 or 10), each with its own true rate, fixed for a run and
|
|
1633
|
+
dealt to the items in a random order a run. Every visit is shown the whole list
|
|
1634
|
+
in one order and sees it down to a depth drawn from a reach curve, the share of
|
|
1635
|
+
visits that see position l: (1, 0.6, 0.35) at K = 3,
|
|
1636
|
+
(1, 0.70, 0.50, 0.35, 0.25) at K = 5 and 0.8^l at K = 10. Each item a visit
|
|
1637
|
+
sees yields an outcome at its rate: yes/no, or an amount (log-normal, mean 200,
|
|
1638
|
+
sigma 0.9, capped at 2 000). The caller's whole part: a view counts only where
|
|
1639
|
+
an item was seen; every hour the set is rebuilt from the tallies
|
|
1640
|
+
(`Posteriors.yes_no` on the flat prior, or `Posteriors.amounts` under
|
|
1641
|
+
`Prior.amount(mean: 4.0, variance: 2_000.0, strength: 20.0)`), and every visit
|
|
1642
|
+
of that hour is shown the set's `rank(rng:, shuffle: 0.10)`. 28 days of 24
|
|
1643
|
+
hours, 50 seeded runs a cell (`Random.new(20 261 102 + 50 i + r)` for run r of
|
|
1644
|
+
cell i).
|
|
1645
|
+
|
|
1646
|
+
Measured: lost, the expected outcomes the shown orders missed against the best
|
|
1647
|
+
order (the items by true rate), as a share of the best order's, and the same for
|
|
1648
|
+
a fixed order (one random order a run shown to every visit); the posterior-mean
|
|
1649
|
+
order (the items by `summary(among: [])`'s means) on days 7 and 28, whether its
|
|
1650
|
+
first item is the best one and whether it is the best order exactly. Mixed
|
|
1651
|
+
visits: half shallow (reach (1, 0.5, 0.3, 0.15, 0.1), every rate halved), half
|
|
1652
|
+
deep (reach (1, 0.9, 0.7, 0.55, 0.4), every rate times 1.5), the known weakness
|
|
1653
|
+
of counting views only where seen, as an item shown low is seen mostly by deep
|
|
1654
|
+
visits and reads a rate above the one it would show on top. Bound (asserted,
|
|
1655
|
+
every cell): lost at most half the fixed order's, and the random share within 4
|
|
1656
|
+
standard errors of 0.10 (measured 0.0990 to 0.1003); each cell's own bounds
|
|
1657
|
+
below. The reference is the same model run 100 times a cell in another language
|
|
1658
|
+
and generator, the figures the bounds were set from with margin (fixed lost /
|
|
1659
|
+
lost / best on top day 7 / day 28 / exact order day 28); 50 runs and another
|
|
1660
|
+
generator read near them, not equal to them.
|
|
1661
|
+
|
|
1662
|
+
| Rates (true) | Visits a day | Fixed order lost | Lost | Best on top, day 7 | Best on top, day 28 | Exact order, day 28 | Bound (asserted) | Reference (100 runs) |
|
|
1663
|
+
|---|---|---|---|---|---|---|---|---|
|
|
1664
|
+
| five, close: 5, 4.5, 4, 3.5, 3 % | 300 | 8.0 % | 2.4 % | 74 % | 88 % | 62 % | day 28 >= 75 %, exact >= 35 %, lost <= 4.5 % | 8.3 / 2.8 / 69 / 90 / 54 |
|
|
1665
|
+
| five, close | 1 000 | 7.4 % | 1.6 % | 82 % | 100 % | 94 % | day 28 >= 90 %, exact >= 75 %, lost <= 3.0 % | 7.6 / 1.6 / 91 / 99 / 91 |
|
|
1666
|
+
| five, close | 3 000 | 7.4 % | 1.1 % | 94 % | 100 % | 100 % | day 28 >= 90 %, exact >= 90 %, lost <= 2.0 % | 7.7 / 1.1 / 99 / 100 / 100 |
|
|
1667
|
+
| five, clear: 8, 6, 5, 4, 3 % | 300 | 14.1 % | 2.5 % | 98 % | 100 % | 86 % | day 7 >= 90 % | 12.8 / 2.4 / 100 / 100 / 97 |
|
|
1668
|
+
| five, clear | 1 000 | 14.3 % | 1.7 % | 100 % | 100 % | 100 % | day 7 >= 90 % | 12.8 / 1.7 / 100 / 100 / 100 |
|
|
1669
|
+
| five, clear | 3 000 | 13.9 % | 1.5 % | 100 % | 100 % | 100 % | day 7 >= 90 % | 13.6 / 1.5 / 100 / 100 / 100 |
|
|
1670
|
+
| three: 5, 4, 3 % | 1 000 | 7.7 % | 1.1 % | 98 % | 100 % | 100 % | day 28 >= 90 % | 7.0 / 1.1 / 100 / 100 / 100 |
|
|
1671
|
+
| ten: 6 % down to 1.5 % by 0.5 | 3 000 | 19.6 % | 2.2 % | 100 % | 100 % | 100 % | day 28 >= 90 % | 18.6 / 2.2 / 98 / 100 / 100 |
|
|
1672
|
+
| five amounts: 4, 3, 2.5, 2, 1.5 % | 300 | 13.8 % | 4.3 % | 82 % | 100 % | 68 % | day 28 >= 85 %, lost <= 6 % | 13.1 / 4.3 / 88 / 98 / 53 |
|
|
1673
|
+
| five amounts | 1 000 | 13.7 % | 2.6 % | 92 % | 100 % | 84 % | day 28 >= 95 %, lost <= 4 % | 12.4 / 2.6 / 98 / 100 / 88 |
|
|
1674
|
+
| five amounts | 3 000 | 12.6 % | 1.9 % | 100 % | 100 % | 100 % | day 28 >= 95 %, lost <= 3 % | 12.9 / 1.8 / 100 / 100 / 98 |
|
|
1675
|
+
| five, close, mixed visits | 300 | 6.9 % | 2.3 % | 78 % | 94 % | 70 % | day 28 >= 75 % | 6.4 / 2.4 / - / 90 / 58 |
|
|
1676
|
+
| five, close, mixed visits | 1 000 | 5.5 % | 1.5 % | 82 % | 98 % | 96 % | day 28 >= 90 % | 6.2 / 1.5 / - / 99 / 95 |
|
|
1677
|
+
|
|
1678
|
+
At 300 visits a day the order rule loses a fifth to a third of what a fixed
|
|
1679
|
+
order loses, and the posterior-mean order finds the best item first in
|
|
1680
|
+
most runs by day 28; at 1 000 and more it finds the best order exactly in most.
|
|
1681
|
+
The mixed cells hold their bounds, though the deep visits flatter the items
|
|
1682
|
+
shown low.
|
|
1683
|
+
|
|
1684
|
+
### The odds against the rule's visits (`odds_simulation_spec.rb`)
|
|
1685
|
+
|
|
1686
|
+
Visits: 2 * 10^5 calls of `rank(rng:)` a fixture on one seeded Random (seed
|
|
1687
|
+
20 261 131); every head and every position against its exact value
|
|
1688
|
+
(`pick_probabilities`, `positions`), and the orders by one chi-square test
|
|
1689
|
+
against `order_probability` (each order with n p >= 5 its own bin, the rest
|
|
1690
|
+
pooled into one). Bound (asserted): every head and position within 4 standard
|
|
1691
|
+
errors, sqrt(p (1 - p) / n), the chi-square's p-value above 0.001, and no order
|
|
1692
|
+
seen that the exact odds give no probability. Items a, b, c, ... in the order
|
|
1693
|
+
their pairs are listed; slots count from 0.
|
|
1694
|
+
|
|
1695
|
+
| Fixture | Rule | Largest gap (standard errors) | Orders' chi-square | p |
|
|
1696
|
+
|---|---|---|---|---|
|
|
1697
|
+
| five Betas: [0.02, 19.98], [3, 60], [4, 70], [6, 95], [2, 45] | shuffle 0.1, explore 0.1 | 2.75 (d at slot 2) | 137.5 on 119 degrees | 0.1186 |
|
|
1698
|
+
| four Betas: [0.3, 20], [1, 20], [2, 30], [5, 90]; views 50, 400, 400, 400 | explore 0.2 among items of at least 100 views | 1.12 (a at slot 1) | 15.8 on 23 | 0.8656 |
|
|
1699
|
+
| four Normals near 5 a view: [5.0, 0.4], [5.3, 0.5], [4.8, 0.3], [5.1, 0.8] | shuffle 0.05 | 1.71 (c at slot 1) | 23.3 on 23 | 0.4457 |
|
|
1700
|
+
| the four Betas above, c pinned at slot 2 | shuffle 0.1 | 1.33 (a at slot 3) | 2.3 on 5 | 0.8086 |
|
|
1701
|
+
| five Normals of mixed widths: [1.0, 1.0], [1.5, 0.5], [0.5, 2.0], [1.2, 0.2], [0.8, 0.7] | explore 0.1, the top 2 | 1.34 (e as head) | 12.2 on 19 | 0.8781 |
|
|
1702
|
+
|
|
1703
|
+
Joint draws: 10^6 a fixture (seed 20 261 141), one draw an item, in order; how
|
|
1704
|
+
often each item is the largest against `summary`'s p_best (and, for Normals,
|
|
1705
|
+
`NormalArgmax.probabilities`, which read the same), and the mean of max - X_i
|
|
1706
|
+
against its expected loss. Bound (asserted): each within 4 standard errors of
|
|
1707
|
+
the draws. Measured, the gap of each item in standard errors:
|
|
1708
|
+
|
|
1709
|
+
| Fixture | P(best) | Expected loss |
|
|
1710
|
+
|---|---|---|
|
|
1711
|
+
| the four Normals near 5 a view | 1.60, 0.28, 1.57, 0.82 | 0.87, 0.74, 0.46, 1.34 |
|
|
1712
|
+
| the five Normals of mixed widths | 0.41, 1.14, 0.79, 1.30, 0.01 | 1.20, 0.44, 0.55, 0.04, 0.39 |
|
|
1713
|
+
| four Betas: [12, 88], [15, 85], [9, 91], [3, 17] | 1.37, 1.07, 0.37, 0.24 | 0.26, 0.18, 0.21, 0.31 |
|
|
1714
|
+
| the four Betas with an alpha of 0.3 | 1.44, 1.16, 1.10, 1.61 | 1.20, 0.85, 1.44, 0.76 |
|
|
1715
|
+
|
|
1716
|
+
### Run time
|
|
1717
|
+
|
|
1718
|
+
On one development machine, Ruby 3.4.9 with YJIT, in seconds, both columns
|
|
1719
|
+
measured the same way: each figure is the sum of a file's examples' own run
|
|
1720
|
+
times as RSpec reports them, Ruby's boot and the file's load left out. 0.3.0 ran
|
|
1721
|
+
on 2026-10-31 with the machine otherwise idle, 0.4.0 on 2026-11-02 while other
|
|
1722
|
+
work kept it busy (a load average of 31 to 46), so the two columns differ by the
|
|
1723
|
+
load as well as by the code.
|
|
1724
|
+
|
|
1725
|
+
| File | Examples | 0.3.0 | 0.4.0 |
|
|
1726
|
+
|---|---|---|---|
|
|
1727
|
+
| `allocator_simulation_spec.rb` | 2 | 150 | 173 |
|
|
1728
|
+
| `win_hold_simulation_spec.rb` | 1 | 132 | 160 |
|
|
1729
|
+
| `value_allocation_simulation_spec.rb` | 6 | 56 | 85 |
|
|
1730
|
+
| `two_layer_simulation_spec.rb` | 3 | 68 | 96 |
|
|
1731
|
+
| `tree_recovery_simulation_spec.rb` | 3 | 178 | 515 |
|
|
1732
|
+
| `contrasts_simulation_spec.rb` | 2 | 17 | 19 |
|
|
1733
|
+
| `argmax_accuracy_simulation_spec.rb` | 2 | 232 | 265 |
|
|
1734
|
+
| `change_test_simulation_spec.rb` | 4 | 1 | 1 |
|
|
1735
|
+
| `odds_simulation_spec.rb` | 3 | - | 33 |
|
|
1736
|
+
| `order_simulation_spec.rb` | 13 | - | 667 |
|
|
1737
|
+
| All | 39 | 833 (about 14 minutes) | 2,013 (about 34 minutes) |
|
|
1738
|
+
|
|
1739
|
+
In three other runs the order file alone took 263, 311 and 407 seconds, the
|
|
1740
|
+
more the busier the machine. Without YJIT the eight files of 0.3.0 took about 22
|
|
1741
|
+
minutes in earlier runs on the same machine. The budget is 25 minutes on CI's
|
|
1742
|
+
Ruby 3.4; the job's timeout is 45 minutes (30 before 0.4.0).
|
|
1743
|
+
|
|
1744
|
+
A change that moves any of these counts changes the rule's behaviour; measure
|
|
1745
|
+
again and update these figures in the same change.
|
|
1746
|
+
|
|
153
1747
|
## Development
|
|
154
1748
|
|
|
155
1749
|
```sh
|
|
@@ -157,6 +1751,8 @@ bundle install
|
|
|
157
1751
|
bundle exec rake spec # run the specs
|
|
158
1752
|
bundle exec rake rubocop # lint
|
|
159
1753
|
bundle exec rake # both
|
|
1754
|
+
SIMULATIONS=1 bundle exec rspec spec/simulations # the bandit simulations (minutes)
|
|
1755
|
+
RUBY_YJIT_ENABLE=1 SIMULATIONS=1 bundle exec rspec spec/simulations # as CI runs them, in about 60 % of the time
|
|
160
1756
|
```
|
|
161
1757
|
|
|
162
1758
|
## License
|