bandit_machines 0.7.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +95 -0
- data/Cargo.lock +321 -0
- data/Cargo.toml +29 -0
- data/LICENSE.txt +21 -0
- data/README.md +367 -0
- data/ext/bandit_machines_native/core/CHANGELOG.md +70 -0
- data/ext/bandit_machines_native/core/Cargo.toml +21 -0
- data/ext/bandit_machines_native/core/README.md +152 -0
- data/ext/bandit_machines_native/core/src/bandit.rs +576 -0
- data/ext/bandit_machines_native/core/src/lib.rs +286 -0
- data/ext/bandit_machines_native/core/src/math.rs +66 -0
- data/ext/bandit_machines_native/core/src/pick.rs +71 -0
- data/ext/bandit_machines_native/extconf.rb +2 -0
- data/ext/bandit_machines_native/ffi/Cargo.toml +21 -0
- data/ext/bandit_machines_native/ffi/extconf.rb +14 -0
- data/ext/bandit_machines_native/ffi/src/convert.rs +238 -0
- data/ext/bandit_machines_native/ffi/src/init.rs +128 -0
- data/ext/bandit_machines_native/ffi/src/lib.rs +246 -0
- data/lib/bandit_machines/version.rb +5 -0
- data/lib/bandit_machines.rb +74 -0
- data/sig/bandit_machines.rbs +157 -0
- metadata +87 -0
data/README.md
ADDED
|
@@ -0,0 +1,367 @@
|
|
|
1
|
+
# BanditMachines
|
|
2
|
+
|
|
3
|
+
> You don't know which one is best. You still have to pick one now. 🎰
|
|
4
|
+
|
|
5
|
+
A multi-armed bandit for Ruby. You have a handful of options and no idea which pays best. Ask the bandit which to try, and it hands you an arm and a ticket. When you find out how it went, you hand the ticket back with a reward. It balances trying the arm that looks best against trying the ones it knows too little about, with UCB1 or epsilon-greedy, and it refuses a reward that shows up for an arm you've since reset.
|
|
6
|
+
|
|
7
|
+
Pre-1.0: breaking changes land in minor versions. It is a policy core, not an experiment platform: it picks arms and counts rewards, and everything else is yours.
|
|
8
|
+
|
|
9
|
+
## Quick Start
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
gem 'bandit_machines'
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
```ruby
|
|
16
|
+
VARIANTS = %i[blue_button green_button orange_button].freeze
|
|
17
|
+
bandit = BanditMachines::Bandit.new(BanditMachines::Config.new(1, 0, :lowest_index))
|
|
18
|
+
VARIANTS.each_index { |arm| bandit.open(arm) } # every arm starts closed
|
|
19
|
+
|
|
20
|
+
def show_checkout(bandit, session)
|
|
21
|
+
selection = bandit.select_ucb1(0)
|
|
22
|
+
return render(VARIANTS.first) if selection.is_a?(Symbol) # :no_open_arm, :tickets_full and friends
|
|
23
|
+
|
|
24
|
+
session[:ticket] = selection.ticket
|
|
25
|
+
render(VARIANTS[selection.arm])
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
def checkout_finished(bandit, session, purchased)
|
|
29
|
+
bandit.reward(session[:ticket], purchased ? 1 : 0)
|
|
30
|
+
end
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
`Bandit.new(config)` is scope zero. Two bandits built with `new` are the same scope and issue the same tickets, so one will count the other's rewards. Run more than one and give each its own scope:
|
|
34
|
+
|
|
35
|
+
```ruby
|
|
36
|
+
checkout = BanditMachines::Bandit.with_scope(BanditMachines::Config.new(1, 0, :lowest_index), 1)
|
|
37
|
+
banner = BanditMachines::Bandit.with_scope(BanditMachines::Config.new(1, 0, :lowest_index), 2)
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
One bandit, two ways to select, one way to report. A `Config` is the core's: `max_reward`, `prior`, `tie_break`, in that order, and `bandit.config` hands it back. A new bandit has every arm closed and selects nothing until you open some. Arms are indices, rewards are Integers from `0` to `max_reward`, and any randomness is a number you pass in.
|
|
41
|
+
|
|
42
|
+
## A Message to the Resistance
|
|
43
|
+
|
|
44
|
+
So you have three checkout buttons and a meeting where someone said "let's A/B test it." Two weeks at a third of the traffic each. For two weeks, two thirds of your customers get a button that, by the end, you'll know was worse. Then someone reads a p-value off a dashboard, ships the winner, and the test is over forever, including the day the winner stops winning.
|
|
45
|
+
|
|
46
|
+
Or you skip the meeting and write `variants.max_by { |v| v.conversions.fdiv(v.views) }`. The first variant to get a lucky sale wins every request from then on, and the other two never get a view to prove otherwise.
|
|
47
|
+
|
|
48
|
+
Or you write `rand < 0.1 ? variants.sample : best`. Now it works, and nobody can reproduce what it did on Tuesday, and the reward for a page shown before the redesign gets credited to the page that replaced it.
|
|
49
|
+
|
|
50
|
+
The piece you keep rebuilding is small: counts and sums per option, a rule for picking, and a way to say which pick a result belongs to. That piece is this gem.
|
|
51
|
+
|
|
52
|
+
## Chapter 1: Stardate 2026.316, the AMS Robbins
|
|
53
|
+
|
|
54
|
+
> **AMS Robbins**, NCC-4227, *Robbins*-class exploration scout.
|
|
55
|
+
> Every jump spent on a known system is a jump not spent finding a better one.
|
|
56
|
+
|
|
57
|
+
The scout doesn't mine anything. It decides where the next probe goes. Each system it could visit has a record: how many probes went out, how much they brought back, how many haven't reported yet. A system with a thin record gets the benefit of the doubt. A system with a thick record gets judged on it.
|
|
58
|
+
|
|
59
|
+
On a bridge somewhere, a cadet volunteers:
|
|
60
|
+
|
|
61
|
+
```ruby
|
|
62
|
+
def next_system
|
|
63
|
+
@systems.max_by { |s| s.ore.fdiv(s.probes) }
|
|
64
|
+
end
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
"That," says the scout's navigator, "sends the whole fleet to the first rock that glittered."
|
|
68
|
+
|
|
69
|
+
## The Flight Plan
|
|
70
|
+
|
|
71
|
+
Each ticket the bandit issues is in one of these states:
|
|
72
|
+
|
|
73
|
+
```mermaid
|
|
74
|
+
stateDiagram-v2
|
|
75
|
+
[*] --> outstanding: select, returns a Selection
|
|
76
|
+
outstanding --> counted: reward(ticket, value)
|
|
77
|
+
outstanding --> returned: abandon(ticket)
|
|
78
|
+
outstanding --> returned: revoke_before(serial) / revoke_all
|
|
79
|
+
outstanding --> void: reset(arm) / reset_all
|
|
80
|
+
counted --> [*]
|
|
81
|
+
returned --> [*]
|
|
82
|
+
void --> [*]
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
And the afternoon the green button was redesigned while a customer was still at the till:
|
|
86
|
+
|
|
87
|
+
```mermaid
|
|
88
|
+
sequenceDiagram
|
|
89
|
+
participant C as Checkout
|
|
90
|
+
participant B as Bandit
|
|
91
|
+
participant O as Operator
|
|
92
|
+
|
|
93
|
+
C->>B: select_ucb1
|
|
94
|
+
B-->>C: Selection(arm 1, ticket, generation 0)
|
|
95
|
+
Note over C: customer is still typing a card number
|
|
96
|
+
|
|
97
|
+
O->>B: reset(1)
|
|
98
|
+
B-->>O: 1 ticket voided
|
|
99
|
+
Note over B: arm 1 is generation 1, cold
|
|
100
|
+
|
|
101
|
+
C->>B: reward(ticket, 1)
|
|
102
|
+
B-->>C: :stale_ticket
|
|
103
|
+
Note over B: the new green button owes nothing to the old one
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
## How the Bandit Works
|
|
107
|
+
|
|
108
|
+
### Arms start closed
|
|
109
|
+
|
|
110
|
+
```ruby
|
|
111
|
+
bandit = BanditMachines::Bandit.new(BanditMachines::Config.new(10, 5, :lowest_index))
|
|
112
|
+
bandit.stats(0).open # => false
|
|
113
|
+
bandit.select_ucb1(0) # => :no_open_arm
|
|
114
|
+
|
|
115
|
+
bandit.open(0) # => :opened
|
|
116
|
+
bandit.open(1) # => :opened
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
The bandit has 16 arms and a new one has all of them closed. It explores only what you open, so nothing is ever selected that you didn't name. Until you open an arm, both selections answer `:no_open_arm` and issue no ticket.
|
|
120
|
+
|
|
121
|
+
### Select with UCB1
|
|
122
|
+
|
|
123
|
+
```ruby
|
|
124
|
+
bandit = BanditMachines::Bandit.new(BanditMachines::Config.new(10, 5, :lowest_index))
|
|
125
|
+
2.times { |arm| bandit.open(arm) }
|
|
126
|
+
|
|
127
|
+
first = bandit.select_ucb1(0)
|
|
128
|
+
first.arm # => 0
|
|
129
|
+
first.reason # => :cold_start
|
|
130
|
+
first.ticket # => #<BanditMachines::Ticket>
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
The argument is `pick`, a draw from `0..2**64-1`, and it has no default. UCB1 reads it only to break a tie under `tie_break: :draw`; with any other rule a constant such as `0` is fine. A `Selection` holds the `ticket` and the `reason`; `selection.arm` is the ticket's arm.
|
|
134
|
+
|
|
135
|
+
An arm nobody has tried is *cold*, and cold arms go first, one per call. Once every arm has been tried, UCB1 scores each as `mean / max_reward + sqrt(2 * ln(total tries) / tries)` and takes the highest. A well-tried arm is judged on its mean. A neglected one gets a bonus that grows until it earns another look.
|
|
136
|
+
|
|
137
|
+
```ruby
|
|
138
|
+
bandit.reward(first.ticket, 0) # arm 0 paid nothing
|
|
139
|
+
bandit.reward(bandit.select_ucb1(0).ticket, 10) # arm 1 paid everything
|
|
140
|
+
|
|
141
|
+
third = bandit.select_ucb1(0)
|
|
142
|
+
third.arm # => 1
|
|
143
|
+
third.reason # => :upper_bound
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
The score is computed with integers only, so it's the same on every machine. `ucb1_index_q56(arm)` shows it, scaled by `2**56`. It is never above the true value and less than `2**-26` below it.
|
|
147
|
+
|
|
148
|
+
### Select with epsilon-greedy
|
|
149
|
+
|
|
150
|
+
```ruby
|
|
151
|
+
epsilon = BanditMachines::Ratio.new(1, 10) # numerator, denominator: explore one time in ten
|
|
152
|
+
draws = BanditMachines::Draws.new(rand(2**64), rand(2**64)) # gate: explore or exploit; pick: which arm
|
|
153
|
+
selection = bandit.select_epsilon_greedy(epsilon, draws)
|
|
154
|
+
selection.reason # => :explore or :exploit
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
`Ratio` and `Draws` are the core's two argument types, passed in the core's order. A `Ratio` is `numerator` over `denominator`, between zero and one; `Ratio::ZERO` and `Ratio::ONE` are ready-made. The bandit has no random number generator. You supply two draws from `0..2**64-1`. It explores when `gate * denominator / 2**64` (rounded down) is below `numerator`, and then takes open arm number `pick * open_arms / 2**64`. Otherwise it takes the highest mean, compared exactly. Log the two draws and you can replay the decision.
|
|
158
|
+
|
|
159
|
+
Cold arms still go first, whatever the gate says.
|
|
160
|
+
|
|
161
|
+
### Reward
|
|
162
|
+
|
|
163
|
+
```ruby
|
|
164
|
+
bandit.reward(selection.ticket, 7) # => :rewarded
|
|
165
|
+
bandit.reward(selection.ticket, 7) # => :unknown_ticket, already spent
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
A ticket is good for one reward. A reward above `max_reward` answers `:reward_out_of_range` and leaves the ticket usable, so you can fix the value and try again.
|
|
169
|
+
|
|
170
|
+
### Abandon
|
|
171
|
+
|
|
172
|
+
A selection that will never have a result gets handed back. It's as if it never happened:
|
|
173
|
+
|
|
174
|
+
```ruby
|
|
175
|
+
selection = bandit.select_ucb1(0)
|
|
176
|
+
bandit.abandon(selection.ticket) # => :abandoned
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
### Waiting counts
|
|
180
|
+
|
|
181
|
+
Tickets that are out count as tries. Ask three times before any reward comes back and you get three different cold arms, not the same one three times. An arm with tickets out and no reward yet competes using `prior` as its mean.
|
|
182
|
+
|
|
183
|
+
```ruby
|
|
184
|
+
bandit = BanditMachines::Bandit.new(BanditMachines::Config.new(10, 5, :lowest_index))
|
|
185
|
+
3.times { |arm| bandit.open(arm) }
|
|
186
|
+
3.times.map { bandit.select_ucb1(0).arm } # => [0, 1, 2]
|
|
187
|
+
bandit.stats(0).pending # => 1
|
|
188
|
+
bandit.stats(0).tries # => 1
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
### Ties
|
|
192
|
+
|
|
193
|
+
When two arms score exactly the same, the config's `tie_break` decides. It is required: there is no default, and leaving it out raises Ruby's own `ArgumentError` for the missing argument. An unknown rule such as `:coin` raises `RangeError`.
|
|
194
|
+
|
|
195
|
+
```ruby
|
|
196
|
+
BanditMachines::Config.new(1, 0, :draw)
|
|
197
|
+
BanditMachines::Config.new(1, 0) # => ArgumentError, missing keyword: :tie_break
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
`:lowest_index`, `:highest_index`, `:least_tried` (fewest tries, then lowest index), or `:draw`, which spreads the tie evenly using your `pick`.
|
|
201
|
+
|
|
202
|
+
### Reset
|
|
203
|
+
|
|
204
|
+
```ruby
|
|
205
|
+
bandit = BanditMachines::Bandit.new(BanditMachines::Config.new(10, 5, :lowest_index))
|
|
206
|
+
2.times { |arm| bandit.open(arm) }
|
|
207
|
+
ticket = bandit.select_ucb1(0).ticket
|
|
208
|
+
|
|
209
|
+
bandit.reset(0) # => 1, one ticket voided
|
|
210
|
+
bandit.reward(ticket, 10) # => :stale_ticket
|
|
211
|
+
bandit.stats(0).generation # => 1
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
Reset is for when an arm is no longer the thing it was. Its counts go to zero, its generation goes up, and every ticket issued before the reset is refused. `reset_all` does it to every arm.
|
|
215
|
+
|
|
216
|
+
### Decay
|
|
217
|
+
|
|
218
|
+
```ruby
|
|
219
|
+
bandit.decay(0, BanditMachines::Ratio.new(1, 2)) # => :decayed, arm 0 keeps half its history
|
|
220
|
+
bandit.decay_all(BanditMachines::Ratio.new(9, 10)) # => :decayed
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
Decay is for when the world drifts and old results should weigh less. The argument is `keep`, a `Ratio`. The reward count is multiplied by it and rounded up, the sum is rounded down. Tickets are untouched: a reward that arrives after a decay still counts in full. Nothing decays unless you call it.
|
|
224
|
+
|
|
225
|
+
### Open and close
|
|
226
|
+
|
|
227
|
+
```ruby
|
|
228
|
+
bandit.close(1) # => :closed, arm 1 is no longer selected
|
|
229
|
+
bandit.close(1) # => :already_closed, nothing changed
|
|
230
|
+
bandit.open(1) # => :opened, and now it is again
|
|
231
|
+
bandit.open(1) # => :already_open, nothing changed
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
`open` answers `:opened` or `:already_open`; `close` answers `:closed` or `:already_closed`. All four are honoured calls, none is a refusal: asking for what already holds is a repeat that changes nothing.
|
|
235
|
+
|
|
236
|
+
Opening is always explicit: the constructor opens nothing. A closed arm keeps its history and still accepts rewards for tickets issued before it closed. Wire this to whatever decides an option is available today.
|
|
237
|
+
|
|
238
|
+
### An arm that isn't there
|
|
239
|
+
|
|
240
|
+
```ruby
|
|
241
|
+
bandit.open(16) # => :unknown_arm
|
|
242
|
+
bandit.reset(99) # => :unknown_arm
|
|
243
|
+
bandit.stats(16) # => nil
|
|
244
|
+
bandit.ucb1_index_q56(16) # => nil
|
|
245
|
+
bandit.open(-1) # => RangeError
|
|
246
|
+
```
|
|
247
|
+
|
|
248
|
+
There are 16 arms, `0...16`. An index that is a valid Integer but names no arm is the core's `Refusal::UnknownArm`: `open`, `close`, `reset` and `decay` answer `:unknown_arm` and change nothing, and the two readers answer `nil`. Only an index that could not be an index at all raises: a non-Integer is a `TypeError`, a negative one or one past `2**64-1` a `RangeError`.
|
|
249
|
+
|
|
250
|
+
### Revoke
|
|
251
|
+
|
|
252
|
+
```ruby
|
|
253
|
+
watermark = bandit.next_serial
|
|
254
|
+
# ... an hour later, by your clock ...
|
|
255
|
+
bandit.revoke_before(watermark) # => how many tickets were abandoned
|
|
256
|
+
bandit.revoke_all # => every ticket still out, no serial needed
|
|
257
|
+
```
|
|
258
|
+
|
|
259
|
+
Every ticket has a `serial`, counting up from 1. The bandit has no clock, so expiring old selections is yours: remember a serial, and later revoke everything before it. A serial carries its bandit's scope (below), so one taken from a bandit in another scope answers `:unknown_ticket` and revokes nothing. `next_serial` is `nil` once a bandit has issued all 2**48-1 serials of its scope.
|
|
260
|
+
|
|
261
|
+
To call everything back, use `revoke_all`, not a big serial: `2**64-1` belongs to scope 65535 and is refused everywhere else, and no serial sits above the last ticket of a scope. `revoke_all` never refuses, answers how many tickets it abandoned, and leaves every arm's counts and generation alone. `reset_all` also voids every ticket, but it forgets what the arms learned.
|
|
262
|
+
|
|
263
|
+
### Scopes
|
|
264
|
+
|
|
265
|
+
A ticket is only meaningful to the bandit that issued it. Run more than one bandit and each needs its own scope, an Integer in `0..65535` carried by every ticket it issues:
|
|
266
|
+
|
|
267
|
+
```ruby
|
|
268
|
+
config = BanditMachines::Config.new(1, 0, :lowest_index)
|
|
269
|
+
checkout = BanditMachines::Bandit.with_scope(config, 1)
|
|
270
|
+
banner = BanditMachines::Bandit.with_scope(config, 2)
|
|
271
|
+
[checkout, banner].each { |bandit| bandit.open(0) }
|
|
272
|
+
|
|
273
|
+
ticket = checkout.select_ucb1(0).ticket
|
|
274
|
+
banner.select_ucb1(0)
|
|
275
|
+
ticket.scope # => 1
|
|
276
|
+
ticket.serial # => 281474976710657, scope 1 in the top 16 bits, 1 below
|
|
277
|
+
banner.reward(ticket, 1) # => :unknown_ticket, not one of mine
|
|
278
|
+
checkout.reward(ticket, 1) # => :rewarded
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
`Bandit.new(config)` is scope zero, where serials are plain `1, 2, 3`. The scope is the bandit's identity, not policy, so it isn't part of `Config`: two bandits may share a config and must not share a scope. The bandit can't pick a scope for you: it reads no clock and rolls no dice. Leave two bandits in the same scope and they issue the same tickets, so one will happily count the other's reward.
|
|
282
|
+
|
|
283
|
+
A restarted bandit counts from one again, so a ticket that outlived the restart could redeem a new selection. Resume from a saved count with `Bandit.resumed(config, scope, bandit.issued)`, or give each incarnation a new scope. `resumed` brings back the serial sequence and nothing else: every arm is closed, nothing is outstanding, earlier tickets are refused, and an `issued` above 2**48-1 raises `RangeError`.
|
|
284
|
+
|
|
285
|
+
### Capacity
|
|
286
|
+
|
|
287
|
+
```ruby
|
|
288
|
+
BanditMachines::Bandit::ARMS # => 16, the core's Bandit::ARMS
|
|
289
|
+
BanditMachines::Bandit::TICKETS # => 64, the core's Bandit::TICKETS
|
|
290
|
+
|
|
291
|
+
bandit = BanditMachines::Bandit.new(BanditMachines::Config.new(1, 0, :lowest_index))
|
|
292
|
+
3.times { |arm| bandit.open(arm) }
|
|
293
|
+
64.times { bandit.select_ucb1(0) }
|
|
294
|
+
bandit.select_ucb1(0) # => :tickets_full
|
|
295
|
+
```
|
|
296
|
+
|
|
297
|
+
At most 64 selections can be waiting on a reward. Past that, selection is refused until a ticket is rewarded, abandoned, revoked or voided.
|
|
298
|
+
|
|
299
|
+
### The arithmetic, in the open
|
|
300
|
+
|
|
301
|
+
```ruby
|
|
302
|
+
math = BanditMachines::Math
|
|
303
|
+
math::ONE # => 72057594037927936, 1.0 in Q8.56
|
|
304
|
+
math.mean_q56(5, 1, 10) # => 36028797018963968, sum / (count * range) = 0.5
|
|
305
|
+
math.bonus_q56(100, 4) # => sqrt(2 * ln(100) / 4) in Q8.56
|
|
306
|
+
math.ln_q56(3) # => ln(3) in Q8.56, never above the true value
|
|
307
|
+
math.log2_q56(8) # => 216172782113783808, exactly 3.0
|
|
308
|
+
```
|
|
309
|
+
|
|
310
|
+
`BanditMachines::Math` is the core's `math` module, the integer routines the UCB1 index is built from: `ucb1_index_q56(arm)` is `mean_q56 + bonus_q56` of that arm's numbers.
|
|
311
|
+
|
|
312
|
+
## The Rules of the Scout
|
|
313
|
+
|
|
314
|
+
- **Everything is an Integer.** Rewards, `max_reward`, `prior` and ratio terms are `0..2**32-1`. Draws, serials and arm indices are `0..2**64-1`. Floats, nil, and strings raise `TypeError`, and so does an Array or a Hash where a `Config`, `Ratio` or `Draws` belongs. Out of range raises `RangeError`, and so do a zero `max_reward`, a `prior` above it, a ratio above one and an unknown `tie_break` or `reason` Symbol. Every value checks itself natively when it is built, by the gem or by hand: `Selection.new` needs a `Ticket` and one of the four reasons, `ArmStats.new` needs `true`/`false` and counts in `0..2**64-1`. An invalid one never exists. The value classes are sealed (subclassing raises `TypeError`, `allocate` is undefined); sealing stops accidents, not reflection, so a value the bandit reads must be an instance of exactly its class and its fields are checked again: a forged one raises `TypeError` or `RangeError` and never earns a refusal symbol. A missing argument is Ruby's own `ArgumentError`; the gem raises none.
|
|
315
|
+
- **Ruby mirrors Rust.** `Config`, `Ratio`, `Draws`, `Selection` and `ArmStats` have the fields of the core's structs, in order, and are passed positionally where the core takes them. `tie_break` and `reason` are the core's variants in snake case. A `Ticket` answers `arm`, `generation`, `serial` and `scope`.
|
|
316
|
+
- **Scopes.** A `Ticket` is only meaningful to the bandit that issued it. Two bandits built with `Bandit.new(config)` are both scope zero and alias: they issue the same tickets and honour each other's. Give each bandit its own scope with `Bandit.with_scope(config, scope)`; the core cannot pick one. A `Ticket` or serial from a bandit in another scope is refused with `:unknown_ticket` (not `:stale_ticket`, which means the arm was reset). Tickets can't be built or copied by hand.
|
|
317
|
+
- **Restarts.** Resume from a saved count with `Bandit.resumed(config, scope, issued)`, or give each incarnation a new scope. `resumed` restores the serial sequence only; reopening arms and what they learned is yours.
|
|
318
|
+
- **A call that raises changed nothing.** Every argument is checked before the core is asked, and every class was resolved once, when the extension loaded, so a `TypeError` or `RangeError` never leaves a ticket issued behind it. A selection is decided on a copy of the bandit, and the `Selection` built natively from that answer, without running any Ruby code; the copy is kept only once it exists.
|
|
319
|
+
- **Refusals don't raise, and don't change anything.** `:unknown_arm`, `:no_open_arm`, `:tickets_full`, `:exhausted`, `:stale_ticket`, `:unknown_ticket`, `:reward_out_of_range` and `:saturated` are answers. Only malformed input raises.
|
|
320
|
+
- **Every answer, by method.**
|
|
321
|
+
|
|
322
|
+
| Method | Honoured | Refused | Records the clock |
|
|
323
|
+
| --- | --- | --- | --- |
|
|
324
|
+
| `select_ucb1(pick)`, `select_epsilon_greedy(epsilon, draws)` | a `Selection` | `:no_open_arm`, `:tickets_full`, `:exhausted` | no |
|
|
325
|
+
| `reward(ticket, reward)` | `:rewarded` | `:stale_ticket`, `:unknown_ticket`, `:reward_out_of_range`, `:saturated` | no |
|
|
326
|
+
| `abandon(ticket)` | `:abandoned` | `:stale_ticket`, `:unknown_ticket` | no |
|
|
327
|
+
| `revoke_before(serial)` | the number of tickets revoked (Integer) | `:unknown_ticket` | no |
|
|
328
|
+
| `revoke_all` | the number of tickets revoked (Integer) | none | no |
|
|
329
|
+
| `reset(arm)` | the number of tickets voided (Integer) | `:unknown_arm`, `:exhausted` | no |
|
|
330
|
+
| `reset_all` | the number of tickets voided (Integer) | `:exhausted` | no |
|
|
331
|
+
| `decay(arm, keep)` | `:decayed` | `:unknown_arm` | no |
|
|
332
|
+
| `decay_all(keep)` | `:decayed` | none | no |
|
|
333
|
+
| `open(arm)` | `:opened`, `:already_open` | `:unknown_arm` | no |
|
|
334
|
+
| `close(arm)` | `:closed`, `:already_closed` | `:unknown_arm` | no |
|
|
335
|
+
|
|
336
|
+
No answer records a clock: the bandit has none, and nothing it answers depends on the time.
|
|
337
|
+
|
|
338
|
+
`stats(arm)` and `ucb1_index_q56(arm)` answer `nil` for an index that names no arm. Those eight symbols are every variant of the core's `Refusal`; there is no catch-all symbol. Two of them cannot be reached in practice: `:exhausted` needs `2**64-1` selections or resets of one arm, and `:saturated` needs an arm's reward count or sum to pass `2**64-1`.
|
|
339
|
+
- **No hidden randomness, no hidden clock.** The same calls with the same draws give the same answers.
|
|
340
|
+
- **Frozen means frozen.** `freeze` the bandit and every method that would change it raises `FrozenError`.
|
|
341
|
+
|
|
342
|
+
## What You Think You're Doing vs Reality
|
|
343
|
+
|
|
344
|
+
### You Think: "I'll show whichever variant has the best conversion rate."
|
|
345
|
+
### Reality: You're showing whichever variant got lucky first.
|
|
346
|
+
|
|
347
|
+
A rate built from four views is noise. Picking by rate alone locks in the early leader and starves the others of the views that would correct it. UCB1 adds a bonus for how little you know. Epsilon-greedy spends a fixed share of picks looking around.
|
|
348
|
+
|
|
349
|
+
### You Think: "The reward came back, so I'll add it to that arm."
|
|
350
|
+
### Reality: That arm was replaced an hour ago.
|
|
351
|
+
|
|
352
|
+
A result belongs to the selection that produced it, not to whatever sits in that slot now. The ticket carries the arm's generation. Reset the arm and the old ticket is dead.
|
|
353
|
+
|
|
354
|
+
### You Think: "UCB1 has a proven regret bound."
|
|
355
|
+
### Reality: For rewards that don't drift and feedback that doesn't wait.
|
|
356
|
+
|
|
357
|
+
Delayed rewards, resets and decay are tools for the real world. They work. They are not covered by the textbook proof, and this gem doesn't pretend otherwise.
|
|
358
|
+
|
|
359
|
+
## What's Yours
|
|
360
|
+
|
|
361
|
+
The bandit lives in your process and forgets everything when the process does. Persisting counts, sharing them between workers, and carrying a ticket across a request boundary are yours: `Ticket` is an in-memory object, not something to serialize. The random draws are yours, and so is their quality. Mapping a messy outcome onto `0..max_reward` is yours. Deciding when the world has changed enough to `reset` or `decay` is yours. Which arms are eligible, and what an arm means, is yours. The bandit recommends; it never routes traffic or flips a flag.
|
|
362
|
+
|
|
363
|
+
## License
|
|
364
|
+
|
|
365
|
+
MIT License. See [LICENSE.txt](LICENSE.txt) for details.
|
|
366
|
+
|
|
367
|
+
**Remember: the best arm so far is not the best arm. It's the best arm you've looked at.**
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## [0.7.1](https://github.com/seuros/bandit_machines/compare/bandit-machines-v0.7.0...bandit-machines-v0.7.1) (2026-11-16)
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
### Bug Fixes
|
|
7
|
+
|
|
8
|
+
* rebuild bandit on matryona 0.5 and drop core slice indexing ([2157556](https://github.com/seuros/bandit_machines/commit/2157556821757abf6e4fd400f7ad6e2807dfb992))
|
|
9
|
+
|
|
10
|
+
## [0.7.0](https://github.com/seuros/bandit_machines/compare/bandit-machines-v0.6.0...bandit-machines-v0.7.0) (2026-11-14)
|
|
11
|
+
|
|
12
|
+
|
|
13
|
+
### âš BREAKING CHANGES
|
|
14
|
+
|
|
15
|
+
* adopt matryona-machines 0.2: symbols! tables, exact-class Ticket, exact Data fields
|
|
16
|
+
|
|
17
|
+
### Features
|
|
18
|
+
|
|
19
|
+
* adopt matryona-machines 0.2: symbols! tables, exact-class Ticket, exact Data fields ([6b58ea2](https://github.com/seuros/bandit_machines/commit/6b58ea2370184d1156158928cd3c0a3e6764df2a))
|
|
20
|
+
|
|
21
|
+
## [0.6.0](https://github.com/seuros/bandit_machines/compare/bandit-machines-v0.5.0...bandit-machines-v0.6.0) (2026-11-14)
|
|
22
|
+
|
|
23
|
+
|
|
24
|
+
### âš BREAKING CHANGES
|
|
25
|
+
|
|
26
|
+
* seal values, check exact classes natively, clone-then-keep selection
|
|
27
|
+
|
|
28
|
+
### Features
|
|
29
|
+
|
|
30
|
+
* seal values, check exact classes natively, clone-then-keep selection ([b120e24](https://github.com/seuros/bandit_machines/commit/b120e242333aeff876a2b3a6cf4a51550ad96277))
|
|
31
|
+
|
|
32
|
+
## [0.5.0](https://github.com/seuros/bandit_machines/compare/bandit-machines-v0.4.0...bandit-machines-v0.5.0) (2026-11-13)
|
|
33
|
+
|
|
34
|
+
|
|
35
|
+
### âš BREAKING CHANGES
|
|
36
|
+
|
|
37
|
+
* handles are issued through periscope-machines. Every instance can be built with a scope; a handle from another scope is refused before any lookup; all handle kinds of an instance share one sequence; resumed restores the sequence after a restart. A ticket from another bandit answers UnknownTicket. Serials count from one; next_serial is an Option; revoke_before returns a Result.
|
|
38
|
+
|
|
39
|
+
### Features
|
|
40
|
+
|
|
41
|
+
* scope ticket serials through periscope-machines, add Bandit::resumed, issued and revoke_all ([1217d87](https://github.com/seuros/bandit_machines/commit/1217d879479a315a7407ed99d8b493283a347682))
|
|
42
|
+
|
|
43
|
+
## [0.4.0](https://github.com/seuros/bandit_machines/compare/bandit-machines-v0.3.0...bandit-machines-v0.4.0) (2026-11-13)
|
|
44
|
+
|
|
45
|
+
|
|
46
|
+
### âš BREAKING CHANGES
|
|
47
|
+
|
|
48
|
+
* see CONVENTIONS: one exhaustive Refusal per crate, Result from every mutator, honoured idempotent repeats, Ruby mirrors the core. No compatibility aliases. open/close return Opening/Closing; tie_break is required; Selection mirrors the core fields.
|
|
49
|
+
|
|
50
|
+
### Features
|
|
51
|
+
|
|
52
|
+
* name open and close outcomes, Config, Ratio and Draws as Ruby values, unknown arm answers :unknown_arm ([d94221c](https://github.com/seuros/bandit_machines/commit/d94221cbbf96f6a56638865e9f270a11f98c6077))
|
|
53
|
+
|
|
54
|
+
## [0.3.0](https://github.com/seuros/bandit_machines/compare/bandit-machines-v0.2.0...bandit-machines-v0.3.0) (2026-11-13)
|
|
55
|
+
|
|
56
|
+
|
|
57
|
+
### âš BREAKING CHANGES
|
|
58
|
+
|
|
59
|
+
* BanditMachines::Bandit.new no longer takes arms: and opens nothing; the caller opens arms explicitly, as in the Rust core.
|
|
60
|
+
|
|
61
|
+
### Features
|
|
62
|
+
|
|
63
|
+
* start every arm closed in the Ruby bandit and drop arms: ([4f4a0fe](https://github.com/seuros/bandit_machines/commit/4f4a0fef6e3aa39bf64dabca933753538f6e1238))
|
|
64
|
+
|
|
65
|
+
## [0.2.0](https://github.com/seuros/bandit_machines/compare/bandit-machines-v0.1.0...bandit-machines-v0.2.0) (2026-11-12)
|
|
66
|
+
|
|
67
|
+
|
|
68
|
+
### Features
|
|
69
|
+
|
|
70
|
+
* initial extraction ([850717f](https://github.com/seuros/bandit_machines/commit/850717fa9108a4c70fee063cae97407e1a383845))
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
[package]
|
|
2
|
+
name = "bandit-machines"
|
|
3
|
+
version = "0.7.1"
|
|
4
|
+
edition.workspace = true
|
|
5
|
+
rust-version.workspace = true
|
|
6
|
+
license.workspace = true
|
|
7
|
+
description = "Bounded UCB1 and epsilon-greedy arm selection with ticketed delayed rewards"
|
|
8
|
+
readme = "README.md"
|
|
9
|
+
exclude = ["src/**/tests.rs", "src/tests/", "tests/"]
|
|
10
|
+
keywords = ["bandit", "ucb1", "epsilon-greedy", "exploration", "no-std"]
|
|
11
|
+
categories = ["no-std", "algorithms"]
|
|
12
|
+
|
|
13
|
+
[dependencies]
|
|
14
|
+
sliderule-machines.workspace = true
|
|
15
|
+
periscope-machines.workspace = true
|
|
16
|
+
|
|
17
|
+
[dev-dependencies]
|
|
18
|
+
matryona-machines = { workspace = true, features = ["testing"] }
|
|
19
|
+
|
|
20
|
+
[lints]
|
|
21
|
+
workspace = true
|
|
@@ -0,0 +1,152 @@
|
|
|
1
|
+
# bandit-machines
|
|
2
|
+
|
|
3
|
+
> **AMS Robbins**, NCC-4227, *Robbins*-class exploration scout.
|
|
4
|
+
|
|
5
|
+
Engine-room flight manual: choose which of a bounded set of arms to try next,
|
|
6
|
+
and learn from rewards that come back late.
|
|
7
|
+
|
|
8
|
+
Allocation-free `no_std` multi-armed bandit over `ARMS` arms and at most
|
|
9
|
+
`TICKETS` unanswered selections. Two policies share one set of statistics:
|
|
10
|
+
UCB1 and epsilon-greedy. Rewards are integers in `0..=max_reward`. There is no
|
|
11
|
+
generator inside: every random decision consumes a `u64` the caller supplies,
|
|
12
|
+
so a run is replayable from its draws.
|
|
13
|
+
|
|
14
|
+
Pre-1.0: breaking changes land in minor versions. This is a policy core, not an experiment platform: it
|
|
15
|
+
does not assign users, persist anything, or know what an arm means.
|
|
16
|
+
|
|
17
|
+
```rust
|
|
18
|
+
use bandit_machines::{
|
|
19
|
+
Bandit, Closing, Config, Draws, Opening, Ratio, Reason, Refusal, TieBreak,
|
|
20
|
+
};
|
|
21
|
+
|
|
22
|
+
let config = Config::new(100, 50, TieBreak::LowestIndex).expect("valid range");
|
|
23
|
+
let mut bandit = Bandit::<4, 16>::new(config);
|
|
24
|
+
assert_eq!(bandit.open(0)?, Opening::Opened);
|
|
25
|
+
assert_eq!(bandit.open(1)?, Opening::Opened);
|
|
26
|
+
// Asking again is honoured and changes nothing.
|
|
27
|
+
assert_eq!(bandit.open(1)?, Opening::AlreadyOpen);
|
|
28
|
+
assert_eq!(bandit.close(3)?, Closing::AlreadyClosed);
|
|
29
|
+
|
|
30
|
+
let first = bandit.select_ucb1(0)?;
|
|
31
|
+
assert_eq!((first.arm(), first.reason), (0, Reason::ColdStart));
|
|
32
|
+
bandit.reward(first.ticket, 80)?;
|
|
33
|
+
|
|
34
|
+
let epsilon = Ratio::new(1, 10).expect("0.1");
|
|
35
|
+
let second = bandit.select_epsilon_greedy(epsilon, Draws { gate: u64::MAX, pick: 0 })?;
|
|
36
|
+
assert_eq!((second.arm(), second.reason), (1, Reason::ColdStart));
|
|
37
|
+
|
|
38
|
+
// The arm is reset while its reward is still in flight: the reward is refused.
|
|
39
|
+
bandit.reset(1)?;
|
|
40
|
+
assert_eq!(bandit.reward(second.ticket, 100), Err(Refusal::StaleTicket));
|
|
41
|
+
|
|
42
|
+
// Another bandit, another scope: it refuses a ticket it did not issue. Two
|
|
43
|
+
// bandits built with `new` are both scope zero and would alias.
|
|
44
|
+
let mut other = Bandit::<4, 16>::with_scope(config, 7);
|
|
45
|
+
other.open(0)?;
|
|
46
|
+
let theirs = other.select_ucb1(0)?.ticket;
|
|
47
|
+
assert_eq!((theirs.scope(), bandit.scope()), (7, 0));
|
|
48
|
+
assert_eq!(bandit.reward(theirs, 1), Err(Refusal::UnknownTicket));
|
|
49
|
+
assert_eq!(other.reward(theirs, 1), Ok(()));
|
|
50
|
+
# Ok::<(), Refusal>(())
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
## Contracts
|
|
54
|
+
|
|
55
|
+
**Eligibility.** Arms start closed: the bandit explores only what the caller
|
|
56
|
+
opens, and selecting before that answers `NoOpenArm`. The Ruby gem starts the
|
|
57
|
+
same way, and its `Config` requires the tie rule just as `Config::new` does.
|
|
58
|
+
`open` and `close` only change whether an arm can be selected, and answer what
|
|
59
|
+
they did: `Opening::Opened` / `Closing::Closed` when eligibility changed,
|
|
60
|
+
`Opening::AlreadyOpen` / `Closing::AlreadyClosed` for an idempotent repeat that
|
|
61
|
+
changed nothing (Ruby: `:opened`, `:closed`, `:already_open`,
|
|
62
|
+
`:already_closed`). An arm past the capacity is refused with `UnknownArm`
|
|
63
|
+
(Ruby: `:unknown_arm`). A
|
|
64
|
+
closed arm keeps its statistics and still accepts the
|
|
65
|
+
rewards of tickets issued before it closed. Whatever decides eligibility (a
|
|
66
|
+
feature gate, a health check) lives with the caller.
|
|
67
|
+
|
|
68
|
+
**Tries.** An arm's tries are its counted rewards plus its outstanding tickets.
|
|
69
|
+
Exploration counts tries, so selecting again before a reward arrives does not
|
|
70
|
+
pile onto the same arm.
|
|
71
|
+
|
|
72
|
+
**Cold start.** An open arm with zero tries is *cold*. Both policies select
|
|
73
|
+
cold arms before doing anything else, one per call, in tie-rule order
|
|
74
|
+
(`Reason::ColdStart`). An arm with tickets out and no reward yet is *warming*
|
|
75
|
+
and competes with the configured `prior` as its mean.
|
|
76
|
+
|
|
77
|
+
**UCB1.** With `t` the tries of all open arms and `n` an arm's tries, the index
|
|
78
|
+
is `mean / max_reward + sqrt(2 ln t / n)`, computed in Q8.56 fixed point by the
|
|
79
|
+
integer routines in `math`. The logarithm is not ours: it is
|
|
80
|
+
`ln_floor` of the `sliderule-machines` crate at 56 fraction bits, the one integer
|
|
81
|
+
logarithm the fleet shares. The computed index never exceeds the true one and
|
|
82
|
+
falls short by less than `2^-26` (`ln` alone: less than `2^-55`). Arms whose
|
|
83
|
+
true indices differ by more than that are always ordered correctly; closer ones
|
|
84
|
+
are ordered by their computed integers, identically on every target.
|
|
85
|
+
`ucb1_index_q56` publishes the integer being compared.
|
|
86
|
+
|
|
87
|
+
**Epsilon-greedy.** Explore when `floor(gate * denominator / 2^64) < numerator`,
|
|
88
|
+
picking open arm number `floor(pick * open / 2^64)`. Otherwise exploit the
|
|
89
|
+
highest mean, compared exactly by cross-multiplication.
|
|
90
|
+
|
|
91
|
+
**Ties.** Exactly equal scores are settled by `TieBreak`: lowest index, highest
|
|
92
|
+
index, fewest tries then lowest index, or a uniform draw from `pick`.
|
|
93
|
+
|
|
94
|
+
**Delayed rewards.** Every selection issues a `Ticket` naming the arm, the
|
|
95
|
+
arm's generation and a serial. A ticket is redeemed once, by `reward` or
|
|
96
|
+
`abandon`. `reset` bumps the generation and voids the arm's tickets, so a late
|
|
97
|
+
reward answers `StaleTicket`; a spent, revoked or foreign ticket answers
|
|
98
|
+
`UnknownTicket`. `revoke_before(serial)` abandons everything issued before a
|
|
99
|
+
serial, which is how a caller with its own clock expires selections that will
|
|
100
|
+
never report. `revoke_all()` abandons every outstanding ticket and cannot
|
|
101
|
+
refuse; no serial does that (`u64::MAX` is another scope's, and the last
|
|
102
|
+
serial of a scope can itself be outstanding). Unlike `reset_all` it keeps
|
|
103
|
+
every arm's statistics and generation. `next_serial` is the serial the next ticket will carry, `None`
|
|
104
|
+
once the scope is spent. Tickets are not a wire format.
|
|
105
|
+
|
|
106
|
+
**Scopes.** A ticket is only meaningful to the bandit that issued it. Serials
|
|
107
|
+
come from a `periscope_machines::Issuer`: the bandit's scope in the top 16
|
|
108
|
+
bits, a sequence counted from one below, never reissued.
|
|
109
|
+
`Bandit::with_scope(config, scope)` chooses the scope and `new(config)` is
|
|
110
|
+
scope zero; `Bandit::scope` and `Ticket::scope` read it. The scope is identity,
|
|
111
|
+
not policy, so it is not in `Config`: replicas share a configuration and must
|
|
112
|
+
not share a scope. `reward` and `abandon` answer `UnknownTicket` for a ticket
|
|
113
|
+
the bandit did not issue, before looking at any arm or slot, and
|
|
114
|
+
`revoke_before` answers it for a serial of another scope. Give each bandit its
|
|
115
|
+
own scope: the core reads no clock and no random source, so it cannot pick one,
|
|
116
|
+
and two bandits left in the same scope (two built with `new`, say) issue the
|
|
117
|
+
same tickets and honour each other's.
|
|
118
|
+
|
|
119
|
+
**No clock.** The bandit reads and records no time; no answer depends on one.
|
|
120
|
+
|
|
121
|
+
**Restarts.** A fresh bandit counts from one again, so a ticket that outlived a
|
|
122
|
+
restart could redeem a new selection. Save `Bandit::issued` and rebuild with
|
|
123
|
+
`Bandit::resumed(config, scope, issued)`, or give each incarnation a new scope.
|
|
124
|
+
`resumed` restores only the serial sequence: every arm is closed with nothing
|
|
125
|
+
observed, no ticket is outstanding, every earlier ticket is refused, and
|
|
126
|
+
reopening arms and restoring what they learned is the caller's job. It is
|
|
127
|
+
`None` when `issued` is beyond `periscope_machines::SEQUENCE_MAX`.
|
|
128
|
+
|
|
129
|
+
**Nonstationarity.** Nothing ages on its own. `reset` / `reset_all` forget an
|
|
130
|
+
arm and fence its late rewards. `decay` / `decay_all` discount history by a
|
|
131
|
+
`Ratio`: count becomes `ceil(count * keep)`, sum `floor(sum * keep)`, tickets
|
|
132
|
+
and generations untouched, so a late reward still lands at full weight.
|
|
133
|
+
|
|
134
|
+
**Refusals.** Every refusal leaves the bandit unchanged. `TicketsFull` when all
|
|
135
|
+
slots are out, `Exhausted` when the scope's 2^48 - 1 serials or an arm's
|
|
136
|
+
generations run out,
|
|
137
|
+
`Saturated` when an arm's `u64` count or sum would overflow (decay, then retry
|
|
138
|
+
the same ticket), `RewardOutOfRange` above `max_reward`.
|
|
139
|
+
|
|
140
|
+
**No panics.** Every arm and slot lookup is checked and no counter is
|
|
141
|
+
subtracted unchecked; how many tickets are outstanding is counted from the
|
|
142
|
+
slots, not kept beside them. Four arms are ruled out by the bandit's own
|
|
143
|
+
invariants yet demanded by the types: a free slot or chosen arm that is not
|
|
144
|
+
there when a ticket is issued, a drawn tie position past `usize`, and an
|
|
145
|
+
outstanding ticket its arm does not count. Each is a `debug_assert!` that
|
|
146
|
+
fails in tests; a release build answers `TicketsFull`, `NoOpenArm` (the first
|
|
147
|
+
two) or `UnknownTicket` with nothing changed (the serial is issued last). Voiding cannot refuse, so there a broken count is left as it
|
|
148
|
+
is and the slot is still freed.
|
|
149
|
+
|
|
150
|
+
The regret bound of UCB1 assumes stationary rewards and immediate feedback.
|
|
151
|
+
Delays, resets and decay are operational tools; they void that proof, not the
|
|
152
|
+
arithmetic.
|