suspect_machines 0.5.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/README.md ADDED
@@ -0,0 +1,335 @@
1
+ # SuspectMachines
2
+
3
+ > Silence is evidence. It is not a death certificate. 📡
4
+
5
+ A phi-accrual failure detector for Ruby. You tell the roster every time a peer's heartbeat arrives. Later you ask how suspicious that peer's silence is, and it answers with a number, phi, that grows the longer the peer stays quiet relative to how regular it used to be. What you do with the number is up to you. The roster never declares anyone dead.
6
+
7
+ ## Quick Start
8
+
9
+ ```bash
10
+ gem 'suspect_machines'
11
+ ```
12
+
13
+ ```ruby
14
+ SM = SuspectMachines
15
+
16
+ normal = SM::Normal.new(100_000_000) # the shape you assume intervals have, with a 100 ms floor on the deviation
17
+ roster = SM::Roster.new(SM::Config.new(normal, 8)) # 8 intervals before any phi is reported
18
+ act_at = SM::Threshold.new(8.0)
19
+
20
+ def now = Process.clock_gettime(Process::CLOCK_MONOTONIC, :nanosecond)
21
+
22
+ def on_heartbeat(roster, peer, incarnation)
23
+ roster.heartbeat(peer, incarnation, now)
24
+ end
25
+
26
+ def on_tick(roster)
27
+ roster.scan(now, act_at, 0, SM::Roster::CAPACITY).suspects.each do |suspect|
28
+ stop_routing_reads_to(suspect.peer) # a hint, cheap to undo
29
+ end
30
+ end
31
+ ```
32
+
33
+ One roster, two verbs: `heartbeat` when you hear from a peer, `suspicion` or `scan` when you want to know who has gone quiet. `SM` is only an alias to keep the lines short. A `Config` is the core's: the distribution you assume and the warm-up, in that order. Peers, incarnations and times are Integers. Times are nanoseconds from a clock that only goes forward.
34
+
35
+ ## A Message to the Resistance
36
+
37
+ So you wrote `dead = now - last_seen > 5.seconds`. It worked for a year. Then a node on a congested rack took 5.2 seconds between two heartbeats, your cluster declared it dead, rebalanced four hundred partitions onto its neighbours, and the neighbours, now twice as busy, started taking 5.2 seconds too.
38
+
39
+ So you raised the timeout to 30 seconds. Now a node that actually dies keeps receiving traffic for half a minute, and the node that heartbeats every 50 ms gets the same generosity as the one that heartbeats every 10 seconds.
40
+
41
+ A fixed timeout is one number pretending to describe every peer and every network on every day. The thing you wanted was: "this peer usually checks in every second, give or take 100 ms, and it has been quiet for 1.5 seconds. How unusual is that?" That question has an answer, and the answer is a probability, not a boolean. This gem computes it.
42
+
43
+ What it gives you is a suspicion level. It does not give you permission to take anything away from the suspect.
44
+
45
+ ## Chapter 1: Stardate 2026.316, the AMS Hayashibara
46
+
47
+ > **AMS Hayashibara**, NCC-4223, *Hayashibara*-class picket vessel.
48
+ > The picket reports how long the lamp has been dark. The admiralty decides what that means.
49
+
50
+ A picket vessel sits at the edge of the formation and watches the signal lamps of the ships beyond it. It keeps a log of how regularly each lamp has flashed. When a lamp goes dark, the picket does not announce that the ship is lost. It reports that this ship, which flashed every sixty seconds for a month, has now been dark for four minutes, and that this is very unlike it.
51
+
52
+ On a bridge somewhere, a cadet volunteers:
53
+
54
+ ```ruby
55
+ def lost?(ship)
56
+ Time.now - ship.last_flash > 300
57
+ end
58
+
59
+ reassign_escort_duty(ship) if lost?(ship)
60
+ ```
61
+
62
+ "That," says the picket's signals officer, "is a ship in fog being relieved of its command by a stopwatch."
63
+
64
+ ## The Flight Plan
65
+
66
+ Each peer the roster knows about is in one of these states:
67
+
68
+ ```mermaid
69
+ stateDiagram-v2
70
+ [*] --> untracked
71
+ untracked --> warming_up: first heartbeat
72
+ warming_up --> warming_up: heartbeat, fewer than min_samples intervals
73
+ warming_up --> measured: heartbeat, min_samples intervals reached
74
+ measured --> measured: heartbeat, oldest interval slides out
75
+ measured --> warming_up: heartbeat with a higher incarnation
76
+ warming_up --> warming_up: heartbeat with a higher incarnation
77
+ warming_up --> untracked: forget
78
+ measured --> untracked: forget
79
+ ```
80
+
81
+ Only `measured` peers have a phi. And one quiet afternoon on node 7:
82
+
83
+ ```mermaid
84
+ sequenceDiagram
85
+ participant N as Node 7
86
+ participant R as Roster
87
+ participant Y as Your code
88
+
89
+ N->>R: heartbeat at 1 s, 2 s, 3 s ... 9 s
90
+ Note over R: mean 1 s, deviation floored at 100 ms
91
+ Y->>R: suspicion(7, 10 s)
92
+ R-->>Y: 0.30 (right on time)
93
+ Note over N: goes quiet
94
+ Y->>R: suspicion(7, 10.3 s)
95
+ R-->>Y: 2.87
96
+ Y->>R: suspicion(7, 10.5 s)
97
+ R-->>Y: 6.54
98
+ Y->>R: scan(11 s, 8.0, 0, 64)
99
+ R-->>Y: [Suspect peer 7, phi 23.1]
100
+ Note over Y: stops sending it new reads. Revokes nothing.
101
+ ```
102
+
103
+ ## How the Picket Works
104
+
105
+ ### Heartbeat
106
+
107
+ ```ruby
108
+ SECOND = 1_000_000_000
109
+ roster = SM::Roster.new(SM::Config.new(SM::Normal.new(100_000_000), 3))
110
+
111
+ roster.heartbeat(7, 1, 0) # => :first peer 7, incarnation 1, now 0 ns
112
+ roster.heartbeat(7, 1, 1 * SECOND) # => #<data SuspectMachines::Recorded interval=1000000000, samples=1>
113
+ roster.heartbeat(7, 1, 2 * SECOND) # => #<data SuspectMachines::Recorded interval=1000000000, samples=2>
114
+ roster.heartbeat(7, 1, 3 * SECOND) # => #<data SuspectMachines::Recorded interval=1000000000, samples=3>
115
+ ```
116
+
117
+ The first heartbeat is only a reference point. Each one after it records the interval since the previous one and answers a `Recorded`: that `interval`, and the `samples` now in the window. The roster keeps the last 128 intervals per peer; the 129th pushes the oldest out.
118
+
119
+ ### Warm-up
120
+
121
+ ```ruby
122
+ roster = SM::Roster.new(SM::Config.new(SM::Normal.new(100_000_000), 3))
123
+ roster.heartbeat(7, 1, 0)
124
+ roster.heartbeat(7, 1, 1 * SECOND)
125
+
126
+ roster.suspicion(7, 60 * SECOND)
127
+ # => #<data SuspectMachines::WarmingUp samples=1, required=3, silence=59000000000>
128
+ roster.last_heartbeat(7) # => 1000000000
129
+ ```
130
+
131
+ Until a peer has `min_samples` intervals on record, the roster has no basis for a phi and says so with a `WarmingUp`: the `samples` it has, the number `required`, and the `silence` since the last heartbeat. It does not guess. A peer that sends one heartbeat and vanishes stays warming up forever, so put a plain timeout of your own on that `silence` (or on `last_heartbeat`) for peers that never finish warming up.
132
+
133
+ ### Suspicion
134
+
135
+ ```ruby
136
+ # after heartbeats at 0, 1, 2 and 3 seconds
137
+ roster.suspicion(7, 4 * SECOND)
138
+ # => #<data SuspectMachines::Measured phi=#<data SuspectMachines::Phi value=0.30102999566398114>>
139
+ roster.suspicion(7, 4 * SECOND + 300_000_000).phi.value # => 2.8696990359293695
140
+ roster.suspicion(7, 4 * SECOND + 500_000_000).phi.millis # => 6542
141
+ roster.suspicion(9, 4 * SECOND) # => :untracked
142
+
143
+ case roster.suspicion(7, now)
144
+ in SM::Measured[phi:] then phi.reaches?(act_at)
145
+ in SM::WarmingUp[silence:] then silence > bootstrap_timeout
146
+ in Symbol => refusal then false # :untracked, :time_regression
147
+ end
148
+ ```
149
+
150
+ `suspicion` answers the core's `Suspicion`: a `Measured` carrying a `Phi`, or a `WarmingUp` while there is none yet. A `Phi` is the core's: its `value` is a finite, non-negative Float, `millis` is the level in thousandths rounded down (the core's `Phi::millis`), and `reaches?(threshold)` is the core's `Phi::reaches`. A `Threshold` is the core's too: a finite Float above zero, and the only thing `reaches?` and `scan` take as a threshold.
151
+
152
+ Phi is `-log10` of the probability that a heartbeat would still show up after this much silence. Phi 1 means about a 10% chance the peer is merely late. Phi 3 means about 0.1%. Phi 8 is a common place to start acting. Asking never changes anything, so ask as often as you like.
153
+
154
+ ### Distributions
155
+
156
+ The probability depends on what you assume intervals look like. You must say which:
157
+
158
+ ```ruby
159
+ SM::Config.new(SM::Normal.new(100_000_000), 8) # Distribution::Normal(Normal::new(min_std_dev))
160
+ SM::Config.new(:exponential, 8) # Distribution::Exponential
161
+ ```
162
+
163
+ A `Normal` uses the mean and standard deviation of the window. A peer that beats like a metronome has a deviation of zero, which would make the first late nanosecond infinitely suspicious, so the deviation is never taken below `min_std_dev`; a zero floor is a `RangeError` from `Normal.new`, as `Normal::new` refuses it in the core. `:exponential` uses only the mean: `phi = silence / (mean * ln 10)`. It climbs in a straight line and is more forgiving of bursty arrivals.
164
+
165
+ ```ruby
166
+ roster = SM::Roster.new(SM::Config.new(:exponential, 1))
167
+ [0, 10, 20].each { |now| roster.heartbeat(1, 0, now) }
168
+ roster.suspicion(1, 50).phi.value # => 1.3028834457097553, three mean intervals of silence
169
+ ```
170
+
171
+ The roster does not check that your traffic matches the shape you picked.
172
+
173
+ ### Acceptable pause
174
+
175
+ ```ruby
176
+ config = SM::Config.new(:exponential, 1).with_acceptable_pause(5)
177
+ config.acceptable_pause # => 5
178
+ roster = SM::Roster.new(config)
179
+ [0, 10, 20].each { |now| roster.heartbeat(1, 0, now) }
180
+ roster.suspicion(1, 24).phi.value # => 0.0
181
+ roster.suspicion(1, 55).phi.value # => 1.3028834457097553
182
+ roster.config == config # => true
183
+ ```
184
+
185
+ If your peers are known to stall for a garbage collection or a fsync, give them that much silence for free. Suspicion starts accruing after the pause. `Config.new` takes the two arguments of the core's `Config::new` and tolerates no pause; `with_acceptable_pause` answers a copy that does, as the core's builder does.
186
+
187
+ ### Scan
188
+
189
+ ```ruby
190
+ scan = roster.scan(now, SM::Threshold.new(8.0), 0, 64) # now, threshold, cursor, limit
191
+ scan.suspects.first.phi.value # => 23.118053405486076, for peer 7, incarnation 1
192
+ scan.next # => nil
193
+ ```
194
+
195
+ `scan` lists every measured peer whose phi has reached the threshold, as a `Scan`. All four arguments are required, as they are in the core: the instant, the threshold, the cursor to start from (zero the first time) and the `limit`, which stands for the length of the buffer the core writes into and is at most `Roster::CAPACITY`. It returns at most `limit` suspects, and `next` is the cursor to resume from when there are more. A peer whose last heartbeat is later than `now` has no silence to measure at that instant and is skipped, not refused (`suspicion` for that peer answers `:time_regression`):
196
+
197
+ ```ruby
198
+ page = roster.scan(now, act_at, 0, 16)
199
+ while page.next
200
+ page = roster.scan(now, act_at, page.next, 16)
201
+ end
202
+ ```
203
+
204
+ Peers still warming up are never listed.
205
+
206
+ ### Statistics
207
+
208
+ ```ruby
209
+ stats = roster.statistics(7)
210
+ stats.samples # => 3
211
+ stats.mean_ns # => 1000000000.0
212
+ stats.std_dev_ns # => 0.0
213
+ stats.min # => 1000000000
214
+ stats.max # => 1000000000
215
+ ```
216
+
217
+ What the roster currently believes about a peer's rhythm, over the intervals in its window. `nil` before the second heartbeat. The deviation shown is the measured one, before the floor.
218
+
219
+ ### Incarnations
220
+
221
+ ```ruby
222
+ roster = SM::Roster.new(SM::Config.new(:exponential, 1))
223
+ roster.heartbeat(1, 5, 100) # => :first
224
+ roster.heartbeat(1, 5, 200) # => #<data SuspectMachines::Recorded interval=100, samples=1>
225
+ roster.heartbeat(1, 4, 300) # => :stale an older incarnation, ignored
226
+ roster.heartbeat(1, 6, 300) # => :restarted statistics start over
227
+ roster.incarnation(1) # => 6
228
+ roster.suspicion(1, 400) # => #<data SuspectMachines::WarmingUp samples=0, required=1, silence=100>
229
+ ```
230
+
231
+ When a peer restarts, have it announce a higher incarnation. A delayed heartbeat from the old process can't refresh the new one, and the new process doesn't inherit the old one's rhythm.
232
+
233
+ ### Forget
234
+
235
+ ```ruby
236
+ roster.forget(1) # => :forgotten
237
+ roster.forget(1) # => :not_tracked
238
+ ```
239
+
240
+ The roster never drops a peer on its own, however suspicious. Membership is yours. When you decide a peer is gone, forget it and its slot is free. Both answers are honoured: forgetting a peer that is not tracked is a repeat that changes nothing, not a refusal.
241
+
242
+ ### Capacity
243
+
244
+ ```ruby
245
+ roster = SM::Roster.new(SM::Config.new(:exponential, 1))
246
+ SuspectMachines::Roster::CAPACITY # => 64
247
+ SuspectMachines::Roster::WINDOW # => 128
248
+ 64.times { |peer| roster.heartbeat(peer, 0, 0) }
249
+ roster.heartbeat(64, 0, 0) # => :full
250
+ roster.forget(3) # => :forgotten
251
+ roster.heartbeat(64, 0, 0) # => :first
252
+ ```
253
+
254
+ Sixty-four peers, 128 intervals each, compiled in. A full roster refuses new peers. It does not evict a tracked one to make room.
255
+
256
+ ### Time only moves forward
257
+
258
+ ```ruby
259
+ roster = SM::Roster.new(SM::Config.new(:exponential, 1))
260
+ roster.heartbeat(1, 0, 100) # => :first
261
+ roster.heartbeat(1, 0, 99) # => :time_regression
262
+ roster.heartbeat(1, 0, 100) # => :same_instant
263
+ roster.suspicion(1, 99) # => :time_regression
264
+ ```
265
+
266
+ Each peer's heartbeats must arrive in increasing time. Two at the same instant carry no interval, so the second is refused. Feed it a monotonic clock in nanoseconds, read where the heartbeat is received.
267
+
268
+ ### One peer on its own
269
+
270
+ ```ruby
271
+ detector = SM::Detector.new(SM::Config.new(SM::Normal.new(100_000_000), 3))
272
+ detector.suspicion(0) # => :no_heartbeat
273
+ detector.heartbeat(0) # => :first
274
+ detector.heartbeat(1 * SECOND) # => #<data SuspectMachines::Recorded interval=1000000000, samples=1>
275
+ detector.suspicion(2 * SECOND) # => #<data SuspectMachines::WarmingUp samples=1, required=3, silence=1000000000>
276
+ detector.samples # => 1
277
+ detector.last_heartbeat # => 1000000000
278
+ detector.statistics # => a Statistics, nil before the second heartbeat
279
+ detector.config # => the Config
280
+ detector.reset # => nil, every interval and the last heartbeat are gone
281
+ ```
282
+
283
+ A `Detector` is the core's single-peer type: the same window (`Detector::WINDOW`, 128 intervals) and the same answers, without peers, incarnations or capacity. It is the one place `suspicion` answers `:no_heartbeat`.
284
+
285
+ ### What each call answers
286
+
287
+ | Call | Honoured | Records `now` | Refused, nothing changed |
288
+ | --- | --- | --- | --- |
289
+ | `Roster#heartbeat(peer, incarnation, now)` | `:first`, a `Recorded`, `:restarted` | every honoured answer: `now` becomes the peer's last heartbeat | `:time_regression`, `:same_instant`, `:stale`, `:full` |
290
+ | `Roster#suspicion(peer, now)` | a `Measured`, or a `WarmingUp` | never: a query | `:untracked`, `:time_regression` |
291
+ | `Roster#scan(now, threshold, cursor, limit)` | a `Scan` | never: a query | none reachable |
292
+ | `Roster#statistics(peer)` | a `Statistics`, or `nil` before the second heartbeat | no clock | a query, never refuses |
293
+ | `Roster#incarnation(peer)`, `Roster#last_heartbeat(peer)` | an Integer, or `nil` for an untracked peer | no clock | a query, never refuses |
294
+ | `Roster#forget(peer)` | `:forgotten`, or the repeat `:not_tracked` | no clock | never refuses |
295
+ | `Roster#size`, `Roster#empty?`, `Roster#config` | an Integer, a boolean, the `Config` | no clock | a query, never refuses |
296
+ | `Detector#heartbeat(now)` | `:first`, a `Recorded` | every honoured answer: `now` becomes the last heartbeat | `:time_regression`, `:same_instant` |
297
+ | `Detector#suspicion(now)` | `:no_heartbeat`, a `WarmingUp`, a `Measured` | never: a query | `:time_regression` |
298
+ | `Detector#reset` | `nil` | no clock; the last heartbeat is forgotten | never refuses |
299
+
300
+ A heartbeat is the observation of `now` itself, so every honoured heartbeat answer depends on `now` and records it; there is no honoured repeat of a heartbeat (the same instant again is `:same_instant`). `suspicion` and `scan` depend on `now` but are queries: they never change the roster or detector, so a later call may ask about an earlier instant, down to the last heartbeat.
301
+
302
+ The six refusal symbols are the six variants of the core's `Refusal`, each with its own symbol; there is no catch-all. One of them, `:overflow`, cannot be reached from Ruby: it is the core's answer to a time past `2**64-1` nanoseconds, and such an Integer raises `RangeError` before the core sees it. It is what `heartbeat`, `suspicion` and `scan` would answer. A roster's `suspicion` never answers `:no_heartbeat`, because a tracked peer always has a heartbeat.
303
+
304
+ `Phi`, `Threshold`, `Normal`, `Config`, `Recorded`, `WarmingUp`, `Measured`, `Statistics`, `Suspect` and `Scan` are frozen `Data` values with the fields of the core types they are named after. Each is checked by the core when it is built, whether the gem built it or you did (and by `with`): a wrong type raises `TypeError`, a value the core would never hold raises `RangeError`. So `Phi.new(-1)`, `Threshold.new(0)`, `Recorded.new(0, 1)` (a zero interval), `WarmingUp.new(3, 3, 0)` (the warm-up is over), `Statistics.new(1, 1.0, 0.0, 2, 1)` (min above max), `Statistics.new(2, 5.0, 0.0, 10, 20)` (a mean below the shortest interval), `Normal.new(0)` (no floor) and `Scan.new([], 64)` (no such slot) all raise `RangeError`; `Measured.new(1.0)` and `Suspect.new(1, 0, 4.0)` raise `TypeError`, because a phi is a `Phi`. The limits are the core's constructors for the compiled window and capacity: `Normal::new`, `Beat::recorded`, `Suspicion::warming_up`, `Statistics::new` and `Scan::new`. The classes are final: subclassing one raises `TypeError` and `allocate` is gone, so no instance skips the check. Sealing stops accidents, not reflection (a named subclass's constant is bound before `inherited` raises, and `Marshal.load` skips `initialize`), so the binding trusts no value it reads: each must be an instance of exactly the gem's class (a subclass raises `TypeError`) and is rebuilt through the core's constructor (a forged field raises `RangeError`). The classes are resolved once, at load. Building an answer runs Ruby code (`new`), so `heartbeat` (on a `Detector` and a `Roster`) decides on a copy, builds the answer, and keeps the copy only if the object is still unfrozen and unchanged: if building raises, the heartbeat is not recorded; if it freezes the object, `FrozenError`; if it re-enters and changes the object, `RuntimeError`, and the outer call is not applied. An enum's unit variant is a symbol and a variant that carries something is a `Data` named after it: a distribution is `:exponential` or a `Normal`, a beat is `:first`, a `Recorded` or `:restarted`, a suspicion is `:no_heartbeat`, a `WarmingUp` or a `Measured`. A phi and a threshold are the core's `Phi` and `Threshold`, as `Data` values carrying a Float `value`. A `Scan`'s `suspects` are the suspects themselves, where the core's `Scan::suspects` counts how many it wrote to the caller's buffer. `Roster#size` is the core's `len`.
305
+
306
+ ## The Rules of the Picket
307
+
308
+ - **Phi is advisory.** It is a statement about arrival statistics. It cannot revoke a lease, depose a leader, or release a lock. Anything that changes ownership needs a durable authority and fencing at the resource.
309
+ - **Assumptions are arguments.** A `Config` takes its distribution and `min_samples` positionally and neither has a default. A `Normal` requires its `min_std_dev`. The acceptable pause is zero until `with_acceptable_pause` says otherwise, as in the core. `scan` has no default cursor or limit.
310
+ - **Peers, incarnations and times are Integers** in `0..2**64-1`. Times are nanoseconds. Floats, nil, and strings raise `TypeError`, and so does anything but a `Config` where one belongs. Out of range raises `RangeError`, and so do a zero `min_samples`, a zero deviation floor, a `min_samples` above the window and a `limit` above the capacity. So does an unknown distribution symbol. A threshold is a `Threshold`; anything else where one belongs raises `TypeError`.
311
+ - **Refusals don't raise.** `:time_regression`, `:same_instant`, `:stale`, `:full` and `:untracked` are answers. A refused heartbeat changes nothing. Only malformed input raises.
312
+ - **Same calls, same floats.** The square root, exponential, logarithm and normal tail are computed in the gem from plain IEEE arithmetic, with no platform math library. Phi is within `1e-12` (absolute or relative) of the exact value for the statistics it was given.
313
+ - **Frozen means frozen.** `freeze` a roster and `heartbeat` and `forget` raise `FrozenError`; `freeze` a detector and `heartbeat` and `reset` do. Reads still work.
314
+
315
+ ## What You Think You're Doing vs Reality
316
+
317
+ ### You Think: "Phi crossed 8, so the node is dead and I can take over its work."
318
+ ### Reality: Phi crossed 8, so the node is quiet from where you're standing.
319
+
320
+ A node in a long GC pause is quiet. A node behind a partitioned switch is quiet to you and chatty to everyone else. Both will come back and carry on writing as if nothing happened. If your takeover depends on the old owner having stopped, phi can't promise that. A lease with a fencing token checked by the storage can. Use phi to decide when to *ask* for a new owner, and let the authority decide whether there is one.
321
+
322
+ ### You Think: "A higher threshold means fewer mistakes."
323
+ ### Reality: A higher threshold means different mistakes.
324
+
325
+ Under `:normal` with a tight deviation, phi goes from 3 to 23 in a few hundred milliseconds. Moving the threshold from 8 to 12 buys almost nothing. What governs false suspicion is whether the window has seen the peer's bad days: a window of 128 calm intervals knows nothing about the weekly backup that stalls heartbeats for four seconds. Set `min_std_dev` and `acceptable_pause` from the stalls you know about.
326
+
327
+ ## What's Yours
328
+
329
+ This is a pre-1.0 policy core; breaking changes land in minor versions. It is not a membership service. You own the clock, the transport that carries heartbeats, and the timer that calls `scan`. You own what a suspicion triggers, and making that reversible. You own membership: adding peers by their first heartbeat, bumping incarnations on restart, and calling `forget`. The roster lives in your process and forgets everything when the process does, so every restart begins with a fresh warm-up. One long outage that ends in a heartbeat enters the window as one very long interval and loosens the peer's statistics until it slides out. If that matters, `forget` the peer on recovery and let it warm up again.
330
+
331
+ ## License
332
+
333
+ MIT License. See [LICENSE.txt](LICENSE.txt) for details.
334
+
335
+ **Remember: the lamp went dark. That is all you know.**
@@ -0,0 +1,48 @@
1
+ # Changelog
2
+
3
+ ## [0.5.1](https://github.com/seuros/suspect_machines/compare/suspect-machines-v0.5.0...suspect-machines-v0.5.1) (2026-11-16)
4
+
5
+
6
+ ### Bug Fixes
7
+
8
+ * revalidate Config#with_acceptable_pause through the core ([52ce530](https://github.com/seuros/suspect_machines/commit/52ce5309b1ffec9922b2e1ca3720f6090637bb96))
9
+
10
+ ## [0.5.0](https://github.com/seuros/suspect_machines/compare/suspect-machines-v0.4.0...suspect-machines-v0.5.0) (2026-11-14)
11
+
12
+
13
+ ### âš  BREAKING CHANGES
14
+
15
+ * adopt matryona-machines 0.2: exact readers, symbols! tables, struct-only field reads
16
+
17
+ ### Features
18
+
19
+ * adopt matryona-machines 0.2: exact readers, symbols! tables, struct-only field reads ([180ab6b](https://github.com/seuros/suspect_machines/commit/180ab6bd465bf6b05cf076c732d036a0ef26045c))
20
+
21
+ ## [0.4.0](https://github.com/seuros/suspect_machines/compare/suspect-machines-v0.3.0...suspect-machines-v0.4.0) (2026-11-14)
22
+
23
+
24
+ ### âš  BREAKING CHANGES
25
+
26
+ * core-validated sealed values, private Normal/Statistics/Suspect/Scan fields, Scan#suspects, clone-then-keep heartbeat
27
+
28
+ ### Features
29
+
30
+ * core-validated sealed values, private Normal/Statistics/Suspect/Scan fields, Scan#suspects, clone-then-keep heartbeat ([e155c5f](https://github.com/seuros/suspect_machines/commit/e155c5f59aa257252775378d9763bd676a669ed9))
31
+
32
+ ## [0.3.0](https://github.com/seuros/suspect_machines/compare/suspect-machines-v0.2.0...suspect-machines-v0.3.0) (2026-11-13)
33
+
34
+
35
+ ### âš  BREAKING CHANGES
36
+
37
+ * see CONVENTIONS: one exhaustive Refusal per crate, Result from every mutator, honoured idempotent repeats, Ruby mirrors the core. No compatibility aliases. Refusal::Duplicate is renamed SameInstant; Roster::forget returns Forgetting.
38
+
39
+ ### Features
40
+
41
+ * Forgetting outcome, SameInstant refusal, Ruby Config, Detector and scan mirror the core ([c936cbe](https://github.com/seuros/suspect_machines/commit/c936cbe6840e915099cacf4589824c9e829d784c))
42
+
43
+ ## [0.2.0](https://github.com/seuros/suspect_machines/compare/suspect-machines-v0.1.0...suspect-machines-v0.2.0) (2026-11-12)
44
+
45
+
46
+ ### Features
47
+
48
+ * initial extraction ([9ae1485](https://github.com/seuros/suspect_machines/commit/9ae1485a042ea92dbdb26c143f19c40797247327))
@@ -0,0 +1,17 @@
1
+ [package]
2
+ name = "suspect-machines"
3
+ version = "0.5.1"
4
+ edition.workspace = true
5
+ rust-version.workspace = true
6
+ license.workspace = true
7
+ description = "Bounded heartbeat interval statistics and advisory phi-accrual suspicion"
8
+ readme = "README.md"
9
+ exclude = ["src/tests/", "src/**/tests.rs", "tests/", "src/test_support.rs"]
10
+ keywords = ["failure-detector", "phi-accrual", "heartbeat", "suspicion", "no-std"]
11
+ categories = ["no-std", "algorithms"]
12
+
13
+ [dev-dependencies]
14
+ matryona-machines = { workspace = true, features = ["testing"] }
15
+
16
+ [lints]
17
+ workspace = true
@@ -0,0 +1,115 @@
1
+ # suspect-machines
2
+
3
+ > **AMS Hayashibara**, NCC-4223, *Hayashibara*-class picket vessel.
4
+
5
+ Engine-room flight manual: turn heartbeat arrival times into a suspicion level,
6
+ and nothing more than a suspicion level.
7
+
8
+ Allocation-free `no_std` phi-accrual failure detector (Hayashibara et al.,
9
+ 2004). A `Detector<W>` keeps the last `W` heartbeat intervals of one peer and
10
+ their mean and population standard deviation. `Roster<P, W>` keeps up to `P`
11
+ such windows keyed by a caller-assigned `u64` peer id and an incarnation
12
+ number. Asked at an explicit instant, either one answers with
13
+ `phi = -log10(P(a heartbeat still arrives after this much silence))`.
14
+
15
+ Everything the detector assumes is stated by the caller:
16
+
17
+ - **Distribution.** `Distribution::Normal(Normal::new(min_std_dev)?)` uses the
18
+ window mean and standard deviation, floored at `min_std_dev` (`Normal::new`
19
+ refuses a zero floor with `ConfigError::ZeroDeviationFloor`).
20
+ `Distribution::Exponential` uses only the mean. Neither is verified against
21
+ the data.
22
+ - **Warm-up.** `Config::new(distribution, min_samples)`: until `min_samples`
23
+ intervals are in the window, the answer is `Suspicion::WarmingUp` with the
24
+ current silence, never a phi computed from too little data.
25
+ - **Acceptable pause.** `with_acceptable_pause` tolerates a fixed extra
26
+ silence before suspicion starts to accrue.
27
+
28
+ Time is explicit: the caller passes a monotonic `Duration` to every call. A
29
+ heartbeat earlier than the previous one is `Refusal::TimeRegression`, an equal
30
+ one is `Refusal::SameInstant`, a time beyond `u64` nanoseconds is
31
+ `Refusal::Overflow`. A refused call changes nothing. A full window drops its
32
+ oldest interval; a full roster answers `Refusal::Full` and never evicts a peer.
33
+ A heartbeat with an older incarnation is `Refusal::Stale`; a newer one discards
34
+ the peer's statistics and starts a new warm-up. `Roster::scan` writes peers at
35
+ or above a threshold into a caller buffer and returns a cursor when it runs out
36
+ of room; a peer whose last heartbeat is later than the scan instant has no
37
+ silence to measure and is skipped, not refused (`Roster::suspicion` refuses it
38
+ with `TimeRegression`). `Roster::forget` never refuses: it answers `Forgetting::Forgotten`, or
39
+ `Forgetting::NotTracked` for the idempotent repeat that changed nothing.
40
+
41
+ Every value has a validating constructor, its fields are private behind
42
+ readers, and every answer passes it: `Normal::new` (a floor of `1 ns` to
43
+ `u64::MAX` ns), `Phi::new` (finite, not negative), `Threshold::new` (finite,
44
+ above zero),
45
+ `Beat::recorded::<W>` (a non-zero interval, `1..=W` samples),
46
+ `Suspicion::warming_up::<W>` (`samples < required <= W`),
47
+ `Statistics::new::<W>` (`1..=W` samples, `1 ns <= min <= max` with one
48
+ sample its own min and max, a mean within `min..=max`, a positively signed
49
+ deviation of at most `(max - min) / 2`), `Suspect::new` (any peer and incarnation)
50
+ and `Scan::new::<P>` (at most `P` suspects, a resume cursor below `P`). Each
51
+ answers `None` (or the `ConfigError`) for anything else.
52
+
53
+ Clock: every honoured heartbeat (`First`, `Recorded`, `Restarted`) records
54
+ `now` as the last heartbeat; it is the observation itself, and there is no
55
+ honoured repeat. `suspicion` and `scan` take `&self` and record nothing.
56
+
57
+ The Ruby gem passes and answers every `Duration` as Integer nanoseconds and
58
+ mirrors `Phi`, `Threshold`, `Normal`, `Config`, `Beat::Recorded`,
59
+ `Suspicion::WarmingUp`, `Suspicion::Measured`, `Statistics`, `Suspect` and
60
+ `Scan` as `Data` values with the same fields (its `Scan#suspects` holds the
61
+ suspects themselves where `Scan::suspects` counts those written to `out`),
62
+ each checked through the constructors above
63
+ when built. Unit variants are symbols.
64
+
65
+ Phi is advisory. It describes arrival statistics under an assumed
66
+ distribution. It is not a verdict that a peer is dead, it does not revoke a
67
+ lease or an election, and it cannot stand in for durable authority or
68
+ resource-side fencing.
69
+
70
+ ```rust
71
+ use core::time::Duration;
72
+ use suspect_machines::{Beat, Config, Detector, Distribution, Normal, Suspicion, Threshold};
73
+
74
+ let normal = Normal::new(Duration::from_millis(100)).expect("a non-zero floor");
75
+ let config = Config::new(Distribution::Normal(normal), 3).expect("valid policy");
76
+ let mut detector = Detector::<16>::new(config).expect("warm-up fits the window");
77
+
78
+ assert_eq!(detector.heartbeat(Duration::from_secs(0)), Ok(Beat::First));
79
+ for second in 1..=3 {
80
+ detector.heartbeat(Duration::from_secs(second)).expect("monotonic");
81
+ }
82
+ // Regular 1 s beats, floor 100 ms: 1.3 s of silence is three deviations late.
83
+ let Ok(Suspicion::Measured(phi)) = detector.suspicion(Duration::from_millis(4_300)) else {
84
+ panic!("warm-up is over");
85
+ };
86
+ assert!((phi.value() - 2.8697).abs() < 1e-4);
87
+ assert!(!phi.reaches(Threshold::new(8.0).expect("finite and positive")));
88
+ ```
89
+
90
+ ## Arithmetic
91
+
92
+ `f64::sqrt`, `exp` and `ln` live in `std`, so the crate carries its own, built
93
+ only from IEEE-754 add, subtract, multiply and divide in a fixed order with
94
+ fixed iteration counts. No `libm`, no fused multiply-add, no lookup of platform
95
+ behavior: the same calls give the same bits on every target.
96
+
97
+ The normal tail is evaluated in the log domain (a positive-term `erf` series
98
+ below `|z| = 2.83`, a 60-level `erfc` continued fraction above), so phi stays
99
+ finite and accurate long after the probability itself would underflow.
100
+
101
+ | routine | domain | bound against the exact value |
102
+ |:--------|:-------|:------------------------------|
103
+ | `sqrt` | finite `x >= 0` | relative `4e-16` |
104
+ | `exp` | `-708 <= x <= 709` | relative `1e-14` |
105
+ | `ln` | positive finite `x` | `1e-15` absolute or `1e-14` relative |
106
+ | normal phi from `z` | finite `z` | `1e-12` absolute or `1e-12` relative |
107
+
108
+ The tests check these against `std`, against 80-digit reference values and
109
+ against an independent quadrature of the Gaussian tail. The bounds cover the
110
+ function evaluation. They do not cover the inputs: intervals are converted to
111
+ `f64` nanoseconds, which is exact up to `2^53` ns (about 104 days) and rounded
112
+ to nearest beyond that.
113
+
114
+ Statistics are recomputed from the window on each read: `O(W)` per suspicion,
115
+ `O(P * W)` per full scan, with an exact integer sum behind the mean.
@@ -0,0 +1,90 @@
1
+ //! Phi-accrual detector for a single peer.
2
+
3
+ use core::time::Duration;
4
+
5
+ use crate::{Beat, Config, ConfigError, Refusal, Statistics, Suspicion, instant, window::Window};
6
+
7
+ /// Heartbeat statistics and suspicion for one peer over the last `W`
8
+ /// intervals.
9
+ #[derive(Debug, Clone, PartialEq, Eq)]
10
+ pub struct Detector<const W: usize> {
11
+ config: Config,
12
+ window: Window<W>,
13
+ }
14
+
15
+ impl<const W: usize> Detector<W> {
16
+ /// Intervals the window holds.
17
+ pub const WINDOW: usize = W;
18
+
19
+ /// An empty detector.
20
+ ///
21
+ /// # Errors
22
+ ///
23
+ /// [`ConfigError::ZeroWindow`] when `W` is zero and
24
+ /// [`ConfigError::WarmUpExceedsWindow`] when the warm-up could never
25
+ /// finish.
26
+ pub const fn new(config: Config) -> Result<Self, ConfigError> {
27
+ match config.fit(W) {
28
+ Ok(config) => Ok(Self {
29
+ config,
30
+ window: Window::new(),
31
+ }),
32
+ Err(error) => Err(error),
33
+ }
34
+ }
35
+
36
+ /// Records a heartbeat observed at `now`, a caller-supplied monotonic
37
+ /// time. A full window drops its oldest interval.
38
+ ///
39
+ /// # Errors
40
+ ///
41
+ /// [`Refusal::Overflow`], [`Refusal::TimeRegression`] or
42
+ /// [`Refusal::SameInstant`]; the detector is unchanged.
43
+ pub fn heartbeat(&mut self, now: Duration) -> Result<Beat, Refusal> {
44
+ self.window.observe(instant(now)?)
45
+ }
46
+
47
+ /// The suspicion at `now`. Reading never changes the detector.
48
+ ///
49
+ /// # Errors
50
+ ///
51
+ /// [`Refusal::Overflow`], or [`Refusal::TimeRegression`] when `now`
52
+ /// precedes the last heartbeat.
53
+ pub fn suspicion(&self, now: Duration) -> Result<Suspicion, Refusal> {
54
+ self.window
55
+ .suspicion(&self.config, instant(now)?)
56
+ .ok_or(Refusal::TimeRegression)
57
+ }
58
+
59
+ /// Statistics of the stored intervals; `None` while there are none.
60
+ #[must_use]
61
+ pub fn statistics(&self) -> Option<Statistics> {
62
+ self.window.statistics()
63
+ }
64
+
65
+ /// Intervals currently in the window.
66
+ #[must_use]
67
+ pub const fn samples(&self) -> usize {
68
+ self.window.len()
69
+ }
70
+
71
+ /// Time of the last recorded heartbeat.
72
+ #[must_use]
73
+ pub const fn last_heartbeat(&self) -> Option<Duration> {
74
+ match self.window.last() {
75
+ Some(last) => Some(Duration::from_nanos(last)),
76
+ None => None,
77
+ }
78
+ }
79
+
80
+ /// The policy this detector was built with.
81
+ #[must_use]
82
+ pub const fn config(&self) -> Config {
83
+ self.config
84
+ }
85
+
86
+ /// Forgets every interval and the last heartbeat; warm-up starts over.
87
+ pub const fn reset(&mut self) {
88
+ self.window.reset();
89
+ }
90
+ }