handoff_machines 0.8.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +113 -0
- data/Cargo.lock +331 -0
- data/Cargo.toml +30 -0
- data/LICENSE.txt +21 -0
- data/README.md +383 -0
- data/ext/handoff_machines_native/core/CHANGELOG.md +81 -0
- data/ext/handoff_machines_native/core/Cargo.toml +22 -0
- data/ext/handoff_machines_native/core/README.md +215 -0
- data/ext/handoff_machines_native/core/src/iter.rs +130 -0
- data/ext/handoff_machines_native/core/src/lib.rs +498 -0
- data/ext/handoff_machines_native/core/src/refusal.rs +83 -0
- data/ext/handoff_machines_native/core/src/transfer.rs +437 -0
- data/ext/handoff_machines_native/core/src/types.rs +440 -0
- data/ext/handoff_machines_native/extconf.rb +2 -0
- data/ext/handoff_machines_native/ffi/Cargo.toml +21 -0
- data/ext/handoff_machines_native/ffi/extconf.rb +14 -0
- data/ext/handoff_machines_native/ffi/src/classes.rs +50 -0
- data/ext/handoff_machines_native/ffi/src/handoff.rs +375 -0
- data/ext/handoff_machines_native/ffi/src/init.rs +62 -0
- data/ext/handoff_machines_native/ffi/src/lib.rs +36 -0
- data/ext/handoff_machines_native/ffi/src/names.rs +86 -0
- data/ext/handoff_machines_native/ffi/src/values.rs +211 -0
- data/lib/handoff_machines/version.rb +4 -0
- data/lib/handoff_machines.rb +263 -0
- data/sig/handoff_machines.rbs +291 -0
- metadata +93 -0
data/README.md
ADDED
|
@@ -0,0 +1,383 @@
|
|
|
1
|
+
# HandoffMachines
|
|
2
|
+
|
|
3
|
+
> Two replicas, one partition, and exactly one of them is allowed to be right.
|
|
4
|
+
|
|
5
|
+
A transfer coordinator for Ruby. When a partition has to move from one replica to another, the coordinator walks the move through four phases: quiesce the source, drain what is in flight, checkpoint, activate the target. It tells you what is owed next and by whom. Each phase ends only when the replica that owes it says so, with a ticket that proves the answer belongs to this move and not to last Tuesday's.
|
|
6
|
+
|
|
7
|
+
## Quick Start
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
gem 'handoff_machines'
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
```ruby
|
|
14
|
+
limits = HandoffMachines::Limits.new(30_000_000_000, 2) # phase_budget, per_replica
|
|
15
|
+
handoff = HandoffMachines::Handoff.new(limits)
|
|
16
|
+
|
|
17
|
+
# Fencing tokens are values your lease authority issues. The source serves under `held`.
|
|
18
|
+
held = HandoffMachines::FencingToken.from_raw(7)
|
|
19
|
+
|
|
20
|
+
# partition, source, target, base, source_token, now
|
|
21
|
+
ticket = handoff.start(42, 1, 2, 100, held, clock.now)
|
|
22
|
+
raise "refused: #{ticket}" if ticket.is_a?(Symbol)
|
|
23
|
+
|
|
24
|
+
orders, = handoff.orders(0, 32)
|
|
25
|
+
orders.each { |order| deliver(order) } # => Order(to: Replica(1), action: :quiesce)
|
|
26
|
+
|
|
27
|
+
# ...and as the replies come back:
|
|
28
|
+
handoff.quiesced(ticket, 1, 102, clock.now) # => Recorded(:drain)
|
|
29
|
+
handoff.drained(ticket, 1, 100, clock.now) # => Recorded(:drain)
|
|
30
|
+
handoff.drained(ticket, 1, 101, clock.now) # => Recorded(:checkpoint)
|
|
31
|
+
handoff.captured(ticket, 1, 9, clock.now) # => Recorded(:checkpoint)
|
|
32
|
+
handoff.installed(ticket, 2, 9, clock.now) # => Recorded(:activate)
|
|
33
|
+
granted = HandoffMachines::FencingToken.from_raw(8) # the authority's answer to the Fence order
|
|
34
|
+
handoff.fenced(ticket, granted, clock.now) # => Recorded(:activate)
|
|
35
|
+
handoff.activated(ticket, 2, granted, clock.now) # => Recorded(:complete)
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
One coordinator, one method per acknowledgement. The Ruby API is the core's, name for name: `HandoffMachines::Handoff` is the core's `Handoff`, every `Data` value has the fields of the core struct or enum variant it mirrors, and arguments come in the core's order. Partitions, replicas, offsets and checkpoints are Integers; fencing tokens are `FencingToken` values. Time is Integer nanoseconds on one monotonic clock, `Process.clock_gettime(Process::CLOCK_MONOTONIC, :nanosecond)`, for instants (`now`, `deadline`) and spans (`phase_budget`) alike. No floats, no milliseconds, no `Time.now`.
|
|
39
|
+
|
|
40
|
+
It is a policy core, not a runtime: it sends nothing, stores nothing and issues no tokens.
|
|
41
|
+
|
|
42
|
+
## A Message to the Resistance
|
|
43
|
+
|
|
44
|
+
So you added a node and the cluster needs to rebalance. Partition 42 has to move from the old box to the new one. You wrote the obvious thing: tell the new node it owns 42, tell the old node it doesn't, copy the data across. It worked in staging.
|
|
45
|
+
|
|
46
|
+
In production the "you don't own it" message sat in a queue for four seconds. For those four seconds both nodes took writes for partition 42. The copy had already been cut, so the old node's last few hundred writes went nowhere. Nobody noticed until a customer asked where their order was.
|
|
47
|
+
|
|
48
|
+
So you added a flag. Then a sleep. Then a "migration_state" column with values like `copying_2` that only one person understands. Then the copy failed halfway and you discovered there was no way back: the old node had stopped taking writes and the new one had half a partition.
|
|
49
|
+
|
|
50
|
+
Every one of those patches is a piece of the same small protocol. Stop the source. Wait for what it already accepted. Cut the copy at a known point. Switch owners in a way the storage layer can enforce. And at every step, know how to undo it. That protocol is this gem.
|
|
51
|
+
|
|
52
|
+
## Chapter 1: Stardate 2026.316, the AMS DeCandia
|
|
53
|
+
|
|
54
|
+
> **AMS DeCandia**, NCC-4234, *DeCandia*-class transfer tender.
|
|
55
|
+
> Cargo changes hands at the airlock, one signature at a time.
|
|
56
|
+
|
|
57
|
+
The tender doesn't carry cargo. It runs the transfer. Two ships dock, and the tender's deck officer holds a clipboard with four lines on it: bay sealed, loaders clear, manifest matched, new crew has the keys. Nobody moves to the next line until the last one is signed, and the officer checks every signature against the transfer number at the top of the page. A signature for an older transfer goes in the bin.
|
|
58
|
+
|
|
59
|
+
On a bridge somewhere, a cadet volunteers:
|
|
60
|
+
|
|
61
|
+
```ruby
|
|
62
|
+
def move(partition, from, to)
|
|
63
|
+
to.assign(partition)
|
|
64
|
+
from.unassign(partition)
|
|
65
|
+
copy(partition, from, to)
|
|
66
|
+
end
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
"That," says the deck officer, "is two captains and no cargo."
|
|
70
|
+
|
|
71
|
+
## The Flight Plan
|
|
72
|
+
|
|
73
|
+
Each transfer the coordinator knows about is in one of these phases:
|
|
74
|
+
|
|
75
|
+
```mermaid
|
|
76
|
+
stateDiagram-v2
|
|
77
|
+
[*] --> quiesce: start, returns a Ticket
|
|
78
|
+
quiesce --> drain: quiesced(until)
|
|
79
|
+
quiesce --> checkpoint: quiesced, nothing in flight
|
|
80
|
+
drain --> drain: drained(offset)
|
|
81
|
+
drain --> checkpoint: last offset drained
|
|
82
|
+
checkpoint --> checkpoint: captured(checkpoint)
|
|
83
|
+
checkpoint --> activate: installed(checkpoint)
|
|
84
|
+
activate --> activate: fenced(token)
|
|
85
|
+
activate --> [*]: activated(token), :complete
|
|
86
|
+
quiesce --> rollback: abort
|
|
87
|
+
drain --> rollback: abort
|
|
88
|
+
checkpoint --> rollback: abort
|
|
89
|
+
activate --> rollback: abort (token required once fenced)
|
|
90
|
+
rollback --> [*]: resumed and discarded, :aborted
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
And the afternoon partition 42 moved from replica 1 to replica 2:
|
|
94
|
+
|
|
95
|
+
```mermaid
|
|
96
|
+
sequenceDiagram
|
|
97
|
+
participant S as Replica 1 (source)
|
|
98
|
+
participant C as Coordinator
|
|
99
|
+
participant A as Lease authority
|
|
100
|
+
participant T as Replica 2 (target)
|
|
101
|
+
|
|
102
|
+
C->>S: quiesce
|
|
103
|
+
S-->>C: quiesced, until 102
|
|
104
|
+
C->>S: drain until 102
|
|
105
|
+
S-->>C: drained 100
|
|
106
|
+
S-->>C: drained 101
|
|
107
|
+
C->>S: capture at 102
|
|
108
|
+
S-->>C: captured checkpoint 9
|
|
109
|
+
C->>T: install checkpoint 9
|
|
110
|
+
T-->>C: installed 9
|
|
111
|
+
C->>A: fence above token 7
|
|
112
|
+
A-->>C: token 8
|
|
113
|
+
C->>T: activate under token 8
|
|
114
|
+
T-->>C: activated 8
|
|
115
|
+
Note over S: token 7 is now stale. The storage layer refuses it.
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
The arrows out of the coordinator are orders you deliver. The arrows in are the methods you call.
|
|
119
|
+
|
|
120
|
+
## How the Coordinator Works
|
|
121
|
+
|
|
122
|
+
### Start
|
|
123
|
+
|
|
124
|
+
```ruby
|
|
125
|
+
limits = HandoffMachines::Limits.new(30_000_000_000, 2)
|
|
126
|
+
handoff = HandoffMachines::Handoff.new(limits)
|
|
127
|
+
handoff.limits # => #<data HandoffMachines::Limits phase_budget=30000000000, per_replica=2>
|
|
128
|
+
|
|
129
|
+
held = HandoffMachines::FencingToken.from_raw(7)
|
|
130
|
+
ticket = handoff.start(42, 1, 2, 100, held, 0)
|
|
131
|
+
ticket # => #<data HandoffMachines::Ticket partition=42, generation=1>
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
`Limits` is `phase_budget` (how long one phase may last before `overdue` reports it) and `per_replica` (how many transfers one replica may take part in). `start(partition, source, target, base, source_token, now)` takes the core's arguments in the core's order: `base` is the lowest offset the source has not durably committed, and `source_token` is the fencing token the source serves under today; its scope is the transfer's token scope. The `Ticket` carries a generation that is never issued twice. Send it along with every order and have the replica send it back.
|
|
135
|
+
|
|
136
|
+
### Orders
|
|
137
|
+
|
|
138
|
+
```ruby
|
|
139
|
+
orders, position = handoff.orders(0, 32)
|
|
140
|
+
orders.first # => #<data HandoffMachines::Order ticket=..., to=#<data HandoffMachines::Replica id=1>, action=:quiesce>
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
Orders are not a queue. They are a restatement of what each transfer is waiting for, so asking twice gives the same answer until somebody acknowledges. A transfer owes at most two orders at a time. Take them in chunks if you like:
|
|
144
|
+
|
|
145
|
+
```ruby
|
|
146
|
+
chunk, position = handoff.orders(0, 4)
|
|
147
|
+
chunk, position = handoff.orders(position, 4) # carries on where the last one stopped
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
`orders(from, limit)` answers at most `limit` orders from position `from` (zero to begin) and the position to resume at.
|
|
151
|
+
|
|
152
|
+
`to` is a `Replica(id)`, or `:authority` when the order is for whoever issues your fencing tokens. `action` says what they must do, one value per core `Action` variant:
|
|
153
|
+
|
|
154
|
+
| `action` | For | Meaning |
|
|
155
|
+
| --- | --- | --- |
|
|
156
|
+
| `:quiesce` | source | stop admitting new work and report the quiesce point |
|
|
157
|
+
| `Drain(until)` | source | finish in-flight offsets below `until` and acknowledge each |
|
|
158
|
+
| `Capture(at)` | source | capture a checkpoint at exactly `at` and report its identity |
|
|
159
|
+
| `Install(checkpoint)` | target | install the named checkpoint |
|
|
160
|
+
| `Fence(above)` | authority | issue the target a token strictly greater than `above` |
|
|
161
|
+
| `Activate(token)` | target | start serving under `token` |
|
|
162
|
+
| `Resume(token)` | source | resume serving, under `token` when the rollback was fenced (else `nil`) |
|
|
163
|
+
| `:discard` | target | stop serving if it started, and drop the installed checkpoint |
|
|
164
|
+
|
|
165
|
+
### Quiesce
|
|
166
|
+
|
|
167
|
+
```ruby
|
|
168
|
+
handoff.quiesced(ticket, 1, 102, 1) # => #<data HandoffMachines::Recorded phase=:drain>
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
The source stopped admitting new work and reports its quiesce point: one past the last offset it accepted. If that equals `base`, nothing is in flight and the answer is `Recorded(:checkpoint)` directly.
|
|
172
|
+
|
|
173
|
+
Every acknowledgement answers the core's `Acked`: `Recorded(phase)` the first time, `Duplicate(phase)` for a repeat, where `phase` is where the transfer stands after the call. Both answer `duplicate?`. A refusal is a Symbol instead.
|
|
174
|
+
|
|
175
|
+
### Drain
|
|
176
|
+
|
|
177
|
+
```ruby
|
|
178
|
+
handoff.drained(ticket, 1, 101, 2) # => Recorded(:drain)
|
|
179
|
+
handoff.outstanding(ticket, 0, 256) # => [100]
|
|
180
|
+
handoff.drained(ticket, 1, 100, 3) # => Recorded(:checkpoint)
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
Every offset between `base` and the quiesce point must be acknowledged, in any order. The phase ends when there are no holes left. `outstanding(ticket, from, limit)` lists at most `limit` holes, ascending from offset `from`; ask again from the last one plus one to page through a long list.
|
|
184
|
+
|
|
185
|
+
Replies get delivered twice. That is not an error:
|
|
186
|
+
|
|
187
|
+
```ruby
|
|
188
|
+
handoff.drained(ticket, 1, 101, 2) # => Recorded(:drain)
|
|
189
|
+
handoff.drained(ticket, 1, 101, 2) # => Duplicate(:drain), it was already acknowledged
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
`Duplicate` is not a refusal. The offset is acknowledged, which is what the source was telling you. It was just true before it said so again. Nothing changes: not the phase, not the window, not the clock.
|
|
193
|
+
|
|
194
|
+
Every acknowledgement works this way, not just `drained`. The coordinator answers a repeat from what the transfer recorded, for as long as the transfer lives, and `Duplicate#phase` is where the transfer stands now:
|
|
195
|
+
|
|
196
|
+
```ruby
|
|
197
|
+
handoff.quiesced(ticket, 1, 102, 1) # => Recorded(:drain)
|
|
198
|
+
handoff.quiesced(ticket, 1, 102, 9) # => Duplicate(:activate), three phases later
|
|
199
|
+
handoff.quiesced(ticket, 1, 105, 9) # => :mismatch, that is not what was recorded
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
Same story, different payload, is not a repeat. A quiesce point, checkpoint or token that differs from the recorded one is `:mismatch`. And `:out_of_phase` means only this: the transfer is not waiting for that acknowledgement and never recorded it. It came too early, or the transfer was aborted before it arrived.
|
|
203
|
+
|
|
204
|
+
A drain can track 256 offsets at a time. If more than that is in flight, confirm what the source has durably committed and the window slides forward:
|
|
205
|
+
|
|
206
|
+
```ruby
|
|
207
|
+
handoff.drained(ticket, 1, 100 + 256, 2) # => :out_of_window
|
|
208
|
+
handoff.committed(ticket, 1, 101, 2) # => Recorded(:drain), window now starts at 101
|
|
209
|
+
handoff.committed(ticket, 1, 101, 2) # => Duplicate(:drain), already committed
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
### Checkpoint
|
|
213
|
+
|
|
214
|
+
```ruby
|
|
215
|
+
handoff.captured(ticket, 1, 9, 4) # => Recorded(:checkpoint) (source captured checkpoint 9)
|
|
216
|
+
handoff.installed(ticket, 2, 9, 5) # => Recorded(:activate) (target installed the same one)
|
|
217
|
+
handoff.installed(ticket, 2, 8, 5) # would have been :mismatch
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
The source captures state at exactly the quiesce point and names it. The target must acknowledge that same name. The coordinator never sees the bytes.
|
|
221
|
+
|
|
222
|
+
### Activate
|
|
223
|
+
|
|
224
|
+
```ruby
|
|
225
|
+
handoff.orders(0, 32).first.first # => Order(to: :authority, action: Fence(above: FencingToken(raw: 7)))
|
|
226
|
+
granted = HandoffMachines::FencingToken.from_raw(8)
|
|
227
|
+
handoff.fenced(ticket, granted, 6) # => Recorded(:activate)
|
|
228
|
+
handoff.activated(ticket, 2, granted, 7) # => Recorded(:complete)
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
First the authority issues the target a token strictly newer than the source's. A token of the source token's scope that isn't newer is `:stale_token`. Then the target acknowledges it is serving under that exact token. `Recorded(:complete)` frees the slot.
|
|
232
|
+
|
|
233
|
+
### Fencing tokens
|
|
234
|
+
|
|
235
|
+
A `FencingToken` is the lease core's token, which the handoff core re-exports: plain data whose `raw` is the issuing authority's scope in the top 16 bits above a sequence number counted from 1. The coordinator issues none. Every method that takes a token (`start`, `fenced`, `activated`, `abort`) reads its `raw`, so a `LeaseMachines::FencingToken` fits as it is: pass `lease.token`, not `lease.token.raw`. This gem does not depend on that one. A bare Integer is not a token and raises `TypeError`; rebuild one that crossed a wire with `HandoffMachines::FencingToken.from_raw(raw)`. Tokens come back as `HandoffMachines::FencingToken` in `Fence#above`, `Activate#token`, `Resume#token` and `Status#token`.
|
|
236
|
+
|
|
237
|
+
Tokens of one scope are ordered. Tokens of two scopes are not: there is no `<`, `order` answers `nil` and `newer_than?` answers `false`. So a transfer reads tokens only inside its source token's scope, and a token of any other scope is `:unknown`, whatever its number, from every call that takes one: `fenced`, `activated` and `abort`. That is the fleet's name for an id that is not of the namespace the call reads it in. It is answered before the transfer's phase or its recorded token is looked at, so it is never folded into "older" (`:stale_token`), "not the one recorded" (`:mismatch`) or "not now" (`:out_of_phase`). While the transfer still waits for the authority's token:
|
|
238
|
+
|
|
239
|
+
```ruby
|
|
240
|
+
other = HandoffMachines::FencingToken.from_raw((3 << 48) | 8) # scope 3, the source token is scope 0
|
|
241
|
+
handoff.fenced(ticket, other, 6) # => :unknown, nothing changed
|
|
242
|
+
handoff.fenced(ticket, held, 6) # => :stale_token, right scope, not newer
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
One transfer therefore lives under one authority scope. If your authority came back under a new scope before `fenced`, give the transfer up with `abort(ticket, nil, now)` and start it again under a source token of the new scope.
|
|
246
|
+
|
|
247
|
+
### Abort
|
|
248
|
+
|
|
249
|
+
```ruby
|
|
250
|
+
handoff.abort(ticket, nil, 5) # => Recorded(:rollback)
|
|
251
|
+
handoff.orders(0, 32).first.map(&:action) # => [Resume(token: nil), :discard]
|
|
252
|
+
handoff.resumed(ticket, 1, 6) # => Recorded(:rollback)
|
|
253
|
+
handoff.discarded(ticket, 2, 6) # => Recorded(:aborted)
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
`abort(ticket, token, now)` takes the rollback token in the core's position; pass `nil` for an unfenced abort.
|
|
257
|
+
|
|
258
|
+
You can abort from any phase. The source always owes `resumed`, because it may have quiesced. The target owes `discarded` only once it was asked to install something.
|
|
259
|
+
|
|
260
|
+
A `resumed` or `discarded` the rollback is not waiting for, because it already arrived or because the target was never asked to install anything, answers `Duplicate(:rollback)` while the rollback is still open. So does a second `abort` with the same token (or the same `nil`); with a different one it is `:mismatch`. Replies the transfer recorded before the abort still answer `Duplicate(:rollback)` too. Once the rollback ends as `Recorded(:aborted)` the ticket is `:stale`.
|
|
261
|
+
|
|
262
|
+
After `fenced`, activation has been ordered and the target may already be serving. A plain abort is refused:
|
|
263
|
+
|
|
264
|
+
```ruby
|
|
265
|
+
handoff.abort(ticket, nil, 8) # => :fence_required
|
|
266
|
+
newer = HandoffMachines::FencingToken.from_raw(9)
|
|
267
|
+
handoff.abort(ticket, newer, 8) # => Recorded(:rollback), source resumes under token 9
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
The rollback token must be of the same scope and beat the activation token, so the target is fenced out the same way the source would have been. One of another scope is `:unknown`, whatever else is wrong with the call; one of the right scope that doesn't beat it is `:stale_token`.
|
|
271
|
+
|
|
272
|
+
### Stale tickets
|
|
273
|
+
|
|
274
|
+
```ruby
|
|
275
|
+
handoff.quiesced(old_ticket, 1, 102, 9) # => :stale
|
|
276
|
+
handoff.quiesced(HandoffMachines::Ticket.new(42, 999), 1, 0, 9) # => :unknown
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
A ticket whose transfer has finished, aborted or been replaced is `:stale`. A generation this coordinator never issued is `:unknown`, and that includes every ticket from a coordinator of another scope. Neither changes anything.
|
|
280
|
+
|
|
281
|
+
### Capacity
|
|
282
|
+
|
|
283
|
+
The coordinator holds 16 transfers at a time, and each replica can take part in at most `per_replica` of them:
|
|
284
|
+
|
|
285
|
+
```ruby
|
|
286
|
+
HandoffMachines::CAPACITY # => 16
|
|
287
|
+
HandoffMachines::WINDOW # => 256
|
|
288
|
+
|
|
289
|
+
handoff.start(43, 1, 2, 0, 3, 0) # => Ticket
|
|
290
|
+
handoff.start(44, 1, 3, 0, 3, 0) # => :replica_busy
|
|
291
|
+
handoff.start(42, 4, 5, 0, 3, 0) # => :busy, 42 is already moving
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
Past sixteen, `start` says `:full` until a transfer completes or finishes rolling back.
|
|
295
|
+
|
|
296
|
+
### Time only moves forward
|
|
297
|
+
|
|
298
|
+
```ruby
|
|
299
|
+
handoff.quiesced(ticket, 1, 102, 10) # => Recorded(:drain)
|
|
300
|
+
handoff.drained(ticket, 1, 100, 9) # => :clock_regressed
|
|
301
|
+
```
|
|
302
|
+
|
|
303
|
+
Each phase gets `Limits#phase_budget` from the moment it begins. The coordinator will tell you who is over:
|
|
304
|
+
|
|
305
|
+
```ruby
|
|
306
|
+
late, position = handoff.overdue(clock.now, 0, 16) # => [[Ticket, ...], 16]
|
|
307
|
+
```
|
|
308
|
+
|
|
309
|
+
`overdue(now, from, limit)` answers at most `limit` tickets from slot `from` (zero to begin) and the slot to resume at.
|
|
310
|
+
|
|
311
|
+
That's all it does about it. A late acknowledgement is still accepted, and nothing is aborted for you. A source that has been quiesced for too long is your call.
|
|
312
|
+
|
|
313
|
+
### Every answer
|
|
314
|
+
|
|
315
|
+
A call that is honoured answers with a value. A call that is not honoured answers with one of the refusal symbols below, and only those, and changes nothing. Each refusal is the snake_case of one core `Refusal` variant; there is no catch-all symbol. `Duplicate(phase)` is an honoured repeat: the transfer already recorded that acknowledgement with that payload, and nothing changed.
|
|
316
|
+
|
|
317
|
+
Every acknowledgement can also refuse with `:clock_regressed`, `:stale` or `:unknown` (the *ticket refusals*). From `fenced`, `activated` and `abort`, `:unknown` can also be a fencing token of another scope than the transfer's source token. Refusals are listed in the order they are checked, and the first that applies is the answer: the ticket refusals first (`:clock_regressed`, then `:unknown` or `:stale` for the ticket), then the rest left to right. `:out_of_phase` always means "not awaited and never recorded": too early, or the transfer was given up first. `:mismatch` means the payload is not the recorded one. `activated`, and the `resumed` or `discarded` that ends a rollback, free the slot: their repeats are `:stale`.
|
|
318
|
+
|
|
319
|
+
| Call | Honoured | Refusals | Records the clock |
|
|
320
|
+
| --- | --- | --- | --- |
|
|
321
|
+
| `start(partition, source, target, base, source_token, now)` | `Ticket` | `:clock_regressed`, `:same_replica`, `:busy`, `:full`, `:replica_busy`, `:exhausted` | yes: the first deadline counts from it |
|
|
322
|
+
| `quiesced(ticket, replica, until_offset, now)` | `Recorded(:drain)`, `Recorded(:checkpoint)`, `Duplicate` (same `until_offset` again) | ticket refusals, `:out_of_phase`, `:wrong_replica`, `:behind_base`, `:mismatch` (different `until_offset`) | `Recorded` yes (a new wait); `Duplicate` no |
|
|
323
|
+
| `drained(ticket, replica, offset, now)` | `Recorded(:drain)`, `Recorded(:checkpoint)`, `Duplicate` (offset already acknowledged or committed) | ticket refusals, `:out_of_phase`, `:wrong_replica`, `:beyond_drain`, `:out_of_window` | `Recorded(:checkpoint)` yes (a new wait); `Recorded(:drain)` and `Duplicate` no |
|
|
324
|
+
| `committed(ticket, replica, offset, now)` | `Recorded(:drain)`, `Duplicate` (offset already committed) | ticket refusals, `:out_of_phase`, `:wrong_replica`, `:unacked` | no |
|
|
325
|
+
| `captured(ticket, replica, checkpoint, now)` | `Recorded(:checkpoint)`, `Duplicate` (same checkpoint again) | ticket refusals, `:out_of_phase`, `:wrong_replica`, `:mismatch` (different checkpoint) | `Recorded` yes (a new wait); `Duplicate` no |
|
|
326
|
+
| `installed(ticket, replica, checkpoint, now)` | `Recorded(:activate)`, `Duplicate` (same checkpoint again) | ticket refusals, `:out_of_phase`, `:wrong_replica`, `:mismatch` (not the captured checkpoint) | `Recorded` yes (a new wait); `Duplicate` no |
|
|
327
|
+
| `fenced(ticket, token, now)` | `Recorded(:activate)`, `Duplicate` (same token again) | ticket refusals, `:unknown` (token of another scope), `:mismatch` (a different token is recorded), `:out_of_phase`, `:stale_token` (not newer than the source token) | `Recorded` yes (a new wait); `Duplicate` no |
|
|
328
|
+
| `activated(ticket, replica, token, now)` | `Recorded(:complete)` | ticket refusals, `:unknown` (token of another scope), `:out_of_phase`, `:wrong_replica`, `:mismatch` (not the fenced token) | no: the transfer is over |
|
|
329
|
+
| `abort(ticket, token, now)` | `Recorded(:rollback)`, `Duplicate` (rollback under way, same `token`) | ticket refusals, `:unknown` (token of another scope), `:mismatch` (rollback under way, different `token`), `:stale_token` (not newer than every token known), `:fence_required` | `Recorded` yes (the rollback is a new wait); `Duplicate` no |
|
|
330
|
+
| `resumed(ticket, replica, now)` | `Recorded(:rollback)`, `Recorded(:aborted)`, `Duplicate` (already recorded) | ticket refusals, `:out_of_phase`, `:wrong_replica` | no: a step within the rollback, its end, or a repeat |
|
|
331
|
+
| `discarded(ticket, replica, now)` | `Recorded(:rollback)`, `Recorded(:aborted)`, `Duplicate` (already recorded or never owed) | ticket refusals, `:out_of_phase`, `:wrong_replica` | no: a step within the rollback, its end, or a repeat |
|
|
332
|
+
| `outstanding(ticket, from, limit)` | `Array` of offsets | `:stale`, `:unknown`, `:out_of_phase` | no, a query |
|
|
333
|
+
|
|
334
|
+
A `Duplicate` carries the phase the transfer stands at now: any phase from the one the acknowledgement first led to up to `:rollback`.
|
|
335
|
+
|
|
336
|
+
**Which answers record the clock.** An honoured call records `now` exactly when its answer depends on it. `start` and every acknowledgement that moves the transfer to a new wait do: the new phase's deadline is `now + phase_budget`. A `Duplicate`, a `Recorded(:drain)` within the drain, a `committed`, a step within the rollback and the acknowledgement that ends the transfer answer the same at any instant and record nothing, so a later call with an earlier `now` is not `:clock_regressed` because of them. `overdue(now, ..)` is a query and records nothing either.
|
|
337
|
+
|
|
338
|
+
`Handoff.new(limits)`, `Handoff.with_scope(limits, scope)`, `limits`, `scope`, `status`, `orders`, `overdue`, `active` and `generation` never refuse. `Handoff.restored(limits, scope, last_generation)` answers a coordinator or raises `RangeError` when `last_generation` is neither `nil` nor a generation of `scope`.
|
|
339
|
+
|
|
340
|
+
## The Rules of the Tender
|
|
341
|
+
|
|
342
|
+
- **Ids are Integers** in `0..2**64-1`: partitions, replicas, offsets, checkpoints. Floats, nil, and strings raise `TypeError`. Out of range raises `RangeError`.
|
|
343
|
+
- **Fencing tokens are values.** Anything with a `raw` reader is a token, the `lease_machines` gem's included; a bare Integer raises `TypeError`, and a `raw` whose sequence number is zero raises `RangeError`.
|
|
344
|
+
- **Time is Integer nanoseconds.** `now`, `Limits#phase_budget` and `Status#deadline` are nanoseconds on one monotonic clock. A Float or any other non-Integer raises `TypeError`. `phase_budget` is in `0..2**64-1`; instants (`now`, `deadline`) run over the whole Rust `Duration` range, `0..Duration::MAX` nanoseconds (about `1.8e28`). `deadline` is `now + phase_budget`, saturating at `Duration::MAX`, so it can be larger than `2**64-1` and still reads back as `now`. Outside its range a value raises `RangeError`.
|
|
345
|
+
- **Values are valid or do not exist.** Every `Data` value is checked at construction through the core constructor of the type it mirrors, whether the coordinator built it or you did: `Limits.new(budget, 0)` (`Limits::new`), `Ticket.new(9, 0)` (`Ticket::new`), `Recorded.new(:quiesce)` and `Duplicate.new(:complete)` (`Acked::recorded`, `Acked::duplicate`), `Order.new(ticket, Replica.new(1), Fence.new(token))` (`Order::new`: only the authority fences) and a `Status` no live transfer reports (`Status::new`) all raise `RangeError`. A value of the wrong type raises `TypeError`; an Integer field outside `0..2**64-1` raises `RangeError`.
|
|
346
|
+
- **The gem keeps its own classes.** `Fence`, `Activate`, `Resume` and `Status` accept any token with a `raw` reader and keep it as a `HandoffMachines::FencingToken`, so `Fence.new(lease_token) == Fence.new(HandoffMachines::FencingToken.new(lease_token.raw))`.
|
|
347
|
+
- **The binding never loses work.** Every argument is read and checked (a token's `raw` included) before the coordinator is asked anything. Building the answer runs Ruby (`Ticket.new`, `Recorded.new`, `Duplicate.new`), so the coordinator decides on a copy, the answer is built, and the copy is kept only if the coordinator is still unfrozen (`FrozenError` otherwise) and unchanged (`RuntimeError` otherwise: something re-entered it while the answer was built). A call that raised changed nothing. `orders`, `overdue`, `outstanding` and `status` are queries.
|
|
348
|
+
- **The binding is the authority on values.** Every `Data` class is sealed (no `allocate`, no subclass), but Ruby binds a subclass constant before `inherited` raises, `Data#initialize` can be bound directly and `Marshal.load` skips `initialize`. So the native side reads a `Limits`, `Ticket`, `Replica` or action only when it is exactly this gem's class (a subclass raises `TypeError`, whatever it holds; a token may also be a sibling gem's, read by `raw`), and checks every field again as the core's constructor does: a forged value raises `TypeError` or `RangeError` and never reaches the coordinator.
|
|
349
|
+
- **Only `:quiesce` and `:discard` are Symbols.** Every other action is its `Data`: `Order.new(ticket, replica, :resume)` raises `RangeError`, because the order the coordinator restates is `Resume.new(nil)` (or `Resume.new(token)`).
|
|
350
|
+
- **Values mirror the core.** `Limits`, `Ticket`, `Order` and `Status` have the core struct's fields in the core's order. An enum variant with a field is a `Data` named after it (`Recorded`, `Duplicate`, `Replica`, `Drain`, `Capture`, `Install`, `Fence`, `Activate`, `Resume`); one without is a Symbol (`:authority`, `:quiesce`, `:discard`, every phase). No argument has a default the core lacks: `orders`, `outstanding` and `overdue` take `from` and `limit` every time, and `limit` is the one argument the core does not have, because Ruby gets an Array where the core hands out an iterator.
|
|
351
|
+
- **Tickets are plain data.** A `Ticket` is a `Data` with a partition and a generation, so it survives a trip over the wire. Anything that isn't a `Ticket` raises `TypeError`, and so does a constructor given anything but a `Limits`.
|
|
352
|
+
- **Scopes.** A ticket is only meaningful to the coordinator that issued it. Generations count from 1, so two coordinators hand out the same tickets for the same partition, and a reply sent to the wrong one would be recorded there. Give each coordinator its own scope: `Handoff.with_scope(limits, scope)`, any Integer in `0..65_535` (a non-Integer raises `TypeError`, anything outside the range raises `RangeError`). `Handoff.new(limits)` is scope zero. The scope rides in the top 16 bits of every generation; `handoff.scope` and `ticket.scope` read it back. A ticket from another scope is `:unknown` to every acknowledgement and to `outstanding`, and changes nothing. The coordinator can't pick a scope for you, and two coordinators left in the same scope still mix up each other's tickets.
|
|
353
|
+
- **Refusals don't raise.** Every acknowledgement answers with a `Recorded` or `Duplicate` whose phase is one of `HandoffMachines::PHASES`, or with a refusal from `HandoffMachines::REFUSALS`. Only malformed input raises.
|
|
354
|
+
- **A refusal changes nothing.** Not the phase, not the window, not the clock.
|
|
355
|
+
- **A repeat is not a refusal.** Every acknowledgement answers `Duplicate` when the transfer already recorded it with the same payload. Treat it as success. It changes nothing either. Only `activated` never does: it frees the slot, so its repeat is `:stale`.
|
|
356
|
+
- **Acknowledgements come from the right replica.** `quiesced`, `drained`, `committed`, `captured` and `resumed` from the source; `installed`, `activated` and `discarded` from the target. Anyone else is `:wrong_replica`.
|
|
357
|
+
- **Frozen means frozen.** `freeze` the coordinator and `start` and every acknowledgement raise `FrozenError`. Reads still work.
|
|
358
|
+
|
|
359
|
+
## What You Think You're Doing vs Reality
|
|
360
|
+
|
|
361
|
+
### You Think: "I'll tell the old node to stop, then tell the new node to start."
|
|
362
|
+
### Reality: The old node didn't get the message.
|
|
363
|
+
|
|
364
|
+
A quiesce order is a request. The source that never received it, or received it and then stalled for ten seconds, will keep writing with a clear conscience. The only thing that actually stops it is a storage layer that has seen token 8 and refuses token 7. The coordinator makes sure a token 8 exists before the target is told to serve, and that it is greater than 7. Getting your storage to check it on every write is the part you can't skip.
|
|
365
|
+
|
|
366
|
+
### You Think: "If it goes wrong, I'll just cancel."
|
|
367
|
+
### Reality: Cancel what, exactly?
|
|
368
|
+
|
|
369
|
+
Before the checkpoint, cancelling means telling the source to carry on. After the target installed something, it also means telling the target to throw it away. After the target was told to activate, it means fencing the target out with a newer token, because it may have taken writes. Each phase has a different undo, and `abort` is the one place that knows which.
|
|
370
|
+
|
|
371
|
+
## What's Yours
|
|
372
|
+
|
|
373
|
+
The coordinator lives in your process and forgets everything when the process does. A fresh coordinator counts from 1 again, so a ticket that outlived a restart could name a new transfer. Either resume the sequence or give each incarnation a scope no earlier one used. To resume, write `generation` somewhere durable before you hand out a ticket, and build the next coordinator with `Handoff.restored(limits, scope, last_generation)` so old tickets stay old. The scope is yours to pass, never read out of the saved value: a value from another scope, one that lost its top bits, or a bare `0` (no generation in any scope, scope zero included) raises `RangeError`. A coordinator that issued nothing answers `nil`, and that is the only spelling of "nothing issued yet": store it as a null and `restored` gives you back an equal fresh coordinator. Transfers that were in flight are not restored: start them again, and let your lease authority's highest token decide who owns a partition that was mid-activation.
|
|
374
|
+
|
|
375
|
+
Replica ids are whatever you say they are. The coordinator compares them and cannot tell a restarted replica from an impostor. Fencing tokens must come from one serialized authority, and the resource they protect has to reject older ones atomically with each write. If you abort without a token while the authority might already have granted the target one, report the grant with `fenced` first.
|
|
376
|
+
|
|
377
|
+
Delivery, retries, checkpoint bytes and retiring the source after `Recorded(:complete)` are yours too. So is choosing which partition moves where. This gem moves ownership. It does not rank placement.
|
|
378
|
+
|
|
379
|
+
## License
|
|
380
|
+
|
|
381
|
+
MIT License. See [LICENSE.txt](LICENSE.txt) for details.
|
|
382
|
+
|
|
383
|
+
**Remember: for four seconds, both nodes thought they owned it. The question is whether your storage agreed.**
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## [0.8.1](https://github.com/seuros/handoff_machines/compare/handoff-machines-v0.8.0...handoff-machines-v0.8.1) (2026-11-16)
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
### Bug Fixes
|
|
7
|
+
|
|
8
|
+
* Status::new refuses tokens no fence or abort could admit ([97b04ef](https://github.com/seuros/handoff_machines/commit/97b04efdfaa2a470d3644601e5e61e9ac21700ff))
|
|
9
|
+
|
|
10
|
+
## [0.8.0](https://github.com/seuros/handoff_machines/compare/handoff-machines-v0.7.0...handoff-machines-v0.8.0) (2026-11-16)
|
|
11
|
+
|
|
12
|
+
|
|
13
|
+
### ⚠ BREAKING CHANGES
|
|
14
|
+
|
|
15
|
+
* generation is nil before the first, restored takes nil
|
|
16
|
+
|
|
17
|
+
### Features
|
|
18
|
+
|
|
19
|
+
* generation is nil before the first, restored takes nil ([837a82b](https://github.com/seuros/handoff_machines/commit/837a82bdf67ebfbe64e9e63eaaa8dd6f7d0dda2a))
|
|
20
|
+
|
|
21
|
+
## [0.7.0](https://github.com/seuros/handoff_machines/compare/handoff-machines-v0.6.0...handoff-machines-v0.7.0) (2026-11-14)
|
|
22
|
+
|
|
23
|
+
|
|
24
|
+
### ⚠ BREAKING CHANGES
|
|
25
|
+
|
|
26
|
+
* adopt matryona-machines 0.2 and name every hook's real field
|
|
27
|
+
|
|
28
|
+
### Features
|
|
29
|
+
|
|
30
|
+
* adopt matryona-machines 0.2 and name every hook's real field ([fcee7fe](https://github.com/seuros/handoff_machines/commit/fcee7fef7c1c8dee5e5ec4908969130ea304ace6))
|
|
31
|
+
|
|
32
|
+
## [0.6.0](https://github.com/seuros/handoff_machines/compare/handoff-machines-v0.5.0...handoff-machines-v0.6.0) (2026-11-14)
|
|
33
|
+
|
|
34
|
+
|
|
35
|
+
### ⚠ BREAKING CHANGES
|
|
36
|
+
|
|
37
|
+
* private validated fields, refuse bare :resume and a drain at its quiesce point, exact-class reads, clone-then-keep acks
|
|
38
|
+
|
|
39
|
+
### Features
|
|
40
|
+
|
|
41
|
+
* private validated fields, refuse bare :resume and a drain at its quiesce point, exact-class reads, clone-then-keep acks ([ff05be6](https://github.com/seuros/handoff_machines/commit/ff05be61aedb4e27ac599d94d8ee2d28afb7b308))
|
|
42
|
+
|
|
43
|
+
## [0.5.0](https://github.com/seuros/handoff_machines/compare/handoff-machines-v0.4.0...handoff-machines-v0.5.0) (2026-11-13)
|
|
44
|
+
|
|
45
|
+
|
|
46
|
+
### ⚠ BREAKING CHANGES
|
|
47
|
+
|
|
48
|
+
* converted to the fleet conventions. One exhaustive Refusal per crate, Result from every operation that can refuse, a refusal changes nothing, provable idempotent repeats are honoured, Ruby mirrors the core. No compatibility aliases. Requires lease-machines >= 0.4. Ruby takes a token by its raw reader; a bare Integer is a TypeError.
|
|
49
|
+
|
|
50
|
+
### Features
|
|
51
|
+
|
|
52
|
+
* fencing tokens are scoped values; a token of another scope is Unknown before anything else is read ([77e57ef](https://github.com/seuros/handoff_machines/commit/77e57ef727f205497d89171dd03db9f2aa6578aa))
|
|
53
|
+
|
|
54
|
+
## [0.4.0](https://github.com/seuros/handoff_machines/compare/handoff-machines-v0.3.0...handoff-machines-v0.4.0) (2026-11-13)
|
|
55
|
+
|
|
56
|
+
|
|
57
|
+
### ⚠ BREAKING CHANGES
|
|
58
|
+
|
|
59
|
+
* handles are issued through periscope-machines. Every instance can be built with a scope; a handle from another scope is refused before any lookup; all handle kinds of an instance share one sequence; resumed restores the sequence after a restart. restored(limits, scope, last_generation) returns an Option.
|
|
60
|
+
|
|
61
|
+
### Features
|
|
62
|
+
|
|
63
|
+
* scope ticket generations through periscope-machines; restored takes the scope and refuses a saved generation from another scope ([978b653](https://github.com/seuros/handoff_machines/commit/978b653d386062d925d002370930bf09d2d2dbf8))
|
|
64
|
+
|
|
65
|
+
## [0.3.0](https://github.com/seuros/handoff_machines/compare/handoff-machines-v0.2.0...handoff-machines-v0.3.0) (2026-11-13)
|
|
66
|
+
|
|
67
|
+
|
|
68
|
+
### ⚠ BREAKING CHANGES
|
|
69
|
+
|
|
70
|
+
* see CONVENTIONS: one exhaustive Refusal per crate, Result from every mutator, honoured idempotent repeats only when provable, Duration time, Ruby mirrors the core. No compatibility aliases. Requires offset-machines >= 0.3. A replay with a different payload is Mismatch; the Ruby class is Handoff and takes Limits.
|
|
71
|
+
|
|
72
|
+
### Features
|
|
73
|
+
|
|
74
|
+
* every acknowledgement answers Result<Acked, Refusal>, honouring recorded repeats as Duplicate ([9d6f347](https://github.com/seuros/handoff_machines/commit/9d6f3472e5c84fd887d87eb4b9aa05b482ffd1e1))
|
|
75
|
+
|
|
76
|
+
## [0.2.0](https://github.com/seuros/handoff_machines/compare/handoff-machines-v0.1.0...handoff-machines-v0.2.0) (2026-11-12)
|
|
77
|
+
|
|
78
|
+
|
|
79
|
+
### Features
|
|
80
|
+
|
|
81
|
+
* initial extraction ([acfda8e](https://github.com/seuros/handoff_machines/commit/acfda8e11b70570a4728955c14a04bc31d5cbba3))
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
[package]
|
|
2
|
+
name = "handoff-machines"
|
|
3
|
+
version = "0.8.1"
|
|
4
|
+
edition.workspace = true
|
|
5
|
+
rust-version.workspace = true
|
|
6
|
+
license.workspace = true
|
|
7
|
+
description = "Bounded partition ownership transfer phases with generation-tagged acknowledgements and fenced activation"
|
|
8
|
+
readme = "README.md"
|
|
9
|
+
exclude = ["src/**/tests.rs", "src/tests/", "tests/"]
|
|
10
|
+
keywords = ["handoff", "partition", "rebalance", "fencing", "no-std"]
|
|
11
|
+
categories = ["no-std", "algorithms"]
|
|
12
|
+
|
|
13
|
+
[dependencies]
|
|
14
|
+
lease-machines.workspace = true
|
|
15
|
+
offset-machines.workspace = true
|
|
16
|
+
periscope-machines.workspace = true
|
|
17
|
+
|
|
18
|
+
[dev-dependencies]
|
|
19
|
+
matryona-machines = { workspace = true, features = ["testing"] }
|
|
20
|
+
|
|
21
|
+
[lints]
|
|
22
|
+
workspace = true
|