handoff_machines 0.8.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,215 @@
1
+ # handoff-machines
2
+
3
+ > **AMS DeCandia**, NCC-4234, *DeCandia*-class transfer tender.
4
+
5
+ Engine-room flight manual: move a partition between replicas one acknowledged phase at a time.
6
+
7
+ Allocation-free `no_std` coordinator policy for partition
8
+ ownership transfer: quiesce the source, drain what is in flight, checkpoint,
9
+ then activate the target under a newer fencing token. `Handoff<N, WORDS>` holds
10
+ at most `N` concurrent transfers, each with a drain window of `WORDS * 64`
11
+ offsets, and refuses rather than grows.
12
+
13
+ It coordinates ownership migration. It does not rank placement, replicate data,
14
+ issue fencing tokens, send messages or persist anything.
15
+
16
+ - `start` issues a `Ticket` (`partition()`, `generation()`). Generations are strictly
17
+ increasing and never reused. Every acknowledgement carries its ticket: an
18
+ issued generation that is no longer the partition's transfer is `Stale`, one
19
+ never issued is `Unknown`.
20
+ - **Scopes.** A ticket means something only to the coordinator that issued
21
+ it. A generation is a `u64` in the
22
+ [`periscope-machines`](https://crates.io/crates/periscope-machines) layout:
23
+ the coordinator's scope in the top 16 bits, a sequence counted from 1 in
24
+ the low 48. Two coordinators count the same sequence, so without scopes a
25
+ ticket for the same partition given to the wrong one is acknowledged there.
26
+ `Handoff::new(limits)` is scope zero and issues 1, 2, 3;
27
+ `Handoff::with_scope(limits, scope)` issues the same sequence under `scope`;
28
+ `Handoff::scope()` and `Ticket::scope()` read it back. A ticket of another
29
+ scope was never issued here: every acknowledgement and `outstanding` answer
30
+ `Unknown` before any transfer is looked up, and nothing changes. The
31
+ coordinator reads no clock and no random source, so it cannot pick a scope;
32
+ two coordinators left in the same scope still alias. A scope holds 2^48 - 1
33
+ generations, then `start` answers `Exhausted`.
34
+ - **Restarts.** A fresh coordinator counts from 1 again, so a ticket that
35
+ outlived a restart could name a new transfer. Either resume the sequence,
36
+ or give each incarnation a scope no earlier one used.
37
+ `Handoff::restored(limits, scope, last_generation)` resumes: it takes the
38
+ scope and what `generation()` answered, an `Option<u64>`, and is `None`
39
+ when a saved value is not a generation of `scope`. The scope is never read
40
+ out of the saved value, so one that lost its top bits, or the `0` a store
41
+ answers for a missing row, cannot land a coordinator in scope zero (`0` is
42
+ no generation there either). "Nothing issued yet" is spelled one way,
43
+ `None`, which is what `generation()` answers then and restores an equal
44
+ fresh coordinator.
45
+ - An idempotent repeat (`Acked::Duplicate`) is answered only from a live
46
+ transfer whose ticket matches, scope included. A ticket whose transfer
47
+ ended is `Stale` even though the coordinator can tell it issued it: it does
48
+ not keep how the transfer ended.
49
+ - Each phase advances only on an explicit acknowledgement from the replica that
50
+ holds the role: `quiesced` (source, with the quiesce point), `drained` (source,
51
+ one in-flight offset at a time), `captured` (source, checkpoint identity),
52
+ `installed` (target, the same identity), `fenced` (the authority's token,
53
+ strictly above the source's), `activated` (target, the same token).
54
+ - Fencing tokens are `lease-machines` `FencingToken`s, re-exported here, and
55
+ they carry their authority's scope. Tokens of two scopes have no order, so a
56
+ transfer reads tokens only inside its source token's scope. `fenced`,
57
+ `activated` and `abort` answer `Unknown` to a token of any other scope,
58
+ whatever its number, and they answer it first: after the clock and the
59
+ ticket, before the transfer's phase, the replica or any recorded token is
60
+ looked at. `Unknown` is the name every crate gives an id that is not of
61
+ the namespace the call reads it in; a foreign token is never folded into
62
+ "older" (`StaleToken`), "not the one recorded" (`Mismatch`) or "not now"
63
+ (`OutOfPhase`). Those three are only ever said of a token of the
64
+ transfer's scope: `StaleToken` when it does not exceed the ones the
65
+ transfer knows, `Mismatch` when a different one is recorded. `start` has
66
+ nothing to compare its source token with: that token names the scope.
67
+ One transfer therefore lives under one authority scope: if
68
+ the authority came back under a new scope before `fenced`, give the
69
+ transfer up with an unfenced `abort` and start it again under a source
70
+ token of the new scope.
71
+ - Drain readiness is an `offset-machines` `Tracker` per transfer, built
72
+ already assigned (`Tracker::assigned`) at the transfer's base: the ticket
73
+ is checked before the tracker is reached, so the tracker keeps one
74
+ generation of its own and never meets another. The phase ends when the contiguous acknowledged prefix
75
+ reaches the quiesce point. `committed` slides the window for drains wider
76
+ than it; `outstanding` lists what is still missing, resumably.
77
+ - `abort` works from every phase and moves to `Rollback`: the source owes
78
+ `resumed`, and the target owes `discarded` once it was asked to install. After
79
+ `fenced`, activation has been ordered and the target may be serving, so `abort`
80
+ demands a token above the activation token and the source resumes under it.
81
+ - `orders(from)` restates what is owed, at most two orders per transfer. Orders
82
+ are derived from state, not queued: nothing overflows, and a pass can stop
83
+ anywhere and resume from `Orders::position()`.
84
+ - Every mutating call takes `now`. An earlier `now` than the last recorded one is
85
+ `ClockRegressed`. Each phase gets `Limits::phase_budget`; `overdue` lists
86
+ transfers past it. That is advice: the coordinator never aborts by itself.
87
+ - **Which answers record the clock.** An honoured call records `now` exactly
88
+ when its answer depends on it: `start`, and every acknowledgement that moves
89
+ the transfer to a new wait (`quiesced`, a `drained` that ends the drain,
90
+ `captured`, `installed`, `fenced`, `abort`), because the new phase's
91
+ deadline is `now + phase_budget`. A `Duplicate`, a `drained` or
92
+ `committed` within the drain, a `resumed` or `discarded` while the rollback
93
+ still waits for the other, and the acknowledgement that ends the transfer
94
+ (`activated`, the last of the rollback) hold whatever the time and record
95
+ nothing. `overdue(now, ..)` is a query.
96
+ - **Values valid by construction.** `Limits::new` (`None` for `per_replica`
97
+ zero), `Ticket::new` (`None` for a zero sequence), `Acked::recorded` and
98
+ `Acked::duplicate` (`None` for a phase no acknowledgement answers),
99
+ `Order::new` (`None` unless only the authority is asked to fence, and it
100
+ is asked nothing else) and `Status::new` (`None` for a view no live
101
+ transfer reports) are what the Ruby binding builds every value through.
102
+ Their fields are private: a `Ticket`, `Order`, `Status` or `Limits` exists
103
+ only as its constructor or the coordinator built it, and is read through
104
+ accessors named after the fields.
105
+ `Phase::is_final` says whether a phase ends a transfer.
106
+ - **No unreachable arms.** Each step carries what its order restates, so an
107
+ order is never missing a field. A transfer is built without a refusal:
108
+ `Tracker::assigned` takes a generation that is never zero, so `start` has
109
+ no arm for a fresh drain tracker refusing. `restored` carries
110
+ `periscope-machines`' `Issuer::restored` whole, whose only refusals are the
111
+ two **Restarts** names (another scope, a zero sequence), so no sequence
112
+ can fall out of reach. Tokens are ranked with
113
+ `FencingToken::order`, which has no answer across scopes: `supersedes`
114
+ answers `Unknown` there, the answer `homed` already gave, and never
115
+ compares raw values of two scopes. One arm remains that the types demand
116
+ and the invariants rule out, a `debug_assert!` with the most conservative
117
+ refusal and nothing changed: the drain tracker refusing an `ack` or a
118
+ `mark_committed` for a reason the transfer rules out (`OutOfPhase`).
119
+ The tracker is asked only under its own current generation, so `Stale`
120
+ cannot come back, but its one `Refusal` type still names it, `Exhausted`,
121
+ and the variant the other call answers.
122
+ Nothing in the crate panics.
123
+ - Every call that can refuse returns `Result<T, Refusal>`. `Refusal` is the one
124
+ enum for "not honoured", and a refused call changes nothing. No public enum
125
+ is `#[non_exhaustive]`: a new variant is a breaking change every caller
126
+ matches on.
127
+ - A network may deliver any acknowledgement twice, so every acknowledgement
128
+ method (`quiesced`, `drained`, `committed`, `captured`, `installed`, `fenced`,
129
+ `activated`, `abort`, `resumed`, `discarded`) returns `Result<Acked, Refusal>`:
130
+ `Acked::Recorded(phase)` the first time, `Acked::Duplicate(phase)` for an
131
+ idempotent repeat. A repeat is honoured, not refused, and changes nothing,
132
+ not even the observed clock.
133
+ - A repeat is answered from what the transfer recorded, for as long as the
134
+ transfer lives, rollback included: `quiesced` with the recorded quiesce
135
+ point, `drained` of an acknowledged or committed offset, `committed` at or
136
+ below the window base, `captured` and `installed` with the recorded
137
+ checkpoint, `fenced` with the recorded token, `abort` during the rollback
138
+ with the same token (or the same absence of one), and a `resumed` or
139
+ `discarded` already recorded or never owed.
140
+ - A replay whose payload differs from the record is not a repeat: a different
141
+ quiesce point, checkpoint, activation token or rollback token is `Mismatch`.
142
+ From the wrong replica it is `WrongReplica`.
143
+ - `OutOfPhase` is what is left: the transfer is not waiting for the
144
+ acknowledgement and never recorded it, because it came too early or because
145
+ the transfer was given up first. `activated` and the acknowledgement that
146
+ ends a rollback free the slot, so their repeats are `Stale` like any other
147
+ finished ticket's.
148
+ - Checks run in one order, and the first that fails is the answer. Every
149
+ acknowledgement: `ClockRegressed`, then the ticket (`Unknown`, `Stale`).
150
+ Then, for the three that take a fencing token:
151
+
152
+ | Call | After the clock and the ticket |
153
+ | --- | --- |
154
+ | `fenced` | `Unknown` (token scope), then `Duplicate` or `Mismatch` against a recorded token, `OutOfPhase`, `StaleToken` |
155
+ | `activated` | `Unknown` (token scope), `OutOfPhase`, `WrongReplica`, `Mismatch` |
156
+ | `abort` | `Unknown` (token scope, when a token is given), then `Duplicate` or `Mismatch` during a rollback, `StaleToken`, `FenceRequired` |
157
+
158
+ ```rust
159
+ use core::time::Duration;
160
+ use handoff_machines::{Acked, Action, FencingToken, Handoff, Limits, Phase, Recipient, Refusal};
161
+
162
+ # fn example() -> Option<()> {
163
+ let limits = Limits::new(Duration::from_secs(30), 2)?;
164
+ let mut handoff = Handoff::<8, 4>::new(limits); // 8 transfers, 256-offset drains
165
+ let now = Duration::ZERO;
166
+ let held = FencingToken::from_raw(7)?; // scope zero, sequence 7
167
+ // Partition 42 from replica 1 to replica 2; offsets below 100 are committed.
168
+ let ticket = handoff.start(42, 1, 2, 100, held, now).ok()?;
169
+ let order = handoff.orders(0).next()?;
170
+ assert_eq!((order.to(), order.action()), (Recipient::Replica(1), Action::Quiesce));
171
+
172
+ let acked = |phase| Ok(Acked::Recorded(phase));
173
+ assert_eq!(handoff.quiesced(ticket, 1, 102, now), acked(Phase::Drain));
174
+ assert_eq!(handoff.quiesced(ticket, 1, 102, now), Ok(Acked::Duplicate(Phase::Drain)));
175
+ assert_eq!(handoff.drained(ticket, 1, 101, now), acked(Phase::Drain));
176
+ assert_eq!(handoff.drained(ticket, 1, 101, now), Ok(Acked::Duplicate(Phase::Drain)));
177
+ assert_eq!(handoff.outstanding(ticket, 0).ok()?.next(), Some(100));
178
+ assert_eq!(handoff.drained(ticket, 1, 100, now), acked(Phase::Checkpoint));
179
+ assert_eq!(handoff.captured(ticket, 1, 9, now), acked(Phase::Checkpoint));
180
+ assert_eq!(handoff.installed(ticket, 2, 9, now), acked(Phase::Activate));
181
+ let granted = FencingToken::from_raw(8)?; // from your lease authority
182
+ // A token of another authority's scope has no order against the source's.
183
+ let foreign = FencingToken::from_raw((3 << 48) | 8)?;
184
+ assert_eq!(handoff.fenced(ticket, foreign, now), Err(Refusal::Unknown));
185
+ assert_eq!(handoff.fenced(ticket, held, now), Err(Refusal::StaleToken));
186
+ assert_eq!(handoff.fenced(ticket, granted, now), acked(Phase::Activate));
187
+ assert_eq!(handoff.activated(ticket, 2, granted, now), acked(Phase::Complete));
188
+ assert_eq!(handoff.active(), 0);
189
+ # Some(())
190
+ # }
191
+ # assert_eq!(example(), Some(()));
192
+ ```
193
+
194
+ ## What stays outside
195
+
196
+ 1. **Durable replica identities.** Replicas are `u64`s the caller vouches for.
197
+ The coordinator compares them; it cannot tell a restarted replica from an
198
+ impostor, and an acknowledgement is believed if its ticket and role match.
199
+ 2. **Serialized fenced activation.** The token passed to `fenced` must come from
200
+ one serialized authority (`lease-machines` models its decisions), and the
201
+ resource must reject lower tokens atomically with each write. The coordinator
202
+ only checks that the token is of the source token's scope and above it. Without resource-side
203
+ fencing, a source that missed `Quiesce` keeps writing.
204
+ 3. **Transfer replay protection.** Generations refuse stale acknowledgements
205
+ inside one coordinator lifetime. Across restarts, persist `generation()` before
206
+ exposing a ticket and rebuild with `Handoff::restored`, or build every
207
+ incarnation with a scope of its own. In-flight transfers
208
+ are not restored: start them again under a new generation, and let the
209
+ authority's highest token say who owns a partition that was mid-activation.
210
+ 4. **Delivery and timing.** Orders are restated until acknowledged; delivery,
211
+ retries and deduplication at the replicas are the adapter's. Checkpoint bytes,
212
+ their integrity and the source's retirement after `Complete` are too.
213
+ 5. **An abort without a token before `fenced` assumes no grant exists.** If the
214
+ authority may already have issued the target a token, report it with `fenced`
215
+ first, then abort with a newer one.
@@ -0,0 +1,130 @@
1
+ use core::time::Duration;
2
+
3
+ use offset_machines::{Gaps, Tracker};
4
+
5
+ use crate::{Order, Ticket, transfer::Transfer};
6
+
7
+ /// Resumable pass over outstanding orders; see [`crate::Handoff::orders`].
8
+ #[derive(Debug, Clone)]
9
+ pub struct Orders<'a, const WORDS: usize> {
10
+ slots: &'a [Option<Transfer<WORDS>>],
11
+ position: usize,
12
+ }
13
+
14
+ impl<'a, const WORDS: usize> Orders<'a, WORDS> {
15
+ pub(crate) const fn new(slots: &'a [Option<Transfer<WORDS>>], from: usize) -> Self {
16
+ Self {
17
+ slots,
18
+ position: from,
19
+ }
20
+ }
21
+
22
+ /// Where a later pass continues after the orders yielded so far.
23
+ #[must_use]
24
+ pub const fn position(&self) -> usize {
25
+ self.position
26
+ }
27
+ }
28
+
29
+ impl<const WORDS: usize> Iterator for Orders<'_, WORDS> {
30
+ type Item = Order;
31
+
32
+ fn next(&mut self) -> Option<Order> {
33
+ while let Some(slot) = self.slots.get(self.position / 2) {
34
+ let part = self.position % 2;
35
+ // position / 2 indexes a slice of non-zero-sized slots, so
36
+ // position < 2 * len <= isize::MAX * 2 and the step cannot wrap.
37
+ self.position += 1;
38
+ let order = slot.as_ref().and_then(|transfer| {
39
+ let (to, action) = transfer.order(part)?;
40
+ Some(Order {
41
+ ticket: transfer.ticket,
42
+ to,
43
+ action,
44
+ })
45
+ });
46
+ if order.is_some() {
47
+ return order;
48
+ }
49
+ }
50
+ None
51
+ }
52
+ }
53
+
54
+ /// Resumable pass over overdue transfers; see [`crate::Handoff::overdue`].
55
+ #[derive(Debug, Clone)]
56
+ pub struct Overdue<'a, const WORDS: usize> {
57
+ slots: &'a [Option<Transfer<WORDS>>],
58
+ now: Duration,
59
+ position: usize,
60
+ }
61
+
62
+ impl<'a, const WORDS: usize> Overdue<'a, WORDS> {
63
+ pub(crate) const fn new(
64
+ slots: &'a [Option<Transfer<WORDS>>],
65
+ now: Duration,
66
+ from: usize,
67
+ ) -> Self {
68
+ Self {
69
+ slots,
70
+ now,
71
+ position: from,
72
+ }
73
+ }
74
+
75
+ /// Where a later pass continues after the tickets yielded so far.
76
+ #[must_use]
77
+ pub const fn position(&self) -> usize {
78
+ self.position
79
+ }
80
+ }
81
+
82
+ impl<const WORDS: usize> Iterator for Overdue<'_, WORDS> {
83
+ type Item = Ticket;
84
+
85
+ fn next(&mut self) -> Option<Ticket> {
86
+ while let Some(slot) = self.slots.get(self.position) {
87
+ self.position += 1;
88
+ if let Some(transfer) = slot
89
+ && self.now >= transfer.deadline
90
+ {
91
+ return Some(transfer.ticket);
92
+ }
93
+ }
94
+ None
95
+ }
96
+ }
97
+
98
+ /// Offsets a drain still waits for; see [`crate::Handoff::outstanding`].
99
+ #[derive(Debug, Clone)]
100
+ pub struct Outstanding<'a, const WORDS: usize> {
101
+ gaps: Gaps<'a, WORDS>,
102
+ next: u64,
103
+ until: u64,
104
+ }
105
+
106
+ impl<'a, const WORDS: usize> Outstanding<'a, WORDS> {
107
+ pub(crate) fn new(tracker: &'a Tracker<WORDS>, from: u64, until: u64) -> Self {
108
+ let frontier = tracker.frontier().unwrap_or_else(|| tracker.base());
109
+ Self {
110
+ gaps: tracker.gaps(from),
111
+ next: from.max(frontier),
112
+ until,
113
+ }
114
+ }
115
+ }
116
+
117
+ impl<const WORDS: usize> Iterator for Outstanding<'_, WORDS> {
118
+ type Item = u64;
119
+
120
+ fn next(&mut self) -> Option<u64> {
121
+ if let Some(gap) = self.gaps.next() {
122
+ return Some(gap);
123
+ }
124
+ (self.next < self.until).then(|| {
125
+ let offset = self.next;
126
+ self.next += 1;
127
+ offset
128
+ })
129
+ }
130
+ }