offset_machines 0.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +73 -0
- data/Cargo.lock +307 -0
- data/Cargo.toml +27 -0
- data/LICENSE.txt +21 -0
- data/README.md +270 -0
- data/ext/offset_machines_native/core/CHANGELOG.md +52 -0
- data/ext/offset_machines_native/core/Cargo.toml +17 -0
- data/ext/offset_machines_native/core/README.md +81 -0
- data/ext/offset_machines_native/core/src/lib.rs +362 -0
- data/ext/offset_machines_native/core/src/refusal.rs +43 -0
- data/ext/offset_machines_native/core/src/status.rs +121 -0
- data/ext/offset_machines_native/extconf.rb +2 -0
- data/ext/offset_machines_native/ffi/Cargo.toml +21 -0
- data/ext/offset_machines_native/ffi/extconf.rb +14 -0
- data/ext/offset_machines_native/ffi/src/init.rs +32 -0
- data/ext/offset_machines_native/ffi/src/lib.rs +230 -0
- data/lib/offset_machines/version.rb +5 -0
- data/lib/offset_machines.rb +39 -0
- data/sig/offset_machines.rbs +52 -0
- metadata +85 -0
data/README.md
ADDED
|
@@ -0,0 +1,270 @@
|
|
|
1
|
+
# OffsetMachines
|
|
2
|
+
|
|
3
|
+
> Offset 102 finishing does not mean offset 101 did.
|
|
4
|
+
|
|
5
|
+
An acknowledgement tracker for one partition, for Ruby. Your workers finish offsets in whatever order they like. You tell the tracker which ones are done. It tells you the one number that is safe to commit, and which offsets you are still waiting for. You persist that number yourself and confirm it back. Only then does the window move.
|
|
6
|
+
|
|
7
|
+
This is an initial release and the API is unstable. It is a policy core, not a consumer runtime: it does not fetch, run handlers, talk to a broker, or store anything.
|
|
8
|
+
|
|
9
|
+
## Quick Start
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
gem 'offset_machines'
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
```ruby
|
|
16
|
+
tracker = OffsetMachines::Tracker.new
|
|
17
|
+
tracker.assign(1, 100) # generation 1, the store says 100 is next
|
|
18
|
+
|
|
19
|
+
def finished(tracker, generation, offset, store)
|
|
20
|
+
case tracker.ack(generation, offset) # :recorded and :duplicate both mean "it is acked"
|
|
21
|
+
when :stale
|
|
22
|
+
return # the partition was reassigned. This worker's result is not ours.
|
|
23
|
+
when :out_of_window
|
|
24
|
+
raise "offset #{offset} is past the window, stop fetching ahead"
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
candidate = tracker.commit_candidate or return
|
|
28
|
+
store.commit(candidate)
|
|
29
|
+
tracker.mark_committed(generation, candidate)
|
|
30
|
+
end
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
One tracker per partition. Generations and offsets are Integers. There is no clock in here at all.
|
|
34
|
+
|
|
35
|
+
## A Message to the Resistance
|
|
36
|
+
|
|
37
|
+
So you made your consumer concurrent. Ten workers on one partition, and throughput looks great. Offset 105 finishes first, so you commit 106. Then the process dies with 101 still on a worker's desk. The broker hands you back everything from 106 onward, and 101 is never seen again.
|
|
38
|
+
|
|
39
|
+
You know what the docs told you to do. "Commit only what you have processed." So you keep a `highest_done` variable. That is the bug you just had, with a name. So you keep a sorted set of finished offsets and walk it under a mutex. Then a rebalance takes the partition away and gives it back, and a worker from before the rebalance acks an offset into the new assignment's set.
|
|
40
|
+
|
|
41
|
+
The piece you keep rebuilding is a bitmap with a floor: everything below the floor is committed, bits above it are finished work, and the floor only moves across an unbroken run of bits. Plus a generation number, so a worker from a previous assignment gets told no. That bitmap is this gem.
|
|
42
|
+
|
|
43
|
+
## Chapter 1: Stardate 2026.316, the AMS Kreps
|
|
44
|
+
|
|
45
|
+
> **AMS Kreps**, NCC-4220, *Kreps*-class log tender.
|
|
46
|
+
> The log is only as committed as its first hole.
|
|
47
|
+
|
|
48
|
+
The tender doesn't carry the log. It carries the tally sheet. Crews report which entries they have finished, in any order, and the tender keeps one mark: the entry up to which nothing is missing. The ship is named for Jay Kreps, who wrote down why the log is the thing everything else hangs off.
|
|
49
|
+
|
|
50
|
+
On a bridge somewhere, a cadet volunteers:
|
|
51
|
+
|
|
52
|
+
```ruby
|
|
53
|
+
def on_done(offset)
|
|
54
|
+
@highest = [@highest, offset].max
|
|
55
|
+
consumer.commit(@highest + 1)
|
|
56
|
+
end
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
"That," says the tender's tallymaster, "commits every entry you haven't finished yet, provided one after it did."
|
|
60
|
+
|
|
61
|
+
## The Flight Plan
|
|
62
|
+
|
|
63
|
+
Each offset in the window is in one of these states:
|
|
64
|
+
|
|
65
|
+
```mermaid
|
|
66
|
+
stateDiagram-v2
|
|
67
|
+
[*] --> outstanding: assign(generation, base)
|
|
68
|
+
outstanding --> acked: ack, returns :recorded
|
|
69
|
+
acked --> acked: ack again, returns :duplicate
|
|
70
|
+
acked --> committed: mark_committed past it
|
|
71
|
+
committed --> committed: ack again, returns :duplicate
|
|
72
|
+
outstanding --> [*]: newer assign discards the window
|
|
73
|
+
acked --> [*]: newer assign discards the window
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
And the afternoon three workers finished out of order:
|
|
77
|
+
|
|
78
|
+
```mermaid
|
|
79
|
+
sequenceDiagram
|
|
80
|
+
participant W as Workers
|
|
81
|
+
participant T as Tracker
|
|
82
|
+
participant S as Your store
|
|
83
|
+
|
|
84
|
+
W->>T: ack(1, 102)
|
|
85
|
+
T-->>W: :recorded
|
|
86
|
+
Note over T: candidate nil. 100 is missing.
|
|
87
|
+
|
|
88
|
+
W->>T: ack(1, 100)
|
|
89
|
+
T-->>W: :recorded
|
|
90
|
+
Note over T: candidate 101. gap at 101.
|
|
91
|
+
|
|
92
|
+
S->>T: commit_candidate
|
|
93
|
+
T-->>S: 101
|
|
94
|
+
S->>S: persist 101
|
|
95
|
+
S->>T: mark_committed(1, 101)
|
|
96
|
+
T-->>S: :advanced
|
|
97
|
+
|
|
98
|
+
W->>T: ack(1, 101)
|
|
99
|
+
T-->>W: :recorded
|
|
100
|
+
Note over T: candidate 103. No gaps.
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
## How the Tracker Works
|
|
104
|
+
|
|
105
|
+
### Assign
|
|
106
|
+
|
|
107
|
+
```ruby
|
|
108
|
+
tracker = OffsetMachines::Tracker.new
|
|
109
|
+
|
|
110
|
+
tracker.ack(1, 100) # => :stale, nothing is assigned yet
|
|
111
|
+
tracker.assign(1, 100) # => :assigned, generation 1, window starts at 100
|
|
112
|
+
tracker.assign(1, 500) # => :stale, not newer than generation 1
|
|
113
|
+
tracker.generation # => 1
|
|
114
|
+
tracker.base # => 100
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
A new tracker is unassigned and refuses everything. `assign` takes a generation strictly greater than the current one and the offset your store says is next. Generation 0 is never accepted.
|
|
118
|
+
|
|
119
|
+
`assign` answers `:assigned`, or refuses with `:stale`.
|
|
120
|
+
|
|
121
|
+
```ruby
|
|
122
|
+
tracker = OffsetMachines::Tracker.assigned(1, 100) # generation 1, window at 100
|
|
123
|
+
tracker.ack(1, 100) # => :recorded
|
|
124
|
+
OffsetMachines::Tracker.assigned(0, 100) # raises RangeError
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
`Tracker.assigned(generation, base)` is `new` followed by an honoured `assign`, in one step with no refusal to handle: on a new tracker `assign` refuses only generation 0, and `assigned` raises `RangeError` for it instead, because the core's constructor takes a generation that is never zero.
|
|
128
|
+
|
|
129
|
+
### Ack
|
|
130
|
+
|
|
131
|
+
```ruby
|
|
132
|
+
tracker.ack(1, 102) # => :recorded, finished out of order
|
|
133
|
+
tracker.ack(1, 100) # => :recorded
|
|
134
|
+
tracker.ack(1, 100) # => :duplicate, already acked
|
|
135
|
+
tracker.ack(1, 99) # => :duplicate, below the window, already committed
|
|
136
|
+
tracker.ack(2, 101) # => :stale, not the current generation
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
An ack records that an offset finished processing. It does not commit anything.
|
|
140
|
+
|
|
141
|
+
`ack` answers `:recorded` or `:duplicate`, or refuses with `:stale` or `:out_of_window`. `:duplicate` is not a refusal: the offset is acked (or already committed), which is what you asked for. It was just true before you asked.
|
|
142
|
+
|
|
143
|
+
### Commit candidate
|
|
144
|
+
|
|
145
|
+
```ruby
|
|
146
|
+
tracker.commit_candidate # => 101
|
|
147
|
+
tracker.frontier # => 103
|
|
148
|
+
tracker.acked # => 2
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
The candidate is one past the unbroken run of acked offsets starting at `base`. It is the number you would hand a broker as "next offset to read". It is `nil` while `base` itself is unacked. `frontier` is one past the highest acked offset, and `acked` counts the bits.
|
|
152
|
+
|
|
153
|
+
### Acked?
|
|
154
|
+
|
|
155
|
+
```ruby
|
|
156
|
+
tracker.acked?(99) # => true, below the window: committed
|
|
157
|
+
tracker.acked?(100) # => true, recorded
|
|
158
|
+
tracker.acked?(101) # => false, still outstanding
|
|
159
|
+
tracker.acked?(900) # => false, beyond the window nothing is recorded
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
`acked?(offset)` asks whether an offset is already acknowledged under the current generation: it is below `base`, or its bit is set in the window. It is true exactly when `ack(generation, offset)` would answer `:duplicate`, and it changes nothing. It takes no generation because it reads only the current one. A new tracker answers `false` for everything, and so does a fresh assignment for every offset at or above its base.
|
|
163
|
+
|
|
164
|
+
### Gaps
|
|
165
|
+
|
|
166
|
+
```ruby
|
|
167
|
+
tracker.gaps(0, 10) # => [101]
|
|
168
|
+
tracker.ack(1, 105) # => :recorded
|
|
169
|
+
tracker.gaps(0, 10) # => [101, 103, 104]
|
|
170
|
+
tracker.gaps(0, 2) # => [101, 103]
|
|
171
|
+
tracker.gaps(104, 2) # => [104]
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
`gaps(from, limit)` lists the unacked offsets below the frontier, ascending, at most `limit` of them. To page through, ask again from the last one plus one. These are the offsets holding your commit back.
|
|
175
|
+
|
|
176
|
+
### Mark committed
|
|
177
|
+
|
|
178
|
+
```ruby
|
|
179
|
+
tracker.mark_committed(1, 102) # => :unacked, 101 isn't done
|
|
180
|
+
tracker.mark_committed(1, 101) # => :advanced
|
|
181
|
+
tracker.base # => 101
|
|
182
|
+
tracker.mark_committed(1, 101) # => :duplicate
|
|
183
|
+
tracker.commit_candidate # => nil, 101 is the new floor and it's unacked
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
`commit_candidate` only proposes. The window slides when you call `mark_committed` after your store has the offset. You may confirm less than the candidate. You may not confirm more.
|
|
187
|
+
|
|
188
|
+
`mark_committed` answers `:advanced` or `:duplicate`, or refuses with `:stale` or `:unacked`. `:duplicate` means the offset is at or below the floor, so it is already committed. Confirming twice is harmless.
|
|
189
|
+
|
|
190
|
+
```ruby
|
|
191
|
+
tracker.ack(1, 101) # => :recorded
|
|
192
|
+
tracker.commit_candidate # => 103
|
|
193
|
+
tracker.status
|
|
194
|
+
# => #<data OffsetMachines::Status generation=1, base=101, commit_candidate=103, frontier=106, acked=3>
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
### Reassign
|
|
198
|
+
|
|
199
|
+
```ruby
|
|
200
|
+
tracker.reassign(101) # => 2, the new generation
|
|
201
|
+
tracker.acked # => 0, the old window is gone
|
|
202
|
+
tracker.ack(1, 101) # => :stale
|
|
203
|
+
tracker.mark_committed(1, 102) # => :stale
|
|
204
|
+
tracker.ack(2, 101) # => :recorded
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
`reassign(base)` is `assign` with the next generation. It answers the new generation as an Integer, or refuses with `:exhausted` when the generation is already `2**64-1`. Every ack from the previous generation is thrown away, including the ones that had not been committed. Those offsets will be delivered again. That is what a rebalance means.
|
|
208
|
+
|
|
209
|
+
### Capacity
|
|
210
|
+
|
|
211
|
+
The window holds 256 offsets starting at `base`. Past that, `ack` refuses with `:out_of_window` and records nothing:
|
|
212
|
+
|
|
213
|
+
```ruby
|
|
214
|
+
tracker = OffsetMachines::Tracker.new
|
|
215
|
+
tracker.assign(1, 0)
|
|
216
|
+
tracker.capacity # => 256
|
|
217
|
+
OffsetMachines::CAPACITY # => 256
|
|
218
|
+
|
|
219
|
+
tracker.ack(1, 255) # => :recorded
|
|
220
|
+
tracker.ack(1, 256) # => :out_of_window
|
|
221
|
+
tracker.gaps(0, 1000).size # => 255
|
|
222
|
+
|
|
223
|
+
256.times { |i| tracker.ack(1, i) }
|
|
224
|
+
tracker.mark_committed(1, 256) # => :advanced
|
|
225
|
+
tracker.ack(1, 256) # => :recorded
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
The window never grows. An offset that falls outside it is a signal to stop fetching until the slow one finishes, or to shrink your concurrency.
|
|
229
|
+
|
|
230
|
+
## The Rules of the Tender
|
|
231
|
+
|
|
232
|
+
- **Generations and offsets are Integers** in `0..2**64-1`. Floats, nil, and strings raise `TypeError`. Out of range raises `RangeError`.
|
|
233
|
+
- **Refusals don't raise.** `:stale`, `:out_of_window`, `:unacked`, and `:exhausted` are answers. Only malformed input raises. A refused call changes nothing, not one bit.
|
|
234
|
+
- **A repeat is not a refusal.** `:duplicate` from `ack` or `mark_committed` means what you asked for already holds. It is honoured, and it also changes nothing.
|
|
235
|
+
- **Every answer, by method.**
|
|
236
|
+
|
|
237
|
+
| Method | Honoured | Refused | Records the clock |
|
|
238
|
+
| --- | --- | --- | --- |
|
|
239
|
+
| `Tracker.assigned(generation, base)` | a `Tracker` | raises `RangeError` / `TypeError` | never |
|
|
240
|
+
| `assign(generation, base)` | `:assigned` | `:stale` | never |
|
|
241
|
+
| `reassign(base)` | the new generation (Integer) | `:exhausted` | never |
|
|
242
|
+
| `ack(generation, offset)` | `:recorded`, `:duplicate` | `:stale`, `:out_of_window` | never |
|
|
243
|
+
| `mark_committed(generation, offset)` | `:advanced`, `:duplicate` | `:stale`, `:unacked` | never |
|
|
244
|
+
| `Status.new(generation, base, commit_candidate, frontier, acked)` | a `Status` | raises `RangeError` / `TypeError` | never |
|
|
245
|
+
|
|
246
|
+
Each refusal has its own symbol, the snake_case of the core's `Refusal` variant. There is no catch-all. The tracker takes no time at all, so no answer depends on an instant and none records one.
|
|
247
|
+
|
|
248
|
+
- **A `Status` is valid or does not exist.** `Status.new` (positional, in the order above, or by keyword) checks its five values through the core's `Status::new`: they must be a view some tracker of this window could report. A non-Integer raises `TypeError` (the two marks may be `nil`); anything else no tracker reports, such as a frontier past the window, a candidate above the frontier, or an `acked` count that doesn't fit the marks, raises `RangeError`. `Status` is final: subclassing it raises `TypeError` and `allocate` is gone. That stops accidents, not reflection (`Marshal.load`, `Data#initialize` bound by hand); no method of the gem reads a `Status` back, so a forged one never reaches the tracker. `Tracker` is sealed the same way: it wraps native state, so subclassing raises `TypeError` and `allocate` is gone.
|
|
249
|
+
|
|
250
|
+
- **The last offset is reserved.** Offset `2**64-1` is never ackable, so every commit candidate fits in an Integer the tracker accepts back.
|
|
251
|
+
- **One tracker, one partition.** Offsets from two partitions in one tracker is a bug the tracker can't see.
|
|
252
|
+
- **Frozen means frozen.** `freeze` the tracker and `assign`, `reassign`, `ack`, and `mark_committed` all raise `FrozenError`. Reads still work.
|
|
253
|
+
- **Native is required.** There is no pure-Ruby fallback.
|
|
254
|
+
|
|
255
|
+
## What You Think You're Doing vs Reality
|
|
256
|
+
|
|
257
|
+
### You Think: "I'll commit the highest offset I've finished."
|
|
258
|
+
### Reality: You commit every unfinished offset below it.
|
|
259
|
+
|
|
260
|
+
A commit is a claim about everything before a number, not about the number. The only number you can honestly commit is the end of the unbroken run. `commit_candidate` is that number, and it stays put until the slow offset finishes, no matter how far ahead the fast ones get.
|
|
261
|
+
|
|
262
|
+
## What's Yours
|
|
263
|
+
|
|
264
|
+
The tracker lives in your process and forgets everything when the process does. After a restart or a rebalance, `assign` a newer generation at the offset your store holds. Where generations come from is yours too: a consumer-group epoch, a lease token, a counter you persist. Fetching, running handlers, retrying a failed offset, the commit itself and its durability are all outside the gem. An offset is acked when you say so. If you ack before the side effect is durable, the tracker will cheerfully propose committing past work you can still lose.
|
|
265
|
+
|
|
266
|
+
## License
|
|
267
|
+
|
|
268
|
+
MIT License. See [LICENSE.txt](LICENSE.txt) for details.
|
|
269
|
+
|
|
270
|
+
**Remember: the fast offsets will finish first. The question is whether you commit past the slow one.**
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## [0.6.0](https://github.com/seuros/offset_machines/compare/offset-machines-v0.5.0...offset-machines-v0.6.0) (2026-11-16)
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
### ⚠ BREAKING CHANGES
|
|
7
|
+
|
|
8
|
+
* seal Tracker, add Tracker.assigned and word-skipping gaps
|
|
9
|
+
|
|
10
|
+
### Features
|
|
11
|
+
|
|
12
|
+
* seal Tracker, add Tracker.assigned and word-skipping gaps ([e1e46e8](https://github.com/seuros/offset_machines/commit/e1e46e845103b9b356fa480938dd49df29056ce4))
|
|
13
|
+
|
|
14
|
+
## [0.5.0](https://github.com/seuros/offset_machines/compare/offset-machines-v0.4.0...offset-machines-v0.5.0) (2026-11-14)
|
|
15
|
+
|
|
16
|
+
|
|
17
|
+
### ⚠ BREAKING CHANGES
|
|
18
|
+
|
|
19
|
+
* build the binding on matryona-machines 0.2; optional marks, compiler-checked symbols; ship no test files
|
|
20
|
+
|
|
21
|
+
### Features
|
|
22
|
+
|
|
23
|
+
* build the binding on matryona-machines 0.2; optional marks, compiler-checked symbols; ship no test files ([57a0356](https://github.com/seuros/offset_machines/commit/57a03561720799184005b4ff9b9debe857f80fb7))
|
|
24
|
+
|
|
25
|
+
## [0.4.0](https://github.com/seuros/offset_machines/compare/offset-machines-v0.3.0...offset-machines-v0.4.0) (2026-11-14)
|
|
26
|
+
|
|
27
|
+
|
|
28
|
+
### ⚠ BREAKING CHANGES
|
|
29
|
+
|
|
30
|
+
* validate Status through the core, seal it, and consolidate tracker internals
|
|
31
|
+
|
|
32
|
+
### Features
|
|
33
|
+
|
|
34
|
+
* validate Status through the core, seal it, and consolidate tracker internals ([3cc7dd3](https://github.com/seuros/offset_machines/commit/3cc7dd3e160c178f1d368464b1dd4029a60be7bd))
|
|
35
|
+
|
|
36
|
+
## [0.3.0](https://github.com/seuros/offset_machines/compare/offset-machines-v0.2.0...offset-machines-v0.3.0) (2026-11-13)
|
|
37
|
+
|
|
38
|
+
|
|
39
|
+
### ⚠ BREAKING CHANGES
|
|
40
|
+
|
|
41
|
+
* see CONVENTIONS: one exhaustive Refusal per crate, Result from every mutator, honoured idempotent repeats, Ruby mirrors the core. No compatibility aliases. Ruby ack answers :recorded; reassign answers :exhausted instead of nil.
|
|
42
|
+
|
|
43
|
+
### Features
|
|
44
|
+
|
|
45
|
+
* report refusals through Result<_, Refusal>, keep duplicate acks and commits honoured, add is_acked ([d1ac915](https://github.com/seuros/offset_machines/commit/d1ac9152bfc994abcc5f6bab20220a3961b5043b))
|
|
46
|
+
|
|
47
|
+
## [0.2.0](https://github.com/seuros/offset_machines/compare/offset-machines-v0.1.0...offset-machines-v0.2.0) (2026-11-12)
|
|
48
|
+
|
|
49
|
+
|
|
50
|
+
### Features
|
|
51
|
+
|
|
52
|
+
* initial extraction ([e061fd5](https://github.com/seuros/offset_machines/commit/e061fd59c8326783c70701e249d48f55805996e8))
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
[package]
|
|
2
|
+
name = "offset-machines"
|
|
3
|
+
version = "0.6.0"
|
|
4
|
+
edition.workspace = true
|
|
5
|
+
rust-version.workspace = true
|
|
6
|
+
license.workspace = true
|
|
7
|
+
description = "Bounded per-partition acknowledgement bitmap with contiguous commit candidates and generation-checked assignment"
|
|
8
|
+
readme = "README.md"
|
|
9
|
+
exclude = ["src/**/tests.rs", "src/tests/", "tests/"]
|
|
10
|
+
keywords = ["offset", "acknowledgement", "kafka", "partition", "no-std"]
|
|
11
|
+
categories = ["no-std", "algorithms"]
|
|
12
|
+
|
|
13
|
+
[dev-dependencies]
|
|
14
|
+
matryona-machines = { workspace = true, features = ["testing"] }
|
|
15
|
+
|
|
16
|
+
[lints]
|
|
17
|
+
workspace = true
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
# offset-machines
|
|
2
|
+
|
|
3
|
+
> **AMS Kreps**, NCC-4220, *Kreps*-class log tender.
|
|
4
|
+
|
|
5
|
+
Engine-room flight manual: tally finished offsets and find the first hole in the log.
|
|
6
|
+
|
|
7
|
+
Pre-1.0. Allocation-free `no_std` per-partition acknowledgement
|
|
8
|
+
bitmap: record which offsets finished processing, learn the contiguous commit
|
|
9
|
+
candidate, and list the gaps behind the highest acknowledged offset.
|
|
10
|
+
|
|
11
|
+
This is not a commit log, consumer runtime or offset store. The caller owns
|
|
12
|
+
execution, storage and the commit itself. The tracker takes explicit inputs,
|
|
13
|
+
has a compile-time window (`Tracker<WORDS>` covers `WORDS * 64` offsets) and
|
|
14
|
+
refuses rather than grows.
|
|
15
|
+
|
|
16
|
+
- `assign(generation, base) -> Result<(), Refusal>` installs a strictly newer
|
|
17
|
+
assignment generation and restarts the window; older or equal generations
|
|
18
|
+
(and zero) are refused as `Stale`.
|
|
19
|
+
- `Tracker::assigned(generation, base) -> Tracker` builds a tracker already
|
|
20
|
+
assigned: what an honoured `assign` makes of a fresh one. The generation is
|
|
21
|
+
a `NonZeroU64`, the one value `assign` refuses on a fresh tracker being
|
|
22
|
+
zero, so it cannot fail.
|
|
23
|
+
- `reassign(base) -> Result<u64, Refusal>` bumps the generation by one and
|
|
24
|
+
returns it; `Exhausted` when the generation is already `u64::MAX`.
|
|
25
|
+
- `ack(generation, offset) -> Result<Ack, Refusal>` answers `Ack::Recorded`, or
|
|
26
|
+
`Ack::Duplicate` for an idempotent repeat (already acked, or below the
|
|
27
|
+
window). Refused as `Stale` (wrong generation, or unassigned) or
|
|
28
|
+
`OutOfWindow`. Offset `u64::MAX` is never ackable.
|
|
29
|
+
- `is_acked(offset) -> bool` is the read-only form of that answer: `true` when
|
|
30
|
+
`offset` is below the base (committed) or recorded in the window, which is
|
|
31
|
+
exactly when `ack` under the current generation answers `Ack::Duplicate`.
|
|
32
|
+
`false` beyond the window, for `u64::MAX` and while unassigned.
|
|
33
|
+
- `commit_candidate()` is one past the contiguous acknowledged prefix, or
|
|
34
|
+
`None`. The adapter persists it however it likes.
|
|
35
|
+
- `mark_committed(generation, offset) -> Result<Commit, Refusal>` is the
|
|
36
|
+
adapter's confirmation of persistence. Only then does the window slide:
|
|
37
|
+
`Commit::Advanced`. Partial confirmation up to the candidate is allowed. An
|
|
38
|
+
offset at or below the base is an idempotent repeat, `Commit::Duplicate`.
|
|
39
|
+
Refused as `Stale` (wrong generation, or unassigned) or `Unacked` (beyond
|
|
40
|
+
the candidate).
|
|
41
|
+
- `status() -> Status` is the five readers as one value. `Status::new::<WORDS>(..)
|
|
42
|
+
-> Option<Status>` is how a `Status` is built from loose numbers: `None` for
|
|
43
|
+
any combination no `Tracker<WORDS>` reports (a frontier past the window, a
|
|
44
|
+
candidate above the frontier, a count that doesn't fit the marks, a base or
|
|
45
|
+
count while unassigned). The fields are private; a `Status` is valid by
|
|
46
|
+
construction.
|
|
47
|
+
- `gaps(from)` iterates unacknowledged offsets below the frontier; resume by
|
|
48
|
+
passing the last yielded offset plus one. Output is bounded by the window.
|
|
49
|
+
|
|
50
|
+
Every call that is not honoured returns `Err(Refusal)` and leaves the tracker
|
|
51
|
+
bit-for-bit unchanged. `Refusal` is the crate's only refusal type: `Stale`,
|
|
52
|
+
`OutOfWindow`, `Unacked`, `Exhausted`. No enum is `#[non_exhaustive]`: a new
|
|
53
|
+
variant is a breaking change every caller matches on. Idempotent repeats are honoured (`Ok`)
|
|
54
|
+
and also change nothing.
|
|
55
|
+
|
|
56
|
+
There is no clock: no operation takes or records an instant.
|
|
57
|
+
|
|
58
|
+
Crash recovery, persistence, rebalancing protocol and retry policy are adapter
|
|
59
|
+
concerns; after a restart, `assign` a newer generation at the stored offset.
|
|
60
|
+
|
|
61
|
+
```rust
|
|
62
|
+
use offset_machines::{Ack, Commit, Refusal, Tracker};
|
|
63
|
+
let mut tracker = Tracker::<4>::new(); // 256-offset window
|
|
64
|
+
assert_eq!(tracker.ack(1, 100), Err(Refusal::Stale)); // nothing assigned yet
|
|
65
|
+
assert_eq!(tracker.assign(1, 100), Ok(()));
|
|
66
|
+
assert_eq!(tracker.ack(1, 100), Ok(Ack::Recorded));
|
|
67
|
+
assert_eq!(tracker.ack(1, 102), Ok(Ack::Recorded));
|
|
68
|
+
assert_eq!(tracker.ack(1, 102), Ok(Ack::Duplicate));
|
|
69
|
+
assert!(tracker.is_acked(99) && tracker.is_acked(102) && !tracker.is_acked(101));
|
|
70
|
+
assert_eq!(tracker.commit_candidate(), Some(101));
|
|
71
|
+
assert_eq!(tracker.gaps(0).next(), Some(101));
|
|
72
|
+
assert_eq!(tracker.mark_committed(1, 102), Err(Refusal::Unacked));
|
|
73
|
+
assert_eq!(tracker.mark_committed(1, 101), Ok(Commit::Advanced));
|
|
74
|
+
assert_eq!(tracker.mark_committed(1, 101), Ok(Commit::Duplicate));
|
|
75
|
+
assert_eq!(tracker.base(), 101);
|
|
76
|
+
assert_eq!(tracker.reassign(101), Ok(2));
|
|
77
|
+
|
|
78
|
+
// Already assigned, with no refusal to handle: the generation is never zero.
|
|
79
|
+
let fresh = Tracker::<4>::assigned(core::num::NonZeroU64::MIN, 500);
|
|
80
|
+
assert_eq!((fresh.generation(), fresh.base()), (1, 500));
|
|
81
|
+
```
|