journal_machines 0.7.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +95 -0
- data/Cargo.lock +321 -0
- data/Cargo.toml +25 -0
- data/LICENSE.txt +21 -0
- data/README.md +401 -0
- data/ext/journal_machines_native/core/CHANGELOG.md +77 -0
- data/ext/journal_machines_native/core/Cargo.toml +18 -0
- data/ext/journal_machines_native/core/README.md +261 -0
- data/ext/journal_machines_native/core/src/checked.rs +161 -0
- data/ext/journal_machines_native/core/src/lib.rs +518 -0
- data/ext/journal_machines_native/core/src/pins.rs +277 -0
- data/ext/journal_machines_native/core/src/policy.rs +95 -0
- data/ext/journal_machines_native/core/src/refusal.rs +105 -0
- data/ext/journal_machines_native/core/src/replace.rs +258 -0
- data/ext/journal_machines_native/core/src/segment.rs +160 -0
- data/ext/journal_machines_native/extconf.rb +2 -0
- data/ext/journal_machines_native/ffi/Cargo.toml +21 -0
- data/ext/journal_machines_native/ffi/extconf.rb +14 -0
- data/ext/journal_machines_native/ffi/src/answers.rs +98 -0
- data/ext/journal_machines_native/ffi/src/classes.rs +46 -0
- data/ext/journal_machines_native/ffi/src/convert.rs +80 -0
- data/ext/journal_machines_native/ffi/src/handles.rs +49 -0
- data/ext/journal_machines_native/ffi/src/init.rs +86 -0
- data/ext/journal_machines_native/ffi/src/lib.rs +432 -0
- data/ext/journal_machines_native/ffi/src/values.rs +97 -0
- data/lib/journal_machines/version.rb +4 -0
- data/lib/journal_machines.rb +138 -0
- data/sig/journal_machines.rbs +209 -0
- metadata +97 -0
data/README.md
ADDED
|
@@ -0,0 +1,401 @@
|
|
|
1
|
+
# JournalMachines
|
|
2
|
+
|
|
3
|
+
> Deleting a log segment is easy. Deleting it at the right time is the job.
|
|
4
|
+
|
|
5
|
+
Segment bookkeeping for a segmented log, in Ruby. You tell the journal what your log files look like: how big the active one is, which records in the old ones are dead, who is still reading, how far recovery has got. It tells you when to roll, which segments are worth compacting, and when an old file may finally be deleted. It never touches a file itself.
|
|
6
|
+
|
|
7
|
+
## Quick Start
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
gem 'journal_machines'
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
```ruby
|
|
14
|
+
HOUR = 3600 * 1_000_000_000 # time is Integer nanoseconds
|
|
15
|
+
clock = -> { Process.clock_gettime(Process::CLOCK_MONOTONIC, :nanosecond) }
|
|
16
|
+
|
|
17
|
+
# max_bytes, max_records, max_age, min_dead_per_mille. nil turns a threshold off.
|
|
18
|
+
policy = JournalMachines::Policy.new(64 * 1024 * 1024, nil, HOUR, 500)
|
|
19
|
+
journal = JournalMachines::Journal.new(policy, 0, clock.call) # first offset 0, opened now
|
|
20
|
+
checkpoint = journal.pin_checkpoint(1, 0) # recovery replays from offset 0
|
|
21
|
+
|
|
22
|
+
# Writer loop: ask before every append batch.
|
|
23
|
+
if journal.roll_decision(file.size, file.records, clock.call) != :keep
|
|
24
|
+
file.fsync_and_close
|
|
25
|
+
rolled = journal.roll(file.size, file.records, clock.call)
|
|
26
|
+
file = open_segment(rolled.opened, rolled.next_offset)
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
# Compactor loop.
|
|
30
|
+
journal.compaction_candidates(nil, 64).each do |segment|
|
|
31
|
+
replacement = journal.propose(segment.id)
|
|
32
|
+
bytes, live = rewrite_and_fsync(segment) # your code, your fsync
|
|
33
|
+
journal.confirm_rewritten(replacement, bytes, live)
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
# Reaper loop: only what the journal lets go.
|
|
37
|
+
journal.reclaimable(nil, 64).each do |segment|
|
|
38
|
+
reclaimed = journal.reclaim(segment.replacement)
|
|
39
|
+
delete_file(reclaimed.segment, reclaimed.revision) if reclaimed.is_a?(JournalMachines::Reclaimed)
|
|
40
|
+
end
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
One journal, three loops. Offsets, sizes and times are Integers. Time is Integer nanoseconds on one monotonic clock, `Process.clock_gettime(Process::CLOCK_MONOTONIC, :nanosecond)`, for instants (`now`, a segment's `opened_at`) and spans (`max_age`) alike. No floats, no seconds, no `Time.now`.
|
|
44
|
+
|
|
45
|
+
## A Message to the Resistance
|
|
46
|
+
|
|
47
|
+
So you wrote an append-only log. Appending was the easy part. Then the disk filled, and you wrote a cleaner. The cleaner copied the live records of an old segment into a new file and deleted the old one.
|
|
48
|
+
|
|
49
|
+
Then a consumer that had been paused for an hour woke up holding a byte position into the file you deleted. So you added a check: don't delete while someone is reading. Then the machine lost power between the delete and the fsync of the new file, and the new file came back zero bytes long. So you added another check: fsync first. Then recovery replayed from the last checkpoint, which pointed into a segment the cleaner had judged finished. So you added a third check, in a third place, guarded by a third boolean.
|
|
50
|
+
|
|
51
|
+
Every one of those was the same rule, discovered one outage at a time. An original may go only when its replacement is durable *and* nobody who needs it is still behind it. The rule is small. The trouble is that it lived in three places and was enforced by remembering. This gem is the one place.
|
|
52
|
+
|
|
53
|
+
## Chapter 1: Stardate 2026.316, the AMS Rosenblum
|
|
54
|
+
|
|
55
|
+
> **AMS Rosenblum**, NCC-4228, *Rosenblum*-class archive cutter.
|
|
56
|
+
> Nothing leaves the hold until its copy is stowed and nobody is still reading it.
|
|
57
|
+
|
|
58
|
+
The archive cutter doesn't write the logbooks and doesn't shred them. It keeps the register of volumes: which one is open, which are closed, which have been recopied onto fresh stock, and who has one checked out. The shredder crew works from the register and from nothing else.
|
|
59
|
+
|
|
60
|
+
On a bridge somewhere, a cadet volunteers:
|
|
61
|
+
|
|
62
|
+
```ruby
|
|
63
|
+
def clean(segment)
|
|
64
|
+
compacted = rewrite(segment)
|
|
65
|
+
File.delete(segment.path)
|
|
66
|
+
compacted
|
|
67
|
+
end
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
"That," says the cutter's archivist, "deletes the only copy between two lines, while the reader on deck four is halfway through it."
|
|
71
|
+
|
|
72
|
+
## The Flight Plan
|
|
73
|
+
|
|
74
|
+
Every segment the journal knows about is in one of these states:
|
|
75
|
+
|
|
76
|
+
```mermaid
|
|
77
|
+
stateDiagram-v2
|
|
78
|
+
[*] --> active
|
|
79
|
+
active --> sealed: roll
|
|
80
|
+
sealed --> proposed: propose
|
|
81
|
+
proposed --> durable: confirm_rewritten / confirm_superseded
|
|
82
|
+
proposed --> sealed: abort
|
|
83
|
+
durable --> sealed: abort
|
|
84
|
+
durable --> reclaimable: last pin moves past the segment
|
|
85
|
+
reclaimable --> durable: a new pin lands on it
|
|
86
|
+
reclaimable --> sealed: reclaim, rewritten (new revision)
|
|
87
|
+
reclaimable --> [*]: reclaim, superseded
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
And the afternoon the cleaner wanted segment 1 while a reader was still in it:
|
|
91
|
+
|
|
92
|
+
```mermaid
|
|
93
|
+
sequenceDiagram
|
|
94
|
+
participant C as Compactor
|
|
95
|
+
participant J as Journal
|
|
96
|
+
participant R as Reader
|
|
97
|
+
|
|
98
|
+
R->>J: pin_reader(3)
|
|
99
|
+
J-->>R: Pin
|
|
100
|
+
C->>J: propose(1)
|
|
101
|
+
J-->>C: Replacement
|
|
102
|
+
C->>C: rewrite live records, fsync
|
|
103
|
+
C->>J: reclaim(replacement)
|
|
104
|
+
J-->>C: :not_durable
|
|
105
|
+
C->>J: confirm_rewritten(replacement, 30, 4)
|
|
106
|
+
J-->>C: :recorded
|
|
107
|
+
C->>J: reclaim(replacement)
|
|
108
|
+
J-->>C: :pinned
|
|
109
|
+
Note over C: keeps both files
|
|
110
|
+
R->>J: advance(pin, 10)
|
|
111
|
+
J-->>R: :moved
|
|
112
|
+
C->>J: reclaim(replacement)
|
|
113
|
+
J-->>C: Reclaimed
|
|
114
|
+
Note over C: deletes revision 0. The reader never noticed.
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
## How the Journal Works
|
|
118
|
+
|
|
119
|
+
### Roll
|
|
120
|
+
|
|
121
|
+
```ruby
|
|
122
|
+
policy = JournalMachines::Policy.new(100, 10, 50, 500)
|
|
123
|
+
policy.max_age # => 50
|
|
124
|
+
journal = JournalMachines::Journal.new(policy, 0, 1000)
|
|
125
|
+
journal.policy == policy # => true
|
|
126
|
+
|
|
127
|
+
journal.roll_decision(99, 9, 1049) # => :keep
|
|
128
|
+
journal.roll_decision(100, 9, 1000) # => #<data JournalMachines::Due reason=:size>
|
|
129
|
+
journal.roll_decision(1, 10, 1000) # => #<data JournalMachines::Due reason=:records>
|
|
130
|
+
journal.roll_decision(1, 1, 1050) # => #<data JournalMachines::Due reason=:age>
|
|
131
|
+
journal.roll_decision(0, 0, 9999) # => :keep, an empty segment is never due
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
`roll_decision` answers the core's `Roll`: the symbol `:keep`, or a `Due(reason)` naming the first threshold reached. It only reads. You pass what you observed about the active segment, bytes, records and the time, and it compares them with the thresholds, in that order. Leave a threshold out and it never triggers.
|
|
135
|
+
|
|
136
|
+
`now` and `max_age` are nanoseconds, so the segment above was opened at 1000 ns and is due by age 50 ns later. Real code passes the monotonic clock reading and a `max_age` like `30 * 60 * 1_000_000_000`. `segment.opened_at` reads back the same nanoseconds.
|
|
137
|
+
|
|
138
|
+
When you have actually sealed the file, say so:
|
|
139
|
+
|
|
140
|
+
```ruby
|
|
141
|
+
rolled = journal.roll(100, 9, 1000)
|
|
142
|
+
rolled.sealed # => 1
|
|
143
|
+
rolled.opened # => 2
|
|
144
|
+
rolled.next_offset # => 9
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
`Policy.new(max_bytes, max_records, max_age, min_dead_per_mille)` holds the three roll thresholds and the dead-record ratio for compaction. `Journal.new(policy, base_offset, now)` opens one active segment at `base_offset`, at instant `now`. Neither has defaults: every argument is yours to choose. A threshold of `nil` never triggers. A threshold of zero, or a ratio above 1000, raises `RangeError` from `Policy.new`, so a bad policy never exists.
|
|
148
|
+
|
|
149
|
+
A roll doesn't have to be due. Sealing early is your call. Sealing an empty segment is `:empty`.
|
|
150
|
+
|
|
151
|
+
### Compaction candidates
|
|
152
|
+
|
|
153
|
+
The journal doesn't know which records are dead. You do, so you tell it:
|
|
154
|
+
|
|
155
|
+
```ruby
|
|
156
|
+
policy = JournalMachines::Policy.new(nil, nil, nil, 500) # no roll thresholds, compact at half dead
|
|
157
|
+
journal = JournalMachines::Journal.new(policy, 0, 0)
|
|
158
|
+
3.times { journal.roll(100, 10, 0) } # segments 1, 2, 3 sealed; 4 active
|
|
159
|
+
|
|
160
|
+
journal.observe_live(1, 5) # => :recorded, half dead
|
|
161
|
+
journal.observe_live(2, 6) # => :recorded, not dead enough
|
|
162
|
+
journal.observe_live(3, 0) # => :recorded, all dead
|
|
163
|
+
|
|
164
|
+
journal.compaction_candidates(nil, 64).map(&:id) # => [1, 3]
|
|
165
|
+
journal.compaction_candidates(nil, 1).map(&:id) # => [1]
|
|
166
|
+
journal.compaction_candidates(1, 1).map(&:id) # => [3]
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
A sealed segment is a candidate when at least `min_dead_per_mille` of its stored records are dead (500 here, so half) and it has no replacement in flight. Candidates come oldest first, at most `limit` at a time. Pass the last id you handled to get the next batch.
|
|
170
|
+
|
|
171
|
+
### Replacement
|
|
172
|
+
|
|
173
|
+
```ruby
|
|
174
|
+
replacement = journal.propose(1)
|
|
175
|
+
replacement.class # => JournalMachines::Replacement
|
|
176
|
+
journal.segment(1).state # => :proposed
|
|
177
|
+
journal.propose(1) # => :busy
|
|
178
|
+
|
|
179
|
+
journal.confirm_rewritten(replacement, 40, 5) # => :recorded
|
|
180
|
+
journal.confirm_rewritten(replacement, 40, 5) # => :already_durable, a repeat
|
|
181
|
+
journal.confirm_rewritten(replacement, 41, 5) # => :mismatch, that is not what was recorded
|
|
182
|
+
journal.confirm_superseded(replacement) # => :mismatch
|
|
183
|
+
journal.segment(1).state # => :reclaimable
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
Confirming twice with the same numbers is not an error. The attempt is durable with exactly that, which is what you asked for, so the answer is `:already_durable`. Confirming it again with anything else, other bytes, another record count, or the other method, is `:mismatch`: the journal never recorded that, so it won't tell you it did. The first confirmation stands. If it was wrong, `abort` and propose again.
|
|
187
|
+
|
|
188
|
+
`propose` marks a segment as being replaced. `confirm_rewritten(replacement, bytes, live_records)` says a compacted copy of the same offsets is on disk and synced. `confirm_superseded(replacement)` says something else now covers the range, a snapshot or an archive, or nothing at all because every record was dead. The journal takes your word for it. It cannot see your disk.
|
|
189
|
+
|
|
190
|
+
The core has one call here, `confirm_durable(replacement, durable)`, where `durable` is `Rewritten { bytes, live_records }` or `Superseded`. Ruby has one method per variant, named after it and taking that variant's fields in order. Both answer `:recorded` the first time.
|
|
191
|
+
|
|
192
|
+
Changed your mind, or the rewrite failed?
|
|
193
|
+
|
|
194
|
+
```ruby
|
|
195
|
+
journal.abort(replacement) # => :aborted
|
|
196
|
+
journal.segment(1).state # => :sealed
|
|
197
|
+
journal.abort(replacement) # => :stale
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
### Pins
|
|
201
|
+
|
|
202
|
+
A pin says "everything at or after this offset is still needed."
|
|
203
|
+
|
|
204
|
+
```ruby
|
|
205
|
+
journal = JournalMachines::Journal.new(policy, 0, 0)
|
|
206
|
+
2.times { journal.roll(100, 10, 0) } # segment 1 holds 0...10, segment 2 holds 10...20
|
|
207
|
+
|
|
208
|
+
reader = journal.pin_reader(3)
|
|
209
|
+
reader.class # => JournalMachines::Pin
|
|
210
|
+
journal.low_pin # => 3
|
|
211
|
+
journal.segment(1).retention # => :pinned
|
|
212
|
+
|
|
213
|
+
journal.advance(reader, 10) # => :moved
|
|
214
|
+
journal.advance(reader, 10) # => :unchanged, already there
|
|
215
|
+
journal.advance(reader, 4) # => :regressed
|
|
216
|
+
journal.segment(1).retention # => :unpinned
|
|
217
|
+
|
|
218
|
+
journal.release(reader) # => :released
|
|
219
|
+
journal.release(reader) # => :stale
|
|
220
|
+
journal.advance(reader, 11) # => :stale
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
Release is checked against the acquisition, not the slot. Once released, a pin handle never names a pin again, even after the journal hands the same slot to someone else: `advance`, `checkpoint` and `pin_offset` answer `:stale` (or `nil`) and leave the new holder alone. A second `release` is `:stale` too, and changes nothing. The journal keeps no list of released pins, and it can't work the answer out: pins, segments and attempts are numbered from one sequence, so "one of my numbers that no pin holds" is just as true of a segment id. It only honours a repeat it can prove. A pin it never issued, from a journal of another scope or with a number it hasn't reached, is `:stale` as well.
|
|
224
|
+
|
|
225
|
+
Moving a pin to where it already sits is `:unchanged`, an answer and not a refusal. Moving it back is `:regressed`, a refusal: the journal may already have let go of what lies behind the pin, so it cannot promise it again.
|
|
226
|
+
|
|
227
|
+
A checkpoint pin also carries the generation of the consumer assignment that reported it:
|
|
228
|
+
|
|
229
|
+
```ruby
|
|
230
|
+
checkpoint = journal.pin_checkpoint(3, 20) # assignment generation 3, offset 20
|
|
231
|
+
|
|
232
|
+
journal.checkpoint(checkpoint, 3, 25) # => :advanced
|
|
233
|
+
journal.checkpoint(checkpoint, 2, 99) # => :stale_assignment
|
|
234
|
+
journal.checkpoint(checkpoint, 4, 8) # => :reassigned
|
|
235
|
+
journal.pin_offset(checkpoint) # => 8
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
A consumer that was fenced off can't drag retention forward with an offset it committed before it lost the partition. A newer assignment takes over at whatever offset it reports, even a lower one.
|
|
239
|
+
|
|
240
|
+
If the generation and the offset come from an offset tracker, hand over the tracker instead of two loose integers:
|
|
241
|
+
|
|
242
|
+
```ruby
|
|
243
|
+
tracker = OffsetMachines::Tracker.new # gem 'offset_machines'
|
|
244
|
+
journal.pin_checkpoint_from(tracker) # => :unassigned, nothing pinned
|
|
245
|
+
|
|
246
|
+
tracker.assign(3, 20)
|
|
247
|
+
checkpoint = journal.pin_checkpoint_from(tracker)
|
|
248
|
+
journal.pin_offset(checkpoint) # => 20
|
|
249
|
+
|
|
250
|
+
tracker.ack(3, 20) # acknowledged, not committed
|
|
251
|
+
journal.checkpoint_from(checkpoint, tracker) # => :unchanged
|
|
252
|
+
tracker.mark_committed(3, 21)
|
|
253
|
+
journal.checkpoint_from(checkpoint, tracker) # => :advanced, now at 21
|
|
254
|
+
|
|
255
|
+
zombie = OffsetMachines::Tracker.new
|
|
256
|
+
zombie.assign(2, 99)
|
|
257
|
+
journal.checkpoint_from(checkpoint, zombie) # => :stale_assignment
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
`pin_checkpoint_from(tracker)` is `pin_checkpoint(tracker.generation, tracker.base)` and `checkpoint_from(pin, tracker)` is `checkpoint(pin, tracker.generation, tracker.base)`, with the same answers. Both numbers are read from the one object you pass, so they can't come from two different trackers. This gem doesn't depend on `offset_machines`: any object with public `generation` and `base` readers fits, `OffsetMachines::Tracker` being the one they were written for. The binding asks for both before calling either, so an object missing one raises `TypeError` naming it (`expected an offset tracker responding to #generation and #base, Object lacks #base`) and runs neither.
|
|
261
|
+
|
|
262
|
+
### Reclaim
|
|
263
|
+
|
|
264
|
+
```ruby
|
|
265
|
+
journal = JournalMachines::Journal.new(policy, 0, 0)
|
|
266
|
+
journal.roll(100, 10, 0)
|
|
267
|
+
reader = journal.pin_reader(3)
|
|
268
|
+
replacement = journal.propose(1)
|
|
269
|
+
|
|
270
|
+
journal.reclaim(replacement) # => :not_durable
|
|
271
|
+
journal.confirm_rewritten(replacement, 30, 4) # => :recorded
|
|
272
|
+
journal.reclaim(replacement) # => :pinned
|
|
273
|
+
journal.reclaimable(nil, 64) # => []
|
|
274
|
+
|
|
275
|
+
journal.advance(reader, 10) # => :moved
|
|
276
|
+
reclaimed = journal.reclaim(replacement)
|
|
277
|
+
reclaimed.revision # => 0, the file to delete
|
|
278
|
+
reclaimed.bytes # => 100
|
|
279
|
+
reclaimed.replaced_by # => 4, the revision now serving 0...10: the attempt number
|
|
280
|
+
```
|
|
281
|
+
|
|
282
|
+
Two conditions, both required: the replacement is confirmed durable, and no pin of either kind sits below the segment's end. Miss one and you get a symbol and nothing changes. A rewritten segment keeps its id and its offsets under a new revision. A superseded one leaves the journal and `replaced_by` is `nil`.
|
|
283
|
+
|
|
284
|
+
### Capacity
|
|
285
|
+
|
|
286
|
+
The journal tracks 64 segments and 32 pins. Past that it refuses:
|
|
287
|
+
|
|
288
|
+
```ruby
|
|
289
|
+
journal = JournalMachines::Journal.new(policy, 0, 0)
|
|
290
|
+
journal.segment_capacity # => 64
|
|
291
|
+
63.times { journal.roll(1, 1, 0) }
|
|
292
|
+
journal.roll(1, 1, 0) # => :segments_full
|
|
293
|
+
|
|
294
|
+
32.times { journal.pin_reader(0) }
|
|
295
|
+
journal.pin_reader(0) # => :pins_full
|
|
296
|
+
```
|
|
297
|
+
|
|
298
|
+
A `:segments_full` roll means your writer keeps appending to the segment it has until you reclaim something. Nothing is dropped to make room.
|
|
299
|
+
|
|
300
|
+
### Time only moves forward
|
|
301
|
+
|
|
302
|
+
```ruby
|
|
303
|
+
journal = JournalMachines::Journal.new(JournalMachines::Policy.new(nil, nil, 10, 500), 0, 100)
|
|
304
|
+
journal.roll_decision(1, 1, 99) # => :time_regression
|
|
305
|
+
journal.roll(1, 1, 99) # => :time_regression
|
|
306
|
+
```
|
|
307
|
+
|
|
308
|
+
A clock reading earlier than the moment the active segment was opened is refused, not guessed at.
|
|
309
|
+
|
|
310
|
+
### Every answer
|
|
311
|
+
|
|
312
|
+
A call that is honoured answers with a value. A call that is not honoured answers with one of the refusal symbols below, and only those, and changes nothing. A repeat of something that already holds is honoured: it gets its own answer in the Honoured column, marked *repeat*, and it changes nothing either.
|
|
313
|
+
|
|
314
|
+
| Call | Honoured | Refusals | Records the clock |
|
|
315
|
+
| --- | --- | --- | --- |
|
|
316
|
+
| `Journal.new`, `with_scope`, `resumed` | a `Journal` | none; bad input raises | yes: the first segment opens at `now` |
|
|
317
|
+
| `roll_decision(bytes, records, now)` | `:keep`, `Due(:size)`, `Due(:records)`, `Due(:age)` | `:time_regression`, `:overflow`, `:no_active` | no, a read: the journal keeps no watermark, and `now` is judged against the active segment's opening |
|
|
318
|
+
| `roll(bytes, records, now)` | `Rolled` | `:empty`, `:time_regression`, `:overflow`, `:segments_full`, `:exhausted`, `:no_active` | yes: the new active segment opens at `now` |
|
|
319
|
+
| `observe_live(id, live_records)` | `:recorded` | `:unknown`, `:active`, `:exceeds` | no clock |
|
|
320
|
+
| `pin_reader(offset)` | `Pin` | `:pins_full`, `:exhausted` | no clock |
|
|
321
|
+
| `pin_checkpoint(assignment, offset)`, `pin_checkpoint_from(tracker)` | `Pin` | `:unassigned`, `:pins_full`, `:exhausted` | no clock |
|
|
322
|
+
| `advance(pin, offset)` | `:moved`, `:unchanged` (*repeat*) | `:stale`, `:wrong_kind`, `:regressed` | no clock |
|
|
323
|
+
| `checkpoint(pin, assignment, offset)`, `checkpoint_from(pin, tracker)` | `:advanced`, `:reassigned`, `:unchanged` (*repeat*) | `:stale`, `:wrong_kind`, `:unassigned`, `:stale_assignment`, `:regressed` | no clock |
|
|
324
|
+
| `release(pin)` | `:released` | `:stale` | no clock |
|
|
325
|
+
| `propose(id)` | `Replacement` | `:unknown`, `:active`, `:busy`, `:exhausted` | no clock |
|
|
326
|
+
| `confirm_rewritten(replacement, bytes, live_records)` | `:recorded`, `:already_durable` (*repeat*) | `:stale`, `:mismatch`, `:exceeds` | no clock |
|
|
327
|
+
| `confirm_superseded(replacement)` | `:recorded`, `:already_durable` (*repeat*) | `:stale`, `:mismatch` | no clock |
|
|
328
|
+
| `reclaim(replacement)` | `Reclaimed` | `:stale`, `:not_durable`, `:pinned` | no clock |
|
|
329
|
+
| `abort(replacement)` | `:aborted` | `:stale` | no clock |
|
|
330
|
+
| `Policy.new`, `Due.new`, `Rolled.new`, `Reclaimed.new`, `Segment.new` | the value | none; a value no journal could hold or answer raises `RangeError`, a wrong type `TypeError` | no clock |
|
|
331
|
+
|
|
332
|
+
Each refusal is the snake_case of one core `Refusal` variant. There is no catch-all symbol. The only instants the journal keeps are its segments' opening times, and only the constructors and an honoured `roll` set one; no repeat takes a clock at all.
|
|
333
|
+
|
|
334
|
+
The repeats. Only one the journal can prove from its own state is honoured:
|
|
335
|
+
|
|
336
|
+
| Repeat | Meaning |
|
|
337
|
+
| --- | --- |
|
|
338
|
+
| `:unchanged` | The pin already sits at that offset (for a checkpoint: under that same assignment generation). |
|
|
339
|
+
| `:already_durable` | This attempt was confirmed before by the same method with the same bytes and record count. |
|
|
340
|
+
|
|
341
|
+
Each refusal means one thing wherever it shows up:
|
|
342
|
+
|
|
343
|
+
| Refusal | Meaning |
|
|
344
|
+
| --- | --- |
|
|
345
|
+
| `:no_active` | There is no active segment. Not reachable with the compiled capacity of 64. |
|
|
346
|
+
| `:empty` | The active segment holds no records, so there is nothing to seal. |
|
|
347
|
+
| `:time_regression` | `now` is earlier than the moment the active segment was opened. |
|
|
348
|
+
| `:overflow` | The offset range would pass `2**64 - 1`. |
|
|
349
|
+
| `:segments_full` | All 64 segment slots are taken. Reclaim one. |
|
|
350
|
+
| `:pins_full` | All 32 pin slots are held. Release one. |
|
|
351
|
+
| `:exhausted` | The journal ran out of numbers for segment ids, pin generations and attempts. That takes `2**48 - 1` of them, all kinds together, per scope. |
|
|
352
|
+
| `:unknown` | No segment has that id. |
|
|
353
|
+
| `:active` | That is the active segment, still being written. |
|
|
354
|
+
| `:busy` | The segment already has a replacement in flight. |
|
|
355
|
+
| `:exceeds` | More live records than the segment stores. |
|
|
356
|
+
| `:mismatch` | The attempt is already durable, but not with this: `confirm_superseded` after `confirm_rewritten` or the reverse, or `confirm_rewritten` with other bytes or another record count. The first confirmation stands. |
|
|
357
|
+
| `:stale` | The `Pin` or `Replacement` is not current: released, aborted, reclaimed, superseded by a newer attempt, or one this journal never issued (another scope, or a number it hasn't reached). Releasing a pin twice is `:stale`. |
|
|
358
|
+
| `:wrong_kind` | `advance` on a checkpoint pin, or `checkpoint` on a reader pin. |
|
|
359
|
+
| `:unassigned` | Assignment generation zero is not an assignment. |
|
|
360
|
+
| `:stale_assignment` | The assignment generation is older than the one the checkpoint holds. |
|
|
361
|
+
| `:regressed` | The offset is behind the pin. |
|
|
362
|
+
| `:not_durable` | The replacement hasn't been confirmed durable. |
|
|
363
|
+
| `:pinned` | A pin still sits below the segment's end. |
|
|
364
|
+
|
|
365
|
+
The queries `active`, `segment`, `covering`, `segments`, `compaction_candidates`, `reclaimable`, `pin_offset`, `low_pin`, `pins`, `policy`, `size`, `empty?`, `segment_capacity` and `pin_capacity` never refuse. They answer a value, `nil` or an empty array. `segments(after, limit)`, `compaction_candidates(after, limit)` and `reclaimable(after, limit)` take both arguments every time: `after` is the last segment id you were given, or `nil` to begin, and `limit` bounds the Array (the core hands out an iterator instead). `after` has to be a number this journal issued: one whose segment has since been reclaimed still works as a position, but a number it never issued (another scope, sequence zero such as a bare `0`, or one it hasn't reached) is no position here, and the scan answers `[]`. Begin with `nil`, not `0`.
|
|
366
|
+
|
|
367
|
+
## The Rules of the Archivist
|
|
368
|
+
|
|
369
|
+
- **Everything is an Integer** in `0..2**64-1`: offsets, sizes, counts, times, ids, generations. Floats, nil, and strings raise `TypeError`. Out of range raises `RangeError`. A threshold may be `nil`, never zero. A missing argument raises `ArgumentError`: nothing has a default.
|
|
370
|
+
- **Time is Integer nanoseconds.** `now` (in `Journal.new`, `roll_decision` and `roll`), `max_age` and `Segment#opened_at` are nanoseconds on one monotonic clock: `Process.clock_gettime(Process::CLOCK_MONOTONIC, :nanosecond)`. A Float or any other non-Integer raises `TypeError`; a negative value or one past `2**64-1` raises `RangeError`.
|
|
371
|
+
- **Scopes.** A `Pin`, a `Replacement` and a segment id are only meaningful to the `Journal` that issued them. They are numbered from 1, so two journals hand out the same ones, and a handle given to the wrong journal would name a live pin or attempt there. One journal never repeats itself, though: segment ids, pin generations and attempt numbers come from a single sequence (the first segment is 1, and every roll, pin and proposal takes the next number), so a generation or an attempt passed where a segment id goes names no segment. Segment ids still ascend; they aren't consecutive. Give each journal its own scope: `Journal.with_scope(policy, base_offset, now, scope)`, any Integer in `0..65_535` (a non-Integer raises `TypeError`, anything outside the range raises `RangeError`). `Journal.new(policy, base_offset, now)` is scope zero. The scope rides in the top 16 bits of every segment id, pin generation and attempt number (and so of a rewritten segment's `revision`); `journal.scope`, `pin.scope` and `replacement.scope` read it back. A journal that never issued it, of another scope or not that far along, answers `:stale` to a pin or replacement (`nil` from `pin_offset`) and `:unknown` to a segment id (`nil` from `segment`, `[]` from the scans), and changes nothing. The journal can't pick a scope for you, and two journals left in the same scope still mix up each other's handles. `Pin` and `Replacement` objects can't be built by hand. They are values: two handles to the same pin acquisition or replacement attempt are `==` and `eql?` and hash alike, so the `replacement` of two reads of one proposed segment compares equal, and so do the two `Segment`s.
|
|
372
|
+
- **Values are valid or do not exist.** `Policy`, `Due`, `Rolled`, `Reclaimed` and `Segment` check themselves through the core when they are built, by hand or by the journal: `Due.new(:nope)`, `Rolled.new(2, 2, 10)`, `Reclaimed.new(1, 0, 9, 1)` or a `Segment` that is `:active` with records all raise `RangeError`; a String for a Symbol, `true` for `retention`, or an Integer for a `Replacement` raises `TypeError`. What the journal answers always passes, so `==` against a value you built means what it says. The values, and the `Pin` and `Replacement` handles, are final: subclassing one raises `TypeError` and `allocate` is gone. That stops accidents, not reflection (a named subclass still leaves its constant behind; `Marshal.load` and a hand-bound `Data#initialize` skip the check), so the extension is the authority: the `Policy` a journal opens with must be exactly a `Policy` (`TypeError` otherwise, a subclass included), and its fields are read natively and checked again by the core (`RangeError`).
|
|
373
|
+
- **The journal never loses work.** Every argument is converted before the journal is asked, and the value classes are resolved once, when the extension loads. So are the classes of `Journal`, `Pin` and `Replacement`: a handle is built with no Ruby code run, an instance of the class the gem defined even if the constant is later redefined, and a redefined `Object.const_get` is never consulted. Building an answer runs Ruby code (`new`, which a reopened class can redefine), so `roll` and `reclaim` decide on a copy, build the `Rolled` or `Reclaimed`, and keep the copy only if the journal is still unfrozen and unchanged: a `new` that raises applies nothing, one that freezes the journal raises `FrozenError`, one that calls back in and changes it raises `RuntimeError`, and the outer call is not applied.
|
|
374
|
+
- **The binding never loses work.** Everything that can raise (a bad argument, a frozen journal, a tracker whose readers raise) happens before the journal is asked; after it has changed, the binding only builds values from its answer. The scans read their segments into a buffer before building a single Ruby value.
|
|
375
|
+
- **Refusals don't raise.** `:segments_full`, `:pinned`, `:not_durable`, `:stale` and the rest are answers. A refused call changes nothing. Only malformed input raises.
|
|
376
|
+
- **Restarts.** A journal built again with `new` or `with_scope` numbers from 1 again, so a pin or replacement that outlived the restart could name a new entry. Either resume the numbering, `Journal.resumed(policy, base_offset, now, scope, issued)` with the `journal.issued` you saved from the journal before it, or give each incarnation its own scope. `resumed` restores the numbering and nothing else: segments and pins are yours to replay, and the one active segment takes the next number. It raises `RangeError` for an `issued` outside `0..2**48-2`. Save `issued` before you hand a number out: resuming from too low a count reissues.
|
|
377
|
+
- **Repeats aren't refusals, when they can be proved.** `:unchanged` and `:already_durable` mean the journal can see that exactly what you asked for was already done. Treat them as success. They change nothing. A repeat it can't prove is refused: a second `release`, `reclaim` or `abort` is `:stale` (it keeps neither the pins it released nor how an attempt ended), a second confirmation with different contents is `:mismatch`.
|
|
378
|
+
- **No pins means no objection.** The journal protects what is pinned. If nobody pinned a checkpoint, nothing stops a durable replacement from being reclaimed.
|
|
379
|
+
- **Frozen means frozen.** `freeze` the journal and every mutating call raises `FrozenError`.
|
|
380
|
+
|
|
381
|
+
## What You Think You're Doing vs Reality
|
|
382
|
+
|
|
383
|
+
### You Think: "The compacted file is written, so the old one can go."
|
|
384
|
+
### Reality: Written is not durable, and durable is not unread.
|
|
385
|
+
|
|
386
|
+
A file you wrote and didn't sync is a promise from the page cache. A file you synced can still be the wrong one to keep alone, if a reader holds a position in the original or recovery intends to replay through it. `reclaim` won't say yes until you've confirmed the first and every pin has cleared the second.
|
|
387
|
+
|
|
388
|
+
### You Think: "The consumer acknowledged it, so the checkpoint can move."
|
|
389
|
+
### Reality: Acknowledged is not committed.
|
|
390
|
+
|
|
391
|
+
Move the checkpoint pin to the offset your commit actually persisted, not to the one a worker says it finished. `pin_checkpoint_from` and `checkpoint_from` pair with `offset_machines` for exactly this: they pin at the tracker's confirmed base, never at what was merely acknowledged.
|
|
392
|
+
|
|
393
|
+
## What's Yours
|
|
394
|
+
|
|
395
|
+
Everything that touches a disk. You write the records, seal the files, rewrite the compacted copies, fsync, and delete. The journal keeps metadata in your process and loses it when the process dies. After a restart, rebuild a journal from what's on disk and from your own durable manifest (see Restarts above for its numbering), then re-pin readers and checkpoints before letting the reaper run. A half-finished replacement found on disk after a crash is yours to finish or discard. This is not a log writer, not a byte compactor and not a Raft log. It's an initial release and the API will move.
|
|
396
|
+
|
|
397
|
+
## License
|
|
398
|
+
|
|
399
|
+
MIT License. See [LICENSE.txt](LICENSE.txt) for details.
|
|
400
|
+
|
|
401
|
+
**Remember: anyone can delete a file. The question is whether someone was still reading it.**
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## [0.8.1](https://github.com/seuros/journal_machines/compare/journal-machines-v0.8.0...journal-machines-v0.8.1) (2026-11-16)
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
### Bug Fixes
|
|
7
|
+
|
|
8
|
+
* fix wrapped classes at load so building a handle runs no Ruby ([f87e049](https://github.com/seuros/journal_machines/commit/f87e049d521da4c49f9b160a5247c2a94307aaf5))
|
|
9
|
+
|
|
10
|
+
## [0.8.0](https://github.com/seuros/journal_machines/compare/journal-machines-v0.7.0...journal-machines-v0.8.0) (2026-11-16)
|
|
11
|
+
|
|
12
|
+
|
|
13
|
+
### ⚠ BREAKING CHANGES
|
|
14
|
+
|
|
15
|
+
* replace Segment pinned bool with a Retention enum
|
|
16
|
+
|
|
17
|
+
### Features
|
|
18
|
+
|
|
19
|
+
* replace Segment pinned bool with a Retention enum ([98d6904](https://github.com/seuros/journal_machines/commit/98d690430cea94988b5ceb7389d7238999896dcd))
|
|
20
|
+
|
|
21
|
+
## [0.7.0](https://github.com/seuros/journal_machines/compare/journal-machines-v0.6.0...journal-machines-v0.7.0) (2026-11-14)
|
|
22
|
+
|
|
23
|
+
|
|
24
|
+
### ⚠ BREAKING CHANGES
|
|
25
|
+
|
|
26
|
+
* adopt matryona-machines 0.2: exact-class Pin/Replacement handles, positive thresholds, compiler-checked symbol tables
|
|
27
|
+
|
|
28
|
+
### Features
|
|
29
|
+
|
|
30
|
+
* adopt matryona-machines 0.2: exact-class Pin/Replacement handles, positive thresholds, compiler-checked symbol tables ([53ef3b1](https://github.com/seuros/journal_machines/commit/53ef3b10ded3b1b994e2da14f04ecd5dcf5c7a7c))
|
|
31
|
+
|
|
32
|
+
## [0.6.0](https://github.com/seuros/journal_machines/compare/journal-machines-v0.5.0...journal-machines-v0.6.0) (2026-11-14)
|
|
33
|
+
|
|
34
|
+
|
|
35
|
+
### ⚠ BREAKING CHANGES
|
|
36
|
+
|
|
37
|
+
* validate values through the core, value handles, native answers, read Policy by exact class, clone-then-keep roll/reclaim
|
|
38
|
+
|
|
39
|
+
### Features
|
|
40
|
+
|
|
41
|
+
* validate values through the core, value handles, native answers, read Policy by exact class, clone-then-keep roll/reclaim ([5b00d40](https://github.com/seuros/journal_machines/commit/5b00d406d92bdfb6f1ec331b60c61243ec013994))
|
|
42
|
+
|
|
43
|
+
## [0.5.0](https://github.com/seuros/journal_machines/compare/journal-machines-v0.4.0...journal-machines-v0.5.0) (2026-11-13)
|
|
44
|
+
|
|
45
|
+
|
|
46
|
+
### ⚠ BREAKING CHANGES
|
|
47
|
+
|
|
48
|
+
* handles are issued through periscope-machines. Every instance can be built with a scope; a handle from another scope is refused before any lookup; all handle kinds of an instance share one sequence; resumed restores the sequence after a restart. Handles of a journal no longer share raw values; release of a pin that is not held is Stale; the Ruby owner tag is gone.
|
|
49
|
+
|
|
50
|
+
### Features
|
|
51
|
+
|
|
52
|
+
* draw segment ids, pin generations and attempts from one scoped periscope sequence; add Journal::resumed ([6a9b959](https://github.com/seuros/journal_machines/commit/6a9b9597aceffa452dee5d74903f99b184a15d2e))
|
|
53
|
+
|
|
54
|
+
## [0.4.0](https://github.com/seuros/journal_machines/compare/journal-machines-v0.3.0...journal-machines-v0.4.0) (2026-11-13)
|
|
55
|
+
|
|
56
|
+
|
|
57
|
+
### ⚠ BREAKING CHANGES
|
|
58
|
+
|
|
59
|
+
* see CONVENTIONS: one exhaustive Refusal per crate, Result from every mutator, honoured idempotent repeats only when provable, Duration time, Ruby mirrors the core. No compatibility aliases. Requires offset-machines >= 0.3. A repeated confirm with another payload is Mismatch; releasing a pin that is not held is Stale.
|
|
60
|
+
|
|
61
|
+
### Features
|
|
62
|
+
|
|
63
|
+
* one Refusal enum with Result everywhere, provable repeats only, Duration time, Ruby Policy and _from pin calls ([f820899](https://github.com/seuros/journal_machines/commit/f8208990722b8407f206f8714c5708720f72dc74))
|
|
64
|
+
|
|
65
|
+
## [0.3.0](https://github.com/seuros/journal_machines/compare/journal-machines-v0.2.0...journal-machines-v0.3.0) (2026-11-13)
|
|
66
|
+
|
|
67
|
+
|
|
68
|
+
### Features
|
|
69
|
+
|
|
70
|
+
* add pin_checkpoint_from and checkpoint_from to the Ruby journal ([a42f1da](https://github.com/seuros/journal_machines/commit/a42f1dace271d39d47264133d63bcd3d1101e441))
|
|
71
|
+
|
|
72
|
+
## [0.2.0](https://github.com/seuros/journal_machines/compare/journal-machines-v0.1.0...journal-machines-v0.2.0) (2026-11-12)
|
|
73
|
+
|
|
74
|
+
|
|
75
|
+
### Features
|
|
76
|
+
|
|
77
|
+
* initial extraction ([1e4f7a9](https://github.com/seuros/journal_machines/commit/1e4f7a9c3344bd27293fc3f51ee66e22d266e183))
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
[package]
|
|
2
|
+
name = "journal-machines"
|
|
3
|
+
version = "0.8.1"
|
|
4
|
+
edition.workspace = true
|
|
5
|
+
rust-version.workspace = true
|
|
6
|
+
license.workspace = true
|
|
7
|
+
description = "Bounded log segment metadata: roll decisions, compaction candidates, and pin-gated reclamation"
|
|
8
|
+
readme = "README.md"
|
|
9
|
+
exclude = ["src/**/tests.rs", "src/tests/", "tests/"]
|
|
10
|
+
keywords = ["journal", "segment", "compaction", "retention", "no-std"]
|
|
11
|
+
categories = ["no-std", "algorithms"]
|
|
12
|
+
[dependencies]
|
|
13
|
+
offset-machines.workspace = true
|
|
14
|
+
periscope-machines.workspace = true
|
|
15
|
+
[dev-dependencies]
|
|
16
|
+
matryona-machines = { workspace = true, features = ["testing"] }
|
|
17
|
+
[lints]
|
|
18
|
+
workspace = true
|