journal_machines 0.7.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,261 @@
1
+ # journal-machines
2
+
3
+ > **AMS Rosenblum**, NCC-4228, *Rosenblum*-class archive cutter.
4
+
5
+ Engine-room flight manual: know which log segments exist, and which may go.
6
+
7
+ Pre-1.0. Allocation-free `no_std` segment metadata policy for a
8
+ segmented log: when the active segment should roll, which sealed segments are
9
+ worth compacting, and when an original may be deleted.
10
+
11
+ This is not a log writer, a byte compactor or a Raft log. The journal never
12
+ touches a file. The caller writes, seals, compacts, fsyncs and deletes; the
13
+ journal holds the bookkeeping and refuses the unsafe order. `Journal<SEGMENTS,
14
+ PINS>` has compile-time capacities and refuses rather than grows.
15
+
16
+ Every call that can refuse returns `Result<T, Refusal>`. `T` says only what
17
+ was honoured; `Refusal` is the one enum for "this call was not honoured", and
18
+ a refused call leaves the journal exactly as it was. An idempotent repeat is
19
+ honoured, not refused: when what the caller asked for already holds, the call
20
+ answers `Ok` with a variant that says so, and still changes nothing. Only a
21
+ repeat the journal can prove is honoured: it must see, in its own state, that
22
+ the same call with the same arguments was already applied.
23
+ No public enum is `#[non_exhaustive]`: a new variant is a breaking change every
24
+ caller matches on.
25
+
26
+ Time is `core::time::Duration` on a monotonic clock whose epoch the caller
27
+ picks: `now`, a segment's `opened_at` and the policy's `max_age`. Offsets,
28
+ record and byte counts, ids, revisions and generations are `u64`.
29
+
30
+ - `Rolled::new(sealed, opened, next_offset)`, `Reclaimed::new(segment,
31
+ revision, bytes, replaced_by)` and `Segment::new(id, revision, base_offset,
32
+ records, stored_records, live_records, bytes, opened_at, state,
33
+ retention, replacement)` answer `Some` only for a value some journal could have
34
+ answered (issued ids of one scope, revisions and attempts drawn after their
35
+ segment, a segment whose counts, state, pin and replacement agree, storing its
36
+ whole range at revision 0). Their
37
+ fields are private: read them through the accessor of the same name, so no
38
+ value skips the check.
39
+ - `SegmentId`, `Pin` and `Replacement` are values: two handles to the same
40
+ identity are equal and hash alike. They have no order: an id carries its
41
+ journal's scope, so ids of two journals compare by nothing meaningful.
42
+ - `Segment::retention` is `Retention::Pinned` while a pin sits below the
43
+ segment's end, otherwise `Retention::Unpinned` (always, while active).
44
+ - `Policy::new(max_bytes, max_records, max_age, min_dead_per_mille)` takes
45
+ `Option<NonZeroU64>` sizes and an `Option<Duration>` age; `None` disables a
46
+ threshold. It answers `None` for a per-mille above 1000 or a zero age.
47
+ - `Journal::new(policy, base_offset, now)` opens the first segment at `now`,
48
+ in scope zero. `Journal::with_scope(policy, base_offset, now, scope)` does
49
+ the same under `scope`; `scope()` reads it back. See Scopes below.
50
+ - `Journal::resumed(policy, base_offset, now, scope, issued) -> Option<Journal>`
51
+ does the same after a restart, continuing the identity sequence after the
52
+ `issued()` count saved from the journal before it. See Restarts below.
53
+ - `roll_decision(bytes, records, now) -> Result<Roll, Refusal>` reads the
54
+ caller's observation of the active segment and answers `Keep` or
55
+ `Due(Size | Records | Age)` against the `Policy`. An empty segment is never
56
+ due. Refuses `NoActive`, `TimeRegression`, `Overflow`.
57
+ - `roll(bytes, records, now) -> Result<Rolled, Refusal>` records that the
58
+ caller sealed the active segment and opened the next. Refuses `NoActive`,
59
+ `Empty`, `TimeRegression`, `Overflow`, `SegmentsFull`, `Exhausted`.
60
+ - `observe_live(id, live) -> Result<(), Refusal>` records how many stored
61
+ records are still live. Refuses `Unknown`, `Active`, `Exceeds`.
62
+ `compaction_candidates(after)` iterates sealed segments whose dead share
63
+ meets `min_dead_per_mille`, oldest first; resume by passing the last id.
64
+ - `pin_reader(offset)` and `pin_checkpoint(assignment, offset)`
65
+ `-> Result<Pin, Refusal>` retain every offset at or after the pin. Refuse
66
+ `PinsFull`, `Exhausted`, and for a checkpoint `Unassigned`.
67
+ - `advance(pin, offset) -> Result<Advanced, Refusal>` moves a reader pin
68
+ forward and answers `Moved`, or `Unchanged` when the pin already sits
69
+ there. Refuses `Stale`, `WrongKind`, `Regressed`.
70
+ - `checkpoint(pin, assignment, offset) -> Result<Checkpointed, Refusal>`
71
+ answers `Advanced`, `Reassigned` when a newer assignment generation takes
72
+ over, or `Unchanged` when the same generation reports the offset the pin
73
+ already sits at. Refuses `Stale`, `WrongKind`, `Unassigned`,
74
+ `StaleAssignment`, `Regressed`: an older assignment cannot move the pin.
75
+ - `release(pin) -> Result<(), Refusal>` is checked against the acquisition
76
+ generation. Refuses `Stale` for a pin that is not held: released already,
77
+ or never issued here. Whoever holds the slot now keeps it.
78
+ - `propose(id) -> Result<Replacement, Refusal>` opens a replacement attempt.
79
+ Refuses `Unknown`, `Active`, `Busy`, `Exhausted`.
80
+ - `confirm_durable(handle, Rewritten { .. } | Superseded)`
81
+ `-> Result<Confirmed, Refusal>` records the caller's word that the
82
+ replacement is durable and answers `Recorded`, or `AlreadyDurable` when the
83
+ attempt was confirmed before with that same `Durable`. Refuses `Stale`,
84
+ `Exceeds`, and `Mismatch` when the attempt is already durable with a
85
+ different `Durable` (the other variant, or other `bytes` or `live_records`):
86
+ the first confirmation stands.
87
+ - `abort(handle) -> Result<(), Refusal>` gives up and keeps the original.
88
+ Refuses `Stale`.
89
+ - `reclaim(handle) -> Result<Reclaimed, Refusal>` succeeds only when the
90
+ replacement is durable **and** no pin of either kind sits below the
91
+ segment's end. Refuses `Stale`, `NotDurable`, `Pinned { offset }`. A
92
+ rewritten segment keeps its slot under a new revision; a superseded one
93
+ leaves the journal.
94
+
95
+ ## Contract
96
+
97
+ ### Scopes
98
+
99
+ Segment ids, `Pin` and `Replacement` handles are only meaningful to the
100
+ journal that issued them. Each is built on an integer (a segment id; a slot
101
+ and a pin generation; a segment id and an attempt number) drawn from one
102
+ sequence the journal counts from one, so two journals issue the same ones,
103
+ and a handle given to the wrong journal names a live pin or attempt there. Give each journal its own scope with
104
+ `Journal::with_scope`: the scope rides in the top 16 bits of every segment
105
+ id, pin generation and attempt number, in the
106
+ [`periscope-machines`](https://crates.io/crates/periscope-machines) layout,
107
+ and `SegmentId::scope()`, `Pin::scope()` and `Replacement::scope()` read it.
108
+ `Journal::new` is scope zero and issues the plain numbers 1, 2, 3.
109
+
110
+ One sequence serves all three kinds: the active segment takes 1, and every
111
+ roll, pin and proposal takes the next number. No two handles of a journal
112
+ share a raw value, so a pin generation or an attempt (which is also a
113
+ rewritten segment's `revision`) passed to `SegmentId::from_raw` names no
114
+ segment. Segment ids still ascend in offset order; they are not consecutive.
115
+
116
+ An identity this journal never issued, of another scope or beyond what it
117
+ has issued, names nothing here, and is refused before anything is looked up,
118
+ with nothing changed: a pin or a replacement is
119
+ `Stale` (`None` from `pin_offset`), a segment id is `Unknown` (`None` from
120
+ `segment`, an empty iterator as a resume position). A `Replacement` is
121
+ refused if either its attempt or its segment was never issued here. The
122
+ journal reads no clock and no random source, so it cannot pick a scope; two
123
+ journals left in the same scope still alias. A scope holds 2^48 - 1
124
+ identities, all kinds together. A rewritten segment's `revision` is the
125
+ attempt that rewrote it, so it carries the scope too.
126
+
127
+ ### Restarts
128
+
129
+ A journal started again with `new` or `with_scope` counts from one again, so
130
+ a handle that outlived the restart could name a new entry. Either resume the
131
+ sequence, `Journal::resumed(policy, base_offset, now, scope, issued)` with
132
+ the `issued()` saved from the journal before it, or give each incarnation a
133
+ scope of its own. `resumed` restores the sequence and nothing else: segments
134
+ and pins are replayed by the caller, and the one active segment takes the
135
+ next number. It is `None` when `issued` is more than a scope holds or leaves
136
+ no identity for the active segment. Resuming from a count lower than what was
137
+ really issued reissues: save `issued()` before handing an identity out.
138
+
139
+ ### Repeats
140
+
141
+ `Advanced::Unchanged`, `Checkpointed::Unchanged` and
142
+ `Confirmed::AlreadyDurable` are answered from a live pin or a live attempt,
143
+ which only a handle this journal issued reaches. No repeat is honoured on a
144
+ handle that is no longer live. A second `release` is `Stale`: pin
145
+ generations share the sequence with segment ids and attempts, so "issued
146
+ here and held in no slot" is as true of a segment id as of a released pin,
147
+ and the journal keeps no record of which numbers were pins. A second
148
+ `reclaim` or `abort` is `Stale` too: an attempt that is no longer current was
149
+ aborted, reclaimed as a rewrite or reclaimed as superseded, and the journal
150
+ does not keep which.
151
+
152
+ `Stale` always means a handle that is not current, pin or replacement, or one
153
+ this journal never issued.
154
+ `Mismatch` always means a repeat that carries something other than what was
155
+ recorded.
156
+ `Regressed` stays a refusal: a pin asked to move back is asking for retention
157
+ the journal can no longer promise.
158
+ `Exhausted` always means the journal's identity sequence is spent: 2^48 - 1
159
+ segment ids, pin generations and attempts together per scope. `Overflow` is
160
+ only the offset range.
161
+ `active`, `segment`, `covering`, the iterators, `pin_offset`, `low_pin` and
162
+ `pins` read only and never refuse.
163
+
164
+ ### Resume positions
165
+
166
+ `segments`, `compaction_candidates` and `reclaimable` take `after:
167
+ Option<SegmentId>`, a position rather than a handle: nothing is looked up by
168
+ it, and the segment it named may have left the journal since. It must still
169
+ be a number this journal issued (`Issuer::owns`): one of another scope, a
170
+ zero sequence or one not reached is no position, and the iterator is empty.
171
+ That is the same line every other call draws for identities, and it keeps a
172
+ bare `0` (a common stand-in for "from the start") from silently meaning
173
+ "everything". `None` is the one way to begin. Any issued number works, a pin
174
+ generation or an attempt included, because one sequence orders them all.
175
+
176
+ ### Which answers record the clock
177
+
178
+ The journal keeps no watermark. The only instants it holds are its segments'
179
+ opening times, and a regression is judged against the active segment's.
180
+ `new`, `with_scope`, `resumed` and an honoured `roll` record `now` as the
181
+ opening of the new active segment: their answer depends on the instant.
182
+ `roll_decision` reads only and records nothing; its answer is about the
183
+ instant supplied. No other call takes a clock, and no honoured repeat
184
+ (`Unchanged`, `AlreadyDurable`) records anything.
185
+
186
+ ### Unreachable arms
187
+
188
+ There are none, and no slot is indexed by a number that could be out of
189
+ range. A lookup by segment id or pin hands back the slot itself, so nothing
190
+ is looked up twice; `roll` takes the active slot and the one after it as one
191
+ pair, whose absence is `SegmentsFull`; a superseded segment leaves by a walk
192
+ over the tracked slots; and the issuer's `Exhausted` is the journal's own
193
+ `Exhausted` (`From<periscope_machines::Refusal>`).
194
+
195
+ Checkpoint pins compose with `offset-machines`: `pin_checkpoint_from(&tracker)`
196
+ and `checkpoint_from(pin, &tracker)` pin at the tracker's confirmed-committed
197
+ base under its assignment generation, never at merely acknowledged offsets. An
198
+ unassigned tracker is refused as `Unassigned`, an older assignment as
199
+ `StaleAssignment`; a tracker whose base has not moved is `Unchanged`. The Ruby gem offers the same two calls over any object
200
+ with `generation` and `base` readers, such as `OffsetMachines::Tracker`.
201
+
202
+ Durability is the caller's claim; the journal cannot check it. Filesystem
203
+ operations, crash recovery and rebuilding this metadata after a restart stay
204
+ outside. Persist what you need and replay it into a resumed journal.
205
+
206
+ ```rust
207
+ use core::{num::NonZeroU64, time::Duration};
208
+ use journal_machines::{
209
+ Advanced, Checkpointed, Confirmed, Durable, Journal, Policy, Refusal, Roll, RollReason,
210
+ };
211
+ # fn main() -> Result<(), Box<dyn core::error::Error>> {
212
+
213
+ let hour = Duration::from_secs(3600);
214
+ let policy = Policy::new(NonZeroU64::new(1024), None, Some(hour), 500).ok_or("policy")?;
215
+ let mut journal = Journal::<8, 4>::new(policy, 0, Duration::ZERO);
216
+ let now = Duration::from_secs(5);
217
+ assert_eq!(journal.roll_decision(2048, 10, now), Ok(Roll::Due(RollReason::Size)));
218
+ assert_eq!(journal.roll_decision(1, 1, hour), Ok(Roll::Due(RollReason::Age)));
219
+ let rolled = journal.roll(2048, 10, now)?;
220
+
221
+ let checkpoint = journal.pin_checkpoint(1, 0)?;
222
+ let reader = journal.pin_reader(0)?;
223
+ assert_eq!(journal.observe_live(rolled.sealed(), 2), Ok(()));
224
+ assert_eq!(journal.compaction_candidates(None).next().map(|s| s.id()), Some(rolled.sealed()));
225
+
226
+ let replacement = journal.propose(rolled.sealed())?;
227
+ assert_eq!(journal.reclaim(replacement), Err(Refusal::NotDurable));
228
+ let rewritten = Durable::Rewritten { bytes: 400, live_records: 2 };
229
+ assert_eq!(journal.confirm_durable(replacement, rewritten), Ok(Confirmed::Recorded));
230
+ assert_eq!(journal.confirm_durable(replacement, rewritten), Ok(Confirmed::AlreadyDurable));
231
+ assert_eq!(journal.confirm_durable(replacement, Durable::Superseded), Err(Refusal::Mismatch));
232
+ assert_eq!(journal.reclaim(replacement), Err(Refusal::Pinned { offset: 0 }));
233
+
234
+ assert_eq!(journal.advance(reader, 10), Ok(Advanced::Moved));
235
+ assert_eq!(journal.advance(reader, 10), Ok(Advanced::Unchanged));
236
+ assert_eq!(journal.release(reader), Ok(()));
237
+ assert_eq!(journal.release(reader), Err(Refusal::Stale));
238
+ assert_eq!(journal.advance(reader, 11), Err(Refusal::Stale));
239
+
240
+ // One sequence for every kind: no two handles share a raw value.
241
+ assert_eq!((rolled.sealed().raw(), rolled.opened().raw()), (1, 2));
242
+ assert_eq!((checkpoint.generation(), reader.generation()), (3, 4));
243
+ assert_eq!((replacement.attempt(), journal.issued()), (5, 5));
244
+
245
+ // A journal of another scope issues the same sequence numbers under its own
246
+ // scope, and neither takes the other's handles.
247
+ let mut other = Journal::<8, 4>::with_scope(policy, 0, Duration::ZERO, 7);
248
+ let theirs = other.pin_checkpoint(1, 0)?;
249
+ assert_eq!((checkpoint.scope(), theirs.scope()), (0, 7));
250
+ assert_eq!(journal.release(theirs), Err(Refusal::Stale));
251
+ assert_eq!(other.checkpoint(checkpoint, 1, 10), Err(Refusal::Stale));
252
+ assert_eq!(journal.checkpoint(checkpoint, 1, 10), Ok(Checkpointed::Advanced));
253
+ assert_eq!(journal.checkpoint(checkpoint, 1, 10), Ok(Checkpointed::Unchanged));
254
+ assert_eq!(journal.reclaim(replacement)?.bytes(), 2048);
255
+
256
+ // After a restart, continue the sequence from the saved count.
257
+ let resumed = Journal::<8, 4>::resumed(policy, 20, now, 0, journal.issued()).ok_or("resume")?;
258
+ assert_eq!(resumed.active().map(|segment| segment.id().raw()), Some(6));
259
+ # Ok(())
260
+ # }
261
+ ```
@@ -0,0 +1,161 @@
1
+ //! Validating constructors for the public-field answers: a value exists only
2
+ //! when some journal could have answered it.
3
+ use core::time::Duration;
4
+
5
+ use periscope_machines::{scope_of, sequence_of};
6
+
7
+ use crate::{Reclaimed, Replacement, Retention, Rolled, Segment, SegmentId, SegmentState};
8
+
9
+ /// Whether `raw` is a number a journal issues: its sequence is not zero.
10
+ const fn issued(raw: u64) -> bool {
11
+ sequence_of(raw) != 0
12
+ }
13
+
14
+ /// Whether `raw` is an identity drawn after segment `id` from the same
15
+ /// sequence: same scope, later number. Attempts, and so revisions, are.
16
+ const fn drawn_after(id: SegmentId, raw: u64) -> bool {
17
+ scope_of(raw) == id.scope() && sequence_of(raw) > sequence_of(id.raw())
18
+ }
19
+
20
+ /// Whether `revision` is one a segment `id` can carry: zero for the
21
+ /// original, otherwise an attempt drawn after it.
22
+ const fn revision_of(id: SegmentId, revision: u64) -> bool {
23
+ revision == 0 || drawn_after(id, revision)
24
+ }
25
+
26
+ /// Whether `attempt` can be a later attempt on segment `id` than the one that
27
+ /// wrote `revision`.
28
+ const fn attempt_on(id: SegmentId, revision: u64, attempt: u64) -> bool {
29
+ drawn_after(id, attempt) && attempt > revision
30
+ }
31
+
32
+ impl Rolled {
33
+ /// A roll some journal could have recorded, or `None`: both ids issued
34
+ /// (a non-zero sequence), of one scope, the opened one drawn after the
35
+ /// sealed one, and a next offset past at least one sealed record.
36
+ #[must_use]
37
+ pub const fn new(sealed: SegmentId, opened: SegmentId, next_offset: u64) -> Option<Self> {
38
+ if issued(sealed.raw()) && drawn_after(sealed, opened.raw()) && next_offset > 0 {
39
+ Some(Self {
40
+ sealed,
41
+ opened,
42
+ next_offset,
43
+ })
44
+ } else {
45
+ None
46
+ }
47
+ }
48
+ }
49
+
50
+ impl Reclaimed {
51
+ /// A reclaim some journal could have answered, or `None`: an issued
52
+ /// segment id; a `revision` that is zero or an attempt drawn after it;
53
+ /// and a `replaced_by`, when present, that is a later attempt still.
54
+ #[must_use]
55
+ pub const fn new(
56
+ segment: SegmentId,
57
+ revision: u64,
58
+ bytes: u64,
59
+ replaced_by: Option<u64>,
60
+ ) -> Option<Self> {
61
+ let replaced = match replaced_by {
62
+ Some(attempt) => attempt_on(segment, revision, attempt),
63
+ None => true,
64
+ };
65
+ if issued(segment.raw()) && revision_of(segment, revision) && replaced {
66
+ Some(Self {
67
+ segment,
68
+ revision,
69
+ bytes,
70
+ replaced_by,
71
+ })
72
+ } else {
73
+ None
74
+ }
75
+ }
76
+ }
77
+
78
+ impl Segment {
79
+ /// A view some journal could have answered, fields in reader order, or
80
+ /// `None`.
81
+ ///
82
+ /// The id is issued and the revision is zero or an attempt drawn after
83
+ /// it. An active segment holds nothing yet: no revision, records, stored
84
+ /// or live records, bytes, pin (it is [`Retention::Unpinned`]) or replacement. A closed one holds at least
85
+ /// one record within `u64` offsets, stores no more than its range (all of
86
+ /// it at revision 0: only a rewrite, which assigns a revision, stores
87
+ /// fewer) and has no more live records than it stores. `Sealed` has no replacement; the
88
+ /// other three have one on this segment, drawn after its revision.
89
+ /// `Durable` is pinned, `Reclaimable` is not.
90
+ // One argument per field, in the order the readers and Ruby name them.
91
+ #[expect(clippy::too_many_arguments)]
92
+ #[must_use]
93
+ pub const fn new(
94
+ id: SegmentId,
95
+ revision: u64,
96
+ base_offset: u64,
97
+ records: u64,
98
+ stored_records: u64,
99
+ live_records: u64,
100
+ bytes: u64,
101
+ opened_at: Duration,
102
+ state: SegmentState,
103
+ retention: Retention,
104
+ replacement: Option<Replacement>,
105
+ ) -> Option<Self> {
106
+ Self {
107
+ id,
108
+ revision,
109
+ base_offset,
110
+ records,
111
+ stored_records,
112
+ live_records,
113
+ bytes,
114
+ opened_at,
115
+ state,
116
+ retention,
117
+ replacement,
118
+ }
119
+ .checked()
120
+ }
121
+
122
+ /// This view, when it upholds what [`Segment::new`] checks.
123
+ pub(crate) const fn checked(self) -> Option<Self> {
124
+ let attempt = match self.replacement {
125
+ Some(replacement) => {
126
+ replacement.segment.raw() == self.id.raw()
127
+ && attempt_on(self.id, self.revision, replacement.attempt)
128
+ }
129
+ None => true,
130
+ };
131
+ let replaced = self.replacement.is_some();
132
+ let pinned = matches!(self.retention, Retention::Pinned);
133
+ // Only a rewrite stores fewer records than the range, and a rewrite
134
+ // assigns its attempt as the revision: revision 0 stores them all.
135
+ let closed = self.records > 0
136
+ && self.base_offset.checked_add(self.records).is_some()
137
+ && self.stored_records <= self.records
138
+ && (self.revision != 0 || self.stored_records == self.records)
139
+ && self.live_records <= self.stored_records;
140
+ let state = match self.state {
141
+ SegmentState::Active => {
142
+ self.revision == 0
143
+ && self.records == 0
144
+ && self.stored_records == 0
145
+ && self.live_records == 0
146
+ && self.bytes == 0
147
+ && !pinned
148
+ && !replaced
149
+ }
150
+ SegmentState::Sealed => closed && !replaced,
151
+ SegmentState::Proposed => closed && replaced,
152
+ SegmentState::Durable => closed && replaced && pinned,
153
+ SegmentState::Reclaimable => closed && replaced && !pinned,
154
+ };
155
+ if issued(self.id.raw()) && revision_of(self.id, self.revision) && attempt && state {
156
+ Some(self)
157
+ } else {
158
+ None
159
+ }
160
+ }
161
+ }