merkle_machines 0.6.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +91 -0
- data/Cargo.lock +314 -0
- data/Cargo.toml +28 -0
- data/LICENSE.txt +21 -0
- data/README.md +326 -0
- data/ext/merkle_machines_native/core/CHANGELOG.md +66 -0
- data/ext/merkle_machines_native/core/Cargo.toml +20 -0
- data/ext/merkle_machines_native/core/README.md +114 -0
- data/ext/merkle_machines_native/core/src/lib.rs +16 -0
- data/ext/merkle_machines_native/core/src/refusal.rs +54 -0
- data/ext/merkle_machines_native/core/src/request.rs +60 -0
- data/ext/merkle_machines_native/core/src/schema.rs +193 -0
- data/ext/merkle_machines_native/core/src/survey/buffer.rs +61 -0
- data/ext/merkle_machines_native/core/src/survey.rs +552 -0
- data/ext/merkle_machines_native/extconf.rb +2 -0
- data/ext/merkle_machines_native/ffi/Cargo.toml +21 -0
- data/ext/merkle_machines_native/ffi/extconf.rb +14 -0
- data/ext/merkle_machines_native/ffi/src/init.rs +50 -0
- data/ext/merkle_machines_native/ffi/src/lib.rs +453 -0
- data/ext/merkle_machines_native/ffi/src/values.rs +47 -0
- data/lib/merkle_machines/survey.rb +43 -0
- data/lib/merkle_machines/values.rb +63 -0
- data/lib/merkle_machines/version.rb +5 -0
- data/lib/merkle_machines.rb +8 -0
- data/sig/merkle_machines.rbs +142 -0
- metadata +92 -0
data/README.md
ADDED
|
@@ -0,0 +1,326 @@
|
|
|
1
|
+
# MerkleMachines
|
|
2
|
+
|
|
3
|
+
> Two replicas disagree. Find out where before you ship the whole dataset. ðŸ§
|
|
4
|
+
|
|
5
|
+
A divergence survey for Ruby. You have two digest trees of the same shape, one per replica, and their roots differ. The survey tells you which nodes to fetch child digests for, compares what you bring back, and hands you the ranges of leaves that actually differ. It holds a fixed number of nodes and a fixed number of ranges, so a badly diverged pair slows the survey down instead of growing it.
|
|
6
|
+
|
|
7
|
+
## Quick Start
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
gem 'merkle_machines'
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
```ruby
|
|
14
|
+
schema = MerkleMachines::Schema.new(fanout: 16, depth: 3, partitioning: 1, digest: 1)
|
|
15
|
+
versions = MerkleMachines::Versions.new(local: mine.version, remote: theirs.version)
|
|
16
|
+
# One scope per survey whose requests could meet: two Survey.new are both
|
|
17
|
+
# scope zero, open the same sessions, and honour each other's requests.
|
|
18
|
+
survey = MerkleMachines::Survey.with_scope(1)
|
|
19
|
+
|
|
20
|
+
case survey.begin(schema, theirs.schema, versions, mine.root, theirs.root)
|
|
21
|
+
when :identical then return
|
|
22
|
+
when :diverged then nil
|
|
23
|
+
else raise 'trees are not comparable'
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
until survey.phase == :complete
|
|
27
|
+
if (request = survey.next_request)
|
|
28
|
+
survey.answer(request, versions,
|
|
29
|
+
mine.children(request.node.level, request.node.index),
|
|
30
|
+
theirs.children(request.node.level, request.node.index))
|
|
31
|
+
end
|
|
32
|
+
survey.drain(MerkleMachines::OUTPUT_CAPACITY).each { |leaves| repair(leaves) }
|
|
33
|
+
end
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
One survey, a handful of methods. Digests are Strings you computed. Everything else is an Integer.
|
|
37
|
+
|
|
38
|
+
## A Message to the Resistance
|
|
39
|
+
|
|
40
|
+
So your two replicas drifted. A write landed on one during a partition, or a repair job died halfway, and now the row counts match and the checksums don't.
|
|
41
|
+
|
|
42
|
+
You know the textbook answer. "Use a Merkle tree." Hash the leaves, hash the hashes, compare the roots, walk down where they differ. Then you write the walk. It recurses, so a pair of replicas that disagree everywhere allocates a frontier the size of the tree. It talks to the other node from inside the recursion, so a dropped connection loses your place. Someone compacts the remote tree halfway through and you spend an afternoon repairing leaves that were never different. And one day a replica is rebuilt with sixteen children per node instead of eight, the walk compares child 3 with child 3 anyway, and every leaf looks wrong.
|
|
43
|
+
|
|
44
|
+
The walk is a small state machine: which nodes still need their children fetched, which requests are out, which leaf ranges are ready. With a refusal for trees that can't be compared, and a refusal for a tree that moved. That machine is this gem.
|
|
45
|
+
|
|
46
|
+
## Chapter 1: Stardate 2026.316, the AMS Merkle
|
|
47
|
+
|
|
48
|
+
> **AMS Merkle**, NCC-4224, *Merkle*-class survey cutter.
|
|
49
|
+
> Two charts of the same sector. Mark where they disagree, and nothing else.
|
|
50
|
+
|
|
51
|
+
The cutter doesn't draw charts and doesn't correct them. Two ships surveyed the same sector and came back with two charts. The cutter lays them side by side, quadrant by quadrant, and asks for a closer look only where the summaries differ. It carries a small plotting table, so it asks for a few quadrants at a time and files its findings as it goes.
|
|
52
|
+
|
|
53
|
+
On a bridge somewhere, a cadet volunteers:
|
|
54
|
+
|
|
55
|
+
```ruby
|
|
56
|
+
def diff(node)
|
|
57
|
+
return [node.leaf] if node.leaf?
|
|
58
|
+
node.children.zip(remote.children(node)).flat_map { |a, b| a.digest == b.digest ? [] : diff(a) }
|
|
59
|
+
end
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
"That," says the cutter's navigator, "is a network call inside a recursion inside a hope that nobody edits the other chart."
|
|
63
|
+
|
|
64
|
+
## The Flight Plan
|
|
65
|
+
|
|
66
|
+
A survey moves through four phases:
|
|
67
|
+
|
|
68
|
+
```mermaid
|
|
69
|
+
stateDiagram-v2
|
|
70
|
+
[*] --> idle
|
|
71
|
+
idle --> complete: begin, roots equal
|
|
72
|
+
idle --> surveying: begin, roots differ
|
|
73
|
+
surveying --> surveying: next_request / answer / requeue
|
|
74
|
+
surveying --> draining: last node resolved
|
|
75
|
+
draining --> complete: drain empties the buffer
|
|
76
|
+
surveying --> complete: last node resolved, buffer already empty
|
|
77
|
+
complete --> surveying: begin again
|
|
78
|
+
surveying --> idle: abort
|
|
79
|
+
draining --> idle: abort
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
And one exchange, over a tree of four leaves where only leaf 2 differs:
|
|
83
|
+
|
|
84
|
+
```mermaid
|
|
85
|
+
sequenceDiagram
|
|
86
|
+
participant A as Your code
|
|
87
|
+
participant S as Survey
|
|
88
|
+
participant R as Remote replica
|
|
89
|
+
|
|
90
|
+
A->>S: begin(schema, schema, versions, root, root)
|
|
91
|
+
S-->>A: :diverged
|
|
92
|
+
A->>S: next_request
|
|
93
|
+
S-->>A: Request(level 0, index 0)
|
|
94
|
+
A->>R: children of the root, at version 57
|
|
95
|
+
R-->>A: two digests
|
|
96
|
+
A->>S: answer(request, versions, mine, theirs)
|
|
97
|
+
S-->>A: :accepted
|
|
98
|
+
A->>S: next_request
|
|
99
|
+
S-->>A: Request(level 1, index 1)
|
|
100
|
+
A->>R: children of node 1, at version 57
|
|
101
|
+
R-->>A: two digests
|
|
102
|
+
A->>S: answer(request, versions, mine, theirs)
|
|
103
|
+
S-->>A: :accepted
|
|
104
|
+
A->>S: drain(8)
|
|
105
|
+
S-->>A: [2...3]
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
## How the Survey Works
|
|
109
|
+
|
|
110
|
+
### Schema
|
|
111
|
+
|
|
112
|
+
```ruby
|
|
113
|
+
schema = MerkleMachines::Schema.new(fanout: 2, depth: 2, partitioning: 1, digest: 1)
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
`fanout` is children per node, from 2 to 64. `depth` is the number of levels below the root, so this tree has four leaves. `partitioning` and `digest` are numbers you choose to name how keys map to leaves and how digests are computed. The survey never interprets them. It only checks that both sides say the same thing.
|
|
117
|
+
|
|
118
|
+
A shape that is not a tree is not a `Schema`. A fanout outside 2..64, or more leaves than fit in 64 bits, raises `RangeError` from `Schema.new`:
|
|
119
|
+
|
|
120
|
+
```ruby
|
|
121
|
+
MerkleMachines::Schema.new(fanout: 1, depth: 2, partitioning: 1, digest: 1) # RangeError
|
|
122
|
+
MerkleMachines::Schema.new(fanout: 2, depth: 64, partitioning: 1, digest: 1) # RangeError
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
### Begin
|
|
126
|
+
|
|
127
|
+
```ruby
|
|
128
|
+
versions = MerkleMachines::Versions.new(local: 40, remote: 57)
|
|
129
|
+
survey = MerkleMachines::Survey.new
|
|
130
|
+
|
|
131
|
+
survey.begin(schema, schema, versions, 'root-a', 'root-b') # => :diverged
|
|
132
|
+
survey.phase # => :surveying
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
A schema answers the questions the core's does:
|
|
136
|
+
|
|
137
|
+
```ruby
|
|
138
|
+
schema.leaves # => 4
|
|
139
|
+
schema.width(1) # => 2 nodes at level 1
|
|
140
|
+
schema.span(MerkleMachines::Node.new(1, 1)) # => 2...4, the leaves below that node
|
|
141
|
+
schema.span(MerkleMachines::Node.new(3, 0)) # => nil, not a node of this tree
|
|
142
|
+
schema.span(MerkleMachines::Node::ROOT) # => 0...4
|
|
143
|
+
schema.compatible(schema.with(fanout: 4)) # => :fanout_mismatch
|
|
144
|
+
schema.compatible(schema) # => :compatible
|
|
145
|
+
survey.schema # => the open session's Schema, nil when idle
|
|
146
|
+
survey.versions # => the Versions it pinned, nil when idle
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
Equal roots answer `:identical` and the survey is `:complete` on the spot. Trees that can't be compared are refused by name, and no session opens:
|
|
150
|
+
|
|
151
|
+
```ruby
|
|
152
|
+
other = schema.with(fanout: 4)
|
|
153
|
+
MerkleMachines::Survey.new.begin(schema, other, versions, 'a', 'b') # => :fanout_mismatch
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
The others are `:depth_mismatch`, `:partitioning_mismatch` and `:digest_mismatch`. A survey that is still working answers `:busy`.
|
|
157
|
+
|
|
158
|
+
### Request
|
|
159
|
+
|
|
160
|
+
```ruby
|
|
161
|
+
request = survey.next_request
|
|
162
|
+
# => #<data MerkleMachines::Request session=1, node=#<data MerkleMachines::Node level=0, index=0>>
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
A request names one node: `request.node` is a `Node` with a `level` (0 is the root) and an `index` within that level. Fetch that node's child digests from both trees, in child order. `nil` means nothing is queued right now: answers are still out, the range buffer is full, or the survey is finished. Call it as many times as you want requests in the air.
|
|
166
|
+
|
|
167
|
+
### Answer
|
|
168
|
+
|
|
169
|
+
```ruby
|
|
170
|
+
survey.answer(request, versions, %w[left same], %w[left changed]) # => :accepted
|
|
171
|
+
|
|
172
|
+
request = survey.next_request
|
|
173
|
+
# => #<data MerkleMachines::Request session=1, node=#<data MerkleMachines::Node level=1, index=1>>
|
|
174
|
+
survey.answer(request, versions, %w[x y], %w[z y]) # => :accepted
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
The survey compares the two lists position by position and queues the children that differ. Children that match are never visited again. An answer it can't use is refused and changes nothing:
|
|
178
|
+
|
|
179
|
+
```ruby
|
|
180
|
+
survey = MerkleMachines::Survey.new
|
|
181
|
+
survey.begin(schema, schema, versions, 'root-a', 'root-b')
|
|
182
|
+
request = survey.next_request
|
|
183
|
+
|
|
184
|
+
survey.answer(request, versions.with(remote: 58), %w[a b], %w[a c]) # => :version_changed
|
|
185
|
+
survey.answer(request, versions, %w[a], %w[a c]) # => :arity
|
|
186
|
+
survey.answer(request, versions, %w[a b], %w[a b]) # => :inconsistent
|
|
187
|
+
elsewhere = request.with(node: request.node.with(level: 1))
|
|
188
|
+
survey.answer(elsewhere, versions, %w[a b], %w[a c]) # => :unexpected
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
`:inconsistent` means the parents differed and no child does, so the digests didn't come from one tree. `:stale` is the answer to a request from a session that was aborted or replaced, or from a survey in another scope.
|
|
192
|
+
|
|
193
|
+
### Drain
|
|
194
|
+
|
|
195
|
+
```ruby
|
|
196
|
+
survey.answer(request, versions, %w[a b], %w[a c]) # => :accepted
|
|
197
|
+
survey.answer(survey.next_request, versions, %w[x y], %w[z y])
|
|
198
|
+
|
|
199
|
+
survey.phase # => :draining
|
|
200
|
+
survey.drain(8) # => [2...3]
|
|
201
|
+
survey.phase # => :complete
|
|
202
|
+
survey.pop_range # => nil, nothing buffered
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
Ranges are leaf indices, end excluded. `drain(limit)` hands back at most `limit` of them, sorted, with neighbours joined, and keeps the rest buffered; `MerkleMachines::OUTPUT_CAPACITY` takes everything there can be. `pop_range` takes just the lowest one, or `nil`. Two calls can return ranges that touch each other, because the second one was found after you took the first.
|
|
206
|
+
|
|
207
|
+
### Requeue
|
|
208
|
+
|
|
209
|
+
A request you sent and never heard back about goes back in the queue:
|
|
210
|
+
|
|
211
|
+
```ruby
|
|
212
|
+
survey = MerkleMachines::Survey.new
|
|
213
|
+
survey.begin(schema, schema, versions, 'root-a', 'root-b')
|
|
214
|
+
request = survey.next_request
|
|
215
|
+
|
|
216
|
+
survey.requeue(request) # => :requeued
|
|
217
|
+
survey.requeue(request) # => :unexpected, it is not in flight any more
|
|
218
|
+
survey.next_request == request # => true
|
|
219
|
+
survey.requeue_all # => 1, for when the whole connection dropped
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
`requeue` answers `:stale` for a request from a session that was aborted or replaced, or from a survey in another scope, and `:unexpected` for a node that is not in flight. Neither changes anything.
|
|
223
|
+
|
|
224
|
+
### Abort
|
|
225
|
+
|
|
226
|
+
```ruby
|
|
227
|
+
survey.abort # => :aborted
|
|
228
|
+
survey.phase # => :idle
|
|
229
|
+
survey.abort # => :already_idle, there was no session to drop and nothing changed
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
Abort when a tree moved under you. The buffered ranges are discarded, every outstanding request becomes `:stale`, and the next `begin` pins the new versions.
|
|
233
|
+
|
|
234
|
+
### Scopes
|
|
235
|
+
|
|
236
|
+
A request is only meaningful to the survey that issued it. Run more than one survey (one per peer, say) and each needs its own scope, an Integer in `0..65535` carried by every session it opens and every request of that session:
|
|
237
|
+
|
|
238
|
+
```ruby
|
|
239
|
+
north = MerkleMachines::Survey.with_scope(1)
|
|
240
|
+
south = MerkleMachines::Survey.with_scope(2)
|
|
241
|
+
[north, south].each { |survey| survey.begin(schema, schema, versions, 'mine', 'theirs') }
|
|
242
|
+
|
|
243
|
+
request = north.next_request
|
|
244
|
+
south.next_request
|
|
245
|
+
request.scope # => 1
|
|
246
|
+
request.session # => 281474976710657, scope 1 in the top 16 bits, session 1 below
|
|
247
|
+
south.answer(request, versions, %w[a b], %w[a c]) # => :stale, not one of mine
|
|
248
|
+
south.requeue(request) # => :stale
|
|
249
|
+
north.answer(request, versions, %w[a b], %w[a c]) # => :accepted
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
`Survey.new` is scope zero, where sessions are plain `1, 2, 3`. The survey can't pick a scope for you: it reads no clock and rolls no dice. Leave two surveys in the same scope and they open the same sessions, so an answer meant for one lands in the other.
|
|
253
|
+
|
|
254
|
+
A restarted survey opens session 1 again, so a request that outlived the restart could name a node in flight in a new session. Resume from a saved count with `Survey.resumed(scope, survey.issued)`, or give each incarnation a new scope. `resumed` brings back the session sequence and nothing else: the survey is idle, earlier requests are `:stale`, and an `issued` above 2**48-1 raises `RangeError`.
|
|
255
|
+
|
|
256
|
+
### What each call answers
|
|
257
|
+
|
|
258
|
+
| Call | Honoured | Refused, nothing changed |
|
|
259
|
+
| --- | --- | --- |
|
|
260
|
+
| `begin` | `:identical`, `:diverged` | `:fanout_mismatch`, `:depth_mismatch`, `:partitioning_mismatch`, `:digest_mismatch`, `:busy`, `:undersized`, `:exhausted` |
|
|
261
|
+
| `answer` | `:accepted` | `:stale`, `:unexpected`, `:version_changed`, `:arity`, `:inconsistent` |
|
|
262
|
+
| `requeue` | `:requeued` | `:stale`, `:unexpected` |
|
|
263
|
+
| `requeue_all` | the number requeued | never refuses |
|
|
264
|
+
| `drain(limit)` | an Array of Ranges, empty when none is buffered | never refuses |
|
|
265
|
+
| `pop_range` | a Range, or `nil` when none is buffered | never refuses |
|
|
266
|
+
| `next_request` | a `Request`, or `nil` when nothing is queued | never refuses |
|
|
267
|
+
| `abort` | `:aborted`, or `:already_idle` when no session was open (an idempotent repeat) | never refuses |
|
|
268
|
+
| `schema`, `versions` | the open session's `Schema` / `Versions`, or `nil` when the survey is idle | a query, never refuses |
|
|
269
|
+
| `session` | the latest session id issued in this scope, open or not; `nil` while none was ever begun | a query, never refuses |
|
|
270
|
+
|
|
271
|
+
No answer records a clock: the survey takes no time at all. `Versions` are tree versions the caller pins, not instants, and nothing about them is remembered beyond the open session.
|
|
272
|
+
|
|
273
|
+
`session` after `abort` is the aborted session. A resumed survey is idle until its next `begin` and answers the last session issued before the restart, or `nil` when it was resumed from a count of zero.
|
|
274
|
+
|
|
275
|
+
`:undersized` means the tree is deeper than `PENDING_CAPACITY`, and `:exhausted` means the survey has opened all 2**48-1 sessions of its scope. Every refusal has its own symbol; there is no catch-all. The core's `Refusal::Schema(Mismatch)` answers with the mismatch's symbol, the same four `Schema#compatible` uses.
|
|
276
|
+
|
|
277
|
+
### Capacity
|
|
278
|
+
|
|
279
|
+
```ruby
|
|
280
|
+
MerkleMachines::PENDING_CAPACITY # => 64
|
|
281
|
+
MerkleMachines::OUTPUT_CAPACITY # => 64
|
|
282
|
+
MerkleMachines::MAX_FANOUT # => 64
|
|
283
|
+
|
|
284
|
+
survey.pending # nodes held: queued, in flight, or answered and waiting for room
|
|
285
|
+
survey.in_flight # requests out
|
|
286
|
+
survey.buffered # ranges waiting for drain
|
|
287
|
+
```
|
|
288
|
+
|
|
289
|
+
A survey holds at most 64 nodes and 64 ranges. When the range buffer is full, answered nodes wait and `next_request` returns `nil` until you call `drain` or `pop_range`. Nothing is dropped and no answer is refused for lack of room.
|
|
290
|
+
|
|
291
|
+
## The Rules of the Navigator
|
|
292
|
+
|
|
293
|
+
- **Digests are Strings.** Use binary strings (`.b`). They are compared byte for byte, and two strings with the same bytes in incompatible encodings are different digests. Anything else raises `TypeError`.
|
|
294
|
+
- **Numbers are Integers** in `0..2**64-1`. `fanout`, `depth`, and `level` stop at 255. Floats, nil, and strings raise `TypeError`. Out of range raises `RangeError`.
|
|
295
|
+
- **Schemas, versions, nodes, and requests are value objects.** `Schema`, `Versions`, `Node` and `Request` are frozen `Data` values with the fields of the core types they are named after. An Array in their place raises `TypeError`. They are sealed (subclassing raises `TypeError`, `allocate` is undefined), and since sealing stops accidents, not reflection, the binding is the authority: every value it reads must be an instance of exactly its class, its fields are read straight from the struct and checked again by the core's constructors, so a forged value raises `TypeError` or `RangeError` and never reaches the survey.
|
|
296
|
+
- **Values are valid or do not exist.** Every value is checked by the core when it is built, by you or by the gem: a wrong type raises `TypeError`, a value the core has none for raises `RangeError` (`Schema.new(fanout: 1, ...)`, `Node.new(256, 0)`, `Request.new(0, Node::ROOT)`: sequence zero is no session).
|
|
297
|
+
- **A call that raises changed nothing.** Every check, and every class lookup, runs before the survey moves; the `Request` from `next_request` and the Ranges from `pop_range` and `drain` are built natively from what the core returned, without running any Ruby code (not even `initialize`), so no work is lost to a raise after it.
|
|
298
|
+
- **Scopes.** A `Request` is only meaningful to the survey that issued it. Give each survey its own scope with `Survey.with_scope(scope)`; the core cannot pick one. A `Request` from a survey in another scope is `:stale`. Two surveys in the same scope, `Survey.new` twice included, open the same sessions and alias each other's requests.
|
|
299
|
+
- **Restarts.** Resume from a saved count with `Survey.resumed(scope, issued)`, or give each incarnation a new scope. `resumed` restores the session sequence only; the survey is idle and comparing again is yours.
|
|
300
|
+
- **Refusals don't raise.** Every refusal is a Symbol, and a refused call changes nothing. Only malformed input raises.
|
|
301
|
+
- **One version of each tree per session.** Every answer names the versions its digests were read at. Different versions are refused.
|
|
302
|
+
- **Frozen means frozen.** `freeze` the survey and every method that would change it raises `FrozenError`.
|
|
303
|
+
|
|
304
|
+
## What You Think You're Doing vs Reality
|
|
305
|
+
|
|
306
|
+
### You Think: "Merkle trees find the difference with the fewest messages."
|
|
307
|
+
### Reality: They find it without sending everything.
|
|
308
|
+
|
|
309
|
+
The survey visits the nodes whose digests differ and skips the rest. That is all it promises. It makes no claim about minimal exchange or round count: two replicas that disagree on every leaf cost you every internal node, which is more than shipping the leaves outright. How many requests you keep in the air, how you batch them, and when you give up and copy the lot are your decisions.
|
|
310
|
+
|
|
311
|
+
### You Think: "Equal digests, equal data."
|
|
312
|
+
### Reality: Equal digests, and you chose the hash.
|
|
313
|
+
|
|
314
|
+
The survey trusts a matching digest and never looks below it. A collision in your digest hides a difference, and the survey can't know.
|
|
315
|
+
|
|
316
|
+
## What's Yours
|
|
317
|
+
|
|
318
|
+
This is an early release and the API will change. It is a policy core, not a sync runtime.
|
|
319
|
+
|
|
320
|
+
You build the trees and compute every digest. The survey does no hashing. You talk to the other replica, in whatever protocol and batch size you like, and you decide when a request is lost. You keep each tree readable at the version you pinned for as long as the session runs, or you abort and start over. You pick the `partitioning` and `digest` numbers and make sure equal numbers really mean equal layouts. You decide what a differing leaf range means: the survey locates divergence, it doesn't merge, repair, or pick a winner. And the survey lives in your process, so a restart means a new `begin`.
|
|
321
|
+
|
|
322
|
+
## License
|
|
323
|
+
|
|
324
|
+
MIT License. See [LICENSE.txt](LICENSE.txt) for details.
|
|
325
|
+
|
|
326
|
+
**Remember: the root told you they differ. The survey tells you where. Fixing it is still your job.**
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## [0.7.1](https://github.com/seuros/merkle_machines/compare/merkle-machines-v0.7.0...merkle-machines-v0.7.1) (2026-11-16)
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
### Bug Fixes
|
|
7
|
+
|
|
8
|
+
* index-free merkle survey core; Request#scope reads natively ([5e40549](https://github.com/seuros/merkle_machines/commit/5e40549c8b2bf82827a563797aa3551863da6b66))
|
|
9
|
+
|
|
10
|
+
## [0.7.0](https://github.com/seuros/merkle_machines/compare/merkle-machines-v0.6.0...merkle-machines-v0.7.0) (2026-11-14)
|
|
11
|
+
|
|
12
|
+
|
|
13
|
+
### âš BREAKING CHANGES
|
|
14
|
+
|
|
15
|
+
* adopt matryona-machines 0.2: native exact_fields readers for every value, native begin/answer/requeue, no reader or deconstruct runs
|
|
16
|
+
|
|
17
|
+
### Features
|
|
18
|
+
|
|
19
|
+
* adopt matryona-machines 0.2: native exact_fields readers for every value, native begin/answer/requeue, no reader or deconstruct runs ([b360c77](https://github.com/seuros/merkle_machines/commit/b360c775def2c804b7a4c5145cd3b7e855ff667d))
|
|
20
|
+
|
|
21
|
+
## [0.6.0](https://github.com/seuros/merkle_machines/compare/merkle-machines-v0.5.0...merkle-machines-v0.6.0) (2026-11-14)
|
|
22
|
+
|
|
23
|
+
|
|
24
|
+
### âš BREAKING CHANGES
|
|
25
|
+
|
|
26
|
+
* seal values, check exact classes natively, clone-then-keep requests and drains, native fields helper
|
|
27
|
+
|
|
28
|
+
### Features
|
|
29
|
+
|
|
30
|
+
* seal values, check exact classes natively, clone-then-keep requests and drains, native fields helper ([eb6ebfa](https://github.com/seuros/merkle_machines/commit/eb6ebfa88033b8f934743af11f8b370d88d60baa))
|
|
31
|
+
|
|
32
|
+
## [0.5.0](https://github.com/seuros/merkle_machines/compare/merkle-machines-v0.4.0...merkle-machines-v0.5.0) (2026-11-13)
|
|
33
|
+
|
|
34
|
+
|
|
35
|
+
### âš BREAKING CHANGES
|
|
36
|
+
|
|
37
|
+
* handles are issued through periscope-machines. Every instance can be built with a scope; a handle from another scope is refused before any lookup; all handle kinds of an instance share one sequence; resumed restores the sequence after a restart. A request from another survey answers Stale.
|
|
38
|
+
|
|
39
|
+
### Features
|
|
40
|
+
|
|
41
|
+
* scope sessions through periscope-machines, add Survey::resumed and issued ([0e54402](https://github.com/seuros/merkle_machines/commit/0e54402d670ce201fe4d18cf61544276005b87be))
|
|
42
|
+
|
|
43
|
+
## [0.4.0](https://github.com/seuros/merkle_machines/compare/merkle-machines-v0.3.0...merkle-machines-v0.4.0) (2026-11-13)
|
|
44
|
+
|
|
45
|
+
|
|
46
|
+
### âš BREAKING CHANGES
|
|
47
|
+
|
|
48
|
+
* see CONVENTIONS: one exhaustive Refusal per crate, Result from every mutator, honoured idempotent repeats, Ruby mirrors the core. No compatibility aliases. requeue returns Result<(), Refusal>; abort returns Aborting; Schema.new raises where the core has no schema.
|
|
49
|
+
|
|
50
|
+
### Features
|
|
51
|
+
|
|
52
|
+
* requeue and abort answer their real outcome, Ruby Request carries a Node and drain replaces take_ranges ([445d667](https://github.com/seuros/merkle_machines/commit/445d667f19c51a2bbf225af73d2897420a9a4d9a))
|
|
53
|
+
|
|
54
|
+
## [0.3.0](https://github.com/seuros/merkle_machines/compare/merkle-machines-v0.2.0...merkle-machines-v0.3.0) (2026-11-13)
|
|
55
|
+
|
|
56
|
+
|
|
57
|
+
### Features
|
|
58
|
+
|
|
59
|
+
* implement Default for LeafRange and name the drain buffer out ([d6e4c5e](https://github.com/seuros/merkle_machines/commit/d6e4c5ec9b67a705d52dcffea34d3a95fad059e7))
|
|
60
|
+
|
|
61
|
+
## [0.2.0](https://github.com/seuros/merkle_machines/compare/merkle-machines-v0.1.0...merkle-machines-v0.2.0) (2026-11-12)
|
|
62
|
+
|
|
63
|
+
|
|
64
|
+
### Features
|
|
65
|
+
|
|
66
|
+
* initial extraction ([43dd5e9](https://github.com/seuros/merkle_machines/commit/43dd5e92b117d600f3828fb106b9e4fa798f3f5e))
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
[package]
|
|
2
|
+
name = "merkle-machines"
|
|
3
|
+
version = "0.7.1"
|
|
4
|
+
edition.workspace = true
|
|
5
|
+
rust-version.workspace = true
|
|
6
|
+
license.workspace = true
|
|
7
|
+
description = "Bounded resumable digest-tree comparison that locates differing leaf ranges"
|
|
8
|
+
readme = "README.md"
|
|
9
|
+
exclude = ["src/**/tests.rs", "src/tests/", "tests/"]
|
|
10
|
+
keywords = ["merkle", "anti-entropy", "reconciliation", "digest", "no-std"]
|
|
11
|
+
categories = ["no-std", "algorithms"]
|
|
12
|
+
|
|
13
|
+
[dependencies]
|
|
14
|
+
periscope-machines.workspace = true
|
|
15
|
+
|
|
16
|
+
[dev-dependencies]
|
|
17
|
+
matryona-machines = { workspace = true, features = ["testing"] }
|
|
18
|
+
|
|
19
|
+
[lints]
|
|
20
|
+
workspace = true
|
|
@@ -0,0 +1,114 @@
|
|
|
1
|
+
# merkle-machines
|
|
2
|
+
|
|
3
|
+
> **AMS Merkle**, NCC-4224, *Merkle*-class survey cutter.
|
|
4
|
+
|
|
5
|
+
Engine-room flight manual: chart where two digest trees disagree, a bounded
|
|
6
|
+
number of nodes at a time.
|
|
7
|
+
|
|
8
|
+
Allocation-free `no_std` comparison traversal over two fixed-depth digest trees
|
|
9
|
+
of the same shape. The survey hands out requests for the child digests of nodes
|
|
10
|
+
whose digests differ, the caller answers with both trees' children, and the
|
|
11
|
+
survey emits the ranges of leaves that differ. It stores no digests and computes
|
|
12
|
+
none: hashing, storage and the exchange with the other replica belong to the
|
|
13
|
+
caller.
|
|
14
|
+
|
|
15
|
+
Two trees are comparable only when their `Schema` matches: fanout, depth, and the
|
|
16
|
+
caller's identifiers for key-to-leaf partitioning and digest algorithm. A
|
|
17
|
+
mismatch is refused by name and opens no session. A session pins one version of
|
|
18
|
+
each tree; an answer read from any other version is refused, because digests
|
|
19
|
+
from a tree that moved cannot be compared with digests from before it moved.
|
|
20
|
+
|
|
21
|
+
Buffers are fixed. `P` bounds the nodes held at once (queued, in flight or
|
|
22
|
+
partly expanded) and must be at least the tree depth; `O` bounds the buffered
|
|
23
|
+
leaf ranges and must be at least one. An answered node keeps its differing
|
|
24
|
+
children as a bit mask and releases them only as slots and output space allow,
|
|
25
|
+
so an answer is never refused for lack of room and a full buffer delays the
|
|
26
|
+
traversal instead of losing work. Lost requests are requeued one by one or all
|
|
27
|
+
at once. Every refusal is a `Refusal`, the crate's one refusal type, and leaves
|
|
28
|
+
the survey unchanged: `begin`, `answer` and `requeue` return
|
|
29
|
+
`Result<_, Refusal>`. Requests carry a session
|
|
30
|
+
id, so answers to an aborted or replaced session are stale. `abort` never
|
|
31
|
+
refuses: it answers `Aborting::Aborted`, or `Aborting::AlreadyIdle` when no
|
|
32
|
+
session was open, an idempotent repeat that changes nothing.
|
|
33
|
+
|
|
34
|
+
**Scopes.** A request is only meaningful to the survey that issued it. Session
|
|
35
|
+
ids come from a `periscope_machines::Issuer`: the survey's scope in the top 16
|
|
36
|
+
bits, a sequence counted from one below, never reissued.
|
|
37
|
+
`Survey::with_scope(scope)` chooses the scope and `new()` is scope zero;
|
|
38
|
+
`Survey::scope` and `Request::scope` read it. `answer` and `requeue` answer
|
|
39
|
+
`Stale` for a request whose session the survey did not issue, before looking at
|
|
40
|
+
any node. Give each survey its own scope: the core reads no clock and no random
|
|
41
|
+
source, so it cannot pick one, and two surveys left in the same scope (two
|
|
42
|
+
`new()` included) issue the same sessions and honour each other's requests. A
|
|
43
|
+
scope is `Exhausted` after 2^48 - 1 sessions. `session()` is the latest session
|
|
44
|
+
issued in the scope, open or not (`Issuer::last`): `None` until the first
|
|
45
|
+
session begins, the aborted session after `abort`. `Request::new` rebuilds a
|
|
46
|
+
request that crossed a wire and is `None` for a session of sequence zero, which
|
|
47
|
+
no survey issues.
|
|
48
|
+
|
|
49
|
+
**Restarts.** A fresh survey opens session one again, so a request that
|
|
50
|
+
outlived a restart could name a node in flight in a new session. Save
|
|
51
|
+
`Survey::issued` and rebuild with `Survey::resumed(scope, issued)`, or give
|
|
52
|
+
each incarnation a new scope. `resumed` restores only the session sequence: the
|
|
53
|
+
survey is idle, every earlier request is `Stale`, and comparing again is the
|
|
54
|
+
caller's job. It is `None` when `issued` is beyond
|
|
55
|
+
`periscope_machines::SEQUENCE_MAX`. A resumed survey's `session()` is the last
|
|
56
|
+
session issued before the restart, which is not open, or `None` when resumed
|
|
57
|
+
from zero.
|
|
58
|
+
|
|
59
|
+
**No clock.** The survey reads and records no time; `Versions` are tree
|
|
60
|
+
versions the caller pins for one session, not instants.
|
|
61
|
+
|
|
62
|
+
The Ruby gem mirrors `Schema`, `Versions`, `Node` and `Request` as `Data`
|
|
63
|
+
values of the same names and fields, each checked by the core when built: where
|
|
64
|
+
`Schema::new` or `Request::new` answers `None`, or a field does not fit the
|
|
65
|
+
core's type, `new` raises `RangeError` (`TypeError` for a wrong type). A
|
|
66
|
+
`LeafRange` is a half-open Ruby `Range`.
|
|
67
|
+
|
|
68
|
+
Buffered ranges are sorted, disjoint and joined when they touch. Ranges taken at
|
|
69
|
+
different times may still touch each other. The traversal visits only nodes
|
|
70
|
+
whose digests differ, deepest first; it makes no claim about minimal exchange or
|
|
71
|
+
round count. Equal digests are trusted to mean equal subtrees, so a digest
|
|
72
|
+
collision hides a difference, and a parent that differs while none of its
|
|
73
|
+
children do is refused as inconsistent.
|
|
74
|
+
|
|
75
|
+
`pop_range` takes one range; `drain(out)` writes as many as fit into the front
|
|
76
|
+
of `out`, returns the count and leaves the rest untouched. Build the buffer as
|
|
77
|
+
`[LeafRange::default(); N]`: the default is the empty range `0..0`, a
|
|
78
|
+
placeholder the survey never emits. A `LeafRange` never runs backwards: its
|
|
79
|
+
fields are private, read through `start()` and `end()`, and
|
|
80
|
+
`LeafRange::new(start, end)` answers `None` when `start > end`, so `len()`
|
|
81
|
+
cannot underflow.
|
|
82
|
+
|
|
83
|
+
```rust
|
|
84
|
+
use merkle_machines::{Begun, LeafRange, Phase, Schema, Survey, Versions};
|
|
85
|
+
|
|
86
|
+
// Two trees of four leaves; only leaf 2 differs.
|
|
87
|
+
let ours = [[10_u64, 20], [1, 2], [3, 4]];
|
|
88
|
+
let theirs = [[10_u64, 21], [1, 2], [9, 4]];
|
|
89
|
+
|
|
90
|
+
let schema = Schema::new(2, 2, 1, 1).expect("valid shape");
|
|
91
|
+
let versions = Versions { local: 40, remote: 57 };
|
|
92
|
+
let mut survey = Survey::<4, 4>::new();
|
|
93
|
+
assert_eq!(survey.begin(schema, schema, versions, &100_u64, &101_u64), Ok(Begun::Diverged));
|
|
94
|
+
|
|
95
|
+
while let Some(request) = survey.next_request() {
|
|
96
|
+
let node = request.node();
|
|
97
|
+
let row = if node.level == 0 { 0 } else { 1 + node.index as usize };
|
|
98
|
+
assert_eq!(survey.answer(request, versions, &ours[row], &theirs[row]), Ok(()));
|
|
99
|
+
}
|
|
100
|
+
assert_eq!(survey.pop_range(), LeafRange::new(2, 3));
|
|
101
|
+
assert_eq!(survey.phase(), Phase::Complete);
|
|
102
|
+
|
|
103
|
+
// Another survey, another scope: it refuses a request it did not issue.
|
|
104
|
+
use merkle_machines::Refusal;
|
|
105
|
+
let mut north = Survey::<4, 4>::with_scope(1);
|
|
106
|
+
let mut south = Survey::<4, 4>::with_scope(2);
|
|
107
|
+
north.begin(schema, schema, versions, &100_u64, &101_u64).expect("comparable");
|
|
108
|
+
south.begin(schema, schema, versions, &100_u64, &101_u64).expect("comparable");
|
|
109
|
+
let request = north.next_request().expect("the root");
|
|
110
|
+
south.next_request().expect("the root");
|
|
111
|
+
assert_eq!(request.scope(), 1);
|
|
112
|
+
assert_eq!(south.answer(request, versions, &ours[0], &theirs[0]), Err(Refusal::Stale));
|
|
113
|
+
assert_eq!(north.answer(request, versions, &ours[0], &theirs[0]), Ok(()));
|
|
114
|
+
```
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
#![no_std]
|
|
2
|
+
#![cfg_attr(not(test), deny(clippy::indexing_slicing))]
|
|
3
|
+
#![doc = include_str!("../README.md")]
|
|
4
|
+
|
|
5
|
+
mod refusal;
|
|
6
|
+
mod request;
|
|
7
|
+
mod schema;
|
|
8
|
+
mod survey;
|
|
9
|
+
|
|
10
|
+
pub use refusal::Refusal;
|
|
11
|
+
pub use request::{Request, Versions};
|
|
12
|
+
pub use schema::{LeafRange, MAX_FANOUT, Mismatch, Node, Schema};
|
|
13
|
+
pub use survey::{Aborting, Begun, Phase, Survey};
|
|
14
|
+
|
|
15
|
+
#[cfg(test)]
|
|
16
|
+
mod tests;
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
//! The one refusal type of the crate.
|
|
2
|
+
|
|
3
|
+
use core::fmt;
|
|
4
|
+
|
|
5
|
+
use crate::schema::Mismatch;
|
|
6
|
+
|
|
7
|
+
/// Why a call was refused. A refused call changes nothing.
|
|
8
|
+
///
|
|
9
|
+
/// Every operation that can refuse answers with this type:
|
|
10
|
+
/// [`Survey::begin`](crate::Survey::begin), [`Survey::answer`](crate::Survey::answer)
|
|
11
|
+
/// and [`Survey::requeue`](crate::Survey::requeue).
|
|
12
|
+
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
|
13
|
+
pub enum Refusal {
|
|
14
|
+
/// The two trees are not comparable.
|
|
15
|
+
Schema(Mismatch),
|
|
16
|
+
/// Pending capacity is below the tree depth, or output capacity is zero.
|
|
17
|
+
Undersized,
|
|
18
|
+
/// A session is still surveying or draining; abort it first.
|
|
19
|
+
Busy,
|
|
20
|
+
/// Every session id of this survey's scope has been issued.
|
|
21
|
+
Exhausted,
|
|
22
|
+
/// The request does not belong to the open session: it is from an earlier
|
|
23
|
+
/// session, from another survey's scope, or no session is open.
|
|
24
|
+
Stale,
|
|
25
|
+
/// The node is not in flight: never requested, already answered or requeued.
|
|
26
|
+
Unexpected,
|
|
27
|
+
/// The digests were read from a different tree version than the session pinned.
|
|
28
|
+
VersionChanged,
|
|
29
|
+
/// A child digest list does not hold exactly `fanout` digests.
|
|
30
|
+
Arity,
|
|
31
|
+
/// The parents differed but no child does: the digests are not from one tree.
|
|
32
|
+
Inconsistent,
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
impl fmt::Display for Refusal {
|
|
36
|
+
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
|
37
|
+
f.write_str(match self {
|
|
38
|
+
Self::Schema(Mismatch::Fanout) => "the trees differ in fanout",
|
|
39
|
+
Self::Schema(Mismatch::Depth) => "the trees differ in depth",
|
|
40
|
+
Self::Schema(Mismatch::Partitioning) => "the trees differ in partitioning",
|
|
41
|
+
Self::Schema(Mismatch::Digest) => "the trees differ in digest",
|
|
42
|
+
Self::Undersized => "capacity is below what the tree needs",
|
|
43
|
+
Self::Busy => "a session is still open",
|
|
44
|
+
Self::Exhausted => "every session of this scope has been issued",
|
|
45
|
+
Self::Stale => "request does not belong to the open session",
|
|
46
|
+
Self::Unexpected => "node is not in flight",
|
|
47
|
+
Self::VersionChanged => "digests were read from another tree version",
|
|
48
|
+
Self::Arity => "child digest list does not match the fanout",
|
|
49
|
+
Self::Inconsistent => "parents differ but no child does",
|
|
50
|
+
})
|
|
51
|
+
}
|
|
52
|
+
}
|
|
53
|
+
|
|
54
|
+
impl core::error::Error for Refusal {}
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
//! What travels between a survey and the replica answering it.
|
|
2
|
+
|
|
3
|
+
use periscope_machines::{scope_of, sequence_of};
|
|
4
|
+
|
|
5
|
+
use crate::schema::Node;
|
|
6
|
+
|
|
7
|
+
/// Tree versions the digests of one session must all be read from.
|
|
8
|
+
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
|
9
|
+
pub struct Versions {
|
|
10
|
+
/// Version of the local tree.
|
|
11
|
+
pub local: u64,
|
|
12
|
+
/// Version of the remote tree.
|
|
13
|
+
pub remote: u64,
|
|
14
|
+
}
|
|
15
|
+
|
|
16
|
+
/// A request for the child digests of one node, on both trees.
|
|
17
|
+
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
|
18
|
+
pub struct Request {
|
|
19
|
+
session: u64,
|
|
20
|
+
node: Node,
|
|
21
|
+
}
|
|
22
|
+
|
|
23
|
+
impl Request {
|
|
24
|
+
/// Rebuild a request that crossed a wire. A forged request is refused by
|
|
25
|
+
/// [`Survey::answer`] unless it names a node that is in flight.
|
|
26
|
+
///
|
|
27
|
+
/// `None` when `session` has sequence number zero: no survey issues that
|
|
28
|
+
/// id, so it names no session at all.
|
|
29
|
+
#[must_use]
|
|
30
|
+
pub const fn new(session: u64, node: Node) -> Option<Self> {
|
|
31
|
+
if sequence_of(session) == 0 {
|
|
32
|
+
return None;
|
|
33
|
+
}
|
|
34
|
+
Some(Self { session, node })
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
/// A request for a session the survey issued, so never sequence zero.
|
|
38
|
+
pub(crate) const fn issued(session: u64, node: Node) -> Self {
|
|
39
|
+
Self { session, node }
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
/// Session the request belongs to: a scoped id, the issuing survey's
|
|
43
|
+
/// scope in the top 16 bits and its session count (from one) below.
|
|
44
|
+
#[must_use]
|
|
45
|
+
pub const fn session(&self) -> u64 {
|
|
46
|
+
self.session
|
|
47
|
+
}
|
|
48
|
+
|
|
49
|
+
/// The scope of the survey that issued the request's session.
|
|
50
|
+
#[must_use]
|
|
51
|
+
pub const fn scope(&self) -> u16 {
|
|
52
|
+
scope_of(self.session)
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
/// Node whose children are wanted.
|
|
56
|
+
#[must_use]
|
|
57
|
+
pub const fn node(&self) -> Node {
|
|
58
|
+
self.node
|
|
59
|
+
}
|
|
60
|
+
}
|