join_machines 0.6.2-arm64-darwin
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +91 -0
- data/LICENSE.txt +21 -0
- data/README.md +278 -0
- data/ext/join_machines_native/core/CHANGELOG.md +66 -0
- data/ext/join_machines_native/core/README.md +176 -0
- data/ext/join_machines_native/extconf.rb +2 -0
- data/lib/join_machines/version.rb +4 -0
- data/lib/join_machines.rb +65 -0
- data/lib/join_machines_native/join_machines_native.bundle +0 -0
- data/sig/join_machines.rbs +140 -0
- metadata +62 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: f4f7d5762e2bfadfb7955859a1e282ed7f8c0555089b26cd6e183ba708ecc82b
|
|
4
|
+
data.tar.gz: 6ef2d4b563a3c292d828badddad73bd877182754aacbe339c6460c26b79f5057
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: c1a473002520f80883c6e5582a0f6bfe994857a141951dbbe4a6de19f8180bd3916b9debafeef0ac7e139e1329d14fbc628777fe58424dd6e8d92a5653cf52df
|
|
7
|
+
data.tar.gz: 3d8c986552c37d5f73a0f887ade6ec98258abc53df2654e07de43384d3b3dd21e25388bc246fa6628888c0d512863c51226cfcfcdf5f3290b38c93d088931347
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## [0.6.2](https://github.com/seuros/join_machines/compare/join_machines/v0.6.1...join_machines/v0.6.2) (2026-11-16)
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
### Bug Fixes
|
|
7
|
+
|
|
8
|
+
* fix wrapped classes at load so building a value runs no Ruby ([7df0f34](https://github.com/seuros/join_machines/commit/7df0f34bb6f9ee574054bef9b858c6c51db27152))
|
|
9
|
+
|
|
10
|
+
## [0.6.1](https://github.com/seuros/join_machines/compare/join_machines/v0.6.0...join_machines/v0.6.1) (2026-11-16)
|
|
11
|
+
|
|
12
|
+
|
|
13
|
+
### Bug Fixes
|
|
14
|
+
|
|
15
|
+
* validate values from their stored members, not deconstruct ([7af4e49](https://github.com/seuros/join_machines/commit/7af4e498acb6f9e48b3b323bf0fb0ccc57331ab3))
|
|
16
|
+
|
|
17
|
+
## [0.6.0](https://github.com/seuros/join_machines/compare/join_machines/v0.5.1...join_machines/v0.6.0) (2026-11-14)
|
|
18
|
+
|
|
19
|
+
|
|
20
|
+
### ⚠ BREAKING CHANGES
|
|
21
|
+
|
|
22
|
+
* read sibling watermarks through matryona own_or_sibling, one refusal message
|
|
23
|
+
|
|
24
|
+
### Features
|
|
25
|
+
|
|
26
|
+
* read sibling watermarks through matryona own_or_sibling, one refusal message ([6634897](https://github.com/seuros/join_machines/commit/6634897f7e7684feb52c3cf0ddb7c805467f4991))
|
|
27
|
+
|
|
28
|
+
## [0.5.1](https://github.com/seuros/join_machines/compare/join_machines/v0.5.0...join_machines/v0.5.1) (2026-11-14)
|
|
29
|
+
|
|
30
|
+
|
|
31
|
+
### Bug Fixes
|
|
32
|
+
|
|
33
|
+
* **deps:** require matryona-machines >= 0.3 with symbols! tables for refusals, advancing, inserting ([a287139](https://github.com/seuros/join_machines/commit/a287139080c0b42d4ea0435f9cecaf5213924b85))
|
|
34
|
+
|
|
35
|
+
## [0.5.0](https://github.com/seuros/join_machines/compare/join_machines/v0.4.0...join_machines/v0.5.0) (2026-11-14)
|
|
36
|
+
|
|
37
|
+
|
|
38
|
+
### ⚠ BREAKING CHANGES
|
|
39
|
+
|
|
40
|
+
* adopt matryona-machines 0.2 (exact_fields, symbols!, frozen-first)
|
|
41
|
+
|
|
42
|
+
### Features
|
|
43
|
+
|
|
44
|
+
* adopt matryona-machines 0.2 (exact_fields, symbols!, frozen-first) ([84d94d2](https://github.com/seuros/join_machines/commit/84d94d2fc841d1dcbc3d0cdd8da938dcd3547298))
|
|
45
|
+
|
|
46
|
+
## [0.4.0](https://github.com/seuros/join_machines/compare/join_machines/v0.3.0...join_machines/v0.4.0) (2026-11-14)
|
|
47
|
+
|
|
48
|
+
|
|
49
|
+
### ⚠ BREAKING CHANGES
|
|
50
|
+
|
|
51
|
+
* seal values, native Bounds#reach, exact-class reads, re-check frozen before keeping
|
|
52
|
+
|
|
53
|
+
### Features
|
|
54
|
+
|
|
55
|
+
* seal values, native Bounds#reach, exact-class reads, re-check frozen before keeping ([3432d10](https://github.com/seuros/join_machines/commit/3432d1010e389333e15a2adf9dd702f0f7adf64d))
|
|
56
|
+
|
|
57
|
+
## [0.3.0](https://github.com/seuros/join_machines/compare/join_machines/v0.2.0...join_machines/v0.3.0) (2026-11-13)
|
|
58
|
+
|
|
59
|
+
|
|
60
|
+
### ⚠ BREAKING CHANGES
|
|
61
|
+
|
|
62
|
+
* converted to the fleet conventions. One exhaustive Refusal per crate, Result from every operation that can refuse, a refusal changes nothing, provable idempotent repeats are honoured, Ruby mirrors the core. No compatibility aliases. Requires window-machines >= 0.3 and re-exports its Advancing. Refusal::Duplicate is gone; a retired record is Late on replay.
|
|
63
|
+
|
|
64
|
+
### Features
|
|
65
|
+
|
|
66
|
+
* honour identical re-inserts as Duplicate, refuse changed ones as Mismatch, retire only when both watermarks have passed ([e06e9fe](https://github.com/seuros/join_machines/commit/e06e9fe31e3e93b80dd1fa99bfb5bc60839b2bd2))
|
|
67
|
+
|
|
68
|
+
## [0.2.0](https://github.com/seuros/join_machines/compare/join_machines/v0.1.2...join_machines/v0.2.0) (2026-11-13)
|
|
69
|
+
|
|
70
|
+
|
|
71
|
+
### ⚠ BREAKING CHANGES
|
|
72
|
+
|
|
73
|
+
* see CONVENTIONS: one exhaustive Refusal per crate, Result from every mutator, honoured idempotent repeats, Ruby mirrors the core. No compatibility aliases. advance_watermark and advance_watermark_from return Result<Advancing, Refusal>.
|
|
74
|
+
|
|
75
|
+
### Features
|
|
76
|
+
|
|
77
|
+
* watermark advances answer Advancing or WatermarkRegressed, Ruby takes Bounds and Record ([eb4f557](https://github.com/seuros/join_machines/commit/eb4f557d1449c4791204000ca818b4a7d8bdec55))
|
|
78
|
+
|
|
79
|
+
## [0.1.2](https://github.com/seuros/join_machines/compare/join_machines/v0.1.1...join_machines/v0.1.2) (2026-11-13)
|
|
80
|
+
|
|
81
|
+
|
|
82
|
+
### Features
|
|
83
|
+
|
|
84
|
+
* expose the window-machines Watermark seam and default output elements ([51df744](https://github.com/seuros/join_machines/commit/51df744004344e4679adf8ccfe308ad68337eca8))
|
|
85
|
+
|
|
86
|
+
## [0.1.1](https://github.com/seuros/join_machines/compare/join_machines-v0.1.0...join_machines/v0.1.1) (2026-11-12)
|
|
87
|
+
|
|
88
|
+
|
|
89
|
+
### Features
|
|
90
|
+
|
|
91
|
+
* initial extraction ([a045343](https://github.com/seuros/join_machines/commit/a04534332b17bd40113de9bdcb41c6a24c6eefa6))
|
data/LICENSE.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Abdelkader Boudih
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,278 @@
|
|
|
1
|
+
# JoinMachines
|
|
2
|
+
|
|
3
|
+
> Two streams, one key, and a window of time in which they are allowed to meet. 🤝
|
|
4
|
+
|
|
5
|
+
An interval-join ledger for Ruby. You tell it about every record on two inputs: an id, a key, an event time. It tells you which left and right records share a key and landed close enough in time, hands you those pairs in batches as small as you like, and tells you which records can never match anything again so you can throw their payloads away. It never sees a payload. It keeps the bookkeeping, you keep the data.
|
|
6
|
+
|
|
7
|
+
Pre-1.0: breaking changes land in minor versions. It is a policy core, not a stream processor.
|
|
8
|
+
|
|
9
|
+
## Quick Start
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
gem 'join_machines'
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
```ruby
|
|
16
|
+
# A click joins an impression of the same ad shown up to 30 s before it.
|
|
17
|
+
join = JoinMachines::Join.new(JoinMachines::Bounds.new(before: 30_000, after: 0))
|
|
18
|
+
|
|
19
|
+
def on_record(join, side, record, store)
|
|
20
|
+
case join.insert(side, JoinMachines::Record.new(record.id, record.ad_id, record.event_ms))
|
|
21
|
+
when :late then return count_late(record)
|
|
22
|
+
when :mismatch then return count_conflict(record) # this id is held with other contents
|
|
23
|
+
when :full then raise 'join is full: advance the watermarks and retire'
|
|
24
|
+
end # :accepted, or :duplicate for a replay of a held record: nothing new is owed
|
|
25
|
+
store[side][record.id] = record
|
|
26
|
+
|
|
27
|
+
until (pairs = join.drain(100)).empty?
|
|
28
|
+
pairs.each { |pair| emit(store[:left][pair.left], store[:right][pair.right]) }
|
|
29
|
+
end
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
def on_watermark(join, side, event_ms, store)
|
|
33
|
+
join.advance_watermark(side, event_ms)
|
|
34
|
+
until (gone = join.retire(100)).empty?
|
|
35
|
+
gone.each { |r| store[r.side].delete(r.record.id) }
|
|
36
|
+
end
|
|
37
|
+
end
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
Ids, keys, and times are Integers. Time is event time in whatever unit you pick, as long as both inputs agree. Each side's watermark speaks for its own input only: one input running well ahead of the other makes nothing `:late` on the slower one.
|
|
41
|
+
|
|
42
|
+
## A Message to the Resistance
|
|
43
|
+
|
|
44
|
+
So you have impressions on one topic and clicks on another, and somebody in a meeting said "just join them." Thirty seconds of attribution window. How hard can it be.
|
|
45
|
+
|
|
46
|
+
You put the impressions in a hash by ad id. A click arrives, you look it up, you emit. Then you notice the hash only grows, so you add a sweeper that deletes anything older than thirty seconds by the wall clock. Then a consumer lags for a minute, the sweeper deletes impressions whose clicks are still sitting in the partition, and attribution quietly drops by four percent. Nothing raises. The dashboard just gets a little sadder.
|
|
47
|
+
|
|
48
|
+
So you stop sweeping by wall clock and sweep by event time. Whose event time? The click that arrives after the sweep was allowed to be up to thirty seconds *behind* the impression's stream, not its own. Then one hot key matches nine thousand buffered records at once and the handler that was supposed to emit "a pair" blocks the consumer while it emits nine thousand.
|
|
49
|
+
|
|
50
|
+
The piece you keep rebuilding is small: which records are held, which pairs are owed, and when is a record provably finished. With a hard bound on all three. That piece is this gem.
|
|
51
|
+
|
|
52
|
+
## Chapter 1: Stardate 2026.316, the AMS Widom
|
|
53
|
+
|
|
54
|
+
> **AMS Widom**, NCC-4232, *Widom*-class stream tender.
|
|
55
|
+
> Two convoys pass in the dark. The tender logs who was abeam of whom, and for how long it still matters.
|
|
56
|
+
|
|
57
|
+
The tender carries no cargo. Freighters from two lanes report in as they pass: hull number, consignment code, time of passage. The tender's log pairs every hull on one lane with every hull on the other that carried the same code within the agreed interval, and strikes a hull from the log only when the other lane's traffic control certifies that nothing that early is still inbound.
|
|
58
|
+
|
|
59
|
+
On a bridge somewhere, a cadet volunteers:
|
|
60
|
+
|
|
61
|
+
```ruby
|
|
62
|
+
def on_click(click)
|
|
63
|
+
impression = @impressions[click.ad_id]
|
|
64
|
+
emit(impression, click) if impression && click.at - impression.at <= 30
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
every(30.seconds) { @impressions.delete_if { |_, i| i.at < Time.now - 30 } }
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
"That," says the tender's logkeeper, "is a join whose correctness depends on nobody ever being late. Have you met a convoy?"
|
|
71
|
+
|
|
72
|
+
## The Flight Plan
|
|
73
|
+
|
|
74
|
+
Each record the join holds goes through this:
|
|
75
|
+
|
|
76
|
+
```mermaid
|
|
77
|
+
stateDiagram-v2
|
|
78
|
+
[*] --> refused: insert → :mismatch, :pending, :late, :full
|
|
79
|
+
[*] --> held: insert → :accepted
|
|
80
|
+
held --> held: insert of the same record → :duplicate
|
|
81
|
+
held --> held: pairs with it are drained
|
|
82
|
+
held --> retirable: opposite watermark passes its reach and its own watermark reaches it
|
|
83
|
+
retirable --> [*]: retire, returns a Retired
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
And the join as a whole only ever owes pairs for one record at a time:
|
|
87
|
+
|
|
88
|
+
```mermaid
|
|
89
|
+
sequenceDiagram
|
|
90
|
+
participant C as Caller
|
|
91
|
+
participant J as Join
|
|
92
|
+
|
|
93
|
+
C->>J: insert(:right, Record(1, key 9, t=8))
|
|
94
|
+
J-->>C: :accepted (nothing to match)
|
|
95
|
+
C->>J: insert(:right, Record(2, key 9, t=15))
|
|
96
|
+
J-->>C: :accepted
|
|
97
|
+
C->>J: insert(:left, Record(100, key 9, t=10))
|
|
98
|
+
J-->>C: :accepted, 2 pairs owed
|
|
99
|
+
C->>J: insert(:left, Record(101, ...))
|
|
100
|
+
J-->>C: :pending
|
|
101
|
+
C->>J: drain(1)
|
|
102
|
+
J-->>C: [Pair(9, 100, 1)]
|
|
103
|
+
C->>J: drain(1)
|
|
104
|
+
J-->>C: [Pair(9, 100, 2)]
|
|
105
|
+
C->>J: advance_watermark(:right, 15)
|
|
106
|
+
C->>J: retire(10)
|
|
107
|
+
J-->>C: [] (the left watermark has not reached 100)
|
|
108
|
+
C->>J: advance_watermark(:left, 10)
|
|
109
|
+
C->>J: retire(10)
|
|
110
|
+
J-->>C: [Retired(:left, Record(100, 9, 10)), Retired(:right, Record(1, 9, 8))]
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
## How the Join Works
|
|
114
|
+
|
|
115
|
+
### Bounds
|
|
116
|
+
|
|
117
|
+
```ruby
|
|
118
|
+
bounds = JoinMachines::Bounds.new(before: 2, after: 5) # or Bounds.new(2, 5)
|
|
119
|
+
join = JoinMachines::Join.new(bounds)
|
|
120
|
+
join.bounds # => #<data JoinMachines::Bounds before=2, after=5>
|
|
121
|
+
bounds.reach(:left, 10) # => #<data JoinMachines::Reach first=8, last=15>
|
|
122
|
+
bounds.reach(:right, 10) # => #<data JoinMachines::Reach first=5, last=12>
|
|
123
|
+
bounds.reach(:left, 10).include?(15) # => true
|
|
124
|
+
JoinMachines.other(:left) # => :right
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
A left record at time `t` matches right records with the same key at times `t - 2` through `t + 5`, both ends included. `Bounds.new(0, 0)` is an exact-time join. `Join.new` takes the `Bounds` value, the way the core's constructor does; the bounds are fixed for the life of the join. `reach(side, timestamp)` is the range of opposite-side times a record on `side` can match.
|
|
128
|
+
|
|
129
|
+
### Insert
|
|
130
|
+
|
|
131
|
+
```ruby
|
|
132
|
+
Record = JoinMachines::Record # id, key, timestamp
|
|
133
|
+
|
|
134
|
+
join.insert(:right, Record.new(1, 9, 8)) # => :accepted
|
|
135
|
+
join.insert(:right, Record.new(2, 9, 15)) # => :accepted
|
|
136
|
+
join.insert(:right, Record.new(3, 9, 16)) # => :accepted
|
|
137
|
+
join.insert(:left, Record.new(100, 9, 10)) # => :accepted
|
|
138
|
+
join.pending? # => true
|
|
139
|
+
join.pending_pairs # => 2
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
`insert(side, record)` takes a `Record`, the same value `retire` hands back. An accepted record is held, and the join now owes one pair for every held record on the other side with the same key inside the interval. The ids are yours. They only have to be unique among the records currently held on the same side.
|
|
143
|
+
|
|
144
|
+
Replaying a record the join still holds is fine. The same id with the same key and timestamp answers `:duplicate`: honoured, nothing changes, and no pair is owed a second time. The same id with a different key or timestamp answers `:mismatch`, a refusal. A held id is answered this way before anything else is looked at, so a replay is `:duplicate` even while pairs are owed or after the side's watermark has passed the record.
|
|
145
|
+
|
|
146
|
+
```ruby
|
|
147
|
+
join.insert(:left, Record.new(100, 9, 10)) # => :duplicate, already held, still owes 2 pairs
|
|
148
|
+
join.insert(:left, Record.new(100, 9, 11)) # => :mismatch, id 100 is held with timestamp 10
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
Held means held right now. A record that matched and was drained is still held. A record that was retired is not, but it cannot come back: it was let go only after its own side's watermark had reached it, and that watermark never moves back, so the same record replayed is `:late`, every time, and no pair is handed over twice. The join does this without keeping any history, which also means it does not remember the id itself: a *different* record under a retired id is judged like any new one. Keep one id for one record and replays across retirement are safe.
|
|
152
|
+
|
|
153
|
+
### Drain
|
|
154
|
+
|
|
155
|
+
```ruby
|
|
156
|
+
join.drain(1) # => [#<data JoinMachines::Pair key=9, left=100, right=1>]
|
|
157
|
+
join.pending? # => true
|
|
158
|
+
join.drain(8) # => [#<data JoinMachines::Pair key=9, left=100, right=2>]
|
|
159
|
+
join.pending? # => false
|
|
160
|
+
join.drain(8) # => []
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
`drain(max)` hands over at most `max` owed pairs. Whatever didn't fit stays owed, in order, and the next call picks up exactly where this one stopped. Every pair comes out once. While pairs are owed, `insert` and `retire` answer `:pending`:
|
|
164
|
+
|
|
165
|
+
```ruby
|
|
166
|
+
join.insert(:left, Record.new(101, 9, 14)) # => :accepted, owes pairs with 2 and 3
|
|
167
|
+
join.insert(:left, Record.new(102, 9, 14)) # => :pending
|
|
168
|
+
join.drain(8).size # => 2
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
### Watermarks
|
|
172
|
+
|
|
173
|
+
```ruby
|
|
174
|
+
join.advance_watermark(:right, 14) # => :advanced
|
|
175
|
+
join.advance_watermark(:right, 14) # => :repeated, already there, nothing changed
|
|
176
|
+
join.advance_watermark(:right, 13) # => :watermark_regressed, time went backwards
|
|
177
|
+
join.watermark(:right) # => 14
|
|
178
|
+
join.watermark(:left) # => nil
|
|
179
|
+
join.insert(:right, Record.new(4, 9, 14)) # => :late
|
|
180
|
+
join.insert(:right, Record.new(4, 9, 15)) # => :accepted
|
|
181
|
+
join.drain(8)
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
Each side has its own watermark: your promise that no more records at or below that time will arrive on that side. The join holds you to it. A record at or below its own side's watermark is `:late` and is not held. The *other* side's watermark never makes a record late: if its partner is still held, the record is accepted and the pair is owed, however far apart the two watermarks are. Advancing a watermark removes nothing by itself. Repeating the value a side already holds is not a refusal: the promise you are making already stands, so the answer is `:repeated`. Only a value behind the side's watermark is refused, with `:watermark_regressed`.
|
|
185
|
+
|
|
186
|
+
If you already track a stream's progress in a watermark object, hand it over instead of unwrapping it:
|
|
187
|
+
|
|
188
|
+
```ruby
|
|
189
|
+
clicks = WindowMachines::Watermark.new # from the window_machines gem, if you use it
|
|
190
|
+
clicks.advance(14) # => :advanced
|
|
191
|
+
join.advance_watermark_from(:right, clicks) # => :repeated, the side already agrees
|
|
192
|
+
join.watermark(:right) # => 14
|
|
193
|
+
join.side_watermark(:right) # => #<data JoinMachines::Watermark value=14>
|
|
194
|
+
join.side_watermark(:left) # => #<data JoinMachines::Watermark value=nil>
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
`advance_watermark_from(side, watermark)` reads `watermark.value` and answers like `advance_watermark`: `:advanced` when the side moved up to it, `:repeated` when the side already equals it, `:watermark_regressed` and no change when it is behind. Any object with a `value` reader returning an Integer or `nil` works; this gem does not depend on `window_machines`. A `WindowMachines::Watermark` fits as it is, whether you advanced it yourself, built it with `Watermark.at(saved)`, or took the copy `Sessions#watermark` answers; only its `value` is read, and it is never advanced for you. A `nil` value means nothing was observed yet: `:repeated` while the side is `nil` too, `:watermark_regressed` once the side has a value. An object without `value` raises `TypeError`. This gem's own `Watermark` is read natively instead, and a subclass of it is not taken for a sibling's: it raises `TypeError`. A `value` that freezes the join raises `FrozenError` and nothing is advanced.
|
|
198
|
+
|
|
199
|
+
`side_watermark(side)` is the other direction: the side's progress as a `JoinMachines::Watermark`, a frozen `Data` with the one reader `value` (an Integer, or `nil` before the first watermark). That is exactly the shape `advance_watermark_from` reads, so one join's side can be handed to another join. It is a copy taken at the call, not a live view, and it does not advance; `watermark(side)` is the same number without the wrapper.
|
|
200
|
+
|
|
201
|
+
### Retire
|
|
202
|
+
|
|
203
|
+
```ruby
|
|
204
|
+
join.retirable # => 0
|
|
205
|
+
join.advance_watermark(:right, 15) # left 100 reached right times 8..15
|
|
206
|
+
join.retirable # => 0, a replay of left 100 would not be late yet
|
|
207
|
+
join.advance_watermark(:left, 10) # now it would be; right 1 reached left times 3..10
|
|
208
|
+
join.retirable # => 2
|
|
209
|
+
join.retire(10)
|
|
210
|
+
# => [#<data JoinMachines::Retired side=:left,
|
|
211
|
+
# record=#<data JoinMachines::Record id=100, key=9, timestamp=10>>,
|
|
212
|
+
# #<data JoinMachines::Retired side=:right,
|
|
213
|
+
# record=#<data JoinMachines::Record id=1, key=9, timestamp=8>>]
|
|
214
|
+
```
|
|
215
|
+
|
|
216
|
+
A held record is finished when two things are true. The *other* side's watermark has reached the last time it could match (`timestamp + after` for a left record, `timestamp + before` for a right one, stopping at `2**64-1`), so nothing that can still arrive pairs with it. And its *own* side's watermark has reached its timestamp, so the same record offered again is `:late`. The first makes dropping it lossless, the second makes it final. A record whose own side lags stays held until that side catches up. `retire(max)` removes up to `max` finished records and tells you which, so you can drop their payloads: each `Retired` is the `side` and the `Record` (`id`, `key`, `timestamp`) that was held. Anything over `max` stays `retirable` for the next call.
|
|
217
|
+
|
|
218
|
+
### Capacity
|
|
219
|
+
|
|
220
|
+
```ruby
|
|
221
|
+
JoinMachines::CAPACITY # => 64
|
|
222
|
+
join.capacity(:left) # => 64
|
|
223
|
+
join.capacity(:right) # => 64
|
|
224
|
+
join.size(:left) # => 1
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
Each side holds 64 records. Past that, `insert` says `:full`. A record is never evicted to make room, because evicting it would silently lose its future pairs. Advance the watermarks, retire, and try again, or shard by key across joins.
|
|
228
|
+
|
|
229
|
+
## The Rules of the Tender
|
|
230
|
+
|
|
231
|
+
- **Ids, keys, times, bounds, and limits are Integers** in `0..2**64-1`. Floats, nil, and strings raise `TypeError`. Out of range raises `RangeError`. A side that isn't `:left` or `:right` raises `RangeError` (or `TypeError` if it isn't a Symbol).
|
|
232
|
+
- **Refusals don't raise.** `:mismatch`, `:pending`, `:late`, `:full` and `:watermark_regressed` are answers, each with its own symbol and no catch-all, and a refused call changes nothing. Only malformed input raises.
|
|
233
|
+
- **Every answer, by method.**
|
|
234
|
+
|
|
235
|
+
| Method | Honoured | Refused | Records the clock |
|
|
236
|
+
| --- | --- | --- | --- |
|
|
237
|
+
| `insert(side, record)` | `:accepted`, or `:duplicate` when exactly that record is already held | `:mismatch`, `:pending`, `:late`, `:full` | no clock, nothing recorded |
|
|
238
|
+
| `drain(max)` | an Array of `Pair`, `[]` when none is owed | never refuses | no clock, nothing recorded |
|
|
239
|
+
| `advance_watermark(side, value)` | `:advanced`, or `:repeated` when the side already held `value` | `:watermark_regressed` | no clock; `:advanced` records the side's watermark, `:repeated` nothing |
|
|
240
|
+
| `advance_watermark_from(side, watermark)` | `:advanced`, or `:repeated` when the side already equals it | `:watermark_regressed` | no clock; `:advanced` records the side's watermark, `:repeated` nothing |
|
|
241
|
+
| `retire(max)` | an Array of `Retired`, `[]` when none is retirable | `:pending` | no clock, nothing recorded |
|
|
242
|
+
| `Reach#include?(timestamp)` | `true` or `false` | never refuses | no clock, nothing recorded |
|
|
243
|
+
- **Arguments are the core's values.** `Join.new` takes a `Bounds`, `insert` takes a `Record`. A Hash, an Array or loose Integers in their place raise `TypeError` (a wrong argument count is Ruby's own `ArgumentError`).
|
|
244
|
+
- **Values are valid or do not exist.** `Reach`, `Bounds`, `Pair`, `Record`, `Retired` and `Watermark` are frozen `Data` checked by the core when built, by hand or by the gem, `with` included: a wrong type raises `TypeError`, an Integer outside `0..2**64-1` or a side other than `:left`/`:right` raises `RangeError`, and `Reach.new(5, 3)` raises `RangeError` (a reach holds at least one timestamp). `Retired` holds a `Record` and nothing else. Every value class is sealed: `allocate` is gone and subclassing raises `TypeError`. Sealing stops accidents, not reflection (a named `class X < Record` still binds its constant, `Data#initialize` can be bound directly, `Marshal.load` skips `initialize`), so the native side is the authority: `Join.new`, `insert`, `Retired.new`, `Bounds#reach`, `Reach#include?` and `advance_watermark_from` (for this gem's own `Watermark`) accept only an instance of exactly the class, read its members without calling any method, and check them through the core again: `TypeError` for anything else, `RangeError` for an invalid member. The classes are resolved once when the gem loads; swapping a constant later changes nothing the gem builds or accepts.
|
|
245
|
+
- **A call that raises changed nothing.** `drain` and `retire` build their whole answer before the join keeps the change, so if building it raises every pair is still owed and every record still held. If building it changed the join (a redefined `Pair.new` or `Retired.new` calling back in), the outer call raises `RuntimeError`, is not applied, and that inner change stands. If building it froze the join, the call raises `FrozenError` and is not applied.
|
|
246
|
+
- **No clock is recorded.** Everything here is event time you pass in; nothing reads or records a monotonic clock. The only progress kept is each side's watermark, recorded on `:advanced` and left alone on `:repeated` or `:watermark_regressed`. `advance_watermark_from` reads a watermark (a `WindowMachines::Watermark` or `side_watermark`'s answer) and keeps only its value.
|
|
247
|
+
- **Repeats are honoured only while provable.** `:duplicate` means the join holds that exact record on that side right now, and changes nothing. It is never a refusal. A retired record leaves no trace, so a replay of it is not `:duplicate`; it is `:late`, because its own side's watermark had reached it before it was let go and never moves back.
|
|
248
|
+
- **One record's pairs at a time.** Drain what the last insert owes before inserting or retiring.
|
|
249
|
+
- **Late is judged on one watermark. Retirement is judged on both.** A record is `:late` at or below its own side's watermark and for no other reason, so a record whose partner is held is never turned away because of the other input. `retire` releases a held record at `t` only when the other side's watermark is at or past the end of its reach (`>= bounds.reach(side, t).last`: nothing still to come can match it) *and* its own side's watermark is at or past `t` (a replay is `:late`). So nothing matchable is ever refused, a retired record replayed is `:late` for good, and every pair is handed over exactly once.
|
|
250
|
+
- **A lagging side keeps its records.** A side whose own watermark is behind holds its records longer, however far the other side has run, and a side whose watermark never moves retires nothing. That costs slots, never pairs: the bound is the capacity, and past it `insert` says `:full`. Advance both watermarks to release records.
|
|
251
|
+
- **Time saturates at the edges.** A reach that would run below 0 or past `2**64-1` stops there.
|
|
252
|
+
- **Frozen means frozen.** `freeze` the join and `insert`, `drain`, `advance_watermark`, `advance_watermark_from`, and `retire` raise `FrozenError`.
|
|
253
|
+
|
|
254
|
+
## What You Think You're Doing vs Reality
|
|
255
|
+
|
|
256
|
+
### You Think: "I'll expire old records after the window has passed."
|
|
257
|
+
### Reality: Passed according to whom?
|
|
258
|
+
|
|
259
|
+
Your wall clock says thirty seconds went by. The click stream, three partitions behind, has not been told. A record is finished when the stream it is waiting on says it has moved on, and the only thing that can say that is that stream's watermark. `retire` asks that, and asks the record's own stream whether it could still be delivered again.
|
|
260
|
+
|
|
261
|
+
### You Think: "A record matches, I emit a pair."
|
|
262
|
+
### Reality: A record matches four hundred, and your output buffer holds fifty.
|
|
263
|
+
|
|
264
|
+
A join's output per input is not bounded by one. `drain(max)` makes the batch size your decision, and keeps the remainder owed instead of dropped or buffered somewhere you can't see.
|
|
265
|
+
|
|
266
|
+
## What's Yours
|
|
267
|
+
|
|
268
|
+
The payloads. The join stores three Integers per record and nothing else, so you keep a store keyed by side and id and delete from it when `retire` says so. Hashing a wide key down to an Integer is also yours, and so is checking the real key on a pair if your hash can collide.
|
|
269
|
+
|
|
270
|
+
The watermarks. The join believes them. Advance one too eagerly and that side's own records come back `:late`; the other side's never do. Never advance one and both sides fill up and stay `:full`.
|
|
271
|
+
|
|
272
|
+
Everything else a stream processor does: reading inputs, running the user's function on a pair, checkpointing, restoring after a crash. The join lives in your process and forgets everything when the process does. Pairs you drained before a crash are not remembered as drained by anyone but you.
|
|
273
|
+
|
|
274
|
+
## License
|
|
275
|
+
|
|
276
|
+
MIT License. See [LICENSE.txt](LICENSE.txt) for details.
|
|
277
|
+
|
|
278
|
+
**Remember: the record you swept on a timer was thirty seconds old. Its match was thirty-one seconds late.**
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## [0.7.1](https://github.com/seuros/join_machines/compare/join-machines-v0.7.0...join-machines-v0.7.1) (2026-11-16)
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
### Bug Fixes
|
|
7
|
+
|
|
8
|
+
* validate values from their stored members, not deconstruct ([7af4e49](https://github.com/seuros/join_machines/commit/7af4e498acb6f9e48b3b323bf0fb0ccc57331ab3))
|
|
9
|
+
|
|
10
|
+
## [0.7.0](https://github.com/seuros/join_machines/compare/join-machines-v0.6.0...join-machines-v0.7.0) (2026-11-14)
|
|
11
|
+
|
|
12
|
+
|
|
13
|
+
### ⚠ BREAKING CHANGES
|
|
14
|
+
|
|
15
|
+
* adopt matryona-machines 0.2 (exact_fields, symbols!, frozen-first)
|
|
16
|
+
|
|
17
|
+
### Features
|
|
18
|
+
|
|
19
|
+
* adopt matryona-machines 0.2 (exact_fields, symbols!, frozen-first) ([84d94d2](https://github.com/seuros/join_machines/commit/84d94d2fc841d1dcbc3d0cdd8da938dcd3547298))
|
|
20
|
+
|
|
21
|
+
## [0.6.0](https://github.com/seuros/join_machines/compare/join-machines-v0.5.0...join-machines-v0.6.0) (2026-11-14)
|
|
22
|
+
|
|
23
|
+
|
|
24
|
+
### ⚠ BREAKING CHANGES
|
|
25
|
+
|
|
26
|
+
* seal values, native Bounds#reach, exact-class reads, re-check frozen before keeping
|
|
27
|
+
|
|
28
|
+
### Features
|
|
29
|
+
|
|
30
|
+
* seal values, native Bounds#reach, exact-class reads, re-check frozen before keeping ([3432d10](https://github.com/seuros/join_machines/commit/3432d1010e389333e15a2adf9dd702f0f7adf64d))
|
|
31
|
+
|
|
32
|
+
## [0.5.0](https://github.com/seuros/join_machines/compare/join-machines-v0.4.0...join-machines-v0.5.0) (2026-11-13)
|
|
33
|
+
|
|
34
|
+
|
|
35
|
+
### ⚠ BREAKING CHANGES
|
|
36
|
+
|
|
37
|
+
* converted to the fleet conventions. One exhaustive Refusal per crate, Result from every operation that can refuse, a refusal changes nothing, provable idempotent repeats are honoured, Ruby mirrors the core. No compatibility aliases. Requires window-machines >= 0.3 and re-exports its Advancing. Refusal::Duplicate is gone; a retired record is Late on replay.
|
|
38
|
+
|
|
39
|
+
### Features
|
|
40
|
+
|
|
41
|
+
* honour identical re-inserts as Duplicate, refuse changed ones as Mismatch, retire only when both watermarks have passed ([e06e9fe](https://github.com/seuros/join_machines/commit/e06e9fe31e3e93b80dd1fa99bfb5bc60839b2bd2))
|
|
42
|
+
|
|
43
|
+
## [0.4.0](https://github.com/seuros/join_machines/compare/join-machines-v0.3.0...join-machines-v0.4.0) (2026-11-13)
|
|
44
|
+
|
|
45
|
+
|
|
46
|
+
### ⚠ BREAKING CHANGES
|
|
47
|
+
|
|
48
|
+
* see CONVENTIONS: one exhaustive Refusal per crate, Result from every mutator, honoured idempotent repeats, Ruby mirrors the core. No compatibility aliases. advance_watermark and advance_watermark_from return Result<Advancing, Refusal>.
|
|
49
|
+
|
|
50
|
+
### Features
|
|
51
|
+
|
|
52
|
+
* watermark advances answer Advancing or WatermarkRegressed, Ruby takes Bounds and Record ([eb4f557](https://github.com/seuros/join_machines/commit/eb4f557d1449c4791204000ca818b4a7d8bdec55))
|
|
53
|
+
|
|
54
|
+
## [0.3.0](https://github.com/seuros/join_machines/compare/join-machines-v0.2.0...join-machines-v0.3.0) (2026-11-13)
|
|
55
|
+
|
|
56
|
+
|
|
57
|
+
### Features
|
|
58
|
+
|
|
59
|
+
* expose the window-machines Watermark seam and default output elements ([51df744](https://github.com/seuros/join_machines/commit/51df744004344e4679adf8ccfe308ad68337eca8))
|
|
60
|
+
|
|
61
|
+
## [0.2.0](https://github.com/seuros/join_machines/compare/join-machines-v0.1.0...join-machines-v0.2.0) (2026-11-12)
|
|
62
|
+
|
|
63
|
+
|
|
64
|
+
### Features
|
|
65
|
+
|
|
66
|
+
* initial extraction ([a045343](https://github.com/seuros/join_machines/commit/a04534332b17bd40113de9bdcb41c6a24c6eefa6))
|
|
@@ -0,0 +1,176 @@
|
|
|
1
|
+
# join-machines
|
|
2
|
+
|
|
3
|
+
> **AMS Widom**, NCC-4232, *Widom*-class stream tender.
|
|
4
|
+
|
|
5
|
+
Engine-room flight manual: log which hulls on two lanes passed abeam of each other.
|
|
6
|
+
|
|
7
|
+
Allocation-free `no_std` two-input interval join over record metadata. Each
|
|
8
|
+
record is an identity, a `u64` key and an event time; a left record at `t`
|
|
9
|
+
matches right records with the same key at `t - before ..= t + after`. The join
|
|
10
|
+
holds at most `L` left and `R` right records and stores no payloads.
|
|
11
|
+
|
|
12
|
+
Inserting a record owes one pair per matching held opposite record. `drain`
|
|
13
|
+
writes owed pairs into a caller buffer of any size and resumes where it stopped,
|
|
14
|
+
so a full buffer never loses or repeats a pair. While pairs are owed, `insert`
|
|
15
|
+
and `retire` are refused with `Pending`; that is the only backpressure the core
|
|
16
|
+
needs to stay bounded.
|
|
17
|
+
|
|
18
|
+
Each input has its own `window_machines::Watermark`. A record at or below its
|
|
19
|
+
own side's watermark is `Late`; the opposite watermark never makes a record
|
|
20
|
+
late. A held record becomes retirable when both watermarks have finished it:
|
|
21
|
+
the opposite one has reached the end of its reach, the last time it can match,
|
|
22
|
+
and its own has reached its timestamp. `retire` removes such records into a
|
|
23
|
+
bounded buffer so the caller can release their payloads. So a record whose
|
|
24
|
+
partner is still held is always admitted and paired, and a retired record
|
|
25
|
+
offered again is `Late`. Reaches saturate at the `u64` edges. The exact
|
|
26
|
+
inequalities are in the Contract below.
|
|
27
|
+
|
|
28
|
+
Refusals (`Mismatch`, `Pending`, `Late`, `Full`, `WatermarkRegressed`) are one
|
|
29
|
+
`Refusal` enum and change nothing. A full side is never evicted.
|
|
30
|
+
|
|
31
|
+
`insert` returns `Result<Inserting, Refusal>`. A record whose identity is held
|
|
32
|
+
on that side is answered from the held record, before any other check:
|
|
33
|
+
|
|
34
|
+
| `insert(side, record)` | Answer |
|
|
35
|
+
| --- | --- |
|
|
36
|
+
| identity not held, admitted | `Ok(Inserting::Accepted)`: held, its pairs owed |
|
|
37
|
+
| identity held with the same key and timestamp | `Ok(Inserting::Duplicate)`: nothing changes, no pair is owed twice |
|
|
38
|
+
| identity held with another key or timestamp | `Err(Refusal::Mismatch)` |
|
|
39
|
+
| identity not held, pairs still owed | `Err(Refusal::Pending)` |
|
|
40
|
+
| identity not held, at or below its own side's watermark | `Err(Refusal::Late)` |
|
|
41
|
+
| identity not held, no free slot | `Err(Refusal::Full)` |
|
|
42
|
+
|
|
43
|
+
Held means live right now. A record that has matched and been drained is still
|
|
44
|
+
held, so replaying it is `Duplicate`, also while its pairs are still owed and
|
|
45
|
+
after its own watermark has passed it. A retired record is no longer held, and
|
|
46
|
+
the join keeps no history of identities, yet the same record can never be
|
|
47
|
+
admitted again: it was retired only once its own watermark had reached it, and
|
|
48
|
+
that watermark never moves back, so a replay is `Late`. The identity itself is
|
|
49
|
+
not remembered: other contents under it are judged as any new record is
|
|
50
|
+
(`Accepted` if they are above the side's watermark, never `Mismatch`).
|
|
51
|
+
|
|
52
|
+
`advance_watermark` returns `Result<Advancing, Refusal>`: `Advancing::Advanced`
|
|
53
|
+
when the side moved forward, `Advancing::Repeated` when it already held exactly
|
|
54
|
+
that value (an idempotent repeat that changes nothing), and
|
|
55
|
+
`Err(Refusal::WatermarkRegressed)` for a value behind it. `Advancing` is
|
|
56
|
+
`window_machines::Advancing`, re-exported. The join asks the side's
|
|
57
|
+
`Watermark` where it stands and then sets it with `Watermark::at`, so it never
|
|
58
|
+
has to translate sibling refusals (`Overflow`, `Undersized`, `Late`, `Full`)
|
|
59
|
+
that `Watermark::advance` can never give.
|
|
60
|
+
|
|
61
|
+
The sibling seam is public: `Watermark` and `Advancing` are re-exported, `side_watermark(side)`
|
|
62
|
+
returns a side's `Watermark` by value (next to `watermark(side)`, which still
|
|
63
|
+
reads the bare `Option<u64>`), and `advance_watermark_from(side, &Watermark)`
|
|
64
|
+
advances a side from a watermark the caller owns, with the same answers as
|
|
65
|
+
`advance_watermark`. A source watermark with no value yet is
|
|
66
|
+
`Ok(Advancing::Repeated)` while the side is also uninitialized (they already
|
|
67
|
+
agree) and `Err(Refusal::WatermarkRegressed)` once the side has a value; it
|
|
68
|
+
never clears a side.
|
|
69
|
+
|
|
70
|
+
`Reach` keeps its fields private and is valid or does not exist:
|
|
71
|
+
`Reach::new(first, last)` is `None` unless `first <= last`, and `Bounds::reach`
|
|
72
|
+
always answers one (it saturates at both edges). `Record`, `Pair`, `Retired`
|
|
73
|
+
and `Bounds` hold any value of their field types, so their fields stay public
|
|
74
|
+
(`Bounds` behind its infallible `new`).
|
|
75
|
+
|
|
76
|
+
`Pair` and `Retired` implement `Default` so an output buffer is
|
|
77
|
+
`[Pair::default(); N]`. The default is a placeholder, not a
|
|
78
|
+
sentinel: a real zero-valued output equals it, so read only the entries reported.
|
|
79
|
+
|
|
80
|
+
The Ruby gem mirrors `Bounds`, `Reach`, `Record`, `Pair` and `Retired` as
|
|
81
|
+
`Data` values of the same names and fields, checked by the core when built, and takes `Bounds` and `Record`
|
|
82
|
+
where this core does. `side_watermark` answers a `Data` with the one reader
|
|
83
|
+
`value`, which is what its `advance_watermark_from` reads.
|
|
84
|
+
|
|
85
|
+
This is join bookkeeping only: no payload buffering, transport, execution,
|
|
86
|
+
persistence, key hashing or clocks. Key collisions, watermark generation and
|
|
87
|
+
recovery belong to the caller. Pre-1.0: breaking changes land in minor versions.
|
|
88
|
+
|
|
89
|
+
```rust
|
|
90
|
+
use join_machines::{
|
|
91
|
+
Advancing, Bounds, Inserting, Join, Pair, Record, Refusal, Side, Watermark,
|
|
92
|
+
};
|
|
93
|
+
let mut join = Join::<4, 4>::new(Bounds::new(2, 5));
|
|
94
|
+
let record = |id, timestamp| Record { id, key: 9, timestamp };
|
|
95
|
+
assert_eq!(join.insert(Side::Right, record(1, 8)), Ok(Inserting::Accepted));
|
|
96
|
+
assert_eq!(join.insert(Side::Right, record(2, 15)), Ok(Inserting::Accepted));
|
|
97
|
+
assert_eq!(join.insert(Side::Left, record(100, 10)), Ok(Inserting::Accepted));
|
|
98
|
+
assert_eq!(join.insert(Side::Left, record(101, 10)), Err(Refusal::Pending));
|
|
99
|
+
// A replay of a held record is honoured; the same id with other contents is not.
|
|
100
|
+
assert_eq!(join.insert(Side::Left, record(100, 10)), Ok(Inserting::Duplicate));
|
|
101
|
+
assert_eq!(join.insert(Side::Left, record(100, 11)), Err(Refusal::Mismatch));
|
|
102
|
+
|
|
103
|
+
let mut out = [Pair::default(); 1];
|
|
104
|
+
assert_eq!(join.drain(&mut out), 1);
|
|
105
|
+
assert_eq!(out[0], Pair { key: 9, left: 100, right: 1 });
|
|
106
|
+
assert_eq!(join.drain(&mut out), 1);
|
|
107
|
+
assert_eq!(out[0].right, 2);
|
|
108
|
+
assert!(!join.pending());
|
|
109
|
+
|
|
110
|
+
// The right input's progress is tracked by a watermark the caller owns.
|
|
111
|
+
let mut right = Watermark::default();
|
|
112
|
+
assert_eq!(right.advance(15), Ok(Advancing::Advanced));
|
|
113
|
+
assert_eq!(join.advance_watermark_from(Side::Right, &right), Ok(Advancing::Advanced));
|
|
114
|
+
assert_eq!(join.advance_watermark(Side::Right, 15), Ok(Advancing::Repeated));
|
|
115
|
+
assert_eq!(join.advance_watermark(Side::Right, 14), Err(Refusal::WatermarkRegressed));
|
|
116
|
+
assert_eq!(join.side_watermark(Side::Right), right);
|
|
117
|
+
assert_eq!(join.watermark(Side::Right), Some(15));
|
|
118
|
+
// Right 15 ends left 100 (it reaches 8..=15), but a replay of it would not be
|
|
119
|
+
// late yet: it stays until the left watermark reaches it too.
|
|
120
|
+
assert_eq!(join.retirable(), 0);
|
|
121
|
+
assert_eq!(join.advance_watermark(Side::Left, 10), Ok(Advancing::Advanced));
|
|
122
|
+
// Left 100 goes, and so does right 1: it reaches left times 3..=10.
|
|
123
|
+
assert_eq!(join.retirable(), 2);
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
## Contract
|
|
127
|
+
|
|
128
|
+
Write `wl` and `wr` for the left and right watermarks (unset until first
|
|
129
|
+
advanced) and `⊕` for addition saturating at `u64::MAX`.
|
|
130
|
+
|
|
131
|
+
- **Match.** Left `l` and right `r` match when the keys are equal and
|
|
132
|
+
`l.timestamp - before <= r.timestamp <= l.timestamp + after`.
|
|
133
|
+
- **Late.** `insert` of a record at `t` that is not held answers `Late` when
|
|
134
|
+
its own side's watermark is set and `t <= w_own`, and on no other count. The
|
|
135
|
+
opposite watermark never refuses a record. Watermarks never move back, so a
|
|
136
|
+
record that is `Late` once is `Late` for good.
|
|
137
|
+
- **Nothing matchable is refused.** A record whose partner is held is admitted
|
|
138
|
+
and paired however far the two watermarks have drifted apart, unless its own
|
|
139
|
+
side has declared it late (or the side is `Full`, or pairs are `Pending`).
|
|
140
|
+
- **Retire.** A held record at `t` is retirable when both hold:
|
|
141
|
+
1. *Nothing can match it any more.* The opposite watermark has reached the
|
|
142
|
+
end of its reach: `wr >= t ⊕ after` for a left record, `wl >= t ⊕ before`
|
|
143
|
+
for a right one. An opposite record is admitted only strictly above its
|
|
144
|
+
own watermark, so every one still to come lies past the reach. The
|
|
145
|
+
comparison is inclusive for that reason, and exact: one short, a record
|
|
146
|
+
at the end of the reach is still admitted and matches. A reach that
|
|
147
|
+
saturates is reached only by a watermark of `u64::MAX`, where no opposite
|
|
148
|
+
record can be admitted at all.
|
|
149
|
+
2. *A replay of it is late.* Its own watermark has reached it: `wl >= t`
|
|
150
|
+
for a left record, `wr >= t` for a right one. This is `insert`'s `Late`
|
|
151
|
+
test itself, so the two cannot disagree.
|
|
152
|
+
|
|
153
|
+
`retire` and `retirable` use this one test. With `Bounds::new(0, 0)` a
|
|
154
|
+
record at `t` is retirable once both watermarks are at or past `t`.
|
|
155
|
+
- **A lagging side keeps its records.** A record is held until its *own*
|
|
156
|
+
side's watermark reaches it, whatever the opposite one says: a side whose
|
|
157
|
+
watermark is never advanced never retires anything. Memory stays bounded all
|
|
158
|
+
the same: the side fills and `insert` answers `Full`. Advance both
|
|
159
|
+
watermarks to release records.
|
|
160
|
+
- **Exactly once, and complete.** As long as an identity names one record
|
|
161
|
+
(same side, key and timestamp) for the life of the join: no `(left, right)`
|
|
162
|
+
pair is drained twice, because a pair is owed only when one of its records
|
|
163
|
+
is `Accepted`, a held record replayed is `Duplicate`, and a retired one is
|
|
164
|
+
`Late`; and every two accepted records that match are drained as a pair,
|
|
165
|
+
because the earlier one is still held when the later one is accepted
|
|
166
|
+
(rule 1 kept it).
|
|
167
|
+
- **No clock.** Every time here is event time; the core reads and records no
|
|
168
|
+
monotonic clock. An honoured `advance_watermark`/`advance_watermark_from`
|
|
169
|
+
records the side's watermark (progress, not a clock) on `Advanced` and
|
|
170
|
+
nothing on `Repeated`.
|
|
171
|
+
- **No unreachable arms.** No `panic!`, `assert!`, `unreachable!` or `expect`
|
|
172
|
+
outside tests. The owed pair carries the opposite record it names, so
|
|
173
|
+
`drain` never looks a slot up again.
|
|
174
|
+
- **Refusals change nothing**, and none is remembered: each is an answer about
|
|
175
|
+
the join as it is now. `Late` stays `Late` only because watermarks, which
|
|
176
|
+
honoured calls record, never regress.
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative 'join_machines/version'
|
|
4
|
+
|
|
5
|
+
module JoinMachines
|
|
6
|
+
# Every value below is checked by the core when it is built, by hand or by
|
|
7
|
+
# the gem, `with` included: TypeError for a wrong type, RangeError for a
|
|
8
|
+
# value the core rejects. `hook` names the private native check, handed the
|
|
9
|
+
# value itself once stored: it reads the members from the struct, never
|
|
10
|
+
# through `deconstruct` or a reader, so neither can lie to it.
|
|
11
|
+
module Checked
|
|
12
|
+
def self.by(hook)
|
|
13
|
+
Module.new do
|
|
14
|
+
define_method(:initialize) do |**fields|
|
|
15
|
+
super(**fields)
|
|
16
|
+
JoinMachines.__send__(hook, self)
|
|
17
|
+
end
|
|
18
|
+
end
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
# Inclusive range of opposite-side event times a record can match. Like the
|
|
23
|
+
# core's `Reach::new` it holds Integers in 0..2**64-1 with `first <= last`.
|
|
24
|
+
# `include?(timestamp)` is native (the core's `contains`).
|
|
25
|
+
Reach = Data.define(:first, :last) { include Checked.by(:__reach) }
|
|
26
|
+
|
|
27
|
+
# Interval bounds: a pair matches when `left - before <= right <= left + after`
|
|
28
|
+
# in event time. Both are Integers in 0..2**64-1. `reach(side, timestamp)` is
|
|
29
|
+
# native: the opposite-side timestamps reachable from a record on `side`, a
|
|
30
|
+
# Reach saturating at the edges of 0..2**64-1.
|
|
31
|
+
Bounds = Data.define(:before, :after) { include Checked.by(:__bounds) }
|
|
32
|
+
|
|
33
|
+
# A matched pair of record ids sharing a join key; Integers in 0..2**64-1.
|
|
34
|
+
Pair = Data.define(:key, :left, :right) { include Checked.by(:__pair) }
|
|
35
|
+
# Join metadata for one externally stored record; Integers in 0..2**64-1.
|
|
36
|
+
Record = Data.define(:id, :key, :timestamp) { include Checked.by(:__record) }
|
|
37
|
+
# A record the join dropped; its payload can be released. `side` is :left or
|
|
38
|
+
# :right, `record` a Record.
|
|
39
|
+
Retired = Data.define(:side, :record) { include Checked.by(:__retired) }
|
|
40
|
+
|
|
41
|
+
# One input's event-time progress, as `Join#side_watermark` answers it: a
|
|
42
|
+
# frozen copy whose `value` is an Integer in 0..2**64-1, or nil before the
|
|
43
|
+
# first watermark. It reads the way `Join#advance_watermark_from` expects, so
|
|
44
|
+
# it can be handed to another join; it does not advance.
|
|
45
|
+
Watermark = Data.define(:value) { include Checked.by(:__watermark) }
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
# Native is required: no algorithm fallback. It validates and builds the Data
|
|
49
|
+
# values above, resolves their classes once at load and adds `Bounds#reach`
|
|
50
|
+
# and `Reach#include?`, so they must exist first.
|
|
51
|
+
require 'join_machines_native/join_machines_native'
|
|
52
|
+
|
|
53
|
+
module JoinMachines
|
|
54
|
+
private_class_method :__reach, :__bounds, :__pair, :__record, :__retired, :__watermark
|
|
55
|
+
private_constant :Checked
|
|
56
|
+
|
|
57
|
+
# Sealed: no subclass can skip the check by overriding `initialize`, and no
|
|
58
|
+
# bare `allocate` builds a value that never ran it. Sealing stops
|
|
59
|
+
# accidents, not reflection, so the native side reads only instances of
|
|
60
|
+
# exactly these classes and re-validates their members through the core.
|
|
61
|
+
[Reach, Bounds, Pair, Record, Retired, Watermark].each do |value|
|
|
62
|
+
value.singleton_class.undef_method(:allocate)
|
|
63
|
+
value.define_singleton_method(:inherited) { |_| raise TypeError, "#{value} cannot be subclassed" }
|
|
64
|
+
end
|
|
65
|
+
end
|
|
Binary file
|
|
@@ -0,0 +1,140 @@
|
|
|
1
|
+
module JoinMachines
|
|
2
|
+
VERSION: String
|
|
3
|
+
# Live records each input holds.
|
|
4
|
+
CAPACITY: Integer
|
|
5
|
+
|
|
6
|
+
type side = :left | :right
|
|
7
|
+
|
|
8
|
+
# Refusals: the call was not honoured and changed nothing. Every reason has
|
|
9
|
+
# its own symbol; there is no catch-all.
|
|
10
|
+
type refusal = :mismatch | :pending | :late | :full | :watermark_regressed
|
|
11
|
+
|
|
12
|
+
# The core's `Advancing`: what an accepted watermark did.
|
|
13
|
+
type advancing = :advanced | :repeated
|
|
14
|
+
|
|
15
|
+
# The core's `Side::other`: the opposite input.
|
|
16
|
+
def self.other: (side) -> side
|
|
17
|
+
|
|
18
|
+
# Every Data value is checked by the core when built, `with` included:
|
|
19
|
+
# TypeError for a wrong type, RangeError for a value the core rejects
|
|
20
|
+
# (an Integer outside 0..2**64-1, a side other than :left or :right).
|
|
21
|
+
# Every value class is sealed: no `allocate`, subclassing raises TypeError.
|
|
22
|
+
# Where the binding reads one back it takes only an instance of exactly the
|
|
23
|
+
# class (TypeError otherwise) and re-validates its members (RangeError).
|
|
24
|
+
|
|
25
|
+
# Mirrors the core's `Reach`, validated by `Reach::new`: RangeError unless
|
|
26
|
+
# first <= last.
|
|
27
|
+
class Reach < Data
|
|
28
|
+
attr_reader first: Integer
|
|
29
|
+
attr_reader last: Integer
|
|
30
|
+
def self.new: (Integer first, Integer last) -> Reach
|
|
31
|
+
| (first: Integer, last: Integer) -> Reach
|
|
32
|
+
# The core's `contains`.
|
|
33
|
+
def include?: (Integer timestamp) -> bool
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
# Mirrors the core's `Bounds`: any two Integers in 0..2**64-1.
|
|
37
|
+
class Bounds < Data
|
|
38
|
+
attr_reader before: Integer
|
|
39
|
+
attr_reader after: Integer
|
|
40
|
+
def self.new: (Integer before, Integer after) -> Bounds
|
|
41
|
+
| (before: Integer, after: Integer) -> Bounds
|
|
42
|
+
def reach: (side, Integer timestamp) -> Reach
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
# Mirrors the core's `Pair`: any three Integers in 0..2**64-1.
|
|
46
|
+
class Pair < Data
|
|
47
|
+
attr_reader key: Integer
|
|
48
|
+
attr_reader left: Integer
|
|
49
|
+
attr_reader right: Integer
|
|
50
|
+
def self.new: (Integer key, Integer left, Integer right) -> Pair
|
|
51
|
+
| (key: Integer, left: Integer, right: Integer) -> Pair
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
# Mirrors the core's `Record`: any three Integers in 0..2**64-1.
|
|
55
|
+
class Record < Data
|
|
56
|
+
attr_reader id: Integer
|
|
57
|
+
attr_reader key: Integer
|
|
58
|
+
attr_reader timestamp: Integer
|
|
59
|
+
def self.new: (Integer id, Integer key, Integer timestamp) -> Record
|
|
60
|
+
| (id: Integer, key: Integer, timestamp: Integer) -> Record
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
# Mirrors the core's `Retired`: TypeError unless `record` is a Record.
|
|
64
|
+
class Retired < Data
|
|
65
|
+
attr_reader side: side
|
|
66
|
+
attr_reader record: Record
|
|
67
|
+
def self.new: (side side, Record record) -> Retired
|
|
68
|
+
| (side: side, record: Record) -> Retired
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
# Anything that reads like WindowMachines::Watermark.
|
|
72
|
+
interface _Watermark
|
|
73
|
+
def value: () -> Integer?
|
|
74
|
+
end
|
|
75
|
+
|
|
76
|
+
# What `Join#side_watermark` answers: a frozen copy of one input's progress.
|
|
77
|
+
# `value` is nil before the first watermark, else an Integer in
|
|
78
|
+
# 0..2**64-1. It satisfies `_Watermark`.
|
|
79
|
+
class Watermark < Data
|
|
80
|
+
attr_reader value: Integer?
|
|
81
|
+
def self.new: (Integer? value) -> Watermark
|
|
82
|
+
| (value: Integer?) -> Watermark
|
|
83
|
+
end
|
|
84
|
+
|
|
85
|
+
class Join
|
|
86
|
+
def self.new: (Bounds bounds) -> Join
|
|
87
|
+
def bounds: () -> Bounds
|
|
88
|
+
def capacity: (side) -> Integer
|
|
89
|
+
# Live records held for one input (the core's `len`).
|
|
90
|
+
def size: (side) -> Integer
|
|
91
|
+
def empty?: () -> bool
|
|
92
|
+
# The core's `contains`.
|
|
93
|
+
def include?: (side, Integer id) -> bool
|
|
94
|
+
# Honoured: :accepted, or :duplicate (the core's `Inserting`) when exactly
|
|
95
|
+
# this record is already held on that side: nothing changes and no pair is
|
|
96
|
+
# owed twice. Refused: :mismatch (the id is held with another key or
|
|
97
|
+
# timestamp), :pending, :late (at or below its own side's watermark; the
|
|
98
|
+
# other side's watermark never makes a record late), :full. A held id is
|
|
99
|
+
# answered first.
|
|
100
|
+
def insert: (side, Record record) -> (:accepted | :duplicate | :mismatch | :pending | :late | :full)
|
|
101
|
+
def pending?: () -> bool
|
|
102
|
+
def pending_pairs: () -> Integer
|
|
103
|
+
# At most `max` owed pairs, [] when none is owed. Never refuses. A raise
|
|
104
|
+
# while building the answer drains nothing; FrozenError if building it
|
|
105
|
+
# froze the join, RuntimeError if it changed it; nothing drained.
|
|
106
|
+
def drain: (Integer max) -> Array[Pair]
|
|
107
|
+
# Honoured: :advanced, and :repeated when the side already held exactly
|
|
108
|
+
# `value` (nothing changes). Refused: :watermark_regressed.
|
|
109
|
+
def advance_watermark: (side, Integer value) -> (advancing | :watermark_regressed)
|
|
110
|
+
# Reads this gem's Watermark natively (TypeError for a subclass of it),
|
|
111
|
+
# any other object through `value`; FrozenError, nothing advanced, if
|
|
112
|
+
# `value` froze the join. Honoured: :advanced, and :repeated when the side
|
|
113
|
+
# already equals it, a nil value against a side with no watermark
|
|
114
|
+
# included. Refused: :watermark_regressed, a nil value against a side that
|
|
115
|
+
# holds one included.
|
|
116
|
+
def advance_watermark_from: (side, _Watermark watermark) -> (advancing | :watermark_regressed)
|
|
117
|
+
# The side's watermark as an Integer, nil before the first.
|
|
118
|
+
def watermark: (side) -> Integer?
|
|
119
|
+
# The same state as a Watermark value, never nil itself.
|
|
120
|
+
def side_watermark: (side) -> Watermark
|
|
121
|
+
# Held records both watermarks have finished: the other side's is at or
|
|
122
|
+
# past the end of the record's reach, and its own is at or past its
|
|
123
|
+
# timestamp (so a replay of it is :late).
|
|
124
|
+
def retirable: () -> Integer
|
|
125
|
+
# Honoured: the retired records, [] when none is retirable. Refused: :pending.
|
|
126
|
+
# A raise while building the answer retires nothing; FrozenError if
|
|
127
|
+
# building it froze the join, RuntimeError if it changed it; nothing
|
|
128
|
+
# retired.
|
|
129
|
+
def retire: (Integer max) -> (Array[Retired] | :pending)
|
|
130
|
+
end
|
|
131
|
+
|
|
132
|
+
# Private native hooks each Data value's `initialize` runs on the built
|
|
133
|
+
# value: nil, or TypeError/RangeError for one the core rejects.
|
|
134
|
+
private def self.__reach: (Reach value) -> nil
|
|
135
|
+
private def self.__bounds: (Bounds value) -> nil
|
|
136
|
+
private def self.__pair: (Pair value) -> nil
|
|
137
|
+
private def self.__record: (Record value) -> nil
|
|
138
|
+
private def self.__retired: (Retired value) -> nil
|
|
139
|
+
private def self.__watermark: (Watermark value) -> nil
|
|
140
|
+
end
|
metadata
ADDED
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
|
2
|
+
name: join_machines
|
|
3
|
+
version: !ruby/object:Gem::Version
|
|
4
|
+
version: 0.6.2
|
|
5
|
+
platform: arm64-darwin
|
|
6
|
+
authors:
|
|
7
|
+
- Abdelkader Boudih
|
|
8
|
+
autorequire:
|
|
9
|
+
bindir: bin
|
|
10
|
+
cert_chain: []
|
|
11
|
+
date: 2026-11-16 00:00:00.000000000 Z
|
|
12
|
+
dependencies: []
|
|
13
|
+
description: 'Matches left and right record IDs that share a key within a time window,
|
|
14
|
+
emits the pairs in resumable batches and retires records past the watermark. Metadata
|
|
15
|
+
only: not a stream processor or payload store.'
|
|
16
|
+
email:
|
|
17
|
+
- oss@seuros.com
|
|
18
|
+
executables: []
|
|
19
|
+
extensions: []
|
|
20
|
+
extra_rdoc_files: []
|
|
21
|
+
files:
|
|
22
|
+
- CHANGELOG.md
|
|
23
|
+
- LICENSE.txt
|
|
24
|
+
- README.md
|
|
25
|
+
- ext/join_machines_native/core/CHANGELOG.md
|
|
26
|
+
- ext/join_machines_native/core/README.md
|
|
27
|
+
- ext/join_machines_native/extconf.rb
|
|
28
|
+
- lib/join_machines.rb
|
|
29
|
+
- lib/join_machines/version.rb
|
|
30
|
+
- lib/join_machines_native/join_machines_native.bundle
|
|
31
|
+
- sig/join_machines.rbs
|
|
32
|
+
homepage: https://github.com/seuros/join_machines
|
|
33
|
+
licenses:
|
|
34
|
+
- MIT
|
|
35
|
+
metadata:
|
|
36
|
+
source_code_uri: https://github.com/seuros/join_machines
|
|
37
|
+
rubygems_mfa_required: 'true'
|
|
38
|
+
cargo_crate_name: join_machines_native
|
|
39
|
+
cargo_manifest_path: ext/join_machines_native/ffi/Cargo.toml
|
|
40
|
+
post_install_message:
|
|
41
|
+
rdoc_options: []
|
|
42
|
+
require_paths:
|
|
43
|
+
- lib
|
|
44
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
|
45
|
+
requirements:
|
|
46
|
+
- - ">="
|
|
47
|
+
- !ruby/object:Gem::Version
|
|
48
|
+
version: '4.0'
|
|
49
|
+
- - "<"
|
|
50
|
+
- !ruby/object:Gem::Version
|
|
51
|
+
version: 4.1.dev
|
|
52
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
53
|
+
requirements:
|
|
54
|
+
- - ">="
|
|
55
|
+
- !ruby/object:Gem::Version
|
|
56
|
+
version: '0'
|
|
57
|
+
requirements: []
|
|
58
|
+
rubygems_version: 3.5.23
|
|
59
|
+
signing_key:
|
|
60
|
+
specification_version: 4
|
|
61
|
+
summary: Bounded two-input interval-join bookkeeping for Ruby
|
|
62
|
+
test_files: []
|