@kairos-es/read 0.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +28 -0
- package/README.md +538 -0
- package/dist/cjs/EventLogDurability.js +184 -0
- package/dist/cjs/EventLogDurability.js.map +1 -0
- package/dist/cjs/ProjectionRunner.js +478 -0
- package/dist/cjs/ProjectionRunner.js.map +1 -0
- package/dist/cjs/ProjectionStore.js +233 -0
- package/dist/cjs/ProjectionStore.js.map +1 -0
- package/dist/cjs/foldIntoRef.js +36 -0
- package/dist/cjs/foldIntoRef.js.map +1 -0
- package/dist/cjs/inMemoryProjectionStore.js +138 -0
- package/dist/cjs/inMemoryProjectionStore.js.map +1 -0
- package/dist/cjs/index.js +128 -0
- package/dist/cjs/index.js.map +1 -0
- package/dist/cjs/projectionWiringFault.js +532 -0
- package/dist/cjs/projectionWiringFault.js.map +1 -0
- package/dist/cjs/runProjection.js +117 -0
- package/dist/cjs/runProjection.js.map +1 -0
- package/dist/cjs/runProjections.js +144 -0
- package/dist/cjs/runProjections.js.map +1 -0
- package/dist/cjs/superviseOnProgress.js +580 -0
- package/dist/cjs/superviseOnProgress.js.map +1 -0
- package/dist/cjs/testing.js +143 -0
- package/dist/cjs/testing.js.map +1 -0
- package/dist/dts/EventLogDurability.d.ts +182 -0
- package/dist/dts/EventLogDurability.d.ts.map +1 -0
- package/dist/dts/ProjectionRunner.d.ts +557 -0
- package/dist/dts/ProjectionRunner.d.ts.map +1 -0
- package/dist/dts/ProjectionStore.d.ts +475 -0
- package/dist/dts/ProjectionStore.d.ts.map +1 -0
- package/dist/dts/foldIntoRef.d.ts +39 -0
- package/dist/dts/foldIntoRef.d.ts.map +1 -0
- package/dist/dts/inMemoryProjectionStore.d.ts +11 -0
- package/dist/dts/inMemoryProjectionStore.d.ts.map +1 -0
- package/dist/dts/index.d.ts +185 -0
- package/dist/dts/index.d.ts.map +1 -0
- package/dist/dts/projectionWiringFault.d.ts +260 -0
- package/dist/dts/projectionWiringFault.d.ts.map +1 -0
- package/dist/dts/runProjection.d.ts +185 -0
- package/dist/dts/runProjection.d.ts.map +1 -0
- package/dist/dts/runProjections.d.ts +480 -0
- package/dist/dts/runProjections.d.ts.map +1 -0
- package/dist/dts/superviseOnProgress.d.ts +587 -0
- package/dist/dts/superviseOnProgress.d.ts.map +1 -0
- package/dist/dts/testing.d.ts +207 -0
- package/dist/dts/testing.d.ts.map +1 -0
- package/dist/esm/EventLogDurability.js +175 -0
- package/dist/esm/EventLogDurability.js.map +1 -0
- package/dist/esm/ProjectionRunner.js +468 -0
- package/dist/esm/ProjectionRunner.js.map +1 -0
- package/dist/esm/ProjectionStore.js +223 -0
- package/dist/esm/ProjectionStore.js.map +1 -0
- package/dist/esm/foldIntoRef.js +29 -0
- package/dist/esm/foldIntoRef.js.map +1 -0
- package/dist/esm/inMemoryProjectionStore.js +131 -0
- package/dist/esm/inMemoryProjectionStore.js.map +1 -0
- package/dist/esm/index.js +185 -0
- package/dist/esm/index.js.map +1 -0
- package/dist/esm/package.json +4 -0
- package/dist/esm/projectionWiringFault.js +524 -0
- package/dist/esm/projectionWiringFault.js.map +1 -0
- package/dist/esm/runProjection.js +109 -0
- package/dist/esm/runProjection.js.map +1 -0
- package/dist/esm/runProjections.js +137 -0
- package/dist/esm/runProjections.js.map +1 -0
- package/dist/esm/superviseOnProgress.js +571 -0
- package/dist/esm/superviseOnProgress.js.map +1 -0
- package/dist/esm/testing.js +133 -0
- package/dist/esm/testing.js.map +1 -0
- package/package.json +41 -0
- package/src/EventLogDurability.ts +201 -0
- package/src/ProjectionRunner.ts +923 -0
- package/src/ProjectionStore.ts +528 -0
- package/src/foldIntoRef.ts +63 -0
- package/src/inMemoryProjectionStore.ts +163 -0
- package/src/index.ts +218 -0
- package/src/projectionWiringFault.ts +694 -0
- package/src/runProjection.ts +270 -0
- package/src/runProjections.ts +623 -0
- package/src/superviseOnProgress.ts +897 -0
- package/src/testing.ts +290 -0
- package/testing/package.json +6 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
BSD 3-Clause License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026, Neverbland
|
|
4
|
+
|
|
5
|
+
Redistribution and use in source and binary forms, with or without
|
|
6
|
+
modification, are permitted provided that the following conditions are met:
|
|
7
|
+
|
|
8
|
+
1. Redistributions of source code must retain the above copyright notice, this
|
|
9
|
+
list of conditions and the following disclaimer.
|
|
10
|
+
|
|
11
|
+
2. Redistributions in binary form must reproduce the above copyright notice,
|
|
12
|
+
this list of conditions and the following disclaimer in the documentation
|
|
13
|
+
and/or other materials provided with the distribution.
|
|
14
|
+
|
|
15
|
+
3. Neither the name of the copyright holder nor the names of its
|
|
16
|
+
contributors may be used to endorse or promote products derived from
|
|
17
|
+
this software without specific prior written permission.
|
|
18
|
+
|
|
19
|
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
|
20
|
+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
21
|
+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
|
22
|
+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
|
23
|
+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
24
|
+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
|
25
|
+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
|
26
|
+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
|
27
|
+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
28
|
+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
package/README.md
ADDED
|
@@ -0,0 +1,538 @@
|
|
|
1
|
+
# @kairos-es/read
|
|
2
|
+
|
|
3
|
+
The kairos-es read side: a query-driven projection runner and the
|
|
4
|
+
`ProjectionStore` checkpoint port with its in-memory implementation (ADR-0007).
|
|
5
|
+
|
|
6
|
+
Platform-neutral — `effect`, `@kairos-es/core` and `@kairos-es/codec` are peer
|
|
7
|
+
dependencies and nothing else, so the all-in-memory permutation — a property of one
|
|
8
|
+
materialisation's (log, view) pairing — needs no database. The durable
|
|
9
|
+
`ProjectionStore` lives in `@kairos-es/read-postgres`.
|
|
10
|
+
|
|
11
|
+
Everything here is the read side and only the read side. `retryOnConflict`, the
|
|
12
|
+
decider-level retry for `AppendConditionFailed` (ADR-0008), is a **write**-side
|
|
13
|
+
combinator and lives in [`@kairos-es/core`](../core) beside the error it matches,
|
|
14
|
+
so retrying your own appends never means depending on this package.
|
|
15
|
+
|
|
16
|
+
## Wiring a read model
|
|
17
|
+
|
|
18
|
+
A read model is a view store bound to the `CheckpointKey` it advances
|
|
19
|
+
(`forKey(store, key)`), the codec slices whose union query drives its
|
|
20
|
+
subscription, and a per-batch `apply`. `runProjection` forks the daemon into the
|
|
21
|
+
enclosing `Scope`; `projectionLayer` is the same thing as a `Layer` whose scope
|
|
22
|
+
owns it. Both also require an `EventLogDurability` declaration, provided **once**
|
|
23
|
+
beside the `DcbEventStore` layer.
|
|
24
|
+
|
|
25
|
+
```ts
|
|
26
|
+
import { event, projection, SerializerDefault } from '@kairos-es/codec'
|
|
27
|
+
import {
|
|
28
|
+
composeProjections,
|
|
29
|
+
DcbEventStoreInMemory,
|
|
30
|
+
tagIdentity,
|
|
31
|
+
} from '@kairos-es/core'
|
|
32
|
+
import {
|
|
33
|
+
checkpointKey,
|
|
34
|
+
EphemeralEventLog,
|
|
35
|
+
foldIntoRef,
|
|
36
|
+
forKey,
|
|
37
|
+
makeInMemoryProjectionStore,
|
|
38
|
+
ProjectionId,
|
|
39
|
+
projectionLayer,
|
|
40
|
+
runProjection,
|
|
41
|
+
} from '@kairos-es/read'
|
|
42
|
+
import { Effect, Layer, Ref, Schema } from 'effect'
|
|
43
|
+
|
|
44
|
+
const CourseId = tagIdentity('course')
|
|
45
|
+
|
|
46
|
+
const SeatTaken = event('SeatTaken', { courseId: Schema.String }, (payload) => [
|
|
47
|
+
CourseId.make(payload.courseId),
|
|
48
|
+
])
|
|
49
|
+
|
|
50
|
+
// Tag-narrowed, never type-only: every read-model slice carries at least one tag,
|
|
51
|
+
// so the composed subscription's union query stays servable.
|
|
52
|
+
const slices = {
|
|
53
|
+
seats: projection<number>(0)
|
|
54
|
+
.on(SeatTaken, (count) => count + 1)
|
|
55
|
+
.build({ tags: [CourseId.make('c1')] }),
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
// The read model, plus the view it materialises into. The view is returned
|
|
59
|
+
// alongside it because a read model is queried through its VIEW, never through
|
|
60
|
+
// the runner — `projectionLayer` deliberately provides nothing.
|
|
61
|
+
const makeCourseSeats = Effect.gen(function* () {
|
|
62
|
+
const store = yield* makeInMemoryProjectionStore
|
|
63
|
+
const view = yield* Ref.make(composeProjections(slices).initialState)
|
|
64
|
+
return {
|
|
65
|
+
view,
|
|
66
|
+
readModel: {
|
|
67
|
+
// `ProjectionStore` is multi-key, because one store instance serves many
|
|
68
|
+
// read models. `forKey` binds it to this one's checkpoint, so `commit` has
|
|
69
|
+
// no key argument left to get wrong — and the read model carries the pair
|
|
70
|
+
// rather than two fields that could disagree.
|
|
71
|
+
store: forKey(store, checkpointKey(ProjectionId.make('course-seats'))),
|
|
72
|
+
slices,
|
|
73
|
+
// The folded-state shape: fold the batch purely, then ONE write. This is
|
|
74
|
+
// what a non-transactional view store's contract requires.
|
|
75
|
+
apply: foldIntoRef(slices, view),
|
|
76
|
+
},
|
|
77
|
+
}
|
|
78
|
+
})
|
|
79
|
+
|
|
80
|
+
// `EphemeralEventLog` sits BESIDE the store layer, because what it declares is a
|
|
81
|
+
// fact about the LOG, not about any one read model: `DcbEventStore` exposes no
|
|
82
|
+
// durability, so the application that chose the layer is the only honest source,
|
|
83
|
+
// and one declaration here satisfies every runner in the graph. Swap it for
|
|
84
|
+
// `DurableEventLog` when the log is.
|
|
85
|
+
const inMemory = Layer.mergeAll(
|
|
86
|
+
DcbEventStoreInMemory,
|
|
87
|
+
EphemeralEventLog,
|
|
88
|
+
SerializerDefault,
|
|
89
|
+
)
|
|
90
|
+
|
|
91
|
+
// Either fork the daemon into a scope you own — catch-up starts at once, and
|
|
92
|
+
// closing the scope interrupts it and waits for it...
|
|
93
|
+
const program = Effect.scoped(
|
|
94
|
+
Effect.gen(function* () {
|
|
95
|
+
const { readModel, view } = yield* makeCourseSeats
|
|
96
|
+
yield* runProjection(readModel)
|
|
97
|
+
// ...append through `DcbEventStore` and let the runner catch up, then read
|
|
98
|
+
// the materialised view.
|
|
99
|
+
return yield* Ref.get(view)
|
|
100
|
+
}),
|
|
101
|
+
).pipe(Effect.provide(inMemory))
|
|
102
|
+
|
|
103
|
+
// ...or let a Layer own it, so the daemon starts when the application's layer
|
|
104
|
+
// graph is built and is interrupted when the graph is released.
|
|
105
|
+
//
|
|
106
|
+
// `onStalled` is wired here rather than left out, because this shape DISCARDS the
|
|
107
|
+
// runner: `projectionLayer` provides nothing, so there is no fibre to await and a
|
|
108
|
+
// `ProjectionStalled` would otherwise reach you only as a log line. The restart is
|
|
109
|
+
// the metric; the stall is the page.
|
|
110
|
+
|
|
111
|
+
// Read by this process's readiness probe.
|
|
112
|
+
const stalledProjections = new Set<string>()
|
|
113
|
+
|
|
114
|
+
const courseSeats = Layer.unwrapEffect(
|
|
115
|
+
Effect.map(makeCourseSeats, ({ readModel }) =>
|
|
116
|
+
projectionLayer(readModel, {
|
|
117
|
+
// Whatever "a human must look at this" means in your deployment: fail a
|
|
118
|
+
// health check, exit non-zero, fire a pager. Infallible by signature, so it
|
|
119
|
+
// can neither fail the daemon nor widen its requirements.
|
|
120
|
+
onStalled: ({ key }) =>
|
|
121
|
+
Effect.sync(() => stalledProjections.add(key.projection)),
|
|
122
|
+
}),
|
|
123
|
+
),
|
|
124
|
+
)
|
|
125
|
+
|
|
126
|
+
const main = Layer.launch(Layer.provide(courseSeats, inMemory))
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
## Testing a read model: `@kairos-es/read/testing`
|
|
130
|
+
|
|
131
|
+
A projection is a **daemon**, so `runProjection` returns as soon as the fibre is
|
|
132
|
+
forked — which is the only honest thing it can do, but it means every test of a
|
|
133
|
+
read model needs one more step: *run this until it has consumed the log, then
|
|
134
|
+
assert the view*. A published subpath ships that step, so nobody has to invent it
|
|
135
|
+
(and nobody has to guess a `sleep`, which is wrong when it is short and expensive
|
|
136
|
+
when it is long).
|
|
137
|
+
|
|
138
|
+
```ts
|
|
139
|
+
import { runProjectionUntil, storedCheckpoint } from '@kairos-es/read/testing'
|
|
140
|
+
|
|
141
|
+
const { readModel, view } = yield* makeCourseSeats
|
|
142
|
+
const positions = yield* appendSomeEvents
|
|
143
|
+
|
|
144
|
+
// Fork the daemon, then wait until the stored checkpoint reaches that position.
|
|
145
|
+
// Returns the `ProjectionRunner`, so `Fiber.poll`/`Fiber.interrupt` stay available.
|
|
146
|
+
yield* runProjectionUntil(readModel, lastPosition(positions), {
|
|
147
|
+
// The runner's own tuning and the wait's, in ONE flat object.
|
|
148
|
+
batchSize: BatchSize.make(2),
|
|
149
|
+
batchWindow: '5 millis',
|
|
150
|
+
timeout: '20 seconds',
|
|
151
|
+
})
|
|
152
|
+
|
|
153
|
+
expect(yield* Ref.get(view)).toEqual(/* … */)
|
|
154
|
+
expect(yield* storedCheckpoint(readModel.store)).toBe(lastPosition(positions))
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
`awaitCheckpoint(readModel.store, target, options?)` is the wait on its own, for a
|
|
158
|
+
projection started some other way (`projectionLayer`, a scope you own). It POLLS
|
|
159
|
+
the stored checkpoint rather than sleeping, so a slow machine takes longer instead
|
|
160
|
+
of failing, and a genuine hang becomes a failed assertion naming the position that
|
|
161
|
+
was never reached. `storedCheckpoint` reads it once, and `lastPosition` turns what
|
|
162
|
+
`append` reported into a target, loudly, rather than yielding `undefined` from an
|
|
163
|
+
empty fixture.
|
|
164
|
+
|
|
165
|
+
`awaitCheckpoint` and `storedCheckpoint` take the **keyed** store a read model
|
|
166
|
+
already carries (`readModel.store`) and `runProjectionUntil` takes the read model
|
|
167
|
+
itself, so there is no key to pass and none to transpose; `lastPosition` is a pure
|
|
168
|
+
function over what `append` returned, and takes no store at all. It adds no peer
|
|
169
|
+
dependency — no test framework, no `expect`, just `Effect`s (and that one pure
|
|
170
|
+
function) you run under whatever runner you already have — which is why it is a
|
|
171
|
+
subpath of this package rather than a package of its own. It is deliberately a
|
|
172
|
+
**testing** surface: there is no lag reporting and no production "wait until my
|
|
173
|
+
write is projected" hook, because a read model is eventually consistent by
|
|
174
|
+
construction.
|
|
175
|
+
|
|
176
|
+
## One store, many read models — one read model, one key per store
|
|
177
|
+
|
|
178
|
+
`ProjectionStore` is **multi-key**: `readCheckpoint(key)`, `resetCheckpoint(key)`
|
|
179
|
+
and `commit({ key, … })`, because one store instance (one `SqlClient`, one pool)
|
|
180
|
+
serves however many read models an application runs, and keeping their keys
|
|
181
|
+
independent is contract behaviour the shared suite proves against every backend.
|
|
182
|
+
|
|
183
|
+
A read model has exactly **one** key, so it is given `forKey(store, key)`: the
|
|
184
|
+
same three operations with the key already supplied, so there is none to
|
|
185
|
+
transpose, omit, or read from the wrong variable. `ReadModel` carries that bound
|
|
186
|
+
store **instead of** a `key` field, which is what turns "one materialisation, one
|
|
187
|
+
key, one runner" from a convention every wiring happened to follow into something a
|
|
188
|
+
wiring cannot express otherwise. The key is still readable as `store.key` — the
|
|
189
|
+
runner's log annotations and its `ProjectionStalled` payload name it.
|
|
190
|
+
|
|
191
|
+
A key is unique **within one store**, which is a consequence of R1 rather than a
|
|
192
|
+
policy: the checkpoint is co-located with its view so the pair commits atomically,
|
|
193
|
+
so a key is only ever resolved against the store holding it and there is no global
|
|
194
|
+
namespace to collide in. That is the direction this heading did not use to name.
|
|
195
|
+
ONE read model materialised in several stores carries ONE `ProjectionId` across
|
|
196
|
+
them — that shared id is what its single identity means, and it is what the
|
|
197
|
+
graduation case asserts as its acceptance criterion. Distinctness is forced only
|
|
198
|
+
where two materialisations of one read model cohabit a store, which is what
|
|
199
|
+
`PartitionId` is for: two of them in one store under one key would trade cursors,
|
|
200
|
+
each advancing the other's, and the wiring gate refuses that pairing at
|
|
201
|
+
construction. That rung is the one only `runProjections` can reach, a single
|
|
202
|
+
materialisation having no sibling to collide with.
|
|
203
|
+
|
|
204
|
+
`forKey` is a **free function over the port**, not a method on it: one
|
|
205
|
+
implementation every backend gets for nothing, a port that stays three methods,
|
|
206
|
+
and — the sharp reason — a store **decorator** (`{ ...inner, commit: … }`) cannot
|
|
207
|
+
copy a `forKey` that closed over the undecorated store and silently route past
|
|
208
|
+
its own decoration.
|
|
209
|
+
|
|
210
|
+
Every read-model slice must carry at least one tag (a `system:…` tag for a
|
|
211
|
+
genuinely global slice), so a composed subscription's union query stays servable.
|
|
212
|
+
The wiring gate **checks this at construction**, for either entry point, and the
|
|
213
|
+
wiring is refused as a defect naming the convention — including for a lone tagless
|
|
214
|
+
slice, which the store's own servable grammar would
|
|
215
|
+
accept (a single type-only item is the indexed fast path) but which breaks the day
|
|
216
|
+
a second, tagged slice is composed with it.
|
|
217
|
+
|
|
218
|
+
The same wiring over a real domain, with the events coming from the write side's
|
|
219
|
+
own deciders rather than from hand-assembled fixtures, is
|
|
220
|
+
[`examples/course-subscriptions/src/courseRoster.ts`](../../examples/course-subscriptions/src/courseRoster.ts) —
|
|
221
|
+
a course's capacity, roster and seats remaining, run and resumed from its
|
|
222
|
+
checkpoint in
|
|
223
|
+
[`test/courseRoster.spec.ts`](../../examples/course-subscriptions/test/courseRoster.spec.ts).
|
|
224
|
+
|
|
225
|
+
## Graduating a slice from memory to Postgres
|
|
226
|
+
|
|
227
|
+
Because the view store is a **value** selected per MATERIALISATION, a slice that
|
|
228
|
+
started in an in-memory view graduates to a durable one with its **slices, its
|
|
229
|
+
composed query, its `CheckpointKey` and its runner untouched**. Two things change
|
|
230
|
+
and only two: the `store` it is handed, and the `apply` that writes through it —
|
|
231
|
+
`foldIntoRef` becomes the view's own statements, which also widens the read
|
|
232
|
+
model's `E` from `never` to whatever those writes can fail with. The runner is
|
|
233
|
+
generic in that `E` and needs no retuning; a fault from `apply` travels through
|
|
234
|
+
`commit` to the supervisor, which is what makes "retry with backoff" the response
|
|
235
|
+
to a flaky view store.
|
|
236
|
+
|
|
237
|
+
That is demonstrated rather than asserted, and it is the same roster both ways.
|
|
238
|
+
[`examples/course-subscriptions/src/courseRosterSql.ts`](../../examples/course-subscriptions/src/courseRosterSql.ts)
|
|
239
|
+
re-points that example's roster at Postgres — importing its slices, its key
|
|
240
|
+
derivation and its pure `courseRosterOf` verbatim — and is the repo's worked SQL
|
|
241
|
+
`apply`, hybrid on purpose: the capacity row's whole batch coalesces into one
|
|
242
|
+
upsert, while each seat is its own row and its own insert.
|
|
243
|
+
[`read-postgres/test/courseRosterGraduation.test.ts`](../read-postgres/test/courseRosterGraduation.test.ts)
|
|
244
|
+
then runs that one slice record into a `Ref` and into Postgres **side by side over
|
|
245
|
+
one durable log** — legal permutations 2 and 3 at once — and asserts the two
|
|
246
|
+
composed queries are equal, the two checkpoint keys are equal, and the two rosters
|
|
247
|
+
agree field for field.
|
|
248
|
+
|
|
249
|
+
So that pair demonstrates two things rather than one. Graduation is the sequential
|
|
250
|
+
reading — memory *then* Postgres, with the `store` and the `apply` the only things
|
|
251
|
+
that change. Running both **at once** is the other, and it is the section below. The
|
|
252
|
+
two wirings live in the example, which is where a consumer reads worked code and
|
|
253
|
+
where `courseRoster.ts` still needs no database at all; what lives in
|
|
254
|
+
`@kairos-es/read-postgres`'s test directory is the COMPARISON, because running that
|
|
255
|
+
needs a container and a live Postgres log.
|
|
256
|
+
|
|
257
|
+
## Several materialisations of one read model
|
|
258
|
+
|
|
259
|
+
Per-read-model view-store selection is a **floor rather than a ceiling**. One slice
|
|
260
|
+
record can feed a durable Postgres materialisation for the big queryable read model
|
|
261
|
+
and an ephemeral in-memory one for hot in-process reads, at the same time, over one
|
|
262
|
+
log.
|
|
263
|
+
|
|
264
|
+
R1 and R2 fix the shape: one atomic `commit` cannot span two stores, and one
|
|
265
|
+
checkpoint cannot carry two durability classes, so this is N checkpoints, N runners
|
|
266
|
+
and N subscriptions. They do not contend — each owns its checkpoint in its own
|
|
267
|
+
store, and the guarded compare-and-set arbitrates without any lease. Nor are they
|
|
268
|
+
shards: no stream is divided and no work is shared, so the sharding of one read
|
|
269
|
+
model that ADR-0007 refuses is not what this is.
|
|
270
|
+
|
|
271
|
+
`runProjections({ slices, projection, materialisations })` wires it, holding one
|
|
272
|
+
`slices` value and one `ProjectionId` for the whole call so the subscription query
|
|
273
|
+
and the decode table cannot diverge across materialisations. Each entry — a
|
|
274
|
+
`Materialisation` — keeps its own **unbound** `store`, its own `apply` and its own
|
|
275
|
+
tuning and supervision hooks, and the call applies `forKey` per entry itself. The
|
|
276
|
+
guarantee is exactly **one query, one decode, N applies**. What it does not hold is
|
|
277
|
+
the folds: a SQL `apply` re-implements the fold rather than reusing `foldIntoRef`,
|
|
278
|
+
so agreement between materialisations is something a test demonstrates rather than
|
|
279
|
+
something the type system gives you.
|
|
280
|
+
|
|
281
|
+
One consequence surprises, so it is worth stating plainly: keys are store-local, so
|
|
282
|
+
under one `ProjectionId` and the default partition the N materialisations'
|
|
283
|
+
`CheckpointKey`s are **identical**. That is the intended shape — a key names a
|
|
284
|
+
cursor *within* a store — and the discriminator is the store. Each runner carries
|
|
285
|
+
its own, so `runners[i].store` is entry *i*'s cursor and
|
|
286
|
+
`awaitCheckpoint(runners[i].store, target)` observes how far that one materialisation
|
|
287
|
+
has got, with no `forKey` for the caller to re-derive. What the store cannot reach is
|
|
288
|
+
a **hook**: `onRestart`/`onStalled` are handed a payload rather than a handle, and
|
|
289
|
+
`info.key` does not tell the entries apart, so which entry's hook fired is the
|
|
290
|
+
discriminator there — which is why the tuning and the hooks are per entry rather than
|
|
291
|
+
per call.
|
|
292
|
+
|
|
293
|
+
Two materialisations that would share one store **under the same resolved
|
|
294
|
+
`CheckpointKey`** are a **construction-time fault** — both conditions, since
|
|
295
|
+
sharing a store is perfectly legal under distinct partitions and equal keys across
|
|
296
|
+
DIFFERENT stores are the intended shape of a read model materialised twice. The fix
|
|
297
|
+
is usually that they were never two materialisations: several views in one store
|
|
298
|
+
want ONE MATERIALISATION whose single `apply` writes them all inside the one
|
|
299
|
+
`commit`, under one checkpoint. Use a distinct partition only where two
|
|
300
|
+
materialisations genuinely must cohabit a store.
|
|
301
|
+
|
|
302
|
+
What the gate cannot see is the other side of the same mistake: two entries whose
|
|
303
|
+
`apply`s write **the same view**. Different stores satisfy every rule it has, and
|
|
304
|
+
the view is opaque inside `apply`, so nothing refuses it — and both runners then
|
|
305
|
+
fold the whole stream into that one target under independent checkpoints, applying
|
|
306
|
+
every event twice. **One materialisation owns one view target**, and that one is on
|
|
307
|
+
you rather than on the gate. Every other rule the read side enforces runs in the
|
|
308
|
+
**same pass** — two of them per entry, two once over the shared query and slice
|
|
309
|
+
record — so a call refused for any reason at all has forked nothing, not even the
|
|
310
|
+
entries that were sound.
|
|
311
|
+
|
|
312
|
+
And if a read model is small enough to fold per query, consider not maintaining it
|
|
313
|
+
at all. `modelAtHead` from `@kairos-es/codec` reads the log at query time and is
|
|
314
|
+
consistent as of the `head` its read returned, which makes it MORE current than any
|
|
315
|
+
maintained view of the same slices — a maintained view is consistent as of a
|
|
316
|
+
checkpoint that trails that head. No table, no checkpoint, no runner, and no rung
|
|
317
|
+
of the construction gate applies to it.
|
|
318
|
+
|
|
319
|
+
All three over one course's roster:
|
|
320
|
+
[`examples/course-subscriptions/src/courseRosterMaterialisations.ts`](../../examples/course-subscriptions/src/courseRosterMaterialisations.ts),
|
|
321
|
+
whose
|
|
322
|
+
[`test/courseRosterMaterialisations.spec.ts`](../../examples/course-subscriptions/test/courseRosterMaterialisations.spec.ts)
|
|
323
|
+
is what establishes that the three folds agree — empirically, by comparing the
|
|
324
|
+
materialised figures, because nothing structural gives it.
|
|
325
|
+
|
|
326
|
+
## Supervision: retry while the checkpoint moves
|
|
327
|
+
|
|
328
|
+
The runner's supervisor is a module of its own — `superviseOnProgress`, which is
|
|
329
|
+
**package-internal and deliberately not exported**: it is this runner's supervisor,
|
|
330
|
+
not a general combinator on offer, and its signature says so (a `CheckpointKey`, a
|
|
331
|
+
`Position`, a `ProjectionStoreError`). What you tune is exported
|
|
332
|
+
(`ProjectionSupervisionOptions`, `MaxNoProgressRestarts`) and what you catch is
|
|
333
|
+
exported (`ProjectionStalled`); the mechanism between them is ours to change.
|
|
334
|
+
|
|
335
|
+
What it does is **retry an effect while an externally observed progress signal keeps
|
|
336
|
+
moving, and give up with `ProjectionStalled` when it stops.** For a projection the
|
|
337
|
+
signal is the stored checkpoint, re-read through `readCheckpoint` after every
|
|
338
|
+
failure. It is separate from the runner because supervision is the hardest thing
|
|
339
|
+
here to test through the whole pipeline — it knows nothing of `subscribe`, decoding,
|
|
340
|
+
batching or committing, so every subtle part of it is pinned against a stub progress
|
|
341
|
+
signal instead of an event store, a rigged view store and a hand-driven clock loop.
|
|
342
|
+
|
|
343
|
+
It splits two decisions that are usually conflated. The **schedule** decides how
|
|
344
|
+
fast to restart — a capped, jittered exponential that resets after
|
|
345
|
+
`restartResetAfter`, so a sustained outage sawtooths back to `restartMinDelay`
|
|
346
|
+
rather than pinning at the ceiling. The **circuit-breaker** decides whether to
|
|
347
|
+
keep restarting, by re-reading `progress` after every failure and counting only
|
|
348
|
+
the *consecutive* restarts that moved it not at all.
|
|
349
|
+
|
|
350
|
+
Keying on the signal rather than on a failure count is what makes a competing
|
|
351
|
+
writer behave: two processes accidentally maintaining one read model both lose
|
|
352
|
+
their guard on every batch, but the winner keeps advancing the shared checkpoint,
|
|
353
|
+
so every one of the loser's restarts scores as progress, its counter resets, and
|
|
354
|
+
it resynchronises at the schedule's bounded rate. Only a signal that is genuinely
|
|
355
|
+
stuck — a poison event, a persistently broken view store — trips the breaker. The
|
|
356
|
+
budget for that is sized in **wall clock**, roughly five minutes on the defaults,
|
|
357
|
+
so a view store that is merely *unavailable* is back inside it.
|
|
358
|
+
|
|
359
|
+
What that paragraph describes is one store under one key, and it is a **fault**: two
|
|
360
|
+
*deliberate* materialisations of one read model never reach it, their checkpoints
|
|
361
|
+
living in different stores with no shared cursor to contend for. That is why
|
|
362
|
+
`runProjections` refuses the one-store pairing at construction rather than leaving
|
|
363
|
+
this mechanism to absorb it — it would, indefinitely and silently, each
|
|
364
|
+
materialisation holding an arbitrary subset of the events.
|
|
365
|
+
|
|
366
|
+
Tuning is two documented option sets in **one flat object**.
|
|
367
|
+
`ProjectionPipelineOptions` (`batchSize`, `batchWindow`) sizes a transaction and a
|
|
368
|
+
latency ceiling; `ProjectionSupervisionOptions` (`restartMinDelay`,
|
|
369
|
+
`restartMaxDelay`, `restartResetAfter`, `maxNoProgressRestarts`, `onRestart`,
|
|
370
|
+
`onStalled`) sizes a recovery policy. `ProjectionRunnerOptions` is their union —
|
|
371
|
+
flat, because the ubiquitous `{ ...OPTIONS, oneField }` override would silently
|
|
372
|
+
drop its siblings under a nested shape. Every figure is bounded at construction:
|
|
373
|
+
the two counts are branded (`BatchSize.make(…)`, `MaxNoProgressRestarts.make(…)`),
|
|
374
|
+
and the four `Duration`s are rejected as defects before anything is forked —
|
|
375
|
+
including under `runProjections`, where every entry's figures are judged before the
|
|
376
|
+
first daemon exists. Retune `maxNoProgressRestarts` by the wall clock it buys, never
|
|
377
|
+
by the count.
|
|
378
|
+
|
|
379
|
+
## The restart is the metric, the stall is the page
|
|
380
|
+
|
|
381
|
+
The supervisor has two outcomes and one hook each, and they are not equally
|
|
382
|
+
important. `onRestart` fires on **every restart**, before the backoff sleep — the
|
|
383
|
+
outcome that resolves itself, so it is a rate to graph and to alert on only if it
|
|
384
|
+
climbs. `onStalled` fires **once**, at the give-up, immediately after the error is
|
|
385
|
+
logged and immediately before `ProjectionStalled` is raised — a poison event or a
|
|
386
|
+
view store broken rather than merely unavailable, both of which need a human.
|
|
387
|
+
|
|
388
|
+
**Wire `onStalled` in production.** The recommended shape is `projectionLayer`,
|
|
389
|
+
which provides nothing and therefore discards the daemon fibre — deliberately,
|
|
390
|
+
because a projection is a background process and a read model is queried through
|
|
391
|
+
its view — so there is nothing to `Fiber.await`, and a stall's only other route out
|
|
392
|
+
of the process is a single `Effect.logError`. The hook is what turns that into a
|
|
393
|
+
failed health check, a non-zero exit, or a page.
|
|
394
|
+
|
|
395
|
+
Its payload is the `ProjectionStalled` about to be raised, field for field and
|
|
396
|
+
value for value: `key`, `noProgressRestarts`, the stuck `position` (so the
|
|
397
|
+
offending event is one query away) and the raw `reason` rather than the string the
|
|
398
|
+
log line carries. Both hooks are `Effect<void>`, so neither can fail the daemon nor
|
|
399
|
+
widen its requirements — and neither traps a **defect**, which takes the fibre down
|
|
400
|
+
as any other bug would. The ordering is what makes that safe: the log line has
|
|
401
|
+
already landed by the time your hook is called, so a broken pager cannot buy
|
|
402
|
+
silence.
|
|
403
|
+
|
|
404
|
+
**The hooks are the primary channel; the log lines are a thin lossy default.**
|
|
405
|
+
Every annotation on the two supervision lines is a string or a number, on purpose
|
|
406
|
+
— JSON-safe rather than uniformly stringy. Be precise about which `bigint` is the
|
|
407
|
+
hazard, because the obvious reading is wrong: `effect@3.22`'s `Logger.json`
|
|
408
|
+
special-cases a **top-level** annotation value, so a bare `bigint` renders fine. A
|
|
409
|
+
`bigint` **nested inside** an annotation's object is the one that survives into the
|
|
410
|
+
single `JSON.stringify` of the whole record, with nothing catching the throw, and
|
|
411
|
+
raises `TypeError: Do not know how to serialize a BigInt` — losing the entire line
|
|
412
|
+
and taking the daemon with it as a defect. A raw fault is exactly such an object:
|
|
413
|
+
`CheckpointSuperseded` carries `expected: Position`. So the log line gets
|
|
414
|
+
`String(fault)` and the hooks get the fault untouched: anything that needs to match
|
|
415
|
+
on a tag, read a payload, or count by fault type takes `onRestart`/`onStalled`; the
|
|
416
|
+
log line is for everyone else.
|
|
417
|
+
|
|
418
|
+
**The label is the fault's, not the supervisor's.** `String(fault)` is the whole
|
|
419
|
+
reduction — there is no table of tag-to-label anywhere, and nothing to hand the
|
|
420
|
+
supervisor. `Error.prototype.toString` is `name: message`, and `Data.TaggedError`
|
|
421
|
+
sets `name` to the tag, so a fault that fills `message` arrives as
|
|
422
|
+
`ProjectionStoreError: connection reset` and one that does not arrives as its bare
|
|
423
|
+
tag. The library's own faults all fill it from the field that says why —
|
|
424
|
+
`ProjectionStoreError` and `CodecError` from `reason`, `PipelineDied` from its
|
|
425
|
+
`defect`, `CheckpointSuperseded` from `expected`, and `ProjectionStalled` from the
|
|
426
|
+
budget it spent and the position it is stuck at. A `bigint` field is no obstacle:
|
|
427
|
+
`message` is typed `string`, so interpolating one yields digits, and the hazard
|
|
428
|
+
above — which is about the annotation values a logger is handed — never arises
|
|
429
|
+
through it. **Your read model's `E` gets the same treatment**, so put a `message` on
|
|
430
|
+
any error you want to recognise in a log: it costs one getter over the field you
|
|
431
|
+
already have, and it pays in `Cause.pretty` and every other reader of that error,
|
|
432
|
+
not only here.
|
|
433
|
+
|
|
434
|
+
## R2: one log, one declaration
|
|
435
|
+
|
|
436
|
+
The three legal (event log, view store) pairings are all-in-memory, durable log
|
|
437
|
+
with an in-memory view, and durable log with a durable view. A **durable view
|
|
438
|
+
over an in-memory log is rejected at construction** (R2): its checkpoint would
|
|
439
|
+
outlive the log, leaving the view frozen and permanently stale without ever
|
|
440
|
+
erroring.
|
|
441
|
+
|
|
442
|
+
Each pairing is a property of ONE materialisation, not of a read model: a read
|
|
443
|
+
model materialised twice can have two of them in play at once over one log, which
|
|
444
|
+
is a deployment property rather than a fourth permutation.
|
|
445
|
+
|
|
446
|
+
The comparison's two inputs are declared in the two places that know them. The
|
|
447
|
+
view store carries its own class on the `ProjectionStore` it implements. The log's
|
|
448
|
+
is the **`EventLogDurability` service** — `DurableEventLog` or
|
|
449
|
+
`EphemeralEventLog`, provided beside the `DcbEventStore` layer — because
|
|
450
|
+
`DcbEventStore` deliberately exposes no durability and the application that chose
|
|
451
|
+
the layer is the only honest source.
|
|
452
|
+
|
|
453
|
+
A **context tag rather than a field on `ReadModel`**, because one log is *one*
|
|
454
|
+
fact. Authored per read model, it would be stated N times at N sites, N copies
|
|
455
|
+
could disagree over the identical store, and nothing anywhere could notice:
|
|
456
|
+
`runProjection` only ever sees the pair it was handed. Since horizontal scale here
|
|
457
|
+
is a runner per read model — and one read model materialised N ways is N runners
|
|
458
|
+
too — N > 1 is the expected case.
|
|
459
|
+
|
|
460
|
+
The cost is that `EventLogDurability` is a **required** input of `runProjection`,
|
|
461
|
+
`projectionLayer` and `runProjections` alike, with no default — and it is costliest
|
|
462
|
+
to forget at `runProjections`, where one missing layer refuses N materialisations at
|
|
463
|
+
once. That is deliberate in both directions:
|
|
464
|
+
defaulting to `'ephemeral'` would make the production wiring the one you have to
|
|
465
|
+
remember, and defaulting to `'durable'` would turn an omission into the frozen
|
|
466
|
+
view R2 exists to prevent. The store layers do not supply it either — the tag is a
|
|
467
|
+
read-side concept and `@kairos-es/core` knows nothing of the read side, a backend
|
|
468
|
+
declaring it would move the fact away from the only party that knows it, and two
|
|
469
|
+
providers of one tag resolve by merge order rather than loudly.
|
|
470
|
+
|
|
471
|
+
## More apply throughput in one runner: coalesce the batch per row
|
|
472
|
+
|
|
473
|
+
ADR-0007's escape hatch for when apply **throughput** rather than isolation is the
|
|
474
|
+
constraint (horizontal scale of one read model — sharding, competing consumers —
|
|
475
|
+
is deliberately not offered). It is a technique, not a library symbol, because all
|
|
476
|
+
of it is a **fold your read model performs over its own batch**: collapse each
|
|
477
|
+
touched row's N events into the one write that row needs, then issue those writes
|
|
478
|
+
inside the one `commit`. A batch of N events touching R rows costs R statements
|
|
479
|
+
rather than N. There is still one writer, one transaction and one checkpoint
|
|
480
|
+
advance.
|
|
481
|
+
|
|
482
|
+
Why per **row** and not per event: one DCB event fans out to several rows, so no
|
|
483
|
+
event-derived key can own a row — which is precisely what sank sharding, and the
|
|
484
|
+
argument lives in ADR-0007 beside the alternative it rejects.
|
|
485
|
+
|
|
486
|
+
Coalescing need not be total, and the **hybrid** is the shape worth seeing: one
|
|
487
|
+
row's whole batch becomes an upsert while a row written per event stays N inserts.
|
|
488
|
+
A plain sequential list is all the ordering that needs, since it preserves exactly
|
|
489
|
+
the order the fold emitted — which is ascending position order, because the store
|
|
490
|
+
delivers strictly ascending and `groupedWithin` keeps that.
|
|
491
|
+
|
|
492
|
+
```ts
|
|
493
|
+
const apply = (batch: Arr.NonEmptyReadonlyArray<DecodedEvent>) =>
|
|
494
|
+
// `Effect.suspend` so the fold runs when this effect RUNS — inside `commit`'s
|
|
495
|
+
// bracket, where a transaction can roll it back — and not when the runner calls
|
|
496
|
+
// `apply` to build the effect it hands to `commit`.
|
|
497
|
+
Effect.suspend(() =>
|
|
498
|
+
Effect.all(
|
|
499
|
+
// One write per row it can collapse, plus whatever cannot fold into that.
|
|
500
|
+
Arr.flatMap([...coalesceByRow(batch)], ([row, delta]) => [
|
|
501
|
+
upsertTotals(row, delta.applied),
|
|
502
|
+
appendAudit(row, delta.positions),
|
|
503
|
+
]),
|
|
504
|
+
{ discard: true },
|
|
505
|
+
),
|
|
506
|
+
)
|
|
507
|
+
```
|
|
508
|
+
|
|
509
|
+
**On a transactional SQL view store these writes cannot run in parallel, and do
|
|
510
|
+
not need to.** `sql.withTransaction` acquires one connection and pins it into the
|
|
511
|
+
fibre context, and `pg` dispatches one statement at a time per connection, so the
|
|
512
|
+
statements queue in the driver whatever a caller does. What coalescing buys there
|
|
513
|
+
is far **fewer round trips** on that pinned connection, never parallel ones.
|
|
514
|
+
|
|
515
|
+
On a **non-transactional** view store the port already requires a single atomic
|
|
516
|
+
per-batch write, and `foldIntoRef` is coalescing taken to its limit: fold the whole
|
|
517
|
+
batch, then write exactly once.
|
|
518
|
+
|
|
519
|
+
## Where the checkpoint cannot travel with the view
|
|
520
|
+
|
|
521
|
+
R1 says a read model's checkpoint lives with its view, committed atomically with
|
|
522
|
+
it — which is what `ProjectionStore.commit` is. A view store offering **no
|
|
523
|
+
transaction at all** (a search index, an object store, an HTTP API) cannot
|
|
524
|
+
satisfy that: there is nothing to bundle the checkpoint into, so the mode is
|
|
525
|
+
at-least-once and the projection's own writes must be **idempotent**.
|
|
526
|
+
|
|
527
|
+
The recommended shape is a **position-keyed write that no-ops on replay** — in
|
|
528
|
+
SQL terms an `INSERT … ON CONFLICT DO NOTHING` keyed by the event's position, and
|
|
529
|
+
in a search index or an object store the equivalent: derive the document or
|
|
530
|
+
object key from the position so re-applying the same event overwrites itself
|
|
531
|
+
instead of accumulating. The runner's guarantee then degrades from exactly-once
|
|
532
|
+
to "every event applied at least once, and applying it twice is
|
|
533
|
+
indistinguishable from applying it once".
|
|
534
|
+
|
|
535
|
+
This is documented rather than enforced because the library cannot force a read
|
|
536
|
+
model's schema to be position-keyed: the key is a property of the reader's own
|
|
537
|
+
rows, which the port has no knowledge of. No helper is shipped for it, and no
|
|
538
|
+
such view store is implemented here.
|