clanka 0.2.40 → 0.2.42
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/Agent.d.ts +11 -0
- package/dist/Agent.d.ts.map +1 -1
- package/dist/Agent.js +50 -3
- package/dist/Agent.js.map +1 -1
- package/dist/ScriptPreprocessing.d.ts.map +1 -1
- package/dist/ScriptPreprocessing.js +66 -0
- package/dist/ScriptPreprocessing.js.map +1 -1
- package/dist/ScriptPreprocessing.test.js +1 -0
- package/dist/ScriptPreprocessing.test.js.map +1 -1
- package/package.json +1 -1
- package/src/Agent.ts +55 -1
- package/src/ScriptPreprocessing.test.ts +1 -0
- package/src/ScriptPreprocessing.ts +103 -0
- package/src/fixtures/patch18-broken.txt +1032 -0
- package/src/fixtures/patch18-fixed.txt +1032 -0
|
@@ -0,0 +1,1032 @@
|
|
|
1
|
+
const spec = String.raw`# EventLogServerUnencrypted
|
|
2
|
+
|
|
3
|
+
## Overview
|
|
4
|
+
|
|
5
|
+
Update \
|
|
6
|
+
t the unencrypted event-log server specification so the server can append its own
|
|
7
|
+
client-visible events, while still supporting client-originated writes over the
|
|
8
|
+
existing ` + "`EventLogRemote`" + String.raw` websocket protocol.
|
|
9
|
+
|
|
10
|
+
The unencrypted server must no longer treat transport history as purely
|
|
11
|
+
per-public-key. Instead it must introduce a persisted **store id** concept:
|
|
12
|
+
|
|
13
|
+
- every client ` + "`publicKey`" + String.raw` resolves to a ` + "`StoreId`" + String.raw`
|
|
14
|
+
- transport history is stored and sequenced per ` + "`StoreId`" + String.raw`, not per public key
|
|
15
|
+
- multiple public keys may resolve to the same store id and therefore observe
|
|
16
|
+
the same outbound history and sequence numbers
|
|
17
|
+
- client writes are appended to the resolved store feed
|
|
18
|
+
- server-authored writes target a store id directly and are broadcast to all
|
|
19
|
+
clients mapped to that store
|
|
20
|
+
|
|
21
|
+
The module remains intentionally **read-time compaction, not ingest-time
|
|
22
|
+
compaction**:
|
|
23
|
+
|
|
24
|
+
- all accepted client writes are stored and processed immediately
|
|
25
|
+
- server-authored writes are stored immediately and optionally processed through
|
|
26
|
+
handlers / reactivity depending on configuration
|
|
27
|
+
- compaction only affects what the server sends back from
|
|
28
|
+
` + "`RequestChanges(startSequence)`" + String.raw`, and only for entries older than a
|
|
29
|
+
configurable fixed duration
|
|
30
|
+
|
|
31
|
+
The design should still reuse existing EventLog patterns as much as possible:
|
|
32
|
+
|
|
33
|
+
- ` + "`EventLog.group(...)`" + String.raw` remains the handler authoring API
|
|
34
|
+
- grouping / payload decoding behavior from ` + "`EventLog.groupCompaction(...)`" + String.raw`
|
|
35
|
+
should be reused or factored into shared internal helpers
|
|
36
|
+
- Reactivity invalidation semantics should match ` + "`EventLog`" + String.raw` whenever handler
|
|
37
|
+
replay is enabled
|
|
38
|
+
- ` + "`SqlEventLogJournal`" + String.raw` remains the preferred journal implementation for
|
|
39
|
+
the server-side processing journal when persistence is needed
|
|
40
|
+
- transport history and publicKey-to-store mapping should stay behind narrow
|
|
41
|
+
service contracts so a future SQL-backed implementation can be added without
|
|
42
|
+
reshaping the runtime
|
|
43
|
+
|
|
44
|
+
## User Decisions Captured
|
|
45
|
+
|
|
46
|
+
The specification reflects the following clarified requirements:
|
|
47
|
+
|
|
48
|
+
1. authorization is handled by a new required ` + "`EventLogServerAuth`" + String.raw` service
|
|
49
|
+
for **client-originated** reads and writes
|
|
50
|
+
2. backend handlers should reuse ` + "`EventLog.group(...)`" + String.raw`
|
|
51
|
+
3. compaction applies only to events older than a configurable fixed duration
|
|
52
|
+
4. unauthorized client writes reject the whole batch
|
|
53
|
+
5. ` + "`Ack.sequenceNumbers`" + String.raw` must still use one sequence per originally
|
|
54
|
+
submitted event slot
|
|
55
|
+
6. server reactivity invalidation must run for all accepted incoming client
|
|
56
|
+
events and for server-authored writes only when handler replay is enabled
|
|
57
|
+
7. unencrypted protocol responses need an explicit error message type
|
|
58
|
+
8. outbound compaction applies to the feed returned from
|
|
59
|
+
` + "`RequestChanges(startSequence)`" + String.raw`
|
|
60
|
+
9. server-authored events must be able to target a ` + "`StoreId`" + String.raw` directly
|
|
61
|
+
10. public keys must map to store ids through a persistable mapping abstraction
|
|
62
|
+
11. server-authored writes bypass ` + "`EventLogServerAuth`" + String.raw`
|
|
63
|
+
12. whether server-authored writes run handlers / reactivity must be
|
|
64
|
+
configurable per write
|
|
65
|
+
|
|
66
|
+
## Goals
|
|
67
|
+
|
|
68
|
+
- Make ` + "`EventLogServerUnencrypted`" + String.raw` feature-complete enough to back a real
|
|
69
|
+
backend event-log endpoint.
|
|
70
|
+
- Allow multiple client identities to share a single store feed through a
|
|
71
|
+
persisted publicKey-to-store mapping.
|
|
72
|
+
- Allow the backend to append its own events to a store feed and have those
|
|
73
|
+
events broadcast to all mapped clients.
|
|
74
|
+
- Allow the backend to execute the same event handlers authored with
|
|
75
|
+
` + "`EventLog.group(...)`" + String.raw` that are already used by clients.
|
|
76
|
+
- Ensure accepted client writes trigger the same Reactivity invalidation pattern
|
|
77
|
+
used by ` + "`EventLog`" + String.raw`, and ensure server-authored writes can opt into that
|
|
78
|
+
same processing path.
|
|
79
|
+
- Finish the already-started unencrypted protocol support in ` + "`EventLogRemote`" + String.raw`.
|
|
80
|
+
|
|
81
|
+
## Non-Goals
|
|
82
|
+
|
|
83
|
+
- Adding encrypted-protocol error frames in the same change.
|
|
84
|
+
- Mutating or deleting stored raw events as part of compaction. Compaction is a
|
|
85
|
+
read-time projection.
|
|
86
|
+
- Replacing ` + "`EventLog.group(...)`" + String.raw` with a new handler DSL.
|
|
87
|
+
- Requiring a dedicated SQL transport-storage adapter in the same PR.
|
|
88
|
+
- Requiring a production-ready SQL implementation of the publicKey-to-store
|
|
89
|
+
registry in the same PR, as long as the service contract is explicitly
|
|
90
|
+
durable-friendly and the shipped runtime does not hard-code ephemeral
|
|
91
|
+
derivation.
|
|
92
|
+
- Adding cross-store fan-out APIs in the initial change. Targeting one store per
|
|
93
|
+
write is sufficient.
|
|
94
|
+
|
|
95
|
+
## Module Surface
|
|
96
|
+
|
|
97
|
+
### Primary module
|
|
98
|
+
|
|
99
|
+
- ` + "`packages/effect/src/unstable/eventlog/EventLogServerUnencrypted.ts`" + String.raw`
|
|
100
|
+
|
|
101
|
+
### Existing modules that must change
|
|
102
|
+
|
|
103
|
+
- ` + "`packages/effect/src/unstable/eventlog/EventLogRemote.ts`" + String.raw`
|
|
104
|
+
- ` + "`packages/effect/src/unstable/eventlog/EventLog.ts`" + String.raw` (internal refactors /
|
|
105
|
+
shared helpers only, unless a small public adjustment is the cleanest option)
|
|
106
|
+
- optionally ` + "`packages/effect/src/unstable/eventlog/EventJournal.ts`" + String.raw` only if a
|
|
107
|
+
small helper type is needed; avoid unnecessary public API churn
|
|
108
|
+
|
|
109
|
+
### Tests
|
|
110
|
+
|
|
111
|
+
- ` + "`packages/effect/test/unstable/eventlog/EventLogServerUnencrypted.test.ts`" + String.raw`
|
|
112
|
+
- update/add targeted tests around ` + "`EventLogRemote.ts`" + String.raw` if needed
|
|
113
|
+
|
|
114
|
+
## Public API
|
|
115
|
+
|
|
116
|
+
### 1. ` + "`StoreId`" + String.raw`
|
|
117
|
+
|
|
118
|
+
Add a stable server-visible store identifier.
|
|
119
|
+
|
|
120
|
+
Suggested shape:
|
|
121
|
+
|
|
122
|
+
```ts
|
|
123
|
+
export type StoreIdTypeId = "effect/eventlog/EventLogServerUnencrypted/StoreId"
|
|
124
|
+
|
|
125
|
+
export type StoreId = string & Brand<StoreIdTypeId>
|
|
126
|
+
|
|
127
|
+
export const StoreId = Schema.String.pipe(Schema.brand(StoreIdTypeId))
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
The exact encoding can differ, but it must be:
|
|
131
|
+
|
|
132
|
+
- stable across process restarts
|
|
133
|
+
- serializable / persistable by user-provided infrastructure
|
|
134
|
+
- suitable as the key for transport history and store-mapping persistence
|
|
135
|
+
|
|
136
|
+
### 2. ` + "`EventLogServerStoreRegistry`" + String.raw`
|
|
137
|
+
|
|
138
|
+
Add a required service responsible for resolving public keys to store ids.
|
|
139
|
+
|
|
140
|
+
```ts
|
|
141
|
+
export class EventLogServerStoreRegistry extends ServiceMap.Service<EventLogServerStoreRegistry, {
|
|
142
|
+
readonly resolve: (publicKey: string) => Effect.Effect<StoreId, EventLogServerStoreRegistryError>
|
|
143
|
+
readonly assign: (options: {
|
|
144
|
+
readonly publicKey: string
|
|
145
|
+
readonly storeId: StoreId
|
|
146
|
+
}) => Effect.Effect<void, EventLogServerStoreRegistryError>
|
|
147
|
+
}>()("effect/eventlog/EventLogServerUnencrypted/EventLogServerStoreRegistry") {}
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
#### Required semantics
|
|
151
|
+
|
|
152
|
+
- ` + "`resolve(publicKey)`" + String.raw` is used by all client-originated paths:
|
|
153
|
+
- ` + "`WriteEntriesUnencrypted`" + String.raw`
|
|
154
|
+
- ` + "`RequestChanges`" + String.raw`
|
|
155
|
+
- two public keys assigned the same store id must observe the same outbound
|
|
156
|
+
history and sequence numbers
|
|
157
|
+
- the mapping is domain state, not an ephemeral hash or in-memory-only default
|
|
158
|
+
- the main server runtime must require this service explicitly
|
|
159
|
+
- a helper such as ` + "`layerStoreRegistryMemory`" + String.raw` is recommended for tests and
|
|
160
|
+
local development
|
|
161
|
+
- the service contract must support durable implementations, even if only a
|
|
162
|
+
memory helper ships in the first PR
|
|
163
|
+
|
|
164
|
+
#### Notes
|
|
165
|
+
|
|
166
|
+
- ` + "`assign(...)`" + String.raw` is included so tests and local setups can configure mappings
|
|
167
|
+
without reaching into implementation details
|
|
168
|
+
- additional helpers such as ` + "`assignMany`" + String.raw`, ` + "`remove`" + String.raw`, or read-only admin APIs
|
|
169
|
+
are acceptable but not required
|
|
170
|
+
|
|
171
|
+
### 3. ` + "`EventLogServerStoreRegistryError`" + String.raw`
|
|
172
|
+
|
|
173
|
+
Add a structured error for store-resolution failures.
|
|
174
|
+
|
|
175
|
+
```ts
|
|
176
|
+
export class EventLogServerStoreRegistryError extends Data.TaggedError("EventLogServerStoreRegistryError")<{
|
|
177
|
+
readonly reason: "NotMapped" | "AlreadyAssigned" | "Unavailable"
|
|
178
|
+
readonly publicKey?: string | undefined
|
|
179
|
+
readonly storeId?: StoreId | undefined
|
|
180
|
+
readonly message?: string | undefined
|
|
181
|
+
}> {}
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
Required behavior:
|
|
185
|
+
|
|
186
|
+
- an unmapped client public key must not silently derive a store id
|
|
187
|
+
- write / read requests for an unmapped public key must fail before any
|
|
188
|
+
transport persistence or subscription creation
|
|
189
|
+
- protocol mapping for ` + "`NotMapped`" + String.raw` should use ` + "`ErrorUnencrypted`" + String.raw` with code
|
|
190
|
+
` + '"InvalidRequest"' + String.raw` unless the implementation intentionally wants to conceal
|
|
191
|
+
store existence behind an auth failure; whichever mapping is chosen must be
|
|
192
|
+
consistent across read and write paths
|
|
193
|
+
|
|
194
|
+
### 4. ` + "`EventLogServerAuth`" + String.raw`
|
|
195
|
+
|
|
196
|
+
Keep the required auth service from the original spec for client-originated
|
|
197
|
+
traffic.
|
|
198
|
+
|
|
199
|
+
```ts
|
|
200
|
+
export class EventLogServerAuth extends ServiceMap.Service<EventLogServerAuth, {
|
|
201
|
+
readonly authorizeWrite: (options: {
|
|
202
|
+
readonly publicKey: string
|
|
203
|
+
readonly entries: ReadonlyArray<Entry>
|
|
204
|
+
}) => Effect.Effect<void, EventLogServerAuthError>
|
|
205
|
+
readonly authorizeRead: (publicKey: string) => Effect.Effect<void, EventLogServerAuthError>
|
|
206
|
+
}>()("effect/eventlog/EventLogServerUnencrypted/EventLogServerAuth") {}
|
|
207
|
+
```
|
|
208
|
+
|
|
209
|
+
#### Semantics
|
|
210
|
+
|
|
211
|
+
- ` + "`authorizeWrite`" + String.raw` is called once per inbound client write request before any
|
|
212
|
+
persistence, handler execution, reactivity invalidation, or store resolution
|
|
213
|
+
- authorization is batch-wide: if it fails, the entire write request is
|
|
214
|
+
rejected
|
|
215
|
+
- ` + "`authorizeRead`" + String.raw` is called before creating a ` + "`RequestChanges`" + String.raw` subscription
|
|
216
|
+
- server-authored writes targeting a store id bypass this service entirely
|
|
217
|
+
- a helper layer such as ` + "`layerAuthAllowAll`" + String.raw` is recommended for tests and local
|
|
218
|
+
development, but the service itself must remain required by the main server
|
|
219
|
+
constructor APIs
|
|
220
|
+
|
|
221
|
+
### 5. ` + "`EventLogServerAuthError`" + String.raw`
|
|
222
|
+
|
|
223
|
+
Add a tagged error type for auth rejections.
|
|
224
|
+
|
|
225
|
+
```ts
|
|
226
|
+
export class EventLogServerAuthError extends Data.TaggedError("EventLogServerAuthError")<{
|
|
227
|
+
readonly reason: "Unauthorized" | "Forbidden"
|
|
228
|
+
readonly publicKey: string
|
|
229
|
+
readonly message?: string | undefined
|
|
230
|
+
}> {}
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
Exact fields may vary, but the type must be structured enough to map to an
|
|
234
|
+
unencrypted protocol error frame.
|
|
235
|
+
|
|
236
|
+
### 6. ` + "`Storage`" + String.raw`
|
|
237
|
+
|
|
238
|
+
Update the transport-history storage abstraction so it is keyed by ` + "`StoreId`" + String.raw`, not
|
|
239
|
+
public key.
|
|
240
|
+
|
|
241
|
+
```ts
|
|
242
|
+
export class Storage extends ServiceMap.Service<Storage, {
|
|
243
|
+
readonly getId: Effect.Effect<RemoteId>
|
|
244
|
+
readonly write: (options: {
|
|
245
|
+
readonly storeId: StoreId
|
|
246
|
+
readonly entries: ReadonlyArray<Entry>
|
|
247
|
+
}) => Effect.Effect<{
|
|
248
|
+
readonly sequenceNumbers: ReadonlyArray<number>
|
|
249
|
+
readonly committed: ReadonlyArray<RemoteEntry>
|
|
250
|
+
}>
|
|
251
|
+
readonly entries: (
|
|
252
|
+
storeId: StoreId,
|
|
253
|
+
startSequence: number
|
|
254
|
+
) => Effect.Effect<ReadonlyArray<RemoteEntry>>
|
|
255
|
+
readonly changes: (
|
|
256
|
+
storeId: StoreId,
|
|
257
|
+
startSequence: number
|
|
258
|
+
) => Effect.Effect<Queue.Dequeue<RemoteEntry, Cause.Done>, never, Scope.Scope>
|
|
259
|
+
}>()("effect/eventlog/EventLogServerUnencrypted/Storage") {}
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
#### Required semantics
|
|
263
|
+
|
|
264
|
+
- transport history is sequenced independently per store id
|
|
265
|
+
- duplicate detection is per store id and entry id
|
|
266
|
+
- ` + "`sequenceNumbers`" + String.raw` must contain one output sequence per input slot, even when
|
|
267
|
+
an input is a duplicate
|
|
268
|
+
- ` + "`committed`" + String.raw` contains only newly committed entries that still need replay into
|
|
269
|
+
the processing journal / handlers
|
|
270
|
+
- ` + "`entries`" + String.raw` and ` + "`changes`" + String.raw` expose the shared store feed used by all mapped
|
|
271
|
+
public keys
|
|
272
|
+
|
|
273
|
+
A helper layer such as ` + "`layerStorageMemory`" + String.raw` is recommended for tests and local
|
|
274
|
+
workflows.
|
|
275
|
+
|
|
276
|
+
### 7. ` + "`EventLogServerUnencrypted`" + String.raw` runtime service
|
|
277
|
+
|
|
278
|
+
Add a server runtime service, parallel to the client-side ` + "`EventLog`" + String.raw` runtime,
|
|
279
|
+
responsible for:
|
|
280
|
+
|
|
281
|
+
- registering outbound compactors
|
|
282
|
+
- registering reactivity mappings
|
|
283
|
+
- processing accepted client entries through the server journal + handlers
|
|
284
|
+
- serving store-backed outbound change feeds to client public keys
|
|
285
|
+
- exposing a server-authored write path that targets store ids directly
|
|
286
|
+
|
|
287
|
+
Suggested shape:
|
|
288
|
+
|
|
289
|
+
```ts
|
|
290
|
+
export class EventLogServerUnencrypted extends ServiceMap.Service<EventLogServerUnencrypted, {
|
|
291
|
+
readonly ingestClient: (options: {
|
|
292
|
+
readonly publicKey: string
|
|
293
|
+
readonly entries: ReadonlyArray<Entry>
|
|
294
|
+
}) => Effect.Effect<{
|
|
295
|
+
readonly storeId: StoreId
|
|
296
|
+
readonly sequenceNumbers: ReadonlyArray<number>
|
|
297
|
+
readonly committed: ReadonlyArray<RemoteEntry>
|
|
298
|
+
}, EventLogServerAuthError | EventLogServerStoreRegistryError | EventJournal.EventJournalError>
|
|
299
|
+
readonly requestChanges: (
|
|
300
|
+
publicKey: string,
|
|
301
|
+
startSequence: number
|
|
302
|
+
) => Effect.Effect<Queue.Dequeue<RemoteEntry, EventLogRemote.EventLogRemoteError>, never, Scope.Scope>
|
|
303
|
+
readonly writeToStore: (options: {
|
|
304
|
+
readonly storeId: StoreId
|
|
305
|
+
readonly entries: ReadonlyArray<Entry>
|
|
306
|
+
readonly runHandlers?: boolean | undefined
|
|
307
|
+
}) => Effect.Effect<{
|
|
308
|
+
readonly sequenceNumbers: ReadonlyArray<number>
|
|
309
|
+
readonly committed: ReadonlyArray<RemoteEntry>
|
|
310
|
+
}, EventJournal.EventJournalError>
|
|
311
|
+
readonly registerCompaction: (options: {
|
|
312
|
+
readonly events: ReadonlyArray<string>
|
|
313
|
+
readonly olderThan: Duration.DurationInput
|
|
314
|
+
readonly effect: (options: {
|
|
315
|
+
readonly entries: ReadonlyArray<Entry>
|
|
316
|
+
readonly write: (entry: Entry) => Effect.Effect<void>
|
|
317
|
+
}) => Effect.Effect<void>
|
|
318
|
+
}) => Effect.Effect<void, never, Scope.Scope>
|
|
319
|
+
readonly registerReactivity: (keys: Record<string, ReadonlyArray<string>>) => Effect.Effect<void, never, Scope.Scope>
|
|
320
|
+
}>()("effect/eventlog/EventLogServerUnencrypted") {}
|
|
321
|
+
```
|
|
322
|
+
|
|
323
|
+
#### Required behavior
|
|
324
|
+
|
|
325
|
+
- ` + "`ingestClient(...)`" + String.raw` is the client-originated write path used by the websocket
|
|
326
|
+
handler
|
|
327
|
+
- ` + "`writeToStore(...)`" + String.raw` is the server-authored write path
|
|
328
|
+
- ` + "`writeToStore(...)`" + String.raw` bypasses auth and store resolution because the caller already
|
|
329
|
+
supplies a target store id
|
|
330
|
+
- when ` + "`runHandlers`" + String.raw` is omitted, it should default to ` + "`true`" + String.raw`
|
|
331
|
+
- when ` + "`runHandlers: true`" + String.raw`, committed entries must be replayed into the processing
|
|
332
|
+
journal and must trigger the same handler / reactivity behavior as client
|
|
333
|
+
writes
|
|
334
|
+
- when ` + "`runHandlers: false`" + String.raw`, committed entries are transport-only:
|
|
335
|
+
- they are persisted to storage
|
|
336
|
+
- they are visible to ` + "`RequestChanges`" + String.raw` readers for that store
|
|
337
|
+
- they are **not** replayed into the processing journal
|
|
338
|
+
- they do **not** execute handlers
|
|
339
|
+
- they do **not** trigger reactivity invalidation
|
|
340
|
+
- they do **not** participate in conflict detection until a future feature
|
|
341
|
+
explicitly introduces a second processing mode
|
|
342
|
+
|
|
343
|
+
### 8. ` + "`layer`" + String.raw`
|
|
344
|
+
|
|
345
|
+
Add a layer constructor, parallel to ` + "`EventLog.layer(...)`" + String.raw`, for building the
|
|
346
|
+
server runtime from handler layers and a processing journal.
|
|
347
|
+
|
|
348
|
+
```ts
|
|
349
|
+
export const layer: <Groups extends EventGroup.Any>(
|
|
350
|
+
schema: EventLog.EventLogSchema<Groups>
|
|
351
|
+
) => Layer.Layer<
|
|
352
|
+
EventLogServerUnencrypted,
|
|
353
|
+
never,
|
|
354
|
+
EventGroup.ToService<Groups>
|
|
355
|
+
| EventJournal.EventJournal
|
|
356
|
+
| EventLogServerStoreRegistry
|
|
357
|
+
| EventLogServerAuth
|
|
358
|
+
| Storage
|
|
359
|
+
>
|
|
360
|
+
```
|
|
361
|
+
|
|
362
|
+
#### Notes
|
|
363
|
+
|
|
364
|
+
- handlers are still authored with ` + "`EventLog.group(...)`" + String.raw`
|
|
365
|
+
- the server runtime should read those handler registrations from the produced
|
|
366
|
+
service map exactly like ` + "`EventLog`" + String.raw` does
|
|
367
|
+
- this keeps domain handler code portable between client and server runtimes
|
|
368
|
+
|
|
369
|
+
### 9. ` + "`groupCompaction`" + String.raw`
|
|
370
|
+
|
|
371
|
+
Keep the existing compaction helper from the original spec, but clarify that it
|
|
372
|
+
now operates on per-store transport history served through client public keys.
|
|
373
|
+
|
|
374
|
+
```ts
|
|
375
|
+
export const groupCompaction: <Events extends Event.Any, R>(
|
|
376
|
+
group: EventGroup.EventGroup<Events>,
|
|
377
|
+
options: {
|
|
378
|
+
readonly olderThan: Duration.DurationInput
|
|
379
|
+
},
|
|
380
|
+
effect: (options: {
|
|
381
|
+
readonly primaryKey: string
|
|
382
|
+
readonly entries: ReadonlyArray<Entry>
|
|
383
|
+
readonly events: ReadonlyArray<Event.TaggedPayload<Events>>
|
|
384
|
+
readonly write: <Tag extends Event.Tag<Events>>(
|
|
385
|
+
tag: Tag,
|
|
386
|
+
payload: Event.PayloadWithTag<Events, Tag>
|
|
387
|
+
) => Effect.Effect<void, never, Event.PayloadSchemaWithTag<Events, Tag>["EncodingServices"]>
|
|
388
|
+
}) => Effect.Effect<void, never, R>
|
|
389
|
+
) => Layer.Layer<never, never, EventLogServerUnencrypted | R | Event.PayloadSchema<Events>["DecodingServices"]>
|
|
390
|
+
```
|
|
391
|
+
|
|
392
|
+
Behavior remains the same as the prior spec except that:
|
|
393
|
+
|
|
394
|
+
- eligible entries come from the resolved store feed
|
|
395
|
+
- a compacted response is shared by all public keys reading that store from the
|
|
396
|
+
same cursor
|
|
397
|
+
|
|
398
|
+
### 10. ` + "`groupReactivity`" + String.raw`
|
|
399
|
+
|
|
400
|
+
Add a server-local helper mirroring ` + "`EventLog.groupReactivity(...)`" + String.raw` so a server
|
|
401
|
+
runtime can register per-event invalidation keys without depending on the
|
|
402
|
+
client-side ` + "`EventLog`" + String.raw` service.
|
|
403
|
+
|
|
404
|
+
### 11. ` + "`makeHandler`" + String.raw` and ` + "`makeHandlerHttp`" + String.raw`
|
|
405
|
+
|
|
406
|
+
Add websocket / HTTP upgrade handlers parallel to ` + "`EventLogServer.makeHandler`" + String.raw`
|
|
407
|
+
and ` + "`makeHandlerHttp`" + String.raw`.
|
|
408
|
+
|
|
409
|
+
```ts
|
|
410
|
+
export const makeHandler: Effect.Effect<
|
|
411
|
+
(socket: Socket.Socket) => Effect.Effect<void, Socket.SocketError>,
|
|
412
|
+
never,
|
|
413
|
+
EventLogServerUnencrypted | Storage | EventLogServerAuth | EventLogServerStoreRegistry
|
|
414
|
+
>
|
|
415
|
+
|
|
416
|
+
export const makeHandlerHttp: Effect.Effect<
|
|
417
|
+
Effect.Effect<
|
|
418
|
+
HttpServerResponse.HttpServerResponse,
|
|
419
|
+
HttpServerError.HttpServerError | Socket.SocketError,
|
|
420
|
+
HttpServerRequest.HttpServerRequest | Scope.Scope
|
|
421
|
+
>,
|
|
422
|
+
never,
|
|
423
|
+
EventLogServerUnencrypted | Storage | EventLogServerAuth | EventLogServerStoreRegistry
|
|
424
|
+
>
|
|
425
|
+
```
|
|
426
|
+
|
|
427
|
+
## EventLogRemote Changes
|
|
428
|
+
|
|
429
|
+
The existing unencrypted support in ` + "`EventLogRemote.ts`" + String.raw` is incomplete and must
|
|
430
|
+
be finished as part of this work.
|
|
431
|
+
|
|
432
|
+
### Protocol additions
|
|
433
|
+
|
|
434
|
+
Add an explicit unencrypted error response:
|
|
435
|
+
|
|
436
|
+
```ts
|
|
437
|
+
export class ErrorUnencrypted extends Schema.Class<ErrorUnencrypted>(
|
|
438
|
+
"effect/eventlog/EventLogRemote/ErrorUnencrypted"
|
|
439
|
+
)({
|
|
440
|
+
_tag: Schema.tag("Error"),
|
|
441
|
+
requestTag: Schema.String,
|
|
442
|
+
id: Schema.optional(Schema.Number),
|
|
443
|
+
publicKey: Schema.optional(Schema.String),
|
|
444
|
+
code: Schema.Literal("Unauthorized", "InvalidRequest", "InternalServerError"),
|
|
445
|
+
message: Schema.String
|
|
446
|
+
}) {}
|
|
447
|
+
```
|
|
448
|
+
|
|
449
|
+
The exact schema can differ, but it must support:
|
|
450
|
+
|
|
451
|
+
- correlating write failures to a pending ` + "`WriteEntries`" + String.raw` request id
|
|
452
|
+
- indicating read-subscription failures for ` + "`RequestChanges`" + String.raw`
|
|
453
|
+
- communicating auth failures in a machine-readable form
|
|
454
|
+
- communicating store-resolution failures for unmapped public keys
|
|
455
|
+
|
|
456
|
+
Update:
|
|
457
|
+
|
|
458
|
+
- ` + "`ProtocolResponseUnencrypted`" + String.raw`
|
|
459
|
+
- ` + "`ProtocolResponseUnencryptedMsgpack`" + String.raw`
|
|
460
|
+
- ` + "`decodeResponseUnencrypted`" + String.raw`
|
|
461
|
+
- ` + "`encodeResponseUnencrypted`" + String.raw`
|
|
462
|
+
|
|
463
|
+
### Codec correctness
|
|
464
|
+
|
|
465
|
+
Complete the unfinished unencrypted protocol path in ` + "`EventLogRemote.ts`" + String.raw`:
|
|
466
|
+
|
|
467
|
+
- ` + "`fromSocketUnencrypted`" + String.raw` must use the unencrypted request encoder and response
|
|
468
|
+
decoder consistently
|
|
469
|
+
- the misspelled ` + "`encodeRequestUnsencrypted`" + String.raw` helper should be corrected to
|
|
470
|
+
` + "`encodeRequestUnencrypted`" + String.raw`
|
|
471
|
+
- if the typo is already exported, keep a temporary backwards-compatible alias
|
|
472
|
+
or deprecate it in-place rather than silently breaking consumers
|
|
473
|
+
|
|
474
|
+
### Error surfacing
|
|
475
|
+
|
|
476
|
+
Because the server now sends explicit unencrypted protocol errors, the remote
|
|
477
|
+
client API must surface them.
|
|
478
|
+
|
|
479
|
+
#### Required behavior
|
|
480
|
+
|
|
481
|
+
- a write rejection for a pending request must fail the corresponding
|
|
482
|
+
` + "`remote.write(...)`" + String.raw` effect with ` + "`EventLogRemoteError`" + String.raw`
|
|
483
|
+
- a rejected ` + "`RequestChanges`" + String.raw` subscription must fail the returned dequeue, or an
|
|
484
|
+
equivalent fallible subscription mechanism, with ` + "`EventLogRemoteError`" + String.raw`
|
|
485
|
+
- downstream ` + "`EventLog.registerRemote(...)`" + String.raw` consumption must handle those
|
|
486
|
+
failures by logging and continuing its normal reconnection / retry behavior
|
|
487
|
+
|
|
488
|
+
Since these APIs are unstable, it is acceptable to widen the public error types
|
|
489
|
+
if needed.
|
|
490
|
+
|
|
491
|
+
## Runtime Architecture
|
|
492
|
+
|
|
493
|
+
### 1. Separate transport history, store mapping, and processing journal
|
|
494
|
+
|
|
495
|
+
The server has three distinct responsibilities:
|
|
496
|
+
|
|
497
|
+
1. maintain a per-store transport feed used by ` + "`RequestChanges`" + String.raw`
|
|
498
|
+
2. resolve client public keys to store ids through a durable-friendly registry
|
|
499
|
+
3. run handlers / conflicts / reactivity against a backend processing journal
|
|
500
|
+
|
|
501
|
+
Use:
|
|
502
|
+
|
|
503
|
+
- the new unencrypted ` + "`Storage`" + String.raw` service for (1)
|
|
504
|
+
- the new ` + "`EventLogServerStoreRegistry`" + String.raw` service for (2)
|
|
505
|
+
- the existing ` + "`EventJournal`" + String.raw` service for (3)
|
|
506
|
+
|
|
507
|
+
This keeps the design aligned with existing abstractions while allowing store
|
|
508
|
+
broadcast semantics.
|
|
509
|
+
|
|
510
|
+
### 2. Stable journal origin per store id
|
|
511
|
+
|
|
512
|
+
Accepted client writes and server-authored writes that replay handlers must be
|
|
513
|
+
replayed into the processing journal via ` + "`journal.writeFromRemote(...)`" + String.raw`. That
|
|
514
|
+
requires a stable ` + "`RemoteId`" + String.raw`.
|
|
515
|
+
|
|
516
|
+
The runtime should derive a deterministic ` + "`RemoteId`" + String.raw` from ` + "`StoreId`" + String.raw`, not from
|
|
517
|
+
public key.
|
|
518
|
+
|
|
519
|
+
Recommended approach:
|
|
520
|
+
|
|
521
|
+
- compute a stable 16-byte digest from the UTF-8 encoded store id
|
|
522
|
+
- brand the first 16 bytes as ` + "`RemoteId`" + String.raw`
|
|
523
|
+
- keep this derivation internal to the server runtime
|
|
524
|
+
|
|
525
|
+
This ensures:
|
|
526
|
+
|
|
527
|
+
- all public keys mapped to the same store share the same processing-journal
|
|
528
|
+
remote origin
|
|
529
|
+
- server-authored writes targeting that store participate in the same remote
|
|
530
|
+
sequence space whenever handler replay is enabled
|
|
531
|
+
|
|
532
|
+
### 3. Client ingest pipeline
|
|
533
|
+
|
|
534
|
+
For each ` + "`WriteEntriesUnencrypted`" + String.raw` request:
|
|
535
|
+
|
|
536
|
+
1. decode request
|
|
537
|
+
2. authorize the entire batch via ` + "`EventLogServerAuth.authorizeWrite`" + String.raw`
|
|
538
|
+
3. resolve ` + "`publicKey -> storeId`" + String.raw` via the store registry
|
|
539
|
+
4. if auth or store resolution fails, send ` + "`ErrorUnencrypted`" + String.raw` and do not persist
|
|
540
|
+
anything
|
|
541
|
+
5. persist raw entries into transport ` + "`Storage.write({ storeId, entries })`" + String.raw`
|
|
542
|
+
6. send ` + "`Ack`" + String.raw` with ` + "`sequenceNumbers.length === request.entries.length`" + String.raw`
|
|
543
|
+
7. replay only ` + "`committed`" + String.raw` entries into ` + "`EventJournal.writeFromRemote(...)`" + String.raw`
|
|
544
|
+
8. during replay, run backend handlers and reactivity invalidation exactly once
|
|
545
|
+
per committed entry
|
|
546
|
+
9. do not compact during ingest
|
|
547
|
+
|
|
548
|
+
### 4. Server-authored write pipeline
|
|
549
|
+
|
|
550
|
+
For each ` + "`writeToStore({ storeId, entries, runHandlers })`" + String.raw` call:
|
|
551
|
+
|
|
552
|
+
1. validate the supplied entries and target store id
|
|
553
|
+
2. persist raw entries into transport ` + "`Storage.write({ storeId, entries })`" + String.raw`
|
|
554
|
+
3. broadcast those committed entries to all current and future readers of that
|
|
555
|
+
store via the normal ` + "`RequestChanges`" + String.raw` path
|
|
556
|
+
4. if ` + "`runHandlers`" + String.raw` is ` + "`true`" + String.raw`, replay committed entries into the
|
|
557
|
+
processing journal using the deterministic store remote id
|
|
558
|
+
5. if ` + "`runHandlers`" + String.raw` is ` + "`false`" + String.raw`, stop after transport persistence and
|
|
559
|
+
do not touch the processing journal
|
|
560
|
+
|
|
561
|
+
This path bypasses ` + "`EventLogServerAuth`" + String.raw` entirely.
|
|
562
|
+
|
|
563
|
+
### 5. Shared-store visibility semantics
|
|
564
|
+
|
|
565
|
+
Because the feed is keyed by store id:
|
|
566
|
+
|
|
567
|
+
- two public keys assigned to the same store must receive the same event stream
|
|
568
|
+
when reading from the same ` + "`startSequence`" + String.raw`
|
|
569
|
+
- a client write from public key A must be visible to public key B if both map to
|
|
570
|
+
the same store
|
|
571
|
+
- a server-authored write to store S must be visible to all clients mapped to S
|
|
572
|
+
- a newly mapped public key requesting sequence ` + "`0`" + String.raw` for an existing store should
|
|
573
|
+
observe the full store backlog
|
|
574
|
+
|
|
575
|
+
### 6. Duplicate handling
|
|
576
|
+
|
|
577
|
+
If the same entry id is written again for the same store:
|
|
578
|
+
|
|
579
|
+
- ` + "`Ack.sequenceNumbers`" + String.raw` or equivalent server-write metadata still includes a
|
|
580
|
+
sequence for that input slot
|
|
581
|
+
- the duplicate entry is not re-committed to storage
|
|
582
|
+
- the duplicate entry is not re-run through handlers
|
|
583
|
+
- the duplicate entry does not re-trigger reactivity invalidation
|
|
584
|
+
|
|
585
|
+
Duplicate identity is store-local. The same entry id written to two different
|
|
586
|
+
stores may be treated as distinct transport entries.
|
|
587
|
+
|
|
588
|
+
### 7. Handler execution semantics
|
|
589
|
+
|
|
590
|
+
Server-side handler execution should match the remote-consumption path already
|
|
591
|
+
implemented inside ` + "`EventLog`" + String.raw`.
|
|
592
|
+
|
|
593
|
+
For each committed entry whose replay path is enabled:
|
|
594
|
+
|
|
595
|
+
- locate the handler via the service key compiled from ` + "`EventLog.group(...)`" + String.raw`
|
|
596
|
+
- decode the payload with the event schema
|
|
597
|
+
- decode conflict payloads with the same schema
|
|
598
|
+
- merge handler-required services into the effect environment
|
|
599
|
+
- execute the handler
|
|
600
|
+
- log handler failures the same way ` + "`EventLog`" + String.raw` does, rather than failing the
|
|
601
|
+
websocket protocol loop
|
|
602
|
+
|
|
603
|
+
Implementation should extract or share the common decode / execute / invalidate
|
|
604
|
+
logic from ` + "`EventLog.ts`" + String.raw` rather than copying it into two places.
|
|
605
|
+
|
|
606
|
+
### 8. Reactivity semantics
|
|
607
|
+
|
|
608
|
+
Reactivity invalidation must behave the same as accepted remote writes in the
|
|
609
|
+
client-side ` + "`EventLog`" + String.raw` runtime whenever replay is enabled.
|
|
610
|
+
|
|
611
|
+
For each committed entry whose replay path is enabled:
|
|
612
|
+
|
|
613
|
+
- look up registered reactivity keys for ` + "`entry.event`" + String.raw`
|
|
614
|
+
- invalidate ` + "`{ [key]: [entry.primaryKey] }`" + String.raw` for each configured key
|
|
615
|
+
- perform invalidation after successful handler execution, matching current
|
|
616
|
+
` + "`EventLog`" + String.raw` semantics
|
|
617
|
+
|
|
618
|
+
When ` + "`runHandlers: false`" + String.raw` is used for a server-authored write, reactivity does not
|
|
619
|
+
run.
|
|
620
|
+
|
|
621
|
+
## Outbound Compaction
|
|
622
|
+
|
|
623
|
+
### 1. When compaction runs
|
|
624
|
+
|
|
625
|
+
Compaction runs only while serving ` + "`RequestChanges(publicKey, startSequence)`" + String.raw`.
|
|
626
|
+
|
|
627
|
+
- the server first resolves ` + "`publicKey -> storeId`" + String.raw`
|
|
628
|
+
- compaction operates on that store's transport history
|
|
629
|
+
- recent entries remain unmodified in the outbound stream
|
|
630
|
+
- only entries older than the configured threshold are eligible
|
|
631
|
+
- authorization happens before the subscription is created
|
|
632
|
+
|
|
633
|
+
### 2. Eligibility rule
|
|
634
|
+
|
|
635
|
+
For a compactor registered with ` + "`olderThan`" + String.raw`, an entry is eligible when:
|
|
636
|
+
|
|
637
|
+
```ts
|
|
638
|
+
entry.createdAtMillis <= now - olderThan
|
|
639
|
+
```
|
|
640
|
+
|
|
641
|
+
If an entry is newer than the cutoff, it must be sent through unchanged even if
|
|
642
|
+
its event tag belongs to a compaction group.
|
|
643
|
+
|
|
644
|
+
### 3. Compaction grouping model
|
|
645
|
+
|
|
646
|
+
Use the same high-level grouping semantics as ` + "`EventLog.groupCompaction(...)`" + String.raw`:
|
|
647
|
+
|
|
648
|
+
- only event tags registered by the compactor participate
|
|
649
|
+
- decode payloads with the event schemas from the ` + "`EventGroup`" + String.raw`
|
|
650
|
+
- group the candidate entries by ` + "`primaryKey`" + String.raw`
|
|
651
|
+
- pass ` + "`{ primaryKey, entries, events, write(...) }`" + String.raw` to the compaction effect
|
|
652
|
+
- ` + "`write(...)`" + String.raw` encodes replacement entries back into ` + "`Entry`" + String.raw`
|
|
653
|
+
|
|
654
|
+
### 4. Sequence assignment for compacted outputs
|
|
655
|
+
|
|
656
|
+
Compaction happens on raw transport history that already has ` + "`remoteSequence`" + String.raw`
|
|
657
|
+
values. Replacement entries sent to clients must preserve cursor monotonicity.
|
|
658
|
+
|
|
659
|
+
Rule:
|
|
660
|
+
|
|
661
|
+
- each compacted output entry must be emitted as a ` + "`RemoteEntry`" + String.raw`
|
|
662
|
+
- its ` + "`remoteSequence`" + String.raw` must be the **maximum** raw sequence consumed for that
|
|
663
|
+
output entry's source bucket
|
|
664
|
+
- when a single primary-key bucket produces multiple replacement entries, each
|
|
665
|
+
emitted replacement entry may share that maximum sequence
|
|
666
|
+
|
|
667
|
+
### 5. Filtering relative to ` + "`startSequence`" + String.raw`
|
|
668
|
+
|
|
669
|
+
The server must compact first, then apply the outbound sequence filter using the
|
|
670
|
+
representative ` + "`remoteSequence`" + String.raw` of the emitted entries.
|
|
671
|
+
|
|
672
|
+
That means old raw entries below ` + "`startSequence`" + String.raw` may still influence a compacted
|
|
673
|
+
replacement entry whose representative sequence is at or above the cursor.
|
|
674
|
+
|
|
675
|
+
### 6. Streaming behavior
|
|
676
|
+
|
|
677
|
+
` + "`requestChanges(publicKey, startSequence)`" + String.raw` should:
|
|
678
|
+
|
|
679
|
+
1. authorize the read request
|
|
680
|
+
2. resolve the public key to a store id
|
|
681
|
+
3. subscribe to raw storage changes for that store
|
|
682
|
+
4. read the initial backlog plus live updates
|
|
683
|
+
5. compact the eligible historical portion on each emission batch
|
|
684
|
+
6. emit recent raw entries unchanged
|
|
685
|
+
7. preserve monotonic ordering by ` + "`remoteSequence`" + String.raw`
|
|
686
|
+
|
|
687
|
+
A simple and acceptable first implementation may compact the initial backlog and
|
|
688
|
+
then pass through live updates unchanged until they age past the threshold on a
|
|
689
|
+
future subscription.
|
|
690
|
+
|
|
691
|
+
## Authorization, Store Resolution, and Protocol Errors
|
|
692
|
+
|
|
693
|
+
### Client write rejection
|
|
694
|
+
|
|
695
|
+
If ` + "`authorizeWrite`" + String.raw` fails:
|
|
696
|
+
|
|
697
|
+
- send ` + "`ErrorUnencrypted`" + String.raw` with a write-correlated request id
|
|
698
|
+
- do not send ` + "`Ack`" + String.raw`
|
|
699
|
+
- do not resolve the store id
|
|
700
|
+
- do not persist transport entries
|
|
701
|
+
- do not replay into the processing journal
|
|
702
|
+
- do not run handlers
|
|
703
|
+
- do not invalidate reactivity
|
|
704
|
+
|
|
705
|
+
### Client read rejection
|
|
706
|
+
|
|
707
|
+
If ` + "`authorizeRead`" + String.raw` fails:
|
|
708
|
+
|
|
709
|
+
- send ` + "`ErrorUnencrypted`" + String.raw` describing the rejected ` + "`RequestChanges`" + String.raw`
|
|
710
|
+
- do not resolve the store id
|
|
711
|
+
- do not create a subscription
|
|
712
|
+
- do not leak queue resources or a live FiberMap registration
|
|
713
|
+
|
|
714
|
+
### Store resolution failure
|
|
715
|
+
|
|
716
|
+
If the public key cannot be resolved to a store id:
|
|
717
|
+
|
|
718
|
+
- send ` + "`ErrorUnencrypted`" + String.raw` for the relevant request
|
|
719
|
+
- do not persist transport entries
|
|
720
|
+
- do not create a subscription
|
|
721
|
+
- do not leak resources
|
|
722
|
+
- do not fall back to hashing the public key into an implicit store id
|
|
723
|
+
|
|
724
|
+
### Internal failures
|
|
725
|
+
|
|
726
|
+
Internal operational failures should be mapped as follows:
|
|
727
|
+
|
|
728
|
+
- expected auth failures -> ` + "`ErrorUnencrypted`" + String.raw`
|
|
729
|
+
- expected store-resolution failures -> ` + "`ErrorUnencrypted`" + String.raw`
|
|
730
|
+
- expected request validation failures in the unencrypted path ->
|
|
731
|
+
` + "`ErrorUnencrypted`" + String.raw`
|
|
732
|
+
- unexpected defects / decode corruption / chunk reassembly corruption -> log and
|
|
733
|
+
use the existing connection failure behavior unless an explicit protocol error
|
|
734
|
+
can be sent safely
|
|
735
|
+
|
|
736
|
+
## Persistence Requirements
|
|
737
|
+
|
|
738
|
+
### 1. Store mapping persistence
|
|
739
|
+
|
|
740
|
+
The publicKey-to-store mapping is durable domain state.
|
|
741
|
+
|
|
742
|
+
Therefore:
|
|
743
|
+
|
|
744
|
+
- the runtime must not derive store ids from public keys on the fly as its only
|
|
745
|
+
mapping strategy
|
|
746
|
+
- the mapping must live behind the required
|
|
747
|
+
` + "`EventLogServerStoreRegistry`" + String.raw` service
|
|
748
|
+
- the service contract must support implementations backed by durable storage
|
|
749
|
+
- a memory helper is acceptable for tests and local development only
|
|
750
|
+
|
|
751
|
+
### 2. Transport history persistence
|
|
752
|
+
|
|
753
|
+
This specification still does **not** require ` + "`SqlEventLogJournal`" + String.raw` to also become the
|
|
754
|
+
transport history store used by outbound ` + "`RequestChanges`" + String.raw`.
|
|
755
|
+
|
|
756
|
+
Reason:
|
|
757
|
+
|
|
758
|
+
- transport history is keyed by ` + "`StoreId`" + String.raw`
|
|
759
|
+
- read-time compaction needs direct access to each store's outbound stream
|
|
760
|
+
- keeping the transport feed as a dedicated server storage abstraction keeps the
|
|
761
|
+
protocol-serving path simpler and mirrors the existing encrypted server module
|
|
762
|
+
|
|
763
|
+
### 3. Relationship to ` + "`SqlEventLogJournal`" + String.raw`
|
|
764
|
+
|
|
765
|
+
` + "`SqlEventLogJournal`" + String.raw` remains a good fit for the **processing journal** side of
|
|
766
|
+
this feature because it already supports:
|
|
767
|
+
|
|
768
|
+
- ` + "`writeFromRemote(...)`" + String.raw`
|
|
769
|
+
- remote sequence tracking
|
|
770
|
+
- conflict-aware effect execution
|
|
771
|
+
- local journal persistence
|
|
772
|
+
|
|
773
|
+
Implementation should keep the transport-store and store-registry contracts
|
|
774
|
+
narrow enough that future SQL-backed adapters can be added without refactoring
|
|
775
|
+
the runtime.
|
|
776
|
+
|
|
777
|
+
## Testing Requirements
|
|
778
|
+
|
|
779
|
+
Add focused tests that follow existing repository patterns.
|
|
780
|
+
|
|
781
|
+
### Core server tests
|
|
782
|
+
|
|
783
|
+
Create ` + "`packages/effect/test/unstable/eventlog/EventLogServerUnencrypted.test.ts`" + String.raw`
|
|
784
|
+
covering at minimum:
|
|
785
|
+
|
|
786
|
+
1. **client writes run handlers for the resolved store**
|
|
787
|
+
- compose a runtime from ` + "`EventLog.group(...)`" + String.raw`
|
|
788
|
+
- map a public key to a store id
|
|
789
|
+
- write an unencrypted batch through the server handler or runtime ingest path
|
|
790
|
+
- assert handler side effects ran
|
|
791
|
+
|
|
792
|
+
2. **two public keys mapped to the same store share transport history**
|
|
793
|
+
- assign public keys A and B to the same store
|
|
794
|
+
- write from A
|
|
795
|
+
- request changes from B
|
|
796
|
+
- assert B receives the same entries and sequence numbers
|
|
797
|
+
|
|
798
|
+
3. **server-authored writes broadcast to mapped clients**
|
|
799
|
+
- assign two public keys to the same store
|
|
800
|
+
- call ` + "`writeToStore({ storeId, ... })`" + String.raw`
|
|
801
|
+
- assert both readers observe the appended entries
|
|
802
|
+
|
|
803
|
+
4. **server-authored writes bypass auth**
|
|
804
|
+
- provide an auth service that rejects all client traffic
|
|
805
|
+
- call ` + "`writeToStore`" + String.raw`
|
|
806
|
+
- assert the write still succeeds and is observable by mapped clients
|
|
807
|
+
|
|
808
|
+
5. **server-authored writes with ` + "`runHandlers: true`" + String.raw` run handlers and reactivity**
|
|
809
|
+
- register a handler and Reactivity observer
|
|
810
|
+
- write to the store with replay enabled
|
|
811
|
+
- assert handler side effects and invalidation occur exactly once per
|
|
812
|
+
committed entry
|
|
813
|
+
|
|
814
|
+
6. **server-authored writes with ` + "`runHandlers: false`" + String.raw` are transport-only**
|
|
815
|
+
- write to the store with replay disabled
|
|
816
|
+
- assert clients receive the events
|
|
817
|
+
- assert handlers do not run
|
|
818
|
+
- assert reactivity does not invalidate
|
|
819
|
+
|
|
820
|
+
7. **unauthorized client write rejects the full batch**
|
|
821
|
+
- ` + "`authorizeWrite`" + String.raw` fails
|
|
822
|
+
- assert ` + "`ErrorUnencrypted`" + String.raw` is returned
|
|
823
|
+
- assert no ` + "`Ack`" + String.raw`
|
|
824
|
+
- assert storage and handler state remain unchanged
|
|
825
|
+
|
|
826
|
+
8. **unmapped public key rejects read and write requests**
|
|
827
|
+
- do not assign a store id
|
|
828
|
+
- assert both client write and ` + "`RequestChanges`" + String.raw` fail with the expected
|
|
829
|
+
protocol error
|
|
830
|
+
|
|
831
|
+
9. **duplicate writes are idempotent but still ack every input slot**
|
|
832
|
+
- write the same entries twice into the same store
|
|
833
|
+
- assert the second ack has the same number of sequence numbers as inputs
|
|
834
|
+
- assert handlers/reactivity only run once per unique entry id
|
|
835
|
+
|
|
836
|
+
10. **read-time compaction only affects old entries in a shared store feed**
|
|
837
|
+
- store an old history batch and a recent batch
|
|
838
|
+
- assert only the old portion is compacted
|
|
839
|
+
- assert recent entries pass through raw
|
|
840
|
+
|
|
841
|
+
11. **compacted outputs preserve cursor progression across public keys mapped to the same store**
|
|
842
|
+
- request from sequence ` + "`0`" + String.raw` and from a later cursor through different keys
|
|
843
|
+
- assert representative output sequences are monotonic and compatible with
|
|
844
|
+
follow-up requests
|
|
845
|
+
|
|
846
|
+
### Remote client tests
|
|
847
|
+
|
|
848
|
+
Add or update tests around ` + "`EventLogRemote`" + String.raw` to verify:
|
|
849
|
+
|
|
850
|
+
1. ` + "`fromSocketUnencrypted`" + String.raw` uses the unencrypted codec path
|
|
851
|
+
2. ` + "`ErrorUnencrypted`" + String.raw` fails pending writes with ` + "`EventLogRemoteError`" + String.raw`
|
|
852
|
+
3. rejected read subscriptions surface an ` + "`EventLogRemoteError`" + String.raw`
|
|
853
|
+
4. store-resolution failures surface as the expected unencrypted protocol error
|
|
854
|
+
5. chunked unencrypted messages round-trip correctly
|
|
855
|
+
|
|
856
|
+
## Validation Expectations
|
|
857
|
+
|
|
858
|
+
Any implementation produced from this spec must run:
|
|
859
|
+
|
|
860
|
+
- ` + "`pnpm lint-fix`" + String.raw`
|
|
861
|
+
- targeted tests, at minimum:
|
|
862
|
+
- ` + "`pnpm test packages/effect/test/unstable/eventlog/EventLogServerUnencrypted.test.ts`" + String.raw`
|
|
863
|
+
- any updated ` + "`EventLog`" + String.raw` / ` + "`EventLogRemote`" + String.raw` tests
|
|
864
|
+
- ` + "`pnpm check:tsgo`" + String.raw`
|
|
865
|
+
- ` + "`pnpm docgen`" + String.raw`
|
|
866
|
+
- ` + "`pnpm codegen`" + String.raw` if new module exports or barrel changes are introduced
|
|
867
|
+
- add a changeset for the unstable eventlog package changes
|
|
868
|
+
|
|
869
|
+
## Implementation Plan
|
|
870
|
+
|
|
871
|
+
The work should be broken into the following atomic, validation-safe tasks.
|
|
872
|
+
Tasks are intentionally grouped so each step can pass linting, tests, and type
|
|
873
|
+
checking on its own.
|
|
874
|
+
|
|
875
|
+
### Task 1: Finish unencrypted protocol plumbing in ` + "`EventLogRemote`" + String.raw`
|
|
876
|
+
|
|
877
|
+
Scope:
|
|
878
|
+
|
|
879
|
+
- add ` + "`ErrorUnencrypted`" + String.raw` to the protocol model
|
|
880
|
+
- fix unencrypted encode/decode usage in ` + "`fromSocketUnencrypted`" + String.raw`
|
|
881
|
+
- correct the misspelled unencrypted request encoder export
|
|
882
|
+
- plumb protocol errors into pending write handling and read-subscription
|
|
883
|
+
handling
|
|
884
|
+
- update/add focused remote protocol tests, including store-resolution failures
|
|
885
|
+
|
|
886
|
+
Why this task is grouped:
|
|
887
|
+
|
|
888
|
+
- protocol message additions and client-side handling must land together or the
|
|
889
|
+
code will not typecheck and tests will fail
|
|
890
|
+
- this task is independently shippable because it only completes the unfinished
|
|
891
|
+
unencrypted client transport behavior
|
|
892
|
+
|
|
893
|
+
Validation for this task:
|
|
894
|
+
|
|
895
|
+
- targeted ` + "`EventLogRemote`" + String.raw` tests
|
|
896
|
+
- any affected ` + "`EventLog`" + String.raw` tests if public remote signatures widen
|
|
897
|
+
|
|
898
|
+
### Task 2: Add store-id, registry, auth, and transport-storage foundations
|
|
899
|
+
|
|
900
|
+
Scope:
|
|
901
|
+
|
|
902
|
+
- define ` + "`StoreId`" + String.raw`, ` + "`EventLogServerStoreRegistry`" + String.raw`, and
|
|
903
|
+
` + "`EventLogServerStoreRegistryError`" + String.raw`
|
|
904
|
+
- define ` + "`EventLogServerAuth`" + String.raw` and ` + "`EventLogServerAuthError`" + String.raw`
|
|
905
|
+
- define store-keyed ` + "`Storage`" + String.raw`, ` + "`layerStorageMemory`" + String.raw`, and a memory
|
|
906
|
+
store-registry helper
|
|
907
|
+
- implement duplicate-aware write results with one ack sequence per input entry
|
|
908
|
+
- add focused registry / storage / auth tests
|
|
909
|
+
|
|
910
|
+
Why this task is grouped:
|
|
911
|
+
|
|
912
|
+
- store resolution, auth, and per-store transport sequencing are inseparable
|
|
913
|
+
foundations for every later runtime behavior
|
|
914
|
+
- splitting duplicate-aware sequencing away from storage would create failing
|
|
915
|
+
intermediate states
|
|
916
|
+
|
|
917
|
+
Validation for this task:
|
|
918
|
+
|
|
919
|
+
- new registry / storage / auth tests
|
|
920
|
+
- compile checks for the new public APIs
|
|
921
|
+
|
|
922
|
+
### Task 3: Build the store-aware client ingest runtime and handler/reactivity integration
|
|
923
|
+
|
|
924
|
+
Scope:
|
|
925
|
+
|
|
926
|
+
- add ` + "`EventLogServerUnencrypted`" + String.raw` runtime service and ` + "`layer(...)`" + String.raw`
|
|
927
|
+
- wire accepted client writes through the store registry, transport storage, and
|
|
928
|
+
processing journal using deterministic store-derived remote ids
|
|
929
|
+
- reuse ` + "`EventLog.group(...)`" + String.raw`-compiled handlers
|
|
930
|
+
- extract or share common handler execution / reactivity helpers from
|
|
931
|
+
` + "`EventLog.ts`" + String.raw`
|
|
932
|
+
- add ` + "`groupReactivity`" + String.raw`
|
|
933
|
+
- add tests proving handlers and reactivity fire exactly once per committed
|
|
934
|
+
client entry
|
|
935
|
+
- add tests proving two public keys mapped to the same store share history
|
|
936
|
+
|
|
937
|
+
Why this task is grouped:
|
|
938
|
+
|
|
939
|
+
- handler execution, deterministic store identity, and shared-feed semantics all
|
|
940
|
+
depend on the same ingest path
|
|
941
|
+
- the resulting runtime is independently shippable even before server-authored
|
|
942
|
+
writes are exposed
|
|
943
|
+
|
|
944
|
+
Validation for this task:
|
|
945
|
+
|
|
946
|
+
- new server runtime tests
|
|
947
|
+
- existing ` + "`EventLog`" + String.raw` tests to ensure shared refactors do not regress client
|
|
948
|
+
behavior
|
|
949
|
+
|
|
950
|
+
### Task 4: Add server-authored store writes with configurable replay
|
|
951
|
+
|
|
952
|
+
Scope:
|
|
953
|
+
|
|
954
|
+
- add ` + "`writeToStore(...)`" + String.raw` or the equivalent public server-authored write API
|
|
955
|
+
- bypass auth for that path
|
|
956
|
+
- support ` + "`runHandlers: true | false`" + String.raw` semantics
|
|
957
|
+
- ensure transport-only writes are still broadcast to clients
|
|
958
|
+
- add tests for broadcast visibility, auth bypass, replay-enabled behavior, and
|
|
959
|
+
replay-disabled behavior
|
|
960
|
+
|
|
961
|
+
Why this task is grouped:
|
|
962
|
+
|
|
963
|
+
- the write API, replay switch, and broadcast tests must land together to avoid
|
|
964
|
+
ambiguous intermediate behavior
|
|
965
|
+
- this task is independently shippable once the client ingest runtime exists
|
|
966
|
+
|
|
967
|
+
Validation for this task:
|
|
968
|
+
|
|
969
|
+
- focused ` + "`EventLogServerUnencrypted`" + String.raw` tests for server-authored writes
|
|
970
|
+
- any affected ` + "`EventLog`" + String.raw` tests if shared helpers change again
|
|
971
|
+
|
|
972
|
+
### Task 5: Add read-time compaction over store feeds
|
|
973
|
+
|
|
974
|
+
Scope:
|
|
975
|
+
|
|
976
|
+
- add server-side ` + "`groupCompaction(...)`" + String.raw` with ` + "`olderThan`" + String.raw`
|
|
977
|
+
- implement compaction of outbound initial backlog / eligible batches for a
|
|
978
|
+
resolved store feed
|
|
979
|
+
- preserve representative remote sequences for cursor compatibility across all
|
|
980
|
+
public keys mapped to the same store
|
|
981
|
+
- add tests for old-vs-recent eligibility and cursor progression
|
|
982
|
+
|
|
983
|
+
Why this task is grouped:
|
|
984
|
+
|
|
985
|
+
- helper registration, runtime projection logic, and cursor-preservation tests
|
|
986
|
+
must land together to avoid broken feed semantics
|
|
987
|
+
- it is independently shippable once store-aware ingest and server writes exist
|
|
988
|
+
|
|
989
|
+
Validation for this task:
|
|
990
|
+
|
|
991
|
+
- server compaction tests
|
|
992
|
+
- existing event log compaction tests, if any shared helper extraction affects
|
|
993
|
+
client behavior
|
|
994
|
+
|
|
995
|
+
### Task 6: Expose websocket / HTTP handlers and complete integration coverage
|
|
996
|
+
|
|
997
|
+
Scope:
|
|
998
|
+
|
|
999
|
+
- implement ` + "`makeHandler`" + String.raw` and ` + "`makeHandlerHttp`" + String.raw`
|
|
1000
|
+
- connect auth, store resolution, storage, runtime, chunk handling, ack/error
|
|
1001
|
+
responses, and change subscriptions
|
|
1002
|
+
- add end-to-end websocket-level tests for authorized writes, unmapped-key
|
|
1003
|
+
errors, server-authored broadcast behavior, and change subscriptions
|
|
1004
|
+
- regenerate exports if necessary and add the required changeset
|
|
1005
|
+
|
|
1006
|
+
Why this task is grouped:
|
|
1007
|
+
|
|
1008
|
+
- the transport handlers are the first point where all preceding contracts meet;
|
|
1009
|
+
partial integration would leave validation broken
|
|
1010
|
+
- this task yields the complete user-facing module
|
|
1011
|
+
|
|
1012
|
+
Validation for this task:
|
|
1013
|
+
|
|
1014
|
+
- end-to-end ` + "`EventLogServerUnencrypted`" + String.raw` tests
|
|
1015
|
+
- ` + "`pnpm lint-fix`" + String.raw`
|
|
1016
|
+
- ` + "`pnpm check:tsgo`" + String.raw`
|
|
1017
|
+
- ` + "`pnpm docgen`" + String.raw`
|
|
1018
|
+
- ` + "`pnpm codegen`" + String.raw` if needed
|
|
1019
|
+
|
|
1020
|
+
## Open Follow-Ups
|
|
1021
|
+
|
|
1022
|
+
These are explicitly out of scope for the initial implementation but should be
|
|
1023
|
+
kept in mind while designing the API:
|
|
1024
|
+
|
|
1025
|
+
- a SQL-backed transport storage implementation for unencrypted server history
|
|
1026
|
+
- a SQL-backed ` + "`EventLogServerStoreRegistry`" + String.raw` helper or companion module
|
|
1027
|
+
- optional success acknowledgements for ` + "`RequestChanges`" + String.raw` subscriptions
|
|
1028
|
+
- extending explicit protocol error frames to the encrypted server path
|
|
1029
|
+
- richer typed convenience APIs for server-authored writes so callers do not
|
|
1030
|
+
need to manually construct ` + "`Entry`" + String.raw` values when that ergonomics becomes
|
|
1031
|
+
important
|
|
1032
|
+
`
|