foldkit-sync 0.4.0 → 0.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +815 -256
- package/dist/index.d.mts +54 -8
- package/dist/index.d.mts.map +1 -1
- package/dist/index.mjs +83 -25
- package/dist/index.mjs.map +1 -1
- package/package.json +4 -3
package/README.md
CHANGED
|
@@ -1,29 +1,123 @@
|
|
|
1
1
|
# `foldkit-sync`
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
3
|
+
Local-first replication for part of a Foldkit application.
|
|
4
|
+
|
|
5
|
+
A durable edit applies locally immediately, is persisted to an outbox, and is
|
|
6
|
+
sent when the network allows. The server decides the authoritative order. The
|
|
7
|
+
replica then replays that committed order and rebases any still-pending local
|
|
8
|
+
edits on top.
|
|
9
|
+
|
|
10
|
+
The important part is what **does not** change: your application's `Model`,
|
|
11
|
+
`Message`, and `update` remain the state machine. Sync does not introduce a
|
|
12
|
+
second reducer. You declare which Model slice is shared and which existing
|
|
13
|
+
Messages are durable; Sync derives replay, persistence, protocol codecs, and the
|
|
14
|
+
server journal contract from that declaration.
|
|
15
|
+
|
|
16
|
+
> **Sync turns a subset of your existing Foldkit Messages into durable
|
|
17
|
+
> operations. They still run through your normal `update`; Sync adds persistence,
|
|
18
|
+
> optimistic replay, server ordering, and reconciliation around them.**
|
|
19
|
+
|
|
20
|
+
**Use it when** a user should keep editing offline, the edit must survive a
|
|
21
|
+
reload or network failure, and several replicas must eventually converge under
|
|
22
|
+
one authoritative server order. **Do not use it for** server-owned disposable
|
|
23
|
+
facts — use [`foldkit-remote`](../remote) for those — or for peer-to-peer / CRDT
|
|
24
|
+
replication where every peer independently accepts writes.
|
|
25
|
+
|
|
26
|
+
It is the client half of [replicated Foldkit state](../../docs/replication.md).
|
|
27
|
+
[`foldkit-durable`](../durable) is an optional server-side ordered journal.
|
|
28
|
+
|
|
29
|
+
## Which state belongs in Sync?
|
|
30
|
+
|
|
31
|
+
A useful ownership rule is:
|
|
32
|
+
|
|
33
|
+
| State | Owner |
|
|
34
|
+
| --- | --- |
|
|
35
|
+
| Route, selected tab, transient errors | ordinary local `Model` / `update` |
|
|
36
|
+
| Server-owned data the client may refetch | `foldkit-remote` |
|
|
37
|
+
| Client-authored state that must survive offline and converge | `foldkit-sync` |
|
|
38
|
+
| Authoritative server ordering, idempotent append, snapshots | `foldkit-durable` |
|
|
39
|
+
|
|
40
|
+
The difference between Remote and Sync is especially important:
|
|
41
|
+
|
|
42
|
+
```text
|
|
43
|
+
Remote
|
|
44
|
+
server owns the fact
|
|
45
|
+
client caches it
|
|
46
|
+
refetch is recovery
|
|
47
|
+
|
|
48
|
+
Sync
|
|
49
|
+
client creates the durable operation
|
|
50
|
+
operation must survive offline
|
|
51
|
+
replay + reconciliation is recovery
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
A Surface may read local state, Remote state, and Sync-owned state at the same
|
|
55
|
+
time. Those observation boundaries do not change who owns each datum.
|
|
56
|
+
|
|
57
|
+
## The replica mental model
|
|
58
|
+
|
|
59
|
+
A replica has two pieces of state:
|
|
60
|
+
|
|
61
|
+
```text
|
|
62
|
+
authoritative committed snapshot
|
|
63
|
+
+
|
|
64
|
+
pending local operations
|
|
65
|
+
=
|
|
66
|
+
optimistic shared state the UI sees
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
If the server has committed `A B C` and this device has pending `D E`:
|
|
70
|
+
|
|
71
|
+
```text
|
|
72
|
+
server order: A B C
|
|
73
|
+
local pending: D E
|
|
74
|
+
|
|
75
|
+
visible state = replay(A, B, C, D, E)
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
If another device gets operation `X` committed before this replica's pending
|
|
79
|
+
work, the next exchange may produce:
|
|
80
|
+
|
|
81
|
+
```text
|
|
82
|
+
new server order: A B C X
|
|
83
|
+
local pending: D E
|
|
84
|
+
|
|
85
|
+
new visible state = replay(A, B, C, X, D, E)
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
The local edit never needed to wait for that exchange to appear. Reconciliation
|
|
89
|
+
changes the base underneath it and replays the still-pending operations on top.
|
|
90
|
+
|
|
91
|
+
The full lifecycle of one durable Message is:
|
|
92
|
+
|
|
93
|
+
```text
|
|
94
|
+
user dispatches durable Message
|
|
95
|
+
|
|
|
96
|
+
v
|
|
97
|
+
replay through application update immediately
|
|
98
|
+
|
|
|
99
|
+
v
|
|
100
|
+
persist operation in local outbox
|
|
101
|
+
|
|
|
102
|
+
v
|
|
103
|
+
optimistic state is visible now
|
|
104
|
+
|
|
|
105
|
+
network later
|
|
106
|
+
v
|
|
107
|
+
server assigns authoritative order
|
|
108
|
+
|
|
|
109
|
+
v
|
|
110
|
+
replica adopts committed operations / checkpoint
|
|
111
|
+
|
|
|
112
|
+
v
|
|
113
|
+
drop acknowledged or rejected pending operations
|
|
114
|
+
|
|
|
115
|
+
v
|
|
116
|
+
replay remaining pending operations on the new base
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
That is the core of `foldkit-sync`. Transport, browser mounting, presence, and
|
|
120
|
+
LWW fields are layers around this mechanism.
|
|
27
121
|
|
|
28
122
|
## Install
|
|
29
123
|
|
|
@@ -31,15 +125,12 @@ covers the mental model and when not to use it.
|
|
|
31
125
|
pnpm add foldkit-sync
|
|
32
126
|
```
|
|
33
127
|
|
|
34
|
-
`foldkit` and `effect` are peer dependencies; `foldkit-surface` comes with
|
|
35
|
-
|
|
128
|
+
`foldkit` and `effect` are peer dependencies; `foldkit-surface` comes with the
|
|
129
|
+
package.
|
|
36
130
|
|
|
37
|
-
##
|
|
131
|
+
## Sixty seconds: choose what is shared and what is durable
|
|
38
132
|
|
|
39
|
-
|
|
40
|
-
`Surface.application` and a writable projection it derives the shared schema,
|
|
41
|
-
the durable Message subset, the initial snapshot, and replay, and exposes a
|
|
42
|
-
read-only Surface over the same projection.
|
|
133
|
+
Start with an ordinary Foldkit application:
|
|
43
134
|
|
|
44
135
|
```ts
|
|
45
136
|
import { Schema } from 'effect'
|
|
@@ -49,7 +140,9 @@ import { MessageSet, Projection, Surface } from 'foldkit-surface'
|
|
|
49
140
|
import { DocumentId, Sync } from 'foldkit-sync'
|
|
50
141
|
|
|
51
142
|
const Model = Schema.Struct({
|
|
52
|
-
todos: Schema.Array(
|
|
143
|
+
todos: Schema.Array(
|
|
144
|
+
Schema.Struct({ id: Schema.String, title: Schema.String }),
|
|
145
|
+
),
|
|
53
146
|
selectedTodoId: Schema.NullOr(Schema.String),
|
|
54
147
|
lastError: Schema.NullOr(Schema.String),
|
|
55
148
|
})
|
|
@@ -62,10 +155,15 @@ const Message = defineMessageUnion({
|
|
|
62
155
|
})
|
|
63
156
|
type Message = typeof Message.Type
|
|
64
157
|
|
|
65
|
-
const initial: Model = {
|
|
158
|
+
const initial: Model = {
|
|
159
|
+
todos: [],
|
|
160
|
+
selectedTodoId: null,
|
|
161
|
+
lastError: null,
|
|
162
|
+
}
|
|
66
163
|
|
|
67
164
|
type Return = Update.Return<Model, Message>
|
|
68
|
-
|
|
165
|
+
|
|
166
|
+
// This is still the application's one transition function.
|
|
69
167
|
const update = (model: Model, message: Message): Return =>
|
|
70
168
|
Message.match<Return>(message, {
|
|
71
169
|
CreatedTodo: ({ id, title }) => ({
|
|
@@ -74,74 +172,386 @@ const update = (model: Model, message: Message): Return =>
|
|
|
74
172
|
RenamedTodo: ({ id, title }) => ({
|
|
75
173
|
model: {
|
|
76
174
|
...model,
|
|
77
|
-
todos: model.todos.map(todo =>
|
|
175
|
+
todos: model.todos.map(todo =>
|
|
176
|
+
todo.id === id ? { ...todo, title } : todo,
|
|
177
|
+
),
|
|
78
178
|
},
|
|
79
179
|
}),
|
|
80
|
-
SelectedTodo: ({ id }) => ({
|
|
180
|
+
SelectedTodo: ({ id }) => ({
|
|
181
|
+
model: { ...model, selectedTodoId: id },
|
|
182
|
+
}),
|
|
81
183
|
})
|
|
82
184
|
|
|
83
185
|
const App = Surface.application({ Model, Message, initial, update })
|
|
84
186
|
|
|
85
187
|
const TodoSync = Sync.forApplication(App).make({
|
|
86
188
|
documentId: DocumentId.make('todos'),
|
|
87
|
-
|
|
88
|
-
|
|
189
|
+
|
|
190
|
+
// This slice of Model is replicated.
|
|
191
|
+
// The writable Projection lets Sync install checkpoints/reconciled state back
|
|
192
|
+
// into the application Model without owning a second copy of the schema.
|
|
193
|
+
shared: Projection.pick(App.fields.todos),
|
|
194
|
+
|
|
195
|
+
// Only these existing Messages become durable operations.
|
|
196
|
+
durable: MessageSet.make(App, [
|
|
197
|
+
Message.CreatedTodo,
|
|
198
|
+
Message.RenamedTodo,
|
|
199
|
+
]),
|
|
89
200
|
})
|
|
90
201
|
```
|
|
91
202
|
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
203
|
+
Read that declaration as:
|
|
204
|
+
|
|
205
|
+
```text
|
|
206
|
+
Model
|
|
207
|
+
todos -> shared / replicated
|
|
208
|
+
selectedTodoId -> local only
|
|
209
|
+
lastError -> local only
|
|
210
|
+
|
|
211
|
+
Messages
|
|
212
|
+
CreatedTodo -> durable
|
|
213
|
+
RenamedTodo -> durable
|
|
214
|
+
SelectedTodo -> local only
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
`Sync.forApplication(App)` derives the shared schema, initial shared snapshot,
|
|
218
|
+
replay function, operation codecs, and server journal contract from the same
|
|
219
|
+
application declaration. There is no parallel sync-specific version of
|
|
220
|
+
`update` to keep aligned.
|
|
221
|
+
|
|
222
|
+
## Wiring: the contract joins an assembly
|
|
223
|
+
|
|
224
|
+
The contract joins a `foldkit-bundle` assembly contract-only, so the Module
|
|
225
|
+
sees it:
|
|
226
|
+
|
|
227
|
+
```ts
|
|
228
|
+
const wiring = TodoSync.wiring()
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
It routes nothing, runs nothing, and subscribes to nothing — Sync owns the
|
|
232
|
+
runtime through `mount` rather than joining it — and `wiring.contract` is the
|
|
233
|
+
same contract above.
|
|
234
|
+
|
|
235
|
+
## What makes a Message durable?
|
|
236
|
+
|
|
237
|
+
A durable Message must be something another machine can replay later and obtain
|
|
238
|
+
the same shared state.
|
|
239
|
+
|
|
240
|
+
In practice that means it must be a deterministic, state-only transition of the
|
|
241
|
+
shared projection.
|
|
242
|
+
|
|
243
|
+
Derived replay protects that boundary. It refuses a durable Message when the
|
|
244
|
+
application's `update`:
|
|
245
|
+
|
|
246
|
+
- returns a Command — a live external effect cannot be replayed later; or
|
|
247
|
+
- changes a field outside the shared projection — that change would disappear
|
|
248
|
+
when only the shared slice is persisted/replayed.
|
|
249
|
+
|
|
250
|
+
For example:
|
|
251
|
+
|
|
252
|
+
```text
|
|
253
|
+
RequestedChargeCard
|
|
254
|
+
-> runs a Command / talks to a provider
|
|
255
|
+
-> local intent, NOT durable
|
|
256
|
+
|
|
257
|
+
CardCharged
|
|
258
|
+
-> pure fact about state
|
|
259
|
+
-> durable and replayable
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
A common pattern is therefore:
|
|
263
|
+
|
|
264
|
+
```text
|
|
265
|
+
local intent Message
|
|
266
|
+
|
|
|
267
|
+
v
|
|
268
|
+
Command / nondeterministic work
|
|
269
|
+
|
|
|
270
|
+
v
|
|
271
|
+
durable fact Message
|
|
272
|
+
```
|
|
273
|
+
|
|
274
|
+
Replay runs the durable fact through the same application `update` on every
|
|
275
|
+
replica. If an application genuinely needs different replay semantics, `make`
|
|
276
|
+
accepts a custom `replay`, but that custom replay does not get the derived
|
|
277
|
+
guardrails.
|
|
95
278
|
|
|
96
|
-
|
|
97
|
-
the shared projection. Replay refuses one whose `update` returns a Command (a
|
|
98
|
-
live effect cannot be replayed) or changes a field outside the projection (the
|
|
99
|
-
change would be silently lost), naming the Message and the fields. A Message that
|
|
100
|
-
needs an effect stays local and emits a durable fact once the effect settles:
|
|
101
|
-
`RequestedChargeCard` runs the Command; `CardCharged` is what replicates.
|
|
102
|
-
Pass `replay` to `make` when an application needs a custom reducer over the
|
|
103
|
-
shared slice; a custom replay is not guarded.
|
|
279
|
+
## Running a replica
|
|
104
280
|
|
|
105
|
-
|
|
106
|
-
|
|
281
|
+
The low-level runtime is a `Replica`. It owns the local outbox, committed shared
|
|
282
|
+
snapshot, cursor, optimistic projection, and synchronization state.
|
|
107
283
|
|
|
108
284
|
```ts
|
|
109
285
|
import { Effect } from 'effect'
|
|
110
|
-
import {
|
|
286
|
+
import { ReplicaId, Sync } from 'foldkit-sync'
|
|
287
|
+
|
|
288
|
+
const program = Effect.gen(function* () {
|
|
289
|
+
// One durable storage identity per document + replica/writer.
|
|
290
|
+
const storage = yield* Sync.indexedDb('todos/tab-1')
|
|
291
|
+
|
|
292
|
+
const replica = yield* TodoSync.openReplica(
|
|
293
|
+
ReplicaId.make('tab-1'),
|
|
294
|
+
storage,
|
|
295
|
+
)
|
|
296
|
+
|
|
297
|
+
// submit validates/replays first, then persists the operation in the outbox.
|
|
298
|
+
yield* replica.submit(
|
|
299
|
+
Message.CreatedTodo({ id: 't1', title: 'Milk' }),
|
|
300
|
+
)
|
|
301
|
+
|
|
302
|
+
// This already includes the pending operation. No server round trip required.
|
|
303
|
+
const optimistic = yield* replica.shared
|
|
304
|
+
|
|
305
|
+
return optimistic
|
|
306
|
+
}).pipe(Effect.scoped)
|
|
307
|
+
```
|
|
308
|
+
|
|
309
|
+
`replica.submit(message)` does not mean “send this now.” It means:
|
|
310
|
+
|
|
311
|
+
1. verify the Message is valid and durable;
|
|
312
|
+
2. replay it against the current optimistic shared state;
|
|
313
|
+
3. assign a stable local operation identity;
|
|
314
|
+
4. persist it to the local outbox;
|
|
315
|
+
5. make the new optimistic projection observable; and
|
|
316
|
+
6. wake the exchange loop if one is running.
|
|
317
|
+
|
|
318
|
+
A Message replay rejects never enters the outbox.
|
|
319
|
+
|
|
320
|
+
## Optimistic state and rebasing
|
|
321
|
+
|
|
322
|
+
Internally the replica keeps:
|
|
323
|
+
|
|
324
|
+
```text
|
|
325
|
+
committed
|
|
326
|
+
the authoritative shared snapshot up to cursor N
|
|
327
|
+
|
|
328
|
+
pending
|
|
329
|
+
locally-authored operations not yet resolved by the server
|
|
330
|
+
```
|
|
331
|
+
|
|
332
|
+
The value returned by `replica.shared` is:
|
|
333
|
+
|
|
334
|
+
```text
|
|
335
|
+
pending.reduce(replay, committed)
|
|
336
|
+
```
|
|
337
|
+
|
|
338
|
+
That simple rule gives optimistic UI and reconciliation the same semantics.
|
|
339
|
+
There is no special “optimistic reducer.”
|
|
340
|
+
|
|
341
|
+
When synchronization advances the committed base, Sync removes pending
|
|
342
|
+
operations that became committed, were explicitly acknowledged, or were
|
|
343
|
+
rejected, then the remaining outbox is replayed over the new committed state.
|
|
344
|
+
|
|
345
|
+
This also explains why durable Messages have to be replay-safe: a pending
|
|
346
|
+
operation may run many times as the authoritative base changes beneath it.
|
|
347
|
+
|
|
348
|
+
## Synchronization
|
|
349
|
+
|
|
350
|
+
A replica exchanges two things with the server:
|
|
351
|
+
|
|
352
|
+
```text
|
|
353
|
+
request
|
|
354
|
+
current cursor
|
|
355
|
+
pending operations
|
|
356
|
+
|
|
357
|
+
response
|
|
358
|
+
newly committed operations
|
|
359
|
+
acknowledgements
|
|
360
|
+
rejections
|
|
361
|
+
optional checkpoint
|
|
362
|
+
```
|
|
363
|
+
|
|
364
|
+
`replica.synchronize` performs one exchange through the `Transport` Effect
|
|
365
|
+
service:
|
|
366
|
+
|
|
367
|
+
```ts
|
|
368
|
+
const once = replica.synchronize.pipe(
|
|
369
|
+
Effect.provide(Sync.transport.socket({ url: 'wss://example.com/sync' })),
|
|
370
|
+
)
|
|
371
|
+
```
|
|
372
|
+
|
|
373
|
+
`replica.start` is the long-running convenience loop. It exchanges once, then
|
|
374
|
+
wakes after every submit. Transport failures are recorded in
|
|
375
|
+
`status.lastError`; the loop survives and retries on the next wake.
|
|
376
|
+
|
|
377
|
+
```ts
|
|
378
|
+
Effect.runFork(
|
|
379
|
+
Effect.provide(
|
|
380
|
+
replica.start,
|
|
381
|
+
Sync.transport.socket({ url: 'wss://example.com/sync' }),
|
|
382
|
+
),
|
|
383
|
+
)
|
|
384
|
+
```
|
|
385
|
+
|
|
386
|
+
The transport does not decide reconciliation. It only carries the exchange;
|
|
387
|
+
the replica validates the response and applies the protocol rules.
|
|
388
|
+
|
|
389
|
+
### Acknowledgements
|
|
390
|
+
|
|
391
|
+
An acknowledgement says an operation the replica sent is durably accepted, so
|
|
392
|
+
it may leave the outbox even if its committed payload is not repeated in this
|
|
393
|
+
response.
|
|
394
|
+
|
|
395
|
+
### Rejections
|
|
396
|
+
|
|
397
|
+
A rejection says the optimistic operation did not commit. Sync removes it from
|
|
398
|
+
the pending set and recomputes the optimistic state from the authoritative base
|
|
399
|
+
plus whatever pending operations remain.
|
|
400
|
+
|
|
401
|
+
```text
|
|
402
|
+
before response
|
|
403
|
+
committed: A B C
|
|
404
|
+
pending: D E
|
|
405
|
+
visible: A B C D E
|
|
406
|
+
|
|
407
|
+
server rejects D
|
|
408
|
+
|
|
409
|
+
committed: A B C
|
|
410
|
+
pending: E
|
|
411
|
+
visible: A B C E
|
|
412
|
+
```
|
|
413
|
+
|
|
414
|
+
`replica.status.rejected` keeps recent rejected operation ids so the UI can
|
|
415
|
+
explain that an optimistic edit was refused.
|
|
416
|
+
|
|
417
|
+
### Checkpoints
|
|
418
|
+
|
|
419
|
+
A server may compact old history. If the replica is too far behind to replay the
|
|
420
|
+
missing prefix, the response can include a checkpoint:
|
|
421
|
+
|
|
422
|
+
```text
|
|
423
|
+
checkpoint = authoritative shared snapshot + cursor
|
|
424
|
+
```
|
|
425
|
+
|
|
426
|
+
The replica adopts the checkpoint as its committed base, then rebases its local
|
|
427
|
+
pending operations on top. A checkpoint behind the replica's current cursor is
|
|
428
|
+
rejected with `CheckpointRegressionError` rather than silently moving history
|
|
429
|
+
backwards.
|
|
430
|
+
|
|
431
|
+
## Replica status and UI state
|
|
432
|
+
|
|
433
|
+
The public replica surface deliberately exposes enough state for UI and recovery
|
|
434
|
+
without exposing the application Messages themselves:
|
|
435
|
+
|
|
436
|
+
```ts
|
|
437
|
+
const status = yield* replica.status
|
|
438
|
+
// {
|
|
439
|
+
// pending: number,
|
|
440
|
+
// cursor: Sequence,
|
|
441
|
+
// lastError: string | undefined,
|
|
442
|
+
// rejected: ReadonlyArray<OpId>
|
|
443
|
+
// }
|
|
444
|
+
```
|
|
445
|
+
|
|
446
|
+
`replica.statusChanges` emits that status initially and after every submit and
|
|
447
|
+
exchange. `replica.changes` emits the status and optimistic `shared` value
|
|
448
|
+
together from one replica snapshot, so a UI can subscribe once without racing
|
|
449
|
+
two separate reads. `replica.snapshot` is the same read, once: the status,
|
|
450
|
+
`shared`, `committed`, and `nextLocalSequence`, the local sequence the next
|
|
451
|
+
submitted operation takes.
|
|
452
|
+
|
|
453
|
+
## Sync + Durable
|
|
454
|
+
|
|
455
|
+
Sync owns **client replication semantics**. Durable can own **server ordering and
|
|
456
|
+
persistence**.
|
|
457
|
+
|
|
458
|
+
```text
|
|
459
|
+
client
|
|
460
|
+
-----------------------------------------------
|
|
461
|
+
Foldkit update
|
|
462
|
+
|
|
|
463
|
+
| durable Message
|
|
464
|
+
v
|
|
465
|
+
foldkit-sync Replica
|
|
466
|
+
optimistic shared state
|
|
467
|
+
persistent outbox
|
|
468
|
+
|
|
|
469
|
+
| exchange
|
|
470
|
+
v
|
|
471
|
+
-----------------------------------------------
|
|
472
|
+
server
|
|
473
|
+
foldkit-durable Journal
|
|
474
|
+
authoritative order
|
|
475
|
+
snapshot + cursor
|
|
476
|
+
|
|
|
477
|
+
v
|
|
478
|
+
committed operations / checkpoint / rejection
|
|
479
|
+
```
|
|
480
|
+
|
|
481
|
+
The same Sync contract produces the server journal contract, so the two halves
|
|
482
|
+
do not duplicate the Message codec, shared schema, initial snapshot, or replay
|
|
483
|
+
reducer:
|
|
484
|
+
|
|
485
|
+
```ts
|
|
486
|
+
import { Effect } from 'effect'
|
|
487
|
+
import { ActorId, DocumentId, Journal, OpId } from 'foldkit-durable'
|
|
111
488
|
|
|
112
489
|
type Principal = { readonly actorId: string }
|
|
113
490
|
|
|
114
491
|
const server = Effect.gen(function* () {
|
|
115
|
-
const journal = yield*
|
|
116
|
-
//
|
|
492
|
+
const journal = yield* Journal.make({
|
|
493
|
+
// Derived from the same Foldkit application + Sync declaration:
|
|
494
|
+
// operation codec, snapshot codec, empty shared value, replay, and
|
|
495
|
+
// compiled authorization rules when present.
|
|
117
496
|
...TodoSync.journalContract(),
|
|
497
|
+
|
|
118
498
|
file: 'todos.sqlite',
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
actorId: (principal: Principal) => actorId(principal.actorId),
|
|
499
|
+
opId: operation => OpId.make(operation.opId),
|
|
500
|
+
actorId: (principal: Principal) => ActorId.make(principal.actorId),
|
|
122
501
|
})
|
|
123
|
-
|
|
502
|
+
|
|
503
|
+
return yield* journal.load(DocumentId.make('todos'))
|
|
124
504
|
}).pipe(Effect.scoped)
|
|
125
505
|
```
|
|
126
506
|
|
|
127
|
-
|
|
507
|
+
The packages are intentionally separable. A custom server may implement the
|
|
508
|
+
exchange without Durable, and Durable can journal a protocol that did not come
|
|
509
|
+
from Sync.
|
|
510
|
+
|
|
511
|
+
## Mounting the full Foldkit application
|
|
512
|
+
|
|
513
|
+
The raw Replica works with the **shared slice**. `Sync.mount` is the Foldkit
|
|
514
|
+
runtime integration that connects that replica back to the whole application
|
|
515
|
+
Model.
|
|
516
|
+
|
|
517
|
+
A mounted durable Message:
|
|
518
|
+
|
|
519
|
+
```text
|
|
520
|
+
dispatch Message
|
|
521
|
+
|
|
|
522
|
+
v
|
|
523
|
+
application update immediately
|
|
524
|
+
|
|
|
525
|
+
+--> local fields / render
|
|
526
|
+
|
|
|
527
|
+
v
|
|
528
|
+
persist durable Message through replica
|
|
529
|
+
|
|
|
530
|
+
exchange changes shared state later
|
|
531
|
+
v
|
|
532
|
+
install reconciled shared slice back into Model
|
|
533
|
+
```
|
|
128
534
|
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
persisted afterwards in a Command; the shared slice is re-installed from the
|
|
132
|
-
replica when an exchange or a rejection changes it, or when a persist fails.
|
|
535
|
+
So the UI still runs one application reducer. Sync does not replace the runtime
|
|
536
|
+
with a second state machine.
|
|
133
537
|
|
|
134
538
|
```ts
|
|
135
539
|
import { Effect, Scope } from 'effect'
|
|
136
540
|
import { ReplicaId, Sync } from 'foldkit-sync'
|
|
137
541
|
|
|
138
|
-
const container = document.getElementById('app')!
|
|
542
|
+
const container = document.getElementById('app')!
|
|
139
543
|
|
|
140
544
|
const scope = Effect.runSync(Scope.make())
|
|
141
545
|
const storage = Effect.runSync(
|
|
142
|
-
Effect.provideService(
|
|
546
|
+
Effect.provideService(
|
|
547
|
+
Sync.indexedDb('todos/tab-1'),
|
|
548
|
+
Scope.Scope,
|
|
549
|
+
scope,
|
|
550
|
+
),
|
|
551
|
+
)
|
|
552
|
+
const replica = Effect.runSync(
|
|
553
|
+
TodoSync.openReplica(ReplicaId.make('tab-1'), storage),
|
|
143
554
|
)
|
|
144
|
-
const replica = Effect.runSync(TodoSync.openReplica(ReplicaId.make('tab-1'), storage))
|
|
145
555
|
|
|
146
556
|
const mounted = Sync.mount(App, TodoSync, {
|
|
147
557
|
replica,
|
|
@@ -153,183 +563,224 @@ const mounted = Sync.mount(App, TodoSync, {
|
|
|
153
563
|
model.todos.map(todo => h.li([], [todo.title])),
|
|
154
564
|
),
|
|
155
565
|
}),
|
|
156
|
-
onPersistenceFailure: (model, error) => ({
|
|
566
|
+
onPersistenceFailure: (model, error) => ({
|
|
567
|
+
...model,
|
|
568
|
+
lastError: error.message,
|
|
569
|
+
}),
|
|
157
570
|
})
|
|
158
571
|
|
|
159
|
-
// The exchange loop: once, then after every submit, retrying a failed exchange.
|
|
160
572
|
Effect.runFork(
|
|
161
|
-
Effect.provide(
|
|
573
|
+
Effect.provide(
|
|
574
|
+
replica.start,
|
|
575
|
+
Sync.transport.socket({ url: 'wss://example.com/sync' }),
|
|
576
|
+
),
|
|
162
577
|
)
|
|
163
578
|
|
|
164
|
-
mounted.dispatch(
|
|
165
|
-
|
|
166
|
-
|
|
579
|
+
mounted.dispatch(
|
|
580
|
+
Message.CreatedTodo({ id: crypto.randomUUID(), title: 'Milk' }),
|
|
581
|
+
)
|
|
582
|
+
|
|
583
|
+
mounted.model()
|
|
584
|
+
await mounted.dispose()
|
|
167
585
|
```
|
|
168
586
|
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
`url: { init: (model, url) => Filters.reduce(model, url), onUrlChange: url => Message.UrlChanged({ url }) }`;
|
|
176
|
-
`subscriptions` and `resources` carry its write entry and the store layer.
|
|
587
|
+
`dispose()` waits for in-flight persists started by the mount; the Replica itself
|
|
588
|
+
remains separately owned and must be closed with its scope/lifecycle.
|
|
589
|
+
|
|
590
|
+
### URL, Mirror, and agents
|
|
591
|
+
|
|
592
|
+
The optional `url` option routes navigation through the application:
|
|
177
593
|
|
|
178
|
-
`
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
[docs/sync-runtime-binding.md](https://github.com/doeixd/foldkit-plus/blob/main/docs/sync-runtime-binding.md) for what the
|
|
182
|
-
mount guarantees and why no Foldkit change is required.
|
|
594
|
+
- `init(model, url)` reduces the initial URL before the first render;
|
|
595
|
+
- `onUrlChange(url)` maps browser navigation back to a Message;
|
|
596
|
+
- `onUrlRequest` optionally maps link requests to a Message.
|
|
183
597
|
|
|
184
|
-
|
|
598
|
+
A `foldkit-mirror` URL mirror plugs into that seam without becoming another
|
|
599
|
+
state owner.
|
|
185
600
|
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
601
|
+
The mounted host also exposes `model`, `dispatch`, `subscribe`, and `observe`.
|
|
602
|
+
Agent runtimes bind to those same capabilities; `observe` reports application
|
|
603
|
+
Messages the runtime applies, including the completion facts an agent capability
|
|
604
|
+
may wait for. See [runtime binding](../../docs/sync-runtime-binding.md).
|
|
605
|
+
|
|
606
|
+
The Model shows a durable edit at once, before the server has it. An agent that
|
|
607
|
+
must not report success until the server commits the edit waits on
|
|
608
|
+
`mounted.committed` instead, the shared slice as confirmed:
|
|
609
|
+
|
|
610
|
+
```ts
|
|
611
|
+
completion: Agent.when({
|
|
612
|
+
source: mounted.committed,
|
|
613
|
+
predicate: (shared, request) => shared.todos.some(todo => todo.id === request.id),
|
|
614
|
+
})
|
|
615
|
+
```
|
|
616
|
+
|
|
617
|
+
It completes after the exchange that commits the edit, and never for one the
|
|
618
|
+
server rejects. The agent learns nothing about cursors or operations.
|
|
619
|
+
`mounted.committed` is a source (`{ get, subscribe }`), not a Projection: it reads
|
|
620
|
+
the replica, not the Model, and tells subscribers after every exchange.
|
|
621
|
+
|
|
622
|
+
## Fragments: compose one document from features
|
|
623
|
+
|
|
624
|
+
Large applications do not need one giant Sync declaration. Declare feature
|
|
625
|
+
fragments and compose them into one document:
|
|
189
626
|
|
|
190
627
|
```ts
|
|
191
628
|
const AppSync = Sync.forApplication(App)
|
|
629
|
+
|
|
192
630
|
const Todos = AppSync.fragment({
|
|
193
631
|
shared: Projection.pick(App.fields.todos),
|
|
194
|
-
durable: MessageSet.make(App, [
|
|
632
|
+
durable: MessageSet.make(App, [
|
|
633
|
+
Message.CreatedTodo,
|
|
634
|
+
Message.RenamedTodo,
|
|
635
|
+
]),
|
|
195
636
|
})
|
|
637
|
+
|
|
196
638
|
const Members = AppSync.fragment({
|
|
197
639
|
shared: Projection.pick(App.fields.members),
|
|
198
640
|
durable: MessageSet.make(App, [Message.Invited]),
|
|
199
641
|
})
|
|
642
|
+
|
|
200
643
|
const Board = AppSync.make({
|
|
201
644
|
documentId: DocumentId.make('board'),
|
|
202
645
|
...AppSync.compose(Todos, Members),
|
|
203
646
|
})
|
|
204
647
|
```
|
|
205
648
|
|
|
206
|
-
`compose` merges the shared projections and
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
649
|
+
`compose` merges the shared projections and durable Message subsets. It rejects
|
|
650
|
+
incompatible duplicate field declarations, duplicate durable variants, and
|
|
651
|
+
fragments from another application. `Module.validate` catches the separate
|
|
652
|
+
architectural mistake of two contracts claiming the same field ownership.
|
|
210
653
|
|
|
211
|
-
|
|
654
|
+
## Authorization and optimistic rejection
|
|
212
655
|
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
`
|
|
656
|
+
Authorization belongs on the Sync contract because it is policy over a durable
|
|
657
|
+
Message against the authoritative shared snapshot. The rule compiles into
|
|
658
|
+
`journalContract()` and is enforced by the server journal — not trusted to the
|
|
659
|
+
client.
|
|
216
660
|
|
|
217
661
|
```ts
|
|
218
|
-
const Authorized = Sync.forApplication(App)
|
|
662
|
+
const Authorized = Sync.forApplication(App)
|
|
663
|
+
.withPrincipal<{ readonly role: 'admin' | 'guest' }>()
|
|
664
|
+
|
|
219
665
|
const Board = Authorized.make({
|
|
220
666
|
documentId: DocumentId.make('board'),
|
|
221
667
|
...Authorized.compose(Todos, Members),
|
|
222
668
|
authorize: {
|
|
223
|
-
// `message` is exactly `RenamedTodo`; `shared` is the authoritative snapshot.
|
|
224
669
|
RenamedTodo: ({ principal, message, shared }) =>
|
|
225
|
-
principal.role === 'admin' &&
|
|
670
|
+
principal.role === 'admin' &&
|
|
671
|
+
shared.todos.some(todo => todo.id === message.id),
|
|
226
672
|
},
|
|
227
673
|
})
|
|
228
674
|
```
|
|
229
675
|
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
676
|
+
A durable variant without a rule is allowed. A key that is not one of the
|
|
677
|
+
contract's durable Message variants is a type error.
|
|
678
|
+
|
|
679
|
+
The client does not run authorization. It may optimistically show the edit;
|
|
680
|
+
when the server refuses it, reconciliation drops that pending operation and
|
|
681
|
+
rebases the remaining ones. This is why rejection is a normal exchange result,
|
|
682
|
+
not a transport failure.
|
|
236
683
|
|
|
237
|
-
|
|
684
|
+
## Persistence and recovery
|
|
238
685
|
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
686
|
+
The local outbox is part of the feature, not an incidental cache. Losing it can
|
|
687
|
+
lose edits that never reached the server.
|
|
688
|
+
|
|
689
|
+
The built-in browser storage adapter is IndexedDB:
|
|
242
690
|
|
|
243
691
|
```ts
|
|
244
|
-
|
|
245
|
-
|
|
692
|
+
const storage = yield* Sync.indexedDb('todos/tab-1')
|
|
693
|
+
```
|
|
246
694
|
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
695
|
+
Stored replica state includes protocol/schema versions, document/replica
|
|
696
|
+
identity, cursor, committed snapshot, local sequence, and pending operations.
|
|
697
|
+
Persisted and remote operations are decoded strictly against the application
|
|
698
|
+
Message Schema.
|
|
699
|
+
|
|
700
|
+
Important recovery cases:
|
|
701
|
+
|
|
702
|
+
- **Storage absent or evicted.** The replica starts at cursor 0 and can catch up
|
|
703
|
+
from the server, but edits that existed only in the lost outbox are gone.
|
|
704
|
+
- **Two writers use one replica storage.** Compare-and-swap detects the stale
|
|
705
|
+
writer. Give each tab/writer its own replica/storage identity.
|
|
706
|
+
- **Replay refuses a Message.** `submit` fails and writes nothing, so invalid
|
|
707
|
+
durable work never enters the outbox.
|
|
708
|
+
- **Malformed stored data.** Opening fails rather than silently replacing bytes;
|
|
709
|
+
the application can offer an explicit reset/recovery path.
|
|
710
|
+
- **Unsupported storage version.** `UnsupportedReplicaVersionError` names the
|
|
711
|
+
persisted and supported versions so migration/reset can be explicit.
|
|
712
|
+
- **Server compaction.** A checkpoint replaces history the replica can no longer
|
|
713
|
+
replay and pending local work is rebased on it.
|
|
714
|
+
|
|
715
|
+
Application Message/shared-state migrations remain application policy; Sync
|
|
716
|
+
versioning protects its own persisted envelope.
|
|
717
|
+
|
|
718
|
+
## Transport
|
|
719
|
+
|
|
720
|
+
`Transport` is an Effect service for the one exchange operation. Sync ships
|
|
721
|
+
helpers under `Sync.transport`:
|
|
722
|
+
|
|
723
|
+
```text
|
|
724
|
+
Sync.transport.socket(...)
|
|
725
|
+
Sync.transport.loopback(...)
|
|
726
|
+
Sync.transport.fromPromise(...)
|
|
727
|
+
Sync.transport.toPromise(...)
|
|
728
|
+
Sync.transport.serve(...)
|
|
729
|
+
Sync.transport.nativeSocket(...)
|
|
730
|
+
```
|
|
250
731
|
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
empty: { todos: [] },
|
|
256
|
-
durable: () => true,
|
|
257
|
-
replay: (shared, message) => ({ todos: [...shared.todos, message.title] }),
|
|
258
|
-
})
|
|
732
|
+
The reconnecting socket transport uses bounded retries/queueing, exponential
|
|
733
|
+
jittered backoff, and keeps request identities stable when resending queued or
|
|
734
|
+
in-flight exchanges. A server rejection is protocol data; only a wire failure is
|
|
735
|
+
a `TransportError`.
|
|
259
736
|
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
737
|
+
Transport is deliberately below reconciliation. A custom transport can carry
|
|
738
|
+
the same exchange without changing replica semantics.
|
|
739
|
+
|
|
740
|
+
## Presence: ephemeral collaboration state
|
|
741
|
+
|
|
742
|
+
Presence is intentionally **not** a durable Message log. “Alice is viewing this
|
|
743
|
+
page” or a cursor position should expire when the peer disappears, not replay
|
|
744
|
+
next week.
|
|
745
|
+
|
|
746
|
+
`Sync.presence.make` creates a TTL'd peer registry. Values are decoded before
|
|
747
|
+
they enter the registry; a peer that stops refreshing is removed. Changes are
|
|
748
|
+
available both as a callback subscription and as a Stream.
|
|
749
|
+
|
|
750
|
+
Presence can travel through:
|
|
751
|
+
|
|
752
|
+
```text
|
|
753
|
+
Sync.presence.socketChannel(...)
|
|
754
|
+
Sync.presence.loopbackChannel(...)
|
|
267
755
|
```
|
|
268
756
|
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
- The Foldkit-facing contract (`Sync.forApplication(App).make`, optionally with
|
|
272
|
-
a custom `replay`): derives the shared projection, the durable Message subset, the
|
|
273
|
-
initial snapshot, and the journal contract from the application, so none is
|
|
274
|
-
declared twice. `Projection.pick`/`Projection.compose` build the writable projection;
|
|
275
|
-
`TodoSync.journalContract()` builds the durable codecs and reducer.
|
|
276
|
-
- The operation envelope: `replicaId:localSequence` identity, `baseCursor`, and
|
|
277
|
-
protocol/schema versions. The contract's `codec` (`normalizeOperation`,
|
|
278
|
-
`operationFrom`, `committedFrom`, `decodeExchange`) is there for a custom
|
|
279
|
-
transport or server; a server-committed operation is a `CommittedOperation`.
|
|
280
|
-
- A persisted outbox and optimistic projection: a durable Message is visible
|
|
281
|
-
immediately and rebased onto the authoritative order.
|
|
282
|
-
- Reconciliation: committed operations are replayed in order, acknowledged or
|
|
283
|
-
refused pending entries are dropped, and a checkpoint replaces the compacted
|
|
284
|
-
prefix. `replica.start` is the exchange loop: fork it to exchange once and then
|
|
285
|
-
after every `submit`, with a transport failure recorded in `status.lastError`
|
|
286
|
-
and retried on the next wake, so the application does not hand-roll it.
|
|
287
|
-
- A redacted status (`replica.status`): the pending count, the cursor, the last
|
|
288
|
-
exchange failure, and the operations the server refused — enough for a UI to
|
|
289
|
-
explain and recover without exposing Messages or the Model. `statusChanges` is
|
|
290
|
-
the same status, emitted on subscribe and re-emitted after every submit and
|
|
291
|
-
exchange. `changes` carries the status and the optimistic shared value together
|
|
292
|
-
from one read, so a UI holds one subscription instead of two.
|
|
293
|
-
- Strict decoding: an operation is always validated with the application's
|
|
294
|
-
Message schema, and a Message the contract does not call durable is refused.
|
|
295
|
-
- Branded positions: `Sequence` (a committed document position) and
|
|
296
|
-
`LocalSequence` (a replica's own 1-based counter) cannot be confused, so a
|
|
297
|
-
client counter is never passed where a committed position is expected.
|
|
298
|
-
- Presence (`Sync.presence.make`): an ephemeral, TTL'd peer registry,
|
|
299
|
-
deliberately outside the durable log. A peer that stops refreshing is dropped, not
|
|
300
|
-
replayed. It is an Effect driven by the `Clock` (so a `TestClock` makes the
|
|
301
|
-
TTL deterministic), peers live in a `Ref`, and the channel is a `PubSub`.
|
|
302
|
-
Every value is decoded through the required `decodeValue` before it is stored,
|
|
303
|
-
so a hostile peer cannot inject a value your `Update` type does not describe.
|
|
304
|
-
Presence can travel over a socket — `Sync.presence.socketChannel` on the
|
|
305
|
-
client and `Sync.presence.serve` fanning through a `Sync.presence.hub` on the
|
|
306
|
-
server — or in-process via `Sync.presence.loopbackChannel`.
|
|
307
|
-
`presence.changes` is a `Stream` that emits the current peers and re-emits
|
|
308
|
-
them on every change, alongside the `subscribe` callback.
|
|
309
|
-
- The transport seam (`Transport`): an Effect service with a loopback layer, a
|
|
310
|
-
bridge to and from the promise client the replica speaks, and a WebSocket
|
|
311
|
-
client layer. The socket reconnects on an exponential, jittered backoff and
|
|
312
|
-
re-sends queued and in-flight frames with their original ids, so a lost reply
|
|
313
|
-
is answered rather than dropped; retries (`maxRetries`) and the queue
|
|
314
|
-
(`maxQueue`) are bounded, and `Sync.transport.serve` is the server side of a
|
|
315
|
-
connection. A refusal is an exchange result; only a wire failure is a
|
|
316
|
-
`TransportError`.
|
|
757
|
+
with server fan-out through:
|
|
317
758
|
|
|
318
|
-
|
|
759
|
+
```text
|
|
760
|
+
Sync.presence.hub(...)
|
|
761
|
+
Sync.presence.serve(...)
|
|
762
|
+
```
|
|
319
763
|
|
|
320
|
-
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
764
|
+
The registry uses Effect's `Clock`, so `TestClock` can make TTL behavior
|
|
765
|
+
deterministic in tests.
|
|
766
|
+
|
|
767
|
+
The conceptual rule is simple:
|
|
768
|
+
|
|
769
|
+
```text
|
|
770
|
+
shared application fact that must replay later -> durable Message / Sync
|
|
771
|
+
peer status that should disappear when stale -> Presence
|
|
772
|
+
```
|
|
327
773
|
|
|
328
774
|
## Last-writer-wins fields (experimental)
|
|
329
775
|
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
776
|
+
Server arrival order is the default conflict rule: whichever operation the
|
|
777
|
+
server commits first is replayed first. Sometimes one field needs a different
|
|
778
|
+
rule — for example, the newest logical title write should win even if an offline
|
|
779
|
+
device reconnects later.
|
|
780
|
+
|
|
781
|
+
`Sync.lww.register` supplies a schema and pure merge rule for that specialized
|
|
782
|
+
case. It is a field-level helper, **not a general CRDT mode for arbitrary
|
|
783
|
+
Messages**.
|
|
333
784
|
|
|
334
785
|
```ts
|
|
335
786
|
import { Schema } from 'effect'
|
|
@@ -337,16 +788,25 @@ import { ReplicaId, Sync } from 'foldkit-sync'
|
|
|
337
788
|
|
|
338
789
|
const Title = Sync.lww.register(Schema.String)
|
|
339
790
|
const Shared = Schema.Struct({ title: Title.schema })
|
|
340
|
-
const Renamed = Schema.Struct({
|
|
791
|
+
const Renamed = Schema.Struct({
|
|
792
|
+
_tag: Schema.Literal('Renamed'),
|
|
793
|
+
title: Title.schema,
|
|
794
|
+
})
|
|
341
795
|
|
|
342
|
-
const update = (
|
|
796
|
+
const update = (
|
|
797
|
+
model: typeof Shared.Type,
|
|
798
|
+
message: typeof Renamed.Type,
|
|
799
|
+
) => ({
|
|
343
800
|
...model,
|
|
344
801
|
title: Title.merge(model.title, message.title),
|
|
345
802
|
})
|
|
346
803
|
|
|
347
804
|
const message: typeof Renamed.Type = {
|
|
348
805
|
_tag: 'Renamed',
|
|
349
|
-
title: {
|
|
806
|
+
title: {
|
|
807
|
+
stamp: { counter: 1, replicaId: ReplicaId.make('tab-a') },
|
|
808
|
+
value: 'Milk',
|
|
809
|
+
},
|
|
350
810
|
}
|
|
351
811
|
```
|
|
352
812
|
|
|
@@ -355,9 +815,9 @@ order, independent of locale. This is logical ordering, not wall-clock time.
|
|
|
355
815
|
The rule follows the total-order register model described in
|
|
356
816
|
[Replicated Data Types: Specification, Verification, Optimality](https://www.microsoft.com/en-us/research/publication/replicated-data-types-specification-verification-optimality/).
|
|
357
817
|
|
|
358
|
-
Allocate stamps **before dispatch**, never
|
|
359
|
-
`Sync.lww.openClock` persists
|
|
360
|
-
rejected or
|
|
818
|
+
Allocate stamps **before dispatch**, never inside `update` or replay.
|
|
819
|
+
`Sync.lww.openClock` persists the high-water mark independently of the outbox,
|
|
820
|
+
so rejected or never-submitted writes cannot cause timestamp reuse after reload:
|
|
361
821
|
|
|
362
822
|
```ts
|
|
363
823
|
import { Effect } from 'effect'
|
|
@@ -368,87 +828,186 @@ type Renamed = typeof Renamed.Type
|
|
|
368
828
|
|
|
369
829
|
const rename = (replica: Replica<Renamed, Shared>, title: string) =>
|
|
370
830
|
Effect.gen(function* () {
|
|
371
|
-
// A separate database from the replica's outbox; one clock per document/writer.
|
|
372
831
|
const storage = yield* Sync.indexedDb('todos-tab-a-clock')
|
|
373
832
|
const clock = yield* Sync.lww.openClock({
|
|
374
833
|
documentId: DocumentId.make('todos'),
|
|
375
834
|
replicaId: ReplicaId.make('tab-a'),
|
|
376
835
|
storage,
|
|
377
836
|
})
|
|
837
|
+
|
|
378
838
|
const shared = yield* replica.shared
|
|
379
839
|
const stamp = yield* clock.next(shared.title.stamp.counter)
|
|
380
|
-
|
|
840
|
+
|
|
841
|
+
yield* replica.submit({
|
|
842
|
+
_tag: 'Renamed',
|
|
843
|
+
title: { stamp, value: title },
|
|
844
|
+
})
|
|
845
|
+
|
|
381
846
|
yield* clock.close
|
|
382
847
|
}).pipe(Effect.scoped)
|
|
383
848
|
```
|
|
384
849
|
|
|
385
|
-
`next(observedCounter)`
|
|
386
|
-
|
|
387
|
-
counter for the field being edited;
|
|
388
|
-
their maximum. Calling `next()` without an observation advances
|
|
389
|
-
|
|
850
|
+
`next(observedCounter)` persists a counter greater than both its previous saved
|
|
851
|
+
value and the supplied observation before returning the stamp. Pass the latest
|
|
852
|
+
observed counter for the field being edited; if causality spans several
|
|
853
|
+
registers, pass their maximum. Calling `next()` without an observation advances
|
|
854
|
+
only the saved local counter.
|
|
390
855
|
|
|
391
856
|
Allocations on one handle are serialized. Separate handles use storage's
|
|
392
|
-
compare-and-swap check; a stale handle fails and
|
|
393
|
-
Storage failures
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
397
|
-
|
|
398
|
-
|
|
399
|
-
Keep the clock database across reloads and outbox resets. If it is lost or
|
|
400
|
-
deleted, use a fresh replica id instead of resetting the same writer's counter.
|
|
401
|
-
For applications that allocate stamps themselves, preserve this same durable
|
|
402
|
-
high-water mark, including refused writes.
|
|
857
|
+
compare-and-swap check; a stale handle fails and should be closed/reopened.
|
|
858
|
+
Storage failures do not return a stamp. Counter gaps after a crash or rejected
|
|
859
|
+
submission are harmless.
|
|
860
|
+
|
|
861
|
+
Keep the clock storage across reloads and outbox resets. If it is lost, use a
|
|
862
|
+
fresh `ReplicaId` rather than reusing the same writer identity from counter zero.
|
|
403
863
|
|
|
404
864
|
A stamp identifies one immutable write within a register. Repeated delivery of
|
|
405
|
-
the same value is harmless; different
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
|
|
419
|
-
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
|
|
423
|
-
|
|
865
|
+
the same value is harmless; a different value with the same stamp throws. Keep
|
|
866
|
+
the **whole register, including its winning stamp**, in shared snapshots. For
|
|
867
|
+
deletion, a nullable register can retain a stamped `null` tombstone so an old
|
|
868
|
+
offline write cannot resurrect deleted state.
|
|
869
|
+
|
|
870
|
+
The server still orders and authorizes every operation, including writes that
|
|
871
|
+
lose the LWW merge. Replica ids and counters carried in Messages are
|
|
872
|
+
client-authored data, not authenticated identity; enforce writer/clock policy at
|
|
873
|
+
admission when needed.
|
|
874
|
+
|
|
875
|
+
Specialized sets, counters, collaborative text, and arbitrary commutative
|
|
876
|
+
Messages remain outside this helper.
|
|
877
|
+
|
|
878
|
+
## Lower level: `Sync.define`
|
|
879
|
+
|
|
880
|
+
`Sync.forApplication(App).make(...)` is the Foldkit-facing compiler. It builds
|
|
881
|
+
on `Sync.define`, the lower-level protocol primitive.
|
|
882
|
+
|
|
883
|
+
Use `Sync.define` directly when there is no Foldkit application to derive the
|
|
884
|
+
contract from — for example, a non-Foldkit client or a hand-written shared
|
|
885
|
+
projection:
|
|
886
|
+
|
|
887
|
+
```ts
|
|
888
|
+
import { Effect, Schema } from 'effect'
|
|
889
|
+
import {
|
|
890
|
+
DocumentId,
|
|
891
|
+
ReplicaId,
|
|
892
|
+
Sync,
|
|
893
|
+
type TransportClient,
|
|
894
|
+
} from 'foldkit-sync'
|
|
895
|
+
|
|
896
|
+
const Shared = Schema.Struct({ todos: Schema.Array(Schema.String) })
|
|
897
|
+
const Renamed = Schema.Struct({
|
|
898
|
+
_tag: Schema.Literal('Renamed'),
|
|
899
|
+
title: Schema.String,
|
|
900
|
+
})
|
|
901
|
+
|
|
902
|
+
declare const transport: TransportClient
|
|
903
|
+
|
|
904
|
+
const Protocol = Sync.define({
|
|
905
|
+
documentId: DocumentId.make('todos'),
|
|
906
|
+
message: Renamed,
|
|
907
|
+
shared: Shared,
|
|
908
|
+
empty: { todos: [] },
|
|
909
|
+
durable: () => true,
|
|
910
|
+
replay: (shared, message) => ({
|
|
911
|
+
todos: [...shared.todos, message.title],
|
|
912
|
+
}),
|
|
913
|
+
})
|
|
914
|
+
|
|
915
|
+
const program = Effect.gen(function* () {
|
|
916
|
+
const storage = yield* Sync.indexedDb('todos-tab-1')
|
|
917
|
+
const replica = yield* Protocol.openReplica(
|
|
918
|
+
ReplicaId.make('tab-1'),
|
|
919
|
+
storage,
|
|
920
|
+
)
|
|
921
|
+
|
|
922
|
+
yield* replica.submit({ _tag: 'Renamed', title: 'Milk' })
|
|
923
|
+
|
|
924
|
+
yield* Effect.provide(
|
|
925
|
+
replica.synchronize,
|
|
926
|
+
Sync.transport.fromPromise(transport),
|
|
927
|
+
)
|
|
928
|
+
|
|
929
|
+
return yield* replica.shared
|
|
930
|
+
}).pipe(Effect.scoped)
|
|
931
|
+
```
|
|
932
|
+
|
|
933
|
+
Most Foldkit applications should prefer `Sync.forApplication`: it keeps the
|
|
934
|
+
shared schema and replay semantics derived from the application rather than
|
|
935
|
+
re-declaring them.
|
|
936
|
+
|
|
937
|
+
## What Sync owns
|
|
938
|
+
|
|
939
|
+
- **The replication contract.** Shared projection, durable Message subset,
|
|
940
|
+
initial shared value, replay, operation codecs, and `journalContract()`.
|
|
941
|
+
- **The replica.** Persisted outbox, committed base, cursor, optimistic shared
|
|
942
|
+
projection, local operation sequence, synchronization, status, and recovery
|
|
943
|
+
validation.
|
|
944
|
+
- **Reconciliation.** Adopt committed operations/checkpoints, drop
|
|
945
|
+
acknowledged/rejected pending operations, and replay what remains.
|
|
946
|
+
- **The transport seam.** Effect `Transport` plus loopback/promise/WebSocket
|
|
947
|
+
adapters; transport carries exchanges but does not own conflict semantics.
|
|
948
|
+
- **Optional collaboration primitives.** Ephemeral Presence and the specialized
|
|
949
|
+
LWW register/clock helpers.
|
|
950
|
+
|
|
951
|
+
Sync does **not** own:
|
|
952
|
+
|
|
953
|
+
- unrelated local Model fields;
|
|
954
|
+
- server-derived disposable caches (`foldkit-remote`);
|
|
955
|
+
- authoritative server storage/order (`foldkit-durable` or your own server);
|
|
956
|
+
- peer-to-peer CRDT convergence;
|
|
957
|
+
- application Message/shared-state migrations.
|
|
958
|
+
|
|
959
|
+
## How state changes here
|
|
960
|
+
|
|
961
|
+
Durable Messages replay the normal application `update` — which is where
|
|
962
|
+
`evo` lives — so deterministic replay and optimistic application are the
|
|
963
|
+
same code path, not two reducers. Checkpoints and adopted server state
|
|
964
|
+
arrive the other way: structurally installed through the declared writable
|
|
965
|
+
projection. Sync already verifies that durable Messages touch only the
|
|
966
|
+
declared projection; treat any other write as a bug.
|
|
967
|
+
|
|
968
|
+
## Limits
|
|
969
|
+
|
|
970
|
+
- IndexedDB is the built-in storage adapter today.
|
|
971
|
+
- Sync assumes an authoritative server order for each document; it is not a
|
|
972
|
+
peer-to-peer replication protocol.
|
|
973
|
+
- Ordinary concurrent writes resolve by that server order. `Sync.lww.register`
|
|
974
|
+
changes the merge rule for individual fields only; it does not make arbitrary
|
|
975
|
+
Messages commutative.
|
|
976
|
+
- There is no operational transform / collaborative text algorithm.
|
|
977
|
+
- Binding the socket transport and presence server to a platform WebSocket
|
|
978
|
+
server remains application/platform glue. The sync example demonstrates a
|
|
979
|
+
`ws` transport binding; presence over that server is not wired there today.
|
|
980
|
+
- Application schema migration remains application policy. Sync versions and
|
|
981
|
+
validates its own persisted/wire envelope.
|
|
424
982
|
|
|
425
983
|
## Older spellings
|
|
426
984
|
|
|
427
|
-
`Sync` groups
|
|
428
|
-
|
|
429
|
-
spellings keeps working:
|
|
985
|
+
`Sync` groups the package exports; it does not wrap them. Existing top-level
|
|
986
|
+
spellings remain available with the same signatures:
|
|
430
987
|
|
|
431
|
-
- `forApplication`, `mount`, `indexedDb` — same names on the namespace
|
|
432
|
-
- `defineSync`
|
|
433
|
-
- `layerSocket`, `layerLoopback`, `layerFromPromise`, `toPromise`, `serveSocket
|
|
434
|
-
|
|
435
|
-
`.toPromise`, `.serve
|
|
988
|
+
- `forApplication`, `mount`, `indexedDb` — same names on the namespace;
|
|
989
|
+
- `defineSync` -> `Sync.define`; `syncMetrics` -> `Sync.metrics`;
|
|
990
|
+
- `layerSocket`, `layerLoopback`, `layerFromPromise`, `toPromise`, `serveSocket`,
|
|
991
|
+
`nativeSocket` -> `Sync.transport.socket`, `.loopback`, `.fromPromise`,
|
|
992
|
+
`.toPromise`, `.serve`, `.nativeSocket`;
|
|
436
993
|
- `createPresence`, `createPresenceHub`, `servePresence`,
|
|
437
|
-
`socketPresenceChannel
|
|
438
|
-
|
|
439
|
-
|
|
440
|
-
- `
|
|
441
|
-
`
|
|
442
|
-
|
|
443
|
-
and likewise for `replicaId`, `opId`, `sequence` and `localSequence`.
|
|
994
|
+
`socketPresenceChannel`, `loopbackPresenceChannel` -> `Sync.presence.make`,
|
|
995
|
+
`.hub`, `.serve`, `.socketChannel`, `.loopbackChannel`;
|
|
996
|
+
- `lwwRegister`, `openLwwClock` -> `Sync.lww.register`, `Sync.lww.openClock`;
|
|
997
|
+
- `documentId('todos')` and `DocumentId.make('todos')` produce the same branded
|
|
998
|
+
value, and likewise for `replicaId`, `opId`, `sequence`, and
|
|
999
|
+
`localSequence`.
|
|
444
1000
|
|
|
445
1001
|
## See also
|
|
446
1002
|
|
|
447
|
-
- [Replicated state](
|
|
448
|
-
|
|
449
|
-
- [Runtime binding](
|
|
450
|
-
|
|
451
|
-
- [`foldkit-durable`](
|
|
452
|
-
|
|
453
|
-
- [`
|
|
454
|
-
|
|
1003
|
+
- [Replicated state](../../docs/replication.md) — Sync and Durable together,
|
|
1004
|
+
including the server exchange and recovery model.
|
|
1005
|
+
- [Runtime binding](../../docs/sync-runtime-binding.md) — what `Sync.mount`
|
|
1006
|
+
guarantees and why no Foldkit core change is required.
|
|
1007
|
+
- [`foldkit-durable`](../durable) — the authoritative server journal commonly
|
|
1008
|
+
paired with Sync.
|
|
1009
|
+
- [`foldkit-remote`](../remote) — server-owned disposable state rather than
|
|
1010
|
+
client-authored durable state.
|
|
1011
|
+
- [`examples/todo-app`](../../examples/todo-app) — browser mount, fragments,
|
|
1012
|
+
authorization, Mirror, and agent integration.
|
|
1013
|
+
- [`examples/sync`](../../examples/sync) — focused replica/server/transport trace.
|