foldkit-sync 0.3.1 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +823 -266
- package/dist/index.d.mts +348 -184
- package/dist/index.d.mts.map +1 -1
- package/dist/index.mjs +369 -253
- package/dist/index.mjs.map +1 -1
- package/package.json +4 -3
package/README.md
CHANGED
|
@@ -1,29 +1,123 @@
|
|
|
1
1
|
# `foldkit-sync`
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
3
|
+
Local-first replication for part of a Foldkit application.
|
|
4
|
+
|
|
5
|
+
A durable edit applies locally immediately, is persisted to an outbox, and is
|
|
6
|
+
sent when the network allows. The server decides the authoritative order. The
|
|
7
|
+
replica then replays that committed order and rebases any still-pending local
|
|
8
|
+
edits on top.
|
|
9
|
+
|
|
10
|
+
The important part is what **does not** change: your application's `Model`,
|
|
11
|
+
`Message`, and `update` remain the state machine. Sync does not introduce a
|
|
12
|
+
second reducer. You declare which Model slice is shared and which existing
|
|
13
|
+
Messages are durable; Sync derives replay, persistence, protocol codecs, and the
|
|
14
|
+
server journal contract from that declaration.
|
|
15
|
+
|
|
16
|
+
> **Sync turns a subset of your existing Foldkit Messages into durable
|
|
17
|
+
> operations. They still run through your normal `update`; Sync adds persistence,
|
|
18
|
+
> optimistic replay, server ordering, and reconciliation around them.**
|
|
19
|
+
|
|
20
|
+
**Use it when** a user should keep editing offline, the edit must survive a
|
|
21
|
+
reload or network failure, and several replicas must eventually converge under
|
|
22
|
+
one authoritative server order. **Do not use it for** server-owned disposable
|
|
23
|
+
facts — use [`foldkit-remote`](../remote) for those — or for peer-to-peer / CRDT
|
|
24
|
+
replication where every peer independently accepts writes.
|
|
25
|
+
|
|
26
|
+
It is the client half of [replicated Foldkit state](../../docs/replication.md).
|
|
27
|
+
[`foldkit-durable`](../durable) is an optional server-side ordered journal.
|
|
28
|
+
|
|
29
|
+
## Which state belongs in Sync?
|
|
30
|
+
|
|
31
|
+
A useful ownership rule is:
|
|
32
|
+
|
|
33
|
+
| State | Owner |
|
|
34
|
+
| --- | --- |
|
|
35
|
+
| Route, selected tab, transient errors | ordinary local `Model` / `update` |
|
|
36
|
+
| Server-owned data the client may refetch | `foldkit-remote` |
|
|
37
|
+
| Client-authored state that must survive offline and converge | `foldkit-sync` |
|
|
38
|
+
| Authoritative server ordering, idempotent append, snapshots | `foldkit-durable` |
|
|
39
|
+
|
|
40
|
+
The difference between Remote and Sync is especially important:
|
|
41
|
+
|
|
42
|
+
```text
|
|
43
|
+
Remote
|
|
44
|
+
server owns the fact
|
|
45
|
+
client caches it
|
|
46
|
+
refetch is recovery
|
|
47
|
+
|
|
48
|
+
Sync
|
|
49
|
+
client creates the durable operation
|
|
50
|
+
operation must survive offline
|
|
51
|
+
replay + reconciliation is recovery
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
A Surface may read local state, Remote state, and Sync-owned state at the same
|
|
55
|
+
time. Those observation boundaries do not change who owns each datum.
|
|
56
|
+
|
|
57
|
+
## The replica mental model
|
|
58
|
+
|
|
59
|
+
A replica has two pieces of state:
|
|
60
|
+
|
|
61
|
+
```text
|
|
62
|
+
authoritative committed snapshot
|
|
63
|
+
+
|
|
64
|
+
pending local operations
|
|
65
|
+
=
|
|
66
|
+
optimistic shared state the UI sees
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
If the server has committed `A B C` and this device has pending `D E`:
|
|
70
|
+
|
|
71
|
+
```text
|
|
72
|
+
server order: A B C
|
|
73
|
+
local pending: D E
|
|
74
|
+
|
|
75
|
+
visible state = replay(A, B, C, D, E)
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
If another device gets operation `X` committed before this replica's pending
|
|
79
|
+
work, the next exchange may produce:
|
|
80
|
+
|
|
81
|
+
```text
|
|
82
|
+
new server order: A B C X
|
|
83
|
+
local pending: D E
|
|
84
|
+
|
|
85
|
+
new visible state = replay(A, B, C, X, D, E)
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
The local edit never needed to wait for that exchange to appear. Reconciliation
|
|
89
|
+
changes the base underneath it and replays the still-pending operations on top.
|
|
90
|
+
|
|
91
|
+
The full lifecycle of one durable Message is:
|
|
92
|
+
|
|
93
|
+
```text
|
|
94
|
+
user dispatches durable Message
|
|
95
|
+
|
|
|
96
|
+
v
|
|
97
|
+
replay through application update immediately
|
|
98
|
+
|
|
|
99
|
+
v
|
|
100
|
+
persist operation in local outbox
|
|
101
|
+
|
|
|
102
|
+
v
|
|
103
|
+
optimistic state is visible now
|
|
104
|
+
|
|
|
105
|
+
network later
|
|
106
|
+
v
|
|
107
|
+
server assigns authoritative order
|
|
108
|
+
|
|
|
109
|
+
v
|
|
110
|
+
replica adopts committed operations / checkpoint
|
|
111
|
+
|
|
|
112
|
+
v
|
|
113
|
+
drop acknowledged or rejected pending operations
|
|
114
|
+
|
|
|
115
|
+
v
|
|
116
|
+
replay remaining pending operations on the new base
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
That is the core of `foldkit-sync`. Transport, browser mounting, presence, and
|
|
120
|
+
LWW fields are layers around this mechanism.
|
|
27
121
|
|
|
28
122
|
## Install
|
|
29
123
|
|
|
@@ -31,25 +125,24 @@ covers the mental model and when not to use it.
|
|
|
31
125
|
pnpm add foldkit-sync
|
|
32
126
|
```
|
|
33
127
|
|
|
34
|
-
`foldkit` and `effect` are peer dependencies; `foldkit-surface` comes with
|
|
35
|
-
|
|
128
|
+
`foldkit` and `effect` are peer dependencies; `foldkit-surface` comes with the
|
|
129
|
+
package.
|
|
36
130
|
|
|
37
|
-
##
|
|
131
|
+
## Sixty seconds: choose what is shared and what is durable
|
|
38
132
|
|
|
39
|
-
|
|
40
|
-
and a writable projection it derives the shared schema, the durable Message
|
|
41
|
-
subset, the initial snapshot, and replay, and exposes a read-only Surface over
|
|
42
|
-
the same projection.
|
|
133
|
+
Start with an ordinary Foldkit application:
|
|
43
134
|
|
|
44
135
|
```ts
|
|
45
136
|
import { Schema } from 'effect'
|
|
46
137
|
import { defineMessageUnion } from 'foldkit/message'
|
|
47
138
|
import type * as Update from 'foldkit/update'
|
|
48
139
|
import { MessageSet, Projection, Surface } from 'foldkit-surface'
|
|
49
|
-
import {
|
|
140
|
+
import { DocumentId, Sync } from 'foldkit-sync'
|
|
50
141
|
|
|
51
142
|
const Model = Schema.Struct({
|
|
52
|
-
todos: Schema.Array(
|
|
143
|
+
todos: Schema.Array(
|
|
144
|
+
Schema.Struct({ id: Schema.String, title: Schema.String }),
|
|
145
|
+
),
|
|
53
146
|
selectedTodoId: Schema.NullOr(Schema.String),
|
|
54
147
|
lastError: Schema.NullOr(Schema.String),
|
|
55
148
|
})
|
|
@@ -62,10 +155,15 @@ const Message = defineMessageUnion({
|
|
|
62
155
|
})
|
|
63
156
|
type Message = typeof Message.Type
|
|
64
157
|
|
|
65
|
-
const initial: Model = {
|
|
158
|
+
const initial: Model = {
|
|
159
|
+
todos: [],
|
|
160
|
+
selectedTodoId: null,
|
|
161
|
+
lastError: null,
|
|
162
|
+
}
|
|
66
163
|
|
|
67
164
|
type Return = Update.Return<Model, Message>
|
|
68
|
-
|
|
165
|
+
|
|
166
|
+
// This is still the application's one transition function.
|
|
69
167
|
const update = (model: Model, message: Message): Return =>
|
|
70
168
|
Message.match<Return>(message, {
|
|
71
169
|
CreatedTodo: ({ id, title }) => ({
|
|
@@ -74,74 +172,375 @@ const update = (model: Model, message: Message): Return =>
|
|
|
74
172
|
RenamedTodo: ({ id, title }) => ({
|
|
75
173
|
model: {
|
|
76
174
|
...model,
|
|
77
|
-
todos: model.todos.map(todo =>
|
|
175
|
+
todos: model.todos.map(todo =>
|
|
176
|
+
todo.id === id ? { ...todo, title } : todo,
|
|
177
|
+
),
|
|
78
178
|
},
|
|
79
179
|
}),
|
|
80
|
-
SelectedTodo: ({ id }) => ({
|
|
180
|
+
SelectedTodo: ({ id }) => ({
|
|
181
|
+
model: { ...model, selectedTodoId: id },
|
|
182
|
+
}),
|
|
81
183
|
})
|
|
82
184
|
|
|
83
185
|
const App = Surface.application({ Model, Message, initial, update })
|
|
84
186
|
|
|
85
|
-
const TodoSync = forApplication(App).make({
|
|
86
|
-
documentId:
|
|
87
|
-
|
|
88
|
-
|
|
187
|
+
const TodoSync = Sync.forApplication(App).make({
|
|
188
|
+
documentId: DocumentId.make('todos'),
|
|
189
|
+
|
|
190
|
+
// This slice of Model is replicated.
|
|
191
|
+
// The writable Projection lets Sync install checkpoints/reconciled state back
|
|
192
|
+
// into the application Model without owning a second copy of the schema.
|
|
193
|
+
shared: Projection.pick(App.fields.todos),
|
|
194
|
+
|
|
195
|
+
// Only these existing Messages become durable operations.
|
|
196
|
+
durable: MessageSet.make(App, [
|
|
197
|
+
Message.CreatedTodo,
|
|
198
|
+
Message.RenamedTodo,
|
|
199
|
+
]),
|
|
89
200
|
})
|
|
90
201
|
```
|
|
91
202
|
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
203
|
+
Read that declaration as:
|
|
204
|
+
|
|
205
|
+
```text
|
|
206
|
+
Model
|
|
207
|
+
todos -> shared / replicated
|
|
208
|
+
selectedTodoId -> local only
|
|
209
|
+
lastError -> local only
|
|
210
|
+
|
|
211
|
+
Messages
|
|
212
|
+
CreatedTodo -> durable
|
|
213
|
+
RenamedTodo -> durable
|
|
214
|
+
SelectedTodo -> local only
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
`Sync.forApplication(App)` derives the shared schema, initial shared snapshot,
|
|
218
|
+
replay function, operation codecs, and server journal contract from the same
|
|
219
|
+
application declaration. There is no parallel sync-specific version of
|
|
220
|
+
`update` to keep aligned.
|
|
221
|
+
|
|
222
|
+
## What makes a Message durable?
|
|
223
|
+
|
|
224
|
+
A durable Message must be something another machine can replay later and obtain
|
|
225
|
+
the same shared state.
|
|
226
|
+
|
|
227
|
+
In practice that means it must be a deterministic, state-only transition of the
|
|
228
|
+
shared projection.
|
|
229
|
+
|
|
230
|
+
Derived replay protects that boundary. It refuses a durable Message when the
|
|
231
|
+
application's `update`:
|
|
232
|
+
|
|
233
|
+
- returns a Command — a live external effect cannot be replayed later; or
|
|
234
|
+
- changes a field outside the shared projection — that change would disappear
|
|
235
|
+
when only the shared slice is persisted/replayed.
|
|
95
236
|
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
237
|
+
For example:
|
|
238
|
+
|
|
239
|
+
```text
|
|
240
|
+
RequestedChargeCard
|
|
241
|
+
-> runs a Command / talks to a provider
|
|
242
|
+
-> local intent, NOT durable
|
|
243
|
+
|
|
244
|
+
CardCharged
|
|
245
|
+
-> pure fact about state
|
|
246
|
+
-> durable and replayable
|
|
247
|
+
```
|
|
248
|
+
|
|
249
|
+
A common pattern is therefore:
|
|
250
|
+
|
|
251
|
+
```text
|
|
252
|
+
local intent Message
|
|
253
|
+
|
|
|
254
|
+
v
|
|
255
|
+
Command / nondeterministic work
|
|
256
|
+
|
|
|
257
|
+
v
|
|
258
|
+
durable fact Message
|
|
259
|
+
```
|
|
104
260
|
|
|
105
|
-
|
|
106
|
-
|
|
261
|
+
Replay runs the durable fact through the same application `update` on every
|
|
262
|
+
replica. If an application genuinely needs different replay semantics, `make`
|
|
263
|
+
accepts a custom `replay`, but that custom replay does not get the derived
|
|
264
|
+
guardrails.
|
|
265
|
+
|
|
266
|
+
## Running a replica
|
|
267
|
+
|
|
268
|
+
The low-level runtime is a `Replica`. It owns the local outbox, committed shared
|
|
269
|
+
snapshot, cursor, optimistic projection, and synchronization state.
|
|
107
270
|
|
|
108
271
|
```ts
|
|
109
272
|
import { Effect } from 'effect'
|
|
110
|
-
import {
|
|
273
|
+
import { ReplicaId, Sync } from 'foldkit-sync'
|
|
274
|
+
|
|
275
|
+
const program = Effect.gen(function* () {
|
|
276
|
+
// One durable storage identity per document + replica/writer.
|
|
277
|
+
const storage = yield* Sync.indexedDb('todos/tab-1')
|
|
278
|
+
|
|
279
|
+
const replica = yield* TodoSync.openReplica(
|
|
280
|
+
ReplicaId.make('tab-1'),
|
|
281
|
+
storage,
|
|
282
|
+
)
|
|
283
|
+
|
|
284
|
+
// submit validates/replays first, then persists the operation in the outbox.
|
|
285
|
+
yield* replica.submit(
|
|
286
|
+
Message.CreatedTodo({ id: 't1', title: 'Milk' }),
|
|
287
|
+
)
|
|
288
|
+
|
|
289
|
+
// This already includes the pending operation. No server round trip required.
|
|
290
|
+
const optimistic = yield* replica.shared
|
|
291
|
+
|
|
292
|
+
return optimistic
|
|
293
|
+
}).pipe(Effect.scoped)
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
`replica.submit(message)` does not mean “send this now.” It means:
|
|
297
|
+
|
|
298
|
+
1. verify the Message is valid and durable;
|
|
299
|
+
2. replay it against the current optimistic shared state;
|
|
300
|
+
3. assign a stable local operation identity;
|
|
301
|
+
4. persist it to the local outbox;
|
|
302
|
+
5. make the new optimistic projection observable; and
|
|
303
|
+
6. wake the exchange loop if one is running.
|
|
304
|
+
|
|
305
|
+
A Message replay rejects never enters the outbox.
|
|
306
|
+
|
|
307
|
+
## Optimistic state and rebasing
|
|
308
|
+
|
|
309
|
+
Internally the replica keeps:
|
|
310
|
+
|
|
311
|
+
```text
|
|
312
|
+
committed
|
|
313
|
+
the authoritative shared snapshot up to cursor N
|
|
314
|
+
|
|
315
|
+
pending
|
|
316
|
+
locally-authored operations not yet resolved by the server
|
|
317
|
+
```
|
|
318
|
+
|
|
319
|
+
The value returned by `replica.shared` is:
|
|
320
|
+
|
|
321
|
+
```text
|
|
322
|
+
pending.reduce(replay, committed)
|
|
323
|
+
```
|
|
324
|
+
|
|
325
|
+
That simple rule gives optimistic UI and reconciliation the same semantics.
|
|
326
|
+
There is no special “optimistic reducer.”
|
|
327
|
+
|
|
328
|
+
When synchronization advances the committed base, Sync removes pending
|
|
329
|
+
operations that became committed, were explicitly acknowledged, or were
|
|
330
|
+
rejected, then the remaining outbox is replayed over the new committed state.
|
|
331
|
+
|
|
332
|
+
This also explains why durable Messages have to be replay-safe: a pending
|
|
333
|
+
operation may run many times as the authoritative base changes beneath it.
|
|
334
|
+
|
|
335
|
+
## Synchronization
|
|
336
|
+
|
|
337
|
+
A replica exchanges two things with the server:
|
|
338
|
+
|
|
339
|
+
```text
|
|
340
|
+
request
|
|
341
|
+
current cursor
|
|
342
|
+
pending operations
|
|
343
|
+
|
|
344
|
+
response
|
|
345
|
+
newly committed operations
|
|
346
|
+
acknowledgements
|
|
347
|
+
rejections
|
|
348
|
+
optional checkpoint
|
|
349
|
+
```
|
|
350
|
+
|
|
351
|
+
`replica.synchronize` performs one exchange through the `Transport` Effect
|
|
352
|
+
service:
|
|
353
|
+
|
|
354
|
+
```ts
|
|
355
|
+
const once = replica.synchronize.pipe(
|
|
356
|
+
Effect.provide(Sync.transport.socket({ url: 'wss://example.com/sync' })),
|
|
357
|
+
)
|
|
358
|
+
```
|
|
359
|
+
|
|
360
|
+
`replica.start` is the long-running convenience loop. It exchanges once, then
|
|
361
|
+
wakes after every submit. Transport failures are recorded in
|
|
362
|
+
`status.lastError`; the loop survives and retries on the next wake.
|
|
363
|
+
|
|
364
|
+
```ts
|
|
365
|
+
Effect.runFork(
|
|
366
|
+
Effect.provide(
|
|
367
|
+
replica.start,
|
|
368
|
+
Sync.transport.socket({ url: 'wss://example.com/sync' }),
|
|
369
|
+
),
|
|
370
|
+
)
|
|
371
|
+
```
|
|
372
|
+
|
|
373
|
+
The transport does not decide reconciliation. It only carries the exchange;
|
|
374
|
+
the replica validates the response and applies the protocol rules.
|
|
375
|
+
|
|
376
|
+
### Acknowledgements
|
|
377
|
+
|
|
378
|
+
An acknowledgement says an operation the replica sent is durably accepted, so
|
|
379
|
+
it may leave the outbox even if its committed payload is not repeated in this
|
|
380
|
+
response.
|
|
381
|
+
|
|
382
|
+
### Rejections
|
|
383
|
+
|
|
384
|
+
A rejection says the optimistic operation did not commit. Sync removes it from
|
|
385
|
+
the pending set and recomputes the optimistic state from the authoritative base
|
|
386
|
+
plus whatever pending operations remain.
|
|
387
|
+
|
|
388
|
+
```text
|
|
389
|
+
before response
|
|
390
|
+
committed: A B C
|
|
391
|
+
pending: D E
|
|
392
|
+
visible: A B C D E
|
|
393
|
+
|
|
394
|
+
server rejects D
|
|
395
|
+
|
|
396
|
+
committed: A B C
|
|
397
|
+
pending: E
|
|
398
|
+
visible: A B C E
|
|
399
|
+
```
|
|
400
|
+
|
|
401
|
+
`replica.status.rejected` keeps recent rejected operation ids so the UI can
|
|
402
|
+
explain that an optimistic edit was refused.
|
|
403
|
+
|
|
404
|
+
### Checkpoints
|
|
405
|
+
|
|
406
|
+
A server may compact old history. If the replica is too far behind to replay the
|
|
407
|
+
missing prefix, the response can include a checkpoint:
|
|
408
|
+
|
|
409
|
+
```text
|
|
410
|
+
checkpoint = authoritative shared snapshot + cursor
|
|
411
|
+
```
|
|
412
|
+
|
|
413
|
+
The replica adopts the checkpoint as its committed base, then rebases its local
|
|
414
|
+
pending operations on top. A checkpoint behind the replica's current cursor is
|
|
415
|
+
rejected with `CheckpointRegressionError` rather than silently moving history
|
|
416
|
+
backwards.
|
|
417
|
+
|
|
418
|
+
## Replica status and UI state
|
|
419
|
+
|
|
420
|
+
The public replica surface deliberately exposes enough state for UI and recovery
|
|
421
|
+
without exposing the application Messages themselves:
|
|
422
|
+
|
|
423
|
+
```ts
|
|
424
|
+
const status = yield* replica.status
|
|
425
|
+
// {
|
|
426
|
+
// pending: number,
|
|
427
|
+
// cursor: Sequence,
|
|
428
|
+
// lastError: string | undefined,
|
|
429
|
+
// rejected: ReadonlyArray<OpId>
|
|
430
|
+
// }
|
|
431
|
+
```
|
|
432
|
+
|
|
433
|
+
`replica.statusChanges` emits that status initially and after every submit and
|
|
434
|
+
exchange. `replica.changes` emits the status and optimistic `shared` value
|
|
435
|
+
together from one replica snapshot, so a UI can subscribe once without racing
|
|
436
|
+
two separate reads. `replica.snapshot` is the same read, once: the status,
|
|
437
|
+
`shared`, `committed`, and `nextLocalSequence`, the local sequence the next
|
|
438
|
+
submitted operation takes.
|
|
439
|
+
|
|
440
|
+
## Sync + Durable
|
|
441
|
+
|
|
442
|
+
Sync owns **client replication semantics**. Durable can own **server ordering and
|
|
443
|
+
persistence**.
|
|
444
|
+
|
|
445
|
+
```text
|
|
446
|
+
client
|
|
447
|
+
-----------------------------------------------
|
|
448
|
+
Foldkit update
|
|
449
|
+
|
|
|
450
|
+
| durable Message
|
|
451
|
+
v
|
|
452
|
+
foldkit-sync Replica
|
|
453
|
+
optimistic shared state
|
|
454
|
+
persistent outbox
|
|
455
|
+
|
|
|
456
|
+
| exchange
|
|
457
|
+
v
|
|
458
|
+
-----------------------------------------------
|
|
459
|
+
server
|
|
460
|
+
foldkit-durable Journal
|
|
461
|
+
authoritative order
|
|
462
|
+
snapshot + cursor
|
|
463
|
+
|
|
|
464
|
+
v
|
|
465
|
+
committed operations / checkpoint / rejection
|
|
466
|
+
```
|
|
467
|
+
|
|
468
|
+
The same Sync contract produces the server journal contract, so the two halves
|
|
469
|
+
do not duplicate the Message codec, shared schema, initial snapshot, or replay
|
|
470
|
+
reducer:
|
|
471
|
+
|
|
472
|
+
```ts
|
|
473
|
+
import { Effect } from 'effect'
|
|
474
|
+
import { ActorId, DocumentId, Journal, OpId } from 'foldkit-durable'
|
|
111
475
|
|
|
112
476
|
type Principal = { readonly actorId: string }
|
|
113
477
|
|
|
114
478
|
const server = Effect.gen(function* () {
|
|
115
|
-
const journal = yield*
|
|
116
|
-
//
|
|
479
|
+
const journal = yield* Journal.make({
|
|
480
|
+
// Derived from the same Foldkit application + Sync declaration:
|
|
481
|
+
// operation codec, snapshot codec, empty shared value, replay, and
|
|
482
|
+
// compiled authorization rules when present.
|
|
117
483
|
...TodoSync.journalContract(),
|
|
484
|
+
|
|
118
485
|
file: 'todos.sqlite',
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
actorId: (principal: Principal) => actorId(principal.actorId),
|
|
486
|
+
opId: operation => OpId.make(operation.opId),
|
|
487
|
+
actorId: (principal: Principal) => ActorId.make(principal.actorId),
|
|
122
488
|
})
|
|
123
|
-
|
|
489
|
+
|
|
490
|
+
return yield* journal.load(DocumentId.make('todos'))
|
|
124
491
|
}).pipe(Effect.scoped)
|
|
125
492
|
```
|
|
126
493
|
|
|
127
|
-
|
|
494
|
+
The packages are intentionally separable. A custom server may implement the
|
|
495
|
+
exchange without Durable, and Durable can journal a protocol that did not come
|
|
496
|
+
from Sync.
|
|
497
|
+
|
|
498
|
+
## Mounting the full Foldkit application
|
|
499
|
+
|
|
500
|
+
The raw Replica works with the **shared slice**. `Sync.mount` is the Foldkit
|
|
501
|
+
runtime integration that connects that replica back to the whole application
|
|
502
|
+
Model.
|
|
503
|
+
|
|
504
|
+
A mounted durable Message:
|
|
505
|
+
|
|
506
|
+
```text
|
|
507
|
+
dispatch Message
|
|
508
|
+
|
|
|
509
|
+
v
|
|
510
|
+
application update immediately
|
|
511
|
+
|
|
|
512
|
+
+--> local fields / render
|
|
513
|
+
|
|
|
514
|
+
v
|
|
515
|
+
persist durable Message through replica
|
|
516
|
+
|
|
|
517
|
+
exchange changes shared state later
|
|
518
|
+
v
|
|
519
|
+
install reconciled shared slice back into Model
|
|
520
|
+
```
|
|
128
521
|
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
afterwards in a Command; the shared slice is re-installed from the replica when
|
|
132
|
-
an exchange or a rejection changes it, or when a persist fails.
|
|
522
|
+
So the UI still runs one application reducer. Sync does not replace the runtime
|
|
523
|
+
with a second state machine.
|
|
133
524
|
|
|
134
525
|
```ts
|
|
135
526
|
import { Effect, Scope } from 'effect'
|
|
136
|
-
import {
|
|
527
|
+
import { ReplicaId, Sync } from 'foldkit-sync'
|
|
137
528
|
|
|
138
|
-
const container = document.getElementById('app')!
|
|
529
|
+
const container = document.getElementById('app')!
|
|
139
530
|
|
|
140
531
|
const scope = Effect.runSync(Scope.make())
|
|
141
|
-
const storage = Effect.runSync(
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
532
|
+
const storage = Effect.runSync(
|
|
533
|
+
Effect.provideService(
|
|
534
|
+
Sync.indexedDb('todos/tab-1'),
|
|
535
|
+
Scope.Scope,
|
|
536
|
+
scope,
|
|
537
|
+
),
|
|
538
|
+
)
|
|
539
|
+
const replica = Effect.runSync(
|
|
540
|
+
TodoSync.openReplica(ReplicaId.make('tab-1'), storage),
|
|
541
|
+
)
|
|
542
|
+
|
|
543
|
+
const mounted = Sync.mount(App, TodoSync, {
|
|
145
544
|
replica,
|
|
146
545
|
container,
|
|
147
546
|
view: (model, h) => ({
|
|
@@ -151,202 +550,250 @@ const mounted = mount(App, TodoSync, {
|
|
|
151
550
|
model.todos.map(todo => h.li([], [todo.title])),
|
|
152
551
|
),
|
|
153
552
|
}),
|
|
154
|
-
onPersistenceFailure: (model, error) => ({
|
|
553
|
+
onPersistenceFailure: (model, error) => ({
|
|
554
|
+
...model,
|
|
555
|
+
lastError: error.message,
|
|
556
|
+
}),
|
|
155
557
|
})
|
|
156
558
|
|
|
157
|
-
|
|
158
|
-
Effect.
|
|
559
|
+
Effect.runFork(
|
|
560
|
+
Effect.provide(
|
|
561
|
+
replica.start,
|
|
562
|
+
Sync.transport.socket({ url: 'wss://example.com/sync' }),
|
|
563
|
+
),
|
|
564
|
+
)
|
|
565
|
+
|
|
566
|
+
mounted.dispatch(
|
|
567
|
+
Message.CreatedTodo({ id: crypto.randomUUID(), title: 'Milk' }),
|
|
568
|
+
)
|
|
159
569
|
|
|
160
|
-
mounted.
|
|
161
|
-
mounted.
|
|
162
|
-
await mounted.dispose() // waits for in-flight persists; the replica stays open
|
|
570
|
+
mounted.model()
|
|
571
|
+
await mounted.dispose()
|
|
163
572
|
```
|
|
164
573
|
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
`onUrlChange(url)` names the Message for every navigation, and `onUrlRequest`
|
|
168
|
-
the Message for a link click (omitted, the mount follows the link itself: an
|
|
169
|
-
internal one pushed, an external one loaded). A `foldkit-mirror` URL mirror
|
|
170
|
-
plugs in there as
|
|
171
|
-
`url: { init: (model, url) => Filters.reduce(model, url), onUrlChange: url => Message.UrlChanged({ url }) }`;
|
|
172
|
-
`subscriptions` and `resources` carry its write entry and the store layer.
|
|
574
|
+
`dispose()` waits for in-flight persists started by the mount; the Replica itself
|
|
575
|
+
remains separately owned and must be closed with its scope/lifecycle.
|
|
173
576
|
|
|
174
|
-
|
|
175
|
-
are the host an agent binds to; `observe` reports every application Message the
|
|
176
|
-
runtime applies, which a capability with a `completion` contract needs. See
|
|
177
|
-
[docs/sync-runtime-binding.md](https://github.com/doeixd/foldkit-plus/blob/main/docs/sync-runtime-binding.md) for what the
|
|
178
|
-
mount guarantees and why no Foldkit change is required.
|
|
577
|
+
### URL, Mirror, and agents
|
|
179
578
|
|
|
180
|
-
|
|
579
|
+
The optional `url` option routes navigation through the application:
|
|
181
580
|
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
581
|
+
- `init(model, url)` reduces the initial URL before the first render;
|
|
582
|
+
- `onUrlChange(url)` maps browser navigation back to a Message;
|
|
583
|
+
- `onUrlRequest` optionally maps link requests to a Message.
|
|
584
|
+
|
|
585
|
+
A `foldkit-mirror` URL mirror plugs into that seam without becoming another
|
|
586
|
+
state owner.
|
|
587
|
+
|
|
588
|
+
The mounted host also exposes `model`, `dispatch`, `subscribe`, and `observe`.
|
|
589
|
+
Agent runtimes bind to those same capabilities; `observe` reports application
|
|
590
|
+
Messages the runtime applies, including the completion facts an agent capability
|
|
591
|
+
may wait for. See [runtime binding](../../docs/sync-runtime-binding.md).
|
|
592
|
+
|
|
593
|
+
The Model shows a durable edit at once, before the server has it. An agent that
|
|
594
|
+
must not report success until the server commits the edit waits on
|
|
595
|
+
`mounted.committed` instead, the shared slice as confirmed:
|
|
185
596
|
|
|
186
597
|
```ts
|
|
187
|
-
|
|
188
|
-
|
|
598
|
+
completion: Agent.when({
|
|
599
|
+
source: mounted.committed,
|
|
600
|
+
predicate: (shared, request) => shared.todos.some(todo => todo.id === request.id),
|
|
601
|
+
})
|
|
602
|
+
```
|
|
603
|
+
|
|
604
|
+
It completes after the exchange that commits the edit, and never for one the
|
|
605
|
+
server rejects. The agent learns nothing about cursors or operations.
|
|
606
|
+
`mounted.committed` is a source (`{ get, subscribe }`), not a Projection: it reads
|
|
607
|
+
the replica, not the Model, and tells subscribers after every exchange.
|
|
608
|
+
|
|
609
|
+
## Fragments: compose one document from features
|
|
610
|
+
|
|
611
|
+
Large applications do not need one giant Sync declaration. Declare feature
|
|
612
|
+
fragments and compose them into one document:
|
|
613
|
+
|
|
614
|
+
```ts
|
|
615
|
+
const AppSync = Sync.forApplication(App)
|
|
616
|
+
|
|
617
|
+
const Todos = AppSync.fragment({
|
|
189
618
|
shared: Projection.pick(App.fields.todos),
|
|
190
|
-
durable: MessageSet.make(App, [
|
|
619
|
+
durable: MessageSet.make(App, [
|
|
620
|
+
Message.CreatedTodo,
|
|
621
|
+
Message.RenamedTodo,
|
|
622
|
+
]),
|
|
191
623
|
})
|
|
192
|
-
|
|
624
|
+
|
|
625
|
+
const Members = AppSync.fragment({
|
|
193
626
|
shared: Projection.pick(App.fields.members),
|
|
194
627
|
durable: MessageSet.make(App, [Message.Invited]),
|
|
195
628
|
})
|
|
196
|
-
|
|
629
|
+
|
|
630
|
+
const Board = AppSync.make({
|
|
631
|
+
documentId: DocumentId.make('board'),
|
|
632
|
+
...AppSync.compose(Todos, Members),
|
|
633
|
+
})
|
|
197
634
|
```
|
|
198
635
|
|
|
199
|
-
`compose` merges the shared projections and
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
636
|
+
`compose` merges the shared projections and durable Message subsets. It rejects
|
|
637
|
+
incompatible duplicate field declarations, duplicate durable variants, and
|
|
638
|
+
fragments from another application. `Module.validate` catches the separate
|
|
639
|
+
architectural mistake of two contracts claiming the same field ownership.
|
|
203
640
|
|
|
204
|
-
|
|
641
|
+
## Authorization and optimistic rejection
|
|
205
642
|
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
`
|
|
643
|
+
Authorization belongs on the Sync contract because it is policy over a durable
|
|
644
|
+
Message against the authoritative shared snapshot. The rule compiles into
|
|
645
|
+
`journalContract()` and is enforced by the server journal — not trusted to the
|
|
646
|
+
client.
|
|
209
647
|
|
|
210
648
|
```ts
|
|
211
|
-
const Authorized = forApplication(App)
|
|
649
|
+
const Authorized = Sync.forApplication(App)
|
|
650
|
+
.withPrincipal<{ readonly role: 'admin' | 'guest' }>()
|
|
651
|
+
|
|
212
652
|
const Board = Authorized.make({
|
|
213
|
-
documentId:
|
|
653
|
+
documentId: DocumentId.make('board'),
|
|
214
654
|
...Authorized.compose(Todos, Members),
|
|
215
655
|
authorize: {
|
|
216
|
-
// `message` is exactly `RenamedTodo`; `shared` is the authoritative snapshot.
|
|
217
656
|
RenamedTodo: ({ principal, message, shared }) =>
|
|
218
|
-
principal.role === 'admin' &&
|
|
657
|
+
principal.role === 'admin' &&
|
|
658
|
+
shared.todos.some(todo => todo.id === message.id),
|
|
219
659
|
},
|
|
220
660
|
})
|
|
221
661
|
```
|
|
222
662
|
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
not a durable tag is a compile error. The rule runs inside the append
|
|
226
|
-
transaction against the snapshot, so keep it synchronous and local. The client
|
|
227
|
-
does not run it: a replica has no principal, and a refused operation comes back
|
|
228
|
-
as a rejection.
|
|
663
|
+
A durable variant without a rule is allowed. A key that is not one of the
|
|
664
|
+
contract's durable Message variants is a type error.
|
|
229
665
|
|
|
230
|
-
|
|
666
|
+
The client does not run authorization. It may optimistically show the edit;
|
|
667
|
+
when the server refuses it, reconciliation drops that pending operation and
|
|
668
|
+
rebases the remaining ones. This is why rejection is a normal exchange result,
|
|
669
|
+
not a transport failure.
|
|
231
670
|
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
671
|
+
## Persistence and recovery
|
|
672
|
+
|
|
673
|
+
The local outbox is part of the feature, not an incidental cache. Losing it can
|
|
674
|
+
lose edits that never reached the server.
|
|
675
|
+
|
|
676
|
+
The built-in browser storage adapter is IndexedDB:
|
|
235
677
|
|
|
236
678
|
```ts
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
defineSync,
|
|
240
|
-
documentId,
|
|
241
|
-
indexedDb,
|
|
242
|
-
layerFromPromise,
|
|
243
|
-
replicaId,
|
|
244
|
-
type TransportClient,
|
|
245
|
-
} from 'foldkit-sync'
|
|
679
|
+
const storage = yield* Sync.indexedDb('todos/tab-1')
|
|
680
|
+
```
|
|
246
681
|
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
682
|
+
Stored replica state includes protocol/schema versions, document/replica
|
|
683
|
+
identity, cursor, committed snapshot, local sequence, and pending operations.
|
|
684
|
+
Persisted and remote operations are decoded strictly against the application
|
|
685
|
+
Message Schema.
|
|
686
|
+
|
|
687
|
+
Important recovery cases:
|
|
688
|
+
|
|
689
|
+
- **Storage absent or evicted.** The replica starts at cursor 0 and can catch up
|
|
690
|
+
from the server, but edits that existed only in the lost outbox are gone.
|
|
691
|
+
- **Two writers use one replica storage.** Compare-and-swap detects the stale
|
|
692
|
+
writer. Give each tab/writer its own replica/storage identity.
|
|
693
|
+
- **Replay refuses a Message.** `submit` fails and writes nothing, so invalid
|
|
694
|
+
durable work never enters the outbox.
|
|
695
|
+
- **Malformed stored data.** Opening fails rather than silently replacing bytes;
|
|
696
|
+
the application can offer an explicit reset/recovery path.
|
|
697
|
+
- **Unsupported storage version.** `UnsupportedReplicaVersionError` names the
|
|
698
|
+
persisted and supported versions so migration/reset can be explicit.
|
|
699
|
+
- **Server compaction.** A checkpoint replaces history the replica can no longer
|
|
700
|
+
replay and pending local work is rebased on it.
|
|
701
|
+
|
|
702
|
+
Application Message/shared-state migrations remain application policy; Sync
|
|
703
|
+
versioning protects its own persisted envelope.
|
|
704
|
+
|
|
705
|
+
## Transport
|
|
706
|
+
|
|
707
|
+
`Transport` is an Effect service for the one exchange operation. Sync ships
|
|
708
|
+
helpers under `Sync.transport`:
|
|
709
|
+
|
|
710
|
+
```text
|
|
711
|
+
Sync.transport.socket(...)
|
|
712
|
+
Sync.transport.loopback(...)
|
|
713
|
+
Sync.transport.fromPromise(...)
|
|
714
|
+
Sync.transport.toPromise(...)
|
|
715
|
+
Sync.transport.serve(...)
|
|
716
|
+
Sync.transport.nativeSocket(...)
|
|
717
|
+
```
|
|
250
718
|
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
empty: { todos: [] },
|
|
256
|
-
durable: () => true,
|
|
257
|
-
replay: (shared, message) => ({ todos: [...shared.todos, message.title] }),
|
|
258
|
-
})
|
|
719
|
+
The reconnecting socket transport uses bounded retries/queueing, exponential
|
|
720
|
+
jittered backoff, and keeps request identities stable when resending queued or
|
|
721
|
+
in-flight exchanges. A server rejection is protocol data; only a wire failure is
|
|
722
|
+
a `TransportError`.
|
|
259
723
|
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
724
|
+
Transport is deliberately below reconciliation. A custom transport can carry
|
|
725
|
+
the same exchange without changing replica semantics.
|
|
726
|
+
|
|
727
|
+
## Presence: ephemeral collaboration state
|
|
728
|
+
|
|
729
|
+
Presence is intentionally **not** a durable Message log. “Alice is viewing this
|
|
730
|
+
page” or a cursor position should expire when the peer disappears, not replay
|
|
731
|
+
next week.
|
|
732
|
+
|
|
733
|
+
`Sync.presence.make` creates a TTL'd peer registry. Values are decoded before
|
|
734
|
+
they enter the registry; a peer that stops refreshing is removed. Changes are
|
|
735
|
+
available both as a callback subscription and as a Stream.
|
|
736
|
+
|
|
737
|
+
Presence can travel through:
|
|
738
|
+
|
|
739
|
+
```text
|
|
740
|
+
Sync.presence.socketChannel(...)
|
|
741
|
+
Sync.presence.loopbackChannel(...)
|
|
267
742
|
```
|
|
268
743
|
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
- The Foldkit-facing contract (`forApplication(App).make`, optionally with
|
|
272
|
-
a custom `replay`): derives the shared projection, the durable Message subset, the
|
|
273
|
-
initial snapshot, and the journal contract from the application, so none is
|
|
274
|
-
declared twice. `Projection.pick`/`Projection.compose` build the writable projection;
|
|
275
|
-
`TodoSync.journalContract()` builds the durable codecs and reducer.
|
|
276
|
-
- The operation envelope: `replicaId:localSequence` identity, `baseCursor`, and
|
|
277
|
-
protocol/schema versions. The contract's `codec` (`normalizeOperation`,
|
|
278
|
-
`operationFrom`, `committedFrom`, `decodeExchange`) is there for a custom
|
|
279
|
-
transport or server; a server-committed operation is a `CommittedOperation`.
|
|
280
|
-
- A persisted outbox and optimistic projection: a durable Message is visible
|
|
281
|
-
immediately and rebased onto the authoritative order.
|
|
282
|
-
- Reconciliation: committed operations are replayed in order, acknowledged or
|
|
283
|
-
refused pending entries are dropped, and a checkpoint replaces the compacted
|
|
284
|
-
prefix. `replica.start` is the exchange loop: fork it to exchange once and then
|
|
285
|
-
after every `submit`, with a transport failure recorded in `status.lastError`
|
|
286
|
-
and retried on the next wake, so the application does not hand-roll it.
|
|
287
|
-
- A redacted status (`replica.status`): the pending count, the cursor, the last
|
|
288
|
-
exchange failure, and the operations the server refused — enough for a UI to
|
|
289
|
-
explain and recover without exposing Messages or the Model. `statusChanges` is
|
|
290
|
-
the same status, emitted on subscribe and re-emitted after every submit and
|
|
291
|
-
exchange. `changes` carries the status and the optimistic shared value together
|
|
292
|
-
from one read, so a UI holds one subscription instead of two.
|
|
293
|
-
- Strict decoding: an operation is always validated with the application's
|
|
294
|
-
Message schema, and a Message the contract does not call durable is refused.
|
|
295
|
-
- Branded positions: `Sequence` (a committed document position) and
|
|
296
|
-
`LocalSequence` (a replica's own 1-based counter) cannot be confused, so a
|
|
297
|
-
client counter is never passed where a committed position is expected.
|
|
298
|
-
- Presence (`createPresence`): an ephemeral, TTL'd peer registry, deliberately
|
|
299
|
-
outside the durable log. A peer that stops refreshing is dropped, not
|
|
300
|
-
replayed. It is an Effect driven by the `Clock` (so a `TestClock` makes the
|
|
301
|
-
TTL deterministic), peers live in a `Ref`, and the channel is a `PubSub`.
|
|
302
|
-
Every value is decoded through the required `decodeValue` before it is stored,
|
|
303
|
-
so a hostile peer cannot inject a value your `Update` type does not describe.
|
|
304
|
-
Presence can travel over a socket — `socketPresenceChannel` on the client and
|
|
305
|
-
`servePresence` fanning through a `createPresenceHub` on the server — or
|
|
306
|
-
in-process via `loopbackPresenceChannel`. `presence.changes` is a `Stream` that
|
|
307
|
-
emits the current peers and re-emits them on every change, alongside the
|
|
308
|
-
`subscribe` callback.
|
|
309
|
-
- The transport seam (`Transport`): an Effect service with a loopback layer, a
|
|
310
|
-
bridge to and from the promise client the replica speaks, and a WebSocket
|
|
311
|
-
client layer. The socket reconnects on an exponential, jittered backoff and
|
|
312
|
-
re-sends queued and in-flight frames with their original ids, so a lost reply
|
|
313
|
-
is answered rather than dropped; retries (`maxRetries`) and the queue
|
|
314
|
-
(`maxQueue`) are bounded, and `serveSocket` is the server side of a
|
|
315
|
-
connection. A refusal is an exchange result; only a wire failure is a
|
|
316
|
-
`TransportError`.
|
|
744
|
+
with server fan-out through:
|
|
317
745
|
|
|
318
|
-
|
|
746
|
+
```text
|
|
747
|
+
Sync.presence.hub(...)
|
|
748
|
+
Sync.presence.serve(...)
|
|
749
|
+
```
|
|
319
750
|
|
|
320
|
-
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
751
|
+
The registry uses Effect's `Clock`, so `TestClock` can make TTL behavior
|
|
752
|
+
deterministic in tests.
|
|
753
|
+
|
|
754
|
+
The conceptual rule is simple:
|
|
755
|
+
|
|
756
|
+
```text
|
|
757
|
+
shared application fact that must replay later -> durable Message / Sync
|
|
758
|
+
peer status that should disappear when stale -> Presence
|
|
759
|
+
```
|
|
327
760
|
|
|
328
761
|
## Last-writer-wins fields (experimental)
|
|
329
762
|
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
763
|
+
Server arrival order is the default conflict rule: whichever operation the
|
|
764
|
+
server commits first is replayed first. Sometimes one field needs a different
|
|
765
|
+
rule — for example, the newest logical title write should win even if an offline
|
|
766
|
+
device reconnects later.
|
|
767
|
+
|
|
768
|
+
`Sync.lww.register` supplies a schema and pure merge rule for that specialized
|
|
769
|
+
case. It is a field-level helper, **not a general CRDT mode for arbitrary
|
|
770
|
+
Messages**.
|
|
333
771
|
|
|
334
772
|
```ts
|
|
335
773
|
import { Schema } from 'effect'
|
|
336
|
-
import {
|
|
774
|
+
import { ReplicaId, Sync } from 'foldkit-sync'
|
|
337
775
|
|
|
338
|
-
const Title =
|
|
776
|
+
const Title = Sync.lww.register(Schema.String)
|
|
339
777
|
const Shared = Schema.Struct({ title: Title.schema })
|
|
340
|
-
const Renamed = Schema.Struct({
|
|
778
|
+
const Renamed = Schema.Struct({
|
|
779
|
+
_tag: Schema.Literal('Renamed'),
|
|
780
|
+
title: Title.schema,
|
|
781
|
+
})
|
|
341
782
|
|
|
342
|
-
const update = (
|
|
783
|
+
const update = (
|
|
784
|
+
model: typeof Shared.Type,
|
|
785
|
+
message: typeof Renamed.Type,
|
|
786
|
+
) => ({
|
|
343
787
|
...model,
|
|
344
788
|
title: Title.merge(model.title, message.title),
|
|
345
789
|
})
|
|
346
790
|
|
|
347
791
|
const message: typeof Renamed.Type = {
|
|
348
792
|
_tag: 'Renamed',
|
|
349
|
-
title: {
|
|
793
|
+
title: {
|
|
794
|
+
stamp: { counter: 1, replicaId: ReplicaId.make('tab-a') },
|
|
795
|
+
value: 'Milk',
|
|
796
|
+
},
|
|
350
797
|
}
|
|
351
798
|
```
|
|
352
799
|
|
|
@@ -355,80 +802,190 @@ order, independent of locale. This is logical ordering, not wall-clock time.
|
|
|
355
802
|
The rule follows the total-order register model described in
|
|
356
803
|
[Replicated Data Types: Specification, Verification, Optimality](https://www.microsoft.com/en-us/research/publication/replicated-data-types-specification-verification-optimality/).
|
|
357
804
|
|
|
358
|
-
Allocate stamps **before dispatch**, never
|
|
359
|
-
persists
|
|
360
|
-
|
|
805
|
+
Allocate stamps **before dispatch**, never inside `update` or replay.
|
|
806
|
+
`Sync.lww.openClock` persists the high-water mark independently of the outbox,
|
|
807
|
+
so rejected or never-submitted writes cannot cause timestamp reuse after reload:
|
|
361
808
|
|
|
362
809
|
```ts
|
|
363
810
|
import { Effect } from 'effect'
|
|
364
|
-
import {
|
|
811
|
+
import { DocumentId, ReplicaId, Sync, type Replica } from 'foldkit-sync'
|
|
365
812
|
|
|
366
813
|
type Shared = typeof Shared.Type
|
|
367
814
|
type Renamed = typeof Renamed.Type
|
|
368
815
|
|
|
369
816
|
const rename = (replica: Replica<Renamed, Shared>, title: string) =>
|
|
370
817
|
Effect.gen(function* () {
|
|
371
|
-
|
|
372
|
-
const
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
replicaId: replicaId('tab-a'),
|
|
818
|
+
const storage = yield* Sync.indexedDb('todos-tab-a-clock')
|
|
819
|
+
const clock = yield* Sync.lww.openClock({
|
|
820
|
+
documentId: DocumentId.make('todos'),
|
|
821
|
+
replicaId: ReplicaId.make('tab-a'),
|
|
376
822
|
storage,
|
|
377
823
|
})
|
|
824
|
+
|
|
378
825
|
const shared = yield* replica.shared
|
|
379
826
|
const stamp = yield* clock.next(shared.title.stamp.counter)
|
|
380
|
-
|
|
827
|
+
|
|
828
|
+
yield* replica.submit({
|
|
829
|
+
_tag: 'Renamed',
|
|
830
|
+
title: { stamp, value: title },
|
|
831
|
+
})
|
|
832
|
+
|
|
381
833
|
yield* clock.close
|
|
382
834
|
}).pipe(Effect.scoped)
|
|
383
835
|
```
|
|
384
836
|
|
|
385
|
-
`next(observedCounter)`
|
|
386
|
-
|
|
387
|
-
counter for the field being edited;
|
|
388
|
-
their maximum. Calling `next()` without an observation advances
|
|
389
|
-
|
|
837
|
+
`next(observedCounter)` persists a counter greater than both its previous saved
|
|
838
|
+
value and the supplied observation before returning the stamp. Pass the latest
|
|
839
|
+
observed counter for the field being edited; if causality spans several
|
|
840
|
+
registers, pass their maximum. Calling `next()` without an observation advances
|
|
841
|
+
only the saved local counter.
|
|
390
842
|
|
|
391
843
|
Allocations on one handle are serialized. Separate handles use storage's
|
|
392
|
-
compare-and-swap check; a stale handle fails and
|
|
393
|
-
Storage failures
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
397
|
-
|
|
398
|
-
|
|
399
|
-
Keep the clock database across reloads and outbox resets. If it is lost or
|
|
400
|
-
deleted, use a fresh replica id instead of resetting the same writer's counter.
|
|
401
|
-
For applications that allocate stamps themselves, preserve this same durable
|
|
402
|
-
high-water mark, including refused writes.
|
|
844
|
+
compare-and-swap check; a stale handle fails and should be closed/reopened.
|
|
845
|
+
Storage failures do not return a stamp. Counter gaps after a crash or rejected
|
|
846
|
+
submission are harmless.
|
|
847
|
+
|
|
848
|
+
Keep the clock storage across reloads and outbox resets. If it is lost, use a
|
|
849
|
+
fresh `ReplicaId` rather than reusing the same writer identity from counter zero.
|
|
403
850
|
|
|
404
851
|
A stamp identifies one immutable write within a register. Repeated delivery of
|
|
405
|
-
the same value is harmless; different
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
|
|
419
|
-
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
|
|
423
|
-
|
|
852
|
+
the same value is harmless; a different value with the same stamp throws. Keep
|
|
853
|
+
the **whole register, including its winning stamp**, in shared snapshots. For
|
|
854
|
+
deletion, a nullable register can retain a stamped `null` tombstone so an old
|
|
855
|
+
offline write cannot resurrect deleted state.
|
|
856
|
+
|
|
857
|
+
The server still orders and authorizes every operation, including writes that
|
|
858
|
+
lose the LWW merge. Replica ids and counters carried in Messages are
|
|
859
|
+
client-authored data, not authenticated identity; enforce writer/clock policy at
|
|
860
|
+
admission when needed.
|
|
861
|
+
|
|
862
|
+
Specialized sets, counters, collaborative text, and arbitrary commutative
|
|
863
|
+
Messages remain outside this helper.
|
|
864
|
+
|
|
865
|
+
## Lower level: `Sync.define`
|
|
866
|
+
|
|
867
|
+
`Sync.forApplication(App).make(...)` is the Foldkit-facing compiler. It builds
|
|
868
|
+
on `Sync.define`, the lower-level protocol primitive.
|
|
869
|
+
|
|
870
|
+
Use `Sync.define` directly when there is no Foldkit application to derive the
|
|
871
|
+
contract from — for example, a non-Foldkit client or a hand-written shared
|
|
872
|
+
projection:
|
|
873
|
+
|
|
874
|
+
```ts
|
|
875
|
+
import { Effect, Schema } from 'effect'
|
|
876
|
+
import {
|
|
877
|
+
DocumentId,
|
|
878
|
+
ReplicaId,
|
|
879
|
+
Sync,
|
|
880
|
+
type TransportClient,
|
|
881
|
+
} from 'foldkit-sync'
|
|
882
|
+
|
|
883
|
+
const Shared = Schema.Struct({ todos: Schema.Array(Schema.String) })
|
|
884
|
+
const Renamed = Schema.Struct({
|
|
885
|
+
_tag: Schema.Literal('Renamed'),
|
|
886
|
+
title: Schema.String,
|
|
887
|
+
})
|
|
888
|
+
|
|
889
|
+
declare const transport: TransportClient
|
|
890
|
+
|
|
891
|
+
const Protocol = Sync.define({
|
|
892
|
+
documentId: DocumentId.make('todos'),
|
|
893
|
+
message: Renamed,
|
|
894
|
+
shared: Shared,
|
|
895
|
+
empty: { todos: [] },
|
|
896
|
+
durable: () => true,
|
|
897
|
+
replay: (shared, message) => ({
|
|
898
|
+
todos: [...shared.todos, message.title],
|
|
899
|
+
}),
|
|
900
|
+
})
|
|
901
|
+
|
|
902
|
+
const program = Effect.gen(function* () {
|
|
903
|
+
const storage = yield* Sync.indexedDb('todos-tab-1')
|
|
904
|
+
const replica = yield* Protocol.openReplica(
|
|
905
|
+
ReplicaId.make('tab-1'),
|
|
906
|
+
storage,
|
|
907
|
+
)
|
|
908
|
+
|
|
909
|
+
yield* replica.submit({ _tag: 'Renamed', title: 'Milk' })
|
|
910
|
+
|
|
911
|
+
yield* Effect.provide(
|
|
912
|
+
replica.synchronize,
|
|
913
|
+
Sync.transport.fromPromise(transport),
|
|
914
|
+
)
|
|
915
|
+
|
|
916
|
+
return yield* replica.shared
|
|
917
|
+
}).pipe(Effect.scoped)
|
|
918
|
+
```
|
|
919
|
+
|
|
920
|
+
Most Foldkit applications should prefer `Sync.forApplication`: it keeps the
|
|
921
|
+
shared schema and replay semantics derived from the application rather than
|
|
922
|
+
re-declaring them.
|
|
923
|
+
|
|
924
|
+
## What Sync owns
|
|
925
|
+
|
|
926
|
+
- **The replication contract.** Shared projection, durable Message subset,
|
|
927
|
+
initial shared value, replay, operation codecs, and `journalContract()`.
|
|
928
|
+
- **The replica.** Persisted outbox, committed base, cursor, optimistic shared
|
|
929
|
+
projection, local operation sequence, synchronization, status, and recovery
|
|
930
|
+
validation.
|
|
931
|
+
- **Reconciliation.** Adopt committed operations/checkpoints, drop
|
|
932
|
+
acknowledged/rejected pending operations, and replay what remains.
|
|
933
|
+
- **The transport seam.** Effect `Transport` plus loopback/promise/WebSocket
|
|
934
|
+
adapters; transport carries exchanges but does not own conflict semantics.
|
|
935
|
+
- **Optional collaboration primitives.** Ephemeral Presence and the specialized
|
|
936
|
+
LWW register/clock helpers.
|
|
937
|
+
|
|
938
|
+
Sync does **not** own:
|
|
939
|
+
|
|
940
|
+
- unrelated local Model fields;
|
|
941
|
+
- server-derived disposable caches (`foldkit-remote`);
|
|
942
|
+
- authoritative server storage/order (`foldkit-durable` or your own server);
|
|
943
|
+
- peer-to-peer CRDT convergence;
|
|
944
|
+
- application Message/shared-state migrations.
|
|
945
|
+
|
|
946
|
+
## Limits
|
|
947
|
+
|
|
948
|
+
- IndexedDB is the built-in storage adapter today.
|
|
949
|
+
- Sync assumes an authoritative server order for each document; it is not a
|
|
950
|
+
peer-to-peer replication protocol.
|
|
951
|
+
- Ordinary concurrent writes resolve by that server order. `Sync.lww.register`
|
|
952
|
+
changes the merge rule for individual fields only; it does not make arbitrary
|
|
953
|
+
Messages commutative.
|
|
954
|
+
- There is no operational transform / collaborative text algorithm.
|
|
955
|
+
- Binding the socket transport and presence server to a platform WebSocket
|
|
956
|
+
server remains application/platform glue. The sync example demonstrates a
|
|
957
|
+
`ws` transport binding; presence over that server is not wired there today.
|
|
958
|
+
- Application schema migration remains application policy. Sync versions and
|
|
959
|
+
validates its own persisted/wire envelope.
|
|
960
|
+
|
|
961
|
+
## Older spellings
|
|
962
|
+
|
|
963
|
+
`Sync` groups the package exports; it does not wrap them. Existing top-level
|
|
964
|
+
spellings remain available with the same signatures:
|
|
965
|
+
|
|
966
|
+
- `forApplication`, `mount`, `indexedDb` — same names on the namespace;
|
|
967
|
+
- `defineSync` -> `Sync.define`; `syncMetrics` -> `Sync.metrics`;
|
|
968
|
+
- `layerSocket`, `layerLoopback`, `layerFromPromise`, `toPromise`, `serveSocket`,
|
|
969
|
+
`nativeSocket` -> `Sync.transport.socket`, `.loopback`, `.fromPromise`,
|
|
970
|
+
`.toPromise`, `.serve`, `.nativeSocket`;
|
|
971
|
+
- `createPresence`, `createPresenceHub`, `servePresence`,
|
|
972
|
+
`socketPresenceChannel`, `loopbackPresenceChannel` -> `Sync.presence.make`,
|
|
973
|
+
`.hub`, `.serve`, `.socketChannel`, `.loopbackChannel`;
|
|
974
|
+
- `lwwRegister`, `openLwwClock` -> `Sync.lww.register`, `Sync.lww.openClock`;
|
|
975
|
+
- `documentId('todos')` and `DocumentId.make('todos')` produce the same branded
|
|
976
|
+
value, and likewise for `replicaId`, `opId`, `sequence`, and
|
|
977
|
+
`localSequence`.
|
|
424
978
|
|
|
425
979
|
## See also
|
|
426
980
|
|
|
427
|
-
- [Replicated state](
|
|
428
|
-
|
|
429
|
-
- [Runtime binding](
|
|
430
|
-
|
|
431
|
-
- [`foldkit-durable`](
|
|
432
|
-
|
|
433
|
-
- [`
|
|
434
|
-
|
|
981
|
+
- [Replicated state](../../docs/replication.md) — Sync and Durable together,
|
|
982
|
+
including the server exchange and recovery model.
|
|
983
|
+
- [Runtime binding](../../docs/sync-runtime-binding.md) — what `Sync.mount`
|
|
984
|
+
guarantees and why no Foldkit core change is required.
|
|
985
|
+
- [`foldkit-durable`](../durable) — the authoritative server journal commonly
|
|
986
|
+
paired with Sync.
|
|
987
|
+
- [`foldkit-remote`](../remote) — server-owned disposable state rather than
|
|
988
|
+
client-authored durable state.
|
|
989
|
+
- [`examples/todo-app`](../../examples/todo-app) — browser mount, fragments,
|
|
990
|
+
authorization, Mirror, and agent integration.
|
|
991
|
+
- [`examples/sync`](../../examples/sync) — focused replica/server/transport trace.
|