foldkit-sync 0.7.0 → 0.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +424 -898
  2. package/package.json +8 -8
package/README.md CHANGED
@@ -2,103 +2,74 @@
2
2
 
3
3
  Local-first replication for part of a Foldkit application.
4
4
 
5
- A durable edit applies locally immediately, is persisted to an outbox, and is
6
- sent when the network allows. The server decides the authoritative order. The
7
- replica then replays that committed order and rebases any still-pending local
8
- edits on top.
9
-
10
- The important part is what **does not** change: your application's `Model`,
11
- `Message`, and `update` remain the state machine. Sync does not introduce a
12
- second reducer. You declare which Model slice is shared and which existing
13
- Messages are durable; Sync derives replay, persistence, protocol codecs, and the
14
- server journal contract from that declaration.
15
-
16
- > **Sync turns a subset of your existing Foldkit Messages into durable
17
- > operations. They still run through your normal `update`; Sync adds persistence,
18
- > optimistic replay, server ordering, and reconciliation around them.**
19
-
20
- **Use it when** a user should keep editing offline, the edit must survive a
21
- reload or network failure, and several replicas must eventually converge under
22
- one authoritative server order. **Do not use it for** server-owned disposable
23
- facts — use [`foldkit-remote`](../remote) for those — or for peer-to-peer / CRDT
24
- replication where every peer independently accepts writes.
5
+ You name a slice of the Model that several devices share, and the existing
6
+ Messages that change it. From then on an edit applies at once, is saved in a
7
+ local outbox, reaches the server when the network allows, and lands in the same
8
+ order on every device. Nothing about the application changes shape: `Model`,
9
+ `Message`, and `update` stay the one state machine. Sync wraps them; it never
10
+ adds a second reducer.
25
11
 
26
- It is the client half of [replicated Foldkit state](../../docs/replication.md).
27
- [`foldkit-durable`](../durable) is an optional server-side ordered journal.
12
+ > **A durable Message is one of your existing Messages, recorded.** It still
13
+ > runs through your `update`. Sync adds the outbox, the optimistic view, the
14
+ > exchange with the server, and the rebase when the server's answer arrives.
28
15
 
29
- ## Which state belongs in Sync?
16
+ It is the client half of [replicated Foldkit state](../../docs/replication.md).
17
+ [`foldkit-durable`](../durable) is the server half, an ordered journal on
18
+ SQLite. Either works without the other.
30
19
 
31
- A useful ownership rule is:
20
+ ## Is this the right package?
32
21
 
33
- | State | Owner |
34
- | --- | --- |
35
- | Route, selected tab, transient errors | ordinary local `Model` / `update` |
36
- | Server-owned data the client may refetch | `foldkit-remote` |
37
- | Client-authored state that must survive offline and converge | `foldkit-sync` |
38
- | Authoritative server ordering, idempotent append, snapshots | `foldkit-durable` |
22
+ | The state | Owner | Use |
23
+ | --- | --- | --- |
24
+ | A route, a selection, a transient error | the local Model | plain Foldkit |
25
+ | Facts the server owns and the client may refetch | the server | [`foldkit-remote`](../remote) |
26
+ | Edits the user makes that must survive offline and converge | the durable log | **`foldkit-sync`** |
27
+ | The authoritative order and snapshot on the server | the server journal | [`foldkit-durable`](../durable) |
39
28
 
40
- The difference between Remote and Sync is especially important:
29
+ The Remote and Sync line is the one worth memorizing. Remote caches a fact the
30
+ server owns, so refetching is recovery. Sync records an edit the user authored,
31
+ so the outbox is user data and replay is recovery.
41
32
 
42
- ```text
43
- Remote
44
- server owns the fact
45
- client caches it
46
- refetch is recovery
47
-
48
- Sync
49
- client creates the durable operation
50
- operation must survive offline
51
- replay + reconciliation is recovery
52
- ```
33
+ Sync assumes one server order per document. It is not peer-to-peer and not a
34
+ CRDT; every peer accepting writes without a server is a different design.
53
35
 
54
- A Surface may read local state, Remote state, and Sync-owned state at the same
55
- time. Those observation boundaries do not change who owns each datum.
56
-
57
- ## The replica mental model
58
-
59
- A replica has two pieces of state:
36
+ ## The model in one equation
60
37
 
61
38
  ```text
62
- authoritative committed snapshot
63
- +
64
- pending local operations
65
- =
66
- optimistic shared state the UI sees
39
+ what the user sees = committed snapshot + pending local edits, replayed
67
40
  ```
68
41
 
69
- If the server has committed `A B C` and this device has pending `D E`:
42
+ The server has committed `A B C`; this device has typed `D E` since:
70
43
 
71
44
  ```text
72
- server order: A B C
73
- local pending: D E
74
-
75
- visible state = replay(A, B, C, D, E)
45
+ server: A B C
46
+ pending: D E
47
+ visible: replay(A, B, C, D, E)
76
48
  ```
77
49
 
78
- If another device gets operation `X` committed before this replica's pending
79
- work, the next exchange may produce:
50
+ Another device gets `X` in before `D` reaches the server. After the next
51
+ exchange:
80
52
 
81
53
  ```text
82
- new server order: A B C X
83
- local pending: D E
84
-
85
- new visible state = replay(A, B, C, X, D, E)
54
+ server: A B C X
55
+ pending: D E
56
+ visible: replay(A, B, C, X, D, E)
86
57
  ```
87
58
 
88
- The local edit never needed to wait for that exchange to appear. Reconciliation
89
- changes the base underneath it and replays the still-pending operations on top.
59
+ `D` and `E` never waited. The base moved underneath them and they were replayed
60
+ on top. That one move, **rebase**, is also how a rejection rolls back: drop the
61
+ rejected edit from `pending` and replay the rest.
90
62
 
91
- The lifecycle is:
63
+ An edit passes three milestones, and each means something different:
92
64
 
93
- ```text
94
- local durable Message → optimistic update + local persistence
95
- → exchange → authoritative commit or refusal
96
- → new base + replay of remaining pending operations
97
- ```
65
+ | Milestone | Where | What it proves |
66
+ | --- | --- | --- |
67
+ | **visible** | the Model, at once | nothing yet: the user saw it |
68
+ | **saved** | the local outbox | a reload or a dropped connection cannot lose it |
69
+ | **committed** | the server journal | every other device will see it in this order |
98
70
 
99
- These are separate milestones. Showing an optimistic value is not an
100
- acknowledgment from either local storage or the server. Watch persistence and
101
- replica status when deciding what the UI may call saved.
71
+ A UI that says "saved" should read the second, and one that says "shared"
72
+ the third. Both are observable; see [what to show the user](#what-to-show-the-user).
102
73
 
103
74
  ## Install
104
75
 
@@ -107,23 +78,25 @@ pnpm add foldkit-sync
107
78
  ```
108
79
 
109
80
  `foldkit` and `effect` are peer dependencies; `foldkit-surface` comes with the
110
- package.
81
+ package. The browser storage is IndexedDB. The server needs
82
+ [`foldkit-durable`](../durable) or your own implementation of
83
+ [the exchange](#the-exchange).
111
84
 
112
- ## Sixty seconds: choose what is shared and what is durable
85
+ ## Sixty seconds: say what is shared
113
86
 
114
- Start with an ordinary Foldkit application:
87
+ Start from an ordinary application. The example keeps a todo list and a
88
+ selection; only the list is shared.
115
89
 
116
90
  ```ts
117
91
  import { Option, Schema } from 'effect'
118
92
  import { defineMessageUnion } from 'foldkit/message'
93
+ import { modifyFields } from 'foldkit/struct'
119
94
  import type * as Update from 'foldkit/update'
120
95
  import { MessageSet, Projection, Surface } from 'foldkit-surface'
121
96
  import { DocumentId, Sync } from 'foldkit-sync'
122
97
 
123
98
  const Model = Schema.Struct({
124
- todos: Schema.Array(
125
- Schema.Struct({ id: Schema.String, title: Schema.String }),
126
- ),
99
+ todos: Schema.Array(Schema.Struct({ id: Schema.String, title: Schema.String })),
127
100
  selectedTodoId: Schema.Option(Schema.String),
128
101
  lastError: Schema.Option(Schema.String),
129
102
  })
@@ -136,30 +109,22 @@ const Message = defineMessageUnion({
136
109
  })
137
110
  type Message = typeof Message.Type
138
111
 
139
- const initial: Model = {
140
- todos: [],
141
- selectedTodoId: Option.none(),
142
- lastError: Option.none(),
143
- }
112
+ const initial: Model = { todos: [], selectedTodoId: Option.none(), lastError: Option.none() }
144
113
 
145
114
  type Return = Update.Return<Model, Message>
146
115
 
147
- // This is still the application's one transition function.
148
116
  const update = (model: Model, message: Message): Return =>
149
117
  Message.match<Return>(message, {
150
118
  CreatedTodo: ({ id, title }) => ({
151
- model: { ...model, todos: [...model.todos, { id, title }] },
119
+ model: modifyFields(model, { todos: () => [...model.todos, { id, title }] }),
152
120
  }),
153
121
  RenamedTodo: ({ id, title }) => ({
154
- model: {
155
- ...model,
156
- todos: model.todos.map(todo =>
157
- todo.id === id ? { ...todo, title } : todo,
158
- ),
159
- },
122
+ model: modifyFields(model, {
123
+ todos: () => model.todos.map(todo => (todo.id === id ? { ...todo, title } : todo)),
124
+ }),
160
125
  }),
161
126
  SelectedTodo: ({ id }) => ({
162
- model: { ...model, selectedTodoId: Option.some(id) },
127
+ model: modifyFields(model, { selectedTodoId: () => Option.some(id) }),
163
128
  }),
164
129
  })
165
130
 
@@ -167,493 +132,190 @@ const App = Surface.application({ Model, Message, initial, update })
167
132
 
168
133
  const TodoSync = Sync.forApplication(App).make({
169
134
  documentId: DocumentId.make('todos'),
170
-
171
- // This slice of Model is replicated.
172
- // The writable Projection lets Sync install checkpoints/reconciled state back
173
- // into the application Model without owning a second copy of the schema.
174
135
  shared: Projection.pick(App.model.todos),
175
-
176
- // Only these existing Messages become durable operations.
177
- durable: MessageSet.make(App, [
178
- Message.CreatedTodo,
179
- Message.RenamedTodo,
180
- ]),
136
+ durable: MessageSet.make(App, [Message.CreatedTodo, Message.RenamedTodo]),
181
137
  })
182
138
  ```
183
139
 
184
- Read that declaration as:
140
+ Read the declaration as a table:
185
141
 
186
142
  ```text
187
- Model
188
- todos -> shared / replicated
189
- selectedTodoId -> local only
190
- lastError -> local only
191
-
192
- Messages
193
- CreatedTodo -> durable
194
- RenamedTodo -> durable
195
- SelectedTodo -> local only
196
- ```
197
-
198
- `Sync.forApplication(App)` derives the shared schema, initial shared snapshot,
199
- replay function, operation codecs, and server journal contract from the same
200
- application declaration. There is no parallel sync-specific version of
201
- `update` to keep aligned.
202
-
203
- ### Declaration is not a running replica
204
-
205
- The example above defines codecs and replay rules. It has not opened storage,
206
- created an outbox, connected to a server, or mounted a view. For the application
207
- path, continue with [mounting](#mounting-the-full-foldkit-application). Use
208
- [the replica API](#running-a-replica) when you need to control that lifecycle.
209
-
210
- `SelectedTodo` stays local because it is outside the durable MessageSet.
211
- Adding a Message to that set is a semantic decision: its update must be safe
212
- to replay and confined to the shared projection. An intent that starts a
213
- Command should instead produce a deterministic fact, as described next.
214
-
215
- ## Wiring: the contract joins an assembly
143
+ todos shared replicated, persisted, journaled
144
+ selectedTodoId local this device only
145
+ lastError local
216
146
 
217
- The contract joins a `foldkit-bundle` assembly contract-only, so the Module
218
- sees it:
219
-
220
- ```ts
221
- const wiring = TodoSync.wiring()
147
+ CreatedTodo durable recorded in the outbox, committed by the server
148
+ RenamedTodo durable
149
+ SelectedTodo local runs through update and goes nowhere
222
150
  ```
223
151
 
224
- It routes nothing, runs nothing, and subscribes to nothing — Sync owns the
225
- runtime through `mount` rather than joining it — and `wiring.contract` is the
226
- same contract above.
227
-
228
- ## What makes a Message durable?
152
+ From that, `make` derives the shared codec, the initial shared value, the
153
+ operation envelope, replay (your `update` on the shared slice), and the
154
+ server's [journal contract](#the-server). You write none of them.
229
155
 
230
- A durable Message must be something another machine can replay later and obtain
231
- the same shared state.
156
+ `TodoSync` is a value. It has opened no storage, connected to nothing, and
157
+ rendered nothing. The next section does that.
232
158
 
233
- In practice that means it must be a deterministic, state-only transition of the
234
- shared projection.
159
+ ## Run it in the browser
235
160
 
236
- Derived replay protects that boundary. It refuses a durable Message when the
237
- application's `update`:
238
-
239
- - returns a Command — a live external effect cannot be replayed later; or
240
- - changes a field outside the shared projection — that change would disappear
241
- when only the shared slice is persisted/replayed.
242
-
243
- For example:
244
-
245
- ```text
246
- RequestedChargeCard
247
- -> runs a Command / talks to a provider
248
- -> local intent, NOT durable
249
-
250
- CardCharged
251
- -> pure fact about state
252
- -> durable and replayable
253
- ```
254
-
255
- A common pattern is therefore:
256
-
257
- ```text
258
- local intent Message
259
- |
260
- v
261
- Command / nondeterministic work
262
- |
263
- v
264
- durable fact Message
265
- ```
266
-
267
- Replay runs the durable fact through the same application `update` on every
268
- replica. If an application genuinely needs different replay semantics, `make`
269
- accepts a custom `replay`, but that custom replay does not get the derived
270
- guardrails.
271
-
272
- When the work between intent and fact needs nothing but the Model — an id minted
273
- from a local counter, say — return the fact as `Sync.fact(message)` instead of an
274
- ordinary Command. `Sync.mount` applies it straight after the intent's `update`,
275
- in the same transition, and persists it if it is durable. An ordinary Command's
276
- Message arrives later, so a second intent dispatched in the meantime would read
277
- the Model from before the fact and could mint the same id. Outside `mount`, or
278
- once a parent has mapped it, `Sync.fact` is an ordinary Command that yields the
279
- Message.
280
-
281
- ## Running a replica
282
-
283
- The low-level runtime is a `Replica`. It owns the local outbox, committed shared
284
- snapshot, cursor, optimistic projection, and synchronization state.
161
+ Three steps: open storage and a replica, mount the application over it, start
162
+ the exchange loop.
285
163
 
286
164
  ```ts
287
- import { Effect } from 'effect'
165
+ import { Effect, Scope } from 'effect'
288
166
  import { ReplicaId, Sync } from 'foldkit-sync'
289
167
 
290
- const program = Effect.gen(function* () {
291
- // One durable storage identity per document + replica/writer.
292
- const storage = yield* Sync.indexedDb('todos/tab-1')
293
-
294
- const replica = yield* TodoSync.openReplica(
295
- ReplicaId.make('tab-1'),
296
- storage,
297
- )
298
-
299
- // submit validates/replays first, then persists the operation in the outbox.
300
- yield* replica.submit(
301
- Message.CreatedTodo({ id: 't1', title: 'Milk' }),
302
- )
303
-
304
- // This already includes the pending operation. No server round trip required.
305
- const optimistic = yield* replica.shared
306
-
307
- return optimistic
308
- }).pipe(Effect.scoped)
309
- ```
310
-
311
- `replica.submit(message)` does not mean “send this now.” It means:
312
-
313
- 1. verify the Message is valid and durable;
314
- 2. replay it against the current optimistic shared state;
315
- 3. assign a stable local operation identity;
316
- 4. persist it to the local outbox;
317
- 5. make the new optimistic projection observable; and
318
- 6. wake the exchange loop if one is running.
319
-
320
- A Message replay rejects never enters the outbox.
321
-
322
- ## Optimistic state and rebasing
323
-
324
- Internally the replica keeps:
325
-
326
- ```text
327
- committed
328
- the authoritative shared snapshot up to cursor N
329
-
330
- pending
331
- locally-authored operations not yet resolved by the server
332
- ```
333
-
334
- The value returned by `replica.shared` is:
335
-
336
- ```text
337
- pending.reduce(replay, committed)
338
- ```
339
-
340
- That simple rule gives optimistic UI and reconciliation the same semantics.
341
- There is no special “optimistic reducer.”
342
-
343
- When synchronization advances the committed base, Sync removes pending
344
- operations that became committed, were explicitly acknowledged, or were
345
- rejected, then the remaining outbox is replayed over the new committed state.
346
-
347
- This also explains why durable Messages have to be replay-safe: a pending
348
- operation may run many times as the authoritative base changes beneath it.
349
-
350
- ## Synchronization
351
-
352
- A replica exchanges these with the server:
353
-
354
- ```text
355
- request
356
- current cursor
357
- pending operations
358
- the server's epoch, once the replica has heard one
359
-
360
- response
361
- newly committed operations
362
- acknowledgements
363
- rejections
364
- optional checkpoint
365
- optional more (the replica asks again at once)
366
- optional epoch
367
- ```
368
-
369
- `replica.synchronize` performs one exchange through the `Transport` Effect
370
- service:
168
+ const container = document.getElementById('app')!
371
169
 
372
- ```ts
373
- const once = replica.synchronize.pipe(
374
- Effect.provide(Sync.transport.socket({ url: 'wss://example.com/sync' })),
170
+ // One IndexedDB database per document and tab; its connection lives for the page.
171
+ const scope = Effect.runSync(Scope.make())
172
+ const storage = Effect.runSync(
173
+ Effect.provideService(Sync.indexedDb('todos/tab-1'), Scope.Scope, scope),
375
174
  )
376
- ```
175
+ const replica = Effect.runSync(TodoSync.openReplica(ReplicaId.make('tab-1'), storage))
377
176
 
378
- `replica.start` is the long-running convenience loop. It exchanges once, then
379
- wakes after every submit and every notice the transport hears from the server
380
- (`Transport.changes`). A failed exchange, whether the wire failed or the
381
- response was refused, is recorded in `status.lastError` and announced on
382
- `statusChanges`; the loop survives and retries on a backoff from 0.5 s up to
383
- 30 s, or at once on the next submit, so an outbox is delivered even if nobody
384
- types again.
177
+ const mounted = Sync.mount(App, TodoSync, {
178
+ replica,
179
+ container,
180
+ view: (model, h) => ({
181
+ title: 'Todos',
182
+ body: h.ul([], model.todos.map(todo => h.li([], [todo.title]))),
183
+ }),
184
+ onPersistenceFailure: (model, error) =>
185
+ modifyFields(model, { lastError: () => Option.some(error.message) }),
186
+ })
385
187
 
386
- ```ts
387
188
  Effect.runFork(
388
- Effect.provide(
389
- replica.start,
390
- Sync.transport.socket({ url: 'wss://example.com/sync' }),
391
- ),
189
+ Effect.provide(replica.start, Sync.transport.socket({ url: 'wss://example.com/sync' })),
392
190
  )
393
- ```
394
-
395
- The transport does not decide reconciliation. It only carries the exchange;
396
- the replica validates the response and applies the protocol rules.
397
-
398
- ### Acknowledgements
399
191
 
400
- An acknowledgement says an operation the replica sent is durably accepted, so
401
- it may leave the outbox even if its committed payload is not repeated in this
402
- response. While a paged response says there is `more`, an acknowledged operation
403
- stays pending, because its committed copy may be on a later page; it leaves once
404
- that copy arrives or the last page has been read.
405
-
406
- ### Rejections
407
-
408
- A rejection says the optimistic operation did not commit. Sync removes it from
409
- the pending set and recomputes the optimistic state from the authoritative base
410
- plus whatever pending operations remain.
411
-
412
- ```text
413
- before response
414
- committed: A B C
415
- pending: D E
416
- visible: A B C D E
417
-
418
- server rejects D
419
-
420
- committed: A B C
421
- pending: E
422
- visible: A B C E
423
- ```
424
-
425
- `replica.status.rejected` keeps recent rejected operation ids so the UI can
426
- explain that an optimistic edit was refused.
427
-
428
- ### Checkpoints
429
-
430
- A server may compact old history. If the replica is too far behind to replay the
431
- missing prefix, the response can include a checkpoint:
432
-
433
- ```text
434
- checkpoint = authoritative shared snapshot + cursor
192
+ mounted.dispatch(Message.CreatedTodo({ id: crypto.randomUUID(), title: 'Milk' }))
435
193
  ```
436
194
 
437
- The replica adopts the checkpoint as its committed base, then rebases its local
438
- pending operations on top. A checkpoint behind the replica's current cursor is
439
- rejected with `CheckpointRegressionError` rather than silently moving history
440
- backwards.
195
+ What each call does, and does not do:
441
196
 
442
- ### Paged answers
197
+ - **`Sync.indexedDb(name)`** opens one database. Name it by document and
198
+ writer: two tabs on one name race, and the loser fails its write rather than
199
+ merging two histories.
200
+ - **`openReplica(id, storage)`** reads back the committed snapshot, cursor, and
201
+ outbox saved last time, or starts empty. It does no network I/O.
202
+ - **`Sync.mount(App, TodoSync, …)`** runs the ordinary Foldkit runtime with
203
+ your `view` and `update`. A durable Message applies through `update` at once,
204
+ then a Command persists it; if the persist fails, the edit is reverted and
205
+ `onPersistenceFailure` says so in the Model. A local Message is untouched.
206
+ - **`replica.start`** is the exchange loop: once now, then after every submit
207
+ and every notice from the server. A failed exchange is retried on a backoff
208
+ from 0.5 s to 30 s, or at once on the next edit. The loop never fails; it
209
+ reports through `status`.
210
+ - **`mounted.dispatch`** sends a Message the way a view does. The container
211
+ must have an `id`, or `mount` throws.
443
212
 
444
- A server may answer with only part of what follows the cursor and set `more:
445
- true`. `synchronize` then exchanges again at once, until an answer has no
446
- `more` or a round moves the cursor nowhere. A replica that was offline for a
447
- long time therefore catches up in bounded steps within one `synchronize`. The
448
- example servers page with Durable's `read(key, cursor, { limit })`.
213
+ Reload the page: the todos come back from the outbox and the committed
214
+ snapshot, the selection does not. That is the shared/local line, visible.
449
215
 
450
- ### Coalescing a burst
216
+ `mounted` also exposes `model()`, `subscribe`, `observe`, `committed`, and
217
+ `dispose()`, which waits for in-flight persists and leaves the replica open.
218
+ Close the replica with its own scope.
451
219
 
452
- Typing makes one durable Message per keystroke. A contract can merge them while
453
- the server has not seen them. `make({ ..., coalesce: (last, next) => merged })`
454
- is offered each submitted Message together with the last one in the outbox. If
455
- it returns a Message, that Message replaces the last operation under the last
456
- operation's id; if it returns `undefined`, the new Message gets an operation of
457
- its own. Replaying the merged Message must give the same result as replaying
458
- `last` then `next`.
220
+ ## The server
459
221
 
460
- An operation is merged into only while no exchange has carried it. Once one has,
461
- even an exchange that failed, the server may already have committed it as it
462
- was. An operation loaded from storage is never merged into either. So there is
463
- no timer to tune: a burst typed while an exchange is out becomes one operation,
464
- and a single keystroke goes out as quickly as it did before. The local sequence
465
- still advances on every submit.
466
-
467
- ## Replica status and UI state
468
-
469
- The public replica surface deliberately exposes enough state for UI and recovery
470
- without exposing the application Messages themselves:
222
+ The server answers the exchange: it takes a cursor and pending operations,
223
+ commits what it accepts, and returns what the replica is missing. With
224
+ `foldkit-durable`, the same contract supplies the journal's codecs, empty
225
+ snapshot, reducer, and any [authorization](#authorize-on-the-server):
471
226
 
472
227
  ```ts
473
- const status = yield* replica.status
474
- // {
475
- // pending: number,
476
- // cursor: Sequence,
477
- // lastError: string | undefined,
478
- // rejected: ReadonlyArray<OpId>
479
- // }
480
- ```
481
-
482
- `replica.statusChanges` emits that status initially and after every submit and
483
- exchange. `replica.changes` emits the status and optimistic `shared` value
484
- together from one replica snapshot, so a UI can subscribe once without racing
485
- two separate reads. `replica.snapshot` is the same read, once: the status,
486
- `shared`, `committed`, and `nextLocalSequence`, the local sequence the next
487
- submitted operation takes.
488
-
489
- ## Sync + Durable
490
-
491
- Sync owns **client replication semantics**. Durable can own **server ordering and
492
- persistence**.
493
-
494
- ```text
495
- client
496
- -----------------------------------------------
497
- Foldkit update
498
- |
499
- | durable Message
500
- v
501
- foldkit-sync Replica
502
- optimistic shared state
503
- persistent outbox
504
- |
505
- | exchange
506
- v
507
- -----------------------------------------------
508
- server
509
- foldkit-durable Journal
510
- authoritative order
511
- snapshot + cursor
512
- |
513
- v
514
- committed operations / checkpoint / rejection
515
- ```
516
-
517
- The same Sync contract produces the server journal contract, so the two halves
518
- do not duplicate the Message codec, shared schema, initial snapshot, or replay
519
- reducer:
520
-
521
- ```ts
522
- import { Effect } from 'effect'
523
228
  import { ActorId, DocumentId, Journal, OpId } from 'foldkit-durable'
524
229
 
525
230
  type Principal = { readonly actorId: string }
526
231
 
527
- const server = Effect.gen(function* () {
528
- const journal = yield* Journal.make({
529
- // Derived from the same Foldkit application + Sync declaration:
530
- // operation codec, snapshot codec, empty shared value, replay, and
531
- // compiled authorization rules when present.
532
- ...TodoSync.journalContract(),
232
+ const journal = yield* Journal.make({
233
+ ...TodoSync.journalContract(),
234
+ file: 'todos.sqlite',
235
+ opId: operation => OpId.make(operation.opId),
236
+ actorId: (principal: Principal) => ActorId.make(principal.actorId),
237
+ })
238
+ ```
239
+
240
+ The exchange handler itself is yours: it calls `journal.append` for each
241
+ pending operation, `journal.read` for what follows the cursor, and serves the
242
+ result on a socket with `Sync.transport.serve`. [The replicated state
243
+ guide](../../docs/replication.md#3-the-server-an-exchange-over-the-journal) shows
244
+ the whole handler in one page, and [`examples/sync`](../../examples/sync) runs
245
+ it over `ws`.
533
246
 
534
- file: 'todos.sqlite',
535
- opId: operation => OpId.make(operation.opId),
536
- actorId: (principal: Principal) => ActorId.make(principal.actorId),
537
- })
247
+ A server that is not Durable implements the same [exchange](#the-exchange).
538
248
 
539
- return yield* journal.load(DocumentId.make('todos'))
540
- }).pipe(Effect.scoped)
541
- ```
249
+ ## What makes a Message durable
542
250
 
543
- The packages are intentionally separable. A custom server may implement the
544
- exchange without Durable, and Durable can journal a protocol that did not come
545
- from Sync.
251
+ Replay runs a durable Message through `update` on another machine, later,
252
+ possibly many times as the base beneath it moves. So it has to be a
253
+ deterministic, state-only transition of the shared slice. `make` derives replay
254
+ from `update` and refuses a durable Message whose transition:
546
255
 
547
- ## Mounting the full Foldkit application
256
+ - returns a Command, because an effect cannot be replayed; or
257
+ - changes a field outside `shared`, because that change would be lost.
548
258
 
549
- The raw Replica works with the **shared slice**. `Sync.mount` is the Foldkit
550
- runtime integration that connects that replica back to the whole application
551
- Model.
259
+ A refused Message fails `submit` with `ReplayError` and writes nothing, so a
260
+ mistake shows on the first edit instead of as diverging replicas later.
552
261
 
553
- A mounted durable Message:
262
+ The consequence is a pattern you will use everywhere: **an intent mints, a fact
263
+ records**.
554
264
 
555
265
  ```text
556
- dispatch Message
557
- |
266
+ RequestedTodo { title } local: its update returns a Command
267
+ | that reads the clock and mints an id
558
268
  v
559
- application update immediately
560
- |
561
- +--> local fields / render
562
- |
563
- v
564
- persist durable Message through replica
565
- |
566
- exchange changes shared state later
567
- v
568
- install reconciled shared slice back into Model
269
+ CreatedTodo { id, title, at } durable: pure over the shared slice
569
270
  ```
570
271
 
571
- So the UI still runs one application reducer. Sync does not replace the runtime
572
- with a second state machine.
573
-
574
- ```ts
575
- import { Effect, Scope } from 'effect'
576
- import { ReplicaId, Sync } from 'foldkit-sync'
272
+ Put every nondeterministic input (ids, timestamps, the result of a provider
273
+ call) *inside* the fact. `update` for the fact only applies it.
577
274
 
578
- const container = document.getElementById('app')!
275
+ When the intent needs nothing outside the Model, say an id from a counter in
276
+ the shared slice, return the fact as `Sync.fact(message)` instead of an
277
+ ordinary Command. The mount applies it in the intent's own transition, before
278
+ any later Message, so two intents dispatched together cannot read the same
279
+ counter. Outside the mount it is an ordinary Command.
579
280
 
580
- const scope = Effect.runSync(Scope.make())
581
- const storage = Effect.runSync(
582
- Effect.provideService(
583
- Sync.indexedDb('todos/tab-1'),
584
- Scope.Scope,
585
- scope,
586
- ),
587
- )
588
- const replica = Effect.runSync(
589
- TodoSync.openReplica(ReplicaId.make('tab-1'), storage),
590
- )
591
-
592
- const mounted = Sync.mount(App, TodoSync, {
593
- replica,
594
- container,
595
- view: (model, h) => ({
596
- title: 'Todos',
597
- body: h.ul(
598
- [],
599
- model.todos.map(todo => h.li([], [todo.title])),
600
- ),
601
- }),
602
- onPersistenceFailure: (model, error) => ({
603
- ...model,
604
- lastError: Option.some(error.message),
605
- }),
606
- })
607
-
608
- Effect.runFork(
609
- Effect.provide(
610
- replica.start,
611
- Sync.transport.socket({ url: 'wss://example.com/sync' }),
612
- ),
613
- )
614
-
615
- mounted.dispatch(
616
- Message.CreatedTodo({ id: crypto.randomUUID(), title: 'Milk' }),
617
- )
281
+ A durable `update` should also be idempotent where a retry could deliver one
282
+ operation twice, as in `CreatedTodo` ignoring an id it already holds.
618
283
 
619
- mounted.model()
620
- await mounted.dispose()
621
- ```
284
+ ## What to show the user
622
285
 
623
- `dispose()` waits for in-flight persists started by the mount; the Replica itself
624
- remains separately owned and must be closed with its scope/lifecycle.
286
+ The three milestones each have a reader.
625
287
 
626
- The mount replaces the shared slice outside `update` in two cases: when an
627
- exchange changes what the replica holds (it commits, acknowledges or rejects
628
- something), and when a failed persist reverts its edit. A replica status that
629
- changes nothing the Model shows, such as the one a mount starts with, replaces
630
- nothing. `onReinstall(next, previous)` sees both cases and returns the
631
- transition, so local state that points into the shared slice (a selection) can
632
- be carried across the change, and a Command can take it where it has to go (a
633
- DOM that is patched rather than re-rendered). Omitted, the Model is simply
634
- `next`. A durable `Sync.fact` it returns is a new edit, so return one only for
635
- what changed: returned every time, it adds an edit per exchange, and while
636
- storage keeps failing, one per failure.
288
+ **Visible** is the Model. Nothing to do.
637
289
 
638
- ### URL, Mirror, and agents
290
+ **Saved** is the persist. `onPersistenceFailure` runs when it fails, after the
291
+ edit has been reverted; the Model it returns is what the user sees next. The
292
+ error names why: `ReplayError` for a Message that cannot be durable,
293
+ `StorageError` for IndexedDB.
639
294
 
640
- The optional `url` option routes navigation through the application:
295
+ **Committed** is the replica's status:
641
296
 
642
- - `init(model, url)` reduces the initial URL before the first render;
643
- - `onUrlChange(url)` maps browser navigation back to a Message;
644
- - `onUrlRequest` optionally maps link requests to a Message.
297
+ ```ts
298
+ const status = yield* replica.status
299
+ // { pending: number, cursor: Sequence, lastError: string | undefined, rejected: ReadonlyArray<OpId> }
300
+ ```
645
301
 
646
- A `foldkit-mirror` URL mirror plugs into that seam without becoming another
647
- state owner.
302
+ `pending` above zero means edits the server has not confirmed; `lastError` is
303
+ the last failed exchange, cleared by the next success; `rejected` lists recent
304
+ edits the server refused. `replica.statusChanges` is a Stream of the same,
305
+ emitted after every submit and exchange; `replica.changes` pairs it with the
306
+ optimistic `shared` value so one subscription sees both.
648
307
 
649
- The mounted host also exposes `model`, `dispatch`, `subscribe`, and `observe`.
650
- Agent runtimes bind to those same capabilities; `observe` reports application
651
- Messages the runtime applies, including the completion facts an agent capability
652
- may wait for. See [runtime binding](../../docs/sync-runtime-binding.md).
308
+ **A refused edit** arrives as a rejection on the next exchange, not as an
309
+ error. The replica drops it and replays the rest; the mount re-installs the
310
+ shared slice, and the field the user edited reverts. `onPersistenceFailure`
311
+ does not run for this, since the local save succeeded. Watch `rejected` to
312
+ explain the revert.
653
313
 
654
- The Model shows a durable edit at once, before the server has it. An agent that
655
- must not report success until the server commits the edit waits on
656
- `mounted.committed` instead, the shared slice as confirmed:
314
+ **Waiting for the server.** The Model shows an edit before the server has it.
315
+ Something that must not report success until the edit is committed, an agent
316
+ tool for instance, waits on `mounted.committed`: the shared slice as the server
317
+ confirmed it, with no pending edit applied. It is a source, `{ get, subscribe }`,
318
+ told after every exchange:
657
319
 
658
320
  ```ts
659
321
  completion: Agent.when({
@@ -662,29 +324,22 @@ completion: Agent.when({
662
324
  })
663
325
  ```
664
326
 
665
- It completes after the exchange that commits the edit, and never for one the
666
- server rejects. The agent learns nothing about cursors or operations.
667
- `mounted.committed` is a source (`{ get, subscribe }`), not a Projection: it reads
668
- the replica, not the Model, and tells subscribers after every exchange.
327
+ That completes after the exchange that commits the todo and never for one the
328
+ server rejects.
669
329
 
670
- Remote draws the same line over its own cache, with the same words and a
671
- different mechanism: see [what a reader sees while a change is in
672
- flight](../../docs/state-model.md#what-a-reader-sees-while-a-change-is-in-flight).
330
+ ## Growing the application
673
331
 
674
- ## Fragments: compose one document from features
332
+ ### Compose a document from features
675
333
 
676
- Large applications do not need one giant Sync declaration. Declare feature
677
- fragments and compose them into one document:
334
+ Each feature declares the fields it shares and the Messages that change them;
335
+ `compose` merges them into one document:
678
336
 
679
337
  ```ts
680
338
  const AppSync = Sync.forApplication(App)
681
339
 
682
340
  const Todos = AppSync.fragment({
683
341
  shared: Projection.pick(App.model.todos),
684
- durable: MessageSet.make(App, [
685
- Message.CreatedTodo,
686
- Message.RenamedTodo,
687
- ]),
342
+ durable: MessageSet.make(App, [Message.CreatedTodo, Message.RenamedTodo]),
688
343
  })
689
344
 
690
345
  const Members = AppSync.fragment({
@@ -698,424 +353,295 @@ const Board = AppSync.make({
698
353
  })
699
354
  ```
700
355
 
701
- `compose` merges the shared projections and durable Message subsets. It rejects
702
- incompatible duplicate field declarations, duplicate durable variants, and
703
- fragments from another application. `Module.validate` catches the separate
704
- architectural mistake of two contracts claiming the same field ownership.
356
+ `compose` throws on a field declared twice with different codecs, a Message
357
+ declared durable twice, or a fragment from another application. The result is
358
+ still one document: one outbox, one cursor, one server order.
705
359
 
706
- ## Authorization and optimistic rejection
360
+ ### Authorize on the server
707
361
 
708
- Authorization belongs on the Sync contract because it is policy over a durable
709
- Message against the authoritative shared snapshot. The rule compiles into
710
- `journalContract()` and is enforced by the server journal — not trusted to the
711
- client.
362
+ Policy belongs on the contract, beside the Messages it governs, and runs on the
363
+ server inside the journal's commit. The client never runs it.
712
364
 
713
365
  ```ts
714
- const Authorized = Sync.forApplication(App)
715
- .withPrincipal<{ readonly role: 'admin' | 'guest' }>()
366
+ const Authorized = Sync.forApplication(App).withPrincipal<{ readonly role: 'admin' | 'guest' }>()
716
367
 
717
368
  const Board = Authorized.make({
718
369
  documentId: DocumentId.make('board'),
719
370
  ...Authorized.compose(Todos, Members),
720
371
  authorize: {
721
372
  RenamedTodo: ({ principal, message, shared }) =>
722
- principal.role === 'admin' &&
723
- shared.todos.some(todo => todo.id === message.id),
373
+ principal.role === 'admin' && shared.todos.some(todo => todo.id === message.id),
724
374
  },
725
375
  })
726
376
  ```
727
377
 
728
- A durable variant without a rule is allowed. A key that is not one of the
729
- contract's durable Message variants is a type error.
378
+ `message` is exactly that variant, `shared` is the authoritative snapshot, and
379
+ `principal` is whatever the server's transport established for the connection;
380
+ it is never read from the operation. A variant without a rule is allowed. A key
381
+ that is not a durable variant is a type error. The rules ride in
382
+ `journalContract()`, so the server enforces them by spreading the contract.
730
383
 
731
- The client does not run authorization. It may optimistically show the edit;
732
- when the server refuses it, reconciliation drops that pending operation and
733
- rebases the remaining ones. This is why rejection is a normal exchange result,
734
- not a transport failure.
384
+ An edit the policy refuses still shows at once on the client, then comes back
385
+ as a rejection, as [above](#what-to-show-the-user).
735
386
 
736
- ## Persistence and recovery
387
+ ### Coalesce a burst of typing
737
388
 
738
- The local outbox is part of the feature, not an incidental cache. Losing it can
739
- lose edits that never reached the server.
740
-
741
- The built-in browser storage adapter is IndexedDB:
389
+ Typing makes one durable Message per keystroke. `coalesce(last, next)` is
390
+ offered each submit together with the last operation in the outbox; return a
391
+ merged Message to replace it, or `undefined` to keep them apart:
742
392
 
743
393
  ```ts
744
- const storage = yield* Sync.indexedDb('todos/tab-1')
394
+ coalesce: (last, next) =>
395
+ last._tag === 'RenamedTodo' && next._tag === 'RenamedTodo' && last.id === next.id
396
+ ? next
397
+ : undefined
745
398
  ```
746
399
 
747
- Stored replica state includes protocol/schema versions, document/replica
748
- identity, cursor, the server's epoch, committed snapshot, local sequence, and
749
- pending operations.
750
- Persisted and remote operations are decoded strictly against the application
751
- Message Schema.
752
-
753
- A submit writes only its operation. A `Storage` may implement the optional
754
- `append(entry, expectedRevision)`, which records one operation with the
755
- replica's next revision and local sequence. `load` then returns the saved state
756
- with those operations added to its outbox. The IndexedDB adapter keeps them in an
757
- `outbox` store beside the state, so a keystroke costs the size of the operation,
758
- not of the document. An exchange changes the committed snapshot, so it still
759
- saves the whole state, which clears the appended rows. A storage without
760
- `append` saves the whole state on every submit, as before.
761
-
762
- Important recovery cases:
763
-
764
- - **Storage absent or evicted.** The replica starts at cursor 0 and can catch up
765
- from the server, but edits that existed only in the lost outbox are gone.
766
- - **Two writers use one replica storage.** Compare-and-swap detects the stale
767
- writer. Give each tab/writer its own replica/storage identity.
768
- - **Replay refuses a Message.** `submit` fails and writes nothing, so invalid
769
- durable work never enters the outbox.
770
- - **Malformed stored data.** Opening fails rather than silently replacing bytes;
771
- the application can offer an explicit reset/recovery path.
772
- - **Unsupported storage version.** `UnsupportedReplicaVersionError` names the
773
- persisted and supported versions so migration/reset can be explicit.
774
- - **Rolling back past the outbox store.** The IndexedDB adapter upgrades its
775
- database to version 2, adding the `outbox` store. A release from before that
776
- opens it as version 1, which the browser refuses with a `VersionError`, so a
777
- rollback that far needs a new database name or a reset.
778
- - **Another tab blocks the upgrade.** Opening fails with a `StorageError`
779
- rather than waiting; a connection the upgrade opens later is closed.
780
- - **Server compaction.** A checkpoint replaces history the replica can no longer
781
- replay and pending local work is rebased on it.
782
- - **Server reset.** A server that answers with an `epoch` (Durable's
783
- `journal.epoch(key)`) names its history, and the replica stores it and sends
784
- it back as `exchange`'s third argument. A server that finds another epoch
785
- answers from sequence 0 of its own history, and should skip its
786
- cursor-ahead check for that request. The replica then rebuilds its committed
787
- state from that answer and keeps its outbox, which the same exchange delivers.
788
- Everything the old server committed is gone, on every replica; only what was
789
- still pending survives. The replica trusts a server that names a new epoch to
790
- have answered from its start; it does not check. A replica that has never
791
- heard an epoch takes the first one as its own, so a reset before that is not
792
- recognized: such a replica fails every exchange (its cursor is ahead of the
793
- server's) until its storage is cleared.
794
-
795
- Application Message/shared-state migrations remain application policy; Sync
796
- versioning protects its own persisted envelope.
797
-
798
- ## Transport
799
-
800
- `Transport` is an Effect service for the one exchange operation. Sync ships
801
- helpers under `Sync.transport`:
400
+ Replaying the merged Message must equal replaying `last` then `next`. An
401
+ operation is only merged into while no exchange has carried it, so there is no
402
+ timer to tune: keystrokes typed while a request is out become one operation, and
403
+ a lone keystroke goes out as fast as before.
802
404
 
803
- ```text
804
- Sync.transport.socket(...)
805
- Sync.transport.loopback(...)
806
- Sync.transport.fromPromise(...)
807
- Sync.transport.toPromise(...)
808
- Sync.transport.serve(...)
809
- Sync.transport.nativeSocket(...)
810
- ```
405
+ ### The URL, mirrors, and agents
811
406
 
812
- The socket transport reconnects for as long as its layer lives, on an
813
- exponential, jittered backoff capped at `maxRetryDelay` (5 s by default), and
814
- keeps request identities stable when resending queued or in-flight exchanges.
815
- After `maxRetries` consecutive failed connections (5 by default) queued work
816
- fails and new exchanges fail fast while no socket is open. The count resets
817
- only when a connection proves healthy, by answering a frame or by staying open
818
- for `maxRetryDelay`, so a server that accepts each socket and drops it is backed
819
- off like one that refuses it. The queue is bounded by `maxQueue`. A server rejection is protocol data; only a wire failure is
820
- a `TransportError`.
821
-
822
- The server can also tell a client that something changed. Pass `serve` a
823
- `changes` subscription, a function that takes a listener and returns its
824
- unsubscribe, and it sends the socket a `{ notify: true }` frame after each
825
- commit. The client's socket transport exposes these notices, and every
826
- (re)connection, as the `Transport`'s optional `changes` stream, which
827
- `replica.start` wakes on as it does on a submit. A notice carries no data: the
828
- exchange that follows reads from the replica's own cursor, so a lost or repeated
829
- notice is harmless, and a replica that is only reading still sees others' edits
830
- without polling. With Durable, the subscription is `journal.subscribe` filtered
831
- to the document's key; `examples/sync/src/server.ts` wires it.
832
-
833
- Transport is deliberately below reconciliation. A custom transport can carry
834
- the same exchange without changing replica semantics.
835
-
836
- ## Presence: ephemeral collaboration state
837
-
838
- Presence is intentionally **not** a durable Message log. “Alice is viewing this
839
- page” or a cursor position should expire when the peer disappears, not replay
840
- next week.
841
-
842
- `Sync.presence.make` creates a TTL'd peer registry. Values are decoded before
843
- they enter the registry; a peer that stops refreshing is removed. Changes are
844
- available both as a callback subscription and as a Stream. A caret sets a new
845
- value on nearly every keystroke; with `throttle: '50 millis'` a value set after
846
- a quiet interval is sent at once, and those set within the interval wait, only
847
- the latest of them sent when it ends. The peer's own entry changes at once. A departure (`leave`) is sent immediately
848
- and cancels a value still waiting.
849
-
850
- Presence can share the socket transport's connection. `transport.socket` is a
851
- stable handle to it: a send goes to the socket that is open now and is dropped
852
- while none is, and messages from every socket the transport opens arrive
853
- through it. So `Sync.presence.socketChannel(transport.socket)` survives
854
- reconnects without being bound again. After a reconnect the server has
855
- forgotten the peer's value, so announce it again within the time to live.
856
-
857
- Presence can travel through:
407
+ `mount` takes the application's own `subscriptions` and the `resources` Layer
408
+ they need, and `url: { init, onUrlChange, onUrlRequest? }` to route navigation
409
+ through `update`: `init(model, url)` reduces the first URL before the first
410
+ render and `onUrlChange(url)` names the Message for every navigation. A
411
+ [`foldkit-mirror`](../mirror) URL mirror plugs into exactly that seam.
858
412
 
859
- ```text
860
- Sync.presence.socketChannel(...)
861
- Sync.presence.loopbackChannel(...)
862
- ```
413
+ `Mounted` is also the host an agent binds to: `model`, `dispatch`, `subscribe`,
414
+ and `observe`, which reports every application Message the runtime applies.
415
+ See [`examples/todo-app`](../../examples/todo-app) for both.
863
416
 
864
- with server fan-out through:
417
+ ### Carry local state across a reinstall
865
418
 
866
- ```text
867
- Sync.presence.hub(...)
868
- Sync.presence.serve(...)
869
- ```
419
+ When an exchange or a failed persist replaces the shared slice outside
420
+ `update`, `onReinstall(next, previous)` returns the transition instead. Use it
421
+ to keep a selection that points into the slice, or to return a Command that
422
+ patches a DOM the change has to reach. Omitted, the Model is simply `next`. It
423
+ is not called for a status that changed nothing the Model shows.
870
424
 
871
- The registry uses Effect's `Clock`, so `TestClock` can make TTL behavior
872
- deterministic in tests.
425
+ ## Testing without a browser or a server
873
426
 
874
- The conceptual rule is simple:
875
-
876
- ```text
877
- shared application fact that must replay later -> durable Message / Sync
878
- peer status that should disappear when stale -> Presence
879
- ```
880
-
881
- ## Last-writer-wins fields (experimental)
882
-
883
- Server arrival order is the default conflict rule: whichever operation the
884
- server commits first is replayed first. Sometimes one field needs a different
885
- rule — for example, the newest logical title write should win even if an offline
886
- device reconnects later.
887
-
888
- `Sync.lww.register` supplies a schema and pure merge rule for that specialized
889
- case. It is a field-level helper, **not a general CRDT mode for arbitrary
890
- Messages**.
427
+ A `Storage` needs three members, and the loopback transport takes a handler in
428
+ place of a socket:
891
429
 
892
430
  ```ts
893
- import { Schema } from 'effect'
894
- import { ReplicaId, Sync } from 'foldkit-sync'
895
-
896
- const Title = Sync.lww.register(Schema.String)
897
- const Shared = Schema.Struct({ title: Title.schema })
898
- const Renamed = Schema.Struct({
899
- _tag: Schema.Literal('Renamed'),
900
- title: Title.schema,
901
- })
902
-
903
- const update = (
904
- model: typeof Shared.Type,
905
- message: typeof Renamed.Type,
906
- ) => ({
907
- ...model,
908
- title: Title.merge(model.title, message.title),
909
- })
910
-
911
- const message: typeof Renamed.Type = {
912
- _tag: 'Renamed',
913
- title: {
914
- stamp: { counter: 1, replicaId: ReplicaId.make('tab-a') },
915
- value: 'Milk',
916
- },
431
+ import { Effect } from 'effect'
432
+ import type { Storage } from 'foldkit-sync'
433
+
434
+ const memoryStorage = (): Storage => {
435
+ let state: unknown
436
+ return {
437
+ load: () => Effect.sync(() => state),
438
+ save: next => Effect.sync(() => void (state = structuredClone(next))),
439
+ close: Effect.void,
440
+ }
917
441
  }
918
- ```
919
442
 
920
- Higher `counter` wins; equal counters use lexicographic UTF-16 `replicaId`
921
- order, independent of locale. This is logical ordering, not wall-clock time.
922
- The rule follows the total-order register model described in
923
- [Replicated Data Types: Specification, Verification, Optimality](https://www.microsoft.com/en-us/research/publication/replicated-data-types-specification-verification-optimality/).
443
+ const program = Effect.gen(function* () {
444
+ const replica = yield* TodoSync.openReplica(ReplicaId.make('test'), memoryStorage())
445
+ yield* replica.submit(Message.CreatedTodo({ id: 't1', title: 'Milk' }))
446
+ yield* replica.synchronize
447
+ return yield* replica.status
448
+ }).pipe(
449
+ Effect.provide(
450
+ Sync.transport.loopback((_cursor, pending) => ({
451
+ operations: [],
452
+ acknowledged: pending.map(operation => operation.opId),
453
+ rejected: [],
454
+ })),
455
+ ),
456
+ )
457
+ ```
924
458
 
925
- Allocate stamps **before dispatch**, never inside `update` or replay.
926
- `Sync.lww.openClock` persists the high-water mark independently of the outbox,
927
- so rejected or never-submitted writes cannot cause timestamp reuse after reload:
459
+ In Node, `Sync.indexedDb(name, new IDBFactory())` from `fake-indexeddb` tests
460
+ the real storage adapter. [`examples/sync`](../../examples/sync) tests every
461
+ recovery case below against a real journal.
462
+
463
+ ## When things go wrong
464
+
465
+ Each case is designed, not incidental. The ones that change how you build the
466
+ application come first.
467
+
468
+ - **The network is gone.** Edits keep applying and keep being saved. `pending`
469
+ grows; the loop retries. Nothing to handle.
470
+ - **The server rejects an edit.** Dropped from `pending`, the rest replayed,
471
+ the Model re-installed. Explain it from `status.rejected`.
472
+ - **The persist fails.** The edit is reverted and `onPersistenceFailure` runs.
473
+ The user has to redo it.
474
+ - **Local storage was evicted.** The replica starts at cursor 0 and catches up
475
+ from the server, but edits that lived only in the outbox are gone. The
476
+ outbox is user data, not a cache.
477
+ - **Two tabs share one storage name.** Compare-and-swap fails the stale writer
478
+ with `StorageError`. Give each tab its own name and `ReplicaId`.
479
+ - **The replica is far behind.** The server may answer in pages (`more`), and
480
+ `synchronize` asks again at once until caught up. If the server compacted
481
+ the history it needs, it sends a checkpoint instead: a snapshot and cursor
482
+ the replica adopts as its base before replaying its pending edits. A
483
+ checkpoint behind the replica's cursor is refused
484
+ (`CheckpointRegressionError`), never applied backwards.
485
+ - **The server lost its history.** A server names its history with an `epoch`;
486
+ a replica that hears a new one rebuilds its committed state from that answer
487
+ and resends its outbox. Everything the old server committed is gone on every
488
+ replica. A replica that never heard an epoch cannot tell, and fails every
489
+ exchange until its storage is cleared.
490
+ - **Stored state is from another version.** Opening fails with
491
+ `UnsupportedReplicaVersionError` naming both versions; the bytes are left for
492
+ a deliberate migration or reset. Sync versions its own envelope only; your
493
+ Message and snapshot migrations stay yours.
494
+ - **The server answers nonsense.** Malformed exchanges, gaps in the committed
495
+ order, acknowledgements or rejections for operations the replica never sent:
496
+ each fails the exchange rather than touch local work.
497
+
498
+ ## Advanced: the replica
499
+
500
+ `Sync.mount` is the normal path. Underneath, a `Replica` works on the shared
501
+ slice alone, and an application that is not a Foldkit runtime (a worker, a
502
+ server-side actor, a test) uses it directly.
928
503
 
929
504
  ```ts
930
- import { Effect } from 'effect'
931
- import { DocumentId, ReplicaId, Sync, type Replica } from 'foldkit-sync'
932
-
933
- type Shared = typeof Shared.Type
934
- type Renamed = typeof Renamed.Type
935
-
936
- const rename = (replica: Replica<Renamed, Shared>, title: string) =>
937
- Effect.gen(function* () {
938
- const storage = yield* Sync.indexedDb('todos-tab-a-clock')
939
- const clock = yield* Sync.lww.openClock({
940
- documentId: DocumentId.make('todos'),
941
- replicaId: ReplicaId.make('tab-a'),
942
- storage,
943
- })
944
-
945
- const shared = yield* replica.shared
946
- const stamp = yield* clock.next(shared.title.stamp.counter)
947
-
948
- yield* replica.submit({
949
- _tag: 'Renamed',
950
- title: { stamp, value: title },
951
- })
952
-
953
- yield* clock.close
954
- }).pipe(Effect.scoped)
955
- ```
956
-
957
- `next(observedCounter)` persists a counter greater than both its previous saved
958
- value and the supplied observation before returning the stamp. Pass the latest
959
- observed counter for the field being edited; if causality spans several
960
- registers, pass their maximum. Calling `next()` without an observation advances
961
- only the saved local counter.
505
+ const program = Effect.gen(function* () {
506
+ const storage = yield* Sync.indexedDb('todos/tab-1')
507
+ const replica = yield* TodoSync.openReplica(ReplicaId.make('tab-1'), storage)
962
508
 
963
- Allocations on one handle are serialized. Separate handles use storage's
964
- compare-and-swap check; a stale handle fails and should be closed/reopened.
965
- Storage failures do not return a stamp. Counter gaps after a crash or rejected
966
- submission are harmless.
509
+ yield* replica.submit(Message.CreatedTodo({ id: 't1', title: 'Milk' }))
510
+ const optimistic = yield* replica.shared // includes the pending edit
967
511
 
968
- Keep the clock storage across reloads and outbox resets. If it is lost, use a
969
- fresh `ReplicaId` rather than reusing the same writer identity from counter zero.
512
+ yield* replica.synchronize // one exchange
513
+ return optimistic
514
+ }).pipe(Effect.scoped, Effect.provide(Sync.transport.socket({ url: 'wss://example.com/sync' })))
515
+ ```
970
516
 
971
- A stamp identifies one immutable write within a register. Repeated delivery of
972
- the same value is harmless; a different value with the same stamp throws. Keep
973
- the **whole register, including its winning stamp**, in shared snapshots. For
974
- deletion, a nullable register can retain a stamped `null` tombstone so an old
975
- offline write cannot resurrect deleted state.
517
+ `submit(message)` checks that the Message is durable, replays it against the
518
+ current optimistic state, gives it a stable operation id, saves it to the
519
+ outbox, publishes the new optimistic value, and wakes the loop. It sends
520
+ nothing. A Message replay refuses never enters the outbox.
976
521
 
977
- The server still orders and authorizes every operation, including writes that
978
- lose the LWW merge. Replica ids and counters carried in Messages are
979
- client-authored data, not authenticated identity; enforce writer/clock policy at
980
- admission when needed.
522
+ `synchronize` takes one snapshot of the cursor and outbox, performs one
523
+ exchange, validates the answer, and reconciles it with the replica's *current*
524
+ state, so an edit submitted while the request was in flight survives.
981
525
 
982
- Specialized sets, counters, collaborative text, and arbitrary commutative
983
- Messages remain outside this helper.
526
+ The rest of the surface: `committed` (the base alone), `pending`, `cursor`,
527
+ `status`, `statusChanges`, `changes` (status and `shared` from one snapshot),
528
+ `snapshot` (the same plus `nextLocalSequence`), `start`, and `close`.
984
529
 
985
- ## Lower level: `Sync.define`
530
+ ### The exchange
986
531
 
987
- `Sync.forApplication(App).make(...)` is the Foldkit-facing compiler. It builds
988
- on `Sync.define`, the lower-level protocol primitive.
532
+ One request, one answer. A custom server implements this; a custom transport
533
+ carries it.
989
534
 
990
- Use `Sync.define` directly when there is no Foldkit application to derive the
991
- contract from — for example, a non-Foldkit client or a hand-written shared
992
- projection:
535
+ ```text
536
+ request cursor
537
+ pending operations
538
+ epoch, once the replica has heard one
539
+
540
+ answer operations committed after the cursor, contiguous
541
+ acknowledged ids the server accepted without repeating them
542
+ rejected ids the server refused
543
+ checkpoint? { cursor, model } when the history is gone
544
+ more? true when another page follows
545
+ epoch? the server's history
546
+ ```
547
+
548
+ An acknowledged operation leaves the outbox once its committed copy has
549
+ arrived or the last page has been read. A rejected one leaves at once. The
550
+ replica validates every field; see [when things go wrong](#when-things-go-wrong).
551
+
552
+ ### Transports
553
+
554
+ `Transport` is an Effect service with one operation. Sync ships:
555
+
556
+ - `Sync.transport.socket({ url, … })`: a reconnecting WebSocket client. It
557
+ queues exchanges while connecting, retries on a jittered backoff capped at
558
+ `maxRetryDelay` (5 s), keeps request ids stable across resends, and after
559
+ `maxRetries` (5) consecutive failed connections fails queued work and fails
560
+ new exchanges fast until a socket is healthy again. `maxQueue` (64) bounds
561
+ the queue.
562
+ - `Sync.transport.serve(socket, { exchange, changes? })`: the server side of one
563
+ accepted socket. With `changes`, a subscription to the document's commits, it
564
+ sends a notice after each; the client's loop exchanges on every notice, so a
565
+ tab that is only reading still sees others' edits without polling.
566
+ - `Sync.transport.loopback(handler)`, `fromPromise(client)`, `toPromise(shape)`,
567
+ and `nativeSocket` for the platform `WebSocket`.
568
+
569
+ A refusal is data in the answer; only a wire failure is a `TransportError`.
570
+
571
+ ### Presence
572
+
573
+ "Who is viewing this" and "where is their caret" should vanish when the peer
574
+ does, not replay next week. `Sync.presence.make({ id, ttl, decodeValue, channel?,
575
+ throttle? })` is a TTL'd peer registry outside the log: values are decoded before
576
+ they enter it, a peer that stops refreshing is pruned, and `throttle` sends at
577
+ most one value per interval, the latest. `Sync.presence.socketChannel(transport.socket)`
578
+ shares the sync socket across reconnects; `Sync.presence.hub` and `.serve` are
579
+ the server side. [`examples/pages`](../../examples/pages) shows remote carets.
580
+
581
+ ### Last-writer-wins fields (experimental)
582
+
583
+ Server order decides concurrent edits. When one field should instead keep the
584
+ newest logical write even from a device that reconnects late, `Sync.lww.register(schema)`
585
+ gives it a stamped schema and a pure `merge`; `Sync.lww.openClock` allocates
586
+ stamps from a persisted counter, before dispatch and never in `update`. It is a
587
+ per-field rule, not a CRDT mode: sets, counters, and text stay outside.
993
588
 
994
589
  ```ts
995
- import { Effect, Schema } from 'effect'
996
- import {
997
- DocumentId,
998
- ReplicaId,
999
- Sync,
1000
- type TransportClient,
1001
- } from 'foldkit-sync'
1002
-
1003
- const Shared = Schema.Struct({ todos: Schema.Array(Schema.String) })
1004
- const Renamed = Schema.Struct({
1005
- _tag: Schema.Literal('Renamed'),
1006
- title: Schema.String,
1007
- })
590
+ const Title = Sync.lww.register(Schema.String)
591
+ const Shared = Schema.Struct({ title: Title.schema })
592
+ // in update: title: Title.merge(model.title, message.title)
593
+ ```
1008
594
 
1009
- declare const transport: TransportClient
595
+ ### Without a Foldkit application
1010
596
 
1011
- const Protocol = Sync.define({
1012
- documentId: DocumentId.make('todos'),
1013
- message: Renamed,
1014
- shared: Shared,
1015
- empty: { todos: [] },
1016
- durable: () => true,
1017
- replay: (shared, message) => ({
1018
- todos: [...shared.todos, message.title],
1019
- }),
1020
- })
597
+ `Sync.define({ documentId, message, shared, empty, durable, replay })` is the
598
+ protocol primitive `forApplication` compiles to, for a client with no
599
+ application to derive from. Its `replay` is unguarded; the author owns its
600
+ agreement with whatever applies the Message elsewhere.
1021
601
 
1022
- const program = Effect.gen(function* () {
1023
- const storage = yield* Sync.indexedDb('todos-tab-1')
1024
- const replica = yield* Protocol.openReplica(
1025
- ReplicaId.make('tab-1'),
1026
- storage,
1027
- )
602
+ ### Joining an assembly
1028
603
 
1029
- yield* replica.submit({ _tag: 'Renamed', title: 'Milk' })
604
+ `TodoSync.wiring()` adds the contract to a `foldkit-bundle` assembly so
605
+ `Module.validate` can report two owners of one field. It routes nothing and
606
+ runs nothing; Sync owns its runtime through `mount`.
1030
607
 
1031
- yield* Effect.provide(
1032
- replica.synchronize,
1033
- Sync.transport.fromPromise(transport),
1034
- )
608
+ ## What Sync owns, and does not
1035
609
 
1036
- return yield* replica.shared
1037
- }).pipe(Effect.scoped)
1038
- ```
610
+ Owns the replication contract (shared slice, durable set, replay, codecs,
611
+ `journalContract`), the replica (outbox, base, cursor, optimistic view, status),
612
+ reconciliation, the transport seam, and the optional presence and LWW helpers.
1039
613
 
1040
- Most Foldkit applications should prefer `Sync.forApplication`: it keeps the
1041
- shared schema and replay semantics derived from the application rather than
1042
- re-declaring them.
1043
-
1044
- ## What Sync owns
1045
-
1046
- - **The replication contract.** Shared projection, durable Message subset,
1047
- initial shared value, replay, operation codecs, and `journalContract()`.
1048
- - **The replica.** Persisted outbox, committed base, cursor, optimistic shared
1049
- projection, local operation sequence, synchronization, status, and recovery
1050
- validation.
1051
- - **Reconciliation.** Adopt committed operations/checkpoints, drop
1052
- acknowledged/rejected pending operations, and replay what remains.
1053
- - **The transport seam.** Effect `Transport` plus loopback/promise/WebSocket
1054
- adapters; transport carries exchanges but does not own conflict semantics.
1055
- - **Optional collaboration primitives.** Ephemeral Presence and the specialized
1056
- LWW register/clock helpers.
1057
-
1058
- Sync does **not** own:
1059
-
1060
- - unrelated local Model fields;
1061
- - server-derived disposable caches (`foldkit-remote`);
1062
- - authoritative server storage/order (`foldkit-durable` or your own server);
1063
- - peer-to-peer CRDT convergence;
1064
- - application Message/shared-state migrations.
1065
-
1066
- ## How state changes here
1067
-
1068
- Durable Messages replay the normal application `update` — which is where
1069
- `modifyFields` lives — so deterministic replay and optimistic application are the
1070
- same code path, not two reducers. Checkpoints and adopted server state
1071
- arrive the other way: structurally installed through the declared writable
1072
- projection. Sync already verifies that durable Messages touch only the
1073
- declared projection; treat any other write as a bug.
614
+ Does not own the rest of the Model, server-derived caches (`foldkit-remote`),
615
+ the authoritative order and storage (`foldkit-durable` or your server),
616
+ peer-to-peer merging, or migration of your own Messages and snapshots.
1074
617
 
1075
618
  ## Limits
1076
619
 
1077
- - IndexedDB is the built-in storage adapter today.
1078
- - Sync assumes an authoritative server order for each document; it is not a
1079
- peer-to-peer replication protocol.
1080
- - Ordinary concurrent writes resolve by that server order. `Sync.lww.register`
1081
- changes the merge rule for individual fields only; it does not make arbitrary
1082
- Messages commutative.
1083
- - There is no operational transform / collaborative text algorithm.
1084
- - Binding the socket transport and presence server to a platform WebSocket
1085
- server remains application/platform glue. The sync example demonstrates a
1086
- `ws` transport binding. `examples/pages` serves presence over the same socket,
1087
- and shows remote carets.
1088
- - Application schema migration remains application policy. Sync versions and
1089
- validates its own persisted/wire envelope.
620
+ - IndexedDB is the only built-in storage; `Storage` is three members to
621
+ implement.
622
+ - One server order per document; no CRDT and no collaborative text algorithm.
623
+ - Binding the socket transport to a platform WebSocket server is glue the
624
+ examples show with `ws`.
1090
625
 
1091
626
  ## Older spellings
1092
627
 
1093
- `Sync` groups the package exports; it does not wrap them. Existing top-level
1094
- spellings remain available with the same signatures:
1095
-
1096
- - `forApplication`, `mount`, `indexedDb` — same names on the namespace;
1097
- - `defineSync` -> `Sync.define`; `syncMetrics` -> `Sync.metrics`;
1098
- - `layerSocket`, `layerLoopback`, `layerFromPromise`, `toPromise`, `serveSocket`,
1099
- `nativeSocket` -> `Sync.transport.socket`, `.loopback`, `.fromPromise`,
1100
- `.toPromise`, `.serve`, `.nativeSocket`;
1101
- - `createPresence`, `createPresenceHub`, `servePresence`,
1102
- `socketPresenceChannel`, `loopbackPresenceChannel` -> `Sync.presence.make`,
1103
- `.hub`, `.serve`, `.socketChannel`, `.loopbackChannel`;
1104
- - `lwwRegister`, `openLwwClock` -> `Sync.lww.register`, `Sync.lww.openClock`;
1105
- - `documentId('todos')` and `DocumentId.make('todos')` produce the same branded
1106
- value, and likewise for `replicaId`, `opId`, `sequence`, and
1107
- `localSequence`.
628
+ `Sync` groups the exports and wraps nothing; every name is also exported on its
629
+ own with the same signature: `defineSync` is `Sync.define`, `syncMetrics` is
630
+ `Sync.metrics`, `layerSocket`/`layerLoopback`/`layerFromPromise`/`toPromise`/
631
+ `serveSocket`/`nativeSocket` are `Sync.transport.*`, `createPresence`/
632
+ `createPresenceHub`/`servePresence`/`socketPresenceChannel`/
633
+ `loopbackPresenceChannel` are `Sync.presence.*`, and `lwwRegister`/`openLwwClock`
634
+ are `Sync.lww.*`. `documentId('todos')` and `DocumentId.make('todos')` make the
635
+ same branded value, as do the other id constructors.
1108
636
 
1109
637
  ## See also
1110
638
 
1111
- - [Replicated state](../../docs/replication.md) — Sync and Durable together,
1112
- including the server exchange and recovery model.
1113
- - [Runtime binding](../../docs/sync-runtime-binding.md) — what `Sync.mount`
1114
- guarantees and why no Foldkit core change is required.
1115
- - [`foldkit-durable`](../durable) — the authoritative server journal commonly
1116
- paired with Sync.
1117
- - [`foldkit-remote`](../remote) — server-owned disposable state rather than
1118
- client-authored durable state.
1119
- - [`examples/todo-app`](../../examples/todo-app) — browser mount, fragments,
1120
- authorization, Mirror, and agent integration.
1121
- - [`examples/sync`](../../examples/sync) — focused replica/server/transport trace.
639
+ - [Replicated state](../../docs/replication.md): both halves end to end, with
640
+ the server's exchange handler.
641
+ - [`foldkit-durable`](../durable): the journal and what it guarantees.
642
+ - [Runtime binding](../../docs/sync-runtime-binding.md): what `Sync.mount` does
643
+ inside the Foldkit runtime, for the curious.
644
+ - [`examples/todo-app`](../../examples/todo-app): a whole application over
645
+ `Sync.mount`, with fragments, authorization, mirrors, and an agent.
646
+ - [`examples/sync`](../../examples/sync): the replica, the journal, the socket,
647
+ and a test for every failure above.