foldkit-sync 0.6.0 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +423 -786
- package/dist/index.d.mts +148 -20
- package/dist/index.d.mts.map +1 -1
- package/dist/index.mjs +458 -160
- package/dist/index.mjs.map +1 -1
- package/package.json +9 -8
package/README.md
CHANGED
|
@@ -2,103 +2,74 @@
|
|
|
2
2
|
|
|
3
3
|
Local-first replication for part of a Foldkit application.
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
`Message`, and `update` remain the state machine. Sync does not introduce a
|
|
12
|
-
second reducer. You declare which Model slice is shared and which existing
|
|
13
|
-
Messages are durable; Sync derives replay, persistence, protocol codecs, and the
|
|
14
|
-
server journal contract from that declaration.
|
|
15
|
-
|
|
16
|
-
> **Sync turns a subset of your existing Foldkit Messages into durable
|
|
17
|
-
> operations. They still run through your normal `update`; Sync adds persistence,
|
|
18
|
-
> optimistic replay, server ordering, and reconciliation around them.**
|
|
19
|
-
|
|
20
|
-
**Use it when** a user should keep editing offline, the edit must survive a
|
|
21
|
-
reload or network failure, and several replicas must eventually converge under
|
|
22
|
-
one authoritative server order. **Do not use it for** server-owned disposable
|
|
23
|
-
facts — use [`foldkit-remote`](../remote) for those — or for peer-to-peer / CRDT
|
|
24
|
-
replication where every peer independently accepts writes.
|
|
5
|
+
You name a slice of the Model that several devices share, and the existing
|
|
6
|
+
Messages that change it. From then on an edit applies at once, is saved in a
|
|
7
|
+
local outbox, reaches the server when the network allows, and lands in the same
|
|
8
|
+
order on every device. Nothing about the application changes shape: `Model`,
|
|
9
|
+
`Message`, and `update` stay the one state machine. Sync wraps them; it never
|
|
10
|
+
adds a second reducer.
|
|
25
11
|
|
|
26
|
-
|
|
27
|
-
|
|
12
|
+
> **A durable Message is one of your existing Messages, recorded.** It still
|
|
13
|
+
> runs through your `update`. Sync adds the outbox, the optimistic view, the
|
|
14
|
+
> exchange with the server, and the rebase when the server's answer arrives.
|
|
28
15
|
|
|
29
|
-
|
|
16
|
+
It is the client half of [replicated Foldkit state](../../docs/replication.md).
|
|
17
|
+
[`foldkit-durable`](../durable) is the server half, an ordered journal on
|
|
18
|
+
SQLite. Either works without the other.
|
|
30
19
|
|
|
31
|
-
|
|
20
|
+
## Is this the right package?
|
|
32
21
|
|
|
33
|
-
|
|
|
34
|
-
| --- | --- |
|
|
35
|
-
|
|
|
36
|
-
|
|
|
37
|
-
|
|
|
38
|
-
|
|
|
22
|
+
| The state | Owner | Use |
|
|
23
|
+
| --- | --- | --- |
|
|
24
|
+
| A route, a selection, a transient error | the local Model | plain Foldkit |
|
|
25
|
+
| Facts the server owns and the client may refetch | the server | [`foldkit-remote`](../remote) |
|
|
26
|
+
| Edits the user makes that must survive offline and converge | the durable log | **`foldkit-sync`** |
|
|
27
|
+
| The authoritative order and snapshot on the server | the server journal | [`foldkit-durable`](../durable) |
|
|
39
28
|
|
|
40
|
-
The
|
|
29
|
+
The Remote and Sync line is the one worth memorizing. Remote caches a fact the
|
|
30
|
+
server owns, so refetching is recovery. Sync records an edit the user authored,
|
|
31
|
+
so the outbox is user data and replay is recovery.
|
|
41
32
|
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
server owns the fact
|
|
45
|
-
client caches it
|
|
46
|
-
refetch is recovery
|
|
47
|
-
|
|
48
|
-
Sync
|
|
49
|
-
client creates the durable operation
|
|
50
|
-
operation must survive offline
|
|
51
|
-
replay + reconciliation is recovery
|
|
52
|
-
```
|
|
33
|
+
Sync assumes one server order per document. It is not peer-to-peer and not a
|
|
34
|
+
CRDT; every peer accepting writes without a server is a different design.
|
|
53
35
|
|
|
54
|
-
|
|
55
|
-
time. Those observation boundaries do not change who owns each datum.
|
|
56
|
-
|
|
57
|
-
## The replica mental model
|
|
58
|
-
|
|
59
|
-
A replica has two pieces of state:
|
|
36
|
+
## The model in one equation
|
|
60
37
|
|
|
61
38
|
```text
|
|
62
|
-
|
|
63
|
-
+
|
|
64
|
-
pending local operations
|
|
65
|
-
=
|
|
66
|
-
optimistic shared state the UI sees
|
|
39
|
+
what the user sees = committed snapshot + pending local edits, replayed
|
|
67
40
|
```
|
|
68
41
|
|
|
69
|
-
|
|
42
|
+
The server has committed `A B C`; this device has typed `D E` since:
|
|
70
43
|
|
|
71
44
|
```text
|
|
72
|
-
server
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
visible state = replay(A, B, C, D, E)
|
|
45
|
+
server: A B C
|
|
46
|
+
pending: D E
|
|
47
|
+
visible: replay(A, B, C, D, E)
|
|
76
48
|
```
|
|
77
49
|
|
|
78
|
-
|
|
79
|
-
|
|
50
|
+
Another device gets `X` in before `D` reaches the server. After the next
|
|
51
|
+
exchange:
|
|
80
52
|
|
|
81
53
|
```text
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
new visible state = replay(A, B, C, X, D, E)
|
|
54
|
+
server: A B C X
|
|
55
|
+
pending: D E
|
|
56
|
+
visible: replay(A, B, C, X, D, E)
|
|
86
57
|
```
|
|
87
58
|
|
|
88
|
-
|
|
89
|
-
|
|
59
|
+
`D` and `E` never waited. The base moved underneath them and they were replayed
|
|
60
|
+
on top. That one move, **rebase**, is also how a rejection rolls back: drop the
|
|
61
|
+
rejected edit from `pending` and replay the rest.
|
|
90
62
|
|
|
91
|
-
|
|
63
|
+
An edit passes three milestones, and each means something different:
|
|
92
64
|
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
65
|
+
| Milestone | Where | What it proves |
|
|
66
|
+
| --- | --- | --- |
|
|
67
|
+
| **visible** | the Model, at once | nothing yet: the user saw it |
|
|
68
|
+
| **saved** | the local outbox | a reload or a dropped connection cannot lose it |
|
|
69
|
+
| **committed** | the server journal | every other device will see it in this order |
|
|
98
70
|
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
replica status when deciding what the UI may call saved.
|
|
71
|
+
A UI that says "saved" should read the second, and one that says "shared"
|
|
72
|
+
the third. Both are observable; see [what to show the user](#what-to-show-the-user).
|
|
102
73
|
|
|
103
74
|
## Install
|
|
104
75
|
|
|
@@ -107,23 +78,25 @@ pnpm add foldkit-sync
|
|
|
107
78
|
```
|
|
108
79
|
|
|
109
80
|
`foldkit` and `effect` are peer dependencies; `foldkit-surface` comes with the
|
|
110
|
-
package.
|
|
81
|
+
package. The browser storage is IndexedDB. The server needs
|
|
82
|
+
[`foldkit-durable`](../durable) or your own implementation of
|
|
83
|
+
[the exchange](#the-exchange).
|
|
111
84
|
|
|
112
|
-
## Sixty seconds:
|
|
85
|
+
## Sixty seconds: say what is shared
|
|
113
86
|
|
|
114
|
-
Start
|
|
87
|
+
Start from an ordinary application. The example keeps a todo list and a
|
|
88
|
+
selection; only the list is shared.
|
|
115
89
|
|
|
116
90
|
```ts
|
|
117
91
|
import { Option, Schema } from 'effect'
|
|
118
92
|
import { defineMessageUnion } from 'foldkit/message'
|
|
93
|
+
import { modifyFields } from 'foldkit/struct'
|
|
119
94
|
import type * as Update from 'foldkit/update'
|
|
120
95
|
import { MessageSet, Projection, Surface } from 'foldkit-surface'
|
|
121
96
|
import { DocumentId, Sync } from 'foldkit-sync'
|
|
122
97
|
|
|
123
98
|
const Model = Schema.Struct({
|
|
124
|
-
todos: Schema.Array(
|
|
125
|
-
Schema.Struct({ id: Schema.String, title: Schema.String }),
|
|
126
|
-
),
|
|
99
|
+
todos: Schema.Array(Schema.Struct({ id: Schema.String, title: Schema.String })),
|
|
127
100
|
selectedTodoId: Schema.Option(Schema.String),
|
|
128
101
|
lastError: Schema.Option(Schema.String),
|
|
129
102
|
})
|
|
@@ -136,30 +109,22 @@ const Message = defineMessageUnion({
|
|
|
136
109
|
})
|
|
137
110
|
type Message = typeof Message.Type
|
|
138
111
|
|
|
139
|
-
const initial: Model = {
|
|
140
|
-
todos: [],
|
|
141
|
-
selectedTodoId: Option.none(),
|
|
142
|
-
lastError: Option.none(),
|
|
143
|
-
}
|
|
112
|
+
const initial: Model = { todos: [], selectedTodoId: Option.none(), lastError: Option.none() }
|
|
144
113
|
|
|
145
114
|
type Return = Update.Return<Model, Message>
|
|
146
115
|
|
|
147
|
-
// This is still the application's one transition function.
|
|
148
116
|
const update = (model: Model, message: Message): Return =>
|
|
149
117
|
Message.match<Return>(message, {
|
|
150
118
|
CreatedTodo: ({ id, title }) => ({
|
|
151
|
-
model:
|
|
119
|
+
model: modifyFields(model, { todos: () => [...model.todos, { id, title }] }),
|
|
152
120
|
}),
|
|
153
121
|
RenamedTodo: ({ id, title }) => ({
|
|
154
|
-
model: {
|
|
155
|
-
...
|
|
156
|
-
|
|
157
|
-
todo.id === id ? { ...todo, title } : todo,
|
|
158
|
-
),
|
|
159
|
-
},
|
|
122
|
+
model: modifyFields(model, {
|
|
123
|
+
todos: () => model.todos.map(todo => (todo.id === id ? { ...todo, title } : todo)),
|
|
124
|
+
}),
|
|
160
125
|
}),
|
|
161
126
|
SelectedTodo: ({ id }) => ({
|
|
162
|
-
model:
|
|
127
|
+
model: modifyFields(model, { selectedTodoId: () => Option.some(id) }),
|
|
163
128
|
}),
|
|
164
129
|
})
|
|
165
130
|
|
|
@@ -167,438 +132,190 @@ const App = Surface.application({ Model, Message, initial, update })
|
|
|
167
132
|
|
|
168
133
|
const TodoSync = Sync.forApplication(App).make({
|
|
169
134
|
documentId: DocumentId.make('todos'),
|
|
170
|
-
|
|
171
|
-
// This slice of Model is replicated.
|
|
172
|
-
// The writable Projection lets Sync install checkpoints/reconciled state back
|
|
173
|
-
// into the application Model without owning a second copy of the schema.
|
|
174
135
|
shared: Projection.pick(App.model.todos),
|
|
175
|
-
|
|
176
|
-
// Only these existing Messages become durable operations.
|
|
177
|
-
durable: MessageSet.make(App, [
|
|
178
|
-
Message.CreatedTodo,
|
|
179
|
-
Message.RenamedTodo,
|
|
180
|
-
]),
|
|
136
|
+
durable: MessageSet.make(App, [Message.CreatedTodo, Message.RenamedTodo]),
|
|
181
137
|
})
|
|
182
138
|
```
|
|
183
139
|
|
|
184
|
-
Read
|
|
140
|
+
Read the declaration as a table:
|
|
185
141
|
|
|
186
142
|
```text
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
lastError -> local only
|
|
191
|
-
|
|
192
|
-
Messages
|
|
193
|
-
CreatedTodo -> durable
|
|
194
|
-
RenamedTodo -> durable
|
|
195
|
-
SelectedTodo -> local only
|
|
196
|
-
```
|
|
197
|
-
|
|
198
|
-
`Sync.forApplication(App)` derives the shared schema, initial shared snapshot,
|
|
199
|
-
replay function, operation codecs, and server journal contract from the same
|
|
200
|
-
application declaration. There is no parallel sync-specific version of
|
|
201
|
-
`update` to keep aligned.
|
|
202
|
-
|
|
203
|
-
### Declaration is not a running replica
|
|
204
|
-
|
|
205
|
-
The example above defines codecs and replay rules. It has not opened storage,
|
|
206
|
-
created an outbox, connected to a server, or mounted a view. For the application
|
|
207
|
-
path, continue with [mounting](#mounting-the-full-foldkit-application). Use
|
|
208
|
-
[the replica API](#running-a-replica) when you need to control that lifecycle.
|
|
209
|
-
|
|
210
|
-
`SelectedTodo` stays local because it is outside the durable MessageSet.
|
|
211
|
-
Adding a Message to that set is a semantic decision: its update must be safe
|
|
212
|
-
to replay and confined to the shared projection. An intent that starts a
|
|
213
|
-
Command should instead produce a deterministic fact, as described next.
|
|
214
|
-
|
|
215
|
-
## Wiring: the contract joins an assembly
|
|
143
|
+
todos shared replicated, persisted, journaled
|
|
144
|
+
selectedTodoId local this device only
|
|
145
|
+
lastError local
|
|
216
146
|
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
```ts
|
|
221
|
-
const wiring = TodoSync.wiring()
|
|
147
|
+
CreatedTodo durable recorded in the outbox, committed by the server
|
|
148
|
+
RenamedTodo durable
|
|
149
|
+
SelectedTodo local runs through update and goes nowhere
|
|
222
150
|
```
|
|
223
151
|
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
## What makes a Message durable?
|
|
152
|
+
From that, `make` derives the shared codec, the initial shared value, the
|
|
153
|
+
operation envelope, replay (your `update` on the shared slice), and the
|
|
154
|
+
server's [journal contract](#the-server). You write none of them.
|
|
229
155
|
|
|
230
|
-
|
|
231
|
-
|
|
156
|
+
`TodoSync` is a value. It has opened no storage, connected to nothing, and
|
|
157
|
+
rendered nothing. The next section does that.
|
|
232
158
|
|
|
233
|
-
|
|
234
|
-
shared projection.
|
|
159
|
+
## Run it in the browser
|
|
235
160
|
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
- returns a Command — a live external effect cannot be replayed later; or
|
|
240
|
-
- changes a field outside the shared projection — that change would disappear
|
|
241
|
-
when only the shared slice is persisted/replayed.
|
|
242
|
-
|
|
243
|
-
For example:
|
|
244
|
-
|
|
245
|
-
```text
|
|
246
|
-
RequestedChargeCard
|
|
247
|
-
-> runs a Command / talks to a provider
|
|
248
|
-
-> local intent, NOT durable
|
|
249
|
-
|
|
250
|
-
CardCharged
|
|
251
|
-
-> pure fact about state
|
|
252
|
-
-> durable and replayable
|
|
253
|
-
```
|
|
254
|
-
|
|
255
|
-
A common pattern is therefore:
|
|
256
|
-
|
|
257
|
-
```text
|
|
258
|
-
local intent Message
|
|
259
|
-
|
|
|
260
|
-
v
|
|
261
|
-
Command / nondeterministic work
|
|
262
|
-
|
|
|
263
|
-
v
|
|
264
|
-
durable fact Message
|
|
265
|
-
```
|
|
266
|
-
|
|
267
|
-
Replay runs the durable fact through the same application `update` on every
|
|
268
|
-
replica. If an application genuinely needs different replay semantics, `make`
|
|
269
|
-
accepts a custom `replay`, but that custom replay does not get the derived
|
|
270
|
-
guardrails.
|
|
271
|
-
|
|
272
|
-
## Running a replica
|
|
273
|
-
|
|
274
|
-
The low-level runtime is a `Replica`. It owns the local outbox, committed shared
|
|
275
|
-
snapshot, cursor, optimistic projection, and synchronization state.
|
|
161
|
+
Three steps: open storage and a replica, mount the application over it, start
|
|
162
|
+
the exchange loop.
|
|
276
163
|
|
|
277
164
|
```ts
|
|
278
|
-
import { Effect } from 'effect'
|
|
165
|
+
import { Effect, Scope } from 'effect'
|
|
279
166
|
import { ReplicaId, Sync } from 'foldkit-sync'
|
|
280
167
|
|
|
281
|
-
const
|
|
282
|
-
// One durable storage identity per document + replica/writer.
|
|
283
|
-
const storage = yield* Sync.indexedDb('todos/tab-1')
|
|
284
|
-
|
|
285
|
-
const replica = yield* TodoSync.openReplica(
|
|
286
|
-
ReplicaId.make('tab-1'),
|
|
287
|
-
storage,
|
|
288
|
-
)
|
|
289
|
-
|
|
290
|
-
// submit validates/replays first, then persists the operation in the outbox.
|
|
291
|
-
yield* replica.submit(
|
|
292
|
-
Message.CreatedTodo({ id: 't1', title: 'Milk' }),
|
|
293
|
-
)
|
|
294
|
-
|
|
295
|
-
// This already includes the pending operation. No server round trip required.
|
|
296
|
-
const optimistic = yield* replica.shared
|
|
297
|
-
|
|
298
|
-
return optimistic
|
|
299
|
-
}).pipe(Effect.scoped)
|
|
300
|
-
```
|
|
301
|
-
|
|
302
|
-
`replica.submit(message)` does not mean “send this now.” It means:
|
|
303
|
-
|
|
304
|
-
1. verify the Message is valid and durable;
|
|
305
|
-
2. replay it against the current optimistic shared state;
|
|
306
|
-
3. assign a stable local operation identity;
|
|
307
|
-
4. persist it to the local outbox;
|
|
308
|
-
5. make the new optimistic projection observable; and
|
|
309
|
-
6. wake the exchange loop if one is running.
|
|
310
|
-
|
|
311
|
-
A Message replay rejects never enters the outbox.
|
|
312
|
-
|
|
313
|
-
## Optimistic state and rebasing
|
|
314
|
-
|
|
315
|
-
Internally the replica keeps:
|
|
316
|
-
|
|
317
|
-
```text
|
|
318
|
-
committed
|
|
319
|
-
the authoritative shared snapshot up to cursor N
|
|
320
|
-
|
|
321
|
-
pending
|
|
322
|
-
locally-authored operations not yet resolved by the server
|
|
323
|
-
```
|
|
324
|
-
|
|
325
|
-
The value returned by `replica.shared` is:
|
|
326
|
-
|
|
327
|
-
```text
|
|
328
|
-
pending.reduce(replay, committed)
|
|
329
|
-
```
|
|
330
|
-
|
|
331
|
-
That simple rule gives optimistic UI and reconciliation the same semantics.
|
|
332
|
-
There is no special “optimistic reducer.”
|
|
333
|
-
|
|
334
|
-
When synchronization advances the committed base, Sync removes pending
|
|
335
|
-
operations that became committed, were explicitly acknowledged, or were
|
|
336
|
-
rejected, then the remaining outbox is replayed over the new committed state.
|
|
337
|
-
|
|
338
|
-
This also explains why durable Messages have to be replay-safe: a pending
|
|
339
|
-
operation may run many times as the authoritative base changes beneath it.
|
|
340
|
-
|
|
341
|
-
## Synchronization
|
|
342
|
-
|
|
343
|
-
A replica exchanges two things with the server:
|
|
344
|
-
|
|
345
|
-
```text
|
|
346
|
-
request
|
|
347
|
-
current cursor
|
|
348
|
-
pending operations
|
|
349
|
-
|
|
350
|
-
response
|
|
351
|
-
newly committed operations
|
|
352
|
-
acknowledgements
|
|
353
|
-
rejections
|
|
354
|
-
optional checkpoint
|
|
355
|
-
```
|
|
356
|
-
|
|
357
|
-
`replica.synchronize` performs one exchange through the `Transport` Effect
|
|
358
|
-
service:
|
|
168
|
+
const container = document.getElementById('app')!
|
|
359
169
|
|
|
360
|
-
|
|
361
|
-
const
|
|
362
|
-
|
|
170
|
+
// One IndexedDB database per document and tab; its connection lives for the page.
|
|
171
|
+
const scope = Effect.runSync(Scope.make())
|
|
172
|
+
const storage = Effect.runSync(
|
|
173
|
+
Effect.provideService(Sync.indexedDb('todos/tab-1'), Scope.Scope, scope),
|
|
363
174
|
)
|
|
364
|
-
|
|
175
|
+
const replica = Effect.runSync(TodoSync.openReplica(ReplicaId.make('tab-1'), storage))
|
|
365
176
|
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
177
|
+
const mounted = Sync.mount(App, TodoSync, {
|
|
178
|
+
replica,
|
|
179
|
+
container,
|
|
180
|
+
view: (model, h) => ({
|
|
181
|
+
title: 'Todos',
|
|
182
|
+
body: h.ul([], model.todos.map(todo => h.li([], [todo.title]))),
|
|
183
|
+
}),
|
|
184
|
+
onPersistenceFailure: (model, error) =>
|
|
185
|
+
modifyFields(model, { lastError: () => Option.some(error.message) }),
|
|
186
|
+
})
|
|
369
187
|
|
|
370
|
-
```ts
|
|
371
188
|
Effect.runFork(
|
|
372
|
-
Effect.provide(
|
|
373
|
-
replica.start,
|
|
374
|
-
Sync.transport.socket({ url: 'wss://example.com/sync' }),
|
|
375
|
-
),
|
|
189
|
+
Effect.provide(replica.start, Sync.transport.socket({ url: 'wss://example.com/sync' })),
|
|
376
190
|
)
|
|
377
|
-
```
|
|
378
|
-
|
|
379
|
-
The transport does not decide reconciliation. It only carries the exchange;
|
|
380
|
-
the replica validates the response and applies the protocol rules.
|
|
381
|
-
|
|
382
|
-
### Acknowledgements
|
|
383
|
-
|
|
384
|
-
An acknowledgement says an operation the replica sent is durably accepted, so
|
|
385
|
-
it may leave the outbox even if its committed payload is not repeated in this
|
|
386
|
-
response.
|
|
387
191
|
|
|
388
|
-
|
|
389
|
-
|
|
390
|
-
A rejection says the optimistic operation did not commit. Sync removes it from
|
|
391
|
-
the pending set and recomputes the optimistic state from the authoritative base
|
|
392
|
-
plus whatever pending operations remain.
|
|
393
|
-
|
|
394
|
-
```text
|
|
395
|
-
before response
|
|
396
|
-
committed: A B C
|
|
397
|
-
pending: D E
|
|
398
|
-
visible: A B C D E
|
|
399
|
-
|
|
400
|
-
server rejects D
|
|
401
|
-
|
|
402
|
-
committed: A B C
|
|
403
|
-
pending: E
|
|
404
|
-
visible: A B C E
|
|
405
|
-
```
|
|
406
|
-
|
|
407
|
-
`replica.status.rejected` keeps recent rejected operation ids so the UI can
|
|
408
|
-
explain that an optimistic edit was refused.
|
|
409
|
-
|
|
410
|
-
### Checkpoints
|
|
411
|
-
|
|
412
|
-
A server may compact old history. If the replica is too far behind to replay the
|
|
413
|
-
missing prefix, the response can include a checkpoint:
|
|
414
|
-
|
|
415
|
-
```text
|
|
416
|
-
checkpoint = authoritative shared snapshot + cursor
|
|
192
|
+
mounted.dispatch(Message.CreatedTodo({ id: crypto.randomUUID(), title: 'Milk' }))
|
|
417
193
|
```
|
|
418
194
|
|
|
419
|
-
|
|
420
|
-
pending operations on top. A checkpoint behind the replica's current cursor is
|
|
421
|
-
rejected with `CheckpointRegressionError` rather than silently moving history
|
|
422
|
-
backwards.
|
|
423
|
-
|
|
424
|
-
## Replica status and UI state
|
|
195
|
+
What each call does, and does not do:
|
|
425
196
|
|
|
426
|
-
|
|
427
|
-
|
|
197
|
+
- **`Sync.indexedDb(name)`** opens one database. Name it by document and
|
|
198
|
+
writer: two tabs on one name race, and the loser fails its write rather than
|
|
199
|
+
merging two histories.
|
|
200
|
+
- **`openReplica(id, storage)`** reads back the committed snapshot, cursor, and
|
|
201
|
+
outbox saved last time, or starts empty. It does no network I/O.
|
|
202
|
+
- **`Sync.mount(App, TodoSync, …)`** runs the ordinary Foldkit runtime with
|
|
203
|
+
your `view` and `update`. A durable Message applies through `update` at once,
|
|
204
|
+
then a Command persists it; if the persist fails, the edit is reverted and
|
|
205
|
+
`onPersistenceFailure` says so in the Model. A local Message is untouched.
|
|
206
|
+
- **`replica.start`** is the exchange loop: once now, then after every submit
|
|
207
|
+
and every notice from the server. A failed exchange is retried on a backoff
|
|
208
|
+
from 0.5 s to 30 s, or at once on the next edit. The loop never fails; it
|
|
209
|
+
reports through `status`.
|
|
210
|
+
- **`mounted.dispatch`** sends a Message the way a view does. The container
|
|
211
|
+
must have an `id`, or `mount` throws.
|
|
428
212
|
|
|
429
|
-
|
|
430
|
-
|
|
431
|
-
// {
|
|
432
|
-
// pending: number,
|
|
433
|
-
// cursor: Sequence,
|
|
434
|
-
// lastError: string | undefined,
|
|
435
|
-
// rejected: ReadonlyArray<OpId>
|
|
436
|
-
// }
|
|
437
|
-
```
|
|
213
|
+
Reload the page: the todos come back from the outbox and the committed
|
|
214
|
+
snapshot, the selection does not. That is the shared/local line, visible.
|
|
438
215
|
|
|
439
|
-
`
|
|
440
|
-
|
|
441
|
-
|
|
442
|
-
two separate reads. `replica.snapshot` is the same read, once: the status,
|
|
443
|
-
`shared`, `committed`, and `nextLocalSequence`, the local sequence the next
|
|
444
|
-
submitted operation takes.
|
|
216
|
+
`mounted` also exposes `model()`, `subscribe`, `observe`, `committed`, and
|
|
217
|
+
`dispose()`, which waits for in-flight persists and leaves the replica open.
|
|
218
|
+
Close the replica with its own scope.
|
|
445
219
|
|
|
446
|
-
##
|
|
220
|
+
## The server
|
|
447
221
|
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
client
|
|
453
|
-
-----------------------------------------------
|
|
454
|
-
Foldkit update
|
|
455
|
-
|
|
|
456
|
-
| durable Message
|
|
457
|
-
v
|
|
458
|
-
foldkit-sync Replica
|
|
459
|
-
optimistic shared state
|
|
460
|
-
persistent outbox
|
|
461
|
-
|
|
|
462
|
-
| exchange
|
|
463
|
-
v
|
|
464
|
-
-----------------------------------------------
|
|
465
|
-
server
|
|
466
|
-
foldkit-durable Journal
|
|
467
|
-
authoritative order
|
|
468
|
-
snapshot + cursor
|
|
469
|
-
|
|
|
470
|
-
v
|
|
471
|
-
committed operations / checkpoint / rejection
|
|
472
|
-
```
|
|
473
|
-
|
|
474
|
-
The same Sync contract produces the server journal contract, so the two halves
|
|
475
|
-
do not duplicate the Message codec, shared schema, initial snapshot, or replay
|
|
476
|
-
reducer:
|
|
222
|
+
The server answers the exchange: it takes a cursor and pending operations,
|
|
223
|
+
commits what it accepts, and returns what the replica is missing. With
|
|
224
|
+
`foldkit-durable`, the same contract supplies the journal's codecs, empty
|
|
225
|
+
snapshot, reducer, and any [authorization](#authorize-on-the-server):
|
|
477
226
|
|
|
478
227
|
```ts
|
|
479
|
-
import { Effect } from 'effect'
|
|
480
228
|
import { ActorId, DocumentId, Journal, OpId } from 'foldkit-durable'
|
|
481
229
|
|
|
482
230
|
type Principal = { readonly actorId: string }
|
|
483
231
|
|
|
484
|
-
const
|
|
485
|
-
|
|
486
|
-
|
|
487
|
-
|
|
488
|
-
|
|
489
|
-
|
|
232
|
+
const journal = yield* Journal.make({
|
|
233
|
+
...TodoSync.journalContract(),
|
|
234
|
+
file: 'todos.sqlite',
|
|
235
|
+
opId: operation => OpId.make(operation.opId),
|
|
236
|
+
actorId: (principal: Principal) => ActorId.make(principal.actorId),
|
|
237
|
+
})
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
The exchange handler itself is yours: it calls `journal.append` for each
|
|
241
|
+
pending operation, `journal.read` for what follows the cursor, and serves the
|
|
242
|
+
result on a socket with `Sync.transport.serve`. [The replicated state
|
|
243
|
+
guide](../../docs/replication.md#3-the-server-an-exchange-over-the-journal) shows
|
|
244
|
+
the whole handler in one page, and [`examples/sync`](../../examples/sync) runs
|
|
245
|
+
it over `ws`.
|
|
490
246
|
|
|
491
|
-
|
|
492
|
-
opId: operation => OpId.make(operation.opId),
|
|
493
|
-
actorId: (principal: Principal) => ActorId.make(principal.actorId),
|
|
494
|
-
})
|
|
247
|
+
A server that is not Durable implements the same [exchange](#the-exchange).
|
|
495
248
|
|
|
496
|
-
|
|
497
|
-
}).pipe(Effect.scoped)
|
|
498
|
-
```
|
|
249
|
+
## What makes a Message durable
|
|
499
250
|
|
|
500
|
-
|
|
501
|
-
|
|
502
|
-
|
|
251
|
+
Replay runs a durable Message through `update` on another machine, later,
|
|
252
|
+
possibly many times as the base beneath it moves. So it has to be a
|
|
253
|
+
deterministic, state-only transition of the shared slice. `make` derives replay
|
|
254
|
+
from `update` and refuses a durable Message whose transition:
|
|
503
255
|
|
|
504
|
-
|
|
256
|
+
- returns a Command, because an effect cannot be replayed; or
|
|
257
|
+
- changes a field outside `shared`, because that change would be lost.
|
|
505
258
|
|
|
506
|
-
|
|
507
|
-
|
|
508
|
-
Model.
|
|
259
|
+
A refused Message fails `submit` with `ReplayError` and writes nothing, so a
|
|
260
|
+
mistake shows on the first edit instead of as diverging replicas later.
|
|
509
261
|
|
|
510
|
-
|
|
262
|
+
The consequence is a pattern you will use everywhere: **an intent mints, a fact
|
|
263
|
+
records**.
|
|
511
264
|
|
|
512
265
|
```text
|
|
513
|
-
|
|
514
|
-
|
|
|
515
|
-
v
|
|
516
|
-
application update immediately
|
|
517
|
-
|
|
|
518
|
-
+--> local fields / render
|
|
519
|
-
|
|
|
266
|
+
RequestedTodo { title } local: its update returns a Command
|
|
267
|
+
| that reads the clock and mints an id
|
|
520
268
|
v
|
|
521
|
-
|
|
522
|
-
|
|
|
523
|
-
exchange changes shared state later
|
|
524
|
-
v
|
|
525
|
-
install reconciled shared slice back into Model
|
|
269
|
+
CreatedTodo { id, title, at } durable: pure over the shared slice
|
|
526
270
|
```
|
|
527
271
|
|
|
528
|
-
|
|
529
|
-
|
|
272
|
+
Put every nondeterministic input (ids, timestamps, the result of a provider
|
|
273
|
+
call) *inside* the fact. `update` for the fact only applies it.
|
|
530
274
|
|
|
531
|
-
|
|
532
|
-
|
|
533
|
-
|
|
275
|
+
When the intent needs nothing outside the Model, say an id from a counter in
|
|
276
|
+
the shared slice, return the fact as `Sync.fact(message)` instead of an
|
|
277
|
+
ordinary Command. The mount applies it in the intent's own transition, before
|
|
278
|
+
any later Message, so two intents dispatched together cannot read the same
|
|
279
|
+
counter. Outside the mount it is an ordinary Command.
|
|
534
280
|
|
|
535
|
-
|
|
281
|
+
A durable `update` should also be idempotent where a retry could deliver one
|
|
282
|
+
operation twice, as in `CreatedTodo` ignoring an id it already holds.
|
|
536
283
|
|
|
537
|
-
|
|
538
|
-
const storage = Effect.runSync(
|
|
539
|
-
Effect.provideService(
|
|
540
|
-
Sync.indexedDb('todos/tab-1'),
|
|
541
|
-
Scope.Scope,
|
|
542
|
-
scope,
|
|
543
|
-
),
|
|
544
|
-
)
|
|
545
|
-
const replica = Effect.runSync(
|
|
546
|
-
TodoSync.openReplica(ReplicaId.make('tab-1'), storage),
|
|
547
|
-
)
|
|
284
|
+
## What to show the user
|
|
548
285
|
|
|
549
|
-
|
|
550
|
-
replica,
|
|
551
|
-
container,
|
|
552
|
-
view: (model, h) => ({
|
|
553
|
-
title: 'Todos',
|
|
554
|
-
body: h.ul(
|
|
555
|
-
[],
|
|
556
|
-
model.todos.map(todo => h.li([], [todo.title])),
|
|
557
|
-
),
|
|
558
|
-
}),
|
|
559
|
-
onPersistenceFailure: (model, error) => ({
|
|
560
|
-
...model,
|
|
561
|
-
lastError: Option.some(error.message),
|
|
562
|
-
}),
|
|
563
|
-
})
|
|
286
|
+
The three milestones each have a reader.
|
|
564
287
|
|
|
565
|
-
|
|
566
|
-
Effect.provide(
|
|
567
|
-
replica.start,
|
|
568
|
-
Sync.transport.socket({ url: 'wss://example.com/sync' }),
|
|
569
|
-
),
|
|
570
|
-
)
|
|
288
|
+
**Visible** is the Model. Nothing to do.
|
|
571
289
|
|
|
572
|
-
|
|
573
|
-
|
|
574
|
-
|
|
575
|
-
|
|
576
|
-
mounted.model()
|
|
577
|
-
await mounted.dispose()
|
|
578
|
-
```
|
|
290
|
+
**Saved** is the persist. `onPersistenceFailure` runs when it fails, after the
|
|
291
|
+
edit has been reverted; the Model it returns is what the user sees next. The
|
|
292
|
+
error names why: `ReplayError` for a Message that cannot be durable,
|
|
293
|
+
`StorageError` for IndexedDB.
|
|
579
294
|
|
|
580
|
-
|
|
581
|
-
remains separately owned and must be closed with its scope/lifecycle.
|
|
295
|
+
**Committed** is the replica's status:
|
|
582
296
|
|
|
583
|
-
|
|
584
|
-
|
|
585
|
-
|
|
586
|
-
|
|
587
|
-
- `init(model, url)` reduces the initial URL before the first render;
|
|
588
|
-
- `onUrlChange(url)` maps browser navigation back to a Message;
|
|
589
|
-
- `onUrlRequest` optionally maps link requests to a Message.
|
|
297
|
+
```ts
|
|
298
|
+
const status = yield* replica.status
|
|
299
|
+
// { pending: number, cursor: Sequence, lastError: string | undefined, rejected: ReadonlyArray<OpId> }
|
|
300
|
+
```
|
|
590
301
|
|
|
591
|
-
|
|
592
|
-
|
|
302
|
+
`pending` above zero means edits the server has not confirmed; `lastError` is
|
|
303
|
+
the last failed exchange, cleared by the next success; `rejected` lists recent
|
|
304
|
+
edits the server refused. `replica.statusChanges` is a Stream of the same,
|
|
305
|
+
emitted after every submit and exchange; `replica.changes` pairs it with the
|
|
306
|
+
optimistic `shared` value so one subscription sees both.
|
|
593
307
|
|
|
594
|
-
|
|
595
|
-
|
|
596
|
-
|
|
597
|
-
|
|
308
|
+
**A refused edit** arrives as a rejection on the next exchange, not as an
|
|
309
|
+
error. The replica drops it and replays the rest; the mount re-installs the
|
|
310
|
+
shared slice, and the field the user edited reverts. `onPersistenceFailure`
|
|
311
|
+
does not run for this, since the local save succeeded. Watch `rejected` to
|
|
312
|
+
explain the revert.
|
|
598
313
|
|
|
599
|
-
The Model shows
|
|
600
|
-
must not report success until the
|
|
601
|
-
`mounted.committed
|
|
314
|
+
**Waiting for the server.** The Model shows an edit before the server has it.
|
|
315
|
+
Something that must not report success until the edit is committed, an agent
|
|
316
|
+
tool for instance, waits on `mounted.committed`: the shared slice as the server
|
|
317
|
+
confirmed it, with no pending edit applied. It is a source, `{ get, subscribe }`,
|
|
318
|
+
told after every exchange:
|
|
602
319
|
|
|
603
320
|
```ts
|
|
604
321
|
completion: Agent.when({
|
|
@@ -607,29 +324,22 @@ completion: Agent.when({
|
|
|
607
324
|
})
|
|
608
325
|
```
|
|
609
326
|
|
|
610
|
-
|
|
611
|
-
server rejects.
|
|
612
|
-
`mounted.committed` is a source (`{ get, subscribe }`), not a Projection: it reads
|
|
613
|
-
the replica, not the Model, and tells subscribers after every exchange.
|
|
327
|
+
That completes after the exchange that commits the todo and never for one the
|
|
328
|
+
server rejects.
|
|
614
329
|
|
|
615
|
-
|
|
616
|
-
different mechanism: see [what a reader sees while a change is in
|
|
617
|
-
flight](../../docs/state-model.md#what-a-reader-sees-while-a-change-is-in-flight).
|
|
330
|
+
## Growing the application
|
|
618
331
|
|
|
619
|
-
|
|
332
|
+
### Compose a document from features
|
|
620
333
|
|
|
621
|
-
|
|
622
|
-
|
|
334
|
+
Each feature declares the fields it shares and the Messages that change them;
|
|
335
|
+
`compose` merges them into one document:
|
|
623
336
|
|
|
624
337
|
```ts
|
|
625
338
|
const AppSync = Sync.forApplication(App)
|
|
626
339
|
|
|
627
340
|
const Todos = AppSync.fragment({
|
|
628
341
|
shared: Projection.pick(App.model.todos),
|
|
629
|
-
durable: MessageSet.make(App, [
|
|
630
|
-
Message.CreatedTodo,
|
|
631
|
-
Message.RenamedTodo,
|
|
632
|
-
]),
|
|
342
|
+
durable: MessageSet.make(App, [Message.CreatedTodo, Message.RenamedTodo]),
|
|
633
343
|
})
|
|
634
344
|
|
|
635
345
|
const Members = AppSync.fragment({
|
|
@@ -643,368 +353,295 @@ const Board = AppSync.make({
|
|
|
643
353
|
})
|
|
644
354
|
```
|
|
645
355
|
|
|
646
|
-
`compose`
|
|
647
|
-
|
|
648
|
-
|
|
649
|
-
architectural mistake of two contracts claiming the same field ownership.
|
|
356
|
+
`compose` throws on a field declared twice with different codecs, a Message
|
|
357
|
+
declared durable twice, or a fragment from another application. The result is
|
|
358
|
+
still one document: one outbox, one cursor, one server order.
|
|
650
359
|
|
|
651
|
-
|
|
360
|
+
### Authorize on the server
|
|
652
361
|
|
|
653
|
-
|
|
654
|
-
|
|
655
|
-
`journalContract()` and is enforced by the server journal — not trusted to the
|
|
656
|
-
client.
|
|
362
|
+
Policy belongs on the contract, beside the Messages it governs, and runs on the
|
|
363
|
+
server inside the journal's commit. The client never runs it.
|
|
657
364
|
|
|
658
365
|
```ts
|
|
659
|
-
const Authorized = Sync.forApplication(App)
|
|
660
|
-
.withPrincipal<{ readonly role: 'admin' | 'guest' }>()
|
|
366
|
+
const Authorized = Sync.forApplication(App).withPrincipal<{ readonly role: 'admin' | 'guest' }>()
|
|
661
367
|
|
|
662
368
|
const Board = Authorized.make({
|
|
663
369
|
documentId: DocumentId.make('board'),
|
|
664
370
|
...Authorized.compose(Todos, Members),
|
|
665
371
|
authorize: {
|
|
666
372
|
RenamedTodo: ({ principal, message, shared }) =>
|
|
667
|
-
principal.role === 'admin' &&
|
|
668
|
-
shared.todos.some(todo => todo.id === message.id),
|
|
373
|
+
principal.role === 'admin' && shared.todos.some(todo => todo.id === message.id),
|
|
669
374
|
},
|
|
670
375
|
})
|
|
671
376
|
```
|
|
672
377
|
|
|
673
|
-
|
|
674
|
-
|
|
675
|
-
|
|
676
|
-
|
|
677
|
-
|
|
678
|
-
rebases the remaining ones. This is why rejection is a normal exchange result,
|
|
679
|
-
not a transport failure.
|
|
378
|
+
`message` is exactly that variant, `shared` is the authoritative snapshot, and
|
|
379
|
+
`principal` is whatever the server's transport established for the connection;
|
|
380
|
+
it is never read from the operation. A variant without a rule is allowed. A key
|
|
381
|
+
that is not a durable variant is a type error. The rules ride in
|
|
382
|
+
`journalContract()`, so the server enforces them by spreading the contract.
|
|
680
383
|
|
|
681
|
-
|
|
384
|
+
An edit the policy refuses still shows at once on the client, then comes back
|
|
385
|
+
as a rejection, as [above](#what-to-show-the-user).
|
|
682
386
|
|
|
683
|
-
|
|
684
|
-
lose edits that never reached the server.
|
|
387
|
+
### Coalesce a burst of typing
|
|
685
388
|
|
|
686
|
-
|
|
389
|
+
Typing makes one durable Message per keystroke. `coalesce(last, next)` is
|
|
390
|
+
offered each submit together with the last operation in the outbox; return a
|
|
391
|
+
merged Message to replace it, or `undefined` to keep them apart:
|
|
687
392
|
|
|
688
393
|
```ts
|
|
689
|
-
|
|
394
|
+
coalesce: (last, next) =>
|
|
395
|
+
last._tag === 'RenamedTodo' && next._tag === 'RenamedTodo' && last.id === next.id
|
|
396
|
+
? next
|
|
397
|
+
: undefined
|
|
690
398
|
```
|
|
691
399
|
|
|
692
|
-
|
|
693
|
-
|
|
694
|
-
|
|
695
|
-
|
|
400
|
+
Replaying the merged Message must equal replaying `last` then `next`. An
|
|
401
|
+
operation is only merged into while no exchange has carried it, so there is no
|
|
402
|
+
timer to tune: keystrokes typed while a request is out become one operation, and
|
|
403
|
+
a lone keystroke goes out as fast as before.
|
|
696
404
|
|
|
697
|
-
|
|
405
|
+
### The URL, mirrors, and agents
|
|
698
406
|
|
|
699
|
-
|
|
700
|
-
|
|
701
|
-
|
|
702
|
-
|
|
703
|
-
-
|
|
704
|
-
durable work never enters the outbox.
|
|
705
|
-
- **Malformed stored data.** Opening fails rather than silently replacing bytes;
|
|
706
|
-
the application can offer an explicit reset/recovery path.
|
|
707
|
-
- **Unsupported storage version.** `UnsupportedReplicaVersionError` names the
|
|
708
|
-
persisted and supported versions so migration/reset can be explicit.
|
|
709
|
-
- **Server compaction.** A checkpoint replaces history the replica can no longer
|
|
710
|
-
replay and pending local work is rebased on it.
|
|
407
|
+
`mount` takes the application's own `subscriptions` and the `resources` Layer
|
|
408
|
+
they need, and `url: { init, onUrlChange, onUrlRequest? }` to route navigation
|
|
409
|
+
through `update`: `init(model, url)` reduces the first URL before the first
|
|
410
|
+
render and `onUrlChange(url)` names the Message for every navigation. A
|
|
411
|
+
[`foldkit-mirror`](../mirror) URL mirror plugs into exactly that seam.
|
|
711
412
|
|
|
712
|
-
|
|
713
|
-
|
|
413
|
+
`Mounted` is also the host an agent binds to: `model`, `dispatch`, `subscribe`,
|
|
414
|
+
and `observe`, which reports every application Message the runtime applies.
|
|
415
|
+
See [`examples/todo-app`](../../examples/todo-app) for both.
|
|
714
416
|
|
|
715
|
-
|
|
417
|
+
### Carry local state across a reinstall
|
|
716
418
|
|
|
717
|
-
|
|
718
|
-
|
|
719
|
-
|
|
720
|
-
|
|
721
|
-
|
|
722
|
-
Sync.transport.loopback(...)
|
|
723
|
-
Sync.transport.fromPromise(...)
|
|
724
|
-
Sync.transport.toPromise(...)
|
|
725
|
-
Sync.transport.serve(...)
|
|
726
|
-
Sync.transport.nativeSocket(...)
|
|
727
|
-
```
|
|
419
|
+
When an exchange or a failed persist replaces the shared slice outside
|
|
420
|
+
`update`, `onReinstall(next, previous)` returns the transition instead. Use it
|
|
421
|
+
to keep a selection that points into the slice, or to return a Command that
|
|
422
|
+
patches a DOM the change has to reach. Omitted, the Model is simply `next`. It
|
|
423
|
+
is not called for a status that changed nothing the Model shows.
|
|
728
424
|
|
|
729
|
-
|
|
730
|
-
jittered backoff, and keeps request identities stable when resending queued or
|
|
731
|
-
in-flight exchanges. A server rejection is protocol data; only a wire failure is
|
|
732
|
-
a `TransportError`.
|
|
425
|
+
## Testing without a browser or a server
|
|
733
426
|
|
|
734
|
-
|
|
735
|
-
|
|
427
|
+
A `Storage` needs three members, and the loopback transport takes a handler in
|
|
428
|
+
place of a socket:
|
|
736
429
|
|
|
737
|
-
|
|
738
|
-
|
|
739
|
-
|
|
740
|
-
|
|
741
|
-
|
|
430
|
+
```ts
|
|
431
|
+
import { Effect } from 'effect'
|
|
432
|
+
import type { Storage } from 'foldkit-sync'
|
|
433
|
+
|
|
434
|
+
const memoryStorage = (): Storage => {
|
|
435
|
+
let state: unknown
|
|
436
|
+
return {
|
|
437
|
+
load: () => Effect.sync(() => state),
|
|
438
|
+
save: next => Effect.sync(() => void (state = structuredClone(next))),
|
|
439
|
+
close: Effect.void,
|
|
440
|
+
}
|
|
441
|
+
}
|
|
742
442
|
|
|
743
|
-
|
|
744
|
-
|
|
745
|
-
|
|
443
|
+
const program = Effect.gen(function* () {
|
|
444
|
+
const replica = yield* TodoSync.openReplica(ReplicaId.make('test'), memoryStorage())
|
|
445
|
+
yield* replica.submit(Message.CreatedTodo({ id: 't1', title: 'Milk' }))
|
|
446
|
+
yield* replica.synchronize
|
|
447
|
+
return yield* replica.status
|
|
448
|
+
}).pipe(
|
|
449
|
+
Effect.provide(
|
|
450
|
+
Sync.transport.loopback((_cursor, pending) => ({
|
|
451
|
+
operations: [],
|
|
452
|
+
acknowledged: pending.map(operation => operation.opId),
|
|
453
|
+
rejected: [],
|
|
454
|
+
})),
|
|
455
|
+
),
|
|
456
|
+
)
|
|
457
|
+
```
|
|
746
458
|
|
|
747
|
-
|
|
459
|
+
In Node, `Sync.indexedDb(name, new IDBFactory())` from `fake-indexeddb` tests
|
|
460
|
+
the real storage adapter. [`examples/sync`](../../examples/sync) tests every
|
|
461
|
+
recovery case below against a real journal.
|
|
462
|
+
|
|
463
|
+
## When things go wrong
|
|
464
|
+
|
|
465
|
+
Each case is designed, not incidental. The ones that change how you build the
|
|
466
|
+
application come first.
|
|
467
|
+
|
|
468
|
+
- **The network is gone.** Edits keep applying and keep being saved. `pending`
|
|
469
|
+
grows; the loop retries. Nothing to handle.
|
|
470
|
+
- **The server rejects an edit.** Dropped from `pending`, the rest replayed,
|
|
471
|
+
the Model re-installed. Explain it from `status.rejected`.
|
|
472
|
+
- **The persist fails.** The edit is reverted and `onPersistenceFailure` runs.
|
|
473
|
+
The user has to redo it.
|
|
474
|
+
- **Local storage was evicted.** The replica starts at cursor 0 and catches up
|
|
475
|
+
from the server, but edits that lived only in the outbox are gone. The
|
|
476
|
+
outbox is user data, not a cache.
|
|
477
|
+
- **Two tabs share one storage name.** Compare-and-swap fails the stale writer
|
|
478
|
+
with `StorageError`. Give each tab its own name and `ReplicaId`.
|
|
479
|
+
- **The replica is far behind.** The server may answer in pages (`more`), and
|
|
480
|
+
`synchronize` asks again at once until caught up. If the server compacted
|
|
481
|
+
the history it needs, it sends a checkpoint instead: a snapshot and cursor
|
|
482
|
+
the replica adopts as its base before replaying its pending edits. A
|
|
483
|
+
checkpoint behind the replica's cursor is refused
|
|
484
|
+
(`CheckpointRegressionError`), never applied backwards.
|
|
485
|
+
- **The server lost its history.** A server names its history with an `epoch`;
|
|
486
|
+
a replica that hears a new one rebuilds its committed state from that answer
|
|
487
|
+
and resends its outbox. Everything the old server committed is gone on every
|
|
488
|
+
replica. A replica that never heard an epoch cannot tell, and fails every
|
|
489
|
+
exchange until its storage is cleared.
|
|
490
|
+
- **Stored state is from another version.** Opening fails with
|
|
491
|
+
`UnsupportedReplicaVersionError` naming both versions; the bytes are left for
|
|
492
|
+
a deliberate migration or reset. Sync versions its own envelope only; your
|
|
493
|
+
Message and snapshot migrations stay yours.
|
|
494
|
+
- **The server answers nonsense.** Malformed exchanges, gaps in the committed
|
|
495
|
+
order, acknowledgements or rejections for operations the replica never sent:
|
|
496
|
+
each fails the exchange rather than touch local work.
|
|
497
|
+
|
|
498
|
+
## Advanced: the replica
|
|
499
|
+
|
|
500
|
+
`Sync.mount` is the normal path. Underneath, a `Replica` works on the shared
|
|
501
|
+
slice alone, and an application that is not a Foldkit runtime (a worker, a
|
|
502
|
+
server-side actor, a test) uses it directly.
|
|
748
503
|
|
|
749
|
-
```
|
|
750
|
-
|
|
751
|
-
Sync.
|
|
752
|
-
|
|
504
|
+
```ts
|
|
505
|
+
const program = Effect.gen(function* () {
|
|
506
|
+
const storage = yield* Sync.indexedDb('todos/tab-1')
|
|
507
|
+
const replica = yield* TodoSync.openReplica(ReplicaId.make('tab-1'), storage)
|
|
753
508
|
|
|
754
|
-
|
|
509
|
+
yield* replica.submit(Message.CreatedTodo({ id: 't1', title: 'Milk' }))
|
|
510
|
+
const optimistic = yield* replica.shared // includes the pending edit
|
|
755
511
|
|
|
756
|
-
|
|
757
|
-
|
|
758
|
-
Sync.
|
|
512
|
+
yield* replica.synchronize // one exchange
|
|
513
|
+
return optimistic
|
|
514
|
+
}).pipe(Effect.scoped, Effect.provide(Sync.transport.socket({ url: 'wss://example.com/sync' })))
|
|
759
515
|
```
|
|
760
516
|
|
|
761
|
-
|
|
762
|
-
|
|
517
|
+
`submit(message)` checks that the Message is durable, replays it against the
|
|
518
|
+
current optimistic state, gives it a stable operation id, saves it to the
|
|
519
|
+
outbox, publishes the new optimistic value, and wakes the loop. It sends
|
|
520
|
+
nothing. A Message replay refuses never enters the outbox.
|
|
763
521
|
|
|
764
|
-
|
|
522
|
+
`synchronize` takes one snapshot of the cursor and outbox, performs one
|
|
523
|
+
exchange, validates the answer, and reconciles it with the replica's *current*
|
|
524
|
+
state, so an edit submitted while the request was in flight survives.
|
|
765
525
|
|
|
766
|
-
|
|
767
|
-
|
|
768
|
-
|
|
769
|
-
```
|
|
526
|
+
The rest of the surface: `committed` (the base alone), `pending`, `cursor`,
|
|
527
|
+
`status`, `statusChanges`, `changes` (status and `shared` from one snapshot),
|
|
528
|
+
`snapshot` (the same plus `nextLocalSequence`), `start`, and `close`.
|
|
770
529
|
|
|
771
|
-
|
|
530
|
+
### The exchange
|
|
772
531
|
|
|
773
|
-
|
|
774
|
-
|
|
775
|
-
rule — for example, the newest logical title write should win even if an offline
|
|
776
|
-
device reconnects later.
|
|
532
|
+
One request, one answer. A custom server implements this; a custom transport
|
|
533
|
+
carries it.
|
|
777
534
|
|
|
778
|
-
|
|
779
|
-
|
|
780
|
-
|
|
535
|
+
```text
|
|
536
|
+
request cursor
|
|
537
|
+
pending operations
|
|
538
|
+
epoch, once the replica has heard one
|
|
539
|
+
|
|
540
|
+
answer operations committed after the cursor, contiguous
|
|
541
|
+
acknowledged ids the server accepted without repeating them
|
|
542
|
+
rejected ids the server refused
|
|
543
|
+
checkpoint? { cursor, model } when the history is gone
|
|
544
|
+
more? true when another page follows
|
|
545
|
+
epoch? the server's history
|
|
546
|
+
```
|
|
547
|
+
|
|
548
|
+
An acknowledged operation leaves the outbox once its committed copy has
|
|
549
|
+
arrived or the last page has been read. A rejected one leaves at once. The
|
|
550
|
+
replica validates every field; see [when things go wrong](#when-things-go-wrong).
|
|
551
|
+
|
|
552
|
+
### Transports
|
|
553
|
+
|
|
554
|
+
`Transport` is an Effect service with one operation. Sync ships:
|
|
555
|
+
|
|
556
|
+
- `Sync.transport.socket({ url, … })`: a reconnecting WebSocket client. It
|
|
557
|
+
queues exchanges while connecting, retries on a jittered backoff capped at
|
|
558
|
+
`maxRetryDelay` (5 s), keeps request ids stable across resends, and after
|
|
559
|
+
`maxRetries` (5) consecutive failed connections fails queued work and fails
|
|
560
|
+
new exchanges fast until a socket is healthy again. `maxQueue` (64) bounds
|
|
561
|
+
the queue.
|
|
562
|
+
- `Sync.transport.serve(socket, { exchange, changes? })`: the server side of one
|
|
563
|
+
accepted socket. With `changes`, a subscription to the document's commits, it
|
|
564
|
+
sends a notice after each; the client's loop exchanges on every notice, so a
|
|
565
|
+
tab that is only reading still sees others' edits without polling.
|
|
566
|
+
- `Sync.transport.loopback(handler)`, `fromPromise(client)`, `toPromise(shape)`,
|
|
567
|
+
and `nativeSocket` for the platform `WebSocket`.
|
|
568
|
+
|
|
569
|
+
A refusal is data in the answer; only a wire failure is a `TransportError`.
|
|
570
|
+
|
|
571
|
+
### Presence
|
|
572
|
+
|
|
573
|
+
"Who is viewing this" and "where is their caret" should vanish when the peer
|
|
574
|
+
does, not replay next week. `Sync.presence.make({ id, ttl, decodeValue, channel?,
|
|
575
|
+
throttle? })` is a TTL'd peer registry outside the log: values are decoded before
|
|
576
|
+
they enter it, a peer that stops refreshing is pruned, and `throttle` sends at
|
|
577
|
+
most one value per interval, the latest. `Sync.presence.socketChannel(transport.socket)`
|
|
578
|
+
shares the sync socket across reconnects; `Sync.presence.hub` and `.serve` are
|
|
579
|
+
the server side. [`examples/pages`](../../examples/pages) shows remote carets.
|
|
580
|
+
|
|
581
|
+
### Last-writer-wins fields (experimental)
|
|
582
|
+
|
|
583
|
+
Server order decides concurrent edits. When one field should instead keep the
|
|
584
|
+
newest logical write even from a device that reconnects late, `Sync.lww.register(schema)`
|
|
585
|
+
gives it a stamped schema and a pure `merge`; `Sync.lww.openClock` allocates
|
|
586
|
+
stamps from a persisted counter, before dispatch and never in `update`. It is a
|
|
587
|
+
per-field rule, not a CRDT mode: sets, counters, and text stay outside.
|
|
781
588
|
|
|
782
589
|
```ts
|
|
783
|
-
import { Schema } from 'effect'
|
|
784
|
-
import { ReplicaId, Sync } from 'foldkit-sync'
|
|
785
|
-
|
|
786
590
|
const Title = Sync.lww.register(Schema.String)
|
|
787
591
|
const Shared = Schema.Struct({ title: Title.schema })
|
|
788
|
-
|
|
789
|
-
_tag: Schema.Literal('Renamed'),
|
|
790
|
-
title: Title.schema,
|
|
791
|
-
})
|
|
792
|
-
|
|
793
|
-
const update = (
|
|
794
|
-
model: typeof Shared.Type,
|
|
795
|
-
message: typeof Renamed.Type,
|
|
796
|
-
) => ({
|
|
797
|
-
...model,
|
|
798
|
-
title: Title.merge(model.title, message.title),
|
|
799
|
-
})
|
|
800
|
-
|
|
801
|
-
const message: typeof Renamed.Type = {
|
|
802
|
-
_tag: 'Renamed',
|
|
803
|
-
title: {
|
|
804
|
-
stamp: { counter: 1, replicaId: ReplicaId.make('tab-a') },
|
|
805
|
-
value: 'Milk',
|
|
806
|
-
},
|
|
807
|
-
}
|
|
592
|
+
// in update: title: Title.merge(model.title, message.title)
|
|
808
593
|
```
|
|
809
594
|
|
|
810
|
-
|
|
811
|
-
order, independent of locale. This is logical ordering, not wall-clock time.
|
|
812
|
-
The rule follows the total-order register model described in
|
|
813
|
-
[Replicated Data Types: Specification, Verification, Optimality](https://www.microsoft.com/en-us/research/publication/replicated-data-types-specification-verification-optimality/).
|
|
814
|
-
|
|
815
|
-
Allocate stamps **before dispatch**, never inside `update` or replay.
|
|
816
|
-
`Sync.lww.openClock` persists the high-water mark independently of the outbox,
|
|
817
|
-
so rejected or never-submitted writes cannot cause timestamp reuse after reload:
|
|
818
|
-
|
|
819
|
-
```ts
|
|
820
|
-
import { Effect } from 'effect'
|
|
821
|
-
import { DocumentId, ReplicaId, Sync, type Replica } from 'foldkit-sync'
|
|
822
|
-
|
|
823
|
-
type Shared = typeof Shared.Type
|
|
824
|
-
type Renamed = typeof Renamed.Type
|
|
825
|
-
|
|
826
|
-
const rename = (replica: Replica<Renamed, Shared>, title: string) =>
|
|
827
|
-
Effect.gen(function* () {
|
|
828
|
-
const storage = yield* Sync.indexedDb('todos-tab-a-clock')
|
|
829
|
-
const clock = yield* Sync.lww.openClock({
|
|
830
|
-
documentId: DocumentId.make('todos'),
|
|
831
|
-
replicaId: ReplicaId.make('tab-a'),
|
|
832
|
-
storage,
|
|
833
|
-
})
|
|
834
|
-
|
|
835
|
-
const shared = yield* replica.shared
|
|
836
|
-
const stamp = yield* clock.next(shared.title.stamp.counter)
|
|
837
|
-
|
|
838
|
-
yield* replica.submit({
|
|
839
|
-
_tag: 'Renamed',
|
|
840
|
-
title: { stamp, value: title },
|
|
841
|
-
})
|
|
842
|
-
|
|
843
|
-
yield* clock.close
|
|
844
|
-
}).pipe(Effect.scoped)
|
|
845
|
-
```
|
|
595
|
+
### Without a Foldkit application
|
|
846
596
|
|
|
847
|
-
`
|
|
848
|
-
|
|
849
|
-
|
|
850
|
-
|
|
851
|
-
only the saved local counter.
|
|
597
|
+
`Sync.define({ documentId, message, shared, empty, durable, replay })` is the
|
|
598
|
+
protocol primitive `forApplication` compiles to, for a client with no
|
|
599
|
+
application to derive from. Its `replay` is unguarded; the author owns its
|
|
600
|
+
agreement with whatever applies the Message elsewhere.
|
|
852
601
|
|
|
853
|
-
|
|
854
|
-
compare-and-swap check; a stale handle fails and should be closed/reopened.
|
|
855
|
-
Storage failures do not return a stamp. Counter gaps after a crash or rejected
|
|
856
|
-
submission are harmless.
|
|
602
|
+
### Joining an assembly
|
|
857
603
|
|
|
858
|
-
|
|
859
|
-
|
|
604
|
+
`TodoSync.wiring()` adds the contract to a `foldkit-bundle` assembly so
|
|
605
|
+
`Module.validate` can report two owners of one field. It routes nothing and
|
|
606
|
+
runs nothing; Sync owns its runtime through `mount`.
|
|
860
607
|
|
|
861
|
-
|
|
862
|
-
the same value is harmless; a different value with the same stamp throws. Keep
|
|
863
|
-
the **whole register, including its winning stamp**, in shared snapshots. For
|
|
864
|
-
deletion, a nullable register can retain a stamped `null` tombstone so an old
|
|
865
|
-
offline write cannot resurrect deleted state.
|
|
608
|
+
## What Sync owns, and does not
|
|
866
609
|
|
|
867
|
-
|
|
868
|
-
|
|
869
|
-
|
|
870
|
-
admission when needed.
|
|
871
|
-
|
|
872
|
-
Specialized sets, counters, collaborative text, and arbitrary commutative
|
|
873
|
-
Messages remain outside this helper.
|
|
874
|
-
|
|
875
|
-
## Lower level: `Sync.define`
|
|
876
|
-
|
|
877
|
-
`Sync.forApplication(App).make(...)` is the Foldkit-facing compiler. It builds
|
|
878
|
-
on `Sync.define`, the lower-level protocol primitive.
|
|
879
|
-
|
|
880
|
-
Use `Sync.define` directly when there is no Foldkit application to derive the
|
|
881
|
-
contract from — for example, a non-Foldkit client or a hand-written shared
|
|
882
|
-
projection:
|
|
883
|
-
|
|
884
|
-
```ts
|
|
885
|
-
import { Effect, Schema } from 'effect'
|
|
886
|
-
import {
|
|
887
|
-
DocumentId,
|
|
888
|
-
ReplicaId,
|
|
889
|
-
Sync,
|
|
890
|
-
type TransportClient,
|
|
891
|
-
} from 'foldkit-sync'
|
|
892
|
-
|
|
893
|
-
const Shared = Schema.Struct({ todos: Schema.Array(Schema.String) })
|
|
894
|
-
const Renamed = Schema.Struct({
|
|
895
|
-
_tag: Schema.Literal('Renamed'),
|
|
896
|
-
title: Schema.String,
|
|
897
|
-
})
|
|
898
|
-
|
|
899
|
-
declare const transport: TransportClient
|
|
900
|
-
|
|
901
|
-
const Protocol = Sync.define({
|
|
902
|
-
documentId: DocumentId.make('todos'),
|
|
903
|
-
message: Renamed,
|
|
904
|
-
shared: Shared,
|
|
905
|
-
empty: { todos: [] },
|
|
906
|
-
durable: () => true,
|
|
907
|
-
replay: (shared, message) => ({
|
|
908
|
-
todos: [...shared.todos, message.title],
|
|
909
|
-
}),
|
|
910
|
-
})
|
|
911
|
-
|
|
912
|
-
const program = Effect.gen(function* () {
|
|
913
|
-
const storage = yield* Sync.indexedDb('todos-tab-1')
|
|
914
|
-
const replica = yield* Protocol.openReplica(
|
|
915
|
-
ReplicaId.make('tab-1'),
|
|
916
|
-
storage,
|
|
917
|
-
)
|
|
918
|
-
|
|
919
|
-
yield* replica.submit({ _tag: 'Renamed', title: 'Milk' })
|
|
920
|
-
|
|
921
|
-
yield* Effect.provide(
|
|
922
|
-
replica.synchronize,
|
|
923
|
-
Sync.transport.fromPromise(transport),
|
|
924
|
-
)
|
|
925
|
-
|
|
926
|
-
return yield* replica.shared
|
|
927
|
-
}).pipe(Effect.scoped)
|
|
928
|
-
```
|
|
610
|
+
Owns the replication contract (shared slice, durable set, replay, codecs,
|
|
611
|
+
`journalContract`), the replica (outbox, base, cursor, optimistic view, status),
|
|
612
|
+
reconciliation, the transport seam, and the optional presence and LWW helpers.
|
|
929
613
|
|
|
930
|
-
|
|
931
|
-
|
|
932
|
-
|
|
933
|
-
|
|
934
|
-
## What Sync owns
|
|
935
|
-
|
|
936
|
-
- **The replication contract.** Shared projection, durable Message subset,
|
|
937
|
-
initial shared value, replay, operation codecs, and `journalContract()`.
|
|
938
|
-
- **The replica.** Persisted outbox, committed base, cursor, optimistic shared
|
|
939
|
-
projection, local operation sequence, synchronization, status, and recovery
|
|
940
|
-
validation.
|
|
941
|
-
- **Reconciliation.** Adopt committed operations/checkpoints, drop
|
|
942
|
-
acknowledged/rejected pending operations, and replay what remains.
|
|
943
|
-
- **The transport seam.** Effect `Transport` plus loopback/promise/WebSocket
|
|
944
|
-
adapters; transport carries exchanges but does not own conflict semantics.
|
|
945
|
-
- **Optional collaboration primitives.** Ephemeral Presence and the specialized
|
|
946
|
-
LWW register/clock helpers.
|
|
947
|
-
|
|
948
|
-
Sync does **not** own:
|
|
949
|
-
|
|
950
|
-
- unrelated local Model fields;
|
|
951
|
-
- server-derived disposable caches (`foldkit-remote`);
|
|
952
|
-
- authoritative server storage/order (`foldkit-durable` or your own server);
|
|
953
|
-
- peer-to-peer CRDT convergence;
|
|
954
|
-
- application Message/shared-state migrations.
|
|
955
|
-
|
|
956
|
-
## How state changes here
|
|
957
|
-
|
|
958
|
-
Durable Messages replay the normal application `update` — which is where
|
|
959
|
-
`modifyFields` lives — so deterministic replay and optimistic application are the
|
|
960
|
-
same code path, not two reducers. Checkpoints and adopted server state
|
|
961
|
-
arrive the other way: structurally installed through the declared writable
|
|
962
|
-
projection. Sync already verifies that durable Messages touch only the
|
|
963
|
-
declared projection; treat any other write as a bug.
|
|
614
|
+
Does not own the rest of the Model, server-derived caches (`foldkit-remote`),
|
|
615
|
+
the authoritative order and storage (`foldkit-durable` or your server),
|
|
616
|
+
peer-to-peer merging, or migration of your own Messages and snapshots.
|
|
964
617
|
|
|
965
618
|
## Limits
|
|
966
619
|
|
|
967
|
-
- IndexedDB is the built-in storage
|
|
968
|
-
|
|
969
|
-
|
|
970
|
-
-
|
|
971
|
-
|
|
972
|
-
Messages commutative.
|
|
973
|
-
- There is no operational transform / collaborative text algorithm.
|
|
974
|
-
- Binding the socket transport and presence server to a platform WebSocket
|
|
975
|
-
server remains application/platform glue. The sync example demonstrates a
|
|
976
|
-
`ws` transport binding; presence over that server is not wired there today.
|
|
977
|
-
- Application schema migration remains application policy. Sync versions and
|
|
978
|
-
validates its own persisted/wire envelope.
|
|
620
|
+
- IndexedDB is the only built-in storage; `Storage` is three members to
|
|
621
|
+
implement.
|
|
622
|
+
- One server order per document; no CRDT and no collaborative text algorithm.
|
|
623
|
+
- Binding the socket transport to a platform WebSocket server is glue the
|
|
624
|
+
examples show with `ws`.
|
|
979
625
|
|
|
980
626
|
## Older spellings
|
|
981
627
|
|
|
982
|
-
`Sync` groups the
|
|
983
|
-
|
|
984
|
-
|
|
985
|
-
|
|
986
|
-
|
|
987
|
-
|
|
988
|
-
|
|
989
|
-
|
|
990
|
-
- `createPresence`, `createPresenceHub`, `servePresence`,
|
|
991
|
-
`socketPresenceChannel`, `loopbackPresenceChannel` -> `Sync.presence.make`,
|
|
992
|
-
`.hub`, `.serve`, `.socketChannel`, `.loopbackChannel`;
|
|
993
|
-
- `lwwRegister`, `openLwwClock` -> `Sync.lww.register`, `Sync.lww.openClock`;
|
|
994
|
-
- `documentId('todos')` and `DocumentId.make('todos')` produce the same branded
|
|
995
|
-
value, and likewise for `replicaId`, `opId`, `sequence`, and
|
|
996
|
-
`localSequence`.
|
|
628
|
+
`Sync` groups the exports and wraps nothing; every name is also exported on its
|
|
629
|
+
own with the same signature: `defineSync` is `Sync.define`, `syncMetrics` is
|
|
630
|
+
`Sync.metrics`, `layerSocket`/`layerLoopback`/`layerFromPromise`/`toPromise`/
|
|
631
|
+
`serveSocket`/`nativeSocket` are `Sync.transport.*`, `createPresence`/
|
|
632
|
+
`createPresenceHub`/`servePresence`/`socketPresenceChannel`/
|
|
633
|
+
`loopbackPresenceChannel` are `Sync.presence.*`, and `lwwRegister`/`openLwwClock`
|
|
634
|
+
are `Sync.lww.*`. `documentId('todos')` and `DocumentId.make('todos')` make the
|
|
635
|
+
same branded value, as do the other id constructors.
|
|
997
636
|
|
|
998
637
|
## See also
|
|
999
638
|
|
|
1000
|
-
- [Replicated state](../../docs/replication.md)
|
|
1001
|
-
|
|
1002
|
-
- [
|
|
1003
|
-
|
|
1004
|
-
|
|
1005
|
-
|
|
1006
|
-
|
|
1007
|
-
|
|
1008
|
-
|
|
1009
|
-
authorization, Mirror, and agent integration.
|
|
1010
|
-
- [`examples/sync`](../../examples/sync) — focused replica/server/transport trace.
|
|
639
|
+
- [Replicated state](../../docs/replication.md): both halves end to end, with
|
|
640
|
+
the server's exchange handler.
|
|
641
|
+
- [`foldkit-durable`](../durable): the journal and what it guarantees.
|
|
642
|
+
- [Runtime binding](../../docs/sync-runtime-binding.md): what `Sync.mount` does
|
|
643
|
+
inside the Foldkit runtime, for the curious.
|
|
644
|
+
- [`examples/todo-app`](../../examples/todo-app): a whole application over
|
|
645
|
+
`Sync.mount`, with fragments, authorization, mirrors, and an agent.
|
|
646
|
+
- [`examples/sync`](../../examples/sync): the replica, the journal, the socket,
|
|
647
|
+
and a test for every failure above.
|