foldkit-sync 0.2.0 → 0.3.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,49 +1,300 @@
1
1
  # `foldkit-sync`
2
2
 
3
- A local-first replica of a shared Foldkit projection. The application's Message
4
- union and `update` stay authoritative; sync wraps them rather than introducing a
5
- second reducer.
3
+ Keeps part of a Foldkit application's Model working offline and converging with
4
+ a server. An edit applies to the local Model immediately, is written to a
5
+ persistent outbox, and is sent to the server when the network allows. The
6
+ server decides the order; every replica replays that order and ends up with the
7
+ same state.
6
8
 
7
- It is the **client half** of [replicated Foldkit state](https://github.com/doeixd/foldkit-plus/blob/main/docs/replication.md).
8
- Reach for it when clients must keep working offline and converge later, with
9
- `foldkit-durable` (or any server that orders operations) as the authority. The
10
- [guide](https://github.com/doeixd/foldkit-plus/blob/main/docs/replication.md) covers the mental model and when not to use
11
- it.
9
+ The trick is that it is still your application. You declare which Model fields
10
+ replicate and which Messages are durable; the shared codec, the initial
11
+ snapshot, the replay reducer, and the server journal's contract are all derived
12
+ from the `update` you already wrote. There is no second reducer to keep in sync
13
+ with the first.
14
+
15
+ **Use it when** clients must keep working offline, several devices or tabs edit
16
+ the same document, and you have (or can run) a server that orders operations —
17
+ [`foldkit-durable`](https://github.com/doeixd/foldkit-plus/tree/main/packages/durable)
18
+ is one. **Not for** peer-to-peer collaboration: this is server-ordered and
19
+ single-writer-per-document, not a CRDT mesh. Concurrent edits to the same field
20
+ resolve by server arrival order unless you opt that field into the
21
+ [last-writer-wins register](#last-writer-wins-fields-experimental) below, and
22
+ there is no operational transform for collaborative text.
23
+
24
+ It is the **client half** of [replicated Foldkit state](https://github.com/doeixd/foldkit-plus/blob/main/docs/replication.md);
25
+ the [guide](https://github.com/doeixd/foldkit-plus/blob/main/docs/replication.md)
26
+ covers the mental model and when not to use it.
27
+
28
+ ## Install
29
+
30
+ ```bash
31
+ pnpm add foldkit-sync
32
+ ```
33
+
34
+ `foldkit` and `effect` are peer dependencies; `foldkit-surface` comes with it.
35
+ `foldkit-durable` is the server half.
36
+
37
+ ## Quick start
38
+
39
+ `forApplication` is the Foldkit-facing layer. From one `Surface.application`
40
+ and a writable projection it derives the shared schema, the durable Message
41
+ subset, the initial snapshot, and replay, and exposes a read-only Surface over
42
+ the same projection.
43
+
44
+ ```ts
45
+ import { Schema } from 'effect'
46
+ import { defineMessageUnion } from 'foldkit/message'
47
+ import type * as Update from 'foldkit/update'
48
+ import { MessageSet, Projection, Surface } from 'foldkit-surface'
49
+ import { documentId, forApplication } from 'foldkit-sync'
50
+
51
+ const Model = Schema.Struct({
52
+ todos: Schema.Array(Schema.Struct({ id: Schema.String, title: Schema.String })),
53
+ selectedTodoId: Schema.NullOr(Schema.String),
54
+ lastError: Schema.NullOr(Schema.String),
55
+ })
56
+ type Model = typeof Model.Type
57
+
58
+ const Message = defineMessageUnion({
59
+ CreatedTodo: { id: Schema.String, title: Schema.String },
60
+ RenamedTodo: { id: Schema.String, title: Schema.String },
61
+ SelectedTodo: { id: Schema.String },
62
+ })
63
+ type Message = typeof Message.Type
64
+
65
+ const initial: Model = { todos: [], selectedTodoId: null, lastError: null }
66
+
67
+ type Return = Update.Return<Model, Message>
68
+ // The application's ordinary transition. Sync does not add a second one.
69
+ const update = (model: Model, message: Message): Return =>
70
+ Message.match<Return>(message, {
71
+ CreatedTodo: ({ id, title }) => ({
72
+ model: { ...model, todos: [...model.todos, { id, title }] },
73
+ }),
74
+ RenamedTodo: ({ id, title }) => ({
75
+ model: {
76
+ ...model,
77
+ todos: model.todos.map(todo => (todo.id === id ? { ...todo, title } : todo)),
78
+ },
79
+ }),
80
+ SelectedTodo: ({ id }) => ({ model: { ...model, selectedTodoId: id } }),
81
+ })
82
+
83
+ const App = Surface.application({ Model, Message, initial, update })
84
+
85
+ const TodoSync = forApplication(App).make({
86
+ documentId: documentId('todos'),
87
+ shared: Projection.pick(App.fields.todos), // the codec, read, and write
88
+ durable: MessageSet.make(App, [Message.CreatedTodo, Message.RenamedTodo]),
89
+ })
90
+ ```
91
+
92
+ Only `todos` is replicated; `selectedTodoId` and `lastError` stay local. The
93
+ derived `replay` installs the shared slice into the application's initial Model,
94
+ applies the durable Message through `update`, and reads the shared slice back.
95
+
96
+ A durable Message must therefore be a deterministic, state-only transition of
97
+ the shared projection. Replay refuses one whose `update` returns a Command (a
98
+ live effect cannot be replayed) or changes a field outside the projection (the
99
+ change would be silently lost), naming the Message and the fields. A Message that
100
+ needs an effect stays local and emits a durable fact once the effect settles:
101
+ `RequestedChargeCard` runs the Command; `CardCharged` is what replicates.
102
+ Pass `replay` to `make` when an application needs a custom reducer over the
103
+ shared slice; a custom replay is not guarded.
104
+
105
+ On the server, the same contract produces the journal's codecs and reducer, so
106
+ neither is written twice:
12
107
 
13
108
  ```ts
14
109
  import { Effect } from 'effect'
15
- import { defineSync, indexedDb, layerFromPromise } from 'foldkit-sync'
110
+ import { actorId, documentId, makeJournal, opId } from 'foldkit-durable'
16
111
 
17
- const Sync = defineSync({ ... })
112
+ type Principal = { readonly actorId: string }
18
113
 
19
- const replica = await Effect.runPromise(
20
- Effect.gen(function* () {
21
- const storage = yield* indexedDb('todos-tab-1')
22
- return yield* Sync.openReplica('tab-1', storage)
114
+ const server = Effect.gen(function* () {
115
+ const journal = yield* makeJournal({
116
+ // Operation and snapshot codecs, `empty`, `reduce`, and `authorize` if declared.
117
+ ...TodoSync.journalContract(),
118
+ file: 'todos.sqlite',
119
+ // The two packages brand their ids separately, so re-brand at the seam.
120
+ opId: operation => opId(operation.opId),
121
+ actorId: (principal: Principal) => actorId(principal.actorId),
122
+ })
123
+ return yield* journal.load(documentId('todos'))
124
+ }).pipe(Effect.scoped)
125
+ ```
126
+
127
+ ### Mounting
128
+
129
+ `mount` runs the application over an open replica with one reducer. A durable
130
+ Message is applied at once through the application's `update` and persisted
131
+ afterwards in a Command; the shared slice is re-installed from the replica when
132
+ an exchange or a rejection changes it, or when a persist fails.
133
+
134
+ ```ts
135
+ import { Effect, Scope } from 'effect'
136
+ import { indexedDb, layerSocket, mount, replicaId } from 'foldkit-sync'
137
+
138
+ const container = document.getElementById('app')! // Foldkit needs the element to have an id
139
+
140
+ const scope = Effect.runSync(Scope.make())
141
+ const storage = Effect.runSync(Effect.provideService(indexedDb('todos/tab-1'), Scope.Scope, scope))
142
+ const replica = Effect.runSync(TodoSync.openReplica(replicaId('tab-1'), storage))
143
+
144
+ const mounted = mount(App, TodoSync, {
145
+ replica,
146
+ container,
147
+ view: (model, h) => ({
148
+ title: 'Todos',
149
+ body: h.ul(
150
+ [],
151
+ model.todos.map(todo => h.li([], [todo.title])),
152
+ ),
23
153
  }),
24
- )
25
- await Effect.runPromise(
26
- replica.submit(Message.CreatedTodo({ id: crypto.randomUUID(), title: 'Milk' })),
27
- )
28
- await Effect.runPromise(Effect.provide(replica.synchronize, layerFromPromise(transport)))
154
+ onPersistenceFailure: (model, error) => ({ ...model, lastError: error.message }),
155
+ })
156
+
157
+ // The exchange loop: once, then after every submit, retrying a failed exchange.
158
+ Effect.runFork(Effect.provide(replica.start, layerSocket({ url: 'wss://example.com/sync' })))
29
159
 
30
- const shared = Effect.runSync(replica.shared)
160
+ mounted.dispatch(Message.CreatedTodo({ id: crypto.randomUUID(), title: 'Milk' }))
161
+ mounted.model() // the Model after the last transition
162
+ await mounted.dispose() // waits for in-flight persists; the replica stays open
163
+ ```
164
+
165
+ The optional `url` option routes the URL through the application:
166
+ `init(model, url)` reduces it into the Model before the first render,
167
+ `onUrlChange(url)` names the Message for every navigation, and `onUrlRequest`
168
+ the Message for a link click (omitted, the mount follows the link itself: an
169
+ internal one pushed, an external one loaded). A `foldkit-mirror` URL mirror
170
+ plugs in there as
171
+ `url: { init: (model, url) => Filters.reduce(model, url), onUrlChange: url => Message.UrlChanged({ url }) }`;
172
+ `subscriptions` and `resources` carry its write entry and the store layer.
173
+
174
+ `mounted.model`, `mounted.dispatch`, `mounted.subscribe`, and `mounted.observe`
175
+ are the host an agent binds to; `observe` reports every application Message the
176
+ runtime applies, which a capability with a `completion` contract needs. See
177
+ [docs/sync-runtime-binding.md](https://github.com/doeixd/foldkit-plus/blob/main/docs/sync-runtime-binding.md) for what the
178
+ mount guarantees and why no Foldkit change is required.
179
+
180
+ ### Fragments
181
+
182
+ A large application declares one fragment per feature and composes them. `App`
183
+ below is a wider application than the quick start's: it also has a `members`
184
+ field and an `Invited` Message.
185
+
186
+ ```ts
187
+ const Sync = forApplication(App)
188
+ const Todos = Sync.fragment({
189
+ shared: Projection.pick(App.fields.todos),
190
+ durable: MessageSet.make(App, [Message.CreatedTodo, Message.RenamedTodo]),
191
+ })
192
+ const Members = Sync.fragment({
193
+ shared: Projection.pick(App.fields.members),
194
+ durable: MessageSet.make(App, [Message.Invited]),
195
+ })
196
+ const Board = Sync.make({ documentId: documentId('board'), ...Sync.compose(Todos, Members) })
197
+ ```
198
+
199
+ `compose` merges the shared projections and the durable subsets and infers the
200
+ merged shape. A field declared twice with a different codec, a Message declared
201
+ durable twice, or a fragment from another application throws. Two contracts
202
+ over the same field are a different mistake, which `Module.validate` reports.
203
+
204
+ ### Authorization
205
+
206
+ Policy attaches to the contract per durable variant and compiles into the
207
+ journal contract, so the server applies it with no glue. This is the same
208
+ `Board` as above, with a principal fixed and one rule added:
209
+
210
+ ```ts
211
+ const Authorized = forApplication(App).withPrincipal<{ readonly role: 'admin' | 'guest' }>()
212
+ const Board = Authorized.make({
213
+ documentId: documentId('board'),
214
+ ...Authorized.compose(Todos, Members),
215
+ authorize: {
216
+ // `message` is exactly `RenamedTodo`; `shared` is the authoritative snapshot.
217
+ RenamedTodo: ({ principal, message, shared }) =>
218
+ principal.role === 'admin' && shared.todos.some(todo => todo.id === message.id),
219
+ },
220
+ })
221
+ ```
222
+
223
+ `Board.journalContract()` then carries the compiled rules in the shape
224
+ `makeJournal` takes. A durable variant without a rule is allowed; a key that is
225
+ not a durable tag is a compile error. The rule runs inside the append
226
+ transaction against the snapshot, so keep it synchronous and local. The client
227
+ does not run it: a replica has no principal, and a refused operation comes back
228
+ as a rejection.
229
+
230
+ ### Lower level
231
+
232
+ `forApplication(App).make` compiles down to `defineSync`, the protocol
233
+ primitive. Use `defineSync` directly when there is no Foldkit application to
234
+ derive the contract from — a non-Foldkit client, or a hand-written projection.
235
+
236
+ ```ts
237
+ import { Effect, Schema } from 'effect'
238
+ import {
239
+ defineSync,
240
+ documentId,
241
+ indexedDb,
242
+ layerFromPromise,
243
+ replicaId,
244
+ type TransportClient,
245
+ } from 'foldkit-sync'
246
+
247
+ const Shared = Schema.Struct({ todos: Schema.Array(Schema.String) })
248
+ const Renamed = Schema.Struct({ _tag: Schema.Literal('Renamed'), title: Schema.String })
249
+ declare const transport: TransportClient // the application's own server client
250
+
251
+ const Protocol = defineSync({
252
+ documentId: documentId('todos'),
253
+ message: Renamed,
254
+ shared: Shared,
255
+ empty: { todos: [] },
256
+ durable: () => true,
257
+ replay: (shared, message) => ({ todos: [...shared.todos, message.title] }),
258
+ })
259
+
260
+ const program = Effect.gen(function* () {
261
+ const storage = yield* indexedDb('todos-tab-1')
262
+ const replica = yield* Protocol.openReplica(replicaId('tab-1'), storage)
263
+ yield* replica.submit({ _tag: 'Renamed', title: 'Milk' })
264
+ yield* Effect.provide(replica.synchronize, layerFromPromise(transport))
265
+ return yield* replica.shared
266
+ }).pipe(Effect.scoped)
31
267
  ```
32
268
 
33
269
  ## What it owns
34
270
 
271
+ - The Foldkit-facing contract (`forApplication(App).make`, optionally with
272
+ a custom `replay`): derives the shared projection, the durable Message subset, the
273
+ initial snapshot, and the journal contract from the application, so none is
274
+ declared twice. `Projection.pick`/`Projection.compose` build the writable projection;
275
+ `TodoSync.journalContract()` builds the durable codecs and reducer.
35
276
  - The operation envelope: `replicaId:localSequence` identity, `baseCursor`, and
36
- protocol/schema versions.
277
+ protocol/schema versions. The contract's `codec` (`normalizeOperation`,
278
+ `operationFrom`, `committedFrom`, `decodeExchange`) is there for a custom
279
+ transport or server; a server-committed operation is a `CommittedOperation`.
37
280
  - A persisted outbox and optimistic projection: a durable Message is visible
38
281
  immediately and rebased onto the authoritative order.
39
282
  - Reconciliation: committed operations are replayed in order, acknowledged or
40
283
  refused pending entries are dropped, and a checkpoint replaces the compacted
41
- prefix.
284
+ prefix. `replica.start` is the exchange loop: fork it to exchange once and then
285
+ after every `submit`, with a transport failure recorded in `status.lastError`
286
+ and retried on the next wake, so the application does not hand-roll it.
42
287
  - A redacted status (`replica.status`): the pending count, the cursor, the last
43
288
  exchange failure, and the operations the server refused — enough for a UI to
44
- explain and recover without exposing Messages or the Model.
289
+ explain and recover without exposing Messages or the Model. `statusChanges` is
290
+ the same status, emitted on subscribe and re-emitted after every submit and
291
+ exchange. `changes` carries the status and the optimistic shared value together
292
+ from one read, so a UI holds one subscription instead of two.
45
293
  - Strict decoding: an operation is always validated with the application's
46
294
  Message schema, and a Message the contract does not call durable is refused.
295
+ - Branded positions: `Sequence` (a committed document position) and
296
+ `LocalSequence` (a replica's own 1-based counter) cannot be confused, so a
297
+ client counter is never passed where a committed position is expected.
47
298
  - Presence (`createPresence`): an ephemeral, TTL'd peer registry, deliberately
48
299
  outside the durable log. A peer that stops refreshing is dropped, not
49
300
  replayed. It is an Effect driven by the `Clock` (so a `TestClock` makes the
@@ -52,7 +303,9 @@ const shared = Effect.runSync(replica.shared)
52
303
  so a hostile peer cannot inject a value your `Update` type does not describe.
53
304
  Presence can travel over a socket — `socketPresenceChannel` on the client and
54
305
  `servePresence` fanning through a `createPresenceHub` on the server — or
55
- in-process via `loopbackPresenceChannel`.
306
+ in-process via `loopbackPresenceChannel`. `presence.changes` is a `Stream` that
307
+ emits the current peers and re-emits them on every change, alongside the
308
+ `subscribe` callback.
56
309
  - The transport seam (`Transport`): an Effect service with a loopback layer, a
57
310
  bridge to and from the promise client the replica speaks, and a WebSocket
58
311
  client layer. The socket reconnects on an exponential, jittered backoff and
@@ -72,7 +325,7 @@ const shared = Effect.runSync(replica.shared)
72
325
  server is left to the application; the sync example shows a `ws` one for the
73
326
  transport (presence over `ws` is not wired there yet).
74
327
 
75
- ## Last-writer-wins fields (M8, experimental)
328
+ ## Last-writer-wins fields (experimental)
76
329
 
77
330
  Use `lwwRegister` when a field's winner should depend on a write's logical time
78
331
  instead of the order offline clients reconnect. It returns a schema and a pure
@@ -80,7 +333,7 @@ instead of the order offline clients reconnect. It returns a schema and a pure
80
333
 
81
334
  ```ts
82
335
  import { Schema } from 'effect'
83
- import { lwwRegister } from 'foldkit-sync'
336
+ import { lwwRegister, replicaId } from 'foldkit-sync'
84
337
 
85
338
  const Title = lwwRegister(Schema.String)
86
339
  const Shared = Schema.Struct({ title: Title.schema })
@@ -93,7 +346,7 @@ const update = (model: typeof Shared.Type, message: typeof Renamed.Type) => ({
93
346
 
94
347
  const message: typeof Renamed.Type = {
95
348
  _tag: 'Renamed',
96
- title: { stamp: { counter: 1, replicaId: 'tab-a' }, value: 'Milk' },
349
+ title: { stamp: { counter: 1, replicaId: replicaId('tab-a') }, value: 'Milk' },
97
350
  }
98
351
  ```
99
352
 
@@ -108,38 +361,38 @@ write cannot cause timestamp reuse after reload:
108
361
 
109
362
  ```ts
110
363
  import { Effect } from 'effect'
111
- import { indexedDb, openLwwClock } from 'foldkit-sync'
364
+ import { documentId, indexedDb, openLwwClock, replicaId, type Replica } from 'foldkit-sync'
365
+
366
+ type Shared = typeof Shared.Type
367
+ type Renamed = typeof Renamed.Type
112
368
 
113
- const clock = await Effect.runPromise(
369
+ const rename = (replica: Replica<Renamed, Shared>, title: string) =>
114
370
  Effect.gen(function* () {
371
+ // A separate database from the replica's outbox; one clock per document/writer.
115
372
  const storage = yield* indexedDb('todos-tab-a-clock')
116
- return yield* openLwwClock({ documentId: 'todos', replicaId: 'tab-a', storage })
117
- }),
118
- )
119
- try {
120
- const stamp = await Effect.runPromise(
121
- clock.next(Effect.runSync(replica.shared).title.stamp.counter),
122
- )
123
- await Effect.runPromise(
124
- replica.submit({ _tag: 'Renamed', title: { stamp, value: 'Milk' } }),
125
- )
126
- } finally {
127
- await Effect.runPromise(clock.close)
128
- }
373
+ const clock = yield* openLwwClock({
374
+ documentId: documentId('todos'),
375
+ replicaId: replicaId('tab-a'),
376
+ storage,
377
+ })
378
+ const shared = yield* replica.shared
379
+ const stamp = yield* clock.next(shared.title.stamp.counter)
380
+ yield* replica.submit({ _tag: 'Renamed', title: { stamp, value: title } })
381
+ yield* clock.close
382
+ }).pipe(Effect.scoped)
129
383
  ```
130
384
 
131
- Use a **separate database** from the replica's outbox, with one clock per
132
- document/writer. `next(observedCounter)` returns only after persisting a counter
133
- greater than both its saved value and the supplied observation. Pass the latest
134
- observed counter for the field being edited; when causality spans several
135
- registers, pass their maximum. Calling `next()` without an observation advances
136
- only the saved local counter. The clock does not inspect application state.
385
+ `next(observedCounter)` returns only after persisting a counter greater than
386
+ both its saved value and the supplied observation. Pass the latest observed
387
+ counter for the field being edited; when causality spans several registers, pass
388
+ their maximum. Calling `next()` without an observation advances only the saved
389
+ local counter. The clock does not inspect application state.
137
390
 
138
391
  Allocations on one handle are serialized. Separate handles use storage's
139
392
  compare-and-swap check; a stale handle fails and must be closed and reopened.
140
393
  Storage failures reject allocation without returning a stamp. Counter gaps after
141
394
  a crash or failed submission are harmless. Counters are nonnegative safe
142
- integers; exhaustion fails before writing. `close()` waits for accepted
395
+ integers; exhaustion fails before writing. `close` waits for accepted
143
396
  allocations and refuses new ones. Opening takes ownership of storage and closes
144
397
  it on initialization failure too.
145
398
 
@@ -166,6 +419,16 @@ identity. Applications must enforce writer ownership and clock policy at admissi
166
419
  where needed. A losing write can still trigger a Command if the application's
167
420
  update produces one; this helper only resolves state.
168
421
 
169
- These M8 helpers do not change the todo example's persisted schema or claim
170
- arbitrary Messages commute. Specialized sets, counters and collaborative text
171
- remain future work.
422
+ These helpers do not claim arbitrary Messages commute. Specialized sets,
423
+ counters, and collaborative text remain future work.
424
+
425
+ ## See also
426
+
427
+ - [Replicated state](https://github.com/doeixd/foldkit-plus/blob/main/docs/replication.md) — the mental model, and
428
+ when not to use it.
429
+ - [Runtime binding](https://github.com/doeixd/foldkit-plus/blob/main/docs/sync-runtime-binding.md) — what
430
+ `mount` guarantees, and why no Foldkit change is required.
431
+ - [`foldkit-durable`](https://github.com/doeixd/foldkit-plus/tree/main/packages/durable) — the server half; [`foldkit-mirror`](https://github.com/doeixd/foldkit-plus/tree/main/packages/mirror) plugs into the mount's
432
+ `url` option.
433
+ - [`examples/todo-app`](https://github.com/doeixd/foldkit-plus/tree/main/examples/todo-app) — two fragments, owner-only rules, and a browser mount;
434
+ [`examples/sync`](https://github.com/doeixd/foldkit-plus/tree/main/examples/sync) is the in-process trace.