foldkit-sync 0.15.0 → 0.17.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (49) hide show
  1. package/README.md +567 -41
  2. package/dist/diagnostics.d.ts +74 -3
  3. package/dist/diagnostics.d.ts.map +1 -1
  4. package/dist/diagnostics.js +60 -4
  5. package/dist/diagnostics.js.map +1 -1
  6. package/dist/do.d.ts +49 -7
  7. package/dist/do.d.ts.map +1 -1
  8. package/dist/do.js +91 -8
  9. package/dist/do.js.map +1 -1
  10. package/dist/errors.d.ts +92 -1
  11. package/dist/errors.d.ts.map +1 -1
  12. package/dist/errors.js +87 -1
  13. package/dist/errors.js.map +1 -1
  14. package/dist/index.d.ts +9 -5
  15. package/dist/index.d.ts.map +1 -1
  16. package/dist/index.js +6 -2
  17. package/dist/index.js.map +1 -1
  18. package/dist/journal.d.ts +52 -27
  19. package/dist/journal.d.ts.map +1 -1
  20. package/dist/journal.js +176 -64
  21. package/dist/journal.js.map +1 -1
  22. package/dist/make.d.ts +48 -17
  23. package/dist/make.d.ts.map +1 -1
  24. package/dist/make.js +11 -4
  25. package/dist/make.js.map +1 -1
  26. package/dist/migrations.d.ts +37 -0
  27. package/dist/migrations.d.ts.map +1 -0
  28. package/dist/migrations.js +56 -0
  29. package/dist/migrations.js.map +1 -0
  30. package/dist/mount.d.ts +1 -1
  31. package/dist/mount.d.ts.map +1 -1
  32. package/dist/mount.js +12 -6
  33. package/dist/mount.js.map +1 -1
  34. package/dist/refusal.d.ts +33 -0
  35. package/dist/refusal.d.ts.map +1 -0
  36. package/dist/refusal.js +47 -0
  37. package/dist/refusal.js.map +1 -0
  38. package/dist/serverJournal.d.ts +143 -0
  39. package/dist/serverJournal.d.ts.map +1 -0
  40. package/dist/serverJournal.js +503 -0
  41. package/dist/serverJournal.js.map +1 -0
  42. package/dist/sync.d.ts +150 -14
  43. package/dist/sync.d.ts.map +1 -1
  44. package/dist/sync.js +273 -64
  45. package/dist/sync.js.map +1 -1
  46. package/dist/transport.d.ts +10 -6
  47. package/dist/transport.d.ts.map +1 -1
  48. package/dist/transport.js.map +1 -1
  49. package/package.json +11 -8
package/README.md CHANGED
@@ -93,7 +93,7 @@ import { defineMessageUnion } from 'foldkit/message'
93
93
  import { modifyFields } from 'foldkit/struct'
94
94
  import type * as Update from 'foldkit/update'
95
95
  import { MessageSet, Projection, Surface } from 'foldkit-surface'
96
- import { DocumentId, Sync } from 'foldkit-sync'
96
+ import { DocumentId, forApplication } from 'foldkit-sync'
97
97
 
98
98
  const Model = Schema.Struct({
99
99
  todos: Schema.Array(Schema.Struct({ id: Schema.String, title: Schema.String })),
@@ -130,7 +130,7 @@ const update = (model: Model, message: Message): Return =>
130
130
 
131
131
  const App = Surface.application({ Model, Message, initial, update })
132
132
 
133
- const TodoSync = Sync.forApplication(App).make({
133
+ const TodoSync = forApplication(App).make({
134
134
  documentId: DocumentId.make('todos'),
135
135
  shared: Projection.pick(App.model.todos),
136
136
  durable: MessageSet.make(App, [Message.CreatedTodo, Message.RenamedTodo]),
@@ -296,6 +296,11 @@ the exchange by hand, with an effect policy per committed operation.
296
296
 
297
297
  A server that is not Durable implements the same [exchange](#the-exchange).
298
298
 
299
+ Import what a server calls by name (`forApplication`, `serveJournal`,
300
+ `servePresence`), not through the `Sync` namespace object: a bundler keeps
301
+ every member of an object, so `Sync.forApplication` in a server's bundle also
302
+ carries the client's mount, sockets, presence and IndexedDB code.
303
+
299
304
  ## Serving from a Durable Object
300
305
 
301
306
  On Cloudflare, one object per document is the single writer the journal
@@ -303,7 +308,7 @@ wants. `defineDocumentHost` from `foldkit-sync/do` defines that class: each
303
308
  accepted socket exchanges against the same journal through `serveJournal`,
304
309
  and the replica side is the socket transport it already speaks:
305
310
 
306
- ```ts
311
+ ```ts sketch
307
312
  import { DurableObject } from 'cloudflare:workers'
308
313
  import { defineDocumentHost } from 'foldkit-sync/do'
309
314
 
@@ -315,7 +320,7 @@ export class SyncHost extends defineDocumentHost(DurableObject, {
315
320
  }) {}
316
321
  ```
317
322
 
318
- ```ts
323
+ ```ts sketch
319
324
  // The worker: one object per document.
320
325
  export default {
321
326
  fetch: (request: Request, env: Env): Promise<Response> =>
@@ -346,6 +351,23 @@ Sync.transport.socket({ url: 'wss://example.com/sync' })
346
351
  request, while the socket is open: a refusal closes it with 4401, and a
347
352
  principal whose `key` (`Policy.keyOf`) changed with 1012, which the
348
353
  replica reconnects after. Without it a socket serves its first principal until it closes.
354
+ - `server: { principal, reactions: env => [...], runsPerPass? }` gives the
355
+ object its [reactions](#work-after-a-commit-reactions): registered when the
356
+ journal opens, before any socket is served; the alarm set before the reply
357
+ to an exchange that committed, a pass run after it, and `alarm()` running
358
+ them and setting the alarm from what they answer, then calling the base
359
+ class's own `alarm()` if it has one. The reactions' part never throws, so
360
+ the runtime's few retries of a throwing alarm are never what a run depends
361
+ on; it runs one pass at a time, and only ever brings the object's one alarm
362
+ earlier. A reaction declared for another contract is refused when the
363
+ object is made. A `setAlarm` that fails after a commit is reported, the
364
+ application's `settle` still runs, and the next exchange arms again.
365
+ A pass that broke is a `HostDefect` (`Alarm`), and the alarm comes back in
366
+ 30 seconds. The base class's `alarm()` shares that one alarm: it runs on
367
+ every wake, so it decides for itself what is due, and sets its own next
368
+ wake with `armAlarm(ctx.storage, at)` from `foldkit-sync/do`, which only
369
+ ever brings the alarm earlier. A subclass that overrides `alarm()` calls
370
+ `super.alarm()`, or the reactions stop being resumed.
349
371
  - `settle: (env, journal) => Effect` runs after each exchange's appends, before
350
372
  the reply is read, to apply what committed elsewhere — the Remote tables a
351
373
  reader then queries. `journal.recover` makes it crash-safe and idempotent;
@@ -420,16 +442,19 @@ error names why: `ReplayError` for a Message that cannot be durable,
420
442
  **Committed** is the replica's status:
421
443
 
422
444
  ```ts
423
- const status = yield* replica.status
424
- // { pending: number, cursor: Sequence, lastError: string | undefined, rejected: ReadonlyArray<Rejection> }
445
+ const status = Effect.runSync(replica.status)
446
+ // { pending, cursor, lastError: string | undefined, rejected: ReadonlyArray<Rejection>, refused, epoch }
425
447
  ```
426
448
 
427
449
  `pending` above zero means edits the server has not confirmed; `lastError` is
428
450
  the last failed exchange, cleared by the next success; `rejected` lists recent
429
- edits the server refused, most recent first, each `{ opId, reason, operation }`,
430
- where `reason` is an `Option` of the server's words for the person and
431
- `operation` is the refused operation itself, which the replica has dropped
432
- from the outbox, so a page reads what it changed from here. `replica.statusChanges` is a Stream of the same,
451
+ edits the server refused, most recent first, each `{ opId, refusal, operation, message }`,
452
+ where `refusal` is an `Option` of why (a `SyncRefusal`, below),
453
+ `operation` is the refused operation as it was sent, which the replica has dropped
454
+ from the outbox, and `message` its Message encoded at the current version
455
+ (see [changing a durable Message](#change-a-durable-message)), so a page reads
456
+ what it changed from here. `refused` lists the operations this build cannot
457
+ read at all, kept until `replica.dismiss(opId)`. `replica.statusChanges` is a Stream of the same,
433
458
  emitted now and after every submit and exchange (subscribed before its first
434
459
  read, so a subscriber slow to take one misses no change); `replica.changes`
435
460
  pairs it with the optimistic `shared` value so one subscription sees both.
@@ -438,13 +463,48 @@ pairs it with the optimistic `shared` value so one subscription sees both.
438
463
  error. The replica drops it and replays the rest; the mount re-installs the
439
464
  shared slice, and the field the user edited reverts. `onPersistenceFailure`
440
465
  does not run for this, since the local save succeeded. Watch `rejected` to
441
- explain the revert: `journalExchange` gives each rejection a reason, the
442
- `authorize` rule's own when it gave one. Otherwise it gives a fixed sentence:
443
- "Refused by the server" when `authorize` refused with none (a plain `false`),
444
- and "Not a valid operation" when the operation failed (`validate` refused or
445
- threw, `authorize` threw, or the reducer could not apply it). It never sends
446
- an error's message, which may carry internals, so a rule a person should read
447
- about belongs in `authorize` with a `reason`, not in `validate`.
466
+ explain the revert. Each rejection's `refusal` is a `SyncRefusal`, in Remote's
467
+ vocabulary for "not saved", so a page that also shows Remote's refusals
468
+ handles both with one function:
469
+
470
+ - `Forbidden { reason }`: `authorize` or `admit` refused who sent it, or its
471
+ replica belongs to another actor. `reason` is the rule's own words when it
472
+ gave some (a plain `false` gives none), cut to 500 characters. Remote's
473
+ `Denied.Forbidden` has the same shape.
474
+ - `Invalid { message, entity, rule, path }`: the operation failed, with a fixed
475
+ sentence as `message`. "Not a valid operation" when `validate` refused or
476
+ threw, `authorize` threw, or the reducer could not apply it; "Not an
477
+ operation this server accepts" when it did not decode or named another
478
+ document; "Its id was already used for another" for a reused id.
479
+ `journalExchange` names no Entity, so `entity`, `rule` and `path` are empty;
480
+ the shape is Remote's `Refusal.invalid`.
481
+ - `Unreadable { version }`: written at an older Message version that no step
482
+ of the server's history reads. The replica keeps such an operation in
483
+ `status.refused` as well, with its payload (see
484
+ [changing a durable Message](#change-a-durable-message)).
485
+
486
+ ```ts
487
+ import { Option } from 'effect'
488
+ import { SyncRefusal, type Rejection } from 'foldkit-sync'
489
+
490
+ const said = (rejection: Rejection): string =>
491
+ Option.match(rejection.refusal, {
492
+ onNone: () => 'Refused by the server',
493
+ onSome: SyncRefusal.match({
494
+ Forbidden: ({ reason }) => Option.getOrElse(reason, () => 'You may not change this'),
495
+ Invalid: ({ message }) => message,
496
+ Unreadable: () => 'Made by another version of the app, so not saved',
497
+ }),
498
+ })
499
+ ```
500
+
501
+ `refusal` is none when the server named none (an older or a custom server), or
502
+ named one this build cannot read, which the replica reports as
503
+ `UnknownRefusal` and still treats as rejected. The server never sends an
504
+ error's message, which may carry internals, so a rule a person should read
505
+ about belongs in `authorize` with a `reason`, not in `validate`. The
506
+ deprecated `reason` (the words alone, an `Option<string>`) is still filled
507
+ for one release; read `refusal` instead.
448
508
 
449
509
  **Waiting for the server.** The Model shows an edit before the server has it.
450
510
  Something that must not report success until the edit is committed, an agent
@@ -452,10 +512,14 @@ tool for instance, waits on `mounted.committed`: the shared slice as the server
452
512
  confirmed it, with no pending edit applied. It is a source, `{ get, subscribe }`,
453
513
  told after every exchange:
454
514
 
515
+ <!-- docs-check
516
+ import { Agent } from 'foldkit-agent'
517
+ -->
455
518
  ```ts
456
- completion: Agent.when({
519
+ const completion = Agent.when({
457
520
  source: mounted.committed,
458
- predicate: (shared, request) => shared.todos.some(todo => todo.id === request.id),
521
+ predicate: (shared, request: { readonly id: string }) =>
522
+ shared.todos.some(todo => todo.id === request.id),
459
523
  })
460
524
  ```
461
525
 
@@ -469,8 +533,29 @@ server rejects.
469
533
  Each feature declares the fields it shares and the Messages that change them;
470
534
  `compose` merges them into one document:
471
535
 
536
+ <!-- docs-check standalone
537
+ import { Schema } from 'effect'
538
+ import { defineMessageUnion } from 'foldkit/message'
539
+ import { MessageSet, Projection, Surface } from 'foldkit-surface'
540
+ import { DocumentId, forApplication } from 'foldkit-sync'
541
+ const Model = Schema.Struct({
542
+ todos: Schema.Array(Schema.Struct({ id: Schema.String, title: Schema.String })),
543
+ members: Schema.Array(Schema.String),
544
+ })
545
+ const Message = defineMessageUnion({
546
+ CreatedTodo: { id: Schema.String, title: Schema.String },
547
+ RenamedTodo: { id: Schema.String, title: Schema.String },
548
+ Invited: { email: Schema.String },
549
+ })
550
+ const App = Surface.application({
551
+ Model,
552
+ Message,
553
+ initial: { todos: [], members: [] },
554
+ update: model => ({ model }),
555
+ })
556
+ -->
472
557
  ```ts
473
- const AppSync = Sync.forApplication(App)
558
+ const AppSync = forApplication(App)
474
559
 
475
560
  const Todos = AppSync.fragment({
476
561
  shared: Projection.pick(App.model.todos),
@@ -498,9 +583,9 @@ Policy belongs on the contract, beside the Messages it governs, and runs on the
498
583
  server inside the journal's commit. The client never runs it.
499
584
 
500
585
  ```ts
501
- const Authorized = Sync.forApplication(App).withPrincipal<{ readonly role: 'admin' | 'guest' }>()
586
+ const Authorized = forApplication(App).withPrincipal<{ readonly role: 'admin' | 'guest' }>()
502
587
 
503
- const Board = Authorized.make({
588
+ const GuardedBoard = Authorized.make({
504
589
  documentId: DocumentId.make('board'),
505
590
  ...Authorized.compose(Todos, Members),
506
591
  authorize: {
@@ -511,7 +596,8 @@ const Board = Authorized.make({
511
596
  ```
512
597
 
513
598
  A rule returns `true`, `false`, or `{ allowed: false, reason }`; the reason
514
- is sent to the replica whose edit it was, as its rejection's `reason`.
599
+ is sent to the replica whose edit it was, as its rejection's
600
+ `Forbidden` refusal.
515
601
 
516
602
  `message` is exactly that variant, `shared` is the authoritative snapshot, and
517
603
  `principal` is whatever the server's transport established for the connection;
@@ -528,13 +614,29 @@ A fact carries its own nondeterminism, but a client cannot know where its
528
614
  operation lands in the server's order. A stamp writes it in, on the server, as
529
615
  the journal commits:
530
616
 
617
+ <!-- docs-check standalone
618
+ import { Schema } from 'effect'
619
+ import { defineMessageUnion } from 'foldkit/message'
620
+ import { MessageSet, Projection, Surface } from 'foldkit-surface'
621
+ import { DocumentId, forApplication } from 'foldkit-sync'
622
+ const Model = Schema.Struct({
623
+ prices: Schema.Array(Schema.Struct({ id: Schema.String, cents: Schema.Number, at: Schema.Number })),
624
+ })
625
+ -->
531
626
  ```ts
532
627
  const Message = defineMessageUnion({
533
628
  // `at` is absent on what a client sends: the server's sequence once committed.
534
629
  PriceEdited: { id: Schema.String, cents: Schema.Number, at: Schema.optionalKey(Schema.Number) },
535
630
  })
536
631
 
537
- const Prices = Sync.forApplication(App).make({
632
+ const App = Surface.application({
633
+ Model,
634
+ Message,
635
+ initial: { prices: [] },
636
+ update: model => ({ model }),
637
+ })
638
+
639
+ const Prices = forApplication(App).make({
538
640
  documentId: DocumentId.make('prices'),
539
641
  shared: Projection.pick(App.model.prices),
540
642
  durable: MessageSet.make(App, [Message.PriceEdited]),
@@ -558,13 +660,238 @@ records the last sequence it applied can tell which edits it holds by comparing
558
660
  the two. A client that sends the stamped field itself is claiming a commit it
559
661
  did not get; refuse it in the journal's `validate`.
560
662
 
663
+ ### Messages only the server appends
664
+
665
+ Some facts only the server may record: a payment provider confirmed a
666
+ charge, the table absorbed an edit. Declare those Messages `serverOnly`,
667
+ beside `durable`. A replica stores and replays them like any other; it can
668
+ never submit one:
669
+
670
+ <!-- docs-check standalone
671
+ import { Effect, Option, Schema } from 'effect'
672
+ import { defineMessageUnion } from 'foldkit/message'
673
+ import { modifyFields } from 'foldkit/struct'
674
+ import { MessageSet, Projection, Surface } from 'foldkit-surface'
675
+ import { DocumentId, forApplication } from 'foldkit-sync'
676
+ import type { Journal } from 'foldkit-durable/core'
677
+ import type { Operation, WrittenOperation } from 'foldkit-sync'
678
+ const Order = Schema.Struct({ id: Schema.String, paid: Schema.Boolean, receipted: Schema.Boolean })
679
+ const Model = Schema.Struct({ orders: Schema.Array(Order) })
680
+ type Model = typeof Model.Type
681
+ -->
682
+ ```ts
683
+ const Message = defineMessageUnion({
684
+ PlacedOrder: { id: Schema.String },
685
+ ConfirmedPayment: { id: Schema.String, charge: Schema.String },
686
+ SentReceipt: { id: Schema.String },
687
+ })
688
+
689
+ const mark = (model: Model, id: string, change: Partial<typeof Order.Type>): Model =>
690
+ modifyFields(model, {
691
+ orders: () => model.orders.map(order => (order.id === id ? { ...order, ...change } : order)),
692
+ })
693
+
694
+ const App = Surface.application({
695
+ Model,
696
+ Message,
697
+ initial: { orders: [] },
698
+ update: (model, message) => ({
699
+ model: Message.match(message, {
700
+ PlacedOrder: ({ id }) =>
701
+ modifyFields(model, {
702
+ orders: () => [...model.orders, { id, paid: false, receipted: false }],
703
+ }),
704
+ ConfirmedPayment: ({ id }) => mark(model, id, { paid: true }),
705
+ SentReceipt: ({ id }) => mark(model, id, { receipted: true }),
706
+ }),
707
+ }),
708
+ })
709
+
710
+ const Confirmed = MessageSet.make(App, [Message.ConfirmedPayment])
711
+ const Receipted = MessageSet.make(App, [Message.SentReceipt])
712
+
713
+ const Orders = forApplication(App).make({
714
+ documentId: DocumentId.make('orders'),
715
+ shared: Projection.pick(App.model.orders),
716
+ // What a replica may submit.
717
+ durable: MessageSet.make(App, [Message.PlacedOrder]),
718
+ // What only the server appends. A Message in both sets is a type error.
719
+ serverOnly: MessageSet.union(Confirmed, Receipted),
720
+ })
721
+ ```
722
+
723
+ - **A replica submitting one is a type error**, and refused at run time.
724
+ - **The server refuses one any replica sends**, whoever it says it is:
725
+ every reader of a sent entry (`codec.writtenFrom`, `normalizeOperation`,
726
+ `operationFrom`) refuses it, judged after
727
+ [upcasting](#change-a-durable-message), before `admit`, `validate` or
728
+ `authorize` run. `journalExchange` rejects it by its id as `Forbidden`
729
+ ("Only the server makes this change") and reports a `ServerOnlySubmitted`
730
+ diagnostic; the edits beside it still go. No rule about who the principal
731
+ is is involved, so a resolver's mistake cannot let one through.
732
+ - **Replica ids under `server/` belong to the server**: a sent entry naming
733
+ one is rejected as `Invalid`, and so is one whose replica id is longer than
734
+ 128 characters.
735
+ - The contract's `surface` still lists `durable` alone, so an agent bound to
736
+ it cannot dispatch a server-only Message; `authorize` and `stamp` rules may
737
+ name one.
738
+
739
+ The server appends one through `serverJournal` from `foldkit-sync/journal`,
740
+ as a principal it holds in code and never resolves from a request:
741
+
742
+ <!-- docs-check
743
+ declare const journal: Journal<Operation, { readonly orders: Model['orders'] }, { readonly actorId: string }, WrittenOperation>
744
+ -->
745
+ ```ts
746
+ import { serverJournal } from 'foldkit-sync/journal'
747
+
748
+ const serving = Effect.gen(function* () {
749
+ const server = yield* serverJournal({ sync: Orders, journal, server: { actorId: 'orders' } })
750
+ // A payment provider's webhook: its event id names the fact.
751
+ return (event: { readonly id: string; readonly order: string; readonly charge: string }) =>
752
+ server.appendAsServer(Message.ConfirmedPayment({ id: event.order, charge: event.charge }), {
753
+ key: `payment/${event.id}`,
754
+ })
755
+ })
756
+ ```
757
+
758
+ - **`key` names the fact**, and is required: a webhook's event id, a clock
759
+ tick. A repeat with the same key and Message appends nothing new, so a
760
+ redelivered webhook is harmless; the same key with another Message fails
761
+ with `ServerKeyReusedError`, and the first fact stands. A key is a
762
+ non-empty string of at most 200 characters.
763
+ - **It goes through the journal's `append`**, so `validate` and `authorize`
764
+ judge it like any operation.
765
+ - **Keep the server's `actorId`.** The journal binds each replica id to the
766
+ first actor that commits from it, so after a change of `actorId` a
767
+ redelivered key is an `IdentityConflictError`: that fact was the first
768
+ actor's. A new key, and a reaction's append for a new commit, commit as the
769
+ new actor.
770
+ - **A refused append frees its key.** The key is recorded before the append,
771
+ so a redelivery after a deploy appends the bytes first written; when the
772
+ journal refuses the append (`validate`, `authorize`), nothing committed and
773
+ the record is cleared, so the key can name a Message the rules take. A
774
+ redelivery after a refusal is judged afresh: refused again with the same
775
+ error, or committed if the rules now take it, never `ServerKeyReusedError`.
776
+ - **Each key leaves a row.** The journal keeps a replica binding per
777
+ `appendAsServer` key, and one per reaction append (per trigger and
778
+ index), in `foldkit-durable`'s `replicas` table; neither compaction nor
779
+ `reset` deletes them, so they grow with the facts and reactions recorded.
780
+ Nothing prunes them yet.
781
+ - A server that hands `foldkit-durable`'s `append` input it did not first read
782
+ with a `codec` reader gets no server-only check: the journal knows no
783
+ Message's tag. Read every sent entry with `codec.writtenFrom` before
784
+ appending it, as `journalExchange` does. The types hold you to it: what an
785
+ exchange receives (`Sync.transport.serve`'s `exchange`, a `TransportClient`,
786
+ a `loopback` handler) is `ReadonlyArray<unknown>`, and `append` takes only
787
+ a `WrittenOperation`.
788
+
789
+ ### Work after a commit: reactions
790
+
791
+ A reaction runs after each committed Message of a set, in journal order:
792
+ mail a receipt, call a provider, append a server-only Message as a result.
793
+ It is a declared value, given to the `serverJournal`:
794
+
795
+ <!-- docs-check
796
+ declare const sendReceipt: (order: string, idempotencyKey: string) => Effect.Effect<void, Error>
797
+ -->
798
+ ```ts
799
+ import { reaction } from 'foldkit-sync/journal'
800
+
801
+ const receipt = reaction(Orders, {
802
+ name: 'receipt',
803
+ on: Confirmed,
804
+ appends: Receipted,
805
+ run: ({ message, key }) =>
806
+ sendReceipt(message.id, key).pipe(Effect.as([Message.SentReceipt({ id: message.id })])),
807
+ })
808
+
809
+ const running = Effect.gen(function* () {
810
+ // Made before the journal serves its first exchange: the reaction sees every commit after it.
811
+ const server = yield* serverJournal({
812
+ sync: Orders,
813
+ journal,
814
+ server: { actorId: 'orders' },
815
+ reactions: [receipt],
816
+ })
817
+ // One pass; none when every reaction has caught up, else how long to wait.
818
+ const next: Option.Option<unknown> = yield* server.react
819
+ return { server, next }
820
+ })
821
+ ```
822
+
823
+ ```text
824
+ commit -> react: read after the stored cursor -> run (answer recorded)
825
+ -> append the answer as the server -> store the cursor
826
+ ```
827
+
828
+ - **At least once, never exactly once.** The provider and the journal share
829
+ no transaction: a crash or a timeout between the call and the ledger's
830
+ record runs the call again. `key` is the same on every run of one
831
+ reaction on one commit (64 hex characters); give it to the provider as its
832
+ idempotency key, and the provider decides whether a repeat is a no-op. The
833
+ appends are once: their ids come from the reaction and its trigger's id
834
+ (never its place in the order, which a `reset` reuses), and the journal
835
+ answers a repeat from history.
836
+ - **The cursor is stored** beside the journal (`foldkit-durable`'s schema 7),
837
+ one per reaction, so a restart neither walks the log again nor misses what
838
+ committed while it was down. A reaction starts where the journal was when
839
+ `serverJournal` registered it: deploying a mailer does not mail the past.
840
+ - **A run that fails waits**: that reaction stops at the commit, on a backoff
841
+ from 0.5 s doubling to 30 s that the store keeps, and the others go on.
842
+ `react` never fails; it reports `ReactionFailed` and answers the delay. A
843
+ defect and a timeout (`timeout`, 10 s by default) are failures like any
844
+ other.
845
+ - **A run that can never succeed is parked**: the journal refused its append
846
+ (`validate`, `authorize`), or its answer holds a Message `appends` does not
847
+ name, more than 16, or one that will not encode. So is one that has failed
848
+ `attempts` times (8) *and* for at least `parkAfter` (an hour), so an outage
849
+ parks nothing. The journal itself failing counts toward neither. A parked
850
+ run is reported as `ReactionParked` and set aside with its trigger; the
851
+ reaction goes on. `parked` lists them, `retry` runs one again out of order
852
+ (a recorded answer is reused, so no second call), and `dismiss` gives one
853
+ up. After a `reset` a parked run is `stale`: it can be dismissed, not
854
+ retried.
855
+ - **Compact no further than `settledThrough`**, the lowest reaction cursor,
856
+ or a reaction skips what it never saw (`ReactionSkipped`).
857
+ - **What a handler is given** is the committed Message, its id, sequence and
858
+ actor: never the snapshot, which has moved on by the time it runs. Its
859
+ fields are what a client wrote; a handler that mails `message.email` mails
860
+ wherever the client said.
861
+ - A reaction's append commits where the journal is when it runs, after
862
+ whatever committed meanwhile. Two reactions in a cycle (one appends what
863
+ the other runs on) are refused when the journal is made.
864
+
865
+ Who calls `react` is the host's:
866
+
867
+ - **A long-lived process** forks `server.follow`: it runs `react` now, after
868
+ each commit this journal instance hears of, and after the delay `react`
869
+ answers. A reaction's commit reaches the document's sockets through
870
+ `serveJournal`'s notices.
871
+ - **A Durable Object**: give `defineDocumentHost` a `server` (below). It
872
+ registers the reactions when it opens the journal, sets the object's alarm
873
+ before replying to an exchange that committed, runs them without holding
874
+ the reply, and `alarm()` runs them again and sets the alarm from their
875
+ answer. Work after a reply lasts only while the object stays loaded; the
876
+ alarm is what resumes a run eviction cut off.
877
+ - **A stateless Worker over D1**: `ctx.waitUntil(server.react)` in the
878
+ request that committed, with `runsPerPass` at 1 or 2 since `waitUntil`
879
+ lasts about 30 seconds, and a Cron Trigger calling `react` for what failed
880
+ or was cut off. Two isolates may run one reaction at once, a duplicate call
881
+ without a crash. And no other request is told: a reaction's commit reaches
882
+ another replica when it next exchanges, never as a notice, since one
883
+ request cannot wake another's stream.
884
+
885
+ `settle` stays for a read model the reply depends on: it runs before the
886
+ reply and may fail the exchange. Reactions are for everything else.
887
+
561
888
  ### Coalesce a burst of typing
562
889
 
563
890
  Typing makes one durable Message per keystroke. `coalesce(last, next)` is
564
891
  offered each submit together with the last operation in the outbox; return a
565
892
  merged Message to replace it, or `undefined` to keep them apart:
566
893
 
567
- ```ts
894
+ ```ts sketch
568
895
  coalesce: (last, next) =>
569
896
  last._tag === 'RenamedTodo' && next._tag === 'RenamedTodo' && last.id === next.id
570
897
  ? next
@@ -576,6 +903,144 @@ operation is only merged into while no exchange has carried it, so there is no
576
903
  timer to tune: keystrokes typed while a request is out become one operation, and
577
904
  a lone keystroke goes out as fast as before.
578
905
 
906
+ ### Change a durable Message
907
+
908
+ An operation is stored, in the outbox and in the server's journal, as it was
909
+ written, and read later, possibly by a newer build. Renaming a field of a
910
+ durable Message, splitting a variant, or changing what a number means would
911
+ leave what was stored unreadable. So the document declares its Message's
912
+ history, and every reader sees an old operation upcast to the current
913
+ version:
914
+
915
+ <!-- docs-check standalone
916
+ import { Match, Schema } from 'effect'
917
+ import { defineMessageUnion } from 'foldkit/message'
918
+ import { modifyFields } from 'foldkit/struct'
919
+ import { MessageSet, Projection, Surface } from 'foldkit-surface'
920
+ import { DocumentId, forApplication, migrations } from 'foldkit-sync'
921
+ const Model = Schema.Struct({
922
+ notes: Schema.Array(Schema.Struct({ id: Schema.String, name: Schema.String })),
923
+ })
924
+ const Message = defineMessageUnion({
925
+ RenamedNote: { id: Schema.String, name: Schema.String },
926
+ ClearedNotes: {},
927
+ })
928
+ const App = Surface.application({
929
+ Model,
930
+ Message,
931
+ initial: { notes: [] },
932
+ update: (model, message) =>
933
+ Message.match(message, {
934
+ RenamedNote: ({ id, name }) => ({
935
+ model: modifyFields(model, { notes: notes => [...notes, { id, name }] }),
936
+ }),
937
+ ClearedNotes: () => ({ model: modifyFields(model, { notes: () => [] }) }),
938
+ }),
939
+ })
940
+ -->
941
+ ```ts
942
+ // Version 1, as the first build shipped it: a note had a `title`. Kept as
943
+ // long as a stored operation may hold it; only the changed variant is copied.
944
+ const RenamedNoteV1 = Schema.TaggedStruct('RenamedNote', {
945
+ id: Schema.String,
946
+ title: Schema.String,
947
+ })
948
+ const MessageV1 = Schema.Union([RenamedNoteV1, Message.ClearedNotes])
949
+
950
+ // Version 2 is the application's own Message, where `title` became `name`.
951
+ const NoteHistory = migrations(MessageV1).next(App.Message, previous =>
952
+ Match.valueTags(previous, {
953
+ RenamedNote: ({ id, title }) => ({ _tag: 'RenamedNote' as const, id, name: title }),
954
+ ClearedNotes: cleared => cleared,
955
+ }),
956
+ )
957
+
958
+ const Notes = forApplication(App).make({
959
+ documentId: DocumentId.make('notes'),
960
+ shared: Projection.pick(App.model.notes),
961
+ durable: MessageSet.make(App, [Message.RenamedNote, Message.ClearedNotes]),
962
+ migrations: NoteHistory,
963
+ })
964
+ ```
965
+
966
+ Version 1 is the first Schema; each `next` adds a version with its Schema and
967
+ a function from the version before, encoded, to its own, encoded. Each step's
968
+ input is typed by the step before, so a step that returns the old shape does
969
+ not compile. Steps are written once and never changed: a later change adds a
970
+ step. A step must be pure and deterministic, with no clock and no random id:
971
+ the server reads an entry more than once (`admit`, then the journal), and
972
+ each read must find the same Message. It is given the version before checked
973
+ and normalized by that version's Schema. The history must end at
974
+ `App.Message` itself (`make` throws otherwise), and with no `migrations` the
975
+ Message is version 1, written exactly as before.
976
+
977
+ What each side does with it:
978
+
979
+ - **The replica** keeps each operation as it was written, with its
980
+ `messageVersion` (absent for version 1), in its outbox and on the wire, and
981
+ replays it upcast. So an outbox written by the old build opens and sends
982
+ under the new one.
983
+ - **The server** (`journalExchange`) hands every hook, `admit`, `validate`,
984
+ `authorize`, `reduce` and `stamp`, the Message upcast to the current version,
985
+ so a client claiming version 1 is judged exactly as the current Message
986
+ would be. It stores what commits at the current version, and recognizes a
987
+ retransmission by the operation as it was written, so an edit committed
988
+ before a deploy, whose acknowledgement was lost, still reads as committed.
989
+
990
+ When something will not read:
991
+
992
+ - **An older version no step reads** (its version's Schema refuses it, or a
993
+ step throws): a pending operation moves to `status.refused`, kept as
994
+ written with its payload, never sent, until `replica.dismiss(opId)`; the
995
+ rest of the outbox opens and sends. No Schema of this build reads it, so a
996
+ page shows it raw (`operation.message`, `operation.messageVersion`) or not at
997
+ all. The server rejects one it receives as `Unreadable { version }`, and the
998
+ replica keeps it in `refused` too. A committed one fails what has to read
999
+ it, with `UnreadableHistoryError` on the server: every exchange of a
1000
+ replica whose cursor is before it (what that replica sends still commits,
1001
+ and is acknowledged once a build reads the row), a retransmission of its
1002
+ id, and, while the row is after the journal's stored snapshot, an append
1003
+ or load that rebuilds the document from that snapshot. A replica past the
1004
+ row is served, and other documents are not affected. Only a build with a
1005
+ step that reads it recovers it. The replica is told only "This server
1006
+ cannot read the history of this document"; the operation's id and
1007
+ version stay on the error, for the server's logs.
1008
+ - **A newer version** (a rollback, or an old client): the replica refuses to
1009
+ open on one in its outbox (`UnsupportedMessageVersionError`, nothing
1010
+ removed), and an exchange that receives one fails with it, recorded in
1011
+ `status.lastError` and retried on the backoff; the page reloads the new build
1012
+ when it sees that error (`ExchangeFailed` on `Diagnostics`). A server
1013
+ receiving one fails the exchange before appending anything, so the client
1014
+ waits with its edits rather than losing them.
1015
+
1016
+ Deploy order: the release of Sync that reads `messageVersion` goes to every
1017
+ server and client first, and skipping that step **loses edits**. A server on
1018
+ an older release rejects an operation carrying `messageVersion` as `Invalid`,
1019
+ and the replica drops it; a replica on an older release that is sent an
1020
+ `Unreadable` refusal drops that operation too, since it keeps no `refused`
1021
+ list. Then deploy an application's new version server first. A client on the
1022
+ old version keeps syncing until the first operation of the new version
1023
+ commits, then fails its exchanges until it reloads; its outbox is kept and
1024
+ sent after the reload.
1025
+
1026
+ A rollback of the Sync release itself meets a replica state holding
1027
+ `refused`, which an older release refuses to open
1028
+ (`InvalidReplicaHistoryError`). Do not clear storage on that error: the
1029
+ outbox is in the same state, and clearing it loses every unsent edit.
1030
+ Roll forward instead.
1031
+
1032
+ A server that builds an operation itself and appends it (`editsJournal`'s
1033
+ `absorbed`) writes it at the current version:
1034
+ `Notes.journalContract().operation.encode(operation)`. The journal takes a
1035
+ `WrittenOperation`, which is branded so that only Sync makes one: a decoded
1036
+ `Operation` names no version, would read as version 1, and is refused by the
1037
+ type. An entry from a wire is `unknown` until `codec.writtenFrom` reads it.
1038
+ Nothing read as history writes again: `operation.decode` takes only a
1039
+ `WrittenOperation`, and `encode` refuses what `codec.committedFrom` returns,
1040
+ stripped of its commit or not. Changing the shared
1041
+ slice's own Schema is not covered: a stored snapshot or checkpoint of the old
1042
+ shape still needs a migration of your own.
1043
+
579
1044
  ### The URL, mirrors, and agents
580
1045
 
581
1046
  `mount` takes the application's own `subscriptions` and the `resources` Layer
@@ -659,6 +1124,30 @@ whole of it.
659
1124
  A `Storage` needs three members, and the loopback transport takes a handler in
660
1125
  place of a socket:
661
1126
 
1127
+ <!-- docs-check standalone
1128
+ import { Option, Schema } from 'effect'
1129
+ import { defineMessageUnion } from 'foldkit/message'
1130
+ import { MessageSet, Projection, Surface } from 'foldkit-surface'
1131
+ import { DocumentId, Sync } from 'foldkit-sync'
1132
+ const Model = Schema.Struct({
1133
+ todos: Schema.Array(Schema.Struct({ id: Schema.String, title: Schema.String })),
1134
+ })
1135
+ const Message = defineMessageUnion({
1136
+ CreatedTodo: { id: Schema.String, title: Schema.String },
1137
+ RenamedTodo: { id: Schema.String, title: Schema.String },
1138
+ })
1139
+ const App = Surface.application({
1140
+ Model,
1141
+ Message,
1142
+ initial: { todos: [] },
1143
+ update: model => ({ model }),
1144
+ })
1145
+ const TodoSync = Sync.forApplication(App).make({
1146
+ documentId: DocumentId.make('todos'),
1147
+ shared: Projection.pick(App.model.todos),
1148
+ durable: MessageSet.make(App, [Message.CreatedTodo, Message.RenamedTodo]),
1149
+ })
1150
+ -->
662
1151
  ```ts
663
1152
  import { Effect } from 'effect'
664
1153
  import type { Storage } from 'foldkit-sync'
@@ -672,7 +1161,7 @@ const memoryStorage = (): Storage => {
672
1161
  }
673
1162
  }
674
1163
 
675
- const program = Effect.gen(function* () {
1164
+ const inMemory = Effect.gen(function* () {
676
1165
  const replica = yield* TodoSync.openReplica(memoryStorage())
677
1166
  yield* replica.submit(Message.CreatedTodo({ id: 't1', title: 'Milk' }))
678
1167
  yield* replica.synchronize
@@ -681,7 +1170,9 @@ const program = Effect.gen(function* () {
681
1170
  Effect.provide(
682
1171
  Sync.transport.loopback((_cursor, pending) => ({
683
1172
  operations: [],
684
- acknowledged: pending.map(operation => operation.opId),
1173
+ acknowledged: pending.map(
1174
+ entry => TodoSync.codec.writtenFrom(entry, TodoSync.documentId).opId,
1175
+ ),
685
1176
  rejected: [],
686
1177
  })),
687
1178
  ),
@@ -724,8 +1215,17 @@ application come first.
724
1215
  exchange until its storage is cleared.
725
1216
  - **Stored state is from another version.** Opening fails with
726
1217
  `UnsupportedReplicaVersionError` naming both versions; the bytes are left for
727
- a deliberate migration or reset. Sync versions its own envelope only; your
728
- Message and snapshot migrations stay yours.
1218
+ a deliberate migration or reset. That versions Sync's own envelope; a
1219
+ durable Message's versions are the document's history
1220
+ ([change a durable Message](#change-a-durable-message)), and snapshot
1221
+ migrations stay yours.
1222
+ - **An operation from another build of the application.** An older version no
1223
+ step reads lands in `status.refused`, kept until dismissed; a newer one
1224
+ refuses to open the replica, or fails the exchange, with
1225
+ `UnsupportedMessageVersionError`, and nothing is removed. Moving an
1226
+ operation to `refused` is the one write an open makes, under the revision
1227
+ it read: if another tab wrote the store meanwhile, the open fails with
1228
+ `StorageError`, and opening again reads both.
729
1229
  - **The server answers nonsense.** Malformed exchanges, gaps in the committed
730
1230
  order, acknowledgements or rejections for operations the replica never sent:
731
1231
  each fails the exchange rather than touch local work.
@@ -748,7 +1248,7 @@ A replica reports to the reporter in the Effect context it runs in:
748
1248
  host takes one with the upgrade request beside it, used for everything that
749
1249
  upgrade runs:
750
1250
 
751
- ```ts
1251
+ ```ts sketch
752
1252
  export class SyncHost extends defineDocumentHost(DurableObject, {
753
1253
  sync: TodoSync,
754
1254
  openJournal: (_ctx, env) => openTodosJournal(env.DB),
@@ -769,7 +1269,7 @@ slice alone, and an application that is not a Foldkit runtime (a worker, a
769
1269
  server-side actor, a test) uses it directly.
770
1270
 
771
1271
  ```ts
772
- const program = Effect.gen(function* () {
1272
+ const worker = Effect.gen(function* () {
773
1273
  const storage = yield* Sync.indexedDb('todos/worker')
774
1274
  const replica = yield* TodoSync.openReplica(storage)
775
1275
 
@@ -795,7 +1295,9 @@ state, so an edit submitted while the request was in flight survives.
795
1295
 
796
1296
  The rest of the surface: `committed` (the base alone), `pending`, `cursor`,
797
1297
  `status`, `statusChanges`, `changes` (status and `shared` from one snapshot),
798
- `snapshot` (the same plus `nextLocalSequence`), `start`, and `close`.
1298
+ `snapshot` (the same plus `nextLocalSequence`), `start`, `dismiss` (forgets a
1299
+ `status.refused` operation), and `close`. `pending` and a rejection's
1300
+ `operation` are `WrittenOperation`s: as written, with `messageVersion`.
799
1301
 
800
1302
  ### The exchange
801
1303
 
@@ -804,21 +1306,34 @@ carries it.
804
1306
 
805
1307
  ```text
806
1308
  request cursor
807
- pending operations
1309
+ pending operations, as written (messageVersion past 1)
808
1310
  epoch, once the replica has heard one
809
1311
 
810
1312
  answer operations committed after the cursor, contiguous
811
1313
  acknowledged ids the server accepted without repeating them
812
1314
  rejected ids the server refused
813
- reasons? [{ opId, reason }]: why, for some of rejected; at most 500 characters each
1315
+ refusals? [{ opId, refusal }]: why, for some of rejected, one each; an encoded SyncRefusal
1316
+ reasons? [{ opId, reason }]: deprecated words, at most 500 characters each
814
1317
  checkpoint? { cursor, model } when the history is gone
815
1318
  more? true when another page follows
816
1319
  epoch? the server's history
817
1320
  ```
818
1321
 
819
1322
  An acknowledged operation leaves the outbox once its committed copy has
820
- arrived or the last page has been read. A rejected one leaves at once. The
821
- replica validates every field; see [when things go wrong](#when-things-go-wrong).
1323
+ arrived or the last page has been read. A rejected one leaves at once.
1324
+
1325
+ A custom server that reads operations through the codec, or appends them to
1326
+ a Durable journal itself, must do what `journalExchange` does with a newer
1327
+ `messageVersion` than its own (`codec.messageVersion`): fail the whole
1328
+ exchange before appending anything. Durable alone sees such an entry as an
1329
+ `InvalidOperationError`, and rejecting it by id loses the edit. It reads
1330
+ each sent entry with `codec.writtenFrom` before appending it, too: that is
1331
+ where a [server-only Message](#messages-only-the-server-appends), a replica
1332
+ id under `server/` or one past 128 characters is refused. Stored history is
1333
+ read with `journalContract().operation.decode`, which takes the server's own
1334
+ Messages; a journal decoding its rows with `codec.normalizeOperation` would
1335
+ refuse them. The replica validates every field; see
1336
+ [when things go wrong](#when-things-go-wrong).
822
1337
 
823
1338
  ### Transports
824
1339
 
@@ -882,7 +1397,8 @@ const Shared = Schema.Struct({ title: Title.schema })
882
1397
 
883
1398
  `Sync.define({ documentId, message, shared, empty, durable, replay })` is the
884
1399
  protocol primitive `forApplication` compiles to, for a client with no
885
- application to derive from. Its `replay` is unguarded; the author owns its
1400
+ application to derive from. It takes `migrations` (a `Sync.migrations`
1401
+ history) in place of `message`, and reads the Message from it. Its `replay` is unguarded; the author owns its
886
1402
  agreement with whatever applies the Message elsewhere.
887
1403
 
888
1404
  ### Joining an assembly
@@ -893,13 +1409,23 @@ runs nothing; Sync owns its runtime through `mount`.
893
1409
 
894
1410
  ## What Sync owns, and does not
895
1411
 
896
- Owns the replication contract (shared slice, durable set, replay, codecs,
897
- `journalContract`), the replica (outbox, base, cursor, optimistic view, status),
898
- reconciliation, the transport seam, and the optional presence and LWW helpers.
1412
+ Owns the replication contract (shared slice, durable and server-only sets,
1413
+ replay, codecs, `journalContract`), the replica (outbox, base, cursor,
1414
+ optimistic view, status), reconciliation, the transport seam, the server's own
1415
+ appends and reactions (`serverJournal`), and the optional presence and LWW
1416
+ helpers.
1417
+
1418
+ A reaction's cursor and parked runs are stored by `foldkit-durable`, beside
1419
+ the effect ledger that records each run; Sync decides when a run happens and
1420
+ what it appends. Whether an external call happened is the provider's to know.
1421
+
1422
+ Sync applies a durable Message's history wherever it reads an operation; the
1423
+ application owns the history itself (which Schemas it shipped and how each
1424
+ becomes the next).
899
1425
 
900
1426
  Does not own the rest of the Model, server-derived caches (`foldkit-remote`),
901
1427
  the authoritative order and storage (`foldkit-durable` or your server),
902
- peer-to-peer merging, or migration of your own Messages and snapshots.
1428
+ peer-to-peer merging, or migration of the shared slice's own Schema.
903
1429
 
904
1430
  ## Limits
905
1431