foldkit-sync 0.15.0 → 0.16.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (49) hide show
  1. package/README.md +558 -40
  2. package/dist/diagnostics.d.ts +74 -3
  3. package/dist/diagnostics.d.ts.map +1 -1
  4. package/dist/diagnostics.js +60 -4
  5. package/dist/diagnostics.js.map +1 -1
  6. package/dist/do.d.ts +49 -7
  7. package/dist/do.d.ts.map +1 -1
  8. package/dist/do.js +91 -8
  9. package/dist/do.js.map +1 -1
  10. package/dist/errors.d.ts +92 -1
  11. package/dist/errors.d.ts.map +1 -1
  12. package/dist/errors.js +87 -1
  13. package/dist/errors.js.map +1 -1
  14. package/dist/index.d.ts +9 -5
  15. package/dist/index.d.ts.map +1 -1
  16. package/dist/index.js +6 -2
  17. package/dist/index.js.map +1 -1
  18. package/dist/journal.d.ts +52 -27
  19. package/dist/journal.d.ts.map +1 -1
  20. package/dist/journal.js +176 -64
  21. package/dist/journal.js.map +1 -1
  22. package/dist/make.d.ts +48 -17
  23. package/dist/make.d.ts.map +1 -1
  24. package/dist/make.js +11 -4
  25. package/dist/make.js.map +1 -1
  26. package/dist/migrations.d.ts +37 -0
  27. package/dist/migrations.d.ts.map +1 -0
  28. package/dist/migrations.js +56 -0
  29. package/dist/migrations.js.map +1 -0
  30. package/dist/mount.d.ts +1 -1
  31. package/dist/mount.d.ts.map +1 -1
  32. package/dist/mount.js +12 -6
  33. package/dist/mount.js.map +1 -1
  34. package/dist/refusal.d.ts +33 -0
  35. package/dist/refusal.d.ts.map +1 -0
  36. package/dist/refusal.js +47 -0
  37. package/dist/refusal.js.map +1 -0
  38. package/dist/serverJournal.d.ts +143 -0
  39. package/dist/serverJournal.d.ts.map +1 -0
  40. package/dist/serverJournal.js +503 -0
  41. package/dist/serverJournal.js.map +1 -0
  42. package/dist/sync.d.ts +132 -11
  43. package/dist/sync.d.ts.map +1 -1
  44. package/dist/sync.js +273 -64
  45. package/dist/sync.js.map +1 -1
  46. package/dist/transport.d.ts +5 -5
  47. package/dist/transport.d.ts.map +1 -1
  48. package/dist/transport.js.map +1 -1
  49. package/package.json +11 -8
package/README.md CHANGED
@@ -93,7 +93,7 @@ import { defineMessageUnion } from 'foldkit/message'
93
93
  import { modifyFields } from 'foldkit/struct'
94
94
  import type * as Update from 'foldkit/update'
95
95
  import { MessageSet, Projection, Surface } from 'foldkit-surface'
96
- import { DocumentId, Sync } from 'foldkit-sync'
96
+ import { DocumentId, forApplication } from 'foldkit-sync'
97
97
 
98
98
  const Model = Schema.Struct({
99
99
  todos: Schema.Array(Schema.Struct({ id: Schema.String, title: Schema.String })),
@@ -130,7 +130,7 @@ const update = (model: Model, message: Message): Return =>
130
130
 
131
131
  const App = Surface.application({ Model, Message, initial, update })
132
132
 
133
- const TodoSync = Sync.forApplication(App).make({
133
+ const TodoSync = forApplication(App).make({
134
134
  documentId: DocumentId.make('todos'),
135
135
  shared: Projection.pick(App.model.todos),
136
136
  durable: MessageSet.make(App, [Message.CreatedTodo, Message.RenamedTodo]),
@@ -296,6 +296,11 @@ the exchange by hand, with an effect policy per committed operation.
296
296
 
297
297
  A server that is not Durable implements the same [exchange](#the-exchange).
298
298
 
299
+ Import what a server calls by name (`forApplication`, `serveJournal`,
300
+ `servePresence`), not through the `Sync` namespace object: a bundler keeps
301
+ every member of an object, so `Sync.forApplication` in a server's bundle also
302
+ carries the client's mount, sockets, presence and IndexedDB code.
303
+
299
304
  ## Serving from a Durable Object
300
305
 
301
306
  On Cloudflare, one object per document is the single writer the journal
@@ -303,7 +308,7 @@ wants. `defineDocumentHost` from `foldkit-sync/do` defines that class: each
303
308
  accepted socket exchanges against the same journal through `serveJournal`,
304
309
  and the replica side is the socket transport it already speaks:
305
310
 
306
- ```ts
311
+ ```ts sketch
307
312
  import { DurableObject } from 'cloudflare:workers'
308
313
  import { defineDocumentHost } from 'foldkit-sync/do'
309
314
 
@@ -315,7 +320,7 @@ export class SyncHost extends defineDocumentHost(DurableObject, {
315
320
  }) {}
316
321
  ```
317
322
 
318
- ```ts
323
+ ```ts sketch
319
324
  // The worker: one object per document.
320
325
  export default {
321
326
  fetch: (request: Request, env: Env): Promise<Response> =>
@@ -346,6 +351,23 @@ Sync.transport.socket({ url: 'wss://example.com/sync' })
346
351
  request, while the socket is open: a refusal closes it with 4401, and a
347
352
  principal whose `key` (`Policy.keyOf`) changed with 1012, which the
348
353
  replica reconnects after. Without it a socket serves its first principal until it closes.
354
+ - `server: { principal, reactions: env => [...], runsPerPass? }` gives the
355
+ object its [reactions](#work-after-a-commit-reactions): registered when the
356
+ journal opens, before any socket is served; the alarm set before the reply
357
+ to an exchange that committed, a pass run after it, and `alarm()` running
358
+ them and setting the alarm from what they answer, then calling the base
359
+ class's own `alarm()` if it has one. The reactions' part never throws, so
360
+ the runtime's few retries of a throwing alarm are never what a run depends
361
+ on; it runs one pass at a time, and only ever brings the object's one alarm
362
+ earlier. A reaction declared for another contract is refused when the
363
+ object is made. A `setAlarm` that fails after a commit is reported, the
364
+ application's `settle` still runs, and the next exchange arms again.
365
+ A pass that broke is a `HostDefect` (`Alarm`), and the alarm comes back in
366
+ 30 seconds. The base class's `alarm()` shares that one alarm: it runs on
367
+ every wake, so it decides for itself what is due, and sets its own next
368
+ wake with `armAlarm(ctx.storage, at)` from `foldkit-sync/do`, which only
369
+ ever brings the alarm earlier. A subclass that overrides `alarm()` calls
370
+ `super.alarm()`, or the reactions stop being resumed.
349
371
  - `settle: (env, journal) => Effect` runs after each exchange's appends, before
350
372
  the reply is read, to apply what committed elsewhere — the Remote tables a
351
373
  reader then queries. `journal.recover` makes it crash-safe and idempotent;
@@ -420,16 +442,19 @@ error names why: `ReplayError` for a Message that cannot be durable,
420
442
  **Committed** is the replica's status:
421
443
 
422
444
  ```ts
423
- const status = yield* replica.status
424
- // { pending: number, cursor: Sequence, lastError: string | undefined, rejected: ReadonlyArray<Rejection> }
445
+ const status = Effect.runSync(replica.status)
446
+ // { pending, cursor, lastError: string | undefined, rejected: ReadonlyArray<Rejection>, refused, epoch }
425
447
  ```
426
448
 
427
449
  `pending` above zero means edits the server has not confirmed; `lastError` is
428
450
  the last failed exchange, cleared by the next success; `rejected` lists recent
429
- edits the server refused, most recent first, each `{ opId, reason, operation }`,
430
- where `reason` is an `Option` of the server's words for the person and
431
- `operation` is the refused operation itself, which the replica has dropped
432
- from the outbox, so a page reads what it changed from here. `replica.statusChanges` is a Stream of the same,
451
+ edits the server refused, most recent first, each `{ opId, refusal, operation, message }`,
452
+ where `refusal` is an `Option` of why (a `SyncRefusal`, below),
453
+ `operation` is the refused operation as it was sent, which the replica has dropped
454
+ from the outbox, and `message` its Message encoded at the current version
455
+ (see [changing a durable Message](#change-a-durable-message)), so a page reads
456
+ what it changed from here. `refused` lists the operations this build cannot
457
+ read at all, kept until `replica.dismiss(opId)`. `replica.statusChanges` is a Stream of the same,
433
458
  emitted now and after every submit and exchange (subscribed before its first
434
459
  read, so a subscriber slow to take one misses no change); `replica.changes`
435
460
  pairs it with the optimistic `shared` value so one subscription sees both.
@@ -438,13 +463,48 @@ pairs it with the optimistic `shared` value so one subscription sees both.
438
463
  error. The replica drops it and replays the rest; the mount re-installs the
439
464
  shared slice, and the field the user edited reverts. `onPersistenceFailure`
440
465
  does not run for this, since the local save succeeded. Watch `rejected` to
441
- explain the revert: `journalExchange` gives each rejection a reason, the
442
- `authorize` rule's own when it gave one. Otherwise it gives a fixed sentence:
443
- "Refused by the server" when `authorize` refused with none (a plain `false`),
444
- and "Not a valid operation" when the operation failed (`validate` refused or
445
- threw, `authorize` threw, or the reducer could not apply it). It never sends
446
- an error's message, which may carry internals, so a rule a person should read
447
- about belongs in `authorize` with a `reason`, not in `validate`.
466
+ explain the revert. Each rejection's `refusal` is a `SyncRefusal`, in Remote's
467
+ vocabulary for "not saved", so a page that also shows Remote's refusals
468
+ handles both with one function:
469
+
470
+ - `Forbidden { reason }`: `authorize` or `admit` refused who sent it, or its
471
+ replica belongs to another actor. `reason` is the rule's own words when it
472
+ gave some (a plain `false` gives none), cut to 500 characters. Remote's
473
+ `Denied.Forbidden` has the same shape.
474
+ - `Invalid { message, entity, rule, path }`: the operation failed, with a fixed
475
+ sentence as `message`. "Not a valid operation" when `validate` refused or
476
+ threw, `authorize` threw, or the reducer could not apply it; "Not an
477
+ operation this server accepts" when it did not decode or named another
478
+ document; "Its id was already used for another" for a reused id.
479
+ `journalExchange` names no Entity, so `entity`, `rule` and `path` are empty;
480
+ the shape is Remote's `Refusal.invalid`.
481
+ - `Unreadable { version }`: written at an older Message version that no step
482
+ of the server's history reads. The replica keeps such an operation in
483
+ `status.refused` as well, with its payload (see
484
+ [changing a durable Message](#change-a-durable-message)).
485
+
486
+ ```ts
487
+ import { Option } from 'effect'
488
+ import { SyncRefusal, type Rejection } from 'foldkit-sync'
489
+
490
+ const said = (rejection: Rejection): string =>
491
+ Option.match(rejection.refusal, {
492
+ onNone: () => 'Refused by the server',
493
+ onSome: SyncRefusal.match({
494
+ Forbidden: ({ reason }) => Option.getOrElse(reason, () => 'You may not change this'),
495
+ Invalid: ({ message }) => message,
496
+ Unreadable: () => 'Made by another version of the app, so not saved',
497
+ }),
498
+ })
499
+ ```
500
+
501
+ `refusal` is none when the server named none (an older or a custom server), or
502
+ named one this build cannot read, which the replica reports as
503
+ `UnknownRefusal` and still treats as rejected. The server never sends an
504
+ error's message, which may carry internals, so a rule a person should read
505
+ about belongs in `authorize` with a `reason`, not in `validate`. The
506
+ deprecated `reason` (the words alone, an `Option<string>`) is still filled
507
+ for one release; read `refusal` instead.
448
508
 
449
509
  **Waiting for the server.** The Model shows an edit before the server has it.
450
510
  Something that must not report success until the edit is committed, an agent
@@ -452,10 +512,14 @@ tool for instance, waits on `mounted.committed`: the shared slice as the server
452
512
  confirmed it, with no pending edit applied. It is a source, `{ get, subscribe }`,
453
513
  told after every exchange:
454
514
 
515
+ <!-- docs-check
516
+ import { Agent } from 'foldkit-agent'
517
+ -->
455
518
  ```ts
456
- completion: Agent.when({
519
+ const completion = Agent.when({
457
520
  source: mounted.committed,
458
- predicate: (shared, request) => shared.todos.some(todo => todo.id === request.id),
521
+ predicate: (shared, request: { readonly id: string }) =>
522
+ shared.todos.some(todo => todo.id === request.id),
459
523
  })
460
524
  ```
461
525
 
@@ -469,8 +533,29 @@ server rejects.
469
533
  Each feature declares the fields it shares and the Messages that change them;
470
534
  `compose` merges them into one document:
471
535
 
536
+ <!-- docs-check standalone
537
+ import { Schema } from 'effect'
538
+ import { defineMessageUnion } from 'foldkit/message'
539
+ import { MessageSet, Projection, Surface } from 'foldkit-surface'
540
+ import { DocumentId, forApplication } from 'foldkit-sync'
541
+ const Model = Schema.Struct({
542
+ todos: Schema.Array(Schema.Struct({ id: Schema.String, title: Schema.String })),
543
+ members: Schema.Array(Schema.String),
544
+ })
545
+ const Message = defineMessageUnion({
546
+ CreatedTodo: { id: Schema.String, title: Schema.String },
547
+ RenamedTodo: { id: Schema.String, title: Schema.String },
548
+ Invited: { email: Schema.String },
549
+ })
550
+ const App = Surface.application({
551
+ Model,
552
+ Message,
553
+ initial: { todos: [], members: [] },
554
+ update: model => ({ model }),
555
+ })
556
+ -->
472
557
  ```ts
473
- const AppSync = Sync.forApplication(App)
558
+ const AppSync = forApplication(App)
474
559
 
475
560
  const Todos = AppSync.fragment({
476
561
  shared: Projection.pick(App.model.todos),
@@ -498,9 +583,9 @@ Policy belongs on the contract, beside the Messages it governs, and runs on the
498
583
  server inside the journal's commit. The client never runs it.
499
584
 
500
585
  ```ts
501
- const Authorized = Sync.forApplication(App).withPrincipal<{ readonly role: 'admin' | 'guest' }>()
586
+ const Authorized = forApplication(App).withPrincipal<{ readonly role: 'admin' | 'guest' }>()
502
587
 
503
- const Board = Authorized.make({
588
+ const GuardedBoard = Authorized.make({
504
589
  documentId: DocumentId.make('board'),
505
590
  ...Authorized.compose(Todos, Members),
506
591
  authorize: {
@@ -511,7 +596,8 @@ const Board = Authorized.make({
511
596
  ```
512
597
 
513
598
  A rule returns `true`, `false`, or `{ allowed: false, reason }`; the reason
514
- is sent to the replica whose edit it was, as its rejection's `reason`.
599
+ is sent to the replica whose edit it was, as its rejection's
600
+ `Forbidden` refusal.
515
601
 
516
602
  `message` is exactly that variant, `shared` is the authoritative snapshot, and
517
603
  `principal` is whatever the server's transport established for the connection;
@@ -528,13 +614,29 @@ A fact carries its own nondeterminism, but a client cannot know where its
528
614
  operation lands in the server's order. A stamp writes it in, on the server, as
529
615
  the journal commits:
530
616
 
617
+ <!-- docs-check standalone
618
+ import { Schema } from 'effect'
619
+ import { defineMessageUnion } from 'foldkit/message'
620
+ import { MessageSet, Projection, Surface } from 'foldkit-surface'
621
+ import { DocumentId, forApplication } from 'foldkit-sync'
622
+ const Model = Schema.Struct({
623
+ prices: Schema.Array(Schema.Struct({ id: Schema.String, cents: Schema.Number, at: Schema.Number })),
624
+ })
625
+ -->
531
626
  ```ts
532
627
  const Message = defineMessageUnion({
533
628
  // `at` is absent on what a client sends: the server's sequence once committed.
534
629
  PriceEdited: { id: Schema.String, cents: Schema.Number, at: Schema.optionalKey(Schema.Number) },
535
630
  })
536
631
 
537
- const Prices = Sync.forApplication(App).make({
632
+ const App = Surface.application({
633
+ Model,
634
+ Message,
635
+ initial: { prices: [] },
636
+ update: model => ({ model }),
637
+ })
638
+
639
+ const Prices = forApplication(App).make({
538
640
  documentId: DocumentId.make('prices'),
539
641
  shared: Projection.pick(App.model.prices),
540
642
  durable: MessageSet.make(App, [Message.PriceEdited]),
@@ -558,13 +660,235 @@ records the last sequence it applied can tell which edits it holds by comparing
558
660
  the two. A client that sends the stamped field itself is claiming a commit it
559
661
  did not get; refuse it in the journal's `validate`.
560
662
 
663
+ ### Messages only the server appends
664
+
665
+ Some facts only the server may record: a payment provider confirmed a
666
+ charge, the table absorbed an edit. Declare those Messages `serverOnly`,
667
+ beside `durable`. A replica stores and replays them like any other; it can
668
+ never submit one:
669
+
670
+ <!-- docs-check standalone
671
+ import { Effect, Option, Schema } from 'effect'
672
+ import { defineMessageUnion } from 'foldkit/message'
673
+ import { modifyFields } from 'foldkit/struct'
674
+ import { MessageSet, Projection, Surface } from 'foldkit-surface'
675
+ import { DocumentId, forApplication } from 'foldkit-sync'
676
+ import type { Journal } from 'foldkit-durable/core'
677
+ import type { Operation, WrittenOperation } from 'foldkit-sync'
678
+ const Order = Schema.Struct({ id: Schema.String, paid: Schema.Boolean, receipted: Schema.Boolean })
679
+ const Model = Schema.Struct({ orders: Schema.Array(Order) })
680
+ type Model = typeof Model.Type
681
+ -->
682
+ ```ts
683
+ const Message = defineMessageUnion({
684
+ PlacedOrder: { id: Schema.String },
685
+ ConfirmedPayment: { id: Schema.String, charge: Schema.String },
686
+ SentReceipt: { id: Schema.String },
687
+ })
688
+
689
+ const mark = (model: Model, id: string, change: Partial<typeof Order.Type>): Model =>
690
+ modifyFields(model, {
691
+ orders: () => model.orders.map(order => (order.id === id ? { ...order, ...change } : order)),
692
+ })
693
+
694
+ const App = Surface.application({
695
+ Model,
696
+ Message,
697
+ initial: { orders: [] },
698
+ update: (model, message) => ({
699
+ model: Message.match(message, {
700
+ PlacedOrder: ({ id }) =>
701
+ modifyFields(model, {
702
+ orders: () => [...model.orders, { id, paid: false, receipted: false }],
703
+ }),
704
+ ConfirmedPayment: ({ id }) => mark(model, id, { paid: true }),
705
+ SentReceipt: ({ id }) => mark(model, id, { receipted: true }),
706
+ }),
707
+ }),
708
+ })
709
+
710
+ const Confirmed = MessageSet.make(App, [Message.ConfirmedPayment])
711
+ const Receipted = MessageSet.make(App, [Message.SentReceipt])
712
+
713
+ const Orders = forApplication(App).make({
714
+ documentId: DocumentId.make('orders'),
715
+ shared: Projection.pick(App.model.orders),
716
+ // What a replica may submit.
717
+ durable: MessageSet.make(App, [Message.PlacedOrder]),
718
+ // What only the server appends. A Message in both sets is a type error.
719
+ serverOnly: MessageSet.union(Confirmed, Receipted),
720
+ })
721
+ ```
722
+
723
+ - **A replica submitting one is a type error**, and refused at run time.
724
+ - **The server refuses one any replica sends**, whoever it says it is:
725
+ every reader of a sent entry (`codec.writtenFrom`, `normalizeOperation`,
726
+ `operationFrom`) refuses it, judged after
727
+ [upcasting](#change-a-durable-message), before `admit`, `validate` or
728
+ `authorize` run. `journalExchange` rejects it by its id as `Forbidden`
729
+ ("Only the server makes this change") and reports a `ServerOnlySubmitted`
730
+ diagnostic; the edits beside it still go. No rule about who the principal
731
+ is is involved, so a resolver's mistake cannot let one through.
732
+ - **Replica ids under `server/` belong to the server**: a sent entry naming
733
+ one is rejected as `Invalid`, and so is one whose replica id is longer than
734
+ 128 characters.
735
+ - The contract's `surface` still lists `durable` alone, so an agent bound to
736
+ it cannot dispatch a server-only Message; `authorize` and `stamp` rules may
737
+ name one.
738
+
739
+ The server appends one through `serverJournal` from `foldkit-sync/journal`,
740
+ as a principal it holds in code and never resolves from a request:
741
+
742
+ <!-- docs-check
743
+ declare const journal: Journal<Operation, { readonly orders: Model['orders'] }, { readonly actorId: string }, WrittenOperation>
744
+ -->
745
+ ```ts
746
+ import { serverJournal } from 'foldkit-sync/journal'
747
+
748
+ const serving = Effect.gen(function* () {
749
+ const server = yield* serverJournal({ sync: Orders, journal, server: { actorId: 'orders' } })
750
+ // A payment provider's webhook: its event id names the fact.
751
+ return (event: { readonly id: string; readonly order: string; readonly charge: string }) =>
752
+ server.appendAsServer(Message.ConfirmedPayment({ id: event.order, charge: event.charge }), {
753
+ key: `payment/${event.id}`,
754
+ })
755
+ })
756
+ ```
757
+
758
+ - **`key` names the fact**, and is required: a webhook's event id, a clock
759
+ tick. A repeat with the same key and Message appends nothing new, so a
760
+ redelivered webhook is harmless; the same key with another Message fails
761
+ with `ServerKeyReusedError`, and the first fact stands. A key is a
762
+ non-empty string of at most 200 characters.
763
+ - **It goes through the journal's `append`**, so `validate` and `authorize`
764
+ judge it like any operation.
765
+ - **Keep the server's `actorId`.** The journal binds each replica id to the
766
+ first actor that commits from it, so after a change of `actorId` a
767
+ redelivered key is an `IdentityConflictError`: that fact was the first
768
+ actor's. A new key, and a reaction's append for a new commit, commit as the
769
+ new actor.
770
+ - **A refused append frees its key.** The key is recorded before the append,
771
+ so a redelivery after a deploy appends the bytes first written; when the
772
+ journal refuses the append (`validate`, `authorize`), nothing committed and
773
+ the record is cleared, so the key can name a Message the rules take. A
774
+ redelivery after a refusal is judged afresh: refused again with the same
775
+ error, or committed if the rules now take it, never `ServerKeyReusedError`.
776
+ - **Each key leaves a row.** The journal keeps a replica binding per
777
+ `appendAsServer` key, and one per reaction append (per trigger and
778
+ index), in `foldkit-durable`'s `replicas` table; neither compaction nor
779
+ `reset` deletes them, so they grow with the facts and reactions recorded.
780
+ Nothing prunes them yet.
781
+ - A server that hands `foldkit-durable`'s `append` input it did not first read
782
+ with a `codec` reader gets no server-only check: the journal knows no
783
+ Message's tag. Read every sent entry with `codec.writtenFrom` before
784
+ appending it, as `journalExchange` does.
785
+
786
+ ### Work after a commit: reactions
787
+
788
+ A reaction runs after each committed Message of a set, in journal order:
789
+ mail a receipt, call a provider, append a server-only Message as a result.
790
+ It is a declared value, given to the `serverJournal`:
791
+
792
+ <!-- docs-check
793
+ declare const sendReceipt: (order: string, idempotencyKey: string) => Effect.Effect<void, Error>
794
+ -->
795
+ ```ts
796
+ import { reaction } from 'foldkit-sync/journal'
797
+
798
+ const receipt = reaction(Orders, {
799
+ name: 'receipt',
800
+ on: Confirmed,
801
+ appends: Receipted,
802
+ run: ({ message, key }) =>
803
+ sendReceipt(message.id, key).pipe(Effect.as([Message.SentReceipt({ id: message.id })])),
804
+ })
805
+
806
+ const running = Effect.gen(function* () {
807
+ // Made before the journal serves its first exchange: the reaction sees every commit after it.
808
+ const server = yield* serverJournal({
809
+ sync: Orders,
810
+ journal,
811
+ server: { actorId: 'orders' },
812
+ reactions: [receipt],
813
+ })
814
+ // One pass; none when every reaction has caught up, else how long to wait.
815
+ const next: Option.Option<unknown> = yield* server.react
816
+ return { server, next }
817
+ })
818
+ ```
819
+
820
+ ```text
821
+ commit -> react: read after the stored cursor -> run (answer recorded)
822
+ -> append the answer as the server -> store the cursor
823
+ ```
824
+
825
+ - **At least once, never exactly once.** The provider and the journal share
826
+ no transaction: a crash or a timeout between the call and the ledger's
827
+ record runs the call again. `key` is the same on every run of one
828
+ reaction on one commit (64 hex characters); give it to the provider as its
829
+ idempotency key, and the provider decides whether a repeat is a no-op. The
830
+ appends are once: their ids come from the reaction and its trigger's id
831
+ (never its place in the order, which a `reset` reuses), and the journal
832
+ answers a repeat from history.
833
+ - **The cursor is stored** beside the journal (`foldkit-durable`'s schema 7),
834
+ one per reaction, so a restart neither walks the log again nor misses what
835
+ committed while it was down. A reaction starts where the journal was when
836
+ `serverJournal` registered it: deploying a mailer does not mail the past.
837
+ - **A run that fails waits**: that reaction stops at the commit, on a backoff
838
+ from 0.5 s doubling to 30 s that the store keeps, and the others go on.
839
+ `react` never fails; it reports `ReactionFailed` and answers the delay. A
840
+ defect and a timeout (`timeout`, 10 s by default) are failures like any
841
+ other.
842
+ - **A run that can never succeed is parked**: the journal refused its append
843
+ (`validate`, `authorize`), or its answer holds a Message `appends` does not
844
+ name, more than 16, or one that will not encode. So is one that has failed
845
+ `attempts` times (8) *and* for at least `parkAfter` (an hour), so an outage
846
+ parks nothing. The journal itself failing counts toward neither. A parked
847
+ run is reported as `ReactionParked` and set aside with its trigger; the
848
+ reaction goes on. `parked` lists them, `retry` runs one again out of order
849
+ (a recorded answer is reused, so no second call), and `dismiss` gives one
850
+ up. After a `reset` a parked run is `stale`: it can be dismissed, not
851
+ retried.
852
+ - **Compact no further than `settledThrough`**, the lowest reaction cursor,
853
+ or a reaction skips what it never saw (`ReactionSkipped`).
854
+ - **What a handler is given** is the committed Message, its id, sequence and
855
+ actor: never the snapshot, which has moved on by the time it runs. Its
856
+ fields are what a client wrote; a handler that mails `message.email` mails
857
+ wherever the client said.
858
+ - A reaction's append commits where the journal is when it runs, after
859
+ whatever committed meanwhile. Two reactions in a cycle (one appends what
860
+ the other runs on) are refused when the journal is made.
861
+
862
+ Who calls `react` is the host's:
863
+
864
+ - **A long-lived process** forks `server.follow`: it runs `react` now, after
865
+ each commit this journal instance hears of, and after the delay `react`
866
+ answers. A reaction's commit reaches the document's sockets through
867
+ `serveJournal`'s notices.
868
+ - **A Durable Object**: give `defineDocumentHost` a `server` (below). It
869
+ registers the reactions when it opens the journal, sets the object's alarm
870
+ before replying to an exchange that committed, runs them without holding
871
+ the reply, and `alarm()` runs them again and sets the alarm from their
872
+ answer. Work after a reply lasts only while the object stays loaded; the
873
+ alarm is what resumes a run eviction cut off.
874
+ - **A stateless Worker over D1**: `ctx.waitUntil(server.react)` in the
875
+ request that committed, with `runsPerPass` at 1 or 2 since `waitUntil`
876
+ lasts about 30 seconds, and a Cron Trigger calling `react` for what failed
877
+ or was cut off. Two isolates may run one reaction at once, a duplicate call
878
+ without a crash. And no other request is told: a reaction's commit reaches
879
+ another replica when it next exchanges, never as a notice, since one
880
+ request cannot wake another's stream.
881
+
882
+ `settle` stays for a read model the reply depends on: it runs before the
883
+ reply and may fail the exchange. Reactions are for everything else.
884
+
561
885
  ### Coalesce a burst of typing
562
886
 
563
887
  Typing makes one durable Message per keystroke. `coalesce(last, next)` is
564
888
  offered each submit together with the last operation in the outbox; return a
565
889
  merged Message to replace it, or `undefined` to keep them apart:
566
890
 
567
- ```ts
891
+ ```ts sketch
568
892
  coalesce: (last, next) =>
569
893
  last._tag === 'RenamedTodo' && next._tag === 'RenamedTodo' && last.id === next.id
570
894
  ? next
@@ -576,6 +900,141 @@ operation is only merged into while no exchange has carried it, so there is no
576
900
  timer to tune: keystrokes typed while a request is out become one operation, and
577
901
  a lone keystroke goes out as fast as before.
578
902
 
903
+ ### Change a durable Message
904
+
905
+ An operation is stored, in the outbox and in the server's journal, as it was
906
+ written, and read later, possibly by a newer build. Renaming a field of a
907
+ durable Message, splitting a variant, or changing what a number means would
908
+ leave what was stored unreadable. So the document declares its Message's
909
+ history, and every reader sees an old operation upcast to the current
910
+ version:
911
+
912
+ <!-- docs-check standalone
913
+ import { Match, Schema } from 'effect'
914
+ import { defineMessageUnion } from 'foldkit/message'
915
+ import { modifyFields } from 'foldkit/struct'
916
+ import { MessageSet, Projection, Surface } from 'foldkit-surface'
917
+ import { DocumentId, forApplication, migrations } from 'foldkit-sync'
918
+ const Model = Schema.Struct({
919
+ notes: Schema.Array(Schema.Struct({ id: Schema.String, name: Schema.String })),
920
+ })
921
+ const Message = defineMessageUnion({
922
+ RenamedNote: { id: Schema.String, name: Schema.String },
923
+ ClearedNotes: {},
924
+ })
925
+ const App = Surface.application({
926
+ Model,
927
+ Message,
928
+ initial: { notes: [] },
929
+ update: (model, message) =>
930
+ Message.match(message, {
931
+ RenamedNote: ({ id, name }) => ({
932
+ model: modifyFields(model, { notes: notes => [...notes, { id, name }] }),
933
+ }),
934
+ ClearedNotes: () => ({ model: modifyFields(model, { notes: () => [] }) }),
935
+ }),
936
+ })
937
+ -->
938
+ ```ts
939
+ // Version 1, as the first build shipped it: a note had a `title`. Kept as
940
+ // long as a stored operation may hold it; only the changed variant is copied.
941
+ const RenamedNoteV1 = Schema.TaggedStruct('RenamedNote', {
942
+ id: Schema.String,
943
+ title: Schema.String,
944
+ })
945
+ const MessageV1 = Schema.Union([RenamedNoteV1, Message.ClearedNotes])
946
+
947
+ // Version 2 is the application's own Message, where `title` became `name`.
948
+ const NoteHistory = migrations(MessageV1).next(App.Message, previous =>
949
+ Match.valueTags(previous, {
950
+ RenamedNote: ({ id, title }) => ({ _tag: 'RenamedNote' as const, id, name: title }),
951
+ ClearedNotes: cleared => cleared,
952
+ }),
953
+ )
954
+
955
+ const Notes = forApplication(App).make({
956
+ documentId: DocumentId.make('notes'),
957
+ shared: Projection.pick(App.model.notes),
958
+ durable: MessageSet.make(App, [Message.RenamedNote, Message.ClearedNotes]),
959
+ migrations: NoteHistory,
960
+ })
961
+ ```
962
+
963
+ Version 1 is the first Schema; each `next` adds a version with its Schema and
964
+ a function from the version before, encoded, to its own, encoded. Each step's
965
+ input is typed by the step before, so a step that returns the old shape does
966
+ not compile. Steps are written once and never changed: a later change adds a
967
+ step. A step must be pure and deterministic, with no clock and no random id:
968
+ the server reads an entry more than once (`admit`, then the journal), and
969
+ each read must find the same Message. It is given the version before checked
970
+ and normalized by that version's Schema. The history must end at
971
+ `App.Message` itself (`make` throws otherwise), and with no `migrations` the
972
+ Message is version 1, written exactly as before.
973
+
974
+ What each side does with it:
975
+
976
+ - **The replica** keeps each operation as it was written, with its
977
+ `messageVersion` (absent for version 1), in its outbox and on the wire, and
978
+ replays it upcast. So an outbox written by the old build opens and sends
979
+ under the new one.
980
+ - **The server** (`journalExchange`) hands every hook, `admit`, `validate`,
981
+ `authorize`, `reduce` and `stamp`, the Message upcast to the current version,
982
+ so a client claiming version 1 is judged exactly as the current Message
983
+ would be. It stores what commits at the current version, and recognizes a
984
+ retransmission by the operation as it was written, so an edit committed
985
+ before a deploy, whose acknowledgement was lost, still reads as committed.
986
+
987
+ When something will not read:
988
+
989
+ - **An older version no step reads** (its version's Schema refuses it, or a
990
+ step throws): a pending operation moves to `status.refused`, kept as
991
+ written with its payload, never sent, until `replica.dismiss(opId)`; the
992
+ rest of the outbox opens and sends. No Schema of this build reads it, so a
993
+ page shows it raw (`operation.message`, `operation.messageVersion`) or not at
994
+ all. The server rejects one it receives as `Unreadable { version }`, and the
995
+ replica keeps it in `refused` too. A committed one fails what has to read
996
+ it, with `UnreadableHistoryError` on the server: every exchange of a
997
+ replica whose cursor is before it (what that replica sends still commits,
998
+ and is acknowledged once a build reads the row), a retransmission of its
999
+ id, and, while the row is after the journal's stored snapshot, an append
1000
+ or load that rebuilds the document from that snapshot. A replica past the
1001
+ row is served, and other documents are not affected. Only a build with a
1002
+ step that reads it recovers it. The replica is told only "This server
1003
+ cannot read the history of this document"; the operation's id and
1004
+ version stay on the error, for the server's logs.
1005
+ - **A newer version** (a rollback, or an old client): the replica refuses to
1006
+ open on one in its outbox (`UnsupportedMessageVersionError`, nothing
1007
+ removed), and an exchange that receives one fails with it, recorded in
1008
+ `status.lastError` and retried on the backoff; the page reloads the new build
1009
+ when it sees that error (`ExchangeFailed` on `Diagnostics`). A server
1010
+ receiving one fails the exchange before appending anything, so the client
1011
+ waits with its edits rather than losing them.
1012
+
1013
+ Deploy order: the release of Sync that reads `messageVersion` goes to every
1014
+ server and client first, and skipping that step **loses edits**. A server on
1015
+ an older release rejects an operation carrying `messageVersion` as `Invalid`,
1016
+ and the replica drops it; a replica on an older release that is sent an
1017
+ `Unreadable` refusal drops that operation too, since it keeps no `refused`
1018
+ list. Then deploy an application's new version server first. A client on the
1019
+ old version keeps syncing until the first operation of the new version
1020
+ commits, then fails its exchanges until it reloads; its outbox is kept and
1021
+ sent after the reload.
1022
+
1023
+ A rollback of the Sync release itself meets a replica state holding
1024
+ `refused`, which an older release refuses to open
1025
+ (`InvalidReplicaHistoryError`). Do not clear storage on that error: the
1026
+ outbox is in the same state, and clearing it loses every unsent edit.
1027
+ Roll forward instead.
1028
+
1029
+ A server that builds an operation itself and appends it (`editsJournal`'s
1030
+ `absorbed`) writes it at the current version:
1031
+ `Notes.journalContract().operation.encode(operation)`. The journal takes a
1032
+ `WrittenOperation`, which is branded so that only Sync makes one: a decoded
1033
+ `Operation` names no version, would read as version 1, and is refused by the
1034
+ type. Changing the shared
1035
+ slice's own Schema is not covered: a stored snapshot or checkpoint of the old
1036
+ shape still needs a migration of your own.
1037
+
579
1038
  ### The URL, mirrors, and agents
580
1039
 
581
1040
  `mount` takes the application's own `subscriptions` and the `resources` Layer
@@ -659,6 +1118,30 @@ whole of it.
659
1118
  A `Storage` needs three members, and the loopback transport takes a handler in
660
1119
  place of a socket:
661
1120
 
1121
+ <!-- docs-check standalone
1122
+ import { Option, Schema } from 'effect'
1123
+ import { defineMessageUnion } from 'foldkit/message'
1124
+ import { MessageSet, Projection, Surface } from 'foldkit-surface'
1125
+ import { DocumentId, Sync } from 'foldkit-sync'
1126
+ const Model = Schema.Struct({
1127
+ todos: Schema.Array(Schema.Struct({ id: Schema.String, title: Schema.String })),
1128
+ })
1129
+ const Message = defineMessageUnion({
1130
+ CreatedTodo: { id: Schema.String, title: Schema.String },
1131
+ RenamedTodo: { id: Schema.String, title: Schema.String },
1132
+ })
1133
+ const App = Surface.application({
1134
+ Model,
1135
+ Message,
1136
+ initial: { todos: [] },
1137
+ update: model => ({ model }),
1138
+ })
1139
+ const TodoSync = Sync.forApplication(App).make({
1140
+ documentId: DocumentId.make('todos'),
1141
+ shared: Projection.pick(App.model.todos),
1142
+ durable: MessageSet.make(App, [Message.CreatedTodo, Message.RenamedTodo]),
1143
+ })
1144
+ -->
662
1145
  ```ts
663
1146
  import { Effect } from 'effect'
664
1147
  import type { Storage } from 'foldkit-sync'
@@ -672,7 +1155,7 @@ const memoryStorage = (): Storage => {
672
1155
  }
673
1156
  }
674
1157
 
675
- const program = Effect.gen(function* () {
1158
+ const inMemory = Effect.gen(function* () {
676
1159
  const replica = yield* TodoSync.openReplica(memoryStorage())
677
1160
  yield* replica.submit(Message.CreatedTodo({ id: 't1', title: 'Milk' }))
678
1161
  yield* replica.synchronize
@@ -724,8 +1207,17 @@ application come first.
724
1207
  exchange until its storage is cleared.
725
1208
  - **Stored state is from another version.** Opening fails with
726
1209
  `UnsupportedReplicaVersionError` naming both versions; the bytes are left for
727
- a deliberate migration or reset. Sync versions its own envelope only; your
728
- Message and snapshot migrations stay yours.
1210
+ a deliberate migration or reset. That versions Sync's own envelope; a
1211
+ durable Message's versions are the document's history
1212
+ ([change a durable Message](#change-a-durable-message)), and snapshot
1213
+ migrations stay yours.
1214
+ - **An operation from another build of the application.** An older version no
1215
+ step reads lands in `status.refused`, kept until dismissed; a newer one
1216
+ refuses to open the replica, or fails the exchange, with
1217
+ `UnsupportedMessageVersionError`, and nothing is removed. Moving an
1218
+ operation to `refused` is the one write an open makes, under the revision
1219
+ it read: if another tab wrote the store meanwhile, the open fails with
1220
+ `StorageError`, and opening again reads both.
729
1221
  - **The server answers nonsense.** Malformed exchanges, gaps in the committed
730
1222
  order, acknowledgements or rejections for operations the replica never sent:
731
1223
  each fails the exchange rather than touch local work.
@@ -748,7 +1240,7 @@ A replica reports to the reporter in the Effect context it runs in:
748
1240
  host takes one with the upgrade request beside it, used for everything that
749
1241
  upgrade runs:
750
1242
 
751
- ```ts
1243
+ ```ts sketch
752
1244
  export class SyncHost extends defineDocumentHost(DurableObject, {
753
1245
  sync: TodoSync,
754
1246
  openJournal: (_ctx, env) => openTodosJournal(env.DB),
@@ -769,7 +1261,7 @@ slice alone, and an application that is not a Foldkit runtime (a worker, a
769
1261
  server-side actor, a test) uses it directly.
770
1262
 
771
1263
  ```ts
772
- const program = Effect.gen(function* () {
1264
+ const worker = Effect.gen(function* () {
773
1265
  const storage = yield* Sync.indexedDb('todos/worker')
774
1266
  const replica = yield* TodoSync.openReplica(storage)
775
1267
 
@@ -795,7 +1287,9 @@ state, so an edit submitted while the request was in flight survives.
795
1287
 
796
1288
  The rest of the surface: `committed` (the base alone), `pending`, `cursor`,
797
1289
  `status`, `statusChanges`, `changes` (status and `shared` from one snapshot),
798
- `snapshot` (the same plus `nextLocalSequence`), `start`, and `close`.
1290
+ `snapshot` (the same plus `nextLocalSequence`), `start`, `dismiss` (forgets a
1291
+ `status.refused` operation), and `close`. `pending` and a rejection's
1292
+ `operation` are `WrittenOperation`s: as written, with `messageVersion`.
799
1293
 
800
1294
  ### The exchange
801
1295
 
@@ -804,21 +1298,34 @@ carries it.
804
1298
 
805
1299
  ```text
806
1300
  request cursor
807
- pending operations
1301
+ pending operations, as written (messageVersion past 1)
808
1302
  epoch, once the replica has heard one
809
1303
 
810
1304
  answer operations committed after the cursor, contiguous
811
1305
  acknowledged ids the server accepted without repeating them
812
1306
  rejected ids the server refused
813
- reasons? [{ opId, reason }]: why, for some of rejected; at most 500 characters each
1307
+ refusals? [{ opId, refusal }]: why, for some of rejected, one each; an encoded SyncRefusal
1308
+ reasons? [{ opId, reason }]: deprecated words, at most 500 characters each
814
1309
  checkpoint? { cursor, model } when the history is gone
815
1310
  more? true when another page follows
816
1311
  epoch? the server's history
817
1312
  ```
818
1313
 
819
1314
  An acknowledged operation leaves the outbox once its committed copy has
820
- arrived or the last page has been read. A rejected one leaves at once. The
821
- replica validates every field; see [when things go wrong](#when-things-go-wrong).
1315
+ arrived or the last page has been read. A rejected one leaves at once.
1316
+
1317
+ A custom server that reads operations through the codec, or appends them to
1318
+ a Durable journal itself, must do what `journalExchange` does with a newer
1319
+ `messageVersion` than its own (`codec.messageVersion`): fail the whole
1320
+ exchange before appending anything. Durable alone sees such an entry as an
1321
+ `InvalidOperationError`, and rejecting it by id loses the edit. It reads
1322
+ each sent entry with `codec.writtenFrom` before appending it, too: that is
1323
+ where a [server-only Message](#messages-only-the-server-appends), a replica
1324
+ id under `server/` or one past 128 characters is refused. Stored history is
1325
+ read with `journalContract().operation.decode`, which takes the server's own
1326
+ Messages; a journal decoding its rows with `codec.normalizeOperation` would
1327
+ refuse them. The replica validates every field; see
1328
+ [when things go wrong](#when-things-go-wrong).
822
1329
 
823
1330
  ### Transports
824
1331
 
@@ -882,7 +1389,8 @@ const Shared = Schema.Struct({ title: Title.schema })
882
1389
 
883
1390
  `Sync.define({ documentId, message, shared, empty, durable, replay })` is the
884
1391
  protocol primitive `forApplication` compiles to, for a client with no
885
- application to derive from. Its `replay` is unguarded; the author owns its
1392
+ application to derive from. It takes `migrations` (a `Sync.migrations`
1393
+ history) in place of `message`, and reads the Message from it. Its `replay` is unguarded; the author owns its
886
1394
  agreement with whatever applies the Message elsewhere.
887
1395
 
888
1396
  ### Joining an assembly
@@ -893,13 +1401,23 @@ runs nothing; Sync owns its runtime through `mount`.
893
1401
 
894
1402
  ## What Sync owns, and does not
895
1403
 
896
- Owns the replication contract (shared slice, durable set, replay, codecs,
897
- `journalContract`), the replica (outbox, base, cursor, optimistic view, status),
898
- reconciliation, the transport seam, and the optional presence and LWW helpers.
1404
+ Owns the replication contract (shared slice, durable and server-only sets,
1405
+ replay, codecs, `journalContract`), the replica (outbox, base, cursor,
1406
+ optimistic view, status), reconciliation, the transport seam, the server's own
1407
+ appends and reactions (`serverJournal`), and the optional presence and LWW
1408
+ helpers.
1409
+
1410
+ A reaction's cursor and parked runs are stored by `foldkit-durable`, beside
1411
+ the effect ledger that records each run; Sync decides when a run happens and
1412
+ what it appends. Whether an external call happened is the provider's to know.
1413
+
1414
+ Sync applies a durable Message's history wherever it reads an operation; the
1415
+ application owns the history itself (which Schemas it shipped and how each
1416
+ becomes the next).
899
1417
 
900
1418
  Does not own the rest of the Model, server-derived caches (`foldkit-remote`),
901
1419
  the authoritative order and storage (`foldkit-durable` or your server),
902
- peer-to-peer merging, or migration of your own Messages and snapshots.
1420
+ peer-to-peer merging, or migration of the shared slice's own Schema.
903
1421
 
904
1422
  ## Limits
905
1423