foldkit-remote 0.12.0 → 0.14.0

package/README.md

@@ -361,7 +361,15 @@ const overHttp = Remote.clientLayer(
 ```
 
 `headers` is read for each request, so a token that changes reaches the next
-call. Neither carries live data.
+call. A 403 whose body does not say `forbidden` itself, as Remote's server's
+does, comes from middleware in front of it refusing a session (a bare
+`{ error }` included): it fails a mutation as `Forbidden` and any other call
+with the status. Neither carries live data. For live data over the same `POST`s,
+`Remote.httpWithLive` keeps reads, queries, and mutations as `http` sends
+them and streams live requirements as a `text/event-stream` of changes;
+[`serveFetch`](../remote-server/README.md#over-fetch-workers) is the other
+end. Abandoning the stream aborts the request, which interrupts the
+subscription server-side.
 
 For a server in a worker or in the page (a sandbox), `foldkit-remote/port`
 speaks Effect's own RPC worker protocol over a `MessagePort`, so live data,
@@ -534,7 +542,8 @@ When a read sits at `Initial` and you expected data, ask the Model why with
 `Data.why`; see [Why is it still `Initial`?](#why-is-it-still-initial).
 
 `RemoteData.match` is exhaustive, so adding or omitting a state is visible at
-compile time.
+compile time. `RemoteData.answered(data)` is whether a read has an answer yet:
+`Ready`, `Refreshing`, `Failed` or `NotFound`, not `Initial` or `Loading`.
 
 ### Drawing one: `RemoteData.render`
 
@@ -669,8 +678,8 @@ the same way, with the request's error; see
 Live events carry cursors. One behind the Model's is a duplicate and is
 dropped; one ahead of the next is a **gap**, recorded in `RemoteModel.gaps`
 and not applied, so the screen keeps what the server last said rather than
-a history with a hole in it. The host resynchronizes and clears the gap; see
-[Live data](#live-data).
+a history with a hole in it. The live entry then restarts the stream and
+refetches what it covers; see [When a stream breaks](#when-a-stream-breaks).
 
 ## Normalized entities and selections
 
@@ -868,9 +877,12 @@ const ready = await Effect.runPromise(
 )
 ```
 
-`actives` is the record `Data.wiring` takes. The passes are bounded (8 by
+`actives` is the record `Data.wiring` takes. A `Surface.each` family is one
+entry of it: its parent's requirements join its instances', so the parent is
+fetched before what it reveals. The passes are bounded (8 by
 default, `{ passes }` to change it); a Surface still reading after the last
-fails the Effect with `RemoteUnsatisfied`, which names it, rather than
+fails the Effect with `RemoteUnsatisfied`, which names it — a still-reading
+family instance as `Child[key]` — rather than
 rendering it loading. A read the server leaves unanswered settles as missing,
 so it ends the loop; what runs out the passes is a chain whose every read
 reveals one more.
@@ -1065,7 +1077,9 @@ could match — predicate containment, which Remote deliberately does not do.
 missing a field the *body* reads), every match could be shown (no row missing a
 field the *Selection* reads), and the list is terminal at both ends. So
 empty-and-complete and empty-and-partial stay different answers — one means
-"none", the other means "none that I can see yet".
+"none", the other means "none that I can see yet". An input no interpreter may
+answer (a search holding NUL, which the server refuses too) gives no items and
+`complete: false`, rather than throwing in `update`.
 
 Nothing is created: no connection, nothing new to retain, nothing new to fetch.
 The server stays authoritative for which rows exist.
@@ -1204,7 +1218,8 @@ identity and by `entity\0id\0field` mark.
 A broken live stream is not a failed read. Its `ReadFailed` carries the
 `stream`, and records a gap in `RemoteModel.gaps` rather than failing any
 field: nothing was being read, and the values on screen are still what the
-server last said.
+server last said. The stream restarts on its own; see
+[When a stream breaks](#when-a-stream-breaks).
 
 ### When a field is withheld
 
@@ -1295,11 +1310,126 @@ A mutation that deletes says so. The server's outcome carries
 server names no list. Patches apply first, so an entity named both ways is
 deleted; a retry of the same request deletes nothing again.
 
+**Which lists a mutation changed is worked out from its answer.** Each patch
+says which row it wrote; the fields whose value differs from what the client
+held are what changed (an answer returning a whole row changes only what is
+new). Every loaded list over that Entity is judged by its query's body against
+the store the answer left:
+
+- a list whose body reads none of the changed fields stands;
+- a row it holds that still matches, or a row it does not hold that still does
+  not, leaves it standing;
+- a held row that stops matching leaves it, with nothing fetched: whether a
+  row belongs is the body's, judged exactly;
+- a row that starts matching, a row new to the client, a held row whose
+  ordering field changed, or a row whose order moves it in from past what a
+  list has loaded, is placed where the body's order puts it among the rows
+  loaded (`placeIn`), when
+  the order is placeable (`Query.placement`); a row that sorts past what is
+  loaded is left out of it (and out of what showed it), for the next page. A
+  row that was itself where a page met rows not loaded (the last before a
+  cursor, the first after one) cannot be told to have moved past it, its old
+  values being gone, so it invalidates the list instead;
+- anything else invalidates it, and its read entry fetches it again: a row the
+  client cannot judge (it lacks a field the body reads), an order it cannot
+  place (text with no declared collation, the server's own), or a row an
+  optimistic overlay shows; a read of it already in flight is restarted, since
+  it could answer with the rows before the write;
+- a list over a `Query.make` query, which declares no body, is invalidated by
+  any change to its Entity;
+- a row a list does not hold that changes a member the server's own read rule
+  filters on (an owner, a team: declared on the Entity as `ReadFilter`, which
+  `foldkit-guard` attaches) invalidates it: whether the row now belongs is the
+  server's to say, since the client runs no such rule. A row it holds is
+  judged by the body as usual. A new row counts, so under such a rule a create
+  refetches the lists over its Entity;
+- a list whose body declares no order leaves its order to the server, by
+  fields the client cannot see, so any change to a row it holds invalidates
+  it. Declare the order in the body (`Query.orderBy`) to keep a list that a
+  change provably did not move.
+
+**A placement can be checked against the server.** `Remote.make({ ...,
+verifyPlacement: true })` places as usual, and also fetches each list it placed
+what the server said in (an answer, as above, or a read or a live patch, as
+below); when the server's first page arrives it is compared with the order
+placed, and a disagreement is kept in `model.remote.placementMismatches` (and
+`Data.inspect`): an order the page does not share, or a row placing took out
+that the page still lists. A pending edit is not checked: the server does not have it
+yet. Turn it on in development and tests: a collation the client
+and the server disagree on then fails an assertion instead of misplacing a row
+for a reader.
+
+So a rename updates every view of the row and fetches nothing, a status change
+moves a row out of the lists filtered by status and into the others in their
+order, and a price change moves the row within a list sorted by price, all
+without a request.
+
+**A pending edit, and the server's word on a row, are placed the same way.**
+While a request's `optimistic` patch or a `Data.overlay` is pending, a list
+that holds the row shows it where the edit puts it, or leaves it out once it
+stops matching; the server's pages are untouched, and lifting the edit puts
+the row back. A read or a live `EntityPatched` that brings a new value for a
+row a list holds moves it in the list's pages, as an answer does. Neither
+brings a row into a list, since what the list selects of it may not be
+loaded, and neither fetches: a list it cannot place stands as it is.
+
+**An inserted row is drawn in order.** A `ConnectionChange.prepend` or
+`append` (a pending create's, a confirmed one's) and a live
+`ConnectionInsert` say a row is in a list; they do not say where. In a list
+the client can place, a read draws the row where the body's order puts it,
+or leaves it out when it sorts past what is loaded; one it cannot place (a
+field the order reads is missing, the order is the server's) stays at the end
+its change names. `Data.visibleItems(remote, connection)` gives a list's edges
+as its reads draw them, for a persistence `snapshot` that stores a list as
+shown.
+`Data.refresh` stays for what an answer cannot say, such as a server-side
+effect on rows it did not return.
+
 `Data.mutation(model, requestId)` reads what became of it from the Model:
 `Pending`, `Applied`, `Failed` with the error the server or transport gave, or
 `Unknown` for an id never started here (or settled so long ago it left the
 bounded ledger). A retry reuses its id, and the latest outcome wins.
 
+A mutation can say what the server may refuse it with, as data rather than a
+message to parse:
+
+```ts
+const SetPrice = Mutation.make('SetPrice', {
+  Input: { id: Schema.String, price: Schema.Number },
+  Output: {},
+  Refusal: Schema.Union([Refusal.field('price', Schema.String), Refusal.conflict]),
+})
+
+Data.refusal(model, requestId, SetPrice) // Option<{ _tag: 'Field', key: 'price', reason: string } | { _tag: 'Conflict' }>
+```
+
+A mutation can also be a declared write: `Mutation.write(name, write)` over a
+`foldkit-entity` `Write.update`, `Write.insert` or `Write.delete`. Its input is
+the write's, it answers nothing of its own, and `Data.mutate` shows an update's
+or an insert's row until the server answers, so it needs no `optimistic` list;
+a delete shows nothing until its answer removes the row everywhere. `keys` names the input keys
+an author changed: only those are shown, and the request carries them for the
+server to write; the input is still sent and validated whole.
+
+```ts
+const EditProject = Mutation.write(
+  'EditProject',
+  Write.update(Entity.input(Project, EditProjectInput), { id: 'id' }),
+)
+Data.mutate(model, EditProject, { id: 'p1', name: 'Apollo II', status: 'active' }, { keys: ['name'] })
+```
+
+`Refusal.field(key, reason)` names an input key and why; `Refusal.conflict`
+says the row moved on. The server sends the refusal encoded by this schema and
+`Data.refusal` decodes it by the same one; it is none while the mutation is
+pending or applied, or when it failed for another reason.
+
+A server may also refuse who is asking, whatever the mutation
+(`RemoteServer.forbid`). Then `Data.mutation(model, requestId)` is `Failed`
+with an error of `_tag: 'Forbidden'`, and `reason` when the server gave one,
+for the person who asked. Asking again as the same person is refused again,
+so show it rather than retry.
+
 Optimistic entity patches are **layers over the base store**, not inverse
 patches. If multiple mutations overlap, the visible cache is recomputed as base
 plus the still-pending layers, so settling one does not require trying to undo
@@ -1346,6 +1476,10 @@ const back = Data.lift(previewed, 'post-preview')
   `Failed`, with an `Overlaid` error naming what is missing, since nothing will
   fetch it. Overlay every field the Selection reads, or preview through a
   smaller Selection.
+- `Data.overlays(model)` lists the ids shown now, in the order they are drawn.
+  A caller that overlays from state of its own reconciles against it:
+  `foldkit-sync/remote` shows a replica's pending cell edits this way, so the
+  durable edit stays Sync's and Remote only draws it.
 
 ### Reading past what is only pending
 
@@ -1403,14 +1537,54 @@ ordered cursors:
 - the next cursor is applied;
 - a cursor ahead of the expected value is a **gap** and is not applied.
 
-A gap is recorded in `RemoteModel.gaps` so the host can resynchronize rather than
-silently accepting missing history. An in-order event or `GapCleared` clears it.
-
 Entity patches update normalized fields, deletes create tombstones, and
 connection insert/remove/invalidate events reconcile the same connection model
 used by query pages and optimistic overlays. A stream failure becomes a Remote
 failure Message rather than mutating anything out of band.
 
+### When a stream breaks
+
+A stream breaks when its transport fails or when an event arrives ahead of
+its cursor (a gap). Either way, what the server said meanwhile is lost: no
+server replays it, since `after` only numbers the events that follow. So
+recovery is a refetch, and the live entry does it on its own:
+
+```text
+break -> gap marked, restarts + 1 -> backoff
+      -> RefreshStarted (what the stream covers) + GapCleared -> resubscribe from the cursor
+                    \-> read entry refetches
+```
+
+- **The backoff is Sync's transport policy:** 50 ms after the first break,
+  doubling for each break in a row, at most 5 s, with ±20% jitter.
+  `retryBase` and `maxRetryDelay` in the `Data.subscriptions` options change
+  it. An event applied after a resubscribe starts it over.
+- **A gap stays until the restart.** An in-order event that a broken stream
+  still delivers is applied, but does not clear it. The event that arrived
+  ahead was dropped, and only the refetch brings back what it said.
+- **What a stream covers is its requirements.** A list the server watches
+  for it (a `ConnectionInvalidate` it sends) is the source's to re-send when
+  the stream opens; the cloudflare example's poll invalidates its list on
+  every open for this reason.
+
+`Data.liveStatus(model, active)` says how what one active entry reads live is
+doing, for a view that shows it:
+
+```ts
+Match.value(Data.liveStatus(model, Watched)).pipe(
+  Match.tagsExhaustive({
+    Idle: () => 'not live',
+    Live: () => 'live',
+    Reconnecting: ({ attempt }) => `reconnecting (attempt ${attempt})`,
+  }),
+)
+```
+
+`Reconnecting` lasts from the break until the stream has resubscribed, and
+carries the error, which is none for a gap. `RemoteModel.streams` holds each
+stream's restarts, its breaks in a row and its last error. `forget` clears it
+with every other server-derived fact.
+
 ## Retention and garbage collection
 
 Remote is a cache, so it should be allowed to forget facts no active feature
@@ -1481,6 +1655,58 @@ snapshot from another version/scope, an oversized snapshot, or malformed data is
 discarded and the planner refetches. `Hydrated` is itself a Remote Message, so
 hydration still changes the application through the reducer.
 
+### A cache kept across reloads: `Data.persistence`
+
+The calls above are the parts. `Data.persistence` puts them together as a
+wiring: it restores the snapshot stored under a key, and saves the cache under
+that key whenever it changes.
+
+```ts
+const persistence = Data.persistence({
+  key: model => `remote-cache:${model.userId}`,
+  scope: model => model.userId,
+  maxBytes: 512_000,
+})
+
+// Assembled beside `Data.wiring`; the runtime provides a `KeyValueStore`,
+// such as `BrowserKeyValueStore.layerLocalStorage`.
+const placements = Page.assemble(Data.wiring(actives), persistence)
+```
+
+The restore and the save each run as Subscriptions, when the key changes or the
+cache does:
+
+```text
+key -> restore -> Hydrated { from: key } -> restoredFrom = key -> save on each change
+```
+
+- **A restore keeps what the Model already holds.** It hydrates with
+  `merge: 'preserve-existing'`, so a fact resumed from the server, or read since
+  the page started, wins over the older snapshot.
+- **So a change of principal is `Data.forget`, in the same `update`.** What one
+  person read, left in memory, would be kept over the next one's snapshot and
+  then saved under their key; `forget` clears it as the key changes. The
+  `scope` check refuses a snapshot written for another reader; it cannot see
+  facts that are only in memory.
+- **Nothing is saved before its own key is restored.** `Hydrated` records the
+  key it came from in `model.remote.restoredFrom`. Until that key is restored,
+  nothing is saved, so the empty store at startup cannot overwrite the
+  snapshot it has not read yet. When the key changes (a different principal),
+  saving stops again until the new key's snapshot is in.
+- **A store that cannot be read restores nothing, and so nothing is saved over
+  it.** A stored snapshot from another version or scope, or one that will not
+  decode, is removed, and an empty snapshot is restored in its place.
+- **Saving is debounced** (`debounce`, 250 ms by default) and best-effort: a
+  full store keeps the snapshot it had, and the next change tries again. The
+  write waits; the snapshot does not: it is serialized on each change of the
+  cache, so under a steady stream of live changes keep it small, with
+  `maxBytes` or a `snapshot` that stores only what a page shows. A
+  cache grown past `maxBytes` removes its key, so an older, smaller snapshot
+  cannot outlive it.
+
+`snapshot` replaces `snapshotOf` when an application keeps something else, such
+as only the edges a page shows.
+
 ## Server rendering: `Remote.resume`
 
 A snapshot is built for a cache that survived a reload: it keeps the whole
@@ -1635,6 +1861,8 @@ Data.explain(model, projects, { surfaces: actives })
 // activation: [{ surface: 'ProjectPage', path: ['route'], tag: 'Project' }]
 ```
 
+A `Surface.each` family reports each reading instance as `Child[key]`.
+
 There is still no **executor**, because what answers a query is a
 `RemoteClient` Layer in the runtime rather than a value in the Model — the same
 purity that lets this be replayed from a recorded Model is what keeps it out of
@@ -1773,7 +2001,7 @@ const ref = Query.first(25)(ProjectsByOwner.ref({ ownerId }))
 const page = yield* Remote.query(ref)
 
 updateRemote(remote, Remote.queryMessage(ref, page))
-Remote.visibleItems(remote, ref)
+Data.visibleItems(remote, ref)
 ```
 
 `items`, `hasNext`, `hasPrevious`, and `isGapped` describe the known region.
@@ -1782,6 +2010,31 @@ a refresh/query result settles it. A `ConnectionMerged` page with
 `refreshes: true` replaces a stale connection's pages rather than merging into
 them.
 
+### Where a row falls: `placeIn`
+
+A query whose body declares a placeable order (`Query.placement` in
+`foldkit-entity`: numbers, booleans, and text under a portable collation) can
+have a changed row placed in its loaded pages by the client, by the same order
+the server compiles. Such a connection also reads every field its body filters
+and orders by, so the rows it holds can be compared with one that changed.
+
+```ts
+// Schematic: `ProjectsByRank` is a Query.define ordered by a number, and
+// `connection` one of `remote.connections`. test/placement.test.ts runs it.
+placeIn(remote.entities, connection, ProjectsByRank.body!, {}, 'Project:p7')
+// Before { segment, edge } | End { segment } | Outside | Unknown { reason }
+```
+
+- `Before` and `End` are positions among loaded edges: before an edge, or at
+  the end of a segment that ends the list (`Terminal`).
+- `Outside` is a row that sorts past a `Cursor` boundary or into a gap between
+  segments: it belongs to rows the client has not read, so it is not shown.
+- `Unknown` is anything not judged exactly: a body that is not placeable, a
+  row or an edge lacking a field the order reads, keys that do not compare.
+
+It is pure and changes nothing. A mutation's answer is placed through it, as
+above.
+
 ### Mutations by hand
 
 `Remote.mutateInto(bound, model, mutation, input, requestId, { optimistic, now })` is