foldkit-remote 0.5.0 → 0.7.0

package/README.md

@@ -22,6 +22,26 @@ disposable view of it. If an edit is client-authored and must survive offline,
 restart, or network failure until it converges, that belongs to
 [`foldkit-sync`](../sync) and [`foldkit-durable`](../durable), not Remote.
 
+## Find what you need
+
+This README is long because it covers everything Remote does. Read the
+[sixty-second example](#sixty-seconds-one-entity-one-surface) first, then jump
+to what you are doing:
+
+| Task | Start here |
+| --- | --- |
+| See it work before there is a server | [A backend held in memory](../remote-server/README.md#a-backend-held-in-memory) |
+| Understand `Initial`, `Loading`, or a stale value | [`RemoteData`](#remotedata-what-does-the-model-know-right-now) |
+| Find out why a read stays `Initial` | [`Data.why`](#remotedata-what-does-the-model-know-right-now) |
+| Show a failed read, and let the user retry | [When a read fails](#when-a-read-fails) |
+| Load an ordered, paged list | [Queries and pagination](#queries-and-pagination) |
+| Filter a list already on screen | [Filtering a loaded list](#filtering-a-loaded-list-without-asking-the-server) |
+| Save, and show the result before the server answers | [Mutations](#mutations-and-optimistic-state) |
+| Fetch outside an active screen | [Policies and prefetch](#reading-policies-and-prefetch) |
+| Keep data across a reload, or seed it from SSR | [Hydration](#persistence-and-hydration) |
+| Release data no active feature needs | [Retention](#retention-and-garbage-collection) |
+| Answer the requests on the server | [How the server packages fit](#how-the-server-packages-fit) |
+
 ## Which state belongs here?
 
 A useful rule across Foldkit Plus is **one owner per datum**:
@@ -55,67 +75,17 @@ ownership.
 
 ## The mental model
 
-The client side is a Foldkit Submodel:
-
 ```text
-                         pure
-                    Projection read
-                         |
-                         v
-+-------------+     +---------+      +------+
-| Foldkit     |---->| Surface |----->| View |
-| Model       |     +---------+      +------+
-|             |
-| Remote.Model|
-| normalized  |
-| cache       |
-+------+------+ 
-       ^
-       | Remote Message
-       |
-+------+-------+
-| Data.reduce |
-+------+-------+
-       ^
-       |
-       | read/query/mutate/live
-       |
-+------+--------+
-| RemoteClient |
-+------+--------+
-       |
-       v
-     server
+Surface Projection → requirements → active Subscription → RemoteClient I/O
+        ↑                                                    |
+        └──────── read Model ← Data.reduce ← Remote Message ──┘
 ```
 
-The other half of the model is the **requirement planner**:
-
-```text
-Surface says:
-  "I need Project p1: id, name, owner.name"
-
-                    |
-                    v
-Remote compares that requirement
-with the normalized cache in Model
-
-                    |
-                    v
-missing:
-  Project:p1.name
-  User:u7.name
-
-                    |
-                    v
-active subscription performs one read
-
-                    |
-                    v
-result -> Remote Message -> Data.reduce -> Model
-```
-
-That separation is why reading stays pure. Rendering a Surface never starts a
-request.
+A Projection reads the current cache and carries requirements such as
+`Project:p1.name` and `User:u7.name`. An active Subscription compares those
+requirements with the cache, then requests missing or stale fields. The
+result returns as a Message; only reducing it changes what the next read sees.
+Rendering the same Projection twice starts no work by itself.
 
 ## Four pieces to remember
 
@@ -133,7 +103,7 @@ Everything else builds on those four pieces.
 ## Install
 
 ```bash
-pnpm add foldkit-remote
+pnpm add foldkit-remote foldkit-entity
 ```
 
 `foldkit` and `effect` are peer dependencies; `foldkit-surface` comes with the
@@ -164,12 +134,10 @@ const ProjectSummary = Entity.select(Project, {
 ```
 
 The entity comes from [`foldkit-entity`](../entity), which declares a domain
-without Remote in it. Remote also has an `Entity.make` of its own, and accepts
-either — but only a `foldkit-entity` entity has addressable `fields`, so only
-that one can be given [relations](#entities-declared-with-foldkit-entity),
-derived members, or a [query body](#queries-and-pagination). Reach for
-`Entity.make` when a domain is small enough that Remote is all it will ever
-need; reach for `Entity.define` otherwise, which is most of the time.
+without Remote in it, so the same declaration can serve the server's database
+binding and your forms too. It is also what gives an entity
+[relations](#entities-declared-with-foldkit-entity), derived members, and a
+[query body](#queries-and-pagination) to point at.
 
 Embed Remote's Submodel in the application and bind the domain to that field:
 
@@ -218,7 +186,7 @@ const App = Surface.application({
 })
 
 const Data = Remote.make({
-  model: App.fields.remote,
+  model: App.model.remote,
   entities: [Project],
 })
 ```
@@ -291,7 +259,14 @@ Remote.Model
 ProjectPage now reads Ready(...)
 ```
 
-Finally, provide the client implementation to the runtime:
+The first example starts on the `home` route, so no project read runs yet.
+Your route update must activate `{ _tag: 'project', projectId: 'p1' }`, and the
+runtime must install `subscriptions`. Expect `Initial` until work starts,
+`Loading` during the first request, and `Ready` once its result is reduced.
+
+Finally, provide the client implementation to the runtime. Here `rpcClient`
+is your configured Effect RPC client, not a value created by the declarations
+above; the [server guide](../remote-server/README.md) supplies the other side:
 
 ```ts
 const clientLayer = Remote.clientLayer(rpcClient)
@@ -301,6 +276,21 @@ const clientLayer = Remote.clientLayer(rpcClient)
 `RemoteClient` service used by subscriptions and Commands. The transport itself
 is not owned by Remote.
 
+To see the page work before you have a server, give it rows instead. A backend
+held in memory answers through the same handlers a real server uses:
+
+```ts
+import { RemoteServer } from 'foldkit-remote-server'
+
+const clientLayer = RemoteServer.memory({
+  domain: Data,
+  rows: { Project: [{ id: 'p1', name: 'Apollo', status: 'active' }] },
+}).layer
+```
+
+It is for a first run, a test or a demo. See
+[`RemoteServer.memory`](../remote-server/README.md#a-backend-held-in-memory).
+
 ## One list per application with `Data.wiring`
 
 The integration steps above — spread `Remote.messages`, reduce by tag, derive
@@ -354,15 +344,19 @@ missing ------------------------------------------------> Loading
 
 server says entity is absent ---------------------------> NotFound
 stored value fails Selection decoding -----------------> Failed
+a read or a list's query fails ------------------------> Failed (with the old value, if any)
 ```
 
 The states are:
 
 - **`Initial`** — required data is absent and **nothing is currently fetching it**.
-- **`Loading`** — required data is absent and a read is in flight.
+- **`Loading`** — required data is absent and a read, or a list's query, is in
+  flight.
 - **`Ready`** — every selected field is present and decodes.
 - **`Refreshing`** — the current value remains visible while it is being refetched.
-- **`Failed`** — stored server data does not decode against the Selection.
+- **`Failed`** — stored server data does not decode against the Selection, or
+  the read or query behind it failed. A value that was already shown is kept as
+  `previous`.
 - **`NotFound`** — the entity is represented by a tombstone: a live event
   deleted it, or the server was asked for it by id and answered without it. That
   id is then known absent, whether it never existed, is gone, or is not this
@@ -376,6 +370,21 @@ no active Surface may remain `Initial` forever. Rendering a spinner for
 `Initial` can therefore hide an activation/wiring mistake; `Loading` is the
 state that actually means "wait for this request."
 
+When a read sits at `Initial` and you expected data, ask the Model why:
+
+```ts
+Data.why(model, projection, { surfaces }) // the record you give Data.subscriptions
+// { state: 'Initial', reason: 'NotFetching', surfaces: ['ProjectPage'],
+//   message: 'ProjectPage reads it and is active, yet nothing is fetching it. …' }
+```
+
+`NotObserved` means no active Surface reads it: the Surface is not in the
+record, or its params are `undefined` for this Model. `NotFetching` means one
+does, and nothing started a request, which almost always means Remote's
+Subscriptions are not installed in the runtime. Without `surfaces` it can only
+say `Unknown`. For every other state it says what the state means in words,
+and a failed request's message says that nothing retries it on its own.
+
 `RemoteData.match` is exhaustive, so adding or omitting a state is visible at
 compile time.
 
@@ -571,7 +580,7 @@ A mutation's `optimistic` operations show over the store while it is in flight.
 `Data.lift`: a preview, in every Selection and view, of something not yet sent.
 
 ```ts
-const previewed = Data.overlay(model, 'post-preview', [Project.patch(id, { name: draft })])
+const previewed = Data.overlay(model, 'post-preview', [Remote.patch(Project, id, { name: draft })])
 const back = Data.lift(previewed, 'post-preview')
 ```
 
@@ -623,7 +632,7 @@ const ProjectsByOwner = Query.make('ProjectsByOwner', {
 })
 
 const Data = Remote.make({
-  model: App.fields.remote,
+  model: App.model.remote,
   entities: [User, Project],
   queries: [ProjectsByOwner],
 })
@@ -706,6 +715,159 @@ their required fields. `Data.next` / `Data.previous` preserve the page size and
 use the loaded boundaries; `hasNext` / `hasPrevious` come from those boundaries,
 not from guessing based on row counts.
 
+### Filtering a loaded list without asking the server
+
+A search box over a page you already have should not be a round trip.
+`Data.filtered` narrows a loaded list by a query body:
+
+```ts
+const found = Data.filtered(model, projects, ActiveProjects, {})
+
+found.items     // decoded exactly as the list decodes them
+found.complete  // whether the answer was about the whole list
+```
+
+**It filters a list; it does not run a query**, and the distinction is the
+design. "Which rows match" would need to know this list holds every row the body
+could match — predicate containment, which Remote deliberately does not do.
+"Which rows *of this list* match" is decidable from what is already here.
+
+`complete` is false unless three things hold: every edge could be judged (no row
+missing a field the *body* reads), every match could be shown (no row missing a
+field the *Selection* reads), and the list is terminal at both ends. So
+empty-and-complete and empty-and-partial stay different answers — one means
+"none", the other means "none that I can see yet".
+
+Nothing is created: no connection, nothing new to retain, nothing new to fetch.
+The server stays authoritative for which rows exist.
+
+### Judging the rows you already hold
+
+A connection's rows are edges the server delivered, and a query body describes
+*which rows* — so until something evaluates the body on the client, nothing here
+can say whether a row belongs to a query. `Remote.matching` says:
+
+```ts
+const judged = Remote.matching(Data.storeOf(model), ProjectsByOwner, { ownerId: 'u1' })
+
+judged.matched  // keys satisfying the body, in the order it asks for
+judged.skipped  // keys held, but missing a field the body reads
+```
+
+It is pure, and it runs the **same interpreter the server checks itself
+against** — the reference implementation in
+[`foldkit-entity`](../entity) — so a body means one thing in both places. The
+conformance suite is run over a store as well as over rows and a table.
+
+Three things about it are deliberate.
+
+**It answers "which of the rows I hold match", never "which rows match."** Those
+are different questions, and the difference is not recoverable from a list of
+keys — so it is in the shape. A caller that knows it holds the whole population
+(a connection terminal at both ends) is the one that can turn this into a
+complete answer.
+
+**`skipped` is the honest half.** A row missing a field the body reads cannot be
+judged: it is neither a match nor a non-match. Dropping it silently would turn
+"I could not tell" into "no", so it is named instead, and the caller decides
+whether to fetch the field, ask the server, or say the answer is partial.
+
+**It takes the input decoded and encodes it itself.** A store holds wire values,
+so a comparison has to happen in that space — and handing an interpreter a
+decoded value is a mistake *no runtime error catches*: the answer is simply
+empty, which reads exactly like a correct one. Doing the encoding here means a
+caller cannot get it wrong.
+
+It is **not authorization**. On the server a compiled `where` is conjoined with
+the binding's `visible` rule; there is no `visible` here. This is safe only
+because the client holds only rows the server already released to it, and a
+local filter is not an access decision.
+
+### An input that changes as fast as someone types
+
+A connection's identity is its query plus its **input**, which is exactly right
+for caching and exactly wrong for a search box. Bound naively, every keystroke
+is a different connection, a different request, and a different thing to retain:
+
+```ts
+// Don't: `postSearch` changes per keystroke, so the query's input does too.
+h.OnInput(text => Message.Searched({ text }))
+```
+
+Remote has no debounce, and should not: its job is to be a faithful function of
+the Model, so if the Model says the search is `Engi` then `Engi` is what the
+query asks. **The debounce belongs between the input Message and the Model field
+the query reads** — which is ordinary application state, and
+[`foldkit-primitives`](../primitives) already has the piece:
+
+```ts
+import { debounce } from 'foldkit-primitives/time'
+
+const SearchInput = debounce({ name: 'PostSearch', value: Schema.String })
+const SearchBox = Bundle.declare(SearchInput, 'search')
+
+// `latest` is what the box shows, so typing stays immediate.
+// The settled OutMessage is what moves the field the query reads.
+Page.at(SearchBox, {
+  args: { delayMs: 250 },
+  onOut: out => model => ({ model: evo(model, { postSearch: () => out.value }) }),
+})
+```
+
+Two fields, deliberately: the one the box draws changes on every keystroke, and
+the one the query reads changes a quarter second after the last one.
+[`examples/entity`](../../examples/entity) does exactly this.
+
+The same applies to any high-frequency query input — a slider, a map viewport, a
+date scrubber. What makes a search box the usual case is that the query is often
+written *for* it: `Expr.contains` over an empty string matches everything, so an
+empty box and a filled one are one query rather than two.
+
+### A page that overruns its window is refused
+
+The client asks for `first: 25`, and a page carrying more than that is a
+protocol disagreement rather than a windfall: the client cannot tell which of
+the edges the window meant, and the connection's boundaries stop describing what
+it holds. Such a page becomes a `QueryFailed` with a named protocol error, and
+none of its edges reach the store — a connection already loaded is left exactly
+as it was, so a refresh that overruns cannot replace good rows with a rejected
+page.
+
+A window with neither `first` nor `last` bounds nothing: `after`/`before` says
+where to start, not how much to take.
+
+### When a read fails
+
+A failure is kept on the Model until something settles it, and the read says
+so. Queries fail per connection, and entity reads per field. A read with
+nothing to show reads `Failed` with the error. One that had a value (a failed
+refresh, or a failed "load more") reads `Failed` with that value as
+`previous`, which `RemoteData.render` draws as data with a `Stale` freshness,
+so it stays on screen. A failure reaches every read that needs it: a list
+fails when one of its rows' fields failed, and a project read fails when its
+owner's did.
+
+**It is not retried on its own.** A persistent error would otherwise be asked
+again every time some unrelated read restarted the entry. What failed is left
+out of the plan, and everything else is still fetched: the rows a failed list
+holds, the other fields of an entity. What asks again:
+
+- `Data.refresh(model, projection)` — the retry button. It works on something
+  that never loaded, too.
+- The value arriving anyway: a page for a list, or a read, live patch or
+  mutation result that writes the field.
+- The server invalidating a list over a live stream, or deleting the entity.
+- Retention dropping it: something nothing reads any more forgets its failure,
+  so coming back to it later asks the server again.
+
+`Remote.inspect(model.remote).failures` lists what failed, by connection
+identity and by `entity\0id\0field` mark.
+
+A broken live stream is not a failed read. Its `ReadFailed` carries the
+`stream`, and records a gap in `RemoteModel.gaps` rather than failing any
+field: nothing was being read, and the values on screen are still what the
+server last said.
+
 ## Mutations and optimistic state
 
 Register mutations on the domain:
@@ -730,7 +892,7 @@ case 'ClickedRename': {
     RenameProject,
     { id, name },
     {
-      optimistic: [Project.patch(id, { name })],
+      optimistic: [Remote.patch(Project, id, { name })],
     },
   )
 
@@ -744,6 +906,12 @@ case 'ClickedRename': {
 2. it returns the Command that calls `RemoteClient` and eventually emits
    `MutationSucceeded` or `MutationFailed`.
 
+`Remote.patch(entity, id, values)` builds one entity's patch, and
+`Remote.ref(entity, id)` names one row. Both take the entity you declared,
+whether with `foldkit-entity` or Remote's `Entity.make`, and check `values`
+against its fields in their wire shape: a relation is written as its ref key,
+such as `'User:u1'`.
+
 Those settlement Messages go through `Data.reduce` like every other Remote fact.
 The request id comes from the Remote Model's sequence, so `update` stays pure.
 Settling is idempotent per `requestId`.
@@ -771,10 +939,10 @@ import { ConnectionChange } from 'foldkit-remote'
 
 const { model: started, command } = Data.mutate(model, AddComment, input, {
   optimistic: ({ tempId }) => [
-    Comment.patch(tempId, { id: tempId, body: input.body }),
+    Remote.patch(Comment, tempId, { id: tempId, body: input.body }),
     ConnectionChange.prepend(
       CommentsForPost.ref({ postId }),
-      Comment.ref(tempId),
+      Remote.ref(Comment, tempId),
     ),
   ],
 })
@@ -837,7 +1005,8 @@ Sync operation
 ```
 
 If losing an unsent edit would be data loss, it belongs to Sync rather than
-Remote.
+Remote. For the same reason, an optimistic overlay is only a temporary view of
+server-owned data: do not persist one and treat it as a queue of edits.
 
 ## Live data
 
@@ -882,27 +1051,34 @@ and refetch; Sync recovery must preserve unsent client intent.**
 
 ## Persistence and hydration
 
-Persist only the disposable entity cache, not session machinery such as live
-cursors, optimistic layers, mutation bookkeeping, gaps, or retention roots.
+A snapshot is `{ entities, connections }`. The entity cache is disposable and
+always goes; a connection goes only if named, and keeps its edges alone — never
+a cursor, which may name server state that is gone. Session machinery (live
+cursors, optimistic layers, mutation bookkeeping, gaps, retention roots) is
+never in one.
 
 ```ts
-import { emptyStore, RemotePersistence } from 'foldkit-remote'
+import { RemotePersistence } from 'foldkit-remote'
 
 // SSR: prefetch on the server, embed, then hydrate on the client.
-const snapshot = RemotePersistence.dehydrate(
-  Data.storeOf(loaded),
+// `snapshotOf` names the connections that survive; none, here.
+const text = RemotePersistence.dehydrate(
+  RemotePersistence.snapshotOf(loaded.remote),
   { scope: userId },
 )
 
+const restored =
+  RemotePersistence.hydrate(text, { scope: userId }) ?? RemotePersistence.emptySnapshot
+
 Data.reduce(model, {
   _tag: 'Hydrated',
-  entities:
-    RemotePersistence.hydrate(snapshot, { scope: userId }) ?? emptyStore,
+  entities: restored.entities,
+  connections: restored.connections,
   merge: 'preserve-existing',
 })
 
 // Or persist through Effect's KeyValueStore.
-RemotePersistence.save(store, {
+RemotePersistence.save(RemotePersistence.snapshotOf(model.remote), {
   key: 'remote-cache',
   scope: userId,
   maxBytes: 512_000,
@@ -974,6 +1150,7 @@ into private state:
 ```ts
 Data.inspect(model)            // serializable cache/domain summary
 Data.plan(model, projection)   // the entity plan a read would execute
+Data.explain(model, query)     // what one query read is, and currently is
 Remote.planQueries(...)        // missing/stale query work
 Remote.inspectEntity(...)      // one normalized entity
 ```
@@ -992,6 +1169,53 @@ asked. That is deliberate. Runtime activity is not a second source of truth
 here, and a view that rendered from it would stop being reproducible from the
 Model.
 
+### Explaining one query read
+
+A query read is assembled from more pieces than it looks like: a definition, an
+input, a connection identity that excludes the window, a window that does not,
+and a Selection that is a slice asked of the server. `Data.explain` puts them in
+one serializable value, beside what the read answers from this Model right now:
+
+```ts
+Data.explain(model, projects)
+// {
+//   domain: 'remote',
+//   query: 'ProjectsByOwner',
+//   input: { ownerId: 'u1' },
+//   identity: 'ProjectsByOwner\u0000{"ownerId":"u1"}',
+//   window: { first: 25 },
+//   select: { entity: 'Project', fields: ['id', 'name'] },
+//   body: 'FROM Project\nWHERE Project.ownerId = $ownerId\nORDER BY Project.name DESC',
+//   dependencies: { fields: [...], inputs: ['ownerId'], operations: ['eq'] },
+//   state: 'Ready',
+// }
+```
+
+`body` is the query's meaning as readable text, and it is neither SQL nor
+whatever the backend compiled — an input shows as `$ownerId` rather than a bound
+parameter, and `contains` is named rather than rendered as somebody's `like`. A
+definition made with `Query.make` has no `body`, because its meaning lives on
+the server that answers it.
+
+`state` comes from the projection's own read, so an explanation and the view
+cannot disagree about whether the data is there.
+
+A Projection cannot say which Surfaces read it — several may — so pass the
+active record `subscriptions` takes and `explain` reports all of them, plus why
+each is active where that is a readable fact (`Surface.when`) rather than a
+callback (`Surface.at`):
+
+```ts
+Data.explain(model, projects, { surfaces: actives })
+// surfaces:   ['ProjectPage']
+// activation: [{ surface: 'ProjectPage', path: ['route'], tag: 'Project' }]
+```
+
+There is still no **executor**, because what answers a query is a
+`RemoteClient` Layer in the runtime rather than a value in the Model — the same
+purity that lets this be replayed from a recorded Model is what keeps it out of
+reach.
+
 ## Advanced: the kernel
 
 Everything above compiles to the lower-level exports below; there is not a
@@ -1008,6 +1232,11 @@ place in an application Model. It exposes the Remote `Model`, `initial`,
 `Remote.at(definition, modelRef)` binds that domain to a `ModelRef`;
 `Remote.make` combines definition and binding in one step.
 
+`Entity.make(name, schema)` is Remote's own entity descriptor, and what a
+`foldkit-entity` Entity is compiled to (`Entity.from`). Remote accepts either
+wherever it takes an entity. Declaring one directly skips `foldkit-entity`,
+and with it relations, derived members and query bodies, which is only worth
+it for a domain that Remote is all it will ever need.
 `Selection.make(Entity, { ... })`, `Selection.connection(...)`, and
 `Entity.patch(ref, values)` are the kernel constructors behind the entity
 helpers. `Remote.select(bound, selection)(id)` is the lower-level form of