@spooky-sync/core 0.0.1-canary.240 → 0.0.1-canary.241

package/AGENTS.md

@@ -11,24 +11,35 @@ The sync engine. A `Sp00kyClient<S>` owns a local database (memory or IndexedDB)
    │  spky generate
    ▼
 schema.gen.ts (typed schema + SURQL_SCHEMA constant)
-   │  passed to SyncedDbConfig
+   │  passed to Sp00kyConfig
    ▼
-Sp00kyClient<S>  ── local store (memory | IndexedDB)
-       │            ↑↓ DBSP reactive query layer
-       │            ↑↓ mutation queue
-       └── SSP ── remote SurrealDB
+Sp00kyClient<S>  ── facade: every method = one saga run, selector or subscription
+       │
+       ├── Runtime (client/runtime.ts): one immutable ClientState, lanes, timers,
+       │     subscriptions, dirty -> materialize scheduling. `query:status`,
+       │     `activity:changed` and `health:changed` are all notified from here,
+       │     off state transitions - never emitted by a saga, so a change made
+       │     outside a sync round (a socket drop, say) still reaches subscribers
+       ├── Sagas (query/ mutation/ sync/ boot/): pure generators yielding effects
+       └── Interpreter (kernel/interpreter.ts) -> adapters (services/):
+             local store · remote socket · SSP wasm circuit · tabs broker · blobs
 ```
 
-Local mutations are applied optimistically, ingested into a DBSP layer that drives reactive query updates, then drained to the remote via SSP. Live updates flow back the same path.
+Local writes are applied optimistically (row + outbox entry in one local
+transaction), rendered through the outbox overlay, and drained to the remote in
+batches. Server membership arrives through one batched edge read (registration,
+poll, LIVE dirt) and bodies are fetched once across all queries. Rows are always
+re-materialized from state, never patched in place.
 
 ## Key exports (`src/index.ts`)
 
-- `Sp00kyClient<S>` — main class. Methods: `init()`, `create(id, payload)`, `update(table, id, payload, options?)`, `delete(table, idOrSelector)`, `query(table, opts?)`, `run(backend, route, payload)`, `bucket(name)`, `remoteQuery(sql, vars?)`, `useRemote(fn)`, `authenticate(token)`, `signOut()`. Plus `pendingMutationCount` / `subscribeToPendingMutations(cb)` (outbox depth) and `fetchingQueryCount` / `subscribeToFetchActivity(cb)` (queries mid-fetch, aggregate) for traffic indicators.
+- `Sp00kyClient<S>` — main class. Methods: `init()`, `create(id, payload)`, `update(table, id, payload, options?)`, `delete(table, id)`, `query(table, opts?)`, `preload(query, { signal? })`, `run(backend, route, payload)`, `bucket(name)`, `remoteQuery(sql, vars?)`, `useRemote(fn)`, `authenticate(token)`, `auth.signOut()`. Traffic indicators: `pendingMutationCount` / `subscribeToPendingMutations(cb)` (outbox depth), `fetchingQueryCount` / `subscribeToFetchActivity(cb)`. Failed-writes tray: `failedMutationCount`, `subscribeToFailedMutations(cb)`, `listFailedMutations()`, `retryFailedMutation(id)`, `discardFailedMutation(id)`.
+- `FailedMutation` — a row of the `_00_failed_mutations` tray (`mutationType`, `recordId`, `data`, `beforeRecord`, `error`, `revert`).
 - `BucketHandle` — file storage handle (`put`, `get`, `delete`, `exists`), plus the cache-aware `read`, `url`, `pin`/`unpin`, `evict`, `prefetch`. `get` is always remote; `read`/`url` go through the blob cache.
 - `services/blobs/` — durable cache for bucket bytes: OPFS holds the files, `_00_blob` holds a manifest that reconcile rebuilds from disk (so a wiped local store costs metadata, not the offline cache). Nothing expires on a timer; eviction is LRU under a byte budget only, skipping pinned and on-screen entries. Config: `blobCache` in `types.ts`.
 - `AuthService` — token management, sign-in/sign-out events.
 - `CrdtManager`, `CrdtField`, `cursorColorFromName`, `CURSOR_COLORS` — Loro-CRDT integration.
-- Types: `Sp00kyConfig`, `SyncedDbConfig` (re-exported by client-solid as the consumer-facing shape), `QueryTimeToLive`, `PersistenceClient`, `StoreType`, `UpdateOptions`, `RunOptions`.
+- Types: `Sp00kyConfig` (client-solid wraps it as `SyncedDbConfig`), `QueryTimeToLive`, `PersistenceClient`, `StoreType`, `UpdateOptions`, `RunOptions`, `PreloadOptions`.
 - Subpath: `@spooky-sync/core/otel` — `createOtelTransmit(endpoint)` for piping pino logs to OpenTelemetry.
 
 ## Common gotchas
@@ -40,15 +51,23 @@ Local mutations are applied optimistically, ingested into a DBSP layer that driv
 - **`db.remoteQuery(sql, vars?)` is the one-shot remote read.** It runs one SurrealQL statement through the client's remote path (connect gate, per-statement timeout, concurrency limit) and returns the SurrealDB result array; results are not synced into the local cache. Use it for counts and other reads the query builder can't express.
 - **`db.useRemote(fn)` hands out the bare SDK client.** Same cache bypass, but it skips the connect gate (a call during a reconnect fails with "Specify a namespace to use"). Reserve it for SDK features `remoteQuery` cannot reach (transactions, `live`).
 - **`@parent` columns are auto-populated.** Defined in the `.surql` schema as `record<user>` with the `-- @parent` annotation; never write them yourself — they're set from the auth context server-side.
+- **`db.preload` is a registered query.** Resolved before on this device (a `_00_view` row exists) it returns at once; never resolved it waits until the server's membership and every body are local. It holds no subscriber, so it is evicted a ttl after registration unless a view mounts the same query. `refresh`/`staleTime` are ignored; pass `signal` to abort a cold wait.
+- **A rejected write is not lost.** Application errors roll the local row back (create/update/delete) and move the mutation to `_00_failed_mutations`; the tray API above lists, retries (as a fresh write) or discards it. Network errors keep the row queued.
 
-## Module map (under `src/modules/`)
+## Module map (under `src/`)
 
-- `data/` — local store + DBSP reactive query layer.
-- `sync/` — SSP client, mutation queue, live-update ingestion.
-- `cache/` — query result cache with TTL.
-- `crdt/` — Loro CRDT manager for collaborative text fields.
-- `auth/` — token storage, sign-in flow.
-- `devtools/` — devtools bridge (talks to the browser extension and `@spooky-sync/devtools-mcp`).
+- `kernel/` — `Effect` union, saga driver + lanes, constants, interpreter.
+- `state/` — `ClientState`, lifecycle machine, reducers (they mark queries dirty), selectors (`overlay`, `needed`, `planFetch`, `settled`, `evictable`, `phaseTimings`).
+- `query/` — keys, membership rules, render set, SQL, and the sagas: `register`, `membership`, `fetch`, `materialize`, `lifecycle`.
+- `mutation/` — outbox rows (v2: `tableName`, `beforeRecord`, `createdAt`), `write`, `push` (drain + rollback), `tray`, jobs.
+- `sync/` — `poll`, `live`, `connection` (health, self-heal), `tabs` relay, pure policy.
+- `boot/` — `boot`, `auth-flip`, `bucket-switch`, `preload`.
+- `client/` — `runtime`, `router` (event -> saga + lane), `services` (adapters), `sp00ky-client` (facade), `bucket-handle`.
+- `testing/` — `runPure` (drive a saga with canned effect results), fake adapters/services, state builders.
+- `modules/` — auth, crdt, feature-flag, app-release, devtools, ref-tables (unchanged adapters).
+- `services/` — local store engines, remote socket + supervisor, SSP wasm wrapper, tabs broker, blobs, persistence, logger.
+
+Rules: nothing under `kernel/state/query/mutation/sync/boot` imports `services/` or touches `Date`, `setTimeout`, `crypto`; those directories are gated at 100% coverage. Legacy services are reached through the typed `service` effect (`kernel/effects.ts`, `ServiceCalls`).
 
 ## Pointers
 

package/README.md

@@ -1,21 +1,61 @@
-## Register Query Flow
+# @spooky-sync/core
 
-1. Query registering (local) - Data Module
-2. Query get local data - Data Module
-3. (Query registing (remote) - Sync Module)
+The sync engine behind `Sp00kyClient`. Everything that decides something is a
+pure function or a generator saga that yields effect descriptions; the only
+effectful code is the adapters under `services/` and the interpreter in
+`kernel/interpreter.ts`. State is one immutable `ClientState` value.
 
-## Query Local Updates
+## Layout
 
-1. Local Mutation is executed - Data Module
-2. Mutation is enqueued for remote - Sync Moduke
-3. New Record is ingested to DBSP - Data Module
-4. Related Queries are updated - Data Module
-5. (Sync Module syncs the mutation to remote - Sync Module)
+| Directory | Holds |
+| --- | --- |
+| `kernel/` | `Effect` union, the saga driver, lanes, constants, the interpreter |
+| `state/` | `ClientState`, the query lifecycle machine, reducers, selectors |
+| `query/` | keys, membership rules, render set, SQL builders, the query sagas (register, membership, fetch, materialize, lifecycle) |
+| `mutation/` | outbox rows, write / push / rollback / tray sagas |
+| `sync/` | poll, LIVE, connection health, shared-tabs relay sagas |
+| `boot/` | boot, auth flip, bucket switch, preload sagas |
+| `client/` | runtime (lanes, timers, subscriptions), router, adapters, the public facade |
+| `services/` | adapters: local store engines, remote socket, SSP wasm, tabs broker, blobs |
 
-## Query Live Updates
+## Query lifecycle
 
-1. Live Update is received - Sync Module
-2. Version is compared with local version - Sync Module
-3. If remote version is higher, the record is updated - Sync Module
-4. Record is ingested to DBSP Module - Data Module
-5. Related Queries are updated - Data Module
+A query is `cold` (never resolved on this device: rows come from a local
+predicate scan, bindings show a loader), `cached` (a durable `_00_view` row
+seeded its membership: rows paint from the store at once), `live` (a server
+membership set was accepted this session) or `view-lost` (the server dropped
+the view; rows are kept while it re-registers).
+
+1. `registerLocal` computes the keys, reads `_00_view`, builds the local SSP
+   view and publishes the entry. The runtime materializes it on the next tick.
+2. `registerRemote` registers on the server and reads edges + meta in one
+   request; `applyMembership` decides (non-empty always; empty only when the
+   server vouches for it) and writes `_00_view`.
+3. `fetchRows` pulls every body any query is missing, once, in parallel
+   chunks, and records versions. `settled` is a selector over that state.
+4. Later changes arrive as LIVE edge events (which only mark membership dirty)
+   or the poll; both go through the same batched re-read.
+
+Rendered rows are always `(membership ∪ (writes ∩ localView)) − deletes`
+where `writes`/`deletes` come from the outbox mirror in state, so unsynced
+local mutations show and a rejected one disappears the moment it is rolled
+back.
+
+## Mutations
+
+`write` runs one local transaction (row + `_00_pending_mutations` entry with
+`beforeRecord`), feeds the circuit and starts `drain`. The drain pushes up to
+50 statements per request and judges each on its own: accepted rows are
+acked (kept in the overlay until membership names them), rejected rows are
+rolled back and moved to `_00_failed_mutations` in the same transaction that
+deletes the pending row, a transport cut leaves the tail queued behind a
+backoff. The tray is exposed as `failedMutationCount`,
+`subscribeToFailedMutations`, `listFailedMutations`, `retryFailedMutation`
+and `discardFailedMutation`.
+
+## Testing
+
+Sagas run against canned effect results with `src/testing/run-pure.ts`; the
+interpreter and facade run against `src/testing/adapters.ts`. `pnpm test:run`
+runs the suite, `pnpm test:coverage` enforces the thresholds in
+`vitest.config.ts` (the saga core is held at 100%).