@spooky-sync/core 0.0.1-canary.32 → 0.0.1-canary.321

package/AGENTS.md

@@ -0,0 +1,81 @@
+# `@spooky-sync/core` — agent guide
+
+## What this package is
+
+The sync engine. A `Sp00kyClient<S>` owns a local database (memory or IndexedDB), a remote SurrealDB connection, a mutation queue, and a CRDT manager. It's framework-agnostic — UI bindings live in `@spooky-sync/client-solid` (or future `client-react`, etc.). Most app code touches this package indirectly through a UI binding's `useDb()` hook.
+
+## Mental model
+
+```
+.surql schema
+   │  spky generate
+   ▼
+schema.gen.ts (typed schema + SURQL_SCHEMA constant)
+   │  passed to Sp00kyConfig
+   ▼
+Sp00kyClient<S>  ── facade: every method = one saga run, selector or subscription
+       │
+       ├── Runtime (client/runtime.ts): one immutable ClientState, lanes, timers,
+       │     subscriptions, dirty -> materialize scheduling. `query:status`,
+       │     `activity:changed` and `health:changed` are all notified from here,
+       │     off state transitions - never emitted by a saga, so a change made
+       │     outside a sync round (a socket drop, say) still reaches subscribers
+       ├── Sagas (query/ mutation/ sync/ boot/): pure generators yielding effects
+       └── Interpreter (kernel/interpreter.ts) -> adapters (services/):
+             local store · remote socket · SSP wasm circuit · tabs broker · blobs
+```
+
+Local writes are applied optimistically (row + outbox entry in one local
+transaction), rendered through the outbox overlay, and drained to the remote in
+batches. Server membership arrives through one batched edge read (registration,
+poll, LIVE dirt) and bodies are fetched once across all queries. Rows are always
+re-materialized from state, never patched in place.
+
+## Key exports (`src/index.ts`)
+
+- `Sp00kyClient<S>` — main class. Methods: `init()`, `create(id, payload)`, `update(table, id, payload, options?)`, `delete(table, id)`, `query(table, opts?)`, `preload(query, { signal? })`, `run(backend, route, payload)`, `bucket(name)`, `remoteQuery(sql, vars?)`, `useRemote(fn)`, `authenticate(token)`, `auth.signOut()`. Admin impersonation (`impersonation.enabled` in sp00ky.yml, admins only, server-enforced): `auth.impersonate(targetId, reason)`, `auth.stopImpersonating()`, `auth.impersonation`, `auth.subscribeImpersonation(cb)`, `auth.searchImpersonationTargets(search)`; while active the client injects a warning bar with a Stop button and shifts the page down onto a hazard-striped layer. `impersonationBanner` in the client config restyles it (`theme`), replaces it (`mode: 'custom'` + `client.acknowledgeImpersonationBanner()`, else it falls back after ~2.5s) or turns it off (`mode: 'none'`). See `/docs/client/impersonation`. Traffic indicators: `pendingMutationCount` / `subscribeToPendingMutations(cb)` (outbox depth), `fetchingQueryCount` / `subscribeToFetchActivity(cb)` (queries a user waits on; a query that already shows rows and is only re-registering, as after a reload, is not counted). Failed-writes tray: `failedMutationCount`, `subscribeToFailedMutations(cb)`, `listFailedMutations()`, `retryFailedMutation(id)`, `discardFailedMutation(id)`.
+- `FailedMutation` — a row of the `_00_failed_mutations` tray (`mutationType`, `recordId`, `data`, `beforeRecord`, `error`, `revert`).
+- `BucketHandle` — file storage handle (`put`, `get`, `delete`, `exists`), plus the cache-aware `read`, `url`, `pin`/`unpin`, `evict`, `prefetch`. `get` is always remote; `read`/`url` go through the blob cache.
+- `services/blobs/` — durable cache for bucket bytes: OPFS holds the files, `_00_blob` holds a manifest that reconcile rebuilds from disk (so a wiped local store costs metadata, not the offline cache). Nothing expires on a timer; eviction is LRU under a byte budget only, skipping pinned and on-screen entries. Config: `blobCache` in `types.ts`.
+- `AuthService` — token management, sign-in/sign-out events.
+- `CrdtManager`, `CrdtField`, `cursorColorFromName`, `CURSOR_COLORS` — Loro-CRDT integration.
+- Types: `Sp00kyConfig` (client-solid wraps it as `SyncedDbConfig`), `QueryTimeToLive`, `PersistenceClient`, `StoreType`, `UpdateOptions`, `RunOptions`, `PreloadOptions`.
+- Subpath: `@spooky-sync/core/otel` — `createOtelTransmit(endpoint)` for piping pino logs to OpenTelemetry.
+- `client.webPush` (`WebPushModule`, `db.webPush` in the Solid bindings) - Web Push on the page: `support()`, `info()`, `subscribe(opts)` (call from a click), `unsubscribe(opts)`, `sync()`, `isSubscribed()`, `update(opts)`, `devices()`, `notify(msg)`, `cancel(id)`, `test()`, `bridge()`, `onMessage(cb)`. Server calls are `fn::push::*`; typed `WebPushError`; config `webPush` in `types.ts`. Sign-out runs through `AuthService.onBeforeSignOut`.
+- Subpaths for service workers, workers and Node (no wasm, no pino, no DOM; `scripts/check-worker-entries.mjs` enforces it in `build`): `@spooky-sync/core/pure` (protocol builders), `@spooky-sync/core/live` (`createLiveFeed`, the live subscriber over the pure-JS SDK), `@spooky-sync/core/sw` (`installPushHandlers`, `getBridgedToken`). Built by the second tsdown config; chunks are prefixed `worker-`. See `docs/web-push-design.md` at the repo root.
+
+## Common gotchas
+
+- **Use UI hooks, not the client directly.** In a Solid app, call `useDb()` from `@spooky-sync/client-solid`. Touch `Sp00kyClient` only inside `provider.client` for advanced flows.
+- **Mutations are optimistic.** `db.create` / `db.update` / `db.delete` return immediately; the queue drains in the background. Inspect progress via `pendingMutationCount`.
+- **Record IDs are full strings.** `db.create('thread:abc123', {...})` — *not* `db.create('thread', { id: 'abc123' })`. The first arg is `'<table>:<id>'`. Generate IDs with `Uuid` from `surrealdb` (re-exported by `client-solid`).
+- **`db.update` takes `(table, id, payload, options?)`**. The `debounced` option (`{ debounced: true }` or `{ debounced: { delay, key, flushOnHide } }`) coalesces rapid updates — use it for CRDT text/title fields. With `flushOnHide` (default `true`) the merged patch is also kept in the local `_00_pending_writes` table and pushed as soon as the page is hidden, so a reload or closed tab inside the delay does not lose it (the next boot moves a leftover patch into the outbox). `flushOnHide: false` keeps the patch in memory only.
+- **`db.remoteQuery(sql, vars?)` is the one-shot remote read.** It runs one SurrealQL statement through the client's remote path (connect gate, per-statement timeout, concurrency limit) and returns the SurrealDB result array; results are not synced into the local cache. Use it for counts and other reads the query builder can't express.
+- **`db.useRemote(fn)` hands out the bare SDK client.** Same cache bypass, but it skips the connect gate (a call during a reconnect fails with "Specify a namespace to use"). Reserve it for SDK features `remoteQuery` cannot reach (transactions, `live`).
+- **Raw remote calls are gated by the query allowlist.** When the generated schema carries `policy.queryAllowlist: 'warn' | 'enforce'` (from `sync.queryAllowlist` in `sp00ky.yml`), `remoteQuery`, `useRemote` and `queryRaw` throw unless the client config sets `allowRawRemote: true`; builder queries are unaffected. A registration the SSP refuses as not allowlisted (403 `not_allowlisted`) is marked `remote: 'failed'` after one attempt with an error-level log carrying the surql and a `spky generate` hint; the view stays cold (`membershipKnown` false), it is never read as confirmed empty.
+- **`@parent` columns are auto-populated.** Defined in the `.surql` schema as `record<user>` with the `-- @parent` annotation; never write them yourself — they're set from the auth context server-side.
+- **`db.preload` is a registered query.** Resolved before on this device (a `_00_view` row exists) it returns at once; never resolved it waits until the server's membership and every body are local. It holds no subscriber, so it is evicted a ttl after registration unless a view mounts the same query. `refresh`/`staleTime` are ignored; pass `signal` to abort a cold wait.
+- **A never-resolved query paints only vouched rows.** Before the server answers (`cold`), the local predicate scan shows a cached body only if some `_00_view` row names it (as a member or subquery child) or a write for it is in flight. A body no view names is an orphan (deleted upstream while nothing watched it): it stays hidden, and the GC deletes it 30 s after boot and then hourly. A `_00_view` row no query has resolved for 14 days is retired by the same sweep, so that query paints cold again.
+- **A rejected write is not lost.** Application errors roll the local row back (create/update/delete) and move the mutation to `_00_failed_mutations`; the tray API above lists, retries (as a fresh write) or discards it. Network errors keep the row queued.
+
+## Module map (under `src/`)
+
+- `kernel/` — `Effect` union, saga driver + lanes, constants, interpreter.
+- `state/` — `ClientState`, lifecycle machine, reducers (they mark queries dirty), selectors (`overlay`, `needed`, `planFetch`, `settled`, `evictable`, `phaseTimings`).
+- `query/` — keys, membership rules, render set, SQL, and the sagas: `register`, `membership`, `fetch`, `materialize`, `lifecycle`.
+- `mutation/` — outbox rows (v2: `tableName`, `beforeRecord`, `createdAt`), `write`, `push` (drain + rollback), `tray`, jobs.
+- `sync/` — `poll`, `live`, `connection` (health, self-heal), `tabs` relay, pure policy.
+- `boot/` — `boot`, `auth-flip`, `bucket-switch`, `preload`.
+- `client/` — `runtime`, `router` (event -> saga + lane), `services` (adapters), `sp00ky-client` (facade), `bucket-handle`.
+- `testing/` — `runPure` (drive a saga with canned effect results), fake adapters/services, state builders.
+- `modules/` — auth, crdt, feature-flag, app-release, devtools, ref-tables (unchanged adapters).
+- `services/` — local store engines, remote socket + supervisor, SSP wasm wrapper, tabs broker, blobs, persistence, logger.
+
+Rules: nothing under `kernel/state/query/mutation/sync/boot` imports `services/` or touches `Date`, `setTimeout`, `crypto`; those directories are gated at 100% coverage. Legacy services are reached through the typed `service` effect (`kernel/effects.ts`, `ServiceCalls`).
+
+## Pointers
+
+- UI bindings: `node_modules/@spooky-sync/client-solid/AGENTS.md`
+- Query builder DSL: `node_modules/@spooky-sync/query-builder/AGENTS.md`
+- Schema codegen / migrations: `node_modules/@spooky-sync/cli/AGENTS.md`
+- Live introspection: `node_modules/@spooky-sync/devtools-mcp/AGENTS.md`

package/README.md

@@ -1,21 +1,61 @@
-## Register Query Flow
+# @spooky-sync/core
 
-1. Query registering (local) - Data Module
-2. Query get local data - Data Module
-3. (Query registing (remote) - Sync Module)
+The sync engine behind `Sp00kyClient`. Everything that decides something is a
+pure function or a generator saga that yields effect descriptions; the only
+effectful code is the adapters under `services/` and the interpreter in
+`kernel/interpreter.ts`. State is one immutable `ClientState` value.
 
-## Query Local Updates
+## Layout
 
-1. Local Mutation is executed - Data Module
-2. Mutation is enqueued for remote - Sync Moduke
-3. New Record is ingested to DBSP - Data Module
-4. Related Queries are updated - Data Module
-5. (Sync Module syncs the mutation to remote - Sync Module)
+| Directory | Holds |
+| --- | --- |
+| `kernel/` | `Effect` union, the saga driver, lanes, constants, the interpreter |
+| `state/` | `ClientState`, the query lifecycle machine, reducers, selectors |
+| `query/` | keys, membership rules, render set, SQL builders, the query sagas (register, membership, fetch, materialize, lifecycle) |
+| `mutation/` | outbox rows, write / push / rollback / tray sagas |
+| `sync/` | poll, LIVE, connection health, shared-tabs relay sagas |
+| `boot/` | boot, auth flip, bucket switch, preload sagas |
+| `client/` | runtime (lanes, timers, subscriptions), router, adapters, the public facade |
+| `services/` | adapters: local store engines, remote socket, SSP wasm, tabs broker, blobs |
 
-## Query Live Updates
+## Query lifecycle
 
-1. Live Update is received - Sync Module
-2. Version is compared with local version - Sync Module
-3. If remote version is higher, the record is updated - Sync Module
-4. Record is ingested to DBSP Module - Data Module
-5. Related Queries are updated - Data Module
+A query is `cold` (never resolved on this device: rows come from a local
+predicate scan, bindings show a loader), `cached` (a durable `_00_view` row
+seeded its membership: rows paint from the store at once), `live` (a server
+membership set was accepted this session) or `view-lost` (the server dropped
+the view; rows are kept while it re-registers).
+
+1. `registerLocal` computes the keys, reads `_00_view`, builds the local SSP
+   view and publishes the entry. The runtime materializes it on the next tick.
+2. `registerRemote` registers on the server and reads edges + meta in one
+   request; `applyMembership` decides (non-empty always; empty only when the
+   server vouches for it) and writes `_00_view`.
+3. `fetchRows` pulls every body any query is missing, once, in parallel
+   chunks, and records versions. `settled` is a selector over that state.
+4. Later changes arrive as LIVE edge events (which only mark membership dirty)
+   or the poll; both go through the same batched re-read.
+
+Rendered rows are always `(membership ∪ (writes ∩ localView)) − deletes`
+where `writes`/`deletes` come from the outbox mirror in state, so unsynced
+local mutations show and a rejected one disappears the moment it is rolled
+back.
+
+## Mutations
+
+`write` runs one local transaction (row + `_00_pending_mutations` entry with
+`beforeRecord`), feeds the circuit and starts `drain`. The drain pushes up to
+50 statements per request and judges each on its own: accepted rows are
+acked (kept in the overlay until membership names them), rejected rows are
+rolled back and moved to `_00_failed_mutations` in the same transaction that
+deletes the pending row, a transport cut leaves the tail queued behind a
+backoff. The tray is exposed as `failedMutationCount`,
+`subscribeToFailedMutations`, `listFailedMutations`, `retryFailedMutation`
+and `discardFailedMutation`.
+
+## Testing
+
+Sagas run against canned effect results with `src/testing/run-pure.ts`; the
+interpreter and facade run against `src/testing/adapters.ts`. `pnpm test:run`
+runs the suite, `pnpm test:coverage` enforces the thresholds in
+`vitest.config.ts` (the saga core is held at 100%).