@spooky-sync/core 0.0.1-canary.33 → 0.0.1-canary.330

package/AGENTS.md

@@ -0,0 +1,84 @@
+# `@spooky-sync/core` — agent guide
+
+## What this package is
+
+The sync engine. A `Sp00kyClient<S>` owns a local database (memory or IndexedDB), a remote SurrealDB connection, a mutation queue, and a CRDT manager. It's framework-agnostic — UI bindings live in `@spooky-sync/client-solid` (or future `client-react`, etc.). Most app code touches this package indirectly through a UI binding's `useDb()` hook.
+
+## Mental model
+
+```
+.surql schema
+   │  spky generate
+   ▼
+schema.gen.ts (typed schema + SURQL_SCHEMA constant)
+   │  passed to Sp00kyConfig
+   ▼
+Sp00kyClient<S>  ── facade: every method = one saga run, selector or subscription
+       │
+       ├── Runtime (client/runtime.ts): one immutable ClientState, lanes, timers,
+       │     subscriptions, dirty -> materialize scheduling. `query:status`,
+       │     `activity:changed` and `health:changed` are all notified from here,
+       │     off state transitions - never emitted by a saga, so a change made
+       │     outside a sync round (a socket drop, say) still reaches subscribers
+       ├── Sagas (query/ mutation/ sync/ boot/): pure generators yielding effects
+       └── Interpreter (kernel/interpreter.ts) -> adapters (services/):
+             local store · remote socket · SSP wasm circuit · tabs broker · blobs
+```
+
+Local writes are applied optimistically (row + outbox entry in one local
+transaction), rendered through the outbox overlay, and drained to the remote in
+batches. Server membership arrives through one batched edge read (registration,
+poll, LIVE dirt) and bodies are fetched once across all queries. Rows are always
+re-materialized from state, never patched in place.
+
+## Key exports (`src/index.ts`)
+
+- `Sp00kyClient<S>` — main class. Methods: `init()`, `create(id, payload)`, `update(table, id, payload, options?)`, `delete(table, id)`, `query(table, opts?)`, `preload(query, { signal? })`, `run(backend, route, payload)`, `bucket(name)`, `remoteQuery(sql, vars?)`, `useRemote(fn)`, `authenticate(token)`, `auth.signOut()`. Admin impersonation (`impersonation.enabled` in sp00ky.yml, admins only, server-enforced): `auth.impersonate(targetId, reason)`, `auth.stopImpersonating()`, `auth.impersonation`, `auth.subscribeImpersonation(cb)`, `auth.searchImpersonationTargets(search)`; while active the client injects a warning bar with a Stop button and shifts the page down onto a hazard-striped layer. `impersonationBanner` in the client config restyles it (`theme`), replaces it (`mode: 'custom'` + `client.acknowledgeImpersonationBanner()`, else it falls back after ~2.5s) or turns it off (`mode: 'none'`). See `/docs/client/impersonation`. Traffic indicators: `pendingMutationCount` / `subscribeToPendingMutations(cb)` (outbox depth), `fetchingQueryCount` / `subscribeToFetchActivity(cb)` (queries a user waits on; a query that already shows rows and is only re-registering, as after a reload, is not counted). Failed-writes tray: `failedMutationCount`, `subscribeToFailedMutations(cb)`, `listFailedMutations()`, `retryFailedMutation(id)`, `discardFailedMutation(id)`.
+- `FailedMutation` — a row of the `_00_failed_mutations` tray (`mutationType`, `recordId`, `data`, `beforeRecord`, `error`, `revert`).
+- `BucketHandle` — file storage handle (`put`, `get`, `delete`, `exists`), plus the cache-aware `read`, `url`, `pin`/`unpin`, `evict`, `prefetch`. `get` is always remote; `read`/`url` go through the blob cache.
+- `services/blobs/` — durable cache for bucket bytes: OPFS holds the files, `_00_blob` holds a manifest that reconcile rebuilds from disk (so a wiped local store costs metadata, not the offline cache). Nothing expires on a timer; eviction is LRU under a byte budget only, skipping pinned and on-screen entries. Config: `blobCache` in `types.ts`.
+- `AuthService` — token management, sign-in/sign-out events. `subscribe(cb, { immediate? })` calls `cb` at once with the current user unless `immediate: false`, then on every change.
+- `CrdtManager`, `CrdtField`, `cursorColorFromName`, `CURSOR_COLORS`: collaborative `@crdt` fields on Loro. Client methods: `openCrdtField(table, recordId, field, draft?)` -> `Promise<CrdtField>` (every open of one field in a tab shares one `LoroDoc`; pair each with a close), `closeCrdtField(table, recordId, field)` -> `Promise<void>` (resolves once unsent edits are in the outbox; offline, once they are saved on the device, never waiting on the network), `createCrdtDraft(table, field)` -> a detached `CrdtField` for a record not created yet (`db.create` first, then pass it to `openCrdtField` as `draft`). `CrdtField`: `getDoc()`, `hasContent()`, `pushCursorState(bytes)`, `onCursorUpdate = cb` (a setter). Config: `crdt: true` preloads Loro, `crdtDebounceMs` (default 300, max wait 2 s). Design: `docs/crdt-v2.md` at the repo root.
+- Types: `Sp00kyConfig` (client-solid wraps it as `SyncedDbConfig`), `QueryTimeToLive`, `PersistenceClient`, `StoreType`, `UpdateOptions`, `RunOptions`, `PreloadOptions`.
+- Subpath: `@spooky-sync/core/otel` — `createOtelTransmit(endpoint)` for piping pino logs to OpenTelemetry.
+- `client.webPush` (`WebPushModule`, `db.webPush` in the Solid bindings) - Web Push on the page: `support()`, `info()`, `subscribe(opts)` (call from a click), `unsubscribe(opts)`, `sync()`, `isSubscribed()`, `update(opts)`, `devices()`, `notify(msg)`, `cancel(id)`, `test()`, `bridge()`, `onMessage(cb)`. Server calls are `fn::push::*`; typed `WebPushError`; config `webPush` in `types.ts`. Sign-out runs through `AuthService.onBeforeSignOut`.
+- Subpaths for service workers, workers and Node (no wasm, no pino, no DOM; `scripts/check-worker-entries.mjs` enforces it in `build`): `@spooky-sync/core/pure` (protocol builders), `@spooky-sync/core/live` (`createLiveFeed`, the live subscriber over the pure-JS SDK; a feed that shares views with another through the same `sessionId` must also pass the same `ownerSecret`, or a signed-out feed's second registration is refused 409 `owner_mismatch`), `@spooky-sync/core/sw` (`installPushHandlers`, `getBridgedToken`). Built by the second tsdown config; chunks are prefixed `worker-`. See `docs/web-push-design.md` at the repo root.
+
+## Common gotchas
+
+- **Use UI hooks, not the client directly.** In a Solid app, call `useDb()` from `@spooky-sync/client-solid`. Touch `Sp00kyClient` only inside `provider.client` for advanced flows.
+- **Mutations are optimistic.** `db.create` / `db.update` / `db.delete` return immediately; the queue drains in the background. Inspect progress via `pendingMutationCount`.
+- **Record IDs are full strings.** `db.create('thread:abc123', {...})` — *not* `db.create('thread', { id: 'abc123' })`. The first arg is `'<table>:<id>'`. Generate IDs with `Uuid` from `surrealdb` (re-exported by `client-solid`).
+- **`db.update` takes `(table, id, payload, options?)`**. The `debounced` option (`{ debounced: true }` or `{ debounced: { delay, key, flushOnHide } }`) coalesces rapid updates, e.g. a plain text input saved as you type (never for `@crdt` fields, which are not written with `db.update`). With `flushOnHide` (default `true`) the merged patch is also kept in the local `_00_pending_writes` table and pushed as soon as the page is hidden, so a reload or closed tab inside the delay does not lose it (the next boot moves a leftover patch into the outbox). `flushOnHide: false` keeps the patch in memory only.
+- **`db.remoteQuery(sql, vars?)` is the one-shot remote read.** It runs one SurrealQL statement through the client's remote path (connect gate, per-statement timeout, concurrency limit) and returns the SurrealDB result array; results are not synced into the local cache. Use it for counts and other reads the query builder can't express.
+- **`db.useRemote(fn)` hands out the bare SDK client.** Same cache bypass, but it skips the connect gate (a call during a reconnect fails with "Specify a namespace to use"). Reserve it for SDK features `remoteQuery` cannot reach (transactions, `live`).
+- **Raw remote calls are gated by the query allowlist.** When the generated schema carries `policy.queryAllowlist: 'warn' | 'enforce'` (from `sync.queryAllowlist` in `sp00ky.yml`), `remoteQuery`, `useRemote` and `queryRaw` throw unless the client config sets `allowRawRemote: true`; builder queries are unaffected. A registration the SSP refuses as not allowlisted (403 `not_allowlisted`) is marked `remote: 'failed'` after one attempt with an error-level log carrying the surql and a `spky generate` hint; the view stays cold (`membershipKnown` false), it is never read as confirmed empty.
+- **A `@crdt` field is server-written plain text.** Edit it only through `openCrdtField` / `useCrdtField`; set it only in the `db.create` that makes the record (its initial text; a later client write keeps the old value). Reading it (lists, search, `where`) is fine, but it trails live edits until the SSP folds the edit log (a few seconds after the last edit by default). Edits travel as `<table>__crdt_log` rows through the ordinary outbox; each tab claims a random Loro peer in `_00_crdt_peer`, and log rows whose blob is not an update made by that peer are refused everywhere (`modules/crdt/acceptance.ts`, mirrored in Rust). Unsent edits survive reloads in `_00_crdt_local`. Dart (`spooky_core`) does not implement CRDT fields yet.
+- **`@parent` columns are auto-populated.** Defined in the `.surql` schema as `record<user>` with the `-- @parent` annotation; never write them yourself — they're set from the auth context server-side.
+- **`db.preload` is a registered query.** Resolved before on this device (a `_00_view` row exists) it returns at once; never resolved it waits until the server's membership and every body are local. A signed-out session without `enableAnonymousLiveQueries` is never sent membership, so its preload returns at once instead of blocking. It holds no subscriber, so it is evicted a ttl after registration unless a view mounts the same query. `refresh`/`staleTime` are ignored; pass `signal` to abort a cold wait.
+- **A never-resolved query paints only vouched rows.** Before the server answers (`cold`), the local predicate scan shows a cached body only if some `_00_view` row names it (as a member or subquery child) or a write for it is in flight. A body no view names is an orphan (deleted upstream while nothing watched it): it stays hidden, and the GC deletes it 30 s after boot and then hourly. A `_00_view` row no query has resolved for 14 days is retired by the same sweep, so that query paints cold again.
+- **`database.store` defaults to `'memory'`.** `'indexeddb'` keeps one IndexedDB database per user bucket (`sp00ky-<bucket>`) and needs `@surrealdb/wasm` >= 3.1.0: the 3.0.x engine opened the store but failed every write (`An IndexedDB error occured: idb error`), so each session fell back to memory. 3.1.0 ships the engine as `@surrealdb/wasm-native`; a Vite app lists it in `optimizeDeps.exclude` next to `@surrealdb/wasm`. A store that still cannot open after a drop runs the session in memory (error log `falling back to in-memory`). Under `localEngine: 'sqlite'`, `'indexeddb'` means OPFS.
+- **Boot owns the first identity.** The facade subscribes to auth with `immediate: false`: boot opens the bucket the hint names and `auth.restoreSession` announces the user. `LocalDatabaseService.switchStore` also waits for an in-flight `connect`, and lets statements already past the gate finish before it closes a client: the 3.1 engine streams results from its worker, and a statement whose client closed under it stayed pending until its 30 s deadline with every later local statement queued behind it.
+- **A rejected write is not lost.** Application errors roll the local row back (create/update/delete) and move the mutation to `_00_failed_mutations`; the tray API above lists, retries (as a fresh write) or discards it. Network errors keep the row queued.
+
+## Module map (under `src/`)
+
+- `kernel/` — `Effect` union, saga driver + lanes, constants, interpreter.
+- `state/` — `ClientState`, lifecycle machine, reducers (they mark queries dirty), selectors (`overlay`, `needed`, `planFetch`, `settled`, `evictable`, `phaseTimings`).
+- `query/` — keys, membership rules, render set, SQL, and the sagas: `register`, `membership`, `fetch`, `materialize`, `lifecycle`.
+- `mutation/` — outbox rows (v2: `tableName`, `beforeRecord`, `createdAt`), `write`, `push` (drain + rollback), `tray`, jobs.
+- `sync/` — `poll`, `live`, `connection` (health, self-heal), `tabs` relay, pure policy.
+- `boot/` — `boot`, `auth-flip`, `bucket-switch`, `preload`.
+- `client/` — `runtime`, `router` (event -> saga + lane), `services` (adapters), `sp00ky-client` (facade), `bucket-handle`.
+- `testing/` — `runPure` (drive a saga with canned effect results), fake adapters/services, state builders.
+- `modules/` — auth, crdt, feature-flag, app-release, devtools, ref-tables (unchanged adapters).
+- `services/` — local store engines, remote socket + supervisor, SSP wasm wrapper, tabs broker, blobs, persistence, logger.
+
+Rules: nothing under `kernel/state/query/mutation/sync/boot` imports `services/` or touches `Date`, `setTimeout`, `crypto`; those directories are gated at 100% coverage. Legacy services are reached through the typed `service` effect (`kernel/effects.ts`, `ServiceCalls`).
+
+## Pointers
+
+- UI bindings: `node_modules/@spooky-sync/client-solid/AGENTS.md`
+- Query builder DSL: `node_modules/@spooky-sync/query-builder/AGENTS.md`
+- Schema codegen / migrations: `node_modules/@spooky-sync/cli/AGENTS.md`
+- Live introspection: `node_modules/@spooky-sync/devtools-mcp/AGENTS.md`

package/README.md

@@ -1,21 +1,61 @@
-## Register Query Flow
+# @spooky-sync/core
 
-1. Query registering (local) - Data Module
-2. Query get local data - Data Module
-3. (Query registing (remote) - Sync Module)
+The sync engine behind `Sp00kyClient`. Everything that decides something is a
+pure function or a generator saga that yields effect descriptions; the only
+effectful code is the adapters under `services/` and the interpreter in
+`kernel/interpreter.ts`. State is one immutable `ClientState` value.
 
-## Query Local Updates
+## Layout
 
-1. Local Mutation is executed - Data Module
-2. Mutation is enqueued for remote - Sync Moduke
-3. New Record is ingested to DBSP - Data Module
-4. Related Queries are updated - Data Module
-5. (Sync Module syncs the mutation to remote - Sync Module)
+| Directory | Holds |
+| --- | --- |
+| `kernel/` | `Effect` union, the saga driver, lanes, constants, the interpreter |
+| `state/` | `ClientState`, the query lifecycle machine, reducers, selectors |
+| `query/` | keys, membership rules, render set, SQL builders, the query sagas (register, membership, fetch, materialize, lifecycle) |
+| `mutation/` | outbox rows, write / push / rollback / tray sagas |
+| `sync/` | poll, LIVE, connection health, shared-tabs relay sagas |
+| `boot/` | boot, auth flip, bucket switch, preload sagas |
+| `client/` | runtime (lanes, timers, subscriptions), router, adapters, the public facade |
+| `services/` | adapters: local store engines, remote socket, SSP wasm, tabs broker, blobs |
 
-## Query Live Updates
+## Query lifecycle
 
-1. Live Update is received - Sync Module
-2. Version is compared with local version - Sync Module
-3. If remote version is higher, the record is updated - Sync Module
-4. Record is ingested to DBSP Module - Data Module
-5. Related Queries are updated - Data Module
+A query is `cold` (never resolved on this device: rows come from a local
+predicate scan, bindings show a loader), `cached` (a durable `_00_view` row
+seeded its membership: rows paint from the store at once), `live` (a server
+membership set was accepted this session) or `view-lost` (the server dropped
+the view; rows are kept while it re-registers).
+
+1. `registerLocal` computes the keys, reads `_00_view`, builds the local SSP
+   view and publishes the entry. The runtime materializes it on the next tick.
+2. `registerRemote` registers on the server and reads edges + meta in one
+   request; `applyMembership` decides (non-empty always; empty only when the
+   server vouches for it) and writes `_00_view`.
+3. `fetchRows` pulls every body any query is missing, once, in parallel
+   chunks, and records versions. `settled` is a selector over that state.
+4. Later changes arrive as LIVE edge events (which only mark membership dirty)
+   or the poll; both go through the same batched re-read.
+
+Rendered rows are always `(membership ∪ (writes ∩ localView)) − deletes`
+where `writes`/`deletes` come from the outbox mirror in state, so unsynced
+local mutations show and a rejected one disappears the moment it is rolled
+back.
+
+## Mutations
+
+`write` runs one local transaction (row + `_00_pending_mutations` entry with
+`beforeRecord`), feeds the circuit and starts `drain`. The drain pushes up to
+50 statements per request and judges each on its own: accepted rows are
+acked (kept in the overlay until membership names them), rejected rows are
+rolled back and moved to `_00_failed_mutations` in the same transaction that
+deletes the pending row, a transport cut leaves the tail queued behind a
+backoff. The tray is exposed as `failedMutationCount`,
+`subscribeToFailedMutations`, `listFailedMutations`, `retryFailedMutation`
+and `discardFailedMutation`.
+
+## Testing
+
+Sagas run against canned effect results with `src/testing/run-pure.ts`; the
+interpreter and facade run against `src/testing/adapters.ts`. `pnpm test:run`
+runs the suite, `pnpm test:coverage` enforces the thresholds in
+`vitest.config.ts` (the saga core is held at 100%).