@syncular/server 0.31.0 → 0.32.1

package/README.md

@@ -128,7 +128,7 @@ After restoring an authoritative database, keep traffic stopped and call
 `rotatePartitionLogEpoch({ storage, partition })` for every restored
 partition. The rotation clears stale client cursors and requires version 2
 clients to reset their server-derived rows while preserving the outbox. Follow
-the complete [backup and restore runbook](https://syncular.dev/server-backup-restore/).
+the complete [backup and restore runbook](https://syncular.dev/server-operations/#backup-and-restore).
 
 ## Adapter error hooks
 
@@ -142,7 +142,7 @@ adapter answers instead. Return `undefined` to keep `sync.internal_error`. A
 throw, a non-`SyncError` return, a code that is not an own catalog entry, or
 `details` that is not a JSON document is contained as `sync.internal_error`.
 The hook covers the sync, operations, segments, blobs, admin, and realtime
-surfaces; `createSyncularAdminRoutes` takes its own `mapError`. It does not
+surfaces. It does not
 change a remote operation's `operation.*` envelope, and a mapped `retryAfterMs`
 is delivery metadata. Client retry scheduling remains unchanged.
 
@@ -533,10 +533,10 @@ context). The demo server wires it behind `SYNCULAR_DEMO_EVENTS=1`.
 
 | Event | When | Key fields |
 | --- | --- | --- |
-| `request.handled` | Once per `POST /sync`, after the response bytes are fully produced (or the request was rejected up front) | `kind` (`sync`), `partition`, `actorId`, `durationMs`, `bytesIn`, `bytesOut`, `outcome` (`ok` \| `schema_floor` \| `rejected` \| `error`), `errorCode?`, `pushCommits`, `pulled`, `subscriptions` |
+| `request.handled` | Once per `POST /sync`, after the response bytes are fully produced (or the request was rejected up front) | `kind` (`sync`), `partition`, `actorId`, `durationMs`, `bytesIn`, `bytesOut`, `outcome` (`ok` \| `schema_floor` \| `epoch_acquired` \| `reset` \| `rejected` \| `error`; `epoch_acquired` answers a first round without `logEpoch`, `reset` a `logEpoch` the log no longer matches), `errorCode?`, `clientId?` (absent when the request did not decode), `pushCommits`, `pulled`, `subscriptions` |
 | `push.applied` | A `PUSH_COMMIT` applied, or replayed from the idempotency cache (§2.3) | `clientId`, `clientCommitId`, `operations`, `commitSeq?`, `replay` |
-| `push.rejected` | A commit rejected (§6.3) | `clientId`, `clientCommitId`, `operations`, `code` (§10.2), `opIndex` |
-| `push.conflicted` | A commit terminated by a version conflict (§6.2) | `clientId`, `clientCommitId`, `operations`, `opIndex` |
+| `push.rejected` | A commit rejected (§6.3) | `clientId`, `clientCommitId`, `operations`, `code` (§10.2), `opIndex`, `table?` and `rowId?` of the terminating operation |
+| `push.conflicted` | A commit terminated by a version conflict (§6.2) | `clientId`, `clientCommitId`, `operations`, `opIndex`, `table?` and `rowId?` of the conflicting operation |
 | `reaction.queued` | A planned reaction committed with its source push | `clientId`, `clientCommitId`, `commitSeq`, `idempotencyKey`, `reactionType`, `version` |
 | `reaction.started` | A worker claimed and began one attempt | `workerId`, `idempotencyKey`, `reactionType`, `version`, `attempt` |
 | `reaction.retried` | A retryable attempt failed and was rescheduled | started fields plus `nextAttemptAtMs`, `errorCode` |
@@ -554,26 +554,31 @@ context). The demo server wires it behind `SYNCULAR_DEMO_EVENTS=1`.
 | `scopes.resolve_failed` | The host `resolveScopes` callback threw — the §3.2/§3.4 fail-loud path | `phase` (`request` \| `realtime` \| `segment-download`), `message` |
 
 All events also carry `type`, `atMs`, and (where a request identity
-exists) `partition` / `actorId`.
+exists) `partition` / `actorId`. The events emitted while serving one sync
+request (`request.handled`, `push.*`, `pull.served`, `lease.issued`,
+`reaction.queued`, `scopes.resolve_failed`) share one `requestId`, so a sink
+can group a request's events.
 
 ## Admin / console surface (`SyncularAdmin`)
 
-The operator-facing read surface over the server core lives in this package
-and adds **zero** wire protocol. Authorization for these reads is entirely the
-host's. It delivers the 80% operator value (who's
-connected, what's flowing,
-horizon health, the event tail) as a handful of read-only, partition-scoped,
-JSON-able queries.
+The operator-facing surface over the server core lives in this package and
+adds no wire protocol. Authorization is entirely the host's. It answers
+who is connected, what is flowing and failing, why a client does or does not
+see a row, and how far the horizon trails, through partition-scoped JSON-able
+queries plus three operator actions.
 
 ### The event ring (the "event stream")
 
 `RingBufferEvents` is a `SyncularServerEvents` sink that retains the last N
 events in memory (bounded — oldest dropped when full) with a
-`query({type?, sinceMs?, clientId?, actorId?, limit})` and a
+`query({type?, partition?, sinceMs?, clientId?, actorId?, limit})` and a
 `subscribe(listener)` hook for live tails (the SSE route below). It is the
-event stream without any infrastructure dependency. Compose it with any
-other sink so the console tail and your logs/metrics see the same
-emissions:
+event stream without any infrastructure dependency. `evictedThroughMs` is
+the newest `atMs` the ring dropped; the admin reports a window that starts
+at or before it as incomplete. The `admin` option of `createSyncularHono`
+and the Workers Durable Object create and compose the ring themselves. A
+script that queries `SyncularAdmin` directly composes it with any other sink
+so the ring and your logs/metrics see the same emissions:
 
 ```ts
 import {
@@ -590,28 +595,51 @@ const admin = SyncularAdmin.fromConfig(config, { ring });
 
 ### Query surface
 
-Every method is read-only. All are partition-scoped except the two fleet
-reads (`listPartitions` / `partitionsOverview`), which enumerate partitions
-by design:
+All methods are partition-scoped except the two fleet reads
+(`listPartitions` / `partitionsOverview`), which enumerate partitions by
+design. Row reads need the server `schema` (`fromConfig` passes it).
 
 | Method | Returns |
 | --- | --- |
-| `listClients(partition)` | Known clients: `clientId`, `actorId`, `cursor`, `lag` (commits not yet pulled: `maxCommitSeq − max(cursor, 0)`), `updatedAtMs`, `subscriptions[]`, and an `active` flag (cursor touched within the §4.6 active window). |
-| `clientDetail(partition, clientId, {eventLimit?})` | One client's drill-down: `{exists, client?, lease?, events}` — the record (with lag), its §7.3 lease when a lease store is wired, and its slice of the event tail. Answers "why is this client stale" in one read. |
-| `listCommits(partition, {afterSeq?, limit?, table?})` | Commit-log **metadata** (never payloads), newest first: `commitSeq`, `clientId`, `clientCommitId`, `actorId`, `createdAtMs`, `changeCount`, `tables[]`. |
+| `tables()` / `schemaVersion` | The schema tables with `columns[]` (`name`, `type`, `nullable`, `primaryKey`), and the server schema version. |
+| `search(partition, query)` | Resolves one pasted id: clients whose id contains it or whose actor equals it, actors containing it, rows with that rowId in any table, the commit with that seq, and write outcomes with that clientCommitId. |
+| `actors(partition)` | Each actor with its devices (clients), open sockets, worst lag, and failures in the last 5 minutes (the `errors().byClient` counts summed per actor). |
+| `tableCounts(partition)` | Row count per schema table. |
+| `listRows(partition, table, {afterRowId?, limit?, scope?})` | One page of decoded rows ordered by rowId, optionally only rows whose stored `scope` (`{variable, value}`) matches, read through the scope index; `nextAfterRowId` continues. |
+| `rowDelivery(partition, table, rowId)` | The trace verdict for every known client on one row, sorted by verdict then last seen: who has the row and who does not. |
+| `findWrite(partition, clientCommitId)` | Every stored idempotency outcome for that clientCommitId (applied with `commitSeq`, or rejected with per-operation results; conflict bytes as base64) and the ring events that carry it. |
+| `listClients(partition, {limit?, sort?})` | `{clients, total}`. Each client: `clientId`, `actorId`, `wireVersion`, `schemaVersion?` (highest among its subscriptions), `cursor`, `lag` (`maxCommitSeq − max(cursor, 0)`), `updatedAtMs`, `subscriptions[]`, `active` (touched within the §4.6 active window), `connected?` and `connectedSinceMs?` (open realtime sockets on this process; present when a hub is wired, which `fromConfig` does when `config.realtime` is a `RealtimeHub`). `sort`: `seen` (default) or `lag`; `limit` default 200. |
+| `clientDetail(partition, clientId, {eventLimit?})` | `{exists, client?, lease?, events}`: the record, its §7.3 lease when a lease store is wired, and its slice of the event tail. |
+| `listCommits(partition, {afterSeq?, beforeSeq?, limit?, table?, changeSample?})` | Commit metadata, newest first: `commitSeq`, `clientId`, `clientCommitId`, `actorId`, `createdAtMs`, `changeCount`, `tables[]`, and with `changeSample` the first N changes as `sample[]` (`table`, `rowId`, `op`). Pass the oldest `commitSeq` of a page as `beforeSeq` for the next older page. |
+| `commitDetail(partition, commitSeq)` | The commit's metadata plus `changes[]` (`table`, `rowId`, `op`, `rowVersion?`, `scopes`, `values?` rendered per SPEC §11, `undecodable?` for a payload written under an older column set). Throws `sync.not_found` for a missing or pruned commit. |
 | `listReactions(partition, {statuses?, types?, limit?})` | Durable reaction lifecycle rows, newest first, including source commit, attempts, lease, completion, and bounded failure information. |
-| `inspectRow(partition, table, rowId)` | `{exists, serverVersion?, scopes?}` — current row version + stored scopes, payload **not** decoded. |
+| `inspectRow(partition, table, rowId, {historyLimit?})` | `{exists, serverVersion?, scopes?, values?, history}`: the row decoded per SPEC §11 and its retained changes, newest first (`commitSeq`, `op`, `rowVersion?`, `actorId`, `clientId`, `createdAtMs`). Throws `sync.unknown_table` for a table outside the schema. |
+| `traceRow(partition, {table, rowId, clientId})` | Whether the client receives the row: a `verdict` (`client-unknown`, `row-missing`, `no-subscription`, `row-deleted`, `scope-mismatch`, `scope-denied`, `pending-pull`, `delivered`, evaluated in that order) with the row, its newest change, the horizon, the client, the actor's `allowed` scopes from `resolveScopes` (`outage` for the §7.3.3 sentinel), each subscription on the table with `requestedMatch`, `mismatches[]` and `effectiveMatch` (requested ∩ allowed, §3.2), and the client's retained push rejections. |
 | `scopeActivity(partition, {variable, value}, {limit?})` | Recent commits touching one scope key, via the §3.1 change-scope index (never a log scan). |
-| `horizonStatus(partition)` | `{maxCommitSeq, horizonSeq, retainedCommits, activeCursorFloor, recommendedHorizonSeq, recommendation}` — the horizon a prune pass would reach now (§4.6) + a coarse `up-to-date` / `prune-recommended`. |
-| `segmentStats()` / `blobStats(partition)` / `stats(partition)` | Counts/bytes where the stores expose them (segments split rows/sqlite). `undefined` when a store omits `stats()`. |
-| `metrics(partition, {windowMs?, buckets?})` | Ring-derived request/push health over a trailing window (default 5 min): request count/rate, error share, p50/p95 duration, push applied/rejected/conflicted, and per-bucket counts for a sparkline. Zero new server state; all zeros when no ring is wired. |
-| `listPartitionRegistry()` / `partitionsOverview()` | Every authenticated partition; the registry carries log continuity and last-authenticated time, while the overview adds retained backlog, client counts, and the prune recommendation. |
-| `events({type?, sinceMs?, clientId?, actorId?, limit?})` | The ring tail, newest first. Empty when no ring is wired (`hasEventStream` reports which). |
+| `horizonStatus(partition)` | `{maxCommitSeq, horizonSeq, retainedCommits, activeCursorFloor, recommendedHorizonSeq, recommendation}`: the horizon a prune pass would reach now (§4.6) and `up-to-date` / `prune-recommended`. |
+| `stats(partition)` | `{segments?, blobs?}` counts/bytes where the stores expose them (segments split rows/sqlite). |
+| `metrics(partition, {windowMs?, buckets?})` | Ring-derived request/push health over a trailing window (default 5 min): request count/rate, error share, p50/p95 duration, push applied/rejected/conflicted, and per-bucket counts, errors, p95 and push outcomes. `coverage: {complete, fromMs}` is false when the ring dropped events inside the window. All zeros when no ring is wired. |
+| `errors(partition, {windowMs?})` | Ring failures in the window: `byCode[]` (`source` `request`/`push`/`reaction`/`scopes`, `code`, `count`, distinct `clients` and `actors`, `lastAtMs`, `sampleClientIds`, `sampleRows`) and every client with a request or push failure in `byClient[]`, most failures first, with `coverage`. |
+| `listPartitionRegistry()` / `partitionsOverview({limit?, offset?})` | Every authenticated partition; the overview pages them (default 100) as `{partitions, total}` with retained backlog, client counts, and the prune recommendation. |
+| `events({type?, partition?, requestId?, sinceMs?, clientId?, actorId?, limit?})` | The ring tail, newest first. Empty when no ring is wired (`hasEventStream` reports which). |
 | `subscribeEvents(listener)` | Live events as they land in the ring; returns the unsubscribe function, `undefined` when no ring is wired. |
 
+Operator actions:
+
+| Method | Effect |
+| --- | --- |
+| `retryReaction(partition, idempotencyKey)` | Requeues a dead-lettered reaction and returns it. Throws `sync.invalid_request` when the key names no dead-lettered reaction. |
+| `prune(partition)` | One §4.6 prune pass now; returns `{previousHorizonSeq, horizonSeq}` and emits `prune.completed`. |
+| `revokeLease(partition, clientId)` | Revokes the client's §7.3 lease (§7.3.4), emits `lease.revoked`, returns the revoked record. Throws `sync.invalid_request` when the client holds none. |
+
 The query surface leans on **additive, optional** storage/store methods
-(`ServerStorage.listClientRecords` / `listCommitMetadata` / `scopeActivity` /
-`getRowScopes` / `listPartitions`; `SegmentStore.stats`; `BlobStore.stats`)
+(`ServerStorage.listClientRecords` / `listCommitMetadata` / `getCommit` /
+`listRowChanges` / `listRows` / `countRows` / `listPushResults` /
+`scopeActivity` / `listPartitions`; `SegmentStore.stats`;
+`BlobStore.stats`). `listRowChanges` reads the partition's retained log
+and `listPushResults` reads the stored idempotency outcomes, both without a
+dedicated index, so the write path pays nothing for them and their cost is
+bounded by retention
 — the established
 optional-method pattern. `SqliteServerStorage`, `PostgresServerStorage`,
 `D1ServerStorage`, and the memory/sqlite stores implement them; the shared
@@ -625,72 +653,94 @@ stores (memory/sqlite) omit the marker.
 
 ### HTTP routes + the single console page
 
-`@syncular/server-hono` exports `createSyncularAdminRoutes(admin, opts)`,
-a mountable Hono sub-app. **The auth seam is required**: the factory throws
-if you omit the `authorize` guard — there is no default-open admin. Every
-endpoint (including the page) runs the guard first; a falsy result is a 401.
+`createSyncularHono` mounts the console with its `admin` option, and
+`createWorkersFetchHandler` takes the same guards (see the
+`@syncular/server-workers` README). The option throws without the
+`authorize` guard, so the admin never mounts open. Every endpoint, including
+the page, runs the guard first; a falsy result is a 401. The `POST` actions
+also run `authorizeActions`; without it, or when it returns falsy, they
+answer 403 `sync.forbidden`.
 
 ```ts
-import { createSyncularAdminRoutes } from '@syncular/server-hono';
-
-const routes = createSyncularAdminRoutes(admin, {
-  defaultPartition: 'main',
-  authorize: ({ request }) => isOperator(request), // YOUR check — mandatory
+import { createSyncularHono } from '@syncular/server-hono';
+
+const app = createSyncularHono({
+  config,
+  authenticate,
+  admin: {
+    authorize: ({ request }) => isOperator(request), // YOUR check, mandatory
+    authorizeActions: ({ request }) => isOnCall(request), // optional actions
+    // path: '/admin', ringCapacity: 1000 (the defaults)
+  },
 });
-app.route('/admin', routes);
 ```
 
+The option creates a `RingBufferEvents` of `ringCapacity` events and
+composes it into `config.events` and, when `config.realtime` is a
+`RealtimeHub`, into the hub's events (`RealtimeHub.addEventSink`), so the
+console sees HTTP and socket rounds. Admin errors go to `config.onError` and
+`config.mapError`.
+
 | Route | Mirrors |
 | --- | --- |
 | `GET /` | The console page (see below). |
-| `GET /clients` | `listClients` |
+| `GET /info` | `{partition, hasEventStream, actions, realtime, schemaVersion?, tables}` |
+| `GET /search?q` | `search` |
+| `GET /actors` | `actors` |
+| `GET /tables` | `tables()` with each table's `rowCount` |
+| `GET /rows/:table?afterRowId&limit&scope=variable:value` | `listRows` |
+| `GET /rows/:table/:rowId/delivery` | `rowDelivery` |
+| `GET /writes/:clientCommitId` | `findWrite` |
+| `GET /clients?limit&sort` | `listClients` |
 | `GET /clients/:clientId?eventLimit` | `clientDetail` |
-| `GET /commits?afterSeq&limit&table` | `listCommits` |
+| `GET /commits?afterSeq&beforeSeq&limit&table&sample` | `listCommits` (`sample` is `changeSample`) |
+| `GET /commits/:commitSeq` | `commitDetail` |
 | `GET /reactions?status&type&limit` | `listReactions` |
-| `GET /rows/:table/:rowId` | `inspectRow` |
+| `GET /rows/:table/:rowId?historyLimit` | `inspectRow` |
+| `GET /trace?table&rowId&clientId` | `traceRow` |
 | `GET /scope-activity?variable&value&limit` | `scopeActivity` |
 | `GET /horizon` | `horizonStatus` |
 | `GET /stats` | `stats` |
 | `GET /metrics?windowMs` | `metrics` |
-| `GET /partitions` | `partitionsOverview` (fleet view) |
-| `GET /events?type&sinceMs&clientId&actorId&limit` | `events` (ring tail) |
-| `GET /events/stream?type&clientId&actorId&limit` | live SSE tail (backlog replay, then `subscribeEvents`) |
-
-`?partition=` selects the partition (falls back to `defaultPartition`).
-`GET /partitions` is the one cross-partition endpoint: the guard still runs
-on it, with an empty `partition` in its context unless the query passes one
-— a host that authorizes per partition should treat it accordingly.
-
-`GET /` (or `/admin`) serves a **single static HTML page** — zero
-framework, no build step, no React. It fetches the sibling JSON endpoints
-(relative to its own mount path, so it works under any prefix and the same
-guard covers its XHRs) and renders: a metrics statusbar (push rate,
-conflicts, error share, p95, ASCII sparkline), the fleet view (which doubles
-as the partition picker), horizon, store stats, clients with cursor lag and
-a per-client drill-down (subscriptions with full scope sets, lease status,
-the client's event slice), recent commits, the row inspector, scope
-activity, and the event tail — with an auto-refresh toggle (2 s poll).
-This is a deliberate one-file console: 5% of the code a full console app
-would cost, the 80% operator value.
+| `GET /errors?windowMs` | `errors` |
+| `GET /partitions?limit&offset` | `partitionsOverview` (fleet view) |
+| `GET /events?type&requestId&sinceMs&clientId&actorId&limit` | `events` (ring tail) |
+| `GET /events/stream?type&requestId&clientId&actorId&limit` | live SSE tail (backlog replay, then `subscribeEvents`) |
+| `POST /reactions/:idempotencyKey/retry` | `retryReaction` |
+| `POST /prune` | `prune` |
+| `POST /clients/:clientId/revoke-lease` | `revokeLease` |
+
+`?partition=` selects the partition. Without it, a deployment with exactly
+one partition in its registry uses that one; any other count answers 400
+`sync.invalid_request` with `details.parameter: 'partition'`, after the
+guard ran with an empty partition. Every endpoint reads only the resolved
+partition, event tail included. A malformed integer query parameter answers
+400 `sync.invalid_request`. `GET /partitions` is the one cross-partition
+endpoint: its guard receives an empty `partition`, so a guard that grants
+single partitions does not grant the fleet view.
+
+`GET /` serves a **single static HTML page**: no framework, no build step,
+no external assets (about 110 KiB, 31 KiB gzipped). It fetches the sibling
+JSON endpoints relative to its own mount path, so it works under any prefix
+and the same guard covers its requests. Operator actions appear only when
+`/info` reports `actions: true`. The views and the debugging workflows are
+on the docs page [Admin console](https://syncular.dev/server-admin-console/).
 
 **Live tail over SSE.** `GET /events/stream` streams the ring as
 server-sent events: a recent backlog replays first, then events arrive as
 they land (a `subscribe` hook on the ring), with a comment ping every 5 s
 (under Bun.serve's 10 s default idle timeout, so a quiet stream survives).
 It is plain web-streams, so it serves identically on Bun, Node, and
-Workers. The page upgrades its event panel to the stream automatically and
-falls back to the 2-second poll when the stream is unavailable (no ring, or
-`EventSource` cannot pass the host's auth headers — cookie-authorized
-setups stream fine). The tail can carry sensitive identifiers (actorIds,
+Workers. The page opens the stream when `/info` reports `hasEventStream`;
+when the stream fails (for example, `EventSource` cannot send the host's
+auth headers, while cookie-authorized setups stream fine), events reload
+with every refresh. The tail can carry sensitive identifiers (actorIds,
 session ids), the same as `GET /events`; the admin guard gates both.
 
 The demo server (`apps/demo`) mounts the admin behind a dev guard:
 `SYNCULAR_DEMO_ADMIN=1` enables `/admin` (optionally token-gated with
-`SYNCULAR_DEMO_ADMIN_TOKEN`), so the console is inspectable live.
-
-> Docs-site coverage of the console is a follow-up: the docs app is owned
-> by a concurrent workstream this round (the schema-bump page), so this
-> README is the console's documentation home for now.
+`SYNCULAR_DEMO_ADMIN_TOKEN`; `SYNCULAR_DEMO_ADMIN_WRITES=1` adds the
+operator actions), so the console is inspectable live.
 
 ## Segment storage on S3 / R2 (`S3SegmentStore`)
 
@@ -795,7 +845,7 @@ honor these conditional headers.
 
 Even so, the counters are **approximate**, and `stats()` marks them
 `approximate: true` (an additive field the admin surface carries through, so
-`admin.segmentStats()` / `admin.stats()` expose it and the console labels the
+`admin.stats()` exposes it and the console labels the
 numbers honestly). They can drift: a crash between the segment PUT and the
 accumulator CAS, lifecycle GC deleting objects the accumulator still counts,
 or enough concurrent writers to exhaust the CAS retry budget. They are a
@@ -957,7 +1007,7 @@ GET/HEAD.
 object (`{keyPrefix}stats/blobs.json`) folded read-modify-write on `put` under
 the same ETag compare-and-swap as segments, and marks the result
 `approximate: true` (the marker already present on `BlobStoreStats`, carried
-through `admin.blobStats()`). A `HEAD` before the PUT detects an idempotent
+through `admin.stats()`). A `HEAD` before the PUT detects an idempotent
 re-upload so distinct blobs count once; the sweep decrements on delete. Same
 caveats as segment stats — a health gauge, not an invoice. The in-process
 stores (memory / sqlite) count on demand and omit the marker.