@syncular/server 0.15.47 → 0.16.1

package/README.md

@@ -15,6 +15,11 @@ registry, command mutations use the ordinary serialized push path, and
 specified in [`docs/REMOTE.md`](../../docs/REMOTE.md) and the practical setup is
 in the [remote operations guide](https://syncular.dev/guide-remote-operations/).
 
+Remote query registration requires generated `relationPlans` for each selected
+SQL statement. Run `syncular generate` before upgrading existing query modules.
+The server uses those boundaries to bind every physical table occurrence to
+the authenticated partition, including quoted self joins and CTE bodies.
+
 Application intent belongs in immutable domain event rows written in the same
 commit as the state change. `SyncularServerEvents` below remains operational
 telemetry. See the [domain event guide](https://syncular.dev/guide-domain-events/).
@@ -105,6 +110,12 @@ Log the cause for operators and stop startup. Do not catch readiness errors in
 authentication or convert them to a 401; request-time schema checks are only a
 defensive fallback.
 
+After restoring an authoritative database, keep traffic stopped and call
+`rotatePartitionLogEpoch({ storage, partition })` for every restored
+partition. The rotation clears stale client cursors and requires version 2
+clients to reset their server-derived rows while preserving the outbox. Follow
+the complete [backup and restore runbook](https://syncular.dev/server-backup-restore/).
+
 ## Write validators and recovery metadata
 
 `validators` is the server-authoritative seam for row business rules that
@@ -507,7 +518,7 @@ by design:
 | `horizonStatus(partition)` | `{maxCommitSeq, horizonSeq, retainedCommits, activeCursorFloor, recommendedHorizonSeq, recommendation}` — the horizon a prune pass would reach now (§4.6) + a coarse `up-to-date` / `prune-recommended`. |
 | `segmentStats()` / `blobStats(partition)` / `stats(partition)` | Counts/bytes where the stores expose them (segments split rows/sqlite). `undefined` when a store omits `stats()`. |
 | `metrics(partition, {windowMs?, buckets?})` | Ring-derived request/push health over a trailing window (default 5 min): request count/rate, error share, p50/p95 duration, push applied/rejected/conflicted, and per-bucket counts for a sparkline. Zero new server state; all zeros when no ring is wired. |
-| `listPartitions()` / `partitionsOverview()` | Every partition the storage holds state for; the overview adds retained backlog, client counts (known/active), and the prune recommendation per partition — the fleet view. |
+| `listPartitionRegistry()` / `partitionsOverview()` | Every authenticated partition; the registry carries log continuity and last-authenticated time, while the overview adds retained backlog, client counts, and the prune recommendation. |
 | `events({type?, sinceMs?, clientId?, actorId?, limit?})` | The ring tail, newest first. Empty when no ring is wired (`hasEventStream` reports which). |
 | `subscribeEvents(listener)` | Live events as they land in the ring; returns the unsubscribe function, `undefined` when no ring is wired. |
 
@@ -921,10 +932,22 @@ at or below it. Nothing prunes automatically — the host schedules it.
 
 **When to run.** A periodic job per partition — hourly to daily is the
 sensible range; there is no benefit below the granularity of your
-`activeWindowMs`. Prune is cheap when there is nothing to do (one cursor
-scan + two point reads), so err on the side of running it often rather
-than letting a backlog build. Pass `events` to get `prune.completed`
-per pass.
+`activeWindowMs`. Pass `events` to get `prune.completed` per pass.
+
+Pruning verifies the captured log epoch and updates the horizon together with
+commit/change/scope deletion in one transaction. Concurrent passes cannot
+lower the horizon. A retry cleans up eligible records even when the horizon
+already covers them. A restore invalidates a pending pass with
+`sync.storage.prune_epoch_mismatch`; recompute its retention inputs.
+Unregistered partitions cannot be pruned. D1 maintenance enters the owning
+Durable Object's existing write queue through
+`SyncularRealtimeHost.pruneCommitLog`.
+
+Custom storage adapters must add `getPartitionLogEpoch(partition)` and replace
+`pruneCommitsThrough(partition, seq)` with
+`pruneCommitsThrough(partition, { logEpoch, throughSeq })`, returning
+`{ previousHorizonSeq, horizonSeq, removedCommits }` from the transaction.
+`setHorizonSeq` remains monotonic and is no longer used by the pruning helper.
 
 **The retention floors (§4.6, encoded in `RetentionPolicy`).** The
 horizon never advances past `min(cursor)` of *active* clients — clients
@@ -1143,3 +1166,23 @@ deterministic in-process sqlite loopback):
 ```sh
 SYNCULAR_PG_URL=postgres://user:pass@localhost:5432/db bun run bench
 ```
+
+Custom storage adapters must implement
+`getActiveClientCursorFloor(partition, cutoffMs)`. Return the minimum cursor
+whose `updatedAtMs >= cutoffMs`, or `null` when no client qualifies. Preserve
+negative bootstrap cursors. Pruning and admin horizon status use this scalar
+aggregate; `listClientCursors` remains the explicit listing interface.
+
+SQLite image builders now return `Promise<Uint8Array>` and receive
+`rowBatches`, an iterable or async iterable of row arrays. Replace custom
+builders' `input.rows` loop with `for await (const rows of input.rowBatches)`,
+insert each batch into the dedicated image database, and count rows during
+consumption. Write the final row count into `_syncular_segment` before
+serialization. Await `buildSqliteImage(input)` when calling the built-in
+Bun or Node builder directly.
+
+The server shares in-flight builds for the same storage pair and artifact
+identity after authorization. Sharing is local to one process. Signed URL
+grants remain per request. The first eligibility probe has at most
+`limitSnapshotRows + 1` rows; subsequent builder batches have at most 5,000
+rows. The image database and serialized output still consume memory.

package/dist/admin.d.ts

@@ -223,11 +223,7 @@ export declare class SyncularAdmin {
         readonly windowMs?: number;
         readonly buckets?: number;
     }): AdminMetrics;
-    /**
-     * Every partition the storage knows (commit log + client records) — the
-     * fleet-view backing and the console's partition picker. Fails loud when
-     * the backend omits the optional `listPartitions`.
-     */
+    /** Every authenticated partition in the storage-backed registry. */
     listPartitions(): Promise<string[]>;
     /**
      * The fleet view: one row per known partition — retained backlog, client

package/dist/admin.js

@@ -173,11 +173,7 @@ export class SyncularAdmin {
         const nowMs = this.#clock();
         const maxCommitSeq = await this.#storage.getMaxCommitSeq(partition);
         const horizonSeq = await this.#storage.getHorizonSeq(partition);
-        const cursors = await this.#storage.listClientCursors(partition);
-        const activeCursors = cursors
-            .filter((c) => c.updatedAtMs >= nowMs - this.#retention.activeWindowMs)
-            .map((c) => c.cursor);
-        const activeCursorFloor = activeCursors.length > 0 ? Math.min(...activeCursors) : null;
+        const activeCursorFloor = await this.#storage.getActiveClientCursorFloor(partition, nowMs - this.#retention.activeWindowMs);
         const cursorFloor = activeCursorFloor ?? Number.MAX_SAFE_INTEGER;
         const forcedSeq = await this.#storage.getCommitSeqBefore(partition, nowMs - this.#retention.ageForceMs);
         const retainFloor = maxCommitSeq - this.#retention.minRetainedCommits;
@@ -283,14 +279,9 @@ export class SyncularAdmin {
             buckets: { widthMs, counts, errors },
         };
     }
-    /**
-     * Every partition the storage knows (commit log + client records) — the
-     * fleet-view backing and the console's partition picker. Fails loud when
-     * the backend omits the optional `listPartitions`.
-     */
+    /** Every authenticated partition in the storage-backed registry. */
     async listPartitions() {
-        const list = required(this.#storage.listPartitions?.bind(this.#storage), 'storage');
-        return list();
+        return (await this.#storage.listPartitionRegistry()).map((entry) => entry.partition);
     }
     /**
      * The fleet view: one row per known partition — retained backlog, client

package/dist/authoritative-query.d.ts

@@ -9,12 +9,20 @@ export interface BoundAuthoritativeQuery {
     readonly params: readonly AuthoritativeQueryValue[];
 }
 declare const PARTITION_BIND: unique symbol;
-/**
- * Turn generated local SQL into a partition-local authoritative statement.
- * Only relations declared by the generated descriptor are rewritten. Values
- * remain parameters; request data is never interpolated into SQL.
- */
-export declare function prepareAuthoritativeQuery(sql: string, params: readonly AuthoritativeQueryValue[], declaredTables: readonly string[], tables: ReadonlyMap<string, CompiledTable>): PreparedAuthoritativeQuery;
+/** Compiler-proven occurrences in the exact generated positional statement. */
+export interface AuthoritativeRelationPlan {
+    readonly sql: string;
+    readonly relations: readonly {
+        readonly table: string;
+        readonly start: number;
+        readonly end: number;
+        readonly alias?: string;
+    }[];
+}
+/** Validate trusted generated metadata before registration or storage execution. */
+export declare function validateAuthoritativeRelationPlan(plan: AuthoritativeRelationPlan, declaredTables: readonly string[]): void;
+/** Bind every compiler-proven table occurrence to the authenticated partition. */
+export declare function prepareAuthoritativeQuery(plan: AuthoritativeRelationPlan, params: readonly AuthoritativeQueryValue[], declaredTables: readonly string[], tables: ReadonlyMap<string, CompiledTable>): PreparedAuthoritativeQuery;
 export declare function bindAuthoritativePartition(prepared: PreparedAuthoritativeQuery, partition: string): BoundAuthoritativeQuery;
 export declare function postgresPlaceholders(sql: string): string;
 export {};

package/dist/authoritative-query.js

@@ -1,24 +1,5 @@
 import { quoteIdent, SYNC_PARTITION_COLUMN } from './relational-rows.js';
 const PARTITION_BIND = Symbol('syncular.authoritative_partition');
-const RESERVED_ALIAS = new Set([
-    'on',
-    'where',
-    'group',
-    'order',
-    'inner',
-    'left',
-    'right',
-    'full',
-    'outer',
-    'natural',
-    'join',
-    'cross',
-    'using',
-    'limit',
-    'having',
-]);
-const IDENT = '[A-Za-z_][A-Za-z0-9_]*';
-const TABLE_REF_RE = new RegExp(`\\b(FROM|(?:NATURAL\\s+)?(?:(?:LEFT|RIGHT|FULL)(?:\\s+OUTER)?|INNER|CROSS)?\\s*JOIN)\\s+((?:\\(\\s*)*)(${IDENT})(?:\\s+(?:AS\\s+)?((?!(?:${[...RESERVED_ALIAS].join('|')})\\b)${IDENT}))?`, 'gi');
 function protectedSqlEnd(sql, index) {
     const char = sql[index];
     const next = sql[index + 1];
@@ -48,97 +29,94 @@ function protectedSqlEnd(sql, index) {
     }
     return undefined;
 }
-function maskedSql(sql) {
-    let out = '';
-    let index = 0;
-    while (index < sql.length) {
-        const end = protectedSqlEnd(sql, index);
-        if (end === undefined)
-            out += sql[index];
-        else
-            out += sql.slice(index, end).replace(/[^\n]/g, ' ');
-        index = end ?? index + 1;
+/** Validate trusted generated metadata before registration or storage execution. */
+export function validateAuthoritativeRelationPlan(plan, declaredTables) {
+    if (plan === undefined ||
+        typeof plan.sql !== 'string' ||
+        !Array.isArray(plan.relations)) {
+        throw new Error('registered query requires generated relation plans; regenerate queries');
     }
-    return out;
-}
-/**
- * Turn generated local SQL into a partition-local authoritative statement.
- * Only relations declared by the generated descriptor are rewritten. Values
- * remain parameters; request data is never interpolated into SQL.
- */
-export function prepareAuthoritativeQuery(sql, params, declaredTables, tables) {
-    if (maskedSql(sql).includes(';'))
-        throw new Error('registered query must be one SELECT');
+    const sql = plan.sql;
     const declared = new Set(declaredTables);
-    const masked = maskedSql(sql);
-    const replacements = [];
     const found = new Set();
-    for (const match of masked.matchAll(TABLE_REF_RE)) {
-        const rawTable = match[3];
-        const table = tables.get(rawTable);
-        if (table === undefined)
-            continue;
-        if (!declared.has(table.name)) {
-            throw new Error('registered query table metadata does not match its SQL');
+    let previousEnd = 0;
+    for (const relation of plan.relations) {
+        if (!Number.isSafeInteger(relation.start) ||
+            !Number.isSafeInteger(relation.end) ||
+            relation.start < previousEnd ||
+            relation.end <= relation.start ||
+            relation.end > sql.length) {
+            throw new Error('registered query relation boundaries do not match its SQL; regenerate queries');
         }
-        if (!table.materialize) {
-            throw new Error('registered query targets a non-materialized table');
+        previousEnd = relation.end;
+        if (!declared.has(relation.table)) {
+            throw new Error('registered query table metadata does not match its SQL');
         }
-        let alias = match[4];
-        if (alias !== undefined && RESERVED_ALIAS.has(alias.toLowerCase())) {
-            alias = undefined;
+        const spelling = sql.slice(relation.start, relation.end);
+        const quote = spelling[0];
+        const name = quote === '['
+            ? spelling.slice(1, -1)
+            : quote === '"' || quote === '`'
+                ? spelling
+                    .slice(1, -1)
+                    .split(quote + quote)
+                    .join(quote)
+                : spelling;
+        if (name.toLowerCase() !== relation.table.toLowerCase()) {
+            throw new Error('registered query relation name does not match its SQL; regenerate queries');
         }
-        const matchStart = match.index ?? 0;
-        const afterOperator = match[1].length;
-        const relative = match[0]
-            .toLowerCase()
-            .indexOf(rawTable.toLowerCase(), afterOperator);
-        const start = matchStart + relative;
-        const projection = table.columns.map((column) => quoteIdent(column.name));
-        replacements.push({
-            start,
-            end: start + rawTable.length,
-            text: `(SELECT ${projection.join(', ')} FROM ${quoteIdent(table.name)} WHERE ${quoteIdent(SYNC_PARTITION_COLUMN)}=/*syncular_partition*/?)${alias === undefined ? ` AS ${quoteIdent(table.name)}` : ''}`,
-        });
-        found.add(table.name);
+        found.add(relation.table);
     }
     if (found.size !== declared.size ||
         [...declared].some((table) => !found.has(table))) {
         throw new Error('registered query table metadata does not match its SQL');
     }
-    let rewritten = sql;
-    for (const replacement of replacements.sort((left, right) => right.start - left.start)) {
-        rewritten =
-            rewritten.slice(0, replacement.start) +
-                replacement.text +
-                rewritten.slice(replacement.end);
-    }
+}
+/** Bind every compiler-proven table occurrence to the authenticated partition. */
+export function prepareAuthoritativeQuery(plan, params, declaredTables, tables) {
+    validateAuthoritativeRelationPlan(plan, declaredTables);
+    const sql = plan.sql;
+    const replacements = plan.relations.map((relation) => {
+        const table = tables.get(relation.table);
+        if (table === undefined)
+            throw new Error('registered query targets an unknown table');
+        if (!table.materialize)
+            throw new Error('registered query targets a non-materialized table');
+        return {
+            start: relation.start,
+            end: relation.end,
+            text: `(SELECT ${table.columns.map((column) => quoteIdent(column.name)).join(', ')} FROM ${quoteIdent(table.name)} WHERE ${quoteIdent(SYNC_PARTITION_COLUMN)}=?)${relation.alias === undefined ? ` AS ${quoteIdent(table.name)}` : ''}`,
+        };
+    });
     const bound = [];
     let anonymousIndex = 0;
     let rendered = '';
-    for (let index = 0; index < rewritten.length; index += 1) {
-        if (rewritten.startsWith('/*syncular_partition*/?', index)) {
-            rendered += '?';
+    let nextRelation = 0;
+    for (let index = 0; index < sql.length; index += 1) {
+        const replacement = replacements[nextRelation];
+        if (replacement?.start === index) {
+            rendered += replacement.text;
             bound.push(PARTITION_BIND);
-            index += '/*syncular_partition*/?'.length - 1;
+            index = replacement.end - 1;
+            nextRelation += 1;
             continue;
         }
-        const protectedEnd = protectedSqlEnd(rewritten, index);
+        const protectedEnd = protectedSqlEnd(sql, index);
         if (protectedEnd !== undefined) {
-            rendered += rewritten.slice(index, protectedEnd);
+            rendered += sql.slice(index, protectedEnd);
             index = protectedEnd - 1;
             continue;
         }
-        const char = rewritten[index];
+        const char = sql[index];
         if (char !== '?') {
             rendered += char;
             continue;
         }
         let end = index + 1;
-        while (end < rewritten.length && /[0-9]/.test(rewritten[end])) {
+        while (end < sql.length && /[0-9]/.test(sql[end])) {
             end += 1;
         }
-        const numbered = rewritten.slice(index + 1, end);
+        const numbered = sql.slice(index + 1, end);
         const parameterIndex = numbered.length > 0
             ? Number.parseInt(numbered, 10) - 1
             : anonymousIndex++;

package/dist/blob-handlers.js

@@ -7,7 +7,7 @@
  * adapters map these onto `PUT/GET <mount>/blobs/{blobId}`.
  */
 import { blobIdFor, isBlobId } from './blob-store.js';
-import { clockOf, RESOLVER_OUTAGE } from './context.js';
+import { clockOf, RESOLVER_OUTAGE, touchAuthenticatedPartition, } from './context.js';
 import { SyncError, syncError } from './errors.js';
 import { emitEvent } from './events.js';
 import { compileSchema } from './schema.js';
@@ -16,6 +16,7 @@ import { issueBlobUploadUrl, issueBlobUrl } from './signed-url.js';
 const DEFAULT_MAX_BLOB_BYTES = 64 * 1024 * 1024;
 /** `PUT <mount>/blobs/{blobId}` (§5.9.3). Host auth is the adapter's job. */
 export async function handleBlobUpload(ctx, request) {
+    await touchAuthenticatedPartition(ctx);
     const store = ctx.blobs;
     if (store === undefined) {
         throw syncError('blob.not_found', 'this server has no blob store (§5.9)');
@@ -50,6 +51,7 @@ export async function handleBlobUpload(ctx, request) {
  * reference index passes the §3.4 scope check for the actor.
  */
 export async function handleBlobDownload(ctx, blobId) {
+    await touchAuthenticatedPartition(ctx);
     const events = ctx.events;
     if (events === undefined)
         return downloadBlob(ctx, blobId);
@@ -213,6 +215,7 @@ async function downloadBlob(ctx, blobId) {
  * result when no presigned-upload store is configured (client streams direct).
  */
 export async function handleBlobUploadGrant(ctx, request) {
+    await touchAuthenticatedPartition(ctx);
     const store = ctx.blobs;
     if (store === undefined) {
         throw syncError('blob.not_found', 'this server has no blob store (§5.9)');

package/dist/context.d.ts

@@ -14,7 +14,7 @@ import type { ServerSchema } from './schema.js';
 import type { SegmentStore } from './segment-store.js';
 import type { BlobPresignConfig, BlobUploadPresignConfig, SegmentUrlConfig } from './signed-url.js';
 import type { SqliteImageBuilder } from './sqlite-image.js';
-import type { ServerStorage, StoredCommit } from './storage.js';
+import type { PartitionRegistryEntry, ServerStorage, StoredCommit } from './storage.js';
 import type { CommitValidator, ValidatorRegistry } from './validate.js';
 /** SSP2 body content type (§1.1). */
 export declare const SSP2_CONTENT_TYPE = "application/vnd.syncular.sync.v2";
@@ -166,3 +166,5 @@ export interface SyncRequestContext extends SyncServerConfig {
 }
 export declare function clockOf(ctx: SyncServerConfig): () => number;
 export declare function limitsOf(ctx: SyncServerConfig): ServerLimits;
+/** Refresh the registry after host authentication and return log continuity. */
+export declare function touchAuthenticatedPartition(ctx: SyncRequestContext): Promise<PartitionRegistryEntry>;

package/dist/context.js

@@ -21,3 +21,7 @@ export function clockOf(ctx) {
 export function limitsOf(ctx) {
     return { ...DEFAULT_LIMITS, ...ctx.limits };
 }
+/** Refresh the registry after host authentication and return log continuity. */
+export function touchAuthenticatedPartition(ctx) {
+    return ctx.storage.touchPartition(ctx.partition, clockOf(ctx)(), crypto.randomUUID());
+}

package/dist/d1-storage.d.ts

@@ -1,5 +1,6 @@
+import type { CommitPruneQuery, CommitPruneResult } from './storage.js';
 import type { CompiledSchema, CompiledTable } from './schema.js';
-import type { AuthoritativeQueryRequest, AuthoritativeQueryResult, ClientCursorInfo, ClientRecord, CommitMetadata, CommitMetadataQuery, CommitWindowQuery, IndexRowScanQuery, PrunedReactionCounts, ReactionClaimQuery, ReactionFailureUpdate, ReactionListQuery, ReactionPruneQuery, RowScanQuery, ScopeActivityQuery, ScopeCommitActivity, ServerStorage, StorageTransaction, StoredCommit, StoredPushResult, StoredReaction, StoredRow } from './storage.js';
+import type { AuthoritativeQueryRequest, AuthoritativeQueryResult, ClientCursorInfo, ClientRecord, CommitMetadata, CommitMetadataQuery, CommitWindowQuery, IndexRowScanQuery, PartitionRegistryEntry, PrunedReactionCounts, ReactionClaimQuery, ReactionFailureUpdate, ReactionListQuery, ReactionPruneQuery, RowScanQuery, ScopeActivityQuery, ScopeCommitActivity, ServerStorage, StorageTransaction, StoredCommit, StoredPushResult, StoredReaction, StoredRow } from './storage.js';
 export interface D1PreparedStatement {
     bind(...values: unknown[]): D1PreparedStatement;
     first<T = Record<string, unknown>>(): Promise<T | null>;
@@ -36,12 +37,16 @@ export declare class D1ServerStorage implements ServerStorage {
     /** Resolve a table's compiled schema; row operations require `ensureSchema`. */
     table(name: string): CompiledTable;
     ensureSchema(schema: CompiledSchema): Promise<void>;
+    touchPartition(partition: string, authenticatedAtMs: number, initialLogEpoch: string): Promise<PartitionRegistryEntry>;
+    rotatePartitionLogEpoch(partition: string, logEpoch: string, authenticatedAtMs: number): Promise<PartitionRegistryEntry>;
+    listPartitionRegistry(): Promise<PartitionRegistryEntry[]>;
     begin(partition: string): Promise<StorageTransaction>;
     getMaxCommitSeq(partition: string): Promise<number>;
     queryAuthoritative(partition: string, query: AuthoritativeQueryRequest): Promise<AuthoritativeQueryResult>;
+    getPartitionLogEpoch(partition: string): Promise<string | undefined>;
     getHorizonSeq(partition: string): Promise<number>;
     setHorizonSeq(partition: string, seq: number): Promise<void>;
-    pruneCommitsThrough(partition: string, seq: number): Promise<number>;
+    pruneCommitsThrough(partition: string, query: CommitPruneQuery): Promise<CommitPruneResult>;
     getCommitSeqBefore(partition: string, createdBeforeMs: number): Promise<number>;
     getRow(partition: string, table: string, rowId: string): Promise<StoredRow | undefined>;
     getPushResult(partition: string, clientId: string, clientCommitId: string): Promise<StoredPushResult | undefined>;
@@ -58,6 +63,7 @@ export declare class D1ServerStorage implements ServerStorage {
     scanRowsByIndex(partition: string, query: IndexRowScanQuery): Promise<StoredRow[]>;
     getClientRecord(partition: string, clientId: string): Promise<ClientRecord | undefined>;
     putClientRecord(partition: string, record: ClientRecord): Promise<void>;
+    getActiveClientCursorFloor(partition: string, cutoffMs: number): Promise<number | null>;
     listClientCursors(partition: string): Promise<ClientCursorInfo[]>;
     listRowsReferencingBlob(partition: string, blobId: string): Promise<{
         readonly table: string;