@syncular/server 0.15.48 → 0.17.0

package/README.md

@@ -15,6 +15,11 @@ registry, command mutations use the ordinary serialized push path, and
 specified in [`docs/REMOTE.md`](../../docs/REMOTE.md) and the practical setup is
 in the [remote operations guide](https://syncular.dev/guide-remote-operations/).
 
+Remote query registration requires generated `relationPlans` for each selected
+SQL statement. Run `syncular generate` before upgrading existing query modules.
+The server uses those boundaries to bind every physical table occurrence to
+the authenticated partition, including quoted self joins and CTE bodies.
+
 Application intent belongs in immutable domain event rows written in the same
 commit as the state change. `SyncularServerEvents` below remains operational
 telemetry. See the [domain event guide](https://syncular.dev/guide-domain-events/).
@@ -105,6 +110,11 @@ Log the cause for operators and stop startup. Do not catch readiness errors in
 authentication or convert them to a 401; request-time schema checks are only a
 defensive fallback.
 
+For D1, finish `storage.migrateSchema(compileSchema(schema))` across separate
+Worker invocations before admitting traffic. Each call returns `complete` and
+`statementsExecuted`; the default budget is 50 statements. See
+[D1 schema migration](https://syncular.dev/server-workers/#schema-migration).
+
 After restoring an authoritative database, keep traffic stopped and call
 `rotatePartitionLogEpoch({ storage, partition })` for every restored
 partition. The rotation clears stale client cursors and requires version 2
@@ -927,10 +937,22 @@ at or below it. Nothing prunes automatically — the host schedules it.
 
 **When to run.** A periodic job per partition — hourly to daily is the
 sensible range; there is no benefit below the granularity of your
-`activeWindowMs`. Prune is cheap when there is nothing to do (one cursor
-scan + two point reads), so err on the side of running it often rather
-than letting a backlog build. Pass `events` to get `prune.completed`
-per pass.
+`activeWindowMs`. Pass `events` to get `prune.completed` per pass.
+
+Pruning verifies the captured log epoch and updates the horizon together with
+commit/change/scope deletion in one transaction. Concurrent passes cannot
+lower the horizon. A retry cleans up eligible records even when the horizon
+already covers them. A restore invalidates a pending pass with
+`sync.storage.prune_epoch_mismatch`; recompute its retention inputs.
+Unregistered partitions cannot be pruned. D1 maintenance enters the owning
+Durable Object's existing write queue through
+`SyncularRealtimeHost.pruneCommitLog`.
+
+Custom storage adapters must add `getPartitionLogEpoch(partition)` and replace
+`pruneCommitsThrough(partition, seq)` with
+`pruneCommitsThrough(partition, { logEpoch, throughSeq })`, returning
+`{ previousHorizonSeq, horizonSeq, removedCommits }` from the transaction.
+`setHorizonSeq` remains monotonic and is no longer used by the pruning helper.
 
 **The retention floors (§4.6, encoded in `RetentionPolicy`).** The
 horizon never advances past `min(cursor)` of *active* clients — clients
@@ -1094,13 +1116,19 @@ cannot silently return.
 ### commitSeq allocation under concurrency
 
 Per-partition `commitSeq` is dense and gap-free (§2.1). `appendCommit`
-allocates it with `UPDATE sync_partitions SET max_commit_seq =
-max_commit_seq + 1 … RETURNING`, which takes a row-level write lock on the
-partition row for the transaction's duration — concurrent pushes to the
-same partition serialize on that row; cross-partition pushes never
-contend. A Postgres `SEQUENCE` is deliberately **not** used: it would leave
+allocates it with an upsert that increments `sync_partitions.max_commit_seq`
+and returns the allocated value. A common table expression feeds that value
+into the commit metadata insert in the same SQL statement. The upsert takes a
+row-level write lock on the partition for the transaction duration. Concurrent
+pushes to that partition serialize; separate partitions use separate locks. A Postgres `SEQUENCE` is deliberately **not** used: it would leave
 gaps on rollback, which the §4.5 pull-window arithmetic does not tolerate.
 
+Each change insert expands its scope object into the inverted scope entries in
+the same statement. Serialized scopes bind as text before JSONB parsing; this
+avoids driver-specific JSON string encoding. Historical string-form scopes
+remain readable. Empty scopes still produce a change row, and repeated scopes
+within a commit retain one index entry.
+
 ### Multi-instance fanout (LISTEN/NOTIFY)
 
 Behind a load balancer, a commit applied on instance A fans out to A's
@@ -1149,3 +1177,32 @@ deterministic in-process sqlite loopback):
 ```sh
 SYNCULAR_PG_URL=postgres://user:pass@localhost:5432/db bun run bench
 ```
+
+Custom storage adapters must implement
+`getActiveClientCursorFloor(partition, cutoffMs)`. Return the minimum cursor
+whose `updatedAtMs >= cutoffMs`, or `null` when no client qualifies. Preserve
+negative bootstrap cursors. Pruning and admin horizon status use this scalar
+aggregate; `listClientCursors` remains the explicit listing interface.
+
+SQLite image builders now return `Promise<Uint8Array>` and receive
+`rowBatches`, an iterable or async iterable of row arrays. Replace custom
+builders' `input.rows` loop with `for await (const rows of input.rowBatches)`,
+insert each batch into the dedicated image database, and count rows during
+consumption. Write the final row count into `_syncular_segment` before
+serialization. Await `buildSqliteImage(input)` when calling the built-in
+Bun or Node builder directly.
+
+The server shares in-flight builds for the same storage pair and artifact
+identity after authorization. Sharing is local to one process. Signed URL
+grants remain per request. The first eligibility probe has at most
+`limitSnapshotRows + 1` rows; subsequent builder batches have at most 5,000
+rows. The image database and serialized output still consume memory.
+
+
+Realtime acknowledgements call `advanceClientCursor(partition, clientId,
+actorId, logEpoch, cursor, updatedAtMs)`. Custom storage adapters must implement
+this atomic update: advance the cursor and activity timestamp to their respective
+maxima, preserve registration fields, and require a matching actor and current
+partition log epoch. Leave missing records unchanged. SQLite, Postgres, and D1
+perform one update without reading or serializing the subscription list. HTTP
+registration keeps its existing cursor and subscription replacement rules.

package/dist/admin.js

@@ -173,11 +173,7 @@ export class SyncularAdmin {
         const nowMs = this.#clock();
         const maxCommitSeq = await this.#storage.getMaxCommitSeq(partition);
         const horizonSeq = await this.#storage.getHorizonSeq(partition);
-        const cursors = await this.#storage.listClientCursors(partition);
-        const activeCursors = cursors
-            .filter((c) => c.updatedAtMs >= nowMs - this.#retention.activeWindowMs)
-            .map((c) => c.cursor);
-        const activeCursorFloor = activeCursors.length > 0 ? Math.min(...activeCursors) : null;
+        const activeCursorFloor = await this.#storage.getActiveClientCursorFloor(partition, nowMs - this.#retention.activeWindowMs);
         const cursorFloor = activeCursorFloor ?? Number.MAX_SAFE_INTEGER;
         const forcedSeq = await this.#storage.getCommitSeqBefore(partition, nowMs - this.#retention.ageForceMs);
         const retainFloor = maxCommitSeq - this.#retention.minRetainedCommits;

package/dist/authoritative-query.d.ts

@@ -9,12 +9,20 @@ export interface BoundAuthoritativeQuery {
     readonly params: readonly AuthoritativeQueryValue[];
 }
 declare const PARTITION_BIND: unique symbol;
-/**
- * Turn generated local SQL into a partition-local authoritative statement.
- * Only relations declared by the generated descriptor are rewritten. Values
- * remain parameters; request data is never interpolated into SQL.
- */
-export declare function prepareAuthoritativeQuery(sql: string, params: readonly AuthoritativeQueryValue[], declaredTables: readonly string[], tables: ReadonlyMap<string, CompiledTable>): PreparedAuthoritativeQuery;
+/** Compiler-proven occurrences in the exact generated positional statement. */
+export interface AuthoritativeRelationPlan {
+    readonly sql: string;
+    readonly relations: readonly {
+        readonly table: string;
+        readonly start: number;
+        readonly end: number;
+        readonly alias?: string;
+    }[];
+}
+/** Validate trusted generated metadata before registration or storage execution. */
+export declare function validateAuthoritativeRelationPlan(plan: AuthoritativeRelationPlan, declaredTables: readonly string[]): void;
+/** Bind every compiler-proven table occurrence to the authenticated partition. */
+export declare function prepareAuthoritativeQuery(plan: AuthoritativeRelationPlan, params: readonly AuthoritativeQueryValue[], declaredTables: readonly string[], tables: ReadonlyMap<string, CompiledTable>): PreparedAuthoritativeQuery;
 export declare function bindAuthoritativePartition(prepared: PreparedAuthoritativeQuery, partition: string): BoundAuthoritativeQuery;
 export declare function postgresPlaceholders(sql: string): string;
 export {};

package/dist/authoritative-query.js

@@ -1,24 +1,5 @@
 import { quoteIdent, SYNC_PARTITION_COLUMN } from './relational-rows.js';
 const PARTITION_BIND = Symbol('syncular.authoritative_partition');
-const RESERVED_ALIAS = new Set([
-    'on',
-    'where',
-    'group',
-    'order',
-    'inner',
-    'left',
-    'right',
-    'full',
-    'outer',
-    'natural',
-    'join',
-    'cross',
-    'using',
-    'limit',
-    'having',
-]);
-const IDENT = '[A-Za-z_][A-Za-z0-9_]*';
-const TABLE_REF_RE = new RegExp(`\\b(FROM|(?:NATURAL\\s+)?(?:(?:LEFT|RIGHT|FULL)(?:\\s+OUTER)?|INNER|CROSS)?\\s*JOIN)\\s+((?:\\(\\s*)*)(${IDENT})(?:\\s+(?:AS\\s+)?((?!(?:${[...RESERVED_ALIAS].join('|')})\\b)${IDENT}))?`, 'gi');
 function protectedSqlEnd(sql, index) {
     const char = sql[index];
     const next = sql[index + 1];
@@ -48,97 +29,94 @@ function protectedSqlEnd(sql, index) {
     }
     return undefined;
 }
-function maskedSql(sql) {
-    let out = '';
-    let index = 0;
-    while (index < sql.length) {
-        const end = protectedSqlEnd(sql, index);
-        if (end === undefined)
-            out += sql[index];
-        else
-            out += sql.slice(index, end).replace(/[^\n]/g, ' ');
-        index = end ?? index + 1;
+/** Validate trusted generated metadata before registration or storage execution. */
+export function validateAuthoritativeRelationPlan(plan, declaredTables) {
+    if (plan === undefined ||
+        typeof plan.sql !== 'string' ||
+        !Array.isArray(plan.relations)) {
+        throw new Error('registered query requires generated relation plans; regenerate queries');
     }
-    return out;
-}
-/**
- * Turn generated local SQL into a partition-local authoritative statement.
- * Only relations declared by the generated descriptor are rewritten. Values
- * remain parameters; request data is never interpolated into SQL.
- */
-export function prepareAuthoritativeQuery(sql, params, declaredTables, tables) {
-    if (maskedSql(sql).includes(';'))
-        throw new Error('registered query must be one SELECT');
+    const sql = plan.sql;
     const declared = new Set(declaredTables);
-    const masked = maskedSql(sql);
-    const replacements = [];
     const found = new Set();
-    for (const match of masked.matchAll(TABLE_REF_RE)) {
-        const rawTable = match[3];
-        const table = tables.get(rawTable);
-        if (table === undefined)
-            continue;
-        if (!declared.has(table.name)) {
-            throw new Error('registered query table metadata does not match its SQL');
+    let previousEnd = 0;
+    for (const relation of plan.relations) {
+        if (!Number.isSafeInteger(relation.start) ||
+            !Number.isSafeInteger(relation.end) ||
+            relation.start < previousEnd ||
+            relation.end <= relation.start ||
+            relation.end > sql.length) {
+            throw new Error('registered query relation boundaries do not match its SQL; regenerate queries');
         }
-        if (!table.materialize) {
-            throw new Error('registered query targets a non-materialized table');
+        previousEnd = relation.end;
+        if (!declared.has(relation.table)) {
+            throw new Error('registered query table metadata does not match its SQL');
         }
-        let alias = match[4];
-        if (alias !== undefined && RESERVED_ALIAS.has(alias.toLowerCase())) {
-            alias = undefined;
+        const spelling = sql.slice(relation.start, relation.end);
+        const quote = spelling[0];
+        const name = quote === '['
+            ? spelling.slice(1, -1)
+            : quote === '"' || quote === '`'
+                ? spelling
+                    .slice(1, -1)
+                    .split(quote + quote)
+                    .join(quote)
+                : spelling;
+        if (name.toLowerCase() !== relation.table.toLowerCase()) {
+            throw new Error('registered query relation name does not match its SQL; regenerate queries');
         }
-        const matchStart = match.index ?? 0;
-        const afterOperator = match[1].length;
-        const relative = match[0]
-            .toLowerCase()
-            .indexOf(rawTable.toLowerCase(), afterOperator);
-        const start = matchStart + relative;
-        const projection = table.columns.map((column) => quoteIdent(column.name));
-        replacements.push({
-            start,
-            end: start + rawTable.length,
-            text: `(SELECT ${projection.join(', ')} FROM ${quoteIdent(table.name)} WHERE ${quoteIdent(SYNC_PARTITION_COLUMN)}=/*syncular_partition*/?)${alias === undefined ? ` AS ${quoteIdent(table.name)}` : ''}`,
-        });
-        found.add(table.name);
+        found.add(relation.table);
     }
     if (found.size !== declared.size ||
         [...declared].some((table) => !found.has(table))) {
         throw new Error('registered query table metadata does not match its SQL');
     }
-    let rewritten = sql;
-    for (const replacement of replacements.sort((left, right) => right.start - left.start)) {
-        rewritten =
-            rewritten.slice(0, replacement.start) +
-                replacement.text +
-                rewritten.slice(replacement.end);
-    }
+}
+/** Bind every compiler-proven table occurrence to the authenticated partition. */
+export function prepareAuthoritativeQuery(plan, params, declaredTables, tables) {
+    validateAuthoritativeRelationPlan(plan, declaredTables);
+    const sql = plan.sql;
+    const replacements = plan.relations.map((relation) => {
+        const table = tables.get(relation.table);
+        if (table === undefined)
+            throw new Error('registered query targets an unknown table');
+        if (!table.materialize)
+            throw new Error('registered query targets a non-materialized table');
+        return {
+            start: relation.start,
+            end: relation.end,
+            text: `(SELECT ${table.columns.map((column) => quoteIdent(column.name)).join(', ')} FROM ${quoteIdent(table.name)} WHERE ${quoteIdent(SYNC_PARTITION_COLUMN)}=?)${relation.alias === undefined ? ` AS ${quoteIdent(table.name)}` : ''}`,
+        };
+    });
     const bound = [];
     let anonymousIndex = 0;
     let rendered = '';
-    for (let index = 0; index < rewritten.length; index += 1) {
-        if (rewritten.startsWith('/*syncular_partition*/?', index)) {
-            rendered += '?';
+    let nextRelation = 0;
+    for (let index = 0; index < sql.length; index += 1) {
+        const replacement = replacements[nextRelation];
+        if (replacement?.start === index) {
+            rendered += replacement.text;
             bound.push(PARTITION_BIND);
-            index += '/*syncular_partition*/?'.length - 1;
+            index = replacement.end - 1;
+            nextRelation += 1;
             continue;
         }
-        const protectedEnd = protectedSqlEnd(rewritten, index);
+        const protectedEnd = protectedSqlEnd(sql, index);
         if (protectedEnd !== undefined) {
-            rendered += rewritten.slice(index, protectedEnd);
+            rendered += sql.slice(index, protectedEnd);
             index = protectedEnd - 1;
             continue;
         }
-        const char = rewritten[index];
+        const char = sql[index];
         if (char !== '?') {
             rendered += char;
             continue;
         }
         let end = index + 1;
-        while (end < rewritten.length && /[0-9]/.test(rewritten[end])) {
+        while (end < sql.length && /[0-9]/.test(sql[end])) {
             end += 1;
         }
-        const numbered = rewritten.slice(index + 1, end);
+        const numbered = sql.slice(index + 1, end);
         const parameterIndex = numbered.length > 0
             ? Number.parseInt(numbered, 10) - 1
             : anonymousIndex++;

package/dist/d1-storage.d.ts

@@ -1,3 +1,4 @@
+import type { CommitPruneQuery, CommitPruneResult } from './storage.js';
 import type { CompiledSchema, CompiledTable } from './schema.js';
 import type { AuthoritativeQueryRequest, AuthoritativeQueryResult, ClientCursorInfo, ClientRecord, CommitMetadata, CommitMetadataQuery, CommitWindowQuery, IndexRowScanQuery, PartitionRegistryEntry, PrunedReactionCounts, ReactionClaimQuery, ReactionFailureUpdate, ReactionListQuery, ReactionPruneQuery, RowScanQuery, ScopeActivityQuery, ScopeCommitActivity, ServerStorage, StorageTransaction, StoredCommit, StoredPushResult, StoredReaction, StoredRow } from './storage.js';
 export interface D1PreparedStatement {
@@ -36,15 +37,23 @@ export declare class D1ServerStorage implements ServerStorage {
     /** Resolve a table's compiled schema; row operations require `ensureSchema`. */
     table(name: string): CompiledTable;
     ensureSchema(schema: CompiledSchema): Promise<void>;
+    /** Run once per Worker invocation. Resume an incomplete result in another invocation. */
+    migrateSchema(schema: CompiledSchema, options?: {
+        readonly maxStatements?: number;
+    }): Promise<{
+        readonly complete: boolean;
+        readonly statementsExecuted: number;
+    }>;
     touchPartition(partition: string, authenticatedAtMs: number, initialLogEpoch: string): Promise<PartitionRegistryEntry>;
     rotatePartitionLogEpoch(partition: string, logEpoch: string, authenticatedAtMs: number): Promise<PartitionRegistryEntry>;
     listPartitionRegistry(): Promise<PartitionRegistryEntry[]>;
     begin(partition: string): Promise<StorageTransaction>;
     getMaxCommitSeq(partition: string): Promise<number>;
     queryAuthoritative(partition: string, query: AuthoritativeQueryRequest): Promise<AuthoritativeQueryResult>;
+    getPartitionLogEpoch(partition: string): Promise<string | undefined>;
     getHorizonSeq(partition: string): Promise<number>;
     setHorizonSeq(partition: string, seq: number): Promise<void>;
-    pruneCommitsThrough(partition: string, seq: number): Promise<number>;
+    pruneCommitsThrough(partition: string, query: CommitPruneQuery): Promise<CommitPruneResult>;
     getCommitSeqBefore(partition: string, createdBeforeMs: number): Promise<number>;
     getRow(partition: string, table: string, rowId: string): Promise<StoredRow | undefined>;
     getPushResult(partition: string, clientId: string, clientCommitId: string): Promise<StoredPushResult | undefined>;
@@ -61,6 +70,9 @@ export declare class D1ServerStorage implements ServerStorage {
     scanRowsByIndex(partition: string, query: IndexRowScanQuery): Promise<StoredRow[]>;
     getClientRecord(partition: string, clientId: string): Promise<ClientRecord | undefined>;
     putClientRecord(partition: string, record: ClientRecord): Promise<void>;
+    advanceClientCursor(partition: string, clientId: string, actorId: string, logEpoch: string, cursor: number, updatedAtMs: number): Promise<void>;
+    updateClientCursor(partition: string, clientId: string, cursor: number, updatedAtMs: number): Promise<void>;
+    getActiveClientCursorFloor(partition: string, cutoffMs: number): Promise<number | null>;
     listClientCursors(partition: string): Promise<ClientCursorInfo[]>;
     listRowsReferencingBlob(partition: string, blobId: string): Promise<{
         readonly table: string;