@syncular/client 0.19.0 → 0.20.1

package/README.md

@@ -457,10 +457,17 @@ still happen during rapid hot-module replacement, or in an embedded/test host
 that shares OPFS data without sharing the same Web Locks and BroadcastChannel
 coordination domain.
 
-Pool acquisition failures surface as `ClientSyncError` with code
-`client.storage_busy` and `retryable === true`. Treat that as a startup state:
-close the competing instance or let it finish shutting down, then create the
-handle again. **Do not delete, rename, or silently replace the database with an
+Persistent worker startup retries retryable `client.storage_busy` failures
+while retaining the leader lease. It opens the same directory at most seven
+times, waiting 50, 100, 200, 400, 800, and 1000 ms between attempts. These delays
+total 2550 ms; browser scheduling and storage operations can extend elapsed time.
+Other errors fail immediately. Direct `openPersistentWasmDatabase` calls remain
+single attempts.
+
+If ownership remains unavailable, handle creation rejects with `ClientSyncError`,
+code `client.storage_busy`, and `retryable === true`, then releases the worker
+and leader lease. Close the competing instance or let it finish shutting down,
+then create the handle again. **Do not delete, rename, or silently replace the database with an
 in-memory one**; the local database and pending outbox may be perfectly healthy.
 Missing or obsolete OPFS APIs instead use the non-retryable
 `client.storage_unavailable` code.
@@ -584,8 +591,97 @@ they own their buffer.
 
 Tests drive the real worker entry in a bun `Worker` with bun:sqlite
 injected through the bootstrap's database-factory override
-(`test/worker-rpc.test.ts`); the OPFS path itself is browser-only and is
-exercised by `apps/demo`.
+(`test/worker-rpc.test.ts`). The separate Chromium gate exercises actual
+sqlite-wasm, OPFS SAH pools and FTS through the production worker and HTTP paths:
+
+```sh
+cd packages/web-client
+bunx --no-install playwright install chromium
+cd ../..
+bun run test:browser
+```
+
+`test/opfs-bootstrap.browser.test.ts` covers five interruption boundaries:
+
+- Partial segment download: hold the asynchronous transfer, then terminate the worker.
+- Before image import: inject an exception before applying the verified image.
+- Mid-import: terminate the worker after an actual OPFS database page write during
+  `INSERT SELECT`. A commit notification must not precede termination.
+- After import: inject an exception after the transaction commits, before `SUB_END`.
+- After checkpoint: complete the round before terminating the worker.
+
+Each case reloads the page using the same replica identity and requires successful
+startup in the same browser session. It checks SQLite and FTS integrity, search
+results, exact rows, persisted client identity, and subscription checkpoints.
+Incomplete sections must download again; completed sections must not. A held
+synchronous test request after a physical page write keeps the mid-import crash
+boundary stable until worker termination. Two additional ownership cases verify
+recovery when a competing owner closes during retry and bounded failure while
+that owner remains live, including pending outbox preservation.
+
+Each case owns a temporary browser profile and a loopback server. A small SQLite
+page cache forces dirty pages to disk before COMMIT. Tests use no timing sleeps,
+COOP/COEP requirement, existing profiles or application replicas. Failed profiles
+and `recovery-test.json` remain at the path printed by the test; CI retains them
+as an artifact. Successful profiles are removed. `bun run check` keeps its
+Bun-only test lane; CI runs the browser gate separately.
+
+`openPersistentWasmDatabase` corrects the SAH-pool VFS's reserved-lock callback
+before the first SQL statement. sqlite-wasm 3.53 reports a reserved lock
+unconditionally, suppressing SQLite's hot-journal recovery after an interrupted
+write. The pool already excludes other owners. Syncular reports no competing
+writer through SQLite's I/O method binding, matching the other OPFS VFS, and
+retains that callback across database closes and pool opens. The correction ships
+in the client package and preserves the existing database format, DELETE journal
+mode and FULL synchronous mode.
+
+On 2026-09-12, Chromium 153 with sqlite-wasm 3.53.0 failed the mid-import case
+without this correction and passed all five cases with it. The fixture serves
+the unmodified dependency. Bounded worker startup retries handle transient
+`client.storage_busy` while the browser releases old handles. This correction enables
+rollback from an intact hot journal; it does not repair existing corruption
+whose recovery journal has already been lost or overwritten.
+
+## Live sync progress
+
+`onProgress` observes work during a sync round. It immediately delivers the latest
+snapshot when one exists and returns an unsubscribe function. `progressSnapshot()`
+reads the cached snapshot synchronously, including on worker handles. Worker
+leaders and followers forward events while download and import are running.
+
+```ts
+const stop = client.onProgress((progress) => {
+  if (progress.phase === 'download') {
+    renderDownload(progress.bytesReceived, progress.bytesTotal);
+  } else if (progress.phase === 'import') {
+    renderImport(progress.rowsProcessed, progress.rowsTotal);
+  }
+  if (progress.state === 'failed') showError(progress.errorCode);
+});
+// Stop listening when the view is disposed. Sync continues.
+stop();
+```
+
+Each snapshot identifies the round with `attempt`, and the current payload with
+`subscriptionId`, `table`, and optional `segmentId`. Counters reset for each
+payload; `syncUntilIdle` starts a new attempt for each round. `phase` is `request`,
+`download`, or `import`; `state` is `running`, `complete`, or `failed`. Unknown
+totals remain absent. A new attempt clears prior errors and counters.
+
+HTTP downloads report decoded body bytes at 64 KiB intervals or larger network
+chunks. Custom segment downloaders receive `request.onProgress`; signed-URL
+implementations receive an optional second callback argument. Call it with the
+cumulative received byte count. Buffered custom transports report a final count
+through the core, without intermediate download updates.
+
+Import reports every 1,024 rows and its final count. `rowsProcessed` includes work
+inside the current transaction; a failed import can roll those rows back. SQLite
+images commit in chunks of at most 1,024 rows and yield after each commit.
+Earlier committed chunks survive a later interruption. `complete` follows checkpoint persistence
+and optimistic-state reconciliation for the round. More bootstrap pages can
+remain; use subscription coverage to determine whether a query is complete.
+Listener exceptions do not fail sync. Listeners should hand UI work off and avoid
+reentering database commands during an import.
 
 ## Snapshot API migration
 
@@ -628,3 +724,19 @@ request. Status and diagnostics use `COUNT(*)` without parsing pending bodies.
 Optimistic replay still reads the remaining outbox after each response. The
 100/1,000/10,000-commit workload and measured limits are recorded in
 [the reliability RFC](../../docs/RFC-RELIABILITY-DX.md#9-implementation-evidence-2026-09-05).
+
+## Reads during bootstrap and eviction
+
+Reactive queries read cached snapshots while window registration runs. Coverage
+continues to distinguish complete, pending, and missing windows; registration
+errors remain observable. Retained owners reuse acknowledged units immediately.
+
+Both segment formats yield between committed import chunks automatically.
+Window eviction deletes at most 1,024 unpinned rows per chunk and resumes from a
+durable pending record after interruption. Security purges remain atomic.
+See [windowing](https://syncular.dev/concepts-windowing/) and
+[bootstrap](https://syncular.dev/concepts-bootstrap/) for coverage and recovery.
+
+Custom database adapters implementing `withSqliteImage` must return a promise and
+await their callback before detaching the image. The callback can await between
+committed import chunks; no write transaction spans that await.

package/dist/apply.d.ts

@@ -43,12 +43,15 @@ export declare function deleteScopedRows(db: ClientDatabase, table: CompiledClie
  * scopes (same local-scope-column rule and fail-closed clause as
  * {@link deleteScopedRows}) EXCEPT rows whose primary key is in
  * `pinnedRowIds` (E1 — pinned by a still-pending outbox commit). Returns
- * `true` iff any pinned row was left behind, so the caller knows to defer
- * the rest of the eviction until the outbox drains. Also removes the
+ * whether unpinned work remains and whether any scoped rows remain. Each
+ * call removes at most 1,024 rows; the caller commits before yielding. Also removes the
  * evicted rows' `server_version` with them (E2 — no residual version
  * cache), which is automatic since the version column is per-row.
  */
-export declare function evictScopedRows(db: ClientDatabase, table: CompiledClientTable, effective: ScopeMap, pinnedRowIds: ReadonlySet<string>): boolean;
+export declare function evictScopedRows(db: ClientDatabase, table: CompiledClientTable, effective: ScopeMap, pinnedRowIds: ReadonlySet<string>): {
+    remaining: boolean;
+    deferred: boolean;
+};
 /** Descriptor fields a sqlite image is validated against (§5.3). */
 export interface SqliteSegmentDescriptor {
     readonly table: string;
@@ -57,18 +60,20 @@ export interface SqliteSegmentDescriptor {
     readonly scopeDigest: string;
 }
 /**
- * Apply a §5.3 sqlite-image segment in ONE local transaction: validate
+ * Apply a §5.3 sqlite image in committed chunks of at most 1,024 rows: validate
  * the in-file metadata against the descriptor, validate the data table's
  * column names/order against the generated schema, run the §5.6
- * first-page clear when fresh, then copy every row with a single
- * one primary-key upsert — `_syncular_version` lands in
+ * first-page clear when fresh, then bulk-copy each chunk through a
+ * primary-key upsert — `_syncular_version` lands in
  * `_sync_version` exactly like a rows segment's per-row `serverVersion`.
  */
 export declare function applySqliteSegment(db: ClientDatabase, schema: CompiledClientSchema, table: CompiledClientTable, bytes: Uint8Array, descriptor: SqliteSegmentDescriptor, options: {
     readonly clearFirst: boolean;
     readonly effective: ScopeMap;
     readonly transaction?: ApplyTransaction;
-}): number;
+    readonly onProgress?: (rowsProcessed: number) => void;
+    readonly isCurrent?: () => boolean;
+}): Promise<number>;
 /**
  * Apply a decoded rows segment: each block in one local transaction
  * (§5.2/§1.4); `clearFirst` implements the §5.6 fresh-bootstrap first-page
@@ -81,4 +86,8 @@ export declare function applyRowsSegment(db: ClientDatabase, schema: CompiledCli
     readonly clearFirst: boolean;
     readonly effective: ScopeMap;
     readonly transaction?: ApplyTransaction;
+    readonly onProgress?: (rowsProcessed: number) => void;
+    readonly isCurrent?: () => boolean;
 }, encryption?: EncryptionConfig): Promise<number>;
+/** Let worker messages run only after the current SQLite transaction commits. */
+export declare function yieldToHost(): Promise<void>;

package/dist/apply.js

@@ -138,15 +138,15 @@ export function deleteScopedRows(db, table, effective) {
  * scopes (same local-scope-column rule and fail-closed clause as
  * {@link deleteScopedRows}) EXCEPT rows whose primary key is in
  * `pinnedRowIds` (E1 — pinned by a still-pending outbox commit). Returns
- * `true` iff any pinned row was left behind, so the caller knows to defer
- * the rest of the eviction until the outbox drains. Also removes the
+ * whether unpinned work remains and whether any scoped rows remain. Each
+ * call removes at most 1,024 rows; the caller commits before yielding. Also removes the
  * evicted rows' `server_version` with them (E2 — no residual version
  * cache), which is automatic since the version column is per-row.
  */
 export function evictScopedRows(db, table, effective, pinnedRowIds) {
     const entries = Object.entries(effective);
     if (entries.length === 0)
-        return false;
+        return { remaining: false, deferred: false };
     const clauses = [];
     const params = [];
     for (const [variable, values] of entries) {
@@ -155,7 +155,7 @@ export function evictScopedRows(db, table, effective, pinnedRowIds) {
             throw new ClientSyncError('sync.scope_revoked', `table ${JSON.stringify(table.name)} has no local scope-column mapping for ${JSON.stringify(variable)} (§4.8/§3.3 fail-closed)`);
         }
         if (values.length === 0)
-            return false;
+            return { remaining: false, deferred: false };
         clauses.push(`${quoteIdent(column)} IN (${values.map(() => '?').join(', ')})`);
         params.push(...values);
     }
@@ -166,31 +166,28 @@ export function evictScopedRows(db, table, effective, pinnedRowIds) {
         pinnedClause = ` AND ${pk} NOT IN (${ids.map(() => '?').join(', ')})`;
         params.push(...ids);
     }
-    db.exec(`DELETE FROM ${quoteIdent(table.name)} WHERE ${clauses.join(' AND ')}${pinnedClause}`, params);
-    if (pinnedRowIds.size === 0)
-        return false;
-    // A pin still matters only if a pinned row actually falls inside this
-    // unit's effective scopes; check by re-selecting the survivors.
-    const survivors = db.query(`SELECT ${pk} AS pk FROM ${quoteIdent(table.name)} WHERE ${clauses.join(' AND ')}`, params.slice(0, params.length - pinnedRowIds.size));
-    for (const row of survivors) {
-        if (pinnedRowIds.has(String(row.pk)))
-            return true;
-    }
-    return false;
+    const target = quoteIdent(table.name);
+    db.exec(`DELETE FROM ${target} WHERE ${pk} IN (
+      SELECT ${pk} FROM ${target} WHERE ${clauses.join(' AND ')}${pinnedClause} LIMIT 1024
+    )`, params);
+    const remaining = db.query(`SELECT 1 FROM ${target} WHERE ${clauses.join(' AND ')}${pinnedClause} LIMIT 1`, params).length > 0;
+    const deferred = remaining ||
+        db.query(`SELECT 1 FROM ${target} WHERE ${clauses.join(' AND ')} LIMIT 1`, params.slice(0, params.length - pinnedRowIds.size)).length > 0;
+    return { remaining, deferred };
 }
 const IMAGE_ALIAS = 'syncular_image';
 function imageInvalid(detail) {
     throw new ClientSyncError('sync.invalid_request', `sqlite segment rejected: ${detail} (§5.3)`);
 }
 /**
- * Apply a §5.3 sqlite-image segment in ONE local transaction: validate
+ * Apply a §5.3 sqlite image in committed chunks of at most 1,024 rows: validate
  * the in-file metadata against the descriptor, validate the data table's
  * column names/order against the generated schema, run the §5.6
- * first-page clear when fresh, then copy every row with a single
- * one primary-key upsert — `_syncular_version` lands in
+ * first-page clear when fresh, then bulk-copy each chunk through a
+ * primary-key upsert — `_syncular_version` lands in
  * `_sync_version` exactly like a rows segment's per-row `serverVersion`.
  */
-export function applySqliteSegment(db, schema, table, bytes, descriptor, options) {
+export async function applySqliteSegment(db, schema, table, bytes, descriptor, options) {
     const withImage = db.withSqliteImage?.bind(db);
     if (withImage === undefined) {
         throw new ClientSyncError('sync.invalid_request', 'received a sqlite segment but the database backend cannot import images (§4.2: do not advertise accept bit 2)');
@@ -198,7 +195,7 @@ export function applySqliteSegment(db, schema, table, bytes, descriptor, options
     if (descriptor.table !== table.name) {
         imageInvalid(`descriptor table ${JSON.stringify(descriptor.table)}`);
     }
-    return withImage(bytes, IMAGE_ALIAS, () => {
+    return withImage(bytes, IMAGE_ALIAS, async () => {
         // 1. Metadata vs descriptor (§5.3 rule 2). A file that is not a
         //    SQLite database or lacks the metadata table fails right here.
         let meta;
@@ -244,30 +241,58 @@ export function applySqliteSegment(db, schema, table, bytes, descriptor, options
             expected.some((name, index) => actual[index] !== name)) {
             throw new ClientSyncError('sync.schema_mismatch', `sqlite segment for ${JSON.stringify(table.name)} does not match the generated schema: columns [${actual.join(', ')}] (§5.3)`);
         }
-        // 3. One transaction: fresh-bootstrap clear, then primary-key upsert.
+        const primaryKeys = info.filter((column) => Number(column.pk) > 0);
+        if (primaryKeys.length !== 1 || primaryKeys[0]?.name !== table.primaryKey)
+            imageInvalid('image primary key does not match generated schema');
+        const source = `${IMAGE_ALIAS}.${quoteIdent(table.name)}`;
+        const count = Number(db.query(`SELECT count(*) AS n FROM ${source}`)[0]?.n);
+        if (count !== descriptor.rowCount)
+            imageInvalid('image row count does not match descriptor');
         const names = table.columns.map((column) => quoteIdent(column.name));
         const insertNames = [...names, quoteIdent(SYNC_VERSION_COLUMN)];
+        const primaryKey = quoteIdent(table.primaryKey);
         const updates = insertNames
-            .filter((name) => name !== quoteIdent(table.primaryKey))
+            .filter((name) => name !== primaryKey)
             .map((name) => `${name}=excluded.${name}`)
             .join(', ');
-        return (options.transaction ?? ((fn) => db.transaction(fn)))(() => {
-            if (options.clearFirst) {
-                deleteScopedRows(db, table, options.effective);
+        let after;
+        let applied = 0;
+        for (;;) {
+            if (options.isCurrent?.() === false)
+                return applied;
+            const boundary = db.query(`SELECT ${primaryKey} AS boundary FROM ${source}
+         ${after === undefined ? '' : `WHERE ${primaryKey} > ?`}
+         ORDER BY ${primaryKey} LIMIT 1 OFFSET 1023`, after === undefined ? [] : [after])[0]?.boundary;
+            if (boundary === null)
+                imageInvalid('image primary key must not be null');
+            const predicates = [];
+            const params = [];
+            if (after !== undefined) {
+                predicates.push(`${primaryKey} > ?`);
+                params.push(after);
             }
-            db.exec(`INSERT INTO ${quoteIdent(table.name)}
-           (${insertNames.join(', ')})
-         SELECT ${[...names, quoteIdent('_syncular_version')].join(', ')}
-         FROM ${IMAGE_ALIAS}.${quoteIdent(table.name)}
-         WHERE true
-         ON CONFLICT (${quoteIdent(table.primaryKey)}) DO UPDATE SET ${updates}`);
-            const counted = db.query(`SELECT count(*) AS n FROM ${IMAGE_ALIAS}.${quoteIdent(table.name)}`)[0];
-            const applied = Number(counted?.n ?? 0);
-            if (applied !== descriptor.rowCount) {
-                imageInvalid(`image holds ${applied} rows, descriptor says ${descriptor.rowCount}`);
+            if (boundary !== undefined) {
+                predicates.push(`${primaryKey} <= ?`);
+                params.push(boundary);
             }
-            return applied;
-        });
+            (options.transaction ?? ((fn) => db.transaction(fn)))(() => {
+                if (after === undefined && options.clearFirst)
+                    deleteScopedRows(db, table, options.effective);
+                db.exec(`INSERT INTO ${quoteIdent(table.name)} (${insertNames.join(', ')})
+           SELECT ${[...names, quoteIdent('_syncular_version')].join(', ')}
+           FROM ${source} WHERE ${predicates.join(' AND ') || 'true'}
+           ON CONFLICT (${primaryKey}) DO UPDATE SET ${updates}`, params);
+                applied += Number(db.query('SELECT changes() AS n')[0]?.n ?? 0);
+            });
+            options.onProgress?.(applied);
+            await yieldToHost();
+            if (boundary === undefined || applied === count)
+                break;
+            after = boundary;
+        }
+        if (applied !== count)
+            imageInvalid('image applied row count does not match descriptor');
+        return applied;
     });
 }
 /**
@@ -284,6 +309,8 @@ export async function applyRowsSegment(db, schema, table, segment, options, encr
     let first = true;
     const blocks = segment.blocks.length > 0 ? segment.blocks : [[]];
     for (const block of blocks) {
+        if (options.isCurrent?.() === false)
+            break;
         // §5.11: decrypt this block's rows before opening the sync transaction
         // (WebCrypto is async; the SQLite transaction is not). Decrypt failure
         // aborts before any write in this block.
@@ -297,6 +324,8 @@ export async function applyRowsSegment(db, schema, table, segment, options, encr
                 : row.values;
             rows.push({ values, serverVersion: row.serverVersion });
         }
+        if (options.isCurrent?.() === false)
+            break;
         const clearThisBlock = first && options.clearFirst;
         first = false;
         if (!clearThisBlock && rows.length === 0)
@@ -308,8 +337,25 @@ export async function applyRowsSegment(db, schema, table, segment, options, encr
             for (const row of rows) {
                 upsertLocalRow(db, table, row.values, row.serverVersion);
                 applied += 1;
+                if (applied % 1024 === 0)
+                    options.onProgress?.(applied);
             }
         });
+        options.onProgress?.(applied);
+        await yieldToHost();
     }
+    options.onProgress?.(applied);
     return applied;
 }
+/** Let worker messages run only after the current SQLite transaction commits. */
+export function yieldToHost() {
+    return new Promise((resolve) => {
+        const channel = new MessageChannel();
+        channel.port1.onmessage = () => {
+            channel.port1.close();
+            channel.port2.close();
+            resolve();
+        };
+        channel.port2.postMessage(null);
+    });
+}

package/dist/bun-database.d.ts

@@ -21,7 +21,7 @@ export declare class BunClientDatabase implements ClientDatabase {
      * §5.3 image import: bun:sqlite attaches files, not buffers, so the
      * image lands in a private temp file for the duration of the ATTACH.
      */
-    withSqliteImage<T>(bytes: Uint8Array, alias: string, fn: () => T): T;
+    withSqliteImage<T>(bytes: Uint8Array, alias: string, fn: () => T | Promise<T>): Promise<T>;
     close(): void;
 }
 export declare function openBunDatabase(path?: string): ClientDatabase;

package/dist/bun-database.js

@@ -50,7 +50,7 @@ export class BunClientDatabase {
      * §5.3 image import: bun:sqlite attaches files, not buffers, so the
      * image lands in a private temp file for the duration of the ATTACH.
      */
-    withSqliteImage(bytes, alias, fn) {
+    async withSqliteImage(bytes, alias, fn) {
         assertImageAlias(alias);
         const dir = mkdtempSync(join(tmpdir(), 'syncular-image-'));
         const path = join(dir, 'segment.db');
@@ -58,7 +58,7 @@ export class BunClientDatabase {
             writeFileSync(path, bytes);
             this.db.run(`ATTACH DATABASE ? AS ${alias}`, [path]);
             try {
-                return fn();
+                return await fn();
             }
             finally {
                 this.db.run(`DETACH DATABASE ${alias}`);

package/dist/client.d.ts

@@ -1,3 +1,4 @@
+import { type SyncProgress, type SyncProgressListener } from './progress.js';
 /**
  * SyncClient implements the client side of SPEC.md §§3–8.
  *
@@ -90,8 +91,8 @@ export interface SyncClientLimits {
     readonly maxSnapshotPages?: number;
     /**
      * §4.2 accept bitmask; defaults to inline + external rows (0b0011)
-     * plus sqlite images (bit 2) when the database backend implements
-     * `withSqliteImage` and a segment downloader is configured (§5.3).
+     * plus SQLite images and signed URLs when supported. Both image chunks
+     * and rows blocks commit before yielding to local reads (§7.7).
      */
     readonly accept?: number;
     /**
@@ -294,6 +295,8 @@ export declare class SyncClient {
      */
     onInvalidate(listener: InvalidationListener): () => void;
     /** Subscribe to exact revisioned observer transactions (SPEC §7.5). */
+    onProgress(listener: SyncProgressListener): () => void;
+    progressSnapshot(): SyncProgress | undefined;
     onChange(listener: ClientChangeListener): () => void;
     /** Subscribe to host wake signals raised by startup and realtime. */
     onSyncNeeded(listener: (reason: 'startup' | 'hello' | WakeReason) => void): () => void;
@@ -380,8 +383,8 @@ export declare class SyncClient {
      * Set the live window units for a base (§4.8): a value-sharded family of
      * subscriptions, one per unit. Computes the diff against the registry —
      * added units get fresh subscriptions (image-lane bootstrap on the next
-     * sync); removed units are unsubscribed and evicted, fused in one local
-     * transaction (E1–E4). Idempotent: calling with the same units is a
+     * sync); removed units are unsubscribed with their first eviction chunk
+     * in one local transaction (E1–E4). Idempotent: calling with the same units is a
      * no-op. Re-entry (a unit removed then re-added) cancels any deferred
      * eviction and fresh-bootstraps.
      *