@docstack/pouchdb-adapter-googledrive 0.1.7 → 0.2.0

package/CHANGELOG.md

@@ -1,6 +1,66 @@
 # Changelog
 
-## 0.1.7 — unreleased
+## 0.1.8/9
+
+### Fixed
+
+- **A connectivity blip on the read that verifies a metadata commit no longer
+  deletes the committed change log.** The commit had landed; the failed read made
+  the writer treat it as unpublished and clean up a log the metadata referenced — a
+  dangling reference, and the document behind it gone despite a successful write. A
+  landed write is now reported as committed even when its verification read fails.
+  A hard kill at the same point survived only by accident: severed connectivity made
+  the cleanup delete fail too. A blip restores connectivity in time for the delete
+  to succeed, which is what made this the worse case.
+
+- **One missing change log no longer stops every future write.** A reference to a
+  change log that answers 404 - damage the verify-blip bug above left in real
+  folders - kept the log permanently unprocessed, and every write burned its whole
+  catch-up budget re-reading it, then threw a message blaming concurrent writers.
+  A folder in that state accepted no writes from any device, indefinitely, while
+  reporting active and converging on schedule. A 404 is now an answer: the log is
+  written off, the reference pruned and tombstoned on the first load that proves it
+  dead, and the catch-up error names unreadable logs instead of guessing at
+  writers. Transient failures keep their retry. (ADR-0008 / finding 0007)
+
+### Added
+
+- **Sync progress for consumers.** The changes feed now reports `pending` — the
+  CouchDB field PouchDB's progress pipeline already understands — so replication
+  `change` events and `activeTasks` can drive a progress bar as
+  `docs_written / (docs_written + pending)`. The count was already being computed
+  to cut each batch, then thrown away. It travels two ways, because PouchDB reads
+  them differently: per change through `onChange(change, pending, lastSeq)` — the
+  only form pouchdb-replication actually consumes for its events — and on the
+  complete response, CouchDB style. Verified end to end: a replication against a
+  memory target reports a decreasing `pending` on every change event. This is also
+  the only percentage source left:
+  `update_seq` ratios stopped meaning anything when sequence numbers went sparse.
+
+- **`onSyncProgress` adapter option** — reports change-log replay during `load()`
+  (`{ phase: 'replay', done, total }`), the cold-connect phase where a busy folder
+  (72 logs in one field report) previously looked frozen. Always reaches `total`
+  even when a straggler download fails; a throwing callback cannot fail the load.
+
+- `tests/dead_log.test.ts` — five cases around the write outage: writes continue
+  past a dangling reference, the folder heals on first load, the log's own writer
+  does not resurrect a pruned reference, transient failures keep their retry, and
+  the error names the unreadable log.
+
+- `tests/sync_progress.test.ts` — seven cases for the progress feedback, including
+  the end-to-end check that `pending` survives the trip through pouchdb-replication
+  onto its 'change' events.
+
+- `tests/interruption_recovery.test.ts` — six scenarios for a tab killed mid-write
+  (after the log upload; after the metadata commit), a connectivity blip on exactly
+  the verification read, a connectivity pause and recovery, a frozen client
+  resuming against a folder that moved on, and polling through an outage. The
+  invariant throughout: an acknowledged write survives any interruption that
+  follows; a rejected one may vanish or become visible later, but never takes an
+  acknowledged write with it.
+
+
+## 0.1.7 — 2026-08-27
 
 ### Fixed
 

package/README.md

@@ -1,7 +1,13 @@
+[![npm](https://img.shields.io/npm/v/@docstack/pouchdb-adapter-googledrive)](https://www.npmjs.com/package/@docstack/pouchdb-adapter-googledrive)
+[![Docs](https://img.shields.io/badge/docs-onyx.ac-blue)](https://onyx.ac/products/docstack/docs)
+[![License](https://img.shields.io/badge/license-CC--BY--SA--4.0-lightgrey)](#license)
+
 # PouchDB Adapter for Google Drive
 
 A persistent, serverless PouchDB adapter that uses Google Drive as a backend storage. Designed for high concurrency, large datasets (via lazy loading), and offline resilience.
 
+Part of [DocStack](https://onyx.ac/products/docstack): it is what makes a DocStack stack sync to the user's *own* Drive, so an application gets multi-device sync and real backup with no server, no storage bill and no custody of anyone's data. It works with plain PouchDB just as well.
+
 ## Features
 
 - **🚀 Append-Only Log**: Uses an efficient append-only log pattern for fast, conflict-free writes.
@@ -80,6 +86,44 @@ A tick costs one `files.list`, whatever has changed; only a tick that sees a new
 `md5Checksum` (or, failing that, a new `modifiedTime`) goes on to fetch anything.
 `db.close()` and `db.destroy()` stop it.
 
+### Sync progress
+
+Two signals, one per phase where a UI would otherwise show nothing:
+
+**Replication progress** rides PouchDB's standard pipeline. The adapter's changes
+feed reports `pending` (the CouchDB field), so replication `change` events and
+`activeTasks` carry it without any adapter-specific wiring:
+
+```typescript
+const rep = PouchDB.replicate(remote, local);
+rep.on('change', info => {
+  const done = info.docs_written;
+  const pct = Math.round(done / (done + info.pending) * 100);
+  // Freeze the denominator at cycle start, or clamp the bar monotone —
+  // other devices keep writing, so `pending` can grow mid-cycle.
+});
+```
+
+Do not derive progress from `update_seq` arithmetic: sequence numbers are sparse
+(they carry a writer slot in the low digits), so ratios of them mean nothing.
+
+**Connect progress** covers the cold load, where a busy folder replays dozens of
+change logs before the database is usable:
+
+```typescript
+GoogleDriveAdapter({
+  accessToken: '...',
+  folderId: 'my-folder-id',
+  onSyncProgress: ({ phase, done, total }) => {
+    // phase 'replay': applying change logs, done of total for this load.
+  }
+});
+```
+
+The callback is fire-and-forget: it always reaches `total` even when a log download
+fails (that log is retried on a later load), and an exception it throws cannot fail
+the load.
+
 ## Concurrent writers
 
 Several clients may share one folder. What that costs, and what it does not:
@@ -118,6 +162,12 @@ The adapter implements a **"Remote-First"** architecture:
 - **Caching**: Changes are indexed locally but bodies are cached in an LRU cache.
 - **Resilience**: Writes use optimistic locking to handle multi-client concurrency safer.
 
+## Documentation
+
+- [Sync to Google Drive](https://onyx.ac/products/docstack/docs/guides/google-drive/) — registering the adapter, OAuth scope, one folder per database, restoring on a new device
+- [Sync to a remote](https://onyx.ac/products/docstack/docs/guides/sync/) — the replication lifecycle this adapter plugs into
+- [DocStack](https://onyx.ac/products/docstack) — the document data layer it was built for
+
 ## License
 
 CC-BY-SA-4.0

package/lib/adapter.js

@@ -569,7 +569,8 @@ function GoogleDriveAdapter(PouchDB) {
                     if (batch.length === 0)
                         return;
                     const emit = (bodies) => {
-                        for (const { id, entry } of batch) {
+                        for (let i = 0; i < batch.length; i++) {
+                            const { id, entry } = batch[i];
                             const change = {
                                 id,
                                 seq: entry.seq,
@@ -579,9 +580,11 @@ function GoogleDriveAdapter(PouchDB) {
                             // makes a filtered replication drop the deletion.
                             if (bodies)
                                 change.doc = bodies[id];
-                            if (opts.onChange)
-                                opts.onChange(change);
                             lastSeq = Math.max(lastSeq, entry.seq);
+                            // Same progress contract as the initial pass; for a live
+                            // batch the horizon is the batch itself.
+                            if (opts.onChange)
+                                opts.onChange(change, batch.length - (i + 1), lastSeq);
                         }
                     };
                     if (opts.include_docs) {
@@ -668,9 +671,10 @@ function GoogleDriveAdapter(PouchDB) {
                 const bodies = opts.include_docs
                     ? await loadChangeBodies(batch)
                     : null;
-                for (const { id, entry } of batch) {
+                for (let i = 0; i < batch.length; i++) {
                     if (complete)
                         break;
+                    const { id, entry } = batch[i];
                     const change = {
                         id: id,
                         seq: entry.seq,
@@ -678,17 +682,32 @@ function GoogleDriveAdapter(PouchDB) {
                     };
                     if (bodies)
                         change.doc = bodies[id];
+                    lastSeq = Math.max(lastSeq, entry.seq);
+                    // The second and third arguments are how progress actually
+                    // reaches a consumer: pouchdb-replication reads `pending` from
+                    // onChange's arguments (never from the complete response) and
+                    // surfaces it on its own 'change' events. `pending` here is
+                    // CouchDB's meaning - eligible changes remaining AFTER this row,
+                    // both the rest of this batch and everything beyond the limit.
                     if (opts.onChange)
-                        opts.onChange(change);
+                        opts.onChange(change, pending.length - (i + 1), lastSeq);
                     if (returnDocs)
                         results.push(change);
-                    lastSeq = Math.max(lastSeq, entry.seq);
                 }
                 // ✅ Call opts.complete() ONLY for non-live modes
                 // PouchDB replication will infinite-loop reconnect if we call complete() on a live feed
                 if (opts.complete && !complete && !opts.live) {
-                    log('_changes calling complete callback with', { results_count: results.length, last_seq: lastSeq });
-                    opts.complete(null, { results, last_seq: lastSeq });
+                    // `pending` is what CouchDB reports and what PouchDB's whole
+                    // progress pipeline runs on: replication surfaces it on 'change'
+                    // events and in activeTasks' total_items. The number was already
+                    // computed to cut the batch; consumers turn it into a progress
+                    // bar as docs_written / (docs_written + pending). It is also the
+                    // only percentage source left standing - update_seq arithmetic
+                    // stopped meaning anything when sequence numbers went sparse
+                    // (ADR-0003).
+                    const remaining = Math.max(0, pending.length - batch.length);
+                    log('_changes calling complete callback with', { results_count: results.length, last_seq: lastSeq, pending: remaining });
+                    opts.complete(null, { results, last_seq: lastSeq, pending: remaining });
                 }
             }
             // Start processing (async) which will wait for load() via getIndexKeys()

package/lib/drive.d.ts

@@ -56,6 +56,13 @@ export declare class DriveHandler {
      *  between. Anything still in here but missing from the remote changeLogIds was
      *  dropped that way and gets put back - see reconcileOwnLogs(). */
     private ownLogIds;
+    /** Change logs proven gone: the metadata references them and Drive answers 404.
+     *  A dangling reference is damage some earlier defect already did (the 0.1.8
+     *  verify-blip bug deleted committed logs); what it must not do is compound -
+     *  a log that can only 404 for ever kept hasUnprocessedLogs() true for ever,
+     *  and every write spent its whole catch-up budget re-reading it and threw.
+     *  One lost change became a permanent, silent write outage. See finding 0007. */
+    private deadLogIds;
     /** This writer's reservation in the low digits of every sequence number it mints.
      *  Derived from writerId; changes only when writerId is re-rolled off a
      *  contested slot. */
@@ -80,6 +87,8 @@ export declare class DriveHandler {
     private pendingDownloads;
     private pendingFinds;
     private log;
+    /** Feed the consumer's progress callback, if any. Its failures are its own. */
+    private reportProgress;
     constructor(options: GoogleDriveAdapterOptions, dbName: string);
     get seq(): number;
     /** Load the database (Index Only) */
@@ -144,6 +153,13 @@ export declare class DriveHandler {
     /** Put back any such log. Returns `latest` by identity when there is nothing to
      *  repair, so callers can tell the two cases apart. */
     private reconcileOwnLogs;
+    /** Drop references this client has proven dead from `changeLogIds`, moving them
+     *  into `retiredLogIds` - the tombstone that stops the log's own writer from
+     *  restoring them (its reconcile respects retirement) and any client from
+     *  re-adopting them. Without the tombstone, prune-and-restore would ping-pong
+     *  between the pruning reader and the writer that still remembers the log.
+     *  Returns `latest` by identity when there is nothing to prune. */
+    private pruneDeadLogs;
     /**
      * Read-modify-write `_meta.json`.
      *

package/lib/drive.js

@@ -74,6 +74,18 @@ class DriveHandler {
     log(...args) {
         console.log(`[googledrive-drive] [${this.meta.dbName}]`, ...args);
     }
+    /** Feed the consumer's progress callback, if any. Its failures are its own. */
+    reportProgress(phase, done, total) {
+        const cb = this.options.onSyncProgress;
+        if (!cb)
+            return;
+        try {
+            cb({ phase, done, total });
+        }
+        catch (e) {
+            this.log('onSyncProgress callback threw', e);
+        }
+    }
     constructor(options, dbName) {
         this.folderId = null;
         this.meta = {
@@ -100,6 +112,13 @@ class DriveHandler {
          *  between. Anything still in here but missing from the remote changeLogIds was
          *  dropped that way and gets put back - see reconcileOwnLogs(). */
         this.ownLogIds = new Set();
+        /** Change logs proven gone: the metadata references them and Drive answers 404.
+         *  A dangling reference is damage some earlier defect already did (the 0.1.8
+         *  verify-blip bug deleted committed logs); what it must not do is compound -
+         *  a log that can only 404 for ever kept hasUnprocessedLogs() true for ever,
+         *  and every write spent its whole catch-up budget re-reading it and threw.
+         *  One lost change became a permanent, silent write outage. See finding 0007. */
+        this.deadLogIds = new Set();
         /** This writer's reservation in the low digits of every sequence number it mints.
          *  Derived from writerId; changes only when writerId is re-rolled off a
          *  contested slot. */
@@ -185,7 +204,8 @@ class DriveHandler {
                 // compaction retiring it was dropped by another writer's
                 // read-modify-write. Put it back before replaying, so this load sees
                 // its own writes - and so the next reader does too.
-                if (this.hasOrphanedOwnLogs(this.meta)) {
+                if (this.hasOrphanedOwnLogs(this.meta) ||
+                    this.meta.changeLogIds.some(id => this.deadLogIds.has(id))) {
                     await this.commitMeta((latest, repaired) => repaired ? latest : null);
                 }
                 if (this.meta.snapshotIndexId !== this.currentSnapshotIndexId) {
@@ -252,6 +272,7 @@ class DriveHandler {
                 const pendingLogs = [...this.meta.changeLogIds, ...discovered]
                     .filter(id => !this.processedLogIds.has(id) && !retired.has(id));
                 if (pendingLogs.length > 0) {
+                    this.reportProgress('replay', 0, pendingLogs.length);
                     this.log(`Downloading ${pendingLogs.length} change logs, ${LOG_DOWNLOAD_CONCURRENCY} at a time`);
                     // Bounded, not unbounded: a cold boot can hold dozens of pending
                     // logs, and firing them all at once is exactly the burst Drive's
@@ -265,7 +286,23 @@ class DriveHandler {
                             return { id, changes };
                         }
                         catch (e) {
-                            this.log(`Failed to download change log ${id}`, e);
+                            // A 404 is an answer, not an outage: the file is gone and
+                            // no retry will bring it back. Recording it as processed
+                            // costs the changes it held - already lost - and keeps
+                            // that loss from also stopping every future write. Any
+                            // other failure stays retryable, as it should: retrying
+                            // is right when the file is there and the network was not.
+                            if (e?.status === 404) {
+                                this.log(`Change log ${id} is gone for good, writing it off`, e?.message);
+                                this.deadLogIds.add(id);
+                                this.processedLogIds.add(id);
+                                // Not ours to defend any more either - reconcile
+                                // would resurrect the reference for ever.
+                                this.ownLogIds.delete(id);
+                            }
+                            else {
+                                this.log(`Failed to download change log ${id}`, e);
+                            }
                             return { id, changes: null };
                         }
                     });
@@ -282,9 +319,15 @@ class DriveHandler {
                         return a.id < b.id ? -1 : a.id > b.id ? 1 : 0;
                     });
                     const foundNew = {};
+                    let replayed = 0;
                     for (const { id, changes } of logResults) {
+                        // A failed download still counts toward the cycle - the bar
+                        // must reach total even when a straggler is left for the next
+                        // load, or it sits at 71/72 looking stuck.
+                        replayed++;
                         if (!changes) {
                             this.log(`Skipping failed log ${id}`);
+                            this.reportProgress('replay', replayed, logResults.length);
                             continue;
                         }
                         let changesArray = Array.isArray(changes) ? changes : [changes];
@@ -302,6 +345,7 @@ class DriveHandler {
                             };
                         }
                         this.processedLogIds.add(id);
+                        this.reportProgress('replay', replayed, logResults.length);
                     }
                     if (Object.keys(foundNew).length > 0) {
                         this.log('Load complete, notifying of changes', Object.keys(foundNew).length);
@@ -324,6 +368,14 @@ class DriveHandler {
                         return { ...latest, changeLogIds: [...latest.changeLogIds, ...missing] };
                     });
                 }
+                // A reference proven dead during THIS replay (the 404s land in the
+                // download loop above, after the pre-replay repair has run) is
+                // pruned now, so a damaged folder heals on the first load that
+                // notices - by any client, reader or writer - rather than carrying
+                // a permanent 404 and a permanent log line until someone writes.
+                if (this.meta.changeLogIds.some(id => this.deadLogIds.has(id))) {
+                    await this.commitMeta((latest, repaired) => repaired ? latest : null);
+                }
                 // 2b. Load Local Documents Store (Pinned in meta)
                 if (this.meta.localDocsId) {
                     this.log('Loading local docs store', this.meta.localDocsId);
@@ -742,7 +794,15 @@ class DriveHandler {
             // itself never does.)
             if (current && this.hasUnprocessedLogs(current.meta)) {
                 if (++catchUps > META_COMMIT_RETRIES) {
-                    throw new Error('Could not catch up with concurrent writers');
+                    // Naming what actually blocked us. The old message blamed
+                    // concurrent writers unconditionally, and sent a real
+                    // investigation looking for a second writer when the cause was
+                    // a change log that could not be read.
+                    const unreadable = current.meta.changeLogIds
+                        .filter(id => !this.processedLogIds.has(id));
+                    throw new Error(unreadable.length > 0
+                        ? `Could not catch up: change log(s) ${unreadable.join(', ')} could not be read after ${META_COMMIT_RETRIES} attempts`
+                        : 'Could not catch up with concurrent writers');
                 }
                 await this.load();
                 // Only the low-level appendChange() callers are checked here.
@@ -1227,6 +1287,26 @@ class DriveHandler {
         this.log('Restoring change logs dropped by another writer', missing);
         return { ...latest, changeLogIds: [...latest.changeLogIds, ...missing] };
     }
+    /** Drop references this client has proven dead from `changeLogIds`, moving them
+     *  into `retiredLogIds` - the tombstone that stops the log's own writer from
+     *  restoring them (its reconcile respects retirement) and any client from
+     *  re-adopting them. Without the tombstone, prune-and-restore would ping-pong
+     *  between the pruning reader and the writer that still remembers the log.
+     *  Returns `latest` by identity when there is nothing to prune. */
+    pruneDeadLogs(latest) {
+        if (this.deadLogIds.size === 0)
+            return latest;
+        const dead = latest.changeLogIds.filter(id => this.deadLogIds.has(id));
+        if (dead.length === 0)
+            return latest;
+        this.log('Pruning dead change-log references', dead);
+        const retired = [...new Set([...(latest.retiredLogIds || []), ...dead])];
+        return {
+            ...latest,
+            changeLogIds: latest.changeLogIds.filter(id => !this.deadLogIds.has(id)),
+            retiredLogIds: retired.slice(-RETIRED_LOG_HISTORY)
+        };
+    }
     /**
      * Read-modify-write `_meta.json`.
      *
@@ -1242,7 +1322,7 @@ class DriveHandler {
                 const current = await this.readRemoteMeta();
                 if (!current)
                     throw new Error('Meta missing');
-                const reconciled = this.reconcileOwnLogs(current.meta);
+                const reconciled = this.pruneDeadLogs(this.reconcileOwnLogs(current.meta));
                 const next = modify(reconciled, reconciled !== current.meta);
                 if (!next)
                     return null;
@@ -1260,7 +1340,21 @@ class DriveHandler {
                     this.meta = next;
                     return next;
                 }
-                const after = await this.readMetaBody(current.pointer.fileId);
+                // From here on the write HAS landed. Whatever the verify read does,
+                // this attempt must never surface as "not committed" - a caller told
+                // that would clean up a change log the metadata now references,
+                // leaving a dangling reference and losing the document behind it. A
+                // read failure is a read failure, not a failed commit: report the
+                // commit as landed, unverified.
+                let after;
+                try {
+                    after = await this.readMetaBody(current.pointer.fileId);
+                }
+                catch (e) {
+                    this.log('Verify read failed after a landed write, reporting the commit as-is', e);
+                    this.meta = next;
+                    return next;
+                }
                 if (after && opts.verify(after)) {
                     // Adopt the remote view rather than our own - it carries whatever
                     // else landed alongside us.

package/lib/types.d.ts

@@ -17,6 +17,19 @@ export interface GoogleDriveAdapterOptions extends DriveClientOptions {
     cacheSize?: number;
     /** Enable debug logging */
     debug?: boolean;
+    /**
+     * Called as load() replays change logs, the phase where a cold connect to a
+     * busy folder can otherwise look frozen (a real-world boot has been seen with
+     * 72 logs pending). `done` counts logs applied so far out of `total` for THIS
+     * load; a later load that finds new logs starts a fresh cycle at 0. Errors
+     * thrown by the callback are swallowed - progress reporting must never be able
+     * to fail a load.
+     */
+    onSyncProgress?: (progress: {
+        phase: 'replay';
+        done: number;
+        total: number;
+    }) => void;
     /** Enable test mode (emulates Google Drive API) */
     testMode?: boolean;
     /** Test server URL (defaults to http://localhost:3000) */

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@docstack/pouchdb-adapter-googledrive",
-  "version": "0.1.7",
+  "version": "0.2.0",
   "description": "PouchDB adapter for Google Drive",
   "main": "lib/index.js",
   "types": "lib/index.d.ts",
@@ -26,7 +26,7 @@
   "bugs": {
     "url": "https://github.com/onyx-ac/docstack-pouchdb-adapter-gdrive/issues"
   },
-  "homepage": "https://onyx.ac/docstack",
+  "homepage": "https://onyx.ac/products/docstack",
   "engines": {
     "node": ">=18"
   },