@docstack/pouchdb-adapter-googledrive 0.0.3 → 0.0.5

package/DOCUMENTATION.md

@@ -0,0 +1,54 @@
+# Architecture & Design Documentation
+
+## 1. Core Principles
+The `pouchdb-adapter-googledrive` implementation is built on three core pillars to ensure data integrity and performance on a file-based remote storage system.
+
+### A. Append-Only Log (Storage)
+Instead of modifying a single database file (which is prone to conflicts), we use an **Append-Only** strategy.
+- **Changes**: Every write operation (or batch of writes) creates a **new, immutable file** (e.g., `changes-{seq}-{uuid}.ndjson`).
+- **Snapshots**: Periodically, the log is compacted into a `snapshot` file.
+- **Benefit**: Historical data is preserved until compaction, and file-write conflicts are minimized.
+
+### B. Optimistic Concurrency Control (OCC)
+To prevent race conditions (two clients writing simultaneously), we use **ETag-based locking** on a single entry point: `_meta.json`.
+- **The Lock**: `_meta.json` holds the current Sequence Number and the list of active log files.
+- **The Protocol**:
+    1. Reader fetches `_meta.json` and its `ETag`.
+    2. Writer prepares a new change file and uploads it (orphaned initially).
+    3. Writer attempts to update `_meta.json` with the new file reference, sending `If-Match: <Old-ETag>`.
+    4. **Success**: The change is now officially part of the DB.
+    5. **Failure (412/409)**: Another client updated the DB. The writer deletes its orphaned file, pulls the new state, and retries the logical operation.
+
+### C. Remote-First "Lazy" Loading (Memory Optimization)
+To support large databases without exhausting client memory, we separate **Metadata** from **Content**.
+
+#### Storage Structure
+- `_meta.json`: Root pointer. Small.
+- `snapshot-index.json`: A map of `{ docId: { rev, filePointer } }`. Medium size (~100 bytes/doc). Loaded at startup.
+- `snapshot-data.json`: The actual document bodies. Large. **Never fully loaded.**
+- `changes-*.ndjson`: Recent updates.
+
+#### Client Startup Sequence
+1.  **Fetch Meta**: Download `_meta.json` and get the `snapshotIndexId`.
+2.  **Fetch Index**: Download `snapshot-index.json`. This builds the "Revision Tree" in memory.
+3.  **Replay Logs**: Download and parse only the small `changes-*.ndjson` files created since the snapshot to update the in-memory Index.
+4.  **Ready**: The client is now ready to query keys. No document content has been downloaded yet.
+
+#### On-Demand Usage
+- **`db.get(id)`**: 
+    1. Look up `id` in the **Memory Index** to find the `filePointer`.
+    2. Check **LRU Cache**.
+    3. If missing, fetch the specific file/range from Google Drive.
+- **`db.allDocs({ keys: [...] })`**: Efficiently looks up pointers and fetches only requested docs.
+
+## 2. Technical Patterns
+
+### Atomic Compaction
+Compaction is a critical maintenance task that merges the `snapshot-data` with recent `changes` to create a new baseline.
+- **Safe**: Limits memory usage by streaming/batching.
+- **Atomic**: Uploads the new snapshot as a new file. Swaps the pointer in `_meta.json` using OCC.
+- **Zero-Downtime**: Clients can continue reading/writing to the old logs while compaction runs. Writes that happen *during* compaction are detected via the ETag check, causing the compaction to abort/retry safeley.
+
+### Conflict Handling
+- **PouchDB Level**: Standard CouchDB revision conflicts (409) are preserved. A "winner" is chosen deterministically, but conflicting revisions are kept in the tree (requires `snapshot-index` to store the full revision tree, not just the winner).
+- **Adapter Level**: Drive API 409s handling (retry logic) ensures the transport layer is reliable.

package/README.md

@@ -8,6 +8,11 @@ A persistent, serverless PouchDB adapter that uses Google Drive as a backend sto
 - **⚡ Lazy Loading**: Optimizes memory and bandwidth by loading only the **Index** into memory. Document bodies are fetched on-demand.
 - **🛡️ Optimistic Concurrency Control**: Uses ETag-based locking on metadata to prevent race conditions.
 - **📦 Auto-Compaction**: Automatically merges logs for performance.
+- **🌍 Universal**: Works natively in Node.js 18+, Browsers, and Edge environments (no `googleapis` dependency).
+
+## Requirements
+
+- **Node.js 18+** (for global `fetch` support) or a modern browser.
 
 ## Installation
 
@@ -17,32 +22,23 @@ npm install @docstack/pouchdb-adapter-googledrive
 
 ## Usage
 
-The adapter is initialized as a plugin with your Google Drive configuration.
+The adapter is initialized as a plugin with your Google Drive access token.
 
 ```typescript
 import PouchDB from 'pouchdb-core';
 import GoogleDriveAdapter from '@docstack/pouchdb-adapter-googledrive';
-import { google } from 'googleapis';
-
-// 1. Setup Google Drive Client
-const oauth2Client = new google.auth.OAuth2(CLIENT_ID, SECRET, REDIRECT);
-oauth2Client.setCredentials({ access_token: '...' });
-const drive = google.drive({ version: 'v3', auth: oauth2Client });
 
-// 2. Initialize the Adapter Plugin with Config
+// 1. Initialize the Adapter Plugin Factory
 const adapterPlugin = GoogleDriveAdapter({
-  drive: drive,
-  folderName: 'my-app-db-folder', // Root folder
-  pollingIntervalMs: 2000
+  accessToken: 'YOUR_GOOGLE_ACCESS_TOKEN',
+  folderName: 'my-app-db-folder', // Root folder in Drive
+  pollingIntervalMs: 2000         // Optional: check for remote changes
 });
 
-// 3. Register Plugin
+// 2. Register Plugin
 PouchDB.plugin(adapterPlugin);
-// Also needs replication plugin if using replicate()
-// PouchDB.plugin(require('pouchdb-replication'));
 
-// 4. Create Database
-// No need to pass 'drive' here anymore!
+// 3. Create Database
 const db = new PouchDB('user_db', {
   adapter: 'googledrive'
 });
@@ -50,6 +46,20 @@ const db = new PouchDB('user_db', {
 await db.post({ title: 'Hello World' });
 ```
 
+### Dynamic Tokens
+
+If your token expires, you can provide an async function that returns a valid token:
+
+```typescript
+const adapterPlugin = GoogleDriveAdapter({
+  accessToken: async () => {
+    const session = await getMySession();
+    return session.accessToken;
+  },
+  folderName: 'my-app-db'
+});
+```
+
 ## Architecture
 
 The adapter implements a **"Remote-First"** architecture:

package/lib/client.d.ts

@@ -0,0 +1,29 @@
+export interface DriveFile {
+    id: string;
+    name: string;
+    mimeType: string;
+    parents?: string[];
+    etag?: string;
+}
+export interface DriveClientOptions {
+    accessToken: string | (() => Promise<string>);
+}
+export declare class GoogleDriveClient {
+    private options;
+    constructor(options: DriveClientOptions);
+    private getToken;
+    private fetch;
+    listFiles(q: string): Promise<DriveFile[]>;
+    getFile(fileId: string): Promise<any>;
+    getFileMetadata(fileId: string): Promise<DriveFile>;
+    createFile(name: string, parents: string[] | undefined, mimeType: string, content: string): Promise<{
+        id: string;
+        etag: string;
+    }>;
+    updateFile(fileId: string, content: string, expectedEtag?: string): Promise<{
+        id: string;
+        etag: string;
+    }>;
+    deleteFile(fileId: string): Promise<void>;
+    private buildMultipart;
+}

package/lib/client.js

@@ -0,0 +1,129 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.GoogleDriveClient = void 0;
+const BASE_URL = 'https://www.googleapis.com/drive/v3/files';
+const UPLOAD_URL = 'https://www.googleapis.com/upload/drive/v3/files';
+class GoogleDriveClient {
+    constructor(options) {
+        this.options = options;
+    }
+    async getToken() {
+        if (typeof this.options.accessToken === 'function') {
+            return await this.options.accessToken();
+        }
+        return this.options.accessToken;
+    }
+    async fetch(url, init) {
+        const token = await this.getToken();
+        const headers = new Headers(init.headers);
+        headers.set('Authorization', `Bearer ${token}`);
+        const res = await fetch(url, { ...init, headers });
+        const method = init.method || 'GET';
+        if (!res.ok) {
+            // Basic error handling
+            const text = await res.text();
+            let errorMsg = `Drive API Error: ${res.status} ${res.statusText} (${method} ${url})`;
+            try {
+                const json = JSON.parse(text);
+                if (json.error && json.error.message) {
+                    errorMsg += ` - ${json.error.message}`;
+                }
+            }
+            catch { }
+            const err = new Error(errorMsg);
+            err.status = res.status;
+            throw err;
+        }
+        return res;
+    }
+    async listFiles(q) {
+        const params = new URLSearchParams({
+            q,
+            fields: 'files(id, name, mimeType, parents, etag)',
+            spaces: 'drive',
+            pageSize: '1000' // Ensure we get enough
+        });
+        const res = await this.fetch(`${BASE_URL}?${params.toString()}`, { method: 'GET' });
+        const data = await res.json();
+        return data.files || [];
+    }
+    async getFile(fileId) {
+        // Try getting media
+        try {
+            const params = new URLSearchParams({ alt: 'media' });
+            const res = await this.fetch(`${BASE_URL}/${fileId}?${params.toString()}`, { method: 'GET' });
+            // Standard fetch handles JSON/Text transparency? 
+            // We expect JSON mostly, but sometimes we might want text.
+            // PouchDB adapter flow: downloadJson, downloadNdjson
+            // Let's rely on content-type or caller expectation?
+            // The usage in `drive.ts` expects parsed JSON/NDJSON lines.
+            // Let's return the raw Text or JSON based on Content-Type?
+            const contentType = res.headers.get('content-type');
+            if (contentType && contentType.includes('application/json')) {
+                return await res.json();
+            }
+            return await res.text();
+        }
+        catch (e) {
+            throw e;
+        }
+    }
+    // Single metadata get (for etag check)
+    async getFileMetadata(fileId) {
+        const params = new URLSearchParams({ fields: 'id, name, mimeType, parents, etag' });
+        const res = await this.fetch(`${BASE_URL}/${fileId}?${params.toString()}`, { method: 'GET' });
+        return await res.json();
+    }
+    async createFile(name, parents, mimeType, content) {
+        const metadata = {
+            name,
+            mimeType,
+            parents
+        };
+        // Folders or empty content can use simple metadata-only POST
+        if (!content && mimeType === 'application/vnd.google-apps.folder') {
+            const res = await this.fetch(`${BASE_URL}?fields=id,etag`, {
+                method: 'POST',
+                headers: { 'Content-Type': 'application/json' },
+                body: JSON.stringify(metadata)
+            });
+            return await res.json();
+        }
+        const multipartBody = this.buildMultipart(metadata, content, mimeType);
+        const res = await this.fetch(`${UPLOAD_URL}?uploadType=multipart&fields=id,etag`, {
+            method: 'POST',
+            headers: {
+                'Content-Type': `multipart/related; boundary=${multipartBody.boundary}`
+            },
+            body: multipartBody.body
+        });
+        return await res.json();
+    }
+    async updateFile(fileId, content, expectedEtag) {
+        // Update content (media) usually, but sometimes meta?
+        // In our usage (saveMeta), we update body.
+        const res = await this.fetch(`${UPLOAD_URL}/${fileId}?uploadType=media&fields=id,etag`, {
+            method: 'PATCH',
+            headers: expectedEtag ? { 'If-Match': expectedEtag, 'Content-Type': 'application/json' } : { 'Content-Type': 'application/json' },
+            body: content
+        });
+        return await res.json();
+    }
+    async deleteFile(fileId) {
+        await this.fetch(`${BASE_URL}/${fileId}`, { method: 'DELETE' });
+    }
+    buildMultipart(metadata, content, contentType) {
+        const boundary = '-------' + Math.random().toString(36).substring(2);
+        const delimiter = `\r\n--${boundary}\r\n`;
+        const closeDelimiter = `\r\n--${boundary}--`;
+        const body = delimiter +
+            'Content-Type: application/json\r\n\r\n' +
+            JSON.stringify(metadata) +
+            delimiter +
+            `Content-Type: ${contentType}\r\n\r\n` +
+            content +
+            closeDelimiter;
+        return { body, boundary };
+    }
+}
+exports.GoogleDriveClient = GoogleDriveClient;

package/lib/drive.d.ts

@@ -10,7 +10,8 @@ import { GoogleDriveAdapterOptions, ChangeEntry, IndexEntry } from './types';
  *   └── changes-*.ndjson     # Append logs
  */
 export declare class DriveHandler {
-    private drive;
+    private client;
+    private options;
     private folderId;
     private folderName;
     private parents;
@@ -64,6 +65,7 @@ export declare class DriveHandler {
     private notifyListeners;
     onChange(cb: any): void;
     stopPolling(): void;
+    private escapeQuery;
     deleteFolder(): Promise<void>;
     getNextSeq(): number;
 }

package/lib/drive.js

@@ -2,6 +2,7 @@
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.DriveHandler = void 0;
 const cache_1 = require("./cache");
+const client_1 = require("./client");
 const DEFAULT_COMPACTION_THRESHOLD = 100; // entries
 const DEFAULT_SIZE_THRESHOLD = 1024 * 1024; // 1MB
 const DEFAULT_CACHE_SIZE = 1000; // Number of docs
@@ -32,7 +33,8 @@ class DriveHandler {
         this.currentLogSizeEstimate = 0;
         this.listeners = [];
         this.pollingInterval = null;
-        this.drive = options.drive;
+        this.client = new client_1.GoogleDriveClient(options);
+        this.options = options;
         this.folderId = options.folderId || null;
         this.folderName = options.folderName || dbName;
         this.parents = options.parents || [];
@@ -40,9 +42,7 @@ class DriveHandler {
         this.compactionSizeThreshold = options.compactionSizeThreshold || DEFAULT_SIZE_THRESHOLD;
         this.meta.dbName = dbName;
         this.docCache = new cache_1.LRUCache(options.cacheSize || DEFAULT_CACHE_SIZE);
-        if (options.pollingIntervalMs) {
-            this.startPolling(options.pollingIntervalMs);
-        }
+        // Polling will be started in load() after folderId is resolved
     }
     // Public getter for Sequence (used by adapter)
     get seq() {
@@ -108,6 +108,10 @@ class DriveHandler {
                 }
             }
         }
+        // 3. Start Polling (if enabled)
+        if (this.options.pollingIntervalMs) {
+            this.startPolling(this.options.pollingIntervalMs);
+        }
     }
     // Migration helper
     filesFromLegacySnapshot(snapshot) {
@@ -267,7 +271,7 @@ class DriveHandler {
                 return await this.tryAppendChanges(changes);
             }
             catch (err) {
-                if (err.code === 412 || err.code === 409) {
+                if (err.status === 412 || err.status === 409) {
                     // Reload and RETRY
                     await this.load();
                     // Check conflicts against Index (Metadata sufficient)
@@ -359,16 +363,8 @@ class DriveHandler {
         });
         // 2. Upload Data File
         const dataContent = JSON.stringify(snapshotData);
-        const dataRes = await this.drive.files.create({
-            requestBody: {
-                name: `snapshot-data-${Date.now()}.json`,
-                parents: [this.folderId],
-                mimeType: 'application/json'
-            },
-            media: { mimeType: 'application/json', body: dataContent },
-            fields: 'id'
-        });
-        const dataFileId = dataRes.data.id;
+        const dataRes = await this.client.createFile(`snapshot-data-${Date.now()}.json`, [this.folderId], 'application/json', dataContent);
+        const dataFileId = dataRes.id;
         // 3. Create Index pointing to this Data File
         const newIndexEntries = {};
         for (const id of Object.keys(snapshotData.docs)) {
@@ -384,16 +380,8 @@ class DriveHandler {
             createdAt: Date.now()
         };
         const indexContent = JSON.stringify(snapshotIndex);
-        const indexRes = await this.drive.files.create({
-            requestBody: {
-                name: `snapshot-index-${Date.now()}.json`,
-                parents: [this.folderId],
-                mimeType: 'application/json'
-            },
-            media: { mimeType: 'application/json', body: indexContent },
-            fields: 'id'
-        });
-        const newIndexId = indexRes.data.id;
+        const indexRes = await this.client.createFile(`snapshot-index-${Date.now()}.json`, [this.folderId], 'application/json', indexContent);
+        const newIndexId = indexRes.id;
         // 4. Update Meta
         await this.atomicUpdateMeta((latest) => {
             const remainingLogs = latest.changeLogIds.filter(id => !oldLogIds.includes(id));
@@ -424,7 +412,7 @@ class DriveHandler {
                 return;
             }
             catch (err) {
-                if (err.code === 412 || err.code === 409) {
+                if (err.status === 412 || err.status === 409) {
                     attempt++;
                     await new Promise(r => setTimeout(r, Math.random() * 500 + 100));
                     continue;
@@ -435,45 +423,38 @@ class DriveHandler {
     }
     // Reused helpers
     async findOrCreateFolder() {
-        const q = `name = '${this.folderName}' and mimeType = 'application/vnd.google-apps.folder' and trashed = false`;
-        const res = await this.drive.files.list({ q, spaces: 'drive', fields: 'files(id)' });
-        if (res.data.files && res.data.files.length > 0)
-            return res.data.files[0].id;
-        const createRes = await this.drive.files.create({
-            requestBody: { name: this.folderName, mimeType: 'application/vnd.google-apps.folder', parents: this.parents.length ? this.parents : undefined },
-            fields: 'id'
-        });
-        return createRes.data.id;
+        const safeName = this.escapeQuery(this.folderName);
+        const q = `name = '${safeName}' and mimeType = 'application/vnd.google-apps.folder' and trashed = false`;
+        const files = await this.client.listFiles(q);
+        if (files.length > 0)
+            return files[0].id;
+        const createRes = await this.client.createFile(this.folderName, this.parents.length ? this.parents : undefined, 'application/vnd.google-apps.folder', '');
+        return createRes.id;
     }
     async findFile(name) {
-        const q = `name = '${name}' and '${this.folderId}' in parents and trashed = false`;
-        const res = await this.drive.files.list({ q, spaces: 'drive', fields: 'files(id, etag)' });
-        if (res.data.files && res.data.files.length > 0)
-            return { id: res.data.files[0].id, etag: res.data.files[0].etag };
+        if (!this.folderId)
+            return null;
+        const safeName = this.escapeQuery(name);
+        const q = `name = '${safeName}' and '${this.folderId}' in parents and trashed = false`;
+        const files = await this.client.listFiles(q);
+        if (files.length > 0)
+            return { id: files[0].id, etag: files[0].etag || '' };
         return null;
     }
     async downloadJson(fileId) {
-        const res = await this.drive.files.get({ fileId, alt: 'media' });
-        return res.data;
+        return await this.client.getFile(fileId);
     }
     async downloadFileAny(fileId) {
-        const res = await this.drive.files.get({ fileId, alt: 'media' });
-        if (typeof res.data === 'string') {
-            // NDJSON or JSON string
-            try {
-                return JSON.parse(res.data);
-            }
-            catch {
-                // NDJSON?
-                const lines = res.data.trim().split('\n').filter((l) => l);
-                return lines.map((line) => JSON.parse(line));
-            }
-        }
-        return res.data;
+        return await this.client.getFile(fileId);
     }
     async downloadNdjson(fileId) {
-        const res = await this.drive.files.get({ fileId, alt: 'media' });
-        const content = typeof res.data === 'string' ? res.data : JSON.stringify(res.data);
+        const data = await this.client.getFile(fileId);
+        // data will likely be a string if NDJSON is returned and getFile sees weird content-type
+        // Or if getFile auto-parsed standard "application/json" but NDJSON is just text.
+        // Google Drive might return application/json for everything if we aren't careful?
+        // Actually .ndjson is separate.
+        // Safest: Handle string or object.
+        const content = typeof data === 'string' ? data : JSON.stringify(data);
         const lines = content.trim().split('\n').filter((l) => l);
         return lines.map((line) => JSON.parse(line));
     }
@@ -481,33 +462,20 @@ class DriveHandler {
         const lines = changes.map(c => JSON.stringify(c)).join('\n') + '\n';
         const startSeq = changes[0].seq;
         const name = `changes-${startSeq}-${Math.random().toString(36).substring(7)}.ndjson`;
-        const res = await this.drive.files.create({
-            requestBody: { name, parents: [this.folderId], mimeType: 'application/x-ndjson' },
-            media: { mimeType: 'application/x-ndjson', body: lines },
-            fields: 'id'
-        });
+        const res = await this.client.createFile(name, [this.folderId], 'application/x-ndjson', lines);
         this.currentLogSizeEstimate += new Blob([lines]).size;
-        return res.data.id;
+        return res.id;
     }
     async saveMeta(meta, expectedEtag = null) {
         const content = JSON.stringify(meta);
         const metaFile = await this.findFile('_meta.json');
         if (metaFile) {
-            const res = await this.drive.files.update({
-                fileId: metaFile.id,
-                headers: expectedEtag ? { 'If-Match': expectedEtag } : undefined,
-                media: { mimeType: 'application/json', body: content },
-                fields: 'id, etag'
-            });
-            this.metaEtag = res.data.etag;
+            const res = await this.client.updateFile(metaFile.id, content, expectedEtag || undefined);
+            this.metaEtag = res.etag;
         }
         else {
-            const res = await this.drive.files.create({
-                requestBody: { name: '_meta.json', parents: [this.folderId], mimeType: 'application/json' },
-                media: { mimeType: 'application/json', body: content },
-                fields: 'id, etag'
-            });
-            this.metaEtag = res.data.etag;
+            const res = await this.client.createFile('_meta.json', [this.folderId], 'application/json', content);
+            this.metaEtag = res.etag;
         }
     }
     async countTotalChanges() {
@@ -521,12 +489,12 @@ class DriveHandler {
     async cleanupOldFiles(oldIndexId, oldLogIds) {
         if (oldIndexId)
             try {
-                await this.drive.files.delete({ fileId: oldIndexId });
+                await this.client.deleteFile(oldIndexId);
             }
             catch { }
         for (const id of oldLogIds)
             try {
-                await this.drive.files.delete({ fileId: id });
+                await this.client.deleteFile(id);
             }
             catch { }
     }
@@ -538,6 +506,7 @@ class DriveHandler {
                 const metaFile = await this.findFile('_meta.json');
                 if (!metaFile)
                     return;
+                // Etag check
                 if (metaFile.etag !== this.metaEtag) {
                     await this.load();
                     this.notifyListeners();
@@ -566,8 +535,11 @@ class DriveHandler {
     onChange(cb) { this.listeners.push(cb); }
     stopPolling() { if (this.pollingInterval)
         clearInterval(this.pollingInterval); }
+    escapeQuery(value) {
+        return value.replace(/'/g, "\\'");
+    }
     async deleteFolder() { if (this.folderId)
-        await this.drive.files.delete({ fileId: this.folderId }); }
+        await this.client.deleteFile(this.folderId); }
     getNextSeq() { return this.meta.seq + 1; }
 }
 exports.DriveHandler = DriveHandler;

package/lib/types.d.ts

@@ -1,9 +1,6 @@
-/** Google Drive API client type */
-export type DriveClient = any;
+import { DriveClientOptions } from './client';
 /** Options for configuring the Google Drive adapter */
-export interface GoogleDriveAdapterOptions {
-    /** Configured Google Drive client (googleapis) */
-    drive: DriveClient;
+export interface GoogleDriveAdapterOptions extends DriveClientOptions {
     /** Specific folder ID to use as the DB root */
     folderId?: string;
     /** Folder name to search/create if no ID provided */

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@docstack/pouchdb-adapter-googledrive",
-  "version": "0.0.3",
+  "version": "0.0.5",
   "description": "PouchDB adapter for Google Drive",
   "main": "lib/index.js",
   "types": "lib/index.d.ts",
@@ -23,8 +23,10 @@
     "url": "https://github.com/onyx-ac/docstack-pouchdb-adapter-gdrive/issues"
   },
   "homepage": "https://onyx.ac/docstack",
+  "engines": {
+    "node": ">=18"
+  },
   "dependencies": {
-    "googleapis": "^126.0.0",
     "pouchdb-core": "^7.3.1"
   },
   "devDependencies": {