gdrive-syncer 3.0.0 → 3.1.1

package/Readme.md

@@ -216,15 +216,35 @@ Patterns support glob-style matching with the following features:
 
 ### Features
 
+- **Nested folder support** - Recursively syncs files in subdirectories
 - **Multiple sync folders** - Configure multiple sync pairs in one config
 - **Local & Global configs** - Per-project or machine-wide configurations
-- **File patterns** - Filter files by pattern
+- **File patterns** - Filter files by pattern (applied to filenames, not paths)
 - **Auto-discovery** - Finds local config in current or parent directories
-- **Git-style diffs** - Colored diff output showing changes
-- **Automatic backups** - Creates timestamped backups before download
+- **Git-style diffs** - Colored diff output showing changes with full paths
+- **Automatic backups** - Creates timestamped backups before download (preserves folder structure)
 - **Two-way sync** - Upload local changes or download from Drive
+- **Optional deletion** - Prompts to delete orphan files/folders after sync
 - **Sync All** - Run operations on all syncs within selected config
 - **Registered Local Configs** - Run sync on multiple projects from anywhere
+- **Auto-create folders** - Creates missing folders on Drive during upload
+
+### Sync Behavior
+
+The sync compares files between local and Drive, showing:
+
+| Status | Diff | Upload | Download |
+|--------|------|--------|----------|
+| **Modified** | Shows diff | Updates on Drive | Updates locally |
+| **Local only** | Listed | Uploads to Drive | Prompts to delete |
+| **Drive only** | Listed | Prompts to delete | Downloads locally |
+| **Empty folders** | Listed | Prompts to delete | — |
+
+**Delete prompts:**
+- **Upload**: After uploading, prompts to delete files/folders on Drive that don't exist locally
+- **Download**: After downloading, prompts to delete local files that don't exist on Drive (backup is created first)
+
+This ensures you can keep Drive in sync with local (or vice versa) without accumulating orphan files.
 
 ### Registered Local Configs
 

package/docs/PERFORMANCE_DECISIONS.md

@@ -0,0 +1,142 @@
+# Performance Optimization Decisions
+
+This document records design decisions made for performance optimizations, including rationale and trade-offs considered.
+
+## Overview
+
+The diff/sync operations were slow with nested folders and many files (e.g., 30+ files across multiple folder levels). Root cause: all Google Drive API calls were sequential and blocking.
+
+---
+
+## Decision 1: Parallel API Calls (Implemented)
+
+**Problem:** Sequential API calls caused O(n) latency where n = number of files + folders.
+
+**Solution:** Use async/parallel operations with batched concurrency.
+
+**Implementation:**
+- `listAsync()` - Non-blocking folder listing using `child_process.exec` with promises
+- `downloadAsync()` - Non-blocking file downloads
+- `downloadParallel()` - Batch downloads with configurable concurrency
+- `listDriveFilesRecursiveOptimized()` - Breadth-first traversal querying multiple folders in parallel
+
+**Concurrency Limit:** 5 concurrent requests per batch
+
+**Rationale for batch size of 5:**
+- Google Drive API limit: ~1,000 queries per 100 seconds per user (~10/sec sustained)
+- Batch size of 5 is conservative, allowing headroom for other operations
+- Can be increased to 8-10 if needed, but diminishing returns due to network latency
+
+**Expected Performance Improvement:**
+| Scenario | Before (sequential) | After (parallel, batch=5) |
+|----------|---------------------|---------------------------|
+| 30 files in nested folders | ~10-15 seconds | ~3-5 seconds |
+| Download 30 files | ~6-15 seconds | ~1.5-3 seconds |
+
+---
+
+## Decision 2: Session-Scoped Folder Caching (Implemented)
+
+**Problem:** When uploading multiple files to the same nested folder structure (e.g., 10 files to `root/a/b/c/`), the `findOrCreateDriveFolder` function would query the same parent folders repeatedly.
+
+**Solution:** Session-scoped folder cache that is cleared at the start of each sync operation.
+
+**Implementation:**
+- `folderCache` - Map structure: `parentId -> Map(folderName -> folderId)`
+- `clearFolderCache()` - Called at start of `runSyncOperation()` to ensure fresh data
+- `getCachedFolderId()` / `cacheFolderId()` - Cache lookup and storage helpers
+- Cache is used in `findOrCreateDriveFolder()` to avoid redundant API calls
+
+**Why this is safe:**
+1. **Fresh data on every operation** - Cache is cleared at start, so we always query Drive for current state
+2. **Short-lived** - Cache only exists for duration of single sync (seconds)
+3. **Consistent within operation** - Folder structure shouldn't change during a 3-5 second sync
+
+**Benefit:** If uploading 10 files to `root/a/b/c/`:
+- Without cache: Query `a`, `b`, `c` 10 times each = 30 API calls
+- With cache: Query `a`, `b`, `c` once each = 3 API calls
+
+**Trade-off considered:** Risk of stale data if folder modified mid-sync.
+- Acceptable risk given short operation duration (seconds)
+- User can simply re-run sync if concurrent modification occurs
+
+---
+
+## Decision 3: Breadth-First vs Depth-First Listing
+
+**Problem:** Original `listDriveFilesRecursive` used depth-first traversal, which creates a chain of blocking calls.
+
+**Solution:** Breadth-first traversal with level-by-level parallel queries.
+
+**How it works:**
+```
+Level 0: Query root folder (1 API call)
+Level 1: Query all subfolders in parallel (e.g., 5 folders = 1 batch)
+Level 2: Query all sub-subfolders in parallel (e.g., 10 folders = 2 batches)
+...
+```
+
+**Benefit:** Folders at the same depth are queried concurrently instead of sequentially.
+
+**Trade-off:** Slightly more memory usage to hold the queue, but negligible for typical folder structures.
+
+---
+
+## Decision 4: Delta Sync (Implemented)
+
+**Problem:** Previously, ALL files were downloaded from Drive to a temp directory for comparison, even if they hadn't changed. With 30 files, this meant 30 downloads every sync.
+
+**Solution:** Compare file metadata (size + modification time) before downloading. Only download files where metadata indicates a potential change.
+
+**Implementation:**
+- `parseListOutput()` now extracts `sizeBytes` and `modifiedTime` from gdrive output
+- `listLocalFilesRecursive()` now includes `sizeBytes` and `modifiedTime` from `fs.statSync()`
+- Comparison logic in `runSyncOperation()`:
+  1. Get local files with metadata
+  2. For each Drive file, check if local exists with same size AND local is not older
+  3. Skip download for unchanged files
+  4. Only download and compare files where metadata differs
+
+**Unchanged file detection:**
+```javascript
+const sizeMatch = driveFile.sizeBytes === localFile.sizeBytes;
+const timeMatch = driveTime <= localTime + 1000; // 1 second tolerance
+
+if (sizeMatch && timeMatch) {
+  // Skip download - metadata indicates unchanged
+}
+```
+
+**Performance improvement:**
+
+| Scenario | Before | After |
+|----------|--------|-------|
+| 30 files, none changed | Download 30 files | Download 0 files |
+| 30 files, 5 changed | Download 30 files | Download 5 files |
+| 30 files, all changed | Download 30 files | Download 30 files |
+
+**Trade-offs:**
+- Relies on accurate timestamps from Google Drive
+- 1-second tolerance to handle clock differences
+- Files with same size but different content AND older local time could be missed (rare edge case)
+
+**Why not use hashes?**
+- Google Drive provides `md5Checksum` but requires additional API call per file
+- Size + time comparison is sufficient for most use cases and much faster
+- Can add optional hash verification in future if needed
+
+---
+
+## Future Considerations
+
+1. **Configurable concurrency** - Allow users to adjust batch size via config if they have different rate limit needs
+2. **Progress callbacks** - Add progress reporting for large sync operations
+3. **Retry with exponential backoff** - Handle transient API failures gracefully
+4. **Optional hash verification** - Add `--verify` flag to compare MD5 checksums for critical syncs
+
+---
+
+## References
+
+- Google Drive API Quotas: https://developers.google.com/drive/api/guides/limits
+- Rate limit: 1,000 queries per 100 seconds per user

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "gdrive-syncer",
-  "version": "3.0.0",
+  "version": "3.1.1",
   "description": "Google Drive Syncer",
   "main": "./index.js",
   "bin": "./run.js",

package/run.js

@@ -6,6 +6,7 @@ const { cfgAdd, cfgRm, cfgShow } = require('./src/cfgManager');
 const { runSync } = require('./src/sync');
 const { runList, runSearch, runDelete, runMkdir, runListSync } = require('./src/list');
 const { envInit, envRun, envShow, envRemove, envRegister, envUnregister, envMigrate } = require('./src/envSync');
+const { checkForUpdates } = require('./src/versionCheck');
 
 const [, , ...args] = process.argv;
 const [firstArg] = args;
@@ -81,10 +82,17 @@ const showHelp = () => {
   console.clear();
   intro(color.bgCyan(color.black(' GDrive Syncer ')));
 
+  // Check for updates and show at start if available
+  const updateMessage = await checkForUpdates();
+  if (updateMessage) {
+    note(updateMessage, 'Update Available');
+  }
+
   try {
     // Check for help
     if (firstArg === 'help' || firstArg === '--help' || firstArg === '-h') {
       showHelp();
+      if (updateMessage) note(updateMessage, 'Update Available');
       outro(color.green('Done!'));
       return;
     }
@@ -92,6 +100,7 @@ const showHelp = () => {
     // Check for direct command
     if (firstArg && commands[firstArg]) {
       await commands[firstArg].handler();
+      if (updateMessage) note(updateMessage, 'Update Available');
       outro(color.green('Done!'));
       return;
     }
@@ -119,6 +128,7 @@ const showHelp = () => {
 
     if (category === 'help') {
       showHelp();
+      if (updateMessage) note(updateMessage, 'Update Available');
       outro(color.green('Done!'));
       return;
     }
@@ -176,6 +186,8 @@ const showHelp = () => {
       await commands[action].handler();
     }
 
+    // Show update message at exit
+    if (updateMessage) note(updateMessage, 'Update Available');
     outro(color.green('Done!'));
   } catch (e) {
     log.error(e.message);