aspect-sync 0.1.16 → 0.1.17

package/README.md

@@ -4,15 +4,14 @@ A CLI tool to sync files from external services (via rclone) to the Aspect platf
 
 ## Features
 
-- **Rclone Integration**: Sync files from any rclone-supported remote (Dropbox, Google Drive, S3, etc.)
-- **Directory Structure Preservation**: Maintains folder hierarchy from source to destination
-- **Idempotent Syncs**: Safe to re-run - automatically skips already-uploaded files and reuses existing directories
-- **Upload-Core Uploads**: Efficient chunked uploads through Aspect's edge worker with retry logic
-- **Dynamic CLI Progress**: Real-time upload progress display with in-place updates
-- **Concurrent Uploads**: Configurable parallel chunk uploads for optimal performance
-- **Session Isolation**: Auto-generated nanosecond-precision sessions for safe concurrent syncs
-- **Automatic Cleanup**: Removes local files after successful upload
-- **Retry Logic**: Automatic retry for failed uploads (part-level and file-level)
+- **Rclone integration**: Lists and downloads from any rclone-supported remote.
+- **Aspect-compatible path planning**: Preserves source hierarchy while sanitizing names that Aspect rejects.
+- **Duplicate preservation when supported**: Preserves duplicate file and folder names when the rclone provider exposes stable IDs.
+- **Idempotent reruns**: Reuses existing directories and skips assets that are already uploaded.
+- **Batched migrations**: Downloads and uploads large remotes in size-limited batches.
+- **Progress display**: Shows download, upload, skip, and failure progress in the terminal.
+- **Session isolation**: Uses unique session directories so concurrent syncs do not collide.
+- **Automatic cleanup**: Removes local staged files after successful upload unless `--keep-local` is set.
 
 ## Prerequisites
 
@@ -61,19 +60,19 @@ my-s3:
 **Important:** The remote names are shown **with a colon** (`:`) but you use them **without the colon** in the `--remote` parameter.
 
 Examples:
-- If you see `dropbox:` → use `--remote dropbox`
-- If you see `my-dropbox:` → use `--remote my-dropbox`
-- If you see `gdrive:` → use `--remote gdrive`
+- If you see `dropbox:`, use `--remote dropbox`
+- If you see `my-dropbox:`, use `--remote my-dropbox`
+- If you see `gdrive:`, use `--remote gdrive`
 
 ### Test Your Remote
 
 Before syncing, test that your remote works:
 ```bash
 # List files in the root of your remote
-rclone ls dropbox:
+rclone lsjson dropbox: --recursive
 
 # List files in a specific folder
-rclone ls dropbox:my-folder
+rclone lsjson dropbox:my-folder --recursive
 ```
 
 This should list files in your cloud storage. If you see files listed, your remote is configured correctly!
@@ -112,24 +111,58 @@ npx aspect-sync \
 
 ### Command Options
 
-| Option | Description | Required | Default |
-|--------|-------------|----------|---------|
-| `--remote <name>` | rclone remote name from your rclone config (without colon). Use `rclone listremotes` to see your configured remotes. | Yes | - |
-| `--path <path>` | Remote path to sync from | Yes | - |
-| `--directory-id <id>` | Aspect directory ID to upload to | Yes | - |
-| `--project-id <id>` | Aspect project ID | Yes | - |
-| `--api-key <key>` | Aspect API key (or set `ASPECT_API_KEY` env var) | Yes | - |
-| `--api-url <url>` | Aspect API URL (or set `ASPECT_API_URL` env var) | No | `https://api.aspect.inc` |
-| `--edge-worker-url <url>` | Aspect edge worker URL (or set `ASPECT_EDGE_WORKER_URL` env var) | No | `https://mackinac.aspect.inc` |
-| `--upload-concurrent <number>` | Max concurrent chunk uploads | No | `16` |
-| `--batch-size <size>` | Run the migration in download/upload batches (e.g., `500GB`, `1TB`) so data is uploaded as soon as each batch finishes downloading. | No | Disabled |
-| `--keep-local` | Keep local files after upload (for debugging) | No | `false` |
-| `--check` | Run in check-only mode (list remote files and compare with Aspect without downloading/uploading) | No | `false` |
-| `--temp-dir <path>` | Base temporary directory for synced files | No | `~/.aspect/sync` |
-| `--session <name>` | Session name for isolation. If not provided, auto-generates timestamp with nanosecond precision (YYYY-MM-DD_HHh-MMm-SSs-NNNNNNNNNns). Creates subdirectory in temp-dir. | No | Auto-generated |
-| `--rclone-extra <arg>` | Additional rclone argument appended to the sync command. Repeat the flag to pass multiple values. | No | Not set |
-
-Repeat `--rclone-extra` for each passthrough flag you need. Example: `--rclone-extra --drive-shared-with-me`.
+Required:
+
+| Option | Description |
+|--------|-------------|
+| `--remote <name>` | rclone remote name from your rclone config, without the trailing colon. |
+| `--path <path>` | Remote path to sync from. |
+| `--directory-id <id>` | Aspect directory ID to upload into. |
+| `--project-id <id>` | Aspect project ID. |
+| `--api-key <key>` | Aspect API key. Can also be set with `ASPECT_API_KEY`. |
+
+Aspect API options:
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `--api-url <url>` | Aspect API URL. Can also be set with `ASPECT_API_URL`. | `https://api.aspect.inc` |
+| `--edge-worker-url <url>` | Aspect edge worker URL. Can also be set with `ASPECT_EDGE_WORKER_URL`. | `https://mackinac.aspect.inc` |
+
+Sync behavior:
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `--upload-concurrent <number>` | Maximum concurrent chunk uploads to Aspect. | `16` |
+| `--batch-size <size>` | Enables batched mode with a maximum batch size, for example `500GB` or `1TB`. | Disabled |
+| `--keep-local` | Keep local staged files after upload for debugging. | `false` |
+| `--check` | Check remote files against Aspect without downloading or uploading. | `false` |
+| `--temp-dir <path>` | Base temporary directory for sync sessions. | `~/.aspect/sync` |
+| `--session <name>` | Session name. If omitted, a nanosecond-precision timestamp is generated. | Auto-generated |
+
+rclone tuning:
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `--rclone-transfers <number>` | Parallel file transfers for bulk rclone sync. Also controls rclone checkers. | `4` |
+| `--rclone-multi-thread-streams <number>` | Streams per large file for rclone transfers. | `4` |
+| `--rclone-multi-thread-chunk-size <size>` | rclone multi-thread chunk size. | `64MiB` |
+| `--rclone-multi-thread-cutoff <size>` | Minimum file size for rclone multi-threading. | `64MiB` |
+| `--rclone-buffer-size <size>` | rclone buffer size per transfer. | `64M` |
+| `--rclone-no-mmap` | Disable rclone memory-mapped I/O. | `false` |
+| `--rclone-extra <arg>` | Extra rclone argument. Repeat for multiple args. | Not set |
+
+Repeat `--rclone-extra` for each passthrough value. Example:
+
+```bash
+aspect-sync \
+  --remote gdrive \
+  --path /Videos \
+  --directory-id <directory-id> \
+  --project-id <project-id> \
+  --api-key <api-key> \
+  --rclone-extra --drive-root-folder-id \
+  --rclone-extra <folder-id>
+```
 
 ### Environment Variables
 
@@ -147,6 +180,8 @@ aspect-sync \
   --project-id <project-id>
 ```
 
+Analytics is enabled only when a PostHog key is available at build time or runtime. Runtime values can be supplied with `ASPECT_POSTHOG_KEY`/`ASPECT_POSTHOG_HOST` or `POSTHOG_KEY`/`POSTHOG_HOST`. Set `ASPECT_DISABLE_POSTHOG=true` or `POSTHOG_DISABLED=true` to disable analytics.
+
 ## Examples
 
 ### Sync from Dropbox
@@ -173,6 +208,19 @@ aspect-sync \
   --rclone-extra --drive-shared-with-me
 ```
 
+### Sync a Specific Google Drive Folder by ID
+
+```bash
+aspect-sync \
+  --remote gdrive \
+  --path / \
+  --directory-id abc-123-def \
+  --project-id xyz-789-uvw \
+  --api-key your-api-key \
+  --rclone-extra --drive-root-folder-id \
+  --rclone-extra 1truv6wvzgXM3CkGMzuC8c-hrpMk5RAAs
+```
+
 ### Sync from S3 Bucket
 
 ```bash
@@ -223,41 +271,67 @@ Sessions are stored in `~/.aspect/sync/{session-name}` by default. The full path
 
 ## How It Works
 
-1. **Session Creation**: Creates isolated session directory with nanosecond-precision timestamp
-2. **Rclone Sync**: Downloads files from the remote to the session directory
-3. **Directory Discovery**: Fetches existing directory tree from Aspect to avoid duplicates
-4. **Directory Creation**: Creates only missing directories, reuses existing ones
-5. **Asset Deduplication**: Checks if each asset already exists before uploading
-6. **File Upload**: Uploads new assets using multipart upload (20MB chunks)
-7. **Progress Display**: Shows real-time upload progress in the CLI
-8. **Cleanup**: Deletes local files after successful upload (unless `--keep-local` is set)
+1. **Session creation**: Creates an isolated local session directory.
+2. **Remote listing**: Uses `rclone lsjson` to list files and directories with IDs when the remote exposes them.
+3. **Path planning**: Sanitizes names for Aspect and assigns unique targets for duplicate files and folders.
+4. **Directory planning**: Fetches the existing Aspect directory tree and creates only missing directories.
+5. **Asset existence check**: Skips assets that already exist and are fully uploaded.
+6. **Download**: Downloads unchanged files with bulk `rclone sync --files-from`; downloads renamed or duplicate files by ID when needed.
+7. **Upload**: Uploads staged files through `@aspect/upload-core` and the Aspect edge worker.
+8. **Cleanup**: Deletes local staged files after successful upload unless `--keep-local` is set.
+
+Aspect upload chunk size is controlled by the server. There is no CLI flag for upload chunk size.
+
+## Name Sanitization And Duplicates
+
+Aspect enforces cross-platform file and folder names. Sanitization works for all rclone providers because it happens inside `aspect-sync` after remote listing. During planning, each path segment is sanitized before upload:
+
+- forbidden characters such as `:`, `*`, `?`, `"`, `<`, `>`, and `|` are replaced with `_`
+- trailing dots/spaces are trimmed
+- reserved Windows names such as `CON`, `AUX`, and `COM1` are prefixed with `_`
+- names are truncated to the backend byte limit
+
+Some providers, especially Google Drive, allow duplicate names in the same folder. `aspect-sync` preserves those without changing the source when rclone exposes stable IDs:
+
+- duplicate files use parenthesized suffixes, for example `IMG_1891.MOV`, `IMG_1891 (2).MOV`
+- duplicate folders use numeric suffixes, for example `Shoot`, `Shoot 2`
+
+Provider behavior:
+
+- Providers without duplicate folder ambiguity work normally.
+- Duplicate files are preserved when rclone exposes stable file IDs. If identity is required but no file ID is available, the sync fails instead of guessing.
+- Duplicate folders are preserved when rclone exposes stable folder IDs and supports querying children by folder ID. Google Drive is the primary supported provider for this flow.
+- Duplicate folders without stable folder IDs fail with a clear error instead of silently merging folders.
+
+When a duplicate Google Drive folder is detected, `aspect-sync` queries each folder by its Drive ID and downloads descendants by file ID so the source folder identity is preserved.
 
 ## Idempotent Behavior
 
 The syncer is **fully idempotent** - you can safely re-run the same sync multiple times:
 
-✅ **Existing directories are reused** - The tool fetches the directory tree first and only creates missing directories
+**Existing directories are reused** - The tool fetches the directory tree first and only creates missing directories.
 
-✅ **Existing assets are skipped** - Before uploading, checks if an asset with the same name, size, and location already exists
+**Existing assets are skipped** - Before uploading, checks if an asset with the same name, size, and location already exists.
   - If found and fully uploaded: Skips upload
   - If found but incomplete: Deletes incomplete asset and re-uploads
   - If not found: Proceeds with upload
 
-✅ **Clean re-runs** - If a sync fails halfway:
-  ```bash
-  # First run - uploads 50 of 100 files then crashes
-  aspect-sync --remote dropbox --path /videos --directory-id abc --project-id xyz --api-key key
+**Clean re-runs** - If a sync fails halfway:
 
-  # Re-run - skips the 50 uploaded files, uploads remaining 50
-  aspect-sync --remote dropbox --path /videos --directory-id abc --project-id xyz --api-key key
-  ```
+```bash
+# First run - uploads 50 of 100 files then crashes
+aspect-sync --remote dropbox --path /videos --directory-id abc --project-id xyz --api-key key
+
+# Re-run - skips the 50 uploaded files, uploads remaining 50
+aspect-sync --remote dropbox --path /videos --directory-id abc --project-id xyz --api-key key
+```
 
 **Output on re-run:**
 ```
-✓ video1.mp4 (100%)
-⊘ video2.mp4 (Skipped - already uploaded)  ← Already uploaded
-⊘ video3.mp4 (Skipped - already uploaded)  ← Already uploaded
-↑ video4.mp4 (45%)                        ← Uploading now
+video1.mp4 (100%)
+video2.mp4 (Skipped - already uploaded)
+video3.mp4 (Skipped - already uploaded)
+video4.mp4 (45%)
 
 Total: 100 | Success: 1 | Failed: 0 | Skipped: 98 | In Progress: 1 | Queued: 0
 ```
@@ -267,29 +341,23 @@ Total: 100 | Success: 1 | Failed: 0 | Skipped: 98 | In Progress: 1 | Queued: 0
 The tool shows dynamic, in-place progress updates:
 
 ```
-✓ video1.mp4 (100%)
-✓ video2.mp4 (100%)
-⊘ video3.mp4 (Skipped - already uploaded)
-↑ video4.mp4 (47%)
-↑ video5.mp4 (23%)
+video1.mp4 (100%)
+video2.mp4 (100%)
+video3.mp4 (Skipped - already uploaded)
+video4.mp4 (47%)
+video5.mp4 (23%)
 
 Total: 10 | Success: 2 | Failed: 0 | Skipped: 1 | In Progress: 2 | Queued: 5 | Speed: 45.3 Mbps | Time remaining: 2m 15s
 ```
 
-**Status Indicators:**
-- `✓` = Successfully uploaded
-- `✗` = Failed upload
-- `⊘` = Skipped (already exists on server)
-- `↑` = Currently uploading
-
 Only files that are actively uploading are shown (not the entire queue).
 
 ## Error Handling
 
 ### Retry Logic
 
-- **Part-level retries**: Each 20MB chunk is retried up to 3 times on failure
-- **File-level retries**: Each file upload is retried up to 3 times on failure
+- **Chunk retries**: Upload chunks are retried on transient failures.
+- **File retries**: Failed files can be retried by rerunning the sync.
 - **Exponential backoff**: Delays increase exponentially between retries
 
 ### Common Errors
@@ -348,7 +416,7 @@ Verify your rclone remote is configured:
 
 ```bash
 rclone listremotes
-rclone ls dropbox:
+rclone lsjson dropbox: --recursive
 ```
 
 ## License

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "aspect-sync",
-  "version": "0.1.16",
+  "version": "0.1.17",
   "description": "CLI tool to sync files from external services to Aspect via rclone",
   "main": "dist/index.js",
   "bin": {