@fazetitans/fscopy 1.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 [fullname]
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,409 @@
+<p align="center">
+  <img src="assets/banner.png" alt="fscopy banner" width="600">
+</p>
+
+<h1 align="center">
+  <img src="assets/logo.png" alt="fscopy logo" width="40" height="40" style="vertical-align: middle;">
+  fscopy
+</h1>
+
+<p align="center">
+  <a href="https://www.npmjs.com/package/@fazetitans/fscopy"><img src="https://img.shields.io/npm/v/@fazetitans/fscopy.svg" alt="npm version"></a>
+  <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a>
+</p>
+
+<p align="center">
+  <strong>Fast CLI tool to copy Firestore collections between Firebase projects</strong>
+</p>
+
+Transfer documents between Firebase projects with support for subcollections, filtering, parallel transfers, and merge mode. Built with [Bun](https://bun.sh) for maximum performance.
+
+## Features
+
+- **Subcollection support** - Recursively copy nested collections
+- **Document filtering** - Filter documents with `--where` clauses
+- **Exclude patterns** - Skip subcollections by name or glob pattern
+- **Merge mode** - Update existing documents instead of overwriting
+- **Parallel transfers** - Copy multiple collections simultaneously
+- **Clear destination** - Optionally delete destination data before transfer
+- **Sync mode** - Delete destination docs not present in source
+- **Document transform** - Transform data during transfer with custom JS/TS functions
+- **Collection renaming** - Rename collections in destination for backups or migrations
+- **ID modification** - Add prefix or suffix to document IDs to avoid conflicts
+- **Webhook notifications** - Send Slack, Discord, or custom webhooks on completion
+- **Resume transfers** - Continue interrupted transfers from saved state
+- **Interactive mode** - Guided setup with prompts for project and collection selection
+- **Progress bar** - Real-time progress with ETA
+- **Automatic retry** - Exponential backoff on network errors
+- **Dry run mode** - Preview changes before applying (enabled by default)
+- **Flexible config** - INI, JSON, or CLI arguments
+
+## Installation
+
+### With Bun (recommended)
+
+```bash
+# Global install
+bun add -g @fazetitans/fscopy
+
+# Or run directly
+bunx @fazetitans/fscopy --help
+```
+
+### With npm
+
+```bash
+npm install -g @fazetitans/fscopy
+```
+
+### Prerequisites
+
+1. [Bun](https://bun.sh) or Node.js 18+
+1. Google Cloud authentication: `gcloud auth application-default login`
+1. Firestore read access on source project, write access on destination
+
+## Quick Start
+
+```bash
+# 1. Generate config file
+fscopy --init config.ini
+
+# 2. Edit config.ini with your project IDs and collections
+
+# 3. Preview transfer (dry run)
+fscopy -f config.ini
+
+# 4. Execute transfer
+fscopy -f config.ini -d false -y
+```
+
+## Usage
+
+### Basic Transfer
+
+```bash
+# Using config file
+fscopy -f config.ini
+
+# Using CLI arguments
+fscopy \
+  --source-project my-source \
+  --dest-project my-dest \
+  -c users orders products
+```
+
+### With Subcollections
+
+```bash
+# Include all subcollections
+fscopy -f config.ini -s
+
+# Exclude specific subcollections
+fscopy -f config.ini -s --exclude logs --exclude "temp/*"
+```
+
+### Filtering Documents
+
+```bash
+# Single filter
+fscopy -f config.ini --where "status == active"
+
+# Multiple filters (AND)
+fscopy -f config.ini -w "active == true" -w "createdAt > 2024-01-01"
+
+# Supported operators: ==, !=, <, >, <=, >=
+```
+
+### Advanced Options
+
+```bash
+# Merge mode (update instead of overwrite)
+fscopy -f config.ini --merge
+
+# Parallel transfers (3 collections at once)
+fscopy -f config.ini --parallel 3
+
+# With logging
+fscopy -f config.ini --log transfer.log
+
+# Limit documents per collection
+fscopy -f config.ini --limit 100
+
+# Quiet mode (no progress bar)
+fscopy -f config.ini -q
+
+# Clear destination before transfer (DESTRUCTIVE)
+fscopy -f config.ini --clear
+
+# Sync mode: delete orphan docs in destination
+fscopy -f config.ini --delete-missing
+
+# Interactive mode with prompts
+fscopy -i
+
+# Transform documents during transfer
+fscopy -f config.ini --transform ./transforms/anonymize.ts
+
+# Rename collections in destination
+fscopy -f config.ini -r users:users_backup -r orders:orders_2024
+
+# Add prefix to document IDs
+fscopy -f config.ini --id-prefix backup_
+
+# Add suffix to document IDs
+fscopy -f config.ini --id-suffix _archived
+
+# Send notification to Slack/Discord
+fscopy -f config.ini --webhook https://hooks.slack.com/services/...
+
+# Resume an interrupted transfer
+fscopy -f config.ini --resume
+```
+
+### Collection Renaming
+
+Rename collections during transfer for backups or migrations:
+
+```bash
+# Backup with dated collection names
+fscopy -f config.ini -r users:users_2024_12_29 -r orders:orders_2024_12_29
+
+# Multiple renames in one command
+fscopy -f config.ini --rename-collection users:users_v2 --rename-collection products:catalog
+```
+
+Subcollections are automatically renamed along with their parent collection.
+
+### ID Modification
+
+Add prefix or suffix to document IDs to avoid conflicts when merging:
+
+```bash
+# Add prefix: user123 → backup_user123
+fscopy -f config.ini --id-prefix backup_
+
+# Add suffix: user123 → user123_v2
+fscopy -f config.ini --id-suffix _v2
+
+# Combine both: user123 → old_user123_archived
+fscopy -f config.ini --id-prefix old_ --id-suffix _archived
+```
+
+### Document Transform
+
+Transform documents during transfer using a custom function:
+
+```bash
+# Create a transform file
+cat > anonymize.ts << 'EOF'
+export function transform(doc, meta) {
+    // Anonymize email addresses
+    if (doc.email) {
+        doc.email = `user_${meta.id}@example.com`;
+    }
+    // Remove sensitive fields
+    delete doc.password;
+    delete doc.ssn;
+    // Return null to skip the document
+    if (doc.deleted) return null;
+    return doc;
+}
+EOF
+
+# Use the transform
+fscopy -f config.ini -t ./anonymize.ts
+```
+
+The transform function receives:
+
+- `doc` - The document data as an object
+- `meta` - Metadata with `id` (document ID) and `path` (full document path)
+
+Return the transformed document, or `null` to skip it.
+
+### Webhook Notifications
+
+Get notified when transfers complete (success or failure):
+
+```bash
+# Slack webhook
+fscopy -f config.ini --webhook https://hooks.slack.com/services/XXX/YYY/ZZZ
+
+# Discord webhook
+fscopy -f config.ini --webhook https://discord.com/api/webhooks/123/abc
+
+# Custom webhook (receives raw JSON payload)
+fscopy -f config.ini --webhook https://api.example.com/webhook
+```
+
+The webhook receives a POST request with:
+
+- `source` / `destination` - Project IDs
+- `collections` - List of transferred collections
+- `stats` - Documents transferred, deleted, errors
+- `duration` - Transfer time in seconds
+- `dryRun` - Whether it was a dry run
+- `success` - Boolean status
+- `error` - Error message (if failed)
+
+Slack and Discord webhooks are automatically formatted with rich messages.
+
+### Resume Interrupted Transfers
+
+Large migrations can be resumed if interrupted:
+
+```bash
+# Start a transfer (state is saved automatically to .fscopy-state.json)
+fscopy -f config.ini -d false
+
+# If interrupted (Ctrl+C, network error, etc.), resume from where it left off
+fscopy -f config.ini --resume
+
+# Use a custom state file
+fscopy -f config.ini --state-file ./my-transfer.state.json
+fscopy -f config.ini --resume --state-file ./my-transfer.state.json
+```
+
+The state file tracks:
+
+- Completed document IDs per collection
+- Transfer statistics
+- Source/destination project validation
+
+State files are automatically deleted on successful completion.
+
+## Configuration
+
+### INI Format (recommended)
+
+```bash
+fscopy --init config.ini
+```
+
+```ini
+[projects]
+source = my-source-project
+dest = my-dest-project
+
+[transfer]
+collections = users, orders, products
+includeSubcollections = true
+dryRun = true
+batchSize = 500
+limit = 0
+
+[options]
+; where = status == active
+; exclude = logs, cache, temp/*
+merge = false
+parallel = 1
+clear = false
+deleteMissing = false
+; transform = ./transforms/anonymize.ts
+; renameCollection = users:users_backup, orders:orders_2024
+; idPrefix = backup_
+; idSuffix = _v2
+```
+
+### JSON Format
+
+```bash
+fscopy --init config.json
+```
+
+```json
+{
+    "sourceProject": "my-source-project",
+    "destProject": "my-dest-project",
+    "collections": ["users", "orders"],
+    "includeSubcollections": true,
+    "dryRun": true,
+    "batchSize": 500,
+    "limit": 0,
+    "where": ["status == active"],
+    "exclude": ["logs", "cache"],
+    "merge": false,
+    "parallel": 1,
+    "clear": false,
+    "deleteMissing": false,
+    "transform": null,
+    "renameCollection": {},
+    "idPrefix": null,
+    "idSuffix": null
+}
+```
+
+## CLI Reference
+
+| Option                     | Alias | Type    | Default | Description                       |
+| -------------------------- | ----- | ------- | ------- | --------------------------------- |
+| `--init`                   |       | string  |         | Generate config template          |
+| `--config`                 | `-f`  | string  |         | Path to config file               |
+| `--source-project`         |       | string  |         | Source Firebase project           |
+| `--dest-project`           |       | string  |         | Destination project               |
+| `--collections`            | `-c`  | array   |         | Collections to transfer           |
+| `--include-subcollections` | `-s`  | boolean | `false` | Include subcollections            |
+| `--where`                  | `-w`  | array   |         | Filter documents                  |
+| `--exclude`                | `-x`  | array   |         | Exclude subcollections            |
+| `--merge`                  | `-m`  | boolean | `false` | Merge instead of overwrite        |
+| `--parallel`               | `-p`  | number  | `1`     | Parallel transfers                |
+| `--dry-run`                | `-d`  | boolean | `true`  | Preview without writing           |
+| `--batch-size`             | `-b`  | number  | `500`   | Documents per batch               |
+| `--limit`                  | `-l`  | number  | `0`     | Limit docs (0 = no limit)         |
+| `--retries`                |       | number  | `3`     | Retries on error                  |
+| `--log`                    |       | string  |         | Log file path                     |
+| `--quiet`                  | `-q`  | boolean | `false` | No progress bar                   |
+| `--yes`                    | `-y`  | boolean | `false` | Skip confirmation                 |
+| `--clear`                  |       | boolean | `false` | Clear destination before transfer |
+| `--delete-missing`         |       | boolean | `false` | Delete dest docs not in source    |
+| `--interactive`            | `-i`  | boolean | `false` | Interactive mode with prompts     |
+| `--transform`              | `-t`  | string  |         | Path to JS/TS transform file      |
+| `--rename-collection`      | `-r`  | array   |         | Rename collection (source:dest)   |
+| `--id-prefix`              |       | string  |         | Add prefix to document IDs        |
+| `--id-suffix`              |       | string  |         | Add suffix to document IDs        |
+| `--webhook`                |       | string  |         | Webhook URL for notifications     |
+| `--resume`                 |       | boolean | `false` | Resume from saved state           |
+| `--state-file`             |       | string  | `.fscopy-state.json` | State file path      |
+
+## How It Works
+
+1. **Authentication** - Uses Google Application Default Credentials (ADC)
+2. **Document counting** - Counts total documents for progress bar
+3. **Batch processing** - Transfers documents in configurable batches
+4. **Retry logic** - Automatic retry with exponential backoff on failures
+5. **Subcollection discovery** - Uses `listCollections()` to find nested data
+
+## Notes
+
+- **Dry run is ON by default** - Use `-d false` for actual transfer
+- **Documents are overwritten** - Use `--merge` to update instead
+- **Where filters apply to root only** - Subcollections are copied in full
+- **Exclude patterns support globs** - e.g., `temp/*`, `*/logs`
+- **Progress bar shows ETA** - Based on documents processed
+- **Clear is destructive** - `--clear` deletes all destination docs before transfer
+- **Delete-missing syncs** - `--delete-missing` removes orphan docs after transfer
+- **Transform applies to all** - Transform function is applied to both root and subcollection docs
+- **Same project allowed** - Source and destination can be the same project when using `--rename-collection` or `--id-prefix`/`--id-suffix`
+
+## Development
+
+```bash
+# Clone and install
+git clone https://github.com/fazetitans/fscopy.git
+cd fscopy
+bun install
+
+# Run locally
+bun start -- -f config.ini
+
+# Run tests
+bun test
+
+# Type check & lint
+bun run type-check
+bun run lint
+```
+
+## License
+
+[MIT](LICENSE)

package/package.json

@@ -0,0 +1,74 @@
+{
+    "name": "@fazetitans/fscopy",
+    "version": "1.0.0",
+    "description": "Fast CLI tool to copy Firestore collections between Firebase projects with filtering, parallel transfers, and subcollection support",
+    "type": "module",
+    "bin": {
+        "fscopy": "./src/cli.ts"
+    },
+    "scripts": {
+        "start": "bun src/cli.ts",
+        "dev": "bun --watch src/cli.ts",
+        "test": "bun test",
+        "test:watch": "bun test --watch",
+        "type-check": "tsc --noEmit",
+        "lint": "eslint src/**/*.ts",
+        "lint:fix": "eslint src/**/*.ts --fix",
+        "format": "prettier --write src/**/*.ts",
+        "format:check": "prettier --check src/**/*.ts",
+        "prepublishOnly": "bun run type-check && bun run lint && bun test"
+    },
+    "keywords": [
+        "firestore",
+        "firebase",
+        "transfer",
+        "migration",
+        "cli",
+        "copy",
+        "backup",
+        "sync",
+        "google-cloud",
+        "database",
+        "nosql",
+        "bun"
+    ],
+    "author": "fazetitans",
+    "license": "MIT",
+    "repository": {
+        "type": "git",
+        "url": "git+https://github.com/fazetitans/fscopy.git"
+    },
+    "homepage": "https://github.com/fazetitans/fscopy#readme",
+    "bugs": {
+        "url": "https://github.com/fazetitans/fscopy/issues"
+    },
+    "engines": {
+        "node": ">=18.0.0"
+    },
+    "files": [
+        "src/cli.ts",
+        "README.md",
+        "LICENSE"
+    ],
+    "dependencies": {
+        "@inquirer/prompts": "^8.1.0",
+        "cli-progress": "^3.12.0",
+        "firebase-admin": "^13.6.0",
+        "ini": "^6.0.0",
+        "yargs": "^17.7.2"
+    },
+    "devDependencies": {
+        "@eslint/js": "^9.39.2",
+        "@types/cli-progress": "^3.11.6",
+        "@types/ini": "^4.1.1",
+        "@types/node": "^25.0.3",
+        "@types/yargs": "^17.0.35",
+        "@typescript-eslint/eslint-plugin": "^8.51.0",
+        "@typescript-eslint/parser": "^8.51.0",
+        "bun-types": "^1.3.5",
+        "eslint": "^9.39.2",
+        "prettier": "^3.7.4",
+        "typescript": "^5.9.3",
+        "typescript-eslint": "^8.51.0"
+    }
+}