sqlew 4.0.1 → 4.0.2

package/docs/cli/DATA_EXPORT_IMPORT.md

@@ -2,6 +2,14 @@
 
 > JSON-based project data migration for sharing and multi-database workflows
 
+## 📢 v4.0.2 Update: JSON is Now Required for Cross-Database Migration
+
+Starting from v4.0.2, **JSON export/import is the ONLY supported method for cross-database migrations** (e.g., SQLite → MySQL, MySQL → PostgreSQL).
+
+SQL dump (`db:dump`) no longer supports cross-database format conversion. See [Complete Cross-Database Migration Guide](#complete-cross-database-migration-guide-v402) below for step-by-step instructions.
+
+---
+
 ## Overview
 
 sqlew provides a complete JSON export/import system for migrating project data between databases. This is useful for:
@@ -21,35 +29,27 @@ solutions including schema + data using SQL dumps.
 ### Export a Project
 
 ```bash
-# You have to install sqlew via npm to use CLI mode
-cd /path/to/your/project
-npm install sqlew
+# Export all projects (no installation required!)
+npx sqlew db:export backup.json
 
-# Export all projects
-node node_modules/sqlew/dist/cli.js db:export --output=full-backup.json
-
-# or Export specific project to file
-node node_modules/sqlew/dist/cli.js db:export --project=my-project --output=backup.json
+# Export specific project to file
+npx sqlew db:export backup.json project=my-project
 
 # Export to stdout (pipe to another command)
-node node_modules/sqlew/dist/cli.js db:export --project=visualizer
+npx sqlew db:export project=visualizer
 ```
 
 ### Import a Project
 
 ```bash
-# You have to install sqlew via npm to use CLI mode (We recommend install per project)
-cd /path/to/your/other-project
-npm install sqlew
-
 # Import from JSON export
-node node_modules/sqlew/dist/cli.js db:import --source=backup.json
+npx sqlew db:import backup.json
 
 # Import with custom project name
-node node_modules/sqlew/dist/cli.js db:import --source=backup.json --project-name=new-name
+npx sqlew db:import backup.json project-name=new-name
 
 # Dry-run validation (no actual import)
-node node_modules/sqlew/dist/cli.js db:import --source=backup.json --dry-run
+npx sqlew db:import backup.json dry-run=true
 ```
 
 ## Export Command
@@ -57,17 +57,16 @@ node node_modules/sqlew/dist/cli.js db:import --source=backup.json --dry-run
 ### Syntax
 
 ```bash
-node node_modules/sqlew/dist/cli.js db:export [options]
+npx sqlew db:export [output-file] [key=value ...]
 ```
 
 ### Options
 
-| Option             | Description                     | Default           |
-|--------------------|---------------------------------|-------------------|
-| `--project <name>` | Export specific project by name | All projects      |
-| `--output <file>`  | Output file path                | stdout            |
-| `--db-path <path>` | Database file path              | `.sqlew/sqlew.db` |
-| `--config <path>`  | Config file path                | Auto-detect       |
+| Option            | Description                     | Default           |
+|-------------------|---------------------------------|-------------------|
+| `project=<name>`  | Export specific project by name | All projects      |
+| `db-path=<path>`  | Database file path              | `.sqlew/sqlew.db` |
+| `config=<path>`   | Config file path                | Auto-detect       |
 
 ### What Gets Exported
 
@@ -96,19 +95,19 @@ node node_modules/sqlew/dist/cli.js db:export [options]
 ### Syntax
 
 ```bash
-node node_modules/sqlew/dist/cli.js db:import --source=<file> [options]
+npx sqlew db:import <source-file> [key=value ...]
 ```
 
 ### Options
 
-| Option                  | Description                   | Default            |
-|-------------------------|-------------------------------|--------------------|
-| `--source <file>`       | JSON export file path         | **Required**       |
-| `--project-name <name>` | Target project name           | Use name from JSON |
-| `--skip-if-exists`      | Skip import if project exists | `true`             |
-| `--dry-run`             | Validate only, don't import   | `false`            |
-| `--db-path <path>`      | Database file path            | `.sqlew/sqlew.db`  |
-| `--config <path>`       | Config file path              | Auto-detect        |
+| Option                   | Description                   | Default            |
+|--------------------------|-------------------------------|--------------------|
+| `<source-file>`          | JSON export file path         | **Required**       |
+| `project-name=<name>`    | Target project name           | Use name from JSON |
+| `skip-if-exists=true`    | Skip import if project exists | `true`             |
+| `dry-run=true`           | Validate only, don't import   | `false`            |
+| `db-path=<path>`         | Database file path            | `.sqlew/sqlew.db`  |
+| `config=<path>`          | Config file path              | Auto-detect        |
 
 ### Import Process
 
@@ -147,34 +146,24 @@ node node_modules/sqlew/dist/cli.js db:import --source=<file> [options]
 - Idempotent operations (safe to retry on failure)
 - Comprehensive error messages with validation details
 
-## Installation for CLI Usage
+## CLI Usage
 
-For users who need to use export/import commands, install sqlew per-project:
+**No installation required!** The unified entry point allows direct use via npx:
 
 ```bash
-# Install in your project directory
-cd /path/to/your/project
-npm install sqlew
-
-# Now you can use CLI commands
-node node_modules/sqlew/dist/cli.js db:export --output=backup.json
-node node_modules/sqlew/dist/cli.js db:import --source=backup.json
-```
+# Export data
+npx sqlew db:export backup.json
 
-**Tip**: Add a shortcut to your `package.json` for convenience:
+# Import data
+npx sqlew db:import backup.json
 
-```json
-{
-  "scripts": {
-    "sqlew": "node node_modules/sqlew/dist/cli.js"
-  }
-}
+# SQL dump (same-database backup)
+npx sqlew db:dump sqlite backup.sql
 ```
 
-Then you can use: `npm run sqlew db:export --output=backup.json`
-
-**Note**: The MCP server (`npx sqlew`) and CLI commands are both included in the same `sqlew` package. You only need to
-install once.
+**Note**: Both MCP server mode and CLI commands use the same `sqlew` entry point. The first argument determines the mode:
+- `db:export`, `db:import`, `db:dump`, `query` → CLI mode
+- No argument or MCP-related args → MCP server mode
 
 ## Use Cases
 
@@ -188,20 +177,16 @@ consolidate all project contexts into one shared database.
 ```bash
 # Step 1: Export from each project's SQLite database
 cd ~/project-a
-npm install sqlew
-node node_modules/sqlew/dist/cli.js db:export --project=project-a --output=/tmp/project-a.json
+npx sqlew db:export /tmp/project-a.json project=project-a
 
 cd ~/project-b
-npm install sqlew
-node node_modules/sqlew/dist/cli.js db:export --project=project-b --output=/tmp/project-b.json
+npx sqlew db:export /tmp/project-b.json project=project-b
 
 cd ~/project-c
-npm install sqlew
-node node_modules/sqlew/dist/cli.js db:export --project=project-c --output=/tmp/project-c.json
+npx sqlew db:export /tmp/project-c.json project=project-c
 
 # Step 2: Create shared database and import all projects
 cd ~/shared-database
-npm install sqlew
 
 # Configure to use single MySQL database (edit .sqlew/config.toml)
 # [database]
@@ -212,9 +197,9 @@ npm install sqlew
 # password = "mypassword"
 # database = "shared_sqlew_db"
 
-node node_modules/sqlew/dist/cli.js db:import --source=/tmp/project-a.json
-node node_modules/sqlew/dist/cli.js db:import --source=/tmp/project-b.json
-node node_modules/sqlew/dist/cli.js db:import --source=/tmp/project-c.json
+npx sqlew db:import /tmp/project-a.json
+npx sqlew db:import /tmp/project-b.json
+npx sqlew db:import /tmp/project-c.json
 
 # Step 3: Configure each project to use shared database
 # In each project's .mcp.json:
@@ -245,11 +230,11 @@ node node_modules/sqlew/dist/cli.js db:import --source=/tmp/project-c.json
 
 ```bash
 # Export from source database
-node node_modules/sqlew/dist/cli.js db:export --project=main --output=main-export.json
+npx sqlew db:export main-export.json project=main
 
 # Import to different database (different machine or different database type)
 # This works because the project doesn't exist in the target database yet
-node node_modules/sqlew/dist/cli.js db:import --source=main-export.json --db-path=/path/to/new/database.db
+npx sqlew db:import main-export.json db-path=/path/to/new/database.db
 ```
 
 **Note**: Import skips if project name exists.
@@ -258,45 +243,361 @@ node node_modules/sqlew/dist/cli.js db:import --source=main-export.json --db-pat
 
 ```bash
 # Backup with SQL dump (preserves schema + data)
-node node_modules/sqlew/dist/cli.js db:dump --format=sqlite --output=backup-$(date +%Y%m%d).sql
+npx sqlew db:dump sqlite backup-$(date +%Y%m%d).sql
 
 # Or simple SQLite file copy
 cp .sqlew/sqlew.db .sqlew/backup-$(date +%Y%m%d).db
 ```
 
-See `node node_modules/sqlew/dist/cli.js db:dump --help` for full backup options.
+See `npx sqlew db:dump --help` for full backup options.
 
 ### Project Sharing
 
 ```bash
 # Developer A: Export project
-node node_modules/sqlew/dist/cli.js db:export --project=feature-x --output=feature-x.json
+npx sqlew db:export feature-x.json project=feature-x
 
 # Developer B: Import project
-node node_modules/sqlew/dist/cli.js db:import --source=feature-x.json
+npx sqlew db:import feature-x.json
 ```
 
 ### Multi-Project Consolidation
 
 ```bash
 # Export from different databases
-node node_modules/sqlew/dist/cli.js db:export --project=visualizer --output=vis.json
-node node_modules/sqlew/dist/cli.js db:export --project=api --output=api.json
+npx sqlew db:export vis.json project=visualizer
+npx sqlew db:export api.json project=api
 
 # Import to single database
-node node_modules/sqlew/dist/cli.js db:import --source=vis.json
-node node_modules/sqlew/dist/cli.js db:import --source=api.json
+npx sqlew db:import vis.json
+npx sqlew db:import api.json
 ```
 
 ### Cross-Database Migration
 
 ```bash
 # Export from SQLite
-node node_modules/sqlew/dist/cli.js db:export --output=data.json --db-path=.sqlew/sqlew.db
+npx sqlew db:export data.json db-path=.sqlew/sqlew.db
+
+# Import to MySQL (configure .sqlew/config.toml for MySQL first)
+npx sqlew db:import data.json
+```
+
+---
+
+## Complete Cross-Database Migration Guide (v4.0.2+)
+
+This section provides step-by-step instructions for migrating your sqlew database between different database systems.
+
+### Pre-Migration Checklist
+
+Before starting a migration, ensure you have:
+
+- [ ] **Backup your current database** (copy `.sqlew/sqlew.db` or use `db:dump`)
+- [ ] **Note your current sqlew version** (`npm list sqlew`)
+- [ ] **Target database is created and accessible**
+- [ ] **Database credentials are available**
+- [ ] **Required privileges**: SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, DROP, REFERENCES
+
+### Migration: SQLite → MySQL
+
+#### Step 1: Export from SQLite
+
+```bash
+cd /path/to/your/project
+
+# Export all data to JSON
+npx sqlew db:export migration-backup.json
+```
+
+#### Step 2: Prepare MySQL Database
+
+```sql
+-- Connect to MySQL as admin
+mysql -u root -p
+
+-- Create database (UTF-8 required for proper text handling)
+CREATE DATABASE sqlew_db CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
+
+-- Create user with required privileges
+CREATE USER 'sqlew_user'@'localhost' IDENTIFIED BY 'your-secure-password';
+GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, INDEX, DROP, REFERENCES
+  ON sqlew_db.* TO 'sqlew_user'@'localhost';
+FLUSH PRIVILEGES;
+```
+
+#### Step 3: Configure config.toml for MySQL
+
+Create or edit `.sqlew/config.toml`:
+
+```toml
+[database]
+type = "mysql"
+
+[database.connection]
+host = "localhost"
+port = 3306
+database = "sqlew_db"
+
+[database.auth]
+type = "direct"
+user = "sqlew_user"
+password = "your-secure-password"
+
+[project]
+name = "your-project-name"
+```
+
+#### Step 4: Import to MySQL
+
+```bash
+# Import data to MySQL (config.toml will be used automatically)
+npx sqlew db:import migration-backup.json
+```
+
+#### Step 5: Verify Migration
+
+```bash
+# Test MCP server connection
+npx sqlew --config-path=.sqlew/config.toml
+
+# Or test with MCP Inspector
+npx @modelcontextprotocol/inspector npx sqlew
+```
+
+---
+
+### Migration: SQLite → PostgreSQL
+
+#### Step 1: Export from SQLite
+
+```bash
+cd /path/to/your/project
+
+# Export all data to JSON
+npx sqlew db:export migration-backup.json
+```
+
+#### Step 2: Prepare PostgreSQL Database
+
+```sql
+-- Connect to PostgreSQL as admin
+psql -U postgres
+
+-- Create database
+CREATE DATABASE sqlew_db WITH ENCODING 'UTF8';
+
+-- Create user with required privileges
+CREATE USER sqlew_user WITH PASSWORD 'your-secure-password';
+GRANT ALL PRIVILEGES ON DATABASE sqlew_db TO sqlew_user;
+
+-- Connect to the new database and grant schema privileges
+\c sqlew_db
+GRANT ALL ON SCHEMA public TO sqlew_user;
+```
+
+#### Step 3: Configure config.toml for PostgreSQL
+
+Create or edit `.sqlew/config.toml`:
+
+```toml
+[database]
+type = "postgres"
+
+[database.connection]
+host = "localhost"
+port = 5432
+database = "sqlew_db"
+
+[database.auth]
+type = "direct"
+user = "sqlew_user"
+password = "your-secure-password"
+
+[project]
+name = "your-project-name"
+```
+
+#### Step 4: Import to PostgreSQL
+
+```bash
+# Import data to PostgreSQL
+npx sqlew db:import migration-backup.json
+```
+
+#### Step 5: Verify Migration
+
+```bash
+# Test MCP server connection
+npx sqlew --config-path=.sqlew/config.toml
+```
+
+---
+
+### Migration: MySQL → PostgreSQL
+
+#### Step 1: Export from MySQL
+
+First, configure config.toml to connect to MySQL:
+
+```toml
+[database]
+type = "mysql"
+
+[database.connection]
+host = "localhost"
+port = 3306
+database = "sqlew_db"
+
+[database.auth]
+type = "direct"
+user = "sqlew_user"
+password = "mysql-password"
+```
+
+Then export:
+
+```bash
+npx sqlew db:export migration-backup.json
+```
+
+#### Step 2: Prepare PostgreSQL Database
+
+```sql
+-- Connect to PostgreSQL as admin
+psql -U postgres
+
+-- Create database
+CREATE DATABASE sqlew_db WITH ENCODING 'UTF8';
+
+-- Create user
+CREATE USER sqlew_user WITH PASSWORD 'postgres-password';
+GRANT ALL PRIVILEGES ON DATABASE sqlew_db TO sqlew_user;
+\c sqlew_db
+GRANT ALL ON SCHEMA public TO sqlew_user;
+```
+
+#### Step 3: Update config.toml for PostgreSQL
+
+```toml
+[database]
+type = "postgres"
+
+[database.connection]
+host = "localhost"
+port = 5432
+database = "sqlew_db"
+
+[database.auth]
+type = "direct"
+user = "sqlew_user"
+password = "postgres-password"
+
+[project]
+name = "your-project-name"
+```
+
+#### Step 4: Import to PostgreSQL
+
+```bash
+npx sqlew db:import migration-backup.json
+```
+
+---
+
+### Post-Migration Verification
+
+After any migration, verify your data:
+
+#### 1. Check Row Counts
+
+Use your database client to compare row counts:
+
+```sql
+-- Count decisions
+SELECT COUNT(*) FROM v4_decisions;
+
+-- Count tasks
+SELECT COUNT(*) FROM v4_tasks;
+
+-- Count file changes
+SELECT COUNT(*) FROM v4_file_changes;
+```
+
+#### 2. Test MCP Server
+
+```bash
+# Start MCP server with new config
+npx sqlew
+
+# Or use MCP Inspector for interactive testing
+npx @modelcontextprotocol/inspector npx sqlew
+```
+
+#### 3. Verify in Claude Code
+
+Update your `.mcp.json` to use the new database:
+
+```json
+{
+  "mcpServers": {
+    "sqlew": {
+      "command": "npx",
+      "args": ["sqlew", "--config-path", "/path/to/.sqlew/config.toml"]
+    }
+  }
+}
+```
+
+---
+
+### Troubleshooting
+
+#### Connection Refused
+
+```
+Error: connect ECONNREFUSED 127.0.0.1:3306
+```
+
+**Solution**: Ensure the database server is running and accepting connections on the specified port.
+
+#### Authentication Failed
 
-# Import to MySQL
-node node_modules/sqlew/dist/cli.js db:import --source=data.json --db-path=mysql://localhost/sqlew
 ```
+Error: Access denied for user 'sqlew_user'@'localhost'
+```
+
+**Solution**: Verify username and password in config.toml. Check that the user has proper privileges.
+
+#### Database Does Not Exist
+
+```
+Error: Unknown database 'sqlew_db'
+```
+
+**Solution**: Create the database first (see Step 2 in migration guides above).
+
+#### Permission Denied
+
+```
+Error: permission denied for schema public
+```
+
+**Solution**: Grant schema privileges to the user:
+```sql
+-- PostgreSQL
+GRANT ALL ON SCHEMA public TO sqlew_user;
+```
+
+#### Import Skipped (Project Exists)
+
+```
+Project "my-project" already exists in target database
+```
+
+**Solution**: Use `--project-name` to specify a different name, or manually delete the existing project from the target database.
+
+---
 
 ## Error Handling
 
@@ -328,10 +629,10 @@ Solution: Ensure all referenced entities exist in export
 
 ### Dry-Run Validation
 
-Always test imports with `--dry-run` first:
+Always test imports with `dry-run=true` first:
 
 ```bash
-node node_modules/sqlew/dist/cli.js db:import --source=data.json --dry-run
+npx sqlew db:import data.json dry-run=true
 ```
 
 This validates:
@@ -365,31 +666,30 @@ This validates:
 
 ## Comparison with db:dump
 
-| Feature            | db:export (JSON)         | db:dump (SQL)                            |
-|--------------------|--------------------------|------------------------------------------|
-| Format             | JSON data only           | SQL DDL + data                           |
-| Schema             | Not included             | Full schema included                     |
-| Use Case           | Project migration        | **Backup/restore, database replication** |
-| Cross-DB           | ✅ Yes                    | ❌ No (dialect-specific)                  |
-| Size               | Smaller (~40% reduction) | Larger (includes schema)                 |
-| Import Speed       | Slower (ID remapping)    | Faster (direct SQL execution)            |
-| Conflict Handling  | Smart deduplication      | Overwrite or fail                        |
-| Restore Capability | ❌ Skips if exists        | ✅ Full restore                           |
-
-**When to use db:export (JSON)**:
-
-- Migrating projects between different sqlew databases
-- Sharing specific projects with team members
-- Merging multiple projects into one database
-- Cross-database migration (SQLite → MySQL → PostgreSQL)
-
-**When to use db:dump (SQL)** - **RECOMMENDED FOR BACKUP**:
-
-- **Full database backup with schema** ✅
-- **Database restore/recovery** ✅
-- Database replication
-- Development → Production deployment
-- Same database type migration
+| Feature            | db:export (JSON)                     | db:dump (SQL)                         |
+|--------------------|--------------------------------------|---------------------------------------|
+| Format             | JSON data only                       | SQL DDL + data                        |
+| Schema             | Not included                         | Full schema included                  |
+| Use Case           | **Cross-DB migration**, sharing      | **Same-DB backup/restore**            |
+| Cross-DB           | ✅ **Yes (ONLY option for cross-DB)** | ❌ No (v4.0.2+ same-DB only)           |
+| Size               | Smaller (~40% reduction)             | Larger (includes schema)              |
+| Import Speed       | Slower (ID remapping)                | Faster (direct SQL execution)         |
+| Conflict Handling  | Smart deduplication                  | Overwrite or fail                     |
+| Restore Capability | ❌ Skips if exists                    | ✅ Full restore                        |
+
+**When to use db:export (JSON)** - **REQUIRED FOR CROSS-DATABASE**:
+
+- ✅ **Cross-database migration** (SQLite → MySQL → PostgreSQL) - **ONLY option**
+- ✅ Migrating projects between different sqlew databases
+- ✅ Sharing specific projects with team members
+- ✅ Merging multiple projects into one database
+
+**When to use db:dump (SQL)** - **SAME-DATABASE ONLY**:
+
+- ✅ **Full database backup with schema** (same DB type)
+- ✅ **Database restore/recovery** (same DB type)
+- ✅ Database replication (same DB type)
+- ❌ Cross-database migration (use JSON instead)
 
 See [Database Migration Guide](DATABASE_MIGRATION.md) for complete `db:dump` documentation.