sqlew 3.7.3 → 3.7.4

package/docs/{DATABASE_MIGRATION.md → cli/DATABASE_MIGRATION.md}

@@ -1,20 +1,45 @@
 # Database Migration Guide
 
-This guide explains how to generate complete SQL dumps from your mcp-sqlew database for use with SQLite, MySQL/MariaDB, or PostgreSQL.
+This guide explains how to generate complete SQL dumps from your mcp-sqlew database for use with SQLite, MySQL/MariaDB,
+or PostgreSQL.
 
 ## Overview
 
-The `db:dump` CLI tool generates complete SQL dumps (CREATE TABLE + INSERT statements) compatible with SQLite, MySQL/MariaDB, and PostgreSQL formats. The generated SQL can be imported directly into an empty database without additional setup.
+The `db:dump` CLI tool generates complete SQL dumps (CREATE TABLE + INSERT statements) compatible with SQLite,
+MySQL/MariaDB, and PostgreSQL formats. The generated SQL can be imported directly into an empty database without
+additional setup.
+
+## Installation
+
+To use the `db:dump` CLI command, install sqlew in your project:
+
+```bash
+cd /path/to/your/project
+npm install sqlew
+```
+
+**Tip**: Add a shortcut to your `package.json` for convenience:
+
+```json
+{
+  "scripts": {
+    "sqlew": "node node_modules/sqlew/dist/cli.js"
+  }
+}
+```
+
+Then you can use: `npm run sqlew db:dump --format=mysql --output=dump.sql`
 
 ## Usage
 
 ### Basic Syntax
 
 ```bash
-npx sqlew db:dump [--from <source>] --format <target> [options]
+node node_modules/sqlew/dist/cli.js db:dump [--from <source>] --format <target> [options]
 ```
 
 **Parameters:**
+
 - `--from <source>`: Source database (sqlite, mysql, postgresql). Default: sqlite
 - `--format <target>`: Target SQL format (sqlite, mysql, postgresql). **Required**
 - `--output <file>`: Output file path (omit for stdout)
@@ -29,12 +54,14 @@ npx sqlew db:dump [--from <source>] --format <target> [options]
 ### Generate SQL Dumps
 
 **From SQLite (default):**
+
 ```bash
-npx sqlew db:dump --format mysql --output dump-mysql.sql
-npx sqlew db:dump --format postgresql --output dump-pg.sql
+node node_modules/sqlew/dist/cli.js db:dump --format mysql --output dump-mysql.sql
+node node_modules/sqlew/dist/cli.js db:dump --format postgresql --output dump-pg.sql
 ```
 
 **From MySQL:**
+
 ```bash
 # Configure connection via environment variables
 export MYSQL_HOST=127.0.0.1
@@ -43,11 +70,12 @@ export MYSQL_USER=youruser
 export MYSQL_PASSWORD=yourpass
 export MYSQL_DATABASE=mcp_context
 
-npx sqlew db:dump --from mysql --format sqlite --output dump-sqlite.sql
-npx sqlew db:dump --from mysql --format postgresql --output dump-pg.sql
+node node_modules/sqlew/dist/cli.js db:dump --from mysql --format sqlite --output dump-sqlite.sql
+node node_modules/sqlew/dist/cli.js db:dump --from mysql --format postgresql --output dump-pg.sql
 ```
 
 **From PostgreSQL:**
+
 ```bash
 # Configure connection via environment variables
 export PG_HOST=localhost
@@ -56,8 +84,8 @@ export PG_USER=postgres
 export PG_PASSWORD=yourpass
 export PG_DATABASE=mcp_context
 
-npx sqlew db:dump --from postgresql --format sqlite --output dump-sqlite.sql
-npx sqlew db:dump --from postgresql --format mysql --output dump-mysql.sql
+node node_modules/sqlew/dist/cli.js db:dump --from postgresql --format sqlite --output dump-sqlite.sql
+node node_modules/sqlew/dist/cli.js db:dump --from postgresql --format mysql --output dump-mysql.sql
 ```
 
 ### Selective Table Export
@@ -65,7 +93,7 @@ npx sqlew db:dump --from postgresql --format mysql --output dump-mysql.sql
 Export only specific tables:
 
 ```bash
-npx sqlew db:dump --format mysql --tables t_decisions,t_tasks,m_agents --output partial.sql
+node node_modules/sqlew/dist/cli.js db:dump --format mysql --tables t_decisions,t_tasks,m_agents --output partial.sql
 ```
 
 ### Custom Chunk Size
@@ -73,7 +101,7 @@ npx sqlew db:dump --format mysql --tables t_decisions,t_tasks,m_agents --output
 For large tables, adjust INSERT batch size:
 
 ```bash
-npx sqlew db:dump --format mysql --chunk-size 500 --output dump.sql
+node node_modules/sqlew/dist/cli.js db:dump --format mysql --chunk-size 500 --output dump.sql
 ```
 
 ### Data-Only Dumps
@@ -81,10 +109,11 @@ npx sqlew db:dump --format mysql --chunk-size 500 --output dump.sql
 For advanced use cases where you manage schema separately:
 
 ```bash
-npx sqlew db:dump --format mysql --exclude-schema --output data-only.sql
+node node_modules/sqlew/dist/cli.js db:dump --format mysql --exclude-schema --output data-only.sql
 ```
 
 This generates INSERT statements without CREATE TABLE, useful for:
+
 - Importing into databases with existing schema
 - Backup/restore of data only
 - Data transfer between identical schemas
@@ -95,21 +124,23 @@ Handle duplicate keys when importing into existing databases:
 
 ```bash
 # Ignore duplicates (safe for adding new data)
-npx sqlew db:dump --format mysql --on-conflict ignore --output dump.sql
+node node_modules/sqlew/dist/cli.js db:dump --format mysql --on-conflict ignore --output dump.sql
 
 # Update existing rows (sync/overwrite mode)
-npx sqlew db:dump --format mysql --on-conflict replace --output dump.sql
+node node_modules/sqlew/dist/cli.js db:dump --format mysql --on-conflict replace --output dump.sql
 
 # Fail on duplicates (default, strict mode)
-npx sqlew db:dump --format mysql --on-conflict error --output dump.sql
+node node_modules/sqlew/dist/cli.js db:dump --format mysql --on-conflict error --output dump.sql
 ```
 
 **Modes:**
+
 - `error` (default): Standard INSERT, fails if duplicate keys exist
 - `ignore`: Skip duplicate rows (INSERT IGNORE / ON CONFLICT DO NOTHING)
 - `replace`: Update existing rows with new values (ON DUPLICATE KEY UPDATE / ON CONFLICT DO UPDATE)
 
 **Use cases:**
+
 - `ignore`: Importing into a database that may already have some data
 - `replace`: Synchronizing data from one database to another
 - `error`: Fresh database migration where duplicates indicate a problem
@@ -120,14 +151,14 @@ npx sqlew db:dump --format mysql --on-conflict error --output dump.sql
 
 The `db:dump` tool supports **all 6 migration paths** between SQLite, MySQL/MariaDB, and PostgreSQL:
 
-| Source | Target | Command |
-|--------|--------|---------|
-| SQLite | MySQL | `--format mysql` |
-| SQLite | PostgreSQL | `--format postgresql` |
-| MySQL | SQLite | `--from mysql --format sqlite` |
-| PostgreSQL | SQLite | `--from postgresql --format sqlite` |
-| MySQL | PostgreSQL | `--from mysql --format postgresql` |
-| PostgreSQL | MySQL | `--from postgresql --format mysql` |
+| Source     | Target     | Command                             |
+|------------|------------|-------------------------------------|
+| SQLite     | MySQL      | `--format mysql`                    |
+| SQLite     | PostgreSQL | `--format postgresql`               |
+| MySQL      | SQLite     | `--from mysql --format sqlite`      |
+| PostgreSQL | SQLite     | `--from postgresql --format sqlite` |
+| MySQL      | PostgreSQL | `--from mysql --format postgresql`  |
+| PostgreSQL | MySQL      | `--from postgresql --format mysql`  |
 
 ---
 
@@ -137,25 +168,26 @@ The tool automatically handles database-specific data type conversions:
 
 ### Boolean Values
 
-| SQLite | MySQL | PostgreSQL |
-|--------|-------|------------|
-| 0/1 | 0/1 (TINYINT) | FALSE/TRUE |
+| SQLite | MySQL         | PostgreSQL |
+|--------|---------------|------------|
+| 0/1    | 0/1 (TINYINT) | FALSE/TRUE |
 
 ### Binary Data
 
-| SQLite | MySQL | PostgreSQL |
-|--------|-------|------------|
+| SQLite     | MySQL         | PostgreSQL     |
+|------------|---------------|----------------|
 | BLOB (hex) | X'hex' (BLOB) | '\xhex'::bytea |
 
 ### Identifiers
 
-| SQLite | MySQL | PostgreSQL |
-|--------|-------|------------|
-| "table" | \`table\` | "table" |
+| SQLite  | MySQL     | PostgreSQL |
+|---------|-----------|------------|
+| "table" | \`table\` | "table"    |
 
 ### Text and Numeric Types
 
 All databases handle TEXT, VARCHAR, INTEGER, and REAL types consistently. The tool preserves:
+
 - Unix epoch timestamps (stored as INTEGER)
 - UTF-8 text encoding
 - NULL values
@@ -167,11 +199,13 @@ All databases handle TEXT, VARCHAR, INTEGER, and REAL types consistently. The to
 The generated SQL is complete and self-contained. Import it directly into an **empty database**:
 
 **SQLite:**
+
 ```bash
 sqlite3 your-database.db < dump-sqlite.sql
 ```
 
 **MySQL/MariaDB:**
+
 ```bash
 # Create empty database first
 mysql -e "CREATE DATABASE mydb CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"
@@ -180,6 +214,7 @@ mysql mydb < dump-mysql.sql
 ```
 
 **PostgreSQL:**
+
 ```bash
 # Create empty database first
 createdb mydb
@@ -193,13 +228,16 @@ Use `--on-conflict ignore` or `--on-conflict replace` when generating the dump (
 ## Transaction Safety
 
 All generated SQL dumps are wrapped in database transactions:
+
 - **MySQL/MariaDB**: `START TRANSACTION;` ... `COMMIT;`
 - **PostgreSQL**: `BEGIN;` ... `COMMIT;`
 - **SQLite**: `BEGIN TRANSACTION;` ... `COMMIT;`
 
-If the import fails at any point, all changes are automatically rolled back, leaving the database in its original state. This prevents partial imports that would leave the database in an inconsistent state.
+If the import fails at any point, all changes are automatically rolled back, leaving the database in its original state.
+This prevents partial imports that would leave the database in an inconsistent state.
 
 **Benefits:**
+
 - Atomic imports: Either all data is imported successfully or nothing is changed
 - Safe to retry: Failed imports don't leave partial data that needs cleanup
 - Consistent state: Database is never left in an intermediate state
@@ -243,5 +281,6 @@ PG_DATABASE=mcp_context     # Default: mcp_context
 ## Support
 
 For issues or questions:
+
 - GitHub Issues: https://github.com/sin5ddd/mcp-sqlew/issues
 - Documentation: `/docs` directory in repository

package/docs/cli/DATA_EXPORT_IMPORT.md

@@ -0,0 +1,400 @@
+# Data Export/Import
+
+> JSON-based project data migration for sharing and multi-database workflows
+
+## Overview
+
+sqlew provides a complete JSON export/import system for migrating project data between databases. This is useful for:
+
+- **Project Sharing** - Share context with team members or between machines
+- **Database Migration** - Move projects to different sqlew databases (different machine, different DB type)
+- **Multi-Project Consolidation** - Merge multiple project exports into one database
+
+**⚠️ Important**: Import skips if project name already exists (default: `--skip-if-exists=true`). This is **NOT a
+backup/restore solution**.
+
+**For backup/restore, use `db:dump` instead**: See [Database Migration Guide](DATABASE_MIGRATION.md) for full backup
+solutions including schema + data using SQL dumps.
+
+## Quick Start
+
+### Export a Project
+
+```bash
+# You have to install sqlew via npm to use CLI mode
+cd /path/to/your/project
+npm install sqlew
+
+# Export all projects
+node node_modules/sqlew/dist/cli.js db:export --output=full-backup.json
+
+# or Export specific project to file
+node node_modules/sqlew/dist/cli.js db:export --project=my-project --output=backup.json
+
+# Export to stdout (pipe to another command)
+node node_modules/sqlew/dist/cli.js db:export --project=visualizer
+```
+
+### Import a Project
+
+```bash
+# You have to install sqlew via npm to use CLI mode (We recommend install per project)
+cd /path/to/your/other-project
+npm install sqlew
+
+# Import from JSON export
+node node_modules/sqlew/dist/cli.js db:import --source=backup.json
+
+# Import with custom project name
+node node_modules/sqlew/dist/cli.js db:import --source=backup.json --project-name=new-name
+
+# Dry-run validation (no actual import)
+node node_modules/sqlew/dist/cli.js db:import --source=backup.json --dry-run
+```
+
+## Export Command
+
+### Syntax
+
+```bash
+node node_modules/sqlew/dist/cli.js db:export [options]
+```
+
+### Options
+
+| Option             | Description                     | Default           |
+|--------------------|---------------------------------|-------------------|
+| `--project <name>` | Export specific project by name | All projects      |
+| `--output <file>`  | Output file path                | stdout            |
+| `--db-path <path>` | Database file path              | `.sqlew/sqlew.db` |
+| `--config <path>`  | Config file path                | Auto-detect       |
+
+### What Gets Exported
+
+- **Master Tables** (filtered to used entries only):
+    - Agents (m_agents)
+    - Context keys (m_context_keys)
+    - Files (m_files)
+    - Tags (m_tags)
+    - Scopes (m_scopes)
+    - Layers (m_layers)
+    - Project metadata (m_projects)
+
+- **Transaction Tables** (all data for selected project):
+    - Decisions with context (t_decisions, t_decision_context)
+    - Tasks with details (t_tasks, t_task_details)
+    - File changes (t_file_changes)
+    - Constraints (t_constraints)
+
+- **Junction Tables** (relationships):
+    - Task tags, file links, decision links, dependencies
+    - Decision tags, scopes
+    - Constraint tags
+
+## Import Command
+
+### Syntax
+
+```bash
+node node_modules/sqlew/dist/cli.js db:import --source=<file> [options]
+```
+
+### Options
+
+| Option                  | Description                   | Default            |
+|-------------------------|-------------------------------|--------------------|
+| `--source <file>`       | JSON export file path         | **Required**       |
+| `--project-name <name>` | Target project name           | Use name from JSON |
+| `--skip-if-exists`      | Skip import if project exists | `true`             |
+| `--dry-run`             | Validate only, don't import   | `false`            |
+| `--db-path <path>`      | Database file path            | `.sqlew/sqlew.db`  |
+| `--config <path>`       | Config file path              | Auto-detect        |
+
+### Import Process
+
+1. **Validation** - Checks JSON format, required fields, data types
+2. **Conflict Detection** - Checks if project name already exists
+3. **Topological Sort** - Resolves task dependencies (circular detection)
+4. **ID Remapping** - Creates new IDs for all imported data
+5. **Master Table Merge** - Reuses existing agents/tags/files by name
+6. **Transaction Import** - Imports with fresh IDs and translated foreign keys
+7. **Junction Table Import** - Restores all relationships
+8. **Validation** - Verifies all foreign key references
+
+### Smart Features
+
+**ID Remapping**:
+
+- All imported data gets fresh auto-incremented IDs
+- Original IDs are mapped to new IDs during transaction
+- Foreign key references automatically updated
+
+**Master Table Deduplication**:
+
+- Agents, tags, scopes, files reused if they already exist (by name/path)
+- Prevents duplicate entries for common metadata
+- Project-scoped master tables (m_files, m_tags, m_scopes) isolated by project_id
+
+**Dependency Resolution**:
+
+- Task dependencies sorted topologically
+- Circular dependency detection prevents invalid imports
+- Dependencies always imported before dependents
+
+**Transaction Safety**:
+
+- All-or-nothing semantics (full rollback on any error)
+- Idempotent operations (safe to retry on failure)
+- Comprehensive error messages with validation details
+
+## Installation for CLI Usage
+
+For users who need to use export/import commands, install sqlew per-project:
+
+```bash
+# Install in your project directory
+cd /path/to/your/project
+npm install sqlew
+
+# Now you can use CLI commands
+node node_modules/sqlew/dist/cli.js db:export --output=backup.json
+node node_modules/sqlew/dist/cli.js db:import --source=backup.json
+```
+
+**Tip**: Add a shortcut to your `package.json` for convenience:
+
+```json
+{
+  "scripts": {
+    "sqlew": "node node_modules/sqlew/dist/cli.js"
+  }
+}
+```
+
+Then you can use: `npm run sqlew db:export --output=backup.json`
+
+**Note**: The MCP server (`npx sqlew`) and CLI commands are both included in the same `sqlew` package. You only need to
+install once.
+
+## Use Cases
+
+### Multi-Project Single Database (Permission-Constrained Environments)
+
+**Scenario**: You work on multiple projects but don't have permissions to create separate MySQL databases. You want to
+consolidate all project contexts into one shared database.
+
+**Solution**: Use export/import to merge multiple project contexts:
+
+```bash
+# Step 1: Export from each project's SQLite database
+cd ~/project-a
+npm install sqlew
+node node_modules/sqlew/dist/cli.js db:export --project=project-a --output=/tmp/project-a.json
+
+cd ~/project-b
+npm install sqlew
+node node_modules/sqlew/dist/cli.js db:export --project=project-b --output=/tmp/project-b.json
+
+cd ~/project-c
+npm install sqlew
+node node_modules/sqlew/dist/cli.js db:export --project=project-c --output=/tmp/project-c.json
+
+# Step 2: Create shared database and import all projects
+cd ~/shared-database
+npm install sqlew
+
+# Configure to use single MySQL database (edit .sqlew/config.toml)
+# [database]
+# type = "mysql"
+# host = "localhost"
+# port = 3306
+# user = "myuser"
+# password = "mypassword"
+# database = "shared_sqlew_db"
+
+node node_modules/sqlew/dist/cli.js db:import --source=/tmp/project-a.json
+node node_modules/sqlew/dist/cli.js db:import --source=/tmp/project-b.json
+node node_modules/sqlew/dist/cli.js db:import --source=/tmp/project-c.json
+
+# Step 3: Configure each project to use shared database
+# In each project's .mcp.json:
+# {
+#   "mcpServers": {
+#     "sqlew": {
+#       "command": "npx",
+#       "args": ["sqlew", "--config-path=/path/to/shared-database/.sqlew/config.toml"]
+#     }
+#   }
+# }
+```
+
+**Benefits**:
+
+- ✅ Single database for multiple projects (saves database quota)
+- ✅ Cross-project context visibility (search decisions across all projects)
+- ✅ Centralized backup and maintenance
+- ✅ Works with permission-constrained MySQL/PostgreSQL environments
+
+**Trade-offs**:
+
+- ⚠️ All projects share same database connection pool
+- ⚠️ Requires manual config path in each project's .mcp.json
+- ⚠️ Project isolation maintained via project_id, not separate databases
+
+### Database Migration (Cross-Machine or Cross-Database)
+
+```bash
+# Export from source database
+node node_modules/sqlew/dist/cli.js db:export --project=main --output=main-export.json
+
+# Import to different database (different machine or different database type)
+# This works because the project doesn't exist in the target database yet
+node node_modules/sqlew/dist/cli.js db:import --source=main-export.json --db-path=/path/to/new/database.db
+```
+
+**Note**: Import skips if project name exists.
+
+**For backup/restore, use `db:dump` instead**:
+
+```bash
+# Backup with SQL dump (preserves schema + data)
+node node_modules/sqlew/dist/cli.js db:dump --format=sqlite --output=backup-$(date +%Y%m%d).sql
+
+# Or simple SQLite file copy
+cp .sqlew/sqlew.db .sqlew/backup-$(date +%Y%m%d).db
+```
+
+See `node node_modules/sqlew/dist/cli.js db:dump --help` for full backup options.
+
+### Project Sharing
+
+```bash
+# Developer A: Export project
+node node_modules/sqlew/dist/cli.js db:export --project=feature-x --output=feature-x.json
+
+# Developer B: Import project
+node node_modules/sqlew/dist/cli.js db:import --source=feature-x.json
+```
+
+### Multi-Project Consolidation
+
+```bash
+# Export from different databases
+node node_modules/sqlew/dist/cli.js db:export --project=visualizer --output=vis.json
+node node_modules/sqlew/dist/cli.js db:export --project=api --output=api.json
+
+# Import to single database
+node node_modules/sqlew/dist/cli.js db:import --source=vis.json
+node node_modules/sqlew/dist/cli.js db:import --source=api.json
+```
+
+### Cross-Database Migration
+
+```bash
+# Export from SQLite
+node node_modules/sqlew/dist/cli.js db:export --output=data.json --db-path=.sqlew/sqlew.db
+
+# Import to MySQL
+node node_modules/sqlew/dist/cli.js db:import --source=data.json --db-path=mysql://localhost/sqlew
+```
+
+## Error Handling
+
+### Common Errors
+
+**Project Already Exists**:
+
+```
+Error: Project "my-project" already exists in target database
+```
+
+Solution: Use `--project-name` to specify different name, or remove existing project
+
+**Circular Dependencies**:
+
+```
+Error: Circular dependency detected in task dependencies
+```
+
+Solution: Fix dependency graph in source database before exporting
+
+**Invalid Foreign Keys**:
+
+```
+Error: Foreign key constraint violation for task_id=123
+```
+
+Solution: Ensure all referenced entities exist in export
+
+### Dry-Run Validation
+
+Always test imports with `--dry-run` first:
+
+```bash
+node node_modules/sqlew/dist/cli.js db:import --source=data.json --dry-run
+```
+
+This validates:
+
+- JSON format and structure
+- Project name conflicts
+- Task dependency cycles
+- Foreign key references
+- Data type correctness
+
+## Technical Details
+
+### Performance
+
+- **Batch Inserts**: 10-row batches to avoid SQLite limits
+- **Transaction Scope**: Single transaction for atomicity
+- **Memory Efficient**: Streams large datasets
+
+### Limitations
+
+- **Max JSON Size**: Limited by available memory
+- **SQLite Batch Limit**: 500 UNION ALL clauses (handled automatically)
+- **Cross-Database**: JSON format only (no SQL dump)
+
+### Data Integrity
+
+- **Foreign Key Validation**: All references validated before insertion
+- **Orphan Cleanup**: Invalid references automatically filtered
+- **View Restoration**: Views temporarily dropped and restored during import
+- **Idempotent Operations**: Safe to retry on network/disk failures
+
+## Comparison with db:dump
+
+| Feature            | db:export (JSON)         | db:dump (SQL)                            |
+|--------------------|--------------------------|------------------------------------------|
+| Format             | JSON data only           | SQL DDL + data                           |
+| Schema             | Not included             | Full schema included                     |
+| Use Case           | Project migration        | **Backup/restore, database replication** |
+| Cross-DB           | ✅ Yes                    | ❌ No (dialect-specific)                  |
+| Size               | Smaller (~40% reduction) | Larger (includes schema)                 |
+| Import Speed       | Slower (ID remapping)    | Faster (direct SQL execution)            |
+| Conflict Handling  | Smart deduplication      | Overwrite or fail                        |
+| Restore Capability | ❌ Skips if exists        | ✅ Full restore                           |
+
+**When to use db:export (JSON)**:
+
+- Migrating projects between different sqlew databases
+- Sharing specific projects with team members
+- Merging multiple projects into one database
+- Cross-database migration (SQLite → MySQL → PostgreSQL)
+
+**When to use db:dump (SQL)** - **RECOMMENDED FOR BACKUP**:
+
+- **Full database backup with schema** ✅
+- **Database restore/recovery** ✅
+- Database replication
+- Development → Production deployment
+- Same database type migration
+
+See [Database Migration Guide](DATABASE_MIGRATION.md) for complete `db:dump` documentation.
+
+## See Also
+
+- [Database Migration](DATABASE_MIGRATION.md) - SQLite → MySQL/PostgreSQL migration
+- [CHANGELOG.md](../../CHANGELOG.md#374) - v3.7.4 release notes
+- [Architecture](../ARCHITECTURE.md) - Technical architecture overview