sqlew 3.6.9 → 3.7.0

package/docs/DATABASE_AUTH.md

@@ -0,0 +1,445 @@
+# Database Authentication Configuration
+
+This document describes the authentication configuration system for multi-database support in mcp-sqlew v3.7.0+.
+
+## Overview
+
+The configuration system supports multiple database backends with various authentication methods:
+
+- **SQLite**: Local file-based database (default, no authentication)
+- **PostgreSQL**: Remote PostgreSQL with direct authentication or IAM authentication (planned)
+- **MySQL**: Remote MySQL with direct authentication or IAM authentication (planned)
+
+**Note**: SSH tunneling is NOT supported by this software. Users must set up SSH tunnels manually using the `ssh` command and connect to localhost.
+
+## Configuration Structure
+
+All configuration is defined in `.sqlew/config.toml` under the `[database]` section.
+
+### SQLite Configuration (Default)
+
+The simplest configuration for local development:
+
+```toml
+[database]
+path = ".sqlew/sqlew.db"
+```
+
+### PostgreSQL/MySQL Configuration
+
+Remote databases require three main sections:
+
+1. **Database Type**: Specifies the database system
+2. **Connection**: Server address and database name
+3. **Authentication**: Credentials and authentication method
+
+## Configuration Sections
+
+### 1. Database Type
+
+```toml
+[database]
+type = "postgres"  # or "mysql" or "sqlite"
+```
+
+Valid values: `sqlite`, `postgres`, `mysql`
+
+### 2. Connection Configuration
+
+Required for PostgreSQL and MySQL:
+
+```toml
+[database.connection]
+host = "localhost"          # Database server hostname or IP
+port = 5432                 # PostgreSQL: 5432, MySQL: 3306
+database = "sqlew"          # Database name
+```
+
+**Fields**:
+- `host` (required): Database server hostname or IP address
+- `port` (required): Database server port (1-65535)
+- `database` (required): Target database name
+
+### 3. Authentication Configuration
+
+```toml
+[database.auth]
+type = "direct"             # Authentication method
+user = "postgres"           # Database username
+password = "your-password"  # Database password
+```
+
+**Fields**:
+- `type` (required): Authentication method
+  - `direct`: Standard username/password (or localhost connection through manual SSH tunnel)
+  - `aws-iam`: AWS RDS IAM authentication (planned)
+  - `gcp-iam`: GCP Cloud SQL IAM authentication (planned)
+- `user` (optional): Database username
+- `password` (optional): Database password
+
+**Note**: For databases behind bastion hosts, users must set up SSH tunnels manually using the `ssh -L` command, then connect to localhost.
+
+### 4. SSL/TLS Configuration (Optional)
+
+For encrypted database connections:
+
+```toml
+[database.auth.ssl]
+ca = "/path/to/ca-cert.pem"           # CA certificate
+cert = "/path/to/client-cert.pem"     # Client certificate (mutual TLS)
+key = "/path/to/client-key.pem"       # Client private key (mutual TLS)
+rejectUnauthorized = true             # Reject self-signed certificates
+```
+
+**Fields**:
+- `ca` (optional): Certificate Authority certificate (file path or content)
+- `cert` (optional): Client certificate for mutual TLS (file path or content)
+- `key` (optional): Client private key for mutual TLS (file path or content)
+- `rejectUnauthorized` (optional): Whether to reject unauthorized certificates (default: `true`)
+
+**Security Note**: Setting `rejectUnauthorized = false` is not recommended for production.
+
+### 5. Manual SSH Tunnel Setup (For Databases Behind Bastion Hosts)
+
+**SSH tunneling is NOT supported by this software.** Users must set up SSH tunnels manually before connecting.
+
+**Manual Tunnel Setup:**
+
+```bash
+# Example: Connect to database behind bastion host
+ssh -L 3307:db.internal.example.com:3306 user@bastion.example.com
+
+# Then configure sqlew to connect to localhost:
+```
+
+```toml
+[database]
+type = "mysql"
+
+[database.connection]
+host = "localhost"      # Connect to tunnel endpoint
+port = 3307             # Forwarded port (not 3306!)
+database = "sqlew_db"
+
+[database.auth]
+type = "direct"
+user = "mysql_user"
+password = "db-password"
+```
+
+**SSH Tunnel Command Options:**
+- `-L localport:remotehost:remoteport`: Port forwarding specification
+- `-N`: Don't execute remote command (tunnel only)
+- `-f`: Run in background
+- `-o ServerAliveInterval=60`: Keep connection alive
+
+**Example with background tunnel:**
+```bash
+ssh -f -N -L 3307:db.internal:3306 user@bastion.example.com
+```
+
+## Complete Examples
+
+### Example 1: PostgreSQL with Direct Authentication
+
+```toml
+[database]
+type = "postgres"
+
+[database.connection]
+host = "db.example.com"
+port = 5432
+database = "sqlew_production"
+
+[database.auth]
+type = "direct"
+user = "postgres"
+password = "secure-password"
+```
+
+### Example 2: PostgreSQL with SSL/TLS
+
+```toml
+[database]
+type = "postgres"
+
+[database.connection]
+host = "db.example.com"
+port = 5432
+database = "sqlew_production"
+
+[database.auth]
+type = "direct"
+user = "postgres"
+password = "secure-password"
+
+[database.auth.ssl]
+ca = "/path/to/ca-cert.pem"
+rejectUnauthorized = true
+```
+
+### Example 3: PostgreSQL via Manual SSH Tunnel
+
+**Step 1**: Set up SSH tunnel manually:
+```bash
+ssh -L 5433:db.internal.example.com:5432 deploy@bastion.example.com
+```
+
+**Step 2**: Configure sqlew to use localhost:
+```toml
+[database]
+type = "postgres"
+
+[database.connection]
+host = "localhost"       # Tunnel endpoint
+port = 5433              # Forwarded port (not 5432!)
+database = "sqlew_production"
+
+[database.auth]
+type = "direct"
+user = "postgres"
+password = "db-password"
+```
+
+### Example 4: MySQL via Manual SSH Tunnel with SSL
+
+**Step 1**: Set up SSH tunnel manually:
+```bash
+ssh -f -N -L 53306:mysql.internal.example.com:3306 deploy@jump.example.com
+```
+
+**Step 2**: Configure sqlew with SSL:
+```toml
+[database]
+type = "mysql"
+
+[database.connection]
+host = "localhost"       # Tunnel endpoint
+port = 53306             # Forwarded port
+database = "sqlew_db"
+
+[database.auth]
+type = "direct"
+user = "mysql_user"
+password = "db-password"
+
+[database.auth.ssl]
+ca = "/path/to/ca.pem"
+rejectUnauthorized = true
+```
+
+### Example 5: MySQL with Inline SSL Certificate
+
+```toml
+[database]
+type = "mysql"
+
+[database.connection]
+host = "db.example.com"
+port = 3306
+database = "sqlew_db"
+
+[database.auth]
+type = "direct"
+user = "mysql_user"
+password = "db-password"
+
+[database.auth.ssl]
+ca = """-----BEGIN CERTIFICATE-----
+MIIEGzCCAwOgAwIBAgIQDKU...
+...certificate content...
+-----END CERTIFICATE-----"""
+rejectUnauthorized = true
+```
+
+## Validation Rules
+
+The configuration loader validates all settings and provides clear error messages:
+
+### Database Type Validation
+- Must be one of: `sqlite`, `postgres`, `mysql`
+
+### Connection Validation
+- Required for PostgreSQL and MySQL
+- `host`: Must be specified
+- `port`: Must be between 1 and 65535
+- `database`: Must be specified
+
+### Authentication Validation
+
+**Direct Authentication**:
+- `user`: Required
+- `password`: Required
+
+### SSL Validation
+- `rejectUnauthorized`: Must be a boolean (if specified)
+
+## Default Values
+
+The configuration loader applies sensible defaults:
+
+### SSL Defaults
+- `rejectUnauthorized`: true
+
+## Error Handling
+
+### Configuration Parsing Errors
+
+If the TOML file is malformed:
+
+```
+⚠️  Failed to load config file: .sqlew/config.toml
+   Error: Unexpected character at line 5
+   Using default configuration
+```
+
+### Validation Errors
+
+If configuration is invalid:
+
+```
+⚠️  Configuration validation failed: .sqlew/config.toml
+   - database.connection.host is required
+   - database.auth.user is required for direct authentication
+   Using default configuration
+```
+
+## Loading Configuration
+
+The configuration is loaded automatically when the MCP server starts. You can also load it programmatically:
+
+```typescript
+import { loadConfigFile, validateDatabaseConfig } from './config/loader.js';
+
+// Load and validate
+const config = loadConfigFile('.sqlew/config.toml');
+const validation = validateDatabaseConfig(config.database);
+
+if (!validation.valid) {
+  console.error('Invalid configuration:', validation.errors);
+}
+```
+
+## Backward Compatibility
+
+The new authentication system is fully backward compatible:
+
+- Existing SQLite configurations continue to work
+- The `path` field in `[database]` is still supported
+- If no configuration is provided, SQLite with default path is used
+
+## Security Best Practices
+
+1. **Use Private Keys**: Prefer SSH private key authentication over passwords
+2. **Enable SSL**: Use SSL/TLS for all production database connections
+3. **Validate Certificates**: Keep `rejectUnauthorized = true` in production
+4. **Secure Credentials**: Never commit config.toml with passwords to version control
+5. **Restrict Permissions**: Set appropriate file permissions on private keys (chmod 600)
+6. **Environment Variables**: Consider using environment variables for sensitive data (future enhancement)
+
+## Troubleshooting
+
+### Connection Failures
+
+**PostgreSQL connection refused**:
+- Check that PostgreSQL is running
+- Verify host and port are correct
+- Ensure firewall allows connections
+- Check pg_hba.conf for authentication settings
+
+**Manual SSH tunnel issues**:
+- Run `ssh -v -L ...` for verbose debugging
+- Check SSH key permissions (`chmod 600 ~/.ssh/id_rsa`)
+- Verify bastion host is reachable
+- Ensure SSH port (usually 22) is open
+- Check if local port is already in use (`netstat -an | grep <port>`)
+
+**SSL certificate errors**:
+- Verify CA certificate path is correct
+- Check certificate is not expired
+- Ensure certificate chain is complete
+- For development, temporarily set `rejectUnauthorized = false` (not for production)
+
+### Configuration Validation
+
+Use the test suite to validate your configuration:
+
+```bash
+npx tsc
+node dist/tests/config-loader.test.js
+```
+
+## API Reference
+
+### Functions
+
+#### `loadConfigFile(configPath?: string): SqlewConfig`
+
+Loads and parses configuration from TOML file.
+
+**Parameters**:
+- `configPath` (optional): Path to config file (default: `.sqlew/config.toml`)
+
+**Returns**: Parsed and validated configuration
+
+**Example**:
+```typescript
+const config = loadConfigFile('.sqlew/config.toml');
+```
+
+#### `validateDatabaseConfig(config: DatabaseConfig): { valid: boolean; errors: string[] }`
+
+Validates database configuration.
+
+**Parameters**:
+- `config`: Database configuration object
+
+**Returns**: Validation result with errors if any
+
+**Example**:
+```typescript
+const validation = validateDatabaseConfig(config.database);
+if (!validation.valid) {
+  console.error('Validation errors:', validation.errors);
+}
+```
+
+#### `normalizeDatabaseConfig(config: DatabaseConfig): DatabaseConfig`
+
+Normalizes database configuration by applying defaults.
+
+**Parameters**:
+- `config`: Partial database configuration
+
+**Returns**: Complete database configuration with defaults
+
+**Example**:
+```typescript
+const normalized = normalizeDatabaseConfig({
+  type: 'postgres',
+  connection: { host: 'localhost', port: 5432, database: 'test' },
+  auth: {
+    type: 'direct',
+    user: 'postgres',
+    password: 'pass',
+    ssl: { ca: '/path/to/ca.pem' }
+  }
+});
+// normalized.auth.ssl.rejectUnauthorized === true (default applied)
+```
+
+## Future Enhancements
+
+Planned authentication methods:
+
+- **AWS RDS IAM Authentication**: Token-based authentication for AWS RDS
+- **GCP Cloud SQL IAM Authentication**: Token-based authentication for GCP Cloud SQL
+- **Environment Variable Substitution**: Reference environment variables in config
+- **Credential Encryption**: Encrypt sensitive fields in config file
+
+## Related Documentation
+
+- [Configuration Guide](./CONFIGURATION.md)
+- [Database Setup](./DATABASE_SETUP.md)
+- [Architecture Overview](../ARCHITECTURE.md)
+- [Type Definitions](../src/config/types.ts)

package/docs/DATABASE_MIGRATION.md

@@ -0,0 +1,247 @@
+# Database Migration Guide
+
+This guide explains how to generate complete SQL dumps from your mcp-sqlew database for use with SQLite, MySQL/MariaDB, or PostgreSQL.
+
+## Overview
+
+The `db:dump` CLI tool generates complete SQL dumps (CREATE TABLE + INSERT statements) compatible with SQLite, MySQL/MariaDB, and PostgreSQL formats. The generated SQL can be imported directly into an empty database without additional setup.
+
+## Usage
+
+### Basic Syntax
+
+```bash
+npx sqlew db:dump [--from <source>] --format <target> [options]
+```
+
+**Parameters:**
+- `--from <source>`: Source database (sqlite, mysql, postgresql). Default: sqlite
+- `--format <target>`: Target SQL format (sqlite, mysql, postgresql). **Required**
+- `--output <file>`: Output file path (omit for stdout)
+- `--tables <list>`: Comma-separated list of specific tables to dump
+- `--chunk-size <n>`: Rows per INSERT statement (default: 100)
+- `--on-conflict <mode>`: Duplicate key handling (error, ignore, replace). Default: error
+- `--exclude-schema`: Exclude CREATE TABLE statements (data-only dump)
+- `--db-path <path>`: SQLite database path (default: .sqlew/sqlew.db)
+
+**Note:** By default, the dump includes both schema (CREATE TABLE) and data (INSERT) for complete migration.
+
+### Generate SQL Dumps
+
+**From SQLite (default):**
+```bash
+npx sqlew db:dump --format mysql --output dump-mysql.sql
+npx sqlew db:dump --format postgresql --output dump-pg.sql
+```
+
+**From MySQL:**
+```bash
+# Configure connection via environment variables
+export MYSQL_HOST=127.0.0.1
+export MYSQL_PORT=3306
+export MYSQL_USER=youruser
+export MYSQL_PASSWORD=yourpass
+export MYSQL_DATABASE=mcp_context
+
+npx sqlew db:dump --from mysql --format sqlite --output dump-sqlite.sql
+npx sqlew db:dump --from mysql --format postgresql --output dump-pg.sql
+```
+
+**From PostgreSQL:**
+```bash
+# Configure connection via environment variables
+export PG_HOST=localhost
+export PG_PORT=5432
+export PG_USER=postgres
+export PG_PASSWORD=yourpass
+export PG_DATABASE=mcp_context
+
+npx sqlew db:dump --from postgresql --format sqlite --output dump-sqlite.sql
+npx sqlew db:dump --from postgresql --format mysql --output dump-mysql.sql
+```
+
+### Selective Table Export
+
+Export only specific tables:
+
+```bash
+npx sqlew db:dump --format mysql --tables t_decisions,t_tasks,m_agents --output partial.sql
+```
+
+### Custom Chunk Size
+
+For large tables, adjust INSERT batch size:
+
+```bash
+npx sqlew db:dump --format mysql --chunk-size 500 --output dump.sql
+```
+
+### Data-Only Dumps
+
+For advanced use cases where you manage schema separately:
+
+```bash
+npx sqlew db:dump --format mysql --exclude-schema --output data-only.sql
+```
+
+This generates INSERT statements without CREATE TABLE, useful for:
+- Importing into databases with existing schema
+- Backup/restore of data only
+- Data transfer between identical schemas
+
+### Conflict Resolution
+
+Handle duplicate keys when importing into existing databases:
+
+```bash
+# Ignore duplicates (safe for adding new data)
+npx sqlew db:dump --format mysql --on-conflict ignore --output dump.sql
+
+# Update existing rows (sync/overwrite mode)
+npx sqlew db:dump --format mysql --on-conflict replace --output dump.sql
+
+# Fail on duplicates (default, strict mode)
+npx sqlew db:dump --format mysql --on-conflict error --output dump.sql
+```
+
+**Modes:**
+- `error` (default): Standard INSERT, fails if duplicate keys exist
+- `ignore`: Skip duplicate rows (INSERT IGNORE / ON CONFLICT DO NOTHING)
+- `replace`: Update existing rows with new values (ON DUPLICATE KEY UPDATE / ON CONFLICT DO UPDATE)
+
+**Use cases:**
+- `ignore`: Importing into a database that may already have some data
+- `replace`: Synchronizing data from one database to another
+- `error`: Fresh database migration where duplicates indicate a problem
+
+---
+
+## Supported Migration Paths
+
+The `db:dump` tool supports **all 6 migration paths** between SQLite, MySQL/MariaDB, and PostgreSQL:
+
+| Source | Target | Command |
+|--------|--------|---------|
+| SQLite | MySQL | `--format mysql` |
+| SQLite | PostgreSQL | `--format postgresql` |
+| MySQL | SQLite | `--from mysql --format sqlite` |
+| PostgreSQL | SQLite | `--from postgresql --format sqlite` |
+| MySQL | PostgreSQL | `--from mysql --format postgresql` |
+| PostgreSQL | MySQL | `--from postgresql --format mysql` |
+
+---
+
+## Data Type Mappings
+
+The tool automatically handles database-specific data type conversions:
+
+### Boolean Values
+
+| SQLite | MySQL | PostgreSQL |
+|--------|-------|------------|
+| 0/1 | 0/1 (TINYINT) | FALSE/TRUE |
+
+### Binary Data
+
+| SQLite | MySQL | PostgreSQL |
+|--------|-------|------------|
+| BLOB (hex) | X'hex' (BLOB) | '\xhex'::bytea |
+
+### Identifiers
+
+| SQLite | MySQL | PostgreSQL |
+|--------|-------|------------|
+| "table" | \`table\` | "table" |
+
+### Text and Numeric Types
+
+All databases handle TEXT, VARCHAR, INTEGER, and REAL types consistently. The tool preserves:
+- Unix epoch timestamps (stored as INTEGER)
+- UTF-8 text encoding
+- NULL values
+
+---
+
+## Importing Generated SQL
+
+The generated SQL is complete and self-contained. Import it directly into an **empty database**:
+
+**SQLite:**
+```bash
+sqlite3 your-database.db < dump-sqlite.sql
+```
+
+**MySQL/MariaDB:**
+```bash
+# Create empty database first
+mysql -e "CREATE DATABASE mydb CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"
+# Import dump
+mysql mydb < dump-mysql.sql
+```
+
+**PostgreSQL:**
+```bash
+# Create empty database first
+createdb mydb
+# Import dump
+psql -d mydb -f dump-pg.sql
+```
+
+**For existing databases with data:**
+Use `--on-conflict ignore` or `--on-conflict replace` when generating the dump (see Conflict Resolution section above).
+
+## Transaction Safety
+
+All generated SQL dumps are wrapped in database transactions:
+- **MySQL/MariaDB**: `START TRANSACTION;` ... `COMMIT;`
+- **PostgreSQL**: `BEGIN;` ... `COMMIT;`
+- **SQLite**: `BEGIN TRANSACTION;` ... `COMMIT;`
+
+If the import fails at any point, all changes are automatically rolled back, leaving the database in its original state. This prevents partial imports that would leave the database in an inconsistent state.
+
+**Benefits:**
+- Atomic imports: Either all data is imported successfully or nothing is changed
+- Safe to retry: Failed imports don't leave partial data that needs cleanup
+- Consistent state: Database is never left in an intermediate state
+
+---
+
+## Environment Variables
+
+### MySQL Configuration
+
+```bash
+MYSQL_HOST=127.0.0.1        # Default: 127.0.0.1
+MYSQL_PORT=3306             # Default: 3306
+MYSQL_USER=root             # Default: root
+MYSQL_PASSWORD=             # Default: empty
+MYSQL_DATABASE=mcp_context  # Default: mcp_context
+```
+
+### PostgreSQL Configuration
+
+```bash
+PG_HOST=localhost           # Default: localhost
+PG_PORT=5432                # Default: 5432
+PG_USER=postgres            # Default: postgres
+PG_PASSWORD=                # Default: empty
+PG_DATABASE=mcp_context     # Default: mcp_context
+```
+
+---
+
+## Best Practices
+
+1. **Test on a copy** - Always test migrations on a database copy first
+2. **Create schema first** - Run Knex migrations before importing data
+3. **Review SQL before import** - Inspect generated SQL file for correctness
+4. **Verify row counts** - Compare source and target table row counts after import
+5. **Backup original data** - Keep backups before performing migrations
+
+---
+
+## Support
+
+For issues or questions:
+- GitHub Issues: https://github.com/sin5ddd/mcp-sqlew/issues
+- Documentation: `/docs` directory in repository