sqlew 4.0.5 → 4.1.1

package/docs/CROSS_DATABASE.md

@@ -0,0 +1,153 @@
+# Cross-Database Compatibility
+
+sqlew supports multiple database backends with unified query behavior.
+
+## Supported Databases
+
+| Database | Version | Status |
+|----------|---------|--------|
+| SQLite | 3.x | Primary (default) |
+| MySQL | 8.0+ | Supported |
+| MariaDB | 10.5+ | Supported |
+| PostgreSQL | 12+ | Supported |
+
+## Configuration
+
+### SQLite (Default)
+
+```toml
+[database]
+type = "sqlite"
+path = ".sqlew/sqlew.db"
+```
+
+### MySQL / MariaDB
+
+```toml
+[database]
+type = "mysql"  # or "mariadb"
+
+[database.connection]
+host = "localhost"
+port = 3306
+database = "sqlew"
+
+[database.auth]
+type = "direct"
+user = "sqlew_user"
+password = "your_password"
+```
+
+### PostgreSQL
+
+```toml
+[database]
+type = "postgres"
+
+[database.connection]
+host = "localhost"
+port = 5432
+database = "sqlew"
+
+[database.auth]
+type = "direct"
+user = "sqlew_user"
+password = "your_password"
+```
+
+## Database-Specific Differences
+
+### String Aggregation
+
+| Database | Function |
+|----------|----------|
+| SQLite | `GROUP_CONCAT(column)` |
+| MySQL | `GROUP_CONCAT(column)` |
+| MariaDB | `GROUP_CONCAT(column)` |
+| PostgreSQL | `string_agg(column, ',')` |
+
+sqlew automatically detects the database type and uses the appropriate function.
+
+### GROUP BY Strictness
+
+PostgreSQL enforces strict GROUP BY rules:
+- All non-aggregated SELECT columns must appear in GROUP BY
+- MySQL/MariaDB/SQLite are more lenient
+
+sqlew includes all necessary columns in GROUP BY for cross-database compatibility.
+
+### Example Query Difference
+
+**MySQL/SQLite:**
+```sql
+SELECT d.key_id, ck.key_name, GROUP_CONCAT(t.name) as tags
+FROM v4_decisions d
+JOIN v4_context_keys ck ON d.key_id = ck.id
+GROUP BY d.key_id
+```
+
+**PostgreSQL:**
+```sql
+SELECT d.key_id, ck.key_name, string_agg(t.name, ',') as tags
+FROM v4_decisions d
+JOIN v4_context_keys ck ON d.key_id = ck.id
+GROUP BY d.key_id, ck.key_name
+```
+
+## Data Migration
+
+### Cross-Database Migration
+
+Use JSON export/import for migrating between database types:
+
+```bash
+# Export from source database
+sqlew db:export backup.json
+
+# Import to target database (after changing config)
+sqlew db:import backup.json
+```
+
+**Note:** SQL dump (`db:dump`) is for same-database-type operations only.
+
+### Export Options
+
+```bash
+# Export all data
+sqlew db:export full-backup.json
+
+# Export with version tracking
+sqlew db:export --version 4.1.0 backup.json
+```
+
+## Adapter Implementation
+
+Database adapters are located in `src/adapters/`:
+
+| File | Purpose |
+|------|---------|
+| `sqlite-adapter.ts` | SQLite (better-sqlite3) |
+| `mysql-adapter.ts` | MySQL/MariaDB (mysql2) |
+| `postgresql-adapter.ts` | PostgreSQL (pg) |
+
+Each adapter implements the `DatabaseAdapter` interface with database-specific:
+- Connection handling
+- Transaction management
+- String aggregation
+- Type conversions
+
+## Testing Cross-Database Compatibility
+
+```bash
+# Run tests on all configured databases
+npm run test:cross-db
+
+# Run tests on specific database
+DB_TYPE=postgres npm test
+```
+
+## Version History
+
+- **v4.1.0**: PostgreSQL compatibility fixes (string_agg, GROUP BY)
+- **v4.0.2**: JSON-only cross-database migration
+- **v3.7.0**: Multi-database adapter architecture

package/docs/DATABASE_AUTH.md

@@ -1,302 +1,107 @@
 # Database Authentication Configuration
 
-This document describes the authentication configuration system for multi-database support in mcp-sqlew v4.0.0+.
+This document describes the authentication configuration for multi-database support in sqlew v4.0.0+.
 
-## Overview
+## Supported Authentication
 
-The configuration system supports multiple database backends with various authentication methods:
-
-- **SQLite**: Local file-based database (default, no authentication)
-- **PostgreSQL**: Remote PostgreSQL with direct authentication or IAM authentication (planned)
-- **MySQL**: Remote MySQL with direct authentication or IAM authentication (planned)
-
-**Note**: SSH tunneling is NOT supported by this software. Users must set up SSH tunnels manually using the `ssh` command and connect to localhost.
+| Method | Status | Description |
+|--------|--------|-------------|
+| **Direct (Password)** | ✅ Supported | Standard username/password authentication |
+| **SSH Tunnel** | ✅ Manual | User-managed SSH port forwarding |
+| SSL/TLS Certificates | 🔮 Planned | Client certificate authentication |
+| AWS RDS IAM | 🔮 Planned | Token-based authentication for AWS RDS |
+| GCP Cloud SQL IAM | 🔮 Planned | Token-based authentication for GCP Cloud SQL |
 
 ## Configuration Structure
 
-All configuration is defined in `.sqlew/config.toml` under the `[database]` section.
-
-### SQLite Configuration (Default)
+All configuration is defined in `.sqlew/config.toml`.
 
-The simplest configuration for local development:
+### SQLite (Default)
 
 ```toml
 [database]
 path = ".sqlew/sqlew.db"
 ```
 
-### PostgreSQL/MySQL Configuration
-
-Remote databases require three main sections:
-
-1. **Database Type**: Specifies the database system
-2. **Connection**: Server address and database name
-3. **Authentication**: Credentials and authentication method
-
-## Configuration Sections
-
-### 1. Database Type
-
-```toml
-[database]
-type = "postgres"  # or "mysql" or "sqlite"
-```
-
-Valid values: `sqlite`, `postgres`, `mysql`
-
-### 2. Connection Configuration
-
-Required for PostgreSQL and MySQL:
-
-```toml
-[database.connection]
-host = "localhost"          # Database server hostname or IP
-port = 5432                 # PostgreSQL: 5432, MySQL: 3306
-database = "sqlew"          # Database name
-```
-
-**Fields**:
-- `host` (required): Database server hostname or IP address
-- `port` (required): Database server port (1-65535)
-- `database` (required): Target database name
-
-### 3. Authentication Configuration
-
-```toml
-[database.auth]
-type = "direct"             # Authentication method
-user = "postgres"           # Database username
-password = "your-password"  # Database password
-```
-
-**Fields**:
-- `type` (required): Authentication method
-  - `direct`: Standard username/password (or localhost connection through manual SSH tunnel)
-  - `aws-iam`: AWS RDS IAM authentication (planned)
-  - `gcp-iam`: GCP Cloud SQL IAM authentication (planned)
-- `user` (optional): Database username
-- `password` (optional): Database password
-
-**Note**: For databases behind bastion hosts, users must set up SSH tunnels manually using the `ssh -L` command, then connect to localhost.
-
-### 4. SSL/TLS Configuration (Optional)
-
-For encrypted database connections:
-
-```toml
-[database.auth.ssl]
-ca = "/path/to/ca-cert.pem"           # CA certificate
-cert = "/path/to/client-cert.pem"     # Client certificate (mutual TLS)
-key = "/path/to/client-key.pem"       # Client private key (mutual TLS)
-rejectUnauthorized = true             # Reject self-signed certificates
-```
-
-**Fields**:
-- `ca` (optional): Certificate Authority certificate (file path or content)
-- `cert` (optional): Client certificate for mutual TLS (file path or content)
-- `key` (optional): Client private key for mutual TLS (file path or content)
-- `rejectUnauthorized` (optional): Whether to reject unauthorized certificates (default: `true`)
-
-**Security Note**: Setting `rejectUnauthorized = false` is not recommended for production.
-
-### 5. Manual SSH Tunnel Setup (For Databases Behind Bastion Hosts)
-
-**SSH tunneling is NOT supported by this software.** Users must set up SSH tunnels manually before connecting.
-
-**Manual Tunnel Setup:**
-
-```bash
-# Example: Connect to database behind bastion host
-ssh -L 3307:db.internal.example.com:3306 user@bastion.example.com
-
-# Then configure sqlew to connect to localhost:
-```
-
-```toml
-[database]
-type = "mysql"
-
-[database.connection]
-host = "localhost"      # Connect to tunnel endpoint
-port = 3307             # Forwarded port (not 3306!)
-database = "sqlew_db"
-
-[database.auth]
-type = "direct"
-user = "mysql_user"
-password = "db-password"
-```
-
-**SSH Tunnel Command Options:**
-- `-L localport:remotehost:remoteport`: Port forwarding specification
-- `-N`: Don't execute remote command (tunnel only)
-- `-f`: Run in background
-- `-o ServerAliveInterval=60`: Keep connection alive
-
-**Example with background tunnel:**
-```bash
-ssh -f -N -L 3307:db.internal:3306 user@bastion.example.com
-```
-
-## Complete Examples
-
-### Example 1: PostgreSQL with Direct Authentication
+### PostgreSQL
 
 ```toml
 [database]
 type = "postgres"
 
 [database.connection]
-host = "db.example.com"
+host = "localhost"
 port = 5432
-database = "sqlew_production"
+database = "sqlew_db"
 
 [database.auth]
 type = "direct"
 user = "postgres"
-password = "secure-password"
+password = "your-password"
 ```
 
-### Example 2: PostgreSQL with SSL/TLS
+### MySQL/MariaDB
 
 ```toml
 [database]
-type = "postgres"
+type = "mysql"
 
 [database.connection]
-host = "db.example.com"
-port = 5432
-database = "sqlew_production"
+host = "localhost"
+port = 3306
+database = "sqlew_db"
 
 [database.auth]
 type = "direct"
-user = "postgres"
-password = "secure-password"
-
-[database.auth.ssl]
-ca = "/path/to/ca-cert.pem"
-rejectUnauthorized = true
-```
-
-### Example 3: PostgreSQL via Manual SSH Tunnel
-
-**Step 1**: Set up SSH tunnel manually:
-```bash
-ssh -L 5433:db.internal.example.com:5432 deploy@bastion.example.com
+user = "mysql_user"
+password = "your-password"
 ```
 
-**Step 2**: Configure sqlew to use localhost:
-```toml
-[database]
-type = "postgres"
-
-[database.connection]
-host = "localhost"       # Tunnel endpoint
-port = 5433              # Forwarded port (not 5432!)
-database = "sqlew_production"
-
-[database.auth]
-type = "direct"
-user = "postgres"
-password = "db-password"
-```
+## SSH Tunnel (Manual Setup)
 
-### Example 4: MySQL via Manual SSH Tunnel with SSL
+**SSH tunneling is NOT built into sqlew.** Set up tunnels manually before connecting.
 
-**Step 1**: Set up SSH tunnel manually:
 ```bash
-ssh -f -N -L 53306:mysql.internal.example.com:3306 deploy@jump.example.com
-```
-
-**Step 2**: Configure sqlew with SSL:
-```toml
-[database]
-type = "mysql"
-
-[database.connection]
-host = "localhost"       # Tunnel endpoint
-port = 53306             # Forwarded port
-database = "sqlew_db"
-
-[database.auth]
-type = "direct"
-user = "mysql_user"
-password = "db-password"
-
-[database.auth.ssl]
-ca = "/path/to/ca.pem"
-rejectUnauthorized = true
+# Example: Forward local port 5433 to remote database
+ssh -L 5433:db.internal.example.com:5432 user@bastion.example.com
 ```
 
-### Example 5: MySQL with Inline SSL Certificate
+Then configure sqlew to connect to localhost:
 
 ```toml
 [database]
-type = "mysql"
+type = "postgres"
 
 [database.connection]
-host = "db.example.com"
-port = 3306
+host = "localhost"    # Tunnel endpoint
+port = 5433           # Forwarded port
 database = "sqlew_db"
 
 [database.auth]
 type = "direct"
-user = "mysql_user"
+user = "postgres"
 password = "db-password"
-
-[database.auth.ssl]
-ca = """-----BEGIN CERTIFICATE-----
-MIIEGzCCAwOgAwIBAgIQDKU...
-...certificate content...
------END CERTIFICATE-----"""
-rejectUnauthorized = true
 ```
 
-## Validation Rules
-
-The configuration loader validates all settings and provides clear error messages:
-
-### Database Type Validation
-- Must be one of: `sqlite`, `postgres`, `mysql`
+**Useful SSH options:**
+- `-N`: Don't execute remote command (tunnel only)
+- `-f`: Run in background
+- `-o ServerAliveInterval=60`: Keep connection alive
 
-### Connection Validation
-- Required for PostgreSQL and MySQL
-- `host`: Must be specified
-- `port`: Must be between 1 and 65535
-- `database`: Must be specified
+## Validation Rules
 
-### Authentication Validation
+### Connection
+- `host`: Required for PostgreSQL/MySQL
+- `port`: 1-65535
+- `database`: Required for PostgreSQL/MySQL
 
-**Direct Authentication**:
+### Authentication
+- `type`: Must be `direct`
 - `user`: Required
 - `password`: Required
 
-### SSL Validation
-- `rejectUnauthorized`: Must be a boolean (if specified)
-
-## Default Values
-
-The configuration loader applies sensible defaults:
-
-### SSL Defaults
-- `rejectUnauthorized`: true
-
 ## Error Handling
 
-### Configuration Parsing Errors
-
-If the TOML file is malformed:
-
-```
-⚠️  Failed to load config file: .sqlew/config.toml
-   Error: Unexpected character at line 5
-   Using default configuration
-```
-
-### Validation Errors
-
-If configuration is invalid:
-
 ```
 ⚠️  Configuration validation failed: .sqlew/config.toml
    - database.connection.host is required
@@ -304,141 +109,50 @@ If configuration is invalid:
    Using default configuration
 ```
 
-## Loading Configuration
-
-The configuration is loaded automatically when the MCP server starts. You can also load it programmatically:
-
-```typescript
-import { loadConfigFile, validateDatabaseConfig } from './config/loader.js';
-
-// Load and validate
-const config = loadConfigFile('.sqlew/config.toml');
-const validation = validateDatabaseConfig(config.database);
-
-if (!validation.valid) {
-  console.error('Invalid configuration:', validation.errors);
-}
-```
-
-## Backward Compatibility
-
-The new authentication system is fully backward compatible:
-
-- Existing SQLite configurations continue to work
-- The `path` field in `[database]` is still supported
-- If no configuration is provided, SQLite with default path is used
-
 ## Security Best Practices
 
-1. **Use Private Keys**: Prefer SSH private key authentication over passwords
-2. **Enable SSL**: Use SSL/TLS for all production database connections
-3. **Validate Certificates**: Keep `rejectUnauthorized = true` in production
-4. **Secure Credentials**: Never commit config.toml with passwords to version control
-5. **Restrict Permissions**: Set appropriate file permissions on private keys (chmod 600)
-6. **Environment Variables**: Consider using environment variables for sensitive data (future enhancement)
-
-## Troubleshooting
-
-### Connection Failures
-
-**PostgreSQL connection refused**:
-- Check that PostgreSQL is running
-- Verify host and port are correct
-- Ensure firewall allows connections
-- Check pg_hba.conf for authentication settings
-
-**Manual SSH tunnel issues**:
-- Run `ssh -v -L ...` for verbose debugging
-- Check SSH key permissions (`chmod 600 ~/.ssh/id_rsa`)
-- Verify bastion host is reachable
-- Ensure SSH port (usually 22) is open
-- Check if local port is already in use (`netstat -an | grep <port>`)
-
-**SSL certificate errors**:
-- Verify CA certificate path is correct
-- Check certificate is not expired
-- Ensure certificate chain is complete
-- For development, temporarily set `rejectUnauthorized = false` (not for production)
-
-### Configuration Validation
-
-Use the test suite to validate your configuration:
+1. **Never commit passwords** - Don't commit config.toml with passwords to git
+2. **Use SSH tunnels** - For databases behind firewalls
+3. **Restrict access** - Limit database user permissions
 
-```bash
-npx tsc
-node dist/tests/config-loader.test.js
-```
-
-## API Reference
+---
 
-### Functions
+## Future Authentication Methods
 
-#### `loadConfigFile(configPath?: string): SqlewConfig`
+> **Status**: Planned for future releases. If you need these features, please [open an issue](https://github.com/sin5ddd/mcp-sqlew/issues) - we'll prioritize based on demand!
 
-Loads and parses configuration from TOML file.
+### SSL/TLS Client Certificates
 
-**Parameters**:
-- `configPath` (optional): Path to config file (default: `.sqlew/config.toml`)
-
-**Returns**: Parsed and validated configuration
-
-**Example**:
-```typescript
-const config = loadConfigFile('.sqlew/config.toml');
+```toml
+# PLANNED - Not yet implemented
+[database.auth.ssl]
+ca = "/path/to/ca-cert.pem"
+cert = "/path/to/client-cert.pem"
+key = "/path/to/client-key.pem"
+rejectUnauthorized = true
 ```
 
-#### `validateDatabaseConfig(config: DatabaseConfig): { valid: boolean; errors: string[] }`
-
-Validates database configuration.
-
-**Parameters**:
-- `config`: Database configuration object
+### AWS RDS IAM Authentication
 
-**Returns**: Validation result with errors if any
-
-**Example**:
-```typescript
-const validation = validateDatabaseConfig(config.database);
-if (!validation.valid) {
-  console.error('Validation errors:', validation.errors);
-}
+```toml
+# PLANNED - Not yet implemented
+[database.auth]
+type = "aws-iam"
+region = "us-east-1"
 ```
 
-#### `normalizeDatabaseConfig(config: DatabaseConfig): DatabaseConfig`
-
-Normalizes database configuration by applying defaults.
-
-**Parameters**:
-- `config`: Partial database configuration
+### GCP Cloud SQL IAM Authentication
 
-**Returns**: Complete database configuration with defaults
-
-**Example**:
-```typescript
-const normalized = normalizeDatabaseConfig({
-  type: 'postgres',
-  connection: { host: 'localhost', port: 5432, database: 'test' },
-  auth: {
-    type: 'direct',
-    user: 'postgres',
-    password: 'pass',
-    ssl: { ca: '/path/to/ca.pem' }
-  }
-});
-// normalized.auth.ssl.rejectUnauthorized === true (default applied)
+```toml
+# PLANNED - Not yet implemented
+[database.auth]
+type = "gcp-iam"
+project = "my-project"
 ```
 
-## Future Enhancements
-
-Planned authentication methods:
-
-- **AWS RDS IAM Authentication**: Token-based authentication for AWS RDS
-- **GCP Cloud SQL IAM Authentication**: Token-based authentication for GCP Cloud SQL
-- **Environment Variable Substitution**: Reference environment variables in config
-- **Credential Encryption**: Encrypt sensitive fields in config file
+---
 
 ## Related Documentation
 
 - [Configuration Guide](./CONFIGURATION.md)
-- [Architecture Overview](./ARCHITECTURE.md)
-- [Type Definitions](../src/config/types.ts)
+- [Cross Database Guide](./CROSS_DATABASE.md)