sqlew 3.7.3 → 3.8.0

package/CHANGELOG.md

@@ -7,6 +7,353 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ---
 
+## [3.8.0] - 2025-11-09
+
+### BREAKING CHANGES
+
+**Batch Action Naming Standardization**
+
+`task.batch_create` has been renamed to `task.create_batch` to follow the `<verb>_batch` naming pattern used by other batch actions (`set_batch`, `record_batch`).
+
+**Migration Required:**
+```typescript
+// ❌ Old (v3.7.x and earlier)
+task({ action: "batch_create", tasks: [...] })
+
+// ✅ New (v3.8.0+)
+task({ action: "create_batch", tasks: [...] })
+```
+
+**Rationale:**
+- Achieves 100% consistency across all batch actions
+- Improves alphabetical sorting in IDE auto-completion (create → create_batch)
+- Aligns with industry standard (REST APIs, GraphQL, ORMs use suffix pattern)
+- See docs/ADR-batch-naming-standard.md for full justification
+
+**Impact:**
+- All code using `task.batch_create` must update to `task.create_batch`
+- Simple find-replace migration (estimated 2-5 minutes per integration)
+- No database schema changes required
+
+### Removed
+
+**Config Tool Removed (Phase 6)**
+
+The orphaned config tool has been removed in favor of CLI-only configuration:
+
+- **Deleted**: `src/tools/config.ts` (307 lines)
+- **Removed**: `ConfigAction` type from `src/types.ts` and `src/types/actions.ts`
+- **Removed**: ConfigAction import from `src/utils/parameter-validator.ts`
+- **Updated**: README.md with CLI-only config approach documentation
+- **Updated**: docs/CONFIGURATION.md already documented config tool removal
+
+**Why removed:**
+- Config tool was never registered in `tool-registry.ts` (orphaned code)
+- Messaging system deprecated (primary use case eliminated)
+- File-based configuration (`.sqlew/config.toml`) is clearer and more maintainable
+- Runtime updates were confusing (changes lost on restart unless manually synced to file)
+- Configuration drift between `m_config` table and config file
+
+**Migration Path:**
+- ✅ **Use `.sqlew/config.toml`** for all configuration (persistent, version-controlled)
+- ✅ **Use CLI arguments** for one-time overrides (`--autodelete-message-hours=48`)
+- ✅ **Internal config operations** preserved (`src/database/config/config-ops.ts`)
+- ✅ **m_config table** preserved (used internally by retention logic)
+
+**Impact:**
+- Cleaner codebase with ~300 lines removed
+- No functional impact - tool was never registered
+- Configuration via file and CLI arguments only
+
+**Message Tool Completely Removed**
+
+The deprecated message tool has been completely removed from the codebase:
+
+- **Deleted**: `src/tools/messaging.ts` (599 lines)
+- **Removed**: Message tool entry from `tool-registry.ts`
+- **Removed**: Message tool handler from `tool-handlers.ts`
+- **Removed**: Message imports from `cli.ts`
+- **Updated**: `MessageAction` type changed to `never` (backward compatibility stub)
+- **CLI**: `sqlew query messages` now returns error message
+
+**Migration Path**:
+- No action required - messaging system was already marked deprecated in v3.6.6
+- The `t_agent_messages` table was dropped in v3.6.6
+- All message tool actions returned deprecation warnings since v3.6.6
+
+**Impact**:
+- Cleaner codebase with ~700 lines removed
+- No functional impact - messaging system was unused
+- MessageAction type remains as deprecated stub for backward compatibility
+
+### Added
+
+**Layer Expansion (5→9 layers)**
+
+Added 4 new layers to enable better task classification and semantic validation:
+
+**New FILE_REQUIRED layers:**
+- `documentation` - README, CHANGELOG, API docs, architecture docs (file_actions required)
+
+**New FILE_OPTIONAL layers:**
+- `planning` - Research, spike tasks, investigation (file_actions optional)
+- `coordination` - Multi-agent orchestration, task delegation (file_actions optional)
+- `review` - Code review, design review, verification (file_actions optional)
+
+**Existing layers** (5→6 FILE_REQUIRED):
+- presentation, business, data, infrastructure, cross-cutting (file_actions required or empty array)
+
+**Benefits:**
+- Documentation layer enforces file operations for docs work
+- Planning layers allow pure research tasks without file boilerplate
+- Better semantic task classification across the development lifecycle
+
+**file_actions Parameter for Tasks**
+
+Introduced semantic `file_actions` parameter to replace generic `watch_files`:
+
+```typescript
+// New file_actions parameter
+task.create({
+  title: "Implement OAuth",
+  layer: "business",
+  file_actions: [
+    { action: "create", path: "src/auth/oauth.ts" },
+    { action: "edit", path: "src/api/router.ts" },
+    { action: "delete", path: "src/auth/legacy.ts" }
+  ]
+});
+
+// Backward compatible - watch_files still works
+task.create({
+  title: "Update config",
+  layer: "infrastructure",
+  watch_files: ["config.toml"]  // Auto-converts to file_actions
+});
+```
+
+**Layer-Based Validation:**
+- FILE_REQUIRED layers (6) → Must provide `file_actions` or `[]`
+- FILE_OPTIONAL layers (3) → Can omit `file_actions` entirely
+- Clear error messages with layer-specific guidance
+
+**Benefits:**
+- Self-documenting: `action: 'create'` vs `action: 'edit'` shows intent
+- Prevents forgotten file watchers (validation enforced)
+- No boilerplate for planning tasks (can omit parameter)
+- Better token efficiency with automatic file watching
+
+**PostgreSQL Adapter Implementation**
+
+Full PostgreSQL 12+ support with complete adapter implementation:
+
+**Adapter Features:**
+- All 15 abstract methods implemented (`insertReturning`, `upsert`, `jsonExtract`, etc.)
+- PostgreSQL-specific SQL syntax (RETURNING, ON CONFLICT, string_agg)
+- Strict type handling (TRUE/FALSE for booleans, not 1/0)
+- Timezone-aware timestamp functions
+- Transaction support with savepoints
+
+**Migration Compatibility:**
+- All 22 migrations tested and verified on PostgreSQL 16.10
+- Cross-database helper functions for view creation
+- Proper CASCADE handling for foreign key dependencies
+- Sequence management after explicit ID inserts
+- GROUP BY strictness compliance
+
+**Supported Databases:**
+- SQLite 3.x (default, development)
+- MySQL 8.0 / MariaDB 10+ (production)
+- PostgreSQL 12+ (production) ✨ NEW
+
+**Configuration:**
+```toml
+[database]
+type = "postgres"
+
+[database.connection]
+host = "localhost"
+port = 5432
+database = "sqlew_db"
+
+[database.auth]
+type = "direct"
+user = "sqlew_user"
+password = "secret"
+```
+
+### Fixed
+
+**Batch Action Parameter Parsing**
+
+Fixed MCP client array serialization issue affecting all batch actions:
+
+**Problem:** MCP client serializes array parameters as JSON strings:
+```typescript
+// MCP sends:
+decisions: "[{\"key\": \"test\", \"value\": \"val\"}]"  // String!
+
+// Expected:
+decisions: [{key: "test", value: "val"}]  // Array
+```
+
+**Solution:** Added JSON parsing in `tool-handlers.ts` for all batch actions:
+- `decision.set_batch` - parse `decisions` parameter
+- `file.record_batch` - parse `file_changes` parameter
+- `task.create_batch` - parse `tasks` parameter
+
+**Impact:** All batch actions now work correctly with array parameters from MCP client.
+
+**Help System Synchronization**
+
+Fixed critical bug where help database was severely out of sync with code:
+
+**Problem:** 25 actions missing from `m_help_actions` table, causing help system to lie about available actions.
+
+**Missing Actions:**
+- decision: 9 actions (quick_set, search_advanced, set_batch, has_updates, etc.)
+- task: 14 actions (update, get, list, move, link, archive, create_batch, etc.)
+- constraint: 1 action (use_case)
+
+**Solution:** Created migration `20251109020000_fix_missing_help_actions_v3_8_0.ts` to:
+- Add all 25 missing actions with correct descriptions
+- Idempotent checks to prevent duplicates
+- Full synchronization between code and database
+
+**Impact:** Help system now accurately reports all available actions.
+
+### Changed
+
+**Tool Registry Schema Fix**
+
+Added `additionalProperties: true` to all tool schemas in `tool-registry.ts`:
+
+**Problem:** MCP couldn't pass action-specific parameters (key, value, tags, etc.) because schemas only defined `action` property.
+
+**Solution:**
+```typescript
+{
+  name: 'decision',
+  inputSchema: {
+    type: 'object',
+    properties: {
+      action: { ... }
+    },
+    required: ['action'],
+    additionalProperties: true,  // ← CRITICAL FIX
+  },
+}
+```
+
+**Impact:** All MCP tools now accept action-specific parameters correctly.
+
+---
+
+## [3.7.4] - 2025-11-08
+
+### Added - Complete JSON Import/Export System
+
+**Full-featured data migration system with smart ID remapping and dependency resolution**
+
+#### New Features
+
+- **db:import CLI Command** - Import project data from JSON exports with automatic ID remapping
+  - Smart conflict detection (skip-if-exists, project-name override)
+  - Dry-run mode for validation before import
+  - Comprehensive error messages with validation details
+- **Topological Sort Algorithm** - Resolves task dependencies during import
+  - Circular dependency detection prevents import of invalid dependency graphs
+  - BFS-based topological sorting ensures dependencies imported before dependents
+  - Preserves all task relationships and blocking constraints
+- **Smart ID Remapping** - Handles complex foreign key relationships
+  - Master table merge logic (reuse existing entries by name/path)
+  - Transaction table ID translation with bidirectional mapping
+  - Junction table relationship preservation
+  - Automatic orphan cleanup for invalid references
+
+#### Import System Architecture
+
+- **4 Core Modules**:
+  1. `import.ts` - Main orchestrator with transaction management
+  2. `master-tables.ts` - Master table merge logic (m_files, m_tags, m_scopes, etc.)
+  3. `topological-sort.ts` - Dependency graph analysis and sorting
+  4. `db-import.ts` - CLI command with argument parsing and validation
+
+#### Data Migration Strategy
+
+- **ID Remapping**: All imported data gets fresh auto-incremented IDs (no ID preservation)
+- **Master Table Deduplication**: Reuse existing entries for agents, tags, scopes, files by name/path
+- **Transaction Atomicity**: All-or-nothing semantics (full rollback on any error)
+- **Project Isolation**: Each import creates independent project with no cross-contamination
+
+#### CLI Examples
+
+```bash
+# Import project from JSON export
+npx sqlew db:import --source=project-backup.json
+
+# Import with custom project name
+npx sqlew db:import --source=data.json --project-name=my-project
+
+# Dry-run validation (no actual import)
+npx sqlew db:import --source=data.json --dry-run
+
+# Export project for migration
+npx sqlew db:export --project=visualizer --output=visualizer-data.json
+```
+
+#### Technical Details
+
+- **Batch Inserts** - 10-row batches to avoid SQLite UNION ALL limits
+- **Foreign Key Validation** - Validates all foreign key references before insertion
+- **View Handling** - Temporarily drops/restores views during schema changes
+- **Idempotent Operations** - Safe to retry on failure
+- **Error Recovery** - Detailed error messages with validation guidance
+
+#### Use Cases
+
+- **Multi-Project Single Database** - Consolidate multiple projects when database creation permissions are limited
+- **Project Sharing** - Share context with team members or between machines
+- **Cross-Database Migration** - Move projects between different databases (different machine, SQLite → MySQL, etc.)
+
+**Note**: Import uses `--skip-if-exists=true` by default. This is NOT a backup/restore solution for the same database.
+Use database-level backups (SQLite file copy, MySQL dump) for backup/restore scenarios.
+
+#### Impact
+
+- ✅ **Complete migration solution** - Export from one database, import to another
+- ✅ **Multi-project support** - Merge multiple project exports into single database
+- ✅ **Permission-friendly** - Works for users who can't create multiple databases
+- ✅ **Data integrity** - Zero data loss, all relationships preserved
+- ✅ **Production ready** - Comprehensive error handling and validation
+- ✅ **Cross-database compatible** - JSON format works across SQLite, MySQL, PostgreSQL
+
+---
+
+### Fixed - Multi-Project Migration (HOTFIX)
+
+**Critical fix for v3.7.0-v3.7.2 migration in multi-project scenarios**
+
+#### Problem
+
+- Users upgrading from v3.6.10 to v3.7.0+ could end up with duplicate projects
+- Migration 20251104000000 created project #1 with fake name "default-project"
+- Users creating second project manually resulted in namespace conflicts
+
+#### Solution
+
+- Enhanced migration idempotency checks
+- Improved project consolidation logic
+- Better handling of existing project scenarios
+
+#### Impact
+
+- ✅ **Safe multi-project migration** - No duplicate projects created
+- ✅ **Backward compatible** - Works for both fresh installs and upgrades
+- ✅ **Data preservation** - All existing data maintained correctly
+
+---
+
 ## [3.7.3] - 2025-11-06
 
 ### Fixed - Master Tables Namespace Collision Bug
@@ -582,6 +929,85 @@ if (!hasProjectId) {
 
 ---
 
+### Added - Environment Variable Support for Project Root
+
+**New `SQLEW_PROJECT_ROOT` environment variable for project-relative databases**
+
+#### Problem
+- Junie AI and other MCP clients require absolute paths in configuration
+- Users had to hardcode project-specific paths in MCP server config
+- No easy way to use project-relative database paths
+
+#### Solution
+- Added `SQLEW_PROJECT_ROOT` environment variable support (inspired by serena-mcp pattern)
+- MCP clients can now pass project directory via environment variable
+- Database automatically created at `$SQLEW_PROJECT_ROOT/.sqlew/sqlew.db`
+
+#### Priority Order
+1. `SQLEW_PROJECT_ROOT` environment variable (NEW - highest priority)
+2. `--db-path` CLI argument (absolute path)
+3. `--config-path` CLI argument (absolute path)
+4. `database.path` in config file (absolute path)
+5. `process.cwd()` fallback
+
+#### Junie AI Configuration Example
+```json
+{
+  "mcpServers": {
+    "sqlew": {
+      "command": "npx",
+      "args": ["sqlew"],
+      "env": {
+        "SQLEW_PROJECT_ROOT": "{projectDir}"
+      }
+    }
+  }
+}
+```
+
+**Note:** Junie AI uses `{projectDir}` variable which expands to the current project's absolute path. This ensures each project gets its own isolated database without hardcoded paths. Other MCP clients may use different variable names like `${workspaceFolder}` (VS Code/Cline) - check your client's documentation.
+
+#### Impact
+- ✅ **Project-relative databases** without hardcoded absolute paths
+- ✅ **Cleaner MCP configuration** (no per-project path updates needed)
+- ✅ **Compatible with Junie AI, Claude Desktop, and other MCP clients**
+- ✅ **No breaking changes** (environment variable is optional)
+
+---
+
+### Fixed - MCP Protocol Compliance (EPIPE Fix)
+
+**Eliminated console output for strict JSON-RPC protocol compliance**
+
+#### Problem
+- EPIPE (broken pipe) errors when running with Junie AI on Windows
+- Console output to stdout/stderr violated MCP JSON-RPC protocol requirements
+- Strict MCP clients (like Junie AI) expect pure JSON-RPC on stdio streams
+
+#### Changes
+- **Redirected all diagnostic output to debug log file** - stdout/stderr reserved exclusively for JSON-RPC
+- Modified `safeConsoleError()` to write to debug log instead of stderr
+- Replaced 50+ console.log/console.error calls across codebase:
+  - `src/database.ts` - Database initialization messages
+  - `src/watcher/file-watcher.ts` - File watcher status and events
+  - `src/watcher/gitignore-parser.ts` - .gitignore loading warnings
+  - `src/tools/tasks.ts` - Task file registration warnings
+  - `src/config/example-generator.ts` - First launch messages
+
+#### Technical Details
+- **MCP Protocol Requirement**: stdin/stdout/stderr must carry only JSON-RPC messages
+- **Debug Logging**: All diagnostic messages now use `debugLog()` with appropriate levels (INFO, WARN, ERROR)
+- **Zero stdout pollution**: Server starts silently, waits for JSON-RPC requests
+- **Tested with Junie AI**: Confirmed no EPIPE errors on Windows
+
+#### Impact
+- ✅ **Works with strict MCP clients** (Junie AI, etc.)
+- ✅ **Maintains full diagnostics** via debug log file
+- ✅ **Pure JSON-RPC protocol** compliance
+- ✅ **No breaking changes** to MCP tool functionality
+
+---
+
 ### Fixed - Windows Absolute Path Handling
 
 **Fixed path normalization for Windows environments**
@@ -1002,7 +1428,7 @@ This version was replaced by v3.2.4. Use v3.2.4 or later.
 - New triggers: `trg_log_task_create`, `trg_log_task_status_change`, `trg_update_task_timestamp`
 
 #### MCP Actions (task tool)
-- `create`, `update`, `get`, `list`, `move`, `link`, `archive`, `batch_create`
+- `create`, `update`, `get`, `list`, `move`, `link`, `archive`, `create_batch`
 - `watch_files` - Start file watching for auto-transitions
 
 #### Documentation

package/README.md

@@ -94,10 +94,21 @@ It's Ready!
 
 ## Configuration
 
+### Database Support
+
+sqlew supports multiple database backends for different deployment scenarios:
+
+| Database | Use Case | Status |
+|----------|----------|--------|
+| **SQLite** | Development, small projects | ✅ Default |
+| **MySQL 8.0 / MariaDB 10+** | Production, shared environments | ✅ Supported |
+| **PostgreSQL 12+** | Production, enterprise | ✅ v3.8.0+ |
+
 ### Optional Config File
 
 On first run, `.sqlew/config.toml` will be created for persistent settings:
 
+**SQLite (Default):**
 ```toml
 [database]
 path = ".sqlew/custom.db"
@@ -107,12 +118,70 @@ ignore_weekend = true
 message_hours = 48
 ```
 
+**PostgreSQL:**
+```toml
+[database]
+type = "postgres"
+
+[database.connection]
+host = "localhost"
+port = 5432
+database = "sqlew_db"
+
+[database.auth]
+type = "direct"
+user = "sqlew_user"
+password = "secret"
+```
+
+**MySQL/MariaDB:**
+```toml
+[database]
+type = "mysql"
+
+[database.connection]
+host = "localhost"
+port = 3306
+database = "sqlew_db"
+
+[database.auth]
+type = "direct"
+user = "sqlew_user"
+password = "secret"
+```
+
 Also `.sqlew/config.example.toml` is created for reference.
 
 **Settings Priority:** CLI args > config.toml > database defaults
 
 See [docs/CONFIGURATION.md](docs/CONFIGURATION.md) for all options and validation rules.
 
+### CLI Configuration (Recommended)
+
+Configuration is managed via **`.sqlew/config.toml`** file and **CLI arguments only**. The MCP `config` tool has been removed for simplicity.
+
+**Why CLI-only configuration?**
+- **No drift:** Single source of truth (config file)
+- **Version control:** Commit config to git, share with team
+- **Clear documentation:** Config file documents project requirements
+- **Type safety:** TOML validation catches errors at startup
+
+**Common CLI arguments:**
+```bash
+# Custom database path
+npx sqlew /path/to/database.db
+
+# Auto-deletion settings
+npx sqlew --autodelete-message-hours=48
+npx sqlew --autodelete-file-history-days=30
+npx sqlew --autodelete-ignore-weekend
+
+# Custom config file
+npx sqlew --config-path=.sqlew/custom.toml
+```
+
+For persistent settings, edit `.sqlew/config.toml` instead of using CLI arguments.
+
 ## Quick Start
 
 install it, launch claude, exit claude and launch Claude again.
@@ -219,7 +288,7 @@ All tools support:
 ### For Developers
 
 - [Configuration Guide](docs/CONFIGURATION.md) - TOML config file setup
-- [Database Migration](docs/DATABASE_MIGRATION.md) - SQLite → MySQL/PostgreSQL migration
+- [CLI Mode Overview](docs/cli/README.md) - Database migration, export/import commands
 - [Building from Source](docs/ARCHITECTURE.md#development) - Setup instructions
 - [Migration Guides](docs/MIGRATION_v2.md) - Version upgrade guides
 
@@ -245,7 +314,7 @@ Support development via [GitHub Sponsors](https://github.com/sponsors/sin5ddd) -
 
 ## Version
 
-Current version: **3.7.0**
+Current version: **3.7.4**
 See [CHANGELOG.md](CHANGELOG.md) for release history.
 
 ## License

package/assets/sample-agents/sqlew-scrum-master.md

@@ -91,7 +91,7 @@ stats({ action: "example" })       // Statistics and monitoring patterns
 // Step 3: Copy the relevant example, modify values, execute
 // Example from action: "example" output:
 task({
-  action: "batch_create",
+  action: "create_batch",
   tasks: [
     { title: "Design API", priority: 3, assigned_agent: "architect" },
     { title: "Implement API", priority: 3, assigned_agent: "backend" },
@@ -117,10 +117,10 @@ task({ action: "create", title: "Task", tags: "backend,api" })
 task({ action: "create", title: "Task", tags: ["backend", "api"] })
 
 // ❌ WRONG - atomic as string
-task({ action: "batch_create", tasks: [...], atomic: "true" })
+task({ action: "create_batch", tasks: [...], atomic: "true" })
 
 // ✅ CORRECT - atomic as boolean
-task({ action: "batch_create", tasks: [...], atomic: false })
+task({ action: "create_batch", tasks: [...], atomic: false })
 ```
 
 ### When Stuck or Getting Errors
@@ -154,7 +154,7 @@ Before executing ANY sqlew tool call:
 
 **Get Correct Syntax**: Always use `task({ action: "example" })` for current parameter format.
 
-**Token Optimization**: Use `batch_create` for multiple related tasks instead of individual `create` calls.
+**Token Optimization**: Use `create_batch` for multiple related tasks instead of individual `create` calls.
 
 ### Dependency Management
 - Use `add_dependency` to establish blocker → blocked relationships