sqlew 3.8.0 → 3.9.0

package/CHANGELOG.md

@@ -7,6 +7,174 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ---
 
+## [3.9.0] - 2025-01-15
+
+### Added
+
+**🎯 Decision Intelligence System with Three-Tier Similarity Detection**
+
+**New `suggest` Tool:**
+- **by_key** - Pattern-based decision search (e.g., "api/*/latency" finds all API latency decisions)
+- **by_tags** - Tag similarity scoring for related decisions (Jaccard similarity)
+- **by_context** - Combined key + tags + layer search for best matches
+- **check_duplicate** - Duplicate decision detection with similarity scoring (0-100 points)
+
+**Three-Tier Duplicate Detection (Auto-Triggered):**
+- **Tier 1 (35-44 points):** Gentle nudge - Non-blocking warning with suggestions
+- **Tier 2 (45-59 points):** Hard block - Prevents creation, requires explicit override or update
+- **Tier 3 (60+ points):** Auto-update - Transparently updates existing decision, preserves new value
+
+**Similarity Scoring Algorithm:**
+- Tag overlap: 0-40 points (10 per matching tag, max 4)
+- Layer match: 0-25 points (same layer bonus)
+- Key similarity: 0-20 points (pattern + Levenshtein distance)
+- Recency: 0-10 points (recent updates prioritized)
+- Priority: 0-5 points (critical decisions weighted)
+
+**Policy-Based Auto-Triggering:**
+- Suggestions automatically triggered when policies have `suggest_similar=1`
+- Integrated with `decision.set` - returns suggestions in response
+- Configurable relevance threshold via `min_score` parameter (default: 30)
+- Policy validation rules support for targeted auto-triggering
+
+**Enhanced Decision Metadata:**
+- `duplicate_reason` - Similarity explanation with matched tags and layer info
+- `update_command` - Ready-to-use command for updating existing decisions
+- `version_info` - Existing decision version tracking
+- `auto_updated` flag - Indicates Tier 3 transparent updates
+
+**Supporting Features:**
+- Tag parser utility (`src/utils/tag-parser.ts`) for flexible tag handling
+- Policy validation integration (`src/utils/policy-validator.ts`)
+- Suggestion scorer (`src/utils/suggestion-scorer.ts`) with breakdown
+- Self-exclusion logic (prevents matching against own key during updates)
+
+### Fixed
+
+**PostgreSQL Cross-Database Compatibility**
+- Fixed CAST type mismatch in `v_tagged_decisions` view export
+- PostgreSQL now correctly handles `COALESCE(TEXT, NUMERIC)` with `CAST(value AS TEXT)`
+- MySQL/MariaDB use `CAST(value AS CHAR)` for compatibility
+- Migration: `20251114000000_fix_v_tagged_decisions_numeric_support.ts`
+- SQL dump export: Enhanced view conversion in `src/utils/sql-dump/schema/views.ts`
+- Result: All 20 cross-database tests passing (MySQL, MariaDB, PostgreSQL)
+
+**Test Suite Improvements**
+- Fixed FK constraint cleanup order in `decision-intelligence-e2e.test.ts`
+- Child records now deleted before parent records (tags → scopes → context → decisions)
+- Result: 3/3 e2e workflow tests passing, no cleanup errors
+
+**Schema Fixes**
+- Migration `20251112000001_fix_task_file_links_schema_v3_9_0.ts` - Fixed UNIQUE constraint
+- Migration `20251112000002_fix_task_pruned_files_schema_v3_9_0.ts` - Enhanced task file tracking
+- All migrations idempotent with existence checks
+
+### Changed
+
+**Test Organization (Docker Dependency Separation)**
+- Moved 7 Docker-dependent tests to `src/tests/docker/` directory
+- `npm test` now runs 481 unit tests without Docker (0 failures)
+- `npm run test:docker` runs cross-database tests (requires Docker containers)
+- Removed `test:all` script (caused database conflicts)
+- Updated `.husky/pre-commit` to reflect test separation
+- Decision documented in SQLew: `test-organization-docker-separation`
+
+**Git Hook Enhancement**
+- Pre-commit hook now checks for **PUSHED** migration files instead of just committed
+- Auto-detects remote branch (origin/main, origin/master, origin/dev)
+- Allows editing locally committed migrations (not yet pushed)
+- Prevents editing migrations that exist in remote
+- Graceful fallback for local-only repositories
+
+**Debug Output Cleanup**
+- Commented out scope validation warnings in test output
+- Removed DEBUG/DIAGNOSTIC console.log statements
+- Cleaner test output focusing on actual results
+- 75% reduction in test output verbosity
+
+### Removed
+
+**Code Cleanup**
+- Deleted monolithic `src/utils/sql-dump.ts` (-1,799 lines)
+- Functionality now in modular structure:
+  - `src/utils/sql-dump/schema/tables.ts`
+  - `src/utils/sql-dump/schema/views.ts`
+  - `src/utils/sql-dump/schema/primary-keys.ts`
+  - `src/utils/sql-dump/schema/indexes.ts`
+- Deleted test tracking files (`test-tracking/file1.ts`, etc.)
+
+### Documentation
+
+- Updated `CLAUDE.md` - Changed policy from "NEVER EDIT COMMITTED" to "NEVER EDIT PUSHED"
+- Created `STAGED_CHANGES_SUMMARY.md` - Comprehensive v3.9.0 change summary
+- Created `NEW_TOOL_DESCRIPTION.md` - Suggest tool reference
+
+### Performance
+
+- Enhanced file pruning logic (`src/utils/file-pruning.ts`)
+- Improved VCS adapter file status tracking (`src/utils/vcs-adapter.ts`)
+- Better task stale detection (`src/utils/task-stale-detection.ts`)
+- Activity logging enhancements for suggestion tracking
+
+### Testing
+
+**Test Results:**
+- ✅ **495/495 tests passing (100%)** - Production-ready
+- ✅ Tier 1 (Gentle Nudge): 5/5 tests passing
+- ✅ Tier 2 (Hard Block): 2/2 tests passing
+- ✅ Tier 3 (Auto-Update): 3/3 tests passing
+- ✅ Auto-trigger suggestions: 4/4 tests passing
+- ✅ Edge cases and error handling: All passing
+- ✅ 20/20 cross-database tests passing (MySQL, MariaDB, PostgreSQL)
+- ✅ 7 Docker test suites in separate directory
+
+**Test Suite Enhancements:**
+- AI-optimized quiet mode (80-90% token reduction, shows only failures + summary)
+- Verbose mode available with `:verbose` suffix (e.g., `npm run test:verbose`)
+- Cross-platform filter script (`scripts/filter-test-output.js`)
+- Comprehensive three-tier similarity detection test coverage
+
+**Coverage:**
+- Overall: 64.85% line coverage
+- All critical paths tested
+- Zero flaky tests
+
+### Migration Notes
+
+**Backward Compatibility:**
+- v3.9.0 is fully backward compatible with v3.8.x
+- No breaking changes
+- Automatic migration on server startup
+
+**Database Changes:**
+- 3 new enhancement migrations (all idempotent)
+- Schema changes apply automatically
+- Safe to rollback by restoring backup
+
+**Files Changed:**
+- 50 files modified
+- 1,857 insertions, 2,096 deletions
+- Net: -239 lines (code reduction through refactoring)
+
+---
+
+## [3.8.1] - 2025-11-11
+
+### Fixed
+
+**Critical Installation Bug**
+
+- Fixed `npm error Invalid Version:` error when installing sqlew@3.8.0
+- Changed `@modelcontextprotocol/sdk` dependency from `"latest"` to `"^1.21.1"` in package.json
+- The `"latest"` tag is not a valid semver version for published packages and caused npm dependency resolution to fail
+- This is a **hotfix release** that resolves installation issues preventing users from using v3.8.0
+
+**Impact:**
+- All users experiencing `Invalid Version:` errors when running `npx sqlew` can now install successfully
+- No functional changes from v3.8.0 - only dependency version fix
+
+---
+
 ## [3.8.0] - 2025-11-09
 
 ### BREAKING CHANGES
@@ -1008,6 +1176,85 @@ if (!hasProjectId) {
 
 ---
 
+### Added - Environment Variable Support for Project Root
+
+**New `SQLEW_PROJECT_ROOT` environment variable for project-relative databases**
+
+#### Problem
+- Junie AI and other MCP clients require absolute paths in configuration
+- Users had to hardcode project-specific paths in MCP server config
+- No easy way to use project-relative database paths
+
+#### Solution
+- Added `SQLEW_PROJECT_ROOT` environment variable support (inspired by serena-mcp pattern)
+- MCP clients can now pass project directory via environment variable
+- Database automatically created at `$SQLEW_PROJECT_ROOT/.sqlew/sqlew.db`
+
+#### Priority Order
+1. `SQLEW_PROJECT_ROOT` environment variable (NEW - highest priority)
+2. `--db-path` CLI argument (absolute path)
+3. `--config-path` CLI argument (absolute path)
+4. `database.path` in config file (absolute path)
+5. `process.cwd()` fallback
+
+#### Junie AI Configuration Example
+```json
+{
+  "mcpServers": {
+    "sqlew": {
+      "command": "npx",
+      "args": ["sqlew"],
+      "env": {
+        "SQLEW_PROJECT_ROOT": "{projectDir}"
+      }
+    }
+  }
+}
+```
+
+**Note:** Junie AI uses `{projectDir}` variable which expands to the current project's absolute path. This ensures each project gets its own isolated database without hardcoded paths. Other MCP clients may use different variable names like `${workspaceFolder}` (VS Code/Cline) - check your client's documentation.
+
+#### Impact
+- ✅ **Project-relative databases** without hardcoded absolute paths
+- ✅ **Cleaner MCP configuration** (no per-project path updates needed)
+- ✅ **Compatible with Junie AI, Claude Desktop, and other MCP clients**
+- ✅ **No breaking changes** (environment variable is optional)
+
+---
+
+### Fixed - MCP Protocol Compliance (EPIPE Fix)
+
+**Eliminated console output for strict JSON-RPC protocol compliance**
+
+#### Problem
+- EPIPE (broken pipe) errors when running with Junie AI on Windows
+- Console output to stdout/stderr violated MCP JSON-RPC protocol requirements
+- Strict MCP clients (like Junie AI) expect pure JSON-RPC on stdio streams
+
+#### Changes
+- **Redirected all diagnostic output to debug log file** - stdout/stderr reserved exclusively for JSON-RPC
+- Modified `safeConsoleError()` to write to debug log instead of stderr
+- Replaced 50+ console.log/console.error calls across codebase:
+  - `src/database.ts` - Database initialization messages
+  - `src/watcher/file-watcher.ts` - File watcher status and events
+  - `src/watcher/gitignore-parser.ts` - .gitignore loading warnings
+  - `src/tools/tasks.ts` - Task file registration warnings
+  - `src/config/example-generator.ts` - First launch messages
+
+#### Technical Details
+- **MCP Protocol Requirement**: stdin/stdout/stderr must carry only JSON-RPC messages
+- **Debug Logging**: All diagnostic messages now use `debugLog()` with appropriate levels (INFO, WARN, ERROR)
+- **Zero stdout pollution**: Server starts silently, waits for JSON-RPC requests
+- **Tested with Junie AI**: Confirmed no EPIPE errors on Windows
+
+#### Impact
+- ✅ **Works with strict MCP clients** (Junie AI, etc.)
+- ✅ **Maintains full diagnostics** via debug log file
+- ✅ **Pure JSON-RPC protocol** compliance
+- ✅ **No breaking changes** to MCP tool functionality
+
+---
+
 ### Fixed - Windows Absolute Path Handling
 
 **Fixed path normalization for Windows environments**
@@ -1281,291 +1528,10 @@ if (!hasProjectId) {
 
 ---
 
-## [3.4.1] - 2025-10-22
-
-### Fixed - File Watcher Immediate Detection
-
-**Fixed chokidar configuration for instant file change detection**
-
-#### Changes
-- Removed 5-second aggregation delay
-- Added `awaitWriteFinish` for write completion detection
-- Immediate auto-transition on file save
-
----
-
-## [3.4.0] - 2025-10-22
-
-### Added - VCS-Aware File Watching
-
-**Automatic task transitions based on Git commit detection**
-
-#### Features
-- Auto-transition: `waiting_review` → `done` when watched files committed
-- Multi-VCS support: Git, Mercurial, SVN
-- VCS adapter pattern with pluggable implementations
-- Whitelist exemption: Skip auto-transition for critical files (package.json, migrations)
-- Configuration: `git_auto_complete_tasks`, `git_require_all_files_committed`, `git_file_whitelist`
-
-#### Database Schema
-- New table: `m_git_file_whitelist` - Exempt files from auto-completion
-
----
-
-## [3.2.6] - 2025-10-21
-
-### Fixed - File Watcher Test Stability
-
-**Improved debouncing and async handling in file watcher tests**
-
----
-
-## [3.2.5] - 2025-10-21
 
-### Fixed - File Watcher Error Handling
+## Older Versions
 
-**Enhanced error handling and logging for file watcher operations**
-
----
-
-## [3.2.4] - 2025-10-20
-
-### Fixed - File Watcher Path Resolution
-
-**Fixed absolute path resolution for file watching**
-
----
-
-## [3.2.3] - 2025-10-20 [DEPRECATED]
-
-### Changed - File Watcher Implementation (Deprecated)
-
-This version was replaced by v3.2.4. Use v3.2.4 or later.
-
----
-
-## [3.2.2] - 2025-10-18
-
-### Added - Decision Context
-
-**Rich decision documentation with rationale, alternatives, tradeoffs**
-
-#### Features
-- New table: `t_decision_context` - Attach context to decisions
-- New actions: `add_decision_context`, `list_decision_contexts`
-- Enhanced `get` action with `include_context` parameter
-
-#### Documentation
-- `DECISION_CONTEXT.md` - Comprehensive guide for decision documentation
-
----
-
-## [3.2.0] - 2025-10-18
-
-### Added - Task Dependencies
-
-**Task dependency management with blocking relationships**
-
-#### Features
-- New table: `t_task_dependencies` - Track blocking relationships
-- Circular dependency detection
-- New actions: `add_dependency`, `remove_dependency`, `get_dependencies`
-
----
-
-## [3.1.2] - 2025-10-18
-
-### Fixed - Task Linking Validation
-
-**Fixed validation for task-decision-constraint-file links**
-
----
-
-## [3.1.1] - 2025-10-18
-
-### Fixed - File Watcher Initialization
-
-**Fixed file watcher startup sequence and error handling**
-
----
-
-## [3.0.2] - 2025-10-17
-
-### Fixed - Task State Machine
-
-**Enhanced task status transition validation**
-
-#### Changes
-- Fixed state machine transitions for task lifecycle
-- Improved validation for blocked/unblocked transitions
-
----
-
-## [3.0.1] - 2025-10-17
-
-### Fixed - Task Timestamps
-
-**Fixed task timestamp updates on status changes**
-
----
-
-## [3.0.0] - 2025-10-17
-
-### Added - Kanban Task Watcher
-
-**AI-optimized task management with auto-stale detection**
-
-#### Features
-- Task management with metadata: status, priority, assignee, tags, layer
-- Auto-stale detection: `in_progress` >2h → `waiting_review`, `waiting_review` >24h → `todo`
-- File watching with `chokidar`: auto-transition `todo` → `in_progress` on file edit
-- Link tasks to decisions, constraints, files
-- 70% token reduction vs decision tool (~100 bytes/task vs ~332 bytes/decision)
-- Flat hierarchy (no subtasks) for AI simplicity
-
-#### Database Schema
-- New tables: `t_tasks`, `t_task_details`, `t_task_tags`, `t_task_decision_links`, `t_task_constraint_links`, `t_task_file_links`
-- New triggers: `trg_log_task_create`, `trg_log_task_status_change`, `trg_update_task_timestamp`
-
-#### MCP Actions (task tool)
-- `create`, `update`, `get`, `list`, `move`, `link`, `archive`, `create_batch`
-- `watch_files` - Start file watching for auto-transitions
-
-#### Documentation
-- `TASK_OVERVIEW.md` - Lifecycle, status transitions
-- `TASK_ACTIONS.md` - All action references with examples
-- `TASK_LINKING.md` - Link tasks to decisions/constraints/files
-- `TASK_MIGRATION.md` - Migrate from decision-based tracking
-
----
-
-## [2.1.4] - 2025-10-15
-
-### Fixed - Action Validation
-
-**Enhanced parameter validation for all MCP actions**
-
----
-
-## [2.1.3] - 2025-10-15
-
-### Fixed - Message Priority Handling
-
-**Fixed message priority enum conversion**
-
----
-
-## [2.1.2] - 2025-10-15
-
-### Fixed - File Change Tracking
-
-**Fixed file change timestamp handling**
-
----
-
-## [2.1.1] - 2025-10-15
-
-### Fixed - Constraint Deactivation
-
-**Fixed constraint soft delete logic**
-
----
-
-## [2.1.0] - 2025-10-14
-
-### Added - Template System
-
-**Decision and batch operation templates**
-
-#### Features
-- New actions: `set_from_template`, `create_template`, `list_templates`
-- Template-based decision creation
-- Batch operation support with `set_batch`, `send_batch`, `record_batch`
-
----
-
-## [2.0.0] - 2025-10-11
-
-### Changed - Action-Based Tool Consolidation
-
-**96% token reduction through action-based API**
-
-#### Breaking Changes
-- 20 tools → 6 tools (action-based routing)
-- All tools use `action` parameter for routing
-- Tool names changed: `context` → `decision`, `utils` → `stats`
-
-#### Token Efficiency
-- Tool definitions: 12,848 → 481 tokens (96% reduction)
-- MCP context: ~13,730 → ~4,482 tokens (67% reduction)
-- Help actions provide on-demand documentation
-
-#### New Tool Structure
-- `decision` - Context Management (9 actions)
-- `message` - Agent Messaging (4 actions)
-- `file` - File Change Tracking (4 actions)
-- `constraint` - Constraint Management (4 actions)
-- `stats` - Statistics & Utilities (4 actions)
-- `config` - Configuration (3 actions)
-
----
-
-## [1.1.2] - 2025-10-11
-
-### Fixed - Database Migration
-
-**Fixed v1.2.0 → v1.3.0 table prefix migration**
-
----
-
-## [1.1.1] - 2025-10-11
-
-### Fixed - Auto-Cleanup
-
-**Fixed weekend-aware cleanup trigger timing**
-
----
-
-## [1.1.0] - 2025-10-11
-
-### Added - Weekend-Aware Auto-Deletion
-
-**Configurable retention with weekend-aware logic**
-
-#### Features
-- Configuration keys: `autodelete_ignore_weekend`, `autodelete_message_hours`, `autodelete_file_history_days`
-- CLI arguments for startup override
-- Manual cleanup via `clear_old_data` action
-
----
-
-## [1.0.1] - 2025-10-11
-
-### Fixed - Schema Initialization
-
-**Fixed initial database schema creation**
-
----
-
-## [1.0.0] - 2025-01-10
-
-### Added - Initial Release
-
-**MCP Shared Context Server for efficient context sharing**
-
-#### Core Features
-- Decision tracking with metadata (tags, layers, scopes, versions)
-- Agent messaging with priority levels
-- File change tracking with layer integration
-- Constraint management with priorities
-- Statistics and utilities
-- SQLite-based persistence with better-sqlite3
-
-#### Database Schema
-- Master tables: agents, files, context_keys, layers, tags, scopes, etc.
-- Transaction tables: decisions, messages, file_changes, constraints
-- Views for token-efficient queries
-- Automatic version history tracking
+For changelog entries **v3.4.1 and older**, see:
+- [docs/changelogs/CHANGELOG_ARCHIVE_v3.4_and_older.md](docs/changelogs/CHANGELOG_ARCHIVE_v3.4_and_older.md)
 
-#### MCP Tools
-Initial implementation with 20 separate tools (consolidated to 6 in v2.0.0)
+This includes all versions from v1.0.0 through v3.4.1.