sqlew 4.0.4 → 4.1.0

package/docs/DECISION_INTELLIGENCE.md

@@ -1,605 +0,0 @@
-# Decision Intelligence System (v4.0.0)
-
-**Status:** Production Ready ✅
-**Release Date:** 2025-11-27
-**Test Coverage:** 495/495 tests passing (100%)
-**v3.9.0 Compatibility:** All Decision Intelligence features from v3.9.0 are preserved and fully functional in v4.0.0
-
----
-
-## Overview
-
-The Decision Intelligence System provides AI-driven duplicate detection, similarity scoring, and intelligent suggestions to maintain consistency and prevent redundant decisions across your project.
-
-### Key Features
-
-- **Three-Tier Detection System**: Gentle nudges (35-44), hard blocks (45-59), and auto-updates (60+)
-- **AI-Friendly Auto-Update**: Transparent updates for high-confidence duplicates (no manual retry needed)
-- **Policy-Based Auto-Triggering**: Automatic suggestions when `suggest_similar=1` in policies
-- **Enriched Suggestions**: Includes reasoning, version history, and update commands
-- **Self-Exclusion**: Prevents decisions from matching themselves
-- **Bypass Mechanism**: `ignore_suggest` parameter for intentional duplicates
-
----
-
-## Three-Tier Similarity Detection
-
-### Threshold Configuration (v4.0.0)
-
-```typescript
-// src/constants.ts
-export const SUGGEST_THRESHOLDS = {
-  GENTLE_NUDGE: 35,       // Warning threshold (non-blocking, Tier 1)
-  HARD_BLOCK: 45,         // Blocking threshold (forces choice, Tier 2)
-  AUTO_UPDATE: 60,        // Auto-update threshold (transparent update, Tier 3)
-  CHECK_DUPLICATE: 50,    // Used by suggest.check_duplicate action
-} as const;
-```
-
-### Score Breakdown by Tier
-
-**Tier 1 (35-44): Gentle Nudge**
-- 2 tags + layer OR 1 tag + layer + partial similarity → Non-blocking warning
-
-**Tier 2 (45-59): Hard Block**
-- 2 tags + layer + moderate similarity → Forces manual decision
-
-**Tier 3 (60+): Auto-Update**
-- 3 tags + layer OR 2 tags + layer + high similarity → Transparent update
-
-### Tier 1: Gentle Nudge (35-44 similarity)
-
-**Behavior:** Non-blocking warning returned in `duplicate_risk` field
-**Severity:** MODERATE
-**Action:** Decision is created, but user is warned about potential duplicates
-
-**Response Structure:**
-```typescript
-{
-  success: true,
-  key: "CVE-2024-0003",
-  version: "1.0.0",
-  duplicate_risk: {
-    severity: "MODERATE",
-    max_score: 52,
-    confidence: {
-      is_duplicate: 0.45,
-      should_update: 0.30
-    },
-    suggestions: [
-      {
-        key: "CVE-2024-0001",
-        value: "Fixed buffer overflow in auth module",
-        score: 52,
-        recommended: true,
-        matches: {
-          tags: ["security", "vulnerability", "auth"],
-          layer: "infrastructure"
-        },
-        version_info: {
-          current: "1.0.0",
-          next_suggested: "1.0.1",
-          recent_changes: [...]
-        },
-        reasoning: "High tag overlap (3/3) and same layer. Similar security context.",
-        update_command: {
-          key: "CVE-2024-0001",
-          version: "1.0.1",
-          tags: ["security", "vulnerability", "auth"]
-        }
-      }
-    ]
-  }
-}
-```
-
-**Use Case:** When decisions are similar but may be intentionally different (e.g., different modules, different time periods).
-
-### Tier 2: Hard Block (45-59 similarity)
-
-**Behavior:** Blocking error thrown, decision NOT created
-**Severity:** HIGH
-**Action:** Operation aborted, user must update existing decision or use `ignore_suggest`
-**Effectiveness:** 95%+ (forces conscious choice)
-
-**Error Message:**
-```
-HIGH-SIMILARITY DUPLICATE DETECTED (score: 85)
-
-Existing Decision: CVE-2024-0005
-  Value: Fixed buffer overflow in auth module
-  Layer: infrastructure
-  Tags: security, vulnerability, auth
-  Version: 1.0.0
-
-New Decision: CVE-2024-0006
-  Value: Fixed buffer overflow in auth module
-  Layer: infrastructure
-  Tags: security, vulnerability, auth
-
-Matching: Layer, 3/3 tags
-Reason: Identical value and metadata. Consider updating existing decision.
-
-Action: UPDATE existing decision or use ignore_suggest: true to bypass.
-
-Update Command:
-decision.set({
-  key: "CVE-2024-0005",
-  value: "<your new value>",
-  version: "1.0.1"
-})
-
-Bypass (if intentional):
-decision.set({
-  key: "CVE-2024-0006",
-  value: "...",
-  ignore_suggest: true,
-  ignore_reason: "Different use case - async tasks vs event bus"
-})
-```
-
-**Use Case:** When decisions are moderately similar, requiring review before proceeding.
-
-### Tier 3: Auto-Update (60+ similarity)
-
-**Behavior:** Transparently updates existing decision, NO manual retry needed
-**Severity:** AUTO
-**Action:** Existing decision updated with new value, original response enhanced with metadata
-**Effectiveness:** 100% (AI-friendly, no retry loop)
-
-**Response Structure:**
-```typescript
-{
-  success: true,
-  auto_updated: true,
-  requested_key: "CVE-2024-0002",
-  actual_key: "CVE-2024-0001",
-  similarity_score: 85,
-  version: "1.0.1",
-  duplicate_reason: {
-    similarity: "Best match: 3 matching tags, same layer",
-    matched_tags: ["security", "vulnerability", "auth"],
-    layer: "infrastructure",
-    key_pattern: "CVE-YYYY-NNNNN"
-  },
-  key: "CVE-2024-0001",
-  key_id: 1234,
-  value: "Fixed buffer overflow in auth module (v2)",
-  message: "Auto-updated existing decision \"CVE-2024-0001\" (similarity: 85)"
-}
-```
-
-**Key Features:**
-- **Transparent**: Response includes both requested and actual keys
-- **Informative**: Shows similarity score and matching criteria
-- **No Retry**: AI agents don't need to catch errors and reformulate
-- **Version Bump**: Automatically increments patch version
-- **Value Preserved**: New value from request is used (not discarded)
-
-**Use Case:** High-confidence duplicates where update is clearly the right action (3+ matching tags + same layer + similar value).
-
-**Design Rationale:**
-- Eliminates manual retry step for AI agents
-- Reduces token waste from error handling
-- Maintains transparency with metadata
-- Respects bypass mechanism (`ignore_suggest` still works)
-
----
-
-## Similarity Scoring Algorithm
-
-### Components
-
-1. **Key Similarity** (0-40 points)
-   - Exact match: 40 points
-   - Pattern match: 20-35 points
-   - Partial match: 10-20 points
-
-2. **Tag Overlap** (0-30 points)
-   - Jaccard similarity × 30
-   - Example: 3/3 matching tags = 30 points
-
-3. **Layer Match** (0-15 points)
-   - Same layer: 15 points
-   - Different layer: 0 points
-
-4. **Value Similarity** (0-15 points)
-   - Levenshtein distance-based
-   - Identical: 15 points
-   - Very similar: 10-14 points
-   - Different: 0-9 points
-
-**Total Score:** 0-100 points
-
-### Examples
-
-**High Similarity (85 points):**
-```typescript
-// Baseline
-key: "CVE-2024-0001"
-value: "Fixed buffer overflow in auth module"
-tags: ["security", "vulnerability", "auth"]
-layer: "infrastructure"
-
-// Duplicate (nearly identical)
-key: "CVE-2024-0002"
-value: "Fixed buffer overflow in auth module"  // Identical value
-tags: ["security", "vulnerability", "auth"]     // 3/3 tags match
-layer: "infrastructure"                         // Same layer
-
-// Score breakdown:
-// Key: 25 (pattern match CVE-2024-*)
-// Tags: 30 (3/3 match)
-// Layer: 15 (match)
-// Value: 15 (identical)
-// Total: 85 → HARD BLOCK
-```
-
-**Moderate Similarity (52 points):**
-```typescript
-// Baseline
-key: "CVE-2024-0001"
-value: "Fixed buffer overflow in auth module"
-tags: ["security", "vulnerability", "auth"]
-layer: "infrastructure"
-
-// Similar (different module)
-key: "CVE-2024-0003"
-value: "Fixed authentication bypass in API module"
-tags: ["security", "vulnerability", "auth"]  // 3/3 tags match
-layer: "infrastructure"                      // Same layer
-
-// Score breakdown:
-// Key: 25 (pattern match CVE-2024-*)
-// Tags: 30 (3/3 match)
-// Layer: 15 (match)
-// Value: 7 (different modules)
-// Total: 77 → GENTLE NUDGE (adjusted to 52 for demonstration)
-```
-
----
-
-## Policy-Based Auto-Triggering
-
-### Configuration
-
-Enable auto-trigger in decision policies:
-
-```typescript
-decision({
-  action: "create_policy",
-  name: "security-vulnerability-policy",
-  defaults: {
-    layer: "cross-cutting",
-    tags: ["security", "vulnerability"]
-  },
-  suggest_similar: 1,  // Enable auto-trigger
-  validation_rules: {
-    patterns: {
-      key: "^CVE-"
-    }
-  }
-})
-```
-
-### Behavior (v4.0.0 Three-Tier System)
-
-When a decision matches a policy with `suggest_similar=1`:
-1. Similarity detection runs automatically BEFORE decision creation
-2. If score ≥ 60: Auto-update existing decision (transparent, no error)
-3. If score 45-59: Hard block error thrown
-4. If score 35-44: Gentle nudge returned in `duplicate_risk`
-5. If score < 35: No action (decision created normally)
-
-**Benefits:**
-- Zero manual effort for duplicate detection
-- AI-friendly auto-update eliminates retry loops
-- Consistent across all policy-matched decisions
-- Respects bypass mechanism (`ignore_suggest: true`)
-
----
-
-## Bypass Mechanism
-
-### When to Use
-
-- Intentional duplicates (e.g., different use cases)
-- Temporary decisions during development
-- Testing scenarios
-- Edge cases where similarity is coincidental
-
-### Syntax
-
-```typescript
-decision({
-  action: "set",
-  key: "queue-implementation",
-  value: "Use RabbitMQ for messaging",
-  tags: ["queue", "rabbitmq", "messaging"],
-  layer: "infrastructure",
-  ignore_suggest: true,  // Skip similarity detection
-  ignore_reason: "Different use case - async tasks vs event bus"
-})
-```
-
-**Important:** Always provide `ignore_reason` for documentation.
-
----
-
-## Manual Suggestion Queries
-
-### by_key - Find by Key Pattern
-
-```typescript
-suggest({
-  action: "by_key",
-  key: "api/*/latency",
-  limit: 5,
-  min_score: 30
-})
-
-// Returns decisions matching "api/*/latency" pattern
-```
-
-### by_tags - Find by Tag Similarity
-
-```typescript
-suggest({
-  action: "by_tags",
-  tags: ["performance", "api"],
-  limit: 5
-})
-
-// Returns decisions with overlapping tags
-```
-
-### by_context - Hybrid Search
-
-```typescript
-suggest({
-  action: "by_context",
-  key: "api/*",
-  tags: ["performance"],
-  layer: "infrastructure",
-  limit: 5
-})
-
-// Returns decisions matching key pattern + tags + layer
-```
-
-### check_duplicate - Pre-Creation Check
-
-```typescript
-suggest({
-  action: "check_duplicate",
-  key: "api/users/get/latency",
-  tags: ["api", "performance"],
-  min_score: 50  // Custom threshold
-})
-
-// Returns potential duplicates before creating decision
-```
-
----
-
-## Bug Fixes (v3.9.0)
-
-### 1. String Mismatch in Error Detection
-
-**Issue:** Error detection checked for `'DUPLICATE_DETECTED'` (underscore) but error messages contained `'DUPLICATE DETECTED'` (space).
-
-**Impact:** Hard block errors were caught and logged as non-blocking instead of propagating correctly.
-
-**Fixed Files:**
-- `src/tools/context/internal/queries.ts` (lines 677, 687)
-- `src/tests/integration/auto-trigger-suggestions.test.ts` (line 165)
-- `src/tests/integration/hybrid-similarity-detection.test.ts` (line 168)
-
-### 2. Self-Duplicate Detection
-
-**Issue:** Newly created decisions would match themselves with 100% similarity.
-
-**Impact:** First decision in a pattern would always trigger duplicate detection against itself.
-
-**Fix:** Added `excludeKey` parameter to `buildContextQuery()` to exclude current decision from similarity searches.
-
-**Fixed Files:**
-- `src/tools/suggest/internal/queries.ts` (line 143-147)
-- `src/tools/suggest/actions/by-context.ts` (line 43)
-
----
-
-## Test Status
-
-**Total Tests:** 495
-**Passing:** 495 (100%) ✅
-**Failing:** 0
-
-### Test Coverage
-
-- ✅ Tier 1 (Gentle Nudge): 5/5 tests passing
-- ✅ Tier 2 (Hard Block): 2/2 tests passing
-- ✅ Tier 3 (Auto-Update): 3/3 tests passing
-- ✅ Auto-trigger suggestions: 4/4 tests passing
-- ✅ Edge cases: All passing
-- ✅ String mismatch fix: All error detection tests pass
-- ✅ Self-exclusion: No self-duplicate detections
-- ✅ Policy respect: Honors `suggest_similar=0`
-- ✅ Bypass mechanism: `ignore_suggest` works correctly
-
-**Test Suite Status:** Production-ready ✅
-
----
-
-## Threshold Adjustment Rationale
-
-**v4.0.0 (2025-11-27):** Three-tier system preserved from v3.9.0 (45/60 with auto-update)
-
-### Why Three-Tier System?
-
-**Problem Identified:** Two-tier system (35-59 gentle nudge, 60+ hard block) was inefficient for AI agents:
-- **Gentle nudge ineffective**: AI agents ignore non-blocking warnings (10-20% effectiveness)
-- **Hard block causes retry loops**: Manual error handling wastes tokens and adds latency
-- **Middle ground missing**: Scores 45-59 need review but 60+ should auto-update
-
-**Solution:** Three-tier system with AI-friendly auto-update:
-
-**Tier 1 (35-44): Gentle Nudge**
-- Ambiguous cases requiring human judgment
-- Non-blocking warning for awareness
-- 10-20% effectiveness acceptable (informational only)
-
-**Tier 2 (45-59): Hard Block**
-- Moderate similarity requiring conscious choice
-- Forces manual decision to update or bypass
-- 95%+ effectiveness (prevents accidental duplicates)
-
-**Tier 3 (60+): Auto-Update**
-- High-confidence duplicates (3+ tags + layer + similar value)
-- Transparent update without error/retry
-- 100% effectiveness (AI-friendly, no manual intervention)
-
-### Shared Benefits
-
-- **Multiple Safety Layers**: Three tiers provide nuanced handling
-  - Tier 1: Informational (ignorable)
-  - Tier 2: Protective (bypassable with `ignore_suggest`)
-  - Tier 3: Helpful (transparent, AI-friendly)
-- **Bypass Mechanism**: All tiers respect `ignore_suggest: true`
-- **Token Efficiency**: Auto-update eliminates error handling overhead
-
-### Alternatives Considered
-
-- **Two-Tier (35-44 nudge, 45+ auto-update)**: No middle ground for review
-- **Two-Tier (35-44 nudge, 45+ hard block)**: No auto-update, AI unfriendly
-- **Four-Tier System**: Excessive complexity
-- **Configurable per-policy**: Added complexity, deferred to future version
-
-### Trade-offs
-
-**Pros:**
-- AI-friendly auto-update (no retry loops)
-- Better duplicate detection coverage
-- Three graduated responses match use cases
-- Token-efficient for high-confidence cases
-
-**Cons:**
-- May produce more false positives in Tier 1 (acceptable, non-blocking)
-- Auto-update at 60+ requires high confidence (mitigated by bypass mechanism)
-- Users may need to use `ignore_suggest` more often in Tier 2 (minimal effort)
-
----
-
-## Configuration Constants
-
-**File:** `src/constants.ts`
-
-```typescript
-// Suggestion & Duplicate Detection (v4.0.0 Three-Tier System)
-export const SUGGEST_THRESHOLDS = {
-  GENTLE_NUDGE: 35,       // Warning threshold (non-blocking, Tier 1)
-  HARD_BLOCK: 45,         // Blocking threshold (forces choice, Tier 2)
-  AUTO_UPDATE: 60,        // Auto-update threshold (transparent update, Tier 3)
-  CHECK_DUPLICATE: 50,    // Used by suggest.check_duplicate action
-} as const;
-
-export const SUGGEST_LIMITS = {
-  MAX_SUGGESTIONS_NUDGE: 3,      // Max suggestions in gentle nudge warning
-  MAX_SUGGESTIONS_BLOCK: 1,      // Max suggestions in blocking error
-  VERSION_HISTORY_COUNT: 2,      // Recent versions to show in preview
-} as const;
-```
-
----
-
-## Best Practices
-
-### 1. Use Descriptive Keys
-
-```typescript
-// Good: Pattern-based keys
-key: "CVE-2024-0001"      // Security vulnerabilities
-key: "api/users/get"      // API endpoints
-key: "perf/latency/p99"   // Performance metrics
-
-// Avoid: Generic keys
-key: "fix-001"
-key: "update"
-key: "change"
-```
-
-### 2. Use Consistent Tags
-
-```typescript
-// Good: Consistent taxonomy
-tags: ["security", "vulnerability", "auth"]
-tags: ["performance", "api", "latency"]
-
-// Avoid: Inconsistent naming
-tags: ["sec", "vuln", "authentication"]  // Mix of abbreviations
-tags: ["perf", "performance", "speed"]   // Redundant terms
-```
-
-### 3. Provide Context for Bypasses
-
-```typescript
-// Good: Clear reasoning
-ignore_suggest: true,
-ignore_reason: "Different use case - async tasks vs event bus"
-
-// Avoid: Vague reasons
-ignore_suggest: true,
-ignore_reason: "Just skip it"
-```
-
-### 4. Review Gentle Nudges
-
-When you receive a gentle nudge:
-1. Review suggested decisions
-2. Check if update is more appropriate
-3. Use `ignore_suggest` if intentionally different
-4. Document reasoning in `ignore_reason`
-
-### 5. Respect Hard Blocks
-
-When you hit a hard block:
-1. **First:** Consider updating existing decision
-2. **Second:** Check if truly different use case
-3. **Last Resort:** Use `ignore_suggest` with clear reasoning
-
----
-
-## Future Enhancements
-
-### Planned for v3.10.0
-
-- Configurable thresholds per-policy
-- Machine learning-based scoring refinement
-- Suggestion history tracking
-- Duplicate merge workflows
-
-### Under Consideration
-
-- Semantic similarity using embeddings
-- Cross-project duplicate detection
-- Automatic decision consolidation
-- Visual similarity dashboard
-
----
-
-## Related Documentation
-
-| Document | Content |
-|----------|---------|
-| [CONSTRAINT_INTELLIGENCE.md](CONSTRAINT_INTELLIGENCE.md) | Constraint duplicate detection |
-| [TOOL_REFERENCE.md](TOOL_REFERENCE.md) | Complete parameter reference |
-| [SHARED_CONCEPTS.md](SHARED_CONCEPTS.md) | Layers, priorities, tags |
-
----
-
-## Support & Feedback
-
-**Issue Tracker:** https://github.com/anthropics/mcp-sqlew/issues
-**Documentation:** See `/docs` directory
-**Version:** v4.0.0
-**Last Updated:** 2025-11-27