claude-self-reflect 4.0.0 → 4.0.2

package/.claude/agents/csr-validator.md

@@ -0,0 +1,151 @@
+---
+name: csr-validator
+description: Validates Claude Self-Reflect system functionality. Use for testing MCP tools, embedding modes, import pipeline, and search. MUST BE USED before releases and after major changes.
+tools: mcp__claude-self-reflect__switch_embedding_mode, mcp__claude-self-reflect__get_embedding_mode, mcp__claude-self-reflect__store_reflection, mcp__claude-self-reflect__csr_reflect_on_past, mcp__claude-self-reflect__csr_quick_check, mcp__claude-self-reflect__csr_search_insights, mcp__claude-self-reflect__get_recent_work, mcp__claude-self-reflect__search_by_recency, mcp__claude-self-reflect__get_timeline, mcp__claude-self-reflect__search_by_file, mcp__claude-self-reflect__search_by_concept, mcp__claude-self-reflect__get_full_conversation, mcp__claude-self-reflect__get_next_results, mcp__claude-self-reflect__csr_get_more, mcp__claude-self-reflect__reload_code, mcp__claude-self-reflect__reload_status, mcp__claude-self-reflect__clear_module_cache, Bash, Read
+model: inherit
+---
+
+You are a focused CSR system validator. Test ONLY through MCP protocol - NEVER import Python modules directly.
+
+## Test Sequence (MANDATORY ORDER)
+
+### 1. Mode Testing
+```
+1. Get current mode (get_embedding_mode)
+2. Switch to CLOUD mode (switch_embedding_mode)
+3. Verify 1024 dimensions
+4. Store test reflection with tag "cloud-test-{timestamp}"
+5. Search for it immediately
+6. Switch to LOCAL mode
+7. Verify 384 dimensions
+8. Store test reflection with tag "local-test-{timestamp}"
+9. Search for it immediately
+```
+
+### 2. MCP Tools Validation (ALL 15+)
+Test each tool with minimal viable input:
+- `csr_reflect_on_past`: Query "test"
+- `csr_quick_check`: Query "system"
+- `store_reflection`: Content with unique timestamp
+- `get_recent_work`: Limit 2
+- `search_by_recency`: Query "import", time_range "today"
+- `get_timeline`: Range "last hour"
+- `search_by_file`: Path "*.py"
+- `search_by_concept`: Concept "testing"
+- `get_full_conversation`: Use any recent ID
+- `csr_search_insights`: Query "performance"
+- `csr_get_more`: After any search
+- `get_next_results`: After any search
+- `reload_status`: Check reload state
+- `clear_module_cache`: If needed
+- `reload_code`: If status shows changes
+
+### 3. Security Scan (CRITICAL)
+```bash
+# Scan for hardcoded paths
+grep -r "/Users/[a-zA-Z]*/\|/home/[a-zA-Z]*/" scripts/ --include="*.py" | grep -v "^#" | head -20
+
+# Scan for API keys/secrets (VOYAGE_KEY, etc)
+grep -r "VOYAGE_KEY\|API_KEY\|SECRET\|PASSWORD" scripts/ --include="*.py" | grep -v "os.environ\|getenv" | head -10
+
+# Check for sensitive patterns in state files
+grep -E "(api_key|secret|password|token)" ~/.claude-self-reflect/config/*.json | head -10
+
+# Find transient test files
+find . -name "*test*.py" -o -name "*benchmark*.py" -o -name "*tmp*" -o -name "*.pyc" | grep -v ".git" | head -20
+```
+
+### 4. Performance Check
+```bash
+# Via Bash tool only
+time python -c "from datetime import datetime; print(datetime.now())"
+ps aux | grep python | head -5
+docker ps --format "table {{.Names}}\t{{.Status}}" | grep qdrant
+```
+
+### 5. State Verification
+```bash
+# Check unified state
+ls -la ~/.claude-self-reflect/config/unified-state.json
+wc -l ~/.claude-self-reflect/config/unified-state.json
+head -20 ~/.claude-self-reflect/config/unified-state.json
+```
+
+### 6. CodeRabbit CLI Analysis
+```bash
+# Run CodeRabbit for code quality check
+echo "=== Running CodeRabbit CLI ==="
+coderabbit --version
+script -q /dev/null coderabbit --prompt-only || echo "CodeRabbit CLI issues detected - terminal mode incompatibility"
+
+# Alternative: Check GitHub PR for CodeRabbit comments
+echo "=== Checking PR CodeRabbit feedback ==="
+gh pr list --state open --limit 1 --json number --jq '.[0].number' | xargs -I {} gh pr view {} --comments | grep -A 5 "coderabbitai" || echo "No open PRs with CodeRabbit feedback"
+```
+
+### 7. Cleanup Transient Files
+```bash
+# List transient files (DO NOT DELETE YET)
+echo "=== Transient files found ==="
+find . -type f \( -name "*test_*.py" -o -name "test_*.py" -o -name "*benchmark*.py" \) -not -path "./.git/*" -not -path "./tests/*"
+
+# Archive or mark for deletion
+echo "=== Suggest archiving to: tests/throwaway/ ==="
+```
+
+## Output Format
+
+```
+CSR VALIDATION REPORT
+====================
+SECURITY SCAN: [PASS/FAIL]
+- Hardcoded paths: [0 found/X found - LIST THEM]
+- API keys exposed: [0 found/X found - LIST THEM]
+- Sensitive data: [none/FOUND - LIST]
+- Transient files: [X files - LIST FOR CLEANUP]
+
+Mode Switching: [PASS/FAIL]
+- Local→Cloud: [✓/✗]
+- Cloud→Local: [✓/✗]
+- Dimensions: [384/1024 verified]
+
+MCP Tools (15/15):
+- csr_reflect_on_past: [✓/✗]
+- [... list all ...]
+
+Performance:
+- Search latency: [Xms]
+- Memory usage: [XMB]
+- Qdrant status: [healthy/unhealthy]
+
+CodeRabbit Analysis: [PASS/FAIL]
+- CLI execution: [✓/✗ - terminal mode issues]
+- PR feedback checked: [✓/✗]
+- Issues found: [none/list]
+
+Critical Issues: [none/list]
+
+CLEANUP NEEDED:
+- [ ] Remove: [list transient files]
+- [ ] Archive: [list test files]
+- [ ] Fix: [list hardcoded paths]
+
+VERDICT: [GREEN/YELLOW/RED]
+```
+
+## Rules
+1. NEVER import Python modules (no `from X import Y`)
+2. Use ONLY mcp__claude-self-reflect__ prefixed tools
+3. Use Bash for system checks ONLY (no Python scripts)
+4. Report EVERY failure, even minor
+5. Test BOTH modes completely
+6. Restore to LOCAL mode at end
+7. Complete in <2 minutes
+
+## Failure Handling
+- If any MCP tool fails: Report exact error, continue testing others
+- If mode switch fails: CRITICAL - stop and report
+- If search returns no results: Note but continue
+- If Bash fails: Try alternative command
+
+Focus: Validate MCP protocol layer functionality, not implementation details.

package/.claude/agents/open-source-maintainer.md

@@ -6,8 +6,42 @@ tools: Read, Write, Edit, Bash, Grep, Glob, LS, WebFetch
 
 You are an open-source project maintainer for the Claude Self Reflect project. Your expertise covers community management, release processes, and maintaining a healthy, welcoming project.
 
+## CRITICAL WORKFLOW - MUST FOLLOW THIS SEQUENCE
+
+### Complete Release Flow (CSR Tester → Open Source Maintainer → NPM)
+1. **Code Review Phase**
+   - Check CodeRabbit feedback on existing PRs
+   - Fix ALL identified issues locally
+   - Create feature branch for fixes
+
+2. **PR Creation Phase**
+   - Create PR with all fixes
+   - Monitor CodeRabbit automated review on the PR
+   - Address any new issues CodeRabbit identifies
+   - Ensure all CI/CD checks pass
+
+3. **PR Merge Phase**
+   - Request review/approval
+   - Merge PR to main branch
+   - Verify merge completed successfully
+
+4. **Release Creation Phase**
+   - Create GitHub release with comprehensive notes
+   - Tag appropriately following semver
+   - Monitor automated workflows
+
+5. **NPM Publication Phase**
+   - Watch CI/CD pipeline for npm publish
+   - Verify package published to npm registry
+   - Test installation: `npm install -g claude-self-reflect@latest`
+
+6. **Post-Release Phase**
+   - Close related issues with release references
+   - Update project documentation
+   - Announce release in discussions/social
+
 ## Core Workflow: Explore, Plan, Execute, Verify
-1. **Explore**: Read relevant files, check git history, review PRs
+1. **Explore**: Read relevant files, check git history, review PRs, check CodeRabbit feedback
 2. **Plan**: Think hard about the release strategy before executing
 3. **Execute**: Implement the release with proper checks
 4. **Verify**: Use independent verification (or ask user to verify)
@@ -81,13 +115,18 @@ git log -p --grep="feature name"
 gh pr list --state merged --limit 10
 ```
 
-### PR Review Process
+### PR Review Process with CodeRabbit
 1. Thank contributor for their time
-2. Run CI/CD checks
-3. Review code for quality and style
-4. Test changes locally
-5. Provide constructive feedback
-6. Merge with descriptive commit message
+2. Check CodeRabbit automated review comments
+   ```bash
+   gh pr view PR_NUMBER --comments | grep -B2 -A10 "coderabbitai"
+   ```
+3. Address any CodeRabbit-identified issues
+4. Run CI/CD checks
+5. Review code for quality and style
+6. Test changes locally
+7. Provide constructive feedback
+8. Merge with descriptive commit message
 
 ### Release Checklist
 

package/mcp-server/src/parallel_search.py

@@ -83,9 +83,14 @@ async def search_single_collection(
                 with_payload=True
             )
 
+            # CRITICAL FIX: Handle None search results (cloud mode issue)
+            if search_results is None:
+                logger.warning(f"Search returned None for collection {collection_name}")
+                search_results = []
+
             # Debug: Log search results
             logger.debug(f"Search of {collection_name} returned {len(search_results)} results")
-            
+
             if should_use_decay and not USE_NATIVE_DECAY:
                 # Apply client-side decay
                 await ctx.debug(f"Using CLIENT-SIDE decay for {collection_name}")

package/mcp-server/src/search_tools.py

@@ -102,9 +102,15 @@ class SearchTools:
                 collection_name=collection_name,
                 query_vector=query_embedding,
                 limit=limit,
-                score_threshold=min_score
+                score_threshold=min_score,
+                with_payload=True  # Explicitly request payloads from Qdrant
             )
-            
+
+            # CRITICAL FIX: Handle None search results (cloud mode issue)
+            if search_results is None:
+                logger.warning(f"Search returned None for collection {collection_name}")
+                search_results = []
+
             # Convert results to dict format
             results = []
             for result in search_results:

package/mcp-server/src/status_unified.py

@@ -0,0 +1,286 @@
+"""Ultra-fast status checker using unified state management.
+
+This module reads from the unified state file for indexing status.
+Designed for <20ms execution time to support status bars and shell scripts.
+"""
+
+import json
+import time
+import sys
+from pathlib import Path
+from collections import defaultdict
+
+# Add scripts directory to path for unified state manager
+scripts_dir = Path(__file__).parent.parent.parent / "scripts"
+if scripts_dir.exists():
+    sys.path.insert(0, str(scripts_dir))
+
+try:
+    from unified_state_manager import UnifiedStateManager
+except ImportError:
+    # Fallback to reading JSON directly if manager not available
+    UnifiedStateManager = None
+
+# Try to import shared utilities
+try:
+    from shared_utils import (
+        extract_project_name_from_path,
+        get_claude_projects_dir,
+        get_csr_config_dir
+    )
+except ImportError:
+    # Fallback implementations
+    def extract_project_name_from_path(file_path: str) -> str:
+        """Extract project name from JSONL file path."""
+        path_obj = Path(file_path)
+        dir_name = path_obj.parent.name
+
+        if dir_name.startswith('-') and 'projects' in dir_name:
+            parts = dir_name.split('-')
+            try:
+                projects_idx = parts.index('projects')
+                if projects_idx + 1 < len(parts):
+                    project_parts = parts[projects_idx + 1:]
+                    return '-'.join(project_parts)
+            except ValueError:
+                pass
+
+        return dir_name.lstrip('-')
+
+    def get_claude_projects_dir() -> Path:
+        """Get Claude projects directory."""
+        import os
+        if 'CLAUDE_PROJECTS_DIR' in os.environ:
+            return Path(os.environ['CLAUDE_PROJECTS_DIR'])
+        return Path.home() / ".claude" / "projects"
+
+    def get_csr_config_dir() -> Path:
+        """Get CSR config directory."""
+        import os
+        if 'CSR_CONFIG_DIR' in os.environ:
+            return Path(os.environ['CSR_CONFIG_DIR'])
+        return Path.home() / '.claude-self-reflect' / 'config'
+
+
+def get_watcher_status() -> dict:
+    """Get streaming watcher status from unified state."""
+    try:
+        if UnifiedStateManager:
+            manager = UnifiedStateManager()
+            state = manager.read_state()
+
+            # Get watcher status from importers section
+            watcher_info = state.get("importers", {}).get("streaming", {})
+            last_run = watcher_info.get("last_run")
+
+            if last_run:
+                from datetime import datetime, timezone
+                last_run_dt = datetime.fromisoformat(last_run)
+                now = datetime.now(timezone.utc)
+                age_seconds = (now - last_run_dt).total_seconds()
+                is_active = age_seconds < 120  # Active if updated in last 2 minutes
+            else:
+                is_active = False
+                age_seconds = float('inf')
+
+            return {
+                "running": is_active,
+                "files_processed": watcher_info.get("files_processed", 0),
+                "last_update_seconds": int(age_seconds) if age_seconds != float('inf') else None,
+                "status": "🟢 active" if is_active else "🔴 inactive"
+            }
+        else:
+            # Fallback to old method if UnifiedStateManager not available
+            watcher_state_file = get_csr_config_dir() / "csr-watcher.json"
+
+            if not watcher_state_file.exists():
+                return {"running": False, "status": "not configured"}
+
+            with open(watcher_state_file) as f:
+                state = json.load(f)
+
+            file_age = time.time() - watcher_state_file.stat().st_mtime
+            is_active = file_age < 120
+
+            return {
+                "running": is_active,
+                "files_processed": len(state.get("imported_files", {})),
+                "last_update_seconds": int(file_age),
+                "status": "🟢 active" if is_active else "🔴 inactive"
+            }
+    except Exception as e:
+        return {"running": False, "status": f"error: {str(e)[:50]}"}
+
+
+def get_status() -> dict:
+    """Get indexing status from unified state with per-project breakdown.
+
+    Returns:
+        dict: JSON structure with overall and per-project indexing status
+    """
+    start_time = time.time()
+
+    try:
+        if UnifiedStateManager:
+            # Use unified state manager for fast access
+            manager = UnifiedStateManager()
+            status = manager.get_status()
+
+            # Get per-project breakdown
+            project_stats = defaultdict(lambda: {"indexed": 0, "total": 0})
+
+            # Count total JSONL files per project
+            projects_dir = get_claude_projects_dir()
+            if projects_dir.exists():
+                for jsonl_file in projects_dir.glob("**/*.jsonl"):
+                    project_name = extract_project_name_from_path(str(jsonl_file))
+                    project_stats[project_name]["total"] += 1
+
+            # Count indexed files per project from unified state
+            state = manager.read_state()
+            for file_path, metadata in state.get("files", {}).items():
+                if metadata.get("status") == "completed":
+                    project_name = extract_project_name_from_path(file_path)
+                    if project_name in project_stats:
+                        project_stats[project_name]["indexed"] += 1
+
+            # Format response
+            result = {
+                "overall": {
+                    "percentage": status["percentage"],
+                    "indexed_files": status["indexed_files"],
+                    "total_files": status["total_files"],
+                    "total_chunks": status["total_chunks"],
+                },
+                "watcher": get_watcher_status(),
+                "projects": dict(project_stats),
+                "execution_time_ms": round((time.time() - start_time) * 1000, 2)
+            }
+
+            return result
+
+        else:
+            # Fallback to old multi-file method
+            return get_status_legacy()
+
+    except Exception as e:
+        return {
+            "error": str(e),
+            "execution_time_ms": round((time.time() - start_time) * 1000, 2)
+        }
+
+
+def get_status_legacy() -> dict:
+    """Legacy status method reading from multiple files (fallback)."""
+    projects_dir = get_claude_projects_dir()
+    project_stats = defaultdict(lambda: {"indexed": 0, "total": 0})
+
+    # Count total JSONL files per project
+    if projects_dir.exists():
+        for jsonl_file in projects_dir.glob("**/*.jsonl"):
+            file_str = str(jsonl_file)
+            project_name = extract_project_name_from_path(file_str)
+            project_stats[project_name]["total"] += 1
+
+    # Read imported-files.json to count indexed files
+    config_dir = get_csr_config_dir()
+    imported_files_path = config_dir / "imported-files.json"
+
+    if imported_files_path.exists():
+        try:
+            with open(imported_files_path, 'r') as f:
+                data = json.load(f)
+                imported_files = data.get("imported_files", {})
+
+                for file_path in imported_files.keys():
+                    # Normalize path
+                    if file_path.startswith("/logs/"):
+                        projects_path = str(get_claude_projects_dir())
+                        normalized_path = file_path.replace("/logs/", projects_path + "/", 1)
+                    else:
+                        normalized_path = file_path
+
+                    # Check if file exists and count it
+                    if Path(normalized_path).exists():
+                        project_name = extract_project_name_from_path(normalized_path)
+                        if project_name in project_stats:
+                            project_stats[project_name]["indexed"] += 1
+        except Exception:
+            pass
+
+    # Calculate overall stats
+    total_files = sum(p["total"] for p in project_stats.values())
+    indexed_files = sum(p["indexed"] for p in project_stats.values())
+    percentage = (indexed_files / max(total_files, 1)) * 100
+
+    return {
+        "overall": {
+            "percentage": percentage,
+            "indexed_files": indexed_files,
+            "total_files": total_files
+        },
+        "watcher": get_watcher_status(),
+        "projects": dict(project_stats)
+    }
+
+
+def main():
+    """CLI interface for status checking."""
+    import argparse
+
+    parser = argparse.ArgumentParser(description="Check Claude Self-Reflect indexing status")
+    parser.add_argument("--format", choices=["json", "text"], default="json",
+                       help="Output format (default: json)")
+    parser.add_argument("--watch", action="store_true",
+                       help="Watch mode - update every 2 seconds")
+
+    args = parser.parse_args()
+
+    if args.watch:
+        try:
+            while True:
+                status = get_status()
+                if args.format == "json":
+                    print(json.dumps(status, indent=2))
+                else:
+                    overall = status.get("overall", {})
+                    print(f"Indexing: {overall.get('percentage', 0):.1f}% "
+                          f"({overall.get('indexed_files', 0)}/{overall.get('total_files', 0)})")
+
+                    watcher = status.get("watcher", {})
+                    print(f"Watcher: {watcher.get('status', '🔴 inactive')}")
+
+                    if status.get("execution_time_ms"):
+                        print(f"Time: {status['execution_time_ms']}ms")
+
+                print("\n" + "-" * 40)
+                time.sleep(2)
+
+        except KeyboardInterrupt:
+            print("\nStopped")
+    else:
+        status = get_status()
+        if args.format == "json":
+            print(json.dumps(status, indent=2))
+        else:
+            overall = status.get("overall", {})
+            print(f"Indexing: {overall.get('percentage', 0):.1f}% "
+                  f"({overall.get('indexed_files', 0)}/{overall.get('total_files', 0)} files)")
+
+            watcher = status.get("watcher", {})
+            print(f"Watcher: {watcher.get('status', '🔴 inactive')}")
+
+            # Show per-project if available
+            projects = status.get("projects", {})
+            if projects:
+                print("\nProjects:")
+                for proj, stats in projects.items():
+                    pct = (stats["indexed"] / max(stats["total"], 1)) * 100
+                    print(f"  {proj}: {pct:.1f}% ({stats['indexed']}/{stats['total']})")
+
+            if status.get("execution_time_ms"):
+                print(f"\nExecution time: {status['execution_time_ms']}ms")
+
+
+if __name__ == "__main__":
+    main()

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-self-reflect",
-  "version": "4.0.0",
+  "version": "4.0.2",
   "description": "Give Claude perfect memory of all your conversations - Installation wizard for Python MCP server",
   "keywords": [
     "claude",
@@ -35,6 +35,9 @@
   },
   "files": [
     "installer/*.js",
+    "scripts/auto-migrate.cjs",
+    "scripts/migrate-to-unified-state.py",
+    "scripts/unified_state_manager.py",
     "scripts/csr-status",
     "scripts/session_quality_tracker.py",
     "scripts/ast_grep_final_analyzer.py",
@@ -68,7 +71,7 @@
     "LICENSE"
   ],
   "scripts": {
-    "postinstall": "node installer/postinstall.js"
+    "postinstall": "node installer/postinstall.js && node scripts/auto-migrate.cjs || true"
   },
   "engines": {
     "node": ">=18.0.0"

package/scripts/auto-migrate.cjs

@@ -0,0 +1,84 @@
+#!/usr/bin/env node
+
+const { execSync } = require('child_process');
+const fs = require('fs');
+const path = require('path');
+const os = require('os');
+
+console.log('🔄 Claude Self-Reflect: Checking for required migrations...');
+
+const homeDir = os.homedir();
+const csrConfigDir = path.join(homeDir, '.claude-self-reflect', 'config');
+const unifiedStateFile = path.join(csrConfigDir, 'unified-state.json');
+const legacyFiles = [
+    'imported-files.json',
+    'skipped_files.json',
+    'failed_files.json',
+    'import-status.json',
+    'streaming-state.json'
+];
+
+// Check if migration is needed
+const needsMigration = legacyFiles.some(file =>
+    fs.existsSync(path.join(csrConfigDir, file))
+);
+
+if (!needsMigration && fs.existsSync(unifiedStateFile)) {
+    console.log('✅ Already using Unified State Management v5.0');
+    process.exit(0);
+}
+
+if (needsMigration) {
+    console.log('📦 Legacy state files detected. Running automatic migration...');
+    console.log('📋 Creating backup of existing state files...');
+
+    try {
+        // Check if Python is available
+        try {
+            execSync('python3 --version', { stdio: 'ignore' });
+        } catch {
+            console.log('⚠️  Python 3 not found. Migration will run when you first use the MCP server.');
+            console.log('   To run migration manually: python3 scripts/migrate-to-unified-state.py');
+            process.exit(0);
+        }
+
+        // Check if the migration script exists (npm global install location)
+        const scriptLocations = [
+            path.join(__dirname, 'migrate-to-unified-state.py'),
+            path.join(homeDir, '.claude-self-reflect', 'scripts', 'migrate-to-unified-state.py'),
+            path.join(process.cwd(), 'scripts', 'migrate-to-unified-state.py')
+        ];
+
+        let migrationScript = null;
+        for (const location of scriptLocations) {
+            if (fs.existsSync(location)) {
+                migrationScript = location;
+                break;
+            }
+        }
+
+        if (!migrationScript) {
+            console.log('⚠️  Migration script not found. It will run automatically when the MCP server starts.');
+            process.exit(0);
+        }
+
+        // Run the migration
+        console.log(`🚀 Running migration from: ${migrationScript}`);
+        const result = execSync(`python3 "${migrationScript}"`, {
+            encoding: 'utf-8',
+            stdio: 'pipe'
+        });
+
+        console.log(result);
+        console.log('✅ Migration completed successfully!');
+        console.log('🎉 Now using Unified State Management v5.0 (20x faster!)');
+
+    } catch (error) {
+        console.log('⚠️  Migration encountered an issue:', error.message);
+        console.log('   Your existing state files are preserved.');
+        console.log('   To run migration manually: python3 scripts/migrate-to-unified-state.py');
+        console.log('   For help: https://github.com/ramakay/claude-self-reflect/issues');
+    }
+} else {
+    console.log('✅ Fresh installation - using Unified State Management v5.0');
+}