claude-self-reflect 2.5.18 → 2.6.0

package/.claude/agents/claude-self-reflect-test.md

@@ -13,6 +13,82 @@ You are a resilient and comprehensive testing specialist for Claude Self-Reflect
 - MCP tools enable reflection and memory storage
 - System must handle sensitive API keys securely
 
+## Comprehensive Test Suite
+
+### Available Test Categories
+The project now includes a comprehensive test suite in `/tests/` directory:
+
+1. **MCP Tool Integration** (`test_mcp_tools_comprehensive.py`)
+   - All MCP tools with various parameters
+   - Edge cases and error handling
+   - Cross-project search validation
+
+2. **Memory Decay** (`test_memory_decay.py`)
+   - Decay calculations and half-life variations
+   - Score adjustments and ranking changes
+   - Performance impact measurements
+
+3. **Multi-Project Support** (`test_multi_project.py`)
+   - Project isolation and collection naming
+   - Cross-project search functionality
+   - Metadata storage and retrieval
+
+4. **Embedding Models** (`test_embedding_models.py`)
+   - FastEmbed vs Voyage AI switching
+   - Dimension compatibility (384 vs 1024)
+   - Model performance comparisons
+
+5. **Delta Metadata** (`test_delta_metadata.py`)
+   - Tool usage extraction
+   - File reference tracking
+   - Incremental updates without re-embedding
+
+6. **Performance & Load** (`test_performance_load.py`)
+   - Large conversation imports (>1000 chunks)
+   - Concurrent operations
+   - Memory and CPU monitoring
+
+7. **Data Integrity** (`test_data_integrity.py`)
+   - Duplicate detection
+   - Unicode handling
+   - Chunk ordering preservation
+
+8. **Recovery Scenarios** (`test_recovery_scenarios.py`)
+   - Partial import recovery
+   - Container restart resilience
+   - State file corruption handling
+
+9. **Security** (`test_security.py`)
+   - API key validation
+   - Input sanitization
+   - Path traversal prevention
+
+### Running the Test Suite
+```bash
+# Run ALL tests
+cd ~/projects/claude-self-reflect
+python tests/run_all_tests.py
+
+# Run specific categories
+python tests/run_all_tests.py -c mcp_tools memory_decay multi_project
+
+# Run with verbose output
+python tests/run_all_tests.py -v
+
+# List available test categories
+python tests/run_all_tests.py --list
+
+# Run individual test files
+python tests/test_mcp_tools_comprehensive.py
+python tests/test_memory_decay.py
+python tests/test_multi_project.py
+```
+
+### Test Results Location
+- JSON results: `tests/test_results.json`
+- Contains timestamps, durations, pass/fail counts
+- Useful for tracking test history
+
 ## Key Responsibilities
 
 1. **System State Detection**

package/.claude/agents/mcp-integration.md

@@ -16,6 +16,51 @@ You are an MCP server development specialist for the memento-stack project. You
 - Supports both local (FastEmbed) and cloud (Voyage AI) embeddings
 - MCP determines project from working directory context
 
+## Available Test Suites
+
+### MCP-Specific Tests
+1. **Comprehensive MCP Tool Tests** (`tests/test_mcp_tools_comprehensive.py`)
+   - Tests all MCP tools: reflect_on_past, store_reflection, quick_search, search_summary
+   - Edge case handling and error scenarios
+   - Parameter validation (limit, min_score, use_decay, response_format)
+   - Cross-project search with project="all"
+   - Run with: `python tests/test_mcp_tools_comprehensive.py`
+
+2. **MCP Search Tests** (`scripts/test-mcp-search.py`)
+   - Basic MCP search functionality
+   - Integration with Qdrant backend
+   - Response parsing and formatting
+   - Run with: `python scripts/test-mcp-search.py`
+
+3. **MCP Robustness Tests** (`scripts/test-mcp-robustness.py`)
+   - Error recovery mechanisms
+   - Timeout handling
+   - Connection resilience
+   - Run with: `python scripts/test-mcp-robustness.py`
+
+### Running MCP Tests
+```bash
+# Run all MCP tests
+cd ~/projects/claude-self-reflect
+python tests/run_all_tests.py -c mcp_tools mcp_search
+
+# Test MCP server directly
+cd mcp-server && python test_server.py
+
+# Verify MCP registration in Claude Code
+claude mcp list | grep claude-self-reflect
+
+# Test MCP tools from Python
+python -c "from mcp_server.src.server import reflect_on_past; import asyncio; asyncio.run(reflect_on_past({'query': 'test', 'limit': 5}))"
+```
+
+### MCP Tool Parameters Reference
+- **reflect_on_past**: query, limit, brief, min_score, project, use_decay, response_format, include_raw
+- **store_reflection**: content, tags
+- **quick_search**: query, min_score, project
+- **search_by_file**: file_path, limit, project
+- **search_by_concept**: concept, include_files, limit, project
+
 ## Key Responsibilities
 
 1. **MCP Server Development**

package/.claude/agents/qdrant-specialist.md

@@ -49,6 +49,47 @@ You are a Qdrant vector database specialist for the memento-stack project. Your
 - **Monitor baseline + headroom** - Measure actual usage before setting limits
 - **Use cgroup-aware CPU monitoring** - Docker shows all CPUs but has limits
 
+## Available Test Suites
+
+### Qdrant-Specific Tests
+1. **Multi-Project Support Tests** (`tests/test_multi_project.py`)
+   - Collection isolation verification
+   - Cross-project search functionality
+   - Collection naming consistency
+   - Project metadata storage
+   - Run with: `python tests/test_multi_project.py`
+
+2. **Data Integrity Tests** (`tests/test_data_integrity.py`)
+   - Duplicate detection
+   - Chunk ordering preservation
+   - Unicode and special character handling
+   - Collection consistency checks
+
+3. **Performance Tests** (`tests/test_performance_load.py`)
+   - Large conversation imports (>1000 chunks)
+   - Concurrent search requests
+   - Memory usage patterns
+   - Collection size limits
+
+### How to Run Tests
+```bash
+# Run all Qdrant-related tests
+cd ~/projects/claude-self-reflect
+python tests/run_all_tests.py -c multi_project data_integrity performance
+
+# Check collection health
+docker exec claude-reflection-qdrant curl -s http://localhost:6333/collections | jq
+
+# Verify specific collection
+python -c "from qdrant_client import QdrantClient; c=QdrantClient('localhost', 6333); print(c.get_collection('conv_HASH_local'))"
+```
+
+### Common Issues & Solutions
+- **Dimension mismatch**: Check embedding model (384 for local, 1024 for voyage)
+- **Empty search results**: Verify collection exists and has points
+- **Slow searches**: Check collection size and optimize with filters
+- **Collection not found**: Verify project name normalization and MD5 hash
+
 ### Quality Gates
 - **Follow the workflow**: implementation → review → test → docs → release
 - **Use pre-releases for major changes** - Better to test than break production

package/README.md

@@ -1,6 +1,34 @@
 # Claude Self-Reflect
 
-Claude forgets everything. This fixes that.
+<div align="center">
+
+[![npm version](https://badge.fury.io/js/claude-self-reflect.svg)](https://www.npmjs.com/package/claude-self-reflect)
+[![npm downloads](https://img.shields.io/npm/dm/claude-self-reflect.svg)](https://www.npmjs.com/package/claude-self-reflect)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![GitHub CI](https://github.com/ramakay/claude-self-reflect/actions/workflows/ci.yml/badge.svg)](https://github.com/ramakay/claude-self-reflect/actions/workflows/ci.yml)
+
+[![Claude Code](https://img.shields.io/badge/Claude%20Code-Compatible-6B4FBB)](https://github.com/anthropics/claude-code)
+[![MCP Protocol](https://img.shields.io/badge/MCP-Enabled-FF6B6B)](https://modelcontextprotocol.io/)
+[![Docker](https://img.shields.io/badge/Docker-Ready-2496ED?logo=docker&logoColor=white)](https://www.docker.com/)
+[![Local First](https://img.shields.io/badge/Local%20First-Privacy-4A90E2)](https://github.com/ramakay/claude-self-reflect)
+
+[![GitHub stars](https://img.shields.io/github/stars/ramakay/claude-self-reflect.svg?style=social)](https://github.com/ramakay/claude-self-reflect/stargazers)
+[![GitHub issues](https://img.shields.io/github/issues/ramakay/claude-self-reflect.svg)](https://github.com/ramakay/claude-self-reflect/issues)
+[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/ramakay/claude-self-reflect/pulls)
+
+</div>
+
+**Claude forgets everything. This fixes that.**
+
+Give Claude perfect memory of all your conversations. Search past discussions instantly. Never lose context again.
+
+**100% Local by Default** - Your conversations never leave your machine. No cloud services required, no API keys needed, complete privacy out of the box.
+
+**Blazing Fast Search** - Semantic search across thousands of conversations in milliseconds. Find that discussion about database schemas from three weeks ago in seconds.
+
+**Zero Configuration** - Works immediately after installation. Smart auto-detection handles everything. No manual setup, no environment variables, just install and use.
+
+**Production Ready** - Battle-tested with 600+ conversations across 24 projects. Handles mixed embedding types automatically. Scales from personal use to team deployments.
 
 ## Table of Contents
 
@@ -176,8 +204,33 @@ Recent conversations matter more. Old ones fade. Like your brain, but reliable.
 - [GitHub Issues](https://github.com/ramakay/claude-self-reflect/issues)
 - [Discussions](https://github.com/ramakay/claude-self-reflect/discussions)
 
+## Upgrading to v2.5.19
+
+### 🆕 New Feature: Metadata Enrichment
+v2.5.19 adds searchable metadata to your conversations - concepts, files, and tools!
+
+#### For Existing Users
+```bash
+# Update to latest version
+npm update -g claude-self-reflect
+
+# Run setup - it will detect your existing installation
+claude-self-reflect setup
+# Choose "yes" when asked about metadata enrichment
+
+# Or manually enrich metadata anytime:
+docker compose run --rm importer python /app/scripts/delta-metadata-update-safe.py
+```
+
+#### What You Get
+- `search_by_concept("docker")` - Find conversations by topic
+- `search_by_file("server.py")` - Find conversations that touched specific files
+- Better search accuracy with metadata-based filtering
+
 ## What's New
 
+- **v2.5.19** - Metadata Enrichment! Search by concepts, files, and tools. [Full release notes](docs/releases/v2.5.19-RELEASE-NOTES.md)
+- **v2.5.18** - Security dependency updates
 - **v2.5.17** - Critical CPU fix and memory limit adjustment. [Full release notes](docs/releases/v2.5.17-release-notes.md)
 - **v2.5.16** - (Pre-release only) Initial streaming importer with CPU throttling
 - **v2.5.15** - Critical bug fixes and collection creation improvements

package/installer/setup-wizard-docker.js

@@ -406,6 +406,54 @@ async function importConversations() {
   }
 }
 
+async function enrichMetadata() {
+  console.log('\n🔍 Metadata Enrichment (NEW in v2.5.19!)...');
+  console.log('━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━');
+  console.log('This feature enhances your conversations with searchable metadata:');
+  console.log('   • Concepts: High-level topics (docker, security, testing, etc.)');
+  console.log('   • Files: Track which files were analyzed or edited');
+  console.log('   • Tools: Record which Claude tools were used');
+  console.log('\nEnables powerful searches like:');
+  console.log('   • search_by_concept("docker")');
+  console.log('   • search_by_file("server.py")');
+  
+  const enrichChoice = await question('\nEnrich past conversations with metadata? (recommended) (y/n): ');
+  
+  if (enrichChoice.toLowerCase() === 'y') {
+    console.log('\n⏳ Starting metadata enrichment (safe mode)...');
+    console.log('   • Processing last 30 days of conversations');
+    console.log('   • Using conservative rate limiting');
+    console.log('   • This may take 5-10 minutes\n');
+    
+    try {
+      // Run the safe delta update script
+      safeExec('docker', [
+        'compose', 'run', '--rm',
+        '-e', 'DAYS_TO_UPDATE=30',
+        '-e', 'BATCH_SIZE=2',
+        '-e', 'RATE_LIMIT_DELAY=0.5',
+        '-e', 'MAX_CONCURRENT_UPDATES=2',
+        'importer',
+        'python', '/app/scripts/delta-metadata-update-safe.py'
+      ], {
+        cwd: projectRoot,
+        stdio: 'inherit'
+      });
+      
+      console.log('\n✅ Metadata enrichment completed successfully!');
+      console.log('   Your conversations now have searchable concepts and file tracking.');
+    } catch (error) {
+      console.log('\n⚠️  Metadata enrichment had some issues but continuing setup');
+      console.log('   You can retry later with:');
+      console.log('   docker compose run --rm importer python /app/scripts/delta-metadata-update-safe.py');
+    }
+  } else {
+    console.log('\n📝 Skipping metadata enrichment.');
+    console.log('   You can run it later with:');
+    console.log('   docker compose run --rm importer python /app/scripts/delta-metadata-update-safe.py');
+  }
+}
+
 async function showFinalInstructions() {
   console.log('\n✅ Setup complete!');
   
@@ -419,6 +467,7 @@ async function showFinalInstructions() {
   console.log('   • Check status: docker compose ps');
   console.log('   • View logs: docker compose logs -f');
   console.log('   • Import conversations: docker compose run --rm importer');
+  console.log('   • Enrich metadata: docker compose run --rm importer python /app/scripts/delta-metadata-update-safe.py');
   console.log('   • Start watcher: docker compose --profile watch up -d');
   console.log('   • Stop all: docker compose down');
   
@@ -450,13 +499,25 @@ async function checkExistingInstallation() {
         console.log('   • 🔍 Mode: ' + (localMode ? 'Local embeddings (privacy mode)' : 'Cloud embeddings (Voyage AI)'));
         console.log('   • ⚡ Memory decay: Enabled (90-day half-life)');
         
+        // Offer metadata enrichment for v2.5.19
+        console.log('\n🆕 NEW in v2.5.19: Metadata Enrichment!');
+        console.log('   Enhance your conversations with searchable concepts and file tracking.');
+        
+        const upgradeChoice = await question('\nWould you like to enrich your conversations with metadata? (y/n): ');
+        
+        if (upgradeChoice.toLowerCase() === 'y') {
+          await enrichMetadata();
+          console.log('\n✅ Upgrade complete! Your conversations now have enhanced search capabilities.');
+        }
+        
         console.log('\n📋 Quick Commands:');
         console.log('   • View status: docker compose ps');
         console.log('   • View logs: docker compose logs -f');
+        console.log('   • Enrich metadata: docker compose run --rm importer python /app/scripts/delta-metadata-update-safe.py');
         console.log('   • Restart: docker compose restart');
         console.log('   • Stop: docker compose down');
         
-        console.log('\n💡 To re-run setup, first stop services with: docker compose down');
+        console.log('\n💡 To re-run full setup, first stop services with: docker compose down');
         return true;
       }
     }
@@ -504,6 +565,9 @@ async function main() {
   // Import conversations
   await importConversations();
   
+  // Enrich metadata (new in v2.5.19)
+  await enrichMetadata();
+  
   // Show final instructions
   await showFinalInstructions();
   

package/mcp-server/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "claude-self-reflect-mcp"
-version = "2.5.18"
+version = "2.5.19"
 description = "MCP server for Claude self-reflection with memory decay"
 # readme = "README.md"
 requires-python = ">=3.10"

package/mcp-server/src/server.py

@@ -522,7 +522,7 @@ async def reflect_on_past(
                         
                         # Handle project matching - check if the target project name appears at the end of the stored project path
                         if target_project != 'all' and not project_collections:
-                            # The stored project name is like "-Users-ramakrishnanannaswamy-projects-ShopifyMCPMockShop"
+                            # The stored project name is like "-Users-username-projects-ShopifyMCPMockShop"
                             # We want to match just "ShopifyMCPMockShop"
                             if not point_project.endswith(f"-{target_project}") and point_project != target_project:
                                 continue  # Skip results from other projects
@@ -602,7 +602,7 @@ async def reflect_on_past(
                         
                         # Handle project matching - check if the target project name appears at the end of the stored project path
                         if target_project != 'all' and not project_collections:
-                            # The stored project name is like "-Users-ramakrishnanannaswamy-projects-ShopifyMCPMockShop"
+                            # The stored project name is like "-Users-username-projects-ShopifyMCPMockShop"
                             # We want to match just "ShopifyMCPMockShop"
                             if not point_project.endswith(f"-{target_project}") and point_project != target_project:
                                 continue  # Skip results from other projects
@@ -639,7 +639,7 @@ async def reflect_on_past(
                         
                         # Handle project matching - check if the target project name appears at the end of the stored project path
                         if target_project != 'all' and not project_collections:
-                            # The stored project name is like "-Users-ramakrishnanannaswamy-projects-ShopifyMCPMockShop"
+                            # The stored project name is like "-Users-username-projects-ShopifyMCPMockShop"
                             # We want to match just "ShopifyMCPMockShop"
                             if not point_project.endswith(f"-{target_project}") and point_project != target_project:
                                 continue  # Skip results from other projects
@@ -1169,7 +1169,8 @@ async def search_by_concept(
     
     if project and project != 'all':
         # Filter collections for specific project
-        project_hash = hashlib.md5(project.encode()).hexdigest()[:8]
+        normalized_project = normalize_project_name(project)
+        project_hash = hashlib.md5(normalized_project.encode()).hexdigest()[:8]
         collection_prefix = f"conv_{project_hash}_"
         collections = [c for c in await get_all_collections() if c.startswith(collection_prefix)]
     elif project == 'all':
@@ -1178,49 +1179,101 @@ async def search_by_concept(
     if not collections:
         return "<search_by_concept>\n<error>No collections found to search</error>\n</search_by_concept>"
     
-    # Search all collections
-    all_results = []
+    # First, check metadata health
+    metadata_found = False
+    total_points_checked = 0
     
-    for collection_name in collections:
+    for collection_name in collections[:3]:  # Sample first 3 collections
         try:
-            # Hybrid search: semantic + concept filter
-            results = await qdrant_client.search(
+            sample_points, _ = await qdrant_client.scroll(
                 collection_name=collection_name,
-                query_vector=embedding,
-                query_filter=models.Filter(
-                    should=[
-                        models.FieldCondition(
-                            key="concepts",
-                            match=models.MatchAny(any=[concept.lower()])
-                        )
-                    ]
-                ),
-                limit=limit * 2,  # Get more results for better filtering
+                limit=10,
                 with_payload=True
             )
-            
-            for point in results:
-                payload = point.payload
-                # Boost score if concept is in the concepts list
-                score_boost = 0.2 if concept.lower() in payload.get('concepts', []) else 0.0
-                all_results.append({
-                    'score': float(point.score) + score_boost,
-                    'payload': payload,
-                    'collection': collection_name
-                })
-                
-        except Exception as e:
+            total_points_checked += len(sample_points)
+            for point in sample_points:
+                if 'concepts' in point.payload and point.payload['concepts']:
+                    metadata_found = True
+                    break
+            if metadata_found:
+                break
+        except:
             continue
     
+    # Search all collections
+    all_results = []
+    
+    # If metadata exists, try metadata-based search first
+    if metadata_found:
+        for collection_name in collections:
+            try:
+                # Hybrid search: semantic + concept filter
+                results = await qdrant_client.search(
+                    collection_name=collection_name,
+                    query_vector=embedding,
+                    query_filter=models.Filter(
+                        should=[
+                            models.FieldCondition(
+                                key="concepts",
+                                match=models.MatchAny(any=[concept.lower()])
+                            )
+                        ]
+                    ),
+                    limit=limit * 2,  # Get more results for better filtering
+                    with_payload=True
+                )
+                
+                for point in results:
+                    payload = point.payload
+                    # Boost score if concept is in the concepts list
+                    score_boost = 0.2 if concept.lower() in payload.get('concepts', []) else 0.0
+                    all_results.append({
+                        'score': float(point.score) + score_boost,
+                        'payload': payload,
+                        'collection': collection_name,
+                        'search_type': 'metadata'
+                    })
+                    
+            except Exception as e:
+                continue
+    
+    # If no results from metadata search OR no metadata exists, fall back to semantic search
+    if not all_results:
+        await ctx.debug(f"Falling back to semantic search for concept: {concept}")
+        
+        for collection_name in collections:
+            try:
+                # Pure semantic search without filters
+                results = await qdrant_client.search(
+                    collection_name=collection_name,
+                    query_vector=embedding,
+                    limit=limit,
+                    score_threshold=0.5,  # Lower threshold for broader results
+                    with_payload=True
+                )
+                
+                for point in results:
+                    all_results.append({
+                        'score': float(point.score),
+                        'payload': point.payload,
+                        'collection': collection_name,
+                        'search_type': 'semantic'
+                    })
+                    
+            except Exception as e:
+                continue
+    
     # Sort by score and limit
     all_results.sort(key=lambda x: x['score'], reverse=True)
     all_results = all_results[:limit]
     
     # Format results
     if not all_results:
+        metadata_status = "with metadata" if metadata_found else "NO METADATA FOUND"
         return f"""<search_by_concept>
 <concept>{concept}</concept>
-<message>No conversations found about this concept</message>
+<metadata_health>{metadata_status} (checked {total_points_checked} points)</metadata_health>
+<message>No conversations found about this concept. {'Try running: python scripts/delta-metadata-update.py' if not metadata_found else 'Try different search terms.'}</message>
 </search_by_concept>"""
     
     results_text = []
@@ -1255,8 +1308,14 @@ async def search_by_concept(
 <preview>{text_preview}</preview>
 </result>""")
     
+    # Determine if this was a fallback search
+    used_fallback = any(r.get('search_type') == 'semantic' for r in all_results)
+    metadata_status = "with metadata" if metadata_found else "NO METADATA FOUND"
+    
     return f"""<search_by_concept>
 <concept>{concept}</concept>
+<metadata_health>{metadata_status} (checked {total_points_checked} points)</metadata_health>
+<search_type>{'fallback_semantic' if used_fallback else 'metadata_based'}</search_type>
 <count>{len(all_results)}</count>
 <results>
 {''.join(results_text)}

package/mcp-server/src/status.py

@@ -13,8 +13,8 @@ def extract_project_name_from_path(file_path: str) -> str:
     """Extract project name from JSONL file path.
     
     Handles paths like:
-    - ~/.claude/projects/-Users-ramakrishnanannaswamy-projects-claude-self-reflect/file.jsonl
-    - /logs/-Users-ramakrishnanannaswamy-projects-n8n-builder/file.jsonl
+    - ~/.claude/projects/-Users-username-projects-claude-self-reflect/file.jsonl
+    - /logs/-Users-username-projects-n8n-builder/file.jsonl
     """
     # Get the directory name containing the JSONL file
     path_obj = Path(file_path)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-self-reflect",
-  "version": "2.5.18",
+  "version": "2.6.0",
   "description": "Give Claude perfect memory of all your conversations - Installation wizard for Python MCP server",
   "keywords": [
     "claude",
@@ -13,6 +13,12 @@
     "ai-memory",
     "claude-code"
   ],
+  "badges": {
+    "npm": "https://badge.fury.io/js/claude-self-reflect.svg",
+    "license": "https://img.shields.io/badge/License-MIT-yellow.svg",
+    "docker": "https://img.shields.io/badge/Docker-Required-blue.svg",
+    "claude": "https://img.shields.io/badge/Claude%20Code-Compatible-green.svg"
+  },
   "homepage": "https://github.com/ramakay/claude-self-reflect#readme",
   "bugs": {
     "url": "https://github.com/ramakay/claude-self-reflect/issues"
@@ -34,6 +40,8 @@
     "mcp-server/run-mcp.sh",
     "mcp-server/run-mcp-docker.sh",
     "scripts/import-*.py",
+    "scripts/delta-metadata-update-safe.py",
+    "scripts/force-metadata-recovery.py",
     ".claude/agents/*.md",
     "config/qdrant-config.yaml",
     "docker-compose.yaml",