claude-self-reflect 2.5.18 → 2.5.19

package/.claude/agents/claude-self-reflect-test.md

@@ -13,6 +13,82 @@ You are a resilient and comprehensive testing specialist for Claude Self-Reflect
 - MCP tools enable reflection and memory storage
 - System must handle sensitive API keys securely
 
+## Comprehensive Test Suite
+
+### Available Test Categories
+The project now includes a comprehensive test suite in `/tests/` directory:
+
+1. **MCP Tool Integration** (`test_mcp_tools_comprehensive.py`)
+   - All MCP tools with various parameters
+   - Edge cases and error handling
+   - Cross-project search validation
+
+2. **Memory Decay** (`test_memory_decay.py`)
+   - Decay calculations and half-life variations
+   - Score adjustments and ranking changes
+   - Performance impact measurements
+
+3. **Multi-Project Support** (`test_multi_project.py`)
+   - Project isolation and collection naming
+   - Cross-project search functionality
+   - Metadata storage and retrieval
+
+4. **Embedding Models** (`test_embedding_models.py`)
+   - FastEmbed vs Voyage AI switching
+   - Dimension compatibility (384 vs 1024)
+   - Model performance comparisons
+
+5. **Delta Metadata** (`test_delta_metadata.py`)
+   - Tool usage extraction
+   - File reference tracking
+   - Incremental updates without re-embedding
+
+6. **Performance & Load** (`test_performance_load.py`)
+   - Large conversation imports (>1000 chunks)
+   - Concurrent operations
+   - Memory and CPU monitoring
+
+7. **Data Integrity** (`test_data_integrity.py`)
+   - Duplicate detection
+   - Unicode handling
+   - Chunk ordering preservation
+
+8. **Recovery Scenarios** (`test_recovery_scenarios.py`)
+   - Partial import recovery
+   - Container restart resilience
+   - State file corruption handling
+
+9. **Security** (`test_security.py`)
+   - API key validation
+   - Input sanitization
+   - Path traversal prevention
+
+### Running the Test Suite
+```bash
+# Run ALL tests
+cd ~/projects/claude-self-reflect
+python tests/run_all_tests.py
+
+# Run specific categories
+python tests/run_all_tests.py -c mcp_tools memory_decay multi_project
+
+# Run with verbose output
+python tests/run_all_tests.py -v
+
+# List available test categories
+python tests/run_all_tests.py --list
+
+# Run individual test files
+python tests/test_mcp_tools_comprehensive.py
+python tests/test_memory_decay.py
+python tests/test_multi_project.py
+```
+
+### Test Results Location
+- JSON results: `tests/test_results.json`
+- Contains timestamps, durations, pass/fail counts
+- Useful for tracking test history
+
 ## Key Responsibilities
 
 1. **System State Detection**

package/.claude/agents/mcp-integration.md

@@ -16,6 +16,51 @@ You are an MCP server development specialist for the memento-stack project. You
 - Supports both local (FastEmbed) and cloud (Voyage AI) embeddings
 - MCP determines project from working directory context
 
+## Available Test Suites
+
+### MCP-Specific Tests
+1. **Comprehensive MCP Tool Tests** (`tests/test_mcp_tools_comprehensive.py`)
+   - Tests all MCP tools: reflect_on_past, store_reflection, quick_search, search_summary
+   - Edge case handling and error scenarios
+   - Parameter validation (limit, min_score, use_decay, response_format)
+   - Cross-project search with project="all"
+   - Run with: `python tests/test_mcp_tools_comprehensive.py`
+
+2. **MCP Search Tests** (`scripts/test-mcp-search.py`)
+   - Basic MCP search functionality
+   - Integration with Qdrant backend
+   - Response parsing and formatting
+   - Run with: `python scripts/test-mcp-search.py`
+
+3. **MCP Robustness Tests** (`scripts/test-mcp-robustness.py`)
+   - Error recovery mechanisms
+   - Timeout handling
+   - Connection resilience
+   - Run with: `python scripts/test-mcp-robustness.py`
+
+### Running MCP Tests
+```bash
+# Run all MCP tests
+cd ~/projects/claude-self-reflect
+python tests/run_all_tests.py -c mcp_tools mcp_search
+
+# Test MCP server directly
+cd mcp-server && python test_server.py
+
+# Verify MCP registration in Claude Code
+claude mcp list | grep claude-self-reflect
+
+# Test MCP tools from Python
+python -c "from mcp_server.src.server import reflect_on_past; import asyncio; asyncio.run(reflect_on_past({'query': 'test', 'limit': 5}))"
+```
+
+### MCP Tool Parameters Reference
+- **reflect_on_past**: query, limit, brief, min_score, project, use_decay, response_format, include_raw
+- **store_reflection**: content, tags
+- **quick_search**: query, min_score, project
+- **search_by_file**: file_path, limit, project
+- **search_by_concept**: concept, include_files, limit, project
+
 ## Key Responsibilities
 
 1. **MCP Server Development**

package/.claude/agents/qdrant-specialist.md

@@ -49,6 +49,47 @@ You are a Qdrant vector database specialist for the memento-stack project. Your
 - **Monitor baseline + headroom** - Measure actual usage before setting limits
 - **Use cgroup-aware CPU monitoring** - Docker shows all CPUs but has limits
 
+## Available Test Suites
+
+### Qdrant-Specific Tests
+1. **Multi-Project Support Tests** (`tests/test_multi_project.py`)
+   - Collection isolation verification
+   - Cross-project search functionality
+   - Collection naming consistency
+   - Project metadata storage
+   - Run with: `python tests/test_multi_project.py`
+
+2. **Data Integrity Tests** (`tests/test_data_integrity.py`)
+   - Duplicate detection
+   - Chunk ordering preservation
+   - Unicode and special character handling
+   - Collection consistency checks
+
+3. **Performance Tests** (`tests/test_performance_load.py`)
+   - Large conversation imports (>1000 chunks)
+   - Concurrent search requests
+   - Memory usage patterns
+   - Collection size limits
+
+### How to Run Tests
+```bash
+# Run all Qdrant-related tests
+cd ~/projects/claude-self-reflect
+python tests/run_all_tests.py -c multi_project data_integrity performance
+
+# Check collection health
+docker exec claude-reflection-qdrant curl -s http://localhost:6333/collections | jq
+
+# Verify specific collection
+python -c "from qdrant_client import QdrantClient; c=QdrantClient('localhost', 6333); print(c.get_collection('conv_HASH_local'))"
+```
+
+### Common Issues & Solutions
+- **Dimension mismatch**: Check embedding model (384 for local, 1024 for voyage)
+- **Empty search results**: Verify collection exists and has points
+- **Slow searches**: Check collection size and optimize with filters
+- **Collection not found**: Verify project name normalization and MD5 hash
+
 ### Quality Gates
 - **Follow the workflow**: implementation → review → test → docs → release
 - **Use pre-releases for major changes** - Better to test than break production

package/README.md

@@ -176,8 +176,33 @@ Recent conversations matter more. Old ones fade. Like your brain, but reliable.
 - [GitHub Issues](https://github.com/ramakay/claude-self-reflect/issues)
 - [Discussions](https://github.com/ramakay/claude-self-reflect/discussions)
 
+## Upgrading to v2.5.19
+
+### 🆕 New Feature: Metadata Enrichment
+v2.5.19 adds searchable metadata to your conversations - concepts, files, and tools!
+
+#### For Existing Users
+```bash
+# Update to latest version
+npm update -g claude-self-reflect
+
+# Run setup - it will detect your existing installation
+claude-self-reflect setup
+# Choose "yes" when asked about metadata enrichment
+
+# Or manually enrich metadata anytime:
+docker compose run --rm importer python /app/scripts/delta-metadata-update-safe.py
+```
+
+#### What You Get
+- `search_by_concept("docker")` - Find conversations by topic
+- `search_by_file("server.py")` - Find conversations that touched specific files
+- Better search accuracy with metadata-based filtering
+
 ## What's New
 
+- **v2.5.19** - Metadata Enrichment! Search by concepts, files, and tools. [Full release notes](docs/releases/v2.5.19-RELEASE-NOTES.md)
+- **v2.5.18** - Security dependency updates
 - **v2.5.17** - Critical CPU fix and memory limit adjustment. [Full release notes](docs/releases/v2.5.17-release-notes.md)
 - **v2.5.16** - (Pre-release only) Initial streaming importer with CPU throttling
 - **v2.5.15** - Critical bug fixes and collection creation improvements

package/installer/setup-wizard-docker.js

@@ -406,6 +406,54 @@ async function importConversations() {
   }
 }
 
+async function enrichMetadata() {
+  console.log('\n🔍 Metadata Enrichment (NEW in v2.5.19!)...');
+  console.log('━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━');
+  console.log('This feature enhances your conversations with searchable metadata:');
+  console.log('   • Concepts: High-level topics (docker, security, testing, etc.)');
+  console.log('   • Files: Track which files were analyzed or edited');
+  console.log('   • Tools: Record which Claude tools were used');
+  console.log('\nEnables powerful searches like:');
+  console.log('   • search_by_concept("docker")');
+  console.log('   • search_by_file("server.py")');
+  
+  const enrichChoice = await question('\nEnrich past conversations with metadata? (recommended) (y/n): ');
+  
+  if (enrichChoice.toLowerCase() === 'y') {
+    console.log('\n⏳ Starting metadata enrichment (safe mode)...');
+    console.log('   • Processing last 30 days of conversations');
+    console.log('   • Using conservative rate limiting');
+    console.log('   • This may take 5-10 minutes\n');
+    
+    try {
+      // Run the safe delta update script
+      safeExec('docker', [
+        'compose', 'run', '--rm',
+        '-e', 'DAYS_TO_UPDATE=30',
+        '-e', 'BATCH_SIZE=2',
+        '-e', 'RATE_LIMIT_DELAY=0.5',
+        '-e', 'MAX_CONCURRENT_UPDATES=2',
+        'importer',
+        'python', '/app/scripts/delta-metadata-update-safe.py'
+      ], {
+        cwd: projectRoot,
+        stdio: 'inherit'
+      });
+      
+      console.log('\n✅ Metadata enrichment completed successfully!');
+      console.log('   Your conversations now have searchable concepts and file tracking.');
+    } catch (error) {
+      console.log('\n⚠️  Metadata enrichment had some issues but continuing setup');
+      console.log('   You can retry later with:');
+      console.log('   docker compose run --rm importer python /app/scripts/delta-metadata-update-safe.py');
+    }
+  } else {
+    console.log('\n📝 Skipping metadata enrichment.');
+    console.log('   You can run it later with:');
+    console.log('   docker compose run --rm importer python /app/scripts/delta-metadata-update-safe.py');
+  }
+}
+
 async function showFinalInstructions() {
   console.log('\n✅ Setup complete!');
   
@@ -419,6 +467,7 @@ async function showFinalInstructions() {
   console.log('   • Check status: docker compose ps');
   console.log('   • View logs: docker compose logs -f');
   console.log('   • Import conversations: docker compose run --rm importer');
+  console.log('   • Enrich metadata: docker compose run --rm importer python /app/scripts/delta-metadata-update-safe.py');
   console.log('   • Start watcher: docker compose --profile watch up -d');
   console.log('   • Stop all: docker compose down');
   
@@ -450,13 +499,25 @@ async function checkExistingInstallation() {
         console.log('   • 🔍 Mode: ' + (localMode ? 'Local embeddings (privacy mode)' : 'Cloud embeddings (Voyage AI)'));
         console.log('   • ⚡ Memory decay: Enabled (90-day half-life)');
         
+        // Offer metadata enrichment for v2.5.19
+        console.log('\n🆕 NEW in v2.5.19: Metadata Enrichment!');
+        console.log('   Enhance your conversations with searchable concepts and file tracking.');
+        
+        const upgradeChoice = await question('\nWould you like to enrich your conversations with metadata? (y/n): ');
+        
+        if (upgradeChoice.toLowerCase() === 'y') {
+          await enrichMetadata();
+          console.log('\n✅ Upgrade complete! Your conversations now have enhanced search capabilities.');
+        }
+        
         console.log('\n📋 Quick Commands:');
         console.log('   • View status: docker compose ps');
         console.log('   • View logs: docker compose logs -f');
+        console.log('   • Enrich metadata: docker compose run --rm importer python /app/scripts/delta-metadata-update-safe.py');
         console.log('   • Restart: docker compose restart');
         console.log('   • Stop: docker compose down');
         
-        console.log('\n💡 To re-run setup, first stop services with: docker compose down');
+        console.log('\n💡 To re-run full setup, first stop services with: docker compose down');
         return true;
       }
     }
@@ -504,6 +565,9 @@ async function main() {
   // Import conversations
   await importConversations();
   
+  // Enrich metadata (new in v2.5.19)
+  await enrichMetadata();
+  
   // Show final instructions
   await showFinalInstructions();
   

package/mcp-server/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "claude-self-reflect-mcp"
-version = "2.5.18"
+version = "2.5.19"
 description = "MCP server for Claude self-reflection with memory decay"
 # readme = "README.md"
 requires-python = ">=3.10"

package/mcp-server/src/server.py

@@ -522,7 +522,7 @@ async def reflect_on_past(
                         
                         # Handle project matching - check if the target project name appears at the end of the stored project path
                         if target_project != 'all' and not project_collections:
-                            # The stored project name is like "-Users-ramakrishnanannaswamy-projects-ShopifyMCPMockShop"
+                            # The stored project name is like "-Users-username-projects-ShopifyMCPMockShop"
                             # We want to match just "ShopifyMCPMockShop"
                             if not point_project.endswith(f"-{target_project}") and point_project != target_project:
                                 continue  # Skip results from other projects
@@ -602,7 +602,7 @@ async def reflect_on_past(
                         
                         # Handle project matching - check if the target project name appears at the end of the stored project path
                         if target_project != 'all' and not project_collections:
-                            # The stored project name is like "-Users-ramakrishnanannaswamy-projects-ShopifyMCPMockShop"
+                            # The stored project name is like "-Users-username-projects-ShopifyMCPMockShop"
                             # We want to match just "ShopifyMCPMockShop"
                             if not point_project.endswith(f"-{target_project}") and point_project != target_project:
                                 continue  # Skip results from other projects
@@ -639,7 +639,7 @@ async def reflect_on_past(
                         
                         # Handle project matching - check if the target project name appears at the end of the stored project path
                         if target_project != 'all' and not project_collections:
-                            # The stored project name is like "-Users-ramakrishnanannaswamy-projects-ShopifyMCPMockShop"
+                            # The stored project name is like "-Users-username-projects-ShopifyMCPMockShop"
                             # We want to match just "ShopifyMCPMockShop"
                             if not point_project.endswith(f"-{target_project}") and point_project != target_project:
                                 continue  # Skip results from other projects
@@ -1169,7 +1169,8 @@ async def search_by_concept(
     
     if project and project != 'all':
         # Filter collections for specific project
-        project_hash = hashlib.md5(project.encode()).hexdigest()[:8]
+        normalized_project = normalize_project_name(project)
+        project_hash = hashlib.md5(normalized_project.encode()).hexdigest()[:8]
         collection_prefix = f"conv_{project_hash}_"
         collections = [c for c in await get_all_collections() if c.startswith(collection_prefix)]
     elif project == 'all':
@@ -1178,49 +1179,101 @@ async def search_by_concept(
     if not collections:
         return "<search_by_concept>\n<error>No collections found to search</error>\n</search_by_concept>"
     
-    # Search all collections
-    all_results = []
+    # First, check metadata health
+    metadata_found = False
+    total_points_checked = 0
     
-    for collection_name in collections:
+    for collection_name in collections[:3]:  # Sample first 3 collections
         try:
-            # Hybrid search: semantic + concept filter
-            results = await qdrant_client.search(
+            sample_points, _ = await qdrant_client.scroll(
                 collection_name=collection_name,
-                query_vector=embedding,
-                query_filter=models.Filter(
-                    should=[
-                        models.FieldCondition(
-                            key="concepts",
-                            match=models.MatchAny(any=[concept.lower()])
-                        )
-                    ]
-                ),
-                limit=limit * 2,  # Get more results for better filtering
+                limit=10,
                 with_payload=True
             )
-            
-            for point in results:
-                payload = point.payload
-                # Boost score if concept is in the concepts list
-                score_boost = 0.2 if concept.lower() in payload.get('concepts', []) else 0.0
-                all_results.append({
-                    'score': float(point.score) + score_boost,
-                    'payload': payload,
-                    'collection': collection_name
-                })
-                
-        except Exception as e:
+            total_points_checked += len(sample_points)
+            for point in sample_points:
+                if 'concepts' in point.payload and point.payload['concepts']:
+                    metadata_found = True
+                    break
+            if metadata_found:
+                break
+        except:
             continue
     
+    # Search all collections
+    all_results = []
+    
+    # If metadata exists, try metadata-based search first
+    if metadata_found:
+        for collection_name in collections:
+            try:
+                # Hybrid search: semantic + concept filter
+                results = await qdrant_client.search(
+                    collection_name=collection_name,
+                    query_vector=embedding,
+                    query_filter=models.Filter(
+                        should=[
+                            models.FieldCondition(
+                                key="concepts",
+                                match=models.MatchAny(any=[concept.lower()])
+                            )
+                        ]
+                    ),
+                    limit=limit * 2,  # Get more results for better filtering
+                    with_payload=True
+                )
+                
+                for point in results:
+                    payload = point.payload
+                    # Boost score if concept is in the concepts list
+                    score_boost = 0.2 if concept.lower() in payload.get('concepts', []) else 0.0
+                    all_results.append({
+                        'score': float(point.score) + score_boost,
+                        'payload': payload,
+                        'collection': collection_name,
+                        'search_type': 'metadata'
+                    })
+                    
+            except Exception as e:
+                continue
+    
+    # If no results from metadata search OR no metadata exists, fall back to semantic search
+    if not all_results:
+        await ctx.debug(f"Falling back to semantic search for concept: {concept}")
+        
+        for collection_name in collections:
+            try:
+                # Pure semantic search without filters
+                results = await qdrant_client.search(
+                    collection_name=collection_name,
+                    query_vector=embedding,
+                    limit=limit,
+                    score_threshold=0.5,  # Lower threshold for broader results
+                    with_payload=True
+                )
+                
+                for point in results:
+                    all_results.append({
+                        'score': float(point.score),
+                        'payload': point.payload,
+                        'collection': collection_name,
+                        'search_type': 'semantic'
+                    })
+                    
+            except Exception as e:
+                continue
+    
     # Sort by score and limit
     all_results.sort(key=lambda x: x['score'], reverse=True)
     all_results = all_results[:limit]
     
     # Format results
     if not all_results:
+        metadata_status = "with metadata" if metadata_found else "NO METADATA FOUND"
         return f"""<search_by_concept>
 <concept>{concept}</concept>
-<message>No conversations found about this concept</message>
+<metadata_health>{metadata_status} (checked {total_points_checked} points)</metadata_health>
+<message>No conversations found about this concept. {'Try running: python scripts/delta-metadata-update.py' if not metadata_found else 'Try different search terms.'}</message>
 </search_by_concept>"""
     
     results_text = []
@@ -1255,8 +1308,14 @@ async def search_by_concept(
 <preview>{text_preview}</preview>
 </result>""")
     
+    # Determine if this was a fallback search
+    used_fallback = any(r.get('search_type') == 'semantic' for r in all_results)
+    metadata_status = "with metadata" if metadata_found else "NO METADATA FOUND"
+    
     return f"""<search_by_concept>
 <concept>{concept}</concept>
+<metadata_health>{metadata_status} (checked {total_points_checked} points)</metadata_health>
+<search_type>{'fallback_semantic' if used_fallback else 'metadata_based'}</search_type>
 <count>{len(all_results)}</count>
 <results>
 {''.join(results_text)}

package/mcp-server/src/status.py

@@ -13,8 +13,8 @@ def extract_project_name_from_path(file_path: str) -> str:
     """Extract project name from JSONL file path.
     
     Handles paths like:
-    - ~/.claude/projects/-Users-ramakrishnanannaswamy-projects-claude-self-reflect/file.jsonl
-    - /logs/-Users-ramakrishnanannaswamy-projects-n8n-builder/file.jsonl
+    - ~/.claude/projects/-Users-username-projects-claude-self-reflect/file.jsonl
+    - /logs/-Users-username-projects-n8n-builder/file.jsonl
     """
     # Get the directory name containing the JSONL file
     path_obj = Path(file_path)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-self-reflect",
-  "version": "2.5.18",
+  "version": "2.5.19",
   "description": "Give Claude perfect memory of all your conversations - Installation wizard for Python MCP server",
   "keywords": [
     "claude",
@@ -34,6 +34,8 @@
     "mcp-server/run-mcp.sh",
     "mcp-server/run-mcp-docker.sh",
     "scripts/import-*.py",
+    "scripts/delta-metadata-update-safe.py",
+    "scripts/force-metadata-recovery.py",
     ".claude/agents/*.md",
     "config/qdrant-config.yaml",
     "docker-compose.yaml",

package/scripts/delta-metadata-update-safe.py

@@ -0,0 +1,442 @@
+#!/usr/bin/env python3
+"""
+Safe delta metadata update script for Claude Self-Reflect.
+Updates existing Qdrant points with tool usage metadata without overwhelming the system.
+Includes rate limiting, batch processing, and proper error recovery.
+"""
+
+import os
+import sys
+import json
+import hashlib
+import re
+import time
+import asyncio
+from datetime import datetime, timedelta
+from typing import List, Dict, Any, Set, Tuple, Optional
+import logging
+from pathlib import Path
+from collections import defaultdict
+
+from qdrant_client import QdrantClient
+from qdrant_client.models import Filter, FieldCondition, MatchValue
+
+# Configuration
+QDRANT_URL = os.getenv("QDRANT_URL", "http://localhost:6333")
+LOGS_DIR = os.getenv("LOGS_DIR", os.path.expanduser("~/.claude/projects"))
+STATE_FILE = os.getenv("STATE_FILE", "./config/delta-update-state.json")
+PREFER_LOCAL_EMBEDDINGS = os.getenv("PREFER_LOCAL_EMBEDDINGS", "true").lower() == "true"
+DRY_RUN = os.getenv("DRY_RUN", "false").lower() == "true"
+DAYS_TO_UPDATE = int(os.getenv("DAYS_TO_UPDATE", "7"))
+BATCH_SIZE = int(os.getenv("BATCH_SIZE", "10"))  # Process N conversations at a time
+RATE_LIMIT_DELAY = float(os.getenv("RATE_LIMIT_DELAY", "0.1"))  # Delay between updates
+MAX_CONCURRENT_UPDATES = int(os.getenv("MAX_CONCURRENT_UPDATES", "5"))  # Max parallel updates
+
+# Set up logging
+logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
+logger = logging.getLogger(__name__)
+
+# Initialize Qdrant client
+client = QdrantClient(url=QDRANT_URL, timeout=30)  # Increased timeout
+
+def get_collection_suffix():
+    """Get the collection suffix based on embedding type (for new collections only)."""
+    return "_local" if PREFER_LOCAL_EMBEDDINGS else "_voyage"
+
+def get_existing_collection_suffix(project_hash: str, max_retries: int = 3) -> str:
+    """Detect which collection type actually exists for this project.
+    
+    This function checks for existing collections and returns the actual suffix found.
+    Only falls back to preference when creating new collections.
+    Includes retry logic for resilience against temporary Qdrant unavailability.
+    
+    Args:
+        project_hash: The MD5 hash of the normalized project name
+        max_retries: Maximum number of retry attempts for collection detection
+        
+    Returns:
+        "_voyage" if voyage collection exists, "_local" if local exists,
+        or preference-based suffix if neither exists yet
+    """
+    for attempt in range(max_retries):
+        try:
+            collections = client.get_collections().collections
+            collection_names = [c.name for c in collections]
+            
+            # Check for both possible collection names
+            voyage_name = f"conv_{project_hash}_voyage"
+            local_name = f"conv_{project_hash}_local"
+            
+            # Return the actual collection type that exists
+            if voyage_name in collection_names:
+                logger.debug(f"Found existing Voyage collection: {voyage_name}")
+                return "_voyage"
+            elif local_name in collection_names:
+                logger.debug(f"Found existing Local collection: {local_name}")
+                return "_local"
+            else:
+                # No existing collection - use preference for new ones
+                suffix = get_collection_suffix()
+                logger.debug(f"No existing collection for hash {project_hash}, using preference: {suffix}")
+                return suffix
+        except Exception as e:
+            if attempt < max_retries - 1:
+                wait_time = 0.5 * (attempt + 1)  # Exponential backoff
+                logger.debug(f"Error checking collections (attempt {attempt + 1}/{max_retries}): {e}, retrying in {wait_time}s")
+                time.sleep(wait_time)
+                continue
+            logger.warning(f"Error checking collections after {max_retries} attempts: {e}, falling back to preference")
+            return get_collection_suffix()
+
+def normalize_project_name(project_name: str) -> str:
+    """Normalize project name by removing path-like prefixes."""
+    if project_name.startswith("-"):
+        parts = project_name.split("-")
+        for i, part in enumerate(parts):
+            if part == "projects" and i < len(parts) - 1:
+                return "-".join(parts[i+1:])
+    return project_name
+
+def normalize_path(path: str) -> str:
+    """Normalize file paths for consistency across platforms."""
+    if not path:
+        return ""
+    path = path.replace("/Users/", "~/").replace("\\Users\\", "~\\")
+    path = path.replace("\\", "/")
+    path = re.sub(r'/+', '/', path)
+    return path
+
+def extract_concepts(text: str, tool_usage: Dict[str, Any]) -> Set[str]:
+    """Extract high-level concepts from conversation and tool usage."""
+    concepts = set()
+    
+    concept_patterns = {
+        'security': r'(security|vulnerability|CVE|injection|sanitize|escape|auth|token|JWT)',
+        'performance': r'(performance|optimization|speed|memory|efficient|benchmark|latency)',
+        'testing': r'(test|pytest|unittest|coverage|TDD|spec|assert)',
+        'docker': r'(docker|container|compose|dockerfile|kubernetes|k8s)',
+        'api': r'(API|REST|GraphQL|endpoint|webhook|http|request)',
+        'database': r'(database|SQL|query|migration|schema|postgres|mysql|mongodb|qdrant)',
+        'authentication': r'(auth|login|token|JWT|session|oauth|permission)',
+        'debugging': r'(debug|error|exception|traceback|log|stack|trace)',
+        'refactoring': r'(refactor|cleanup|improve|restructure|optimize|technical debt)',
+        'deployment': r'(deploy|CI/CD|release|production|staging|rollout)',
+        'git': r'(git|commit|branch|merge|pull request|PR|rebase)',
+        'architecture': r'(architecture|design|pattern|structure|component|module)',
+        'mcp': r'(MCP|claude-self-reflect|tool|agent|claude code)',
+        'embeddings': r'(embedding|vector|semantic|similarity|fastembed|voyage)',
+        'search': r'(search|query|find|filter|match|relevance)'
+    }
+    
+    combined_text = text.lower()
+    for concept, pattern in concept_patterns.items():
+        if re.search(pattern, combined_text, re.IGNORECASE):
+            concepts.add(concept)
+    
+    # Check tool usage patterns
+    if tool_usage.get('grep_searches'):
+        concepts.add('search')
+    if tool_usage.get('files_edited') or tool_usage.get('files_created'):
+        concepts.add('development')
+    if any('test' in str(f).lower() for f in tool_usage.get('files_read', [])):
+        concepts.add('testing')
+    if any('docker' in str(cmd).lower() for cmd in tool_usage.get('bash_commands', [])):
+        concepts.add('docker')
+    
+    return concepts
+
+def extract_tool_usage_from_jsonl(jsonl_path: str) -> Dict[str, Any]:
+    """Extract all tool usage from a conversation."""
+    tool_usage = {
+        "files_read": [],
+        "files_edited": [],
+        "files_created": [],
+        "grep_searches": [],
+        "bash_commands": [],
+        "tools_summary": {}
+    }
+    
+    try:
+        with open(jsonl_path, 'r', encoding='utf-8') as f:
+            for line in f:
+                if not line.strip():
+                    continue
+                try:
+                    data = json.loads(line)
+                    if 'message' in data and data['message']:
+                        msg = data['message']
+                        if msg.get('role') == 'assistant' and msg.get('content'):
+                            content = msg['content']
+                            if isinstance(content, list):
+                                for item in content:
+                                    if isinstance(item, dict) and item.get('type') == 'tool_use':
+                                        tool_name = item.get('name', '')
+                                        inputs = item.get('input', {})
+                                        
+                                        # Track tool usage
+                                        tool_usage['tools_summary'][tool_name] = tool_usage['tools_summary'].get(tool_name, 0) + 1
+                                        
+                                        # Extract file paths
+                                        if tool_name == 'Read':
+                                            file_path = inputs.get('file_path')
+                                            if file_path:
+                                                tool_usage['files_read'].append(normalize_path(file_path))
+                                        elif tool_name in ['Edit', 'Write', 'MultiEdit']:
+                                            file_path = inputs.get('file_path')
+                                            if file_path:
+                                                tool_usage['files_edited'].append(normalize_path(file_path))
+                                        elif tool_name == 'Grep':
+                                            pattern = inputs.get('pattern')
+                                            if pattern:
+                                                tool_usage['grep_searches'].append({'pattern': pattern[:100]})
+                                        elif tool_name == 'Bash':
+                                            command = inputs.get('command', '')[:200]
+                                            if command:
+                                                tool_usage['bash_commands'].append(command)
+                except Exception as e:
+                    continue
+    except Exception as e:
+        logger.error(f"Error reading JSONL file {jsonl_path}: {e}")
+    
+    # Deduplicate
+    tool_usage['files_read'] = list(set(tool_usage['files_read']))[:20]
+    tool_usage['files_edited'] = list(set(tool_usage['files_edited']))[:10]
+    
+    return tool_usage
+
+def load_state() -> Dict[str, Any]:
+    """Load the current state from file."""
+    state_path = Path(STATE_FILE)
+    if state_path.exists():
+        try:
+            with open(state_path, 'r') as f:
+                return json.load(f)
+        except Exception as e:
+            logger.warning(f"Could not load state: {e}")
+    return {
+        "last_update": None,
+        "updated_conversations": {},
+        "failed_conversations": {}
+    }
+
+def save_state(state: Dict[str, Any]):
+    """Save the current state to file."""
+    state_path = Path(STATE_FILE)
+    state_path.parent.mkdir(parents=True, exist_ok=True)
+    
+    state["last_update"] = datetime.now().isoformat()
+    
+    try:
+        with open(state_path, 'w') as f:
+            json.dump(state, f, indent=2)
+    except Exception as e:
+        logger.error(f"Could not save state: {e}")
+
+async def update_point_metadata_batch(updates: List[Tuple[str, int, Dict, str]]) -> int:
+    """Update multiple points in a batch with rate limiting."""
+    success_count = 0
+    
+    for conversation_id, chunk_index, metadata, collection_name in updates:
+        try:
+            # Calculate point ID
+            point_id_str = hashlib.md5(
+                f"{conversation_id}_{chunk_index}".encode()
+            ).hexdigest()[:16]
+            point_id = int(point_id_str, 16) % (2**63)
+            
+            if not DRY_RUN:
+                # Update with rate limiting
+                client.set_payload(
+                    collection_name=collection_name,
+                    payload=metadata,
+                    points=[point_id],
+                    wait=False
+                )
+                success_count += 1
+                
+                # Rate limit to avoid overwhelming Qdrant
+                await asyncio.sleep(RATE_LIMIT_DELAY)
+            else:
+                logger.info(f"[DRY RUN] Would update point {point_id}")
+                success_count += 1
+                
+        except Exception as e:
+            logger.debug(f"Failed to update point {conversation_id}_{chunk_index}: {e}")
+    
+    return success_count
+
+async def process_conversation_async(jsonl_file: Path, state: Dict[str, Any]) -> bool:
+    """Process a single conversation file asynchronously."""
+    try:
+        conversation_id = jsonl_file.stem
+        project_name = jsonl_file.parent.name
+        
+        # Check if already updated
+        if conversation_id in state.get("updated_conversations", {}):
+            last_updated = state["updated_conversations"][conversation_id].get("updated_at")
+            file_mtime = jsonl_file.stat().st_mtime
+            if last_updated and last_updated >= file_mtime:
+                logger.debug(f"Skipping {conversation_id} - already updated")
+                return True
+        
+        # Check if previously failed too many times
+        failed_info = state.get("failed_conversations", {}).get(conversation_id, {})
+        if failed_info.get("retry_count", 0) > 3:
+            logger.debug(f"Skipping {conversation_id} - too many failures")
+            return False
+        
+        logger.info(f"Processing: {conversation_id}")
+        
+        # Extract metadata
+        tool_usage = extract_tool_usage_from_jsonl(str(jsonl_file))
+        
+        # Read conversation text (limited)
+        conversation_text = ""
+        with open(jsonl_file, 'r', encoding='utf-8') as f:
+            for i, line in enumerate(f):
+                if i > 100:  # Limit lines to avoid memory issues
+                    break
+                if line.strip():
+                    try:
+                        data = json.loads(line)
+                        if 'message' in data and data['message']:
+                            msg = data['message']
+                            if msg.get('content'):
+                                if isinstance(msg['content'], str):
+                                    conversation_text += msg['content'][:500] + "\n"
+                    except Exception as e:
+                        logger.debug(f"Parse error in {jsonl_file}: {e}")
+                        continue
+        
+        # Extract concepts
+        concepts = extract_concepts(conversation_text[:10000], tool_usage)
+        
+        # Prepare metadata
+        metadata_update = {
+            "files_analyzed": tool_usage.get('files_read', [])[:20],
+            "files_edited": tool_usage.get('files_edited', [])[:10],
+            "tools_used": list(tool_usage.get('tools_summary', {}).keys())[:20],
+            "concepts": list(concepts)[:15],
+            "has_file_metadata": True,
+            "metadata_updated_at": datetime.now().isoformat()
+        }
+        
+        # Determine collection
+        project_hash = hashlib.md5(normalize_project_name(project_name).encode()).hexdigest()[:8]
+        # Use smart detection to find the actual collection type
+        collection_suffix = get_existing_collection_suffix(project_hash)
+        collection_name = f"conv_{project_hash}{collection_suffix}"
+        
+        # Check if collection exists
+        try:
+            collections = client.get_collections().collections
+            if collection_name not in [c.name for c in collections]:
+                logger.warning(f"Collection {collection_name} not found")
+                return False
+        except Exception as e:
+            logger.error(f"Error checking collection: {e}")
+            # Record failure
+            state.setdefault("failed_conversations", {})[conversation_id] = {
+                "error": str(e),
+                "retry_count": failed_info.get("retry_count", 0) + 1,
+                "last_attempt": time.time()
+            }
+            return False
+        
+        # Prepare batch updates
+        updates = []
+        for chunk_index in range(20):  # Most conversations have < 20 chunks
+            updates.append((conversation_id, chunk_index, metadata_update, collection_name))
+        
+        # Process in batch with rate limiting
+        success_count = await update_point_metadata_batch(updates)
+        
+        if success_count > 0:
+            logger.info(f"Updated {success_count} chunks for {conversation_id}")
+            state["updated_conversations"][conversation_id] = {
+                "updated_at": time.time(),
+                "chunks_updated": success_count,
+                "project": project_name
+            }
+            return True
+        else:
+            logger.warning(f"No chunks updated for {conversation_id}")
+            return False
+            
+    except Exception as e:
+        logger.error(f"Failed to process {jsonl_file}: {e}")
+        return False
+
+async def main_async():
+    """Main async function with proper batching and rate limiting."""
+    logger.info("=== Starting Safe Delta Metadata Update ===")
+    logger.info(f"Configuration:")
+    logger.info(f"  Qdrant URL: {QDRANT_URL}")
+    logger.info(f"  Days to update: {DAYS_TO_UPDATE}")
+    logger.info(f"  Batch size: {BATCH_SIZE}")
+    logger.info(f"  Rate limit delay: {RATE_LIMIT_DELAY}s")
+    logger.info(f"  Max concurrent: {MAX_CONCURRENT_UPDATES}")
+    
+    # Load state
+    state = load_state()
+    
+    # Get recent files
+    recent_files = []
+    cutoff_time = datetime.now() - timedelta(days=DAYS_TO_UPDATE)
+    logs_path = Path(LOGS_DIR)
+    
+    if logs_path.exists():
+        for jsonl_file in logs_path.glob("**/*.jsonl"):
+            try:
+                mtime = datetime.fromtimestamp(jsonl_file.stat().st_mtime)
+                if mtime >= cutoff_time:
+                    recent_files.append(jsonl_file)
+            except:
+                continue
+    
+    logger.info(f"Found {len(recent_files)} conversations from the past {DAYS_TO_UPDATE} days")
+    
+    # Process in batches
+    success_count = 0
+    failed_count = 0
+    
+    for i in range(0, len(recent_files), BATCH_SIZE):
+        batch = recent_files[i:i + BATCH_SIZE]
+        logger.info(f"Processing batch {i//BATCH_SIZE + 1}/{(len(recent_files) + BATCH_SIZE - 1)//BATCH_SIZE}")
+        
+        # Create tasks for concurrent processing
+        tasks = []
+        for jsonl_file in batch:
+            task = asyncio.create_task(process_conversation_async(jsonl_file, state))
+            tasks.append(task)
+        
+        # Wait for batch to complete
+        results = await asyncio.gather(*tasks)
+        
+        # Count results
+        batch_success = sum(1 for r in results if r)
+        batch_failed = len(results) - batch_success
+        success_count += batch_success
+        failed_count += batch_failed
+        
+        # Save state after each batch
+        save_state(state)
+        
+        # Add delay between batches to avoid overwhelming the system
+        if i + BATCH_SIZE < len(recent_files):
+            await asyncio.sleep(1.0)
+    
+    # Final save
+    save_state(state)
+    
+    logger.info("=== Delta Update Complete ===")
+    logger.info(f"Successfully updated: {success_count} conversations")
+    logger.info(f"Failed: {failed_count} conversations")
+    logger.info(f"Total conversations in state: {len(state['updated_conversations'])}")
+
+def main():
+    """Entry point."""
+    asyncio.run(main_async())
+
+if __name__ == "__main__":
+    main()

package/scripts/force-metadata-recovery.py

@@ -0,0 +1,305 @@
+#!/usr/bin/env python3
+"""
+Force metadata recovery script for Claude Self-Reflect.
+Fixes conversations that were marked as updated but don't actually have metadata.
+This addresses the point ID mismatch bug in delta-metadata-update.py.
+"""
+
+import os
+import sys
+import json
+import hashlib
+import re
+import asyncio
+from datetime import datetime
+from typing import List, Dict, Any, Set, Optional
+import logging
+from pathlib import Path
+
+from qdrant_client import QdrantClient
+
+# Configuration
+QDRANT_URL = os.getenv("QDRANT_URL", "http://localhost:6333")
+LOGS_DIR = os.getenv("LOGS_DIR", os.path.expanduser("~/.claude/projects"))
+DRY_RUN = os.getenv("DRY_RUN", "false").lower() == "true"
+BATCH_SIZE = int(os.getenv("BATCH_SIZE", "100"))
+
+# Set up logging
+logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
+logger = logging.getLogger(__name__)
+
+# Initialize Qdrant client
+client = QdrantClient(url=QDRANT_URL, timeout=30)
+
+def normalize_path(path: str) -> str:
+    """Normalize file paths for consistency."""
+    if not path:
+        return ""
+    path = path.replace("/Users/", "~/").replace("\\Users\\", "~\\")
+    path = path.replace("\\", "/")
+    path = re.sub(r'/+', '/', path)
+    return path
+
+def extract_concepts(text: str) -> Set[str]:
+    """Extract high-level concepts from text."""
+    concepts = set()
+    
+    concept_patterns = {
+        'security': r'(security|vulnerability|CVE|injection|auth)',
+        'docker': r'(docker|container|compose|kubernetes)',
+        'testing': r'(test|pytest|unittest|coverage)',
+        'api': r'(API|REST|GraphQL|endpoint)',
+        'database': r'(database|SQL|query|migration|qdrant)',
+        'debugging': r'(debug|error|exception|traceback)',
+        'git': r'(git|commit|branch|merge|pull request)',
+        'mcp': r'(MCP|claude-self-reflect|tool|agent)',
+        'embeddings': r'(embedding|vector|semantic|similarity)',
+    }
+    
+    text_lower = text.lower()
+    for concept, pattern in concept_patterns.items():
+        if re.search(pattern, text_lower, re.IGNORECASE):
+            concepts.add(concept)
+    
+    return concepts
+
+def extract_metadata_from_jsonl(jsonl_path: str) -> Dict[str, Any]:
+    """Extract metadata from a JSONL conversation file."""
+    metadata = {
+        "files_analyzed": [],
+        "files_edited": [],
+        "tools_used": set(),
+        "concepts": set(),
+        "text_sample": ""
+    }
+    
+    try:
+        with open(jsonl_path, 'r', encoding='utf-8') as f:
+            line_count = 0
+            for line in f:
+                line_count += 1
+                if line_count > 200:  # Limit processing
+                    break
+                    
+                if not line.strip():
+                    continue
+                    
+                try:
+                    data = json.loads(line)
+                    if 'message' in data and data['message']:
+                        msg = data['message']
+                        
+                        # Extract text for concept analysis
+                        if msg.get('content'):
+                            if isinstance(msg['content'], str):
+                                metadata['text_sample'] += msg['content'][:500] + "\n"
+                            
+                        # Extract tool usage
+                        if msg.get('role') == 'assistant' and msg.get('content'):
+                            content = msg['content']
+                            if isinstance(content, list):
+                                for item in content:
+                                    if isinstance(item, dict) and item.get('type') == 'tool_use':
+                                        tool_name = item.get('name', '')
+                                        metadata['tools_used'].add(tool_name)
+                                        
+                                        inputs = item.get('input', {})
+                                        
+                                        if tool_name == 'Read' and 'file_path' in inputs:
+                                            metadata['files_analyzed'].append(
+                                                normalize_path(inputs['file_path'])
+                                            )
+                                        elif tool_name in ['Edit', 'Write'] and 'file_path' in inputs:
+                                            metadata['files_edited'].append(
+                                                normalize_path(inputs['file_path'])
+                                            )
+                                            
+                except json.JSONDecodeError:
+                    continue
+                    
+    except Exception as e:
+        logger.error(f"Error reading {jsonl_path}: {e}")
+    
+    # Extract concepts from collected text
+    if metadata['text_sample']:
+        metadata['concepts'] = extract_concepts(metadata['text_sample'][:5000])
+    
+    # Convert sets to lists and limit
+    metadata['tools_used'] = list(metadata['tools_used'])[:20]
+    metadata['concepts'] = list(metadata['concepts'])[:15]
+    metadata['files_analyzed'] = list(set(metadata['files_analyzed']))[:20]
+    metadata['files_edited'] = list(set(metadata['files_edited']))[:10]
+    
+    del metadata['text_sample']  # Don't store in Qdrant
+    
+    return metadata
+
+async def find_conversations_without_metadata(collection_name: str) -> List[str]:
+    """Find all unique conversation IDs that don't have metadata."""
+    conversations_without_metadata = set()
+    
+    offset = None
+    total_checked = 0
+    
+    while True:
+        points, next_offset = client.scroll(
+            collection_name=collection_name,
+            limit=BATCH_SIZE,
+            offset=offset,
+            with_payload=True,
+            with_vectors=False
+        )
+        
+        if not points:
+            break
+            
+        for point in points:
+            # Check if metadata is missing
+            if not point.payload.get('concepts') or not point.payload.get('has_file_metadata'):
+                conv_id = point.payload.get('conversation_id')
+                if conv_id:
+                    conversations_without_metadata.add(conv_id)
+        
+        total_checked += len(points)
+        offset = next_offset
+        
+        if offset is None:
+            break
+    
+    logger.info(f"  Checked {total_checked} points, found {len(conversations_without_metadata)} conversations without metadata")
+    return list(conversations_without_metadata)
+
+async def update_conversation_points(collection_name: str, conversation_id: str, metadata: Dict[str, Any]) -> int:
+    """Update all points for a conversation with metadata."""
+    updated_count = 0
+    
+    # Get all points in the collection
+    offset = None
+    while True:
+        points, next_offset = client.scroll(
+            collection_name=collection_name,
+            limit=BATCH_SIZE,
+            offset=offset,
+            with_payload=True,
+            with_vectors=False
+        )
+        
+        if not points:
+            break
+        
+        # Find and update points for this conversation
+        for point in points:
+            if point.payload.get('conversation_id') == conversation_id:
+                if not DRY_RUN:
+                    # Merge metadata with existing payload
+                    updated_payload = {**point.payload, **metadata}
+                    updated_payload['has_file_metadata'] = True
+                    updated_payload['metadata_updated_at'] = datetime.now().isoformat()
+                    
+                    client.set_payload(
+                        collection_name=collection_name,
+                        payload=updated_payload,
+                        points=[point.id],
+                        wait=False
+                    )
+                
+                updated_count += 1
+        
+        offset = next_offset
+        if offset is None:
+            break
+    
+    return updated_count
+
+async def process_collection(collection_name: str):
+    """Process a single collection to add missing metadata."""
+    logger.info(f"\nProcessing collection: {collection_name}")
+    
+    # Find conversations without metadata
+    conversations_without_metadata = await find_conversations_without_metadata(collection_name)
+    
+    if not conversations_without_metadata:
+        logger.info(f"  ✓ All conversations have metadata")
+        return 0
+    
+    logger.info(f"  Found {len(conversations_without_metadata)} conversations needing metadata")
+    
+    # Process each conversation
+    success_count = 0
+    failed_count = 0
+    
+    for conv_id in conversations_without_metadata[:10]:  # Limit for testing
+        # Find the JSONL file
+        jsonl_pattern = f"**/{conv_id}.jsonl"
+        jsonl_files = list(Path(LOGS_DIR).glob(jsonl_pattern))
+        
+        if not jsonl_files:
+            logger.warning(f"  Cannot find JSONL for {conv_id}")
+            failed_count += 1
+            continue
+        
+        jsonl_file = jsonl_files[0]
+        logger.info(f"  Processing {conv_id}")
+        
+        # Extract metadata
+        metadata = extract_metadata_from_jsonl(str(jsonl_file))
+        
+        if not metadata['concepts'] and not metadata['files_analyzed']:
+            logger.warning(f"    No metadata extracted from {conv_id}")
+            failed_count += 1
+            continue
+        
+        # Update points
+        updated_points = await update_conversation_points(collection_name, conv_id, metadata)
+        
+        if updated_points > 0:
+            logger.info(f"    ✓ Updated {updated_points} points with {len(metadata['concepts'])} concepts")
+            success_count += 1
+        else:
+            logger.warning(f"    No points updated for {conv_id}")
+            failed_count += 1
+    
+    logger.info(f"  Collection complete: {success_count} fixed, {failed_count} failed")
+    return success_count
+
+async def main():
+    """Main recovery process."""
+    logger.info("=== Force Metadata Recovery ===")
+    logger.info(f"Qdrant URL: {QDRANT_URL}")
+    logger.info(f"Dry run: {DRY_RUN}")
+    
+    # Get all collections
+    collections = client.get_collections().collections
+    
+    # Focus on collections with potential issues
+    priority_collections = []
+    other_collections = []
+    
+    for collection in collections:
+        name = collection.name
+        if name.startswith('conv_'):
+            other_collections.append(name)
+    
+    logger.info(f"Found {len(priority_collections)} priority collections")
+    logger.info(f"Found {len(other_collections)} other collections")
+    
+    # Process priority collections first
+    total_fixed = 0
+    
+    for collection_name in priority_collections:
+        fixed = await process_collection(collection_name)
+        total_fixed += fixed
+    
+    # Process a sample of other collections
+    for collection_name in other_collections[:5]:
+        fixed = await process_collection(collection_name)
+        total_fixed += fixed
+    
+    logger.info(f"\n=== Recovery Complete ===")
+    logger.info(f"Total conversations fixed: {total_fixed}")
+    
+    if DRY_RUN:
+        logger.info("This was a DRY RUN - no actual updates were made")
+
+if __name__ == "__main__":
+    asyncio.run(main())