claude-self-reflect 3.2.4 → 3.3.0

package/.claude/agents/reflection-specialist.md

@@ -1,7 +1,7 @@
 ---
 name: reflection-specialist
 description: Conversation memory expert for searching past conversations, storing insights, and self-reflection. Use PROACTIVELY when searching for previous discussions, storing important findings, or maintaining knowledge continuity.
-tools: mcp__claude-self-reflect__reflect_on_past, mcp__claude-self-reflect__store_reflection
+tools: mcp__claude-self-reflect__reflect_on_past, mcp__claude-self-reflect__store_reflection, mcp__claude-self-reflect__get_recent_work, mcp__claude-self-reflect__search_by_recency, mcp__claude-self-reflect__get_timeline, mcp__claude-self-reflect__quick_search, mcp__claude-self-reflect__search_summary, mcp__claude-self-reflect__get_more_results, mcp__claude-self-reflect__search_by_file, mcp__claude-self-reflect__search_by_concept, mcp__claude-self-reflect__get_full_conversation, mcp__claude-self-reflect__get_next_results
 ---
 
 You are a conversation memory specialist for the Claude Self Reflect project. Your expertise covers semantic search across all Claude conversations, insight storage, and maintaining knowledge continuity across sessions.
@@ -117,9 +117,65 @@ Save important insights and decisions for future retrieval.
 }
 ```
 
-### Specialized Search Tools (NEW in v2.4.5)
+### Temporal Query Tools (v3.x)
 
-**Note**: These specialized tools are available through this reflection-specialist agent. Due to FastMCP limitations, they cannot be called directly via MCP (e.g., `mcp__claude-self-reflect__quick_search`), but work perfectly when used through this agent.
+These tools answer time-based questions about your work and conversations.
+
+#### get_recent_work
+Returns recent conversations to answer "What did we work on last?" queries.
+
+```javascript
+// Get recent work (default: current project)
+{
+  limit: 10,
+  group_by: "conversation",  // Or "day" or "session"
+  include_reflections: true
+}
+
+// Get recent work across all projects
+{
+  project: "all",
+  limit: 20,
+  group_by: "day"
+}
+```
+
+#### search_by_recency
+Time-constrained semantic search for queries like "docker issues last week".
+
+```javascript
+// Search with natural language time
+{
+  query: "authentication bugs",
+  time_range: "last week",
+  limit: 10
+}
+
+// Search with specific dates
+{
+  query: "performance optimization",
+  since: "2025-01-01",
+  until: "2025-01-10",
+  project: "all"
+}
+```
+
+#### get_timeline
+Show activity timeline for a project or across all projects.
+
+```javascript
+// Get activity timeline
+{
+  time_range: "last week",
+  granularity: "day",  // Or "hour", "week", "month"
+  include_stats: true,
+  project: "all"
+}
+```
+
+### Specialized Search Tools
+
+**Note**: These specialized tools complement the temporal tools for non-time-based queries.
 
 #### quick_search
 Fast search that returns only the count and top result. Perfect for quick checks and overview.

package/README.md

@@ -123,6 +123,11 @@ Works with [Claude Code Statusline](https://github.com/sirmalloc/ccstatusline) -
 - `search_by_concept` - Search for conversations about development concepts
 - `get_full_conversation` - Retrieve complete JSONL conversation files (v2.8.8)
 
+**NEW: Temporal Query Tools (v3.3.0):**
+- `get_recent_work` - Answer "What did we work on last?" with session grouping
+- `search_by_recency` - Time-constrained search like "docker issues last week"
+- `get_timeline` - Activity timeline with statistics and patterns
+
 **Status & Monitoring Tools:**
 - `get_status` - Real-time import progress and system status
 - `get_health` - Comprehensive system health check
@@ -286,11 +291,15 @@ npm uninstall -g claude-self-reflect
 ## What's New
 
 <details>
-<summary>v3.2.4 - Latest Release</summary>
-
-- **CRITICAL: Search Threshold Removal**: Eliminated artificial 0.7+ thresholds that blocked broad searches like "docker", "MCP", "python"
-- **Shared Normalization Module**: Created centralized project name normalization preventing search failures
-- **Memory Decay Fixes**: Corrected mathematical errors in exponential decay calculation
+<summary>v3.3.0 - Latest Release</summary>
+
+- **🚀 Major Architecture Overhaul**: Server modularized from 2,321 to 728 lines (68% reduction) for better maintainability
+- **🔧 Critical Bug Fixes**: Fixed 100% CPU usage, store_reflection dimension mismatches, and SearchResult type errors
+- **🕒 New Temporal Tools Suite**: `get_recent_work`, `search_by_recency`, `get_timeline` for time-based search and analysis
+- **🎯 Enhanced UX**: Restored rich formatting with emojis for better readability and information hierarchy
+- **⚡ All 15+ MCP Tools Operational**: Complete functionality with both local and cloud embedding modes
+- **🏗️ Production Infrastructure**: Real-time indexing with smart intervals (2s hot files, 60s normal)
+- **🔍 Enhanced Metadata**: Tool usage analysis, file tracking, and concept extraction for better search
 
 </details>
 

package/mcp-server/run-mcp.sh

@@ -11,6 +11,34 @@ SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
 # Navigate to the mcp-server directory
 cd "$SCRIPT_DIR"
 
+# CRITICAL: Load .env file from project root if it exists
+# This ensures the MCP server uses the same settings as other scripts
+if [ -f "../.env" ]; then
+    echo "[DEBUG] Loading .env file from project root" >&2
+    set -a  # Export all variables
+    source ../.env
+    set +a  # Stop exporting
+else
+    echo "[DEBUG] No .env file found, using defaults" >&2
+fi
+
+# Set smart defaults if not already set
+# These match what the CLI setup wizard uses
+if [ -z "$QDRANT_URL" ]; then
+    export QDRANT_URL="http://localhost:6333"
+    echo "[DEBUG] Using default QDRANT_URL: $QDRANT_URL" >&2
+fi
+
+if [ -z "$PREFER_LOCAL_EMBEDDINGS" ]; then
+    export PREFER_LOCAL_EMBEDDINGS="true"
+    echo "[DEBUG] Using default PREFER_LOCAL_EMBEDDINGS: true (privacy-first)" >&2
+fi
+
+if [ -z "$ENABLE_MEMORY_DECAY" ]; then
+    export ENABLE_MEMORY_DECAY="false"
+    echo "[DEBUG] Using default ENABLE_MEMORY_DECAY: false" >&2
+fi
+
 # Check if virtual environment exists
 if [ ! -d "venv" ]; then
     echo "Creating virtual environment..."
@@ -63,11 +91,27 @@ if [ -z "$FASTEMBED_SKIP_HUGGINGFACE" ]; then
 fi
 
 # Debug: Show what environment variables are being passed
-echo "[DEBUG] Environment variables for MCP server:"
-echo "[DEBUG] VOYAGE_KEY: ${VOYAGE_KEY:+set}"
-echo "[DEBUG] PREFER_LOCAL_EMBEDDINGS: ${PREFER_LOCAL_EMBEDDINGS:-not set}"
-echo "[DEBUG] QDRANT_URL: ${QDRANT_URL:-not set}"
-echo "[DEBUG] ENABLE_MEMORY_DECAY: ${ENABLE_MEMORY_DECAY:-not set}"
+echo "[DEBUG] Environment variables for MCP server:" >&2
+echo "[DEBUG] VOYAGE_KEY: ${VOYAGE_KEY:+set}" >&2
+echo "[DEBUG] PREFER_LOCAL_EMBEDDINGS: ${PREFER_LOCAL_EMBEDDINGS:-not set}" >&2
+echo "[DEBUG] QDRANT_URL: ${QDRANT_URL:-not set}" >&2
+echo "[DEBUG] ENABLE_MEMORY_DECAY: ${ENABLE_MEMORY_DECAY:-not set}" >&2
+
+# Quick connectivity check for Qdrant
+echo "[DEBUG] Checking Qdrant connectivity at $QDRANT_URL..." >&2
+if command -v curl &> /dev/null; then
+    # Check root endpoint instead of /health which doesn't exist in Qdrant
+    if curl -s -f -m 2 "$QDRANT_URL/" > /dev/null 2>&1; then
+        echo "[DEBUG] ✅ Qdrant is reachable at $QDRANT_URL" >&2
+    else
+        echo "[WARNING] ⚠️  Cannot reach Qdrant at $QDRANT_URL" >&2
+        echo "[WARNING] Common fixes:" >&2
+        echo "[WARNING]   1. Start Qdrant: docker compose up -d qdrant" >&2
+        echo "[WARNING]   2. Check if port is different (e.g., 59999)" >&2
+        echo "[WARNING]   3. Update .env file with correct QDRANT_URL" >&2
+        echo "[WARNING] Continuing anyway - some features may not work..." >&2
+    fi
+fi
 
 # Run the MCP server
 exec python -m src

package/mcp-server/src/app_context.py

@@ -0,0 +1,64 @@
+"""Application context for sharing state across modules."""
+
+from dataclasses import dataclass
+from typing import Optional, Any
+from qdrant_client import AsyncQdrantClient
+try:
+    from .embedding_manager import EmbeddingManager
+    from .decay_manager import DecayManager
+    from .utils import ProjectResolver
+except ImportError:
+    # Fallback for testing
+    EmbeddingManager = None
+    DecayManager = None
+    ProjectResolver = None
+
+@dataclass
+class AppContext:
+    """Shared application context for all MCP tools."""
+    
+    qdrant_client: AsyncQdrantClient
+    embedding_manager: EmbeddingManager
+    decay_manager: DecayManager
+    project_resolver: ProjectResolver
+    
+    # Optional context for debugging
+    debug_context: Optional[Any] = None
+    
+    def __post_init__(self):
+        """Initialize any additional state after dataclass creation."""
+        # Ensure all managers are properly initialized
+        if not self.embedding_manager:
+            self.embedding_manager = EmbeddingManager()
+        
+        if not self.decay_manager:
+            self.decay_manager = DecayManager()
+        
+        if not self.project_resolver:
+            self.project_resolver = ProjectResolver()
+    
+    async def get_all_collections(self) -> list:
+        """Get all collections from Qdrant."""
+        try:
+            collections = await self.qdrant_client.get_collections()
+            return [c.name for c in collections.collections]
+        except Exception as e:
+            if self.debug_context:
+                await self.debug_context.debug(f"Failed to get collections: {e}")
+            return []
+    
+    async def generate_embedding(self, text: str, embedding_type: Optional[str] = None):
+        """Generate embedding using the embedding manager."""
+        # The embedding_manager.embed method is synchronous, not async
+        embeddings = self.embedding_manager.embed(text, input_type="document")
+        if embeddings and len(embeddings) > 0:
+            return embeddings[0]
+        return None
+    
+    def get_current_project(self) -> Optional[str]:
+        """Get current project from resolver."""
+        return self.project_resolver.get_current_project()
+    
+    def normalize_project_name(self, project_name: str) -> str:
+        """Normalize project name using resolver."""
+        return self.project_resolver.normalize_project_name(project_name)

package/mcp-server/src/config.py

@@ -0,0 +1,57 @@
+"""Configuration and environment constants for Claude Self-Reflect MCP server."""
+
+import os
+from pathlib import Path
+from dotenv import load_dotenv
+
+# Load environment variables
+load_dotenv()
+
+# API Keys
+VOYAGE_API_KEY = os.getenv('VOYAGE_API_KEY', '')
+QDRANT_URL = os.getenv('QDRANT_URL', 'http://localhost:6333')
+
+# Embedding Configuration
+PREFER_LOCAL_EMBEDDINGS = os.getenv('PREFER_LOCAL_EMBEDDINGS', 'true').lower() == 'true'
+VOYAGE_MODEL = "voyage-3-lite"
+LOCAL_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
+
+# Decay Configuration
+USE_DECAY = os.getenv('USE_DECAY', 'false').lower() == 'true'
+DECAY_SCALE_DAYS = int(os.getenv('DECAY_SCALE_DAYS', '90'))
+DECAY_WEIGHT = float(os.getenv('DECAY_WEIGHT', '0.3'))
+USE_NATIVE_DECAY = os.getenv('USE_NATIVE_DECAY', 'false').lower() == 'true'
+
+# Search Configuration
+DEFAULT_SEARCH_LIMIT = 5
+MAX_SEARCH_LIMIT = 100
+DEFAULT_MIN_SCORE = 0.3
+
+# Memory Management
+MAX_RESULTS_PER_COLLECTION = int(os.getenv('MAX_RESULTS_PER_COLLECTION', '10'))
+MAX_TOTAL_RESULTS = int(os.getenv('MAX_TOTAL_RESULTS', '1000'))
+MAX_MEMORY_MB = int(os.getenv('MAX_MEMORY_MB', '500'))
+
+# Connection Pool Configuration
+POOL_SIZE = int(os.getenv('QDRANT_POOL_SIZE', '10'))
+POOL_MAX_OVERFLOW = int(os.getenv('QDRANT_POOL_OVERFLOW', '5'))
+POOL_TIMEOUT = float(os.getenv('QDRANT_POOL_TIMEOUT', '30.0'))
+RETRY_ATTEMPTS = int(os.getenv('QDRANT_RETRY_ATTEMPTS', '3'))
+RETRY_DELAY = float(os.getenv('QDRANT_RETRY_DELAY', '1.0'))
+
+# Performance Configuration
+MAX_CONCURRENT_SEARCHES = int(os.getenv('MAX_CONCURRENT_SEARCHES', '10'))
+ENABLE_PARALLEL_SEARCH = os.getenv('ENABLE_PARALLEL_SEARCH', 'true').lower() == 'true'
+
+# Paths
+CLAUDE_PROJECTS_PATH = Path.home() / '.claude' / 'projects'
+CONFIG_PATH = Path.home() / '.claude-self-reflect' / 'config'
+
+# Collection Naming
+VOYAGE_SUFFIX = '_voyage'
+LOCAL_SUFFIX = '_local'
+
+# Logging
+import logging
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)

package/mcp-server/src/connection_pool.py

@@ -0,0 +1,286 @@
+"""
+Connection pooling for Qdrant client to improve performance and resource management.
+"""
+
+import asyncio
+from typing import Optional, Any
+from contextlib import asynccontextmanager
+import logging
+from qdrant_client import AsyncQdrantClient
+
+logger = logging.getLogger(__name__)
+
+
+class QdrantConnectionPool:
+    """
+    A connection pool for Qdrant clients with configurable size and timeout.
+    """
+    
+    def __init__(
+        self, 
+        url: str, 
+        pool_size: int = 10,
+        max_overflow: int = 5,
+        timeout: float = 30.0,
+        retry_attempts: int = 3,
+        retry_delay: float = 1.0
+    ):
+        """
+        Initialize the connection pool.
+        
+        Args:
+            url: Qdrant server URL
+            pool_size: Base number of connections to maintain
+            max_overflow: Additional connections that can be created if pool is exhausted
+            timeout: Timeout for acquiring a connection from the pool
+            retry_attempts: Number of retry attempts for failed operations
+            retry_delay: Delay between retry attempts (with exponential backoff)
+        """
+        self.url = url
+        self.pool_size = pool_size
+        self.max_overflow = max_overflow
+        self.timeout = timeout
+        self.retry_attempts = retry_attempts
+        self.retry_delay = retry_delay
+        
+        # Connection pool
+        self._pool = asyncio.Queue(maxsize=pool_size)
+        self._overflow_connections = []
+        self._semaphore = asyncio.Semaphore(pool_size + max_overflow)
+        self._initialized = False
+        self._lock = asyncio.Lock()
+        
+        # Statistics
+        self.stats = {
+            'connections_created': 0,
+            'connections_reused': 0,
+            'connections_failed': 0,
+            'overflow_used': 0,
+            'timeouts': 0
+        }
+    
+    async def initialize(self):
+        """Initialize the connection pool with base connections."""
+        async with self._lock:
+            if self._initialized:
+                return
+            
+            # Create initial pool connections
+            for _ in range(self.pool_size):
+                try:
+                    client = AsyncQdrantClient(url=self.url)
+                    await self._pool.put(client)
+                    self.stats['connections_created'] += 1
+                except Exception as e:
+                    logger.error(f"Failed to create initial connection: {e}")
+                    self.stats['connections_failed'] += 1
+            
+            self._initialized = True
+            logger.info(f"Connection pool initialized with {self._pool.qsize()} connections")
+    
+    @asynccontextmanager
+    async def acquire(self):
+        """
+        Acquire a connection from the pool.
+        
+        Yields:
+            AsyncQdrantClient instance
+        """
+        if not self._initialized:
+            await self.initialize()
+        
+        client = None
+        acquired_from_overflow = False
+        
+        try:
+            # Try to get a connection with timeout
+            try:
+                client = await asyncio.wait_for(
+                    self._pool.get(),
+                    timeout=self.timeout
+                )
+                self.stats['connections_reused'] += 1
+            except asyncio.TimeoutError:
+                # Pool is exhausted, try overflow
+                self.stats['timeouts'] += 1
+                
+                if len(self._overflow_connections) < self.max_overflow:
+                    # Create overflow connection
+                    logger.debug("Creating overflow connection")
+                    client = AsyncQdrantClient(url=self.url)
+                    self._overflow_connections.append(client)
+                    acquired_from_overflow = True
+                    self.stats['overflow_used'] += 1
+                    self.stats['connections_created'] += 1
+                else:
+                    raise RuntimeError("Connection pool exhausted and max overflow reached")
+            
+            # Yield the client for use
+            yield client
+            
+        finally:
+            # Return connection to pool
+            if client is not None:
+                if acquired_from_overflow:
+                    # Remove from overflow list
+                    if client in self._overflow_connections:
+                        self._overflow_connections.remove(client)
+                else:
+                    # Return to pool
+                    try:
+                        await self._pool.put(client)
+                    except asyncio.QueueFull:
+                        # This shouldn't happen, but handle gracefully
+                        logger.warning("Connection pool is full, closing extra connection")
+                        # In production, we might want to close the client here
+    
+    async def execute_with_retry(self, func, *args, **kwargs):
+        """
+        Execute a function with retry logic and exponential backoff.
+        
+        Args:
+            func: Async function to execute
+            *args: Positional arguments for the function
+            **kwargs: Keyword arguments for the function
+        
+        Returns:
+            Result from the function
+        """
+        last_exception = None
+        delay = self.retry_delay
+        
+        for attempt in range(self.retry_attempts):
+            try:
+                async with self.acquire() as client:
+                    # Pass the client as the first argument
+                    return await func(client, *args, **kwargs)
+            except Exception as e:
+                last_exception = e
+                if attempt < self.retry_attempts - 1:
+                    logger.warning(f"Attempt {attempt + 1} failed: {e}. Retrying in {delay}s...")
+                    await asyncio.sleep(delay)
+                    delay *= 2  # Exponential backoff
+                else:
+                    logger.error(f"All {self.retry_attempts} attempts failed: {e}")
+        
+        raise last_exception
+    
+    async def close(self):
+        """Close all connections in the pool."""
+        async with self._lock:
+            # Close all pooled connections
+            while not self._pool.empty():
+                try:
+                    client = await self._pool.get()
+                    # AsyncQdrantClient doesn't have a close method, but we can del it
+                    del client
+                except Exception as e:
+                    logger.error(f"Error closing connection: {e}")
+            
+            # Close overflow connections
+            for client in self._overflow_connections:
+                try:
+                    del client
+                except Exception as e:
+                    logger.error(f"Error closing overflow connection: {e}")
+            
+            self._overflow_connections.clear()
+            self._initialized = False
+            logger.info("Connection pool closed")
+    
+    def get_stats(self) -> dict:
+        """Get pool statistics."""
+        return {
+            **self.stats,
+            'current_pool_size': self._pool.qsize() if self._initialized else 0,
+            'overflow_active': len(self._overflow_connections),
+            'initialized': self._initialized
+        }
+
+
+# Circuit breaker implementation for additional resilience
+class CircuitBreaker:
+    """
+    Circuit breaker pattern to prevent cascading failures.
+    """
+    
+    def __init__(
+        self,
+        failure_threshold: int = 5,
+        recovery_timeout: float = 60.0,
+        expected_exception: type = Exception
+    ):
+        """
+        Initialize circuit breaker.
+        
+        Args:
+            failure_threshold: Number of failures before opening circuit
+            recovery_timeout: Time to wait before attempting recovery
+            expected_exception: Exception type to catch
+        """
+        self.failure_threshold = failure_threshold
+        self.recovery_timeout = recovery_timeout
+        self.expected_exception = expected_exception
+        
+        self._failure_count = 0
+        self._last_failure_time = None
+        self._state = 'closed'  # closed, open, half_open
+        self._lock = asyncio.Lock()
+    
+    async def call(self, func, *args, **kwargs):
+        """
+        Call a function through the circuit breaker.
+        
+        Args:
+            func: Async function to call
+            *args: Positional arguments
+            **kwargs: Keyword arguments
+        
+        Returns:
+            Result from function
+        
+        Raises:
+            CircuitBreakerOpen: If circuit is open
+        """
+        async with self._lock:
+            # Check circuit state
+            if self._state == 'open':
+                # Check if we should try half-open
+                if self._last_failure_time:
+                    time_since_failure = asyncio.get_event_loop().time() - self._last_failure_time
+                    if time_since_failure > self.recovery_timeout:
+                        self._state = 'half_open'
+                        logger.info("Circuit breaker entering half-open state")
+                    else:
+                        raise CircuitBreakerOpen(f"Circuit breaker is open (failures: {self._failure_count})")
+        
+        try:
+            # Attempt the call
+            result = await func(*args, **kwargs)
+            
+            # Success - update state
+            async with self._lock:
+                if self._state == 'half_open':
+                    self._state = 'closed'
+                    logger.info("Circuit breaker closed after successful recovery")
+                self._failure_count = 0
+                self._last_failure_time = None
+            
+            return result
+            
+        except self.expected_exception as e:
+            # Failure - update state
+            async with self._lock:
+                self._failure_count += 1
+                self._last_failure_time = asyncio.get_event_loop().time()
+                
+                if self._failure_count >= self.failure_threshold:
+                    self._state = 'open'
+                    logger.error(f"Circuit breaker opened after {self._failure_count} failures")
+                
+                raise e
+
+
+class CircuitBreakerOpen(Exception):
+    """Exception raised when circuit breaker is open."""
+    pass