claude-self-reflect 2.5.17 → 2.5.19

package/.claude/agents/claude-self-reflect-test.md

@@ -13,6 +13,82 @@ You are a resilient and comprehensive testing specialist for Claude Self-Reflect
 - MCP tools enable reflection and memory storage
 - System must handle sensitive API keys securely
 
+## Comprehensive Test Suite
+
+### Available Test Categories
+The project now includes a comprehensive test suite in `/tests/` directory:
+
+1. **MCP Tool Integration** (`test_mcp_tools_comprehensive.py`)
+   - All MCP tools with various parameters
+   - Edge cases and error handling
+   - Cross-project search validation
+
+2. **Memory Decay** (`test_memory_decay.py`)
+   - Decay calculations and half-life variations
+   - Score adjustments and ranking changes
+   - Performance impact measurements
+
+3. **Multi-Project Support** (`test_multi_project.py`)
+   - Project isolation and collection naming
+   - Cross-project search functionality
+   - Metadata storage and retrieval
+
+4. **Embedding Models** (`test_embedding_models.py`)
+   - FastEmbed vs Voyage AI switching
+   - Dimension compatibility (384 vs 1024)
+   - Model performance comparisons
+
+5. **Delta Metadata** (`test_delta_metadata.py`)
+   - Tool usage extraction
+   - File reference tracking
+   - Incremental updates without re-embedding
+
+6. **Performance & Load** (`test_performance_load.py`)
+   - Large conversation imports (>1000 chunks)
+   - Concurrent operations
+   - Memory and CPU monitoring
+
+7. **Data Integrity** (`test_data_integrity.py`)
+   - Duplicate detection
+   - Unicode handling
+   - Chunk ordering preservation
+
+8. **Recovery Scenarios** (`test_recovery_scenarios.py`)
+   - Partial import recovery
+   - Container restart resilience
+   - State file corruption handling
+
+9. **Security** (`test_security.py`)
+   - API key validation
+   - Input sanitization
+   - Path traversal prevention
+
+### Running the Test Suite
+```bash
+# Run ALL tests
+cd ~/projects/claude-self-reflect
+python tests/run_all_tests.py
+
+# Run specific categories
+python tests/run_all_tests.py -c mcp_tools memory_decay multi_project
+
+# Run with verbose output
+python tests/run_all_tests.py -v
+
+# List available test categories
+python tests/run_all_tests.py --list
+
+# Run individual test files
+python tests/test_mcp_tools_comprehensive.py
+python tests/test_memory_decay.py
+python tests/test_multi_project.py
+```
+
+### Test Results Location
+- JSON results: `tests/test_results.json`
+- Contains timestamps, durations, pass/fail counts
+- Useful for tracking test history
+
 ## Key Responsibilities
 
 1. **System State Detection**

package/.claude/agents/docker-orchestrator.md

@@ -13,6 +13,26 @@ You are a Docker orchestration specialist for the memento-stack project. You man
 - Services run on host network for local development
 - Production uses Railway deployment
 
+## CRITICAL GUARDRAILS (from v2.5.17 crisis)
+
+### Resource Limit Guidelines
+⚠️ **Memory limits must include baseline usage**
+- Measure baseline: `docker stats --no-stream`
+- Add 200MB+ headroom above baseline
+- Default: 600MB minimum (not 400MB)
+
+⚠️ **CPU monitoring in containers**
+- Containers see all host CPUs but have cgroup limits
+- 1437% CPU = ~90% of actual allocation
+- Use cgroup-aware monitoring: `/sys/fs/cgroup/cpu/cpu.cfs_quota_us`
+
+### Pre-Deployment Checklist
+✅ Test with production data volumes (600+ files)
+✅ Verify STATE_FILE paths match between config and container
+✅ Check volume mounts are writable
+✅ Confirm memory/CPU limits are realistic
+✅ Test graceful shutdown handling
+
 ## Key Responsibilities
 
 1. **Service Management**

package/.claude/agents/import-debugger.md

@@ -16,6 +16,21 @@ You are an import pipeline debugging expert for the memento-stack project. You s
 - Project name must be correctly extracted from path for proper collection naming
 - Collections named using MD5 hash of project name
 
+## CRITICAL GUARDRAILS (from v2.5.17 crisis)
+
+### Pre-Release Testing Checklist
+✅ **Test with actual Claude JSONL files** - Real ~/.claude/projects/*.jsonl files
+✅ **Verify processing metrics** - files_processed, chunks_created must be > 0  
+✅ **Memory limits = baseline + headroom** - Measure actual usage first (typically 400MB base)
+✅ **Run tests to completion** - Don't mark as done without execution proof
+✅ **Handle production backlogs** - Test with 600+ file queues
+
+### Common Failure Patterns
+🚨 **State updates without progress** - high_water_mark changes but processed_files = 0
+🚨 **Memory limit blocking** - "Memory limit exceeded" on every file = limit too low
+🚨 **CPU misreporting** - 1437% CPU might be 90% of container limit
+🚨 **Wrong file format** - Testing with .json when production uses .jsonl
+
 ## Key Responsibilities
 
 1. **JSONL Processing**

package/.claude/agents/mcp-integration.md

@@ -16,6 +16,51 @@ You are an MCP server development specialist for the memento-stack project. You
 - Supports both local (FastEmbed) and cloud (Voyage AI) embeddings
 - MCP determines project from working directory context
 
+## Available Test Suites
+
+### MCP-Specific Tests
+1. **Comprehensive MCP Tool Tests** (`tests/test_mcp_tools_comprehensive.py`)
+   - Tests all MCP tools: reflect_on_past, store_reflection, quick_search, search_summary
+   - Edge case handling and error scenarios
+   - Parameter validation (limit, min_score, use_decay, response_format)
+   - Cross-project search with project="all"
+   - Run with: `python tests/test_mcp_tools_comprehensive.py`
+
+2. **MCP Search Tests** (`scripts/test-mcp-search.py`)
+   - Basic MCP search functionality
+   - Integration with Qdrant backend
+   - Response parsing and formatting
+   - Run with: `python scripts/test-mcp-search.py`
+
+3. **MCP Robustness Tests** (`scripts/test-mcp-robustness.py`)
+   - Error recovery mechanisms
+   - Timeout handling
+   - Connection resilience
+   - Run with: `python scripts/test-mcp-robustness.py`
+
+### Running MCP Tests
+```bash
+# Run all MCP tests
+cd ~/projects/claude-self-reflect
+python tests/run_all_tests.py -c mcp_tools mcp_search
+
+# Test MCP server directly
+cd mcp-server && python test_server.py
+
+# Verify MCP registration in Claude Code
+claude mcp list | grep claude-self-reflect
+
+# Test MCP tools from Python
+python -c "from mcp_server.src.server import reflect_on_past; import asyncio; asyncio.run(reflect_on_past({'query': 'test', 'limit': 5}))"
+```
+
+### MCP Tool Parameters Reference
+- **reflect_on_past**: query, limit, brief, min_score, project, use_decay, response_format, include_raw
+- **store_reflection**: content, tags
+- **quick_search**: query, min_score, project
+- **search_by_file**: file_path, limit, project
+- **search_by_concept**: concept, include_files, limit, project
+
 ## Key Responsibilities
 
 1. **MCP Server Development**

package/.claude/agents/qdrant-specialist.md

@@ -36,6 +36,66 @@ You are a Qdrant vector database specialist for the memento-stack project. Your
    - Analyze embedding quality
    - Compare different embedding models (Voyage vs OpenAI)
 
+## CRITICAL GUARDRAILS (from v2.5.17 crisis)
+
+### Testing Requirements
+- **ALWAYS test with real JSONL files** - Claude uses .jsonl, not .json
+- **Verify actual processing** - Check files_processed > 0, not just state updates
+- **Test memory limits with baseline** - System uses 400MB baseline, set limits accordingly
+- **Never mark tests complete without execution** - Run and verify output
+
+### Resource Management
+- **Default memory to 600MB minimum** - 400MB is too conservative
+- **Monitor baseline + headroom** - Measure actual usage before setting limits
+- **Use cgroup-aware CPU monitoring** - Docker shows all CPUs but has limits
+
+## Available Test Suites
+
+### Qdrant-Specific Tests
+1. **Multi-Project Support Tests** (`tests/test_multi_project.py`)
+   - Collection isolation verification
+   - Cross-project search functionality
+   - Collection naming consistency
+   - Project metadata storage
+   - Run with: `python tests/test_multi_project.py`
+
+2. **Data Integrity Tests** (`tests/test_data_integrity.py`)
+   - Duplicate detection
+   - Chunk ordering preservation
+   - Unicode and special character handling
+   - Collection consistency checks
+
+3. **Performance Tests** (`tests/test_performance_load.py`)
+   - Large conversation imports (>1000 chunks)
+   - Concurrent search requests
+   - Memory usage patterns
+   - Collection size limits
+
+### How to Run Tests
+```bash
+# Run all Qdrant-related tests
+cd ~/projects/claude-self-reflect
+python tests/run_all_tests.py -c multi_project data_integrity performance
+
+# Check collection health
+docker exec claude-reflection-qdrant curl -s http://localhost:6333/collections | jq
+
+# Verify specific collection
+python -c "from qdrant_client import QdrantClient; c=QdrantClient('localhost', 6333); print(c.get_collection('conv_HASH_local'))"
+```
+
+### Common Issues & Solutions
+- **Dimension mismatch**: Check embedding model (384 for local, 1024 for voyage)
+- **Empty search results**: Verify collection exists and has points
+- **Slow searches**: Check collection size and optimize with filters
+- **Collection not found**: Verify project name normalization and MD5 hash
+
+### Quality Gates
+- **Follow the workflow**: implementation → review → test → docs → release
+- **Use pre-releases for major changes** - Better to test than break production
+- **Document quantitative metrics** - "0 files processed" is clearer than "test failed"
+- **Rollback immediately on failure** - Don't push through broken releases
+
 ## Essential Commands
 
 ### Collection Operations

package/Dockerfile.streaming-importer

@@ -14,11 +14,12 @@ RUN pip install --upgrade pip setuptools wheel
 # Install PyTorch CPU version for smaller size
 RUN pip install --no-cache-dir torch==2.3.0 --index-url https://download.pytorch.org/whl/cpu
 
-# Install other dependencies
+# Install other dependencies with updated versions for security
+# Note: fastembed 0.4.0 requires numpy<2, so using compatible version
 RUN pip install --no-cache-dir \
     qdrant-client==1.15.0 \
-    fastembed==0.2.7 \
-    numpy==1.26.0 \
+    fastembed==0.4.0 \
+    numpy==1.26.4 \
     psutil==7.0.0 \
     tenacity==8.2.3 \
     python-dotenv==1.0.0 \

package/README.md

@@ -6,10 +6,7 @@ Claude forgets everything. This fixes that.
 
 - [What You Get](#what-you-get)
 - [Requirements](#requirements)
-- [Quick Install](#quick-install)
-  - [Local Mode (Default)](#local-mode-default---your-data-stays-private)
-  - [Cloud Mode](#cloud-mode-better-search-accuracy)
-- [Uninstall Instructions](#uninstall-instructions)
+- [Quick Install/Uninstall](#quick-installuninstall)
 - [The Magic](#the-magic)
 - [Before & After](#before--after)
 - [Real Examples](#real-examples-that-made-us-build-this)
@@ -18,10 +15,9 @@ Claude forgets everything. This fixes that.
 - [Using It](#using-it)
 - [Key Features](#key-features)
 - [Performance](#performance)
-- [V2.5.16 Critical Updates](#v2516-critical-updates)
 - [Configuration](#configuration)
 - [Technical Stack](#the-technical-stack)
-- [Problems?](#problems)
+- [Problems](#problems)
 - [What's New](#whats-new)
 - [Advanced Topics](#advanced-topics)
 - [Contributors](#contributors)
@@ -30,12 +26,12 @@ Claude forgets everything. This fixes that.
 
 Ask Claude about past conversations. Get actual answers. **100% local by default** - your conversations never leave your machine. Cloud-enhanced search available when you need it.
 
-**✅ Proven at Scale**: Successfully indexed 682 conversation files with 100% reliability. No data loss, no corruption, just seamless conversation memory that works.
+**Proven at Scale**: Successfully indexed 682 conversation files with 100% reliability. No data loss, no corruption, just seamless conversation memory that works.
 
 **Before**: "I don't have access to previous conversations"  
 **After**: 
 ```
-⏺ reflection-specialist(Search FastEmbed vs cloud embedding decision)
+reflection-specialist(Search FastEmbed vs cloud embedding decision)
   ⎿ Done (3 tool uses · 8.2k tokens · 12.4s)
 
 "Found it! Yesterday we decided on FastEmbed for local mode - better privacy, 
@@ -52,9 +48,11 @@ Your conversations become searchable. Your decisions stay remembered. Your conte
 - **Node.js** 16+ (for the setup wizard)
 - **Claude Desktop** app
 
-## Quick Install
+## Quick Install/Uninstall
 
-### Local Mode (Default - Your Data Stays Private)
+### Install
+
+#### Local Mode (Default - Your Data Stays Private)
 ```bash
 # Install and run automatic setup
 npm install -g claude-self-reflect
@@ -69,7 +67,7 @@ claude-self-reflect setup
 # 🔒 Keep all data local - no API keys needed
 ```
 
-### Cloud Mode (Better Search Accuracy)
+#### Cloud Mode (Better Search Accuracy)
 ```bash
 # Step 1: Get your free Voyage AI key
 # Sign up at https://www.voyageai.com/ - it takes 30 seconds
@@ -122,7 +120,7 @@ Here's how your conversations get imported and prioritized:
 ![Import Architecture](docs/diagrams/import-architecture.png)
 
 **The system intelligently prioritizes your conversations:**
-- **🔥 HOT** (< 5 minutes): Switches to 2-second intervals for near real-time import
+- **HOT** (< 5 minutes): Switches to 2-second intervals for near real-time import
 - **🌡️ WARM** (< 24 hours): Normal priority, processed every 60 seconds
 - **❄️ COLD** (> 24 hours): Batch processed, max 5 per cycle to prevent blocking
 
@@ -138,7 +136,7 @@ The reflection specialist automatically activates. No special commands needed.
 
 ## Key Features
 
-### 🎯 Project-Scoped Search
+### Project-Scoped Search
 Searches are **project-aware by default**. Claude automatically searches within your current project:
 
 ```
@@ -162,40 +160,6 @@ Recent conversations matter more. Old ones fade. Like your brain, but reliable.
 - **Scale**: 100% indexing success rate across all conversation types
 - **V2 Migration**: 100% complete - all conversations use token-aware chunking
 
-## V2.5.16 Critical Updates
-
-### 🚨 CPU Performance Fix - RESOLVED
-**Issue**: Streaming importer was consuming **1437% CPU** causing system overload  
-**Solution**: Complete rewrite with production-grade throttling and monitoring  
-**Result**: CPU usage reduced to **<1%** (99.93% improvement)
-
-### ✅ Production-Ready Streaming Importer
-- **Non-blocking CPU monitoring** with cgroup awareness
-- **Queue overflow protection** - data deferred, never dropped
-- **Atomic state persistence** with fsync for crash recovery
-- **Memory management** with 15% GC buffer and automatic cleanup
-- **Proper async signal handling** for clean shutdowns
-
-### 🎯 100% V2 Token-Aware Chunking
-- **Complete Migration**: All collections now use optimized chunking
-- **Configuration**: 400 tokens/1600 chars with 75 token/300 char overlap
-- **Search Quality**: Improved semantic boundaries and context preservation
-- **Memory Efficiency**: Streaming processing prevents OOM during imports
-
-### 📊 Performance Metrics (v2.5.16)
-| Metric | Before | After | Improvement |
-|--------|--------|-------|-------------|
-| CPU Usage | 1437% | <1% | 99.93% ↓ |
-| Memory | 8GB peak | 302MB | 96.2% ↓ |
-| Search Latency | Variable | 3.16ms avg | Consistent |
-| Test Success | Unstable | 21/25 passing | Reliable |
-
-### 🔧 CLI Status Command Fix
-Fixed broken `--status` command in MCP server - now returns:
-- Collection counts and health
-- Real-time CPU and memory usage
-- Search performance metrics
-- Import processing status
 
 ## The Technical Stack
 
@@ -206,15 +170,41 @@ Fixed broken `--status` command in MCP server - now returns:
 - **MCP Server**: Python + FastMCP
 - **Search**: Semantic similarity with time decay
 
-## Problems?
+## Problems
 
 - [Troubleshooting Guide](docs/troubleshooting.md)
 - [GitHub Issues](https://github.com/ramakay/claude-self-reflect/issues)
 - [Discussions](https://github.com/ramakay/claude-self-reflect/discussions)
 
+## Upgrading to v2.5.19
+
+### 🆕 New Feature: Metadata Enrichment
+v2.5.19 adds searchable metadata to your conversations - concepts, files, and tools!
+
+#### For Existing Users
+```bash
+# Update to latest version
+npm update -g claude-self-reflect
+
+# Run setup - it will detect your existing installation
+claude-self-reflect setup
+# Choose "yes" when asked about metadata enrichment
+
+# Or manually enrich metadata anytime:
+docker compose run --rm importer python /app/scripts/delta-metadata-update-safe.py
+```
+
+#### What You Get
+- `search_by_concept("docker")` - Find conversations by topic
+- `search_by_file("server.py")` - Find conversations that touched specific files
+- Better search accuracy with metadata-based filtering
+
 ## What's New
 
-- **v2.5.16** - **CRITICAL PERFORMANCE UPDATE** - Fixed 1437% CPU overload, 100% V2 migration complete, production streaming importer
+- **v2.5.19** - Metadata Enrichment! Search by concepts, files, and tools. [Full release notes](docs/releases/v2.5.19-RELEASE-NOTES.md)
+- **v2.5.18** - Security dependency updates
+- **v2.5.17** - Critical CPU fix and memory limit adjustment. [Full release notes](docs/releases/v2.5.17-release-notes.md)
+- **v2.5.16** - (Pre-release only) Initial streaming importer with CPU throttling
 - **v2.5.15** - Critical bug fixes and collection creation improvements
 - **v2.5.14** - Async importer collection fix - All conversations now searchable
 - **v2.5.11** - Critical cloud mode fix - Environment variables now properly passed to MCP server
@@ -231,6 +221,22 @@ Fixed broken `--status` command in MCP server - now returns:
 - [Architecture details](docs/architecture-details.md)
 - [Contributing](CONTRIBUTING.md)
 
+### Uninstall
+
+For complete uninstall instructions, see [docs/UNINSTALL.md](docs/UNINSTALL.md).
+
+Quick uninstall:
+```bash
+# Remove MCP server
+claude mcp remove claude-self-reflect
+
+# Stop Docker containers
+docker-compose down
+
+# Uninstall npm package
+npm uninstall -g claude-self-reflect
+```
+
 ## Contributors
 
 Special thanks to our contributors:
@@ -240,6 +246,4 @@ Special thanks to our contributors:
 
 ---
 
-Stop reading. Start installing. Your future self will thank you.
-
-MIT License. Built with ❤️ for the Claude community.
+Built with ❤️  by [ramakay](https://github.com/ramakay) for the Claude community.

package/installer/setup-wizard-docker.js

@@ -406,6 +406,54 @@ async function importConversations() {
   }
 }
 
+async function enrichMetadata() {
+  console.log('\n🔍 Metadata Enrichment (NEW in v2.5.19!)...');
+  console.log('━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━');
+  console.log('This feature enhances your conversations with searchable metadata:');
+  console.log('   • Concepts: High-level topics (docker, security, testing, etc.)');
+  console.log('   • Files: Track which files were analyzed or edited');
+  console.log('   • Tools: Record which Claude tools were used');
+  console.log('\nEnables powerful searches like:');
+  console.log('   • search_by_concept("docker")');
+  console.log('   • search_by_file("server.py")');
+  
+  const enrichChoice = await question('\nEnrich past conversations with metadata? (recommended) (y/n): ');
+  
+  if (enrichChoice.toLowerCase() === 'y') {
+    console.log('\n⏳ Starting metadata enrichment (safe mode)...');
+    console.log('   • Processing last 30 days of conversations');
+    console.log('   • Using conservative rate limiting');
+    console.log('   • This may take 5-10 minutes\n');
+    
+    try {
+      // Run the safe delta update script
+      safeExec('docker', [
+        'compose', 'run', '--rm',
+        '-e', 'DAYS_TO_UPDATE=30',
+        '-e', 'BATCH_SIZE=2',
+        '-e', 'RATE_LIMIT_DELAY=0.5',
+        '-e', 'MAX_CONCURRENT_UPDATES=2',
+        'importer',
+        'python', '/app/scripts/delta-metadata-update-safe.py'
+      ], {
+        cwd: projectRoot,
+        stdio: 'inherit'
+      });
+      
+      console.log('\n✅ Metadata enrichment completed successfully!');
+      console.log('   Your conversations now have searchable concepts and file tracking.');
+    } catch (error) {
+      console.log('\n⚠️  Metadata enrichment had some issues but continuing setup');
+      console.log('   You can retry later with:');
+      console.log('   docker compose run --rm importer python /app/scripts/delta-metadata-update-safe.py');
+    }
+  } else {
+    console.log('\n📝 Skipping metadata enrichment.');
+    console.log('   You can run it later with:');
+    console.log('   docker compose run --rm importer python /app/scripts/delta-metadata-update-safe.py');
+  }
+}
+
 async function showFinalInstructions() {
   console.log('\n✅ Setup complete!');
   
@@ -419,6 +467,7 @@ async function showFinalInstructions() {
   console.log('   • Check status: docker compose ps');
   console.log('   • View logs: docker compose logs -f');
   console.log('   • Import conversations: docker compose run --rm importer');
+  console.log('   • Enrich metadata: docker compose run --rm importer python /app/scripts/delta-metadata-update-safe.py');
   console.log('   • Start watcher: docker compose --profile watch up -d');
   console.log('   • Stop all: docker compose down');
   
@@ -450,13 +499,25 @@ async function checkExistingInstallation() {
         console.log('   • 🔍 Mode: ' + (localMode ? 'Local embeddings (privacy mode)' : 'Cloud embeddings (Voyage AI)'));
         console.log('   • ⚡ Memory decay: Enabled (90-day half-life)');
         
+        // Offer metadata enrichment for v2.5.19
+        console.log('\n🆕 NEW in v2.5.19: Metadata Enrichment!');
+        console.log('   Enhance your conversations with searchable concepts and file tracking.');
+        
+        const upgradeChoice = await question('\nWould you like to enrich your conversations with metadata? (y/n): ');
+        
+        if (upgradeChoice.toLowerCase() === 'y') {
+          await enrichMetadata();
+          console.log('\n✅ Upgrade complete! Your conversations now have enhanced search capabilities.');
+        }
+        
         console.log('\n📋 Quick Commands:');
         console.log('   • View status: docker compose ps');
         console.log('   • View logs: docker compose logs -f');
+        console.log('   • Enrich metadata: docker compose run --rm importer python /app/scripts/delta-metadata-update-safe.py');
         console.log('   • Restart: docker compose restart');
         console.log('   • Stop: docker compose down');
         
-        console.log('\n💡 To re-run setup, first stop services with: docker compose down');
+        console.log('\n💡 To re-run full setup, first stop services with: docker compose down');
         return true;
       }
     }
@@ -504,6 +565,9 @@ async function main() {
   // Import conversations
   await importConversations();
   
+  // Enrich metadata (new in v2.5.19)
+  await enrichMetadata();
+  
   // Show final instructions
   await showFinalInstructions();
   

package/mcp-server/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "claude-self-reflect-mcp"
-version = "2.5.17"
+version = "2.5.19"
 description = "MCP server for Claude self-reflection with memory decay"
 # readme = "README.md"
 requires-python = ">=3.10"