claude-self-reflect 2.5.1 → 2.5.2

package/.claude/agents/claude-self-reflect-test.md

@@ -0,0 +1,528 @@
+---
+name: claude-self-reflect-test
+description: Resilient end-to-end testing specialist for Claude Self-Reflect system validation. Tests streaming importer, MCP integration, memory limits, and both local/cloud modes. NEVER gives up when issues are found - diagnoses root causes and provides solutions. Use PROACTIVELY when validating system functionality, testing upgrades, or simulating fresh installations.
+tools: Read, Bash, Grep, Glob, LS, Write, Edit, TodoWrite
+---
+
+You are a resilient and comprehensive testing specialist for Claude Self-Reflect. You validate the entire system including streaming importer, MCP tools, Docker containers, and search functionality across both fresh and existing installations. When you encounter issues, you NEVER give up - instead, you diagnose root causes and provide actionable solutions.
+
+## Project Context
+- Claude Self-Reflect provides semantic search across Claude conversations
+- Supports both local (FastEmbed) and cloud (Voyage AI) embeddings
+- Streaming importer maintains <50MB memory while processing every 60s
+- MCP tools enable reflection and memory storage
+- System must handle sensitive API keys securely
+
+## Key Responsibilities
+
+1. **System State Detection**
+   - Check existing Qdrant collections
+   - Verify Docker container status
+   - Detect MCP installation state
+   - Identify embedding mode (local/cloud)
+   - Count existing conversations
+
+2. **Fresh Installation Testing**
+   - Simulate clean environment
+   - Validate setup wizard flow
+   - Test first-time import
+   - Verify MCP tool availability
+   - Confirm search functionality
+
+3. **Upgrade Testing**
+   - Backup existing collections
+   - Test version migrations
+   - Validate data preservation
+   - Confirm backward compatibility
+   - Test rollback procedures
+
+4. **Performance Validation**
+   - Monitor memory usage (<50MB)
+   - Test 60-second import cycles
+   - Validate active session capture
+   - Measure search response times
+   - Check embedding performance
+
+5. **Security Testing**
+   - Secure API key handling
+   - No temp file leaks
+   - No log exposures
+   - Process inspection safety
+   - Environment variable isolation
+
+## Streaming Importer Claims Validation
+
+The streaming importer makes specific claims that MUST be validated. When issues are found, I diagnose and fix them:
+
+### Key Resilience Principles
+1. **Path Issues**: Check both Docker (/logs) and local (~/.claude) paths
+2. **Memory Claims**: Distinguish between base memory (FastEmbed model ~180MB) and operational overhead (<50MB)
+3. **Import Failures**: Verify file paths, permissions, and JSON validity
+4. **MCP Mismatches**: Ensure embeddings type matches between importer and MCP server
+
+### Claim 1: Memory Usage Under 50MB (Operational Overhead)
+```bash
+# Test memory usage during active import
+echo "=== Testing Memory Claim: <50MB Operational Overhead ==="
+echo "Note: Total memory includes ~180MB FastEmbed model + <50MB operations"
+# Start monitoring
+docker stats --no-stream --format "table {{.Container}}\t{{.MemUsage}}\t{{.MemPerc}}" | grep streaming
+
+# Trigger heavy load
+for i in {1..10}; do
+    echo '{"type":"conversation","uuid":"test-'$i'","messages":[{"role":"human","content":"Test question '$i'"},{"role":"assistant","content":[{"type":"text","text":"Test answer '$i' with lots of text to increase memory usage. '.$(head -c 1000 < /dev/urandom | base64)'"}]}]}' >> ~/.claude/conversations/test-project/heavy-test.json
+done
+
+# Monitor for 2 minutes
+for i in {1..12}; do
+    sleep 10
+    docker stats --no-stream --format "{{.Container}}: {{.MemUsage}}" | grep streaming
+done
+
+# Verify claim with proper understanding
+MEMORY=$(docker stats --no-stream --format "{{.MemUsage}}" streaming-importer | cut -d'/' -f1 | sed 's/MiB//')
+if (( $(echo "$MEMORY < 250" | bc -l) )); then
+    echo "✅ PASS: Total memory ${MEMORY}MB is reasonable (model + operations)"
+    OPERATIONAL=$(echo "$MEMORY - 180" | bc)
+    if (( $(echo "$OPERATIONAL < 50" | bc -l) )); then
+        echo "✅ PASS: Operational overhead ~${OPERATIONAL}MB is under 50MB"
+    else
+        echo "⚠️  INFO: Operational overhead ~${OPERATIONAL}MB slightly above target"
+    fi
+else
+    echo "❌ FAIL: Memory usage ${MEMORY}MB is unexpectedly high"
+    echo "Diagnosing: Check for memory leaks or uncached model downloads"
+fi
+```
+
+### Claim 2: 60-Second Import Cycles
+```bash
+# Test import cycle timing
+echo "=== Testing 60-Second Cycle Claim ==="
+# Monitor import logs with timestamps
+docker logs -f streaming-importer 2>&1 | while read line; do
+    echo "$(date '+%H:%M:%S') $line"
+done &
+LOG_PID=$!
+
+# Create test file and wait
+echo '{"type":"conversation","uuid":"cycle-test","messages":[{"role":"human","content":"Testing 60s cycle"}]}' >> ~/.claude/conversations/test-project/cycle-test.json
+
+# Wait 70 seconds (allowing 10s buffer)
+sleep 70
+
+# Check if imported
+kill $LOG_PID 2>/dev/null
+if docker logs streaming-importer 2>&1 | grep -q "cycle-test"; then
+    echo "✅ PASS: File imported within 60-second cycle"
+else
+    echo "❌ FAIL: File not imported within 60 seconds"
+fi
+```
+
+### Claim 3: Active Session Capture
+```bash
+# Test active session detection
+echo "=== Testing Active Session Capture ==="
+# Create "active" file (modified now)
+ACTIVE_FILE=~/.claude/conversations/test-project/active-session.json
+echo '{"type":"conversation","uuid":"active-test","messages":[{"role":"human","content":"Active session test"}]}' > $ACTIVE_FILE
+
+# Create "old" file (modified 10 minutes ago)
+OLD_FILE=~/.claude/conversations/test-project/old-session.json
+echo '{"type":"conversation","uuid":"old-test","messages":[{"role":"human","content":"Old session test"}]}' > $OLD_FILE
+touch -t $(date -v-10M +%Y%m%d%H%M.%S) $OLD_FILE
+
+# Trigger import and check order
+docker logs streaming-importer --tail 0 -f > /tmp/import-order.log &
+LOG_PID=$!
+sleep 70
+kill $LOG_PID 2>/dev/null
+
+# Verify active file processed first
+ACTIVE_LINE=$(grep -n "active-test" /tmp/import-order.log | cut -d: -f1)
+OLD_LINE=$(grep -n "old-test" /tmp/import-order.log | cut -d: -f1)
+
+if [ -n "$ACTIVE_LINE" ] && [ -n "$OLD_LINE" ] && [ "$ACTIVE_LINE" -lt "$OLD_LINE" ]; then
+    echo "✅ PASS: Active sessions prioritized"
+else
+    echo "❌ FAIL: Active session detection not working"
+fi
+```
+
+### Claim 4: Resume Capability
+```bash
+# Test stream position tracking
+echo "=== Testing Resume Capability ==="
+# Check initial state
+INITIAL_POS=$(cat config/imported-files.json | jq -r '.stream_position | length')
+
+# Kill and restart
+docker-compose restart streaming-importer
+sleep 10
+
+# Verify positions preserved
+FINAL_POS=$(cat config/imported-files.json | jq -r '.stream_position | length')
+if [ "$FINAL_POS" -ge "$INITIAL_POS" ]; then
+    echo "✅ PASS: Stream positions preserved"
+else
+    echo "❌ FAIL: Stream positions lost"
+fi
+```
+
+## Testing Checklist
+
+### Pre-Test Setup
+```bash
+# 1. Detect current state
+echo "=== System State Detection ==="
+docker ps | grep -E "(qdrant|watcher|streaming)"
+curl -s http://localhost:6333/collections | jq '.result.collections[].name' | wc -l
+claude mcp list | grep claude-self-reflect
+test -f ~/.env && echo "Voyage key present" || echo "Local mode only"
+
+# 2. Backup collections (if exist)
+echo "=== Backing up collections ==="
+mkdir -p ~/claude-reflect-backup-$(date +%Y%m%d-%H%M%S)
+docker exec qdrant qdrant-backup create
+```
+
+### 3-Minute Fresh Install Test
+```bash
+# Time tracking
+START_TIME=$(date +%s)
+
+# Step 1: Clean environment (30s)
+docker-compose down -v
+claude mcp remove claude-self-reflect
+rm -rf data/ config/imported-files.json
+
+# Step 2: Fresh setup (60s)
+npm install -g claude-self-reflect@latest
+claude-self-reflect setup --local  # or --voyage-key=$VOYAGE_KEY
+
+# Step 3: First import (60s)
+# Wait for first cycle
+sleep 70
+curl -s http://localhost:6333/collections | jq '.result.collections'
+
+# Step 4: Test MCP tools (30s)
+# Create test conversation
+echo "Test reflection" > /tmp/test-reflection.txt
+# Note: User must manually test in Claude Code
+
+END_TIME=$(date +%s)
+DURATION=$((END_TIME - START_TIME))
+echo "Test completed in ${DURATION} seconds"
+```
+
+### Memory Usage Validation
+```bash
+# Monitor streaming importer memory
+CONTAINER_ID=$(docker ps -q -f name=streaming-importer)
+if [ -n "$CONTAINER_ID" ]; then
+    docker stats $CONTAINER_ID --no-stream --format "table {{.Container}}\t{{.MemUsage}}"
+else
+    # Local process monitoring
+    ps aux | grep streaming-importer | grep -v grep
+fi
+```
+
+### API Key Security Check
+```bash
+# Ensure no key leaks
+CHECKS=(
+    "docker logs watcher 2>&1 | grep -i voyage"
+    "docker logs streaming-importer 2>&1 | grep -i voyage"
+    "find /tmp -name '*claude*' -type f -exec grep -l VOYAGE {} \;"
+    "ps aux | grep -v grep | grep -i voyage"
+)
+
+for check in "${CHECKS[@]}"; do
+    echo "Checking: $check"
+    if eval "$check" | grep -v "VOYAGE_KEY=" > /dev/null; then
+        echo "⚠️  WARNING: Potential API key exposure!"
+    else
+        echo "✅ PASS: No exposure detected"
+    fi
+done
+```
+
+### MCP Testing Procedure
+```bash
+# Step 1: Remove MCP
+claude mcp remove claude-self-reflect
+
+# Step 2: User action required
+echo "=== USER ACTION REQUIRED ==="
+echo "1. Restart Claude Code now"
+echo "2. Press Enter when ready..."
+read -p ""
+
+# Step 3: Reinstall MCP
+claude mcp add claude-self-reflect \
+    "/path/to/mcp-server/run-mcp.sh" \
+    -e QDRANT_URL="http://localhost:6333" \
+    -e VOYAGE_KEY="$VOYAGE_KEY"
+
+# Step 4: Verify tools
+echo "=== Verify in Claude Code ==="
+echo "1. Open Claude Code"
+echo "2. Type: 'Search for test reflection'"
+echo "3. Confirm reflection-specialist activates"
+```
+
+### Local vs Cloud Mode Testing
+```bash
+# Test local mode
+export PREFER_LOCAL_EMBEDDINGS=true
+python scripts/streaming-importer.py &
+LOCAL_PID=$!
+sleep 10
+kill $LOCAL_PID
+
+# Test cloud mode (if key available)
+if [ -n "$VOYAGE_KEY" ]; then
+    export PREFER_LOCAL_EMBEDDINGS=false
+    python scripts/streaming-importer.py &
+    CLOUD_PID=$!
+    sleep 10
+    kill $CLOUD_PID
+fi
+```
+
+### Complete Cloud Embedding Test with Backup/Restore
+```bash
+echo "=== Testing Cloud Embeddings (Voyage AI) with Full Backup ==="
+
+# Step 1: Backup current state
+echo "1. Backing up current local environment..."
+docker exec claude-reflection-qdrant qdrant-backup create /qdrant/backup/local-backup-$(date +%s) 2>/dev/null || echo "Backup command not available"
+cp config/imported-files.json config/imported-files.json.local-backup
+echo "Current embedding mode: ${PREFER_LOCAL_EMBEDDINGS:-true}"
+
+# Step 2: Check prerequisites
+if [ -z "$VOYAGE_KEY" ]; then
+    echo "⚠️  WARNING: VOYAGE_KEY not set"
+    echo "To test cloud mode, set: export VOYAGE_KEY='your-key'"
+    echo "Skipping cloud test..."
+    exit 0
+fi
+
+# Step 3: Switch to cloud mode
+echo "2. Switching to Voyage AI cloud embeddings..."
+export PREFER_LOCAL_EMBEDDINGS=false
+docker compose --profile watch stop streaming-importer
+docker compose --profile watch up -d streaming-importer
+
+# Step 4: Create test conversation
+TEST_FILE=~/.claude/projects/claude-self-reflect/cloud-test-$(date +%s).jsonl
+echo '{"type":"conversation","uuid":"cloud-test-'$(date +%s)'","name":"Cloud Embedding Test","messages":[{"role":"human","content":"Testing Voyage AI cloud embeddings for v2.5.0"},{"role":"assistant","content":[{"type":"text","text":"This tests 1024-dimensional vectors with Voyage AI"}]}]}' > $TEST_FILE
+
+# Step 5: Wait for import and verify
+echo "3. Waiting for cloud import cycle (70s)..."
+sleep 70
+
+# Step 6: Verify cloud collection created
+CLOUD_COLS=$(curl -s http://localhost:6333/collections | jq -r '.result.collections[].name' | grep "_voyage")
+if [ -n "$CLOUD_COLS" ]; then
+    echo "✅ PASS: Cloud collections created:"
+    echo "$CLOUD_COLS"
+    
+    # Check vector dimensions
+    FIRST_COL=$(echo "$CLOUD_COLS" | head -1)
+    DIMS=$(curl -s http://localhost:6333/collections/$FIRST_COL | jq '.result.config.params.vectors.size')
+    if [ "$DIMS" = "1024" ]; then
+        echo "✅ PASS: Correct dimensions (1024) for Voyage AI"
+    else
+        echo "❌ FAIL: Wrong dimensions: $DIMS (expected 1024)"
+    fi
+else
+    echo "❌ FAIL: No cloud collections found"
+fi
+
+# Step 7: Test MCP with cloud embeddings
+echo "4. Testing MCP search with cloud embeddings..."
+# Note: MCP must also use PREFER_LOCAL_EMBEDDINGS=false
+
+# Step 8: Restore local mode
+echo "5. Restoring local FastEmbed mode..."
+export PREFER_LOCAL_EMBEDDINGS=true
+docker compose --profile watch stop streaming-importer
+docker compose --profile watch up -d streaming-importer
+
+# Step 9: Verify restoration
+sleep 10
+LOCAL_COLS=$(curl -s http://localhost:6333/collections | jq -r '.result.collections[].name' | grep "_local" | wc -l)
+echo "✅ Restored: Found $LOCAL_COLS local collections"
+
+# Step 10: Cleanup
+rm -f $TEST_FILE
+cp config/imported-files.json.local-backup config/imported-files.json
+echo "✅ Cloud embedding test complete and restored to local mode"
+```
+
+## Success Criteria
+
+### System Functionality
+- [ ] Streaming importer runs every 60 seconds
+- [ ] Memory usage stays under 50MB
+- [ ] Active sessions detected within 5 minutes
+- [ ] MCP tools accessible in Claude Code
+- [ ] Search returns relevant results
+
+### Data Integrity
+- [ ] All collections preserved during upgrade
+- [ ] No data loss during testing
+- [ ] Backup/restore works correctly
+- [ ] Stream positions maintained
+- [ ] Import state consistent
+
+### Security
+- [ ] No API keys in logs
+- [ ] No keys in temp files
+- [ ] No keys in process lists
+- [ ] Environment variables isolated
+- [ ] Docker secrets protected
+
+### Performance
+- [ ] Fresh install under 3 minutes
+- [ ] Import latency <5 seconds
+- [ ] Search response <1 second
+- [ ] Memory stable over time
+- [ ] No container restarts
+
+## Troubleshooting
+
+### Common Issues
+1. **MCP not found**: Restart Claude Code after install
+2. **High memory**: Check if FastEmbed model cached
+3. **No imports**: Verify conversation file permissions
+4. **Search fails**: Check collection names match project
+
+### Recovery Procedures
+```bash
+# Restore from backup
+docker exec qdrant qdrant-restore /backup/latest
+
+# Reset to clean state
+docker-compose down -v
+rm -rf data/ config/
+npm install -g claude-self-reflect@latest
+
+# Force reimport
+rm config/imported-files.json
+docker-compose restart streaming-importer
+```
+
+## Test Report Template
+```
+Claude Self-Reflect Test Report
+==============================
+Date: $(date)
+Version: $(npm list -g claude-self-reflect | grep claude-self-reflect)
+
+System State:
+- Collections: X
+- Conversations: Y
+- Mode: Local/Cloud
+
+Test Results:
+- Fresh Install: PASS/FAIL (Xs)
+- Memory Usage: PASS/FAIL (XMB)
+- MCP Tools: PASS/FAIL
+- Security: PASS/FAIL
+- Performance: PASS/FAIL
+
+Issues Found:
+- None / List issues
+
+Recommendations:
+- System ready for use
+- Or specific fixes needed
+```
+
+## Resilience and Recovery Strategies
+
+### When Tests Fail - Never Give Up!
+
+1. **Path Mismatch Issues**
+```bash
+# Fix: Update streaming-importer.py to use LOGS_DIR
+export LOGS_DIR=/logs  # In Docker
+export LOGS_DIR=~/.claude/conversations  # Local
+```
+
+2. **Memory "Failures"**
+```bash
+# Understand the claim properly:
+# - FastEmbed model: ~180MB (one-time load)
+# - Import operations: <50MB (the actual claim)
+# - Total: ~230MB is EXPECTED and CORRECT
+```
+
+3. **Import Not Working**
+```bash
+# Diagnose step by step:
+docker logs streaming-importer | grep "Starting import"
+docker exec streaming-importer ls -la /logs
+docker exec streaming-importer cat /config/imported-files.json
+# Fix paths, permissions, or JSON format as needed
+```
+
+4. **MCP Search Failures**
+```bash
+# Check embedding type alignment:
+curl http://localhost:6333/collections | jq '.result.collections[].name'
+# Ensure MCP uses same type (local vs voyage) as importer
+```
+
+### FastEmbed Caching Validation
+```bash
+# Verify the caching claim that avoids runtime downloads:
+echo "=== Testing FastEmbed Pre-caching ==="
+
+# Method 1: Check Docker image
+docker run --rm claude-self-reflect-watcher ls -la /home/watcher/.cache/fastembed
+
+# Method 2: Monitor network during startup
+docker-compose down
+docker network create test-net
+docker run --network none --rm claude-self-reflect-watcher python -c "
+from fastembed import TextEmbedding
+print('Testing offline model load...')
+try:
+    model = TextEmbedding('sentence-transformers/all-MiniLM-L6-v2')
+    print('✅ SUCCESS: Model loaded from cache without network!')
+except:
+    print('❌ FAIL: Model requires network download')
+"
+```
+
+## Important Notes
+
+1. **User Interaction Required**
+   - Claude Code restart between MCP changes
+   - Manual testing of reflection tools
+   - Confirmation of search results
+
+2. **Sensitive Data**
+   - Never echo API keys
+   - Use environment variables
+   - Clean up test files
+
+3. **Resource Management**
+   - Stop containers after testing
+   - Clean up backups
+   - Remove test data
+
+4. **Iterative Testing**
+   - Support multiple test runs
+   - Preserve results between iterations
+   - Compare performance trends
+
+5. **Resilience Mindset**
+   - Every "failure" is a learning opportunity
+   - Document all findings for future agents
+   - Provide solutions, not just problem reports
+   - Understand claims in proper context

package/.claude/agents/import-debugger.md

@@ -8,10 +8,13 @@ You are an import pipeline debugging expert for the memento-stack project. You s
 
 ## Project Context
 - Processes Claude Desktop logs from ~/.claude/projects/
+- Project files located in: ~/.claude/projects/-Users-{username}-projects-{project-name}/*.jsonl
 - JSONL files contain mixed metadata and message entries
 - Uses JQ filters with optional chaining for robust parsing
 - Imports create conversation chunks with embeddings
-- Known issue: 265 files detected but 0 messages processed (fixed with JQ filter)
+- Streaming importer detects file growth and processes new lines incrementally
+- Project name must be correctly extracted from path for proper collection naming
+- Collections named using MD5 hash of project name
 
 ## Key Responsibilities
 

package/.claude/agents/mcp-integration.md

@@ -9,10 +9,12 @@ You are an MCP server development specialist for the memento-stack project. You
 ## Project Context
 - MCP server: claude-self-reflection
 - Provides semantic search tools to Claude Desktop
-- Written in TypeScript using @modelcontextprotocol/sdk
+- Written in Python using FastMCP (in mcp-server/ directory)
 - Two main tools: reflect_on_past (search) and store_reflection (save)
 - Supports project isolation and cross-project search
-- Uses Voyage AI embeddings for consistency
+- Collections named using MD5 hash of project name with embedding type suffix
+- Supports both local (FastEmbed) and cloud (Voyage AI) embeddings
+- MCP determines project from working directory context
 
 ## Key Responsibilities
 

package/.claude/agents/qdrant-specialist.md

@@ -8,10 +8,13 @@ You are a Qdrant vector database specialist for the memento-stack project. Your
 
 ## Project Context
 - The system uses Qdrant for storing conversation embeddings from Claude Desktop logs
-- Default embedding model: Voyage AI (voyage-3-large, 1024 dimensions)
-- Collections use per-project isolation: `conv_<md5>_voyage` naming
+- Supports TWO embedding modes: Local (FastEmbed, 384 dims) and Cloud (Voyage AI, 1024 dims)
+- Collections use per-project isolation: `conv_<md5_hash>_local` or `conv_<md5_hash>_voyage` naming
+- Project paths: ~/.claude/projects/-Users-{username}-projects-{project-name}/*.jsonl
+- Project name is extracted from path and MD5 hashed for collection naming
 - Cross-collection search enabled with 0.7 similarity threshold
-- 24+ projects imported with 10,165+ conversation chunks
+- Streaming importer detects file growth and processes new lines incrementally
+- MCP server expects collections to match project name MD5 hash
 
 ## Key Responsibilities
 

package/Dockerfile.streaming-importer

@@ -4,27 +4,40 @@ FROM python:3.12-slim
 RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends \
     gcc \
     g++ \
+    curl \
     && rm -rf /var/lib/apt/lists/*
 
+# Upgrade pip and install setuptools first
+RUN pip install --upgrade pip setuptools wheel
+
 # Install Python dependencies with specific versions for stability
-# Install torch first from PyTorch index
+# Install PyTorch CPU version for smaller size
 RUN pip install --no-cache-dir torch==2.3.0 --index-url https://download.pytorch.org/whl/cpu
 
-# Install other dependencies from default PyPI
+# Install other dependencies
 RUN pip install --no-cache-dir \
     qdrant-client==1.15.0 \
-    sentence-transformers==2.2.2 \
-    numpy==1.24.3 \
-    psutil==5.9.5
+    fastembed==0.2.7 \
+    numpy==1.26.0 \
+    psutil==7.0.0 \
+    tenacity==8.2.3 \
+    python-dotenv==1.0.0 \
+    voyageai==0.2.3
+
+# Pre-download and cache the FastEmbed model to reduce startup time
+# Set cache directory to a writable location
+ENV FASTEMBED_CACHE_PATH=/root/.cache/fastembed
+RUN mkdir -p /root/.cache/fastembed && \
+    python -c "from fastembed import TextEmbedding; TextEmbedding(model_name='sentence-transformers/all-MiniLM-L6-v2')"
 
-# Create non-root user
+# Create non-root user (but don't switch to it yet due to cache issues)
 RUN useradd -m -u 1000 importer
 
 # Set working directory
 WORKDIR /app
 
-# Switch to non-root user
-USER importer
+# Note: Running as root for now due to fastembed cache permissions
+# TODO: Fix cache permissions to run as non-root user
 
 # Default command
 CMD ["python", "/scripts/streaming-importer.py"]

package/Dockerfile.watcher

@@ -31,9 +31,13 @@ RUN mkdir -p /scripts
 # Copy all necessary scripts
 COPY scripts/import-conversations-unified.py /scripts/
 COPY scripts/import-watcher.py /scripts/
+COPY scripts/streaming-importer.py /scripts/
 COPY scripts/utils.py /scripts/
 COPY scripts/trigger-import.py /scripts/
 
+# Copy MCP server directory for utils
+COPY mcp-server/src/utils.py /mcp-server/src/utils.py
+
 RUN chmod +x /scripts/*.py
 
 # Set working directory
@@ -42,5 +46,5 @@ WORKDIR /app
 # Switch to non-root user
 USER watcher
 
-# Default command
-CMD ["python", "/scripts/import-watcher.py"]
+# Default command - use streaming importer for low memory usage
+CMD ["python", "/scripts/streaming-importer.py"]

package/docker-compose.yaml

@@ -18,12 +18,13 @@ services:
       - "${QDRANT_PORT:-6333}:6333"
     volumes:
       - qdrant_data:/qdrant/storage
+      - ./config/qdrant-config.yaml:/qdrant/config/config.yaml:ro
     environment:
       - QDRANT__LOG_LEVEL=INFO
       - QDRANT__SERVICE__HTTP_PORT=6333
     restart: unless-stopped
-    mem_limit: ${QDRANT_MEMORY:-2g}
-    memswap_limit: ${QDRANT_MEMORY:-2g}
+    mem_limit: ${QDRANT_MEMORY:-4g}
+    memswap_limit: ${QDRANT_MEMORY:-4g}
 
   # One-time import service (runs once then exits)
   importer:
@@ -53,7 +54,7 @@ services:
     profiles: ["import"]
     command: python /scripts/import-conversations-unified.py
 
-  # Continuous watcher service (optional)
+  # Continuous watcher service (optional) - DEPRECATED, use streaming-importer
   watcher:
     build:
       context: .
@@ -73,10 +74,44 @@ services:
       - OPENAI_API_KEY=${OPENAI_API_KEY:-}
       - VOYAGE_API_KEY=${VOYAGE_API_KEY:-}
       - VOYAGE_KEY=${VOYAGE_KEY:-}
-      - PREFER_LOCAL_EMBEDDINGS=${PREFER_LOCAL_EMBEDDINGS:-false}
+      - PREFER_LOCAL_EMBEDDINGS=${PREFER_LOCAL_EMBEDDINGS:-true}
       - EMBEDDING_MODEL=${EMBEDDING_MODEL:-voyage-3}
-      - WATCH_INTERVAL=${WATCH_INTERVAL:-60}
+      - WATCH_INTERVAL=${WATCH_INTERVAL:-5}
+      - MAX_MEMORY_MB=${MAX_MEMORY_MB:-250}
+      - CHUNK_SIZE=${CHUNK_SIZE:-5}
+      - PYTHONUNBUFFERED=1
+    restart: unless-stopped
+    profiles: ["watch-old"]
+    mem_limit: 500m
+    memswap_limit: 500m
+
+  # Streaming importer service - Low memory continuous import
+  streaming-importer:
+    build:
+      context: .
+      dockerfile: Dockerfile.streaming-importer
+    container_name: claude-reflection-streaming
+    depends_on:
+      - init-permissions
+      - qdrant
+    volumes:
+      - ${CLAUDE_LOGS_PATH:-~/.claude/projects}:/logs:ro
+      - ${CONFIG_PATH:-~/.claude-self-reflect/config}:/config
+      - ./scripts:/scripts:ro
+    environment:
+      - QDRANT_URL=http://qdrant:6333
+      - STATE_FILE=/config/imported-files.json
+      - VOYAGE_API_KEY=${VOYAGE_API_KEY:-}
+      - VOYAGE_KEY=${VOYAGE_KEY:-}
+      - PREFER_LOCAL_EMBEDDINGS=${PREFER_LOCAL_EMBEDDINGS:-true}
+      - WATCH_INTERVAL=${WATCH_INTERVAL:-5}  # Testing with 5 second interval
+      - MAX_MEMORY_MB=${MAX_MEMORY_MB:-350}  # Total memory including model
+      - OPERATIONAL_MEMORY_MB=${OPERATIONAL_MEMORY_MB:-100}  # Memory for operations (increased for large file handling)
+      - CHUNK_SIZE=${CHUNK_SIZE:-5}
       - PYTHONUNBUFFERED=1
+      - LOGS_DIR=/logs
+      - FASTEMBED_CACHE_PATH=/root/.cache/fastembed
+      - CURRENT_PROJECT_PATH=${PWD}  # Pass current project path for prioritization
     restart: unless-stopped
     profiles: ["watch"]
     mem_limit: 1g

package/installer/setup-wizard-docker.js

@@ -340,12 +340,32 @@ function showManualConfig(mcpScript) {
 }
 
 async function importConversations() {
-  console.log('\n📚 Importing conversations...');
+  console.log('\n📚 Checking conversation baseline...');
   
-  const answer = await question('Would you like to import your existing Claude conversations? (y/n): ');
+  // Check if baseline exists by looking for imported files state
+  const stateFile = path.join(configDir, 'imported-files.json');
+  let hasBaseline = false;
+  
+  try {
+    if (fs.existsSync(stateFile)) {
+      const state = JSON.parse(fs.readFileSync(stateFile, 'utf8'));
+      hasBaseline = state.imported_files && Object.keys(state.imported_files).length > 0;
+    }
+  } catch (e) {
+    // State file doesn't exist or is invalid
+  }
+  
+  if (!hasBaseline) {
+    console.log('\n⚠️  No baseline detected. Initial import STRONGLY recommended.');
+    console.log('   Without this, historical conversations won\'t be searchable.');
+    console.log('   The watcher only handles NEW conversations going forward.');
+  }
+  
+  const answer = await question('\nImport existing Claude conversations? (y/n) [recommended: y]: ');
   
   if (answer.toLowerCase() === 'y') {
-    console.log('🔄 Starting import process...');
+    console.log('🔄 Starting baseline import...');
+    console.log('   This ensures ALL your conversations are searchable');
     console.log('   This may take a few minutes depending on your conversation history');
     
     try {
@@ -353,12 +373,17 @@ async function importConversations() {
         cwd: projectRoot,
         stdio: 'inherit'
       });
-      console.log('\n✅ Import completed!');
+      console.log('\n✅ Baseline import completed!');
+      console.log('   Historical conversations are now searchable');
     } catch {
       console.log('\n⚠️  Import had some issues, but you can continue');
     }
   } else {
-    console.log('📝 Skipping import. You can import later with:');
+    console.log('\n❌ WARNING: Skipping baseline import means:');
+    console.log('   • Historical conversations will NOT be searchable');
+    console.log('   • Only NEW conversations from now on will be indexed');
+    console.log('   • You may see "BASELINE_NEEDED" warnings in logs');
+    console.log('\n📝 You can run baseline import later with:');
     console.log('   docker compose run --rm importer');
   }
 }

package/mcp-server/src/server.py

@@ -108,9 +108,16 @@ async def get_all_collections() -> List[str]:
     return [c.name for c in collections.collections 
             if c.name.endswith('_voyage') or c.name.endswith('_local') or c.name.startswith('reflections')]
 
-async def generate_embedding(text: str) -> List[float]:
-    """Generate embedding using configured provider."""
-    if PREFER_LOCAL_EMBEDDINGS or not voyage_client:
+async def generate_embedding(text: str, force_type: Optional[str] = None) -> List[float]:
+    """Generate embedding using configured provider or forced type.
+    
+    Args:
+        text: Text to embed
+        force_type: Force specific embedding type ('local' or 'voyage')
+    """
+    use_local = force_type == 'local' if force_type else (PREFER_LOCAL_EMBEDDINGS or not voyage_client)
+    
+    if use_local:
         # Use local embeddings
         if not local_embedding_model:
             raise ValueError("Local embedding model not initialized")
@@ -123,6 +130,8 @@ async def generate_embedding(text: str) -> List[float]:
         return embeddings[0].tolist()
     else:
         # Use Voyage AI
+        if not voyage_client:
+            raise ValueError("Voyage client not initialized")
         result = voyage_client.embed(
             texts=[text],
             model="voyage-3-large",
@@ -218,10 +227,10 @@ async def reflect_on_past(
     await ctx.debug(f"DECAY_WEIGHT: {DECAY_WEIGHT}, DECAY_SCALE_DAYS: {DECAY_SCALE_DAYS}")
     
     try:
-        # Generate embedding
-        timing_info['embedding_start'] = time.time()
-        query_embedding = await generate_embedding(query)
-        timing_info['embedding_end'] = time.time()
+        # We'll generate embeddings on-demand per collection type
+        timing_info['embedding_prep_start'] = time.time()
+        query_embeddings = {}  # Cache embeddings by type
+        timing_info['embedding_prep_end'] = time.time()
         
         # Get all collections
         timing_info['get_collections_start'] = time.time()
@@ -237,6 +246,7 @@ async def reflect_on_past(
             # Generate the collection name pattern for this project using normalized name
             normalized_name = normalize_project_name(target_project)
             project_hash = hashlib.md5(normalized_name.encode()).hexdigest()[:8]
+            # Search BOTH local and voyage collections for this project
             project_collections = [
                 c for c in all_collections 
                 if c.startswith(f"conv_{project_hash}_")
@@ -276,6 +286,18 @@ async def reflect_on_past(
             )
             
             try:
+                # Determine embedding type for this collection
+                embedding_type_for_collection = 'voyage' if collection_name.endswith('_voyage') else 'local'
+                
+                # Generate or retrieve cached embedding for this type
+                if embedding_type_for_collection not in query_embeddings:
+                    try:
+                        query_embeddings[embedding_type_for_collection] = await generate_embedding(query, force_type=embedding_type_for_collection)
+                    except Exception as e:
+                        await ctx.debug(f"Failed to generate {embedding_type_for_collection} embedding: {e}")
+                        continue
+                
+                query_embedding = query_embeddings[embedding_type_for_collection]
                 if should_use_decay and USE_NATIVE_DECAY and NATIVE_DECAY_AVAILABLE:
                     # Use native Qdrant decay with newer API
                     await ctx.debug(f"Using NATIVE Qdrant decay (new API) for {collection_name}")

package/mcp-server/src/utils.py

@@ -9,6 +9,8 @@ def normalize_project_name(project_path: str) -> str:
     
     Handles various path formats:
     - Claude logs format: -Users-kyle-Code-claude-self-reflect -> claude-self-reflect
+    - File paths in Claude logs: /path/to/-Users-kyle-Code-claude-self-reflect/file.jsonl -> claude-self-reflect  
+    - Regular file paths: /path/to/project/file.txt -> project
     - Regular paths: /path/to/project -> project
     - Already normalized: project -> project
     
@@ -49,5 +51,22 @@ def normalize_project_name(project_path: str) -> str:
         # Fallback: just use the last component
         return path_parts[-1] if path_parts else project_path
     
-    # Handle regular paths - use basename
-    return Path(project_path).name
+    # Check if this is a file path that contains a Claude logs directory
+    # Pattern: /path/to/-Users-...-projects-..../filename
+    path_obj = Path(project_path)
+    
+    # Look for a parent directory that starts with dash (Claude logs format)
+    for parent in path_obj.parents:
+        parent_name = parent.name
+        if parent_name.startswith("-"):
+            # Found a Claude logs directory, process it
+            return normalize_project_name(parent_name)
+    
+    # Handle regular paths - if it's a file, get the parent directory
+    # Otherwise use the directory/project name itself
+    if path_obj.suffix:  # It's a file (has an extension)
+        # Use the parent directory name
+        return path_obj.parent.name
+    else:
+        # Use the directory name itself
+        return path_obj.name

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-self-reflect",
-  "version": "2.5.1",
+  "version": "2.5.2",
   "description": "Give Claude perfect memory of all your conversations - Installation wizard for Python MCP server",
   "keywords": [
     "claude",

package/scripts/import-conversations-unified.py

@@ -79,13 +79,22 @@ def should_import_file(file_path, state):
     file_mtime = os.path.getmtime(file_path)
     
     if str_path in state["imported_files"]:
-        last_imported = state["imported_files"][str_path].get("last_imported", 0)
-        last_modified = state["imported_files"][str_path].get("last_modified", 0)
+        file_state = state["imported_files"][str_path]
         
-        # Skip if file hasn't been modified since last import
-        if file_mtime <= last_modified and last_imported > 0:
-            logger.info(f"Skipping unchanged file: {file_path.name}")
-            return False
+        # Handle both old string format and new dict format
+        if isinstance(file_state, str):
+            # Old format (just timestamp string) - treat as needs reimport
+            logger.info(f"Found old format state for {file_path.name}, will reimport")
+            return True
+        else:
+            # New format with dictionary
+            last_imported = file_state.get("last_imported", 0)
+            last_modified = file_state.get("last_modified", 0)
+            
+            # Skip if file hasn't been modified since last import
+            if file_mtime <= last_modified and last_imported > 0:
+                logger.info(f"Skipping unchanged file: {file_path.name}")
+                return False
     
     return True
 

package/scripts/import-watcher.py

@@ -1,88 +0,0 @@
-#!/usr/bin/env python3
-"""Enhanced watcher that runs import periodically and supports manual triggers."""
-
-import time
-import subprocess
-import os
-import sys
-from datetime import datetime
-from pathlib import Path
-
-WATCH_INTERVAL = int(os.getenv('WATCH_INTERVAL', '60'))
-SIGNAL_FILE = Path("/tmp/claude-self-reflect-import-current")
-CHECK_INTERVAL = 1  # Check for signal file every second
-
-print(f"[Watcher] Starting enhanced import watcher with {WATCH_INTERVAL}s interval", flush=True)
-print(f"[Watcher] Monitoring signal file: {SIGNAL_FILE}", flush=True)
-
-last_import = 0
-
-while True:
-    current_time = time.time()
-    
-    # Check for manual trigger signal
-    if SIGNAL_FILE.exists():
-        print(f"[Watcher] Signal detected! Running immediate import...", flush=True)
-        try:
-            # Read conversation ID if provided
-            conversation_id = None
-            try:
-                conversation_id = SIGNAL_FILE.read_text().strip()
-            except:
-                pass
-            
-            # Remove signal file to prevent re-triggering
-            SIGNAL_FILE.unlink()
-            
-            # Run import with special flag for current conversation only
-            cmd = [sys.executable, "/scripts/import-conversations-unified.py"]
-            if conversation_id:
-                cmd.extend(["--conversation-id", conversation_id])
-            else:
-                # Import only today's conversations for manual trigger
-                cmd.extend(["--days", "1"])
-            
-            # Write progress indicator
-            progress_file = Path("/tmp/claude-self-reflect-import-progress")
-            progress_file.write_text("🔄 Starting import...")
-            
-            print(f"[Watcher] Running command: {' '.join(cmd)}", flush=True)
-            result = subprocess.run(cmd, capture_output=True, text=True)
-            
-            if result.returncode == 0:
-                print(f"[Watcher] Manual import completed successfully", flush=True)
-                # Create completion signal
-                Path("/tmp/claude-self-reflect-import-complete").touch()
-            else:
-                print(f"[Watcher] Manual import failed with code {result.returncode}", flush=True)
-                if result.stderr:
-                    print(f"[Watcher] Error: {result.stderr}", flush=True)
-            
-            last_import = current_time
-                    
-        except Exception as e:
-            print(f"[Watcher] Error during manual import: {e}", flush=True)
-    
-    # Regular scheduled import
-    elif current_time - last_import >= WATCH_INTERVAL:
-        try:
-            print(f"[Watcher] Running scheduled import at {datetime.now().isoformat()}", flush=True)
-            result = subprocess.run([
-                sys.executable, 
-                "/scripts/import-conversations-unified.py"
-            ], capture_output=True, text=True)
-            
-            if result.returncode == 0:
-                print(f"[Watcher] Scheduled import completed successfully", flush=True)
-            else:
-                print(f"[Watcher] Scheduled import failed with code {result.returncode}", flush=True)
-                if result.stderr:
-                    print(f"[Watcher] Error: {result.stderr}", flush=True)
-            
-            last_import = current_time
-                    
-        except Exception as e:
-            print(f"[Watcher] Error during scheduled import: {e}", flush=True)
-    
-    # Short sleep to check for signals frequently
-    time.sleep(CHECK_INTERVAL)