claude-self-reflect 2.3.2 → 2.3.4

package/.claude/agents/reflection-specialist.md

@@ -1,17 +1,19 @@
 ---
 name: reflection-specialist
 description: Conversation memory expert for searching past conversations, storing insights, and self-reflection. Use PROACTIVELY when searching for previous discussions, storing important findings, or maintaining knowledge continuity.
-tools: mcp__claude-self-reflection__reflect_on_past, mcp__claude-self-reflection__store_reflection
+tools: mcp__claude-self-reflect__reflect_on_past, mcp__claude-self-reflect__store_reflection
 ---
 
 You are a conversation memory specialist for the Claude Self Reflect project. Your expertise covers semantic search across all Claude conversations, insight storage, and maintaining knowledge continuity across sessions.
 
 ## Project Context
-- Claude Self Reflect provides semantic search across all Claude Desktop conversations
-- Uses Qdrant vector database with Voyage AI embeddings (voyage-3-large, 1024 dimensions)
+- Claude Self Reflect provides semantic search across all Claude conversations
+- Uses Qdrant vector database with two embedding options:
+  - **Local (Default)**: FastEmbed with sentence-transformers/all-MiniLM-L6-v2 (384 dimensions)
+  - **Cloud (Opt-in)**: Voyage AI embeddings (voyage-3-large, 1024 dimensions)
 - Supports per-project isolation and cross-project search capabilities
 - Memory decay feature available for time-based relevance (90-day half-life)
-- 24+ projects imported with 10,165+ conversation chunks indexed
+- Collections named with `_local` or `_voyage` suffix based on embedding type
 
 ## Key Responsibilities
 
@@ -38,29 +40,27 @@ You are a conversation memory specialist for the Claude Self Reflect project. Yo
 ### reflect_on_past
 Search for relevant past conversations using semantic similarity.
 
-```typescript
+```javascript
 // Basic search
 {
   query: "streaming importer fixes",
   limit: 5,
-  minScore: 0.7  // Default threshold
+  min_score: 0.0  // Start with 0 to see all results
 }
 
 // Advanced search with options
 {
   query: "authentication implementation",
   limit: 10,
-  minScore: 0.6,  // Lower for broader results
-  project: "specific-project",  // Filter by project
-  crossProject: true,  // Search across all projects
-  useDecay: true  // Apply time-based relevance
+  min_score: 0.05,  // Common threshold for relevant results
+  use_decay: 1  // Apply time-based relevance (1=enable, 0=disable, -1=default)
 }
 ```
 
 ### store_reflection
 Save important insights and decisions for future retrieval.
 
-```typescript
+```javascript
 // Store with tags
 {
   content: "Fixed streaming importer hanging by filtering session types and yielding buffers properly",
@@ -71,13 +71,17 @@ Save important insights and decisions for future retrieval.
 ## Search Strategy Guidelines
 
 ### Understanding Score Ranges
-- **0.0-0.2**: Very low relevance (rarely useful)
-- **0.2-0.4**: Moderate similarity (often contains relevant results)
-- **0.4-0.6**: Good similarity (usually highly relevant)
-- **0.6-0.8**: Strong similarity (very relevant matches)
-- **0.8-1.0**: Excellent match (nearly identical content)
-
-**Important**: Most semantic searches return scores between 0.2-0.5. Start with minScore=0.7 and lower if needed.
+- **0.0-0.05**: Low relevance but can still be useful (common range for semantic matches)
+- **0.05-0.15**: Moderate relevance (often contains good results)
+- **0.15-0.3**: Good similarity (usually highly relevant)
+- **0.3-0.5**: Strong similarity (very relevant matches)
+- **0.5-1.0**: Excellent match (rare in practice)
+
+**Important**: Real-world semantic search scores are often much lower than expected:
+- **Local embeddings**: Typically 0.02-0.2 range
+- **Cloud embeddings**: Typically 0.05-0.3 range
+- Many relevant results score as low as 0.05-0.1
+- Start with min_score=0.0 to see all results, then adjust based on quality
 
 ### Effective Search Patterns
 1. **Start Broad**: Use general terms first
@@ -170,21 +174,27 @@ If the MCP tools aren't working, here's what you need to know:
    - The exact tool names are: `reflect_on_past` and `store_reflection`
 
 2. **Environment Variables Not Loading**
-   - The MCP server runs via `run-mcp.sh` which sources the `.env` file
+   - The MCP server runs via `/path/to/claude-self-reflect/mcp-server/run-mcp.sh`
+   - The script sources the `.env` file from the project root
    - Key variables that control memory decay:
      - `ENABLE_MEMORY_DECAY`: true/false to enable decay
      - `DECAY_WEIGHT`: 0.3 means 30% weight on recency (0-1 range)
      - `DECAY_SCALE_DAYS`: 90 means 90-day half-life
 
-3. **Changes Not Taking Effect**
-   - After modifying TypeScript files, run `npm run build`
+3. **Local vs Cloud Embeddings Configuration**
+   - Set `PREFER_LOCAL_EMBEDDINGS=true` in `.env` for local mode (default)
+   - Set `PREFER_LOCAL_EMBEDDINGS=false` and provide `VOYAGE_KEY` for cloud mode
+   - Local collections end with `_local`, cloud collections end with `_voyage`
+
+4. **Changes Not Taking Effect**
+   - After modifying Python files, restart the MCP server
    - Remove and re-add the MCP server in Claude:
      ```bash
-     claude mcp remove claude-self-reflection
-     claude mcp add claude-self-reflection /path/to/run-mcp.sh
+     claude mcp remove claude-self-reflect
+     claude mcp add claude-self-reflect "/path/to/claude-self-reflect/mcp-server/run-mcp.sh" -e PREFER_LOCAL_EMBEDDINGS=true
      ```
 
-4. **Debugging MCP Connection**
+5. **Debugging MCP Connection**
    - Check if server is connected: `claude mcp list`
    - Look for: `claude-self-reflection: ✓ Connected`
    - If failed, the error will be shown in the list output
@@ -214,28 +224,33 @@ If the MCP tools aren't working, here's what you need to know:
 
 If recent conversations aren't appearing in search results, you may need to import the latest data.
 
-### Quick Import with Streaming Importer
+### Quick Import with Unified Importer
 
-The streaming importer efficiently processes large conversation files without memory issues:
+The unified importer supports both local and cloud embeddings:
 
 ```bash
-# Activate virtual environment (REQUIRED in managed environment)
-cd /Users/ramakrishnanannaswamy/claude-self-reflect
-source .venv/bin/activate
-
-# Import latest conversations (streaming)
-export VOYAGE_API_KEY=your-voyage-api-key
-python scripts/import-conversations-voyage-streaming.py --limit 5  # Test with 5 files first
+# Activate virtual environment (REQUIRED)
+cd /path/to/claude-self-reflect
+source .venv/bin/activate  # or source venv/bin/activate
+
+# For local embeddings (default)
+export PREFER_LOCAL_EMBEDDINGS=true
+python scripts/import-conversations-unified.py
+
+# For cloud embeddings (Voyage AI)
+export PREFER_LOCAL_EMBEDDINGS=false
+export VOYAGE_KEY=your-voyage-api-key
+python scripts/import-conversations-unified.py
 ```
 
 ### Import Troubleshooting
 
 #### Common Import Issues
 
-1. **Import Hangs After ~100 Messages**
-   - Cause: Mixed session files with non-conversation data
-   - Solution: Streaming importer now filters by session type
-   - Fix applied: Only processes 'chat' sessions, skips others
+1. **JSONL Parsing Issues**
+   - Cause: JSONL files contain one JSON object per line, not a single JSON array
+   - Solution: Import scripts now parse line-by-line
+   - Memory fix: Docker containers need 2GB memory limit for large files
 
 2. **"No New Files to Import" Message**
    - Check imported files list: `cat config-isolated/imported-files.json`
@@ -259,7 +274,7 @@ python scripts/import-conversations-voyage-streaming.py --limit 5  # Test with 5
    ```
 
 5. **Collection Not Found After Import**
-   - Collections use MD5 hash naming: `conv_<md5>_voyage`
+   - Collections use MD5 hash naming: `conv_<md5>_local` or `conv_<md5>_voyage`
    - Check collections: `python scripts/check-collections.py`
    - Restart MCP after new collections are created
 
@@ -268,13 +283,14 @@ python scripts/import-conversations-voyage-streaming.py --limit 5  # Test with 5
 For automatic imports, use the watcher service:
 
 ```bash
-# Start the import watcher
-docker compose -f docker-compose-optimized.yaml up -d import-watcher
+# Start the import watcher (uses settings from .env)
+docker compose up -d import-watcher
 
 # Check watcher logs
 docker compose logs -f import-watcher
 
 # Watcher checks every 60 seconds for new files
+# Set PREFER_LOCAL_EMBEDDINGS=true in .env for local mode
 ```
 
 ### Docker Streaming Importer

package/README.md

@@ -13,31 +13,51 @@ Your conversations become searchable. Your decisions stay remembered. Your conte
 
 ## Install
 
-### Quick Start (Recommended)
+### Quick Start (Local Mode - Default)
 ```bash
-# Step 1: Get your free Voyage AI key
-# Sign up at https://www.voyageai.com/ - it takes 30 seconds
-
-# Step 2: Install and run automatic setup
+# Install and run automatic setup
 npm install -g claude-self-reflect
-claude-self-reflect setup --voyage-key=YOUR_ACTUAL_KEY_HERE
+claude-self-reflect setup
 
 # That's it! The setup will:
 # ✅ Configure everything automatically
 # ✅ Install the MCP in Claude Code  
 # ✅ Start monitoring for new conversations
 # ✅ Verify the reflection tools work
+# 🔒 Keep all data local - no API keys needed
 ```
 
-### Alternative: Local Mode (No API Key)
+### Cloud Mode (Better Search Accuracy)
 ```bash
+# Step 1: Get your free Voyage AI key
+# Sign up at https://www.voyageai.com/ - it takes 30 seconds
+
+# Step 2: Install with Voyage key
 npm install -g claude-self-reflect
-claude-self-reflect setup --local
+claude-self-reflect setup --voyage-key=YOUR_ACTUAL_KEY_HERE
 ```
-*Note: Local mode uses basic embeddings. Semantic search won't be as good.*
+*Note: Cloud mode provides more accurate semantic search but sends conversation data to Voyage AI for processing.*
 
 5 minutes. Everything automatic. Just works.
 
+> [!IMPORTANT]
+> **Security Update v2.3.3** - This version addresses critical security vulnerabilities. Please update immediately.
+>
+> ### 🔒 Privacy & Data Exchange
+> 
+> | Mode | Data Storage | External API Calls | Data Sent | Search Quality |
+> |------|--------------|-------------------|-----------|----------------|
+> | **Local (Default)** | Your machine only | None | Nothing leaves your computer | Good - uses efficient local embeddings |
+> | **Cloud (Opt-in)** | Your machine | Voyage AI | Conversation text for embedding generation | Better - uses state-of-the-art models |
+> 
+> **Disclaimer**: Cloud mode sends conversation content to Voyage AI for processing. Review their [privacy policy](https://www.voyageai.com/privacy) before enabling.
+> 
+> ### Security Fixes in v2.3.3
+> - ✅ Removed hardcoded API keys
+> - ✅ Fixed command injection vulnerabilities
+> - ✅ Patched vulnerable dependencies
+> - ✅ Local embeddings by default using FastEmbed
+ 
 ## The Magic
 
 ![Self Reflection vs The Grind](docs/images/red-reflection.webp)
@@ -123,12 +143,16 @@ Want to customize? See [Configuration Guide](docs/installation-guide.md).
 If you must know:
 
 - **Vector DB**: Qdrant (local, your data stays yours)
-- **Embeddings**: Voyage AI (200M free tokens/month)*
+- **Embeddings**: 
+  - Local (Default): FastEmbed with sentence-transformers/all-MiniLM-L6-v2
+  - Cloud (Optional): Voyage AI (200M free tokens/month)*
 - **MCP Server**: Python + FastMCP
 - **Search**: Semantic similarity with time decay
 
 *We chose Voyage AI for their excellent cost-effectiveness ([66.1% accuracy at one of the lowest costs](https://research.aimultiple.com/embedding-models/#:~:text=Cost%2Deffective%20alternatives%3A%20Voyage%2D3.5%2Dlite%20delivered%20solid%20accuracy%20(66.1%25)%20at%20one%20of%20the%20lowest%20costs%2C%20making%20it%20attractive%20for%20budget%2Dsensitive%20implementations.)). We are not affiliated with Voyage AI.
 
+**Note**: Local mode uses FastEmbed, the same efficient embedding library used by the Qdrant MCP server, ensuring fast and private semantic search without external dependencies.
+
 ### Want More Details?
 
 - [Architecture Deep Dive](docs/architecture-details.md) - How it actually works