claude-self-reflect 2.3.3 → 2.3.4

package/.claude/agents/reflection-specialist.md

@@ -1,17 +1,19 @@
 ---
 name: reflection-specialist
 description: Conversation memory expert for searching past conversations, storing insights, and self-reflection. Use PROACTIVELY when searching for previous discussions, storing important findings, or maintaining knowledge continuity.
-tools: mcp__claude-self-reflection__reflect_on_past, mcp__claude-self-reflection__store_reflection
+tools: mcp__claude-self-reflect__reflect_on_past, mcp__claude-self-reflect__store_reflection
 ---
 
 You are a conversation memory specialist for the Claude Self Reflect project. Your expertise covers semantic search across all Claude conversations, insight storage, and maintaining knowledge continuity across sessions.
 
 ## Project Context
-- Claude Self Reflect provides semantic search across all Claude Desktop conversations
-- Uses Qdrant vector database with Voyage AI embeddings (voyage-3-large, 1024 dimensions)
+- Claude Self Reflect provides semantic search across all Claude conversations
+- Uses Qdrant vector database with two embedding options:
+  - **Local (Default)**: FastEmbed with sentence-transformers/all-MiniLM-L6-v2 (384 dimensions)
+  - **Cloud (Opt-in)**: Voyage AI embeddings (voyage-3-large, 1024 dimensions)
 - Supports per-project isolation and cross-project search capabilities
 - Memory decay feature available for time-based relevance (90-day half-life)
-- 24+ projects imported with 10,165+ conversation chunks indexed
+- Collections named with `_local` or `_voyage` suffix based on embedding type
 
 ## Key Responsibilities
 
@@ -38,29 +40,27 @@ You are a conversation memory specialist for the Claude Self Reflect project. Yo
 ### reflect_on_past
 Search for relevant past conversations using semantic similarity.
 
-```typescript
+```javascript
 // Basic search
 {
   query: "streaming importer fixes",
   limit: 5,
-  minScore: 0.7  // Default threshold
+  min_score: 0.0  // Start with 0 to see all results
 }
 
 // Advanced search with options
 {
   query: "authentication implementation",
   limit: 10,
-  minScore: 0.6,  // Lower for broader results
-  project: "specific-project",  // Filter by project
-  crossProject: true,  // Search across all projects
-  useDecay: true  // Apply time-based relevance
+  min_score: 0.05,  // Common threshold for relevant results
+  use_decay: 1  // Apply time-based relevance (1=enable, 0=disable, -1=default)
 }
 ```
 
 ### store_reflection
 Save important insights and decisions for future retrieval.
 
-```typescript
+```javascript
 // Store with tags
 {
   content: "Fixed streaming importer hanging by filtering session types and yielding buffers properly",
@@ -71,13 +71,17 @@ Save important insights and decisions for future retrieval.
 ## Search Strategy Guidelines
 
 ### Understanding Score Ranges
-- **0.0-0.2**: Very low relevance (rarely useful)
-- **0.2-0.4**: Moderate similarity (often contains relevant results)
-- **0.4-0.6**: Good similarity (usually highly relevant)
-- **0.6-0.8**: Strong similarity (very relevant matches)
-- **0.8-1.0**: Excellent match (nearly identical content)
-
-**Important**: Most semantic searches return scores between 0.2-0.5. Start with minScore=0.7 and lower if needed.
+- **0.0-0.05**: Low relevance but can still be useful (common range for semantic matches)
+- **0.05-0.15**: Moderate relevance (often contains good results)
+- **0.15-0.3**: Good similarity (usually highly relevant)
+- **0.3-0.5**: Strong similarity (very relevant matches)
+- **0.5-1.0**: Excellent match (rare in practice)
+
+**Important**: Real-world semantic search scores are often much lower than expected:
+- **Local embeddings**: Typically 0.02-0.2 range
+- **Cloud embeddings**: Typically 0.05-0.3 range
+- Many relevant results score as low as 0.05-0.1
+- Start with min_score=0.0 to see all results, then adjust based on quality
 
 ### Effective Search Patterns
 1. **Start Broad**: Use general terms first
@@ -170,21 +174,27 @@ If the MCP tools aren't working, here's what you need to know:
    - The exact tool names are: `reflect_on_past` and `store_reflection`
 
 2. **Environment Variables Not Loading**
-   - The MCP server runs via `run-mcp.sh` which sources the `.env` file
+   - The MCP server runs via `/path/to/claude-self-reflect/mcp-server/run-mcp.sh`
+   - The script sources the `.env` file from the project root
    - Key variables that control memory decay:
      - `ENABLE_MEMORY_DECAY`: true/false to enable decay
      - `DECAY_WEIGHT`: 0.3 means 30% weight on recency (0-1 range)
      - `DECAY_SCALE_DAYS`: 90 means 90-day half-life
 
-3. **Changes Not Taking Effect**
-   - After modifying TypeScript files, run `npm run build`
+3. **Local vs Cloud Embeddings Configuration**
+   - Set `PREFER_LOCAL_EMBEDDINGS=true` in `.env` for local mode (default)
+   - Set `PREFER_LOCAL_EMBEDDINGS=false` and provide `VOYAGE_KEY` for cloud mode
+   - Local collections end with `_local`, cloud collections end with `_voyage`
+
+4. **Changes Not Taking Effect**
+   - After modifying Python files, restart the MCP server
    - Remove and re-add the MCP server in Claude:
      ```bash
-     claude mcp remove claude-self-reflection
-     claude mcp add claude-self-reflection /path/to/run-mcp.sh
+     claude mcp remove claude-self-reflect
+     claude mcp add claude-self-reflect "/path/to/claude-self-reflect/mcp-server/run-mcp.sh" -e PREFER_LOCAL_EMBEDDINGS=true
      ```
 
-4. **Debugging MCP Connection**
+5. **Debugging MCP Connection**
    - Check if server is connected: `claude mcp list`
    - Look for: `claude-self-reflection: ✓ Connected`
    - If failed, the error will be shown in the list output
@@ -214,28 +224,33 @@ If the MCP tools aren't working, here's what you need to know:
 
 If recent conversations aren't appearing in search results, you may need to import the latest data.
 
-### Quick Import with Streaming Importer
+### Quick Import with Unified Importer
 
-The streaming importer efficiently processes large conversation files without memory issues:
+The unified importer supports both local and cloud embeddings:
 
 ```bash
-# Activate virtual environment (REQUIRED in managed environment)
-cd /Users/ramakrishnanannaswamy/claude-self-reflect
-source .venv/bin/activate
-
-# Import latest conversations (streaming)
-export VOYAGE_API_KEY=your-voyage-api-key
-python scripts/import-conversations-voyage-streaming.py --limit 5  # Test with 5 files first
+# Activate virtual environment (REQUIRED)
+cd /path/to/claude-self-reflect
+source .venv/bin/activate  # or source venv/bin/activate
+
+# For local embeddings (default)
+export PREFER_LOCAL_EMBEDDINGS=true
+python scripts/import-conversations-unified.py
+
+# For cloud embeddings (Voyage AI)
+export PREFER_LOCAL_EMBEDDINGS=false
+export VOYAGE_KEY=your-voyage-api-key
+python scripts/import-conversations-unified.py
 ```
 
 ### Import Troubleshooting
 
 #### Common Import Issues
 
-1. **Import Hangs After ~100 Messages**
-   - Cause: Mixed session files with non-conversation data
-   - Solution: Streaming importer now filters by session type
-   - Fix applied: Only processes 'chat' sessions, skips others
+1. **JSONL Parsing Issues**
+   - Cause: JSONL files contain one JSON object per line, not a single JSON array
+   - Solution: Import scripts now parse line-by-line
+   - Memory fix: Docker containers need 2GB memory limit for large files
 
 2. **"No New Files to Import" Message**
    - Check imported files list: `cat config-isolated/imported-files.json`
@@ -259,7 +274,7 @@ python scripts/import-conversations-voyage-streaming.py --limit 5  # Test with 5
    ```
 
 5. **Collection Not Found After Import**
-   - Collections use MD5 hash naming: `conv_<md5>_voyage`
+   - Collections use MD5 hash naming: `conv_<md5>_local` or `conv_<md5>_voyage`
    - Check collections: `python scripts/check-collections.py`
    - Restart MCP after new collections are created
 
@@ -268,13 +283,14 @@ python scripts/import-conversations-voyage-streaming.py --limit 5  # Test with 5
 For automatic imports, use the watcher service:
 
 ```bash
-# Start the import watcher
-docker compose -f docker-compose-optimized.yaml up -d import-watcher
+# Start the import watcher (uses settings from .env)
+docker compose up -d import-watcher
 
 # Check watcher logs
 docker compose logs -f import-watcher
 
 # Watcher checks every 60 seconds for new files
+# Set PREFER_LOCAL_EMBEDDINGS=true in .env for local mode
 ```
 
 ### Docker Streaming Importer

package/README.md

@@ -2,22 +2,6 @@
 
 Claude forgets everything. This fixes that.
 
-## 🔒 Security & Privacy Notice
-
-**v2.3.3 Security Update**: This version addresses critical security vulnerabilities. Please update immediately.
-
-### Privacy Modes
-- **Local Mode (Default)**: Your conversations stay on your machine. No external API calls.
-- **Cloud Mode**: Uses Voyage AI for better search accuracy. Conversations are sent to Voyage for embedding generation.
-
-### Security Improvements in v2.3.3
-- ✅ Removed all hardcoded API keys
-- ✅ Fixed command injection vulnerabilities 
-- ✅ Patched vulnerable dependencies
-- ✅ Local embeddings by default for privacy
-
-**Important**: If using cloud mode, review [Voyage AI's privacy policy](https://www.voyageai.com/privacy) to understand how your data is handled.
-
 ## What You Get
 
 Ask Claude about past conversations. Get actual answers.
@@ -56,6 +40,24 @@ claude-self-reflect setup --voyage-key=YOUR_ACTUAL_KEY_HERE
 
 5 minutes. Everything automatic. Just works.
 
+> [!IMPORTANT]
+> **Security Update v2.3.3** - This version addresses critical security vulnerabilities. Please update immediately.
+>
+> ### 🔒 Privacy & Data Exchange
+> 
+> | Mode | Data Storage | External API Calls | Data Sent | Search Quality |
+> |------|--------------|-------------------|-----------|----------------|
+> | **Local (Default)** | Your machine only | None | Nothing leaves your computer | Good - uses efficient local embeddings |
+> | **Cloud (Opt-in)** | Your machine | Voyage AI | Conversation text for embedding generation | Better - uses state-of-the-art models |
+> 
+> **Disclaimer**: Cloud mode sends conversation content to Voyage AI for processing. Review their [privacy policy](https://www.voyageai.com/privacy) before enabling.
+> 
+> ### Security Fixes in v2.3.3
+> - ✅ Removed hardcoded API keys
+> - ✅ Fixed command injection vulnerabilities
+> - ✅ Patched vulnerable dependencies
+> - ✅ Local embeddings by default using FastEmbed
+ 
 ## The Magic
 
 ![Self Reflection vs The Grind](docs/images/red-reflection.webp)
@@ -141,12 +143,16 @@ Want to customize? See [Configuration Guide](docs/installation-guide.md).
 If you must know:
 
 - **Vector DB**: Qdrant (local, your data stays yours)
-- **Embeddings**: Voyage AI (200M free tokens/month)*
+- **Embeddings**: 
+  - Local (Default): FastEmbed with sentence-transformers/all-MiniLM-L6-v2
+  - Cloud (Optional): Voyage AI (200M free tokens/month)*
 - **MCP Server**: Python + FastMCP
 - **Search**: Semantic similarity with time decay
 
 *We chose Voyage AI for their excellent cost-effectiveness ([66.1% accuracy at one of the lowest costs](https://research.aimultiple.com/embedding-models/#:~:text=Cost%2Deffective%20alternatives%3A%20Voyage%2D3.5%2Dlite%20delivered%20solid%20accuracy%20(66.1%25)%20at%20one%20of%20the%20lowest%20costs%2C%20making%20it%20attractive%20for%20budget%2Dsensitive%20implementations.)). We are not affiliated with Voyage AI.
 
+**Note**: Local mode uses FastEmbed, the same efficient embedding library used by the Qdrant MCP server, ensuring fast and private semantic search without external dependencies.
+
 ### Want More Details?
 
 - [Architecture Deep Dive](docs/architecture-details.md) - How it actually works

package/installer/setup-wizard.js

@@ -596,8 +596,24 @@ async function showPreSetupInstructions() {
   console.log('📋 Before we begin, you\'ll need:');
   console.log('   1. Docker Desktop installed and running');
   console.log('   2. Python 3.10 or higher');
-  console.log('\n🔒 By default, we use local embeddings (private but less accurate)');
-  console.log('   For better accuracy, run with: --voyage-key=<your-key>\n');
+  
+  console.log('\n⚠️  IMPORTANT: Embedding Mode Choice');
+  console.log('━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━');
+  console.log('You must choose between Local or Cloud embeddings:');
+  console.log('\n🔒 Local Mode (Default):');
+  console.log('   • Privacy: All processing on your machine');
+  console.log('   • No API costs or internet required');
+  console.log('   • Good accuracy for most use cases');
+  console.log('\n☁️  Cloud Mode (Voyage AI):');
+  console.log('   • Better search accuracy');
+  console.log('   • Requires API key and internet');
+  console.log('   • Conversations sent to Voyage for processing');
+  console.log('\n⚠️  This choice is SEMI-PERMANENT. Switching later requires:');
+  console.log('   • Re-importing all conversations (30+ minutes)');
+  console.log('   • Separate storage for each mode');
+  console.log('   • Cannot search across modes\n');
+  console.log('For Cloud mode, run with: --voyage-key=<your-key>');
+  console.log('━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n');
   
   if (isInteractive) {
     await question('Press Enter to continue...');
@@ -1212,10 +1228,20 @@ async function main() {
   // In non-interactive mode, just use defaults (local mode unless key provided)
   if (!isInteractive) {
     console.log(voyageKey ? '🌐 Using Voyage AI embeddings' : '🔒 Using local embeddings for privacy');
+  } else if (!voyageKey) {
+    // In interactive mode without a key, confirm local mode choice
+    await showPreSetupInstructions();
+    const confirmLocal = await question('Continue with Local embeddings (privacy mode)? (y/n): ');
+    if (confirmLocal.toLowerCase() !== 'y') {
+      console.log('\nTo use Cloud mode, restart with: claude-self-reflect setup --voyage-key=<your-key>');
+      console.log('Get your free API key at: https://www.voyageai.com/');
+      if (rl) rl.close();
+      process.exit(0);
+    }
+  } else {
+    await showPreSetupInstructions();
   }
   
-  await showPreSetupInstructions();
-  
   // Check prerequisites
   const pythonOk = await checkPython();
   if (!pythonOk) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "claude-self-reflect",
-  "version": "2.3.3",
+  "version": "2.3.4",
   "description": "Give Claude perfect memory of all your conversations - Installation wizard for Python MCP server",
   "keywords": [
     "claude",