claude-self-reflect 6.0.5 → 7.1.8

package/.claude/agents/csr-validator.md

@@ -93,7 +93,79 @@ find . -type f \( -name "*test_*.py" -o -name "test_*.py" -o -name "*benchmark*.
 echo "=== Suggest archiving to: tests/throwaway/ ==="
 ```
 
-### 8. NPM Package Validation (Regression Check for #71)
+### 8. Ralph Loop Integration Validation (v7.1+)
+
+**What is Ralph Loop?**
+The Ralph Wiggum technique helps Claude maintain focus on long, complex tasks. With CSR integration, Ralph loops gain cross-session memory through hooks.
+
+**CRITICAL: Runaway Loop Prevention**
+- ALWAYS use `--max-iterations` (PLURAL!) as safety net
+- `--max-iteration` (singular) is IGNORED - loop runs forever!
+- Setting `active: false` does NOT stop loops - must DELETE file
+- To stop: `rm .claude/ralph-loop.local.md`
+
+**v7.1+ Enhanced Features:**
+- **Error Signature Deduplication**: Normalizes errors to avoid storing duplicates
+- **Output Decline Detection**: Circuit breaker pattern (detects >70% output drop)
+- **Confidence-Based Exit**: 0-100 scoring for exit decisions
+- **Anti-Pattern Injection**: "DON'T RETRY THESE" surfaced first
+- **Work Type Tracking**: IMPLEMENTATION/TESTING/DEBUGGING/DOCUMENTATION
+- **Error-Centric Search**: Find past sessions by error pattern, not just task
+
+```bash
+# Check Ralph hooks in settings.json
+echo "=== Ralph Hooks Validation ==="
+grep -c "session_start_hook\|session_end_hook\|precompact" ~/.claude/settings.json && echo "✅ Hooks configured" || echo "❌ Hooks missing"
+
+# Verify hook scripts exist
+echo "=== Hook Scripts ==="
+ls -la src/runtime/hooks/session_start_hook.py 2>/dev/null && echo "✅ SessionStart" || echo "❌ SessionStart missing"
+ls -la src/runtime/hooks/session_end_hook.py 2>/dev/null && echo "✅ SessionEnd" || echo "❌ SessionEnd missing"
+ls -la src/runtime/precompact-hook.sh 2>/dev/null && echo "✅ PreCompact" || echo "❌ PreCompact missing"
+
+# Test RalphState import (NOT RalphStateParser)
+python3 -c "
+import sys
+sys.path.insert(0, 'src/runtime/hooks')
+from ralph_state import RalphState
+print('✅ RalphState imports')
+" || echo "❌ RalphState import failed"
+
+# Check for Ralph-related CLI integration
+grep -l "ralph" installer/setup-wizard-docker.js && echo "✅ CLI integration" || echo "❌ CLI missing Ralph"
+
+# Test v7.1+ enhanced features
+echo "=== Testing v7.1+ Enhanced Features ==="
+python3 -c "
+import sys
+sys.path.insert(0, 'src/runtime/hooks')
+from ralph_state import RalphState
+
+# Quick feature check
+state = RalphState.create_new('test', 'test')
+state.add_error('Error at line 42')
+state.track_output(1000)
+state.update_confidence({'all_tasks_complete': True})
+state.work_type = 'TESTING'
+
+print('✅ Error dedup:', len(state.error_signatures), 'signatures')
+print('✅ Output tracking:', len(state.output_lengths), 'samples')
+print('✅ Confidence:', state.exit_confidence, '%')
+print('✅ Work type:', state.work_type)
+print('✅ All v7.1+ features work')
+"
+```
+
+**Test CSR Search for Ralph Sessions:**
+```python
+# Search for past Ralph sessions
+results = await csr_reflect_on_past("Ralph loop session", limit=3, min_score=0.3)
+
+# Quick check
+quick = await csr_quick_check("Ralph Wiggum")
+```
+
+### 9. NPM Package Validation (Regression Check for #71)
 ```bash
 echo "=== NPM Package Contents Check ==="
 
@@ -166,6 +238,20 @@ CodeRabbit Analysis: [PASS/FAIL]
 - PR feedback checked: [✓/✗]
 - Issues found: [none/list]
 
+Ralph Loop Integration (v7.1+): [PASS/FAIL]
+- Hooks in settings.json: [✓/✗]
+- SessionStart hook: [✓/✗]
+- SessionEnd hook: [✓/✗]
+- PreCompact hook: [✓/✗]
+- RalphState module: [✓/✗]
+- CLI integration: [✓/✗]
+- CSR search for Ralph: [✓/✗]
+- v7.1+ Enhanced Features:
+  - Error signature dedup: [✓/✗]
+  - Output decline detection: [✓/✗]
+  - Confidence scoring: [✓/✗]
+  - Work type tracking: [✓/✗]
+
 Critical Issues: [none/list]
 
 CLEANUP NEEDED:

package/.env.example

@@ -28,3 +28,52 @@ CHUNK_SIZE=50
 MAX_FILES_PER_CYCLE=10
 HOT_WINDOW_MINUTES=15
 MAX_COLD_FILES_PER_CYCLE=3
+
+# ====================================================================================
+# Batch Automation Configuration (OPTIONAL - Requires Anthropic API Key)
+# ====================================================================================
+# Batch automation provides 9.3x better search quality with automated narratives
+# Disabled by default - enable via CLI during installation or manually
+
+# Anthropic API Key (required for batch automation)
+# Get your key from https://console.anthropic.com
+ANTHROPIC_API_KEY=
+
+# Qdrant API Key (optional for standalone mode, required for shared/multi-user)
+# STANDALONE: Leave empty if running locally just for yourself
+# SHARED: Set a strong API key if multiple people access this Qdrant instance
+QDRANT_API_KEY=
+
+# Batch Automation Directories
+CSR_HOME=~/.claude-self-reflect
+CSR_CONFIG_DIR=~/.claude-self-reflect/config
+CSR_BATCH_STATE_DIR=~/.claude-self-reflect/batch_state
+CSR_BATCH_QUEUE_DIR=~/.claude-self-reflect/batch_queue
+
+# Batch Triggers
+BATCH_SIZE_TRIGGER=10                    # Trigger batch after N files
+BATCH_TIME_TRIGGER_MINUTES=30            # Or after N minutes
+
+# Subprocess Settings
+SUBPROCESS_TIMEOUT_SECONDS=1800          # 30 minute timeout for batch operations
+
+# Watcher Timing
+HOT_CHECK_INTERVAL_S=2                   # Check hot files every N seconds
+NORMAL_CHECK_INTERVAL_S=60               # Normal check interval
+WARM_WINDOW_HOURS=24                     # Files < N hours are warm
+MAX_COLD_FILES=5                         # Max cold files per cycle
+
+# ====================================================================================
+# Evaluation System Configuration (OPTIONAL - Session Health Checks)
+# ====================================================================================
+# Automatically runs health checks on session start to verify MCP tools,
+# search quality, and performance are functioning correctly
+
+# Enable automatic evaluation at session start
+EVAL_ON_STARTUP=false                    # Set to true for session-start health checks
+
+# Evaluation timeout settings
+EVAL_TIMEOUT_SECONDS=30                  # Maximum time for evaluation run
+
+# Performance targets
+EVAL_PERFORMANCE_TARGET_MS=500           # Target search latency in milliseconds

package/Dockerfile.batch-monitor

@@ -0,0 +1,36 @@
+FROM python:3.11-slim
+
+WORKDIR /app
+
+# Install system dependencies
+RUN apt-get update && apt-get install -y \
+    gcc \
+    g++ \
+    && rm -rf /var/lib/apt/lists/*
+
+# Copy requirements
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Copy source code
+COPY src/ ./src/
+COPY docs/design/ ./docs/design/
+
+# Create non-root user
+RUN groupadd -r appuser && \
+    useradd -r -g appuser -u 1001 appuser
+
+# Create directories for batch state and set ownership
+RUN mkdir -p /home/appuser/.claude-self-reflect/batch_state && \
+    chown -R appuser:appuser /home/appuser/.claude-self-reflect && \
+    chown -R appuser:appuser /app
+
+# Set Python path
+ENV PYTHONPATH=/app
+ENV HOME=/home/appuser
+
+# Switch to non-root user
+USER appuser
+
+# Run batch monitor
+CMD ["python", "/app/src/runtime/batch_monitor.py"]

package/Dockerfile.batch-watcher

@@ -0,0 +1,38 @@
+FROM python:3.11-slim
+
+WORKDIR /app
+
+# Install system dependencies
+RUN apt-get update && apt-get install -y \
+    gcc \
+    g++ \
+    && rm -rf /var/lib/apt/lists/*
+
+# Copy requirements
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Copy source code
+COPY src/ ./src/
+COPY docs/design/ ./docs/design/
+
+# Create non-root user
+RUN groupadd -r appuser && \
+    useradd -r -g appuser -u 1001 appuser
+
+# Create directories for state and queue and set ownership
+RUN mkdir -p /home/appuser/.claude-self-reflect/config && \
+    mkdir -p /home/appuser/.claude-self-reflect/batch_queue && \
+    mkdir -p /home/appuser/.claude-self-reflect/batch_state && \
+    chown -R appuser:appuser /home/appuser/.claude-self-reflect && \
+    chown -R appuser:appuser /app
+
+# Set Python path
+ENV PYTHONPATH=/app
+ENV HOME=/home/appuser
+
+# Switch to non-root user
+USER appuser
+
+# Run batch watcher
+CMD ["python", "/app/src/runtime/batch_watcher.py"]

package/README.md

@@ -26,6 +26,8 @@ Give Claude perfect memory of all your conversations. Search past discussions in
 
 **100% Local by Default** • **20x Faster** • **Zero Configuration** • **Production Ready**
 
+> **Latest: v7.0 Automated Narratives** - 9.3x better search quality via AI-powered summaries. [Learn more →](#v70-automated-narrative-generation)
+
 ## Why This Exists
 
 Claude starts fresh every conversation. You've solved complex bugs, designed architectures, made critical decisions - all forgotten. Until now.
@@ -39,6 +41,7 @@ Claude starts fresh every conversation. You've solved complex bugs, designed arc
 - [Real Examples](#real-examples)
 - [NEW: Real-time Indexing Status](#new-real-time-indexing-status-in-your-terminal)
 - [Key Features](#key-features)
+- [Ralph Loop Memory Integration](#ralph-loop-memory-integration)
 - [Code Quality Insights](#code-quality-insights)
 - [Architecture](#architecture)
 - [Requirements](#requirements)
@@ -63,7 +66,7 @@ claude-self-reflect setup
 ```
 
 > [!TIP]
-> **v4.0+ Auto-Migration**: Updates from v3.x automatically migrate during npm install - no manual steps needed!
+> **Auto-Migration**: Updates automatically handle breaking changes. Simply run `npm update -g claude-self-reflect`.
 
 <details open>
 <summary>Cloud Mode (Better Search Accuracy)</summary>
@@ -84,18 +87,15 @@ claude-self-reflect setup --voyage-key=YOUR_ACTUAL_KEY_HERE
 
 ## Performance
 
-<details open>
-<summary><b>v4.0 Performance Improvements</b></summary>
-
-| Metric | v3.x | v4.0 | Improvement |
-|--------|------|------|-------------|
+| Metric | Before | After | Improvement |
+|--------|--------|-------|-------------|
 | **Status Check** | 119ms | 6ms | **20x faster** |
 | **Storage Usage** | 100MB | 50MB | **50% reduction** |
 | **Import Speed** | 10/sec | 100/sec | **10x faster** |
 | **Memory Usage** | 500MB | 50MB | **90% reduction** |
 | **Search Latency** | 15ms | 3ms | **5x faster** |
 
-### How We Compare
+### Competitive Comparison
 
 | Feature | Claude Self-Reflect | MemGPT | LangChain Memory |
 |---------|---------------------|---------|------------------|
@@ -106,8 +106,6 @@ claude-self-reflect setup --voyage-key=YOUR_ACTUAL_KEY_HERE
 | **Setup time** | 5 min | 30+ min | 20+ min |
 | **Docker required** | Yes | Python | Python |
 
-</details>
-
 ## The Magic
 
 ![Self Reflection vs The Grind](docs/images/red-reflection.webp)
@@ -177,32 +175,115 @@ Your code quality displayed live as you work:
 
 </details>
 
+## v7.0 Automated Narrative Generation
+
+**9.3x Better Search Quality** • **50% Cost Savings** • **Fully Automated**
+
+v7.0 introduces AI-powered conversation narratives that transform raw conversation excerpts into rich problem-solution summaries with comprehensive metadata extraction.
+
+### Before/After Comparison
+
+| Metric | v6.x (Raw Excerpts) | v7.0 (AI Narratives) | Improvement |
+|--------|---------------------|----------------------|-------------|
+| **Search Quality** | 0.074 | 0.691 | **9.3x better** |
+| **Token Compression** | 100% | 18% | **82% reduction** |
+| **Cost per Conversation** | $0.025 | $0.012 | **50% savings** |
+| **Metadata Richness** | Basic | Tools + Concepts + Files | **Full context** |
+
+### What You Get
+
+**Enhanced Search Results:**
+- **Problem-Solution Patterns**: Conversations structured as challenges encountered and solutions implemented
+- **Rich Metadata**: Automatic extraction of tools used, technical concepts, and files modified
+- **Context Compression**: 82% token reduction while maintaining searchability
+- **Better Relevance**: Search scores improved from 0.074 to 0.691 (9.3x)
+
+**Cost-Effective Processing:**
+- Anthropic Batch API: $0.012 per conversation (vs $0.025 standard)
+- Automatic batch queuing and processing
+- Progress monitoring via Docker containers
+- Evaluation generation for quality assurance
+
+**Fully Automated Workflow:**
+```bash
+# 1. Watch for new conversations
+docker compose up batch-watcher
+
+# 2. Auto-trigger batch processing when threshold reached
+# (Configurable: BATCH_THRESHOLD_FILES, default 10)
+
+# 3. Monitor batch progress
+docker compose logs batch-monitor -f
+
+# 4. Enhanced narratives automatically imported to Qdrant
+```
+
+### Example: Raw Excerpt vs AI Narrative
+
+**Before (v6.x)** - Raw excerpt showing basic conversation flow:
+```
+User: How do I fix the Docker memory issue?
+Assistant: The container was limited to 2GB but only using 266MB...
+```
+
+**After (v7.0)** - Rich narrative with metadata:
+```
+PROBLEM: Docker container memory consumption investigation revealed
+discrepancy between limits (2GB) and actual usage (266MB). Analysis
+required to determine if memory limit was appropriate.
+
+SOLUTION: Discovered issue occurred with MAX_QUEUE_SIZE=1000 outside
+Docker environment. Implemented proper Docker resource constraints
+stabilizing memory at 341MB.
+
+TOOLS USED: Docker, grep, Edit
+CONCEPTS: container-memory, resource-limits, queue-sizing
+FILES: docker-compose.yaml, batch_watcher.py
+```
+
+### Getting Started with Narratives
+
+Narratives are automatically generated for new conversations. To process existing conversations:
+
+```bash
+# Process all existing conversations in batch
+python docs/design/batch_import_all_projects.py
+
+# Monitor batch progress
+docker compose logs batch-monitor -f
+
+# Check completion status
+curl http://localhost:6333/collections/csr_claude-self-reflect_local_384d
+```
+
+For complete documentation, see [Batch Automation Guide](docs/testing/NARRATIVE_TESTING_SUMMARY.md).
+
 ## Key Features
 
 <details>
 <summary><b>MCP Tools Available to Claude</b></summary>
 
-**Search & Memory Tools:**
+**Search & Memory:**
 - `reflect_on_past` - Search past conversations using semantic similarity with time decay (supports quick/summary modes)
 - `store_reflection` - Store important insights or learnings for future reference
 - `get_next_results` - Paginate through additional search results
 - `search_by_file` - Find conversations that analyzed specific files
 - `search_by_concept` - Search for conversations about development concepts
-- `get_full_conversation` - Retrieve complete JSONL conversation files (v2.8.8)
+- `get_full_conversation` - Retrieve complete JSONL conversation files
 
-**NEW: Temporal Query Tools (v3.3.0):**
+**Temporal Queries:**
 - `get_recent_work` - Answer "What did we work on last?" with session grouping
 - `search_by_recency` - Time-constrained search like "docker issues last week"
 - `get_timeline` - Activity timeline with statistics and patterns
 
-**Runtime Configuration Tools (v4.0):**
+**Runtime Configuration:**
 - `switch_embedding_mode` - Switch between local/cloud modes without restart
 - `get_embedding_mode` - Check current embedding configuration
 - `reload_code` - Hot reload Python code changes
 - `reload_status` - Check reload state
 - `clear_module_cache` - Clear Python cache
 
-**Status & Monitoring Tools:**
+**Status & Monitoring:**
 - `get_status` - Real-time import progress and system status
 - `get_health` - Comprehensive system health check
 - `collection_status` - Check Qdrant collection health and stats
@@ -244,6 +325,64 @@ Claude: [Searches across ALL your projects]
 
 </details>
 
+<details>
+<summary><b>Ralph Loop Memory Integration (v7.1+)</b></summary>
+
+<div align="center">
+<img src="docs/images/ralph-loop-csr.png" alt="Ralph Loop with CSR Memory - From hamster wheel to upward spiral" width="800"/>
+</div>
+
+Use the [ralph-wiggum plugin](https://github.com/anthropics/claude-code-plugins/tree/main/ralph-wiggum) for long tasks? CSR automatically gives Ralph loops **cross-session memory**:
+
+**Core Features:**
+- **Automatic backup** before context compaction
+- **Past session retrieval** when starting new Ralph loops
+- **Failed approach tracking** - never repeat the same mistakes
+- **Success pattern learning** - reuse what worked before
+
+**v7.1+ Enhanced Features:**
+- **Error Signature Deduplication** - Normalizes errors (removes line numbers, paths, timestamps) to avoid redundant storage
+- **Output Decline Detection** - Circuit breaker pattern that detects >70% drop in output length
+- **Confidence-Based Exit** - 0-100 scoring based on signals (tasks complete, tests passing, no errors)
+- **Anti-Pattern Injection** - "DON'T RETRY THESE" section surfaces failed approaches first
+- **Work Type Tracking** - Categorizes sessions as IMPLEMENTATION/TESTING/DEBUGGING/DOCUMENTATION
+- **Error-Centric Search** - Finds past sessions by error pattern, not just task description
+
+**Setup (one-time):**
+```bash
+./scripts/ralph/install_hooks.sh        # Install CSR hooks
+./scripts/ralph/install_hooks.sh --check  # Verify installation
+```
+
+**How it works:**
+1. Start a Ralph loop: `/ralph-wiggum:ralph-loop "Build feature X"`
+2. Work naturally - state is tracked in `.claude/ralph-loop.local.md`
+3. When compaction occurs, state is backed up to CSR
+4. New sessions retrieve past learnings from CSR automatically
+5. Anti-patterns and winning strategies are surfaced first
+
+**Files created:**
+- `.ralph_past_sessions.md` - Injected context from past sessions (auto-generated)
+
+**Verified proof (2026-01-04):**
+```
+# Session start hook injects past sessions:
+INFO: Found 2 relevant results:
+  - Anti-patterns: 0
+  - Winning strategies: 0
+  - Similar tasks: 2
+
+# Sessions stored in Qdrant:
+{
+  "tags": ["ralph_session", "outcome_completed"],
+  "timestamp": "2026-01-04T18:13:03.711262+00:00"
+}
+```
+
+[Full documentation →](docs/development/ralph-memory-integration.md)
+
+</details>
+
 <details>
 <summary><b>Memory Decay</b></summary>
 
@@ -296,9 +435,6 @@ Files are categorized by age and processed with priority queuing to ensure newes
 
 ## Requirements
 
-> [!WARNING]
-> **Breaking Change in v4.0**: Collections now use prefixed naming (e.g., `csr_project_local_384d`). Run migration automatically via `npm update`.
-
 <details>
 <summary><b>System Requirements</b></summary>
 
@@ -376,20 +512,44 @@ npm uninstall -g claude-self-reflect
 
 ## Keeping Up to Date
 
-> [!TIP]
-> **For Existing Users**: Simply run `npm update -g claude-self-reflect` to get the latest features and improvements. Updates are automatic and preserve your data.
-
-<details>
-<summary>Recent Improvements</summary>
+```bash
+npm update -g claude-self-reflect
+```
 
-- **20x faster performance** - Status checks, search, and imports
-- **Runtime configuration** - Switch modes without restarting
-- **Unified state management** - Single source of truth
-- **AST-GREP integration** - Code quality analysis
-- **Temporal search tools** - Find recent work and time-based queries
-- **Auto-migration** - Updates handle breaking changes automatically
+Updates are automatic and preserve your data. See [full changelog](docs/release-history.md) for details.
 
-[Full changelog](docs/release-history.md)
+<details>
+<summary><b>Release Evolution</b></summary>
+
+### v7.0 - Automated Narratives (Oct 2025)
+- **9.3x better search quality** via AI-powered conversation summaries
+- **50% cost savings** using Anthropic Batch API ($0.012 per conversation)
+- **82% token compression** while maintaining searchability
+- Rich metadata extraction (tools, concepts, files)
+- Problem-solution narrative structure
+- Automated batch processing with Docker monitoring
+
+### v4.0 - Performance Revolution (Sep 2025)
+- **20x faster** status checks (119ms → 6ms)
+- **50% storage reduction** via unified state management
+- **10x faster imports** (10/sec → 100/sec)
+- **90% memory reduction** (500MB → 50MB)
+- Runtime mode switching (no restart required)
+- Prefixed collection naming (breaking change)
+- Code quality tracking with AST-GREP (100+ patterns)
+
+### v3.3 - Temporal Intelligence (Aug 2025)
+- Time-based search: "docker issues last week"
+- Session grouping: "What did we work on last?"
+- Activity timelines with statistics
+- Recency-aware queries
+
+### v2.8 - Full Context Access (Jul 2025)
+- Complete conversation retrieval
+- JSONL file access for deeper analysis
+- Enhanced debugging capabilities
+
+[View complete changelog →](docs/release-history.md)
 
 </details>