superlocalmemory 2.3.0

package/docs/V2.2.0-OPTIONAL-SEARCH.md

@@ -0,0 +1,666 @@
+# SuperLocalMemory V2.2.0 - Optional Search Components
+
+**Created by:** Varun Pratap Bhardwaj
+**Version:** 2.2.0
+**Date:** February 7, 2026
+
+---
+
+## Overview
+
+SuperLocalMemory V2.2.0 introduces two powerful optional search components that significantly enhance memory retrieval performance and accuracy:
+
+1. **HNSW Index** (`src/hnsw_index.py`) - Fast approximate nearest neighbor search
+2. **Embedding Engine** (`src/embedding_engine.py`) - Local semantic embedding generation
+
+Both components are **completely optional** with graceful fallback to existing TF-IDF methods if dependencies are not installed.
+
+---
+
+## Component #1: HNSW Index
+
+### What is HNSW?
+
+HNSW (Hierarchical Navigable Small World) is a state-of-the-art algorithm for approximate nearest neighbor search. It provides:
+
+- **Sub-10ms search** for 10,000 memories
+- **Sub-50ms search** for 100,000 memories
+- **Incremental updates** without full rebuild
+- **Disk persistence** for instant startup
+
+### Installation
+
+```bash
+# Optional - only install if you want HNSW acceleration
+pip install hnswlib
+```
+
+### Usage
+
+```python
+from hnsw_index import HNSWIndex
+import numpy as np
+
+# Initialize index
+index = HNSWIndex(dimension=384)  # Match your embedding dimension
+
+# Build from vectors
+vectors = np.random.randn(1000, 384).astype('float32')
+memory_ids = list(range(1000))
+index.build(vectors, memory_ids)
+
+# Search for similar vectors
+query = np.random.randn(384).astype('float32')
+results = index.search(query, k=5)
+# Returns: [(memory_id, similarity_score), ...]
+
+# Add single vector (incremental)
+new_vector = np.random.randn(384).astype('float32')
+index.add(new_vector, memory_id=1001)
+
+# Get statistics
+stats = index.get_stats()
+print(f"Indexed: {stats['indexed_count']} vectors")
+print(f"Method: {'HNSW' if stats['use_hnsw'] else 'Linear fallback'}")
+```
+
+### Fallback Behavior
+
+If `hnswlib` is not installed:
+- Automatically falls back to sklearn-based linear search
+- Performance: ~100ms for 10K vectors (vs <10ms with HNSW)
+- No feature degradation - all functionality remains available
+
+### CLI Commands
+
+```bash
+# Show index statistics
+python src/hnsw_index.py stats
+
+# Rebuild from database
+python src/hnsw_index.py rebuild
+
+# Run performance test
+python src/hnsw_index.py test
+```
+
+### Performance Benchmarks
+
+| Dataset Size | HNSW Search | Linear Search | Speedup |
+|--------------|-------------|---------------|---------|
+| 1K vectors   | <1ms        | ~10ms         | 10x     |
+| 10K vectors  | <10ms       | ~100ms        | 10x     |
+| 100K vectors | <50ms       | ~1000ms       | 20x     |
+
+### Configuration
+
+```python
+index = HNSWIndex(
+    dimension=384,              # Vector dimension
+    max_elements=100_000,       # Maximum vectors to index
+    m=16,                       # HNSW M parameter (connections)
+    ef_construction=200,        # Build-time quality
+    ef_search=50                # Search-time quality
+)
+```
+
+**Parameters explained:**
+- `m`: Higher = better accuracy, more memory (typical: 16)
+- `ef_construction`: Higher = better index quality, slower build (typical: 200)
+- `ef_search`: Higher = better accuracy, slower search (typical: 50)
+
+### Security & Limits
+
+```python
+MAX_MEMORIES_FOR_HNSW = 100_000    # Prevents memory exhaustion
+MAX_DIMENSION = 5000                # Typical: 384 for embeddings
+```
+
+---
+
+## Component #2: Embedding Engine
+
+### What is the Embedding Engine?
+
+The Embedding Engine generates semantic embeddings locally using sentence-transformers, enabling:
+
+- **True semantic search** (understands meaning, not just keywords)
+- **100% local processing** (no API calls, no cloud dependencies)
+- **GPU acceleration** (CUDA/Apple Silicon MPS support)
+- **Smart caching** (LRU cache for 10K embeddings)
+
+### Installation
+
+```bash
+# Optional - only install if you want semantic embeddings
+pip install sentence-transformers
+
+# For GPU acceleration (optional)
+pip install torch  # Will auto-detect CUDA/MPS
+```
+
+### Usage
+
+```python
+from embedding_engine import EmbeddingEngine
+
+# Initialize engine
+engine = EmbeddingEngine()  # Auto-detects GPU, downloads model
+
+# Generate single embedding
+text = "SuperLocalMemory is a local memory system for AI assistants"
+embedding = engine.encode(text)
+# Returns: numpy array of shape (384,)
+
+# Batch processing
+texts = [
+    "First memory about authentication",
+    "Second memory about database design",
+    "Third memory about API endpoints"
+]
+embeddings = engine.encode(texts, batch_size=32)
+# Returns: numpy array of shape (3, 384)
+
+# Compute similarity
+emb1 = engine.encode("JWT authentication")
+emb2 = engine.encode("Token-based auth")
+similarity = engine.similarity(emb1, emb2)
+# Returns: 0.85 (high similarity)
+
+# Get statistics
+stats = engine.get_stats()
+print(f"Device: {stats['device']}")  # cuda / mps / cpu
+print(f"Cache size: {stats['cache_size']}")
+```
+
+### Models Available
+
+| Model | Dimension | Size | Speed | Quality | Use Case |
+|-------|-----------|------|-------|---------|----------|
+| **all-MiniLM-L6-v2** (default) | 384 | 80MB | Fast | Good | General use |
+| all-mpnet-base-v2 | 768 | 420MB | Slower | Better | High accuracy |
+| paraphrase-multilingual | 384 | 420MB | Medium | Good | Multiple languages |
+
+```python
+# Use different model
+engine = EmbeddingEngine(model_name="all-mpnet-base-v2")
+```
+
+### Fallback Behavior
+
+If `sentence-transformers` is not installed:
+- Automatically falls back to TF-IDF vectorization
+- Performance: Still fast, but less semantic understanding
+- Dimension: 384 (padded/truncated to match)
+
+### CLI Commands
+
+```bash
+# Show engine statistics
+python src/embedding_engine.py stats
+
+# Generate embeddings for all memories
+python src/embedding_engine.py generate
+
+# Clear embedding cache
+python src/embedding_engine.py clear-cache
+
+# Run performance test
+python src/embedding_engine.py test
+```
+
+### Performance Benchmarks
+
+| Device | Speed (texts/sec) | GPU Memory |
+|--------|-------------------|------------|
+| CPU (Intel i7) | ~100 | N/A |
+| CUDA (RTX 3090) | ~1000 | ~2GB |
+| Apple M1/M2 (MPS) | ~500 | Unified |
+| Cache Hit | ~1,000,000 | N/A |
+
+### GPU Acceleration
+
+```python
+# Auto-detect (recommended)
+engine = EmbeddingEngine()  # Tries: CUDA → MPS → CPU
+
+# Force specific device
+engine = EmbeddingEngine(device='cuda')   # NVIDIA GPU
+engine = EmbeddingEngine(device='mps')    # Apple Silicon
+engine = EmbeddingEngine(device='cpu')    # CPU only
+```
+
+### Caching
+
+The engine uses an LRU cache for repeated queries:
+
+```python
+# First call: generates embedding (~10-100ms)
+emb1 = engine.encode("Same text")
+
+# Second call: cache hit (~0.001ms)
+emb2 = engine.encode("Same text")
+
+# Save cache to disk
+engine.save_cache()
+
+# Clear cache
+engine.clear_cache()
+```
+
+Cache settings:
+- **Max size:** 10,000 entries
+- **Eviction:** LRU (Least Recently Used)
+- **Persistence:** Saved to `~/.claude-memory/embedding_cache.json`
+
+### Security & Limits
+
+```python
+MAX_BATCH_SIZE = 128        # Prevents memory exhaustion
+MAX_TEXT_LENGTH = 10_000    # Characters per input
+CACHE_MAX_SIZE = 10_000     # Cache entries
+```
+
+---
+
+## Integration with MemoryStoreV2
+
+### Adding Embeddings to Database
+
+```python
+from embedding_engine import EmbeddingEngine
+from pathlib import Path
+
+engine = EmbeddingEngine()
+
+# Add embeddings to all memories
+db_path = Path.home() / ".claude-memory" / "memory.db"
+engine.add_to_database(db_path)
+```
+
+This will:
+1. Add `embedding` column to database (if not exists)
+2. Generate embeddings for all memories
+3. Store as JSON arrays in database
+4. Use batch processing for efficiency
+
+### Building HNSW Index from Database
+
+```python
+from hnsw_index import HNSWIndex
+from pathlib import Path
+
+index = HNSWIndex(dimension=384)
+
+# Build from database embeddings
+db_path = Path.home() / ".claude-memory" / "memory.db"
+index.rebuild_from_db(db_path, embedding_column='embedding')
+```
+
+### Complete Search Pipeline
+
+```python
+from memory_store_v2 import MemoryStoreV2
+from embedding_engine import EmbeddingEngine
+from hnsw_index import HNSWIndex
+
+# Initialize components
+store = MemoryStoreV2()
+engine = EmbeddingEngine()
+index = HNSWIndex(dimension=384)
+
+# Add memory with embedding
+content = "Fixed authentication bug using JWT tokens"
+mem_id = store.add_memory(content)
+
+# Generate and store embedding
+embedding = engine.encode(content)
+# Store embedding in database...
+
+# Add to HNSW index
+index.add(embedding, mem_id)
+
+# Search
+query = "auth token issue"
+query_embedding = engine.encode(query)
+similar_ids = index.search(query_embedding, k=5)
+
+# Get full memory details
+for mem_id, score in similar_ids:
+    memory = store.get_by_id(mem_id)
+    print(f"[{mem_id}] Score: {score:.3f}")
+    print(f"  {memory['content'][:100]}...")
+```
+
+---
+
+## Installation Guide
+
+### Minimal Installation (TF-IDF only)
+
+```bash
+# Already included in base requirements
+pip install scikit-learn numpy
+```
+
+### Standard Installation (Recommended)
+
+```bash
+# Add semantic search
+pip install sentence-transformers
+
+# This automatically installs:
+# - torch (PyTorch)
+# - transformers (Hugging Face)
+# - tqdm (progress bars)
+```
+
+### Full Installation (Maximum Performance)
+
+```bash
+# Standard + HNSW acceleration
+pip install sentence-transformers hnswlib
+
+# For GPU acceleration (if you have NVIDIA GPU)
+pip install torch --index-url https://download.pytorch.org/whl/cu118
+```
+
+### Apple Silicon (M1/M2/M3)
+
+```bash
+# MPS (Metal Performance Shaders) support included
+pip install sentence-transformers hnswlib
+
+# torch with MPS support (usually auto-installed)
+pip install torch torchvision torchaudio
+```
+
+---
+
+## Verification
+
+Check what's available:
+
+```bash
+# Test HNSW
+python src/hnsw_index.py stats
+
+# Test Embeddings
+python src/embedding_engine.py stats
+
+# Run comprehensive test
+python test_new_modules.py
+```
+
+---
+
+## Troubleshooting
+
+### Issue: "No module named 'hnswlib'"
+
+**Solution:**
+```bash
+pip install hnswlib
+```
+
+Or just use the fallback - the system will automatically use linear search.
+
+### Issue: "No module named 'sentence_transformers'"
+
+**Solution:**
+```bash
+pip install sentence-transformers
+```
+
+Or just use the fallback - the system will automatically use TF-IDF.
+
+### Issue: "CUDA out of memory"
+
+**Solution:**
+```python
+# Reduce batch size
+engine = EmbeddingEngine()
+embeddings = engine.encode(texts, batch_size=16)  # Default: 32
+
+# Or force CPU
+engine = EmbeddingEngine(device='cpu')
+```
+
+### Issue: "Model download is slow"
+
+**Solution:**
+Models are cached locally after first download:
+- Location: `~/.claude-memory/models/`
+- Size: 80MB (all-MiniLM-L6-v2)
+- Subsequent loads: <1 second
+
+### Issue: "Search is slow even with HNSW"
+
+**Possible causes:**
+1. Fallback mode active (check `index.get_stats()`)
+2. Need to adjust `ef_search` parameter
+3. Index not built yet
+
+**Solution:**
+```python
+# Rebuild index
+index.rebuild_from_db(db_path)
+
+# Increase search quality (slower but more accurate)
+index.ef_search = 100  # Default: 50
+```
+
+---
+
+## API Reference
+
+### HNSWIndex Class
+
+```python
+class HNSWIndex:
+    def __init__(
+        dimension: int = 384,
+        max_elements: int = 100_000,
+        m: int = 16,
+        ef_construction: int = 200,
+        ef_search: int = 50
+    )
+
+    def build(vectors: np.ndarray, memory_ids: List[int])
+    def add(vector: np.ndarray, memory_id: int)
+    def search(query_vector: np.ndarray, k: int = 5) -> List[Tuple[int, float]]
+    def update(memory_id: int, vector: np.ndarray)
+    def delete(memory_id: int)
+    def rebuild_from_db(db_path: Path, embedding_column: str = 'embedding')
+    def get_stats() -> Dict[str, Any]
+```
+
+### EmbeddingEngine Class
+
+```python
+class EmbeddingEngine:
+    def __init__(
+        model_name: str = "all-MiniLM-L6-v2",
+        device: Optional[str] = None,
+        use_cache: bool = True
+    )
+
+    def encode(texts: Union[str, List[str]], batch_size: int = 32) -> np.ndarray
+    def encode_batch(texts: List[str], batch_size: int = 32) -> np.ndarray
+    def similarity(embedding1: np.ndarray, embedding2: np.ndarray) -> float
+    def add_to_database(db_path: Path, embedding_column: str = 'embedding')
+    def save_cache()
+    def clear_cache()
+    def get_stats() -> Dict[str, Any]
+```
+
+---
+
+## Best Practices
+
+### 1. Generate Embeddings Once
+
+```python
+# Do this once after installation
+engine = EmbeddingEngine()
+engine.add_to_database(db_path)
+```
+
+### 2. Rebuild HNSW Periodically
+
+```python
+# After adding many new memories
+index = HNSWIndex()
+index.rebuild_from_db(db_path)
+```
+
+### 3. Use Batch Processing
+
+```python
+# Good: process in batches
+embeddings = engine.encode(texts, batch_size=32)
+
+# Bad: process one at a time (slow)
+embeddings = [engine.encode(text) for text in texts]
+```
+
+### 4. Monitor Cache Hit Rate
+
+```python
+stats = engine.get_stats()
+print(f"Cache size: {stats['cache_size']}/{stats['cache_max_size']}")
+
+# Save cache periodically
+engine.save_cache()
+```
+
+### 5. Choose Right Model for Use Case
+
+- **General use:** all-MiniLM-L6-v2 (default, fast)
+- **High accuracy:** all-mpnet-base-v2 (slower, better)
+- **Multilingual:** paraphrase-multilingual
+
+---
+
+## Performance Tuning
+
+### HNSW Parameters
+
+**For speed priority:**
+```python
+index = HNSWIndex(
+    m=8,                # Fewer connections
+    ef_construction=100,  # Faster build
+    ef_search=25        # Faster search
+)
+```
+
+**For accuracy priority:**
+```python
+index = HNSWIndex(
+    m=32,               # More connections
+    ef_construction=400,  # Better build
+    ef_search=100       # More thorough search
+)
+```
+
+**Balanced (recommended):**
+```python
+index = HNSWIndex(
+    m=16,
+    ef_construction=200,
+    ef_search=50
+)
+```
+
+### Embedding Batch Sizes
+
+| Use Case | Batch Size | Memory | Speed |
+|----------|------------|--------|-------|
+| Interactive (real-time) | 1-8 | Low | Fast response |
+| Standard | 32 | Medium | Balanced |
+| Bulk processing | 64-128 | High | Maximum throughput |
+
+---
+
+## Architecture Diagram
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│  SuperLocalMemory V2.2.0 Search Architecture                │
+├─────────────────────────────────────────────────────────────┤
+│                                                               │
+│  User Query: "authentication token bug"                      │
+│         │                                                     │
+│         ├──> [1] EmbeddingEngine.encode(query)              │
+│         │        ├─> Cache check (0.001ms if hit)           │
+│         │        └─> Model inference (10-100ms if miss)     │
+│         │                                                     │
+│         ├──> [2] HNSWIndex.search(embedding, k=5)           │
+│         │        ├─> HNSW search (<10ms for 10K vectors)    │
+│         │        └─> Linear fallback (~100ms if no HNSW)    │
+│         │                                                     │
+│         └──> [3] MemoryStoreV2.get_by_id(memory_ids)        │
+│                  └─> SQLite lookup (<1ms per memory)        │
+│                                                               │
+│  Total latency: 10-150ms (depending on cache/GPU/HNSW)      │
+│                                                               │
+│  Fallback path (no optional deps):                           │
+│  MemoryStoreV2.search() -> TF-IDF -> Full-text search       │
+│  Total latency: ~100ms (still acceptable)                    │
+└─────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Version History
+
+### V2.2.0 (February 7, 2026)
+
+**Added:**
+- `src/hnsw_index.py` - Fast approximate nearest neighbor search
+- `src/embedding_engine.py` - Local semantic embedding generation
+- Complete test suite (`test_new_modules.py`)
+- Comprehensive documentation (this file)
+
+**Features:**
+- Optional dependencies with graceful fallback
+- Sub-10ms search for 10K memories (with HNSW)
+- GPU acceleration (CUDA/MPS)
+- LRU caching for embeddings
+- Disk persistence for index
+- Security limits and input validation
+- CLI interfaces for both components
+
+**Performance:**
+- 10-20x speedup with HNSW vs linear search
+- 100-1000 texts/sec embedding generation (GPU)
+- <1ms cache hit latency
+
+---
+
+## Credits
+
+**Architecture & Implementation:** Varun Pratap Bhardwaj (Solution Architect)
+
+**Technologies:**
+- hnswlib: Fast approximate nearest neighbor search
+- sentence-transformers: Local embedding generation
+- scikit-learn: TF-IDF fallback
+- PyTorch: GPU acceleration
+
+**License:** MIT License
+
+---
+
+## Support
+
+For issues, questions, or contributions:
+- **GitHub Issues:** https://github.com/varun369/SuperLocalMemoryV2/issues
+- **Documentation:** https://github.com/varun369/SuperLocalMemoryV2/wiki
+
+---
+
+**SuperLocalMemory V2.2.0** - Making AI memory faster and smarter, locally.
+
+*Created by Varun Pratap Bhardwaj, February 2026*

package/docs/WINDOWS-INSTALL-README.txt

@@ -0,0 +1,34 @@
+SuperLocalMemory V2.1.0 - Windows Installation
+==============================================
+
+Thank you for installing SuperLocalMemory V2!
+
+This installer will:
+1. Copy all necessary files to your system
+2. Install Python modules to %USERPROFILE%\.claude-memory\
+3. Configure MCP integration for supported IDEs
+4. Install universal skills for AI tools
+5. Add 'slm' command to your system PATH
+
+System Requirements:
+--------------------
+• Windows 10 or higher (64-bit)
+• Python 3.8 or higher
+• 100 MB disk space
+• Internet connection (for Python packages)
+
+After Installation:
+-------------------
+1. Open Command Prompt or PowerShell
+2. Run: slm status
+3. Test: slm remember "Test memory"
+4. Search: slm recall "test"
+
+Documentation:
+--------------
+• GitHub: https://github.com/varun369/SuperLocalMemoryV2
+• Wiki: https://github.com/varun369/SuperLocalMemoryV2/wiki
+• Issues: https://github.com/varun369/SuperLocalMemoryV2/issues
+
+Copyright (c) 2026 Varun Pratap Bhardwaj
+Licensed under MIT License