superlocalmemory 3.4.17 → 3.4.19

package/docs/v2-archive/SEARCH-ENGINE-V2.2.0.md

@@ -1,749 +0,0 @@
-# SuperLocalMemory V2.2.0 - Search Engine Documentation
-
-**Created by:** [Varun Pratap Bhardwaj](https://github.com/varun369) (Solution Architect)
-**Version:** 2.2.0
-**Release Date:** 2026-02-07
-
----
-
-## Overview
-
-SuperLocalMemory V2.2.0 introduces a professional-grade search engine with four integrated components:
-
-1. **BM25 Search Engine** - Industry-standard keyword ranking
-2. **Query Optimizer** - Spell correction and query expansion
-3. **Cache Manager** - LRU caching for performance
-4. **Hybrid Search System** - Multi-method retrieval fusion
-
-### Why This Matters
-
-Previous versions relied on basic SQLite FTS and TF-IDF. V2.2.0 brings:
-
-- **3x faster search** - BM25 optimized for <30ms on 1K memories
-- **Better relevance** - BM25 outperforms TF-IDF by 15-20% in precision
-- **Query intelligence** - Auto-corrects typos, expands terms
-- **Multi-method fusion** - Combines keyword, semantic, and graph search
-- **Production-ready caching** - 30-50% cache hit rates reduce load
-
----
-
-## Architecture
-
-```
-┌─────────────────────────────────────────────────────────────┐
-│  HYBRID SEARCH ENGINE (hybrid_search.py)                    │
-│  ┌──────────────┐  ┌──────────────┐  ┌─────────────┐       │
-│  │   BM25       │  │   Semantic   │  │   Graph     │       │
-│  │   Search     │  │   (TF-IDF)   │  │   Traversal │       │
-│  └──────────────┘  └──────────────┘  └─────────────┘       │
-│         │                  │                  │             │
-│         └──────────────────┴──────────────────┘             │
-│                           │                                 │
-│                    Weighted Fusion                          │
-│                    (RRF / Scores)                           │
-└─────────────────────────────────────────────────────────────┘
-                           │
-         ┌─────────────────┼─────────────────┐
-         │                 │                 │
-┌────────▼────────┐ ┌──────▼──────┐ ┌───────▼────────┐
-│ Query Optimizer │ │   Cache     │ │  Memory Store  │
-│  - Spell Check  │ │   Manager   │ │     (DB)       │
-│  - Expansion    │ │   (LRU)     │ │                │
-└─────────────────┘ └─────────────┘ └────────────────┘
-```
-
----
-
-## Components
-
-### 1. BM25 Search Engine (`search_engine_v2.py`)
-
-**Pure Python implementation of Okapi BM25 ranking function.**
-
-#### Algorithm
-
-BM25 (Best Match 25) is the gold standard for keyword search, used by:
-- Elasticsearch (default ranking)
-- Lucene/Solr
-- Apache Lucene
-- Microsoft Bing
-
-**Formula:**
-```
-score(D,Q) = Σ IDF(qi) × (f(qi,D) × (k1 + 1)) / (f(qi,D) + k1 × (1 - b + b × |D| / avgdl))
-```
-
-Where:
-- `f(qi,D)` = term frequency in document
-- `|D|` = document length
-- `avgdl` = average document length
-- `k1` = term saturation (default: 1.5)
-- `b` = length normalization (default: 0.75)
-- `IDF(qi)` = inverse document frequency
-
-#### Key Features
-
-- **No Dependencies** - Pure Python, no external libraries
-- **Fast Indexing** - O(n × m) where n=docs, m=avg_tokens
-- **Fast Search** - O(q × p) where q=query_terms, p=postings
-- **Memory Efficient** - Inverted index with compressed postings
-
-#### Performance
-
-| Metric | Target | Typical |
-|--------|--------|---------|
-| Index 1K docs | <500ms | 250ms |
-| Search 1K docs | <30ms | 15-25ms |
-| Memory overhead | <50MB | 30-40MB |
-
-#### Usage
-
-```python
-from search_engine_v2 import BM25SearchEngine
-
-# Initialize
-engine = BM25SearchEngine(k1=1.5, b=0.75)
-
-# Index documents
-documents = ["Python web development", "JavaScript frontend", ...]
-doc_ids = [1, 2, ...]
-engine.index_documents(documents, doc_ids)
-
-# Search
-results = engine.search("Python web", limit=10)
-# Returns: [(doc_id, score), ...]
-
-# Get statistics
-stats = engine.get_stats()
-print(f"Indexed {stats['num_documents']} documents")
-print(f"Vocabulary: {stats['vocabulary_size']} terms")
-```
-
-#### Parameter Tuning
-
-**k1 (Term Frequency Saturation)**
-- Lower (1.2): Better for short documents
-- Higher (2.0): Better for long documents
-- Default (1.5): Balanced for most use cases
-
-**b (Length Normalization)**
-- 0.0: No normalization (good for uniform length docs)
-- 0.5: Moderate normalization
-- 0.75: Standard normalization (default)
-- 1.0: Full normalization (aggressive for long docs)
-
----
-
-### 2. Query Optimizer (`query_optimizer.py`)
-
-**Enhances queries with spell correction and expansion.**
-
-#### Features
-
-**1. Spell Correction**
-- Edit distance (Levenshtein) algorithm
-- Vocabulary-based correction
-- Technical term preservation (API, SQL, JWT, etc.)
-- Max distance: 2 edits
-
-**2. Query Expansion**
-- Co-occurrence based expansion
-- Adds related terms to broaden search
-- Configurable expansion count
-- Minimum co-occurrence threshold
-
-**3. Boolean Operators**
-- AND: `term1 AND term2` (both required)
-- OR: `term1 OR term2` (either required)
-- NOT: `term1 NOT term2` (exclude term2)
-- Phrase: `"exact phrase"` (exact match)
-
-#### Usage
-
-```python
-from query_optimizer import QueryOptimizer
-
-# Initialize with vocabulary
-vocabulary = {'python', 'javascript', 'web', 'development', ...}
-optimizer = QueryOptimizer(vocabulary)
-
-# Build co-occurrence for expansion
-documents = [
-    ['python', 'web', 'development'],
-    ['javascript', 'frontend', 'web'],
-    ...
-]
-optimizer.build_cooccurrence_matrix(documents)
-
-# Spell correction
-corrected = optimizer.spell_correct("pythno")  # → "python"
-
-# Query expansion
-expanded = optimizer.expand_query(['python'], max_expansions=2)
-# Returns: ['python', 'web', 'development']
-
-# Full optimization
-optimized = optimizer.optimize(
-    "pythno web devlopment",
-    enable_spell_correction=True,
-    enable_expansion=False
-)
-# Returns: "python web development"
-
-# Boolean parsing
-parsed = optimizer.parse_boolean_query('python AND (web OR api)')
-```
-
-#### Spell Correction Algorithm
-
-Uses Levenshtein distance with optimizations:
-
-1. **Quick filters:**
-   - Length difference > max_distance → skip
-   - Term in vocabulary → return as-is
-   - Technical terms (≤3 chars) → preserve
-
-2. **Approximate matching:**
-   - Uses `difflib.get_close_matches()` for candidates
-   - Validates with full edit distance
-   - Returns best match within threshold
-
-3. **Performance:**
-   - O(k × m × n) where k=candidates, m,n=string lengths
-   - Typical: <5ms per query
-
----
-
-### 3. Cache Manager (`cache_manager.py`)
-
-**LRU cache for search results with TTL support.**
-
-#### Features
-
-- **LRU Eviction** - Least Recently Used policy
-- **TTL Support** - Time-to-live for cache entries
-- **Thread-Safe** - Optional locking for concurrent access
-- **Size-Based** - Maximum entry count
-- **Analytics** - Hit rate, access counts, eviction tracking
-
-#### Usage
-
-```python
-from cache_manager import CacheManager
-
-# Initialize
-cache = CacheManager(
-    max_size=100,          # Max 100 cached queries
-    ttl_seconds=300,       # 5 minute TTL
-    thread_safe=False      # Single-threaded
-)
-
-# Cache operations
-result = cache.get("python web")
-if result is None:
-    # Cache miss - perform search
-    result = search_engine.search("python web")
-    cache.put("python web", result)
-
-# Statistics
-stats = cache.get_stats()
-print(f"Hit rate: {stats['hit_rate']*100:.1f}%")
-print(f"Evictions: {stats['evictions']}")
-
-# Manual eviction
-cache.evict_expired()  # Remove expired entries
-cache.clear()          # Clear all entries
-```
-
-#### Cache Key Generation
-
-Keys are generated from query + parameters:
-```python
-key = hash(json.dumps({
-    'query': query,
-    'limit': limit,
-    'method': method,
-    # ... other parameters
-}))
-```
-
-This ensures different parameter combinations get separate cache entries.
-
-#### Performance Impact
-
-| Operation | Time | Description |
-|-----------|------|-------------|
-| Cache hit | ~0.1ms | Dictionary lookup |
-| Cache miss | Search time + 0.1ms | Standard search + cache store |
-| Eviction | ~0.01ms | OrderedDict pop |
-
-**Expected hit rates:**
-- Repeated queries: 80-90%
-- Similar queries: 10-20%
-- Overall typical: 30-50%
-
----
-
-### 4. Hybrid Search System (`hybrid_search.py`)
-
-**Multi-method retrieval with score fusion.**
-
-#### Fusion Methods
-
-**1. Weighted Score Fusion**
-- Normalizes scores from each method
-- Combines with configurable weights
-- Best for balanced results
-
-**2. Reciprocal Rank Fusion (RRF)**
-- Rank-based combination
-- Robust to score magnitude differences
-- Standard: `score = Σ 1/(k + rank)` where k=60
-
-**3. Single Method**
-- BM25 only
-- Semantic only
-- Graph only
-
-#### Default Weights
-
-```python
-weights = {
-    'bm25': 0.4,      # 40% - Best for keyword queries
-    'semantic': 0.3,  # 30% - Best for natural language
-    'graph': 0.3      # 30% - Best for conceptual queries
-}
-```
-
-#### Usage
-
-```python
-from hybrid_search import HybridSearchEngine
-from pathlib import Path
-
-# Initialize
-db_path = Path.home() / ".claude-memory" / "memory.db"
-hybrid = HybridSearchEngine(db_path, enable_cache=True)
-
-# BM25 only
-results = hybrid.search("Python web", method="bm25", limit=10)
-
-# Hybrid with default weights
-results = hybrid.search("Python web", method="hybrid", limit=10)
-
-# Custom weights
-results = hybrid.search(
-    query="Python web",
-    method="weighted",
-    weights={'bm25': 0.6, 'semantic': 0.4, 'graph': 0.0},
-    limit=10
-)
-
-# Reciprocal Rank Fusion
-results = hybrid.search("Python web", method="rrf", limit=10)
-
-# Statistics
-stats = hybrid.get_stats()
-print(f"Search time: {stats['last_search_time_ms']:.2f}ms")
-print(f"Fusion time: {stats['last_fusion_time_ms']:.2f}ms")
-print(f"Cache hit rate: {stats['cache']['hit_rate']*100:.1f}%")
-```
-
-#### Result Format
-
-```python
-[
-    {
-        'id': 1,
-        'content': 'Memory content...',
-        'summary': 'Summary...',
-        'score': 0.87,
-        'match_type': 'hybrid',
-        'category': 'development',
-        'tags': ['python', 'web'],
-        # ... other memory fields
-    },
-    ...
-]
-```
-
-#### Weight Tuning Guide
-
-**For keyword queries** (exact terms):
-```python
-weights = {'bm25': 0.7, 'semantic': 0.3, 'graph': 0.0}
-```
-
-**For conceptual queries** (themes):
-```python
-weights = {'bm25': 0.2, 'semantic': 0.3, 'graph': 0.5}
-```
-
-**For balanced queries** (mixed):
-```python
-weights = {'bm25': 0.4, 'semantic': 0.3, 'graph': 0.3}  # Default
-```
-
----
-
-## Performance Benchmarks
-
-### Test Environment
-- MacBook Pro M1 (2021)
-- Python 3.11
-- 1,000 test memories
-- Average memory size: 200 tokens
-
-### Results
-
-| Component | Metric | Target | Actual | Status |
-|-----------|--------|--------|--------|--------|
-| BM25 | Index 1K docs | <500ms | 247ms | ✅ |
-| BM25 | Search 1K docs | <30ms | 18ms | ✅ |
-| Query Optimizer | Spell check | <5ms | 2ms | ✅ |
-| Cache Manager | Get/Put | <0.5ms | 0.12ms | ✅ |
-| Hybrid Search | Combined | <50ms | 35ms | ✅ |
-
-### Scalability
-
-| Documents | BM25 Index | BM25 Search | Hybrid Search |
-|-----------|------------|-------------|---------------|
-| 100 | 25ms | 3ms | 8ms |
-| 500 | 120ms | 10ms | 20ms |
-| 1,000 | 247ms | 18ms | 35ms |
-| 5,000 | 1,200ms | 45ms | 95ms |
-| 10,000 | 2,400ms | 80ms | 180ms |
-
-**Notes:**
-- Index time is one-time cost
-- Search time scales sub-linearly (inverted index efficiency)
-- Hybrid search includes fusion overhead (~10-15ms)
-- These are projected estimates for the optional BM25 engine. See wiki Performance Benchmarks for measured end-to-end search latency.
-
----
-
-## Integration with Memory Store V2
-
-### Automatic Integration
-
-Hybrid search automatically integrates with `MemoryStoreV2`:
-
-```python
-from memory_store_v2 import MemoryStoreV2
-from hybrid_search import HybridSearchEngine
-
-# Initialize
-store = MemoryStoreV2()
-hybrid = HybridSearchEngine(store.db_path)
-
-# Add memories (automatically indexed)
-store.add_memory("Python web development", tags=['python', 'web'])
-
-# Search
-results = hybrid.search("Python", limit=5)
-```
-
-### Backward Compatibility
-
-V2.2.0 maintains full backward compatibility:
-
-```python
-# Old API still works
-results = store.search("Python web", limit=5)
-
-# New API available
-results = hybrid.search("Python web", method="hybrid", limit=5)
-```
-
----
-
-## Installation
-
-### Basic (BM25 + Hybrid)
-
-```bash
-pip install scikit-learn numpy
-```
-
-### Full (All features)
-
-```bash
-pip install -r requirements-search.txt
-```
-
-This includes:
-- scikit-learn (TF-IDF)
-- numpy (numerical computing)
-- sentence-transformers (optional embeddings)
-- hnswlib (optional fast search)
-
----
-
-## Testing
-
-### Run Test Suite
-
-```bash
-python test_search_engine.py
-```
-
-### Expected Output
-
-```
-============================================================
-SuperLocalMemory V2.2.0 - Search Engine Test Suite
-============================================================
-
-✓ PASS: BM25 Basic Functionality
-  → Indexed 4 docs, search returned 2 results
-✓ PASS: BM25 Performance
-  → Search: 18.45ms, Index: 247.32ms (1K docs)
-✓ PASS: Query Optimizer
-  → Spell correction and expansion working correctly
-✓ PASS: Cache Manager
-  → LRU eviction and stats working (hit rate: 33%)
-✓ PASS: Cache TTL
-  → Time-to-live expiration working correctly
-✓ PASS: Search Quality
-  → Relevance ranking correct, top score: 0.873
-✓ PASS: Hybrid Search Integration
-  → All methods working, 35.21ms search time
-✓ PASS: Weighted Fusion
-  → Multiple weight configurations working correctly
-
-============================================================
-TEST SUMMARY
-============================================================
-PASSED:   8
-FAILED:   0
-WARNINGS: 0
-
-✅ All tests passed!
-
-Search Engine V2.2.0 Components:
-  ✓ BM25 Search Engine
-  ✓ Query Optimizer
-  ✓ Cache Manager
-  ✓ Hybrid Search System
-
-Performance Targets:
-  ✓ BM25: <30ms for 1K memories
-  ✓ Hybrid: <50ms for 1K memories
-
-🎉 Ready for production!
-```
-
----
-
-## CLI Usage
-
-### BM25 Search Engine
-
-```bash
-python src/search_engine_v2.py
-```
-
-Output:
-```
-BM25 Search Engine - Demo
-============================================================
-
-Indexing 6 documents...
-✓ Indexed in 3.21ms
-  Vocabulary: 42 unique terms
-  Avg doc length: 8.5 tokens
-
-============================================================
-Search Results:
-============================================================
-
-Query: 'Python programming'
-  Found: 3 results in 1.23ms
-  Query terms: ['python', 'programming']
-    [0.873] doc_0: Python is a high-level programming language...
-    [0.542] doc_2: Machine learning uses Python libraries...
-    [0.234] doc_4: Django is a Python web framework...
-```
-
-### Query Optimizer
-
-```bash
-python src/query_optimizer.py
-```
-
-### Cache Manager
-
-```bash
-python src/cache_manager.py
-```
-
-### Hybrid Search
-
-```bash
-python src/hybrid_search.py "Python web development"
-```
-
----
-
-## Migration from V2.1.0
-
-No migration needed! V2.2.0 is backward compatible.
-
-### Changes
-
-1. **New components** (optional):
-   - `search_engine_v2.py` - BM25 engine
-   - `query_optimizer.py` - Query enhancement
-   - `cache_manager.py` - Result caching
-   - `hybrid_search.py` - Multi-method search
-
-2. **Existing behavior preserved**:
-   - `MemoryStoreV2.search()` still works
-   - Database schema unchanged
-   - API unchanged
-
-### Upgrade Path
-
-**Option 1: Use old API (no changes)**
-```python
-# Works exactly as before
-store = MemoryStoreV2()
-results = store.search("Python web")
-```
-
-**Option 2: Use new hybrid search (recommended)**
-```python
-# Better results, faster search
-hybrid = HybridSearchEngine(store.db_path)
-results = hybrid.search("Python web", method="hybrid")
-```
-
----
-
-## Troubleshooting
-
-### Issue: "scikit-learn not found"
-
-**Solution:**
-```bash
-pip install scikit-learn numpy
-```
-
-### Issue: Search is slow (>50ms)
-
-**Causes:**
-1. Large database (>10K memories)
-2. Complex queries
-3. Cold cache
-
-**Solutions:**
-1. Use BM25 only: `method="bm25"`
-2. Reduce limit: `limit=10` instead of 50
-3. Enable caching: `enable_cache=True`
-
-### Issue: Poor relevance
-
-**Solutions:**
-1. Try hybrid search: `method="hybrid"`
-2. Adjust weights: `weights={'bm25': 0.6, ...}`
-3. Use query expansion: `optimizer.optimize(..., enable_expansion=True)`
-
-### Issue: High memory usage
-
-**Causes:**
-1. Large vocabulary (>100K terms)
-2. Cache too large
-
-**Solutions:**
-1. Reduce BM25 `max_features` (not exposed by default)
-2. Reduce cache size: `CacheManager(max_size=50)`
-
----
-
-## Advanced Topics
-
-### Custom BM25 Parameters
-
-```python
-# For short documents (tweets, logs)
-engine = BM25SearchEngine(k1=1.2, b=0.0)
-
-# For long documents (articles, docs)
-engine = BM25SearchEngine(k1=2.0, b=1.0)
-```
-
-### Custom Fusion Weights
-
-```python
-# Keyword-heavy queries
-results = hybrid.search(
-    "Python FastAPI REST API",
-    weights={'bm25': 0.8, 'semantic': 0.2, 'graph': 0.0}
-)
-
-# Conceptual queries
-results = hybrid.search(
-    "how to optimize performance",
-    weights={'bm25': 0.2, 'semantic': 0.4, 'graph': 0.4}
-)
-```
-
-### Cache Configuration
-
-```python
-# High-traffic scenarios
-cache = CacheManager(
-    max_size=1000,        # Large cache
-    ttl_seconds=600,      # 10 minute TTL
-    thread_safe=True      # Enable locking
-)
-
-# Memory-constrained scenarios
-cache = CacheManager(
-    max_size=50,          # Small cache
-    ttl_seconds=60,       # 1 minute TTL
-    thread_safe=False     # No locking overhead
-)
-```
-
----
-
-## Roadmap
-
-### V2.2.1 (Planned)
-- Query suggestions
-- Fuzzy matching
-- Phrase boosting
-
-### V2.3.0 (Future)
-- Embedding-based search
-- Neural reranking
-- Cross-encoder scoring
-
----
-
-## Credits
-
-**Created by:** Varun Pratap Bhardwaj
-**Role:** Solution Architect & Original Creator
-**GitHub:** [@varun369](https://github.com/varun369)
-
-### Research Papers
-
-1. **BM25:** Robertson & Zaragoza (2009) - "The Probabilistic Relevance Framework: BM25 and Beyond"
-2. **RRF:** Cormack et al. (2009) - "Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods"
-3. **Query Expansion:** Carpineto & Romano (2012) - "A Survey of Automatic Query Expansion in Information Retrieval"
-
----
-
-## License
-
-AGPL-3.0 - See [LICENSE](../LICENSE) file
-
-**Attribution Required:** This notice must be preserved in all copies per AGPL-3.0 terms.
-
----
-
-**Project:** [SuperLocalMemory V2](https://github.com/qualixar/superlocalmemory)
-**Documentation:** [Full Docs](https://github.com/qualixar/superlocalmemory/wiki)
-**Issues:** [Report Issues](https://github.com/qualixar/superlocalmemory/issues)