@ngao/search 0.1.0 → 0.1.2

package/docs/reference/06_RESEARCH_SUMMARY.md

@@ -0,0 +1,519 @@
+# NGAO Search: Research Summary & Key Insights
+
+## Executive Summary
+
+NGAO Search is a **local document/code search system** designed to help developers find, understand, and analyze code and documentation efficiently. The system is optimized for **LLM consumption**, meaning results are structured and formatted specifically for AI models to process, enabling advanced use cases like automated documentation generation, code analysis, and onboarding assistance.
+
+### Key Differentiators
+
+1. **Format-Aware Indexing**: Different file types (Python, Markdown, JS) return different result structures matching their semantics
+2. **LLM-Optimized Output**: All results follow a structured JSON schema designed for AI consumption
+3. **Multi-Level Ranking**: Relevance scoring considers keyword matching, position, scope specificity, recency, and frequency
+4. **Context-Aware Results**: Returns snippets with surrounding context and scope hierarchy
+5. **Semantic Structure**: Extracts and indexes meaningful blocks (functions, classes, sections) with scope hierarchy
+
+---
+
+## Core Architecture
+
+### Three Main Pipelines
+
+```
+INDEXING PIPELINE:
+Files → Parse → Extract Blocks → Multi-Index Ingestion → Persistent Storage
+
+QUERY PIPELINE:
+Query → Parse → Multi-Index Search → Aggregate → Extract Context → Rank → Format
+
+RESULT PIPELINE:
+Ranking → LLM-Optimized Formatting → Output (JSON, CLI, Web API)
+```
+
+### Key Components
+
+| Component | Purpose | Input | Output |
+|-----------|---------|-------|--------|
+| **FileRouter** | Route files to appropriate parser | File path | Parser instance |
+| **Parsers** | Extract structured blocks | Raw file content | List of Block objects |
+| **BlockExtractor** | Normalize blocks and compute metadata | Parsed AST | Normalized Block objects |
+| **IndexBuilder** | Build multiple specialized indexes | Normalized blocks | Inverted, scope, registry indexes |
+| **MultiIndexSearcher** | Search across all indexes | Query terms + filters | Matching blocks with scores |
+| **RankingEngine** | Compute relevance scores | Blocks + query metadata | Ranked results |
+| **ContextExtractor** | Extract surrounding context | Block + source file | Enriched snippet |
+| **ResultFormatter** | Format results for output | Enriched results | LLM-friendly JSON |
+
+---
+
+## Data Flow Step-by-Step
+
+### Indexing Dataflow
+
+```
+1. DISCOVERY: File watcher detects new/modified file
+2. CHANGE DETECTION: Compute hash, compare with stored hash
+3. PARSING: Route to format-specific parser
+4. BLOCK EXTRACTION: Extract functions/classes/sections with scope
+5. INDEXING: Add to inverted, scope, and block registry indexes
+6. PERSISTENCE: Save indexes to SQLite/JSON storage
+```
+
+### Query Dataflow
+
+```
+1. INPUT: User enters "authentication handler"
+2. PARSING: Tokenize → ["auth", "handler"], extract filters
+3. SEARCH: Query inverted index for each term
+4. AGGREGATION: Combine results, deduplicate
+5. CONTEXT: Load source files, extract snippets ±context lines
+6. RANKING: Compute multi-factor relevance score
+7. FORMATTING: Transform to LLM-friendly JSON schema
+8. OUTPUT: Return ranked, formatted results
+```
+
+---
+
+## Index Structure
+
+The system maintains **4 specialized indexes**:
+
+### 1. Inverted Index (70% of storage)
+Maps keywords to source locations:
+```json
+{
+  "authentication": {
+    "files": ["src/auth/handler.py", "docs/auth.md"],
+    "positions": [
+      {
+        "file": "src/auth/handler.py",
+        "block_id": "func_42",
+        "occurrences": [{"line": 45}, {"line": 50}]
+      }
+    ]
+  }
+}
+```
+
+### 2. Scope Hierarchy Index (20% of storage)
+Maps file structure for quick navigation:
+```json
+{
+  "src/auth/handler.py": {
+    "module": { "children": ["class:AuthHandler"] },
+    "class:AuthHandler": { "children": ["method:handle", "method:validate"] }
+  }
+}
+```
+
+### 3. Block Registry (10% of storage)
+Quick lookup for block metadata:
+```json
+{
+  "func_42": {
+    "file": "src/auth/handler.py",
+    "type": "method",
+    "location": {"start": 45, "end": 78},
+    "scope": ["module", "class:AuthHandler"]
+  }
+}
+```
+
+### 4. Semantic Index (Optional, 10-20% additional)
+Dense vector embeddings for similarity search.
+
+---
+
+## Format-Specific Parsing
+
+### Python Files
+Extracts via AST traversal:
+- **Blocks**: Functions, classes, methods, variables
+- **Metadata**: Decorators, type hints, docstrings, signatures
+- **Scope**: Module → Class → Method hierarchy
+
+### Markdown Files
+Extracts via heading hierarchy:
+- **Blocks**: Heading levels (H1-H6), sections, paragraphs, code blocks
+- **Metadata**: Heading hierarchy, language for code blocks
+- **Scope**: Parent heading chain
+
+### JavaScript/TypeScript
+Extracts via Babel/tree-sitter parser:
+- **Blocks**: Functions, classes, components, hooks, exports
+- **Metadata**: JSDoc, type definitions, export types
+- **Scope**: Module → Class → Method hierarchy
+
+### JSON/YAML
+Extracts via flattened key paths:
+- **Blocks**: Key-value pairs, objects, arrays
+- **Metadata**: Nesting level, value types
+- **Scope**: Key path hierarchy (e.g., database.connection.pool_size)
+
+---
+
+## Ranking Algorithm
+
+Results ranked by weighted combination of 5 factors:
+
+```
+Final Score = (
+  0.35 × Keyword Match Score +        # How well query terms match
+  0.25 × Position Score +              # Where in block was found (name vs body)
+  0.15 × Scope Specificity +           # Nested scopes score higher
+  0.15 × Recency Score +               # Recently modified files score higher
+  0.10 × Frequency Score               # How many times keywords appear
+)
+```
+
+### Scoring Details
+
+**Keyword Match (0.35)**
+- Exact phrase: 1.0
+- All terms match: 0.8
+- Partial (X/Y terms): 0.6 × (matched/total)
+- Fuzzy match: 0.2
+
+**Position (0.25)**
+- In identifier/name: 1.0
+- In docstring/comment: 0.9
+- In code body: 0.5
+- In nested scope: 0.3
+
+**Scope Specificity (0.15)**
+- Method/deep nested: 0.8
+- Class-level: 0.7
+- Module-level: 0.5
+
+**Recency (0.15)**
+- Last 24 hours: 1.0
+- Last week: 0.8
+- Last month: 0.6
+- Older: 0.4
+
+**Frequency (0.10)**
+- First occurrence: 1.0
+- Multiple mentions: +0.1 per mention (capped at 0.3)
+
+---
+
+## LLM-Optimized Output Format
+
+Every result follows this universal schema:
+
+```json
+{
+  "rank": 1,
+  "relevance_score": 0.92,
+  
+  "file": {
+    "path": "src/auth/handler.py",
+    "type": "python"
+  },
+  
+  "location": {
+    "start_line": 45,
+    "end_line": 78,
+    "file_tree": ["src/", "auth/", "handler.py"],
+    "scope_hierarchy": ["module", "class:AuthHandler", "method:handle"]
+  },
+  
+  "match": {
+    "type": "method",
+    "name": "handle",
+    "signature": "def handle(self, request: Request) -> Response:",
+    "matched_terms": ["handle", "authentication"],
+    "match_positions": [
+      {"line": 45, "column": 8, "text": "def handle"},
+      {"line": 50, "column": 20, "text": "# Handle authentication"}
+    ]
+  },
+  
+  "content": {
+    "snippet": "[code with context and line numbers]",
+    "context_lines": 15
+  },
+  
+  "metadata": {
+    "docstring": "Handle authentication for requests",
+    "is_public": true,
+    "parameters": ["self", "request"],
+    "return_type": "Response",
+    "decorators": ["@validate_request"]
+  },
+  
+  "tags": ["authentication", "security", "middleware"]
+}
+```
+
+Benefits for LLM processing:
+- ✅ Predictable structure (easy JSON parsing)
+- ✅ All relevant metadata included (full context)
+- ✅ Clear scope hierarchy (understand code location)
+- ✅ Snippet with context (understand purpose)
+- ✅ Relevance scoring (prioritize important results)
+- ✅ Matched terms highlighted (understand what matched)
+
+---
+
+## Performance Characteristics
+
+### Indexing
+| Scenario | Time |
+|----------|------|
+| Small project (100 files) | <5 sec |
+| Medium project (500 files) | 20-30 sec |
+| Large project (2000 files) | 2-3 min |
+| Incremental update (1 file) | <200 ms |
+
+### Querying
+| Operation | Time |
+|-----------|------|
+| Simple keyword query | <50 ms |
+| Multi-term query | <200 ms |
+| With context extraction | <500 ms |
+| Full result set (50 results) | <1 sec |
+
+### Storage
+- **Typical project**: 50-100 MB
+- **With compression**: 20-40 MB
+- **Peak memory during search**: <200 MB
+
+---
+
+## Implementation Phases
+
+```
+PHASE 1 (Weeks 1-2): Foundation
+├─ File discovery, type detection, routing
+├─ Basic text indexing, simple JSON storage
+├─ Python AST parser
+└─ Outcome: Searchable Python codebase
+
+PHASE 2 (Weeks 3-4): Markdown Support
+├─ Markdown parser (heading hierarchy)
+├─ Context extraction for markdown
+└─ Outcome: Both Python and Markdown searchable
+
+PHASE 3 (Weeks 5-6): Advanced Ranking
+├─ Multi-factor relevance scoring
+├─ Configuration and tuning
+└─ Outcome: Smart result ordering
+
+PHASE 4 (Weeks 7-8): Additional Formats
+├─ JavaScript/TypeScript support
+├─ JSON/YAML configuration files
+└─ Outcome: Multi-language support
+
+PHASE 5 (Weeks 9-10): Optimization
+├─ Incremental indexing (10x speedup)
+├─ Query caching
+├─ Parallel processing
+└─ Outcome: Production-ready performance
+
+PHASE 6+ (Optional): Advanced Features
+├─ Semantic search with embeddings
+├─ IDE integrations
+├─ Distributed indexing
+└─ Outcome: Enterprise-grade system
+```
+
+---
+
+## Use Case Examples
+
+### 1. Code Review Context
+**Query**: "authentication flow from request to token"
+**Output**: Full execution path with code snippets, signatures, and docstrings
+**Benefit**: 5 minutes instead of 30 min to understand code
+
+### 2. Onboarding New Team Member
+**Query**: "@llm get_all_public_functions with_docstrings in auth"
+**Output**: All public auth functions formatted for LLM
+**Benefit**: LLM generates architecture overview and examples
+
+### 3. Bug Investigation
+**Query**: "session_timeout references AND modifications"
+**Output**: All uses of session_timeout with call chains
+**Benefit**: LLM analyzes data flow to find bug root cause
+
+### 4. Safe Refactoring
+**Query**: "old_function_name type:reference"
+**Output**: Function definition + all call sites with context
+**Benefit**: Confident, complete refactoring
+
+### 5. Documentation Generation
+**Query**: "@doc get_all_exported_functions with_docstrings"
+**Output**: All exported functions with documentation
+**Benefit**: Auto-generated API docs always in sync
+
+---
+
+## Key Advantages
+
+### For Developers
+- 🔍 Find code quickly without manual searching
+- 📚 Understand context and scope automatically
+- 🎯 See related code patterns
+- 📖 Generate documentation on-the-fly
+
+### For Teams
+- 🏃 Faster onboarding of new members
+- 🐛 Quicker bug resolution
+- 🔄 Consistent refactoring
+- 📊 Automated architecture analysis
+
+### For LLMs
+- 📋 Structured, predictable output format
+- 🎯 Relevant context included
+- 🏗️ Clear code structure (scope hierarchy)
+- 🔗 Linked references (scope + line numbers)
+
+---
+
+## Technical Highlights
+
+### 1. Incremental Indexing
+- **File hash detection**: Only reindex changed files
+- **AST caching**: Parse once, use many times
+- **Speed improvement**: 10x faster re-indexing
+
+### 2. Multi-Index Strategy
+- **Keyword-based**: Fast for simple searches
+- **Scope-based**: Understand code structure
+- **Semantic** (optional): Find similar patterns
+
+### 3. Context-Aware Ranking
+- 5-factor scoring considers keyword quality, position, scope depth, recency, and frequency
+- Weights configurable per use case
+- Optimized for typical developer workflows
+
+### 4. Format Flexibility
+- Parse 5+ file types with specialized handling
+- Each returns appropriate metadata
+- Unified output schema for LLM consistency
+
+---
+
+## Comparison with Alternatives
+
+| Feature | NGAO | grep | IDE | Semantic Search |
+|---------|------|------|-----|-----------------|
+| Structured results | ✅ | ❌ | ✅ (IDE specific) | ✅ |
+| LLM-friendly format | ✅ | ❌ | ❌ | Partial |
+| Scope hierarchy | ✅ | ❌ | ✅ | ❌ |
+| Multi-format support | ✅ | ✅ | Limited | Limited |
+| Local indexing | ✅ | ✅ | ✅ | Partial |
+| Configurable ranking | ✅ | ❌ | ❌ | Limited |
+| CLI + API | ✅ | ✅ | ❌ | ✅ |
+| Context extraction | ✅ | ❌ | ✅ | Limited |
+
+---
+
+## Recommended Implementation Technology
+
+### For Python Implementation (Recommended for MVP)
+**Rationale**: Faster development, rich ecosystem for code parsing
+
+- **Core**: Python 3.10+, Poetry for dependency management
+- **Parsing**: `ast` module (Python), `remark` (Markdown), `tree-sitter` (JS)
+- **Indexing**: SQLite for metadata, JSON for custom indexes
+- **Search**: Native dict-based inverted index, `rapidfuzz` for fuzzy matching
+- **CLI**: Click or Typer for command-line interface
+- **LLM**: Pydantic for structured output, JSON serialization
+
+### For Node.js Implementation (Alternative)
+- **Parsing**: `@babel/parser`, `remark`, `tree-sitter-web`
+- **Indexing**: `better-sqlite3`, `lowdb`
+- **CLI**: `Commander.js`, `Chalk`
+
+---
+
+## Success Criteria
+
+### Phase 1 (MVP)
+- ✅ Index Python and Markdown files
+- ✅ Keyword search returns relevant results
+- ✅ LLM-friendly JSON output format
+- ✅ CLI interface working
+- ✅ >80% test coverage
+
+### Phase 2+ (Production)
+- ✅ <500ms query time (any size codebase)
+- ✅ <100 MB storage for typical project
+- ✅ Incremental indexing working
+- ✅ 5+ file types supported
+- ✅ Relevance scoring tuned
+- ✅ >85% precision in top results
+
+---
+
+## Quick Start (Conceptual)
+
+```bash
+# 1. Install
+pip install ngao-search
+
+# 2. Initialize index
+ngao-search init /path/to/codebase
+
+# 3. Search
+ngao-search query "authentication handler" --format json
+
+# 4. Use with LLM
+results=$(ngao-search query "database query patterns" --format json)
+cat $results | llm prompt "Explain these database patterns and suggest optimizations"
+```
+
+---
+
+## Next Steps
+
+1. **Validate**: Get feedback on architecture from potential users
+2. **Prototype**: Build Phase 1 implementation (Python + Markdown + basic ranking)
+3. **Test**: Create comprehensive test suite with sample queries
+4. **Optimize**: Profile and optimize based on real usage patterns
+5. **Expand**: Add additional file types based on user feedback
+6. **Integrate**: Connect with IDE plugins and LLM workflows
+
+---
+
+## Documentation Files
+
+This research is complete with:
+
+1. **[START_HERE.md](START_HERE.md)** - Entry point and navigation
+2. **[RESEARCH_SUMMARY.md](RESEARCH_SUMMARY.md)** - This executive overview
+3. **[IMPLEMENTATION_OVERVIEW.md](IMPLEMENTATION_OVERVIEW.md)** - TypeScript/Node.js guide
+4. **[QUICK_REFERENCE.md](QUICK_REFERENCE.md)** - Formulas and quick answers
+5. **[DOCUMENTATION_INDEX.md](DOCUMENTATION_INDEX.md)** - Complete navigation
+6. **[DELIVERY_SUMMARY.md](DELIVERY_SUMMARY.md)** - Summary of content
+
+All documents in the reference folder are complementary and should be read together for complete understanding.
+
+---
+
+## Questions Answered
+
+✅ **How to index different file types?** → Format-specific parsers with AST analysis
+✅ **How to return LLM-friendly format?** → Universal JSON schema with all metadata
+✅ **How to rank results relevantly?** → 5-factor scoring algorithm
+✅ **How to provide context?** → Context extraction with configurable window sizes
+✅ **How to scale to large codebases?** → Incremental indexing + parallel processing
+✅ **How to handle different scopes?** → Scope hierarchy indexing
+✅ **How to balance performance and accuracy?** → Multiple optimization layers
+
+---
+
+## Key Insights
+
+1. **Multi-Index is Essential**: Keyword search alone is insufficient; scope hierarchy indexing provides structure understanding
+2. **Format Matters**: Each file type has unique structure; generic parsing loses important information
+3. **Context is Critical**: Search results without context are nearly useless; must include surrounding code/docs
+4. **LLM Optimization is Novel**: Formatting results specifically for AI models enables new use cases
+5. **Ranking is Complex**: Simple keyword relevance insufficient; multi-factor scoring needed
+6. **Incremental Indexing is Crucial**: Full re-indexing on every change is too slow; change detection mandatory
+7. **Caching Layers Essential**: Multiple caching levels (query, block, AST) needed for sub-second performance
+
+---
+
+This comprehensive research provides a solid foundation for implementing NGAO Search as a powerful, LLM-friendly code search system optimized for modern development workflows.
+