@ngao/search 0.1.0 → 0.1.2

package/docs/architecture-design-standards/04_VISUAL_GUIDE.md

@@ -0,0 +1,922 @@
+# Visual Architecture & Decision Trees
+
+## 1. System Architecture Visualization
+
+```
+┌────────────────────────────────────────────────────────────────────┐
+│                         NGAO SEARCH SYSTEM                        │
+│                    Multi-Format LLM-Friendly                      │
+│                        Code/Document Search                       │
+└────────────────────────────────────────────────────────────────────┘
+
+╔════════════════════════════════════════════════════════════════════╗
+║                      INPUT LAYER                                  ║
+╚════════════════════════════════════════════════════════════════════╝
+
+┌─────────────┬─────────────┬──────────────┬──────────────┐
+│   Python    │  Markdown   │ JavaScript   │ JSON/YAML    │
+│   Files     │   Files     │   Files      │ Config Files │
+└──────┬──────┴──────┬──────┴───────┬──────┴──────┬───────┘
+       │             │              │             │
+       └─────────────┼──────────────┼─────────────┘
+                     │
+                     ▼
+╔════════════════════════════════════════════════════════════════════╗
+║                    PARSING LAYER                                  ║
+║  (Format-Specific AST Extraction)                                ║
+╚════════════════════════════════════════════════════════════════════╝
+
+        ┌─────────────────────────────────────────┐
+        │   FileType Router & Parser Selector     │
+        └─────────────────────────────────────────┘
+                        ↓
+     ┌──────────────┬──────────────┬──────────────┬──────────────┐
+     │              │              │              │              │
+     ▼              ▼              ▼              ▼              ▼
+┌────────────┐ ┌───────────┐ ┌──────────┐ ┌──────────┐ ┌─────────┐
+│   Python   │ │ Markdown  │ │ Babel/   │ │  JSON    │ │ Generic │
+│   AST      │ │ remark    │ │ tree-    │ │  Parser  │ │  Text   │
+│ Traversal  │ │ Unified   │ │ sitter   │ │          │ │ Parser  │
+└────────────┘ └───────────┘ └──────────┘ └──────────┘ └─────────┘
+
+Extract:       Extract:      Extract:      Extract:       Extract:
+├─ Functions   ├─ Headings   ├─ Functions  ├─ Keys        ├─ Words
+├─ Classes     ├─ Sections   ├─ Classes    ├─ Values      └─ Lines
+├─ Methods     ├─ Paragraphs ├─ Hooks      └─ Nesting
+├─ Docstrings  └─ Code       └─ JSDoc
+└─ Decorators     blocks
+
+
+            ┌─────────────────────────────────────────┐
+            │  Block Extraction & Normalization       │
+            │  (Compute scope, metadata, line ranges) │
+            └─────────────────────────────────────────┘
+                        ↓
+                     BLOCKS
+                ┌──────────────────┐
+                │ block_id: func_42│
+                │ file: src/auth.. │
+                │ type: method     │
+                │ scope: [class]   │
+                │ lines: 45-78     │
+                │ content: ...     │
+                │ metadata: {...}  │
+                └──────────────────┘
+
+╔════════════════════════════════════════════════════════════════════╗
+║                    INDEXING LAYER                                 ║
+║  (Build Specialized Search Indexes)                              ║
+╚════════════════════════════════════════════════════════════════════╝
+
+        Normalized Blocks
+                ↓
+     ┌──────────┴──────────┬─────────────┬─────────────┐
+     │                     │             │             │
+     ▼                     ▼             ▼             ▼
+┌───────────────┐  ┌──────────────┐ ┌──────────┐ ┌──────────┐
+│   INVERTED    │  │ SCOPE INDEX  │ │  BLOCK   │ │SEMANTIC  │
+│    INDEX      │  │ (Hierarchy)  │ │REGISTRY  │ │(Embeddings)
+├───────────────┤  ├──────────────┤ ├──────────┤ ├──────────┤
+│ keyword→pos   │  │file→scope→   │ │block_id→ │ │block→    │
+│               │  │children      │ │metadata  │ │embedding │
+│auth:          │  │              │ │          │ │          │
+│├─files:[..]   │  │src/auth.py:  │ │func_42: {│ │[0.1,    │
+│└─pos:[45,50]  │  │├─module      │ │file:..., │ │0.2,    │
+│               │  │├─class:Auth  │ │line:45-78│ │0.3,..] │
+│(70% storage)  │  │└─method:h    │ │type:meth │ │         │
+│               │  │              │ │}         │ │(optional)
+│(Search by     │  │(20% storage) │ │          │ │         │
+│ keywords)     │  │              │ │(10%st)  │ │(10-20%  │
+│               │  │(Navigate     │ │          │ │ storage)│
+│               │  │structure)    │ │(Quick    │ │         │
+│               │  │              │ │lookup)   │ │(Semantic│
+│               │  │              │ │          │ │ search) │
+└───────────────┘  └──────────────┘ └──────────┘ └──────────┘
+
+        ┌─────────────────────────────────────┐
+        │  Persist Indexes to Storage         │
+        │ (SQLite + JSON or pure JSON)        │
+        └─────────────────────────────────────┘
+                     ↓
+             .ngao_search/
+          ├─ inverted_index.json
+          ├─ scope_index.json
+          ├─ block_registry.json
+          └─ index_metadata.json
+
+
+╔════════════════════════════════════════════════════════════════════╗
+║                     QUERY LAYER                                   ║
+╚════════════════════════════════════════════════════════════════════╝
+
+    User Query: "find auth handler with retry"
+                        ↓
+        ┌───────────────────────────────┐
+        │  Query Parser & Analyzer      │
+        ├───────────────────────────────┤
+        │ • Tokenize                    │
+        │ • Remove stopwords            │
+        │ • Extract filters (type:...)  │
+        │ • Plan search strategy        │
+        └───────────────────────────────┘
+                ↓
+        Terms: ["auth", "handler", "retry"]
+        Filters: {}
+        Strategy: multi_index
+                ↓
+    ┌──────────────┬────────────┬──────────┐
+    │              │            │          │
+    ▼              ▼            ▼          ▼
+KEYWORD       SCOPE        SEMANTIC      REGEX
+SEARCH        SEARCH       SEARCH (opt)  SEARCH
+ │             │             │            │
+ └──────────────┴─────────────┴────────────┘
+        ↓
+   Aggregate & Deduplicate Results
+   └─→ [block_42, block_88, method_15, ...]
+        ↓
+┌─────────────────────────────────────┐
+│  Context Extraction                 │
+│ • Load source file                  │
+│ • Extract snippet ±context          │
+│ • Preserve formatting               │
+│ • Highlight matches                 │
+└─────────────────────────────────────┘
+        ↓
+┌─────────────────────────────────────┐
+│  Relevance Ranking (Multi-Factor)   │
+│                                     │
+│ Score = 0.35×keyword_match +       │
+│         0.25×position +             │
+│         0.15×scope_specificity +   │
+│         0.15×recency +              │
+│         0.10×frequency              │
+└─────────────────────────────────────┘
+        ↓
+    Sort by Score (DESC)
+    Truncate to max_results
+        ↓
+
+╔════════════════════════════════════════════════════════════════════╗
+║                    OUTPUT LAYER                                   ║
+║  (LLM-Friendly Structured Format)                                ║
+╚════════════════════════════════════════════════════════════════════╝
+
+    ┌─────────────────────────────────────┐
+    │  Transform to Type-Specific Format  │
+    └─────────────────────────────────────┘
+        ↓
+    For Python:
+    {type: "python", signature: "...", decorators: [...]}
+        
+    For Markdown:
+    {type: "markdown", heading_hierarchy: [...]}
+        
+    For JavaScript:
+    {type: "javascript", jsdoc: "..."}
+        
+    For JSON:
+    {type: "json", key_path: [...], value: "..."}
+        ↓
+    ┌──────────────────────────────────────┐
+    │  Wrap in LLM-Friendly Schema         │
+    │ • Structured JSON                    │
+    │ • All metadata included              │
+    │ • Scope hierarchy clear              │
+    │ • Match positions highlighted        │
+    └──────────────────────────────────────┘
+        ↓
+    ┌───────────────┬──────────┬──────────┬────────┐
+    │               │          │          │        │
+    ▼               ▼          ▼          ▼        ▼
+  JSON API        CLI        IDE      Web UI    LLM
+  (HTTP)       (Terminal)  Plugin   Interface  Integration
+
+
+┌─────────────────────────────────────────────────────────────────────┐
+│ RESULT EXAMPLE (LLM-Ready JSON):                                   │
+├─────────────────────────────────────────────────────────────────────┤
+│ {                                                                   │
+│   "rank": 1,                                                       │
+│   "relevance_score": 0.92,                                         │
+│   "file": {"path": "src/auth/handler.py", "type": "python"},      │
+│   "location": {                                                     │
+│     "start_line": 45,                                              │
+│     "end_line": 78,                                                │
+│     "scope_hierarchy": ["module", "class:AuthHandler"]             │
+│   },                                                                │
+│   "match": {                                                        │
+│     "name": "handle",                                              │
+│     "signature": "def handle(self, request) -> Response:",         │
+│     "matched_terms": ["handle", "auth"],                          │
+│     "match_positions": [{"line": 45, "text": "def handle"}, ...]  │
+│   },                                                                │
+│   "content": {                                                      │
+│     "snippet": "[code with context and line numbers]",            │
+│     "context_lines": 15                                            │
+│   },                                                                │
+│   "metadata": {                                                     │
+│     "docstring": "Handle authentication...",                       │
+│     "decorators": ["@validate_request"],                           │
+│     "is_public": true                                              │
+│   }                                                                 │
+│ }                                                                   │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 2. File Type Decision Tree
+
+```
+┌─────────────────────────┐
+│ Incoming File           │
+└────────────┬────────────┘
+             │
+    Check File Extension
+             │
+    ┌────────┼────────┬────────┬────────┐
+    │        │        │        │        │
+   *.py    *.md     *.js    *.json   Other
+    │        │        │        │        │
+    ▼        ▼        ▼        ▼        ▼
+  
+┌─────────┐ ┌──────────┐ ┌──────────┐ ┌─────────┐ ┌─────────┐
+│ Python  │ │ Markdown │ │JavaScript│ │  JSON   │ │ Generic │
+│ Parser  │ │ Parser   │ │ Parser   │ │ Parser  │ │ Parser  │
+└────┬────┘ └────┬─────┘ └────┬─────┘ └────┬────┘ └────┬────┘
+     │           │            │            │           │
+     ▼           ▼            ▼            ▼           ▼
+     
+Parse with  Parse with   Parse with  Parse with  Simple
+`ast`       `remark`     `babel`     JSON API    Regex
+module      unified      parser                  matching
+
+Extract:    Extract:     Extract:    Extract:    Extract:
+├─ Classes  ├─ Headings  ├─ Classes  ├─ Keys     └─ Words
+├─ Funcs    ├─ Sections  ├─ Funcs    ├─ Values
+├─ Methods  ├─ Params    ├─ Hooks    └─ Nesting
+├─ Types    └─ Code      ├─ Exports
+└─ Docstr      blocks    └─ JSDoc
+ 
+  Blocks: Blocks:      Blocks:      Blocks:     Blocks:
+  scope:  scope:       scope:       scope:      scope:
+  module→ H1→H2→...   module→      path→       (none)
+  class→  section      class→       sub_key
+  method  para         method
+  
+  ▼           ▼            ▼            ▼           ▼
+  
+╔═════════════════════════════════════════════════════╗
+║ Normalized Block Objects (Common Format)           ║
+╠═════════════════════════════════════════════════════╣
+║ • file_path                                        ║
+║ • block_id                                         ║
+║ • item_type                                        ║
+║ • name                                             ║
+║ • content                                          ║
+║ • location (start_line, end_line)                 ║
+║ • scope_hierarchy                                  ║
+║ • metadata (type-specific)                         ║
+║ • docstring                                        ║
+╚═════════════════════════════════════════════════════╝
+  ▼
+  → To Indexing Layer
+```
+
+---
+
+## 3. Query Processing Decision Tree
+
+```
+┌────────────────────────┐
+│ Raw User Query         │
+└────────────┬───────────┘
+             │
+    ┌────────────────────────┐
+    │ Parse Query            │
+    ├────────────────────────┤
+    │ Tokenize: split into   │
+    │ individual terms       │
+    └────────────┬───────────┘
+                 │
+         ┌───────┴───────┐
+         │               │
+    Extract Filters   Analyze Terms
+         │               │
+    type:*.py         Remove
+    scope:auth.*      stopwords
+    file:*.py         │
+         │            └─→ Stemming
+         │                 │
+         │           "authentication"
+         │           → "auth"
+         │                 │
+         ├─────────────────┘
+         │
+         ▼
+    ┌──────────────────────────┐
+    │ Plan Search Strategy     │
+    │                          │
+    │ Single term? → Keywords  │
+    │ Multi-term? → Multi-idx  │
+    │ Regex? → Pattern search  │
+    │ Semantic? → Embeddings   │
+    └──────────┬───────────────┘
+                │
+    ┌───────────┴────────────┬──────────┬───────────┐
+    │                        │          │           │
+    ▼                        ▼          ▼           ▼
+    
+KEYWORD SEARCH        SCOPE SEARCH   SEMANTIC    REGEX
+(Inverted Index)      (Hierarchy)    SEARCH      SEARCH
+
+Search for each    Filter by       Find        Match
+term in inverted   scope path      similar     pattern
+index              (scope:auth.*)  embeddings
+                                   
+Find position      Narrow down     Find        Get all
+info in inverted   results to      similar     matching
+index              certain parts   code        files
+                   of structure
+
+Match all          │               │           │
+results            │               │           │
+│                  │               │           │
+├──────────────────┴───────────────┴───────────┴──→
+
+            ┌─────────────────────────────┐
+            │ Aggregate Results           │
+            │ • Combine from all indexes  │
+            │ • Deduplicate blocks        │
+            │ • Prepare for ranking       │
+            └──────────┬──────────────────┘
+                       │
+                       ▼
+            ┌─────────────────────────────┐
+            │ Extract Context             │
+            │ • Load source files         │
+            │ • Get snippet ±context      │
+            │ • Preserve formatting       │
+            └──────────┬──────────────────┘
+                       │
+                       ▼
+            ┌──────────────────────────────┐
+            │ Rank Results                 │
+            │                              │
+            │ For each result compute:     │
+            │  score = 0.35×kw_match +    │
+            │          0.25×position +    │
+            │          0.15×scope +       │
+            │          0.15×recency +     │
+            │          0.10×frequency     │
+            └──────────┬───────────────────┘
+                       │
+                       ▼
+            ┌─────────────────────────────┐
+            │ Sort & Truncate             │
+            │ • Sort by score (DESC)      │
+            │ • Keep top N results        │
+            │ • Return to formatter       │
+            └──────────┬──────────────────┘
+                       │
+                       ▼
+                  Formatted Results
+```
+
+---
+
+## 4. Ranking Factor Contribution
+
+```
+RELEVANCE SCORE BREAKDOWN
+═════════════════════════════════════════════════════════
+
+Query: "handle auth retry" searching in codebase
+
+For each matching block, compute:
+
+FACTOR 1: KEYWORD MATCH (Weight: 0.35)
+───────────────────────────────────────
+
+Block name: "handle_auth_retry"
+Terms matched: ["handle", "auth", "retry"]
+Match score: 3/3 terms = 1.0
+
+   Contribution: 1.0 × 0.35 = 0.35
+   
+        [████████████████████████████████████] 35%
+
+
+FACTOR 2: POSITION (Weight: 0.25)
+─────────────────────────────────
+
+Match location:
+- In function name: ✓ (score: 1.0)
+- In docstring: ✓ (adds: 0.0)
+- In code body: ✓ (adds: 0.0)
+
+Position score: 1.0
+
+   Contribution: 1.0 × 0.25 = 0.25
+   
+        [████████████████████████] 25%
+
+
+FACTOR 3: SCOPE SPECIFICITY (Weight: 0.15)
+──────────────────────────────────────────
+
+Scope: ["module", "class:AuthHandler", "method:handle"]
+Depth: 3
+Score: min(0.5 + (3 × 0.1), 1.0) = 0.8
+
+   Contribution: 0.8 × 0.15 = 0.12
+   
+        [██████████████] 12%
+
+
+FACTOR 4: RECENCY (Weight: 0.15)
+────────────────────────────────
+
+Last modified: 2 hours ago
+Score: 1.0 (within 24 hours)
+
+   Contribution: 1.0 × 0.15 = 0.15
+   
+        [███████████████] 15%
+
+
+FACTOR 5: FREQUENCY (Weight: 0.10)
+──────────────────────────────────
+
+Term occurrences: 
+- "handle": 2 times
+- "auth": 3 times
+- "retry": 1 time
+Total: 6 times in 50 lines
+Normalized: min(6 / 50, 1.0) = 0.12
+
+   Contribution: 0.12 × 0.10 = 0.012
+   
+        [█] ~1%
+
+
+TOTAL RELEVANCE SCORE:
+═════════════════════════════════════════════════════════
+
+0.35 + 0.25 + 0.12 + 0.15 + 0.012 = 0.862
+
+Final Score: 0.86 / 1.0
+         ┌──────────────────────────────────────────┐
+         │████████████████████████████████████░░░░░░│
+         └──────────────────────────────────────────┘
+         86% relevance
+
+Result Rank: Top results sorted by this score
+```
+
+---
+
+## 5. Block Extraction Flowchart
+
+```
+INPUT: Raw source code file
+│
+├─→ PYTHON FILE (*.py)
+│   │
+│   ├─ Parse with ast.parse()
+│   │  │
+│   │  ├─→ ModuleVisitor.visit()
+│   │  │   │
+│   │  │   ├─→ Module docstring
+│   │  │   │
+│   │  │   ├─→ Top-level functions
+│   │  │   │   └─→ For each:
+│   │  │   │       ├─ Extract params
+│   │  │   │       ├─ Get docstring
+│   │  │   │       ├─ Get decorators
+│   │  │   │       └─ Line numbers
+│   │  │   │
+│   │  │   └─→ Class definitions
+│   │  │       └─→ For each class:
+│   │  │           ├─ Get docstring
+│   │  │           ├─ Get decorators
+│   │  │           └─→ Class methods
+│   │  │               └─→ For each method:
+│   │  │                   ├─ Extract params
+│   │  │                   ├─ Get return type
+│   │  │                   ├─ Get docstring
+│   │  │                   └─ Line numbers
+│   │  │
+│   │  └─→ Normalize all blocks
+│   │      ├─ Compute scope_hierarchy
+│   │      ├─ Extract content
+│   │      ├─ Get line ranges
+│   │      └─ Create block_ids
+│   │
+│   └─→ Output: List[Block]
+│
+├─→ MARKDOWN FILE (*.md)
+│   │
+│   ├─ Parse with remark/unified
+│   │  │
+│   │  ├─→ Heading traversal
+│   │  │   ├─ H1 → Top-level section
+│   │  │   ├─ H2 → Subsection
+│   │  │   ├─ H3 → Sub-subsection
+│   │  │   └─ Track hierarchy
+│   │  │
+│   │  ├─→ Content between headings
+│   │  │   ├─ Paragraphs
+│   │  │   ├─ Code blocks (with language)
+│   │  │   ├─ Lists
+│   │  │   └─ Tables
+│   │  │
+│   │  ├─→ Frontmatter (YAML)
+│   │  │   └─ Extract metadata
+│   │  │
+│   │  └─→ Normalize sections
+│   │      ├─ Group content by heading
+│   │      ├─ Create hierarchy path
+│   │      ├─ Line number tracking
+│   │      └─ Create block_ids
+│   │
+│   └─→ Output: List[Block]
+│
+├─→ JAVASCRIPT FILE (*.js/*.jsx)
+│   │
+│   ├─ Parse with @babel/parser
+│   │  │
+│   │  ├─→ Extract top-level:
+│   │  │   ├─ Import statements
+│   │  │   ├─ Export statements
+│   │  │   └─ Variable declarations
+│   │  │
+│   │  ├─→ Function declarations
+│   │  │   └─→ For each:
+│   │  │       ├─ Get parameters
+│   │  │       ├─ Get JSDoc
+│   │  │       └─ Line range
+│   │  │
+│   │  ├─→ Arrow functions (const useHook = ...)
+│   │  │   └─→ For each:
+│   │  │       ├─ Identify hook pattern
+│   │  │       ├─ Get JSDoc
+│   │  │       └─ Line range
+│   │  │
+│   │  ├─→ Class definitions
+│   │  │   └─→ For each:
+│   │  │       ├─ Methods
+│   │  │       ├─ Properties
+│   │  │       └─ Constructor
+│   │  │
+│   │  ├─→ React Components (JSX)
+│   │  │   └─→ For each:
+│   │  │       ├─ Props
+│   │  │       ├─ Return JSX
+│   │  │       └─ JSDoc
+│   │  │
+│   │  └─→ Normalize all blocks
+│   │      ├─ Compute scope
+│   │      ├─ Extract signature
+│   │      └─ Line ranges
+│   │
+│   └─→ Output: List[Block]
+│
+├─→ JSON FILE (*.json)
+│   │
+│   ├─ Parse as JSON
+│   │  │
+│   │  ├─→ Flatten key hierarchy
+│   │  │   └─ database.connection.pool_size
+│   │  │
+│   │  ├─→ For each key-value:
+│   │  │   ├─ Key path
+│   │  │   ├─ Value
+│   │  │   ├─ Type (string|number|bool|object|array)
+│   │  │   └─ Nesting depth
+│   │  │
+│   │  └─→ Normalize blocks
+│   │      ├─ Create scope from path
+│   │      └─ Line number tracking
+│   │
+│   └─→ Output: List[Block]
+│
+└─→ GENERIC TEXT FILE
+    │
+    ├─ Fallback parsing (last resort)
+    │  │
+    │  ├─→ Line-by-line analysis
+    │  ├─→ Regex pattern matching
+    │  └─→ Basic structure detection
+    │
+    └─→ Output: List[Block] (minimal metadata)
+```
+
+---
+
+## 6. Context Extraction Window
+
+```
+SOURCE FILE (Python)
+═════════════════════════════════════════════════════
+
+  40: class AuthHandler:
+  41:     """Handles authentication"""
+  42:     
+  43:     def __init__(self, config):
+  44:         self.config = config
+  45:
+  46:     def handle(self, request):        ← BLOCK START (line 46)
+  47:         """Handle incoming request"""
+  48:         
+  49:         # Validate token
+  50:         token = request.headers.get('Authorization')
+  51:         if not token:
+  52:             return Response(status=401)
+  53:         
+  54:         # Process authentication
+  55:         result = self.validate_token(token)
+  56:         if not result:
+  57:             return Response(status=403)
+  58:         
+  59:         return Response(status=200)      ← BLOCK END (line 59)
+  60:
+  61:     def validate_token(self, token):
+  62:         """Validate token"""
+
+
+CONTEXT EXTRACTION CONFIG
+═════════════════════════════════════════════════════
+
+{
+  "file_type": "python",
+  "context_lines_before": 5,
+  "context_lines_after": 5,
+  "include_docstring": true,
+  "include_parent_class": true,
+  "max_context_lines": 30
+}
+
+
+EXTRACTED CONTEXT WINDOW
+═════════════════════════════════════════════════════
+
+BEFORE:
+  41: │ """Handles authentication"""
+  42: │ 
+  43: │ def __init__(self, config):
+  44: │     self.config = config
+  45: │ 
+
+BLOCK CONTENT:
+  46: │ def handle(self, request):
+  47: │     """Handle incoming request"""
+  48: │     
+  49: │     # Validate token
+  50: │     token = request.headers.get('Authorization')
+  51: │     if not token:
+  52: │         return Response(status=401)
+  53: │     
+  54: │     # Process authentication
+  55: │     result = self.validate_token(token)
+  56: │     if not result:
+  57: │         return Response(status=403)
+  58: │     
+  59: │     return Response(status=200)
+
+AFTER:
+  60: │ 
+  61: │ def validate_token(self, token):
+  62: │     """Validate token"""
+
+
+FORMATTED OUTPUT FOR LLM
+═════════════════════════════════════════════════════
+
+Scope: class AuthHandler → method handle
+Lines: 46-59 (in file src/auth/handler.py)
+
+Code:
+```python
+41  │ """Handles authentication"""
+42  │ 
+43  │ def __init__(self, config):
+44  │     self.config = config
+45  │ 
+46  │ def handle(self, request):
+47  │     """Handle incoming request"""
+48  │     
+49  │     # Validate token
+50  │     token = request.headers.get('Authorization')
+51  │     if not token:
+52  │         return Response(status=401)
+53  │     
+54  │     # Process authentication
+55  │     result = self.validate_token(token)
+56  │     if not result:
+57  │         return Response(status=403)
+58  │     
+59  │     return Response(status=200)
+60  │ 
+61  │ def validate_token(self, token):
+62  │     """Validate token"""
+```
+
+Docstring: "Handle incoming request"
+Method signature: def handle(self, request) -> ?
+Related method: validate_token() called on line 55
+```
+
+---
+
+## 7. Performance Optimization Layers
+
+```
+┌────────────────────────────────────────────────────────┐
+│ OPTIMIZATION LAYER 1: INPUT NORMALIZATION             │
+│ (Query Pre-processing)                                │
+└────────────────────────────────────────────────────────┘
+
+Query: "finding authentication token handler retry logic"
+  ▼
+Tokenize:
+["finding", "authentication", "token", "handler", "retry", "logic"]
+  ▼
+Remove stopwords:
+["authentication", "token", "handler", "retry", "logic"]
+  (removed: "finding", "logic")
+  ▼
+Stem/Lemmatize:
+["auth", "token", "handle", "retry", "logic"]
+  ▼
+Optimized query: 5 terms instead of 6
+Reduced search space by 17%
+
+Search cost reduction: -20-30%
+
+
+┌────────────────────────────────────────────────────────┐
+│ OPTIMIZATION LAYER 2: INDEX STRUCTURE                 │
+│ (Data Structure Optimization)                         │
+└────────────────────────────────────────────────────────┘
+
+Naive inverted index:
+{
+  "auth": [
+    {"file": "f1", "line": 45, "block": "b1"},
+    {"file": "f1", "line": 50, "block": "b1"},
+    {"file": "f2", "line": 100, "block": "b5"},
+    ... (100 entries per keyword)
+  ]
+}
+Size: ~1 MB per keyword
+
+Optimized inverted index:
+{
+  "auth": {
+    "files": [compressed bitmap of file IDs],
+    "positions": [varint-encoded line numbers]
+  }
+}
+Size: ~200 KB per keyword
+
+Size reduction: 80%
+Lookup time: O(1) → O(1) ✓
+
+
+┌────────────────────────────────────────────────────────┐
+│ OPTIMIZATION LAYER 3: QUERY EXECUTION STRATEGY        │
+│ (Smart Search Strategy)                               │
+└────────────────────────────────────────────────────────┘
+
+Multi-term query: ["auth", "handler", "retry"]
+
+Strategy A (Naive): Search all terms, combine results
+Time: O(log n × 3 terms) = O(3 log n)
+Results: 1000 candidates
+
+Strategy B (Optimized): Search smallest first, prune
+1. Count postings for each term:
+   - "retry": 45 results (rarest)
+   - "auth": 200 results
+   - "handler": 500 results
+
+2. Search "retry" first (smallest):
+   45 candidates
+
+3. Intersect with "auth":
+   45 × 200 positions checked
+   ~35 survivors
+
+4. Intersect with "handler":
+   ~25 final results
+
+Time: O(log n + intersection cost)
+Results: 25 candidates
+
+Speedup: 40x fewer candidates
+
+
+┌────────────────────────────────────────────────────────┐
+│ OPTIMIZATION LAYER 4: CACHING                         │
+│ (Multi-Level Cache)                                   │
+└────────────────────────────────────────────────────────┘
+
+Query Layers:
+┌─────────────────────────────────┐
+│ Level 1: Query Result Cache     │  Fast
+│ (LRU, TTL: 1 hour)              │  Hit rate: 30-40%
+│ Size: 50 MB                     │
+└────────────────────────────────┐
+         │
+    Miss │ (70% of queries)
+         │
+┌─────────────────────────────────┐
+│ Level 2: Block Content Cache    │  Medium
+│ (Recently accessed blocks)      │  Hit rate: 60-70%
+│ Size: 100 MB                    │
+└────────────────────────────────┐
+         │
+    Miss │ (30% of queries)
+         │
+┌─────────────────────────────────┐
+│ Level 3: Disk (SQLite/JSON)     │  Slow
+│ Size: ~75 MB                    │  Read time: 50-200 ms
+└─────────────────────────────────┘
+
+Overall performance:
+- Cache hit (L1): ~5 ms (70% speedup vs disk)
+- Cache hit (L2): ~20 ms (10-15x speedup)
+- Cache miss: ~150 ms (disk read)
+- Average (with cache): ~30 ms
+
+Cache speedup factor: 5-6x
+
+
+┌────────────────────────────────────────────────────────┐
+│ OPTIMIZATION LAYER 5: INCREMENTAL INDEXING            │
+│ (Change-Based Updates)                                │
+└────────────────────────────────────────────────────────┘
+
+Full Reindex (without incremental):
+- 1000 files × 100 blocks/file = 100k blocks
+- Time: 10-15 minutes
+- Cost: 100% reprocessing
+
+Incremental Indexing:
+File changed: handler.py (2 KB file)
+  ▼
+1. Compute hash: abc123def456...
+2. Compare with stored: xyz789...
+3. Hash mismatch: FILE CHANGED
+  ▼
+4. Load old blocks for handler.py (from registry)
+   - Remove 12 old blocks from indexes
+   - Time: 10 ms
+  ▼
+5. Parse new version: 50 ms
+6. Extract new blocks: 10 ms
+7. Add to indexes: 20 ms
+8. Update metadata: 5 ms
+  ▼
+Total time: ~95 ms
+Speedup: 6000x faster than full reindex!
+
+
+┌────────────────────────────────────────────────────────┐
+│ OPTIMIZATION LAYER 6: PARALLEL PROCESSING             │
+│ (Multi-threaded Indexing)                             │
+└────────────────────────────────────────────────────────┘
+
+Sequential Processing:
+File1 → Parse (100ms) → Extract (20ms) → Index (10ms)
+File2 → Parse (100ms) → Extract (20ms) → Index (10ms)
+File3 → Parse (100ms) → Extract (20ms) → Index (10ms)
+Total: 390 ms
+
+Parallel Processing (4 workers):
+Worker1: File1 Parse (100ms)
+Worker2: File2 Parse (100ms)
+Worker3: File3 Parse (100ms)
+Worker4: File4 Parse (100ms)  ← All parallel
+───────────────────────────────────
+         100 ms total (parse phase)
+
+Worker1: File1 Extract (20ms)
+Worker2: File2 Extract (20ms)
+Worker3: File3 Extract (20ms)
+Worker4: File4 Extract (20ms)
+───────────────────────────────────
+          20 ms total (extract phase)
+
+Worker1-4: Batch index (40ms)
+───────────────────────────────────
+          40 ms total (index phase)
+
+Parallel total: ~160 ms
+Speedup: 2.4x (limited by I/O and batch overhead)
+```
+
+This visual guide provides comprehensive reference for understanding the system architecture and decision-making processes.
+