graph-indexer 1.0.1 → 1.0.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,144 +1,136 @@
1
1
  # graph-indexer
2
2
 
3
3
  <p align="center">
4
- <strong>Zero-Database • Air-Gapped • AST-Precise • Hybrid Semantic Search</strong><br>
5
- <em>Production-grade MCP code indexer for AI coding agents — surgical context retrieval that cuts token costs by up to 84% and improves code accuracy.</em>
4
+ <strong>Zero-Database · Air-Gapped · AST-Precise · Hybrid Search</strong><br>
5
+ <em>An MCP server that lets AI coding agents search your codebase by concept instead of reading files — reducing context usage by 65–88% per query.</em>
6
6
  </p>
7
7
 
8
8
  <p align="center">
9
9
  <a href="https://www.npmjs.com/package/graph-indexer"><img src="https://img.shields.io/npm/v/graph-indexer?color=blue&style=flat-square" alt="npm version"></a>
10
10
  <a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-yellow.svg?style=flat-square" alt="License: MIT"></a>
11
11
  <a href="https://nodejs.org"><img src="https://img.shields.io/badge/Node.js-%3E%3D18-green?style=flat-square" alt="Node.js 18+"></a>
12
- <a href="#test-results"><img src="https://img.shields.io/badge/Recall@5-100%25-brightgreen?style=flat-square" alt="Recall@5 100%"></a>
12
+ <a href="#results"><img src="https://img.shields.io/badge/Recall@5-100%25-brightgreen?style=flat-square" alt="Recall@5 100%"></a>
13
13
  </p>
14
14
 
15
15
  ---
16
16
 
17
- ## The Problem: Token Waste and Lost Context
17
+ ## What it does
18
18
 
19
- When you use an AI coding agent (Claude, Cursor, GitHub Copilot) on a real codebase, the default workflow wastes **80–90% of your context window**:
19
+ graph-indexer pre-indexes your codebase into an AST-precise search index and exposes it as an MCP server. Instead of reading full files, AI agents call `search_code("payment validation")` and get back the exact functions that match — with their type signatures, docstrings, dependency graph, and call sites — in a fraction of the tokens.
20
20
 
21
- ```
22
- Agent wants: "Find the payment processing function"
23
- Standard workflow:
24
- 1. Read file: src/payments/handlers.ts (850 lines) → ~10,625 tokens
25
- 2. Read file: src/config/stripe.ts (340 lines) → ~4,250 tokens
26
- 3. Read file: src/types/payment.ts (120 lines) → ~1,500 tokens
27
- Total: ~16,375 tokens to find ONE function worth ~200 tokens ⚠️
28
-
29
- Reality for complex refactors:
30
- • 5–10 such reads per session
31
- • ~80,000–160,000 tokens just for context gathering
32
- • At Claude 3.5 Sonnet pricing, that's ~$0.48–$0.96 per refactor
33
- • AND the LLM accuracy *decreases* — it gets lost in the noise
34
- ```
21
+ It runs entirely on your machine. No external database, no cloud APIs, no telemetry.
35
22
 
36
- Three compounding problems:
37
-
38
- 1. **Token burn:** File reads consume 10–50× more tokens than necessary
39
- 2. **Lost in the Middle:** LLMs lose accuracy when relevant code is buried in 1000-line files
40
- 3. **Topology blindness:** The agent can't see cross-file dependencies. Refactors break silently.
23
+ ---
41
24
 
42
- ## The Solution: Surgical, AST-Precise Retrieval
25
+ ## Why it matters
43
26
 
44
- **graph-indexer** is a local MCP server that replaces file reads with **structural code queries**. Instead of reading entire files, your agent retrieves exactly the code it needs.
27
+ When an AI agent works on a real codebase without graph-indexer, it reads files to find code:
45
28
 
46
29
  ```
30
+ Agent task: "Add error handling to the payment processing flow"
31
+
32
+ Without graph-indexer:
33
+ readFile("src/payments/service.ts") → 8,400 tokens
34
+ readFile("src/payments/handlers.ts") → 6,200 tokens
35
+ readFile("src/types/payment.ts") → 1,800 tokens
36
+ Total context consumed: ~16,400 tokens for 3 files
37
+
47
38
  With graph-indexer:
48
- 1. Search: "payment processing" → returns [processPayment, validateCard, chargeCustomer]
49
- 2. One get_chunk() call → ~300 tokens
50
- 3. Dependency topology included → can see full call chain
51
- Total: ~300 tokens vs ~16,375 tokens — 98% savings on this query
39
+ search_code("payment processing") → 650 tokens (5 exact chunks)
40
+ get_chunk("chunk_id") → 280 tokens (full function body)
41
+ Total context consumed: ~930 tokens
52
42
  ```
53
43
 
54
- ### Key Benefits
55
-
56
- | Benefit | Impact | Typical Savings |
57
- | :--- | :--- | :--- |
58
- | **Surgical chunking** | AST-precise: entire functions, never split mid-logic | 84–90% token reduction |
59
- | **Hybrid search** | TF-IDF + semantic vectors (Ollama) rank results intelligently | +0.04 MRR, better relevance |
60
- | **Zero infrastructure** | Pure in-memory index, no external databases | Instant startup, zero DevOps |
61
- | **Air-gapped** | Works entirely locally or with local Ollama | 100% privacy, no API keys |
62
- | **Dependency awareness** | Every result includes topology (who calls this, what it imports) | Prevents broken refactors |
63
- | **Language support** | JavaScript, TypeScript, Python, Rust, Go, PHP, CSS, Java, Kotlin, C#, Ruby | Single tool for polyglot codebases |
64
- | **Fault tolerance** | Degrades gracefully to TF-IDF if Ollama offline | Always works |
44
+ The difference compounds across a session. Agents that read files also lack topology — they can't see which other functions call the one they're modifying, so they miss side effects.
65
45
 
66
46
  ---
67
47
 
68
- ## Test Results: 100% Recall, 96% MRR
48
+ ## Results
69
49
 
70
- graph-indexer was evaluated on 5 production-grade open-source projects totaling **7,503 chunks** across JavaScript, TypeScript, Python, Go, and Ruby.
50
+ Measured across 5 production open-source codebases (7,503 total AST chunks, 65 ground-truth queries). Token savings compare the chunks returned per query against reading the full source files they came from.
71
51
 
72
- ### Test Suite Performance
52
+ ### Token usage per query: with vs. without graph-indexer
73
53
 
74
- | Project | Language | Chunks | Lexical Recall@5 | Hybrid Recall@5 | MRR Improvement |
75
- | :--- | :--- | :--- | :--- | :--- | :--- |
76
- | Axios v1.6.0 | JavaScript | 435 | 1.00 | 1.00 | — |
77
- | Express 4.18.2 | JavaScript | 389 | 1.00 | 1.00 | +0.06 |
78
- | NestJS v10.4.9 | TypeScript | 2,019 | 1.00 | 1.00 | **+0.14** |
79
- | FastAPI 0.103.0 | Python | 3,572 | 1.00 | 1.00 | +0.01 |
80
- | Gin v1.9.1 | Go | 1,088 | 1.00 | 1.00 | +0.01 |
81
- | **Mean** | — | **7,503** | **1.00** | **1.00** | **+0.04** |
54
+ | Project | Language | Codebase size | Without (full files) | With graph-indexer | Savings | Latency |
55
+ | :--- | :--- | ---: | ---: | ---: | ---: | ---: |
56
+ | Axios v1.6.0 | JavaScript | 111k tokens | ~2,290 tok/query | ~655 tok/query | **71.4%** | 0.2 ms |
57
+ | Express 4.18.2 | JavaScript | 145k tokens | ~7,330 tok/query | ~850 tok/query | **88.4%** | 0.2 ms |
58
+ | NestJS v10.4.9 | TypeScript | 700k tokens | ~2,535 tok/query | ~900 tok/query | **64.5%** | 0.7 ms |
59
+ | FastAPI 0.103.0 | Python | 859k tokens | ~4,030 tok/query | ~685 tok/query | **83.0%** | 1.0 ms |
60
+ | Gin v1.9.1 | Go | 131k tokens | ~2,530 tok/query | ~450 tok/query | **82.2%** | 0.4 ms |
61
+ | **Mean** | | | | | **77.9%** | **0.5 ms** |
82
62
 
83
- **What this means:**
84
- - ✅ **100% of relevant code found in top 5 results** across every suite and language
85
- - ✅ **All queries return correct results rank-1** when using exact symbol names (`exact_tokens`)
86
- - ✅ **Mean reciprocal rank improves to 0.96** with semantic search enabled
87
- - ✅ **Hybrid mode adds +0.14 MRR** on NestJS (large polyglot TypeScript codebase)
88
- - ✅ **Graceful degradation:** lexical-only mode still achieves 1.00 recall@5 for exact-name queries
63
+ > "Full files" = reading the source files that contain the top-5 retrieved chunks. "With graph-indexer" = tokens in the returned chunk code snippets only. Measured at 4 chars/token.
89
64
 
90
- ### Query Breakdown
65
+ **Key observations:**
66
+ - NestJS has 700k tokens of source — larger than most LLMs' full context window. graph-indexer reduces each query to ~900 tokens.
67
+ - Express shows 88.4% savings because its core router logic is spread across large files with much non-relevant code.
68
+ - Latency is sub-millisecond for all suites at their real corpus sizes.
91
69
 
92
- 14–15 ground-truth queries per suite covering:
93
- - **Easy (3–5):** Direct function/class names (`HTTPException`, `Router`, `RequestValidator`)
94
- - **Medium (5–7):** Multi-token natural language (`"response send body"`, `"validate request body"`)
95
- - **Hard (4):** Semantic concepts with vocabulary mismatch (`"dependency injection container"`, `"content negotiation"`)
70
+ ### Search quality
96
71
 
97
- **Result:** Even the hardest semantic queries achieve 100% recall@5 when hybrid search is enabled.
72
+ | Project | Chunks | Recall@1 | Recall@3 | Recall@5 | MRR | Queries |
73
+ | :--- | ---: | ---: | ---: | ---: | ---: | ---: |
74
+ | Axios v1.6.0 | 435 | 0.86 | 1.00 | **1.00** | 0.92 | 14 |
75
+ | Express 4.18.2 | 389 | 0.93 | 0.93 | **1.00** | 0.95 | 14 |
76
+ | NestJS v10.4.9 | 2,019 | 1.00 | 1.00 | **1.00** | **1.00** | 15 |
77
+ | FastAPI 0.103.0 | 3,572 | 0.93 | 1.00 | **1.00** | 0.96 | 14 |
78
+ | Gin v1.9.1 | 1,088 | 0.85 | 1.00 | **1.00** | 0.92 | 13 |
79
+ | **Mean** | **7,503** | **0.91** | **0.99** | **1.00** | **0.95** | **70** |
98
80
 
99
- ### Token Savings
81
+ - **Recall@5 = 1.00** across every project and language: the correct answer is always in the top 5 results.
82
+ - **MRR = 0.95** overall: on average, the correct answer is the first or second result.
83
+ - Queries span three difficulty levels: easy (exact symbol names), medium (multi-token descriptions), hard (semantic concepts with vocabulary mismatch).
100
84
 
101
- Average **74.6% token reduction** across all queries when using graph-indexer vs full-file reads:
85
+ ---
102
86
 
103
- ```
104
- Query: "Find the JWT validation logic"
105
- File-based approach: src/auth.ts (850 lines) → 10,625 tokens
106
- graph-indexer: validateToken chunk (35 lines) → 437 tokens
107
- Savings: 95.9% on this query
108
- ```
87
+ ## How it works
109
88
 
110
- ---
89
+ ### Indexing
111
90
 
112
- ## Why graph-indexer Beats Alternatives
91
+ When you run `npm run mcp:index`, graph-indexer:
113
92
 
114
- ### vs. Standard RAG (ChromaDB, Pinecone, Weaviate)
93
+ 1. **Parses** every source file with Tree-sitter to build an AST — no regex, exact code boundaries.
94
+ 2. **Extracts chunks**: named functions, classes, methods, and exports. Each chunk includes its name, docstring, parameters, return type, call sites, and type references.
95
+ 3. **Builds a dependency graph**: bidirectional import map so each chunk knows what it imports and what imports it.
96
+ 4. **Creates two search indexes**:
97
+ - A BM25 inverted index (lexical search, O(1) per-token lookup).
98
+ - A vector embedding per chunk via Ollama `nomic-embed-text` (optional; falls back to lexical-only if Ollama is unavailable).
99
+ 5. **Serializes to disk**: `code-index.json` (metadata + BM25 index) and `code-index.embeddings.bin` (binary float32 vectors).
115
100
 
116
- | Feature | **graph-indexer** | Standard RAG |
117
- | :--- | :--- | :--- |
118
- | **Infrastructure** | Zero-DB, pure in-memory `Map` | Requires external vector database |
119
- | **Privacy** | 100% local + air-gapped | Often sends to cloud APIs |
120
- | **Chunking strategy** | AST-precise (entire functions) | Naive token-count splits — breaks logic in half |
121
- | **Hybrid search** | TF-IDF ⊕ vectors w/ RRF fusion | Dense-only or BM25-only |
122
- | **Code structure** | Bidirectional dependency graph | No topology awareness |
123
- | **Startup time** | <100ms (memory-map index) | 5–30s (load from DB) |
124
- | **Cost** | $0 (local) or $5–50/month (Ollama CPU) | $0–500/month per million queries |
125
- | **Language support** | 11 languages with AST precision | Language-agnostic (no structure) |
101
+ A background file watcher daemon (`watch-daemon.mjs`) re-indexes changed files automatically after the initial index is built.
126
102
 
127
- ### vs. File-Based Search (git grep, ripgrep)
103
+ ### Search
128
104
 
129
- graph-indexer is semantic + structural, not just text matching:
105
+ When an agent calls `search_code("payment validation middleware")`:
130
106
 
131
- | Query | ripgrep | graph-indexer |
132
- | :--- | :--- | :--- |
133
- | `grep -r "ValidationError"` | Returns every line mentioning `ValidationError` (thousands of results) | Returns the definition + all places it's raised, ranked by relevance |
134
- | `grep -r "async function"` | Every async function in the codebase | Only async functions matching semantic context of your query |
135
- | Refactor safety | No way to know if rename is safe | Call graph shows every place the symbol is used |
107
+ 1. **Lexical search** over the BM25 inverted index — returns candidates with BM25 scores in O(query_terms) time.
108
+ 2. **Vector search** over pre-normalized float32 embeddings — returns semantically similar chunks.
109
+ 3. **Reciprocal Rank Fusion (RRF)** merges both ranked lists with weights (lexical 1.5×, vector 1.0×).
110
+ 4. **Scoring adjustments**: exact name match +2.0×, snake_case suffix match +1.4×, path token match +1.4×, test/example files demoted.
111
+ 5. **Returns** top-k chunks with: code snippet, type signature, docstring, call graph, and dependency topology — all in one response.
136
112
 
137
- ---
113
+ ### Architecture
114
+
115
+ ```
116
+ Source files (.ts .js .py .go .rs …)
117
+ │
118
+ ▼
119
+ Tree-sitter AST parser
120
+ │
121
+ ├──► Named chunks (functions, classes, methods)
122
+ │ │
123
+ │ ├──► BM25 inverted index ──┐
124
+ │ └──► Ollama embeddings ──┤── RRF fusion ──► Ranked results
125
+ │ │
126
+ └──► Dependency graph ────────────────┘
127
+ (importedBy / imports) ▼
128
+ MCP tools for agents
129
+ ```
138
130
 
139
131
  ---
140
132
 
141
- ## Quick Start
133
+ ## Getting started
142
134
 
143
135
  ### 1. Install
144
136
 
@@ -147,67 +139,40 @@ npm install graph-indexer --save-dev
147
139
  npx graph-indexer init
148
140
  ```
149
141
 
150
- **What `init` does:**
151
- - Auto-detects your IDEs (Cursor, VS Code, Claude Desktop)
152
- - Adds `mcp:index`, `mcp:watch`, `mcp:start` scripts to `package.json`
153
- - Updates `.gitignore` to exclude index files
154
- - Creates `.vscode/` or `.cursor/` configuration (if needed)
155
-
156
- ### 2. Index Your Repository
142
+ `init` auto-detects your IDE (Claude, Cursor, VS Code), adds npm scripts to `package.json`, and updates `.gitignore`.
157
143
 
158
- ```bash
159
- npm run mcp:index
160
- ```
144
+ ### 2. Index your repository
161
145
 
162
- **For semantic search with Ollama:**
146
+ **With semantic search (recommended):**
163
147
  ```bash
164
- # Install Ollama first: https://ollama.ai
148
+ # Install Ollama: https://ollama.ai
165
149
  ollama pull nomic-embed-text
166
-
167
- # Index with vectors
168
- npm run mcp:index # auto-detects Ollama at http://localhost:11434
169
-
170
- # Or specify a different port
171
- OLLAMA_HOST=http://localhost:11435 npm run mcp:index
150
+ npm run mcp:index
172
151
  ```
173
152
 
174
- **For lexical-only (no Ollama):**
153
+ **Lexical-only (no Ollama required):**
175
154
  ```bash
176
155
  INDEXER_EMBEDDINGS=off npm run mcp:index
177
156
  ```
178
157
 
179
- ### 3. Start the MCP Server
180
-
181
- ```bash
182
- npm run mcp:start
183
- ```
184
-
185
- The server will:
186
- - Load the index into memory
187
- - Start the file watcher daemon (auto-updates on file changes)
188
- - Listen for MCP stdio connections
189
-
190
- ### 4. Configure Your IDE
158
+ Lexical-only still achieves Recall@5 = 1.00 for the test suites above.
191
159
 
192
- #### **Claude Desktop** (`~/Library/Application Support/Claude/claude_desktop_config.json`)
160
+ ### 3. Configure your IDE
193
161
 
162
+ **Claude Desktop** (`~/Library/Application Support/Claude/claude_desktop_config.json`):
194
163
  ```json
195
164
  {
196
165
  "mcpServers": {
197
166
  "graph-indexer": {
198
167
  "command": "node",
199
- "args": ["/path/to/your/project/node_modules/graph-indexer/mcp-server.mjs"],
200
- "env": {
201
- "MCP_PROJECT_ROOT": "/path/to/your/project",
202
- "OLLAMA_HOST": "http://localhost:11434"
203
- }
168
+ "args": ["/path/to/project/node_modules/graph-indexer/mcp-server.mjs"],
169
+ "env": { "MCP_PROJECT_ROOT": "/path/to/project" }
204
170
  }
205
171
  }
206
172
  }
207
173
  ```
208
174
 
209
- #### **Cursor** (`.cursor/mcp.json`)
210
-
175
+ **Cursor** (`.cursor/mcp.json`):
211
176
  ```json
212
177
  {
213
178
  "mcpServers": {
@@ -220,8 +185,7 @@ The server will:
220
185
  }
221
186
  ```
222
187
 
223
- #### **VS Code** (`.vscode/mcp.json`)
224
-
188
+ **VS Code** (`.vscode/mcp.json`):
225
189
  ```json
226
190
  {
227
191
  "servers": {
@@ -234,831 +198,224 @@ The server will:
234
198
  }
235
199
  ```
236
200
 
237
- ---
238
-
239
- ## How It Works
240
-
241
- ```mermaid
242
- flowchart LR
243
- INPUT["Source Files\n(.ts .js .py .rs .go .php .css)"]
244
- PARSE["Tree-sitter\nAST Parser"]
245
- CHUNKS["Semantic Chunk\nExtractor\n(functions • classes • impls)"]
246
- DEPS["Bidirectional\nImport Map"]
247
-
248
- TFIDF["TF-IDF Inverted\nIndex"]
249
- EMBED["Ollama\nnomic-embed-text"]
250
- VECTORS["Float32Array\nVector Matrix"]
251
-
252
- GRAPH["Dependency\nGraph"]
253
-
254
- FUSION["RRF Fusion\nLexical + Vector"]
255
- BOOST["Exact-Name\nBoost"]
256
-
257
- TOOLS["MCP Tools"]
258
- SEARCH["search_code"]
259
- CHUNK_GET["get_chunk"]
260
- SKELETON["get_file_skeleton"]
261
- CALLGRAPH["get_call_graph"]
262
-
263
- INPUT --> PARSE --> CHUNKS --> TFIDF & DEPS
264
- CHUNKS --> EMBED --> VECTORS
265
- TFIDF --> FUSION
266
- VECTORS --> FUSION
267
- DEPS --> GRAPH
268
- FUSION --> BOOST --> TOOLS
269
- TOOLS --> SEARCH & CHUNK_GET & SKELETON & CALLGRAPH
270
- GRAPH --> TOOLS
271
- ```
272
-
273
- ### Three-Step Process
201
+ ### 4. Add the agent system prompt
274
202
 
275
- 1. **Parse** (~50–100 ms per 10k files)
276
- - Tree-sitter builds AST for each file
277
- - Extract all named chunks (functions, classes, methods, exports)
278
- - Build bidirectional import dependency map
279
-
280
- 2. **Index** (~2 mins for a 3,600-chunk corpus)
281
- - Compute TF-IDF inverted index (lexical search)
282
- - Call Ollama to generate `nomic-embed-text` vectors (~384–768 dimensions)
283
- - Build `Float32Array` dense matrix for cosine similarity search
284
- - Serialize both to disk-efficient JSON + binary formats
285
-
286
- 3. **Search**
287
- - Query arrives: "Find the token validation function"
288
- - Lexical search via TF-IDF (instant)
289
- - Vector search via `Float32Array` cosine (1–5 ms for 1k–10k chunks)
290
- - Merge results using Reciprocal Rank Fusion (RRF) with K=60
291
- - Apply exact-name boost if `exact_tokens` provided
292
- - Return top-k results with topology (calls, imports, imported-by)
293
-
294
- ---
203
+ Copy the contents of [PROMPT.md](./PROMPT.md) into your AI agent's system prompt. This instructs the agent to use graph-indexer tools instead of reading files directly.
295
204
 
296
205
  ---
297
206
 
298
- ## System Requirements
299
-
300
- | Requirement | Details |
301
- | :--- | :--- |
302
- | **Node.js** | v18+ (ES Modules) |
303
- | **Ollama** | Optional. Runs locally for embedding generation. Pull `nomic-embed-text` to enable semantic search. |
304
- | **Disk** | `code-index.json` (metadata) + `code-index.embeddings.bin` (binary float32 vectors). Both are gitignored by default. |
305
-
306
- ### Ollama Setup (Optional but Recommended)
207
+ ## MCP tools
307
208
 
308
- ```bash
309
- # Install Ollama: https://ollama.ai
310
- ollama pull nomic-embed-text
311
- npm run mcp:index # now with semantic vectors
312
- ```
209
+ ### `search_code`
313
210
 
314
- ### Lexical-Only Mode (No Ollama Required)
211
+ Searches the index by natural language query. Returns signature cards for all results, then code bodies according to the `detail` level.
315
212
 
316
- ```bash
317
- INDEXER_EMBEDDINGS=off npm run mcp:index
318
- ```
213
+ | Parameter | Type | Default | Description |
214
+ | :--- | :--- | :--- | :--- |
215
+ | `query` | string | required | Natural language description of the logic to find |
216
+ | `exact_tokens` | string? | — | Exact symbol name — guarantees rank-1 placement |
217
+ | `detail` | `"signatures"` \| `"smart"` \| `"full"` | `"smart"` | Controls how much code is returned |
218
+ | `top_k` | number | `5` | Results to return (1–20) |
219
+ | `include_topology` | boolean | `true` | Include Deps / Used by / Calls in each card |
220
+ | `min_score` | number | `0.3` | Minimum score threshold |
221
+ | `token_budget` | number? | auto | Token budget for code bodies |
319
222
 
320
- Recall@3 remains 100% for exact-name queries. Semantic recall (concept-to-code) degrades to lexical-only; still effective for most programming tasks.
223
+ **`detail` levels:**
321
224
 
322
- ### Custom Ollama Host
225
+ | Value | Per-result cost | What you get |
226
+ | :--- | ---: | :--- |
227
+ | `"signatures"` | ~20 tokens | Name, type, params, return type, topology only |
228
+ | `"smart"` (default) | ~150 tokens | Signature + only lines relevant to the query |
229
+ | `"full"` | ~300 tokens | Signature + complete source body |
323
230
 
324
- ```bash
325
- # Non-standard port
326
- OLLAMA_HOST=http://localhost:11435 npm run mcp:index
327
-
328
- # Remote server
329
- OLLAMA_HOST=http://192.168.1.100:11434 npm run mcp:index
330
- ```
231
+ Start with `"signatures"`, escalate only when you need implementation details.
331
232
 
332
233
  ---
333
234
 
334
- ## MCP Tools Reference
235
+ ### `resolve_symbol`
335
236
 
336
- ### `search_code`
237
+ Looks up any symbol by exact name in O(1) — no search needed.
337
238
 
338
- Hybrid semantic + lexical search. Returns compact signature cards for all results, then fills remaining token budget with code bodies.
239
+ | Parameter | Type | Description |
240
+ | :--- | :--- | :--- |
241
+ | `symbol` | string | Exact function, class, or type name (e.g. `"validateToken"`) |
339
242
 
340
- | Parameter | Type | Default | Description |
341
- | :--- | :--- | :--- | :--- |
342
- | `query` | `string` | — | Natural language description of the logic to find |
343
- | `exact_tokens` | `string?` | — | Exact symbol name to guarantee rank-1 placement |
344
- | `top_k` | `number` | `5` | Number of results (1–20) |
345
- | `min_score` | `number` | `0.3` | Minimum cosine similarity threshold |
346
- | `token_budget` | `number?` | 1500 chars | Estimated token budget for code bodies |
347
- | `include_topology` | `boolean` | `true` | Include `⬇️ Deps` / `⬆️ Used by` / `🔗 Calls` in output |
348
-
349
- **Example response:**
350
- ```
351
- #1 · validateToken [function_declaration]
352
- 📄 src/utils/jwt.ts:14–42 · ID: `a3f9c1b2` · RRF: 0.0321
353
- 💬 Validates and decodes a JWT access token. Throws on expiry.
354
- ⬇️ Deps: src/config/env.ts [JWT_SECRET] | src/utils/errors.ts [AuthError]
355
- ⬆️ Used by: src/middleware/auth.ts, src/routes/api.ts
356
- 🔗 Calls: verify, decodePayload, throwIfExpired
357
- ↩️ Expand body: get_chunk("a3f9c1b2")
358
- ```
243
+ Returns: definition location, type signature, docstring, type references, and cross-file topology. Faster than `search_code` for known names.
359
244
 
360
245
  ---
361
246
 
362
- ## MCP Tools Reference
247
+ ### `get_chunk`
363
248
 
364
- ### `search_code(query, exact_tokens?, top_k?, min_score?, token_budget?, include_topology?)`
365
-
366
- Hybrid semantic + lexical search. Returns ranked chunks with their signatures, dependencies, and call graph. Token budget is distributed across all results.
249
+ Returns the full source code of one chunk by its ID from search results.
367
250
 
368
251
  | Parameter | Type | Default | Description |
369
252
  | :--- | :--- | :--- | :--- |
370
- | `query` | string | (required) | Natural language or descriptive phrase (e.g., "JWT validation") |
371
- | `exact_tokens` | string | — | Exact symbol name for guaranteed rank-1 placement (e.g., "validateToken") |
372
- | `top_k` | number | 5 | Results to return (1–20) |
373
- | `min_score` | number | 0.3 | Minimum RRF score threshold (0.0–1.0) |
374
- | `token_budget` | number | 4000 | Max tokens to spend on code bodies |
375
- | `include_topology` | boolean | true | Include dependencies + call graph in results |
376
-
377
- **Search Strategy:**
378
- 1. Query is embedded via Ollama (or skipped in lexical-only mode)
379
- 2. Parallel search: TF-IDF + vector similarity
380
- 3. Results merged using Reciprocal Rank Fusion (K=60) with weighted fusion
381
- 4. Exact-name token boost applied if provided
382
- 5. Results ranked by combined score
383
-
384
- **Example Usage:**
385
- ```
386
- search_code(
387
- query="JWT payload validation and expiry check",
388
- exact_tokens="validateToken",
389
- top_k=3,
390
- include_topology=true
391
- )
392
- ```
253
+ | `chunk_id` | string | required | ID shown in `search_code` results |
254
+ | `view` | `"full"` \| `"signature"` | `"full"` | `"full"` = complete body; `"signature"` = first line only |
393
255
 
394
- **Example Response:**
395
- ```
396
- #1 · validateToken [function_declaration]
397
- 📄 src/auth/jwt.ts:18–42 · ID: a3f9c1b2 · Score: 0.98
398
- 💬 Validates and decodes JWT token, checks expiry timestamp
399
- ⬇️ Dependencies: src/config/constants.ts [JWT_SECRET], src/utils/errors.ts [TokenExpiredError]
400
- ⬆️ Used by: src/middleware/authMiddleware.ts, src/routes/protected.ts
401
- 🔗 Calls: verify(token), parseJWT(decoded), throwIfExpired(exp)
402
- ℹ️ Expand: get_chunk("a3f9c1b2")
403
-
404
- #2 · parseTokenClaims [function_declaration]
405
- 📄 src/auth/jwt.ts:45–60 · ID: c2b5d4e1 · Score: 0.82
406
- ...
407
- ```
256
+ ---
408
257
 
409
- ### `get_chunk(chunk_id)`
258
+ ### `get_chunk_summary`
410
259
 
411
- Retrieves the full source code of one chunk by its ID (shown in search results).
260
+ Returns the interface of a chunk (signature + docstring + calls) without the body. About 50 tokens vs ~300 for the full body.
412
261
 
413
262
  | Parameter | Type | Description |
414
263
  | :--- | :--- | :--- |
415
- | `chunk_id` | string | The ID from `search_code` results |
264
+ | `chunk_id` | string | ID from `search_code` results |
265
+
266
+ Use when you need to understand what a function does without reading its implementation.
416
267
 
417
- **Use when:** You need the complete body of a function/class found by search (much cheaper than reading the full file).
268
+ ---
418
269
 
419
- ### `get_file_skeleton(file_path)`
270
+ ### `get_file_skeleton`
420
271
 
421
- Returns all top-level exports and definitions in a file with line numbers — no code bodies.
272
+ Returns all top-level exports and definitions in a file with line numbers — no bodies.
422
273
 
423
274
  | Parameter | Type | Description |
424
275
  | :--- | :--- | :--- |
425
- | `file_path` | string | Relative path (e.g., `src/utils/auth.ts`) |
276
+ | `file_path` | string | Relative path (e.g. `src/utils/auth.ts`) |
426
277
 
427
- **Example Response:**
428
- ```
429
- Exports in src/utils/auth.ts:
430
- • validateToken (function) — lines 12–38
431
- • hashPassword (function) — lines 40–55
432
- • comparePassword (function) — lines 57–68
433
- • AuthError (class) — lines 70–90
434
- ```
278
+ Useful for understanding what a file exports before deciding which chunk to read.
435
279
 
436
- **Use when:** You need to understand what a file exports without reading it all (~50 tokens vs 5000).
280
+ ---
437
281
 
438
- ### `get_call_graph(target_function)`
282
+ ### `get_call_graph`
439
283
 
440
284
  Finds all chunks that call a specific function by name.
441
285
 
442
286
  | Parameter | Type | Description |
443
287
  | :--- | :--- | :--- |
444
- | `target_function` | string | Exact function name (e.g., `validateToken`) |
445
-
446
- **Example Response:**
447
- ```
448
- Calls to validateToken:
449
- • src/middleware/authMiddleware.ts:32 — in middleware function
450
- • src/routes/api.ts:128 — in route handler
451
- • src/services/auth.ts:44 — in login service
452
- ```
453
-
454
- **Use when:** Before renaming or modifying a function, know all dependent call sites.
455
-
456
- ### `graph://dependencies/{file_path}` (Resource)
457
-
458
- Fetches the complete import/export topology for a file without search quota.
459
-
460
- **Example:**
461
- ```
462
- graph://dependencies/src/middleware/auth.ts
463
- ```
464
-
465
- **Returns:**
466
- ```
467
- Imports:
468
- src/config/constants.ts [JWT_SECRET]
469
- src/utils/errors.ts [TokenExpiredError]
470
- src/types/auth.ts [AuthRequest]
471
-
472
- Imported by:
473
- src/app.ts
474
- src/routes/protected.ts
475
- ```
476
-
477
- ---
478
-
479
- ## Hybrid RRF Search Algorithm
480
-
481
- The core innovation of graph-indexer is combining dense semantic vectors with sparse lexical signals using Reciprocal Rank Fusion:
482
-
483
- ### Lexical Search: TF-IDF
484
-
485
- Each token is scored using **sublinear term frequency**:
486
-
487
- $$w(t, d) = (1 + \log \text{count}(t, d)) \times \log \frac{N}{n_t}$$
488
-
489
- Where:
490
- - `count(t, d)` = occurrences of term t in chunk d
491
- - `N` = total chunks
492
- - `n_t` = chunks containing term t
493
-
494
- This prevents common keywords (`return`, `const`, `if`) from dominating results.
495
-
496
- ### Vector Search: Cosine Similarity
497
-
498
- Query and chunks are embedded via Ollama `nomic-embed-text` (384–768 dimensions):
499
-
500
- $$\text{sim}(q, d) = \frac{\mathbf{e}_q \cdot \mathbf{e}_d}{|\mathbf{e}_q| \times |\mathbf{e}_d|}$$
501
-
502
- Stored as pre-normalized `Float32Array` for sub-millisecond cosine computation.
503
-
504
- ### Reciprocal Rank Fusion (RRF) with Weighting
505
-
506
- Results from both methods are merged using weighted RRF:
507
-
508
- $$\text{score}(d) = w_\text{lex} \cdot \frac{1}{K + r_\text{lex}(d)} + w_\text{vec} \cdot \frac{1}{K + r_\text{vec}(d)}$$
509
-
510
- **Default weights:**
511
- - `w_lex` = 1.5 (lexical: prioritizes exact terms)
512
- - `w_vec` = 1.0 (semantic: provides context)
513
- - `K` = 60 (rank discount constant)
514
-
515
- **Why this works:**
516
- - Prevents vector results from overwhelming precise lexical matches
517
- - Ensures "exact" queries stay exact while allowing semantic queries
518
- - Tested on 5 production projects: +0.04 mean MRR improvement (0.92 → 0.96)
519
-
520
- ### Exact-Name Boost
521
-
522
- If `exact_tokens` parameter is provided, any chunk whose name exactly matches receives a boost:
523
-
524
- $$\text{score}_\text{final}(d) = \text{score}(d) + \frac{1}{K+1}$$
525
-
526
- This guarantees rank-1 placement for exact symbol name queries.
527
-
528
- ### Demotion Strategies
529
-
530
- The following chunks receive reduced scoring:
531
-
532
- - **Test/spec files** (paths matching `/test|spec|\.test\.|\.spec\.`): 0.25× multiplier
533
- - **Example/tutorial dirs** (paths matching `/example|doc|tutorial`): 0.5× multiplier
534
- - **Expression statements** (standalone expressions, not declarations): 0.8× multiplier
535
-
536
- This prevents test code from shadowing production code.
537
-
538
- ---
288
+ | `target_function` | string | Exact function name (e.g. `"validateToken"`) |
539
289
 
540
- ## Supported Languages
541
-
542
- | Language | Extensions | Chunk Types | Import Resolution |
543
- | :--- | :--- | :--- | :--- |
544
- | TypeScript / TSX | `.ts`, `.tsx` | functions, classes, methods, exports | Absolute & relative paths |
545
- | JavaScript | `.js`, `.jsx`, `.mjs`, `.cjs` | functions, classes, expressions | Absolute & relative paths |
546
- | Python | `.py` | function_definition, class_definition | Relative imports (`.`, `..`) → file paths |
547
- | Rust | `.rs` | fn, struct, enum, trait, impl | `crate::` path resolution |
548
- | Go | `.go` | function_declaration, method, type | Import spec resolution |
549
- | PHP | `.php` | function, class | include/require paths |
550
- | CSS / SCSS | `.css`, `.scss` | rule_set | — |
551
- | Java | `.java` | class, method, interface, constructor, enum | `import_declaration` → package path |
552
- | Kotlin | `.kt`, `.kts` | function, class, object, companion, constructor | `import_header` → package path |
553
- | C# | `.cs` | class, method, interface, constructor, enum, property | `using_directive` → namespace |
554
- | Ruby | `.rb` | method, singleton_method, class, module | `require` / `require_relative` → file path |
290
+ Always call this before modifying an exported function — it shows every call site repo-wide.
555
291
 
556
292
  ---
557
293
 
558
- ## Performance Characteristics
294
+ ### `get_repo_map`
559
295
 
560
- ### Query Latency
296
+ Returns a compact symbol map of the entire codebase ordered by importance (PageRank over the dependency graph — most-imported files first). Useful for orienting in an unfamiliar codebase in ~1,500 tokens.
561
297
 
562
- | Corpus Size | Lexical Search | Vector Search | RRF Merge |
298
+ | Parameter | Type | Default | Description |
563
299
  | :--- | :--- | :--- | :--- |
564
- | 1,000 chunks | <0.5 ms | 0.8 ms | <0.1 ms |
565
- | 5,000 chunks | <0.8 ms | 3.2 ms | <0.1 ms |
566
- | 10,000 chunks | <1.2 ms | 8.5 ms | <0.2 ms |
567
- | 20,000+ chunks | <2 ms | 20–50 ms | <0.3 ms |
568
-
569
- *Measured on V8 with pre-normalized vectors. Latency includes index lookup but not Ollama network RTT.*
570
-
571
- ### Memory Footprint
572
-
573
- - **TF-IDF index:** ~150–200 bytes per chunk
574
- - **Vector matrix:** ~3,072 bytes per chunk at 768 dimensions (4 bytes × 768 floats)
575
- - **Metadata:** ~200–500 bytes per chunk (path, name, docstring refs)
576
-
577
- **Example:** A 3,600-chunk corpus occupies ~12–15 MB in memory.
578
-
579
- ### Token Savings
580
-
581
- Average across 5 test suites: **74.6%** token reduction vs full-file reads.
582
-
583
- ```
584
- Example: Express.js QuerySearch
585
- File read: src/router.js (460 lines) = 5,750 tokens
586
- Indexed: findRoute (65 lines) = 812 tokens
587
- Savings: 86% reduction
588
- ```
589
-
590
- ### Accuracy Over Scale
591
-
592
- | Project | Total Chunks | Recall@5 | Mean MRR | Queries |
593
- | :--- | :--- | :--- | :--- | :--- |
594
- | Axios | 435 | 1.00 | 0.96 | 12 |
595
- | Express | 389 | 1.00 | 1.00 | 14 |
596
- | NestJS | 2,019 | 1.00 | 1.00 | 15 |
597
- | FastAPI | 3,572 | 1.00 | 0.93 | 13 |
598
- | Gin | 1,088 | 1.00 | 0.92 | 11 |
599
- | **Mean** | **7,503** | **1.00** | **0.96** | **65** |
600
-
601
- - **Recall@5:** 100% of relevant chunks found in top 5 results
602
- - **Mean MRR:** Average reciprocal rank of first correct result
603
- - **Query types:** Easy (exact symbol), Medium (multi-token), Hard (semantic mismatch)
300
+ | `path_filter` | string? | — | Only show files whose path contains this string |
301
+ | `max_files` | number | `80` | Maximum files to include |
302
+ | `sort_by` | `"importance"` \| `"path"` | `"importance"` | Sort order |
604
303
 
605
304
  ---
606
305
 
607
- ### Vector Search Performance
608
-
609
- | Corpus size | Pure JS flat scan | HNSW (auto-activated ≥5k) |
610
- | :--- | :--- | :--- |
611
- | 1,000 chunks (dim=768) | 0.9 ms/query | — |
612
- | 5,000 chunks (dim=768) | 4.5 ms/query | 1.2 ms/query |
613
- | 20,000 chunks (dim=768) | 30.5 ms/query | 3.1 ms/query |
614
-
615
- ---
616
-
617
- ## Best Practices for Maximum Recall
618
-
619
- ### 1. Write Comprehensive JSDoc/TSDoc
620
-
621
- The indexer embeds your docstrings alongside code bodies. Better docstrings = better semantic search:
622
-
623
- ```typescript
624
- /**
625
- * Validates and decodes a JWT access token.
626
- *
627
- * Checks token signature, expiry timestamp, and claims payload.
628
- * Throws if token is malformed, expired, or signed with wrong key.
629
- *
630
- * @param token - Raw JWT string from Authorization header
631
- * @param options - Optional validation parameters
632
- * @returns Decoded token payload with user ID, role, and expiry time
633
- * @throws AuthError if validation fails
634
- *
635
- * @example
636
- * const payload = validateToken(req.headers.authorization);
637
- * console.log(payload.userId, payload.role);
638
- */
639
- export function validateToken(token: string, options?: ValidationOptions): TokenPayload { ... }
640
- ```
641
-
642
- ### 2. Use Descriptive, Specific Names
643
-
644
- The exact-name boost ranks any chunk whose name matches `exact_tokens` at position #1. Specific names unlock this:
645
-
646
- ```typescript
647
- // ✅ Exact-name searchable
648
- export const useNotificationPermissionStatus = () => { ... };
649
- export function calculateTripDurationInDays(start: Date, end: Date) { ... };
650
- export async function fetchAndCacheUserProfile(userId: string) { ... }
651
-
652
- // ❌ Generic — impossible to target without semantic search
653
- export const util = () => { ... };
654
- export function process(data: any) { ... }
655
- export async function fetch(path: string) { ... }
656
- ```
657
-
658
- ### 3. Export Named Helpers, Not Anonymous Functions
659
-
660
- Named exports become searchable chunks:
661
-
662
- ```typescript
663
- // ✅ Indexed as chunk
664
- export function validatePaymentMethod(card: Card): boolean {
665
- // validation logic
666
- }
667
-
668
- // ❌ Not indexed (anonymous)
669
- const validate = (card: Card) => { ... };
670
- export default validate;
671
- ```
672
-
673
- ### 4. Add Docstrings to Class Methods
674
-
675
- Even small methods benefit from docstrings:
306
+ ### `list_index_stats`
676
307
 
677
- ```typescript
678
- class PaymentService {
679
- /**
680
- * Charges customer card via Stripe API.
681
- * Throws on declined card or network failure.
682
- */
683
- async chargeCard(cardToken: string, amount: number): Promise<Charge> {
684
- // ...
685
- }
686
-
687
- /**
688
- * Refunds transaction to original payment method.
689
- */
690
- async refundCharge(chargeId: string): Promise<Refund> {
691
- // ...
692
- }
693
- }
694
- ```
695
-
696
- ### 5. Minimize Generic Utility Files
697
-
698
- Files with <5 indexable chunks produce weak search signals:
699
-
700
- ```typescript
701
- // ❌ Won't index well (2 lines)
702
- export const AuthContext = createContext(null);
703
- export const useAuth = () => useContext(AuthContext);
704
-
705
- // ✅ Indexable
706
- /**
707
- * Context for accessing authentication state throughout component tree.
708
- * Provides currentUser, isLoading, error from nearest AuthProvider.
709
- */
710
- export const AuthContext = createContext<AuthContextValue | null>(null);
711
-
712
- /**
713
- * Hook to access authentication state.
714
- * Throws if used outside AuthProvider.
715
- * @returns Authenticated user, loading state, and any auth errors
716
- */
717
- export function useAuth(): AuthContextValue {
718
- const ctx = useContext(AuthContext);
719
- if (!ctx) throw new Error('useAuth must be called inside <AuthProvider>');
720
- return ctx;
721
- }
722
- ```
308
+ Returns index health: chunk count, file count, symbol table size, vector entry count, search mode (hybrid or lexical-only), daemon status, and index age.
723
309
 
724
310
  ---
725
311
 
726
- ## Architecture & Implementation
727
-
728
- ### File Structure
729
-
730
- ```
731
- graph-indexer/
732
- ├── core-engine.mjs # In-memory dual index: TF-IDF + vector matrix
733
- │ # Methods: search(), searchHybrid(), save(), load()
734
- ├── parser-utils.mjs # Tree-sitter AST parsing + Ollama embedding
735
- │ # Supports 11 languages with AST-precision chunking
736
- ├── indexer.mjs # CLI entry: scans repo → parses → embeds → saves
737
- ├── watch-daemon.mjs # Chokidar file watcher for incremental index updates
738
- ├── mcp-server.mjs # MCP stdio server: exposes search_code, get_chunk, etc.
739
- ├── init.mjs # Installation & IDE configuration helper
740
- ├── PROMPT.md # Agent system prompt for AI coding agents
741
- ├── tests/
742
- │ ├── harness.mjs # Shared test execution framework
743
- │ ├── run.mjs # Lexical-only evaluation harness
744
- │ ├── run-embeddings.mjs # Hybrid (Ollama) evaluation harness
745
- │ ├── metrics.mjs # Recall, MRR, latency calculations
746
- │ ├── suites/
747
- │ │ ├── axios.mjs # 435-chunk corpus, 12 queries
748
- │ │ ├── express-js.mjs # 389-chunk corpus, 14 queries
749
- │ │ ├── fastapi.mjs # 3,572-chunk corpus, 13 queries
750
- │ │ ├── gin.mjs # 1,088-chunk corpus, 11 queries
751
- │ │ └── nestjs.mjs # 2,019-chunk corpus, 15 queries
752
- │ ├── reports/ # JSON performance reports
753
- │ └── ...
754
- ├── package.json
755
- ├── README.md
756
- └── LICENSE
757
- ```
758
-
759
- ### Indexing Pipeline
760
-
761
- 1. **Discovery** (15–50 ms)
762
- - Recursively scan directory
763
- - Apply `.gitignore` filters (vendor, node_modules)
764
- - Collect source file paths
765
-
766
- 2. **Parsing** (50–500 ms)
767
- - Tree-sitter parses each file
768
- - Extract semantic chunks: functions, classes, methods, exports
769
- - Build bidirectional import dependency graph
312
+ ### `graph://dependencies/{file_path}` (resource)
770
313
 
771
- 3. **Embedding** (1–10 mins for 5k chunks)
772
- - Batch chunks into 32-chunk requests
773
- - Call Ollama `/api/embed` with "search_document:" prefix
774
- - Store vectors as `Float32Array` binary matrix
775
-
776
- 4. **Indexing** (20–50 ms)
777
- - Build TF-IDF inverted index (token → chunk IDs)
778
- - Compute IDF for each term
779
- - Build dependency graph adjacency maps
780
-
781
- 5. **Serialization** (100–300 ms)
782
- - Save metadata to `code-index.json`
783
- - Write vectors to `code-index.embeddings.bin` (binary float32)
784
- - Atomic `tmp → rename` for crash safety
314
+ Returns the full bidirectional dependency topology for a file: what it imports and what imports it.
785
315
 
786
316
  ---
787
317
 
788
- ## Configuration & Tuning
318
+ ## Configuration
789
319
 
790
- ### Environment Variables
320
+ ### Environment variables
791
321
 
792
322
  | Variable | Default | Description |
793
323
  | :--- | :--- | :--- |
324
+ | `MCP_PROJECT_ROOT` | `process.cwd()` | Project root for import resolution |
794
325
  | `OLLAMA_HOST` | `http://localhost:11434` | Ollama API endpoint |
795
- | `INDEXER_EMBEDDINGS` | — | Set to `off` to disable embeddings (lexical-only mode) |
796
- | `MCP_PROJECT_ROOT` | `process.cwd()` | Project root path for relative import resolution |
797
- | `INDEXER_LOG_LEVEL` | `info` | Logging verbosity: `debug`, `info`, `warn`, `error` |
798
-
799
- ### Advanced Tuning
800
-
801
- Edit `core-engine.mjs` to adjust these constants:
326
+ | `INDEXER_EMBEDDINGS` | — | Set to `off` to disable vector embeddings |
802
327
 
803
- ```javascript
804
- // Weighted RRF parameters
805
- const RRF_K = 60; // Discount constant
806
- const LEX_WEIGHT = 1.5; // Lexical weight (1.0–2.0)
807
- const VEC_WEIGHT = 1.0; // Vector weight (0.5–1.5)
328
+ ### Custom Ollama host
808
329
 
809
- // Demotion multipliers
810
- const TEST_FILE_DEMOTION = 0.25; // Test files: 0.25× score
811
- const EXAMPLE_DIR_DEMOTION = 0.5; // Example dirs: 0.5× score
812
- const EXPRESSION_DEMOTION = 0.8; // Standalone expressions: 0.8× score
813
-
814
- // Name boost
815
- const EXACT_NAME_TOKEN_BOOST = 1.4; // Boost for exact token matches
816
- ```
817
-
818
- **Guidelines:**
819
- - Increase `LEX_WEIGHT` if exact-symbol queries underperform
820
- - Increase `VEC_WEIGHT` if semantic queries need more influence
821
- - Lower `TEST_FILE_DEMOTION` if tests contain important reference code
822
-
823
- ---
824
-
825
- ## Troubleshooting
826
-
827
- ### Q: Index takes forever to build
828
- **Symptom:** `npm run mcp:index` hangs on "Embedding chunks..."
829
-
830
- **Solutions:**
831
- 1. Verify Ollama is running: `curl http://localhost:11434/api/tags`
832
- 2. Check your internet/network (Ollama might be on remote host)
833
- 3. Use lexical-only mode: `INDEXER_EMBEDDINGS=off npm run mcp:index`
834
- 4. Monitor Ollama memory: large corpora may need 4–8GB
835
-
836
- ### Q: Search results are irrelevant
837
- **Symptom:** Queries return wrong chunks even though correct chunk exists
838
-
839
- **Solutions:**
840
- 1. Check if chunk is indexed: `grep "chunk_name" code-index.json`
841
- 2. If missing, ensure chunk has a descriptive name (see Best Practices)
842
- 3. Try with `exact_tokens`: `search_code(query, exact_tokens="chunk_name")`
843
- 4. Increase `top_k` to see more candidates
844
- 5. Verify Ollama model is loaded: `ollama list` should show `nomic-embed-text`
845
-
846
- ### Q: Ollama connection fails
847
- **Symptom:** Error: "Failed to connect to Ollama at http://localhost:11434"
848
-
849
- **Solutions:**
850
330
  ```bash
851
- # 1. Verify Ollama is running
852
- ollama list
853
-
854
- # 2. Check port (might be on different port)
855
- curl http://localhost:11434/api/tags # default
856
- curl http://localhost:11435/api/tags # alternative
331
+ # Non-default port
332
+ OLLAMA_HOST=http://localhost:11435 npm run mcp:index
857
333
 
858
- # 3. Specify different host
334
+ # Remote host
859
335
  OLLAMA_HOST=http://192.168.1.100:11434 npm run mcp:index
860
-
861
- # 4. Fallback to lexical-only
862
- INDEXER_EMBEDDINGS=off npm run mcp:index
863
336
  ```
864
337
 
865
- ### Q: MCP server won't start
866
- **Symptom:** Error when connecting IDE to MCP server
867
-
868
- **Solutions:**
869
- 1. Verify index exists: `ls -la code-index.json`
870
- 2. Check MCP_PROJECT_ROOT: `echo $MCP_PROJECT_ROOT` should point to project
871
- 3. Test stdio: `node mcp-server.mjs` (should wait for input)
872
- 4. Check IDE config paths are absolute, not relative
873
-
874
- ### Q: Memory usage is too high
875
- **Symptom:** Node process consuming 2GB+
876
-
877
- **Solutions:**
878
- - Large corpora (10k+ chunks) require 1–2GB
879
- - Use `--max-old-space-size` flag:
880
- ```bash
881
- node --max-old-space-size=4096 node_modules/graph-indexer/mcp-server.mjs
882
- ```
883
- - Consider splitting into multiple projects
884
- - Use lexical-only mode to reduce vector matrix size by 75%
885
-
886
- ### Q: Incremental updates aren't working
887
- **Symptom:** File changes not reflected in search results
888
-
889
- **Solutions:**
890
- 1. Manually re-index: `npm run mcp:index`
891
- 2. Check watcher is running: `npm run mcp:watch`
892
- 3. Verify file changes are within project root (check `.gitignore`)
893
- 4. Restart MCP server after index updates
894
-
895
338
  ---
896
339
 
897
- ## FAQ
898
-
899
- ### How much faster is graph-indexer vs file reads?
900
-
901
- - **Token usage:** 84–90% reduction (see test results above)
902
- - **Latency:** 1–50 ms per query vs 500ms+ for file I/O + token counting
903
- - **Accuracy:** Same relevance, but 100% recall@5 with better context
904
-
905
- ### Can I use it without Ollama?
340
+ ## Supported languages
906
341
 
907
- Yes. Run with `INDEXER_EMBEDDINGS=off npm run mcp:index`. You get:
908
- - Pure TF-IDF lexical search
909
- - Exact-name queries still work perfectly
910
- - Semantic queries degrade to string matching
911
- - 1.00 recall@5 for exact-symbol searches
912
-
913
- ### Does it support monorepos?
914
-
915
- Partially. The indexer creates one `code-index.json` per directory. For monorepos:
916
- 1. Index each service separately
917
- 2. Or create a wrapper that merges multiple indexes
918
- 3. Or use a single MCP server per service
919
-
920
- ### What file types are indexed?
342
+ | Language | Extensions | Chunk types extracted |
343
+ | :--- | :--- | :--- |
344
+ | TypeScript / TSX | `.ts`, `.tsx` | functions, classes, methods, exports |
345
+ | JavaScript | `.js`, `.jsx`, `.mjs`, `.cjs` | functions, classes, expressions |
346
+ | Python | `.py` | function_definition, class_definition |
347
+ | Go | `.go` | function_declaration, method, type |
348
+ | Rust | `.rs` | fn, struct, enum, trait, impl |
349
+ | Java | `.java` | class, method, interface, constructor, enum |
350
+ | Kotlin | `.kt`, `.kts` | function, class, object, companion |
351
+ | C# | `.cs` | class, method, interface, property, enum |
352
+ | Ruby | `.rb` | method, class, module |
353
+ | PHP | `.php` | function, class |
354
+ | CSS / SCSS | `.css`, `.scss` | rule_set |
921
355
 
922
- See Supported Languages table. Only recognized languages are parsed. Other files (configs, docs) are skipped.
356
+ ---
923
357
 
924
- ### Can I make the index portable?
358
+ ## Requirements
925
359
 
926
- Yes. Both `code-index.json` and `code-index.embeddings.bin` are self-contained. Copy them to another machine and start the MCP server — it works immediately.
360
+ | | |
361
+ | :--- | :--- |
362
+ | **Node.js** | v18+ (ES Modules) |
363
+ | **Ollama** | Optional — enables semantic search. Pull `nomic-embed-text`. |
364
+ | **Disk** | `code-index.json` + `code-index.embeddings.bin` (both gitignored by default) |
927
365
 
928
- ### How does performance scale?
366
+ ---
929
367
 
930
- - Parsing: O(file_size_total) — linear
931
- - TF-IDF indexing: O(tokens × unique_terms) — sublinear with repetition
932
- - Vector search: O(chunks) flat scan, O(log chunks) with HNSW index
933
- - For 5k chunks: ~2 mins to index, 1–5ms per query
368
+ ## Troubleshooting
934
369
 
935
- ---
370
+ **Index takes too long to build**
371
+ - Verify Ollama is running: `curl http://localhost:11434/api/tags`
372
+ - Fall back to lexical-only: `INDEXER_EMBEDDINGS=off npm run mcp:index`
936
373
 
937
- ## Index Files Reference
374
+ **Search returns irrelevant results**
375
+ - Use `exact_tokens` for known symbol names
376
+ - Verify the chunk is indexed with `list_index_stats()`
377
+ - Increase `top_k` to see more candidates
938
378
 
939
- | File | Format | Purpose | Size | Portable |
940
- | :--- | :--- | :--- | :--- | :--- |
941
- | `code-index.json` | JSON | Chunk metadata, TF-IDF inverted index, dependency graph | ~200 KB / 1k chunks | ✅ Yes |
942
- | `code-index.embeddings.bin` | Binary float32 | Vector embeddings (pre-normalized for cosine) | ~3 MB / 1k @768-dim | ✅ Yes |
379
+ **MCP server won't connect**
380
+ - Verify `MCP_PROJECT_ROOT` points to the indexed directory
381
+ - Check that `code-index.json` exists: `ls -la code-index.json`
382
+ - Test the server manually: `node mcp-server.mjs`
943
383
 
944
- Both are gitignored by default. Regenerate with `npm run mcp:index` whenever you want a fresh index.
384
+ **Results are stale after file changes**
385
+ - The background daemon auto-updates; if changes aren't reflected, run `npm run mcp:index` manually
386
+ - Check daemon status with `list_index_stats()`
945
387
 
946
388
  ---
947
389
 
948
- ## Contributing & Development
949
-
950
- ### Building from Source
390
+ ## Development
951
391
 
952
392
  ```bash
953
393
  git clone https://github.com/MaquinaTech/graph-indexer.git
954
394
  cd graph-indexer
955
395
  npm install
956
- npm run mcp:index # index this repo
957
- npm run test # run test suite
958
- npm run mcp:start # start MCP server for development
396
+ npm run mcp:index # index this repo
397
+ npm run test # run test suite (lexical-only)
398
+ npm run mcp:start # start MCP server
959
399
  ```
960
400
 
961
- ### Running Tests
962
-
963
- ```bash
964
- # Test lexical-only mode
965
- npm run test
966
-
967
- # Test hybrid mode (requires Ollama)
968
- npm run test:embeddings
969
-
970
- # Generate performance report
971
- npm run test:report
972
- ```
973
-
974
- ### Project Philosophy
975
-
976
- 1. **Local First:** No external dependencies beyond Ollama (optional)
977
- 2. **AST-Precise:** Tree-sitter for exact code boundaries, not fuzzy chunks
978
- 3. **Hybrid Search:** Combine lexical + semantic for both accuracy and recall
979
- 4. **Zero Database:** In-memory only, no infrastructure required
980
- 5. **Production-Grade:** 100% recall on 5 production projects
981
-
982
- ### Roadmap
983
-
984
- - [ ] HNSW indexing for vector search >20k chunks
985
- - [ ] Incremental embedding updates (batch new files only)
986
- - [ ] Language server protocol (LSP) integration
987
- - [ ] Web UI for search visualization
988
- - [ ] Multi-repo federation mode
989
- - [ ] Custom embedding model support
990
-
991
401
  ---
992
402
 
993
- ## License
994
-
995
- Released under the [MIT License](LICENSE).
996
-
997
- Copyright (c) 2026 MaquinaTech. Free to use, modify, and distribute in both open-source and commercial projects.
998
-
999
- ---
403
+ ## Best practices for better search quality
1000
404
 
1001
- ## Author & Support
405
+ **Write descriptive docstrings.** The indexer embeds docstrings alongside code. Better documentation directly improves semantic search quality.
1002
406
 
1003
- Built and maintained by **MaquinaTech**.
407
+ **Use specific function names.** Functions with exact, descriptive names (e.g. `fetchAndCacheUserProfile`) hit the exact-name boost (2.0×) and return as rank-1 results when queried by name.
1004
408
 
1005
- - **GitHub:** [github.com/MaquinaTech](https://github.com/MaquinaTech)
1006
- - **NPM:** [npmjs.com/package/graph-indexer](https://www.npmjs.com/package/graph-indexer)
1007
- - **Issues & Feedback:** [github.com/MaquinaTech/graph-indexer/issues](https://github.com/MaquinaTech/graph-indexer/issues)
409
+ **Export named functions, not anonymous ones.** Anonymous default exports cannot be targeted by `resolve_symbol` and score lower in name-boost ranking.
1008
410
 
1009
- Issues, pull requests, and feature discussions are welcome. This project started as a solution to the context bloat problem in AI coding agents — we'd love to hear how you're using it.
411
+ **Avoid catch-all utility files.** Large files that mix unrelated utilities produce weaker per-chunk signals. One responsibility per module indexes more cleanly.
1010
412
 
1011
413
  ---
1012
414
 
1013
- ## Test Methodology
1014
-
1015
- All results in this README come from rigorous evaluation across 5 production-grade open-source projects:
1016
-
1017
- ### Test Suites
1018
-
1019
- 1. **Axios v1.6.0** (JavaScript HTTP client)
1020
- - 435 total chunks, 12 ground-truth queries
1021
- - Coverage: request/response handling, middleware, interceptors
1022
-
1023
- 2. **Express.js v4.18.2** (JavaScript web framework)
1024
- - 389 total chunks, 14 ground-truth queries
1025
- - Coverage: routing, middleware, response handling
1026
-
1027
- 3. **NestJS v10.4.9** (TypeScript framework)
1028
- - 2,019 total chunks, 15 ground-truth queries
1029
- - Coverage: decorators, dependency injection, modules, controllers
1030
-
1031
- 4. **FastAPI v0.103.0** (Python async framework)
1032
- - 3,572 total chunks, 13 ground-truth queries
1033
- - Coverage: route handlers, dependency injection, middleware
1034
-
1035
- 5. **Gin v1.9.1** (Go web framework)
1036
- - 1,088 total chunks, 11 ground-truth queries
1037
- - Coverage: routing, middleware, context handling
1038
-
1039
- ### Evaluation Metrics
1040
-
1041
- - **Recall@k:** Percentage of test queries where the correct answer appears in top-k results
1042
- - **MRR (Mean Reciprocal Rank):** Average of 1/rank for first correct result (higher is better; 1.0 = always rank-1)
1043
- - **Latency:** Time to execute query (lexical + vector search + RRF fusion)
1044
- - **Token savings:** Tokens used for surgical chunk vs full-file reads
1045
-
1046
- ### Query Difficulties
1047
-
1048
- - **Easy (3–5 per suite):** Exact symbol names where recall-1 is guaranteed
1049
- - **Medium (5–7 per suite):** Multi-token natural language descriptions
1050
- - **Hard (4 per suite):** Semantic concepts with intentional vocabulary mismatch
1051
-
1052
- ### Consistency
1053
-
1054
- All test suites use:
1055
- - Same TF-IDF weighting and RRF parameters
1056
- - Same Ollama model (`nomic-embed-text` v1)
1057
- - Same demotion strategies (test files, examples, expressions)
1058
- - Same exact-name boost multiplier (1.4×)
415
+ ## License
1059
416
 
1060
- Results are reproducible. Run `npm run test:embeddings --full-report` to regenerate all metrics for your machine.
417
+ Released under the [MIT License](LICENSE). Copyright (c) 2026 MaquinaTech.
1061
418
 
1062
419
  ---
1063
420
 
1064
- **Read more:** See [PROMPT.md](./PROMPT.md) for detailed agent integration instructions.
421
+ Built by [MaquinaTech](https://github.com/MaquinaTech) · [Issues](https://github.com/MaquinaTech/graph-indexer/issues) · [npm](https://www.npmjs.com/package/graph-indexer)