graph-indexer 1.0.1 → 1.0.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,144 +1,97 @@
1
1
  # graph-indexer
2
2
 
3
3
  <p align="center">
4
- <strong>Zero-Database • Air-Gapped • AST-Precise • Hybrid Semantic Search</strong><br>
5
- <em>Production-grade MCP code indexer for AI coding agents — surgical context retrieval that cuts token costs by up to 84% and improves code accuracy.</em>
4
+ <strong>Zero-Database · Air-Gapped · AST-Precise · Hybrid Search</strong><br>
5
+ <em>An MCP server that lets AI coding agents search your codebase by concept instead of reading files — reducing context usage by 65–88% per query.</em>
6
6
  </p>
7
7
 
8
8
  <p align="center">
9
9
  <a href="https://www.npmjs.com/package/graph-indexer"><img src="https://img.shields.io/npm/v/graph-indexer?color=blue&style=flat-square" alt="npm version"></a>
10
10
  <a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-yellow.svg?style=flat-square" alt="License: MIT"></a>
11
11
  <a href="https://nodejs.org"><img src="https://img.shields.io/badge/Node.js-%3E%3D18-green?style=flat-square" alt="Node.js 18+"></a>
12
- <a href="#test-results"><img src="https://img.shields.io/badge/Recall@5-100%25-brightgreen?style=flat-square" alt="Recall@5 100%"></a>
12
+ <a href="#results"><img src="https://img.shields.io/badge/Recall@5-100%25-brightgreen?style=flat-square" alt="Recall@5 100%"></a>
13
13
  </p>
14
14
 
15
15
  ---
16
16
 
17
- ## The Problem: Token Waste and Lost Context
17
+ ## Table of Contents
18
18
 
19
- When you use an AI coding agent (Claude, Cursor, GitHub Copilot) on a real codebase, the default workflow wastes **80–90% of your context window**:
19
+ - [What it does](#what-it-does)
20
+ - [Results](#results)
21
+ - [Getting started](#getting-started)
22
+ - [MCP tools](#mcp-tools)
23
+ - [Configuration](#configuration)
24
+ - [Supported languages](#supported-languages)
25
+ - [How it works](#how-it-works)
26
+ - [Best practices](#best-practices)
27
+ - [Troubleshooting](#troubleshooting)
28
+ - [Development](#development)
29
+ - [License](#license)
20
30
 
21
- ```
22
- Agent wants: "Find the payment processing function"
23
- Standard workflow:
24
- 1. Read file: src/payments/handlers.ts (850 lines) → ~10,625 tokens
25
- 2. Read file: src/config/stripe.ts (340 lines) → ~4,250 tokens
26
- 3. Read file: src/types/payment.ts (120 lines) → ~1,500 tokens
27
- Total: ~16,375 tokens to find ONE function worth ~200 tokens ⚠️
28
-
29
- Reality for complex refactors:
30
- • 5–10 such reads per session
31
- • ~80,000–160,000 tokens just for context gathering
32
- • At Claude 3.5 Sonnet pricing, that's ~$0.48–$0.96 per refactor
33
- • AND the LLM accuracy *decreases* — it gets lost in the noise
34
- ```
31
+ ---
35
32
 
36
- Three compounding problems:
33
+ ## What it does
37
34
 
38
- 1. **Token burn:** File reads consume 10–50× more tokens than necessary
39
- 2. **Lost in the Middle:** LLMs lose accuracy when relevant code is buried in 1000-line files
40
- 3. **Topology blindness:** The agent can't see cross-file dependencies. Refactors break silently.
35
+ graph-indexer pre-indexes your codebase into an AST-precise search index and exposes it as an MCP server. Instead of reading full files, AI agents call `search_code("payment validation")` and get back the exact functions that match — with type signatures, docstrings, dependency graph, and call sites — in a fraction of the tokens.
41
36
 
42
- ## The Solution: Surgical, AST-Precise Retrieval
37
+ It runs entirely on your machine. No external database, no cloud APIs, no telemetry.
43
38
 
44
- **graph-indexer** is a local MCP server that replaces file reads with **structural code queries**. Instead of reading entire files, your agent retrieves exactly the code it needs.
39
+ **Token comparison:**
45
40
 
46
41
  ```
42
+ Agent task: "Add error handling to the payment processing flow"
43
+
44
+ Without graph-indexer:
45
+ readFile("src/payments/service.ts") → 8,400 tokens
46
+ readFile("src/payments/handlers.ts") → 6,200 tokens
47
+ readFile("src/types/payment.ts") → 1,800 tokens
48
+ Total: ~ 16,400 tokens
49
+
47
50
  With graph-indexer:
48
- 1. Search: "payment processing" → returns [processPayment, validateCard, chargeCustomer]
49
- 2. One get_chunk() call → ~300 tokens
50
- 3. Dependency topology included → can see full call chain
51
- Total: ~300 tokens vs ~16,375 tokens — 98% savings on this query
51
+ search_code("payment processing") → 650 tokens (5 exact chunks)
52
+ get_chunk("chunk_id") → 280 tokens (full function body)
53
+ Total: ~ 930 tokens
52
54
  ```
53
55
 
54
- ### Key Benefits
55
-
56
- | Benefit | Impact | Typical Savings |
57
- | :--- | :--- | :--- |
58
- | **Surgical chunking** | AST-precise: entire functions, never split mid-logic | 84–90% token reduction |
59
- | **Hybrid search** | TF-IDF + semantic vectors (Ollama) rank results intelligently | +0.04 MRR, better relevance |
60
- | **Zero infrastructure** | Pure in-memory index, no external databases | Instant startup, zero DevOps |
61
- | **Air-gapped** | Works entirely locally or with local Ollama | 100% privacy, no API keys |
62
- | **Dependency awareness** | Every result includes topology (who calls this, what it imports) | Prevents broken refactors |
63
- | **Language support** | JavaScript, TypeScript, Python, Rust, Go, PHP, CSS, Java, Kotlin, C#, Ruby | Single tool for polyglot codebases |
64
- | **Fault tolerance** | Degrades gracefully to TF-IDF if Ollama offline | Always works |
56
+ The difference compounds across a session. Agents that read files also lack topology — they can't see which other functions call the one they're modifying, so they miss side effects.
65
57
 
66
58
  ---
67
59
 
68
- ## Test Results: 100% Recall, 96% MRR
69
-
70
- graph-indexer was evaluated on 5 production-grade open-source projects totaling **7,503 chunks** across JavaScript, TypeScript, Python, Go, and Ruby.
60
+ ## Results
71
61
 
72
- ### Test Suite Performance
62
+ Measured across 5 production open-source codebases (7,503 AST chunks, 70 ground-truth queries across 3 difficulty levels).
73
63
 
74
- | Project | Language | Chunks | Lexical Recall@5 | Hybrid Recall@5 | MRR Improvement |
75
- | :--- | :--- | :--- | :--- | :--- | :--- |
76
- | Axios v1.6.0 | JavaScript | 435 | 1.00 | 1.00 | — |
77
- | Express 4.18.2 | JavaScript | 389 | 1.00 | 1.00 | +0.06 |
78
- | NestJS v10.4.9 | TypeScript | 2,019 | 1.00 | 1.00 | **+0.14** |
79
- | FastAPI 0.103.0 | Python | 3,572 | 1.00 | 1.00 | +0.01 |
80
- | Gin v1.9.1 | Go | 1,088 | 1.00 | 1.00 | +0.01 |
81
- | **Mean** | — | **7,503** | **1.00** | **1.00** | **+0.04** |
64
+ ### Token savings per query
82
65
 
83
- **What this means:**
84
- - ✅ **100% of relevant code found in top 5 results** across every suite and language
85
- - ✅ **All queries return correct results rank-1** when using exact symbol names (`exact_tokens`)
86
- - ✅ **Mean reciprocal rank improves to 0.96** with semantic search enabled
87
- - ✅ **Hybrid mode adds +0.14 MRR** on NestJS (large polyglot TypeScript codebase)
88
- - ✅ **Graceful degradation:** lexical-only mode still achieves 1.00 recall@5 for exact-name queries
66
+ | Project | Language | Without (full files) | With graph-indexer | Savings | Latency |
67
+ | :--- | :--- | ---: | ---: | ---: | ---: |
68
+ | Axios v1.6.0 | JavaScript | ~2,290 tok | ~655 tok | **71.4%** | 0.2 ms |
69
+ | Express 4.18.2 | JavaScript | ~7,330 tok | ~850 tok | **88.4%** | 0.2 ms |
70
+ | NestJS v10.4.9 | TypeScript | ~2,535 tok | ~900 tok | **64.5%** | 0.7 ms |
71
+ | FastAPI 0.103.0 | Python | ~4,030 tok | ~685 tok | **83.0%** | 1.0 ms |
72
+ | Gin v1.9.1 | Go | ~2,530 tok | ~450 tok | **82.2%** | 0.4 ms |
73
+ | **Mean** | | | | **77.9%** | **0.5 ms** |
89
74
 
90
- ### Query Breakdown
75
+ > "Full files" = tokens in the source files containing the top-5 results. "With graph-indexer" = tokens in the returned chunk snippets only. Measured at 4 chars/token. NestJS is 700k tokens of source — larger than most LLMs' context windows.
91
76
 
92
- 14–15 ground-truth queries per suite covering:
93
- - **Easy (3–5):** Direct function/class names (`HTTPException`, `Router`, `RequestValidator`)
94
- - **Medium (5–7):** Multi-token natural language (`"response send body"`, `"validate request body"`)
95
- - **Hard (4):** Semantic concepts with vocabulary mismatch (`"dependency injection container"`, `"content negotiation"`)
77
+ ### Search quality
96
78
 
97
- **Result:** Even the hardest semantic queries achieve 100% recall@5 when hybrid search is enabled.
79
+ | Project | Chunks | Recall@1 | Recall@3 | Recall@5 | MRR |
80
+ | :--- | ---: | ---: | ---: | ---: | ---: |
81
+ | Axios v1.6.0 | 435 | 0.86 | 1.00 | **1.00** | 0.92 |
82
+ | Express 4.18.2 | 389 | 0.93 | 0.93 | **1.00** | 0.95 |
83
+ | NestJS v10.4.9 | 2,019 | 1.00 | 1.00 | **1.00** | **1.00** |
84
+ | FastAPI 0.103.0 | 3,572 | 0.93 | 1.00 | **1.00** | 0.96 |
85
+ | Gin v1.9.1 | 1,088 | 0.85 | 1.00 | **1.00** | 0.92 |
86
+ | **Mean** | **7,503** | **0.91** | **0.99** | **1.00** | **0.95** |
98
87
 
99
- ### Token Savings
100
-
101
- Average **74.6% token reduction** across all queries when using graph-indexer vs full-file reads:
102
-
103
- ```
104
- Query: "Find the JWT validation logic"
105
- File-based approach: src/auth.ts (850 lines) → 10,625 tokens
106
- graph-indexer: validateToken chunk (35 lines) → 437 tokens
107
- Savings: 95.9% on this query
108
- ```
88
+ **Recall@5 = 1.00** across every project: the correct answer is always in the top 5. **MRR = 0.95**: the correct answer is rank 1 or 2 on average.
109
89
 
110
90
  ---
111
91
 
112
- ## Why graph-indexer Beats Alternatives
113
-
114
- ### vs. Standard RAG (ChromaDB, Pinecone, Weaviate)
115
-
116
- | Feature | **graph-indexer** | Standard RAG |
117
- | :--- | :--- | :--- |
118
- | **Infrastructure** | Zero-DB, pure in-memory `Map` | Requires external vector database |
119
- | **Privacy** | 100% local + air-gapped | Often sends to cloud APIs |
120
- | **Chunking strategy** | AST-precise (entire functions) | Naive token-count splits — breaks logic in half |
121
- | **Hybrid search** | TF-IDF ⊕ vectors w/ RRF fusion | Dense-only or BM25-only |
122
- | **Code structure** | Bidirectional dependency graph | No topology awareness |
123
- | **Startup time** | <100ms (memory-map index) | 5–30s (load from DB) |
124
- | **Cost** | $0 (local) or $5–50/month (Ollama CPU) | $0–500/month per million queries |
125
- | **Language support** | 11 languages with AST precision | Language-agnostic (no structure) |
92
+ ## Getting started
126
93
 
127
- ### vs. File-Based Search (git grep, ripgrep)
128
-
129
- graph-indexer is semantic + structural, not just text matching:
130
-
131
- | Query | ripgrep | graph-indexer |
132
- | :--- | :--- | :--- |
133
- | `grep -r "ValidationError"` | Returns every line mentioning `ValidationError` (thousands of results) | Returns the definition + all places it's raised, ranked by relevance |
134
- | `grep -r "async function"` | Every async function in the codebase | Only async functions matching semantic context of your query |
135
- | Refactor safety | No way to know if rename is safe | Call graph shows every place the symbol is used |
136
-
137
- ---
138
-
139
- ---
140
-
141
- ## Quick Start
94
+ **Requirements:** Node.js v18+ · Ollama (optional, for semantic search)
142
95
 
143
96
  ### 1. Install
144
97
 
@@ -147,918 +100,330 @@ npm install graph-indexer --save-dev
147
100
  npx graph-indexer init
148
101
  ```
149
102
 
150
- **What `init` does:**
151
- - Auto-detects your IDEs (Cursor, VS Code, Claude Desktop)
152
- - Adds `mcp:index`, `mcp:watch`, `mcp:start` scripts to `package.json`
153
- - Updates `.gitignore` to exclude index files
154
- - Creates `.vscode/` or `.cursor/` configuration (if needed)
155
-
156
- ### 2. Index Your Repository
103
+ `init` auto-detects your IDE (Claude, Cursor, VS Code), adds npm scripts to `package.json`, updates `.gitignore`, and opens an interactive language selector:
157
104
 
158
- ```bash
159
- npm run mcp:index
160
105
  ```
106
+ ⚙️ Select languages (Arrows/Tab: move, Space: toggle, Enter: confirm):
161
107
 
162
- **For semantic search with Ollama:**
163
- ```bash
164
- # Install Ollama first: https://ollama.ai
165
- ollama pull nomic-embed-text
108
+ ❯ ◯ TypeScript / TSX .ts, .tsx
109
+ ◯ JavaScript .js, .jsx, .mjs, .cjs
110
+ ◯ Python .py
111
+ ◯ Go .go
112
+ ◯ Rust .rs
113
+ ◯ PHP .php
114
+ ◯ Java .java
115
+ ◯ Kotlin .kt, .kts
116
+ ◯ C# .cs
117
+ ◯ Ruby .rb
118
+ ◯ CSS / SCSS .css, .scss
119
+ ```
166
120
 
167
- # Index with vectors
168
- npm run mcp:index # auto-detects Ollama at http://localhost:11434
121
+ Navigate with **↑ ↓**, toggle with **Space**, confirm with **Enter**. Leaving all unselected enables every language. Your selection is saved to `.graph-indexer.json`. Pass `--all-languages` to skip the prompt.
169
122
 
170
- # Or specify a different port
171
- OLLAMA_HOST=http://localhost:11435 npm run mcp:index
172
- ```
123
+ ### 2. Index your codebase
173
124
 
174
- **For lexical-only (no Ollama):**
125
+ **With semantic search (recommended):**
175
126
  ```bash
176
- INDEXER_EMBEDDINGS=off npm run mcp:index
127
+ # Install Ollama: https://ollama.ai
128
+ ollama pull nomic-embed-text
129
+ npm run mcp:index
177
130
  ```
178
131
 
179
- ### 3. Start the MCP Server
180
-
132
+ **Lexical-only (no Ollama required):**
181
133
  ```bash
182
- npm run mcp:start
134
+ INDEXER_EMBEDDINGS=off npm run mcp:index
183
135
  ```
184
136
 
185
- The server will:
186
- - Load the index into memory
187
- - Start the file watcher daemon (auto-updates on file changes)
188
- - Listen for MCP stdio connections
137
+ Lexical-only still achieves Recall@5 = 1.00 on all benchmarks above.
189
138
 
190
- ### 4. Configure Your IDE
139
+ ### 3. Configure your IDE
191
140
 
192
- #### **Claude Desktop** (`~/Library/Application Support/Claude/claude_desktop_config.json`)
141
+ `init` writes the config automatically. Manual examples:
193
142
 
143
+ **Claude Desktop** (`~/Library/Application Support/Claude/claude_desktop_config.json`):
194
144
  ```json
195
145
  {
196
146
  "mcpServers": {
197
147
  "graph-indexer": {
198
- "command": "node",
199
- "args": ["/path/to/your/project/node_modules/graph-indexer/mcp-server.mjs"],
200
- "env": {
201
- "MCP_PROJECT_ROOT": "/path/to/your/project",
202
- "OLLAMA_HOST": "http://localhost:11434"
203
- }
148
+ "command": "npm",
149
+ "args": ["run", "--prefix", "/path/to/project", "mcp:start"],
150
+ "env": { "MCP_PROJECT_ROOT": "/path/to/project" }
204
151
  }
205
152
  }
206
153
  }
207
154
  ```
208
155
 
209
- #### **Cursor** (`.cursor/mcp.json`)
210
-
156
+ **Cursor** (`.cursor/mcp.json`) and **Claude Code** (`.mcp.json`):
211
157
  ```json
212
158
  {
213
159
  "mcpServers": {
214
160
  "graph-indexer": {
215
- "command": "node",
216
- "args": ["${workspaceFolder}/node_modules/graph-indexer/mcp-server.mjs"],
161
+ "command": "npm",
162
+ "args": ["run", "mcp:start"],
217
163
  "env": { "MCP_PROJECT_ROOT": "${workspaceFolder}" }
218
164
  }
219
165
  }
220
166
  }
221
167
  ```
222
168
 
223
- #### **VS Code** (`.vscode/mcp.json`)
224
-
169
+ **VS Code** (`.vscode/mcp.json`):
225
170
  ```json
226
171
  {
227
172
  "servers": {
228
173
  "graph-indexer": {
229
174
  "type": "stdio",
230
- "command": "node",
231
- "args": ["${workspaceFolder}/node_modules/graph-indexer/mcp-server.mjs"]
175
+ "command": "npm",
176
+ "args": ["run", "mcp:start"],
177
+ "env": { "MCP_PROJECT_ROOT": "${workspaceFolder}" }
232
178
  }
233
179
  }
234
180
  }
235
181
  ```
236
182
 
237
- ---
238
-
239
- ## How It Works
240
-
241
- ```mermaid
242
- flowchart LR
243
- INPUT["Source Files\n(.ts .js .py .rs .go .php .css)"]
244
- PARSE["Tree-sitter\nAST Parser"]
245
- CHUNKS["Semantic Chunk\nExtractor\n(functions • classes • impls)"]
246
- DEPS["Bidirectional\nImport Map"]
247
-
248
- TFIDF["TF-IDF Inverted\nIndex"]
249
- EMBED["Ollama\nnomic-embed-text"]
250
- VECTORS["Float32Array\nVector Matrix"]
251
-
252
- GRAPH["Dependency\nGraph"]
253
-
254
- FUSION["RRF Fusion\nLexical + Vector"]
255
- BOOST["Exact-Name\nBoost"]
256
-
257
- TOOLS["MCP Tools"]
258
- SEARCH["search_code"]
259
- CHUNK_GET["get_chunk"]
260
- SKELETON["get_file_skeleton"]
261
- CALLGRAPH["get_call_graph"]
262
-
263
- INPUT --> PARSE --> CHUNKS --> TFIDF & DEPS
264
- CHUNKS --> EMBED --> VECTORS
265
- TFIDF --> FUSION
266
- VECTORS --> FUSION
267
- DEPS --> GRAPH
268
- FUSION --> BOOST --> TOOLS
269
- TOOLS --> SEARCH & CHUNK_GET & SKELETON & CALLGRAPH
270
- GRAPH --> TOOLS
271
- ```
272
-
273
- ### Three-Step Process
274
-
275
- 1. **Parse** (~50–100 ms per 10k files)
276
- - Tree-sitter builds AST for each file
277
- - Extract all named chunks (functions, classes, methods, exports)
278
- - Build bidirectional import dependency map
183
+ ### 4. Add the agent system prompt
279
184
 
280
- 2. **Index** (~2 mins for a 3,600-chunk corpus)
281
- - Compute TF-IDF inverted index (lexical search)
282
- - Call Ollama to generate `nomic-embed-text` vectors (~384–768 dimensions)
283
- - Build `Float32Array` dense matrix for cosine similarity search
284
- - Serialize both to disk-efficient JSON + binary formats
285
-
286
- 3. **Search**
287
- - Query arrives: "Find the token validation function"
288
- - Lexical search via TF-IDF (instant)
289
- - Vector search via `Float32Array` cosine (1–5 ms for 1k–10k chunks)
290
- - Merge results using Reciprocal Rank Fusion (RRF) with K=60
291
- - Apply exact-name boost if `exact_tokens` provided
292
- - Return top-k results with topology (calls, imports, imported-by)
293
-
294
- ---
185
+ Copy [PROMPT.md](./PROMPT.md) into your AI agent's system prompt to instruct it to use graph-indexer tools instead of reading files directly.
295
186
 
296
187
  ---
297
188
 
298
- ## System Requirements
299
-
300
- | Requirement | Details |
301
- | :--- | :--- |
302
- | **Node.js** | v18+ (ES Modules) |
303
- | **Ollama** | Optional. Runs locally for embedding generation. Pull `nomic-embed-text` to enable semantic search. |
304
- | **Disk** | `code-index.json` (metadata) + `code-index.embeddings.bin` (binary float32 vectors). Both are gitignored by default. |
305
-
306
- ### Ollama Setup (Optional but Recommended)
307
-
308
- ```bash
309
- # Install Ollama: https://ollama.ai
310
- ollama pull nomic-embed-text
311
- npm run mcp:index # now with semantic vectors
312
- ```
313
-
314
- ### Lexical-Only Mode (No Ollama Required)
315
-
316
- ```bash
317
- INDEXER_EMBEDDINGS=off npm run mcp:index
318
- ```
319
-
320
- Recall@3 remains 100% for exact-name queries. Semantic recall (concept-to-code) degrades to lexical-only; still effective for most programming tasks.
189
+ ## MCP tools
321
190
 
322
- ### Custom Ollama Host
191
+ ### `search_code`
323
192
 
324
- ```bash
325
- # Non-standard port
326
- OLLAMA_HOST=http://localhost:11435 npm run mcp:index
193
+ Hybrid BM25 + vector search over the index. The primary entry point for agents.
327
194
 
328
- # Remote server
329
- OLLAMA_HOST=http://192.168.1.100:11434 npm run mcp:index
330
- ```
195
+ | Parameter | Type | Default | Description |
196
+ | :--- | :--- | :--- | :--- |
197
+ | `query` | string | required | Natural language description of the logic to find |
198
+ | `exact_tokens` | string? | — | Exact symbol name — guarantees rank-1 placement |
199
+ | `detail` | `"signatures"` \| `"smart"` \| `"full"` | `"smart"` | How much code to return per result |
200
+ | `top_k` | number | `5` | Results to return (1–20) |
201
+ | `include_topology` | boolean | `true` | Include imports / used-by / calls |
202
+ | `min_score` | number | `0.3` | Minimum relevance threshold |
203
+ | `token_budget` | number? | auto | Cap total tokens returned |
204
+
205
+ **`detail` levels:**
206
+
207
+ | Value | Cost | Returns |
208
+ | :--- | ---: | :--- |
209
+ | `"signatures"` | ~20 tok | Name, type, params, return type, topology |
210
+ | `"smart"` (default) | ~150 tok | Signature + lines relevant to the query |
211
+ | `"full"` | ~300 tok | Signature + complete source body |
331
212
 
332
213
  ---
333
214
 
334
- ## MCP Tools Reference
215
+ ### `resolve_symbol`
335
216
 
336
- ### `search_code`
217
+ O(1) lookup by exact name — no search ranking needed.
337
218
 
338
- Hybrid semantic + lexical search. Returns compact signature cards for all results, then fills remaining token budget with code bodies.
339
-
340
- | Parameter | Type | Default | Description |
341
- | :--- | :--- | :--- | :--- |
342
- | `query` | `string` | — | Natural language description of the logic to find |
343
- | `exact_tokens` | `string?` | — | Exact symbol name to guarantee rank-1 placement |
344
- | `top_k` | `number` | `5` | Number of results (1–20) |
345
- | `min_score` | `number` | `0.3` | Minimum cosine similarity threshold |
346
- | `token_budget` | `number?` | 1500 chars | Estimated token budget for code bodies |
347
- | `include_topology` | `boolean` | `true` | Include `⬇️ Deps` / `⬆️ Used by` / `🔗 Calls` in output |
348
-
349
- **Example response:**
350
- ```
351
- #1 · validateToken [function_declaration]
352
- 📄 src/utils/jwt.ts:14–42 · ID: `a3f9c1b2` · RRF: 0.0321
353
- 💬 Validates and decodes a JWT access token. Throws on expiry.
354
- ⬇️ Deps: src/config/env.ts [JWT_SECRET] | src/utils/errors.ts [AuthError]
355
- ⬆️ Used by: src/middleware/auth.ts, src/routes/api.ts
356
- 🔗 Calls: verify, decodePayload, throwIfExpired
357
- ↩️ Expand body: get_chunk("a3f9c1b2")
358
- ```
219
+ | Parameter | Type | Description |
220
+ | :--- | :--- | :--- |
221
+ | `symbol` | string | Exact function, class, or type name (e.g. `"validateToken"`) |
359
222
 
360
223
  ---
361
224
 
362
- ## MCP Tools Reference
225
+ ### `get_chunk`
363
226
 
364
- ### `search_code(query, exact_tokens?, top_k?, min_score?, token_budget?, include_topology?)`
365
-
366
- Hybrid semantic + lexical search. Returns ranked chunks with their signatures, dependencies, and call graph. Token budget is distributed across all results.
227
+ Returns the source code of a single chunk by its ID.
367
228
 
368
229
  | Parameter | Type | Default | Description |
369
230
  | :--- | :--- | :--- | :--- |
370
- | `query` | string | (required) | Natural language or descriptive phrase (e.g., "JWT validation") |
371
- | `exact_tokens` | string | — | Exact symbol name for guaranteed rank-1 placement (e.g., "validateToken") |
372
- | `top_k` | number | 5 | Results to return (1–20) |
373
- | `min_score` | number | 0.3 | Minimum RRF score threshold (0.0–1.0) |
374
- | `token_budget` | number | 4000 | Max tokens to spend on code bodies |
375
- | `include_topology` | boolean | true | Include dependencies + call graph in results |
376
-
377
- **Search Strategy:**
378
- 1. Query is embedded via Ollama (or skipped in lexical-only mode)
379
- 2. Parallel search: TF-IDF + vector similarity
380
- 3. Results merged using Reciprocal Rank Fusion (K=60) with weighted fusion
381
- 4. Exact-name token boost applied if provided
382
- 5. Results ranked by combined score
383
-
384
- **Example Usage:**
385
- ```
386
- search_code(
387
- query="JWT payload validation and expiry check",
388
- exact_tokens="validateToken",
389
- top_k=3,
390
- include_topology=true
391
- )
392
- ```
231
+ | `chunk_id` | string | required | ID from `search_code` results |
232
+ | `view` | `"full"` \| `"signature"` | `"full"` | Full body or first line only |
393
233
 
394
- **Example Response:**
395
- ```
396
- #1 · validateToken [function_declaration]
397
- 📄 src/auth/jwt.ts:18–42 · ID: a3f9c1b2 · Score: 0.98
398
- 💬 Validates and decodes JWT token, checks expiry timestamp
399
- ⬇️ Dependencies: src/config/constants.ts [JWT_SECRET], src/utils/errors.ts [TokenExpiredError]
400
- ⬆️ Used by: src/middleware/authMiddleware.ts, src/routes/protected.ts
401
- 🔗 Calls: verify(token), parseJWT(decoded), throwIfExpired(exp)
402
- ℹ️ Expand: get_chunk("a3f9c1b2")
403
-
404
- #2 · parseTokenClaims [function_declaration]
405
- 📄 src/auth/jwt.ts:45–60 · ID: c2b5d4e1 · Score: 0.82
406
- ...
407
- ```
234
+ ---
408
235
 
409
- ### `get_chunk(chunk_id)`
236
+ ### `get_chunk_summary`
410
237
 
411
- Retrieves the full source code of one chunk by its ID (shown in search results).
238
+ Signature + docstring + calls without the body. ~50 tokens vs ~300 for full.
412
239
 
413
240
  | Parameter | Type | Description |
414
241
  | :--- | :--- | :--- |
415
- | `chunk_id` | string | The ID from `search_code` results |
242
+ | `chunk_id` | string | ID from `search_code` results |
416
243
 
417
- **Use when:** You need the complete body of a function/class found by search (much cheaper than reading the full file).
244
+ ---
418
245
 
419
- ### `get_file_skeleton(file_path)`
246
+ ### `get_file_skeleton`
420
247
 
421
- Returns all top-level exports and definitions in a file with line numbers — no code bodies.
248
+ All top-level exports and definitions in a file with line numbers — no bodies.
422
249
 
423
250
  | Parameter | Type | Description |
424
251
  | :--- | :--- | :--- |
425
- | `file_path` | string | Relative path (e.g., `src/utils/auth.ts`) |
252
+ | `file_path` | string | Relative path (e.g. `src/utils/auth.ts`) |
426
253
 
427
- **Example Response:**
428
- ```
429
- Exports in src/utils/auth.ts:
430
- • validateToken (function) — lines 12–38
431
- • hashPassword (function) — lines 40–55
432
- • comparePassword (function) — lines 57–68
433
- • AuthError (class) — lines 70–90
434
- ```
435
-
436
- **Use when:** You need to understand what a file exports without reading it all (~50 tokens vs 5000).
254
+ ---
437
255
 
438
- ### `get_call_graph(target_function)`
256
+ ### `get_call_graph`
439
257
 
440
- Finds all chunks that call a specific function by name.
258
+ Every chunk across the repo that calls a specific function.
441
259
 
442
260
  | Parameter | Type | Description |
443
261
  | :--- | :--- | :--- |
444
- | `target_function` | string | Exact function name (e.g., `validateToken`) |
445
-
446
- **Example Response:**
447
- ```
448
- Calls to validateToken:
449
- • src/middleware/authMiddleware.ts:32 — in middleware function
450
- • src/routes/api.ts:128 — in route handler
451
- • src/services/auth.ts:44 — in login service
452
- ```
262
+ | `target_function` | string | Exact function name (e.g. `"validateToken"`) |
453
263
 
454
- **Use when:** Before renaming or modifying a function, know all dependent call sites.
455
-
456
- ### `graph://dependencies/{file_path}` (Resource)
457
-
458
- Fetches the complete import/export topology for a file without search quota.
459
-
460
- **Example:**
461
- ```
462
- graph://dependencies/src/middleware/auth.ts
463
- ```
464
-
465
- **Returns:**
466
- ```
467
- Imports:
468
- src/config/constants.ts [JWT_SECRET]
469
- src/utils/errors.ts [TokenExpiredError]
470
- src/types/auth.ts [AuthRequest]
471
-
472
- Imported by:
473
- src/app.ts
474
- src/routes/protected.ts
475
- ```
264
+ Call this before modifying any exported function to find all affected call sites.
476
265
 
477
266
  ---
478
267
 
479
- ## Hybrid RRF Search Algorithm
480
-
481
- The core innovation of graph-indexer is combining dense semantic vectors with sparse lexical signals using Reciprocal Rank Fusion:
482
-
483
- ### Lexical Search: TF-IDF
484
-
485
- Each token is scored using **sublinear term frequency**:
486
-
487
- $$w(t, d) = (1 + \log \text{count}(t, d)) \times \log \frac{N}{n_t}$$
488
-
489
- Where:
490
- - `count(t, d)` = occurrences of term t in chunk d
491
- - `N` = total chunks
492
- - `n_t` = chunks containing term t
493
-
494
- This prevents common keywords (`return`, `const`, `if`) from dominating results.
495
-
496
- ### Vector Search: Cosine Similarity
497
-
498
- Query and chunks are embedded via Ollama `nomic-embed-text` (384–768 dimensions):
499
-
500
- $$\text{sim}(q, d) = \frac{\mathbf{e}_q \cdot \mathbf{e}_d}{|\mathbf{e}_q| \times |\mathbf{e}_d|}$$
501
-
502
- Stored as pre-normalized `Float32Array` for sub-millisecond cosine computation.
503
-
504
- ### Reciprocal Rank Fusion (RRF) with Weighting
505
-
506
- Results from both methods are merged using weighted RRF:
507
-
508
- $$\text{score}(d) = w_\text{lex} \cdot \frac{1}{K + r_\text{lex}(d)} + w_\text{vec} \cdot \frac{1}{K + r_\text{vec}(d)}$$
509
-
510
- **Default weights:**
511
- - `w_lex` = 1.5 (lexical: prioritizes exact terms)
512
- - `w_vec` = 1.0 (semantic: provides context)
513
- - `K` = 60 (rank discount constant)
514
-
515
- **Why this works:**
516
- - Prevents vector results from overwhelming precise lexical matches
517
- - Ensures "exact" queries stay exact while allowing semantic queries
518
- - Tested on 5 production projects: +0.04 mean MRR improvement (0.92 → 0.96)
519
-
520
- ### Exact-Name Boost
521
-
522
- If `exact_tokens` parameter is provided, any chunk whose name exactly matches receives a boost:
523
-
524
- $$\text{score}_\text{final}(d) = \text{score}(d) + \frac{1}{K+1}$$
525
-
526
- This guarantees rank-1 placement for exact symbol name queries.
527
-
528
- ### Demotion Strategies
529
-
530
- The following chunks receive reduced scoring:
268
+ ### `get_repo_map`
531
269
 
532
- - **Test/spec files** (paths matching `/test|spec|\.test\.|\.spec\.`): 0.25× multiplier
533
- - **Example/tutorial dirs** (paths matching `/example|doc|tutorial`): 0.5× multiplier
534
- - **Expression statements** (standalone expressions, not declarations): 0.8× multiplier
270
+ Compact symbol map ordered by PageRank (most-imported files first). Orients agents in an unfamiliar codebase in ~1,500 tokens.
535
271
 
536
- This prevents test code from shadowing production code.
537
-
538
- ---
539
-
540
- ## Supported Languages
541
-
542
- | Language | Extensions | Chunk Types | Import Resolution |
272
+ | Parameter | Type | Default | Description |
543
273
  | :--- | :--- | :--- | :--- |
544
- | TypeScript / TSX | `.ts`, `.tsx` | functions, classes, methods, exports | Absolute & relative paths |
545
- | JavaScript | `.js`, `.jsx`, `.mjs`, `.cjs` | functions, classes, expressions | Absolute & relative paths |
546
- | Python | `.py` | function_definition, class_definition | Relative imports (`.`, `..`) → file paths |
547
- | Rust | `.rs` | fn, struct, enum, trait, impl | `crate::` path resolution |
548
- | Go | `.go` | function_declaration, method, type | Import spec resolution |
549
- | PHP | `.php` | function, class | include/require paths |
550
- | CSS / SCSS | `.css`, `.scss` | rule_set | — |
551
- | Java | `.java` | class, method, interface, constructor, enum | `import_declaration` → package path |
552
- | Kotlin | `.kt`, `.kts` | function, class, object, companion, constructor | `import_header` → package path |
553
- | C# | `.cs` | class, method, interface, constructor, enum, property | `using_directive` → namespace |
554
- | Ruby | `.rb` | method, singleton_method, class, module | `require` / `require_relative` → file path |
274
+ | `path_filter` | string? | — | Limit to files whose path contains this string |
275
+ | `max_files` | number | `80` | Maximum files to include |
276
+ | `sort_by` | `"importance"` \| `"path"` | `"importance"` | Sort order |
555
277
 
556
278
  ---
557
279
 
558
- ## Performance Characteristics
559
-
560
- ### Query Latency
561
-
562
- | Corpus Size | Lexical Search | Vector Search | RRF Merge |
563
- | :--- | :--- | :--- | :--- |
564
- | 1,000 chunks | <0.5 ms | 0.8 ms | <0.1 ms |
565
- | 5,000 chunks | <0.8 ms | 3.2 ms | <0.1 ms |
566
- | 10,000 chunks | <1.2 ms | 8.5 ms | <0.2 ms |
567
- | 20,000+ chunks | <2 ms | 20–50 ms | <0.3 ms |
280
+ ### `list_index_stats`
568
281
 
569
- *Measured on V8 with pre-normalized vectors. Latency includes index lookup but not Ollama network RTT.*
570
-
571
- ### Memory Footprint
572
-
573
- - **TF-IDF index:** ~150–200 bytes per chunk
574
- - **Vector matrix:** ~3,072 bytes per chunk at 768 dimensions (4 bytes × 768 floats)
575
- - **Metadata:** ~200–500 bytes per chunk (path, name, docstring refs)
576
-
577
- **Example:** A 3,600-chunk corpus occupies ~12–15 MB in memory.
578
-
579
- ### Token Savings
580
-
581
- Average across 5 test suites: **74.6%** token reduction vs full-file reads.
582
-
583
- ```
584
- Example: Express.js QuerySearch
585
- File read: src/router.js (460 lines) = 5,750 tokens
586
- Indexed: findRoute (65 lines) = 812 tokens
587
- Savings: 86% reduction
588
- ```
589
-
590
- ### Accuracy Over Scale
591
-
592
- | Project | Total Chunks | Recall@5 | Mean MRR | Queries |
593
- | :--- | :--- | :--- | :--- | :--- |
594
- | Axios | 435 | 1.00 | 0.96 | 12 |
595
- | Express | 389 | 1.00 | 1.00 | 14 |
596
- | NestJS | 2,019 | 1.00 | 1.00 | 15 |
597
- | FastAPI | 3,572 | 1.00 | 0.93 | 13 |
598
- | Gin | 1,088 | 1.00 | 0.92 | 11 |
599
- | **Mean** | **7,503** | **1.00** | **0.96** | **65** |
600
-
601
- - **Recall@5:** 100% of relevant chunks found in top 5 results
602
- - **Mean MRR:** Average reciprocal rank of first correct result
603
- - **Query types:** Easy (exact symbol), Medium (multi-token), Hard (semantic mismatch)
282
+ Index health snapshot: chunk count, file count, symbol table size, vector count, search mode, daemon status, index age.
604
283
 
605
284
  ---
606
285
 
607
- ### Vector Search Performance
286
+ ### `graph://dependencies/{file_path}` (resource)
608
287
 
609
- | Corpus size | Pure JS flat scan | HNSW (auto-activated ≥5k) |
610
- | :--- | :--- | :--- |
611
- | 1,000 chunks (dim=768) | 0.9 ms/query | — |
612
- | 5,000 chunks (dim=768) | 4.5 ms/query | 1.2 ms/query |
613
- | 20,000 chunks (dim=768) | 30.5 ms/query | 3.1 ms/query |
288
+ Full bidirectional dependency topology for a file: what it imports and what imports it.
614
289
 
615
290
  ---
616
291
 
617
- ## Best Practices for Maximum Recall
618
-
619
- ### 1. Write Comprehensive JSDoc/TSDoc
620
-
621
- The indexer embeds your docstrings alongside code bodies. Better docstrings = better semantic search:
622
-
623
- ```typescript
624
- /**
625
- * Validates and decodes a JWT access token.
626
- *
627
- * Checks token signature, expiry timestamp, and claims payload.
628
- * Throws if token is malformed, expired, or signed with wrong key.
629
- *
630
- * @param token - Raw JWT string from Authorization header
631
- * @param options - Optional validation parameters
632
- * @returns Decoded token payload with user ID, role, and expiry time
633
- * @throws AuthError if validation fails
634
- *
635
- * @example
636
- * const payload = validateToken(req.headers.authorization);
637
- * console.log(payload.userId, payload.role);
638
- */
639
- export function validateToken(token: string, options?: ValidationOptions): TokenPayload { ... }
640
- ```
641
-
642
- ### 2. Use Descriptive, Specific Names
643
-
644
- The exact-name boost ranks any chunk whose name matches `exact_tokens` at position #1. Specific names unlock this:
645
-
646
- ```typescript
647
- // ✅ Exact-name searchable
648
- export const useNotificationPermissionStatus = () => { ... };
649
- export function calculateTripDurationInDays(start: Date, end: Date) { ... };
650
- export async function fetchAndCacheUserProfile(userId: string) { ... }
651
-
652
- // ❌ Generic — impossible to target without semantic search
653
- export const util = () => { ... };
654
- export function process(data: any) { ... }
655
- export async function fetch(path: string) { ... }
656
- ```
657
-
658
- ### 3. Export Named Helpers, Not Anonymous Functions
659
-
660
- Named exports become searchable chunks:
661
-
662
- ```typescript
663
- // ✅ Indexed as chunk
664
- export function validatePaymentMethod(card: Card): boolean {
665
- // validation logic
666
- }
292
+ ## Configuration
667
293
 
668
- // ❌ Not indexed (anonymous)
669
- const validate = (card: Card) => { ... };
670
- export default validate;
671
- ```
294
+ ### Environment variables
672
295
 
673
- ### 4. Add Docstrings to Class Methods
296
+ | Variable | Default | Description |
297
+ | :--- | :--- | :--- |
298
+ | `MCP_PROJECT_ROOT` | `process.cwd()` | Project root directory |
299
+ | `OLLAMA_HOST` | `http://localhost:11434` | Ollama API endpoint |
300
+ | `INDEXER_EMBEDDINGS` | — | Set to `off` to disable vector embeddings |
674
301
 
675
- Even small methods benefit from docstrings:
302
+ ### Language selection
676
303
 
677
- ```typescript
678
- class PaymentService {
679
- /**
680
- * Charges customer card via Stripe API.
681
- * Throws on declined card or network failure.
682
- */
683
- async chargeCard(cardToken: string, amount: number): Promise<Charge> {
684
- // ...
685
- }
304
+ `init` saves your selection to `.graph-indexer.json` in the project root:
686
305
 
687
- /**
688
- * Refunds transaction to original payment method.
689
- */
690
- async refundCharge(chargeId: string): Promise<Refund> {
691
- // ...
692
- }
693
- }
306
+ ```json
307
+ { "languages": ["typescript", "javascript", "python"] }
694
308
  ```
695
309
 
696
- ### 5. Minimize Generic Utility Files
697
-
698
- Files with <5 indexable chunks produce weak search signals:
699
-
700
- ```typescript
701
- // ❌ Won't index well (2 lines)
702
- export const AuthContext = createContext(null);
703
- export const useAuth = () => useContext(AuthContext);
704
-
705
- // ✅ Indexable
706
- /**
707
- * Context for accessing authentication state throughout component tree.
708
- * Provides currentUser, isLoading, error from nearest AuthProvider.
709
- */
710
- export const AuthContext = createContext<AuthContextValue | null>(null);
711
-
712
- /**
713
- * Hook to access authentication state.
714
- * Throws if used outside AuthProvider.
715
- * @returns Authenticated user, loading state, and any auth errors
716
- */
717
- export function useAuth(): AuthContextValue {
718
- const ctx = useContext(AuthContext);
719
- if (!ctx) throw new Error('useAuth must be called inside <AuthProvider>');
720
- return ctx;
721
- }
722
- ```
310
+ Valid keys: `typescript`, `javascript`, `python`, `go`, `rust`, `php`, `java`, `kotlin`, `csharp`, `ruby`, `css`.
723
311
 
724
- ---
312
+ Update at any time by re-running `init`:
725
313
 
726
- ## Architecture & Implementation
727
-
728
- ### File Structure
729
-
730
- ```
731
- graph-indexer/
732
- ├── core-engine.mjs # In-memory dual index: TF-IDF + vector matrix
733
- │ # Methods: search(), searchHybrid(), save(), load()
734
- ├── parser-utils.mjs # Tree-sitter AST parsing + Ollama embedding
735
- │ # Supports 11 languages with AST-precision chunking
736
- ├── indexer.mjs # CLI entry: scans repo → parses → embeds → saves
737
- ├── watch-daemon.mjs # Chokidar file watcher for incremental index updates
738
- ├── mcp-server.mjs # MCP stdio server: exposes search_code, get_chunk, etc.
739
- ├── init.mjs # Installation & IDE configuration helper
740
- ├── PROMPT.md # Agent system prompt for AI coding agents
741
- ├── tests/
742
- │ ├── harness.mjs # Shared test execution framework
743
- │ ├── run.mjs # Lexical-only evaluation harness
744
- │ ├── run-embeddings.mjs # Hybrid (Ollama) evaluation harness
745
- │ ├── metrics.mjs # Recall, MRR, latency calculations
746
- │ ├── suites/
747
- │ │ ├── axios.mjs # 435-chunk corpus, 12 queries
748
- │ │ ├── express-js.mjs # 389-chunk corpus, 14 queries
749
- │ │ ├── fastapi.mjs # 3,572-chunk corpus, 13 queries
750
- │ │ ├── gin.mjs # 1,088-chunk corpus, 11 queries
751
- │ │ └── nestjs.mjs # 2,019-chunk corpus, 15 queries
752
- │ ├── reports/ # JSON performance reports
753
- │ └── ...
754
- ├── package.json
755
- ├── README.md
756
- └── LICENSE
314
+ ```bash
315
+ npx graph-indexer init # Re-run interactive selector
316
+ npx graph-indexer init --all-languages # Enable all, no prompt
757
317
  ```
758
318
 
759
- ### Indexing Pipeline
760
-
761
- 1. **Discovery** (15–50 ms)
762
- - Recursively scan directory
763
- - Apply `.gitignore` filters (vendor, node_modules)
764
- - Collect source file paths
765
-
766
- 2. **Parsing** (50–500 ms)
767
- - Tree-sitter parses each file
768
- - Extract semantic chunks: functions, classes, methods, exports
769
- - Build bidirectional import dependency graph
770
-
771
- 3. **Embedding** (1–10 mins for 5k chunks)
772
- - Batch chunks into 32-chunk requests
773
- - Call Ollama `/api/embed` with "search_document:" prefix
774
- - Store vectors as `Float32Array` binary matrix
775
-
776
- 4. **Indexing** (20–50 ms)
777
- - Build TF-IDF inverted index (token → chunk IDs)
778
- - Compute IDF for each term
779
- - Build dependency graph adjacency maps
780
-
781
- 5. **Serialization** (100–300 ms)
782
- - Save metadata to `code-index.json`
783
- - Write vectors to `code-index.embeddings.bin` (binary float32)
784
- - Atomic `tmp → rename` for crash safety
319
+ If `.graph-indexer.json` is absent or has no `languages` key, all installed parsers are used.
785
320
 
786
321
  ---
787
322
 
788
- ## Configuration & Tuning
789
-
790
- ### Environment Variables
323
+ ## Supported languages
791
324
 
792
- | Variable | Default | Description |
325
+ | Language | Extensions | Chunks extracted |
793
326
  | :--- | :--- | :--- |
794
- | `OLLAMA_HOST` | `http://localhost:11434` | Ollama API endpoint |
795
- | `INDEXER_EMBEDDINGS` | — | Set to `off` to disable embeddings (lexical-only mode) |
796
- | `MCP_PROJECT_ROOT` | `process.cwd()` | Project root path for relative import resolution |
797
- | `INDEXER_LOG_LEVEL` | `info` | Logging verbosity: `debug`, `info`, `warn`, `error` |
798
-
799
- ### Advanced Tuning
800
-
801
- Edit `core-engine.mjs` to adjust these constants:
802
-
803
- ```javascript
804
- // Weighted RRF parameters
805
- const RRF_K = 60; // Discount constant
806
- const LEX_WEIGHT = 1.5; // Lexical weight (1.0–2.0)
807
- const VEC_WEIGHT = 1.0; // Vector weight (0.5–1.5)
808
-
809
- // Demotion multipliers
810
- const TEST_FILE_DEMOTION = 0.25; // Test files: 0.25× score
811
- const EXAMPLE_DIR_DEMOTION = 0.5; // Example dirs: 0.5× score
812
- const EXPRESSION_DEMOTION = 0.8; // Standalone expressions: 0.8× score
813
-
814
- // Name boost
815
- const EXACT_NAME_TOKEN_BOOST = 1.4; // Boost for exact token matches
816
- ```
817
-
818
- **Guidelines:**
819
- - Increase `LEX_WEIGHT` if exact-symbol queries underperform
820
- - Increase `VEC_WEIGHT` if semantic queries need more influence
821
- - Lower `TEST_FILE_DEMOTION` if tests contain important reference code
327
+ | TypeScript / TSX | `.ts`, `.tsx` | functions, classes, methods, exports |
328
+ | JavaScript | `.js`, `.jsx`, `.mjs`, `.cjs` | functions, classes, expressions |
329
+ | Python | `.py` | functions, classes |
330
+ | Go | `.go` | functions, methods, types |
331
+ | Rust | `.rs` | fn, struct, enum, trait, impl |
332
+ | Java | `.java` | class, method, interface, constructor, enum |
333
+ | Kotlin | `.kt`, `.kts` | function, class, object, companion |
334
+ | C# | `.cs` | class, method, interface, property, enum |
335
+ | Ruby | `.rb` | method, class, module |
336
+ | PHP | `.php` | function, class |
337
+ | CSS / SCSS | `.css`, `.scss` | rule sets |
822
338
 
823
339
  ---
824
340
 
825
- ## Troubleshooting
826
-
827
- ### Q: Index takes forever to build
828
- **Symptom:** `npm run mcp:index` hangs on "Embedding chunks..."
341
+ ## How it works
829
342
 
830
- **Solutions:**
831
- 1. Verify Ollama is running: `curl http://localhost:11434/api/tags`
832
- 2. Check your internet/network (Ollama might be on remote host)
833
- 3. Use lexical-only mode: `INDEXER_EMBEDDINGS=off npm run mcp:index`
834
- 4. Monitor Ollama memory: large corpora may need 4–8GB
343
+ ### Indexing
835
344
 
836
- ### Q: Search results are irrelevant
837
- **Symptom:** Queries return wrong chunks even though correct chunk exists
345
+ When you run `npm run mcp:index`:
838
346
 
839
- **Solutions:**
840
- 1. Check if chunk is indexed: `grep "chunk_name" code-index.json`
841
- 2. If missing, ensure chunk has a descriptive name (see Best Practices)
842
- 3. Try with `exact_tokens`: `search_code(query, exact_tokens="chunk_name")`
843
- 4. Increase `top_k` to see more candidates
844
- 5. Verify Ollama model is loaded: `ollama list` should show `nomic-embed-text`
347
+ 1. **Parses** every source file with Tree-sitter — no regex, exact AST boundaries.
348
+ 2. **Extracts chunks**: named functions, classes, methods, and exports, each with name, docstring, parameters, return type, call sites, and type references.
349
+ 3. **Builds a dependency graph**: bidirectional import map so each chunk knows what it imports and what imports it.
350
+ 4. **Creates two indexes**: a BM25 inverted index (lexical) and per-chunk float32 embeddings via Ollama `nomic-embed-text` (optional).
351
+ 5. **Serializes to disk**: `code-index.json` + `code-index.embeddings.bin`.
845
352
 
846
- ### Q: Ollama connection fails
847
- **Symptom:** Error: "Failed to connect to Ollama at http://localhost:11434"
353
+ A background watcher daemon re-indexes changed files automatically.
848
354
 
849
- **Solutions:**
850
- ```bash
851
- # 1. Verify Ollama is running
852
- ollama list
853
-
854
- # 2. Check port (might be on different port)
855
- curl http://localhost:11434/api/tags # default
856
- curl http://localhost:11435/api/tags # alternative
355
+ ### Search pipeline
857
356
 
858
- # 3. Specify different host
859
- OLLAMA_HOST=http://192.168.1.100:11434 npm run mcp:index
860
-
861
- # 4. Fallback to lexical-only
862
- INDEXER_EMBEDDINGS=off npm run mcp:index
357
+ ```
358
+ Source files
359
+ │
360
+ ▼
361
+ Tree-sitter AST
362
+ │
363
+ ├──► Chunks ──► BM25 inverted index ──┐
364
+ │ └──► Float32 embeddings ──┤── RRF fusion ──► Ranked results ──► MCP tools
365
+ │ │
366
+ └──► Dependency graph ────────────────┘
863
367
  ```
864
368
 
865
- ### Q: MCP server won't start
866
- **Symptom:** Error when connecting IDE to MCP server
867
-
868
- **Solutions:**
869
- 1. Verify index exists: `ls -la code-index.json`
870
- 2. Check MCP_PROJECT_ROOT: `echo $MCP_PROJECT_ROOT` should point to project
871
- 3. Test stdio: `node mcp-server.mjs` (should wait for input)
872
- 4. Check IDE config paths are absolute, not relative
873
-
874
- ### Q: Memory usage is too high
875
- **Symptom:** Node process consuming 2GB+
876
-
877
- **Solutions:**
878
- - Large corpora (10k+ chunks) require 1–2GB
879
- - Use `--max-old-space-size` flag:
880
- ```bash
881
- node --max-old-space-size=4096 node_modules/graph-indexer/mcp-server.mjs
882
- ```
883
- - Consider splitting into multiple projects
884
- - Use lexical-only mode to reduce vector matrix size by 75%
885
-
886
- ### Q: Incremental updates aren't working
887
- **Symptom:** File changes not reflected in search results
888
-
889
- **Solutions:**
890
- 1. Manually re-index: `npm run mcp:index`
891
- 2. Check watcher is running: `npm run mcp:watch`
892
- 3. Verify file changes are within project root (check `.gitignore`)
893
- 4. Restart MCP server after index updates
369
+ 1. **BM25 lexical search** — O(query_terms) over the inverted index.
370
+ 2. **Vector search** — cosine similarity over pre-normalized float32 embeddings.
371
+ 3. **RRF fusion** merges both lists (lexical 1.5×, vector 1.0×).
372
+ 4. **Score boosts**: exact name match +2.0×, suffix match +1.4×, path token +1.4×, test files demoted.
894
373
 
895
374
  ---
896
375
 
897
- ## FAQ
898
-
899
- ### How much faster is graph-indexer vs file reads?
376
+ ## Best practices
900
377
 
901
- - **Token usage:** 84–90% reduction (see test results above)
902
- - **Latency:** 1–50 ms per query vs 500ms+ for file I/O + token counting
903
- - **Accuracy:** Same relevance, but 100% recall@5 with better context
378
+ **Write descriptive docstrings.** Docstrings are embedded alongside code — better documentation directly improves semantic search quality.
904
379
 
905
- ### Can I use it without Ollama?
380
+ **Use specific function names.** Names like `fetchAndCacheUserProfile` trigger the exact-name boost (2.0×) and reliably appear as rank-1 results when queried by name.
906
381
 
907
- Yes. Run with `INDEXER_EMBEDDINGS=off npm run mcp:index`. You get:
908
- - Pure TF-IDF lexical search
909
- - Exact-name queries still work perfectly
910
- - Semantic queries degrade to string matching
911
- - 1.00 recall@5 for exact-symbol searches
382
+ **Export named functions, not anonymous ones.** Anonymous default exports can't be targeted by `resolve_symbol` and rank lower in name-boost scoring.
912
383
 
913
- ### Does it support monorepos?
384
+ **Avoid catch-all utility files.** Files that mix unrelated utilities produce weaker per-chunk signals. One responsibility per module indexes more cleanly.
914
385
 
915
- Partially. The indexer creates one `code-index.json` per directory. For monorepos:
916
- 1. Index each service separately
917
- 2. Or create a wrapper that merges multiple indexes
918
- 3. Or use a single MCP server per service
919
-
920
- ### What file types are indexed?
921
-
922
- See Supported Languages table. Only recognized languages are parsed. Other files (configs, docs) are skipped.
923
-
924
- ### Can I make the index portable?
925
-
926
- Yes. Both `code-index.json` and `code-index.embeddings.bin` are self-contained. Copy them to another machine and start the MCP server — it works immediately.
927
-
928
- ### How does performance scale?
386
+ ---
929
387
 
930
- - Parsing: O(file_size_total) — linear
931
- - TF-IDF indexing: O(tokens × unique_terms) — sublinear with repetition
932
- - Vector search: O(chunks) flat scan, O(log chunks) with HNSW index
933
- - For 5k chunks: ~2 mins to index, 1–5ms per query
388
+ ## Troubleshooting
934
389
 
935
- ---
390
+ **Index takes too long**
391
+ - Verify Ollama is running: `curl http://localhost:11434/api/tags`
392
+ - Use lexical-only mode: `INDEXER_EMBEDDINGS=off npm run mcp:index`
936
393
 
937
- ## Index Files Reference
394
+ **Search returns irrelevant results**
395
+ - Use `exact_tokens` for known symbol names
396
+ - Check coverage with `list_index_stats()`
397
+ - Increase `top_k` to see more candidates
938
398
 
939
- | File | Format | Purpose | Size | Portable |
940
- | :--- | :--- | :--- | :--- | :--- |
941
- | `code-index.json` | JSON | Chunk metadata, TF-IDF inverted index, dependency graph | ~200 KB / 1k chunks | ✅ Yes |
942
- | `code-index.embeddings.bin` | Binary float32 | Vector embeddings (pre-normalized for cosine) | ~3 MB / 1k @768-dim | ✅ Yes |
399
+ **MCP server won't connect**
400
+ - Verify `MCP_PROJECT_ROOT` points to the indexed directory
401
+ - Check `code-index.json` exists: `ls -la code-index.json`
402
+ - Test manually: `npm run mcp:start`
943
403
 
944
- Both are gitignored by default. Regenerate with `npm run mcp:index` whenever you want a fresh index.
404
+ **Results are stale after file changes**
405
+ - The daemon auto-updates; if stale, run `npm run mcp:index` manually
406
+ - Check daemon status with `list_index_stats()`
945
407
 
946
408
  ---
947
409
 
948
- ## Contributing & Development
949
-
950
- ### Building from Source
410
+ ## Development
951
411
 
952
412
  ```bash
953
413
  git clone https://github.com/MaquinaTech/graph-indexer.git
954
414
  cd graph-indexer
955
415
  npm install
956
- npm run mcp:index # index this repo
957
- npm run test # run test suite
958
- npm run mcp:start # start MCP server for development
959
- ```
960
-
961
- ### Running Tests
962
-
963
- ```bash
964
- # Test lexical-only mode
965
- npm run test
966
-
967
- # Test hybrid mode (requires Ollama)
968
- npm run test:embeddings
969
-
970
- # Generate performance report
971
- npm run test:report
416
+ npm run mcp:index # index this repo
417
+ npm run test # run test suite (lexical-only)
418
+ npm run mcp:start # start MCP server
972
419
  ```
973
420
 
974
- ### Project Philosophy
975
-
976
- 1. **Local First:** No external dependencies beyond Ollama (optional)
977
- 2. **AST-Precise:** Tree-sitter for exact code boundaries, not fuzzy chunks
978
- 3. **Hybrid Search:** Combine lexical + semantic for both accuracy and recall
979
- 4. **Zero Database:** In-memory only, no infrastructure required
980
- 5. **Production-Grade:** 100% recall on 5 production projects
981
-
982
- ### Roadmap
983
-
984
- - [ ] HNSW indexing for vector search >20k chunks
985
- - [ ] Incremental embedding updates (batch new files only)
986
- - [ ] Language server protocol (LSP) integration
987
- - [ ] Web UI for search visualization
988
- - [ ] Multi-repo federation mode
989
- - [ ] Custom embedding model support
990
-
991
421
  ---
992
422
 
993
423
  ## License
994
424
 
995
- Released under the [MIT License](LICENSE).
996
-
997
- Copyright (c) 2026 MaquinaTech. Free to use, modify, and distribute in both open-source and commercial projects.
998
-
999
- ---
1000
-
1001
- ## Author & Support
1002
-
1003
- Built and maintained by **MaquinaTech**.
1004
-
1005
- - **GitHub:** [github.com/MaquinaTech](https://github.com/MaquinaTech)
1006
- - **NPM:** [npmjs.com/package/graph-indexer](https://www.npmjs.com/package/graph-indexer)
1007
- - **Issues & Feedback:** [github.com/MaquinaTech/graph-indexer/issues](https://github.com/MaquinaTech/graph-indexer/issues)
1008
-
1009
- Issues, pull requests, and feature discussions are welcome. This project started as a solution to the context bloat problem in AI coding agents — we'd love to hear how you're using it.
1010
-
1011
- ---
1012
-
1013
- ## Test Methodology
1014
-
1015
- All results in this README come from rigorous evaluation across 5 production-grade open-source projects:
1016
-
1017
- ### Test Suites
1018
-
1019
- 1. **Axios v1.6.0** (JavaScript HTTP client)
1020
- - 435 total chunks, 12 ground-truth queries
1021
- - Coverage: request/response handling, middleware, interceptors
1022
-
1023
- 2. **Express.js v4.18.2** (JavaScript web framework)
1024
- - 389 total chunks, 14 ground-truth queries
1025
- - Coverage: routing, middleware, response handling
1026
-
1027
- 3. **NestJS v10.4.9** (TypeScript framework)
1028
- - 2,019 total chunks, 15 ground-truth queries
1029
- - Coverage: decorators, dependency injection, modules, controllers
1030
-
1031
- 4. **FastAPI v0.103.0** (Python async framework)
1032
- - 3,572 total chunks, 13 ground-truth queries
1033
- - Coverage: route handlers, dependency injection, middleware
1034
-
1035
- 5. **Gin v1.9.1** (Go web framework)
1036
- - 1,088 total chunks, 11 ground-truth queries
1037
- - Coverage: routing, middleware, context handling
1038
-
1039
- ### Evaluation Metrics
1040
-
1041
- - **Recall@k:** Percentage of test queries where the correct answer appears in top-k results
1042
- - **MRR (Mean Reciprocal Rank):** Average of 1/rank for first correct result (higher is better; 1.0 = always rank-1)
1043
- - **Latency:** Time to execute query (lexical + vector search + RRF fusion)
1044
- - **Token savings:** Tokens used for surgical chunk vs full-file reads
1045
-
1046
- ### Query Difficulties
1047
-
1048
- - **Easy (3–5 per suite):** Exact symbol names where recall-1 is guaranteed
1049
- - **Medium (5–7 per suite):** Multi-token natural language descriptions
1050
- - **Hard (4 per suite):** Semantic concepts with intentional vocabulary mismatch
1051
-
1052
- ### Consistency
1053
-
1054
- All test suites use:
1055
- - Same TF-IDF weighting and RRF parameters
1056
- - Same Ollama model (`nomic-embed-text` v1)
1057
- - Same demotion strategies (test files, examples, expressions)
1058
- - Same exact-name boost multiplier (1.4×)
1059
-
1060
- Results are reproducible. Run `npm run test:embeddings --full-report` to regenerate all metrics for your machine.
425
+ Released under the [MIT License](LICENSE). Copyright (c) 2026 MaquinaTech.
1061
426
 
1062
427
  ---
1063
428
 
1064
- **Read more:** See [PROMPT.md](./PROMPT.md) for detailed agent integration instructions.
429
+ Built by [MaquinaTech](https://github.com/MaquinaTech) · [Issues](https://github.com/MaquinaTech/graph-indexer/issues) · [npm](https://www.npmjs.com/package/graph-indexer)