graph-indexer 1.0.2 → 1.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (8) hide show
  1. package/PROMPT.md +143 -218
  2. package/README.md +179 -156
  3. package/indexer.mjs +4 -22
  4. package/init.mjs +179 -68
  5. package/mcp-server.mjs +10 -1
  6. package/package.json +14 -13
  7. package/parser-utils.mjs +83 -20
  8. package/watch-daemon.mjs +23 -19
package/README.md CHANGED
@@ -14,31 +14,44 @@
14
14
 
15
15
  ---
16
16
 
17
- ## What it does
17
+ ## Table of Contents
18
+
19
+ - [What it does](#what-it-does)
20
+ - [Results](#results)
21
+ - [Getting started](#getting-started)
22
+ - [MCP tools](#mcp-tools)
23
+ - [Configuration](#configuration)
24
+ - [Supported languages](#supported-languages)
25
+ - [How it works](#how-it-works)
26
+ - [Best practices](#best-practices)
27
+ - [Troubleshooting](#troubleshooting)
28
+ - [Development](#development)
29
+ - [Security](#security)
30
+ - [License](#license)
18
31
 
19
- graph-indexer pre-indexes your codebase into an AST-precise search index and exposes it as an MCP server. Instead of reading full files, AI agents call `search_code("payment validation")` and get back the exact functions that match — with their type signatures, docstrings, dependency graph, and call sites — in a fraction of the tokens.
32
+ ---
20
33
 
21
- It runs entirely on your machine. No external database, no cloud APIs, no telemetry.
34
+ ## What it does
22
35
 
23
- ---
36
+ graph-indexer pre-indexes your codebase into an AST-precise search index and exposes it as an MCP server. Instead of reading full files, AI agents call `search_code("payment validation")` and get back the exact functions that match — with type signatures, docstrings, dependency graph, and call sites — in a fraction of the tokens.
24
37
 
25
- ## Why it matters
38
+ It runs entirely on your machine. No external database, no cloud APIs, no telemetry.
26
39
 
27
- When an AI agent works on a real codebase without graph-indexer, it reads files to find code:
40
+ **Token comparison:**
28
41
 
29
42
  ```
30
43
  Agent task: "Add error handling to the payment processing flow"
31
44
 
32
45
  Without graph-indexer:
33
- readFile("src/payments/service.ts") → 8,400 tokens
34
- readFile("src/payments/handlers.ts") → 6,200 tokens
35
- readFile("src/types/payment.ts") → 1,800 tokens
36
- Total context consumed: ~16,400 tokens for 3 files
46
+ readFile("src/payments/service.ts") → 8,400 tokens
47
+ readFile("src/payments/handlers.ts") → 6,200 tokens
48
+ readFile("src/types/payment.ts") → 1,800 tokens
49
+ Total: ~ 16,400 tokens
37
50
 
38
51
  With graph-indexer:
39
- search_code("payment processing") → 650 tokens (5 exact chunks)
40
- get_chunk("chunk_id") → 280 tokens (full function body)
41
- Total context consumed: ~930 tokens
52
+ search_code("payment processing") → 650 tokens (5 exact chunks)
53
+ get_chunk("chunk_id") → 280 tokens (full function body)
54
+ Total: ~ 930 tokens
42
55
  ```
43
56
 
44
57
  The difference compounds across a session. Agents that read files also lack topology — they can't see which other functions call the one they're modifying, so they miss side effects.
@@ -47,91 +60,40 @@ The difference compounds across a session. Agents that read files also lack topo
47
60
 
48
61
  ## Results
49
62
 
50
- Measured across 5 production open-source codebases (7,503 total AST chunks, 65 ground-truth queries). Token savings compare the chunks returned per query against reading the full source files they came from.
51
-
52
- ### Token usage per query: with vs. without graph-indexer
63
+ Measured across 5 production open-source codebases (7,503 AST chunks, 70 ground-truth queries across 3 difficulty levels).
53
64
 
54
- | Project | Language | Codebase size | Without (full files) | With graph-indexer | Savings | Latency |
55
- | :--- | :--- | ---: | ---: | ---: | ---: | ---: |
56
- | Axios v1.6.0 | JavaScript | 111k tokens | ~2,290 tok/query | ~655 tok/query | **71.4%** | 0.2 ms |
57
- | Express 4.18.2 | JavaScript | 145k tokens | ~7,330 tok/query | ~850 tok/query | **88.4%** | 0.2 ms |
58
- | NestJS v10.4.9 | TypeScript | 700k tokens | ~2,535 tok/query | ~900 tok/query | **64.5%** | 0.7 ms |
59
- | FastAPI 0.103.0 | Python | 859k tokens | ~4,030 tok/query | ~685 tok/query | **83.0%** | 1.0 ms |
60
- | Gin v1.9.1 | Go | 131k tokens | ~2,530 tok/query | ~450 tok/query | **82.2%** | 0.4 ms |
61
- | **Mean** | | | | | **77.9%** | **0.5 ms** |
65
+ ### Token savings per query
62
66
 
63
- > "Full files" = reading the source files that contain the top-5 retrieved chunks. "With graph-indexer" = tokens in the returned chunk code snippets only. Measured at 4 chars/token.
67
+ | Project | Language | Without (full files) | With graph-indexer | Savings | Latency |
68
+ | :--- | :--- | ---: | ---: | ---: | ---: |
69
+ | Axios v1.6.0 | JavaScript | ~2,290 tok | ~655 tok | **71.4%** | 0.2 ms |
70
+ | Express 4.18.2 | JavaScript | ~7,330 tok | ~850 tok | **88.4%** | 0.2 ms |
71
+ | NestJS v10.4.9 | TypeScript | ~2,535 tok | ~900 tok | **64.5%** | 0.7 ms |
72
+ | FastAPI 0.103.0 | Python | ~4,030 tok | ~685 tok | **83.0%** | 1.0 ms |
73
+ | Gin v1.9.1 | Go | ~2,530 tok | ~450 tok | **82.2%** | 0.4 ms |
74
+ | **Mean** | | | | **77.9%** | **0.5 ms** |
64
75
 
65
- **Key observations:**
66
- - NestJS has 700k tokens of source — larger than most LLMs' full context window. graph-indexer reduces each query to ~900 tokens.
67
- - Express shows 88.4% savings because its core router logic is spread across large files with much non-relevant code.
68
- - Latency is sub-millisecond for all suites at their real corpus sizes.
76
+ > "Full files" = tokens in the source files containing the top-5 results. "With graph-indexer" = tokens in the returned chunk snippets only. Measured at 4 chars/token. NestJS is 700k tokens of source — larger than most LLMs' context windows.
69
77
 
70
78
  ### Search quality
71
79
 
72
- | Project | Chunks | Recall@1 | Recall@3 | Recall@5 | MRR | Queries |
73
- | :--- | ---: | ---: | ---: | ---: | ---: | ---: |
74
- | Axios v1.6.0 | 435 | 0.86 | 1.00 | **1.00** | 0.92 | 14 |
75
- | Express 4.18.2 | 389 | 0.93 | 0.93 | **1.00** | 0.95 | 14 |
76
- | NestJS v10.4.9 | 2,019 | 1.00 | 1.00 | **1.00** | **1.00** | 15 |
77
- | FastAPI 0.103.0 | 3,572 | 0.93 | 1.00 | **1.00** | 0.96 | 14 |
78
- | Gin v1.9.1 | 1,088 | 0.85 | 1.00 | **1.00** | 0.92 | 13 |
79
- | **Mean** | **7,503** | **0.91** | **0.99** | **1.00** | **0.95** | **70** |
80
+ | Project | Chunks | Recall@1 | Recall@3 | Recall@5 | MRR |
81
+ | :--- | ---: | ---: | ---: | ---: | ---: |
82
+ | Axios v1.6.0 | 435 | 0.86 | 1.00 | **1.00** | 0.92 |
83
+ | Express 4.18.2 | 389 | 0.93 | 0.93 | **1.00** | 0.95 |
84
+ | NestJS v10.4.9 | 2,019 | 1.00 | 1.00 | **1.00** | **1.00** |
85
+ | FastAPI 0.103.0 | 3,572 | 0.93 | 1.00 | **1.00** | 0.96 |
86
+ | Gin v1.9.1 | 1,088 | 0.85 | 1.00 | **1.00** | 0.92 |
87
+ | **Mean** | **7,503** | **0.91** | **0.99** | **1.00** | **0.95** |
80
88
 
81
- - **Recall@5 = 1.00** across every project and language: the correct answer is always in the top 5 results.
82
- - **MRR = 0.95** overall: on average, the correct answer is the first or second result.
83
- - Queries span three difficulty levels: easy (exact symbol names), medium (multi-token descriptions), hard (semantic concepts with vocabulary mismatch).
84
-
85
- ---
86
-
87
- ## How it works
88
-
89
- ### Indexing
90
-
91
- When you run `npm run mcp:index`, graph-indexer:
92
-
93
- 1. **Parses** every source file with Tree-sitter to build an AST — no regex, exact code boundaries.
94
- 2. **Extracts chunks**: named functions, classes, methods, and exports. Each chunk includes its name, docstring, parameters, return type, call sites, and type references.
95
- 3. **Builds a dependency graph**: bidirectional import map so each chunk knows what it imports and what imports it.
96
- 4. **Creates two search indexes**:
97
- - A BM25 inverted index (lexical search, O(1) per-token lookup).
98
- - A vector embedding per chunk via Ollama `nomic-embed-text` (optional; falls back to lexical-only if Ollama is unavailable).
99
- 5. **Serializes to disk**: `code-index.json` (metadata + BM25 index) and `code-index.embeddings.bin` (binary float32 vectors).
100
-
101
- A background file watcher daemon (`watch-daemon.mjs`) re-indexes changed files automatically after the initial index is built.
102
-
103
- ### Search
104
-
105
- When an agent calls `search_code("payment validation middleware")`:
106
-
107
- 1. **Lexical search** over the BM25 inverted index — returns candidates with BM25 scores in O(query_terms) time.
108
- 2. **Vector search** over pre-normalized float32 embeddings — returns semantically similar chunks.
109
- 3. **Reciprocal Rank Fusion (RRF)** merges both ranked lists with weights (lexical 1.5×, vector 1.0×).
110
- 4. **Scoring adjustments**: exact name match +2.0×, snake_case suffix match +1.4×, path token match +1.4×, test/example files demoted.
111
- 5. **Returns** top-k chunks with: code snippet, type signature, docstring, call graph, and dependency topology — all in one response.
112
-
113
- ### Architecture
114
-
115
- ```
116
- Source files (.ts .js .py .go .rs …)
117
- │
118
- ▼
119
- Tree-sitter AST parser
120
- │
121
- ├──► Named chunks (functions, classes, methods)
122
- │ │
123
- │ ├──► BM25 inverted index ──┐
124
- │ └──► Ollama embeddings ──┤── RRF fusion ──► Ranked results
125
- │ │
126
- └──► Dependency graph ────────────────┘
127
- (importedBy / imports) ▼
128
- MCP tools for agents
129
- ```
89
+ **Recall@5 = 1.00** across every project: the correct answer is always in the top 5. **MRR = 0.95**: the correct answer is rank 1 or 2 on average.
130
90
 
131
91
  ---
132
92
 
133
93
  ## Getting started
134
94
 
95
+ **Requirements:** Node.js v18+ · Ollama (optional, for semantic search)
96
+
135
97
  ### 1. Install
136
98
 
137
99
  ```bash
@@ -139,9 +101,27 @@ npm install graph-indexer --save-dev
139
101
  npx graph-indexer init
140
102
  ```
141
103
 
142
- `init` auto-detects your IDE (Claude, Cursor, VS Code), adds npm scripts to `package.json`, and updates `.gitignore`.
104
+ `init` auto-detects your IDE (Claude, Cursor, VS Code), adds npm scripts to `package.json`, updates `.gitignore`, and opens an interactive language selector:
105
+
106
+ ```
107
+ ⚙️ Select languages (Arrows/Tab: move, Space: toggle, Enter: confirm):
108
+
109
+ ❯ ◯ TypeScript / TSX .ts, .tsx
110
+ ◯ JavaScript .js, .jsx, .mjs, .cjs
111
+ ◯ Python .py
112
+ ◯ Go .go
113
+ ◯ Rust .rs
114
+ ◯ PHP .php
115
+ ◯ Java .java
116
+ ◯ Kotlin .kt, .kts
117
+ ◯ C# .cs
118
+ ◯ Ruby .rb
119
+ ◯ CSS / SCSS .css, .scss
120
+ ```
143
121
 
144
- ### 2. Index your repository
122
+ Navigate with **↑ ↓**, toggle with **Space**, confirm with **Enter**. Leaving all unselected enables every language. Your selection is saved to `.graph-indexer.json`. Pass `--all-languages` to skip the prompt.
123
+
124
+ ### 2. Index your codebase
145
125
 
146
126
  **With semantic search (recommended):**
147
127
  ```bash
@@ -155,30 +135,32 @@ npm run mcp:index
155
135
  INDEXER_EMBEDDINGS=off npm run mcp:index
156
136
  ```
157
137
 
158
- Lexical-only still achieves Recall@5 = 1.00 for the test suites above.
138
+ Lexical-only still achieves Recall@5 = 1.00 on all benchmarks above.
159
139
 
160
140
  ### 3. Configure your IDE
161
141
 
142
+ `init` writes the config automatically. Manual examples:
143
+
162
144
  **Claude Desktop** (`~/Library/Application Support/Claude/claude_desktop_config.json`):
163
145
  ```json
164
146
  {
165
147
  "mcpServers": {
166
148
  "graph-indexer": {
167
- "command": "node",
168
- "args": ["/path/to/project/node_modules/graph-indexer/mcp-server.mjs"],
149
+ "command": "npm",
150
+ "args": ["run", "--prefix", "/path/to/project", "mcp:start"],
169
151
  "env": { "MCP_PROJECT_ROOT": "/path/to/project" }
170
152
  }
171
153
  }
172
154
  }
173
155
  ```
174
156
 
175
- **Cursor** (`.cursor/mcp.json`):
157
+ **Cursor** (`.cursor/mcp.json`) and **Claude Code** (`.mcp.json`):
176
158
  ```json
177
159
  {
178
160
  "mcpServers": {
179
161
  "graph-indexer": {
180
- "command": "node",
181
- "args": ["${workspaceFolder}/node_modules/graph-indexer/mcp-server.mjs"],
162
+ "command": "npm",
163
+ "args": ["run", "mcp:start"],
182
164
  "env": { "MCP_PROJECT_ROOT": "${workspaceFolder}" }
183
165
  }
184
166
  }
@@ -191,8 +173,9 @@ Lexical-only still achieves Recall@5 = 1.00 for the test suites above.
191
173
  "servers": {
192
174
  "graph-indexer": {
193
175
  "type": "stdio",
194
- "command": "node",
195
- "args": ["${workspaceFolder}/node_modules/graph-indexer/mcp-server.mjs"]
176
+ "command": "npm",
177
+ "args": ["run", "mcp:start"],
178
+ "env": { "MCP_PROJECT_ROOT": "${workspaceFolder}" }
196
179
  }
197
180
  }
198
181
  }
@@ -200,7 +183,7 @@ Lexical-only still achieves Recall@5 = 1.00 for the test suites above.
200
183
 
201
184
  ### 4. Add the agent system prompt
202
185
 
203
- Copy the contents of [PROMPT.md](./PROMPT.md) into your AI agent's system prompt. This instructs the agent to use graph-indexer tools instead of reading files directly.
186
+ Copy [PROMPT.md](./PROMPT.md) into your AI agent's system prompt to instruct it to use graph-indexer tools instead of reading files directly.
204
187
 
205
188
  ---
206
189
 
@@ -208,96 +191,88 @@ Copy the contents of [PROMPT.md](./PROMPT.md) into your AI agent's system prompt
208
191
 
209
192
  ### `search_code`
210
193
 
211
- Searches the index by natural language query. Returns signature cards for all results, then code bodies according to the `detail` level.
194
+ Hybrid BM25 + vector search over the index. The primary entry point for agents.
212
195
 
213
196
  | Parameter | Type | Default | Description |
214
197
  | :--- | :--- | :--- | :--- |
215
198
  | `query` | string | required | Natural language description of the logic to find |
216
199
  | `exact_tokens` | string? | — | Exact symbol name — guarantees rank-1 placement |
217
- | `detail` | `"signatures"` \| `"smart"` \| `"full"` | `"smart"` | Controls how much code is returned |
200
+ | `detail` | `"signatures"` \| `"smart"` \| `"full"` | `"smart"` | How much code to return per result |
218
201
  | `top_k` | number | `5` | Results to return (1–20) |
219
- | `include_topology` | boolean | `true` | Include Deps / Used by / Calls in each card |
220
- | `min_score` | number | `0.3` | Minimum score threshold |
221
- | `token_budget` | number? | auto | Token budget for code bodies |
202
+ | `include_topology` | boolean | `true` | Include imports / used-by / calls |
203
+ | `min_score` | number | `0.3` | Minimum relevance threshold |
204
+ | `token_budget` | number? | auto | Cap total tokens returned |
222
205
 
223
206
  **`detail` levels:**
224
207
 
225
- | Value | Per-result cost | What you get |
208
+ | Value | Cost | Returns |
226
209
  | :--- | ---: | :--- |
227
- | `"signatures"` | ~20 tokens | Name, type, params, return type, topology only |
228
- | `"smart"` (default) | ~150 tokens | Signature + only lines relevant to the query |
229
- | `"full"` | ~300 tokens | Signature + complete source body |
230
-
231
- Start with `"signatures"`, escalate only when you need implementation details.
210
+ | `"signatures"` | ~20 tok | Name, type, params, return type, topology |
211
+ | `"smart"` (default) | ~150 tok | Signature + lines relevant to the query |
212
+ | `"full"` | ~300 tok | Signature + complete source body |
232
213
 
233
214
  ---
234
215
 
235
216
  ### `resolve_symbol`
236
217
 
237
- Looks up any symbol by exact name in O(1) — no search needed.
218
+ O(1) lookup by exact name — no search ranking needed.
238
219
 
239
220
  | Parameter | Type | Description |
240
221
  | :--- | :--- | :--- |
241
222
  | `symbol` | string | Exact function, class, or type name (e.g. `"validateToken"`) |
242
223
 
243
- Returns: definition location, type signature, docstring, type references, and cross-file topology. Faster than `search_code` for known names.
244
-
245
224
  ---
246
225
 
247
226
  ### `get_chunk`
248
227
 
249
- Returns the full source code of one chunk by its ID from search results.
228
+ Returns the source code of a single chunk by its ID.
250
229
 
251
230
  | Parameter | Type | Default | Description |
252
231
  | :--- | :--- | :--- | :--- |
253
- | `chunk_id` | string | required | ID shown in `search_code` results |
254
- | `view` | `"full"` \| `"signature"` | `"full"` | `"full"` = complete body; `"signature"` = first line only |
232
+ | `chunk_id` | string | required | ID from `search_code` results |
233
+ | `view` | `"full"` \| `"signature"` | `"full"` | Full body or first line only |
255
234
 
256
235
  ---
257
236
 
258
237
  ### `get_chunk_summary`
259
238
 
260
- Returns the interface of a chunk (signature + docstring + calls) without the body. About 50 tokens vs ~300 for the full body.
239
+ Signature + docstring + calls without the body. ~50 tokens vs ~300 for full.
261
240
 
262
241
  | Parameter | Type | Description |
263
242
  | :--- | :--- | :--- |
264
243
  | `chunk_id` | string | ID from `search_code` results |
265
244
 
266
- Use when you need to understand what a function does without reading its implementation.
267
-
268
245
  ---
269
246
 
270
247
  ### `get_file_skeleton`
271
248
 
272
- Returns all top-level exports and definitions in a file with line numbers — no bodies.
249
+ All top-level exports and definitions in a file with line numbers — no bodies.
273
250
 
274
251
  | Parameter | Type | Description |
275
252
  | :--- | :--- | :--- |
276
253
  | `file_path` | string | Relative path (e.g. `src/utils/auth.ts`) |
277
254
 
278
- Useful for understanding what a file exports before deciding which chunk to read.
279
-
280
255
  ---
281
256
 
282
257
  ### `get_call_graph`
283
258
 
284
- Finds all chunks that call a specific function by name.
259
+ Every chunk across the repo that calls a specific function.
285
260
 
286
261
  | Parameter | Type | Description |
287
262
  | :--- | :--- | :--- |
288
263
  | `target_function` | string | Exact function name (e.g. `"validateToken"`) |
289
264
 
290
- Always call this before modifying an exported function — it shows every call site repo-wide.
265
+ Call this before modifying any exported function to find all affected call sites.
291
266
 
292
267
  ---
293
268
 
294
269
  ### `get_repo_map`
295
270
 
296
- Returns a compact symbol map of the entire codebase ordered by importance (PageRank over the dependency graph — most-imported files first). Useful for orienting in an unfamiliar codebase in ~1,500 tokens.
271
+ Compact symbol map ordered by PageRank (most-imported files first). Orients agents in an unfamiliar codebase in ~1,500 tokens.
297
272
 
298
273
  | Parameter | Type | Default | Description |
299
274
  | :--- | :--- | :--- | :--- |
300
- | `path_filter` | string? | — | Only show files whose path contains this string |
275
+ | `path_filter` | string? | — | Limit to files whose path contains this string |
301
276
  | `max_files` | number | `80` | Maximum files to include |
302
277
  | `sort_by` | `"importance"` \| `"path"` | `"importance"` | Sort order |
303
278
 
@@ -305,13 +280,13 @@ Returns a compact symbol map of the entire codebase ordered by importance (PageR
305
280
 
306
281
  ### `list_index_stats`
307
282
 
308
- Returns index health: chunk count, file count, symbol table size, vector entry count, search mode (hybrid or lexical-only), daemon status, and index age.
283
+ Index health snapshot: chunk count, file count, symbol table size, vector count, search mode, daemon status, index age.
309
284
 
310
285
  ---
311
286
 
312
287
  ### `graph://dependencies/{file_path}` (resource)
313
288
 
314
- Returns the full bidirectional dependency topology for a file: what it imports and what imports it.
289
+ Full bidirectional dependency topology for a file: what it imports and what imports it.
315
290
 
316
291
  ---
317
292
 
@@ -321,68 +296,116 @@ Returns the full bidirectional dependency topology for a file: what it imports a
321
296
 
322
297
  | Variable | Default | Description |
323
298
  | :--- | :--- | :--- |
324
- | `MCP_PROJECT_ROOT` | `process.cwd()` | Project root for import resolution |
299
+ | `MCP_PROJECT_ROOT` | `process.cwd()` | Project root directory |
325
300
  | `OLLAMA_HOST` | `http://localhost:11434` | Ollama API endpoint |
326
301
  | `INDEXER_EMBEDDINGS` | — | Set to `off` to disable vector embeddings |
327
302
 
328
- ### Custom Ollama host
303
+ ### Language selection
329
304
 
330
- ```bash
331
- # Non-default port
332
- OLLAMA_HOST=http://localhost:11435 npm run mcp:index
305
+ `init` saves your selection to `.graph-indexer.json` in the project root:
306
+
307
+ ```json
308
+ { "languages": ["typescript", "javascript", "python"] }
309
+ ```
310
+
311
+ Valid keys: `typescript`, `javascript`, `python`, `go`, `rust`, `php`, `java`, `kotlin`, `csharp`, `ruby`, `css`.
312
+
313
+ Update at any time by re-running `init`:
333
314
 
334
- # Remote host
335
- OLLAMA_HOST=http://192.168.1.100:11434 npm run mcp:index
315
+ ```bash
316
+ npx graph-indexer init # Re-run interactive selector
317
+ npx graph-indexer init --all-languages # Enable all, no prompt
336
318
  ```
337
319
 
320
+ If `.graph-indexer.json` is absent or has no `languages` key, all installed parsers are used.
321
+
338
322
  ---
339
323
 
340
324
  ## Supported languages
341
325
 
342
- | Language | Extensions | Chunk types extracted |
326
+ | Language | Extensions | Chunks extracted |
343
327
  | :--- | :--- | :--- |
344
328
  | TypeScript / TSX | `.ts`, `.tsx` | functions, classes, methods, exports |
345
329
  | JavaScript | `.js`, `.jsx`, `.mjs`, `.cjs` | functions, classes, expressions |
346
- | Python | `.py` | function_definition, class_definition |
347
- | Go | `.go` | function_declaration, method, type |
330
+ | Python | `.py` | functions, classes |
331
+ | Go | `.go` | functions, methods, types |
348
332
  | Rust | `.rs` | fn, struct, enum, trait, impl |
349
333
  | Java | `.java` | class, method, interface, constructor, enum |
350
334
  | Kotlin | `.kt`, `.kts` | function, class, object, companion |
351
335
  | C# | `.cs` | class, method, interface, property, enum |
352
336
  | Ruby | `.rb` | method, class, module |
353
337
  | PHP | `.php` | function, class |
354
- | CSS / SCSS | `.css`, `.scss` | rule_set |
338
+ | CSS / SCSS | `.css`, `.scss` | rule sets |
339
+
340
+ ---
341
+
342
+ ## How it works
343
+
344
+ ### Indexing
345
+
346
+ When you run `npm run mcp:index`:
347
+
348
+ 1. **Parses** every source file with Tree-sitter — no regex, exact AST boundaries.
349
+ 2. **Extracts chunks**: named functions, classes, methods, and exports, each with name, docstring, parameters, return type, call sites, and type references.
350
+ 3. **Builds a dependency graph**: bidirectional import map so each chunk knows what it imports and what imports it.
351
+ 4. **Creates two indexes**: a BM25 inverted index (lexical) and per-chunk float32 embeddings via Ollama `nomic-embed-text` (optional).
352
+ 5. **Serializes to disk**: `code-index.json` + `code-index.embeddings.bin`.
353
+
354
+ A background watcher daemon re-indexes changed files automatically. It respects
355
+ `.gitignore` and skips `node_modules`, build output, and dot-directories, so it
356
+ never traverses (or exhausts OS file-watcher limits on) dependency trees.
357
+
358
+ ### Search pipeline
359
+
360
+ ```
361
+ Source files
362
+ │
363
+ ▼
364
+ Tree-sitter AST
365
+ │
366
+ ├──► Chunks ──► BM25 inverted index ──┐
367
+ │ └──► Float32 embeddings ──┤── RRF fusion ──► Ranked results ──► MCP tools
368
+ │ │
369
+ └──► Dependency graph ────────────────┘
370
+ ```
371
+
372
+ 1. **BM25 lexical search** — O(query_terms) over the inverted index.
373
+ 2. **Vector search** — cosine similarity over pre-normalized float32 embeddings.
374
+ 3. **RRF fusion** merges both lists (lexical 1.5×, vector 1.0×).
375
+ 4. **Score boosts**: exact name match +2.0×, suffix match +1.4×, path token +1.4×, test files demoted.
355
376
 
356
377
  ---
357
378
 
358
- ## Requirements
379
+ ## Best practices
380
+
381
+ **Write descriptive docstrings.** Docstrings are embedded alongside code — better documentation directly improves semantic search quality.
359
382
 
360
- | | |
361
- | :--- | :--- |
362
- | **Node.js** | v18+ (ES Modules) |
363
- | **Ollama** | Optional — enables semantic search. Pull `nomic-embed-text`. |
364
- | **Disk** | `code-index.json` + `code-index.embeddings.bin` (both gitignored by default) |
383
+ **Use specific function names.** Names like `fetchAndCacheUserProfile` trigger the exact-name boost (2.0×) and reliably appear as rank-1 results when queried by name.
384
+
385
+ **Export named functions, not anonymous ones.** Anonymous default exports can't be targeted by `resolve_symbol` and rank lower in name-boost scoring.
386
+
387
+ **Avoid catch-all utility files.** Files that mix unrelated utilities produce weaker per-chunk signals. One responsibility per module indexes more cleanly.
365
388
 
366
389
  ---
367
390
 
368
391
  ## Troubleshooting
369
392
 
370
- **Index takes too long to build**
393
+ **Index takes too long**
371
394
  - Verify Ollama is running: `curl http://localhost:11434/api/tags`
372
- - Fall back to lexical-only: `INDEXER_EMBEDDINGS=off npm run mcp:index`
395
+ - Use lexical-only mode: `INDEXER_EMBEDDINGS=off npm run mcp:index`
373
396
 
374
397
  **Search returns irrelevant results**
375
398
  - Use `exact_tokens` for known symbol names
376
- - Verify the chunk is indexed with `list_index_stats()`
399
+ - Check coverage with `list_index_stats()`
377
400
  - Increase `top_k` to see more candidates
378
401
 
379
402
  **MCP server won't connect**
380
403
  - Verify `MCP_PROJECT_ROOT` points to the indexed directory
381
- - Check that `code-index.json` exists: `ls -la code-index.json`
382
- - Test the server manually: `node mcp-server.mjs`
404
+ - Check `code-index.json` exists: `ls -la code-index.json`
405
+ - Test manually: `npm run mcp:start`
383
406
 
384
407
  **Results are stale after file changes**
385
- - The background daemon auto-updates; if changes aren't reflected, run `npm run mcp:index` manually
408
+ - The daemon auto-updates; if stale, run `npm run mcp:index` manually
386
409
  - Check daemon status with `list_index_stats()`
387
410
 
388
411
  ---
@@ -400,15 +423,15 @@ npm run mcp:start # start MCP server
400
423
 
401
424
  ---
402
425
 
403
- ## Best practices for better search quality
404
-
405
- **Write descriptive docstrings.** The indexer embeds docstrings alongside code. Better documentation directly improves semantic search quality.
406
-
407
- **Use specific function names.** Functions with exact, descriptive names (e.g. `fetchAndCacheUserProfile`) hit the exact-name boost (2.0×) and return as rank-1 results when queried by name.
426
+ ## Security
408
427
 
409
- **Export named functions, not anonymous ones.** Anonymous default exports cannot be targeted by `resolve_symbol` and score lower in name-boost ranking.
428
+ graph-indexer runs locally and is air-gapped by default — its only outbound call
429
+ is to a local Ollama endpoint for embeddings (skipped entirely with
430
+ `INDEXER_EMBEDDINGS=off`). It never executes the code it indexes, and the index
431
+ artifacts contain source snippets, so keep them git-ignored (as `init` configures).
410
432
 
411
- **Avoid catch-all utility files.** Large files that mix unrelated utilities produce weaker per-chunk signals. One responsibility per module indexes more cleanly.
433
+ See [SECURITY.md](SECURITY.md) for the full threat model and how to report a
434
+ vulnerability.
412
435
 
413
436
  ---
414
437
 
package/indexer.mjs CHANGED
@@ -19,7 +19,7 @@ import fs from 'fs';
19
19
  import path from 'path';
20
20
  import {
21
21
  MAX_FILE_SIZE_BYTES, EXTENSIONS, getParserForFile, buildIgnoreFilter, getLocalEmbeddingsBatch,
22
- extractImportsFromAST, extractSemanticChunks, resolveLocalImports
22
+ extractImportsFromAST, extractSemanticChunks, resolveLocalImports, buildEmbeddingPayload
23
23
  } from './parser-utils.mjs';
24
24
  import { MemoryGraphIndex, writeEmbeddingBinary } from './core-engine.mjs';
25
25
 
@@ -113,27 +113,9 @@ async function main() {
113
113
  console.time("Embedding Generation Duration");
114
114
 
115
115
  const worker = async (batch) => {
116
- const textsToEmbed = batch.map(c => {
117
- const dependencies = indexData.graph.dependencies[c.file_path] || [];
118
- const cleanDeps = dependencies.map(d => {
119
- const depsFile = path.relative(PROJECT_ROOT, d);
120
- const chunkMatches = indexData.chunks.filter(c => c.file_path === depsFile);
121
- return chunkMatches.map(c => c.name).join(' ');
122
- });
123
- const topologicalContext = cleanDeps.length > 0
124
- ? `This code architectural neighborhood connects with: ${cleanDeps.join(', ')}.`
125
- : '';
126
-
127
- return [
128
- `File Location: ${c.file_path}`,
129
- `Symbol Name: ${c.node_type} -> ${c.name}`,
130
- c.docstring ? `Developer Documentation: ${c.docstring}` : '',
131
- c.type_refs?.length ? `Type References: ${c.type_refs.join(', ')}` : '',
132
- topologicalContext,
133
- `--- Source Code ---`,
134
- c.code_snippet
135
- ].filter(Boolean).join('\n');
136
- });
116
+ const textsToEmbed = batch.map(c =>
117
+ buildEmbeddingPayload(c, indexData.graph.dependencies[c.file_path] || [])
118
+ );
137
119
 
138
120
  const embeddingsMatrix = await getLocalEmbeddingsBatch(textsToEmbed, true);
139
121
  if (embeddingsMatrix && embeddingsMatrix.length === batch.length) {