graph-indexer 1.1.0 → 2.0.0

package/README.md

@@ -1,444 +1,322 @@
-# graph-indexer
+<div align="center">
+  <img src="https://raw.githubusercontent.com/MaquinaTech/graph-indexer/main/assets/logo.jpg" alt="Graph Indexer Logo" width="250" />
 
-<p align="center">
-  <strong>Zero-Database · Air-Gapped · AST-Precise · Hybrid Search</strong><br>
-  <em>An MCP server that lets AI coding agents search your codebase by concept instead of reading files — reducing context usage by 65–88% per query.</em>
-</p>
+  <h1>Graph Indexer</h1>
 
-<p align="center">
-  <a href="https://www.npmjs.com/package/graph-indexer"><img src="https://img.shields.io/npm/v/graph-indexer?color=blue&style=flat-square" alt="npm version"></a>
-  <a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-yellow.svg?style=flat-square" alt="License: MIT"></a>
-  <a href="https://nodejs.org"><img src="https://img.shields.io/badge/Node.js-%3E%3D18-green?style=flat-square" alt="Node.js 18+"></a>
-  <a href="#results"><img src="https://img.shields.io/badge/Recall@5-100%25-brightgreen?style=flat-square" alt="Recall@5 100%"></a>
-</p>
+  <p>
+    <em>A local MCP server that gives AI coding agents an AST-precise, topology-aware index of your codebase.</em>
+  </p>
 
----
-
-## Table of Contents
-
-- [What it does](#what-it-does)
-- [Results](#results)
-- [Getting started](#getting-started)
-- [MCP tools](#mcp-tools)
-- [Configuration](#configuration)
-- [Supported languages](#supported-languages)
-- [How it works](#how-it-works)
-- [Best practices](#best-practices)
-- [Troubleshooting](#troubleshooting)
-- [Development](#development)
-- [Security](#security)
-- [License](#license)
+  <a href="https://www.npmjs.com/package/graph-indexer"><img src="https://img.shields.io/npm/v/graph-indexer?color=007acc&style=for-the-badge" alt="npm version"></a>
+  <a href="https://www.npmjs.com/package/graph-indexer"><img src="https://img.shields.io/npm/dt/graph-indexer?color=4caf50&style=for-the-badge" alt="NPM Downloads"></a>
+  <a href="https://nodejs.org"><img src="https://img.shields.io/badge/Node.js-%3E%3D18-brightgreen?style=for-the-badge&logo=nodedotjs" alt="Node.js 18+"></a>
+  <a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge" alt="License: MIT"></a>
+</div>
 
----
+<br />
 
 ## What it does
 
-graph-indexer pre-indexes your codebase into an AST-precise search index and exposes it as an MCP server. Instead of reading full files, AI agents call `search_code("payment validation")` and get back the exact functions that match — with type signatures, docstrings, dependency graph, and call sites — in a fraction of the tokens.
-
-It runs entirely on your machine. No external database, no cloud APIs, no telemetry.
-
-**Token comparison:**
-
-```
-Agent task: "Add error handling to the payment processing flow"
-
-Without graph-indexer:
-  readFile("src/payments/service.ts")   →  8,400 tokens
-  readFile("src/payments/handlers.ts")  →  6,200 tokens
-  readFile("src/types/payment.ts")      →  1,800 tokens
-  Total:                                ~ 16,400 tokens
-
-With graph-indexer:
-  search_code("payment processing")     →    650 tokens  (5 exact chunks)
-  get_chunk("chunk_id")                 →    280 tokens  (full function body)
-  Total:                                ~    930 tokens
-```
-
-The difference compounds across a session. Agents that read files also lack topology — they can't see which other functions call the one they're modifying, so they miss side effects.
-
----
+Graph Indexer is a local [Model Context Protocol](https://modelcontextprotocol.io) server that builds an AST-precise index of your repository with Tree-sitter and serves it to AI coding agents. Instead of grepping text or embedding whole files, it indexes *semantic chunks* — functions, classes, methods — and the call graph and import topology that connect them, so an agent can find the right symbol, see every caller and dependency that would be affected by a change, and resolve references across files. It runs entirely on your machine: the default search path is lexical (BM25 + morphological stemming) and needs no model, no daemon, and no network. Dense vector embeddings, LLM enrichment, and an LLM reranker are all available but off by default — you opt into each one when you have a measured reason to.
 
-## Results
+**Why it matters for AI coding agents:**
 
-Measured across 5 production open-source codebases (7,503 AST chunks, 70 ground-truth queries across 3 difficulty levels).
+- **Up to 98.2% Token Savings:** Delivers the exact AST chunk needed instead of dumping 1M token context files.
+- **Blast radius before every edit.** `get_call_graph` and `find_references` surface every caller, subclass, and dependency a change would touch, so the agent reasons about impact on code it never opened.
+- **Private by default.** Everything runs locally; the default path makes zero network calls and needs no model — your code never leaves your machine.
+- **One command, any language.** Guided setup wires your editors in seconds, and the zero-dependency lexical engine indexes 14 languages out of the box.
 
-### Token savings per query
+## 📦 Prerequisites
 
-| Project | Language | Without (full files) | With graph-indexer | Savings | Latency |
-| :--- | :--- | ---: | ---: | ---: | ---: |
-| Axios v1.6.0 | JavaScript | ~2,290 tok | ~655 tok | **71.4%** | 0.2 ms |
-| Express 4.18.2 | JavaScript | ~7,330 tok | ~850 tok | **88.4%** | 0.2 ms |
-| NestJS v10.4.9 | TypeScript | ~2,535 tok | ~900 tok | **64.5%** | 0.7 ms |
-| FastAPI 0.103.0 | Python | ~4,030 tok | ~685 tok | **83.0%** | 1.0 ms |
-| Gin v1.9.1 | Go | ~2,530 tok | ~450 tok | **82.2%** | 0.4 ms |
-| **Mean** | | | | **77.9%** | **0.5 ms** |
+* **Node.js** 18+ (22+ recommended if you hit >15k chunks for automatic SQLite scaling).
+* **OS:** Agnostic. macOS Apple Silicon required only for the optional MLX GPU acceleration.
+* **Optional:** Ollama for embeddings, enrichment, and reranking (see [Ollama setup](#ollama-setup)).
+* **Optional:** Python 3.10+ for the MLX embedder (macOS Apple Silicon only).
 
-> "Full files" = tokens in the source files containing the top-5 results. "With graph-indexer" = tokens in the returned chunk snippets only. Measured at 4 chars/token. NestJS is 700k tokens of source — larger than most LLMs' context windows.
+## Quick start
 
-### Search quality
-
-| Project | Chunks | Recall@1 | Recall@3 | Recall@5 | MRR |
-| :--- | ---: | ---: | ---: | ---: | ---: |
-| Axios v1.6.0 | 435 | 0.86 | 1.00 | **1.00** | 0.92 |
-| Express 4.18.2 | 389 | 0.93 | 0.93 | **1.00** | 0.95 |
-| NestJS v10.4.9 | 2,019 | 1.00 | 1.00 | **1.00** | **1.00** |
-| FastAPI 0.103.0 | 3,572 | 0.93 | 1.00 | **1.00** | 0.96 |
-| Gin v1.9.1 | 1,088 | 0.85 | 1.00 | **1.00** | 0.92 |
-| **Mean** | **7,503** | **0.91** | **0.99** | **1.00** | **0.95** |
-
-**Recall@5 = 1.00** across every project: the correct answer is always in the top 5. **MRR = 0.95**: the correct answer is rank 1 or 2 on average.
-
----
-
-## Getting started
-
-**Requirements:** Node.js v18+ · Ollama (optional, for semantic search)
-
-### 1. Install
+### 1. Run the Interactive Indexer
+Go to your project root (works for Python, Go, Rust, TS/JS, C#, and 9 more languages) and run:
 
 ```bash
-npm install graph-indexer --save-dev
-npx graph-indexer init
+npx graph-indexer /path/to/your/repo
 ```
 
-`init` auto-detects your IDE (Claude, Cursor, VS Code), adds npm scripts to `package.json`, updates `.gitignore`, and opens an interactive language selector:
+That runs the [guided setup](#guided-setup) against that repo, which leaves it ready to use. It is idempotent — re-run it whenever you add a language or change a setting; it merges into what you already have and never clobbers another tool's config. Every generated file lands in `.graph-indexer/` (git-ignored), so your repo root stays clean.
 
-```
-⚙️  Select languages (Arrows/Tab: move, Space: toggle, Enter: confirm):
-
-  ❯ ◯ TypeScript / TSX         .ts, .tsx
-    ◯ JavaScript               .js, .jsx, .mjs, .cjs
-    ◯ Python                   .py
-    ◯ Go                       .go
-    ◯ Rust                     .rs
-    ◯ PHP                      .php
-    ◯ Java                     .java
-    ◯ Kotlin                   .kt, .kts
-    ◯ C#                       .cs
-    ◯ Ruby                     .rb
-    ◯ CSS / SCSS               .css, .scss
-```
+### 2. Connect Your Agent
 
-Navigate with **↑ ↓**, toggle with **Space**, confirm with **Enter**. Leaving all unselected enables every language. Your selection is saved to `.graph-indexer.json`. Pass `--all-languages` to skip the prompt.
+**If you ran the guided setup and selected your agent in step 4, this is already done — skip ahead.**
 
-### 2. Index your codebase
+The guided setup writes the MCP config automatically (with absolute paths that survive GUI launches) for **Claude Code** (`.mcp.json`), **Claude Desktop**, **Cursor** (`.cursor/mcp.json`), and **VS Code** (`.vscode/mcp.json`). Re-run `npx graph-indexer /path/to/your/repo` at any time to add or refresh a client.
 
-**With semantic search (recommended):**
-```bash
-# Install Ollama: https://ollama.ai
-ollama pull nomic-embed-text
-npm run mcp:index
-```
+<details>
+<summary>Manual wiring (if you skipped setup or use a client not listed above)</summary>
+
+#### Claude Code — CLI
 
-**Lexical-only (no Ollama required):**
 ```bash
-INDEXER_EMBEDDINGS=off npm run mcp:index
+claude mcp add graph-indexer -- npx -y -p graph-indexer idx-mcp --repo /path/to/your/repo
 ```
 
-Lexical-only still achieves Recall@5 = 1.00 on all benchmarks above.
-
-### 3. Configure your IDE
+#### Cursor — `.cursor/mcp.json`
 
-`init` writes the config automatically. Manual examples:
-
-**Claude Desktop** (`~/Library/Application Support/Claude/claude_desktop_config.json`):
 ```json
 {
   "mcpServers": {
     "graph-indexer": {
-      "command": "npm",
-      "args": ["run", "--prefix", "/path/to/project", "mcp:start"],
-      "env": { "MCP_PROJECT_ROOT": "/path/to/project" }
+      "command": "npx",
+      "args": ["-y", "-p", "graph-indexer", "idx-mcp", "--repo", "/path/to/your/repo"]
     }
   }
 }
 ```
 
-**Cursor** (`.cursor/mcp.json`) and **Claude Code** (`.mcp.json`):
+#### VS Code — `.vscode/mcp.json`
+
 ```json
 {
-  "mcpServers": {
+  "servers": {
     "graph-indexer": {
-      "command": "npm",
-      "args": ["run", "mcp:start"],
-      "env": { "MCP_PROJECT_ROOT": "${workspaceFolder}" }
+      "command": "npx",
+      "args": ["-y", "-p", "graph-indexer", "idx-mcp", "--repo", "/path/to/your/repo"]
     }
   }
 }
 ```
 
-**VS Code** (`.vscode/mcp.json`):
+#### Claude Desktop — `claude_desktop_config.json`
+
 ```json
 {
-  "servers": {
+  "mcpServers": {
     "graph-indexer": {
-      "type": "stdio",
-      "command": "npm",
-      "args": ["run", "mcp:start"],
-      "env": { "MCP_PROJECT_ROOT": "${workspaceFolder}" }
+      "command": "npx",
+      "args": ["-y", "-p", "graph-indexer", "idx-mcp", "--repo", "/path/to/your/repo"]
     }
   }
 }
 ```
 
-### 4. Add the agent system prompt
-
-Copy [PROMPT.md](./PROMPT.md) into your AI agent's system prompt to instruct it to use graph-indexer tools instead of reading files directly.
-
----
-
-## MCP tools
-
-### `search_code`
-
-Hybrid BM25 + vector search over the index. The primary entry point for agents.
-
-| Parameter | Type | Default | Description |
-| :--- | :--- | :--- | :--- |
-| `query` | string | required | Natural language description of the logic to find |
-| `exact_tokens` | string? | — | Exact symbol name — guarantees rank-1 placement |
-| `detail` | `"signatures"` \| `"smart"` \| `"full"` | `"smart"` | How much code to return per result |
-| `top_k` | number | `5` | Results to return (1–20) |
-| `include_topology` | boolean | `true` | Include imports / used-by / calls |
-| `min_score` | number | `0.3` | Minimum relevance threshold |
-| `token_budget` | number? | auto | Cap total tokens returned |
-
-**`detail` levels:**
-
-| Value | Cost | Returns |
-| :--- | ---: | :--- |
-| `"signatures"` | ~20 tok | Name, type, params, return type, topology |
-| `"smart"` (default) | ~150 tok | Signature + lines relevant to the query |
-| `"full"` | ~300 tok | Signature + complete source body |
-
----
-
-### `resolve_symbol`
-
-O(1) lookup by exact name — no search ranking needed.
-
-| Parameter | Type | Description |
-| :--- | :--- | :--- |
-| `symbol` | string | Exact function, class, or type name (e.g. `"validateToken"`) |
-
----
-
-### `get_chunk`
-
-Returns the source code of a single chunk by its ID.
-
-| Parameter | Type | Default | Description |
-| :--- | :--- | :--- | :--- |
-| `chunk_id` | string | required | ID from `search_code` results |
-| `view` | `"full"` \| `"signature"` | `"full"` | Full body or first line only |
-
----
-
-### `get_chunk_summary`
+> **Note:** The `-p graph-indexer idx-mcp` form is required — `npx graph-indexer` runs the setup wizard, not the MCP server. If `npx` isn't on your GUI PATH (common on macOS when launching editors from Finder/Dock), run the guided setup instead — it writes absolute paths that always work.
 
-Signature + docstring + calls without the body. ~50 tokens vs ~300 for full.
+</details>
 
-| Parameter | Type | Description |
-| :--- | :--- | :--- |
-| `chunk_id` | string | ID from `search_code` results |
+Once connected, the agent can call `search_code`. A query like `search_code("rate limiting middleware")` returns ranked semantic chunks, not whole files:
 
----
-
-### `get_file_skeleton`
-
-All top-level exports and definitions in a file with line numbers — no bodies.
-
-| Parameter | Type | Description |
-| :--- | :--- | :--- |
-| `file_path` | string | Relative path (e.g. `src/utils/auth.ts`) |
-
----
-
-### `get_call_graph`
-
-Every chunk across the repo that calls a specific function.
+```jsonc
+[
+  {
+    "score": 8.41,
+    "chunk": {
+      "id": "src/middleware/rateLimit.ts:14",
+      "file_path": "src/middleware/rateLimit.ts",
+      "name": "rateLimiter",
+      "node_type": "function_declaration",
+      "start_line": 14, "end_line": 47,
+      "calls": ["tokenBucket", "getClientKey"],
+      "class_context": ""
+    }
+  }
+]
+```
 
-| Parameter | Type | Description |
-| :--- | :--- | :--- |
-| `target_function` | string | Exact function name (e.g. `"validateToken"`) |
+The agent can then call `get_call_graph("rateLimiter")` to see what calls it (the blast radius) before changing it.
 
-Call this before modifying any exported function to find all affected call sites.
+## Guided setup
 
----
+`npx graph-indexer <path>` runs an interactive wizard that leaves the repo ready to use. It is **idempotent** — re-run it whenever you add a language or change a setting; it merges into what you already have and never clobbers another tool's config. Every generated file lands in `.graph-indexer/` (git-ignored), so your repo root stays clean.
 
-### `get_repo_map`
+Before the steps, it scans the repo to pre-select your stack — languages from a bounded file walk, frameworks from your manifests (`package.json`, `pyproject.toml`, `composer.json`, `pom.xml`, `Gemfile`, `*.csproj`, …) — and, if it finds a pre-v1.4 layout, tidies those stray artifacts into `.graph-indexer/` first. Then six steps:
 
-Compact symbol map ordered by PageRank (most-imported files first). Orients agents in an unfamiliar codebase in ~1,500 tokens.
+| Step | You choose | What happens |
+|------|------------|--------------|
+| **1 · Languages** | Languages to index | Detected languages are pre-checked — press Enter to accept. Selecting none indexes **all** supported languages. |
+| **2 · Frameworks** | Prompt add-ons | Filtered to your languages and pre-checked from detection. Sharpens the agent prompt for React, Express/NestJS, FastAPI/Django, Spring, Rails, Laravel/Symfony, ASP.NET, or Android. |
+| **3 · Search engine & LLM** | Retrieval engine | **Press Enter for the recommended default** — lexical search, no LLM, no network. Everything heavier is opt-in (see below). |
+| **4 · Agents & IDEs** | Your coding tools | Multi-select which agents to wire: Claude Code/Desktop, Cursor, VS Code/Copilot, Windsurf, Cline/Roo, JetBrains Junie, Codex (AGENTS.md), Gemini CLI. Pre-checked from your saved choice, else what's detected, else all. Drives steps 5 **and** 7 — deselect a tool and graph-indexer generates nothing for it. |
+| **5 · MCP server wiring** | *(automatic)* | Wires the MCP server for each **selected** editor (VS Code, Cursor, Claude Desktop, Claude Code). **Merge-safe**: your other MCP servers and keys are preserved. |
+| **6 · Project files & daemon** | *(automatic)* | Adds `mcp:*` npm scripts (index + daemon control), a managed `.gitignore` block, and `.graph-indexer/config.json` (which remembers your agent selection). |
+| **7 · Agent instructions** | *(automatic)* | Always writes the canonical layered prompt (`GRAPH_INDEXER_PROMPT.md`) + a `GRAPH_INDEXER_DOMAIN.md` template for your own rules. Then, for each **selected** agent: `@`-imports (no duplication) in `CLAUDE.md` and `GEMINI.md`; rule files for Cursor, Windsurf, and Cline/Roo; and managed blocks in `.github/copilot-instructions.md`, `.junie/guidelines.md`, and `AGENTS.md` (Codex/Zed/Jules) — preserving anything already there. |
 
-| Parameter | Type | Default | Description |
-| :--- | :--- | :--- | :--- |
-| `path_filter` | string? | — | Limit to files whose path contains this string |
-| `max_files` | number | `80` | Maximum files to include |
-| `sort_by` | `"importance"` \| `"path"` | `"importance"` | Sort order |
+It finishes with a grouped summary (created / updated / kept / skipped / warnings), offers to **build the index now**, and prints your next steps.
 
----
+**Most repos need nothing past step 3's default** — the lexical engine has zero dependencies and works in any language. Step 3 only branches when you decline the defaults:
 
-### `list_index_stats`
+- **Storage** — `auto` (in-memory, promoting to SQLite past ~15k chunks), or force in-memory / SQLite.
+- **Embeddings** — `off` (default), `auto`, **Ollama**, **local** (in-process MiniLM, ~25 MB on first index), or **Apple Metal (MLX)** on macOS. Ollama is probed only when you actually pick it; choosing MLX offers to provision its Python venv on the spot.
+- **Enrichment / reranker** — off by default. A measured per-language note is shown before the reranker prompt (it helps Go/Python, regresses JS/TS). Model pickers are provider-aware, and if you select MLX the wizard verifies `mlx_lm.server` end-to-end — installing `mlx-lm`, starting the server, and pre-loading the model — so your first index can't fail against a server that isn't up.
 
-Index health snapshot: chunk count, file count, symbol table size, vector count, search mode, daemon status, index age.
+### Non-interactive & preview
 
----
+| Flag | Effect |
+|------|--------|
+| `--yes`, `-y` | Accept all detected/default selections — no prompts (CI). |
+| `--dry-run` | Print every file action without writing anything. |
+| `--all-languages` | Index every supported language (implies `--yes`). |
+| `--help`, `-h` | Show usage. |
 
-### `graph://dependencies/{file_path}` (resource)
+Setup also runs non-interactively whenever stdin isn't a TTY, so piping into it behaves like `--yes`.
 
-Full bidirectional dependency topology for a file: what it imports and what imports it.
+## How it works
 
----
+- **AST indexing.** Tree-sitter parses each file into a syntax tree, and the indexer extracts one *chunk* per top-level definition (function, class, method, struct, …) with its name, parameters, line range, call sites, and enclosing class. A "god class" is split so its methods become their own chunks. Supported languages: **TypeScript/JavaScript** (`.ts .tsx .js .jsx .mjs .cjs`), **Python**, **Go**, **Rust**, **Java**, **Kotlin**, **C#**, **C**, **Ruby**, **PHP**, **Bash**, and **Swift**, plus **CSS/SCSS**.
+- **Retrieval.** A hybrid ranker fuses a lexical channel (BM25 with camelCase splitting and language-agnostic Porter stemming) with an optional dense-vector channel (local embeddings) via Reciprocal Rank Fusion. With embeddings off, only the lexical channel runs — and it is the default.
+- **Call graph.** `get_call_graph` returns the callers and callees of a symbol — the *blast radius* of a change — so an agent can see what it might break before editing code it never read.
+- **Backend parity.** The in-memory and SQLite backends share the same ranking core, so they return identical top-5 results for the same query (enforced by `test/sqlite.mjs`).
+
+### MCP tools
+
+| Tool | Returns |
+|------|---------|
+| `search_code` | Ranked semantic chunks for a natural-language or symbol query. |
+| `get_chunk` | The full source of one chunk by id. |
+| `get_chunk_summary` | A compact summary of a chunk (signature, calls, context). |
+| `resolve_symbol` | Exact, case-insensitive symbol lookup by name. |
+| `get_file_skeleton` | The top-level structure (symbols + signatures) of a file. |
+| `get_call_graph` | Callers and callees of a symbol — the blast radius. |
+| `find_references` | Where a symbol is used: callers, subclasses, and type references. |
+| `find_routes` | HTTP routes mapped to their handler chunks (NestJS, FastAPI/Flask, Spring, Express/Koa). |
+| `get_subgraph` | A bounded connected subgraph around a seed symbol — its callees, high-confidence callers, and type/inheritance users, in one call. |
+| `get_repo_map` | A high-level map of the repository's modules and topology. |
+| `list_index_stats` | Index health: chunk/file/symbol/vector counts and the active config. |
 
 ## Configuration
 
-### Environment variables
-
-| Variable | Default | Description |
-| :--- | :--- | :--- |
-| `MCP_PROJECT_ROOT` | `process.cwd()` | Project root directory |
-| `OLLAMA_HOST` | `http://localhost:11434` | Ollama API endpoint |
-| `INDEXER_EMBEDDINGS` | — | Set to `off` to disable vector embeddings |
-
-### Language selection
-
-`init` saves your selection to `.graph-indexer.json` in the project root:
-
-```json
-{ "languages": ["typescript", "javascript", "python"] }
-```
-
-Valid keys: `typescript`, `javascript`, `python`, `go`, `rust`, `php`, `java`, `kotlin`, `csharp`, `ruby`, `css`.
-
-Update at any time by re-running `init`:
+Everything beyond the lexical default is opt-in. The server, indexer, and daemon all print their **effective configuration** at startup (storage backend, model names, which optional features are on), and emit a visible warning whenever an opt-in feature has a known trade-off — nothing is enabled silently.
+
+### Headline trade-offs
+
+| Option | Default | When to enable | Cost |
+|--------|---------|----------------|------|
+| `--embeddings` | off | Larger repos where recall matters; lifts success@5 | Requires Ollama or the in-process MiniLM model; slower indexing |
+| `--embed-model qwen3-embedding:4b` | `nomic-embed-text` | Better code recall + symbolic precision | slower indexing; requires Ollama |
+| `--enrichment` | off | Only useful paired with `--rerank`; alone it regresses | slowest indexing |
+| `--rerank` | off | Go/Python repos with weak semantic recall; regresses JS repos | Requires an Ollama 7B model; adds query latency |
+| `--use-sqlite` | `auto` | Repos past ~15k chunks or memory-constrained environments | Slightly higher query latency; needs Node 22+ |
+
+For most repos, the default (lexical + stemming, no embeddings) is the right starting point. Enable embeddings when you notice the agent missing chunks it should find. Enable the reranker only on Go or Python repos after measuring whether it helps — it is known to regress JavaScript repositories.
+
+### All CLI flags
+
+| Flag | Default | Effect |
+|------|---------|--------|
+| `--repo <path>` | current directory | Repository to index / serve. |
+| `--embeddings` | off | Enable the dense-vector channel. |
+| `--embed-model <model>` | `nomic-embed-text` | Ollama embedding model (e.g. `qwen3-embedding:4b`). |
+| `--embed-provider <auto\|ollama\|local\|mlx\|off>` | `auto` | Force the embedding backend (see [Embedder backends](#embedder-backends)). |
+| `--mlx-embed-model <model>` | `mlx-community/all-MiniLM-L6-v2-4bit` | Model loaded by the MLX (Apple Metal) embedder. |
+| `--use-sqlite` | `auto` | Force the disk-backed SQLite backend. |
+| `--enrichment` | off | Enable LLM enrichment of central chunks. |
+| `--enrich-model <model>` | `qwen2.5-coder:1.5b` | Model used for enrichment. |
+| `--enrich-max <n>` | 500 | Cap on new LLM calls per index run. |
+| `--enrich-concurrency <n>` | 4 | Parallel Ollama requests during enrichment. |
+| `--rerank` | off | Enable the LLM reranker (one call per NL query). |
+| `--no-git-signals` | (signals on) | Skip collecting local git churn/recency/co-change. |
+| `--git-rank-boost <0..1>` | 0 | Opt-in weight for git recency/churn in ranking (0 = ranking unchanged). |
+| `--llm-provider <ollama\|mlx>` | `ollama` | LLM backend for enrichment, reranking, and HyDE. `mlx` routes calls to a local `mlx_lm.server`. |
+| `--mlx-lm-host <url>` | `http://localhost:8080` | Endpoint for the `mlx_lm.server` when `--llm-provider mlx` is set. |
+
+### All environment variables
+
+| Variable | Default | Effect |
+|----------|---------|--------|
+| `MCP_PROJECT_ROOT` | current directory | Repository root when `--repo` is not given. |
+| `OLLAMA_HOST` | `http://localhost:11434` | Ollama endpoint for embeddings/enrichment/rerank. |
+| `INDEXER_EMBEDDINGS` | `off` | `on` enables embeddings; `off` always wins over `--embeddings`. |
+| `EMBED_MODEL` | `nomic-embed-text` | Ollama embedding model (overrides config; overridden by `--embed-model`). |
+| `INDEXER_EMBED_PROVIDER` | `auto` | `auto` \| `ollama` \| `local` \| `mlx` \| `off`. |
+| `INDEXER_MLX_EMBED_MODEL` | `mlx-community/all-MiniLM-L6-v2-4bit` | Model loaded by the MLX embedder (overridden by `--mlx-embed-model`). |
+| `INDEXER_MLX_BATCH_SIZE` | 32 | Texts per batch sent to the MLX subprocess (raise on large unified memory). |
+| `INDEXER_STORAGE` | `auto` | `auto` \| `memory` \| `sqlite`. |
+| `ENRICH_MODEL` | (unset) | Naming a model enables enrichment and selects it. |
+| `RERANK_MODEL` | (unset) | Naming a model enables the reranker and selects it. |
+| `INDEXER_GIT_SIGNALS` | (on) | Set to `off` to skip git-signal collection. |
+| `INDEXER_GIT_RANK_BOOST` | 0 | Opt-in git recency/churn ranking weight (0..1). |
+| `INDEXER_LLM_PROVIDER` | `ollama` | LLM backend for enrichment, reranking, and HyDE: `ollama` or `mlx`. |
+| `INDEXER_MLX_LM_HOST` | `http://localhost:8080` | Endpoint for the `mlx_lm.server` when `INDEXER_LLM_PROVIDER=mlx`. |
+| `INDEXER_EMBED_CONCURRENCY` | 4 | Parallel embedding batches; lower to 1 for large models on modest hardware. |
+| `INDEXER_EMBED_TIMEOUT_MS` | 120000 | Per-batch embedding timeout; raise for very large models. |
+
+When embeddings are enabled, the provider is selected in this order, and every fallback is logged (never silent): Ollama with `EMBED_MODEL` if set and reachable → Ollama with `nomic-embed-text` → the in-process MiniLM model (optional `@huggingface/transformers`) → lexical-only with a warning.
+
+### Embedder backends
+
+Set the backend with `--embed-provider` (or `INDEXER_EMBED_PROVIDER`). `mlx` is a native Python embedder that runs the Apple Metal GPU as a reused subprocess — faster in-process vectors than the bundled Xenova path, with no running Ollama daemon. The in-process (`local`) and `mlx` backends produce all-MiniLM-L6-v2-family **384-dim** vectors; the default `auto`→Ollama path uses `nomic-embed-text` (768-dim).
+
+| Backend | `--embed-provider` | Platform | Throughput¹ | Setup |
+|---------|--------------------|----------|-------------|-------|
+| Auto (Ollama → local → off) | `auto` (default) | any | varies | none |
+| In-process (Xenova) | `local` | any | ~18 ch/s | `npm i @huggingface/transformers` |
+| **Apple Metal (MLX)** | **`mlx`** | **macOS Apple Silicon** | **~42 ch/s** | `npm run embed:setup:mlx` |
+| Ollama daemon | `ollama` | any | ~14 ch/s² | `ollama serve` + `ollama pull` |
+
+¹ **Throughput is hardware-dependent.** These figures are the median of 3 cold builds on an **Apple M2 Mac mini (24 GB)** indexing the `express-js` fixture (389 chunks): `node bench/cell.mjs express-js <E0|E0_MLX|O0>`. Expect different numbers on other chips; reproduce on your own machine. Throughput also varies with system load, model warm state, and corpus size — larger repos amortize cold-start better (e.g. Ollama/nomic reaches ~32 ch/s on the larger `gin` corpus).
+
+² On Apple Silicon, Ollama already uses the Metal GPU internally (via llama.cpp). The `mlx` provider's advantage comes from a smaller model (all-MiniLM-L6-v2-4bit, 22M params, 384-dim) and no HTTP round-trip, not from GPU vs CPU.
+
+The index records which provider/model built it (in the `code-index.embeddings.bin.meta.json` sidecar) and queries with the same one, so vectors never get mixed across models. Switching providers or the MLX model between builds is detected and triggers a clean re-embed.
+
+**macOS Apple Silicon (recommended):**
 
 ```bash
-npx graph-indexer init                    # Re-run interactive selector
-npx graph-indexer init --all-languages   # Enable all, no prompt
-```
-
-If `.graph-indexer.json` is absent or has no `languages` key, all installed parsers are used.
-
----
-
-## Supported languages
-
-| Language | Extensions | Chunks extracted |
-| :--- | :--- | :--- |
-| TypeScript / TSX | `.ts`, `.tsx` | functions, classes, methods, exports |
-| JavaScript | `.js`, `.jsx`, `.mjs`, `.cjs` | functions, classes, expressions |
-| Python | `.py` | functions, classes |
-| Go | `.go` | functions, methods, types |
-| Rust | `.rs` | fn, struct, enum, trait, impl |
-| Java | `.java` | class, method, interface, constructor, enum |
-| Kotlin | `.kt`, `.kts` | function, class, object, companion |
-| C# | `.cs` | class, method, interface, property, enum |
-| Ruby | `.rb` | method, class, module |
-| PHP | `.php` | function, class |
-| CSS / SCSS | `.css`, `.scss` | rule sets |
-
----
-
-## How it works
-
-### Indexing
-
-When you run `npm run mcp:index`:
-
-1. **Parses** every source file with Tree-sitter — no regex, exact AST boundaries.
-2. **Extracts chunks**: named functions, classes, methods, and exports, each with name, docstring, parameters, return type, call sites, and type references.
-3. **Builds a dependency graph**: bidirectional import map so each chunk knows what it imports and what imports it.
-4. **Creates two indexes**: a BM25 inverted index (lexical) and per-chunk float32 embeddings via Ollama `nomic-embed-text` (optional).
-5. **Serializes to disk**: `code-index.json` + `code-index.embeddings.bin`.
-
-A background watcher daemon re-indexes changed files automatically. It respects
-`.gitignore` and skips `node_modules`, build output, and dot-directories, so it
-never traverses (or exhausts OS file-watcher limits on) dependency trees.
-
-### Search pipeline
-
-```
-Source files
-     │
-     ▼
-Tree-sitter AST
-     │
-     ├──► Chunks ──► BM25 inverted index ──┐
-     │         └──► Float32 embeddings  ──┤── RRF fusion ──► Ranked results ──► MCP tools
-     │                                     │
-     └──► Dependency graph ────────────────┘
+npm run embed:setup:mlx                                  # one-time: creates embedders/venv-mlx + installs MLX
+npx idx-index --repo . --embeddings --embed-provider mlx
 ```
 
-1. **BM25 lexical search** — O(query_terms) over the inverted index.
-2. **Vector search** — cosine similarity over pre-normalized float32 embeddings.
-3. **RRF fusion** merges both lists (lexical 1.5×, vector 1.0×).
-4. **Score boosts**: exact name match +2.0×, suffix match +1.4×, path token +1.4×, test files demoted.
+`embed:setup:mlx` provisions a **dedicated Python virtualenv** under `embedders/venv-mlx` (so MLX's deps never touch your system Python) and is idempotent — re-running it is a no-op once ready. The interactive setup (`npx graph-indexer`) offers MLX directly and can run this step for you. The first index downloads the model (~90 MB) into the Hugging Face cache; later builds reuse it.
 
----
+Choose a different MLX model with `--mlx-embed-model <id>` (or `INDEXER_MLX_EMBED_MODEL`); it must be an `mlx_embeddings`-compatible sentence model, and the default `mlx-community/all-MiniLM-L6-v2-4bit` is the proven option. Tune batch size with `INDEXER_MLX_BATCH_SIZE` (default 32; lower to 16 on 8 GB machines, raise to 64 on M-series Max/Ultra). `mlx` is macOS-only; requesting it elsewhere fails fast with the alternative to use.
 
-## Best practices
+## Benchmarks
 
-**Write descriptive docstrings.** Docstrings are embedded alongside code — better documentation directly improves semantic search quality.
+Every number here is produced by the eval harness on cold, isolated builds — never hand-edited — with **strict scoring** (exact symbol match, no file-path fallback) against a **held-out** split (~20–25% per language, never used to tune). Reproduce it all with `npm run test:eval`.
 
-**Use specific function names.** Names like `fetchAndCacheUserProfile` trigger the exact-name boost (2.0×) and reliably appear as rank-1 results when queried by name.
+What the measurements say, across 18 real-world fixtures spanning every supported language:
 
-**Export named functions, not anonymous ones.** Anonymous default exports can't be targeted by `resolve_symbol` and rank lower in name-boost scoring.
+- **The zero-dependency lexical default is the right starting point.** It wins the held-out metric outright on 2 fixtures, and 6 more need only a cheap in-process embedder — so **8 of 18 run with no Ollama and no network**. Most repos never need more.
+- **Symbolic lookups are strong** (mean rank-1 ≈ 0.70). Behavioural, natural-language queries are harder (≈ 0.30 on the default path) — that's where the optional embedding and reranker channels earn their keep.
+- **Optional features are measured, not assumed.** Dense embeddings lift recall on larger repos; the LLM reranker is language- and repo-dependent (it lifts 8 fixtures but taxes JavaScript and Spring); enrichment helps only where proven (just rust). Setup surfaces each trade-off before you enable it.
+- **Backend parity:** the in-memory and SQLite backends return byte-identical top-5 on all 18 fixtures (gated by `test/sqlite.mjs`).
 
-**Avoid catch-all utility files.** Files that mix unrelated utilities produce weaker per-chunk signals. One responsibility per module indexes more cleanly.
+Per-fixture best configs, 3× spreads, and copy-paste enable flags live in **[docs/benchmarks/BENCH_PER_FIXTURE.md](docs/benchmarks/BENCH_PER_FIXTURE.md)**.
 
----
+### Structural coverage by language
 
-## Troubleshooting
+`get_call_graph` and `find_references` are richest for typed languages. Every verdict is confirmed by invocation on the real index, not by reading field counts:
 
-**Index takes too long**
-- Verify Ollama is running: `curl http://localhost:11434/api/tags`
-- Use lexical-only mode: `INDEXER_EMBEDDINGS=off npm run mcp:index`
+| Capability | Strong | Limited |
+|---|---|---|
+| **Call graph** (callers/callees) | resolves on every supported language | Java/Spring is class-granular; SCSS effectively none |
+| **Caller precision** | receiver-aware: TS/JS, Python, C#, Swift, PHP | name-only: Go, Rust, Kotlin, Ruby, Bash, C |
+| **Typed `find_references`** | precise: TS/JS, Python · field-precise: C# | heuristic: Java/PHP/Kotlin/Swift/Rust/Go/C · empty: Ruby, Bash, SCSS, dynamic JS/Python |
 
-**Search returns irrelevant results**
-- Use `exact_tokens` for known symbol names
-- Check coverage with `list_index_stats()`
-- Increase `top_k` to see more candidates
+AST chunking and lexical search cover **every** supported language; only the typed cross-reference channel narrows for dynamic ones.
 
-**MCP server won't connect**
-- Verify `MCP_PROJECT_ROOT` points to the indexed directory
-- Check `code-index.json` exists: `ls -la code-index.json`
-- Test manually: `npm run mcp:start`
+### Where it's weakest (honestly)
 
-**Results are stale after file changes**
-- The daemon auto-updates; if stale, run `npm run mcp:index` manually
-- Check daemon status with `list_index_stats()`
+- Behavioural, natural-language recall on the default path (rank-1 ≈ 0.30) — closing it needs the embedding/reranker channel, not lexical tuning.
+- The hardest fixtures stay below 0.65 held-out success@5 even with the full stack (rails 0.50, rust 0.61).
+- Java/Spring `get_call_graph` reports at class granularity (the god-class split only fires ≥200 lines); SCSS has no meaningful call graph.
+- LLM enrichment without reranking regresses precision — enable it only paired with `--rerank`.
 
----
-
-## Development
+## Reproducing the benchmarks
 
 ```bash
-git clone https://github.com/MaquinaTech/graph-indexer.git
-cd graph-indexer
-npm install
-npm run mcp:index   # index this repo
-npm run test        # run test suite (lexical-only)
-npm run mcp:start   # start MCP server
+npm run test:all                   # full unit + integration suite
+npm run test:setup                 # index the benchmark fixtures
+npm run test:eval                  # default lexical path — prints the HELD-OUT block
+npm run test:eval -- --suite css   # any single fixture (18 authored suites)
+npm run test:eval -- --embeddings  # hybrid eval (requires Ollama)
+npm run test:eval -- --verbose     # per-query breakdown incl. file-rank column
+node test/sqlite.mjs               # backend-parity gate (memory ↔ SQLite top-5)
+node bench/cell.mjs <fixture> L1   # cold rebuild + score one fixture
 ```
 
----
-
-## Security
-
-graph-indexer runs locally and is air-gapped by default — its only outbound call
-is to a local Ollama endpoint for embeddings (skipped entirely with
-`INDEXER_EMBEDDINGS=off`). It never executes the code it indexes, and the index
-artifacts contain source snippets, so keep them git-ignored (as `init` configures).
-
-See [SECURITY.md](SECURITY.md) for the full threat model and how to report a
-vulnerability.
-
----
-
-## License
-
-Released under the [MIT License](LICENSE). Copyright (c) 2026 MaquinaTech.
+The harness is [test/evaluate.mjs](test/evaluate.mjs); deeper reproduction notes live in [docs/benchmarks/BENCH_BASELINE.md](docs/benchmarks/BENCH_BASELINE.md) and [docs/benchmarks/BENCH_FULL_SUITE.md](docs/benchmarks/BENCH_FULL_SUITE.md).
 
 ---
 
-Built by [MaquinaTech](https://github.com/MaquinaTech) · [Issues](https://github.com/MaquinaTech/graph-indexer/issues) · [npm](https://www.npmjs.com/package/graph-indexer)
+<div align="center">
+  <sub>MIT licensed · runs entirely on your machine · see <a href="SECURITY.md">SECURITY.md</a> and <a href="TRADEMARK.md">TRADEMARK.md</a></sub>
+</div>