graph-indexer 2.1.0 → 3.0.0

package/CHANGELOG.md

@@ -4,7 +4,266 @@ All notable changes to graph-indexer are documented here. Dates are in YYYY-MM-D
 
 ---
 
-## [Unreleased]
+## [3.0.0] — 2026-09-23
+
+A rewrite around one goal: give coding agents answers that are exact, current and small.
+Search, a real reference graph, change impact and edit checks, from a zero-dependency package
+that needs no model, network or native build. Measured offline against the TypeScript compiler
+and real commits, and end to end with coding agents
+([docs/AGENTIC-BENCHMARK.md](docs/AGENTIC-BENCHMARK.md)).
+
+### Breaking
+
+- **Node.js 22.5+** is required (the index uses the built-in `node:sqlite`).
+- **New tool surface** (eight read-only MCP tools): `search_code`, `search_text`, `read_code`,
+  `find_references`, `call_graph`, `change_impact`, `check_changes`, `outline`. `get_symbol`
+  (one definition by name, which `read_code` replaced) stays callable for clients configured for
+  it. The 2.x tools and their option set are gone.
+- **Lean tool surface.** Only `find_references` asks MCP clients that defer tools behind a tool
+  search to load it up front — it was 62 of the 64 graph-indexer calls in real sessions; the
+  others are found through the tool search. Descriptions, the server instructions and the
+  CLAUDE.md / AGENTS.md block are a few lines each: graph-indexer adds 1,190 tokens to a Claude
+  Code prompt instead of 2,930, re-read at every turn.
+- **Removed:** dense embeddings, Ollama/MLX/in-process embedders, LLM enrichment and reranking,
+  the watch daemon, sealed mode and its attestation, taint tools, and the grammar auto-installer.
+  Search is lexical + structural and needs no model.
+- **CLI:** `graph-indexer init | serve | index | status | search | grep | files | read | symbol |
+  refs | callgraph | impact | check | outline | hook`. `idx-mcp` and `idx-index` remain as aliases of
+  `serve` and `index`.
+- The index lives in `.graph-indexer/index.db` (SQLite); 2.x index files are ignored.
+
+### Added
+
+- **Bundled parsing.** web-tree-sitter 0.25 and 16 grammars (JavaScript, TypeScript, TSX,
+  Python, Go, Java, Kotlin, Scala, C#, Rust, C, C++, Ruby, PHP, Bash, CSS) vendored as
+  WebAssembly with pinned versions and checksums; nothing is downloaded or compiled.
+- **Reference graph.** Every call, instantiation, type use, inheritance, decorator, value and
+  member-read reference is bound to its definition through lexical scopes, the enclosing type and
+  its bases, imports (barrels, tsconfig paths, workspaces, Go modules, Python packages — including
+  a module reached as an attribute of a package alias, `exp.Column` through `from pkg import exp`
+  and star re-exports, where a later import rebinds the name — Rust crates, C includes), packages
+  and globals, with a confidence label. `find_references` includes
+  overload sets and calls through base types, labelled. Ambiguous and standard-library method
+  names (`get`, `set`, `apply`…) stay unbound instead of being guessed; `super()` calls reach the
+  parent's method only. Go implicit interface satisfaction, field reads and struct literal keys;
+  Python `super()`, string annotations and `self` attributes as fields; TypeScript computed keys
+  and `typeof`.
+- **Receiver type inference** across languages: annotations, initialisers, casts, `await`,
+  declared and inferred return types (including generic bounds and defaults, and `Self`/`this`),
+  value roots (`export const NestFactory = new NestFactoryStatic()`), fields and
+  constructor-injected dependencies, collections and their elements (`xs[i]`, loops, array
+  callbacks, `list.get(0)`), optional/nullable and wrapper types, closures and access chains
+  (also fluent chains formatted one call per line, and chains that start with `(await …)`).
+  Callback parameters without annotations take their types from the signature of the function
+  they are passed to (`helpers.connect((err, socket) => socket.send())`). Members of primitives
+  and collections are never guessed.
+- **Hybrid search** over symbols: exact/qualified name channel, BM25F over name, signature, doc,
+  path and body, file-level relevance, a concept thesaurus, interpretable priors and PageRank
+  centrality, with per-file diversification.
+- **Maps and dictionaries** in the type model (`Map<K, V>`, `dict[K, V]`, `HashMap<K, V>`, Go
+  `map[K]V`, classes that extend them): `m.get(k)`, `m[k]` and `m.values()` reach the value type.
+- `find_references` states where other same-name references resolve, separates plausible unbound
+  call sites (in files that use the type or a subclass inheriting the member) from unrelated
+  ones, says when a list of calls is complete, and lists members named by string
+  (`sinon.stub(obj, 'name')`, `jest.spyOn`, `getattr(o, "name")`, `patch("pkg.C.name")`), which a
+  rename has to update too. For a class or interface it lists every type that inherits it,
+  directly or through a subclass (and through which one); `path` limits any list to a directory.
+  Ambiguous names list the alternatives as ready-to-use qualified targets (overloads count as one
+  definition).
+- **`read_code`** (`graph-indexer read`): reads symbols (`Class.method`), line ranges
+  (`path:120-180`) or files, up to 12 per call, with line numbers — and under each one, where
+  every name the code uses is defined (file:line, signature, first doc line), about one line per
+  dozen lines read, so following a call or a type is a read of the listed targets rather than a
+  search. A symbol also shows what it overrides, who uses it and the tests that reference it; a
+  file over 300 lines comes back as its outline unless `full` is set; a definition already shown
+  in the same answer is not listed again.
+- **`search_text`** (`graph-indexer grep`): text search over every file — code, configuration,
+  docs. Each code match carries its enclosing definition and, for identifiers, the symbol it
+  refers to, one compact line per match; when the search did not match the definition of the
+  name it looked for (wrong file, a pattern that misses the definition line), it says where the
+  definition is. `graph-indexer files` finds files by path or glob.
+- **Change impact:** call sites to update, overrides and implementations, transitive dependents
+  by distance, tests that exercise them (including describe blocks), public surface, git
+  co-change history, blind spots, and `diff: true` for the uncommitted working tree; "risk:
+  unknown" when the graph cannot tell.
+- **`check_changes`** (`graph-indexer check`): verifies uncommitted edits without building —
+  syntax errors introduced, calls that no longer fit a changed signature, removed or renamed
+  definitions still in use, callers the edit did not touch — and names the tests to run with the
+  command for the project's runner (jest, vitest, mocha, pytest, unittest, Django, go test,
+  cargo, Maven/Gradle). About 1 s on nestjs.
+- **Freshness:** file watcher plus stat sweeps; changed files are re-indexed before every answer
+  and re-validated before their lines are rendered. Symbol ids stay stable across edits.
+- **Robustness:** the index rebuilds itself when the extractor changes (fingerprint), recovers
+  from interrupted runs, never reuses row ids, skips minified/generated/huge files, and ignores
+  symlinks that point outside the repository. Path arguments accept absolute and `./` paths.
+- **MCP** implemented without an SDK: stdio JSON-RPC, protocol versions 2024-11-05 to
+  2025-11-25 plus the stateless 2026-07-28 revision (`server/discover`), read-only tool
+  annotations, plain-text replies with token caps, and input schemas without `const` or
+  `additionalProperties` (some Gemini-family hosts drop tools whose schemas use them). Server
+  instructions are one paragraph: keep reading and searching text with the agent's own tools;
+  `find_references` for exact uses, `call_graph` for callers of callers, `change_impact` before
+  changing a signature, `check_changes` when done. Only `find_references` is marked
+  always-loaded for clients that defer MCP tools (see Lean tool surface).
+- **`graph-indexer init`** configures Claude Code, Cursor, VS Code, Gemini CLI, Codex, OpenCode and
+  Kilo Code (`opencode.json`, or an existing `kilo.json`), Junie (`.junie/mcp/mcp.json`), Zed
+  (`context_servers` in `.zed/settings.json`) and Devin (instructions and the Claude Code hooks it
+  loads), never rewrites a config file whose comments or JSONC syntax a JSON round trip would lose
+  (it prints the entry to add instead), and adds a
+  short managed block to `CLAUDE.md`/`AGENTS.md` on how to look code up: several lookups per
+  message, the function or the lines needed rather than the file, one search for a definition
+  line across the package, and one check at the end; then graph-indexer for what a text search
+  cannot answer exactly — exact uses (`find_references`), callers of callers, what a change
+  reaches (`change_impact`), what an edit broke and the closest tests (`check_changes`), a
+  definition a search missed (`read_code`). The fourth benchmark round chose this arrangement:
+  sending every read and search through graph-indexer cost more on issue fixes than the rules
+  alone. `init --hooks` adds `graph-indexer hook` to Claude Code (an
+  installed `graph-indexer` binary when there is one, npx otherwise). After an edit it runs the
+  edit check on that file; after a search for a definition that did not find it, it says where
+  the definition is; once the agent is crawling — three searches or reads since its last edit —
+  a read gets where the names in the lines read are defined and a grep for a name several
+  definitions share says which is which; a subagent starts with the lookup rules. Otherwise it
+  stays silent, and it fails open under a deadline. The same hook reads Codex, Copilot CLI,
+  Devin and Gemini CLI payloads, and answers Cursor's events with Cursor's field. A Claude Code plugin (`integrations/claude-code`) bundles the
+  server and the hooks.
+- **Structural helper.** For Claude Code, `init` writes `.claude/agents/code-structure.md` (the
+  plugin ships the same file; `--no-helper` skips it, and a file of that name the user wrote is
+  left alone): a read-only sub-agent on a small model that answers call-site, caller, subclass
+  and impact questions from the index and replies with the `path:line` list alone, saying whether
+  it is complete. The main agent hands the question off and gets back the answer, not the
+  searches and reads behind it. Its instructions are in `src/cli/helper.mjs`, for the MCP tools
+  or, in shells without MCP, the CLI.
+- **Constructors by language.** `initialize` is a constructor only in Ruby (`__init__` and
+  `__new__` in Python, `__construct` in PHP, `constructor` in JavaScript and TypeScript, declared
+  constructors elsewhere), so `new Foo()` no longer shows up as a caller of an ordinary
+  `initialize` method, in the call graph or the edit check. A map built with type arguments
+  (`new Map<string, Foo>()`) types the values taken from it, and `call_graph` lists a recursive
+  function among its own callers.
+- **Anonymous classes and map values.** A class expression without a name (`return class extends
+  ModuleRef {…}`, `mixin(class extends Base {…})`) is a class of its own, `<anonymous>` in the
+  index, so `this` in its methods reaches its bases' members: before, `this.find()` there was
+  bound to a `find` of the class around it and reported as complete. A default-exported anonymous
+  class is a class too. The values of a map reach the code that takes them — `m.forEach(v => …)`,
+  `for (const v of m.values())`, `Array.from(m.values())`, `[...m.values()]` — also when the map
+  is a class that extends `Map<K, V>`; a loop over such a class's `values()` used to be taken for
+  the language's own and left out without a mention. On nestjs this binds 4 more calls and 9 more
+  member reads, and changes no other binding in any of the seven benchmark repositories.
+- **Go, measured against the Go type checker** (`bench/eval-graph-go.mjs`, `bench/oracle-go`), on
+  gin and on caddy and nats-server, which graph-indexer had not been run on before. Precision is
+  now 0.992–1.000 and recall 0.972–0.981 (the dispatch oracle), and implicit implementations
+  0.977–1.000 / 0.938–1.000. Fixes: implicit implementations survive reopening the index (before,
+  they were lost on every server start); modules at major version 2 or later (`…/v2`,
+  `gopkg.in/yaml.v3`) are imported under their package name; interfaces are matched by method
+  signature as well as name, and count promoted methods and type aliases; package-level variables
+  of other files type the locals set from them; `(*T)(nil)` references `T`; library types keep
+  their package (`http.Request`), so their members are not bound by name to the repository's; a
+  local shadows the package function of the same name; embedded fields are named by their type,
+  named collection types (`type M map[string]*T`) type their elements, interface methods keep
+  their result type, and a chain may start with a type assertion (`v.(Module).Info().Name()`, and
+  `(x as Foo).bar()` in TypeScript).
+- **Java, measured against the Java compiler** (`bench/eval-graph-java.mjs`,
+  `bench/oracle-java/Oracle.java`) on spring-petclinic, jsoup and Commons Collections: precision
+  0.767–0.919 → 0.955–0.994, recall 0.791–0.995 → 0.861–0.989. Overloads are methods of their own:
+  calls carry their argument count and bind to the overload that takes it (fixed arity before
+  variadic, inherited overloads when the type's own do not fit), overrides match by parameter
+  count, and `find_references` no longer merges overloads in Java. A class named for a static
+  access (`StringUtil.join(…)`, `Foo::bar`) is a use of it; anonymous classes and enum constants
+  with bodies are subclasses of what they extend; nested types keep their enclosing type; fields
+  inherited from another file type their receivers; an argument is never a method. In every
+  language a member's enclosing type is its parent, not a type found by name.
+- **A product monorepo, measured against the TypeScript compiler** (Twenty, an open-source CRM:
+  28,512 files, a NestJS server, a React front end and a shared package). graph-indexer found 18%
+  of the server's references there, against 96% on the libraries: path aliases were read from the
+  root tsconfig only, and each package of a monorepo declares its own. Each file now resolves
+  through the nearest `tsconfig.json` above it (with `extends`), then the root's: recall 0.182 →
+  0.924, precision 0.999. With it, in every JavaScript/TypeScript repository: a bare call inside a
+  method is not the method itself (it needs `this.`); members of object and type literals are not
+  names in scope; a function or class used as a value is a use in an arrow's body
+  (`@Field(() => Dto)`, `forwardRef(() => X)`), a conditional, `return`, an initialiser, an
+  assignment and `typeof` in a type; destructured values keep their types (`const { a: x } = e`,
+  `({ a }: Deps)`, `({ a }: { a: A })`); `Pick<T, …>`, `Omit<T, …>` and type aliases have the
+  members of the types they are made of.
+- **Go agent round** (`go2`, 57 real Claude Code sessions on caddy, tasks graded by the Go type
+  checker): questions 14/14 with graph-indexer against 13/14 without, at 0.87 of the cost;
+  refactors all solved at 0.81 of the cost and half the time.
+- **Resident process.** The MCP server, or `graph-indexer daemon` when no server runs (the hooks
+  start it and it exits after 30 idle minutes), keeps the index open and in sync and answers hooks
+  and CLI queries over a socket only its owner can reach, inside `.graph-indexer/` (a named pipe
+  on Windows). On a 1,600-file repository a hook drops from 280–570 ms to 70–120 ms and a CLI
+  query from ~370 ms to 110–210 ms, with the same output; without a resident everything works as
+  before. The plugin's hook command is a small client that talks to that socket instead of
+  starting graph-indexer through npx (about a second per call). `GRAPH_INDEXER_RESIDENT=0`
+  turns it off.
+- **`check_changes` names the closest test functions**: the test functions you changed and those
+  that reach the changed code through calls the index sees (directly or through one helper), with
+  a command that runs only them (pytest, unittest, Django, `go test -run`, Maven, Gradle), before
+  the test files that exercise the change.
+- **Resolved indirection after reads**: `read_code`, `get_symbol` and the post-read hook say which code actually runs. For a method, which override a subclass uses, which subclasses inherit it unchanged, and what a `self.x()` call can dispatch to. For registration tables (dict/object literals of handlers, extended with `**Base.TABLE` or `...base`), which entry handles a key and what a subclass table replaces. They also say which method a name built at run time reaches (`getattr(self, f"{key}_sql")`, `dir(cls)` + `endswith`) and what a decorator is. Static analysis only: silent when the lines have no indirection, never repeated in a session. Checked against the running program in the benchmark: precision 1.000 for overrides and inheritors, 0.995 for table keys on sqlglot.
+- **Method resolution order**: which subclasses inherit a member is decided by C3 linearization. With multiple inheritance, the index used to count classes that run another base's override (inheriting-subclass precision on sqlglot went from 0.59 to 1.000).
+- **Subclasses that inherit a changed method**: `check_changes` and `change_impact` list the
+  subclasses (transitively) that inherit a changed method without redeclaring it — they run the
+  new code too — and those that override it, and add the inheriting subclasses' test files to the
+  tests to run. In the fourth round's issue fixes, agents searched for these subclasses by hand
+  after editing, and the check named only the edited class's tests.
+- **OpenCode and Kilo Code plugin** (`integrations/opencode/graph-indexer.js`, written by
+  `init --hooks` to `.opencode/plugins/` or `.kilo/plugin/`): after a read, a search or an edit it
+  appends to the tool's output what the Claude Code hooks add as context, answered by the resident
+  process; with none running it adds nothing and starts one.
+- **Benchmarks** (`bench/`): symbol-search suites (377 queries, 9 repositories), reference
+  accuracy against the TypeScript compiler, and localization replayed from real commits, with
+  pinned fixtures (`bench/fixtures.mjs`). An agentic benchmark (`bench/agentic/`): code questions
+  graded by the TypeScript compiler, multi-site refactors graded by differential `tsc`, and real
+  issues from after the model's training cutoff graded by the hidden tests of their fix, with and
+  without grep, paired statistics and acceptance gates. The model's output is recovered from the
+  growth of the context between calls (transcripts keep only the first streamed usage and redact
+  reasoning), and `anatomy.mjs` reports where cost and time go, the calls before the first edit
+  and the share of code read that the fix changes.
+
+### Results (see docs/BENCHMARKS.md)
+
+- References vs the TypeScript compiler (nestjs, 400 symbols): precision 0.997, recall 0.962
+  (grep: 0.128 / 0.993; name-only: 0.252 / 0.975); on Twenty's server 0.999 / 0.924 (grep
+  0.009).
+- Localization over 169 real commits: file Acc@1 0.544, function MRR@10 0.432 (grep-style
+  ranking 0.485 / 0.374; BM25 0.373 / 0.290).
+- Symbol search (377 queries): rank-1 0.700, MRR 0.759 (2.x on the same queries: 0.552 / 0.646).
+- Agents, four rounds (88 tasks, 359 runs; docs/AGENTIC-BENCHMARK.md). In the fourth (19 new
+  tasks, 122 runs, all solved), graph-indexer as `init` installs it answers code questions at
+  0.44 of the cost of grep alone (95% CI 0.32–0.67) and 0.37 of the time, with half the model's
+  reasoning; questions and refactors together at 0.48; real issues at 0.85 (0.76–0.96), the same
+  as the lookup rules alone (0.86); one-file fixes at 0.85. On questions and refactors it cost
+  0.76, 0.81 and 0.73 of grep in the first three rounds.
+- Agents with a smaller model at half the price per token (25 earlier tasks, 61 runs): code
+  questions at 0.26 of grep's cost (0.18–0.50) and 0.32 of the time, answered exactly in 7 of 7
+  against 5 of 7 with grep alone — a tenth of what the usual model spent with grep; multi-site
+  refactors at 0.69 (0.56–0.87); real issues at 1.05 (0.82–1.46), and 1.19 with the lookup rules
+  alone.
+- Delegated code questions (24 new, 72 runs, all on the smaller model): the structural helper
+  answered 20 of 24 exactly, as many as a general-purpose sub-agent with grep, at 0.24 (0.18–0.33)
+  of its cost and 0.28 of its time; Claude Code's Explore agent came to 0.76. Each handed back
+  about 170 tokens, against the 23k that looking the answer up adds to the asking agent's context.
+  After the fixes that followed the round, the index alone answers all 24 exactly.
+- Real Claude Code sessions (`bench/agentic/session-round.mjs`; 12 new questions and 8 refactors,
+  60 runs): with graph-indexer as `init` installs it every question was answered exactly, 9 of 12
+  without it (the misses were the three two-level caller questions), at 0.86 (0.81–0.92) of the
+  cost and 0.68 of the time; refactors were all solved either way at the same cost. The main agent
+  never handed a question to the structural helper or to Explore on these short tasks.
+- Against graph-indexer 2.x (docs/BENCHMARKS.md section 5): references on 8 repositories in
+  TypeScript, Go and Java at precision 0.955–1.000, where 2.x is at 0.02–0.70 (it lists every chunk
+  that mentions a name; in Go it misses most method calls), with answers of a few hundred tokens
+  (2.x: up to 12,000 on average); symbol search MRR 0.760 against 0.646; localization file Acc@1
+  0.544 against 0.473; 1,200 tokens added to each session against 10,600–11,300; an unchanged
+  repository re-synced in 0.2–1.8 s where 2.x rebuilds (9–180 s). In real Claude Code sessions
+  (nestjs and Twenty, 40 tasks, 120 runs), 3.0 cost 0.43 of 2.x on code questions and 0.46 on
+  refactors, and 0.84–0.86 of no index with half the tokens spent looking code up; 2.x cost
+  1.86–1.97 of no index.
+- Real sessions on the fourth round's 11 issues (66 runs): every run fixed its issue with or without
+  graph-indexer, and no agent called it; its tool definitions, instructions and CLAUDE.md block
+  alone made those sessions cost 1.07–1.10. With the lean surface (below) that fell to 1.02–1.05,
+  and the questions stayed exact at 0.80–0.84 of the cost (126 runs).
+
+---
+
+## [Unreleased 2.x — superseded by 3.0.0]
 
 ### Install-size fix: heavy optional packages no longer ship by default