graph-indexer 2.0.0 → 2.1.0

package/CHANGELOG.md

@@ -4,6 +4,448 @@ All notable changes to graph-indexer are documented here. Dates are in YYYY-MM-D
 
 ---
 
+## [Unreleased]
+
+### Install-size fix: heavy optional packages no longer ship by default
+
+- **`@huggingface/transformers` (→ onnxruntime-node, ~950 MB with the full model/runtime chain)
+  and `hnswlib-node` are no longer `optionalDependencies`.** npm's `optionalDependencies`
+  install eagerly regardless of the name — "optional" only means "don't fail the whole install
+  if this can't build on this platform" — so every `npm i graph-indexer` was downloading the
+  local-embeddings/cross-encoder-rerank stack even for lexical-only and Ollama-only users. They
+  are now `devDependencies` (available for this repo's own tests/CI, never installed as a
+  dependent). The runtime already lazy-`import()`s them with a clear "install it or change
+  provider" error — that design was correct; only the packaging contradicted it. New
+  `npm run embed:setup:local` one-liner installs it on demand, mirroring `embed:setup:mlx`; the
+  `init.mjs` wizard now offers to install it inline when "Local (in-process)" embeddings are
+  selected (or "Auto" falls through because Ollama isn't reachable).
+- **The 15 Tree-sitter language grammars (~250 MB combined) are also no longer bundled.** Same
+  fix, larger surface: `parse/languages.mjs` now resolves each grammar ambiently first (this
+  repo's own devDependencies, or a project that installed one directly), then from a scoped
+  `.graph-indexer/node_modules` install dir, auto-installing whatever's missing there on the
+  first `idx-index`/`idx-mcp`/daemon run for the languages actually selected/detected — cached
+  from then on, works identically whether graph-indexer is a project dependency, a global
+  install, or run via bare `npx` (the install always targets the repo being indexed, not
+  graph-indexer's own location). Each grammar installs independently, so one platform-
+  incompatible package (no native toolchain) only skips that language, not the whole batch — the
+  same graceful degradation `optionalDependencies` used to give for free, now paired with a
+  negative-install-cache so a known-unbuildable grammar isn't retried on every run. `--sealed`
+  disables auto-install entirely and fails the run with the exact remedy command if a configured
+  language's grammar isn't already resolvable, rather than silently shipping a partial index.
+  Verified end-to-end: a fresh `npm install graph-indexer` drops from ~1.5 GB to ~27 MB of
+  `node_modules`; `test:all` (including `test:languages` and `test:seal`) stays green.
+
+---
+
+## [2.1.0] — 2026-06-25
+
+Frontier upgrade (Phases 1–3). Every item is **opt-in**: the default path stays lexical-only,
+in-memory, zero-dependency, with byte-identical `npm run test:eval` output and byte-identical
+memory↔sqlite top-5 parity. Headline additions: code-aware retrieval knobs (B1/B2), an
+inter-procedural receiver-type fixpoint (A3), a **persistent resolved symbol graph** with
+symbol-level **centrality** (A4/A5), a **precise resolver** + cross-file **SCIP** resolution
+(A1/A2), `impact_of_edit` / `explain_symbol` / `tests_for` MCP tools (C1/C4/C5/D1), **sealed mode**
+with a runtime egress guard and **signed attestation** (F1), **taint analysis** (`trace_taint` /
+`find_tainted_sinks`) with index-time serialization across JS/TS/Python/Java/Go (C2), an opt-in
+**learned ranker** (D3, measured a tie — RRF stays the default), **SCIP-backed precise cross-file
+references** for `find_references` (A2 v2), an opt-in **learned-sparse** vocabulary-expansion
+channel (B3, measured net-neutral — lexical stays the default), and a **ColBERT-style
+late-interaction reranker** (B4, measured helps-Go / neutral-JS at production depth — opt-in, default
+off). Every capability ships behind a flag + env + config key with the trade-off printed at startup;
+nothing changes the default path.
+
+### Code-specialized embedding models — documented; NL vector-weight re-tune evaluated and rejected (B1)
+
+- **Code-specialized embedders are now documented as a first-class opt-in.** `qwen3-embedding`,
+  `nomic-embed-code`, and `jina-embeddings-v2-base-code` embed code semantics more faithfully than
+  the general-purpose `nomic-embed-text` default. They are selected with `--embed-model <name>`
+  (no new code — Ollama models are chosen by name) and stamped into the index meta so query and
+  document vectors always share a space. **Measured (honest):** on a clean `qwen3-embedding:16k`
+  re-embed, semantic rank-1 lifted on Go (`gin` 0.20 → 0.40, MRR 0.41 → 0.67) but was neutral on
+  Python (`django` rank-1 0.33 → 0.33) and JavaScript (`express-js` 0.43 → 0.43); it also indexes
+  substantially slower. So a code embedder is a *measure-first* opt-in, not a default.
+- **Rejected: re-weighting the NL vector channel upward for code embedders.** We implemented an
+  opt-in weight profile and swept the natural-language vector weight from 0.4 → 1.2 on
+  `gin`/`django` with `qwen3-embedding`. There was **no recall@5 gain at any weight** (gin s@5 flat
+  at 1.00, django at 0.83); rank-1 was flat-to-negative and symbolic regressed at higher weights
+  (`django` symbolic rank-1 0.72 → 0.61). The existing default weighting (`NL_VECTOR_WEIGHT_PLAIN
+  = 0.4`) is already optimal even for the stronger embedder — the vector channel's job here is
+  low-weight *rescue*, and recall is already saturated. The profile was therefore **not shipped**
+  (no placebo knob); the ranking core is unchanged. Full write-up:
+  `docs/internals/IMPROVEMENT_CODE_EMBED.md`.
+
+### Local cross-encoder reranker provider (B2)
+
+- **`--rerank-provider cross-encoder`** (`RERANK_PROVIDER` / `rerank.provider` config) adds a
+  second, **air-gapped** reranker alongside the generative LLM judge. It scores each
+  (query, candidate) pair with a local MS-MARCO cross-encoder
+  (`Xenova/ms-marco-MiniLM-L-6-v2`, via the optional `@huggingface/transformers` — the same dep
+  as the in-process embedder) and sorts by score. No Ollama, no daemon, **deterministic**, fast
+  (~tens of ms per NL query); the model downloads once on first use.
+- **Measured (honest), core suites, lexical channel:** agent-style semantic rank-1 0.19 → 0.26
+  (held-out 0.30 → 0.35; held-out s@5 0.48 → 0.52), strongest on Go (gin 0.20 → 0.40) and Python
+  (fastapi 0.14 → 0.29). It is **weaker than the generative 7B judge** (0.42 agent-style rank-1)
+  and shows the usual reranker behaviour on JavaScript (mixed) plus a small symbolic wobble (long
+  symbolic queries that pass the NL gate). The generative judge stays the higher-reasoning tier;
+  the cross-encoder is the no-LLM-in-the-loop option. Full write-up:
+  `docs/internals/IMPROVEMENT_CROSS_ENCODER.md`.
+- Implementation: `rerankCrossEncoder` / `crossEncoderScore` in `enrichment.mjs` (sibling to the
+  generative `rerankResults`, lazy-loads transformers, deterministic score-desc/id-asc sort,
+  never mutates the fused score, degrades gracefully); dispatched in `mcp/tools.mjs` on
+  `rerank.provider`, reusing the existing over-fetch pool unchanged. Default path (provider
+  `generative`, reranker off) is byte-identical; memory↔sqlite parity unaffected. New
+  `test/unit.mjs` tests inject the scorer (no model in CI).
+
+### Inter-procedural receiver-type fixpoint (A3)
+
+- **`--interprocedural`** (`INDEXER_INTERPROCEDURAL` / `interprocedural` config) adds an
+  index-time, whole-program fixpoint that propagates function return types along **factory call
+  chains**, so a call-site receiver whose type comes from a multi-hop or unannotated factory
+  resolves to its concrete class. Example: `makeRepoIndirect()` returns `makeRepo()` returns
+  `new OrderRepo()` — the indirect factory has no return type to read, but the fixpoint resolves
+  the chain, so `const m = makeRepoIndirect(); m.save()` is now a **high-confidence** caller of
+  `OrderRepo.save` in `get_call_graph` / `find_references` instead of a name-only match.
+- **Design:** a new `parse/interprocedural.mjs` runs a bounded (`MAX_ITERS = 8`), monotone,
+  deterministic worklist over a per-symbol return-type lattice (conflicts — ≥2 distinct types —
+  are conservatively dropped). It writes `recv_resolved_type` onto the relevant call sites;
+  because `call_sites` is serialized identically by both backends, the resolved data is
+  **parity-free**. `classifyCallers` prefers `recv_resolved_type` and keeps the existing 1-hop
+  `recv_via_call` fallback, so an index built without the pass (or a per-file daemon update) is
+  never worse. The transient capture (`_return_via`) is stripped before serialization.
+- **Invariants:** OFF by default → the index and `get_call_graph` output are byte-identical, and
+  `npm run test:eval` is unchanged (search ranking never reads receiver types). Air-gapped,
+  zero new runtime deps. **Residual limitation (documented):** the watch daemon re-indexes one
+  file at a time and does not re-run the whole-program pass, so cross-file chains refresh on the
+  next full `idx-index`. New `test/callgraph.mjs` tests prove a 2-hop chain promotes only with the
+  fixpoint, the conflict guard holds, and `resolveReturnTypes` is order-independent. Write-up:
+  `docs/internals/IMPROVEMENT_INTERPROCEDURAL.md`.
+
+### Test→code mapping + two compound MCP tools (C5 + D1)
+
+The MCP tool surface grows from 11 to 13. Both new tools are read-only and compose existing
+store primitives — no ranking, parity, or default-path impact.
+
+- **`tests_for(symbol)`** — the test/spec chunks that exercise a symbol: a chunk under a
+  test path (`TEST_FILE_RE`) that calls or references it. Reuses `findCallers` + `findReferers`
+  filtered to tests, deterministically ordered (file, line, id). Tells an agent which tests to
+  run or update before a change, and how the symbol is meant to behave.
+- **`explain_symbol(symbol, target_class?)`** — a single-round-trip overview that previously took
+  four tool calls: signature(s) (`resolve_symbol`), callees (the symbol's `calls`), callers /
+  blast radius + subclasses + type users (`find_references` / `classifyCallers`), the HTTP routes
+  it handles (`find_routes`), the tests that exercise it (`tests_for`), and git recency/co-change.
+  Both `markdown` and `json` (`structuredContent`) formats.
+- `test/mcp.mjs` asserts the 13-tool surface; `test/references.mjs` adds a test→code fixture
+  proving `tests_for` returns only the test chunk (not the production caller) and `explain_symbol`
+  composes definition + callees + callers + tests, in both response formats.
+
+### Persistent resolved symbol graph (A4) — Phase 2
+
+- **`--symbol-graph`** (`INDEXER_SYMBOL_GRAPH` / `symbolGraph` config) materializes a resolved
+  **chunk→chunk** graph at index time: one edge `{from_chunk_id, to_chunk_id, kind, confidence}`
+  per resolved reference, where `kind` ∈ `calls` | `extends` | `type` and `confidence` ∈
+  `high` | `name_only`. It is built by **reusing the query-time resolvers** (`classifyCallers` /
+  `findReferences`) over every defined symbol, so an edge's confidence is identical to what
+  `get_call_graph` / `find_references` report — no logic duplication, no drift.
+- New store method **`getEdges(chunkId, { kind?, direction })`** (`direction: 'in'` = referrers,
+  `'out'` = referents) on both backends, deterministically ordered → byte-identical
+  memory↔sqlite. Stored in a SQLite `edges` table / the in-memory index JSON.
+- `findCallers` / `findReferers` now **read the edge graph when present**, returning the *same*
+  sets as the name-match scan (a set-equivalence test is the load-bearing guard), and **fall back
+  to the scan** when the graph is absent or the name is undefined — so the default path is
+  byte-identical and an undefined-symbol lookup never returns empty-by-edges.
+- **Invariants:** OFF by default → index + call graph byte-identical, `test:eval` unchanged
+  (edges never touch search). Air-gapped, zero new deps. Deterministic edge order. A per-file
+  daemon update **invalidates** the graph (it would otherwise go stale) so callers fall back to
+  the always-correct scan until the next full `idx-index`. New `test/edges.mjs` (6 tests):
+  set-equivalence, getEdges direction/kind/confidence, undefined-name fallback, memory↔sqlite
+  parity, daemon invalidation, deterministic ordering. Foundation for `impact_of_edit` (C4) and
+  symbol-level PageRank (A5). Write-up: `docs/internals/IMPROVEMENT_SYMBOL_GRAPH.md`.
+
+### `impact_of_edit` — precise blast radius (C1 + C4) — Phase 2
+
+- New MCP tool **`impact_of_edit({ symbols?, files? })`** (13 → 14 tools): pass what you are
+  about to change and get the **transitively-affected** code (callers, subclasses, type users by
+  hop depth), the HTTP routes that reach it, the tests to run, the direct same-named referrers to
+  verify, and git co-change — in one call instead of recursively walking `get_call_graph`.
+- It follows **high-confidence** edges transitively (a precise closure that does not explode on
+  ambiguous names) and lists direct `name_only` referrers separately. It is **chunk-precise on a
+  `--symbol-graph` index** (walks `getEdges`) and falls back to query-time `classifyCallers` /
+  `findReferences` otherwise — a new `buildImpact` in `mcp/topology.mjs`, plus a `hasSymbolGraph()`
+  store predicate. Deterministic ordering (depth, file, line, id); both response formats.
+- Additive and read-only — no ranking/parity/default-path impact. `test/edges.mjs` adds 4 tests
+  (transitive depth via the graph AND the fallback, the two agreeing, and the tool composing
+  changed + impacted + tests + routes); `test/mcp.mjs` asserts the 14-tool surface.
+
+### Symbol-level centrality (A5) — Phase 2
+
+- On a `--symbol-graph` index, the indexer now also computes **symbol centrality**: a
+  confidence-weighted **PageRank over the resolved chunk→chunk edges** (`high` edges weighted
+  1.0, `name_only` 0.5). A definition referenced — called / extended / used as a type — by many
+  *other central* symbols scores high; it surfaces the program's load-bearing hubs. New module
+  `mcp/centrality.mjs` (`computeSymbolCentrality`). This is **distinct** from the existing
+  file-level `computePageRank` (import graph): A5 is symbol-granular and call/type-aware.
+- **Surfaced** through two existing tools (no new tool — the surface stays at 14): `explain_symbol`
+  attaches each definition's `{ score, rank, total }` (markdown shows `🎯 centrality #R/T`, with a
+  `(hub)` tag in the top decile); `get_repo_map`'s unfiltered view lists the most-central symbols.
+  Both **omit** centrality entirely when the graph is absent → default output unchanged.
+- **Parity-free, like the edges:** computed **once at index time** and serialized
+  (memory `centrality` map; SQLite `centrality(chunk_id, score, rank)` table), so both backends
+  read byte-identical scores and ranks. It never touches `searchHybrid`, so `test:eval` and the
+  search parity gate are unaffected. New store methods `hasCentrality()`, `getCentrality(id)`,
+  `topCentral(limit)` on both backends. Deterministic (sorted node set, fixed-order power
+  iteration, rounded scores, rank tie-broken by id).
+- **Daemon staleness:** a per-file update invalidates the centrality alongside the edges
+  (it is a whole-program quantity) — both engines drop it until the next full `idx-index`.
+- `test/edges.mjs` adds 8 tests (PageRank determinism, hub-outranks-leaf, confidence weighting,
+  serialization round-trip + store methods, memory↔sqlite parity, daemon invalidation, default-path
+  gating, and the two tools surfacing/omitting it). Full write-up:
+  `docs/internals/IMPROVEMENT_SYMBOL_CENTRALITY.md`.
+
+### Precise resolver provider + impact precision dial (A1) — Phase 2
+
+- **`--resolver precise`** (`INDEXER_RESOLVER` / `resolver` config) adds a pluggable resolver
+  provider for the symbol graph. The default `heuristic` provider keeps edge confidence exactly
+  `{ high, name_only }` (byte-identical). `precise` lifts the **provably-unambiguous** subset into
+  a third, stronger tier — **`resolved`** — a reference is `resolved` when there is no question
+  *which* definition it binds to: a **sole definition** (no same-named rival), or a **type-pinned
+  receiver** (typed receiver, the A3 inter-procedural fixpoint, `this.m()` inside the class, or an
+  explicit target class — type-resolved, so shadow-immune). `high` then means "ambiguous name,
+  import/proximity-disambiguated"; `name_only` means "ambiguous, no evidence."
+- **`impact_of_edit` gains a `precision` dial:** `standard` (default) follows the high-confidence
+  closure (`resolved` + `high`); **`strict`** follows **only `resolved`** edges — a
+  false-positive-free blast radius ("what *definitely* breaks"). Without a graph, `strict` follows
+  the `proven` query-time callers, so it degrades gracefully. Measured on the repo's own source:
+  of 325 `high` calls/refs edges, 303 are provably unambiguous (→ `resolved`), 22 remain heuristic
+  `high`; `name_only` is never promoted.
+- **Additive and parity-free:** the resolver only changes the `confidence` STRING on edges that
+  already exist (never adds, drops, or reorders them), so `findCallers`/`findReferers`
+  set-equivalence holds and both backends round-trip the `resolved` tier identically. `resolved`
+  carries the same A5 centrality weight as `high`, so enabling it does not perturb centrality. The
+  symbol-graph dedupe now keeps the **strongest** confidence (`resolved > high > name_only`); on the
+  default path the values are unchanged so the dedupe is identical. New module `mcp/resolver.mjs`;
+  `classifyCallers` / `findReferences` gain an additive `proven` flag.
+- **Honest scope:** v1 ships the unambiguous-binding tier the engine can already prove from
+  index-time signals. Detecting a local variable that **shadows** an import (and *refuting* that
+  edge), and cross-file precise resolution via stack-graphs, are the documented next layers — both
+  need per-language AST/scope analysis and an external toolchain that does not fit the air-gapped
+  default. `tree-sitter-stack-graphs` has no production Node binding, so a literal integration would
+  require a Rust CLI sidecar (rejected for v1). `test/edges.mjs` adds 5 tests; full write-up:
+  `docs/internals/IMPROVEMENT_PRECISE_RESOLVER.md`.
+
+### Sealed mode (F1) — Phase 3
+
+- **`--sealed [strict|local]`** (`INDEXER_SEALED`) turns graph-indexer's air-gapped default from a
+  behaviour you *trust* into a guarantee you can *verify and enforce*. Two tiers: **`strict`** =
+  zero network egress (lexical-only); **`local`** = loopback only (local Ollama/MLX on this box are
+  allowed, nothing leaves it). Bare `--sealed` = `strict`.
+- **Fail-closed**: `resolveConfig` refuses to construct a sealed config whose enabled features would
+  egress beyond the tier — the build exits 2 with a message naming each offending feature
+  (embeddings/enrichment/rerank/hyde), its reach (loopback vs off-box), and the remedy. No silent
+  downgrade.
+- **Runtime egress guard**: a deny-by-default hook on the in-process network paths (`net`/`tls`
+  sockets, `http(s)`, `fetch`) installed before any provider loads, so even an *accidental* call is
+  caught — turning "we don't call out" into "we *can't*." A best-effort `child_process` denylist
+  covers the obvious out-of-process egress tools (documented limitation: a child has its own socket
+  layer). `list_index_stats` reports `sealed` + `egress_guard: active`.
+- **Attestation**: `idx-index --attest` prints a deterministic manifest (tier, providers, egress
+  posture, any egressing features) — an auditor/CI artifact for regulated/air-gapped deployments.
+- **Opt-in, zero default impact**: `sealed: 'off'` default; sealing only *removes* capability
+  (fail-closed), never changes the store or ranking. The default lexical path is already
+  strict-compatible. New module `seal.mjs`; `test/seal.mjs` adds 11 tests (loopback classifier,
+  per-tier validation, guard block/permit/restore, child denylist, manifest, resolveConfig
+  fail-close). Design: `docs/internals/PHASE3_SEALED_MODE.md`.
+
+### Taint analysis (C2) — Phase 3
+
+- Two new **security** MCP tools (14 → 16): **`trace_taint`** traces untrusted data from a SOURCE
+  (request body/query/params, argv/env, stdin, socket/file reads) to a dangerous SINK (eval/exec,
+  SQL, fs/path, HTML, outbound request) across the call graph, returning the concrete source→sink
+  path tagged with category (rce/sqli/xss/path/ssrf) and confidence. **`find_tainted_sinks`** lists
+  every dangerous sink grouped by category, each flagged with whether an untrusted source reaches
+  it — the attack-surface map.
+- **Confidence model**: a same-function source→sink is `high` (`direct`); a source function that
+  transitively calls a sink function is `medium` (`reachable`); a sanitizer
+  (escape/encode/parameterise/`Number()`/`int()`…) on the path drops it to `low`. Flows are
+  deterministically ordered by category severity, then confidence, then location.
+- **Query-time, read-only, zero default cost**: implemented as on-demand tools (like
+  `get_call_graph`) — no index-time work, no serialization, **no parity surface**, the default index
+  is byte-identical, and the tools cost nothing unless invoked. **Air-gapped** (pure regex catalogs
+  + call-graph traversal; no model, no network). Walks A4 `getEdges` when present, else `chunk.calls`.
+- **Honest scope**: a **finder, not a verifier** (JS/TS + Python in v1). It favours precision but
+  misses flows through dynamic dispatch, reflection, ORM/query-builder indirection, and untyped
+  collections — every tool response says so, and "0 findings" ≠ "no vulnerabilities." New modules
+  `parse/taint-patterns.mjs`, `mcp/taint.mjs`; `test/taint.mjs` adds 9 tests; `test/mcp.mjs` asserts
+  the 16-tool surface. Design: `docs/internals/PHASE3_TAINT_ANALYSIS.md`.
+
+### Learned re-ranker — opt-in, measured ≈neutral, RRF stays default (D3)
+
+- **`--ranker learned` / `INDEXER_RANKER`** adds an opt-in, **zero-dependency** post-fusion
+  learning-to-rank layer: a small linear model re-orders the fused top-N using the RRF score plus the
+  program-structure signals the earlier phases produced — symbol **centrality** (A5), **resolved** /
+  SCIP **in-degree** (A4/A1) — and git recency. It is a *post-fusion* re-rank in the tool layer (like
+  the git boost / LLM reranker — never inside `db.searchHybrid`), so the **default RRF ordering is
+  byte-identical** and inference is a deterministic dot product (parity-safe). New module surface in
+  `search-core.mjs` (`learnedRerank`, `scoreLearned`, `extractRankerFeatures`,
+  `DEFAULT_RANKER_MODEL`); an offline trainer `bench/train-ranker.mjs` (zero-dep logistic regression,
+  **not shipped at runtime**); `test/ranker.mjs` (6 tests).
+- **Honest measurement (the headline):** the trainer was run over all five eval fixtures built with
+  `--symbol-graph` (3 161 candidates, 410 positives). Held-out rank-1 (95 queries, grouped per
+  query) — **RRF-only 0.579 vs the data-fit model 0.579**: the learned model **ties** RRF, it does
+  not beat it. The structure features get near-zero learned weight because the correct symbol is
+  usually *not* the most central / most-called one. Per the honest-metrics + sacred-default
+  invariants, a tie is no reason to displace the simpler, faster default — **RRF remains the default
+  ranker.** What
+  ships is the *capability*: the shipped `DEFAULT_RANKER_MODEL` is conservative + RRF-dominant and
+  measures **byte-identical to RRF** on the benchmark (a safe, never-regressing opt-in), and the
+  trainer lets users fit a *repo-specific* model where their call structure does carry signal. Full
+  write-up: `docs/internals/IMPROVEMENT_LEARNED_RANKER.md`.
+
+### Learned-sparse vocabulary expansion — opt-in, measured net-neutral, lexical stays default (B3)
+
+- **`--learned-sparse` / `INDEXER_LEARNED_SPARSE`** adds an opt-in, **zero-dependency, air-gapped**
+  learned-sparse channel (SPLADE / uniCOIL family): a model expands an NL query into a *weighted term
+  set* — the literal terms plus learned associates — fused as an extra sparse channel via RRF, to
+  close lexical-gap misses the Porter-stemming bridge cannot (stemming normalises morphology; it
+  cannot connect `auth` → `credential`/`token`). Rather than a neural MLM (which needs a downloaded
+  model + `@huggingface/transformers` and can't be measured air-gapped), the shipped provider **learns
+  the associations FROM THE CORPUS** at index time via positive PMI over chunk co-occurrence — genuine
+  learned-sparse, zero new dependency, deterministic. New module `search-sparse.mjs`
+  (`buildSparseModel`, `expandSparseQuery`) + a shared `scoreSparseExpanded` in `search-core.mjs`; the
+  model serialises as an **isolated `sparse_model` artifact** (memory JSON key / single-row sqlite
+  table) so it cannot perturb the lexical postings; `test/sparse.mjs` (7 tests).
+- **Presence-gated + NL-asymmetric + parity-by-construction.** A default index has no model → the
+  channel is inert → `test:eval` **byte-identical**. The channel fires only for natural-language
+  queries with a real expansion, so symbolic/exact lookups are byte-identical. Both backends score
+  through the *same* `scoreSparseExpanded` over identically-ordered terms → memory↔sqlite byte-identical.
+- **Honest measurement (the headline):** in-process A/B over the five core eval suites, swept across
+  the fusion weight — **the learned-sparse channel does not beat the lexical baseline at any weight.**
+  Higher weights regress semantic s@5 (the co-occurrence associates add noise: axios semantic s@5
+  90.9→72.7, gin 83.3→33.3 at 0.5); the best case is ≈neutral at the shipped weight 0.2 (semantic +
+  symbolic rank-1 both exactly preserved). Like BM25F / AST / the code-embedding NL weight before it,
+  this measured net-neutral-to-negative, so **lexical stays the default and B3 is opt-in only.** What
+  ships is the air-gapped infrastructure + the seam for a neural SPLADE provider + the honest finding;
+  a repo with genuine query/code vocabulary mismatch (which the benchmark fixtures lack) may see the
+  recall win — *measure on your own repo*. Full write-up:
+  `docs/internals/IMPROVEMENT_LEARNED_SPARSE.md`.
+
+### ColBERT-style late-interaction reranker — opt-in, measured helps-Go / neutral-JS (B4)
+
+- **`--rerank-provider late-interaction`** adds a third reranker provider (sibling to the generative
+  judge and the B2 cross-encoder): the multi-vector **MaxSim** idea from ColBERT, deployed as a
+  reranker over the over-fetched pool. It keeps MULTIPLE vectors per side and scores
+  `Σ_qi max_dj cos(qi, dj)`, so each query sub-vector finds its best document sub-vector — the
+  compositional-query win single-vector pooling loses. New module `colbert.mjs` (`maxSimScore`,
+  `encodeMultiVector`, `loadChunkVectors`, `rerankLateInteraction`) + `collectVectorsByKey` in
+  `engine/binary.mjs` + `embeddingBinPath()` on both engines; `test/colbert.mjs` (7 tests).
+- **No storage blow-up** (ColBERT's canonical cost): it reuses the document sub-vectors the engine
+  ALREADY stores (base + `|s` summary + `|wN` windows), reading a candidate's vectors from the shared
+  `.embeddings.bin` in one pass. **Air-gapped** (query multi-vectors from the same local embedder, no
+  new model/network), **parity-by-construction** (both backends read the same bin; MaxSim is
+  order-independent → identical re-order), and a post-retrieval re-order that **never mutates
+  `r.score`** and degrades to the original order on any shortfall → **default path byte-identical**.
+- **Honest measurement (at production retrieval depth):** a reranker can only re-order the pool it is
+  handed, so the measurement uses production's over-fetch (pool 15 / topM 12; the harness default of
+  10/8 manufactured a *false* JS regression that vanishes at the right depth). On real Ollama indexes,
+  `gin` (Go) improves — semantic rank-1 **0.00 → 0.40**, overall **0.68 → 0.77**, with **no symbolic
+  regression** (0.88 → 0.88) — while `express-js` (JS) is **neutral** (every metric unchanged: overall
+  0.65, semantic rank-1 0.43, s@5 0.57). **At production depth B4 helps Go and is neutral on
+  JavaScript — the "helps Go/Python" half of the known reranker profile without the JS regression the
+  other rerankers show.** Small-n (gin semantic n=5), so directional. It still needs a query-time
+  model and does not uniformly win across languages, so it stays opt-in (default off), the trade-off
+  is printed at startup, and the default path is byte-identical. Requires `--embeddings` (a no-op
+  without it). Approximates true token-level ColBERT with a sentence embedder over query sub-units; a
+  token-level model is a clean future drop-in behind `encodeMultiVector`. Full write-up:
+  `docs/internals/IMPROVEMENT_LATE_INTERACTION.md`.
+
+### Taint analysis hardening — index-time serialization + Java/Go (C2)
+
+- **Opt-in index-time serialization (`--taint` / `INDEXER_TAINT`).** The taint flow set can now be
+  precomputed once at index time and **serialized into the index** (a `taint` table in SQLite, a
+  `taint` JSON key in memory) so `trace_taint` / `find_tainted_sinks` answer **instantly** instead of
+  walking the call graph per query — the flagship win for large, security-sensitive repos. Computed
+  against the symbol-graph edges when `--symbol-graph` is on (cross-function precision), else
+  `chunk.calls`. Like A4 edges / A5 centrality: **computed once → both backends serve byte-identical
+  flows** (`hasTaint()` / `getTaintFlows()` added to the store contract; an incremental file update
+  clears it; the tools fall back to a live scan when absent). `buildTaintGraph` gains a fast path
+  that serves cached flows filtered to the query's category/depth envelope (a deeper `max_depth` or
+  direct-only query transparently recomputes). **Default index byte-identical** (off → no
+  serialization, query-time as before); air-gapped; `list_index_stats` surfaces the flow count.
+- **`trace_taint` / `find_tainted_sinks` now cover Java and Go** in addition to JS/TS and Python.
+  `parse/taint-patterns.mjs` gains `java` and `go` source / sink / sanitizer catalogs: Java —
+  `request.getParameter` / `@RequestParam` / `System.getenv` sources, `Runtime.exec` / `ProcessBuilder`
+  (rce), JDBC/JPA statement execute (sqli), servlet/JSP writers (xss), `File`/`Paths`/`Files` (path),
+  `URL`/`RestTemplate`/`HttpClient` (ssrf), with `PreparedStatement` / `URLEncoder` / `Integer.parseInt`
+  sanitizers; Go — `r.URL.Query()` / `r.FormValue` / gin·gorilla helpers / `os.Getenv` sources,
+  `exec.Command` (rce), `db.Query`/`Exec` (sqli, with a lookbehind so `r.URL.Query()` is not mistaken
+  for a DB call), response-writer / `template.HTML` (xss), `os.Open`/`filepath.Join` (path),
+  `http.Get`/`client.Do` (ssrf), with `strconv.Atoi` / `html.EscapeString` / `filepath.Clean`
+  sanitizers. `test/taint.mjs` adds Java + Go scan/flow/sanitizer tests plus serialization tests
+  (cache == live, fast-path category/depth filtering, memory↔sqlite parity, incremental-clear).
+  Design: `docs/internals/PHASE3_TAINT_ANALYSIS.md`.
+
+### SCIP resolver provider (A2) — Phase 3
+
+- **`--resolver scip` + `--scip-index <path>`** (`INDEXER_SCIP_INDEX` / `scipIndex` config)
+  completes the A1 precision story with the realistic, air-gapped cross-file backend. Where
+  `--resolver precise` only *relabels* edges the heuristic emitted, `scip` ingests a
+  **locally-generated SCIP index** (the user's own `scip-typescript` / `scip-python` /
+  `scip-java` / `rust-analyzer --scip` — produced out of band, on-box) and resolves edges by
+  **real cross-file bindings**: it promotes a confirmed `calls` edge to the `resolved` tier **and
+  suppresses the wrong-target fan-out** an ambiguous same-name reference would otherwise emit —
+  the disambiguation the index-time heuristic structurally cannot do. (Scoped to `calls` edges in
+  v1; `extends`/`type` edges stay heuristic because the binding relation is kind-agnostic.)
+- **Zero new dependency.** The `.scip` protobuf is decoded by a **hand-rolled, zero-dependency**
+  wire reader (`parse/scip.mjs`) that walks only the four fields A2 needs; a SCIP-shaped JSON file
+  is also accepted. Occurrences align to chunks by 1-based line range (SCIP is 0-based; chunks
+  store no columns → line granularity). Air-gapped (a local file read only) — so **`--sealed
+  strict` + `--resolver scip` is a valid sealed, precisely-resolved index.**
+- **Parity-free, default byte-identical, never-worse.** Like A1, it runs once at index time and
+  serializes one edge list into both backends (no per-backend code; memory↔sqlite identical). The
+  `resolveEdges` provider method exists only on the `scip` provider, so the default
+  `heuristic`/`precise` path is byte-identical. **Absence ≠ refutation, even under partial
+  coverage**: suppression requires that SCIP *actually recorded* the def it drops (`definedChunks`
+  gate), so a covered caller referencing an *uncovered* def (a `.ts`-only run that never saw a
+  `.js` def, a stale `.scip`) keeps the heuristic edge — never dropped. Coverage (matched docs,
+  bindings, resolved-tier count) is printed at index time; a missing/unreadable/0-match index warns
+  loudly and falls back to heuristic (the build never fails).
+- **v2 — precise cross-file references for `find_references`.** The same binding relation is
+  inverted (`buildScipReferers`) into a precise *referenced-by* map and surfaced by
+  `find_references` (and `explain_symbol`) as a **`resolved`-tier reference dimension**. SCIP
+  occurrences are kind-agnostic ("symbol used here"), which is exactly what `find_references`
+  answers — so this single dimension both **disambiguates** same-named symbols (each referer is
+  attached to the one definition SCIP bound it to, not every same-named def the name heuristic
+  matches) and **recovers references the AST/name heuristic misses entirely** (the v1 "recall
+  concern" — including type usages in languages whose `type_refs` channel is empty). It is
+  serialized as an **isolated index artifact** (`scip_refs`), exactly like taint/centrality, so it
+  **cannot perturb the A4 edges or A5 centrality** and is parity-free; empty (and so default
+  byte-identical) on any index built without a SCIP resolver, and dropped on incremental update.
+  `list_index_stats` reports the count.
+- **Module surface:** `parse/scip.mjs` (`loadScip`, `buildScipBindings`, `buildScipReferers`,
+  `normalizeScipPath`) + `createScipResolver` in `mcp/resolver.mjs`; the `scip_refs` artifact in
+  both engines (`hasScipRefs`/`getScipReferers`/`scipRefCount`). `test/scip.mjs` adds 15 tests
+  (protobuf round-trip, alignment, promote + suppress over a real ambiguous-name fixture,
+  **partial-coverage soundness**, **calls-only cross-kind isolation**, memory↔sqlite parity, plus
+  v2: inversion determinism, precise+attributed `find_references` dimension, references parity,
+  **sacred-default byte-identity**, incremental staleness). Write-up:
+  `docs/internals/IMPROVEMENT_SCIP_RESOLVER.md`.
+
+### Sealed-mode attestation signing (F1 hardening)
+
+- The sealed-mode attestation manifest can now be **cryptographically signed and verified** — turning
+  "here is my air-gapped posture" into a tamper-evident, auditor-verifiable statement. `seal.mjs`
+  adds `signManifest` (Ed25519 / RSA / EC) → a `{ manifest, signature: { alg, keyType,
+  publicKeyFingerprint, value } }` envelope, and `verifySignedManifest` → `{ valid, reason }` (never
+  throws). The signature is taken over a `canonicalJson` (recursively key-sorted) form, so it
+  survives a file round-trip and any future manifest key-order change.
+- **CLI:** `idx-index --gen-attestation-key <prefix>` writes an Ed25519 keypair (`.key` private /
+  `.pub` public); `idx-index --attest --sign-key <path>` prints the signed envelope;
+  `idx-index --verify-attestation <file> --pub-key <path>` verifies the signature **and** reports
+  whether the attested manifest still matches the current effective config (policy drift → exit 3).
+- **Zero dependency, air-gapped** (`node:crypto` only — no network, works under `--sealed strict`);
+  the default path is untouched (these are pre-indexing CLI actions that exit before any build).
+  `test/seal.mjs` adds 4 tests (canonical JSON, Ed25519 + RSA sign/verify, tamper/wrong-key/alg
+  rejection, JSON round-trip). Design: `docs/internals/PHASE3_SEALED_MODE.md`.
+
 ## [2.0.0] — 2026-06-21
 
 This is a major release. The public API (MCP tools, CLI flags, config keys) has grown significantly, but the default behaviour — lexical-only search, in-memory store, zero external dependencies — is backward-compatible. Internal modules were reorganized into `engine/`, `mcp/`, and `parse/` (see [Module reorganization](#module-reorganization-breaking-only-for-deep-imports) below) — breaking only for code that deep-imported internal files.