codesearch-mcp 0.2.3-alpha → 0.3.0-alpha

package/CHANGELOG.md

@@ -3,7 +3,37 @@
 All notable changes to this project will be documented in this file.
 Format based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
-## [Unreleased]
+## [0.3.0-alpha] - 2026-10-31
+
+### Added
+
+- **One `tool call` log line per MCP tool call** — at `logging.level` `info` (or `debug`) the MCP server writes one line per completed call, whatever the outcome (result, handler error, unknown tool, refused after a failed startup, cancelled by the client): `tool`, `args` (strings cut at 500 characters, arrays at 20 items), `isError`, `responseChars` (the response size — never its content), `ms`, and the caller: `client` (`{name, version}` from `initialize`), `connectionId` (random, one per server process), `session` (`CLAUDE_CODE_SESSION_ID`, or `null`), `meta` (the request's `_meta` without `progressToken`), `callId` (a per-process counter) and, for Claude Code, `toolUseId`; `cancelled: true` marks a call the client cancelled. Find them with `jq 'select(.msg=="tool call")'`.
+- **Call context on every MCP server log line** — every line of the MCP server carries its process's `connectionId`, and every line written during a tool call, at any level and from any service (a search debug trace, a `rerank failed` warning, a handler error), also carries that call's `callId` and `toolUseId`. `rag-index`, `rag-web` and eval logs are unchanged.
+- **`rag-index eval` reports each search stage separately** _(internal development tool: the eval suites, corpora and results live in this repository and are not shipped in the npm package)_ — every case is searched once and scored at the `vector` stage (the candidate pool's cosine order as the reranker receives it, cut to 10 — not the same as a `rerank: false` search, which fetches a smaller pool) and, when a reranker is configured, the `reranked` stage. Each case also records where its answer sits anywhere in the candidate pool and the pool's size; the run reports pool recall and how many cases got a pool shorter than the size their search requested. Within a run, the `vector` → `reranked` change is reported per category as improved / worsened / unchanged cases, the cases that gained or lost hit@5, and an exact two-sided sign-test p-value. A failed `rerank()` is still scored on the vector-order fallback `search_codebase` returns, but counted and the `reranked` stage flagged as degraded. Result files record the search conditions (similarity threshold, `reranker.overfetch`, `reranker.candidateFloor`, and the pool sizes and HNSW settings the searches reported). `--baseline` compares per stage with the same paired counts, compares pool recall and short pools, names stages present on one side only, and warns on any differing condition or rerank failures. Result files written before this change load as a baseline in the stage their recorded reranker implies, with "no pool data" and a missing-conditions warning. Each category row shows the share of one case, so a single-case move is visible as such. No change to `search_codebase` results.
+- **`rag-index eval` runs on frozen corpora** _(internal development tool: the eval suites, corpora and results live in this repository and are not shipped in the npm package)_ — the default suite directory `benchmarks/eval/` now holds suites written against two pinned repositories, `twenty` (Nx: NestJS + GraphQL server, React front, small packages) and `arbook-backend` (Express + Apollo, schema-first GraphQL), declared in each suite as `corpus: { root, commit }`. Before any search, eval checks that each corpus is checked out at that commit with no change to tracked files, no untracked file the indexer would collect, and untracked `.ragignore` files equal to the copies in `benchmarks/corpus/`; otherwise it exits 1 without searching or writing a result file, naming every difference. Result files record per corpus the commit, its indexed projects, chunk count and a fingerprint of the indexed chunks (positions, text, owners, embeddings), plus the number of live cases; `--baseline` warns when a corpus is on one side only, its commit or fingerprint differs, or either run includes live cases. Metrics, pool recall and stage change are also reported per corpus. The former suites on `RAG` and `fix-flow-crm-backend` moved to `benchmarks/eval-live/` (`--suite benchmarks/eval-live`) as smoke tests. Setup: keep the corpus checkouts at their commits and `rag-index --project <p> --reindex` every corpus project. While it runs, eval prints one progress line per case to stderr (`[12/153] T_ID_012 vector 3 → reranked 1 (4.2s)`).
+- **Voyage reranker** — `reranker.provider: voyage` reranks through Voyage AI (`rerank-3` by default, `rerank-3-lite` as the cheaper alternative) with any embedder, including a local Infinity one. Its defaults are its own: `reranker.baseURL` `https://api.voyageai.com/v1`, endpoint `/v1/rerank`. The key comes only from `reranker.apiKey` — never from `embedding.apiKey` — and a Voyage reranker without one stays disabled, with a warning in the log at `logging.level` `warn` or lower; startup is unaffected.
+- **`rag-setup` asks for a reranker for every embedding provider** — `voyage`, `openai-compatible` (base URL, model, optional key stored as `RERANKER_API_KEY`) or `none`, plus `infinity` (same container) for an Infinity embedder; the default follows the embedding provider (`infinity` → `infinity`, `voyage` → `voyage`, `openai-compatible` → `none`). The Voyage reranker key is always asked separately and stored as `VOYAGE_RERANK_API_KEY`, so it never overwrites the embedding key; a blank key offers to disable the reranker. The wizard always writes an explicit `reranker` block.
+- **Current Voyage embedding models** — `rag-setup` suggests `voyage-code-4`, and the model registry and dashboard suggestions list `voyage-code-4`, `voyage-4-large`, `voyage-4` and `voyage-4-lite` (1024 dimensions by default, up to 2048). `voyage-code-3`, `voyage-3` and `voyage-3-lite` are no longer listed but still work — their dimension is probed at index time. The dashboard suggests `rerank-3` / `rerank-3-lite` for `reranker.model` when the saved `reranker.provider` is `voyage`.
+
+### Changed
+
+- **BREAKING (config): `logging.searchMetrics` is removed** — the `search metrics` line is now a plain `info` line: set `logging.level: info` to get it, and it is never written at `warn`/`error`. A config file that still contains the key loads and ignores it; `rag-config set logging.searchMetrics` fails as an unknown key.
+- **Errors are logged as structured records under `err`** — every caught error in the log now carries `err` with its type, message, stack (up to 30 frames), driver fields (`code`, `errno`, `syscall`, `detail`, `hint`, …) and its cause chain as `causes[]` (up to 5 links); a failed database query keeps its SQL (≤ 500 characters) and parameter count with up to 30 values. `reason` now holds only domain codes, no longer the error message. The indexer's console line names the root cause: `[indexer] Failed: <file> — <message> (<code>)`.
+- **The candidate query always fetches the `poolSaturated` probe row** — `fetchLimit + 1` rows on every search, so `hnsw.ef_search` is now `fetchLimit + 1` for pools of 40 or more (pools of 39 or fewer keep the floor of 40), whatever the logging config. Results near the threshold boundary can shift marginally; `rag-index eval --baseline` against a result taken without `searchMetrics` warns that `efSearch` differs — re-run the baselines once.
+- **Chunks follow declarations and members** — **requires `rag-index --reindex`.** A declaration's chunk now starts at its doc comment (JSDoc or line comments directly above it, no blank line between) or its first decorator, so doc comments and decorators are embedded with the code they belong to, and — when they fit `indexing.chunkSize` together with the declaration's name line — no chunk is a lone `@Injectable()` or a doc comment cut off from the declaration below it; otherwise the decorators alone stay with the name line when they fit with it, and a longer doc comment or decorator is cut by the character splitter. Top-level declarations of at least 200 characters that share no line with another declaration get chunks of their own (enums and `declare module` blocks are packed like other module-scope code); smaller ones (type aliases, one-line constants) are packed together. A class or object literal longer than `indexing.chunkSize` is split by its members: each member of at least 200 characters that shares no line with a sibling or with the class's first or last line gets its own chunk, starting at its doc comment; the header, fields, getters and short constructors are packed together; only a member still longer than `chunkSize` is cut by the character splitter, inside itself. A class that fits `chunkSize` stays one chunk. Import and re-export statements — multi-line ones, `import x = require(…)` and `export … from` included — are removed before packing, so they are never mixed into a code chunk. Short leftover pieces without declaration code (`export default meta;`, a closing brace) are merged into a neighbouring chunk when the result fits `indexing.chunkSize`, no removed import block lies between and — when the leftover lies inside a declaration with chunks of its own, or a unit cut by the character splitter — the neighbour lies inside the same one. A short file prologue before the imports (`'use client';`, `/* @license … */`, a shebang, lint directives — under 60 characters) is removed with them; a longer file-header comment followed by imports stays a chunk of its own.
+- **`Symbol:` in `search_codebase` names the smallest declaration containing all of the chunk's declaration code** — not the declaration of the chunk's first line, so a chunk is never labelled with a method it only overlaps. A chunk of several small members shows their class; a chunk of several small top-level declarations, or of module-scope code only, shows none. Module-scope code (top-level statements, enums, `declare module` blocks) belongs to no declaration, so a chunk packing it with one small declaration shows that declaration. The same rule applies to schema files and the `character` strategy.
+- **`search_codebase` returns at most 2 results per file by default** — new optional `per_file` argument (integer ≥ 1, default 2) caps the results taken from one file; the remaining slots are filled from the next candidates (a larger pool without reranking, the full rerank order with it). Pass a higher `per_file` when several answers are expected in one file. `rag-index eval` is unaffected (it searches with `per_file` equal to its limit).
+- **The MCP server's `rerank` guidance follows the re-measured eval** — `rerank: false` is still advised for "what does X do" / "where is X" lookups (reranking costs them −0.13 MRR@10), but conceptual/architectural queries are no longer described as helped by reranking: on the new chunking it is roughly neutral for them (−0.05, not significant).
+- **`code_chunk` gains a nullable `endLine` column**, added automatically on the next `rag-index`; rows indexed before it keep `null` until re-indexed.
+- **The reranker sees each chunk's file and owning symbol** — every document sent to the reranker starts with one line `File: <path> | Symbol: <owner>` (`File: <path>` for a chunk without an owner; line breaks in either name become spaces), so two same-named methods in different classes are no longer indistinguishable to it. Applies to every reranker provider, with no config key. Result format, stored chunks and the vector stage are unchanged; the order of reranked results can change. No re-index required.
+- **Reranking is off by default for every non-`infinity` embedder without a `reranker` block** — when `reranker.enabled` is absent, reranking is on only with an `infinity` embedder or when the `reranker` block names a server (`provider`, `baseURL` or `url`); a block with only `model` does not turn it on. Before, any other embedding config without a `reranker` block sent every search's rerank request to the embedding host. On a cloud host (OpenAI, Voyage) that could not succeed and fell back to vector order; a self-hosted `openai-compatible` server that does serve `/rerank` (vLLM, an external Infinity) now needs `reranker.enabled: true` or a `reranker.provider`/`baseURL` to keep reranking. An explicit `reranker.enabled` behaves as before.
+- **A malformed rerank response fails at once** — a 2xx response without the provider's result list, or with indices outside the document list, fails the rerank call with an error naming the URL instead of being retried as a network error; search falls back to vector order as for any rerank failure.
+- **`get_dependents` / `get_dependencies` group their direct results by file** — one line per importing/imported file instead of one per `code_dependency` row, listing the distinct names imported (`default` / `* as namespace` / `side-effect` for a nameless import, re-exports grouped after plain imports under one `re-export ` prefix, `export *` for a namespace re-export, capped at 8 then `(+N more)`); `get_dependents` additionally lists up to 3 distinct enclosing declarations using the import (`` · in `PlansPage` ``, `(+N more)` beyond that) — only those resolved unambiguously, so the part can be partial or absent. The direct section header now states the file count (`**Direct:** 8 files`). Transitive sections and the empty-result/grep-fallback text are unchanged. Output format only — no MCP tool signature, config key or CLI command changes, no re-index required.
+
+### Fixed
+
+- **"MCP server is unavailable" names the cause when PostgreSQL refuses every address** — a connection refused on both `::1` and `127.0.0.1` arrives as an `AggregateError` without a message, so the tool response read `MCP server is unavailable: ` with nothing after it. It now lists the refused addresses (`connect ECONNREFUSED ::1:5432; connect ECONNREFUSED 127.0.0.1:5432`), and the log record's `err` carries each of them under `errors`.
+- **`search_codebase` on large projects returns the configured candidate pool** — a `project`-scoped search on a project large enough for PostgreSQL to use the HNSW index (measured from ~5.8 k chunks; the crossover depends on table statistics), or any unscoped search, returned at most 40 candidates and as few as 0 even when hundreds of chunks passed the threshold: the index scan stopped at `hnsw.ef_search` (default 40) before the project filter and threshold were applied. Each search now sets `hnsw.ef_search` to the requested row count (at least 40, at most 1000) and `hnsw.iterative_scan = relaxed_order` for its own query, so the pool sized by `reranker.candidateFloor` / `reranker.overfetch` is actually fetched unless the iterative scan stops at `hnsw.max_scan_tuples` (20 000) first. Reranked results may shift, since the reranker now sees the full pool. pgvector ≥ 0.8 is recommended: an older version gets the raised `ef_search` only, with one `warn` line in the log (hidden at the default `logging.level` `error`). `search metrics` lines and `rag-index eval` search conditions record the `efSearch` / `iterativeScan` used. No config key, MCP tool signature or index change; no re-index required.
 
 ## [0.2.3-alpha] - 2026-10-21
 

package/README.md

@@ -7,7 +7,7 @@ Ask your AI agent things like *"where is the payment retry logic?"* or *"show me
 **Why codesearch-mcp:**
 
 - **Dramatically fewer tokens** — without RAG, answering a cross-project question means discovery calls, directory listings, and reading several wrong files before finding the right one; a workspace of 10 projects can easily mean 50 000+ lines to sift through. With RAG, one `search_codebase` call returns 5 targeted chunks (~200–500 lines total) from across *all* projects at once — the model gets the right code immediately, with no exploration overhead
-- **Reranker narrows the result further** — a cross-encoder reranker (enabled by default) re-scores the candidate pool so the top results are the most relevant, not just the closest vector; fewer results means less context consumed per answer
+- **Reranker narrows the result further** — a cross-encoder reranker (on by default with the local Infinity backend; local or Voyage AI) re-scores the candidate pool so the top results are the most relevant, not just the closest vector; fewer results means less context consumed per answer
 - **Semantic, not grep** — embeddings capture intent; queries in plain English or pseudocode find the relevant code regardless of naming
 - **Local and private** — default setup runs entirely on your machine (Infinity embeddings + PostgreSQL); no code leaves your network
 - **Code-structure aware** — chunks align to function and class boundaries; each result shows which declaration it belongs to (`InvoiceService.create`, `collectFiles`, …), so the model understands context without reading the surrounding file
@@ -31,7 +31,7 @@ See `CHANGELOG.md` for version history.
 ## Requirements
 
 - Node.js 18+
-- PostgreSQL 15+ with pgvector (the symbol index's unique index relies on `NULLS NOT DISTINCT`, added in PostgreSQL 15)
+- PostgreSQL 15+ with pgvector (the symbol index's unique index relies on `NULLS NOT DISTINCT`, added in PostgreSQL 15); pgvector ≥ 0.8 recommended (filtered search on large projects; older versions work with reduced recall)
 - Docker (for local DB + Infinity) — or bring your own PostgreSQL with pgvector + an embedding backend (see [Embedding providers](#embedding-providers))
 
 ## Installation
@@ -106,7 +106,7 @@ Every file or directory argument is **workspace-relative** (relative to `workspa
 
 | Tool | Description |
 |------|-------------|
-| `search_codebase` | Semantic search across all indexed projects. Accepts `include_graph_context` to append direct imports/dependents to each result, `include_neighbors` for surrounding chunks, `threshold` to override the similarity threshold for that query, and `rerank` to force reranking on/off for that call regardless of `reranker.enabled` |
+| `search_codebase` | Semantic search across all indexed projects. Accepts `include_graph_context` to append direct imports/dependents to each result, `include_neighbors` for surrounding chunks, `threshold` to override the similarity threshold for that query, `rerank` to force reranking on/off for that call regardless of `reranker.enabled`, and `per_file` (default 2) to cap how many results come from one file |
 | `list_projects` | List projects with metadata, chunk counts and each project's directory (`path:` line) |
 | `find_symbol` | Find a declaration or member by exact name — bare, qualified (`Class.method`), quoted, or computed — faster than semantic search. Also finds a NestJS GraphQL operation, resolve field or type by its GraphQL name (`kind: graphql_query`/`graphql_mutation`/`graphql_subscription`/`graphql_field`/`graphql_type`) |
 | `list_files` | Browse the directory tree of any workspace directory |
@@ -150,7 +150,7 @@ Pass `include_graph_context: true` to `search_codebase` to see direct imports an
 
 Useful for immediately understanding a file's role in the dependency graph.
 
-Each result also shows a **Symbol** breadcrumb — the declaration the chunk belongs to (`InvoiceService.create`, `collectFiles`, …) — so you can see the enclosing class/method without opening the file. It is omitted for module-scope chunks and for projects indexed before this feature (re-index to populate).
+Each result also shows a **Symbol** breadcrumb — the smallest declaration containing all of the chunk's declaration code (`InvoiceService.create`, `collectFiles`, …) — so you can see the enclosing class/method without opening the file. A chunk packing several small members shows their class. Module-scope code (top-level statements, enums, `declare module`) belongs to no declaration, so a chunk packing it with one small declaration shows that declaration; the breadcrumb is omitted for a chunk of module-scope code only, for chunks packing several small top-level declarations, and for projects indexed before this feature (re-index to populate).
 
 ## Excluding files
 
@@ -174,6 +174,12 @@ npx pino-pretty < ~/.config/codesearch-mcp/logs/indexer/current.log
 # Detailed indexing diagnostics (timing per file: deps / embed / db)
 rag-config set logging.level debug
 
+# See what an agent called (needs logging.level info or debug), grouped by session and connection
+jq -c 'select(.msg=="tool call") | {session, connectionId, callId, tool, args, isError, ms}' \
+  ~/.config/codesearch-mcp/logs/mcp/current.log
+jq -s 'map(select(.msg=="tool call")) | group_by(.session, .connectionId) | map({session: .[0].session, connectionId: .[0].connectionId, calls: map(.tool)})' \
+  ~/.config/codesearch-mcp/logs/mcp/current.log
+
 # Disable file logging
 rag-config set logging.dir false
 ```
@@ -205,10 +211,10 @@ rag-config edit                      # open config in $EDITOR; validates on save
 | `embedding.dimensions` | (derived) | Vector width override; required for models whose native width exceeds 2000 (Matryoshka). Otherwise derived from model → registry → runtime probe |
 | `embedding.batchSize` | `32` | Texts per batch embedding request |
 | `search.similarityThreshold` | `0.6` | Max cosine distance — raise if search returns nothing |
-| `reranker.enabled` | `true` | Enable cross-encoder reranking; requires `reranker.model` |
-| `reranker.provider` | `openai-compatible` | Rerank backend — `openai-compatible` `/rerank` only (Infinity, TEI, vLLM) |
-| `reranker.baseURL` | `embedding.baseURL` | Rerank server base URL (Infinity serves embed + rerank from one server) |
-| `reranker.model` | `mixedbread-ai/mxbai-rerank-xsmall-v1` | Reranker model name |
+| `reranker.enabled` | context | Enable cross-encoder reranking. When absent: on with `infinity` embedding or when the `reranker` block names a server (`provider`/`baseURL`/`url`), off otherwise |
+| `reranker.provider` | `openai-compatible` | Rerank backend — `openai-compatible` `/rerank` (Infinity, vLLM) or `voyage` (Voyage AI, needs `reranker.apiKey`) |
+| `reranker.baseURL` | per-provider | Rerank server base URL — `embedding.baseURL` for `openai-compatible` (Infinity serves embed + rerank from one server), `https://api.voyageai.com/v1` for `voyage` |
+| `reranker.model` | per-provider | Reranker model name — `mixedbread-ai/mxbai-rerank-xsmall-v1` for `openai-compatible`, `rerank-3` for `voyage` |
 | `reranker.overfetch` | `5` | Candidate multiplier before reranking (`limit × N`) |
 | `reranker.candidateFloor` | `80` | Minimum candidate pool fed to the reranker regardless of `limit` — `fetchLimit = min(max(limit × overfetch, candidateFloor), 200)`. Raise on GPU/fast hosts for better recall; lower on weak CPU to cut latency |
 | `indexing.chunkSize` | `1500` | Max characters per chunk |
@@ -218,9 +224,8 @@ rag-config edit                      # open config in $EDITOR; validates on save
 | `indexing.chunkStrategy` | `ast` | How TS/JS files are chunked: `ast` = boundaries aligned to functions/classes, `character` = fixed-size splitter. Re-index to apply |
 | `search.defaultLimit` | `5` | Default result count returned by `search_codebase` |
 | `search.maxLimit` | `20` | Hard cap on result count |
-| `logging.level` | `error` | Log level written to file: `error` \| `warn` \| `info` \| `debug` |
+| `logging.level` | `error` | Log level written to file: `error` \| `warn` \| `info` \| `debug`. `info` adds one `tool call` line per MCP tool call and one `search metrics` line per search |
 | `logging.dir` | `~/.config/codesearch-mcp/logs` | Base log directory, or `"false"` to disable |
-| `logging.searchMetrics` | `false` | Emit one structured search-tuning line per query (at `info`, independent of `logging.level`) — use it to size `candidateFloor`/`overfetch`/`threshold` for your hardware |
 
 ## Embedding providers
 
@@ -234,7 +239,7 @@ rag-config edit                      # open config in $EDITOR; validates on save
 | `openai-compatible` | OpenAI cloud, or any OpenAI-style server (TEI, vLLM, LM Studio, LocalAI, **or Ollama via its `/v1` endpoint**). | `baseURL` (+ `apiKey` for cloud) |
 | `voyage` | Voyage AI cloud, code-specialized embeddings. | Voyage `apiKey` |
 
-Easiest path: `rag-setup` asks for the provider, model, and any API key. Haven't run setup? It's the place to start. To change things later, edit config directly (below).
+Easiest path: `rag-setup` asks for the provider, model, any API key, and a reranker. Haven't run setup? It's the place to start. To change things later, edit config directly (below).
 
 ### Switching provider or changing the model
 
@@ -246,14 +251,14 @@ rag-config set embedding.provider openai-compatible
 rag-config set embedding.baseURL  https://api.openai.com/v1
 rag-config set embedding.model    text-embedding-3-small
 rag-config set embedding.apiKey   env:OPENAI_API_KEY     # key lives in .env, not config.json
-rag-config set reranker.enabled   false                  # OpenAI has no /rerank endpoint
+rag-config set reranker.enabled   false                  # only if rag-setup chose the `infinity` reranker: it stays aimed at the old server
 rag-index --reindex
 
 # Switch to Voyage AI
 rag-config set embedding.provider voyage
-rag-config set embedding.model    voyage-code-3
+rag-config set embedding.model    voyage-code-4
 rag-config set embedding.apiKey   env:VOYAGE_API_KEY     # key lives in .env, not config.json
-rag-config set reranker.enabled   false                  # Voyage has no /rerank endpoint
+rag-config set reranker.enabled   false                  # only if rag-setup chose the `infinity` reranker (or configure the Voyage reranker, below)
 rag-index --reindex
 
 # Change only the model on the current provider
@@ -272,9 +277,9 @@ rag-index --reindex
 Per-provider notes:
 
 - **infinity** — the model set is bound at container startup, so changing `embedding.model` (or the reranker model) **recreates** the managed container on the next `rag-setup`/`rag-index`/dashboard save. Defaults: `jinaai/jina-embeddings-v2-base-code` + reranker `mixedbread-ai/mxbai-rerank-xsmall-v1`. Both models are included in the compose template and reranking is **enabled by default** (`reranker.enabled: true`).
-- **openai-compatible** — point `embedding.baseURL` at the server; set `embedding.apiKey` (as `env:VAR`) for cloud, leave empty for local servers (TEI/vLLM/LM Studio/**Ollama**). E.g. `text-embedding-3-small`, `text-embedding-3-large`. Cloud endpoints (OpenAI, Azure) don't expose `/rerank` — disable the reranker or point it at a separate server (see [Reranker with a cloud embedder](#reranker-with-a-cloud-or-external-embedder)).
+- **openai-compatible** — point `embedding.baseURL` at the server; set `embedding.apiKey` (as `env:VAR`) for cloud, leave empty for local servers (TEI/vLLM/LM Studio/**Ollama**). E.g. `text-embedding-3-small`, `text-embedding-3-large`. Cloud endpoints (OpenAI, Azure) don't expose `/rerank`, so without a `reranker` block reranking stays off unless you configure a separate rerank server; a block from `rag-setup`'s `infinity` reranker choice keeps reranking on at the old server until you set `reranker.enabled false` (see [Reranker with a cloud embedder](#reranker-with-a-cloud-or-external-embedder)).
 - **Ollama** (via `openai-compatible`) — set `embedding.baseURL` to `http://localhost:11434/v1` and `embedding.model` to any model you've pulled (e.g. `bge-m3`). No dedicated code path, no auto-pull — pull the model yourself with `ollama pull <model>` first.
-- **voyage** — set `embedding.apiKey` (as `env:VAR`); e.g. `voyage-code-3`, `voyage-3`. Voyage doesn't expose a `/rerank` endpoint in the OpenAI-compatible format — disable the reranker or point it at a separate server (see [Reranker with a cloud embedder](#reranker-with-a-cloud-or-external-embedder)).
+- **voyage** — set `embedding.apiKey` (as `env:VAR`); e.g. `voyage-code-4` (default in `rag-setup`), `voyage-4-large`, `voyage-4`, `voyage-4-lite`. Older names such as `voyage-code-3` still work — their dimension is probed. Without a `reranker` block, reranking stays off unless you configure one (a block from `rag-setup`'s `infinity` reranker choice keeps it on at the old server until you set `reranker.enabled false`) — e.g. the Voyage reranker (see [Reranker with a cloud embedder](#reranker-with-a-cloud-or-external-embedder)).
 
 The vector **dimension** is derived automatically (model registry → one-shot probe); set `embedding.dimensions` only to force a Matryoshka-reduced width for models whose native size exceeds 2000.
 
@@ -291,23 +296,36 @@ The same "already running?" probe applies everywhere: a reachable backend defaul
 
 ### Reranker with a cloud or external embedder
 
-The reranker is **enabled by default** (`reranker.enabled: true`) and `reranker.baseURL` defaults to `embedding.baseURL`. Cloud embedding endpoints (OpenAI, Voyage, Azure) don't serve `/rerank` — the reranker call will fail silently and fall back to vector order on every query. Fix it one of two ways:
+With a cloud embedder (OpenAI, Voyage, Azure) and no `reranker` block, reranking is **off** — those endpoints don't serve `/rerank`. `rag-setup` asks for a reranker after the embedding step for every provider (`voyage`, `openai-compatible`, `none`, plus `infinity` for an Infinity embedder) and always writes an explicit `reranker` block. To configure one by hand:
+
+These recipes edit whatever `reranker` block is already stored, so each one sets `reranker.enabled true` (an explicit `false` from `rag-setup`'s `none` choice would otherwise win) and clears the keys that would point the requests at the old server.
 
-**Option A — disable reranking** (simplest, recommended for cloud-only setups):
+**Voyage reranker** (works with any embedder; its key is separate from the embedding key):
 
 ```bash
-rag-config set reranker.enabled false
+echo 'VOYAGE_RERANK_API_KEY=pa-…' >> ~/.config/codesearch-mcp/.env
+rag-config set   reranker.enabled  true
+rag-config set   reranker.provider voyage
+rag-config set   reranker.model    rerank-3                  # or rerank-3-lite
+rag-config set   reranker.apiKey   env:VOYAGE_RERANK_API_KEY
+rag-config reset reranker.baseURL                           # a stored Infinity baseURL would win over the Voyage default
+rag-config reset reranker.url
 ```
 
-**Option B — point the reranker at a separate server** (e.g. a local Infinity or TEI instance while using Voyage/OpenAI for embeddings):
+Without `reranker.apiKey` a Voyage reranker stays disabled and a warning is written to the log — visible only at `logging.level` `warn` or lower (the default `error` hides it).
+
+**A separate `/rerank` server** (e.g. a local Infinity or vLLM instance while using Voyage/OpenAI for embeddings):
 
 ```bash
-rag-config set reranker.baseURL http://localhost:7997
-rag-config set reranker.model   mixedbread-ai/mxbai-rerank-xsmall-v1
-# reranker.enabled is already true
+rag-config set   reranker.enabled  true
+rag-config set   reranker.provider openai-compatible       # a stored voyage provider would send the Voyage wire format
+rag-config set   reranker.baseURL  http://localhost:7997
+rag-config set   reranker.model    mixedbread-ai/mxbai-rerank-xsmall-v1
+rag-config reset reranker.url
+rag-config reset reranker.apiKey                          # unless the server needs one
 ```
 
-The embedding and reranking servers are fully independent — any combination works: Voyage embeddings + local Infinity reranker, OpenAI embeddings + remote TEI reranker, etc.
+The embedding and reranking servers are fully independent — any combination works: Voyage embeddings + local Infinity reranker, OpenAI embeddings + remote vLLM reranker, etc.
 
 ### Secrets & external embedders
 
@@ -323,13 +341,15 @@ When reranking is enabled, `search_codebase` fetches a **candidate pool** by cos
 - **Threshold ↔ window** — raising `search.similarityThreshold` lets *more* chunks under the bar, so the pool must be wide enough to reach them; a strict threshold with a small pool silently truncates recall. Widen `candidateFloor`/`overfetch` when you raise the threshold.
 - **Latency** — the reranker cost grows with pool size (on CPU, tens of seconds; a rough guide is a large fixed cost plus a per-candidate increment). Bigger pool = better recall, slower query. GPU/fast hosts can raise `candidateFloor`; ultra-constrained hosts lower it.
 
+The pool is reachable on large projects too. The HNSW index scan stops after `hnsw.ef_search` rows (PostgreSQL default 40) and only then applies the `project` filter and the threshold, so a scoped search on a project of a few thousand chunks or more (measured from ~5.8 k; the crossover depends on table statistics) — or an unscoped one — used to get at most 40 candidates, often far fewer. Each search now sets `hnsw.ef_search` to the requested row count (at least 40, at most 1000) and, on pgvector ≥ 0.8, `hnsw.iterative_scan = relaxed_order` for its own query, so `candidateFloor` / `overfetch` size the pool the reranker actually receives, unless the iterative scan stops at `hnsw.max_scan_tuples` (20 000) first. On an older pgvector only `ef_search` is raised, and one `warn` line per process names the version — or `unknown` if the first version read failed, with no later line naming it. Like the Voyage warning above, it is visible only at `logging.level` `warn` or lower.
+
 To size these for **your** hardware, turn on telemetry:
 
 ```bash
-rag-config set logging.searchMetrics true
+rag-config set logging.level info
 ```
 
-Each search then writes one `info` line (independent of `logging.level`) with, among others: `fetchLimit`, `candidatesReturned`, `poolSaturated`, `maxCosineRankSurvived`, `distinctFilesInResult`, `topFileShare`, and `embedMs`/`rerankMs`/`dbMs`/`totalMs`. Read them as:
+Each search then writes one `search metrics` line with, among others: `fetchLimit`, `candidatesReturned`, `poolSaturated`, `maxCosineRankSurvived`, `distinctFilesInResult`, `topFileShare`, `efSearch`/`iterativeScan` (the HNSW settings sent with the query), and `embedMs`/`rerankMs`/`dbMs`/`totalMs`. Read them as:
 
 - **`poolSaturated: true`** — more chunks were under the threshold than the window took → recall is being truncated; raise `candidateFloor`/`overfetch`.
 - **`maxCosineRankSurvived`** — the deepest cosine rank that survived reranking. If it stays well below `fetchLimit`, the pool is bigger than needed → shrink it to save latency; if it approaches `fetchLimit`, widen.

package/dist/app.js

@@ -46,20 +46,13 @@ const McpCommand_1 = require("./src/commands/McpCommand");
 const ConfigCommand_1 = require("./src/commands/ConfigCommand");
 const WebCommand_1 = require("./src/commands/WebCommand");
 const setup_1 = require("./src/setup");
-/**
- * Load `~/.config/codesearch-mcp/.env` into `process.env` before config is read.
- * All five binaries share this file — `rag-mcp` in particular is spawned by the
- * MCP client and does not inherit shell exports. A real `process.env` value
- * always wins over the file. Minimal `KEY=VALUE` parser (no `dotenv` dep):
- * skips blanks and `#` comments, strips one layer of surrounding quotes.
- */
 function loadDotEnv(file) {
     let text;
     try {
         text = fs.readFileSync(file, 'utf-8');
     }
     catch {
-        return; // absent / unreadable → no-op
+        return;
     }
     for (const rawLine of text.split(/\r?\n/)) {
         const line = rawLine.trim();
@@ -76,7 +69,7 @@ function loadDotEnv(file) {
             val = val.slice(1, -1);
         }
         if (!(key in process.env))
-            process.env[key] = val; // real env wins
+            process.env[key] = val;
     }
 }
 async function main() {
@@ -88,8 +81,6 @@ async function main() {
     const isConfig = argv1 === 'rag-config' || process.argv[2] === 'config' || process.argv[2] === 'rag-config';
     const isWeb = argv1 === 'rag-web' || process.argv[2] === 'web' || process.argv[2] === 'rag-web';
     const isBinary = ['rag-index', 'rag-mcp', 'rag-setup', 'rag-config', 'rag-web'].includes(argv1);
-    // `rag-index eval` — a sub-mode of index, not its own binary, so it sits one
-    // token deeper than the other modes (`rag-index eval` / `index eval`).
     const evalSubArg = isBinary ? process.argv[2] : process.argv[3];
     const isEval = isIndex && evalSubArg === 'eval';
     const mode = isEval ? 'eval' : isIndex ? 'index' : isMcp ? 'mcp' : isSetup ? 'setup' : isConfig ? 'config' : isWeb ? 'web' : (process.argv[2] ?? '');
@@ -99,7 +90,6 @@ async function main() {
         process.exit(0);
     }
     const effectiveArgs = process.argv.slice(isEval ? (isBinary ? 3 : 4) : (isBinary ? 2 : 3));
-    // Arg validation
     if (mode === 'eval') {
         const valueFlags = new Set(['--suite', '--baseline', '--project']);
         const unknown = [];
@@ -108,7 +98,7 @@ async function main() {
             if (valueFlags.has(a)) {
                 const next = effectiveArgs[i + 1];
                 if (next !== undefined && !next.startsWith('-'))
-                    i++; // consume the value
+                    i++;
                 continue;
             }
             unknown.push(a);
@@ -129,7 +119,7 @@ async function main() {
             if (a === '--project') {
                 const next = effectiveArgs[i + 1];
                 if (next !== undefined && !next.startsWith('-'))
-                    i++; // consume the name
+                    i++;
                 continue;
             }
             unknown.push(a);
@@ -158,7 +148,7 @@ async function main() {
                     process.exit(1);
                 }
                 i++;
-                continue; // skip its (validated) value
+                continue;
             }
             console.error(`Unexpected argument: ${effectiveArgs[i]}\n`);
             (0, help_1.printHelp)(isBinary ? argv1 : 'rag-web');
@@ -173,7 +163,6 @@ async function main() {
         await new ConfigCommand_1.ConfigCommand(effectiveArgs).execute();
         process.exit(0);
     }
-    // All other modes require a valid config.json
     const loggerName = mode === 'index' || mode === 'eval' ? 'indexer' : mode === 'mcp' ? 'mcp' : mode === 'web' ? 'web' : undefined;
     let container;
     try {
@@ -193,9 +182,6 @@ async function main() {
         console.error(`workspaceRoot is not configured. ${hint}`);
         process.exit(1);
     }
-    // Construct only the selected command — building every command eagerly would
-    // run each constructor's arg parsing (e.g. WebCommand's --port) for modes that
-    // never use it.
     const factories = {
         index: () => new IndexCommand_1.IndexCommand(container, argv1),
         eval: () => new EvalCommand_1.EvalCommand(container, argv1),

package/dist/src/chunking/AstChunker.d.ts

@@ -0,0 +1,12 @@
+import type { Config } from '../config';
+import type { IChunker, ChunkPiece, ChunkOpts } from '../interfaces/IChunker';
+import type { IChunkPostProcessor, IChunkStrategy } from './interfaces';
+export declare class AstChunker implements IChunker {
+    private readonly declarationStrategy;
+    private readonly characterStrategy;
+    private readonly postProcessors;
+    private readonly declarationPostProcessors;
+    private readonly chunkSize;
+    constructor(declarationStrategy: IChunkStrategy, characterStrategy: IChunkStrategy, postProcessors: IChunkPostProcessor[], declarationPostProcessors: IChunkPostProcessor[], config: Config);
+    chunk(code: string, opts: ChunkOpts): Promise<ChunkPiece[]>;
+}

package/dist/src/chunking/AstChunker.js

@@ -0,0 +1,30 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.AstChunker = void 0;
+const DeclarationTree_1 = require("../declarations/DeclarationTree");
+const SourceLines_1 = require("../declarations/SourceLines");
+class AstChunker {
+    declarationStrategy;
+    characterStrategy;
+    postProcessors;
+    declarationPostProcessors;
+    chunkSize;
+    constructor(declarationStrategy, characterStrategy, postProcessors, declarationPostProcessors, config) {
+        this.declarationStrategy = declarationStrategy;
+        this.characterStrategy = characterStrategy;
+        this.postProcessors = postProcessors;
+        this.declarationPostProcessors = declarationPostProcessors;
+        this.chunkSize = config.indexer.chunkSize;
+    }
+    async chunk(code, opts) {
+        const declarations = opts.declarations ?? DeclarationTree_1.DeclarationTree.EMPTY;
+        const ctx = { lines: new SourceLines_1.SourceLines(code, { jsx: opts.jsx }), declarations, chunkSize: this.chunkSize, removed: [], splitPoints: [], characterSplit: [] };
+        const byDeclarations = !opts.isSchemaFile && opts.strategy !== 'character' && declarations.roots.length > 0;
+        let pieces = await (byDeclarations ? this.declarationStrategy : this.characterStrategy).chunk(ctx, opts);
+        const processors = byDeclarations ? [...this.postProcessors, ...this.declarationPostProcessors] : this.postProcessors;
+        for (const processor of processors)
+            pieces = processor.process(pieces, ctx);
+        return pieces;
+    }
+}
+exports.AstChunker = AstChunker;

package/dist/src/chunking/CharacterChunkStrategy.d.ts

@@ -0,0 +1,8 @@
+import type { ChunkOpts, ChunkPiece } from '../interfaces/IChunker';
+import type { ChunkContext, IChunkStrategy, ITextSplitter } from './interfaces';
+export declare class CharacterChunkStrategy implements IChunkStrategy {
+    private readonly jsSplitter;
+    private readonly schemaSplitter;
+    constructor(jsSplitter: ITextSplitter, schemaSplitter: ITextSplitter);
+    chunk(ctx: ChunkContext, opts: ChunkOpts): Promise<ChunkPiece[]>;
+}

package/dist/src/chunking/CharacterChunkStrategy.js

@@ -0,0 +1,15 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.CharacterChunkStrategy = void 0;
+class CharacterChunkStrategy {
+    jsSplitter;
+    schemaSplitter;
+    constructor(jsSplitter, schemaSplitter) {
+        this.jsSplitter = jsSplitter;
+        this.schemaSplitter = schemaSplitter;
+    }
+    chunk(ctx, opts) {
+        return (opts.isSchemaFile ? this.schemaSplitter : this.jsSplitter).split(ctx.lines.text, 1);
+    }
+}
+exports.CharacterChunkStrategy = CharacterChunkStrategy;

package/dist/src/chunking/DeclarationChunkStrategy.d.ts

@@ -0,0 +1,15 @@
+import type { ChunkPiece } from '../interfaces/IChunker';
+import type { ChunkContext, IChunkStrategy, ISplitPolicy } from './interfaces';
+import { RunPacker } from './RunPacker';
+export declare class DeclarationChunkStrategy implements IChunkStrategy {
+    private readonly policy;
+    private readonly packer;
+    constructor(policy: ISplitPolicy, packer: RunPacker);
+    chunk(ctx: ChunkContext): Promise<ChunkPiece[]>;
+    private splitNode;
+    private coversParent;
+    private splitLevel;
+    private packRuns;
+    private static isShortPrologue;
+    private static gaps;
+}

package/dist/src/chunking/DeclarationChunkStrategy.js

@@ -0,0 +1,87 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.DeclarationChunkStrategy = void 0;
+const FragmentMerger_1 = require("./FragmentMerger");
+const RunPacker_1 = require("./RunPacker");
+class DeclarationChunkStrategy {
+    policy;
+    packer;
+    constructor(policy, packer) {
+        this.policy = policy;
+        this.packer = packer;
+    }
+    async chunk(ctx) {
+        const roots = ctx.declarations.roots;
+        const removed = DeclarationChunkStrategy.gaps({ start: 1, end: ctx.lines.count }, roots)
+            .flatMap(gap => ctx.lines.importRegions(gap.start, gap.end));
+        if (removed.length > 0 && DeclarationChunkStrategy.isShortPrologue(removed[0].start - 1, ctx.lines)) {
+            removed[0] = { start: 1, end: removed[0].end };
+        }
+        ctx.removed.push(...removed);
+        return this.splitLevel({ start: 1, end: ctx.lines.count }, roots, null, removed, ctx);
+    }
+    async splitNode(node, ctx) {
+        if (ctx.lines.length(node.start, node.end) <= ctx.chunkSize)
+            return RunPacker_1.RunPacker.piece(node, ctx.lines);
+        let children = node.children;
+        while (children.length === 1 && this.coversParent(children[0], node, ctx))
+            children = children[0].children;
+        return this.splitLevel(node, children, node, [], ctx);
+    }
+    coversParent(child, parent, ctx) {
+        if (child.end !== parent.end || child.start < parent.start)
+            return false;
+        for (let n = parent.start; n < child.start; n++) {
+            if (!ctx.lines.isBlank(n) && !ctx.lines.isCommentOnly(n))
+                return false;
+        }
+        return true;
+    }
+    async splitLevel(range, children, boundary, removed, ctx) {
+        const splitPoints = children.filter(c => this.policy.isSplitPoint(c, children, boundary, ctx.lines));
+        const packed = children.filter(c => !splitPoints.includes(c));
+        ctx.splitPoints.push(...splitPoints);
+        const pieces = [];
+        let cursor = range.start;
+        for (const sp of splitPoints) {
+            if (cursor < sp.start)
+                pieces.push(...await this.packRuns({ start: cursor, end: sp.start - 1 }, packed, boundary, removed, ctx));
+            pieces.push(...await this.splitNode(sp, ctx));
+            cursor = sp.end + 1;
+        }
+        if (cursor <= range.end)
+            pieces.push(...await this.packRuns({ start: cursor, end: range.end }, packed, boundary, removed, ctx));
+        return pieces;
+    }
+    async packRuns(span, children, owner, removed, ctx) {
+        const pieces = [];
+        for (const run of DeclarationChunkStrategy.gaps(span, removed)) {
+            pieces.push(...await this.packer.pack(run, children, ctx.lines, ctx.chunkSize, ctx.characterSplit, owner));
+        }
+        return pieces;
+    }
+    static isShortPrologue(end, lines) {
+        if (end < 1)
+            return false;
+        for (let n = lines.line(1).startsWith('#!') ? 2 : 1; n <= end; n++) {
+            if (!lines.isBlank(n) && !lines.isCommentOnly(n) && !lines.isDirective(n))
+                return false;
+        }
+        return lines.slice(1, end).replace(/\s+/g, '').length < FragmentMerger_1.MIN_FRAGMENT_CHARS;
+    }
+    static gaps(span, taken) {
+        const out = [];
+        let cursor = span.start;
+        for (const t of [...taken].sort((a, b) => a.start - b.start)) {
+            if (t.end < cursor || t.start > span.end)
+                continue;
+            if (cursor < t.start)
+                out.push({ start: cursor, end: t.start - 1 });
+            cursor = Math.max(cursor, t.end + 1);
+        }
+        if (cursor <= span.end)
+            out.push({ start: cursor, end: span.end });
+        return out;
+    }
+}
+exports.DeclarationChunkStrategy = DeclarationChunkStrategy;

package/dist/src/chunking/FragmentMerger.d.ts

@@ -0,0 +1,11 @@
+import type { ChunkPiece } from '../interfaces/IChunker';
+import type { ChunkContext, IChunkPostProcessor } from './interfaces';
+export declare const MIN_FRAGMENT_CHARS = 60;
+export declare class FragmentMerger implements IChunkPostProcessor {
+    process(pieces: ChunkPiece[], ctx: ChunkContext): ChunkPiece[];
+    private isFragment;
+    private canMerge;
+    private static scope;
+    private static within;
+    private static merge;
+}

package/dist/src/chunking/FragmentMerger.js

@@ -0,0 +1,61 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.FragmentMerger = exports.MIN_FRAGMENT_CHARS = void 0;
+const owner_path_1 = require("../owner-path");
+exports.MIN_FRAGMENT_CHARS = 60;
+class FragmentMerger {
+    process(pieces, ctx) {
+        const queue = [...pieces].sort((a, b) => a.startLine - b.startLine);
+        const out = [];
+        for (let i = 0; i < queue.length; i++) {
+            const piece = queue[i];
+            if (!this.isFragment(piece, ctx)) {
+                out.push(piece);
+                continue;
+            }
+            const scope = FragmentMerger.scope(piece, ctx);
+            const prev = out[out.length - 1];
+            if (prev && FragmentMerger.within(prev, scope) && this.canMerge(prev, piece, ctx)) {
+                out[out.length - 1] = FragmentMerger.merge(prev, piece, ctx);
+                continue;
+            }
+            const next = queue[i + 1];
+            if (next && FragmentMerger.within(next, scope) && this.canMerge(piece, next, ctx)) {
+                queue[i + 1] = FragmentMerger.merge(piece, next, ctx);
+                continue;
+            }
+            out.push(piece);
+        }
+        return out;
+    }
+    isFragment(piece, ctx) {
+        return ctx.lines.significantLength(piece.startLine, piece.endLine) < exports.MIN_FRAGMENT_CHARS
+            && (0, owner_path_1.ownedLines)(piece, ctx.declarations, ctx.lines).length === 0;
+    }
+    canMerge(first, second, ctx) {
+        const start = Math.min(first.startLine, second.startLine);
+        const end = Math.max(first.endLine, second.endLine);
+        if (ctx.removed.some(r => r.start > first.endLine && r.end < second.startLine))
+            return false;
+        return ctx.lines.length(start, end) <= ctx.chunkSize;
+    }
+    static scope(piece, ctx) {
+        let best = null;
+        for (const sp of [...ctx.splitPoints, ...ctx.characterSplit]) {
+            if (sp.start > piece.startLine || sp.end < piece.endLine)
+                continue;
+            if (!best || sp.end - sp.start < best.end - best.start)
+                best = sp;
+        }
+        return best;
+    }
+    static within(piece, scope) {
+        return !scope || (scope.start <= piece.startLine && piece.endLine <= scope.end);
+    }
+    static merge(first, second, ctx) {
+        const startLine = Math.min(first.startLine, second.startLine);
+        const endLine = Math.max(first.endLine, second.endLine);
+        return { content: ctx.lines.slice(startLine, endLine), startLine, endLine };
+    }
+}
+exports.FragmentMerger = FragmentMerger;

package/dist/src/chunking/ImportOnlyFilter.d.ts

@@ -0,0 +1,5 @@
+import type { ChunkPiece } from '../interfaces/IChunker';
+import type { ChunkContext, IChunkPostProcessor } from './interfaces';
+export declare class ImportOnlyFilter implements IChunkPostProcessor {
+    process(pieces: ChunkPiece[], ctx: ChunkContext): ChunkPiece[];
+}

package/dist/src/chunking/ImportOnlyFilter.js

@@ -0,0 +1,14 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.ImportOnlyFilter = void 0;
+class ImportOnlyFilter {
+    process(pieces, ctx) {
+        return pieces.filter((p) => {
+            if (!ctx.lines.isImportOnly(p.startLine, p.endLine))
+                return true;
+            ctx.removed.push({ start: p.startLine, end: p.endLine });
+            return false;
+        });
+    }
+}
+exports.ImportOnlyFilter = ImportOnlyFilter;