codesearch-mcp 0.1.0-alpha → 0.2.1-alpha

package/CHANGELOG.md

@@ -5,6 +5,45 @@ Format based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## [Unreleased]
 
+## [0.2.1-alpha] - 2026-10-02
+
+### Added
+
+- **`find_symbol` indexes every member kind, not just top-level declarations** — class methods (`abstract`, `static`, `async`, generators, `#private`, computed and string-literal names), get/set accessors, constructors, class properties initialised with a function, object-literal methods, and interface/type-literal method signatures all get a `code_symbol` row now, with the dotted `container` of their enclosing scopes (`RepairsService`, `NS.Parser`, `api.handlers`) and a `static` flag. Classes are collected from the whole file — inside namespaces, as class expressions, as `export default class {}`, and inside functions or methods — not only at the top level.
+- **Qualified, quoted and computed lookup** — `find_symbol` accepts `Class.method` to disambiguate same-named members (exact container first, then a `.`-boundary suffix match), a quoted name for string-literal members (`Router.'kebab-name'`), and a computed name (`Stack.[Symbol.iterator]`), alongside the existing bare-name lookup. `find_symbol` falls back to grep whenever the index returns **no hits** (not only when it errors), and marks grep-sourced hits so the caller knows they may be stale.
+- **Qualified call graph** — `code_call.fromSymbol`/`calleeSymbol` now carry the qualified name of the enclosing/resolved declaration (`RepairsService.createRepairOrder`, including constructors and accessors, which previously had no `fromSymbol` at all). `get_callers` accepts a qualified symbol with the same suffix-matching rules as `find_symbol`.
+- **Per-project symbol-index version, enforced full re-index** — `rag-index` without `--reindex` now refuses to run on a project indexed before member indexing (mixed old/new-format rows would silently hide methods from `find_symbol`); `find_symbol`/`get_callers` show a warning when a project in scope is outdated.
+
+### Changed (breaking)
+
+- **Requires PostgreSQL 15+** — the symbol index's new unique index relies on `NULLS NOT DISTINCT`, added in PostgreSQL 15. `rag-index` checks the server version before any schema change and refuses on an older server.
+- **Every project must be re-indexed** — run `rag-index --reindex` on each project after upgrading. Old-format rows (no members, bare call-graph names) are incompatible with the new lookup and are cleared by `--reindex`.
+
+## [0.2.0-alpha] - 2026-09-29
+
+### Added
+
+- **Index consistency — orphan cleanup and project pruning** — `rag-index` now removes orphaned rows left by deleted files, model changes, and deselected projects. Deleted files and `.ragignore`-excluded files are removed from all four index tables (`code_chunk`, `code_symbol`, `code_call`, `code_dependency`) in one transaction per project. `find_symbol`, `get_callers`, `get_dependencies`, and `get_dependents` no longer return results from files that no longer exist on disk. Projects outside the current selection are automatically pruned from the index; each gets a `Removed project <name> (not selected)` log line. A failure pruning one project does not block others or stop indexing, but exits non-zero.
+- **Candidate floor for reranking** — the reranker's candidate pool no longer scales only with `limit`. A new `reranker.candidateFloor` (default `80`) guarantees a minimum pool regardless of `limit`: `fetchLimit = min(max(limit × overfetch, candidateFloor), 200)` when reranking is enabled. Fixes recall starvation at small limits (the default `limit = 5` previously fed the reranker only 15 candidates, so its best matches never entered the window). `reranker.overfetch` default raised `3 → 5`. Applies immediately to search — no re-index.
+- **`logging.searchMetrics` search-tuning telemetry** — a new boolean (default `false`) that emits one structured line per `search_codebase` query at `info`, **independent of `logging.level`**, with the signals needed to size the pool for your hardware: `fetchLimit`, `candidatesReturned`, `poolSaturated`, `maxCosineRankSurvived`, `distinctFilesInResult`, `topFileShare`, `best/worstResultScore`, and `embed/rerank/db/totalMs`. Never logs chunk content, embeddings, or credentials.
+- **Infinity auto-recreates on a missing model, verifying embed + reranker together** — a managed Infinity container that doesn't yet serve the required model set is now automatically recreated (previously only the embedding model was checked, and only in the unreachable branch loaded the reranker). `rag-setup` and the web dashboard verify the full set (embedding + reranker, when on the same server); indexing verifies the embedding model only, so a bad reranker model never blocks or restarts indexing. Classification of managed-vs-external now also checks the compose file's actual published port (not just that a compose file exists), and a crashed or restart-looping container fails fast during the readiness wait instead of hanging out the full 30-minute timeout.
+- **Managed compose file is now the durable source of truth for Infinity's model set** — the model set is written into `INFINITY_MODEL_ID` as a literal (no more `${INFINITY_MODEL_ID:-…}` env interpolation), so a plain `docker compose up -d` after a host reboot or manual restart serves the configured models instead of falling back to a hard-coded default. The write is atomic and preserves the file's permissions.
+- **Web dashboard recreates Infinity in the background** — saving a config that changes the required Infinity model set (e.g. enabling reranking) now triggers the recreate automatically, without blocking the `PUT /api/config` response. New `GET /api/backend-status` endpoint reports progress (`idle`/`recreating`/`ready`/`failed`); the `PUT` response gains a `backend: { action, message }` field. Concurrent saves queue safely — one Docker run at a time, with at most one follow-up applying the latest save.
+- **`InfinityEmbedder` implements `listModels()`** — lists served models via `/models` (2s limit), so the dashboard's model suggestions keep working for the default provider now that `OllamaEmbedder` is gone.
+
+### Changed
+
+- **Reranking enabled by default** — `reranker.enabled` is now `true` by default. The shipped `docker-compose.yml` template includes both the embedding model (`jinaai/jina-embeddings-v2-base-code`) and the reranker model (`mixedbread-ai/mxbai-rerank-xsmall-v1`) in `INFINITY_MODEL_ID`, so a fresh `rag-setup` serves both without extra steps. Cloud embedding providers (OpenAI, Voyage) don't expose a `/rerank` endpoint — disable reranking with `rag-config set reranker.enabled false` or point `reranker.baseURL` at a separate server.
+- **Import/re-export-only chunks are no longer indexed** — the AST chunker now drops chunks whose only significant content is `import` / `export … from` / `export *` / `export {…}` statements (≈5–8% of chunks on a typical corpus). They are never a search answer and previously crowded out real results. The dependency graph is unaffected (edges come from the parser, not chunk text). Takes effect on the next per-file change or `rag-index --reindex`.
+
+### Changed (breaking)
+
+- **The `ollama` embedding provider is removed** — `OllamaEmbedder` and the `ollama` provider enum value are gone; the managed `ollama` Docker service is removed from the shipped `docker-compose.yml`. Ollama remains fully usable by pointing `embedding.provider: openai-compatible` at `http://localhost:11434/v1` (it already exposes an OpenAI-style `/v1/embeddings` endpoint) — there is no capability loss, only the dedicated code path and the auto-pull-on-index convenience.
+- **Default provider is now `infinity`, default model `jinaai/jina-embeddings-v2-base-code`** (was `ollama` / `bge-m3`); default `reranker.model` is now `mixedbread-ai/mxbai-rerank-xsmall-v1`.
+- **A stored config that still selects Ollama now fails to load, with guidance, instead of being silently reinterpreted.** `embedding.provider: ollama`, a legacy top-level `ollama` block, or a provider-less `embedding` block that still points at Ollama's defaults (port `11434`, or a model name Ollama used to own such as `bge-m3`) all raise a clear error naming the fix (`openai-compatible` at `http://localhost:11434/v1`, via `rag-config edit` or `rag-setup`) and the required `rag-index --reindex` afterward. `rag-config set`/`rag-config reset` cannot fix it (they load the same invalid config); `rag-config edit` can, since it opens the file even while invalid.
+- **`rag-index` refuses to silently rebuild an index that was built by `ollama`** — including an install predating the `index_meta` table (treated as a legacy `ollama`/`bge-m3` index). A plain `rag-index` against such an index now prints an explanation to stderr and exits non-zero before touching the backend or the database; `rag-index --reindex` is the explicit confirmation to rebuild it under the new provider. More generally, `rag-index` now exits non-zero on **any** backend-readiness or schema-reconciliation failure — previously it printed "Embedding setup failed" and exited 0, which a script could not detect. `rag-mcp`/`search_codebase` apply the same "pre-`index_meta` index is `ollama`/`bge-m3`" interpretation read-only, so a stale local install fails with the re-index instruction instead of an opaque pgvector dimension-mismatch error.
+- **`rag-setup` now saves the config before provisioning Infinity** — previously Infinity provisioning ran first, so a Docker failure there could abandon the whole wizard without ever writing `config.json`. The database-container step keeps its original position and abort-without-saving behavior.
+
 ## [0.1.0-alpha] - 2026-07-12
 
 ### Added

package/README.md

@@ -1,8 +1,17 @@
 # codesearch-mcp
 
-Semantic codebase search for AI agents. Indexes your local projects into a PostgreSQL vector database and exposes search via the Model Context Protocol (MCP).
+Semantic code search for AI agents — finds code by **meaning**, not keywords. Indexes your local projects into a PostgreSQL vector database and exposes search via the Model Context Protocol (MCP), so Claude can search your entire codebase automatically.
 
-Ask your AI agent things like *"where is the payment retry logic?"* or *"show me how auth tokens are validated"* — across all your projects at once.
+Ask your AI agent things like *"where is the payment retry logic?"* or *"show me how auth tokens are validated"* — it will find the right code even if those exact words never appear.
+
+**Why codesearch-mcp:**
+
+- **Dramatically fewer tokens** — without RAG, answering a cross-project question means discovery calls, directory listings, and reading several wrong files before finding the right one; a workspace of 10 projects can easily mean 50 000+ lines to sift through. With RAG, one `search_codebase` call returns 5 targeted chunks (~200–500 lines total) from across *all* projects at once — the model gets the right code immediately, with no exploration overhead
+- **Reranker narrows the result further** — a cross-encoder reranker (enabled by default) re-scores the candidate pool so the top results are the most relevant, not just the closest vector; fewer results means less context consumed per answer
+- **Semantic, not grep** — embeddings capture intent; queries in plain English or pseudocode find the relevant code regardless of naming
+- **Local and private** — default setup runs entirely on your machine (Infinity embeddings + PostgreSQL); no code leaves your network
+- **Code-structure aware** — chunks align to function and class boundaries; each result shows which declaration it belongs to (`InvoiceService.create`, `collectFiles`, …), so the model understands context without reading the surrounding file
+- **Import graph** — `include_graph_context` appends direct imports and dependents inline with each result, replacing separate `get_dependencies` / `get_dependents` calls
 
 > **Alpha** — works well locally, API may change before 1.0.
 
@@ -10,16 +19,20 @@ See `CHANGELOG.md` for version history.
 
 ## How it works
 
-1. `rag-index` scans your projects, chunks source files, embeds them with [bge-m3](https://huggingface.co/BAAI/bge-m3) via Ollama, and stores vectors in PostgreSQL
+![Architecture](docs/architecture.svg)
+
+1. `rag-index` scans your projects, chunks source files, embeds them using your configured embedding backend (default: [jina-embeddings-v2-base-code](https://huggingface.co/jinaai/jina-embeddings-v2-base-code) via a local [Infinity](https://github.com/michaelfeil/infinity) container), and stores vectors in PostgreSQL
 2. `rag-mcp` exposes an MCP server — Claude calls it automatically when you ask about code
 3. Incremental — only changed files are re-indexed on subsequent runs
+4. Index stays consistent — deleted files, `.ragignore`-excluded files, and deselected projects are removed from all four index tables on the next `rag-index` run
 
-**Stack:** TypeScript · Drizzle ORM · PostgreSQL + pgvector · Ollama (bge-m3) · LangChain text splitters · ts-morph
+**Stack:** TypeScript · Drizzle ORM · PostgreSQL + pgvector · Infinity (jina-embeddings-v2-base-code) · LangChain text splitters · ts-morph
 
 ## Requirements
 
 - Node.js 18+
-- Docker (for local DB + Ollama) — or bring your own PostgreSQL with pgvector + Ollama
+- PostgreSQL 15+ with pgvector (the symbol index's unique index relies on `NULLS NOT DISTINCT`, added in PostgreSQL 15)
+- Docker (for local DB + Infinity) — or bring your own PostgreSQL with pgvector + an embedding backend (see [Embedding providers](#embedding-providers))
 
 ## Installation
 
@@ -35,11 +48,19 @@ rag-setup
 
 Walks you through:
 - **Database** — enter connection details; optionally starts Docker containers automatically
-- **Embedding backend** — pick a provider (`ollama` / `infinity` / `openai-compatible` / `voyage`), its model, and an API key if the provider needs one (see [Embedding providers](#embedding-providers))
+- **Embedding backend** — pick a provider (`infinity` / `openai-compatible` / `voyage`), its model, and an API key if the provider needs one (see [Embedding providers](#embedding-providers))
 - **Logging** — choose log level and directory (defaults: `error` level, `~/.config/codesearch-mcp/logs/`)
 
 Config is saved to `~/.config/codesearch-mcp/config.json`.
 
+## Register with Claude Code
+
+```bash
+claude mcp add codesearch-mcp -s user -- rag-mcp
+```
+
+Then ask Claude anything about your code — it will use the tools below automatically.
+
 ## Index your projects
 
 ```bash
@@ -62,14 +83,6 @@ If `workspaceRoot` is an [Nx](https://nx.dev) workspace (detected by `nx.json`),
 
 No configuration needed — detection is automatic. Plain (non-Nx) workspaces keep the original behavior: each immediate subdirectory with a `package.json` is one project.
 
-## Register with Claude Code
-
-```bash
-claude mcp add codesearch-mcp -s user -- rag-mcp
-```
-
-Then ask Claude anything about your code — it will use the tools below automatically.
-
 ## Web dashboard
 
 ```bash
@@ -78,23 +91,23 @@ rag-web                 # http://127.0.0.1:4319  (--port to change)
 
 A local web UI (bound to `127.0.0.1` only) with a schema-driven config editor,
 an overview of indexed projects, and the interactive dependency graph. The
-config form shows the applied default for any unset key and offers dropdowns for
-model fields (the embedding model is restricted to dimension-compatible models).
+config form shows the applied default for any unset key and suggests dimension-compatible
+models in the model fields (free text is always accepted).
 See [docs/web-dashboard.md](docs/web-dashboard.md) for the full reference.
 
 ## MCP Tools
 
 | Tool | Description |
 |------|-------------|
-| `search_codebase` | Semantic search across all indexed projects. Accepts `include_graph_context` to append direct imports/dependents to each result, and `include_neighbors` for surrounding chunks |
+| `search_codebase` | Semantic search across all indexed projects. Accepts `include_graph_context` to append direct imports/dependents to each result, `include_neighbors` for surrounding chunks, and `threshold` to override the similarity threshold for that query |
 | `list_projects` | List projects with metadata and chunk counts |
-| `find_symbol` | Find a function, class, or type by exact name — faster than semantic search |
+| `find_symbol` | Find a declaration or member by exact name — bare, qualified (`Class.method`), quoted, or computed — faster than semantic search |
 | `list_files` | Browse the directory tree of a project |
 | `read_file` | Read a complete file by absolute path |
 | `read_file_range` | Read a specific line range from an indexed file |
 | `get_dependencies` | Show what files a given file imports (direct or transitive) |
 | `get_dependents` | Show which files import a given file (direct or transitive) |
-| `get_callers` | Find all call sites of a function or method by exact name |
+| `get_callers` | Find all call sites of a function or method by exact name (bare or qualified) |
 
 ## Project metadata
 
@@ -104,6 +117,7 @@ Add `.ragmeta.json` to a project root to enrich `list_projects` and generate a v
 {
   "description": "Main API server",
   "stack": "NestJS, GraphQL, PostgreSQL",
+  "notes": "Uses optimistic locking — see order-service/docs/concurrency.md",
   "relations": {
     "consumed_by": ["crm-frontend"],
     "consumes": ["billing-service", "auth-service"]
@@ -111,7 +125,7 @@ Add `.ragmeta.json` to a project root to enrich `list_projects` and generate a v
 }
 ```
 
-After each indexing run an interactive `project-graph.html` is written to the workspace root.
+After each indexing run an interactive `project-graph.html` is written to the workspace root. The same visualization is available in the web dashboard's Graph tab (`rag-web`).
 
 ## Graph context
 
@@ -166,6 +180,7 @@ rag-config show                      # print current config (password redacted)
 rag-config get <key>                 # print a single value
 rag-config set <key> <value>         # update a value
 rag-config reset <key>               # remove key (reverts to default)
+rag-config edit                      # open config in $EDITOR; validates on save (use this to recover an invalid config)
 ```
 
 | Key | Default | Description |
@@ -176,33 +191,40 @@ rag-config reset <key>               # remove key (reverts to default)
 | `db.password` | — | PostgreSQL password |
 | `db.name` | `rag_<os-username>` | Database name — auto-created on first run |
 | `workspaceRoot` | — | Root of your projects directory |
-| `embedding.provider` | `ollama` | Embedding backend: `ollama` \| `infinity` \| `openai-compatible` \| `voyage` |
-| `embedding.baseURL` | per-provider | Backend base URL (ollama `http://localhost:11434`, OpenAI `https://api.openai.com/v1`, Infinity/TEI/vLLM `http://localhost:7997`, Voyage `https://api.voyageai.com/v1`) |
+| `embedding.provider` | `infinity` | Embedding backend: `infinity` \| `openai-compatible` \| `voyage`. A stored `ollama` value fails validation with migration guidance (see [Embedding providers](#embedding-providers)) |
+| `embedding.baseURL` | per-provider | Backend base URL (Infinity/TEI/vLLM `http://localhost:7997`, OpenAI `https://api.openai.com/v1`, Voyage `https://api.voyageai.com/v1`; Ollama via `openai-compatible` at `http://localhost:11434/v1`) |
 | `embedding.apiKey` | `''` | API key, or `env:VAR_NAME` to read from `.env`/environment. Empty for local servers |
-| `embedding.model` | `bge-m3` | Embedding model |
+| `embedding.model` | `jinaai/jina-embeddings-v2-base-code` | Embedding model |
 | `embedding.dimensions` | (derived) | Vector width override; required for models whose native width exceeds 2000 (Matryoshka). Otherwise derived from model → registry → runtime probe |
 | `embedding.batchSize` | `32` | Texts per batch embedding request |
 | `search.similarityThreshold` | `0.6` | Max cosine distance — raise if search returns nothing |
-| `reranker.enabled` | `false` | Enable cross-encoder reranking; requires `reranker.model` |
-| `reranker.provider` | `openai-compatible` | Rerank backend — `openai-compatible` `/rerank` only (Infinity, TEI, vLLM). Ollama has no rerank API |
+| `reranker.enabled` | `true` | Enable cross-encoder reranking; requires `reranker.model` |
+| `reranker.provider` | `openai-compatible` | Rerank backend — `openai-compatible` `/rerank` only (Infinity, TEI, vLLM) |
 | `reranker.baseURL` | `embedding.baseURL` | Rerank server base URL (Infinity serves embed + rerank from one server) |
-| `reranker.model` | `''` | Reranker model name (e.g. `qllama/bge-reranker-v2-m3`) |
+| `reranker.model` | `mixedbread-ai/mxbai-rerank-xsmall-v1` | Reranker model name |
+| `reranker.overfetch` | `5` | Candidate multiplier before reranking (`limit × N`) |
+| `reranker.candidateFloor` | `80` | Minimum candidate pool fed to the reranker regardless of `limit` — `fetchLimit = min(max(limit × overfetch, candidateFloor), 200)`. Raise on GPU/fast hosts for better recall; lower on weak CPU to cut latency |
+| `indexing.chunkSize` | `1500` | Max characters per chunk |
+| `indexing.chunkOverlap` | `200` | Overlap between chunks |
+| `indexing.embedBatchSize` | `5` | Parallel embedding requests per batch |
 | `indexing.astBatchSize` | `3000` | Max files per ts-morph batch — lower if indexing large projects causes OOM |
 | `indexing.chunkStrategy` | `ast` | How TS/JS files are chunked: `ast` = boundaries aligned to functions/classes, `character` = fixed-size splitter. Re-index to apply |
+| `search.defaultLimit` | `5` | Default result count returned by `search_codebase` |
+| `search.maxLimit` | `20` | Hard cap on result count |
 | `logging.level` | `error` | Log level written to file: `error` \| `warn` \| `info` \| `debug` |
 | `logging.dir` | `~/.config/codesearch-mcp/logs` | Base log directory, or `"false"` to disable |
+| `logging.searchMetrics` | `false` | Emit one structured search-tuning line per query (at `info`, independent of `logging.level`) — use it to size `candidateFloor`/`overfetch`/`threshold` for your hardware |
 
 ## Embedding providers
 
-`embedding.provider` selects the backend. All four produce vectors the same way; they differ in where they run and which models they offer.
+`embedding.provider` selects the backend. All three produce vectors the same way; they differ in where they run and which models they offer. There is no dedicated Ollama provider — Ollama exposes an OpenAI-compatible `/v1/embeddings` endpoint, so it's reached through `openai-compatible` (see below).
 
 ### Choosing a provider
 
 | Provider | When to use | Needs |
 |----------|-------------|-------|
-| `ollama` | Default. Local, zero-setup, general-purpose (`bge-m3`); auto-pulls the model. | Ollama running (Docker or native) |
-| `infinity` | Local, best quality **plus a real reranker** in one container; GPU or CPU. | Docker (managed) or a running Infinity server |
-| `openai-compatible` | OpenAI cloud, or any OpenAI-style server (TEI, vLLM, LM Studio, LocalAI). | `baseURL` (+ `apiKey` for cloud) |
+| `infinity` | Default. Local, best quality **plus a real reranker** in one container; GPU or CPU. | Docker (managed) or a running Infinity server |
+| `openai-compatible` | OpenAI cloud, or any OpenAI-style server (TEI, vLLM, LM Studio, LocalAI, **or Ollama via its `/v1` endpoint**). | `baseURL` (+ `apiKey` for cloud) |
 | `voyage` | Voyage AI cloud, code-specialized embeddings. | Voyage `apiKey` |
 
 Easiest path: `rag-setup` asks for the provider, model, and any API key. Haven't run setup? It's the place to start. To change things later, edit config directly (below).
@@ -212,11 +234,19 @@ Easiest path: `rag-setup` asks for the provider, model, and any API key. Haven't
 Any change to the provider, model, or dimension requires a **re-index** — the mismatch guard blocks search until you do it. The index is a rebuildable cache, so there's no data migration.
 
 ```bash
-# Switch provider (example: OpenAI cloud)
+# Switch to OpenAI cloud
 rag-config set embedding.provider openai-compatible
 rag-config set embedding.baseURL  https://api.openai.com/v1
 rag-config set embedding.model    text-embedding-3-small
 rag-config set embedding.apiKey   env:OPENAI_API_KEY     # key lives in .env, not config.json
+rag-config set reranker.enabled   false                  # OpenAI has no /rerank endpoint
+rag-index --reindex
+
+# Switch to Voyage AI
+rag-config set embedding.provider voyage
+rag-config set embedding.model    voyage-code-3
+rag-config set embedding.apiKey   env:VOYAGE_API_KEY     # key lives in .env, not config.json
+rag-config set reranker.enabled   false                  # Voyage has no /rerank endpoint
 rag-index --reindex
 
 # Change only the model on the current provider
@@ -224,31 +254,106 @@ rag-config set embedding.model <model-name>
 rag-index --reindex
 ```
 
+### Breaking change: the `ollama` provider was removed
+
+> **Migrating from an older version?** A config with `embedding.provider: ollama` (or an old-style top-level `ollama` block, or `embedding.baseURL`/`embedding.model` still pointing at Ollama's defaults with no `provider` set) now **fails to load** with an error explaining the fix.
+>
+> Fix it with `rag-config edit` (opens the file even while it's invalid) or by re-running `rag-setup`: set `embedding.provider` to `openai-compatible` and `embedding.baseURL` to `http://localhost:11434/v1` to keep using Ollama, or switch to `infinity` for a managed local backend.
+>
+> Either way, run `rag-index --reindex` afterward. A plain `rag-index` additionally refuses (non-zero exit, no changes made) to silently rebuild an index built by the old `ollama` provider — including an older install with no `index_meta` row, which is assumed to be a pre-migration Ollama/`bge-m3` index.
+
 Per-provider notes:
 
-- **ollama** — `rag-index` auto-pulls the model if it isn't present locally (`bge-m3`, `nomic-embed-text`, `mxbai-embed-large`, …). No API key. Change model: `rag-config set embedding.model nomic-embed-text`.
-- **infinity** — the model set is bound at container startup, so changing `embedding.model` (or the reranker model) **recreates** the managed container on the next `rag-index`. Defaults: `jinaai/jina-embeddings-v2-base-code` + reranker `Alibaba-NLP/gte-reranker-modernbert-base`.
-- **openai-compatible** — point `embedding.baseURL` at the server; set `embedding.apiKey` (as `env:VAR`) for cloud, leave empty for local servers (TEI/vLLM/LM Studio). E.g. `text-embedding-3-small`, `text-embedding-3-large`.
-- **voyage** — set `embedding.apiKey`; e.g. `voyage-code-3`, `voyage-3`.
+- **infinity** — the model set is bound at container startup, so changing `embedding.model` (or the reranker model) **recreates** the managed container on the next `rag-setup`/`rag-index`/dashboard save. Defaults: `jinaai/jina-embeddings-v2-base-code` + reranker `mixedbread-ai/mxbai-rerank-xsmall-v1`. Both models are included in the compose template and reranking is **enabled by default** (`reranker.enabled: true`).
+- **openai-compatible** — point `embedding.baseURL` at the server; set `embedding.apiKey` (as `env:VAR`) for cloud, leave empty for local servers (TEI/vLLM/LM Studio/**Ollama**). E.g. `text-embedding-3-small`, `text-embedding-3-large`. Cloud endpoints (OpenAI, Azure) don't expose `/rerank` — disable the reranker or point it at a separate server (see [Reranker with a cloud embedder](#reranker-with-a-cloud-or-external-embedder)).
+- **Ollama** (via `openai-compatible`) — set `embedding.baseURL` to `http://localhost:11434/v1` and `embedding.model` to any model you've pulled (e.g. `bge-m3`). No dedicated code path, no auto-pull — pull the model yourself with `ollama pull <model>` first.
+- **voyage** — set `embedding.apiKey` (as `env:VAR`); e.g. `voyage-code-3`, `voyage-3`. Voyage doesn't expose a `/rerank` endpoint in the OpenAI-compatible format — disable the reranker or point it at a separate server (see [Reranker with a cloud embedder](#reranker-with-a-cloud-or-external-embedder)).
 
 The vector **dimension** is derived automatically (model registry → one-shot probe); set `embedding.dimensions` only to force a Matryoshka-reduced width for models whose native size exceeds 2000.
 
 ### Infinity — local embeddings + reranking in one container
 
-Select `infinity` in `rag-setup` (or set `embedding.provider = infinity`) to run [Infinity](https://github.com/michaelfeil/infinity), which serves both `/embeddings` and `/rerank` from a single container — so you get a real cross-encoder reranker locally, which Ollama does not offer. During setup you're asked for an embedding model (default `jinaai/jina-embeddings-v2-base-code`, 768 dims) and an optional reranker model (default `Alibaba-NLP/gte-reranker-modernbert-base`); when a reranker is given, `reranker.*` is auto-wired to the same server.
+`infinity` is the default provider (select it in `rag-setup`, or set `embedding.provider = infinity`) — it runs [Infinity](https://github.com/michaelfeil/infinity), which serves both `/embeddings` and `/rerank` from a single container, so you get a real cross-encoder reranker locally. During setup you're asked for an embedding model (default `jinaai/jina-embeddings-v2-base-code`, 768 dims) and a reranker model (default `mixedbread-ai/mxbai-rerank-xsmall-v1`, enabled by default); `reranker.*` is auto-wired to the same server.
+
+Managed vs. external is decided by a `/health` probe at `embedding.baseURL` (default `http://localhost:7997`) plus the managed compose file — no config flag:
+
+- **Managed** — the app-owned compose file (`~/.config/codesearch-mcp/docker-compose.yml`) declares the `infinity` service and `embedding.baseURL` points at its published port on `localhost`/`127.0.0.1`/`[::1]`. If nothing answers the probe, or the running container doesn't serve the configured model set, `rag-setup`, `rag-index`, and the web dashboard bring up (or recreate) that container. The full model set (embedding + reranker, when on the same server) is written into the compose file first — a plain `docker compose up -d` afterward serves the same set, no env var needed. It **prefers an NVIDIA GPU and falls back to CPU** automatically (GPU layered via `docker-compose.infinity-gpu.yml` only when `nvidia-smi` is present). To opt out of management entirely, remove the `infinity` service from that compose file.
+- **External** — any other `embedding.baseURL` (a remote host, a server on a different port, or a managed compose file that's absent/unreadable) is used as-is; no Docker command ever runs. If it's missing a required model, you're shown the `--model-id` flag to restart it with.
+
+The same "already running?" probe applies everywhere: a reachable backend defaults `rag-setup`'s "start Docker?" prompt to no, and a web-dashboard save that doesn't change the served model set never touches Docker.
 
-Managed vs. external is decided by a `/health` probe at `embedding.baseURL` (default `http://localhost:7997`) — no config flag:
+### Reranker with a cloud or external embedder
 
-- **Managed** — if nothing answers the probe, `rag-setup` (and `rag-index`) bring up the bundled `infinity` Docker service with your model set. It **prefers an NVIDIA GPU and falls back to CPU** automatically (GPU layered via `docker-compose.infinity-gpu.yml` only when `nvidia-smi` is present). Because Infinity binds its models at startup, changing `embedding.model`/reranker model **recreates** the container.
-- **External** — if a server already answers (your own Infinity, remote host, CI), it is used as-is and no Docker command runs. Point `embedding.baseURL` at it.
+The reranker is **enabled by default** (`reranker.enabled: true`) and `reranker.baseURL` defaults to `embedding.baseURL`. Cloud embedding endpoints (OpenAI, Voyage, Azure) don't serve `/rerank` — the reranker call will fail silently and fall back to vector order on every query. Fix it one of two ways:
 
-The same "already running?" probe now also applies to Ollama: a reachable backend defaults `rag-setup`'s "start Docker?" prompt to no.
+**Option A — disable reranking** (simplest, recommended for cloud-only setups):
+
+```bash
+rag-config set reranker.enabled false
+```
+
+**Option B — point the reranker at a separate server** (e.g. a local Infinity or TEI instance while using Voyage/OpenAI for embeddings):
+
+```bash
+rag-config set reranker.baseURL http://localhost:7997
+rag-config set reranker.model   mixedbread-ai/mxbai-rerank-xsmall-v1
+# reranker.enabled is already true
+```
+
+The embedding and reranking servers are fully independent — any combination works: Voyage embeddings + local Infinity reranker, OpenAI embeddings + remote TEI reranker, etc.
 
 ### Secrets & external embedders
 
 API keys live in `~/.config/codesearch-mcp/.env` (chmod 600), **not** in `config.json`. Reference them with `env:VAR_NAME`, e.g. `embedding.apiKey = env:OPENAI_API_KEY` with `OPENAI_API_KEY=sk-…` in `.env`. The file is loaded into the environment at startup (a real environment variable wins over the file), so all binaries — including the client-spawned `rag-mcp` — see it identically.
 
-Switching the embedding model, provider, or dimension is applied by **re-indexing** (`rag-index --reindex`) — the index is a rebuildable cache, so there is no data migration. A single-row `index_meta` table records which model built the index; `search_codebase` and `rag-mcp` refuse with a clear "re-index required" error when the configured model disagrees, rather than returning noise from a foreign vector space. Staying on Ollama + `bge-m3` keeps dimension 1024 and forces no re-index.
+Switching the embedding model, provider, or dimension is applied by **re-indexing** (`rag-index --reindex`) — the index is a rebuildable cache, so there is no data migration. A single-row `index_meta` table records which model built the index; `search_codebase` and `rag-mcp` refuse with a clear "re-index required" error when the configured model disagrees, rather than returning noise from a foreign vector space.
+
+## Tuning search recall
+
+When reranking is enabled, `search_codebase` fetches a **candidate pool** by cosine distance, reranks it with the cross-encoder, and returns the top `limit`. Three knobs and one relationship govern how good the results are and how long they take:
+
+- **Pool size** — `fetchLimit = min(max(limit × overfetch, candidateFloor), 200)`. `candidateFloor` (default `80`) guarantees the reranker a large enough menu even at the default `limit = 5`; without it a small `limit` starves the reranker (only `5 × 5 = 25` candidates) and its best matches never enter the window.
+- **Threshold ↔ window** — raising `search.similarityThreshold` lets *more* chunks under the bar, so the pool must be wide enough to reach them; a strict threshold with a small pool silently truncates recall. Widen `candidateFloor`/`overfetch` when you raise the threshold.
+- **Latency** — the reranker cost grows with pool size (on CPU, tens of seconds; a rough guide is a large fixed cost plus a per-candidate increment). Bigger pool = better recall, slower query. GPU/fast hosts can raise `candidateFloor`; ultra-constrained hosts lower it.
+
+To size these for **your** hardware, turn on telemetry:
+
+```bash
+rag-config set logging.searchMetrics true
+```
+
+Each search then writes one `info` line (independent of `logging.level`) with, among others: `fetchLimit`, `candidatesReturned`, `poolSaturated`, `maxCosineRankSurvived`, `distinctFilesInResult`, `topFileShare`, and `embedMs`/`rerankMs`/`dbMs`/`totalMs`. Read them as:
+
+- **`poolSaturated: true`** — more chunks were under the threshold than the window took → recall is being truncated; raise `candidateFloor`/`overfetch`.
+- **`maxCosineRankSurvived`** — the deepest cosine rank that survived reranking. If it stays well below `fetchLimit`, the pool is bigger than needed → shrink it to save latency; if it approaches `fetchLimit`, widen.
+
+## Index consistency
+
+`rag-index` keeps the four index tables (`code_chunk`, `code_symbol`, `code_call`, `code_dependency`) consistent after every run:
+
+| Event | What happens |
+|---|---|
+| **File deleted from disk** | All rows for that file are removed from the four tables in one transaction on the next `rag-index`. The file disappears from `find_symbol`, `get_callers`, `get_dependencies`, and `get_dependents`. |
+| **File matched by `.ragignore`** | Same as deleted — treated as not collected, removed from all four tables. |
+| **Project deselected** | All rows for that project are removed from the four tables. To bring it back, re-select and re-index — the index is a cache. The removal is printed as `Removed project <name> (not selected)`. |
+| **`workspaceRoot` changed** | All projects indexed under the old root (their paths no longer match anything under the new root) are pruned. Each gets its own `Removed project` line. No confirmation is required — the index is a cache that `rag-index` rebuilds. |
+
+A failure pruning one project does not block the others or stop indexing of the selected projects, but `rag-index` exits non-zero to signal the partial failure.
+
+## Architecture
+
+**Stack:** TypeScript on Node.js 18+ · PostgreSQL 17 + pgvector (HNSW, cosine distance) via Drizzle ORM · ts-morph for AST/symbol/call-graph extraction · `@langchain/textsplitters` for chunk fallback · MCP SDK over stdio · Zod v4 for config validation and JSON-Schema generation · pino + pino-roll for logging · plain `node:http` dashboard with vis-network · Vitest · Docker Compose for Postgres + Infinity.
+
+**Design:**
+- **Composition root + manual DI** — `src/container.ts` builds every service once and wires it through constructors; no module-level singletons, no DI framework.
+- **Dependency inversion** — services depend on interfaces (`IEmbedder`, `IDatabase`, `IAstParser`, `IChunker`, `ISearcher`, `IProjectDiscovery`, `IMonorepoResolver`, `IMcpToolHandler`), so providers (OpenAI-compatible/Voyage/Infinity — Ollama via `openai-compatible`), chunking strategy (`ast`/`character`), and monorepo resolvers are swappable.
+- **Command pattern** — each CLI mode (`rag-index`, `rag-mcp`, `rag-config`, `rag-web`) is its own `ICommand`; `app.ts` only dispatches.
+- **Registry / Open-Closed** — MCP tools are `IMcpToolHandler` implementations registered in a `Map`; adding a tool never touches `McpServer`.
+- **Fallback chains** — vector dimension: config override → model registry → runtime probe; `find_symbol`/`get_dependents`: indexed table → grep; failed rerank → vector order.
+- **Incremental indexing** — mtime fast path, SHA-256 fallback for clones/transfers, AST-aligned chunk boundaries so declarations aren't split arbitrarily.
+- **Index/model consistency guard** — `index_meta` tracks the embedding model that built the index; search refuses with a clear re-index message on mismatch instead of returning noise.
+
+Full internals (per-file responsibilities, every pattern with file:line references) are documented in `CLAUDE.md`.
 
 ## License
 

package/dist/app.js

@@ -42,7 +42,6 @@ const config_1 = require("./src/config");
 const config_store_1 = require("./src/config-store");
 const IndexCommand_1 = require("./src/commands/IndexCommand");
 const McpCommand_1 = require("./src/commands/McpCommand");
-const BackfillCommand_1 = require("./src/commands/BackfillCommand");
 const ConfigCommand_1 = require("./src/commands/ConfigCommand");
 const WebCommand_1 = require("./src/commands/WebCommand");
 const setup_1 = require("./src/setup");
@@ -87,9 +86,7 @@ async function main() {
     const isSetup = argv1 === 'rag-setup' || process.argv[2] === 'setup' || process.argv[2] === 'rag-setup';
     const isConfig = argv1 === 'rag-config' || process.argv[2] === 'config' || process.argv[2] === 'rag-config';
     const isWeb = argv1 === 'rag-web' || process.argv[2] === 'web' || process.argv[2] === 'rag-web';
-    const isBackfill = process.argv[2] === 'backfill' ||
-        (process.argv[2] === 'rag-index' && process.argv[3] === 'backfill');
-    const mode = isBackfill ? 'backfill' : isIndex ? 'index' : isMcp ? 'mcp' : isSetup ? 'setup' : isConfig ? 'config' : isWeb ? 'web' : (process.argv[2] ?? '');
+    const mode = isIndex ? 'index' : isMcp ? 'mcp' : isSetup ? 'setup' : isConfig ? 'config' : isWeb ? 'web' : (process.argv[2] ?? '');
     const isBinary = ['rag-index', 'rag-mcp', 'rag-setup', 'rag-config', 'rag-web'].includes(argv1);
     if (process.argv.includes('--help') || process.argv.includes('-h')) {
         (0, help_1.printHelp)(isBinary ? argv1 : mode);
@@ -131,27 +128,16 @@ async function main() {
             process.exit(1);
         }
     }
-    else if (mode === 'backfill') {
-        const knownBackfillFlags = new Set(['--calls']);
-        const backfillIdx = process.argv.indexOf('backfill');
-        const backfillArgs = backfillIdx >= 0 ? process.argv.slice(backfillIdx + 1) : [];
-        const unknown = backfillArgs.filter(a => !knownBackfillFlags.has(a));
-        if (unknown.length > 0) {
-            console.error(`Unexpected argument: ${unknown[0]}\n`);
-            (0, help_1.printHelp)('rag-index');
-            process.exit(1);
-        }
-    }
     if (mode === 'setup') {
         await (0, setup_1.runSetup)();
-        process.exit(0);
+        process.exit(process.exitCode ?? 0);
     }
     if (mode === 'config') {
         await new ConfigCommand_1.ConfigCommand(effectiveArgs).execute();
         process.exit(0);
     }
     // All other modes require a valid config.json
-    const loggerName = mode === 'index' || mode === 'backfill' ? 'indexer' : mode === 'mcp' ? 'mcp' : mode === 'web' ? 'web' : undefined;
+    const loggerName = mode === 'index' ? 'indexer' : mode === 'mcp' ? 'mcp' : mode === 'web' ? 'web' : undefined;
     let container;
     try {
         container = (0, container_1.createContainer)(loggerName);
@@ -176,7 +162,6 @@ async function main() {
     const factories = {
         index: () => new IndexCommand_1.IndexCommand(container, argv1),
         mcp: () => new McpCommand_1.McpCommand(container),
-        backfill: () => new BackfillCommand_1.BackfillCommand(container),
         web: () => new WebCommand_1.WebCommand(container, effectiveArgs),
     };
     const factory = factories[mode];
@@ -189,6 +174,6 @@ async function main() {
     }
     await factory().execute();
     await container.db.destroy();
-    process.exit(0);
+    process.exit(process.exitCode ?? 0);
 }
 main().catch(err => { console.error(err); process.exit(1); });

package/dist/src/commands/IndexCommand.js

@@ -6,6 +6,7 @@ const project_selector_1 = require("../project-selector");
 const graph_1 = require("../graph");
 const MonorepoAwareDiscoveryService_1 = require("../services/MonorepoAwareDiscoveryService");
 const file_utils_1 = require("../file-utils");
+const legacy_ollama_index_1 = require("../services/legacy-ollama-index");
 class IndexCommand {
     container;
     argv1;
@@ -19,7 +20,57 @@ class IndexCommand {
         const force = args.includes('--reindex');
         const forceSelect = args.includes('--select');
         await db.initDbWithSchema();
-        // Ensure the local model is pulled (Ollama only — no-op for cloud), then
+        // A former Ollama installation the config layer cannot recognise (old
+        // defaults, no `embedding`/`ollama` block) is protected here: without an
+        // explicit `--reindex`, refuse to touch an index built by `ollama` (or a
+        // pre-`index_meta` install, which is indistinguishable from one) before
+        // any backend readiness check, Docker command, or schema work.
+        if (!force) {
+            let identity;
+            try {
+                identity = await (0, legacy_ollama_index_1.resolveIndexedIdentity)(db);
+            }
+            catch (err) {
+                console.error(`\n${err.message}`);
+                process.exitCode = 1;
+                return;
+            }
+            if (identity?.provider === 'ollama') {
+                console.error(`\nThe existing index was built with ${identity.provider}/${identity.model}, but "ollama" is no ` +
+                    `longer a supported embedding provider (configured: ${config.embedding.provider}/${config.embedding.model}). ` +
+                    `Run \`rag-index --reindex\` to confirm rebuilding the index under the configured provider.`);
+                process.exitCode = 1;
+                return;
+            }
+        }
+        const discovered = discovery.discover(config.workspaceRoot);
+        const selected = await new project_selector_1.ProjectSelector(config.workspaceRoot).select(discovered, forceSelect);
+        if (selected.length === 0)
+            return;
+        // Expand any selected nested-monorepo unit (e.g. an Nx repo) into its
+        // sub-projects; ordinary projects pass through unchanged.
+        const projects = (0, MonorepoAwareDiscoveryService_1.toLeafProjects)(selected);
+        // A project indexed before member extraction mixes old-format rows (no
+        // members, bare calleeSymbol) with new ones if left to an incremental run.
+        // Refuse before any backend readiness check, Docker command, or schema
+        // work — the same pattern as the legacy-Ollama guard above.
+        if (!force) {
+            let outdated;
+            try {
+                outdated = await db.listOutdatedProjects(projects.map(p => p.name));
+            }
+            catch (err) {
+                console.error(`\n${err.message}`);
+                process.exitCode = 1;
+                return;
+            }
+            if (outdated.length > 0) {
+                console.error(`\nIndexed before member indexing: ${[...outdated].sort().join(', ')} — run \`rag-index --reindex\`.`);
+                process.exitCode = 1;
+                return;
+            }
+        }
+        // Ensure the local model is served (Infinity only — no-op for cloud), then
         // resolve the active embedding dimension and reconcile it with index_meta
         // (recreate code_chunk on a dimension change, clear rows on a model change)
         // before scanning. A clear error here beats a mid-run vector width mismatch.
@@ -30,18 +81,28 @@ class IndexCommand {
         }
         catch (err) {
             console.error(`\nEmbedding setup failed: ${err.message}`);
+            process.exitCode = 1;
             return;
         }
         const controller = new AbortController();
         const teardown = (0, interrupt_1.setupInterrupt)(controller);
         try {
-            const discovered = discovery.discover(config.workspaceRoot);
-            const selected = await new project_selector_1.ProjectSelector(config.workspaceRoot).select(discovered, forceSelect);
-            if (selected.length === 0)
+            let pruneResult;
+            try {
+                pruneResult = await indexer.pruneProjects(projects.map(p => p.name));
+            }
+            catch (err) {
+                console.error(`\nPruning failed: ${err.message}`);
+                process.exitCode = 1;
                 return;
-            // Expand any selected nested-monorepo unit (e.g. an Nx repo) into its
-            // sub-projects; ordinary projects pass through unchanged.
-            const projects = (0, MonorepoAwareDiscoveryService_1.toLeafProjects)(selected);
+            }
+            const { removed, failed } = pruneResult;
+            for (const name of removed) {
+                console.log(`Removed project ${name} (not selected)`);
+            }
+            if (failed.length > 0) {
+                process.exitCode = 1;
+            }
             const filesPerProject = projects.map(p => (0, file_utils_1.collectFiles)(p.path, (0, file_utils_1.loadIgnore)(p.path), p.path));
             const totalFiles = filesPerProject.reduce((sum, f) => sum + f.length, 0);
             console.log(`\n${projects.length} projects — ${totalFiles} files total`);

package/dist/src/config-error.d.ts

@@ -0,0 +1,7 @@
+/**
+ * Own module (not `config.ts`) so `config-store.ts` can throw it too without
+ * an import cycle — `config.ts` imports from `config-store.ts`.
+ */
+export declare class ConfigError extends Error {
+    constructor(message: string);
+}

package/dist/src/config-error.js

@@ -0,0 +1,14 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.ConfigError = void 0;
+/**
+ * Own module (not `config.ts`) so `config-store.ts` can throw it too without
+ * an import cycle — `config.ts` imports from `config-store.ts`.
+ */
+class ConfigError extends Error {
+    constructor(message) {
+        super(message);
+        this.name = 'ConfigError';
+    }
+}
+exports.ConfigError = ConfigError;