codesearch-mcp 0.0.18-alpha → 0.2.0-alpha

package/CHANGELOG.md

@@ -3,7 +3,53 @@
 All notable changes to this project will be documented in this file.
 Format based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
-## [Unreleased]
+## [0.2.0-alpha] - 2026-09-29
+
+### Added
+
+- **Index consistency — orphan cleanup and project pruning** — `rag-index` now removes orphaned rows left by deleted files, model changes, and deselected projects. Deleted files and `.ragignore`-excluded files are removed from all four index tables (`code_chunk`, `code_symbol`, `code_call`, `code_dependency`) in one transaction per project. `find_symbol`, `get_callers`, `get_dependencies`, and `get_dependents` no longer return results from files that no longer exist on disk. Projects outside the current selection are automatically pruned from the index; each gets a `Removed project <name> (not selected)` log line. A failure pruning one project does not block others or stop indexing, but exits non-zero.
+- **Candidate floor for reranking** — the reranker's candidate pool no longer scales only with `limit`. A new `reranker.candidateFloor` (default `80`) guarantees a minimum pool regardless of `limit`: `fetchLimit = min(max(limit × overfetch, candidateFloor), 200)` when reranking is enabled. Fixes recall starvation at small limits (the default `limit = 5` previously fed the reranker only 15 candidates, so its best matches never entered the window). `reranker.overfetch` default raised `3 → 5`. Applies immediately to search — no re-index.
+- **`logging.searchMetrics` search-tuning telemetry** — a new boolean (default `false`) that emits one structured line per `search_codebase` query at `info`, **independent of `logging.level`**, with the signals needed to size the pool for your hardware: `fetchLimit`, `candidatesReturned`, `poolSaturated`, `maxCosineRankSurvived`, `distinctFilesInResult`, `topFileShare`, `best/worstResultScore`, and `embed/rerank/db/totalMs`. Never logs chunk content, embeddings, or credentials.
+- **Infinity auto-recreates on a missing model, verifying embed + reranker together** — a managed Infinity container that doesn't yet serve the required model set is now automatically recreated (previously only the embedding model was checked, and only in the unreachable branch loaded the reranker). `rag-setup` and the web dashboard verify the full set (embedding + reranker, when on the same server); indexing verifies the embedding model only, so a bad reranker model never blocks or restarts indexing. Classification of managed-vs-external now also checks the compose file's actual published port (not just that a compose file exists), and a crashed or restart-looping container fails fast during the readiness wait instead of hanging out the full 30-minute timeout.
+- **Managed compose file is now the durable source of truth for Infinity's model set** — the model set is written into `INFINITY_MODEL_ID` as a literal (no more `${INFINITY_MODEL_ID:-…}` env interpolation), so a plain `docker compose up -d` after a host reboot or manual restart serves the configured models instead of falling back to a hard-coded default. The write is atomic and preserves the file's permissions.
+- **Web dashboard recreates Infinity in the background** — saving a config that changes the required Infinity model set (e.g. enabling reranking) now triggers the recreate automatically, without blocking the `PUT /api/config` response. New `GET /api/backend-status` endpoint reports progress (`idle`/`recreating`/`ready`/`failed`); the `PUT` response gains a `backend: { action, message }` field. Concurrent saves queue safely — one Docker run at a time, with at most one follow-up applying the latest save.
+- **`InfinityEmbedder` implements `listModels()`** — lists served models via `/models` (2s limit), so the dashboard's model suggestions keep working for the default provider now that `OllamaEmbedder` is gone.
+
+### Changed
+
+- **Reranking enabled by default** — `reranker.enabled` is now `true` by default. The shipped `docker-compose.yml` template includes both the embedding model (`jinaai/jina-embeddings-v2-base-code`) and the reranker model (`mixedbread-ai/mxbai-rerank-xsmall-v1`) in `INFINITY_MODEL_ID`, so a fresh `rag-setup` serves both without extra steps. Cloud embedding providers (OpenAI, Voyage) don't expose a `/rerank` endpoint — disable reranking with `rag-config set reranker.enabled false` or point `reranker.baseURL` at a separate server.
+- **Import/re-export-only chunks are no longer indexed** — the AST chunker now drops chunks whose only significant content is `import` / `export … from` / `export *` / `export {…}` statements (≈5–8% of chunks on a typical corpus). They are never a search answer and previously crowded out real results. The dependency graph is unaffected (edges come from the parser, not chunk text). Takes effect on the next per-file change or `rag-index --reindex`.
+
+### Changed (breaking)
+
+- **The `ollama` embedding provider is removed** — `OllamaEmbedder` and the `ollama` provider enum value are gone; the managed `ollama` Docker service is removed from the shipped `docker-compose.yml`. Ollama remains fully usable by pointing `embedding.provider: openai-compatible` at `http://localhost:11434/v1` (it already exposes an OpenAI-style `/v1/embeddings` endpoint) — there is no capability loss, only the dedicated code path and the auto-pull-on-index convenience.
+- **Default provider is now `infinity`, default model `jinaai/jina-embeddings-v2-base-code`** (was `ollama` / `bge-m3`); default `reranker.model` is now `mixedbread-ai/mxbai-rerank-xsmall-v1`.
+- **A stored config that still selects Ollama now fails to load, with guidance, instead of being silently reinterpreted.** `embedding.provider: ollama`, a legacy top-level `ollama` block, or a provider-less `embedding` block that still points at Ollama's defaults (port `11434`, or a model name Ollama used to own such as `bge-m3`) all raise a clear error naming the fix (`openai-compatible` at `http://localhost:11434/v1`, via `rag-config edit` or `rag-setup`) and the required `rag-index --reindex` afterward. `rag-config set`/`rag-config reset` cannot fix it (they load the same invalid config); `rag-config edit` can, since it opens the file even while invalid.
+- **`rag-index` refuses to silently rebuild an index that was built by `ollama`** — including an install predating the `index_meta` table (treated as a legacy `ollama`/`bge-m3` index). A plain `rag-index` against such an index now prints an explanation to stderr and exits non-zero before touching the backend or the database; `rag-index --reindex` is the explicit confirmation to rebuild it under the new provider. More generally, `rag-index` now exits non-zero on **any** backend-readiness or schema-reconciliation failure — previously it printed "Embedding setup failed" and exited 0, which a script could not detect. `rag-mcp`/`search_codebase` apply the same "pre-`index_meta` index is `ollama`/`bge-m3`" interpretation read-only, so a stale local install fails with the re-index instruction instead of an opaque pgvector dimension-mismatch error.
+- **`rag-setup` now saves the config before provisioning Infinity** — previously Infinity provisioning ran first, so a Docker failure there could abandon the whole wizard without ever writing `config.json`. The database-container step keeps its original position and abort-without-saving behavior.
+
+## [0.1.0-alpha] - 2026-07-12
+
+### Added
+
+- **`infinity` embedding provider** — a first-class local backend (`embedding.provider = infinity`) that serves both `/embeddings` and `/rerank` from a single [Infinity](https://github.com/michaelfeil/infinity) container (e.g. `jinaai/jina-embeddings-v2-base-code` + `Alibaba-NLP/gte-reranker-modernbert-base`), so local reranking works out of the box — something Ollama does not provide. Reuses the OpenAI-compatible HTTP path and adds only container-readiness behavior. Selecting `infinity` in `rag-setup` optionally provisions a reranker model and auto-wires `reranker.*` to the same server. Managed provisioning vs. an external/already-running server is decided by a `/health` probe (no config flag) — the same "already running?" probe now also defaults `rag-setup`'s "start Docker?" prompt to no for a reachable Ollama. The managed container prefers an NVIDIA GPU and falls back to CPU, and is recreated when the configured model set changes (Infinity binds models at startup). The managed container also gets `restart: unless-stopped` (survives host reboot) and disables Infinity's model warmup on CPU (a multi-minute benchmark at every boot; kept on for GPU where it is cheap). `jinaai/jina-embeddings-v2-base-code` (768 dims, the Infinity default) and `BAAI/bge-m3` (1024) are added to the model registry so their dimension resolves without a probe.
+- **Reranking is `openai-compatible`-only** — `reranker.provider` no longer accepts `ollama` (Ollama has no rerank API — the previous `/api/rerank` path always 404'd and silently fell back to vector order); the only valid backend is an OpenAI-compatible `/rerank` server (Infinity, TEI, vLLM). The default provider is now `openai-compatible`, and an existing `reranker.provider: 'ollama'` is migrated automatically on read. The embedding provider stays independent of the rerank server, so Ollama embeddings + an Infinity reranker is a valid combination.
+- **Pluggable embedding backends** — the embedder is now selected by `embedding.provider` via a factory, with three implementations: `ollama` (native `/api/embed`, the **backward-compatible default**), `openai-compatible` (one code path covering **OpenAI cloud, Infinity, TEI, vLLM, LM Studio, LocalAI** via a configurable `embedding.baseURL`), and `voyage` (native schema with mandatory `input_type`). To run a faster local backend or a cloud embedder you now edit config, not code.
+- **Batch embedding with retry** — `IEmbedder` gains `embedBatch(texts, inputType?)`; indexing sends one request per batch (`embedding.batchSize`, default 32) instead of one call per chunk, with bounded exponential backoff on HTTP 429/5xx. Makes cloud/Infinity indexing of a whole monorepo practical.
+- **Query/document input-type asymmetry** — `embed`/`embedBatch` take an optional `inputType` (`query` | `document`); search embeds as `query`, indexing as `document`. Voyage forwards it as `input_type`; Ollama and the OpenAI-compatible `/embeddings` endpoint ignore it (OpenAI has no such field).
+- **Flexible vector dimension** — the `code_chunk.embedding` column width now derives from the active model (`embedding.dimensions` override → model registry → one-shot runtime probe) instead of a hardcoded 1024, capped at ≤ 2000 (pgvector HNSW limit); larger cloud models reduce via the Matryoshka `dimensions`/`output_dimension` request parameter.
+- **`index_meta` source-of-truth table + mismatch guard** — a single-row table records `{ provider, model, dimensions }`. `search_codebase` (before embedding a query) and `rag-mcp` (at start) refuse with a clear "run `rag-index --reindex`" error when the configured model disagrees with the indexed one, instead of returning noise from a foreign vector space. A dimension change drops+recreates `code_chunk` at the new width on re-index; a same-dimension model change clears its rows.
+- **API-key secrets via `.env`** — keys live in `~/.config/codesearch-mcp/.env` (chmod 600), loaded into the environment at startup before config is read (a real env var wins over the file), so the client-spawned `rag-mcp` sees them too. Config values support an `env:VAR_NAME` reference; a missing referenced variable fails with a clear error. The dashboard shows `✔ set via <VAR>` for `env:` refs and masks raw keys.
+- **Auto-pull of local Ollama models with live progress** — `rag-index` now pulls the configured Ollama model if it is not present locally (only for `provider: ollama`; a no-op for cloud providers), so switching to (or back to) a local model that was never pulled just works without re-running `rag-setup`. The pull streams progress: a live single-line bar on a TTY (`downloading [██████░░░░] 62% (245.7/393.2 MB)`), de-duplicated status lines otherwise. `rag-setup`'s in-container pull now renders Ollama's native progress bar (allocates a TTY via `docker exec -t` when interactive) and pulls the model you actually chose (previously hardcoded to `bge-m3`).
+- **Cloud providers skip the Ollama container** — when a non-Ollama provider is selected, `rag-setup`'s Docker helper starts only the database service (`docker compose up -d db`) and skips the Ollama wait + model pull, instead of bringing up an unused Ollama container.
+- **Owner-symbol breadcrumb in search results** — each `search_codebase` result now shows a `Symbol:` line naming the declaration the chunk belongs to (`IndexerService.ingestFile`, `collectFiles`, …), so the enclosing class/method is visible without opening the file. Computed at index time into a new nullable `code_chunk.ownerPath` column (added via `ADD COLUMN IF NOT EXISTS`): the deepest declaration containing the chunk's start line wins, so a method beats its enclosing class. Method/constructor/accessor ranges are extracted purely for this breadcrumb and are **not** written to `code_symbol`, so `find_symbol` is unchanged. Module-scope chunks (imports) and rows indexed before this release show no breadcrumb; re-index to populate. No embedding-dimension change.
+- **AST-aware chunking** — TS/JS files are now chunked along top-level declaration boundaries (functions, classes, interfaces, types, exported variables) instead of at fixed character offsets, so a declaration is no longer split across chunks and search-hit line ranges map to coherent units of code. Declarations larger than `indexing.chunkSize` and the gaps between them (imports, top-level code) are sub-split by the recursive splitter, so no chunk exceeds the size limit and no source line is dropped. Reuses the symbol ranges already produced during indexing — no extra parse. New config key `indexing.chunkStrategy` (`ast` \| `character`, default `ast`); set it to `character` to restore the previous fixed-size behavior. Schema files (`.prisma`/`.graphql`/`.gql`) and files without symbol analysis keep the character splitter. Takes effect on the next re-index; no embedding-dimension or DB-schema change.
+
+### Changed (breaking)
+
+- **Config restructure** — the `ollama.*` section is replaced by `embedding.*` (`provider`, `baseURL`, `apiKey`, `model`, `dimensions`, `batchSize`); `reranker.*` gains `provider`/`baseURL`/`apiKey` mirroring the embedding taxonomy (Infinity can serve both from one server). **Legacy `ollama.host`/`ollama.model` configs are migrated to `embedding.*` automatically on read**, so existing installs keep working without manual edits.
+- **Query translation removed** — the `IEmbedder.translate` method and the `search.autoTranslate` / `search.translateModel` config keys are gone; queries are embedded as-is. The keys are silently ignored if still present.
+- **Re-index required on model/dimension change** — switching the embedding model, provider, or dimension now requires `rag-index --reindex` (a dimension change recreates `code_chunk`). Staying on Ollama + `bge-m3` keeps dimension 1024, backfills `index_meta`, and forces no re-index. The mismatch guard makes any disagreement loud rather than silently corrupting results.
 
 ## [0.0.18-alpha] - 2026-07-04
 

package/README.md

@@ -1,8 +1,17 @@
 # codesearch-mcp
 
-Semantic codebase search for AI agents. Indexes your local projects into a PostgreSQL vector database and exposes search via the Model Context Protocol (MCP).
+Semantic code search for AI agents — finds code by **meaning**, not keywords. Indexes your local projects into a PostgreSQL vector database and exposes search via the Model Context Protocol (MCP), so Claude can search your entire codebase automatically.
 
-Ask your AI agent things like *"where is the payment retry logic?"* or *"show me how auth tokens are validated"* — across all your projects at once.
+Ask your AI agent things like *"where is the payment retry logic?"* or *"show me how auth tokens are validated"* — it will find the right code even if those exact words never appear.
+
+**Why codesearch-mcp:**
+
+- **Dramatically fewer tokens** — without RAG, answering a cross-project question means discovery calls, directory listings, and reading several wrong files before finding the right one; a workspace of 10 projects can easily mean 50 000+ lines to sift through. With RAG, one `search_codebase` call returns 5 targeted chunks (~200–500 lines total) from across *all* projects at once — the model gets the right code immediately, with no exploration overhead
+- **Reranker narrows the result further** — a cross-encoder reranker (enabled by default) re-scores the candidate pool so the top results are the most relevant, not just the closest vector; fewer results means less context consumed per answer
+- **Semantic, not grep** — embeddings capture intent; queries in plain English or pseudocode find the relevant code regardless of naming
+- **Local and private** — default setup runs entirely on your machine (Infinity embeddings + PostgreSQL); no code leaves your network
+- **Code-structure aware** — chunks align to function and class boundaries; each result shows which declaration it belongs to (`InvoiceService.create`, `collectFiles`, …), so the model understands context without reading the surrounding file
+- **Import graph** — `include_graph_context` appends direct imports and dependents inline with each result, replacing separate `get_dependencies` / `get_dependents` calls
 
 > **Alpha** — works well locally, API may change before 1.0.
 
@@ -10,16 +19,17 @@ See `CHANGELOG.md` for version history.
 
 ## How it works
 
-1. `rag-index` scans your projects, chunks source files, embeds them with [bge-m3](https://huggingface.co/BAAI/bge-m3) via Ollama, and stores vectors in PostgreSQL
+1. `rag-index` scans your projects, chunks source files, embeds them using your configured embedding backend (default: [jina-embeddings-v2-base-code](https://huggingface.co/jinaai/jina-embeddings-v2-base-code) via a local [Infinity](https://github.com/michaelfeil/infinity) container), and stores vectors in PostgreSQL
 2. `rag-mcp` exposes an MCP server — Claude calls it automatically when you ask about code
 3. Incremental — only changed files are re-indexed on subsequent runs
+4. Index stays consistent — deleted files, `.ragignore`-excluded files, and deselected projects are removed from all four index tables on the next `rag-index` run
 
-**Stack:** TypeScript · Drizzle ORM · PostgreSQL + pgvector · Ollama (bge-m3) · LangChain text splitters · ts-morph
+**Stack:** TypeScript · Drizzle ORM · PostgreSQL + pgvector · Infinity (jina-embeddings-v2-base-code) · LangChain text splitters · ts-morph
 
 ## Requirements
 
 - Node.js 18+
-- Docker (for local DB + Ollama) — or bring your own PostgreSQL with pgvector + Ollama
+- Docker (for local DB + Infinity) — or bring your own PostgreSQL with pgvector + an embedding backend (see [Embedding providers](#embedding-providers))
 
 ## Installation
 
@@ -35,11 +45,19 @@ rag-setup
 
 Walks you through:
 - **Database** — enter connection details; optionally starts Docker containers automatically
-- **Ollama** — configure API URL
+- **Embedding backend** — pick a provider (`infinity` / `openai-compatible` / `voyage`), its model, and an API key if the provider needs one (see [Embedding providers](#embedding-providers))
 - **Logging** — choose log level and directory (defaults: `error` level, `~/.config/codesearch-mcp/logs/`)
 
 Config is saved to `~/.config/codesearch-mcp/config.json`.
 
+## Register with Claude Code
+
+```bash
+claude mcp add codesearch-mcp -s user -- rag-mcp
+```
+
+Then ask Claude anything about your code — it will use the tools below automatically.
+
 ## Index your projects
 
 ```bash
@@ -62,14 +80,6 @@ If `workspaceRoot` is an [Nx](https://nx.dev) workspace (detected by `nx.json`),
 
 No configuration needed — detection is automatic. Plain (non-Nx) workspaces keep the original behavior: each immediate subdirectory with a `package.json` is one project.
 
-## Register with Claude Code
-
-```bash
-claude mcp add codesearch-mcp -s user -- rag-mcp
-```
-
-Then ask Claude anything about your code — it will use the tools below automatically.
-
 ## Web dashboard
 
 ```bash
@@ -78,15 +88,15 @@ rag-web                 # http://127.0.0.1:4319  (--port to change)
 
 A local web UI (bound to `127.0.0.1` only) with a schema-driven config editor,
 an overview of indexed projects, and the interactive dependency graph. The
-config form shows the applied default for any unset key and offers dropdowns for
-model fields (the embedding model is restricted to dimension-compatible models).
+config form shows the applied default for any unset key and suggests dimension-compatible
+models in the model fields (free text is always accepted).
 See [docs/web-dashboard.md](docs/web-dashboard.md) for the full reference.
 
 ## MCP Tools
 
 | Tool | Description |
 |------|-------------|
-| `search_codebase` | Semantic search across all indexed projects. Accepts `include_graph_context` to append direct imports/dependents to each result, and `include_neighbors` for surrounding chunks |
+| `search_codebase` | Semantic search across all indexed projects. Accepts `include_graph_context` to append direct imports/dependents to each result, `include_neighbors` for surrounding chunks, and `threshold` to override the similarity threshold for that query |
 | `list_projects` | List projects with metadata and chunk counts |
 | `find_symbol` | Find a function, class, or type by exact name — faster than semantic search |
 | `list_files` | Browse the directory tree of a project |
@@ -104,6 +114,7 @@ Add `.ragmeta.json` to a project root to enrich `list_projects` and generate a v
 {
   "description": "Main API server",
   "stack": "NestJS, GraphQL, PostgreSQL",
+  "notes": "Uses optimistic locking — see order-service/docs/concurrency.md",
   "relations": {
     "consumed_by": ["crm-frontend"],
     "consumes": ["billing-service", "auth-service"]
@@ -111,14 +122,16 @@ Add `.ragmeta.json` to a project root to enrich `list_projects` and generate a v
 }
 ```
 
-After each indexing run an interactive `project-graph.html` is written to the workspace root.
+After each indexing run an interactive `project-graph.html` is written to the workspace root. The same visualization is available in the web dashboard's Graph tab (`rag-web`).
 
 ## Graph context
 
 Pass `include_graph_context: true` to `search_codebase` to see direct imports and dependents inline with each result — without extra `get_dependencies` / `get_dependents` calls:
 
 ```
-**File:** src/services/InvoiceService.ts  **Line:** 45
+**File:** src/services/InvoiceService.ts
+**Symbol:** InvoiceService.create
+**Line:** 45
 ...code...
 **Relevance:** 0.32
 **Imports:** src/repositories/InvoiceRepo.ts · src/dto/CreateInvoiceDto.ts
@@ -127,6 +140,8 @@ Pass `include_graph_context: true` to `search_codebase` to see direct imports an
 
 Useful for immediately understanding a file's role in the dependency graph.
 
+Each result also shows a **Symbol** breadcrumb — the declaration the chunk belongs to (`InvoiceService.create`, `collectFiles`, …) — so you can see the enclosing class/method without opening the file. It is omitted for module-scope chunks and for projects indexed before this feature (re-index to populate).
+
 ## Excluding files
 
 Create `.ragignore` in a project root — same syntax as `.gitignore`:
@@ -162,6 +177,7 @@ rag-config show                      # print current config (password redacted)
 rag-config get <key>                 # print a single value
 rag-config set <key> <value>         # update a value
 rag-config reset <key>               # remove key (reverts to default)
+rag-config edit                      # open config in $EDITOR; validates on save (use this to recover an invalid config)
 ```
 
 | Key | Default | Description |
@@ -172,14 +188,169 @@ rag-config reset <key>               # remove key (reverts to default)
 | `db.password` | — | PostgreSQL password |
 | `db.name` | `rag_<os-username>` | Database name — auto-created on first run |
 | `workspaceRoot` | — | Root of your projects directory |
-| `ollama.host` | `http://localhost:11434` | Ollama API URL |
-| `ollama.model` | `bge-m3` | Embedding model |
+| `embedding.provider` | `infinity` | Embedding backend: `infinity` \| `openai-compatible` \| `voyage`. A stored `ollama` value fails validation with migration guidance (see [Embedding providers](#embedding-providers)) |
+| `embedding.baseURL` | per-provider | Backend base URL (Infinity/TEI/vLLM `http://localhost:7997`, OpenAI `https://api.openai.com/v1`, Voyage `https://api.voyageai.com/v1`; Ollama via `openai-compatible` at `http://localhost:11434/v1`) |
+| `embedding.apiKey` | `''` | API key, or `env:VAR_NAME` to read from `.env`/environment. Empty for local servers |
+| `embedding.model` | `jinaai/jina-embeddings-v2-base-code` | Embedding model |
+| `embedding.dimensions` | (derived) | Vector width override; required for models whose native width exceeds 2000 (Matryoshka). Otherwise derived from model → registry → runtime probe |
+| `embedding.batchSize` | `32` | Texts per batch embedding request |
 | `search.similarityThreshold` | `0.6` | Max cosine distance — raise if search returns nothing |
-| `search.autoTranslate` | `false` | Translate non-English queries to English before embedding |
-| `search.translateModel` | `llama3.2` | Ollama model for translation (requires `ollama pull llama3.2`) |
+| `reranker.enabled` | `true` | Enable cross-encoder reranking; requires `reranker.model` |
+| `reranker.provider` | `openai-compatible` | Rerank backend — `openai-compatible` `/rerank` only (Infinity, TEI, vLLM) |
+| `reranker.baseURL` | `embedding.baseURL` | Rerank server base URL (Infinity serves embed + rerank from one server) |
+| `reranker.model` | `mixedbread-ai/mxbai-rerank-xsmall-v1` | Reranker model name |
+| `reranker.overfetch` | `5` | Candidate multiplier before reranking (`limit × N`) |
+| `reranker.candidateFloor` | `80` | Minimum candidate pool fed to the reranker regardless of `limit` — `fetchLimit = min(max(limit × overfetch, candidateFloor), 200)`. Raise on GPU/fast hosts for better recall; lower on weak CPU to cut latency |
+| `indexing.chunkSize` | `1500` | Max characters per chunk |
+| `indexing.chunkOverlap` | `200` | Overlap between chunks |
+| `indexing.embedBatchSize` | `5` | Parallel embedding requests per batch |
 | `indexing.astBatchSize` | `3000` | Max files per ts-morph batch — lower if indexing large projects causes OOM |
+| `indexing.chunkStrategy` | `ast` | How TS/JS files are chunked: `ast` = boundaries aligned to functions/classes, `character` = fixed-size splitter. Re-index to apply |
+| `search.defaultLimit` | `5` | Default result count returned by `search_codebase` |
+| `search.maxLimit` | `20` | Hard cap on result count |
 | `logging.level` | `error` | Log level written to file: `error` \| `warn` \| `info` \| `debug` |
 | `logging.dir` | `~/.config/codesearch-mcp/logs` | Base log directory, or `"false"` to disable |
+| `logging.searchMetrics` | `false` | Emit one structured search-tuning line per query (at `info`, independent of `logging.level`) — use it to size `candidateFloor`/`overfetch`/`threshold` for your hardware |
+
+## Embedding providers
+
+`embedding.provider` selects the backend. All three produce vectors the same way; they differ in where they run and which models they offer. There is no dedicated Ollama provider — Ollama exposes an OpenAI-compatible `/v1/embeddings` endpoint, so it's reached through `openai-compatible` (see below).
+
+### Choosing a provider
+
+| Provider | When to use | Needs |
+|----------|-------------|-------|
+| `infinity` | Default. Local, best quality **plus a real reranker** in one container; GPU or CPU. | Docker (managed) or a running Infinity server |
+| `openai-compatible` | OpenAI cloud, or any OpenAI-style server (TEI, vLLM, LM Studio, LocalAI, **or Ollama via its `/v1` endpoint**). | `baseURL` (+ `apiKey` for cloud) |
+| `voyage` | Voyage AI cloud, code-specialized embeddings. | Voyage `apiKey` |
+
+Easiest path: `rag-setup` asks for the provider, model, and any API key. Haven't run setup? It's the place to start. To change things later, edit config directly (below).
+
+### Switching provider or changing the model
+
+Any change to the provider, model, or dimension requires a **re-index** — the mismatch guard blocks search until you do it. The index is a rebuildable cache, so there's no data migration.
+
+```bash
+# Switch to OpenAI cloud
+rag-config set embedding.provider openai-compatible
+rag-config set embedding.baseURL  https://api.openai.com/v1
+rag-config set embedding.model    text-embedding-3-small
+rag-config set embedding.apiKey   env:OPENAI_API_KEY     # key lives in .env, not config.json
+rag-config set reranker.enabled   false                  # OpenAI has no /rerank endpoint
+rag-index --reindex
+
+# Switch to Voyage AI
+rag-config set embedding.provider voyage
+rag-config set embedding.model    voyage-code-3
+rag-config set embedding.apiKey   env:VOYAGE_API_KEY     # key lives in .env, not config.json
+rag-config set reranker.enabled   false                  # Voyage has no /rerank endpoint
+rag-index --reindex
+
+# Change only the model on the current provider
+rag-config set embedding.model <model-name>
+rag-index --reindex
+```
+
+### Breaking change: the `ollama` provider was removed
+
+> **Migrating from an older version?** A config with `embedding.provider: ollama` (or an old-style top-level `ollama` block, or `embedding.baseURL`/`embedding.model` still pointing at Ollama's defaults with no `provider` set) now **fails to load** with an error explaining the fix.
+>
+> Fix it with `rag-config edit` (opens the file even while it's invalid) or by re-running `rag-setup`: set `embedding.provider` to `openai-compatible` and `embedding.baseURL` to `http://localhost:11434/v1` to keep using Ollama, or switch to `infinity` for a managed local backend.
+>
+> Either way, run `rag-index --reindex` afterward. A plain `rag-index` additionally refuses (non-zero exit, no changes made) to silently rebuild an index built by the old `ollama` provider — including an older install with no `index_meta` row, which is assumed to be a pre-migration Ollama/`bge-m3` index.
+
+Per-provider notes:
+
+- **infinity** — the model set is bound at container startup, so changing `embedding.model` (or the reranker model) **recreates** the managed container on the next `rag-setup`/`rag-index`/dashboard save. Defaults: `jinaai/jina-embeddings-v2-base-code` + reranker `mixedbread-ai/mxbai-rerank-xsmall-v1`. Both models are included in the compose template and reranking is **enabled by default** (`reranker.enabled: true`).
+- **openai-compatible** — point `embedding.baseURL` at the server; set `embedding.apiKey` (as `env:VAR`) for cloud, leave empty for local servers (TEI/vLLM/LM Studio/**Ollama**). E.g. `text-embedding-3-small`, `text-embedding-3-large`. Cloud endpoints (OpenAI, Azure) don't expose `/rerank` — disable the reranker or point it at a separate server (see [Reranker with a cloud embedder](#reranker-with-a-cloud-or-external-embedder)).
+- **Ollama** (via `openai-compatible`) — set `embedding.baseURL` to `http://localhost:11434/v1` and `embedding.model` to any model you've pulled (e.g. `bge-m3`). No dedicated code path, no auto-pull — pull the model yourself with `ollama pull <model>` first.
+- **voyage** — set `embedding.apiKey` (as `env:VAR`); e.g. `voyage-code-3`, `voyage-3`. Voyage doesn't expose a `/rerank` endpoint in the OpenAI-compatible format — disable the reranker or point it at a separate server (see [Reranker with a cloud embedder](#reranker-with-a-cloud-or-external-embedder)).
+
+The vector **dimension** is derived automatically (model registry → one-shot probe); set `embedding.dimensions` only to force a Matryoshka-reduced width for models whose native size exceeds 2000.
+
+### Infinity — local embeddings + reranking in one container
+
+`infinity` is the default provider (select it in `rag-setup`, or set `embedding.provider = infinity`) — it runs [Infinity](https://github.com/michaelfeil/infinity), which serves both `/embeddings` and `/rerank` from a single container, so you get a real cross-encoder reranker locally. During setup you're asked for an embedding model (default `jinaai/jina-embeddings-v2-base-code`, 768 dims) and a reranker model (default `mixedbread-ai/mxbai-rerank-xsmall-v1`, enabled by default); `reranker.*` is auto-wired to the same server.
+
+Managed vs. external is decided by a `/health` probe at `embedding.baseURL` (default `http://localhost:7997`) plus the managed compose file — no config flag:
+
+- **Managed** — the app-owned compose file (`~/.config/codesearch-mcp/docker-compose.yml`) declares the `infinity` service and `embedding.baseURL` points at its published port on `localhost`/`127.0.0.1`/`[::1]`. If nothing answers the probe, or the running container doesn't serve the configured model set, `rag-setup`, `rag-index`, and the web dashboard bring up (or recreate) that container. The full model set (embedding + reranker, when on the same server) is written into the compose file first — a plain `docker compose up -d` afterward serves the same set, no env var needed. It **prefers an NVIDIA GPU and falls back to CPU** automatically (GPU layered via `docker-compose.infinity-gpu.yml` only when `nvidia-smi` is present). To opt out of management entirely, remove the `infinity` service from that compose file.
+- **External** — any other `embedding.baseURL` (a remote host, a server on a different port, or a managed compose file that's absent/unreadable) is used as-is; no Docker command ever runs. If it's missing a required model, you're shown the `--model-id` flag to restart it with.
+
+The same "already running?" probe applies everywhere: a reachable backend defaults `rag-setup`'s "start Docker?" prompt to no, and a web-dashboard save that doesn't change the served model set never touches Docker.
+
+### Reranker with a cloud or external embedder
+
+The reranker is **enabled by default** (`reranker.enabled: true`) and `reranker.baseURL` defaults to `embedding.baseURL`. Cloud embedding endpoints (OpenAI, Voyage, Azure) don't serve `/rerank` — the reranker call will fail silently and fall back to vector order on every query. Fix it one of two ways:
+
+**Option A — disable reranking** (simplest, recommended for cloud-only setups):
+
+```bash
+rag-config set reranker.enabled false
+```
+
+**Option B — point the reranker at a separate server** (e.g. a local Infinity or TEI instance while using Voyage/OpenAI for embeddings):
+
+```bash
+rag-config set reranker.baseURL http://localhost:7997
+rag-config set reranker.model   mixedbread-ai/mxbai-rerank-xsmall-v1
+# reranker.enabled is already true
+```
+
+The embedding and reranking servers are fully independent — any combination works: Voyage embeddings + local Infinity reranker, OpenAI embeddings + remote TEI reranker, etc.
+
+### Secrets & external embedders
+
+API keys live in `~/.config/codesearch-mcp/.env` (chmod 600), **not** in `config.json`. Reference them with `env:VAR_NAME`, e.g. `embedding.apiKey = env:OPENAI_API_KEY` with `OPENAI_API_KEY=sk-…` in `.env`. The file is loaded into the environment at startup (a real environment variable wins over the file), so all binaries — including the client-spawned `rag-mcp` — see it identically.
+
+Switching the embedding model, provider, or dimension is applied by **re-indexing** (`rag-index --reindex`) — the index is a rebuildable cache, so there is no data migration. A single-row `index_meta` table records which model built the index; `search_codebase` and `rag-mcp` refuse with a clear "re-index required" error when the configured model disagrees, rather than returning noise from a foreign vector space.
+
+## Tuning search recall
+
+When reranking is enabled, `search_codebase` fetches a **candidate pool** by cosine distance, reranks it with the cross-encoder, and returns the top `limit`. Three knobs and one relationship govern how good the results are and how long they take:
+
+- **Pool size** — `fetchLimit = min(max(limit × overfetch, candidateFloor), 200)`. `candidateFloor` (default `80`) guarantees the reranker a large enough menu even at the default `limit = 5`; without it a small `limit` starves the reranker (only `5 × 5 = 25` candidates) and its best matches never enter the window.
+- **Threshold ↔ window** — raising `search.similarityThreshold` lets *more* chunks under the bar, so the pool must be wide enough to reach them; a strict threshold with a small pool silently truncates recall. Widen `candidateFloor`/`overfetch` when you raise the threshold.
+- **Latency** — the reranker cost grows with pool size (on CPU, tens of seconds; a rough guide is a large fixed cost plus a per-candidate increment). Bigger pool = better recall, slower query. GPU/fast hosts can raise `candidateFloor`; ultra-constrained hosts lower it.
+
+To size these for **your** hardware, turn on telemetry:
+
+```bash
+rag-config set logging.searchMetrics true
+```
+
+Each search then writes one `info` line (independent of `logging.level`) with, among others: `fetchLimit`, `candidatesReturned`, `poolSaturated`, `maxCosineRankSurvived`, `distinctFilesInResult`, `topFileShare`, and `embedMs`/`rerankMs`/`dbMs`/`totalMs`. Read them as:
+
+- **`poolSaturated: true`** — more chunks were under the threshold than the window took → recall is being truncated; raise `candidateFloor`/`overfetch`.
+- **`maxCosineRankSurvived`** — the deepest cosine rank that survived reranking. If it stays well below `fetchLimit`, the pool is bigger than needed → shrink it to save latency; if it approaches `fetchLimit`, widen.
+
+## Index consistency
+
+`rag-index` keeps the four index tables (`code_chunk`, `code_symbol`, `code_call`, `code_dependency`) consistent after every run:
+
+| Event | What happens |
+|---|---|
+| **File deleted from disk** | All rows for that file are removed from the four tables in one transaction on the next `rag-index`. The file disappears from `find_symbol`, `get_callers`, `get_dependencies`, and `get_dependents`. |
+| **File matched by `.ragignore`** | Same as deleted — treated as not collected, removed from all four tables. |
+| **Project deselected** | All rows for that project are removed from the four tables. To bring it back, re-select and re-index — the index is a cache. The removal is printed as `Removed project <name> (not selected)`. |
+| **`workspaceRoot` changed** | All projects indexed under the old root (their paths no longer match anything under the new root) are pruned. Each gets its own `Removed project` line. No confirmation is required — the index is a cache that `rag-index` rebuilds. |
+
+A failure pruning one project does not block the others or stop indexing of the selected projects, but `rag-index` exits non-zero to signal the partial failure.
+
+## Architecture
+
+**Stack:** TypeScript on Node.js 18+ · PostgreSQL 17 + pgvector (HNSW, cosine distance) via Drizzle ORM · ts-morph for AST/symbol/call-graph extraction · `@langchain/textsplitters` for chunk fallback · MCP SDK over stdio · Zod v4 for config validation and JSON-Schema generation · pino + pino-roll for logging · plain `node:http` dashboard with vis-network · Vitest · Docker Compose for Postgres + Infinity.
+
+**Design:**
+- **Composition root + manual DI** — `src/container.ts` builds every service once and wires it through constructors; no module-level singletons, no DI framework.
+- **Dependency inversion** — services depend on interfaces (`IEmbedder`, `IDatabase`, `IAstParser`, `IChunker`, `ISearcher`, `IProjectDiscovery`, `IMonorepoResolver`, `IMcpToolHandler`), so providers (OpenAI-compatible/Voyage/Infinity — Ollama via `openai-compatible`), chunking strategy (`ast`/`character`), and monorepo resolvers are swappable.
+- **Command pattern** — each CLI mode (`rag-index`, `rag-mcp`, `rag-config`, `rag-web`) is its own `ICommand`; `app.ts` only dispatches.
+- **Registry / Open-Closed** — MCP tools are `IMcpToolHandler` implementations registered in a `Map`; adding a tool never touches `McpServer`.
+- **Fallback chains** — vector dimension: config override → model registry → runtime probe; `find_symbol`/`get_dependents`: indexed table → grep; failed rerank → vector order.
+- **Incremental indexing** — mtime fast path, SHA-256 fallback for clones/transfers, AST-aligned chunk boundaries so declarations aren't split arbitrarily.
+- **Index/model consistency guard** — `index_meta` tracks the embedding model that built the index; search refuses with a clear re-index message on mismatch instead of returning noise.
+
+Full internals (per-file responsibilities, every pattern with file:line references) are documented in `CLAUDE.md`.
 
 ## License
 

package/dist/app.js

@@ -34,17 +34,53 @@ var __importStar = (this && this.__importStar) || (function () {
     };
 })();
 Object.defineProperty(exports, "__esModule", { value: true });
+const fs = __importStar(require("fs"));
 const path = __importStar(require("path"));
 const help_1 = require("./src/help");
 const container_1 = require("./src/container");
 const config_1 = require("./src/config");
+const config_store_1 = require("./src/config-store");
 const IndexCommand_1 = require("./src/commands/IndexCommand");
 const McpCommand_1 = require("./src/commands/McpCommand");
 const BackfillCommand_1 = require("./src/commands/BackfillCommand");
 const ConfigCommand_1 = require("./src/commands/ConfigCommand");
 const WebCommand_1 = require("./src/commands/WebCommand");
 const setup_1 = require("./src/setup");
+/**
+ * Load `~/.config/codesearch-mcp/.env` into `process.env` before config is read.
+ * All five binaries share this file — `rag-mcp` in particular is spawned by the
+ * MCP client and does not inherit shell exports. A real `process.env` value
+ * always wins over the file. Minimal `KEY=VALUE` parser (no `dotenv` dep):
+ * skips blanks and `#` comments, strips one layer of surrounding quotes.
+ */
+function loadDotEnv(file) {
+    let text;
+    try {
+        text = fs.readFileSync(file, 'utf-8');
+    }
+    catch {
+        return; // absent / unreadable → no-op
+    }
+    for (const rawLine of text.split(/\r?\n/)) {
+        const line = rawLine.trim();
+        if (!line || line.startsWith('#'))
+            continue;
+        const eq = line.indexOf('=');
+        if (eq === -1)
+            continue;
+        const key = line.slice(0, eq).trim();
+        if (!key)
+            continue;
+        let val = line.slice(eq + 1).trim();
+        if (val.length >= 2 && ((val[0] === '"' && val.at(-1) === '"') || (val[0] === "'" && val.at(-1) === "'"))) {
+            val = val.slice(1, -1);
+        }
+        if (!(key in process.env))
+            process.env[key] = val; // real env wins
+    }
+}
 async function main() {
+    loadDotEnv(config_store_1.ENV_FILE);
     const argv1 = path.basename(process.argv[1] ?? '');
     const isIndex = argv1 === 'rag-index' || process.argv[2] === 'index' || process.argv[2] === 'rag-index';
     const isMcp = argv1 === 'rag-mcp' || process.argv[2] === 'mcp' || process.argv[2] === 'rag-mcp';
@@ -108,7 +144,7 @@ async function main() {
     }
     if (mode === 'setup') {
         await (0, setup_1.runSetup)();
-        process.exit(0);
+        process.exit(process.exitCode ?? 0);
     }
     if (mode === 'config') {
         await new ConfigCommand_1.ConfigCommand(effectiveArgs).execute();
@@ -153,6 +189,6 @@ async function main() {
     }
     await factory().execute();
     await container.db.destroy();
-    process.exit(0);
+    process.exit(process.exitCode ?? 0);
 }
 main().catch(err => { console.error(err); process.exit(1); });

package/dist/src/commands/IndexCommand.js

@@ -6,6 +6,7 @@ const project_selector_1 = require("../project-selector");
 const graph_1 = require("../graph");
 const MonorepoAwareDiscoveryService_1 = require("../services/MonorepoAwareDiscoveryService");
 const file_utils_1 = require("../file-utils");
+const legacy_ollama_index_1 = require("../services/legacy-ollama-index");
 class IndexCommand {
     container;
     argv1;
@@ -14,11 +15,48 @@ class IndexCommand {
         this.argv1 = argv1;
     }
     async execute() {
-        const { db, indexer, searcher, config, discovery } = this.container;
+        const { db, indexer, searcher, config, discovery, embedder } = this.container;
         const args = process.argv.slice(this.argv1 === 'rag-index' ? 2 : 3);
         const force = args.includes('--reindex');
         const forceSelect = args.includes('--select');
         await db.initDbWithSchema();
+        // A former Ollama installation the config layer cannot recognise (old
+        // defaults, no `embedding`/`ollama` block) is protected here: without an
+        // explicit `--reindex`, refuse to touch an index built by `ollama` (or a
+        // pre-`index_meta` install, which is indistinguishable from one) before
+        // any backend readiness check, Docker command, or schema work.
+        if (!force) {
+            let identity;
+            try {
+                identity = await (0, legacy_ollama_index_1.resolveIndexedIdentity)(db);
+            }
+            catch (err) {
+                console.error(`\n${err.message}`);
+                process.exitCode = 1;
+                return;
+            }
+            if (identity?.provider === 'ollama') {
+                console.error(`\nThe existing index was built with ${identity.provider}/${identity.model}, but "ollama" is no ` +
+                    `longer a supported embedding provider (configured: ${config.embedding.provider}/${config.embedding.model}). ` +
+                    `Run \`rag-index --reindex\` to confirm rebuilding the index under the configured provider.`);
+                process.exitCode = 1;
+                return;
+            }
+        }
+        // Ensure the local model is served (Infinity only — no-op for cloud), then
+        // resolve the active embedding dimension and reconcile it with index_meta
+        // (recreate code_chunk on a dimension change, clear rows on a model change)
+        // before scanning. A clear error here beats a mid-run vector width mismatch.
+        try {
+            await embedder.ensureModelAvailable?.();
+            const meta = await indexer.prepareEmbeddingSchema();
+            console.log(`Embedding: ${meta.provider} / ${meta.model} (${meta.dimensions} dims)`);
+        }
+        catch (err) {
+            console.error(`\nEmbedding setup failed: ${err.message}`);
+            process.exitCode = 1;
+            return;
+        }
         const controller = new AbortController();
         const teardown = (0, interrupt_1.setupInterrupt)(controller);
         try {
@@ -29,6 +67,22 @@ class IndexCommand {
             // Expand any selected nested-monorepo unit (e.g. an Nx repo) into its
             // sub-projects; ordinary projects pass through unchanged.
             const projects = (0, MonorepoAwareDiscoveryService_1.toLeafProjects)(selected);
+            let pruneResult;
+            try {
+                pruneResult = await indexer.pruneProjects(projects.map(p => p.name));
+            }
+            catch (err) {
+                console.error(`\nPruning failed: ${err.message}`);
+                process.exitCode = 1;
+                return;
+            }
+            const { removed, failed } = pruneResult;
+            for (const name of removed) {
+                console.log(`Removed project ${name} (not selected)`);
+            }
+            if (failed.length > 0) {
+                process.exitCode = 1;
+            }
             const filesPerProject = projects.map(p => (0, file_utils_1.collectFiles)(p.path, (0, file_utils_1.loadIgnore)(p.path), p.path));
             const totalFiles = filesPerProject.reduce((sum, f) => sum + f.length, 0);
             console.log(`\n${projects.length} projects — ${totalFiles} files total`);

package/dist/src/commands/McpCommand.js

@@ -7,14 +7,18 @@ class McpCommand {
         this.container = container;
     }
     async execute() {
-        const { db, mcp, logger } = this.container;
+        const { db, mcp, searcher, logger } = this.container;
         let startupError;
         try {
             await db.initDb();
+            // Refuse to serve searches embedded into a foreign vector space — surface
+            // the mismatch through the existing startup-error plumbing so every tool
+            // call reports it clearly.
+            await searcher.assertIndexMatches();
         }
         catch (err) {
             startupError = err instanceof Error ? err : new Error(String(err));
-            logger.error('McpCommand: database connection failed', { reason: startupError.message });
+            logger.error('McpCommand: startup check failed', { reason: startupError.message });
         }
         try {
             await mcp.start(startupError);

package/dist/src/config-error.d.ts

@@ -0,0 +1,7 @@
+/**
+ * Own module (not `config.ts`) so `config-store.ts` can throw it too without
+ * an import cycle — `config.ts` imports from `config-store.ts`.
+ */
+export declare class ConfigError extends Error {
+    constructor(message: string);
+}

package/dist/src/config-error.js

@@ -0,0 +1,14 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.ConfigError = void 0;
+/**
+ * Own module (not `config.ts`) so `config-store.ts` can throw it too without
+ * an import cycle — `config.ts` imports from `config-store.ts`.
+ */
+class ConfigError extends Error {
+    constructor(message) {
+        super(message);
+        this.name = 'ConfigError';
+    }
+}
+exports.ConfigError = ConfigError;