@esankhan3/code-search-mcp 0.1.0 → 0.3.0

package/README.md

@@ -39,9 +39,43 @@ That's the whole setup. Claude Desktop now knows your codebase.
 
 ---
 
-## Three modes, one binary
+## Three bin entries (one install)
 
-### Remote proxy (default)
+The npm package ships three binaries. Pick the one that matches the
+job — none of them need a separate install.
+
+| Bin | Use case |
+|---|---|
+| `code-search-mcp` | The MCP-server entry. Three operating modes below. |
+| `code-search` | Standalone CLI: `index` / `query` / `status` / `reset` / `daemon` / `serve` / `mcp`. |
+| `code-search-daemon` | Long-running indexer with file-watcher and JSON-RPC socket. |
+
+### Standalone CLI
+```sh
+code-search index .                                  # index current dir
+code-search query "where do we verify JWTs"          # hybrid by default
+code-search query "createPet" --mode bm25 --top-k 5  # exact-identifier mode
+code-search status --project petshop                 # JSON report
+code-search reset --project petshop                  # drop the index
+code-search --print-config                           # resolved config (redacted)
+```
+Every `CodeSearchConfig` leaf is settable as `--<dotted-path>` —
+e.g. `--embedding.provider codestral`, `--retrieval.max-chunks 12`,
+`--no-indexing.auto-index`.
+
+### Daemon mode (long-running indexer)
+```sh
+code-search-daemon --workspace /path/to/repos
+```
+Builds the index once, then watches the workspace and debounce-
+reindexes on file events. Listens on a UDS socket at
+`<dataDir>/daemon/<project>.sock` (named pipe on Windows). The CLI
+and MCP server automatically prefer the daemon when it's alive,
+falling back to in-process when it isn't.
+
+### MCP — three modes (`code-search-mcp` argv)
+
+**Remote proxy (default)**
 ```sh
 code-search-mcp                  # default — proxies to remote
 code-search-mcp --remote URL
@@ -51,7 +85,7 @@ Zero local setup, zero local index — perfect for hosted
 deployments where the index lives in the cloud and dev machines
 stay light. Auth via `CODE_SEARCH_API_KEY`.
 
-### Local
+**Local**
 ```sh
 code-search-mcp --local /path/to/repos
 code-search-mcp --local github:my-org/my-pattern
@@ -60,14 +94,15 @@ Discovers every repo under a path (or clones a GitHub org), builds
 the knowledge base, and serves over stdio. Works fully offline if
 your embedder + reranker are local (Ollama).
 
-### Serve
+**Serve**
 ```sh
 code-search-mcp --serve --port 4000 --auth api-key
 ```
 Boots an HTTP server (Streamable HTTP transport, SSE optional)
-with `/mcp`, `/health`, `/status`, and an admin `POST /index`. Use
-this to host one index for a whole team — every dev points their
-client at the same URL.
+with `/mcp`, `/health`, `/ready`, `/version`, `/status`,
+`/metrics` (Prometheus), `/admin/api/status`, and an admin
+`POST /index`. Use this to host one index for a whole team — every
+dev points their client at the same URL.
 
 ---
 
@@ -142,6 +177,45 @@ repos*. So does `impact_analysis`.
 
 ---
 
+## Observability
+
+`code-search-mcp --serve` exposes these endpoints out of the box —
+no scrape config or APM agent required.
+
+| Endpoint | Returns |
+|---|---|
+| `GET /health` | Liveness ping with project + uptime |
+| `GET /ready` | `200` only after first index completes |
+| `GET /version` | Package + Node + platform |
+| `GET /status` | Current indexing phase + last-success history |
+| `GET /metrics` | Prometheus text format (queries, latency histogram, chunks, embedding/LLM calls, errors) |
+| `GET /admin/api/status` | JSON status dashboard for the admin UI |
+
+Set `CODE_SEARCH_STRUCTURED_LOGS=1` for one-JSON-line-per-event
+stderr logs, and `CODE_SEARCH_TELEMETRY_RECORD_QUERIES=1` to opt
+into recording query text in spans (off by default — queries are
+PII-shaped).
+
+---
+
+## Multi-project / multi-tenant
+
+Drop project entries under `<dataDir>/projects/<name>/project.yaml`:
+
+```yaml
+workspace: /Users/me/repos/petshop
+scopes: [team-shop]
+quotas:
+  max_queries_per_minute: 60
+  max_embedding_cost_usd: 5
+```
+
+`ProjectRegistry` reads them at boot. `projectAccessAllowed`
+gates by scope (`*` is admin). `checkProjectQuota` enforces a
+sliding 1-minute window per `(identity, project)`.
+
+---
+
 ## Server features
 
 ### Auth, three flavors
@@ -221,21 +295,64 @@ LanceDB store, same graph files.
 
 ---
 
-## Configuration
+## Configuration — defaults → file → env → CLI flags
+
+Single resolver in `src/core/config.ts`. Five precedence layers
+(lowest → highest):
+
+1. compiled-in `DEFAULTS`
+2. `~/.code-search/config.yaml` (user-global)
+3. `<workspaceDir>/.code-search.yaml` (per-workspace overlay)
+4. `CODE_SEARCH_*` env vars
+5. CLI flags via `--<dotted-path>` (e.g. `--embedding.provider codestral`)
 
-Everything is `CODE_SEARCH_*` env vars, single source of truth in
-`src/core/env-config.ts`:
+Run `code-search --print-config` or `code-search-mcp --print-config`
+any time to see the resolved struct (secrets redacted).
 
-| Var | What it does |
+| Var (`CODE_SEARCH_*`) | Effect |
 |---|---|
-| `CODE_SEARCH_SERVER` | Remote URL (proxy mode) |
-| `CODE_SEARCH_API_KEY` | API key for proxy or serve modes |
-| `CODE_SEARCH_DATA_DIR` | Override `~/.anvil/knowledge-base` |
-| `CODE_SEARCH_REINDEX_INTERVAL` | `30m` / `1h` / `6h` / `0` |
-| `EMBEDDING_PROVIDER` | `auto` / `voyage` / `openai` / `ollama` / … |
-| `EMBEDDING_API_KEY` | Bridged to provider-specific var |
-| `RERANKER_PROVIDER` | `ollama` / `cohere` / `voyage` / `none` |
+| `SERVER` | Remote URL (proxy mode) |
+| `API_KEY` | API key for proxy or serve modes |
+| `DATA_DIR` | Override the index storage root |
+| `PORT` / `HOST` / `TRANSPORT` | HTTP server bind |
+| `AUTH_MODE` / `AUTH_API_KEYS` / `AUTH_JWT_SECRET` | Server auth |
+| `REINDEX_INTERVAL` | `30m` / `1h` / `6h` / `0` |
+| `EMBEDDING_PROVIDER` | `auto` / `codestral` / `voyage` / `openai` / `ollama` / `openai-compatible` / `custom` / `gemini-oauth` |
+| `EMBEDDING_MODEL` / `EMBEDDING_DIMENSIONS` | Model + dims |
+| `EMBEDDING_API_KEY` / `EMBEDDING_BASE_URL` | Credentials |
 | `OLLAMA_HOST` | Default `http://localhost:11434` |
+| `RERANKER_PROVIDER` | `ollama` (default) / `cohere` / `voyage` / `openai-compatible` / `custom` / `none` |
+| `RERANKER_MODEL` / `RERANKER_API_KEY` / `RERANKER_BASE_URL` | Reranker config |
+| `RETRIEVAL_MAX_CHUNKS` / `RETRIEVAL_MAX_TOKENS` | Retrieval tuning |
+| `AUTO_INDEX` | `false` to disable startup auto-index |
+| `DAEMON_DISABLED` / `DAEMON_AUTO_SPAWN` | Daemon backend control |
+| `LLM_MODE` / `LLM_PROVIDER` / `LLM_MODEL` / `LLM_API_KEY` / `LLM_BASE_URL` | Profiling + service-mesh LLM. Pre-promoted to `ANVIL_LLM_*` so the agent-core integration is silent. |
+| `INDEXING_DEBOUNCE_MS` / `CHUNKING_MAX_TOKENS` | Indexer knobs |
+| `METRICS_ENABLED` / `STRUCTURED_LOGS` / `TELEMETRY_RECORD_QUERIES` | Observability |
+
+### Per-workspace config
+
+Drop a `.code-search.yaml` at the root of your workspace:
+
+```yaml
+embedding:
+  provider: codestral
+  model: codestral-embed-2505
+  dimensions: 1024
+reranker:
+  provider: ollama
+  model: qwen2.5-coder:7b           # the default that actually answers
+retrieval:
+  max_chunks: 12
+  hybrid_weights: { vector: 0.55, bm25: 0.3, graph: 0.15 }
+indexing:
+  auto_index: true
+  debounce_ms: 500
+```
+
+CLI flags override env, env overrides this file, this file overrides
+the user-global `~/.code-search/config.yaml`, that overrides
+`DEFAULTS`.
 
 ---
 
@@ -262,11 +379,49 @@ logs a warning instead of pretending it's fine.
 
 ---
 
+## End-to-end smoke test (T1–T8)
+
+The eight-case smoke that pins the standalone product contract lives
+at `scripts/smoke/run-smoke.sh`. It spins up a tiny `pet-shop` git
+fixture under `/tmp/cs-smoke/`, then runs:
+
+| # | Verifies |
+|---|---|
+| **T1** | `--print-config` env + CLI flag layering, redaction |
+| **T2** | `code-search index` produces a real Ollama-backed index; no stale legacy-env warning |
+| **T3** | `code-search query --mode vector` and `--mode bm25` both return the right file |
+| **T4** | `code-search status` reports provider + chunk count |
+| **T5** | Issue-#6 reproduction: `CODE_SEARCH_EMBEDDING_PROVIDER=openai` actually constructs the OpenAI embedder (skipped when `OPENAI_API_KEY` is unset) |
+| **T6** | Vector-space mismatch guard hard-errors on cross-provider query |
+| **T7** | Daemon round-trip: spawn, UDS socket, query through RPC, file-watcher debounce-reindex |
+| **T8** | HTTP serve: `/health` `/ready` `/version` `/metrics` `/admin/api/status` all 200; Prom text format |
+
+Run it locally:
+
+```sh
+npm -w @esankhan3/code-search-mcp run build      # required first
+npm -w @esankhan3/code-search-mcp run smoke      # T1–T8
+```
+
+Requirements: Ollama on `$OLLAMA_HOST` (default `localhost:11434`)
+with `bge-m3` pulled. Pass `OPENAI_API_KEY=...` to exercise the T5
+provider-switch path against the real OpenAI endpoint.
+
+The smoke runs **outside CI** — it requires Ollama + real models —
+so it's a hand-rolled local check, not a release gate. The release
+workflow does enforce a static guard that every bin entry
+(`code-search-mcp` / `code-search` / `code-search-daemon`) is in the
+built tarball before publishing.
+
+---
+
 ## Status
 
 Stable. The retrieval and graph layers move with
 `@anvil/knowledge-core`; the MCP surface is locked. New tools
-land additively.
+land additively. P0–P8 of the standalone-product plan
+(`docs/CODE-SEARCH-MCP-STANDALONE-PLAN.md`) have landed; F1–F4
+follow-up fixes verified by `scripts/smoke/run-smoke.sh`.
 
 ---
 

package/dist/__tests__/config-resolver.test.js

@@ -0,0 +1,153 @@
+import { describe, it, beforeEach, afterEach } from "node:test";
+import assert from "node:assert/strict";
+import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs";
+import { join } from "node:path";
+import { tmpdir, homedir } from "node:os";
+import {
+  resolveCodeSearchConfig,
+  parseCliFlags,
+  toKnowledgeConfig,
+  redactSecrets,
+  DEFAULTS
+} from "../core/config.js";
+const ENV_KEYS = [
+  "CODE_SEARCH_EMBEDDING_PROVIDER",
+  "CODE_SEARCH_EMBEDDING_MODEL",
+  "CODE_SEARCH_EMBEDDING_DIMENSIONS",
+  "CODE_SEARCH_EMBEDDING_API_KEY",
+  "CODE_SEARCH_RERANKER_PROVIDER",
+  "CODE_SEARCH_PORT",
+  "CODE_SEARCH_AUTH_MODE",
+  "CODE_SEARCH_AUTH_API_KEYS",
+  "CODE_SEARCH_DATA_DIR",
+  "CODE_SEARCH_REINDEX_INTERVAL"
+];
+let tmp = "";
+const STASH = {};
+beforeEach(() => {
+  for (const k of ENV_KEYS) {
+    STASH[k] = process.env[k];
+    delete process.env[k];
+  }
+  tmp = mkdtempSync(join(tmpdir(), "cs-cfg-"));
+});
+afterEach(() => {
+  rmSync(tmp, { recursive: true, force: true });
+  for (const k of ENV_KEYS) {
+    if (STASH[k] === void 0) delete process.env[k];
+    else process.env[k] = STASH[k];
+  }
+});
+describe("resolveCodeSearchConfig precedence (P3)", () => {
+  it("returns DEFAULTS when no file / env / CLI is present", () => {
+    const cfg = resolveCodeSearchConfig({ configPath: join(tmp, "missing.yaml") });
+    assert.equal(cfg.embedding.provider, "auto");
+    assert.equal(cfg.server.port, 3100);
+    assert.equal(cfg.indexing.autoIndex, true);
+  });
+  it("env overrides defaults", () => {
+    process.env.CODE_SEARCH_EMBEDDING_PROVIDER = "codestral";
+    process.env.CODE_SEARCH_EMBEDDING_MODEL = "codestral-embed-2505";
+    process.env.CODE_SEARCH_EMBEDDING_API_KEY = "sk-abc";
+    process.env.CODE_SEARCH_PORT = "4242";
+    const cfg = resolveCodeSearchConfig({ configPath: join(tmp, "missing.yaml") });
+    assert.equal(cfg.embedding.provider, "codestral");
+    assert.equal(cfg.embedding.model, "codestral-embed-2505");
+    assert.equal(cfg.embedding.apiKey, "sk-abc");
+    assert.equal(cfg.server.port, 4242);
+  });
+  it("YAML config file overrides defaults", () => {
+    const configPath = join(tmp, "config.yaml");
+    writeFileSync(
+      configPath,
+      `server:
+  port: 5500
+embedding:
+  provider: voyage
+  model: voyage-code-3
+`,
+      "utf-8"
+    );
+    const cfg = resolveCodeSearchConfig({ configPath });
+    assert.equal(cfg.server.port, 5500);
+    assert.equal(cfg.embedding.provider, "voyage");
+    assert.equal(cfg.embedding.model, "voyage-code-3");
+  });
+  it("CLI flags override env vars", () => {
+    process.env.CODE_SEARCH_EMBEDDING_PROVIDER = "voyage";
+    const { patch } = parseCliFlags(["--embedding.provider", "codestral", "--server.port", "6000"]);
+    const cfg = resolveCodeSearchConfig({ configPath: join(tmp, "missing.yaml"), cli: patch });
+    assert.equal(cfg.embedding.provider, "codestral");
+    assert.equal(cfg.server.port, 6e3);
+  });
+  it("per-workspace .code-search.yaml overrides ~/.code-search/config.yaml", () => {
+    const globalPath = join(tmp, "global.yaml");
+    writeFileSync(globalPath, `embedding:
+  provider: voyage
+`, "utf-8");
+    const workspace = join(tmp, "workspace");
+    mkdirSync(workspace, { recursive: true });
+    writeFileSync(join(workspace, ".code-search.yaml"), `embedding:
+  provider: openai
+`, "utf-8");
+    const cfg = resolveCodeSearchConfig({ configPath: globalPath, workspaceDir: workspace });
+    assert.equal(cfg.embedding.provider, "openai");
+  });
+});
+describe("toKnowledgeConfig (P3 adapter)", () => {
+  it("maps unified config to KnowledgeConfig shape", () => {
+    const cfg = resolveCodeSearchConfig({ configPath: join(tmp, "missing.yaml") });
+    const k = toKnowledgeConfig(cfg);
+    assert.equal(k.embedding.provider, cfg.embedding.provider);
+    assert.equal(k.retrieval.maxChunks, cfg.retrieval.maxChunks);
+    assert.deepEqual(k.chunking, cfg.indexing.chunking);
+    const r = typeof k.retrieval.reranker === "string" ? { provider: k.retrieval.reranker } : k.retrieval.reranker;
+    assert.equal(r.provider, cfg.reranker.provider);
+  });
+});
+describe("redactSecrets (P3 --print-config)", () => {
+  it("redacts apiKey/jwtSecret/token fields, preserves shape", () => {
+    const input = {
+      embedding: { apiKey: "sk-super-secret", model: "x" },
+      auth: { apiKeys: ["key-1", "key-2"], jwtSecret: "jwt-super-secret" },
+      github: { token: "gh-token" },
+      server: { port: 3100 }
+    };
+    const out = redactSecrets(input);
+    assert.match(out.embedding.apiKey, /redacted/);
+    assert.equal(out.embedding.model, "x");
+    assert.match(out.auth.apiKeys[0], /redacted/);
+    assert.match(out.auth.jwtSecret, /redacted/);
+    assert.match(out.github.token, /redacted/);
+    assert.equal(out.server.port, 3100);
+  });
+  it("keeps DEFAULTS shape redaction-safe (no secrets exposed)", () => {
+    const redacted = redactSecrets(DEFAULTS);
+    const json = JSON.stringify(redacted);
+    assert.equal(/sk-[a-zA-Z0-9]+/.test(json), false);
+  });
+});
+describe("parseCliFlags", () => {
+  it("handles dotted paths and value coercion", () => {
+    const { patch } = parseCliFlags([
+      "--embedding.provider",
+      "voyage",
+      "--retrieval.max-chunks",
+      "20",
+      "--telemetry.metrics-enabled",
+      "true"
+    ]);
+    assert.equal(patch.embedding.provider, "voyage");
+    assert.equal(patch.retrieval.maxChunks, 20);
+    assert.equal(patch.telemetry.metricsEnabled, true);
+  });
+  it("treats reserved subcommand flags as rest", () => {
+    const { rest } = parseCliFlags(["--local", "/some/path", "--embedding.provider", "codestral"]);
+    assert.ok(rest.includes("--local"));
+  });
+  it("--no-x.y sets boolean false", () => {
+    const { patch } = parseCliFlags(["--no-indexing.auto-index"]);
+    assert.equal(patch.indexing.autoIndex, false);
+  });
+});
+void homedir;

package/dist/__tests__/daemon-rpc.test.js

@@ -0,0 +1,100 @@
+import { describe, it, before, after } from "node:test";
+import assert from "node:assert/strict";
+import { mkdtempSync, rmSync } from "node:fs";
+import { join } from "node:path";
+import { tmpdir } from "node:os";
+import { RpcServer } from "../daemon/rpc-server.js";
+import { DaemonBackend } from "../backends/daemon-client.js";
+const tmp = mkdtempSync(join(tmpdir(), "cs-rpc-"));
+const socketPath = join(tmp, "test.sock");
+let server;
+before(async () => {
+  server = new RpcServer({
+    socketPath,
+    handlers: {
+      "search.code": async (p) => ({ query: p.query, totalTokens: 0, chunks: [] }),
+      "index.status": async () => ({
+        totalChunks: 0,
+        repos: [],
+        embeddingProvider: "stub",
+        lastIndexedAt: null,
+        watching: true,
+        queueDepth: 0,
+        uptimeSec: 1
+      }),
+      "index.force": async () => ({
+        totalChunks: 0,
+        repos: [],
+        embeddingProvider: "stub",
+        lastIndexedAt: null,
+        watching: true,
+        queueDepth: 0,
+        uptimeSec: 1
+      }),
+      "index.invalidate": async () => ({ ok: true }),
+      "health": () => ({ ok: true, uptime: 1 })
+    }
+  });
+  await server.start();
+});
+after(async () => {
+  await server.stop();
+  rmSync(tmp, { recursive: true, force: true });
+});
+describe("daemon RPC roundtrip", { skip: process.platform === "win32" }, () => {
+  it("ping returns true for a live daemon", async () => {
+    const client = new DaemonBackend({
+      project: "test",
+      workspaceDir: null,
+      knowledge: {},
+      preferDaemon: true,
+      socketPath
+    });
+    assert.equal(await client.ping(), true);
+    await client.close();
+  });
+  it("search returns the typed payload", async () => {
+    const client = new DaemonBackend({
+      project: "test",
+      workspaceDir: null,
+      knowledge: {},
+      preferDaemon: true,
+      socketPath
+    });
+    const r = await client.search("hello", { mode: "hybrid", maxResults: 5 });
+    assert.equal(r.query, "hello");
+    assert.equal(Array.isArray(r.chunks), true);
+    await client.close();
+  });
+  it("status / invalidate / forceIndex roundtrip", async () => {
+    const client = new DaemonBackend({
+      project: "test",
+      workspaceDir: null,
+      knowledge: {},
+      preferDaemon: true,
+      socketPath
+    });
+    const s = await client.status();
+    assert.equal(s.embeddingProvider, "stub");
+    assert.equal(s.watching, true);
+    await client.invalidate(["/some/path.ts"]);
+    const after2 = await client.forceIndex({ force: true });
+    assert.equal(after2.embeddingProvider, "stub");
+    await client.close();
+  });
+  it("unknown method returns RPC error", async () => {
+    const client = new DaemonBackend({
+      project: "test",
+      workspaceDir: null,
+      knowledge: {},
+      preferDaemon: true,
+      socketPath
+    });
+    await assert.rejects(
+      // @ts-expect-error — calling a private method on purpose
+      () => client.rpc("notARealMethod", {}, 500),
+      /Method not found/
+    );
+    await client.close();
+  });
+});

package/dist/__tests__/observability.test.js

@@ -0,0 +1,69 @@
+import { describe, it } from "node:test";
+import assert from "node:assert/strict";
+import { Counter, Gauge, Histogram, registry, metrics } from "../observability/metrics.js";
+import { Logger } from "../observability/logger.js";
+describe("Prometheus collectors (P7)", () => {
+  it("Counter renders with labels", () => {
+    const c = new Counter("test_counter", "test counter");
+    c.inc({ kind: "a" });
+    c.inc({ kind: "a" });
+    c.inc({ kind: "b" });
+    const out = c.render();
+    assert.match(out, /# HELP test_counter test counter/);
+    assert.match(out, /# TYPE test_counter counter/);
+    assert.match(out, /test_counter\{kind="a"\} 2/);
+    assert.match(out, /test_counter\{kind="b"\} 1/);
+  });
+  it("Gauge updates last-write per label set", () => {
+    const g = new Gauge("test_gauge", "test gauge");
+    g.set(5, { repo: "a" });
+    g.set(7, { repo: "a" });
+    assert.match(g.render(), /test_gauge\{repo="a"\} 7/);
+  });
+  it("Histogram tracks buckets, sum, count", () => {
+    const h = new Histogram("test_hist", "test", [0.1, 0.5, 1]);
+    h.observe(0.05);
+    h.observe(0.3);
+    h.observe(0.9);
+    const out = h.render();
+    assert.match(out, /test_hist_count\{?\}? 3/);
+    assert.match(out, /test_hist_sum\{?\}? 1.25/);
+    assert.match(out, /test_hist_bucket\{le="0.1"\} 1/);
+    assert.match(out, /test_hist_bucket\{le="\+Inf"\} 3/);
+  });
+  it("registry renders the prelude for all pre-declared metrics", () => {
+    metrics.queriesTotal.inc({ mode: "hybrid" });
+    const out = registry.render();
+    assert.match(out, /code_search_queries_total/);
+    assert.match(out, /code_search_query_duration_seconds/);
+  });
+});
+describe("Logger (P7)", () => {
+  it("emits structured JSON when enabled", () => {
+    const lines = [];
+    const log = new Logger({ structured: true, write: (l) => lines.push(l) });
+    log.info("hello", { query: "auth" });
+    assert.equal(lines.length, 1);
+    const parsed = JSON.parse(lines[0]);
+    assert.equal(parsed.msg, "hello");
+    assert.equal(parsed.query, "auth");
+    assert.equal(parsed.level, "info");
+    assert.ok(parsed.ts);
+  });
+  it("respects level threshold", () => {
+    const lines = [];
+    const log = new Logger({ structured: true, level: "warn", write: (l) => lines.push(l) });
+    log.info("skipped");
+    log.warn("kept");
+    assert.equal(lines.length, 1);
+    assert.match(lines[0], /"kept"/);
+  });
+  it("falls back to text mode when structured=false", () => {
+    const lines = [];
+    const log = new Logger({ structured: false, write: (l) => lines.push(l) });
+    log.error("boom");
+    assert.equal(lines.length, 1);
+    assert.match(lines[0], /ERROR/);
+    assert.match(lines[0], /boom/);
+  });
+});