@fodx/codelens 1.0.0

package/README.md

@@ -0,0 +1,212 @@
+# CodeLens
+
+![CodeLens](docs/codelens-preview.png)
+
+A local, **branch-aware code search & relation graph** — a lens into how
+the code in a repo connects. Indexes the current branch into SQLite (FTS5
+lexical + tree-sitter symbols + source-graph edges: imports / tests / calls /
+defines / belongs_to) and returns compact, ranked, re-queryable handles so you
+can find relevant files/symbols and walk their relationships without flooding
+your context with raw grep/read output.
+
+No chat LLM is involved — retrieval is deterministic. Surfaced as an **MCP
+server** and a **CLI**; works with any MCP-compatible client (Claude Code,
+Cursor, Gemini CLI, opencode, Codex CLI) or directly from the terminal.
+
+- **Branch isolation** — each branch/worktree has its own index; results never
+  leak across branches.
+- **Relations, not just text** — graph edges (imports / tests / callers / …)
+  ranked alongside FTS5 + symbol-name matches.
+- **Fresh** — auto-refreshes changed files before query; `cl_expand` always
+  reads from disk.
+- **Compact** — returns ranked handles; expand only what you need.
+- **Durable** — saved contexts live in a separate DB and survive index rebuilds.
+- **Self-cleaning** — automatic TTL prunes inactive indexes.
+
+## Install (one command)
+
+The installer builds the tool, puts `codelens` on your PATH, and wires the
+MCP server into your agents/IDEs automatically (Claude Code, Cursor, Gemini CLI,
+opencode, Codex CLI). Requires Node.js ≥ 22.5.
+
+**macOS / Linux:**
+```bash
+curl -fsSL https://raw.githubusercontent.com/ex-git/codeLens/main/install.sh | sh
+```
+
+**Windows (PowerShell):**
+```powershell
+irm https://raw.githubusercontent.com/ex-git/codeLens/main/install.ps1 | iex
+```
+
+Then open a new terminal (so PATH picks up) and the agents you use are configured.
+The installer wires the MCP server into every detected agent; to choose
+explicitly or re-run later:
+
+```bash
+codelens install --target all --yes        # wire all agents
+codelens install --target claude,cursor    # wire specific agents
+codelens install --target auto --location=local   # project-local config
+codelens --print-config codex              # print a snippet, no writes
+codelens uninstall                         # remove from all agents
+```
+
+**Upgrade:**
+```bash
+codelens upgrade --check   # is an update available?
+codelens upgrade           # git pull + rebuild
+```
+
+**Uninstall everything:**
+```bash
+curl -fsSL https://raw.githubusercontent.com/ex-git/codeLens/main/install.sh | sh -s -- --uninstall
+```
+
+> Note on clients/LLMs: MCP clients (Claude Code, Cursor, …) bring their own
+> model — CodeLens has no LLM and configures none. The installer only wires the
+> MCP server + routing instructions into each host's config.
+
+## Supported agents / IDEs
+
+The installer (`codelens install --target <id>`) writes the MCP server config
+and routing instructions into each host's real config file. `--location=global`
+(user-wide) is the default; `--location=local` writes project-local config
+instead. All writes are idempotent and removable with `codelens uninstall`.
+
+| Host | target id | Config file (global → local) | Entry shape |
+|---|---|---|---|
+| Claude Code | `claude` | `~/.claude.json` → `.mcp.json` | `mcpServers.codelens = { command, args: [] }` |
+| Cursor | `cursor` | `~/.cursor/mcp.json` → `.cursor/mcp.json` | `mcpServers.codelens = { command, args: [] }` |
+| Gemini CLI | `gemini` | `~/.gemini/settings.json` → `.gemini/settings.json` | `mcpServers.codelens = { command, args: [] }` |
+| opencode | `opencode` | `~/.config/opencode/opencode.json` → `./opencode.json` | `mcp.codelens = { type: "local", command: [cmd], enabled: true }` |
+| Codex CLI | `codex` | `~/.codex/config.toml` → `.codex/config.toml` | TOML `[mcp_servers.codelens]` block (`command`, `args = []`) |
+| Pi Coding Agent | `pi` | `pi install npm:@fodx/codelens` (loads `adapters/pi/codelens.extension.ts`) | Pi extension that bridges the MCP server via `pi.registerTool` |
+
+`command` is the absolute path to the installed `codelens` launcher (written by
+the installer); for a manual snippet use `npx -y @fodx/codelens`.
+
+**Routing instructions** are also written so the host prefers codelens tools for
+discovery over raw grep/read:
+
+| Host | Instructions file (global → local) |
+|---|---|
+| Claude Code | `~/.claude/CLAUDE.md` → `./CLAUDE.md` (+ slash commands in `~/.claude/commands/codelens-*.md`) |
+| Cursor | `~/.cursor/rules/codelens.mdc` → `.cursor/rules/codelens.mdc` (dedicated, `alwaysApply`) |
+| Gemini CLI | `~/.gemini/GEMINI.md` → `./GEMINI.md` |
+| opencode | `~/.config/opencode/AGENTS.md` → `./AGENTS.md` |
+| Codex CLI | `~/.codex/AGENTS.md` → `./AGENTS.md` |
+
+**Print a snippet without writing anything:**
+
+```bash
+codelens --print-config claude    # JSON mcpServers snippet + target path
+codelens --print-config codex     # TOML [mcp_servers.codelens] block
+codelens --print-config pi        # Pi extension manifest pointer
+```
+
+**Pi Coding Agent** — install as a Pi package (the npm tarball ships the
+extension under `adapters/pi/` and is tagged `pi-package`, so it appears in the
+[Pi package gallery](https://pi.dev/packages) once published):
+
+```bash
+pi install npm:@fodx/codelens      # user-wide
+pi install -l npm:@fodx/codelens   # project-local (.pi/settings.json)
+pi -e npm:@fodx/codelens           # try it for one run without installing
+```
+
+> **npm discoverability:** `package.json` is tagged `pi-package` (required for
+> the Pi gallery) plus descriptive keywords (`mcp-server`,
+> `modelcontextprotocol`, `claude-code`, `cursor`, `gemini-cli`, `opencode`,
+> `codex`) so the package is findable via npm search. Only `pi-package` is
+> confirmed to trigger a host gallery listing; the others are for search
+> discoverability and do not imply auto-listing in those hosts' marketplaces.
+
+## Install (MCP) — manual alternative
+
+Add to your MCP client config (Claude Code, Cursor, OpenCode, Gemini CLI,
+Codex CLI):
+
+```json
+{
+  "mcpServers": {
+    "codelens": {
+      "command": "npx",
+      "args": ["-y", "@fodx/codelens"]
+    }
+  }
+}
+```
+
+Or run locally from source:
+
+```bash
+npm install --legacy-peer-deps
+npm run build
+node build/src/server.js
+```
+
+## Quickstart
+
+1. In your repo, the agent calls `cl_current` (builds index if missing) or
+   `cl_refresh` explicitly.
+2. `cl_search(query: "session validation")` → ranked handles.
+3. `cl_related(path: "src/auth/session.ts", types: ["tests"])` → tests.
+4. `cl_expand(path: "src/auth/session.ts", startLine: 12, endLine: 58)` → exact code.
+
+See [`docs/agent-guide.md`](docs/agent-guide.md) for a full walkthrough,
+[`docs/tools.md`](docs/tools.md) for the tool reference, and
+[`docs/routing.md`](docs/routing.md) for agent routing instructions.
+
+
+## Native adapters
+
+See `adapters/` for host-specific hook skeletons that nudge agents toward
+codelens tools instead of raw grep/read.
+
+## CLI (non-MCP usage)
+
+The `codelens` binary also works directly from the terminal:
+
+```bash
+codelens doctor          # health check
+codelens index            # build/update the current branch index
+codelens search "session validation"   # ranked search
+codelens related src/auth/session.ts    # graph neighbors
+codelens stats           # index counts
+codelens current         # repo/branch status
+```
+
+## Development
+
+```bash
+npm run typecheck   # tsc --noEmit
+npm run lint        # eslint
+npm test            # vitest run
+npm run build       # tsc + copy schema assets
+npm run benchmark   # performance gate (search <50ms, cold <3s)
+```
+
+## Docs
+
+- [How CodeLens works](docs/how-it-works.md) — architecture, index layers, branch isolation, freshness, ranking, TTL, saved contexts, usage.
+- [Usage metrics & how "saved" is calculated](docs/usage-metrics.md) — the formula, assumptions, and limits.
+- [CodeLens vs CodeGraph](docs/vs-codegraph.md) — feature comparison.
+
+## Limitations
+
+- **No semantic/vector search** — the ONNX subsystem was removed; ranking is
+  FTS5 + symbol-name + graph proximity. (A real tokenizer can be re-added later.)
+- **Lazy on-demand indexing for very large repos** is future work — `cl_refresh`
+  is eager. The recommended pattern for large repos is one `cl_refresh` then
+  incremental + the file watcher thereafter (cold index ~1.6s for 2000 files).
+- **Routing hooks are advisory** (soft nudges, not hard blocks) per the no-
+  throttling design decision — raw reads remain allowed for editing/verification.
+- **npm package** — published automatically from `v*` git tags via the
+  `publish` workflow; `npx -y @fodx/codelens` works once a tag is pushed. See
+  `.github/workflows/publish.yml`.
+- **Windows not tested** — path normalization handles backslashes, but `fs.watch`
+  recursive behavior and native builds are unverified on Windows.
+
+## License
+
+MIT

package/adapters/pi/codelens.extension.ts

@@ -0,0 +1,280 @@
+/**
+ * CodeLens — Pi Coding Agent extension.
+ *
+ * Bridges the CodeLens MCP server (stdio) into Pi by spawning it as a
+ * long-lived child, performing the MCP handshake (initialize → tools/list),
+ * and registering each tool with `pi.registerTool`. Each tool's `execute()`
+ * forwards to `tools/call`. No external deps — pure child_process + JSON-RPC
+ * over stdio line frames.
+ *
+ * Install: copy/symlink this file to ~/.pi/agent/extensions/codelens.ts
+ * (Pi auto-discovers global extensions there; `/reload` hot-reloads it).
+ *
+ * The server script path resolves from (in order): CODELENS_SERVER env,
+ * the launcher on PATH (~/.local/bin/codelens → handled below by exec'ing
+ * node directly), or the build next to this repo. We exec `node <server.js>`
+ * to avoid depending on the launcher being on PATH inside Pi's spawn env.
+ */
+import { spawn, type ChildProcess } from "node:child_process";
+import { existsSync, readFileSync } from "node:fs";
+import { join, dirname } from "node:path";
+import { fileURLToPath } from "node:url";
+import { homedir } from "node:os";
+
+const SERVER_PATH_FILE = join(homedir(), ".codelens", "server-path");
+
+/** A resolved server spec: either `node <server.js>` or a direct launcher. */
+interface ServerSpec { command: string; args: string[] }
+
+/** Wrap a .js path with the current node binary; otherwise treat as a launcher. */
+function specFor(path: string): ServerSpec {
+  return path.endsWith(".js") || path.endsWith(".mjs")
+    ? { command: process.execPath, args: [path] }
+    : { command: path, args: [] };
+}
+
+function resolveServer(): ServerSpec | null {
+  // 1. Installer-written absolute path (~/.codelens/server-path).
+  try {
+    if (existsSync(SERVER_PATH_FILE)) {
+      const p = readFileSync(SERVER_PATH_FILE, "utf-8").trim();
+      if (p && existsSync(p)) return specFor(p);
+    }
+  } catch { /* ignore */ }
+  // 2. Env override.
+  if (process.env.CODELENS_SERVER) {
+    const p = process.env.CODELENS_SERVER;
+    if (existsSync(p)) return specFor(p);
+  }
+  // 3. install.sh default location.
+  const appBuild = join(homedir(), ".codelens", "app", "build", "src", "server.js");
+  if (existsSync(appBuild)) return specFor(appBuild);
+  // 4. Dev: extension lives at <repo>/adapters/pi/ → <repo>/build/src/server.js.
+  try {
+    const here = dirname(fileURLToPath(import.meta.url));
+    const dev = join(here, "..", "..", "..", "build", "src", "server.js");
+    if (existsSync(dev)) return specFor(dev);
+  } catch { /* ignore */ }
+  return null;
+}
+
+// ── Minimal MCP stdio JSON-RPC client ────────────────────────
+
+interface JsonRpcResponse { id?: number; result?: unknown; error?: { message: string }; }
+interface ToolDef { name: string; description?: string; inputSchema?: Record<string, unknown>; }
+
+class McpStdioClient {
+  private proc: ChildProcess | null = null;
+  private nextId = 1;
+  private pending = new Map<number, (r: JsonRpcResponse) => void>();
+  private buffer = "";
+
+  start(spec: ServerSpec, cwd?: string): void {
+    this.proc = spawn(spec.command, spec.args, {
+      stdio: ["pipe", "pipe", "inherit"],
+      env: process.env,
+      cwd,
+    });
+    this.proc.stdout!.setEncoding("utf-8");
+    this.proc.stdout!.on("data", (chunk: string) => {
+      this.buffer += chunk;
+      let nl: number;
+      while ((nl = this.buffer.indexOf("\n")) >= 0) {
+        const line = this.buffer.slice(0, nl).trim();
+        this.buffer = this.buffer.slice(nl + 1);
+        if (!line) continue;
+        try {
+          const msg = JSON.parse(line) as JsonRpcResponse;
+          if (msg.id !== undefined && this.pending.has(msg.id)) {
+            this.pending.get(msg.id)!(msg);
+            this.pending.delete(msg.id);
+          }
+        } catch { /* not a JSON-RPC response line */ }
+      }
+    });
+  }
+
+  private request(method: string, params: unknown, timeoutMs = 30000): Promise<unknown> {
+    const id = this.nextId++;
+    const frame = JSON.stringify({ jsonrpc: "2.0", id, method, params }) + "\n";
+    return new Promise((resolve, reject) => {
+      const timer = setTimeout(() => reject(new Error(`${method} timed out`)), timeoutMs);
+      this.pending.set(id, (msg) => { clearTimeout(timer); if (msg.error) reject(new Error(msg.error.message)); else resolve(msg.result); });
+      this.proc!.stdin!.write(frame);
+    });
+  }
+
+  async initialize(): Promise<void> {
+    await this.request("initialize", {
+      protocolVersion: "2025-06-18",
+      capabilities: {},
+      clientInfo: { name: "pi-codelens-bridge", version: "1.0.0" },
+    });
+    this.proc!.stdin!.write(JSON.stringify({ jsonrpc: "2.0", method: "notifications/initialized" }) + "\n");
+  }
+
+  async listTools(): Promise<ToolDef[]> {
+    const r = (await this.request("tools/list", {})) as { tools?: ToolDef[] };
+    return r.tools ?? [];
+  }
+
+  async callTool(name: string, args: Record<string, unknown>): Promise<{ content: Array<{ type: string; text: string }>; isError?: boolean }> {
+    const r = (await this.request("tools/call", { name, arguments: args }, 120000)) as { content?: Array<{ type: string; text: string }>; isError?: boolean };
+    return { content: r.content ?? [], isError: r.isError };
+  }
+
+  shutdown(): void {
+    try { this.proc?.stdin?.end(); } catch { /* ignore */ }
+    try { this.proc?.kill("SIGTERM"); } catch { /* ignore */ }
+    this.proc = null;
+  }
+}
+
+// ── Routing instructions injected into Pi's system prompt ─────
+// Tells the LLM to prefer the cl_* tools over raw grep/find/read for code
+// discovery. Only injected once the bridge has registered the tools.
+const ROUTING = `
+## CodeLens (code context index) — routing
+
+A local branch-scoped code index is available via these tools. For codebase
+DISCOVERY, prefer them over grep/find/read:
+- cl_current  — repo/branch/index status (call if unsure whether index is ready)
+- cl_search   — ranked semantic+lexical search over the current branch index;
+                 returns compact handles (path + line range + score). Pass
+                 contentType:"code" for source only. Auto-builds the index on
+                 first use, so you can call it directly.
+- cl_related  — graph neighbors of a file/symbol (tests, importers, callers)
+- cl_expand   — exact current file content by path/range (reads disk, never stale)
+
+Rules:
+- Use cl_search / cl_related BEFORE grep, find, or reading multiple files to
+  locate code. It keeps your context lean and is branch-aware.
+- Use raw read ONLY for the exact file you are about to edit or verify, not for
+  discovery.
+- After \`git checkout\`, results auto-scope to the new branch; call cl_current
+  if you want to confirm index status.
+- If cl_search returns nothing relevant, fall back to grep/read as usual.
+
+You do NOT need the user's permission to use these tools — they are the
+preferred discovery path.`;
+
+// ── Extension entry ──────────────────────────────────────────
+
+export default function (pi: import("@earendil-works/pi-coding-agent").ExtensionAPI) {
+  const spec = resolveServer();
+  if (!spec) {
+    process.stderr.write(`[codelens] server not found. Run \`codelens install\` (writes ~/.codelens/server-path), or set CODELENS_SERVER=<path>/build/src/server.js, or run install.sh.\n`);
+    return;
+  }
+
+  // Mutable bridge holder. Tools' execute() closures reference `client` and
+  // `bridged` directly, so reassigning them on a project switch re-points every
+  // tool at the new server child (no re-registration needed).
+  let client: McpStdioClient | null = null;
+  let bridged = false;
+  let currentCwd = process.cwd();
+  let toolsRegistered = false;
+
+  async function boot(cwd: string): Promise<void> {
+    try { client?.shutdown(); } catch { /* ignore */ }
+    const c = new McpStdioClient();
+    c.start(spec, cwd); // spawn the server with cwd = the current project
+    await c.initialize();
+    const tools = await c.listTools();
+    if (!toolsRegistered) {
+      for (const tool of tools) {
+        pi.registerTool({
+          name: tool.name,
+          label: tool.name,
+          description: tool.description ?? "",
+          parameters: tool.inputSchema ?? { type: "object", properties: {} },
+          async execute(_id, params) {
+            if (!client) throw new Error("codelens bridge not ready");
+            const result = await client.callTool(tool.name, (params ?? {}) as Record<string, unknown>);
+            const text = (result.content ?? []).filter((x) => x?.type === "text").map((x) => x.text).join("\n");
+            if (result.isError) throw new Error(text || `${tool.name} returned an error`);
+            return { content: [{ type: "text", text }], details: {} };
+          },
+        });
+      }
+      toolsRegistered = true;
+    }
+    client = c;
+    currentCwd = cwd;
+    bridged = true;
+  }
+
+  boot(currentCwd).catch((err: unknown) => {
+    process.stderr.write(`[codelens] MCP bridge failed: ${err instanceof Error ? err.message : String(err)}\n`);
+    bridged = false;
+  });
+
+  // Each turn: if Pi's cwd changed (project switch), respawn the server into
+  // the new project so cl_search queries the right repo. Then inject routing.
+  pi.on("before_agent_start", async (event: { systemPrompt?: string; systemPromptOptions?: { cwd?: string } }) => {
+    const cwd = event.systemPromptOptions?.cwd ?? process.cwd();
+    if (bridged && cwd && cwd !== currentCwd) {
+      try { await boot(cwd); } catch { /* keep old bridge on respawn failure */ }
+    }
+    if (!bridged) return {};
+    const base = event.systemPrompt ?? "";
+    if (base.includes("CodeLens (code context index)")) return {};
+    return { systemPrompt: base + "\n" + ROUTING };
+  });
+
+  // ── Slash commands (discoverable via `/`) ────────────────────
+  const call = async (name: string, args: Record<string, unknown> = {}): Promise<string> => {
+    if (!client || !bridged) return "codelens bridge not ready yet — wait a moment or /reload.";
+    const r = await client.callTool(name, args);
+    return (r.content ?? []).filter((c) => c?.type === "text").map((c) => c.text).join("\n");
+  };
+  const fmtBytes = (n: number) => (n < 1024 ? `${n}B` : n < 1048576 ? `${(n/1024).toFixed(1)}KB` : `${(n/1048576).toFixed(2)}MB`);
+
+  pi.registerCommand("codelens-usage", {
+    description: "Show codelens tool usage stats (global, across repos)",
+    handler: async (_args, ctx) => {
+      const text = await call("cl_usage");
+      try {
+        const u = JSON.parse(text);
+        const t = u.totals ?? { calls: 0, bytes_served: 0, bytes_saved: 0 };
+        const lines: string[] = [];
+        lines.push(`cl_* usage (global) — calls: ${t.calls} | served: ${fmtBytes(t.bytes_served)} | saved(est): ${fmtBytes(t.bytes_saved)}`);
+        lines.push("Per tool:");
+        for (const row of (u.perTool ?? [])) lines.push(`  ${row.tool.padEnd(14)} calls=${row.calls}  served=${fmtBytes(row.bytes_served)}  saved=${fmtBytes(row.bytes_saved)}`);
+        if (!(u.perTool ?? []).length) lines.push("  (no calls recorded yet)");
+        lines.push("Per repo:");
+        for (const r of (u.perRepo ?? [])) lines.push(`  ${r.repo_id.slice(0,8)}  calls=${r.calls}  saved=${fmtBytes(r.bytes_saved)}`);
+        if (!(u.perRepo ?? []).length) lines.push("  (none)");
+        ctx.ui.notify(lines.join("\n"), "info");
+      } catch { ctx.ui.notify(text.slice(0, 800), "info"); }
+    },
+  });
+
+  pi.registerCommand("codelens-stats", {
+    description: "Show current index statistics (file/symbol/chunk/edge counts)",
+    handler: async (_args, ctx) => { ctx.ui.notify((await call("cl_stats")).slice(0, 800), "info"); },
+  });
+
+  pi.registerCommand("codelens-doctor", {
+    description: "Run a codelens health check",
+    handler: async (_args, ctx) => { ctx.ui.notify((await call("cl_doctor")).slice(0, 800), "info"); },
+  });
+
+  pi.registerCommand("codelens-search", {
+    description: "Search the current branch index: /codelens-search <query>",
+    handler: async (args, ctx) => {
+      const query = (args ?? "").trim();
+      if (!query) { ctx.ui.notify("Usage: /codelens-search <query>", "warn"); return; }
+      const text = await call("cl_search", { query, limit: 5 });
+      try {
+        const r = JSON.parse(text);
+        const lines = (r.results ?? []).map((h: { score: number; path: string; startLine: number; endLine: number; why?: string[] }) =>
+          `${h.score.toFixed(3)}  ${h.path}:${h.startLine}-${h.endLine}  [${(h.why ?? []).join(",")}]`);
+        ctx.ui.notify(lines.length ? lines.join("\n") : "no results", "info");
+      } catch { ctx.ui.notify(text.slice(0, 800), "info"); }
+    },
+  });
+
+  // Clean up the child on Pi shutdown.
+  pi.on("shutdown", () => { try { client?.shutdown(); } catch { /* ignore */ } });
+}

package/adapters/pi/extension.json

@@ -0,0 +1,10 @@
+{
+  "name": "codelens",
+  "version": "1.0.0",
+  "type": "mcp",
+  "mcp": {
+    "command": "npx",
+    "args": ["-y", "@fodx/codelens"]
+  },
+  "description": "Branch-aware code search & relation graph (MCP server + CLI). Routes agents to cl_search/cl_related/cl_expand instead of raw grep/read."
+}