@fodx/codelens 2.0.0 → 2.1.0

package/docs/agent-guide.md

@@ -11,7 +11,13 @@ End-to-end workflow for a coding agent using codelens.
 2. cl_refresh
    → { indexedFiles: 812, totalChunks: 2400, status: "ready" }
 
-3. cl_search(query: "session validation")
+3. cl_explore(query: "session validation flow")
+   → { query: "session validation flow", count: 3,
+       files: [{ path: "src/auth/session.ts", results: [{ handle: "chk_…", lines: "12-58", preview: "export function validateSession…" }] }],
+       related: [{ sourcePath: "src/auth/session.ts", path: "src/routes/login.ts", edgeType: "imported_by", hops: 1 }] }
+
+   # If you only need to locate a handle, use the leaner search tool:
+   cl_search(query: "session validation")
    → { query: "session validation", count: 1, results: [
        { handle: "chk_…", path: "src/auth/session.ts", lines: "12-58",
          score: 0.92, why: "fts,symbol,graph", preview: "export function validateSession(token: string): boolean" }
@@ -24,6 +30,10 @@ End-to-end workflow for a coding agent using codelens.
      ]
    # TS/JS also populate `calls`/`references` and resolve dynamic import().
 
+   # Before changing shared code, ask for the blast radius.
+   cl_impact(symbol: "validateSession", path: "src/auth/session.ts")
+   → { callers: [...], callees: [...], affectedFiles: [...], affectedTests: [...], confidenceNote: "…" }
+
    # Orientation (optional): outline a file or directory without reading it.
    cl_map(path: "src/auth")
    → files: [{ path: "src/auth/session.ts", symbols: [{ name: "validateSession", kind: "function", signature: "export function validateSession(token: string): boolean" }] }]
@@ -37,6 +47,14 @@ End-to-end workflow for a coding agent using codelens.
 8. cl_load(name: "auth-investigation")   # after compaction
 ```
 
+## Tool choice
+
+- Use `cl_explore` for broad questions: "how does X work?", "show the flow around Y", or surveying an unfamiliar area. It combines search, compact source previews, and relationships in one call.
+- Use `cl_search` when you only need ranked handles/locations.
+- Use `cl_related` to expand from a known file.
+- Use `cl_impact` before editing shared code to see callers/callees/affected tests.
+- Use `cl_expand` (or raw read) for exact current file content before editing.
+
 ## Branch switch
 
 ```
@@ -46,6 +64,10 @@ cl_refresh   → builds feature-b index; main's results no longer leak in
 ```
 
 
+## Freshness and stale results
+
+Query tools reconcile changed files before answering. If a response includes `freshness: "partial"`, `pendingFiles`, or per-result `stale:true`, read those stale files directly (`cl_expand` or raw read) before relying on indexed previews/edges.
+
 ## Saving context across compaction
 
 `cl_save` / `cl_load` persist named handle sets in a **separate** DB that

package/docs/how-it-works.md

@@ -0,0 +1,154 @@
+# How CodeLens works
+
+CodeLens is a **branch-aware local code knowledge retrieval engine**. It indexes
+the current repo/branch into SQLite and lets coding agents ask "what code is
+relevant to X?" via compact ranked handles — instead of grep/find/read flooding
+their context window. No chat LLM is used anywhere in the core path.
+
+## Architecture
+
+```
+Agent (Pi / Claude Code / Cursor / Gemini / opencode / Codex)
+   │  MCP stdio
+   ▼
+CodeLens MCP server  ──▶  CLI (codelens <subcommand>)
+   │
+   ├── Git scope detector        (repo / worktree / branch / HEAD / dirty)
+   ├── Index manager             (per-branch index identity)
+   ├── File scanner              (.gitignore-aware, binary/size filters)
+   ├── FTS5 indexer              (structure-aware chunks + content hash)
+   ├── Tree-sitter symbol extractor (11 grammars, text fallback)
+   ├── Source graph builder      (imports / defines / tests / belongs_to)
+   ├── Graph query (recursive CTE + bounded BFS)
+   ├── Freshness checker          (mtime/size fast → hash on suspicion)
+   ├── TTL pruner                (never-delete guards)
+   ├── Saved-context store       (separate DB, survives rebuilds)
+   └── Usage tracker             (global, actual-file-size savings)
+   │
+   ▼
+SQLite
+   ├── per-branch index DB  (~/.codelens/indexes/index-<repoId>.db)
+   ├── saved-contexts DB   (~/.codelens/contexts/contexts-<repoId>.db)
+   └── global usage DB     (~/.codelens/usage.db)
+```
+
+## The index layers
+
+1. **Files**: path, language, size, mtime, content hash.
+2. **FTS5 lexical**: structure-aware code chunks with line-based fallback and
+   Porter-stemming/BM25 ranking. Code files with extractable symbols are chunked
+   around outermost functions/classes/methods/types; oversized symbols and non-structural
+   gaps (imports, top-level statements, inter-symbol regions) are line-chunked.
+   Leading comment/decorator blocks are attached to the following symbol chunk.
+   Line fallback uses small overlap to reduce boundary loss. Chunks carry
+   `code` vs `prose`, `chunker`, `chunker_version`, and `symbol_id` when aligned
+   to a symbol. The match-only FTS text preserves the original chunk and appends
+   bounded, deduplicated identifier subtokens (for example `validateSession` →
+   `validate session`) so code-identifier queries can match without changing
+   snippets or `cl_expand` content.
+3. **Symbols** (tree-sitter): functions/classes/methods/types/exports/imports
+   with line ranges + signatures + exported flag. Parser-eligible files are
+   parsed once and the same tree-sitter tree is reused for symbols and edges;
+   structure-aware chunking consumes those extracted symbol ranges. 11 grammars
+   shipped; unknown languages fall back to text-only FTS.
+4. **Source graph**: edges `imports`, `defines`, `belongs_to`, `exports`,
+   `tests` (filename heuristics). Resolution handles TS ESM `.js`→`.ts`
+   substitution. Unresolved imports emit no edge (no wrong edges).
+5. *(No vector/semantic layer — removed; ranking is FTS + symbol + graph.)*
+
+## Branch isolation (the core idea)
+
+Every index is scoped to `repoRoot + worktreePath + branch + HEAD`:
+
+```
+index_id = sha256(repoRoot | worktreePath | branch | headSha)
+```
+
+Every table row carries `index_id`; every query filters by it. So `git checkout`
+activates/creates a different index automatically — results from `main` never
+leak into `feature-b`. `cl_current` reports which index is active.
+
+## Freshness (local files are source of truth)
+
+Before query tools answer (`cl_search`, `cl_explore`, `cl_related`, `cl_impact`, `cl_map`), `ensureFreshIndex` runs:
+1. Detect current git scope → activate/create the branch index.
+2. Scan files, diff `mtime`+`size` vs indexed rows (fast); hash only
+   changed/suspicious files.
+3. Reindex changed/new files in per-file transactions; drop deleted. Files with
+   chunks from an older/unknown `chunker_version` are also treated as changed, so
+   chunker improvements roll forward on the next refresh.
+4. Budget-bounded (default 500ms); if incomplete, surface
+   `freshness:"partial"` + `pendingFiles`. A chunker-version bump can require a
+   full budget-bounded re-chunk over several refresh cycles.
+
+`cl_expand` **always reads current disk** (never stale stored text). A file
+watcher (server mode) short-circuits the scan when nothing changed, with a 5s
+periodic full-scan backstop.
+
+**Edit behavior:** when the agent edits a file, the *next query tool call*
+auto-refreshes and reindexes it (lazy, not eager at edit time). If the refresh
+budget is exhausted, known-stale paths are surfaced with `stale:true` and
+`freshness:"partial"` so the agent can read those files directly.
+
+## Ranking (hybrid)
+
+`cl_search` fuses lexical, structural, graph, path, and code/prose signals with
+weights from `src/search/rank.ts`:
+
+```
+score = fts×0.34 + symbol×0.18 + graph×0.22 + code×0.08 + path×0.08 + exact×0.10
+```
+
+- `fts`: BM25 normalized. Queries are expanded with the same bounded identifier
+  splitter used at index time, then quoted through the FTS path; expansion is
+  capped to avoid broadening queries or hurting latency.
+- `symbol`: full signal when the chunk's `symbol_id` name partially matches a
+  query term; lower-strength file-level symbol match is kept as fallback.
+- `exact`: full signal when the chunk's `symbol_id` name exactly matches a query
+  term; lower-strength file-level exact match is kept as fallback.
+- `graph`: 1 if the file is graph-proximate to top lexical results.
+- `path`: query term appears in the file path/name.
+- `code`: modest boost for `code` chunks over `prose` (docs), so code discovery
+  isn't drowned out by markdown. Pass `contentType:"code"` to filter to source
+  only.
+
+`cl_related` runs a recursive-CTE BFS over the `edges` table (direction-aware,
+cycle-guarded, depth-capped at 3).
+
+## Concurrency & recovery
+
+- **WAL mode** + a single-writer queue + a cross-process advisory lease
+  (`index_locks`) so two agent processes on the same repo never corrupt rows;
+  readers see the prior committed snapshot.
+- **Corruption recovery:** `PRAGMA quick_check` on startup + on query error →
+  rebuild the core index. Saved contexts live in a **separate DB** and survive.
+- **Schema versioning:** version guard refuses a newer-than-code DB; migrations
+  are transactional with a pre-migration backup.
+
+## TTL
+
+Inactive indexes are pruned automatically (startup + periodic): inactive branch
+14d, detached 3d, worktree 48h. **Never** deletes the active index, pinned
+indexes, locked indexes, or recently-accessed ones. `cl_prune` (manual),
+`cl_drop` (explicit, refuses active/pinned).
+
+## Saved contexts
+
+`cl_save`/`cl_load` persist named handle sets + notes in a **separate**
+contexts DB (`~/.codelens/contexts/`), keyed by repo. They survive core-index
+rebuilds. Items reference path+symbol (stable across reindex), not chunk ids.
+Pin to prevent TTL deletion.
+
+## Usage tracking
+
+Global (`~/.codelens/usage.db`): per-tool calls, bytes served, and an estimated
+context saving computed from **actual indexed file sizes** of the result files.
+Discovery tools (`cl_search`, `cl_explore`, `cl_related`, `cl_impact`) accrue savings. See
+[`docs/usage-metrics.md`](usage-metrics.md).
+
+## Distribution
+
+Currently builds from source (Node ≥ 22.5) via `install.sh`/`install.ps1`,
+which also wires agent configs + slash commands. A self-contained bundle
+(vendored Node) like some other tools is future work once published. The `cl_*`
+MCP tools are the same surface across every host.

package/docs/routing.md

@@ -14,14 +14,16 @@ right tool for the job.
 
 Prefer codelens when:
 - you don't know the exact name/string (semantic or conceptual search via `cl_search`)
-- you need relationships — importers, tests, callers (`cl_related`) — or a
-  per-file outline / repo map (`cl_map`)
+- you need broad orientation in one call (`cl_explore`) — "how does X work?", flows, or unfamiliar areas
+- you need relationships — importers, tests, callers (`cl_related`) — or blast radius before edits (`cl_impact`)
+- you need a per-file outline / repo map (`cl_map`)
 - the repo is large or unfamiliar, or you'd otherwise grep + read many files
 - branch-scoped correctness matters (results won't leak across branches)
 
 Then use `cl_expand` to read the exact current content of a chosen target (it
 reads from disk — never stale), and `cl_save`/`cl_load` to persist working
-context across compaction.
+context across compaction. If a query result has `stale:true`, read that file
+from disk before relying on indexed snippets/edges.
 
 ## When raw grep/find/read is fine (or better)
 
@@ -44,9 +46,11 @@ branch's index automatically.
 
 ## Freshness
 
-`cl_search` auto-refreshes changed files before returning (budget-bounded). If
-the response carries `freshness: "partial"` and `pendingFiles > 0`, some very
-recent edits may not yet be reflected — call `cl_refresh` or re-query.
+Query tools auto-refresh changed files before returning (budget-bounded). If
+the response carries `freshness: "partial"`, `pendingFiles > 0`, or per-result
+`stale:true`, some recent edits were not reindexed within the budget. Read those
+files directly with `cl_expand`/raw read, then call `cl_refresh` or re-query when
+you need indexed relationships to catch up.
 
 ## Tool quick reference
 
@@ -55,7 +59,9 @@ recent edits may not yet be reflected — call `cl_refresh` or re-query.
 | `cl_current` | repo/branch/index status + freshness |
 | `cl_refresh` | build/update the current branch index |
 | `cl_search` | hybrid ranked search → compact handles |
+| `cl_explore` | one-call grouped search + previews + relationship map |
 | `cl_related` | graph neighbors (imports/tests/callers) |
+| `cl_impact` | callers/callees/affected files/tests before edits |
 | `cl_map` | per-file symbol outline (repo map) |
 | `cl_expand` | exact current file content by path/range |
 | `cl_save` / `cl_load` | persist + reload working context |

package/docs/tools.md

@@ -15,24 +15,37 @@ JSON-serialized text content.
 
 ## cl_search
 - **Input**: `{ query: string, limit?: number=5, cursor?: string, contentType?: "code"|"prose", related?: boolean, snippet?: "none"|"headline"|"compact"|"full" }`
-- **Returns**: `{ indexId, query, count, results:[{handle,path,lines,score,why,preview}], freshness, nextCursor?, pendingFiles?, related? }`
-  - `lines` is `"start-end"`; `why` is a comma-joined signal string; `preview` is a short highlighted snippet (empty when `snippet:"none"`). Use `handle` with `cl_expand`/`cl_save`. Pagination uses the top-level `nextCursor` (no per-result cursor).
+- **Returns**: `{ indexId, query, count, results:[{handle,path,lines,score,why,preview,stale?}], freshness, nextCursor?, pendingFiles?, related? }`
+  - `lines` is `"start-end"`; `why` is a comma-joined signal string; `preview` is a short highlighted snippet (empty when `snippet:"none"`). Use `handle` with `cl_expand`/`cl_save`. Pagination uses the top-level `nextCursor` (no per-result cursor). If `stale:true`, read that file from disk (`cl_expand`/raw read) before relying on indexed content.
 - **Use**: intent-level code discovery.
 - **Identifier matching**: code identifiers are indexed with bounded subtokens (`validateSession` also matches `session`) in the match-only FTS text. Query expansion uses the same bounded splitter and preserves snippets/handles from stored chunk content.
 - **`snippet` (preview verbosity)**: default is signature-first `headline` (richer `compact` for the top ~3 results), which keeps payloads small. `none` returns path+lines only (fetch with `cl_expand`); `compact`/`full` return larger code windows. Explicitly setting `snippet` applies that mode to all results.
 - **`why` signals**: `fts|symbol|exact|graph|path|code` (exact = exact symbol-name match; path = query term in the file path). Ranking is deterministic with a stable tie-break.
 - **Line-range format**: `cl_search` reports `lines` as a `"start-end"` string for compact display; `cl_map` and `cl_expand` use numeric `startLine`/`endLine` for programmatic use. This difference is intentional.
 
+## cl_explore
+- **Input**: `{ query: string, limit?: number=8, cursor?: string, contentType?: "code"|"prose", snippet?: "none"|"headline"|"compact"|"full", relatedDepth?: number=1, maxFiles?: number=6, maxResultsPerFile?: number=3, maxRelated?: number=20 }`
+- **Returns**: `{ indexId, query, count, files:[{path, stale?, results:[{handle,lines,score,why,preview,signature?,collapsed?,stale?}]}], related:[{sourcePath,path,edgeType,hops,confidence,stale?}], freshness, pendingFiles?, nextCursor?, truncated? }`
+- **Use**: broad orientation in one call — "how does X work?", "show the flow around Y", or surveying an unfamiliar area. It fuses `cl_search` + grouped previews + graph relationships; use `cl_search` when you only need to locate handles.
+
 ## cl_related
 - **Input**: `{ path: string, types?: string[], depth?: number=2, direction?: "out"|"in"|"both" }`
-- **Returns**: `{ indexId, results:[{handle,path,edgeType,hops,confidence}] }`
+- **Returns**: `{ indexId, results:[{handle,path,edgeType,hops,confidence,stale?}], freshness?, pendingFiles? }`
 - **Edge types**: `imports|imported_by|tests|calls|references|defines|exports|belongs_to` (TS/JS populate `calls`/`references` and resolve dynamic `import()`).
 
+## cl_impact
+- **Input**: `{ symbol?: string, path?: string, depth?: number=2, includeTests?: boolean=true }`
+- **Returns**: `{ indexId, target?, candidates?, callers, callees, affectedFiles, affectedTests, depth, summary?, confidenceNote, freshness?, pendingFiles? }`
+- **Use**: before changing shared code. Pass `symbol` plus `path` when possible; if a symbol is ambiguous, CodeLens returns `candidates` instead of guessing. Impact is edge-derived and includes confidence/hop counts plus provenance labels (`graph` or conservative `path-heuristic`); use `cl_expand` to inspect exact current code before editing.
+
 ## cl_map
 - **Input**: `{ path?: string, limit?: number=50, all?: boolean }`
-- **Returns**: `{ indexId, files:[{path, symbols:[{name,kind,signature,startLine,endLine,exported}]}], fileCount, truncated }`
+- **Returns**: `{ indexId, files:[{path, stale?, symbols:[{name,kind,signature,startLine,endLine,exported}]}], fileCount, truncated, freshness?, pendingFiles? }`
 - **Use**: outline / repo-map for orientation — per-file symbol signatures from the index (no file re-read). Defaults to exported symbols; pass `all:true` for everything. File-capped (default 50, max 200) with a `truncated` flag.
 
+## Freshness and stale flags
+Query tools perform a budget-bounded reconciliation before answering. If the response has `freshness:"partial"` and `pendingFiles > 0`, some changed files were not reindexed within the budget. Results/files with `stale:true` point at those known-stale paths; read them directly with `cl_expand` or a raw read before depending on indexed previews/edges.
+
 ## cl_expand
 - **Input**: `{ path?: string, handle?: string, startLine?: number, endLine?: number, budget?: number=4000 }`
 - **Returns**: `{ path, startLine, endLine, content, truncated, chars }`

package/docs/usage-metrics.md

@@ -0,0 +1,89 @@
+# Usage metrics — how "saved" is calculated
+
+`cl_usage` reports per-tool call counts, bytes served, and an **estimated**
+context-window saving. The saving is an *estimate*, not a measurement — this
+page explains exactly how it's computed and its limits.
+
+## Which tools are tracked
+
+Only the agent's **retrieval + context-management** tools are tracked as
+"usage" — the ones that represent the agent actually using CodeLens to
+find/read/save code:
+
+| Tool | tracked | calls | bytes_served | bytes_saved |
+|------|---------|-------|--------------|-------------|
+| `cl_search` | ✅ | ✅ | ✅ | ✅ (discovery) |
+| `cl_explore` | ✅ | ✅ | ✅ | ✅ (discovery) |
+| `cl_related` | ✅ | ✅ | ✅ | ✅ (discovery) |
+| `cl_impact` | ✅ | ✅ | ✅ | ✅ (discovery) |
+| `cl_expand` | ✅ | ✅ | ✅ | 0 (it *is* the scoped read step) |
+| `cl_save` / `cl_load` | ✅ | ✅ | ✅ | 0 (context management) |
+
+**Operational tools are NOT tracked** (they're maintenance, not usage):
+`cl_refresh`, `cl_doctor`, `cl_stats`, `cl_prune`, `cl_drop`, `cl_current`,
+`cl_usage`. So `cl_refresh` (building the index) does not appear in the usage
+report, and checking `cl_usage` never inflates the numbers.
+
+Only the **discovery** tools (`cl_search`, `cl_explore`, `cl_related`, `cl_impact`) accrue `bytes_saved` —
+the ones that replace "grep + read a bunch of files" with compact indexed context.
+
+## The formula (refined — actual file sizes)
+
+For a discovery call, CodeLens computes savings from the **real sizes of the
+files in the results**, which it already indexes (`files.size`):
+
+```
+saved = max(0, Σ(distinct result files' indexed sizes) − bytesServed)
+```
+
+- `distinct result files` = the unique paths among the returned handles/files
+  (`cl_search`/`cl_related` carry `results[].path`; `cl_explore` carries
+  `files[].path`; `cl_impact` carries paths in target/candidates/callers/callees/affected lists).
+- `indexed sizes` = `files.size` for those paths in the current branch index.
+- `bytesServed` = bytes of the JSON actually sent to the model.
+- **Capped at 50 distinct files** so a `cl_related` result returning 100
+  importers doesn't inflate the total (the agent wouldn't read all 100
+  without the tool).
+
+**Counterfactual being modeled:** without the index, the agent would `grep` +
+`read` the relevant files (whole-file reads at their real sizes); with the tool
+it got compact handles. The difference is the saved context.
+
+### Fallback (flat proxy)
+
+If the size lookup fails (e.g. index not ready, paths not yet indexed), CodeLens
+falls back to a flat proxy:
+
+```
+saved = max(0, handles × 4096 − bytesServed)
+```
+
+where `4096` is a rough average file size and `handles` = number of result
+entries. This is the older, less-accurate estimate, kept only as a graceful
+fallback.
+
+## Honest limits
+
+1. **It's a counterfactual estimate, not a measurement.** We don't know exactly
+   which files the agent *would have* read without the tool — we assume the
+   result files, whole.
+2. **Whole-file assumption.** We credit the full file size, but a raw `read`
+   might be partial, or the agent might read only the relevant part — so the
+   estimate can overstate for large files with small relevant regions.
+3. **No grep-output cost.** It ignores the bytes `grep` itself would have dumped
+   into context, which would make the real saving *larger*.
+4. **No multi-round exploration.** It counts one call, not the exploration
+   rounds the agent avoids.
+5. The 50-file cap is a judgment call to curb broad relationship/impact results.
+
+Net: treat `saved(est)` as an order-of-magnitude indicator of how much context
+the index tools are sparing, not an audited figure.
+
+## Storage
+
+Usage is **global** (`~/.codelens/usage.db`), aggregated across all repos, with
+a `(tool, repo_id)` key so `cl_usage` can break it down per repo. It survives
+restarts and core-index rebuilds (it's separate from the per-branch index DBs).
+
+Reset with `cl_usage` is not exposed as a tool; from code, `UsageTracker.reset()`
+clears it.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@fodx/codelens",
-  "version": "2.0.0",
+  "version": "2.1.0",
   "description": "Local, branch-aware code search & relation graph. SQLite FTS5 + tree-sitter symbols + source-graph edges, exposed as an MCP server and CLI. No chat LLM required.",
   "license": "MIT",
   "type": "module",
@@ -39,8 +39,10 @@
     "adapters/pi",
     "README.md",
     "docs/agent-guide.md",
+    "docs/how-it-works.md",
     "docs/routing.md",
     "docs/tools.md",
+    "docs/usage-metrics.md",
     "docs/codelens-preview.png"
   ],
   "pi": {
@@ -58,7 +60,8 @@
     "mcp:smoke": "node build/src/server.js --smoke",
     "start": "node build/src/server.js",
     "benchmark": "npx tsx bench/run.ts",
-    "quality": "npx tsx bench/quality.ts"
+    "quality": "npx tsx bench/quality.ts",
+    "eval:agent": "npx tsx bench/agent-eval.ts"
   },
   "dependencies": {
     "@modelcontextprotocol/sdk": "^1.26.0",