xindex 1.0.0

package/.ai/task/task.2026-04-10-search-config.md

@@ -0,0 +1,274 @@
+# Task: Search Result Config — Keyword Ignore, File Ignore & Inline Code Snippets
+
+## Context
+
+Three improvements to xindex, all configurable via `.xindex.json`:
+
+1. **Keyword ignore list** — exclude noisy keywords at index time (case-insensitive exact match). Improves grouping relevance. Requires re-index after config change — acceptable as one-time setup.
+2. **File ignore list** — gitignore-style glob patterns to exclude files from indexing. Same semantics as `.gitignore` but defined in `.xindex.json`. Applied in `WalkFiles` and `WatchFiles` alongside existing `.gitignore` rules.
+3. **Inline code snippets** — when a search result is small (≤ N lines), include actual source code in the output. Configurable via `.xindex.json` defaults + MCP tool parameter overrides.
+
+**Current state:**
+- `.xindex.json` exists but is empty `{}` — file is optional, may not exist at all
+- Keywords: `compromise` NLP → `keyword-extractor` cleanup in `componets/keywords/cleanUpKeywords.ts`
+- File walking: `componets/walkFiles.ts` uses `ignore` package for `.gitignore` rules; `componets/watchFiles.ts` has its own `loadGitignore` + `ignore()` at line 22-31
+- Search results: 1-line summaries only (`1. path:from-to (score) — keywords`)
+- Cluster metadata stores `fromLine`/`toLine` in `componets/index/indexMeta.ts:11` (`IClusterMeta`)
+- MCP `xindex_search` accepts `query` and `limit` only
+- No config loading exists
+- Entry points that create `WalkFiles`/`WatchFiles`: `run.mcp.ts:18,30`, `run.index.ts:10`, `run.watch.ts:13-14`
+
+**Decisions:**
+- `.xindex.json` is **optional** — missing file → all defaults, no error
+- `ignoreKeywords`: exact strings, case-insensitive. No globs/patterns.
+- `ignoreFiles`: gitignore-style glob patterns (reuses existing `ignore` package)
+- `maxSnippetResults: 3`, `maxSnippetLines: 7` — confirmed defaults
+- Ignore list applied at **index time** — re-index + MCP restart required after config change. One-time setup, review in 3mo.
+- File-level results (no cluster) **also get snippets** if total file lines ≤ `maxSnippetLines`
+- Config field names are explicit: `ignoreKeywords`, `ignoreFiles`, `maxSnippetLines`, `maxSnippetResults`
+
+## Diagram
+
+```
+.xindex.json (optional)                   MCP xindex_search
+┌──────────────────────────┐              ┌──────────────────────────────┐
+│ ignoreKeywords: [...]    │              │ query, limit                 │
+│ ignoreFiles: [...]       │              │ maxSnippetResults: 3         │ ← override
+│ maxSnippetLines: 7       │              │ maxSnippetLines: 7           │ ← override
+│ maxSnippetResults: 3     │              └──────┬───────────────────────┘
+└──────┬───────────────────┘                     │
+       │                                         │
+       ├─ ignoreFiles ────┐                      │
+       │                   ▼                     │
+       │            WalkFiles + WatchFiles       │
+       │            (skip matching paths)        │
+       │                                         │
+       ├─ ignoreKeywords ─┐                      │
+       │                   ▼                     │
+       │           CleanUpKeywords               │
+       │           (index time)                  │
+       │                                         │
+       └─ snippet config ────────────────────────┤
+                                                  ▼
+                                          Format results
+                                          ├─ cluster ≤ maxSnippetLines? → readSnippet
+                                          └─ file ≤ maxSnippetLines?    → readSnippet
+
+Data flow (indexing):
+  walkFiles(inputs)                          ← ignoreFiles applied here (NEW)
+    → readFile
+    → ExtractKeywords (compromise NLP)
+    → CleanUpKeywords (keyword-extractor + ignoreKeywords filter)  ← NEW
+    → embed → vectra upsert + objectStore write
+
+Data flow (search):
+  query
+    → extractKeywords → cleanUpKeywords → embed → vectra query
+    → filter by scoreThreshold → objectStore.read for each hit
+    → format results + readSnippet for top N small results  ← NEW
+```
+
+## Steps
+
+### 1. Config schema & loading
+
+- **1.1 Define config type** — create `componets/config/xindexConfig.ts`
+  ```ts
+  export type IXindexConfig = {
+      ignoreKeywords: string[];
+      ignoreFiles: string[];
+      maxSnippetLines: number;
+      maxSnippetResults: number;
+  };
+  ```
+  All fields optional in the JSON file; defaults applied at load time.
+
+- **1.2 Load config** — create `componets/config/loadConfig.ts` as HOF
+  ```ts
+  export type ILoadConfig = () => Promise<IXindexConfig>;
+  export function LoadConfig({configPath, log}: {configPath: string, log: ILogger}): ILoadConfig
+  ```
+  - Read `configPath` (`.xindex.json` in cwd)
+  - `JSON.parse`, apply defaults: `{ignoreKeywords: [], ignoreFiles: [], maxSnippetLines: 7, maxSnippetResults: 3}`
+  - If file missing or empty → return all defaults (no error)
+  - If JSON parse fails → throw with clear message including path
+  - Validate: use `log` to warn if any `ignoreKeywords` entry has length ≤ 1 (no `console.*` — project uses `ILogger`)
+
+- **1.3 Wire into BuildComponents** — modify `componets/buildComponents.ts:6-22`
+  - Current: creates `embed`, `extractKeywords`, `cleanUpKeywords({maxNgrams: 2, minLength: 2})` then `ContentIndexDriver`
+  - `BuildComponents` currently takes no args. Add `{log}: {log: ILogger}` so `LoadConfig` can use it for warnings.
+  - Add: `const loadConfig = LoadConfig({configPath: ".xindex.json", log})` → `const config = await loadConfig()`
+  - Pass `config.ignoreKeywords` to `CleanUpKeywords`: `CleanUpKeywords({maxNgrams: 2, minLength: 2, ignoreKeywords: config.ignoreKeywords})`
+  - Return `config` in the output so callers can access snippet + file ignore settings
+  - `BuildComponents` return type gains `config: IXindexConfig`
+  - All callers need updating to pass `log` and destructure `config`:
+    - `apps/run.mcp.ts:19` — needs `config` for `McpApp` + `ignoreFiles` for `WalkFiles`/`WatchFiles`
+    - `apps/run.index.ts:11` — needs `config.ignoreFiles` for `WalkFiles`
+    - `apps/run.watch.ts:15` — needs `config.ignoreFiles` for `WalkFiles`/`WatchFiles`
+    - `apps/run.search.ts:8` — needs `config` for snippet settings
+
+### 2. Keyword ignore list
+
+- **2.1 Extend CleanUpKeywords** — modify `componets/keywords/cleanUpKeywords.ts:8`
+  - Current signature: `CleanUpKeywords({maxNgrams, minLength}: {maxNgrams: number, minLength: number})`
+  - New signature: `CleanUpKeywords({maxNgrams, minLength, ignoreKeywords = []}: {maxNgrams: number, minLength: number, ignoreKeywords?: string[]})`
+  - Build `ignoreSet = new Set(ignoreKeywords.map(k => k.toLowerCase()))` at factory time (once, not per call)
+  - Add `if (ignoreSet.has(lower)) return false;` into existing filter chain at line 21-27, before the `seen` dedup check
+
+  Exact change at `cleanUpKeywords.ts`:
+  ```ts
+  export function CleanUpKeywords({maxNgrams, minLength, ignoreKeywords = []}: {
+      maxNgrams: number, minLength: number, ignoreKeywords?: string[]
+  }): ICleanUpKeywords {
+      const ignoreSet = new Set(ignoreKeywords.map(k => k.toLowerCase()));
+      return function cleanUpKeywords(keywords) {
+          // ... existing extraction ...
+          const seen = new Set<string>();
+          return extracted.filter((kw: string) => {
+              if (kw.length <= minLength || !/[a-z]/i.test(kw)) return false;
+              const lower = kw.toLowerCase();
+              if (ignoreSet.has(lower)) return false;  // ← NEW
+              if (seen.has(lower)) return false;
+              seen.add(lower);
+              return true;
+          });
+      }
+  }
+  ```
+
+- **2.2 Propagation** — single change at `buildComponents.ts:9` propagates to all consumers:
+  - `ContentIndexDriver` (`contentIndexDriver.ts:28`) passes `cleanUpKeywords` to:
+    - `ClusterLines` (`clusterLines.ts:20`) — uses at `:34-35` (top/bot split keywords) and `:56` (leaf keywords)
+    - `IndexFileContent` (`indexFileContent.ts:10`) — uses at `:19` (file-level keywords)
+    - `SearchContentIndex` (`searchContentIndex.ts:12`) — uses at `:22` (query keywords)
+  - All paths share the same `cleanUpKeywords` instance. No additional wiring needed.
+
+### 3. File ignore list
+
+- **3.1 Extend WalkFiles** — modify `componets/walkFiles.ts:8`
+  - Current signature: `WalkFiles({cwd, log}: {cwd: string, log: ILogger})`
+  - New signature: `WalkFiles({cwd, log, ignoreFiles = []}: {cwd: string, log: ILogger, ignoreFiles?: string[]})`
+  - In `walk()` at line 18-22: the `ignore` instance `ig` is already constructed per-directory with accumulated `.gitignore` rules. Add `ignoreFiles` rules after existing rules:
+    ```ts
+    const ig = ignore();
+    for (const rule of rules) ig.add(rule);
+    for (const pattern of ignoreFiles) ig.add(pattern);  // ← NEW
+    ```
+  - This makes `ignoreFiles` patterns behave identically to `.gitignore` entries — same glob syntax, same matching semantics (relative paths, directory trailing `/`, negation with `!`)
+  - The `ignore` package is already a dependency (`package.json:28`)
+
+- **3.2 Extend WatchFiles** — modify `componets/watchFiles.ts:20`
+  - Current signature: `WatchFiles({cwd, log}: {cwd: string, log: ILogger})`
+  - New signature: `WatchFiles({cwd, log, ignoreFiles = []}: {cwd: string, log: ILogger, ignoreFiles?: string[]})`
+  - In `loadGitignore()` at line 22-31: creates its own `ignore()` instance per watched directory. Add `ignoreFiles` rules after `.gitignore` rules:
+    ```ts
+    async function loadGitignore(dir: string) {
+        const ig = ignore();
+        ig.add(".*");
+        try {
+            const content = await readFile(join(dir, ".gitignore"), "utf8");
+            ig.add(content);
+        } catch {}
+        for (const pattern of ignoreFiles) ig.add(pattern);  // ← NEW
+        return ig;
+    }
+    ```
+
+- **3.3 Wire ignoreFiles to all entry points** — pass `config.ignoreFiles` at each `WalkFiles`/`WatchFiles` construction:
+  - `apps/run.mcp.ts:18` — `WalkFiles({cwd, log})` → `WalkFiles({cwd, log, ignoreFiles: config.ignoreFiles})`
+  - `apps/run.mcp.ts:30` — `WatchFiles({cwd, log})` → `WatchFiles({cwd, log, ignoreFiles: config.ignoreFiles})`
+  - `apps/run.index.ts:10` — `WalkFiles({cwd, log})` → `WalkFiles({cwd, log, ignoreFiles: config.ignoreFiles})`
+  - `apps/run.watch.ts:13` — `WalkFiles({cwd, log})` → `WalkFiles({cwd, log, ignoreFiles: config.ignoreFiles})`
+  - `apps/run.watch.ts:14` — `WatchFiles({cwd, log})` → `WatchFiles({cwd, log, ignoreFiles: config.ignoreFiles})`
+
+### 4. Inline code snippets
+
+- **4.1 Add snippet params to MCP** — modify `apps/mcpApp.ts:34-58`
+  - `McpApp` factory gains `config: IXindexConfig` dependency (add to `mcpApp.ts:23` params)
+  - Extend `xindex_search` schema at line 37:
+    ```ts
+    inputSchema: z.object({
+        query: z.string().describe("Natural language search query"),
+        limit: z.number().int().min(1).max(100).default(10)
+            .describe("Max results to return, 10 by default, 100 max"),
+        maxSnippetResults: z.number().int().min(0).max(20).optional()
+            .describe("How many top results include inline code (default from .xindex.json, 3)"),
+        maxSnippetLines: z.number().int().min(0).max(50).optional()
+            .describe("Max lines in a result to qualify for inline code (default from .xindex.json, 7)"),
+    }),
+    ```
+  - In handler: resolve with config fallback:
+    ```ts
+    const sr = maxSnippetResults ?? config.maxSnippetResults;
+    const sl = maxSnippetLines ?? config.maxSnippetLines;
+    ```
+  - Update `apps/run.mcp.ts:48` — pass `config` to `McpApp`
+
+- **4.2 Read source lines** — create `componets/index/readSnippet.ts`
+  ```ts
+  export type IReadSnippet = (record: IIndexRecord, maxLines: number) => Promise<string | null>;
+  export function ReadSnippet(): IReadSnippet
+  ```
+  Logic:
+  - **Cluster result** (`meta.type === StoreEntryType.cluster`): use `meta.fromLine`/`meta.toLine` directly from `IClusterMeta` (no need to parse from ID). Compute `lineCount = meta.toLine - meta.fromLine + 1`. If `lineCount > maxLines` → return `null`. Extract file path from `record.id` by splitting on last `:` (format `"path/to/file.ts:14-27"`). `readFile(filePath, "utf8")`, split lines, slice `[meta.fromLine-1, meta.toLine]`, return joined with `\n`.
+  - **File result** (`meta.type === StoreEntryType.meta`): `readFile(record.id, "utf8")`, split lines, count. If `lineCount > maxLines` → return `null`. Otherwise return full content.
+  - **Error handling**: on `readFile` failure (file deleted, moved, permission error) → return `null` silently. Search results still display; snippet is just omitted.
+
+- **4.3 Format with code** — update result formatting in both entry points:
+
+  **MCP** (`apps/mcpApp.ts:47-51`): currently `results.map(...)` builds 1-line summaries. Replace with loop:
+  ```ts
+  const readSnippet = ReadSnippet();
+  const lines: string[] = [];
+  for (let i = 0; i < results.length; i++) {
+      const r = results[i];
+      const kw = r.meta.keywords ? ` — ${r.meta.keywords}` : "";
+      lines.push(`${i + 1}. ${r.id} (${r.score.toFixed(2)})${kw}`);
+      if (i < sr) {
+          const snippet = await readSnippet(r, sl);
+          if (snippet) lines.push("```\n" + snippet + "\n```");
+      }
+  }
+  ```
+
+  **CLI** (`apps/run.search.ts:8-31`): destructure config from `BuildComponents()`:
+  ```ts
+  const {searchContentIndex, config} = await BuildComponents({log});
+  ```
+  Then same snippet pattern using `config.maxSnippetResults` and `config.maxSnippetLines`:
+  ```ts
+  const readSnippet = ReadSnippet();
+  // ... in the result loop after existing log lines:
+  if (i < config.maxSnippetResults) {
+      const snippet = await readSnippet(results[i], config.maxSnippetLines);
+      if (snippet) log("```\n" + snippet + "\n```");
+  }
+  ```
+
+## Files Changed
+
+| File | Change |
+|------|--------|
+| `componets/config/xindexConfig.ts` | **NEW** — `IXindexConfig` type with 4 fields |
+| `componets/config/loadConfig.ts` | **NEW** — `LoadConfig` HOF, reads `.xindex.json` (optional), applies defaults, warns via `ILogger` |
+| `componets/keywords/cleanUpKeywords.ts` | Add `ignoreKeywords` param + `ignoreSet` filter |
+| `componets/walkFiles.ts` | Add `ignoreFiles` param, feed into `ignore()` instances |
+| `componets/watchFiles.ts` | Add `ignoreFiles` param, feed into `loadGitignore()` `ignore()` instance |
+| `componets/buildComponents.ts` | Add `{log}` param, load config, pass `ignoreKeywords` to `CleanUpKeywords`, return `config` |
+| `componets/index/readSnippet.ts` | **NEW** — `ReadSnippet` HOF, reads file lines for a search result |
+| `apps/mcpApp.ts` | Add `config` dep, `maxSnippetResults`/`maxSnippetLines` schema params, snippet formatting |
+| `apps/run.mcp.ts` | Pass `config` to `McpApp`, `ignoreFiles` to `WalkFiles`/`WatchFiles`, `log` to `BuildComponents` |
+| `apps/run.search.ts` | Pass `log` to `BuildComponents`, use `config` for snippet formatting |
+| `apps/run.index.ts` | Pass `log` to `BuildComponents`, `ignoreFiles` to `WalkFiles` |
+| `apps/run.watch.ts` | Pass `log` to `BuildComponents`, `ignoreFiles` to `WalkFiles`/`WatchFiles` |
+
+## Example `.xindex.json`
+
+```json
+{
+  "ignoreKeywords": ["import", "export", "const", "function", "return", "async", "await"],
+  "ignoreFiles": ["*.test.ts", "*.spec.ts", "rnd/**", "dist/**"],
+  "maxSnippetLines": 7,
+  "maxSnippetResults": 3
+}
+```

package/.ai/task/task.2026-04-10-watch-indexing.log.md

@@ -0,0 +1,32 @@
+# Log: xindex-watch — Continuous Indexing with File Watcher
+
+### 2026-04-10
+
+- Task created from user notes: "file watcher → apply to indexer → xindex-index runs → indexes provided or cwd → watches for changes → created/updated/moved/deleted → queued to stream → index content"
+- Scouted codebase: IndexApp already uses streamx pipeline (`from → tap → map → run`), WalkFiles is async generator, Writer supports push-based streaming, merge combines streams
+- Key decision: use `node:fs/promises` `watch()` (recursive, async iterable) — no external dep needed
+- Key decision: tagged union `{type:"index"|"remove", path}` as common event shape for walk + watch streams
+- Key decision: merge initial walk stream + watch stream into single pipeline
+- Debounce needed: editors fire multiple events per save (write temp → rename → delete old)
+- RemoveContent needed: Vectra has `deleteItem()` but no HOF wrapper exists yet
+- **Clarification round resolved:**
+  - Watch is always on (no optional flag) — index all, then watch. Default behavior.
+  - Default to cwd when no args
+  - Event→Vectra: created→add, updated→delete+add, deleted→delete, moved→delete(old)+add(new)
+  - Binary filtering: deferred (TODO), keep simple for now
+  - Graceful shutdown: SIGINT → stop processing → ignore queued → exit
+  - Watching individual files: works fine with fs.watch, no issue
+- **Consistency check:** fixed 6 issues — removed optional watch flag, added graceful shutdown to diagram/steps, clarified update=delete+add semantics, marked binary filtering as TODO
+- **User clarification:** separate entry points — `xindex-watch` (new, continuous) vs `xindex-index` (existing, one-time). Both default to cwd.
+- **Design decision:** Vectra `upsertItem` handles both add and update — no need for delete+add on updates, just upsert
+- **Design decision:** WatchApp is a new HOF in `apps/watchApp.ts`; IndexApp stays unchanged; no modifications to MCP/search paths
+- **Design decision:** WatchFiles uses `Writer<FileEvent>` to push events into streamx-compatible stream; `stop()` closes watchers + finishes writer
+- **Implementation pivot:** streamx `Writer`/`merge`/`batchTimed` depend on `@handy/fun` (not installed in xindex). Rewrote to use plain async generators instead — simpler, no new deps. Two-phase approach: walk+index first, then watch+process.
+- **Debounce approach:** collect events in Map (keyed by path, last event wins), flush after 150ms quiet period. Replaces batchTimed.
+- **Watcher uses AbortController** for clean shutdown — `fs.watch` accepts `signal` option natively.
+- **All steps implemented and verified:**
+  - Step 1: RemoveContent HOF + wired into ContentIndexDriver + BuildComponents ✓
+  - Step 2: WatchFiles component with debounced async generator ✓
+  - Step 3: WatchApp with two-phase (walk then watch) ✓
+  - Step 4: run.watch.ts + bin/xindex-watch + package.json + run.index.ts default ✓
+- **Tested:** initial index, file create detection, file delete detection, SIGINT graceful shutdown — all pass

package/.ai/task/task.2026-04-10-watch-indexing.md

@@ -0,0 +1,101 @@
+# Task: xindex-watch — Continuous Indexing with File Watcher
+
+## Context
+
+Two entry points:
+- `xindex-index` — one-time batch job: index all declared paths (default cwd), exit with status
+- `xindex-watch` — **new**: index all, then watch for changes continuously, Ctrl+C to stop
+
+Both default to cwd when no args. Watch handles create, update, move, delete events via a merged stream.
+
+**Current pipeline** (`apps/indexApp.ts`):
+```
+from(walkFiles(inputs))
+  .pipe(tap(log))
+  .pipe(map(readFile → extractKeywords → cleanUp → indexContent))
+  → run()
+```
+
+**Key pieces already in place:**
+- `WalkFiles` (`componets/walkFiles.ts`) — async generator yielding relative paths, .gitignore-aware
+- `Writer` (`packages/streamx/src/writer.ts`) — push-based stream writer (backpressure-aware)
+- `merge` (`packages/streamx/src/merge.ts`) — combines multiple streams
+- `from` / `of` / `run` — streamx core
+
+**Node.js `fs.watch`** (recursive works on all platforms since Node 22+):
+- `fs.watch(dir, {recursive: true})` — macOS (FSEvents), Windows (ReadDirectoryChangesW), Linux (inotify, fd-per-dir)
+- Returns `AsyncIterable<FileChangeInfo>` with `.eventType` ("rename" | "change") and `.filename`
+- "rename" = create, delete, or move; "change" = content modified
+- Need to `stat()` after event to distinguish create vs delete (file exists → create/update; doesn't exist → delete)
+- Known issues: duplicate events per save, null filenames possible → handled by batchTimed dedup
+
+**No external dep needed** — `node:fs/promises` `watch()` returns async iterable directly. If issues arise, chokidar v5 (1 dep, ESM-only) is a drop-in upgrade. See [research](../research/2026-04-10-file-watching.md).
+
+## Goal
+
+Add `xindex-watch` entry point: index all provided paths (or cwd by default), then watch for changes and keep the index up to date continuously. Ctrl+C gracefully stops. Update `xindex-index` to default to cwd but remain a one-time job.
+
+## Diagram
+
+```
+bin/xindex-index → run.index.ts     bin/xindex-watch → run.watch.ts
+  │ one-time, exits                   │ continuous, SIGINT to stop
+  │                                   │
+  ├─ inputs || [cwd]                  ├─ inputs || [cwd]
+  └─ IndexApp(inputs)                 ├─ SIGINT → app.stop()
+     │                                └─ WatchApp(inputs)
+     └─ walkFiles → index → exit           │
+                                           ├── INITIAL: from(walkFiles)
+                                           │     → map(path → {type:"index", path})
+                                           │
+                                           ├── WATCH: from(watchFiles)
+                                           │     → yields {type:"index"|"remove", path}
+                                           │
+                                           └── merge(initial, watch)
+                                                 │
+                                                 ├── batchTimed (debounce)
+                                                 ├── dedup per path
+                                                 ├── flat()
+                                                 ├── tap(log)
+                                                 ├── type:"index"  → readFile → extractKeywords → cleanUp → indexContent
+                                                 ├── type:"remove" → removeContent
+                                                 │
+                                                 ▼
+                                            runs until SIGINT
+
+Event → Vectra:
+  created  → upsert (add)
+  updated  → upsert (overwrite)
+  deleted  → deleteItem
+  moved    → deleteItem(old) + upsert(new)  (two fs.watch events)
+```
+
+## Steps
+
+### 1. RemoveContent + Wiring
+- **RemoveContent HOF** — create `componets/index/removeContent.ts` wrapping Vectra `deleteItem(id)`, matching `indexContent.ts` pattern
+- **ContentIndexDriver** — add `removeContent` to interface and factory
+- **BuildComponents** — expose `removeContent` in return object
+
+### 2. WatchFiles Component
+- **FileEvent type** — define `{type: "index"|"remove", path: string}` in `componets/watchFiles.ts` (exported, shared with WatchApp)
+- **WatchFiles HOF** — create `componets/watchFiles.ts`, uses `Writer<FileEvent>` + `fs.watch(dir, {recursive:true})`; returns `{stream, stop}`
+- **Event mapping** — "change" + stat exists → index; "rename" + stat exists → index; "rename" + stat throws → remove; filter through gitignore rules
+- **Stop method** — close FSWatcher handles + `writer.finish()` to end the stream
+
+### 3. WatchApp
+- **WatchApp HOF** — create `apps/watchApp.ts`: merge walk stream (mapped to FileEvents) + watch stream → `batchTimed(20, 150)` → dedup per path → `flat()` → `tap(log)` → process: "index" → readFile → extractKeywords → cleanUp → indexContent; "remove" → removeContent → `run()`; returns `{run, stop}`
+
+### 4. Entry Points
+- **run.watch.ts** — new entry point: default to cwd, SIGINT → `app.stop()`, log stats on exit
+- **bin/xindex-watch** + package.json — bin shim + scripts entry
+- **run.index.ts** — default to cwd instead of error-exit; IndexApp unchanged (one-time job)
+
+## Edge Cases
+
+- **Gitignore changes** — if `.gitignore` itself is modified, watcher should reload rules (or at minimum, not index newly-ignored files)
+- **Binary files** — TODO: filter by extension or detect encoding; for now, readFile utf8 on everything (may produce garbage keywords for binaries)
+- **Rapid renames** — editor save = delete old + create new; debounce window prevents double-processing
+- **Symlinks** — `fs.watch` recursive doesn't follow symlinks; acceptable default
+- **Index doesn't have the file** — "remove" for a file not in index should be a no-op (log warning, don't throw)
+- **Watching individual files** — `fs.watch` works on single files too; not a primary use case but no issue supporting it

package/.ai/task/task.2026-04-10-xindex-mcp.log.md

@@ -0,0 +1,5 @@
+### 2026-04-10 — Task created
+
+- User researched MCP SDK pattern (Server + StdioServerTransport + tool handlers)
+- Scope: wrap existing xindex BuildComponents as 2 MCP tools (index + search)
+- Integration: add to .claude/settings.json for Claude Code auto-discovery

package/.ai/task/task.2026-04-10-xindex-mcp.md

@@ -0,0 +1,92 @@
+# Task: xindex-mcp — MCP Server for Semantic Code Search
+
+## Context
+
+xindex is a working local semantic code search tool (index files → query by meaning). Goal: wrap it as an MCP server so Claude Code can search the codebase directly.
+
+**Existing xindex pipeline:**
+- `extractKeywords` → `cleanUpKeywords` → `embed` (MiniLM-L6) → `vectra` (upsert/query)
+- `BuildComponents()` wires everything
+- Entry points: `apps/run.index.ts`, `apps/run.search.ts`
+
+**MCP SDK (modern API — `registerTool` + Zod):**
+```ts
+import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
+import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { z } from "zod";
+
+const server = new McpServer({ name: "xindex", version: "0.1.0" });
+
+server.registerTool("xindex_search", {
+  title: "Search codebase",
+  description: "Semantic search over indexed codebase files",
+  inputSchema: z.object({
+    query: z.string().describe("Natural language search query"),
+    limit: z.number().int().min(1).max(50).default(10).describe("Max results"),
+  }),
+  annotations: { readOnlyHint: true },
+}, async ({ query, limit }) => {
+  // call searchContentIndex(query, limit)
+  return { content: [{ type: "text", text: JSON.stringify(results) }] };
+});
+
+const transport = new StdioServerTransport();
+await server.connect(transport);
+```
+
+**Integration:** `.mcp.json` in project root (project scope — shared via git).
+
+**Decisions:**
+- `BuildComponents()` inits once at startup — just factories, fast
+- MCP exposes **search only** — indexing is a separate process (`bin/xindex-index`)
+- Future: file watcher daemon to keep index up to date (out of scope for this task)
+
+## Goal
+
+Create an MCP server that exposes `xindex_search` as a tool, so Claude Code can search the codebase semantically. Indexing runs separately via CLI.
+
+## Diagram
+
+```
+                    ┌─────────────────────────┐
+                    │   bin/xindex-index       │
+                    │   (separate process)     │
+                    │   indexes files → .xindex│
+                    └──────────┬──────────────┘
+                               │ writes
+                               ▼
+                          .xindex/ (vectra)
+                               ▲
+                               │ reads
+┌──────────┐  stdio   ┌───────┴──────────────┐
+│Claude Code├──────────┤  xindex-mcp server   │
+└──────────┘          │                       │
+                      │  tool: xindex_search  │
+                      │   query → extract     │
+                      │   → cleanUp → embed   │
+                      │   → vectra query      │
+                      │   → IIndexRecord[]    │
+                      └───────────────────────┘
+```
+
+## Steps
+
+### 1. MCP Server Setup
+- Install `@modelcontextprotocol/sdk` and `zod` dependencies
+- Create `apps/mcp-server.ts` — McpServer + StdioServerTransport
+- `BuildComponents()` at top level (once), use `searchContentIndex` in tool handler
+
+### 2. Tool: xindex_search
+- Input schema: Zod `z.object({query: z.string(), limit: z.number().default(10)})` with `.describe()` on each field
+- Handler: call `searchContentIndex(query, limit)` → return `IIndexRecord[]` as text content
+- Use `annotations: { readOnlyHint: true }` — search has no side effects
+
+### 3. Integration
+- Add `bin/xindex-mcp` entry point (`#!/usr/bin/env tsx` + import)
+- Add `.mcp.json` to project root: `{"mcpServers": {"xindex": {"command": "npx", "args": ["tsx", "apps/mcp-server.ts"]}}}`
+- Test: index codebase via CLI, restart Claude Code, verify tool appears, search it
+- Debug with: `npx @modelcontextprotocol/inspector npx tsx apps/mcp-server.ts`
+
+## Future (out of scope)
+- File watcher daemon — observe fs events, keep index up to date automatically
+- `xindex_index` tool — allow Claude to trigger indexing directly

package/.ai/task/task.2026-04-10-xindex-mcp.report.md

@@ -0,0 +1,113 @@
+# Research: How to Build an MCP Server (2026)
+
+## Findings
+
+### 1. SDK & API
+
+**Package:** `@modelcontextprotocol/sdk` (latest ^1.0.0)
+**Peer dep:** `zod` ^3.22.0
+
+**Modern API** (use this — old `server.tool()` and `setRequestHandler` are deprecated):
+
+```ts
+import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
+import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
+import { z } from "zod";
+
+const server = new McpServer({ name: "xindex", version: "0.1.0" });
+
+server.registerTool(
+  "xindex_search",
+  {
+    title: "Search codebase",
+    description: "Semantic search over indexed codebase files",
+    inputSchema: z.object({
+      query: z.string().describe("Natural language search query"),
+      limit: z.number().int().min(1).max(50).default(10).describe("Max results"),
+    }),
+    annotations: { readOnlyHint: true },
+  },
+  async ({ query, limit }) => {
+    // call searchContentIndex(query, limit)
+    return {
+      content: [{ type: "text", text: JSON.stringify(results) }],
+    };
+  }
+);
+
+const transport = new StdioServerTransport();
+await server.connect(transport);
+```
+
+### 2. Tool naming & annotations
+
+- **snake_case** with service prefix: `xindex_search`, `xindex_index`
+- **annotations**: `readOnlyHint: true` for search (signals no side effects), `destructiveHint: false` for index
+- **inputSchema**: Zod objects with `.describe()` on each field — descriptions are shown to Claude
+
+### 3. Transport
+
+**Stdio** is correct for local tools. Server reads JSON-RPC from stdin, writes to stdout. Use `console.error()` for logging (stdout is the protocol channel).
+
+**Gotcha with tsx**: Works fine as shebang (`#!/usr/bin/env tsx`). No build step needed.
+
+### 4. Claude Code Integration
+
+**Three scopes:**
+
+| Scope | File | Shared? |
+|-------|------|---------|
+| Local (default) | `~/.claude.json` under project path | No |
+| Project | `.mcp.json` in project root | Yes (git) |
+| User | `~/.claude.json` global | No |
+
+**For this project — use `.mcp.json` (project scope)** so it's checked in and works for anyone who clones.
+
+```json
+{
+  "mcpServers": {
+    "xindex": {
+      "command": "npx",
+      "args": ["tsx", "apps/mcp-server.ts"],
+      "env": {}
+    }
+  }
+}
+```
+
+Or add via CLI:
+```bash
+claude mcp add --transport stdio --scope project xindex -- npx tsx apps/mcp-server.ts
+```
+
+### 5. Package requirements
+
+- `"type": "module"` in package.json — already have this
+- tsconfig: `module` and `moduleResolution` should be `Node16` or `NodeNext` (SDK uses ES module exports with subpath imports)
+- Current tsconfig has `"module": "Node16"` — compatible
+
+### 6. Testing
+
+Use MCP Inspector for debugging:
+```bash
+npx @modelcontextprotocol/inspector npx tsx apps/mcp-server.ts
+```
+
+## Recommendation
+
+Minimal implementation — single file `apps/mcp-server.ts`:
+1. `BuildComponents()` at top level
+2. `server.registerTool("xindex_search", ...)` calling `searchContentIndex`
+3. `StdioServerTransport` + connect
+4. `.mcp.json` in project root for Claude Code discovery
+5. `bin/xindex-mcp` shebang entry point
+
+No build step needed (tsx). No express/HTTP needed (stdio only). ~40 lines of code.
+
+## Sources
+
+- [Official TypeScript SDK](https://github.com/modelcontextprotocol/typescript-sdk)
+- [SDK server docs](https://github.com/modelcontextprotocol/typescript-sdk/blob/main/docs/server.md)
+- [Claude Code MCP docs](https://code.claude.com/docs/en/mcp)
+- [Anthropic MCP server reference](https://github.com/anthropics/skills/blob/main/skills/mcp-builder/reference/node_mcp_server.md)
+- [npm package](https://www.npmjs.com/package/@modelcontextprotocol/sdk)