xindex 1.0.0 → 1.0.2

package/.ai/task/task.2026-04-10-watch-indexing.md

@@ -1,101 +0,0 @@
-# Task: xindex-watch — Continuous Indexing with File Watcher
-
-## Context
-
-Two entry points:
-- `xindex-index` — one-time batch job: index all declared paths (default cwd), exit with status
-- `xindex-watch` — **new**: index all, then watch for changes continuously, Ctrl+C to stop
-
-Both default to cwd when no args. Watch handles create, update, move, delete events via a merged stream.
-
-**Current pipeline** (`apps/indexApp.ts`):
-```
-from(walkFiles(inputs))
-  .pipe(tap(log))
-  .pipe(map(readFile → extractKeywords → cleanUp → indexContent))
-  → run()
-```
-
-**Key pieces already in place:**
-- `WalkFiles` (`componets/walkFiles.ts`) — async generator yielding relative paths, .gitignore-aware
-- `Writer` (`packages/streamx/src/writer.ts`) — push-based stream writer (backpressure-aware)
-- `merge` (`packages/streamx/src/merge.ts`) — combines multiple streams
-- `from` / `of` / `run` — streamx core
-
-**Node.js `fs.watch`** (recursive works on all platforms since Node 22+):
-- `fs.watch(dir, {recursive: true})` — macOS (FSEvents), Windows (ReadDirectoryChangesW), Linux (inotify, fd-per-dir)
-- Returns `AsyncIterable<FileChangeInfo>` with `.eventType` ("rename" | "change") and `.filename`
-- "rename" = create, delete, or move; "change" = content modified
-- Need to `stat()` after event to distinguish create vs delete (file exists → create/update; doesn't exist → delete)
-- Known issues: duplicate events per save, null filenames possible → handled by batchTimed dedup
-
-**No external dep needed** — `node:fs/promises` `watch()` returns async iterable directly. If issues arise, chokidar v5 (1 dep, ESM-only) is a drop-in upgrade. See [research](../research/2026-04-10-file-watching.md).
-
-## Goal
-
-Add `xindex-watch` entry point: index all provided paths (or cwd by default), then watch for changes and keep the index up to date continuously. Ctrl+C gracefully stops. Update `xindex-index` to default to cwd but remain a one-time job.
-
-## Diagram
-
-```
-bin/xindex-index → run.index.ts     bin/xindex-watch → run.watch.ts
-  │ one-time, exits                   │ continuous, SIGINT to stop
-  │                                   │
-  ├─ inputs || [cwd]                  ├─ inputs || [cwd]
-  └─ IndexApp(inputs)                 ├─ SIGINT → app.stop()
-     │                                └─ WatchApp(inputs)
-     └─ walkFiles → index → exit           │
-                                           ├── INITIAL: from(walkFiles)
-                                           │     → map(path → {type:"index", path})
-                                           │
-                                           ├── WATCH: from(watchFiles)
-                                           │     → yields {type:"index"|"remove", path}
-                                           │
-                                           └── merge(initial, watch)
-                                                 │
-                                                 ├── batchTimed (debounce)
-                                                 ├── dedup per path
-                                                 ├── flat()
-                                                 ├── tap(log)
-                                                 ├── type:"index"  → readFile → extractKeywords → cleanUp → indexContent
-                                                 ├── type:"remove" → removeContent
-                                                 │
-                                                 ▼
-                                            runs until SIGINT
-
-Event → Vectra:
-  created  → upsert (add)
-  updated  → upsert (overwrite)
-  deleted  → deleteItem
-  moved    → deleteItem(old) + upsert(new)  (two fs.watch events)
-```
-
-## Steps
-
-### 1. RemoveContent + Wiring
-- **RemoveContent HOF** — create `componets/index/removeContent.ts` wrapping Vectra `deleteItem(id)`, matching `indexContent.ts` pattern
-- **ContentIndexDriver** — add `removeContent` to interface and factory
-- **BuildComponents** — expose `removeContent` in return object
-
-### 2. WatchFiles Component
-- **FileEvent type** — define `{type: "index"|"remove", path: string}` in `componets/watchFiles.ts` (exported, shared with WatchApp)
-- **WatchFiles HOF** — create `componets/watchFiles.ts`, uses `Writer<FileEvent>` + `fs.watch(dir, {recursive:true})`; returns `{stream, stop}`
-- **Event mapping** — "change" + stat exists → index; "rename" + stat exists → index; "rename" + stat throws → remove; filter through gitignore rules
-- **Stop method** — close FSWatcher handles + `writer.finish()` to end the stream
-
-### 3. WatchApp
-- **WatchApp HOF** — create `apps/watchApp.ts`: merge walk stream (mapped to FileEvents) + watch stream → `batchTimed(20, 150)` → dedup per path → `flat()` → `tap(log)` → process: "index" → readFile → extractKeywords → cleanUp → indexContent; "remove" → removeContent → `run()`; returns `{run, stop}`
-
-### 4. Entry Points
-- **run.watch.ts** — new entry point: default to cwd, SIGINT → `app.stop()`, log stats on exit
-- **bin/xindex-watch** + package.json — bin shim + scripts entry
-- **run.index.ts** — default to cwd instead of error-exit; IndexApp unchanged (one-time job)
-
-## Edge Cases
-
-- **Gitignore changes** — if `.gitignore` itself is modified, watcher should reload rules (or at minimum, not index newly-ignored files)
-- **Binary files** — TODO: filter by extension or detect encoding; for now, readFile utf8 on everything (may produce garbage keywords for binaries)
-- **Rapid renames** — editor save = delete old + create new; debounce window prevents double-processing
-- **Symlinks** — `fs.watch` recursive doesn't follow symlinks; acceptable default
-- **Index doesn't have the file** — "remove" for a file not in index should be a no-op (log warning, don't throw)
-- **Watching individual files** — `fs.watch` works on single files too; not a primary use case but no issue supporting it

package/.ai/task/task.2026-04-10-xindex-mcp.log.md

@@ -1,5 +0,0 @@
-### 2026-04-10 — Task created
-
-- User researched MCP SDK pattern (Server + StdioServerTransport + tool handlers)
-- Scope: wrap existing xindex BuildComponents as 2 MCP tools (index + search)
-- Integration: add to .claude/settings.json for Claude Code auto-discovery

package/.ai/task/task.2026-04-10-xindex-mcp.md

@@ -1,92 +0,0 @@
-# Task: xindex-mcp — MCP Server for Semantic Code Search
-
-## Context
-
-xindex is a working local semantic code search tool (index files → query by meaning). Goal: wrap it as an MCP server so Claude Code can search the codebase directly.
-
-**Existing xindex pipeline:**
-- `extractKeywords` → `cleanUpKeywords` → `embed` (MiniLM-L6) → `vectra` (upsert/query)
-- `BuildComponents()` wires everything
-- Entry points: `apps/run.index.ts`, `apps/run.search.ts`
-
-**MCP SDK (modern API — `registerTool` + Zod):**
-```ts
-import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
-import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
-import { z } from "zod";
-
-const server = new McpServer({ name: "xindex", version: "0.1.0" });
-
-server.registerTool("xindex_search", {
-  title: "Search codebase",
-  description: "Semantic search over indexed codebase files",
-  inputSchema: z.object({
-    query: z.string().describe("Natural language search query"),
-    limit: z.number().int().min(1).max(50).default(10).describe("Max results"),
-  }),
-  annotations: { readOnlyHint: true },
-}, async ({ query, limit }) => {
-  // call searchContentIndex(query, limit)
-  return { content: [{ type: "text", text: JSON.stringify(results) }] };
-});
-
-const transport = new StdioServerTransport();
-await server.connect(transport);
-```
-
-**Integration:** `.mcp.json` in project root (project scope — shared via git).
-
-**Decisions:**
-- `BuildComponents()` inits once at startup — just factories, fast
-- MCP exposes **search only** — indexing is a separate process (`bin/xindex-index`)
-- Future: file watcher daemon to keep index up to date (out of scope for this task)
-
-## Goal
-
-Create an MCP server that exposes `xindex_search` as a tool, so Claude Code can search the codebase semantically. Indexing runs separately via CLI.
-
-## Diagram
-
-```
-                    ┌─────────────────────────┐
-                    │   bin/xindex-index       │
-                    │   (separate process)     │
-                    │   indexes files → .xindex│
-                    └──────────┬──────────────┘
-                               │ writes
-                               ▼
-                          .xindex/ (vectra)
-                               ▲
-                               │ reads
-┌──────────┐  stdio   ┌───────┴──────────────┐
-│Claude Code├──────────┤  xindex-mcp server   │
-└──────────┘          │                       │
-                      │  tool: xindex_search  │
-                      │   query → extract     │
-                      │   → cleanUp → embed   │
-                      │   → vectra query      │
-                      │   → IIndexRecord[]    │
-                      └───────────────────────┘
-```
-
-## Steps
-
-### 1. MCP Server Setup
-- Install `@modelcontextprotocol/sdk` and `zod` dependencies
-- Create `apps/mcp-server.ts` — McpServer + StdioServerTransport
-- `BuildComponents()` at top level (once), use `searchContentIndex` in tool handler
-
-### 2. Tool: xindex_search
-- Input schema: Zod `z.object({query: z.string(), limit: z.number().default(10)})` with `.describe()` on each field
-- Handler: call `searchContentIndex(query, limit)` → return `IIndexRecord[]` as text content
-- Use `annotations: { readOnlyHint: true }` — search has no side effects
-
-### 3. Integration
-- Add `bin/xindex-mcp` entry point (`#!/usr/bin/env tsx` + import)
-- Add `.mcp.json` to project root: `{"mcpServers": {"xindex": {"command": "npx", "args": ["tsx", "apps/mcp-server.ts"]}}}`
-- Test: index codebase via CLI, restart Claude Code, verify tool appears, search it
-- Debug with: `npx @modelcontextprotocol/inspector npx tsx apps/mcp-server.ts`
-
-## Future (out of scope)
-- File watcher daemon — observe fs events, keep index up to date automatically
-- `xindex_index` tool — allow Claude to trigger indexing directly

package/.ai/task/task.2026-04-10-xindex-mcp.report.md

@@ -1,113 +0,0 @@
-# Research: How to Build an MCP Server (2026)
-
-## Findings
-
-### 1. SDK & API
-
-**Package:** `@modelcontextprotocol/sdk` (latest ^1.0.0)
-**Peer dep:** `zod` ^3.22.0
-
-**Modern API** (use this — old `server.tool()` and `setRequestHandler` are deprecated):
-
-```ts
-import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
-import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
-import { z } from "zod";
-
-const server = new McpServer({ name: "xindex", version: "0.1.0" });
-
-server.registerTool(
-  "xindex_search",
-  {
-    title: "Search codebase",
-    description: "Semantic search over indexed codebase files",
-    inputSchema: z.object({
-      query: z.string().describe("Natural language search query"),
-      limit: z.number().int().min(1).max(50).default(10).describe("Max results"),
-    }),
-    annotations: { readOnlyHint: true },
-  },
-  async ({ query, limit }) => {
-    // call searchContentIndex(query, limit)
-    return {
-      content: [{ type: "text", text: JSON.stringify(results) }],
-    };
-  }
-);
-
-const transport = new StdioServerTransport();
-await server.connect(transport);
-```
-
-### 2. Tool naming & annotations
-
-- **snake_case** with service prefix: `xindex_search`, `xindex_index`
-- **annotations**: `readOnlyHint: true` for search (signals no side effects), `destructiveHint: false` for index
-- **inputSchema**: Zod objects with `.describe()` on each field — descriptions are shown to Claude
-
-### 3. Transport
-
-**Stdio** is correct for local tools. Server reads JSON-RPC from stdin, writes to stdout. Use `console.error()` for logging (stdout is the protocol channel).
-
-**Gotcha with tsx**: Works fine as shebang (`#!/usr/bin/env tsx`). No build step needed.
-
-### 4. Claude Code Integration
-
-**Three scopes:**
-
-| Scope | File | Shared? |
-|-------|------|---------|
-| Local (default) | `~/.claude.json` under project path | No |
-| Project | `.mcp.json` in project root | Yes (git) |
-| User | `~/.claude.json` global | No |
-
-**For this project — use `.mcp.json` (project scope)** so it's checked in and works for anyone who clones.
-
-```json
-{
-  "mcpServers": {
-    "xindex": {
-      "command": "npx",
-      "args": ["tsx", "apps/mcp-server.ts"],
-      "env": {}
-    }
-  }
-}
-```
-
-Or add via CLI:
-```bash
-claude mcp add --transport stdio --scope project xindex -- npx tsx apps/mcp-server.ts
-```
-
-### 5. Package requirements
-
-- `"type": "module"` in package.json — already have this
-- tsconfig: `module` and `moduleResolution` should be `Node16` or `NodeNext` (SDK uses ES module exports with subpath imports)
-- Current tsconfig has `"module": "Node16"` — compatible
-
-### 6. Testing
-
-Use MCP Inspector for debugging:
-```bash
-npx @modelcontextprotocol/inspector npx tsx apps/mcp-server.ts
-```
-
-## Recommendation
-
-Minimal implementation — single file `apps/mcp-server.ts`:
-1. `BuildComponents()` at top level
-2. `server.registerTool("xindex_search", ...)` calling `searchContentIndex`
-3. `StdioServerTransport` + connect
-4. `.mcp.json` in project root for Claude Code discovery
-5. `bin/xindex-mcp` shebang entry point
-
-No build step needed (tsx). No express/HTTP needed (stdio only). ~40 lines of code.
-
-## Sources
-
-- [Official TypeScript SDK](https://github.com/modelcontextprotocol/typescript-sdk)
-- [SDK server docs](https://github.com/modelcontextprotocol/typescript-sdk/blob/main/docs/server.md)
-- [Claude Code MCP docs](https://code.claude.com/docs/en/mcp)
-- [Anthropic MCP server reference](https://github.com/anthropics/skills/blob/main/skills/mcp-builder/reference/node_mcp_server.md)
-- [npm package](https://www.npmjs.com/package/@modelcontextprotocol/sdk)