xindex 1.0.0 → 1.0.2

package/.ai/task/task.2026-04-10-line-clustering.md

@@ -1,176 +0,0 @@
-# Task: Line-level clustering for block-granular search
-
-## Context
-
-**Current state**: xindex indexes one vector per file. The `id` is the file path, keywords are extracted from the entire file content, and search returns file-level matches. This is too coarse — a 500-line file with mixed concerns returns as a single hit with no indication of *where* in the file the match is.
-
-**User's idea**: split files into semantically coherent blocks (clusters of lines), then index each block separately so search returns `file:fromLine-toLine` references.
-
-**Approach — extend existing pipeline with recursive bisection**:
-1. Keep existing file-level indexing intact — `indexContent(filePath, keywords, meta)` runs first, unchanged
-2. After file-level index: split file content into lines
-3. Bisect into 2 halves → extract keywords for each → embed → compute cosine similarity (dot product of normalized vectors)
-4. If similarity is high (≥ 0.6) → cohesive, no clustering needed. If low → 2 separate clusters.
-5. Recurse: split each cluster into 2 again, test overlap, stop when clusters are cohesive or hit limits (max depth 4 → up to 16 clusters, min 5 lines per cluster)
-6. If only 1 cluster (whole file is cohesive) → skip clustering, file-level entry is enough
-7. If 2+ clusters → index each in persistent Vectra as `<file>:<fromLine>-<toLine>` alongside the file-level entry
-8. Write a manifest at `<file>::manifest` tracking cluster IDs for cleanup on re-index
-9. Both file-level and cluster-level entries coexist — search may return both
-
-**Key files (change targets)**:
-- `componets/index/indexFileContent.ts` — **main change site**: currently calls `indexContent(id, keywords, meta)` once per file. Will call `clusterLines` then loop over clusters.
-- `componets/index/handleFileEvent.ts` — calls `removeContent(path)` on file change. Must delete all clusters for a file, not just one ID.
-- `componets/index/indexContent.ts` — low-level: embeds + upserts one item. No change needed — called per cluster.
-- `componets/index/removeContent.ts` — low-level: deletes one item. No change needed — called per cluster ID.
-- `componets/index/searchContentIndex.ts` — returns `IIndexRecord{score, id, meta}`. No change needed — `id` becomes `file:1-27` naturally.
-- `componets/index/indexMeta.ts` — `IIndexMeta{keywords, id}`. Add `type` tag, add `IClusterMeta`, `IFileManifest` using `IType<>` tagged union.
-- `componets/index/objectStore.ts` — stores `IIndexMeta` as JSON, keyed by MD5(id). Needs a manifest entry per file to track cluster IDs.
-- `componets/index/contentIndexDriver.ts` — wires components together. Must construct `ClusterLines`, `IndexFileContent`, `RemoveFileContent` inside. Currently `IndexFileContent` is constructed by callers.
-- `componets/buildComponents.ts` — top-level builder. Must return `indexFileContent` + `removeFileContent` from driver.
-- `apps/indexApp.ts` — bulk indexer. Calls `indexFileContent` directly via stream (no `HandleFileEvent`). Needs `removeFileContent` for cleanup.
-- `apps/run.index.ts`, `apps/run.watch.ts`, `apps/run.mcp.ts` — entry points. Currently construct `IndexFileContent` manually. Will use driver-provided version.
-- `componets/index/vectraIndex.ts` — creates `LocalIndex(path)`. No change.
-- `componets/llm/embed.ts` — MiniLM-L6 embeddings, returns `number[]`. No change.
-- `test-vectra-memory.ts` — proved VirtualFileStorage works for in-memory cosine queries.
-
-**Raw notes**: recursive split → 2 → 4 → 8 → 16 hard stop. Overlap by keywords via embedding cosine similarity. Final clusters get indexed in persistent store with line references. MCP query returns lines.
-
-## Goal
-
-Extend the existing indexing pipeline with a `ClusterLines` component (HOF pattern) that takes file content, splits it into semantically coherent line clusters using recursive bisection with embedding cosine similarity, and returns cluster descriptors `{fromLine, toLine, content, keywords}[]`. The existing file-level index stays intact — clustering adds block-level entries alongside it.
-
-## Diagram
-
-```
-handleFileEvent (file change/add)
-        │
-        ├── removeFileContent(path)              ◄── clean ALL old data first
-        │       ├── removeContent(path)               delete file-level vectra + meta
-        │       └── read manifest(path::manifest)     if exists:
-        │           ├── removeContent(path:1-10)        delete each cluster
-        │           ├── removeContent(path:11-25)
-        │           └── objectStore.remove(manifest)
-        │
-        └── indexFileContent(path, text)         ◄── create ALL new data
-                │
-                ├── EXISTING: file-level index (unchanged)
-                │   extractKeywords + cleanUpKeywords(text)
-                │   indexContent(path, keywords, {keywords, id: path})
-                │       ├── embed(keywords) → vector
-                │       ├── vectra.upsertItem({id: path, vector})
-                │       └── objectStore.write(path, meta)
-                │
-                ├── NEW: cluster-level index (extension)
-                │   clusterLines(lines, path)
-                │       │
-                │       ▼
-                │   ┌─────────────────┐
-                │   │  Split in half   │
-                │   │  lines[0..n/2]   │
-                │   │  lines[n/2..n]   │
-                │   └────────┬────────┘
-                │            │
-                │            ▼
-                │   ┌──────────────────────────┐
-                │   │  Extract keywords each    │
-                │   │  Embed keywords → vec     │
-                │   │  cosine(vecA, vecB)       │
-                │   └────────┬─────────────────┘
-                │            │
-                │       sim ≥ 0.6? ──yes──► 1 cluster (leaf)
-                │            │
-                │           no → recurse each half  ◄── depth ≤ 4, min 5 lines
-                │            │
-                │            ▼
-                │   clusters[] = {fromLine, toLine, content, keywords}[]
-                │
-                │   clusters.length ≤ 1? → SKIP (file entry is enough)
-                │
-                │   clusters.length > 1? → for each cluster:
-                │       indexContent(id="path:12-45", cluster.keywords, clusterMeta)
-                │
-                └── objectStore.write(path::manifest, {clusterIds})
-
-Three key types in store:
-  path           → file-level entry (vectra + objectStore)
-  path:1-10      → cluster entry (vectra + objectStore)
-  path::manifest → {type:"manifest", clusterIds} (objectStore only)
-```
-
-## Steps
-
-### 1. ClusterLines component — NEW `componets/index/clusterLines.ts`
-1. **Cosine helper** — `cosine(a: number[], b: number[]): number` — dot product of two normalized vectors. Pure function, no deps.
-2. **HOF factory** — `ClusterLines({embed, extractKeywords, cleanUpKeywords, threshold, minLines, maxDepth})` returns `IClusterLines(lines: string[], file: string) → Promise<ILineCluster[]>`. Defaults: threshold=0.6, minLines=5, maxDepth=4.
-3. **ILineCluster type** — `{fromLine: number, toLine: number, content: string, keywords: string}`. `fromLine`/`toLine` are 1-based line numbers.
-4. **Recursive bisection** — split lines at midpoint → join each half → `extractKeywords` + `cleanUpKeywords` → `embed` each → `cosine(vecA, vecB)`. If sim ≥ threshold → leaf cluster. If sim < threshold → recurse on each half.
-5. **Guards** — `lines.length ≤ minLines` or `depth ≥ maxDepth` → leaf. Empty lines → return `[]`. Either half has no keywords → leaf.
-
-### 2. Extend metadata — MODIFY `componets/index/indexMeta.ts` + `objectStore.ts`
-1. **Tag IIndexMeta** — add `type: "meta"` field using `IType<>` pattern: `IType<{type: "meta", keywords: string, id: string}>`. Breaking change — all constructors must add `type: "meta"`.
-2. **Add IClusterMeta type** — `IType<{type: "cluster", keywords: string, id: string, fromLine: number, toLine: number}>`. Cluster-level entries with line ranges.
-3. **Add IFileManifest type** — `IType<{type: "manifest", id: string, clusterIds: string[]}>`. Stored at key `filePath::manifest` in object store.
-4. **IStoreEntry union** — `IIndexMeta | IClusterMeta | IFileManifest`. Discriminated by `type` field.
-5. **Widen objectStore types** — `IObjectStore.write`/`read` accept/return `IStoreEntry`.
-6. **Update indexContent.ts** — widen `meta` param from `IIndexMeta` to `IIndexMeta | IClusterMeta`.
-7. **Update searchContentIndex.ts** — narrow `IStoreEntry` by `type` when reading results from object store.
-
-### 3. RemoveFileContent — NEW `componets/index/removeFileContent.ts`
-1. **HOF factory** — `RemoveFileContent({removeContent, objectStore})` returns `IRemoveFileContent(filePath: string) => Promise<void>`.
-2. **Deletes all layers** — (a) `removeContent(filePath)` to delete file-level vectra item + meta. (b) Read manifest at `filePath::manifest` → if exists, `removeContent(clusterId)` for each → `objectStore.remove(manifestKey)`. (c) All deletes wrapped in try/catch — missing entries are fine (first-time index, no clusters).
-
-### 4. Update indexFileContent — MODIFY `componets/index/indexFileContent.ts`
-1. **Add deps** — `{extractKeywords, cleanUpKeywords, indexContent, clusterLines, objectStore}`. Existing deps stay — file-level index needs `extractKeywords`/`cleanUpKeywords`.
-2. **File-level index (EXISTING, now tagged)** — `extractKeywords(content)` → `cleanUpKeywords` → `indexContent(id, keywords, {type: "meta", keywords, id})`. Runs first, always.
-3. **Cluster-level index (NEW, extension)** — `content.split("\n")` → `clusterLines(lines, id)` → if `clusters.length ≤ 1` → skip (file is cohesive). If `clusters.length > 1` → for each cluster: `indexContent(\`${id}:${fromLine}-${toLine}\`, cluster.keywords, {type: "cluster", ...})`.
-4. **Write manifest** — after all clusters indexed, `objectStore.write(id + "::manifest", {type: "manifest", id, clusterIds})`.
-
-### 5. Wire through driver + builder — MODIFY `contentIndexDriver.ts` + `buildComponents.ts`
-1. **contentIndexDriver.ts** — instantiate `ClusterLines({embed, extractKeywords, cleanUpKeywords})`. Construct `IndexFileContent({extractKeywords, cleanUpKeywords, indexContent, clusterLines, objectStore})` inside driver (currently constructed by callers). Construct `RemoveFileContent({removeContent, objectStore})`. Add `indexFileContent` + `removeFileContent` to `IContentIndexDriver`.
-2. **buildComponents.ts** — destructure `indexFileContent` + `removeFileContent` from `ContentIndexDriver`. Return them. Callers no longer construct `IndexFileContent` themselves.
-
-### 6. Update callers — MODIFY `run.*.ts` + `handleFileEvent.ts` + `indexApp.ts`
-1. **run.index.ts, run.watch.ts, run.mcp.ts** — remove `IndexFileContent(...)` construction. Get `indexFileContent` + `removeFileContent` from `BuildComponents()`.
-2. **handleFileEvent.ts** — replace `removeContent` dep with `removeFileContent`. On `FileEventType.index`: `removeFileContent(path)` first (clean old data), then `indexFileContent(path, text)` (creates file entry + cluster entries). On `FileEventType.remove`: `removeFileContent(path)`.
-3. **indexApp.ts** — currently calls `indexFileContent(id, text)` directly via stream pipeline (no `HandleFileEvent`). Add `removeFileContent` dep. Call `removeFileContent(id)` before `indexFileContent(id, text)` in the `map` callback — otherwise old clusters linger when cluster boundaries change on re-index. Update `IndexApp({walkFiles, indexFileContent, removeFileContent, log})`.
-4. **Import paths** — `run.index.ts` imports `IndexFileContent` from `componets/index/indexFileContent.js`. After moving construction inside driver, remove this import. Same for `run.watch.ts` and `run.mcp.ts`.
-
-### 7. Test end-to-end
-1. **Unit test clusterLines** — feed a file with 2 distinct sections (imports+types vs. implementation), verify ≥2 clusters with correct 1-based line ranges.
-2. **Integration test** — index a multi-concern file, query for a specific concept, verify search returns both `file.ts` (file-level) and `file.ts:12-45` (cluster-level).
-3. **Re-index test** — modify file, re-index, verify old clusters deleted + new ones created.
-4. **Cohesive file test** — index a small/uniform file, verify only file-level entry exists (no clusters, no manifest).
-
-## Decisions
-
-- **Extend, don't replace** — existing file-level indexing stays intact. Clustering is an additional step that runs after. Both levels coexist in the index.
-- **1 cluster = skip** — if the file is cohesive (clustering returns 1 cluster = whole file), no cluster entries are created. File-level entry is enough.
-- **Embedding cosine similarity** for bisection (not Jaccard). Jaccard only matches exact keyword strings — `fetchUser` and `getUser` would score 0% overlap despite being the same concern. Embeddings capture meaning. Cost is acceptable: MiniLM-L6 is local, ~30 embed calls per file at max depth, ~50-100ms total.
-- **Cosine computation**: Option A — direct dot product (3-line helper, vectors already normalized). Fallback to Option B (in-memory Vectra via `VirtualFileStorage`) if direct cosine proves insufficient.
-- **Similarity threshold**: start at 0.55–0.70, tune empirically. Try 0.6 as default.
-- **Min cluster size**: 3–5 lines. Use 5 as default, configurable.
-- **Three tagged types in store** (using `IType<>` pattern): `IIndexMeta{type:"meta"}` at `filePath`, `IClusterMeta{type:"cluster"}` at `filePath:fromLine-toLine`, `IFileManifest{type:"manifest"}` at `filePath::manifest`. All separate keys, discriminated by `type`.
-- **Cleanup on re-index**: `removeFileContent` deletes file-level entry, then reads manifest to delete all cluster entries, then deletes manifest itself. Graceful on missing data.
-- **Move IndexFileContent inside driver** — currently constructed by callers in `run.*.ts`. Moving inside `contentIndexDriver.ts` consolidates wiring since the driver already has all deps.
-
-## Research: existing NPM packages
-
-- **semantic-chunking** (jparkerweb, v2.4.4) — splits text into sentences, embeds each with ONNX model, groups by cosine similarity. Sentence-level, not line-level. Uses its own ONNX pipeline, not BYOE.
-- **semantic-chunker** (johnhenry) — BYOE approach, bring your own embedding function. More flexible. Could plug in our MiniLM-L6 embed.
-- **LangChain RecursiveCharacterTextSplitter** — recursive splitting by character/token boundaries, not semantic. 2026 benchmarks show 512-token recursive splitting at 69% accuracy — good baseline but not meaning-aware.
-- **NAACL 2025 finding**: fixed 200-word chunks match or beat semantic chunking for general RAG. But for *code* with mixed concerns in one file, semantic splitting should outperform fixed-size.
-- **Verdict**: existing packages target prose (sentence-level). Our use case is code (line-level, preserve line boundaries for references). Custom recursive bisection with our existing embed pipeline is the right call — simpler than adapting a prose chunker to respect line boundaries.
-
-## Edge Cases
-
-- **Small files (≤ 5 lines)** — return as single cluster, no splitting attempted.
-- **Empty files** — file-level entry is still indexed (existing pipeline runs first). Clustering returns `[]`, no cluster entries created.
-- **Files with uniform content** (e.g., all imports) — cosine similarity stays high at every split, returns 1 cluster. Expected behavior.
-- **Binary/non-text files** — already filtered upstream by the file walker. Not a concern here.
-- **Legacy index data** — files indexed before this change won't have manifests. On re-index, no old clusters to delete — just index fresh.
-
-## Open Questions
-
-- **Keyword extraction quality**: current keywords come from compromise NLP + keyword-extractor. May need tuning for code (variable names, imports, function signatures).
-- **Threshold tuning**: need to test 0.55 vs 0.60 vs 0.70 on real project files to find the sweet spot.
-- ~~**Object store dual use**~~ — resolved: three tagged types (`IIndexMeta`, `IClusterMeta`, `IFileManifest`) discriminated by `type` field, stored at separate keys. Union `IStoreEntry = IIndexMeta | IClusterMeta | IFileManifest`.

package/.ai/task/task.2026-04-10-object-store.log.md

@@ -1,7 +0,0 @@
-### 2026-04-10 — Task created
-
-- Scouted: vectra currently stores vector + IIndexMeta (keywords, file) together
-- User wants to separate: vectra for vectors only, .xindex/objects/ for meta JSON
-- Hash-based path: md5(id) → xx/yy/xxyyzz.json
-- Need to update indexContent, searchContentIndex, resetIndex, contentIndexDriver
-- New components: objectStore (read/write/clear), indexStructure (manage .xindex/ dirs)

package/.ai/task/task.2026-04-10-object-store.md

@@ -1,81 +0,0 @@
-# Task: Object Store — Separate Meta Storage from Vectra
-
-## Context
-
-Currently vectra stores both vectors AND metadata (`{keywords, file}`) in the same index. Vectra is good for semantic search, not for storage. Goal: split storage into two layers:
-
-- **`.xindex/semantic/`** — vectra stores only vectors + id (for search)
-- **`.xindex/objects/`** — file-based JSON store for meta objects (for storage/retrieval)
-
-**Current state:**
-- `indexContent.ts` — embeds content, upserts `{id, vector, metadata: IIndexMeta}` into vectra
-- `searchContentIndex.ts` — queries vectra, reads `r.item.metadata as IIndexMeta`
-- `resetIndex.ts` — `deleteIndex()` + `createIndex()` on vectra only
-- `IIndexMeta = {keywords: string, file: string}`
-- Index path: `.xindex` (single vectra folder)
-
-**New structure:**
-```
-.xindex/
-├── semantic/     ← vectra (vectors + id only, minimal meta)
-└── objects/      ← JSON files keyed by hash of id
-    └── xx/
-        └── yy/
-            └── xxyyzz.json  ← {keywords, file, ...}
-```
-
-## Goal
-
-Introduce an object store layer that writes `IIndexMeta` as JSON files in `.xindex/objects/`, remove metadata from vectra (keep only vector + id), and decorate `indexContent` and `searchContentIndex` to read/write both layers.
-
-## Diagram
-
-```
-INDEX PIPELINE:
-  file → extractKeywords → cleanUp → keywords
-    │
-    ├── [1] embed(keywords) → vector
-    │     └── vectra.upsert({id, vector})     → .xindex/semantic/
-    │
-    └── [2] objectStore.write(id, meta)       → .xindex/objects/xx/yy/xxyyzz.json
-                                                 {keywords, file}
-
-SEARCH PIPELINE:
-  query → extractKeywords → cleanUp → embed → vector
-    │
-    ├── [1] vectra.query(vector, limit)       → [{score, id}]
-    │
-    └── [2] objectStore.read(id)              → IIndexMeta
-                                                 ↓
-                                          [{score, id, meta}]
-
-RESET:
-  [1] vectra.deleteIndex + createIndex        → .xindex/semantic/ wiped
-  [2] rm -rf .xindex/objects/                 → objects wiped
-```
-
-## Steps
-
-### 1. Object Store HOF
-- Create `componets/index/objectStore.ts` — `ObjectStore({basePath}): IObjectStore`
-- `write(id, meta)` — hash id (md5 → hex), split into `xx/yy/xxyyzz`, `mkdir -p`, write JSON
-- `read(id)` — hash id, read JSON, parse as `IIndexMeta`
-- `clear()` — rm -rf basePath, recreate empty dir
-
-### 2. Update Index Structure
-- Create `componets/index/indexStructure.ts` — `IndexStructure({basePath}): IIndexStructure`
-- Manages `.xindex/` top-level: ensures `semantic/` and `objects/` dirs exist
-- Returns paths: `{semanticPath, objectsPath}`
-- Used by `contentIndexDriver` at init
-
-### 3. Decorate Index/Search
-- Update `IndexContent` — upsert vector+id to vectra (no meta), write meta to objectStore
-- Update `SearchContentIndex` — query vectra for `{score, id}[]`, then `objectStore.read(id)` for each result to attach meta
-- Update `ResetIndex` — call both `vectra.deleteIndex/createIndex` and `objectStore.clear()`
-- Update `ContentIndexDriver` — pass `semanticPath` to `VectraIndex`, create `ObjectStore({basePath: objectsPath})`
-
-## Open Questions
-
-- Hash function: `crypto.createHash('md5')` from Node built-in — fast enough, no deps. Or use simpler hash?
-- Should objectStore support partial updates (upsert) or always overwrite?
-- Should search batch-read objects or read one by one per result?

package/.ai/task/task.2026-04-10-search-config.log.md

@@ -1,46 +0,0 @@
-### 2026-04-10
-
-- Task created from user notes
-- Scouted codebase via xindex search (indexed 167 files)
-- `.xindex.json` exists but empty `{}`, no config loading anywhere
-- `CleanUpKeywords` at `componets/keywords/cleanUpKeywords.ts:8` — HOF takes `{maxNgrams, minLength}`. Add `ignoreKeywords` here.
-- `SearchContentIndex` at `componets/index/searchContentIndex.ts:12` — search pipeline, uses `cleanUpKeywords` on query. Ignore list propagates automatically.
-- `IClusterMeta` at `componets/index/indexMeta.ts:11` — has `fromLine`/`toLine` for reading snippet lines
-- MCP tool at `apps/mcpApp.ts:34` — `xindex_search` schema has `{query, limit}` only. Add snippet params.
-- CLI at `apps/run.search.ts:23-31` — formats results with score + keywords, no snippets
-- `BuildComponents` at `componets/buildComponents.ts:6` — wires everything, no config loading. Config loads here.
-- `ContentIndexDriver` at `componets/index/contentIndexDriver.ts:27` — passes `cleanUpKeywords` to `ClusterLines` and `SearchContentIndex`
-- Entry points: `apps/run.mcp.ts:19` (MCP), `apps/run.search.ts:8` (CLI) — both call `BuildComponents()`
-- User wants explicit config names: `ignoreKeywords`, `snippetLines`, `snippetResults`
-
-**Clarification round — decisions:**
-- Defaults confirmed: `snippetResults: 3`, `snippetLines: 7`
-- `ignoreKeywords`: exact strings, case-insensitive. No globs/patterns.
-- Ignore at **index time** — re-index + MCP restart after config change is acceptable. One-time setup, review in 3mo.
-- File-level results (whole file, no cluster) also get snippets if file total lines ≤ `snippetLines`
-- Task finalized
-
-**Round 2 — user feedback during detail expansion:**
-- Renamed `snippetLines` → `maxSnippetLines`, `snippetResults` → `maxSnippetResults` (user preference for explicit names)
-- Added `ignoreFiles` feature: gitignore-style glob patterns in `.xindex.json` to exclude files from indexing. Reuses existing `ignore` package already in `walkFiles.ts:3` and `watchFiles.ts`
-- Expanded task from 3x3 to 4x3 to accommodate file ignore list as separate step
-- Traced all WalkFiles/WatchFiles consumers: `run.mcp.ts`, `run.index.ts`, `run.watch.ts` — all need `ignoreFiles` plumbed
-- Task ready for implementation
-
-**Round 3 — consistency check (7 findings, all fixed):**
-- [Missing] `.xindex.json` is optional — added to Decisions + diagram label
-- [Drift] Diagram only showed WalkFiles — added WatchFiles
-- [Mismatch] Step 2.3 duplicated validation from 1.2 — removed 2.3, kept in 1.2 only
-- [Mismatch] `console.warn` in LoadConfig violates project `ILogger` pattern — added `log: ILogger` dep to LoadConfig and BuildComponents
-- [Drift] Files Changed table had tentative "(if it creates its own)" for run.index.ts — made definitive
-- [Inconsistency] Step 4.2 parsed fromLine/toLine from ID string — uses `meta.fromLine`/`meta.toLine` directly now
-- [Missing] Step 1.3 vague "WalkFiles consumers" — listed all 5 specific construction sites (run.mcp.ts:18,30, run.index.ts:10, run.watch.ts:13,14)
-
-**Round 4 — implementation:**
-- Implemented all 12 files (3 new, 9 modified) + run.reset.ts (missed in plan, also calls BuildComponents)
-- Phase 1: config type + loadConfig HOF
-- Phase 2: cleanUpKeywords ignoreSet, walkFiles + watchFiles ignoreFiles
-- Phase 3: readSnippet HOF
-- Phase 4: buildComponents wiring ({log} param, config loading, return config)
-- Phase 5: all entry points updated (run.mcp, run.search, run.index, run.watch, run.reset)
-- Verified: keyword ignore filters noisy words, file ignore excludes rnd/**, snippets show for small results (top 3, ≤7 lines)

package/.ai/task/task.2026-04-10-search-config.md

@@ -1,274 +0,0 @@
-# Task: Search Result Config — Keyword Ignore, File Ignore & Inline Code Snippets
-
-## Context
-
-Three improvements to xindex, all configurable via `.xindex.json`:
-
-1. **Keyword ignore list** — exclude noisy keywords at index time (case-insensitive exact match). Improves grouping relevance. Requires re-index after config change — acceptable as one-time setup.
-2. **File ignore list** — gitignore-style glob patterns to exclude files from indexing. Same semantics as `.gitignore` but defined in `.xindex.json`. Applied in `WalkFiles` and `WatchFiles` alongside existing `.gitignore` rules.
-3. **Inline code snippets** — when a search result is small (≤ N lines), include actual source code in the output. Configurable via `.xindex.json` defaults + MCP tool parameter overrides.
-
-**Current state:**
-- `.xindex.json` exists but is empty `{}` — file is optional, may not exist at all
-- Keywords: `compromise` NLP → `keyword-extractor` cleanup in `componets/keywords/cleanUpKeywords.ts`
-- File walking: `componets/walkFiles.ts` uses `ignore` package for `.gitignore` rules; `componets/watchFiles.ts` has its own `loadGitignore` + `ignore()` at line 22-31
-- Search results: 1-line summaries only (`1. path:from-to (score) — keywords`)
-- Cluster metadata stores `fromLine`/`toLine` in `componets/index/indexMeta.ts:11` (`IClusterMeta`)
-- MCP `xindex_search` accepts `query` and `limit` only
-- No config loading exists
-- Entry points that create `WalkFiles`/`WatchFiles`: `run.mcp.ts:18,30`, `run.index.ts:10`, `run.watch.ts:13-14`
-
-**Decisions:**
-- `.xindex.json` is **optional** — missing file → all defaults, no error
-- `ignoreKeywords`: exact strings, case-insensitive. No globs/patterns.
-- `ignoreFiles`: gitignore-style glob patterns (reuses existing `ignore` package)
-- `maxSnippetResults: 3`, `maxSnippetLines: 7` — confirmed defaults
-- Ignore list applied at **index time** — re-index + MCP restart required after config change. One-time setup, review in 3mo.
-- File-level results (no cluster) **also get snippets** if total file lines ≤ `maxSnippetLines`
-- Config field names are explicit: `ignoreKeywords`, `ignoreFiles`, `maxSnippetLines`, `maxSnippetResults`
-
-## Diagram
-
-```
-.xindex.json (optional)                   MCP xindex_search
-┌──────────────────────────┐              ┌──────────────────────────────┐
-│ ignoreKeywords: [...]    │              │ query, limit                 │
-│ ignoreFiles: [...]       │              │ maxSnippetResults: 3         │ ← override
-│ maxSnippetLines: 7       │              │ maxSnippetLines: 7           │ ← override
-│ maxSnippetResults: 3     │              └──────┬───────────────────────┘
-└──────┬───────────────────┘                     │
-       │                                         │
-       ├─ ignoreFiles ────┐                      │
-       │                   ▼                     │
-       │            WalkFiles + WatchFiles       │
-       │            (skip matching paths)        │
-       │                                         │
-       ├─ ignoreKeywords ─┐                      │
-       │                   ▼                     │
-       │           CleanUpKeywords               │
-       │           (index time)                  │
-       │                                         │
-       └─ snippet config ────────────────────────┤
-                                                  ▼
-                                          Format results
-                                          ├─ cluster ≤ maxSnippetLines? → readSnippet
-                                          └─ file ≤ maxSnippetLines?    → readSnippet
-
-Data flow (indexing):
-  walkFiles(inputs)                          ← ignoreFiles applied here (NEW)
-    → readFile
-    → ExtractKeywords (compromise NLP)
-    → CleanUpKeywords (keyword-extractor + ignoreKeywords filter)  ← NEW
-    → embed → vectra upsert + objectStore write
-
-Data flow (search):
-  query
-    → extractKeywords → cleanUpKeywords → embed → vectra query
-    → filter by scoreThreshold → objectStore.read for each hit
-    → format results + readSnippet for top N small results  ← NEW
-```
-
-## Steps
-
-### 1. Config schema & loading
-
-- **1.1 Define config type** — create `componets/config/xindexConfig.ts`
-  ```ts
-  export type IXindexConfig = {
-      ignoreKeywords: string[];
-      ignoreFiles: string[];
-      maxSnippetLines: number;
-      maxSnippetResults: number;
-  };
-  ```
-  All fields optional in the JSON file; defaults applied at load time.
-
-- **1.2 Load config** — create `componets/config/loadConfig.ts` as HOF
-  ```ts
-  export type ILoadConfig = () => Promise<IXindexConfig>;
-  export function LoadConfig({configPath, log}: {configPath: string, log: ILogger}): ILoadConfig
-  ```
-  - Read `configPath` (`.xindex.json` in cwd)
-  - `JSON.parse`, apply defaults: `{ignoreKeywords: [], ignoreFiles: [], maxSnippetLines: 7, maxSnippetResults: 3}`
-  - If file missing or empty → return all defaults (no error)
-  - If JSON parse fails → throw with clear message including path
-  - Validate: use `log` to warn if any `ignoreKeywords` entry has length ≤ 1 (no `console.*` — project uses `ILogger`)
-
-- **1.3 Wire into BuildComponents** — modify `componets/buildComponents.ts:6-22`
-  - Current: creates `embed`, `extractKeywords`, `cleanUpKeywords({maxNgrams: 2, minLength: 2})` then `ContentIndexDriver`
-  - `BuildComponents` currently takes no args. Add `{log}: {log: ILogger}` so `LoadConfig` can use it for warnings.
-  - Add: `const loadConfig = LoadConfig({configPath: ".xindex.json", log})` → `const config = await loadConfig()`
-  - Pass `config.ignoreKeywords` to `CleanUpKeywords`: `CleanUpKeywords({maxNgrams: 2, minLength: 2, ignoreKeywords: config.ignoreKeywords})`
-  - Return `config` in the output so callers can access snippet + file ignore settings
-  - `BuildComponents` return type gains `config: IXindexConfig`
-  - All callers need updating to pass `log` and destructure `config`:
-    - `apps/run.mcp.ts:19` — needs `config` for `McpApp` + `ignoreFiles` for `WalkFiles`/`WatchFiles`
-    - `apps/run.index.ts:11` — needs `config.ignoreFiles` for `WalkFiles`
-    - `apps/run.watch.ts:15` — needs `config.ignoreFiles` for `WalkFiles`/`WatchFiles`
-    - `apps/run.search.ts:8` — needs `config` for snippet settings
-
-### 2. Keyword ignore list
-
-- **2.1 Extend CleanUpKeywords** — modify `componets/keywords/cleanUpKeywords.ts:8`
-  - Current signature: `CleanUpKeywords({maxNgrams, minLength}: {maxNgrams: number, minLength: number})`
-  - New signature: `CleanUpKeywords({maxNgrams, minLength, ignoreKeywords = []}: {maxNgrams: number, minLength: number, ignoreKeywords?: string[]})`
-  - Build `ignoreSet = new Set(ignoreKeywords.map(k => k.toLowerCase()))` at factory time (once, not per call)
-  - Add `if (ignoreSet.has(lower)) return false;` into existing filter chain at line 21-27, before the `seen` dedup check
-
-  Exact change at `cleanUpKeywords.ts`:
-  ```ts
-  export function CleanUpKeywords({maxNgrams, minLength, ignoreKeywords = []}: {
-      maxNgrams: number, minLength: number, ignoreKeywords?: string[]
-  }): ICleanUpKeywords {
-      const ignoreSet = new Set(ignoreKeywords.map(k => k.toLowerCase()));
-      return function cleanUpKeywords(keywords) {
-          // ... existing extraction ...
-          const seen = new Set<string>();
-          return extracted.filter((kw: string) => {
-              if (kw.length <= minLength || !/[a-z]/i.test(kw)) return false;
-              const lower = kw.toLowerCase();
-              if (ignoreSet.has(lower)) return false;  // ← NEW
-              if (seen.has(lower)) return false;
-              seen.add(lower);
-              return true;
-          });
-      }
-  }
-  ```
-
-- **2.2 Propagation** — single change at `buildComponents.ts:9` propagates to all consumers:
-  - `ContentIndexDriver` (`contentIndexDriver.ts:28`) passes `cleanUpKeywords` to:
-    - `ClusterLines` (`clusterLines.ts:20`) — uses at `:34-35` (top/bot split keywords) and `:56` (leaf keywords)
-    - `IndexFileContent` (`indexFileContent.ts:10`) — uses at `:19` (file-level keywords)
-    - `SearchContentIndex` (`searchContentIndex.ts:12`) — uses at `:22` (query keywords)
-  - All paths share the same `cleanUpKeywords` instance. No additional wiring needed.
-
-### 3. File ignore list
-
-- **3.1 Extend WalkFiles** — modify `componets/walkFiles.ts:8`
-  - Current signature: `WalkFiles({cwd, log}: {cwd: string, log: ILogger})`
-  - New signature: `WalkFiles({cwd, log, ignoreFiles = []}: {cwd: string, log: ILogger, ignoreFiles?: string[]})`
-  - In `walk()` at line 18-22: the `ignore` instance `ig` is already constructed per-directory with accumulated `.gitignore` rules. Add `ignoreFiles` rules after existing rules:
-    ```ts
-    const ig = ignore();
-    for (const rule of rules) ig.add(rule);
-    for (const pattern of ignoreFiles) ig.add(pattern);  // ← NEW
-    ```
-  - This makes `ignoreFiles` patterns behave identically to `.gitignore` entries — same glob syntax, same matching semantics (relative paths, directory trailing `/`, negation with `!`)
-  - The `ignore` package is already a dependency (`package.json:28`)
-
-- **3.2 Extend WatchFiles** — modify `componets/watchFiles.ts:20`
-  - Current signature: `WatchFiles({cwd, log}: {cwd: string, log: ILogger})`
-  - New signature: `WatchFiles({cwd, log, ignoreFiles = []}: {cwd: string, log: ILogger, ignoreFiles?: string[]})`
-  - In `loadGitignore()` at line 22-31: creates its own `ignore()` instance per watched directory. Add `ignoreFiles` rules after `.gitignore` rules:
-    ```ts
-    async function loadGitignore(dir: string) {
-        const ig = ignore();
-        ig.add(".*");
-        try {
-            const content = await readFile(join(dir, ".gitignore"), "utf8");
-            ig.add(content);
-        } catch {}
-        for (const pattern of ignoreFiles) ig.add(pattern);  // ← NEW
-        return ig;
-    }
-    ```
-
-- **3.3 Wire ignoreFiles to all entry points** — pass `config.ignoreFiles` at each `WalkFiles`/`WatchFiles` construction:
-  - `apps/run.mcp.ts:18` — `WalkFiles({cwd, log})` → `WalkFiles({cwd, log, ignoreFiles: config.ignoreFiles})`
-  - `apps/run.mcp.ts:30` — `WatchFiles({cwd, log})` → `WatchFiles({cwd, log, ignoreFiles: config.ignoreFiles})`
-  - `apps/run.index.ts:10` — `WalkFiles({cwd, log})` → `WalkFiles({cwd, log, ignoreFiles: config.ignoreFiles})`
-  - `apps/run.watch.ts:13` — `WalkFiles({cwd, log})` → `WalkFiles({cwd, log, ignoreFiles: config.ignoreFiles})`
-  - `apps/run.watch.ts:14` — `WatchFiles({cwd, log})` → `WatchFiles({cwd, log, ignoreFiles: config.ignoreFiles})`
-
-### 4. Inline code snippets
-
-- **4.1 Add snippet params to MCP** — modify `apps/mcpApp.ts:34-58`
-  - `McpApp` factory gains `config: IXindexConfig` dependency (add to `mcpApp.ts:23` params)
-  - Extend `xindex_search` schema at line 37:
-    ```ts
-    inputSchema: z.object({
-        query: z.string().describe("Natural language search query"),
-        limit: z.number().int().min(1).max(100).default(10)
-            .describe("Max results to return, 10 by default, 100 max"),
-        maxSnippetResults: z.number().int().min(0).max(20).optional()
-            .describe("How many top results include inline code (default from .xindex.json, 3)"),
-        maxSnippetLines: z.number().int().min(0).max(50).optional()
-            .describe("Max lines in a result to qualify for inline code (default from .xindex.json, 7)"),
-    }),
-    ```
-  - In handler: resolve with config fallback:
-    ```ts
-    const sr = maxSnippetResults ?? config.maxSnippetResults;
-    const sl = maxSnippetLines ?? config.maxSnippetLines;
-    ```
-  - Update `apps/run.mcp.ts:48` — pass `config` to `McpApp`
-
-- **4.2 Read source lines** — create `componets/index/readSnippet.ts`
-  ```ts
-  export type IReadSnippet = (record: IIndexRecord, maxLines: number) => Promise<string | null>;
-  export function ReadSnippet(): IReadSnippet
-  ```
-  Logic:
-  - **Cluster result** (`meta.type === StoreEntryType.cluster`): use `meta.fromLine`/`meta.toLine` directly from `IClusterMeta` (no need to parse from ID). Compute `lineCount = meta.toLine - meta.fromLine + 1`. If `lineCount > maxLines` → return `null`. Extract file path from `record.id` by splitting on last `:` (format `"path/to/file.ts:14-27"`). `readFile(filePath, "utf8")`, split lines, slice `[meta.fromLine-1, meta.toLine]`, return joined with `\n`.
-  - **File result** (`meta.type === StoreEntryType.meta`): `readFile(record.id, "utf8")`, split lines, count. If `lineCount > maxLines` → return `null`. Otherwise return full content.
-  - **Error handling**: on `readFile` failure (file deleted, moved, permission error) → return `null` silently. Search results still display; snippet is just omitted.
-
-- **4.3 Format with code** — update result formatting in both entry points:
-
-  **MCP** (`apps/mcpApp.ts:47-51`): currently `results.map(...)` builds 1-line summaries. Replace with loop:
-  ```ts
-  const readSnippet = ReadSnippet();
-  const lines: string[] = [];
-  for (let i = 0; i < results.length; i++) {
-      const r = results[i];
-      const kw = r.meta.keywords ? ` — ${r.meta.keywords}` : "";
-      lines.push(`${i + 1}. ${r.id} (${r.score.toFixed(2)})${kw}`);
-      if (i < sr) {
-          const snippet = await readSnippet(r, sl);
-          if (snippet) lines.push("```\n" + snippet + "\n```");
-      }
-  }
-  ```
-
-  **CLI** (`apps/run.search.ts:8-31`): destructure config from `BuildComponents()`:
-  ```ts
-  const {searchContentIndex, config} = await BuildComponents({log});
-  ```
-  Then same snippet pattern using `config.maxSnippetResults` and `config.maxSnippetLines`:
-  ```ts
-  const readSnippet = ReadSnippet();
-  // ... in the result loop after existing log lines:
-  if (i < config.maxSnippetResults) {
-      const snippet = await readSnippet(results[i], config.maxSnippetLines);
-      if (snippet) log("```\n" + snippet + "\n```");
-  }
-  ```
-
-## Files Changed
-
-| File | Change |
-|------|--------|
-| `componets/config/xindexConfig.ts` | **NEW** — `IXindexConfig` type with 4 fields |
-| `componets/config/loadConfig.ts` | **NEW** — `LoadConfig` HOF, reads `.xindex.json` (optional), applies defaults, warns via `ILogger` |
-| `componets/keywords/cleanUpKeywords.ts` | Add `ignoreKeywords` param + `ignoreSet` filter |
-| `componets/walkFiles.ts` | Add `ignoreFiles` param, feed into `ignore()` instances |
-| `componets/watchFiles.ts` | Add `ignoreFiles` param, feed into `loadGitignore()` `ignore()` instance |
-| `componets/buildComponents.ts` | Add `{log}` param, load config, pass `ignoreKeywords` to `CleanUpKeywords`, return `config` |
-| `componets/index/readSnippet.ts` | **NEW** — `ReadSnippet` HOF, reads file lines for a search result |
-| `apps/mcpApp.ts` | Add `config` dep, `maxSnippetResults`/`maxSnippetLines` schema params, snippet formatting |
-| `apps/run.mcp.ts` | Pass `config` to `McpApp`, `ignoreFiles` to `WalkFiles`/`WatchFiles`, `log` to `BuildComponents` |
-| `apps/run.search.ts` | Pass `log` to `BuildComponents`, use `config` for snippet formatting |
-| `apps/run.index.ts` | Pass `log` to `BuildComponents`, `ignoreFiles` to `WalkFiles` |
-| `apps/run.watch.ts` | Pass `log` to `BuildComponents`, `ignoreFiles` to `WalkFiles`/`WatchFiles` |
-
-## Example `.xindex.json`
-
-```json
-{
-  "ignoreKeywords": ["import", "export", "const", "function", "return", "async", "await"],
-  "ignoreFiles": ["*.test.ts", "*.spec.ts", "rnd/**", "dist/**"],
-  "maxSnippetLines": 7,
-  "maxSnippetResults": 3
-}
-```

package/.ai/task/task.2026-04-10-watch-indexing.log.md

@@ -1,32 +0,0 @@
-# Log: xindex-watch — Continuous Indexing with File Watcher
-
-### 2026-04-10
-
-- Task created from user notes: "file watcher → apply to indexer → xindex-index runs → indexes provided or cwd → watches for changes → created/updated/moved/deleted → queued to stream → index content"
-- Scouted codebase: IndexApp already uses streamx pipeline (`from → tap → map → run`), WalkFiles is async generator, Writer supports push-based streaming, merge combines streams
-- Key decision: use `node:fs/promises` `watch()` (recursive, async iterable) — no external dep needed
-- Key decision: tagged union `{type:"index"|"remove", path}` as common event shape for walk + watch streams
-- Key decision: merge initial walk stream + watch stream into single pipeline
-- Debounce needed: editors fire multiple events per save (write temp → rename → delete old)
-- RemoveContent needed: Vectra has `deleteItem()` but no HOF wrapper exists yet
-- **Clarification round resolved:**
-  - Watch is always on (no optional flag) — index all, then watch. Default behavior.
-  - Default to cwd when no args
-  - Event→Vectra: created→add, updated→delete+add, deleted→delete, moved→delete(old)+add(new)
-  - Binary filtering: deferred (TODO), keep simple for now
-  - Graceful shutdown: SIGINT → stop processing → ignore queued → exit
-  - Watching individual files: works fine with fs.watch, no issue
-- **Consistency check:** fixed 6 issues — removed optional watch flag, added graceful shutdown to diagram/steps, clarified update=delete+add semantics, marked binary filtering as TODO
-- **User clarification:** separate entry points — `xindex-watch` (new, continuous) vs `xindex-index` (existing, one-time). Both default to cwd.
-- **Design decision:** Vectra `upsertItem` handles both add and update — no need for delete+add on updates, just upsert
-- **Design decision:** WatchApp is a new HOF in `apps/watchApp.ts`; IndexApp stays unchanged; no modifications to MCP/search paths
-- **Design decision:** WatchFiles uses `Writer<FileEvent>` to push events into streamx-compatible stream; `stop()` closes watchers + finishes writer
-- **Implementation pivot:** streamx `Writer`/`merge`/`batchTimed` depend on `@handy/fun` (not installed in xindex). Rewrote to use plain async generators instead — simpler, no new deps. Two-phase approach: walk+index first, then watch+process.
-- **Debounce approach:** collect events in Map (keyed by path, last event wins), flush after 150ms quiet period. Replaces batchTimed.
-- **Watcher uses AbortController** for clean shutdown — `fs.watch` accepts `signal` option natively.
-- **All steps implemented and verified:**
-  - Step 1: RemoveContent HOF + wired into ContentIndexDriver + BuildComponents ✓
-  - Step 2: WatchFiles component with debounced async generator ✓
-  - Step 3: WatchApp with two-phase (walk then watch) ✓
-  - Step 4: run.watch.ts + bin/xindex-watch + package.json + run.index.ts default ✓
-- **Tested:** initial index, file create detection, file delete detection, SIGINT graceful shutdown — all pass