xindex 1.0.4 → 1.0.6

package/.tmp/watch.log

@@ -0,0 +1,9 @@
+[xindex-1776976173089-3ab4dd18cfbfc8] started, indexing and watching: .
+indexing: notes.md
+indexing: util.ts
+initial index complete, acquiring watcher lock...
+done. indexed: {"indexedAmount":2}
+watching for changes: .
+index: wombat.md
+remove: functional
+remove batch failed: Error saving index: Error: ENOENT: no such file or directory, open '.xindex/semantic/index.json'

package/README.md

@@ -6,6 +6,8 @@ Local semantic code search for your codebase — plus an MCP server so Claude Co
 
 ## Install
 
+Requires **Node ≥22**.
+
 ```bash
 npm i -g xindex
 ```
@@ -39,6 +41,16 @@ Drop this into `.mcp.json` at your project root:
 
 Open the project in Claude Code — it picks up the xindex MCP server and can call `xindex_search`, `xindex_index`, and `xindex_reset` directly. Fewer hallucinations, fewer round-trips.
 
+## Features
+
+- **Local** — everything runs on your machine; embeddings cached on disk
+- **Semantic search** — natural-language queries, not substring match
+- **MCP server** — plugs into Claude Code via `.mcp.json`
+- **Watch mode** — keeps the index warm while you code
+- **Gitignore-aware** — respects `.gitignore` + custom ignore rules
+- **Zero config** — works with defaults; `.xindex.json` is optional
+- **Tolerant** — tolerates unreadable files, oversize files, empty files, binary files, and symlinks; each is skipped with a log line so the run always finishes
+
 ## Claude Code skills (`@xi`)
 
 Two optional [Claude Code skills](https://docs.claude.com/en/docs/claude-code/skills) wrap the MCP tools so you don't have to think about them:
@@ -73,13 +85,12 @@ argument-hint: "[question]"
 Surface-level codebase discovery via xindex. Tool: `xindex_search` (natural-language, meaning-based).
 
 **Steps:**
-1. Draft 5–10 focused queries from $ARGUMENTS (entry points, routing, config, integrations, tests, related patterns).
-2. Run `xindex_search` for each.
-3. If results are empty/sparse/stale → scoped-index the most relevant content-heavy root folders (one path per `xindex_index` call, e.g. `src`, `apps`, `features`, `componets`), then re-search. Prefer scoped over full-repo.
-4. Refine with 2–3 narrower follow-ups.
-5. Return file paths + brief keywords showing why each matched.
+1. Draft 3–7 queries from $ARGUMENTS (entry points, routing, config, tests, patterns); run `xindex_search` in parallel.
+2. If empty/sparse/stale → scoped `xindex_index` on relevant root folders (one path per call, e.g. `src`, `skills`, `agents`), then re-search. Prefer scoped over full-repo.
+3. Run 3–7 narrower follow-ups in parallel based on round-1 hits.
+4. Return file paths + brief keywords showing why each matched.
 
-Output = file links + keywords, not analysis. For reset or full re-index, delegate to `/xindex` (owns safety rules).
+Output = file links + keywords, not analysis. **Escalate to `/ask-cursor` by default** (cheap codebase reasoning); only go to `/ask-claude` for multi-file/pattern analysis or `/ask-claude-opus` for trade-offs. For reset or full re-index, delegate to `/xindex` (owns safety rules).
 ````
 
 `xindex/SKILL.md`:
@@ -90,7 +101,7 @@ name: xindex
 description: Manages xindex semantic search — index, search, reset via MCP tools. For research questions, use /ask-xi.
 argument-hint: "[search query | index | reset]"
 ---
-Full xindex tool management. For research, use `/ask-xi`. Install: `npm i -g xindex`.
+Full xindex tool management. For research, use `/ask-xi`. Install: `npm i -g xindex` ([npm](https://www.npmjs.com/package/xindex)).
 
 **Tools:**
 - `xindex_search` — find files by meaning (synonyms, semantics). Try before grepping blindly.
@@ -106,15 +117,6 @@ $ARGUMENTS
 
 Both skills assume the `xindex` MCP server is registered (see the section above). Restart Claude Code after adding skills.
 
-## Features
-
-- **Local** — everything runs on your machine; embeddings cached on disk
-- **Semantic search** — natural-language queries, not substring match
-- **MCP server** — plugs into Claude Code via `.mcp.json`
-- **Watch mode** — keeps the index warm while you code
-- **Gitignore-aware** — respects `.gitignore` + custom ignore rules
-- **Zero config** — works with defaults; `.xindex.json` is optional
-
 ---
 
 ## CLI reference
@@ -129,7 +131,7 @@ xindex-index apps features
 ```
 
 ### `xindex-search <query...>`
-Search the index. All args are joined into one query. Default limit: 10.
+Search the index. All args are joined into one query. Default limit: 7.
 ```bash
 xindex-search "database migration logic"
 xindex-search file watcher debounce
@@ -157,19 +159,23 @@ xindex-mcp --watch-dir=./src # watch a specific dir
 
 ## MCP tools
 
-| Tool | What it does | Input |
-|------|--------------|-------|
-| `xindex_search` | Semantic search | `query: string`, `limit?: number` (default 5, max 100) |
-| `xindex_index`  | Index paths | `inputs: string[]` (at least one) |
-| `xindex_reset`  | Wipe index (destructive) | — |
+- **`xindex_search`** — semantic search. `query: string`, `limit?: number` (default 7, max 50)
+- **`xindex_index`** — index paths. `inputs: string[]` (at least one)
+- **`xindex_reset`** — wipe index (destructive). No input
 
-Note: CLI `xindex-search` defaults to 10 results; MCP `xindex_search` defaults to 5.
+Note: both CLI `xindex-search` and MCP `xindex_search` default to 7 results; MCP caps at 50.
 
 ## Configuration
 
 ### `.xindex.json` (optional)
 
-Place at your project root. Both fields are optional arrays; unknown keys are ignored.
+Project-root file. All fields optional; unknown keys ignored; missing/empty → defaults.
+
+- **`ignoreKeywords`** — `string[]`, default `[]`. Tokens stripped before embedding — add project slang/boilerplate polluting results. Entries ≤1 char warn.
+- **`ignoreFiles`** — `string[]`, default `[]`. Extra globs excluded during walk/watch, on top of `.gitignore` — add vendored/generated folders.
+- **`maxLines`** — `number`, default `30`. Lines per chunk — tune if chunks feel over/under-sized.
+- **`maxFileBytes`** — `number`, default `1000000`. Skip files over this (1 MB) — raise to index larger generated files.
+- **`followSymlinks`** — `boolean`, default `false`. When `false`, symbolic links encountered during walk/watch are skipped with a log line. Set `true` to follow them (cycles are broken via `realpath` dedup).
 
 ```json
 {
@@ -178,8 +184,7 @@ Place at your project root. Both fields are optional arrays; unknown keys are ig
 }
 ```
 
-- **`ignoreKeywords`** — tokens stripped before embedding (noise words, language keywords, project slang). Keeps search focused on meaningful terms.
-- **`ignoreFiles`** — extra glob patterns excluded during walk/watch, on top of `.gitignore`.
+Override only what you need; re-run `xindex-index .` (or let the watcher pick it up). Invalid JSON throws; wrong-typed fields fall back silently.
 
 ### `.xindex/` folder
 

package/apps/indexApp.ts

@@ -1,4 +1,3 @@
-import {readFile} from "fs/promises";
 import {from} from "../packages/streamx/src/from.js";
 import {batch} from "../packages/streamx/src/batch.js";
 import {map} from "../packages/streamx/src/map.js";
@@ -8,23 +7,24 @@ import {IWalkFiles} from "../componets/walkFiles.js";
 import {IIndexContent} from "../features/indexContent.js";
 import {ILogger} from "../componets/logger.js";
 import {INDEXING_BATCH_SIZE} from "../componets/config/INDEXING_BATCH_SIZE";
+import {SafeIndexBatch} from "../componets/io/safeIndexBatch.js";
 
 export type IIndexApp = (inputs: string[]) => Promise<void>;
 
-export function IndexApp({walkFiles, indexContent, log}: {
+export function IndexApp({walkFiles, indexContent, log, maxFileBytes}: {
     walkFiles: IWalkFiles,
     indexContent: IIndexContent,
     log: ILogger,
+    maxFileBytes: number,
 }): IIndexApp {
+    const safeIndexBatch = SafeIndexBatch({indexContent, log, maxFileBytes});
     return async function indexApp(inputs) {
         await run(
             from(walkFiles(inputs))
                 .pipe(tap(id => log(`indexing: ${id}`)))
                 .pipe(batch(INDEXING_BATCH_SIZE))
                 .pipe(map<string[], string[]>(async (ids) => {
-                    const texts = await Promise.all(ids.map(id => readFile(id, "utf8")));
-                    const items = ids.map((id, i) => ({id, content: `${texts[i]}. ${id}`}));
-                    await indexContent(items);
+                    await safeIndexBatch(ids);
                     return ids;
                 }))
         );

package/apps/run.index.ts

@@ -8,8 +8,8 @@ const appId = AppId();
 const cwd = process.cwd();
 const log = BufferedLoggerToStdOut();
 const {indexContent, getIndexStats, config} = await BuildComponents({log});
-const walkFiles = WalkFiles({cwd, log, ignoreFiles: config.ignoreFiles});
-const indexApp = IndexApp({walkFiles, indexContent, log});
+const walkFiles = WalkFiles({cwd, log, ignoreFiles: config.ignoreFiles, followSymlinks: config.followSymlinks});
+const indexApp = IndexApp({walkFiles, indexContent, log, maxFileBytes: config.maxFileBytes});
 
 const inputs = process.argv.slice(2);
 if (!inputs.length) inputs.push(".");

package/apps/run.mcp.ts

@@ -17,8 +17,8 @@ const watchDisabled = args.includes("--watch-disabled");
 const cwd = process.cwd();
 const log = BufferedLoggerToStdErr();
 const {indexContent, removeContent, getIndexStats, searchContentIndex, resetIndex, config} = await BuildComponents({log});
-const walkFiles = WalkFiles({cwd, log, ignoreFiles: config.ignoreFiles});
-const indexApp = IndexApp({walkFiles, indexContent, log});
+const walkFiles = WalkFiles({cwd, log, ignoreFiles: config.ignoreFiles, followSymlinks: config.followSymlinks});
+const indexApp = IndexApp({walkFiles, indexContent, log, maxFileBytes: config.maxFileBytes});
 const search = SearchApp({searchContentIndex});
 
 const appId = AppId();
@@ -31,7 +31,7 @@ const watcherLock = WatcherLock({
 const watch = watchDisabled ? undefined : {
     watchFiles: WatchFiles({cwd, log, ignoreFiles: config.ignoreFiles}),
     watchDir: watchDirArg ? watchDirArg.split("=")[1] : ".",
-    handleFileEvents: HandleFileEvents({indexContent, removeContent, log}),
+    handleFileEvents: HandleFileEvents({indexContent, removeContent, log, maxFileBytes: config.maxFileBytes}),
     watcherLock,
 };
 

package/apps/run.watch.ts

@@ -11,9 +11,9 @@ import {join} from "path";
 const cwd = process.cwd();
 const log = BufferedLoggerToStdOut();
 const {indexContent, removeContent, getIndexStats, config} = await BuildComponents({log});
-const walkFiles = WalkFiles({cwd, log, ignoreFiles: config.ignoreFiles});
+const walkFiles = WalkFiles({cwd, log, ignoreFiles: config.ignoreFiles, followSymlinks: config.followSymlinks});
 const watchFiles = WatchFiles({cwd, log, ignoreFiles: config.ignoreFiles});
-const handleFileEvents = HandleFileEvents({indexContent, removeContent, log});
+const handleFileEvents = HandleFileEvents({indexContent, removeContent, log, maxFileBytes: config.maxFileBytes});
 
 const appId = AppId();
 const watcherLock = WatcherLock({
@@ -22,7 +22,7 @@ const watcherLock = WatcherLock({
     log,
 });
 
-const app = WatchApp({walkFiles, watchFiles, handleFileEvents, indexContent, log, watcherLock});
+const app = WatchApp({walkFiles, watchFiles, handleFileEvents, indexContent, log, watcherLock, maxFileBytes: config.maxFileBytes});
 
 const inputs = process.argv.slice(2);
 if (!inputs.length) inputs.push(".");

package/apps/watchApp.ts

@@ -3,7 +3,6 @@ import {batch} from "../packages/streamx/src/batch.js";
 import {map} from "../packages/streamx/src/map.js";
 import {tap} from "../packages/streamx/src/tap.js";
 import {run} from "../packages/streamx/src/index.js";
-import {readFile} from "fs/promises";
 import {IWalkFiles} from "../componets/walkFiles.js";
 import {IWatchFiles} from "../componets/watchFiles.js";
 import {IHandleFileEvents} from "../componets/index/handleFileEvent.js";
@@ -12,20 +11,23 @@ import {IWatcherLock} from "../componets/index/watcherLock.js";
 import {WatchFileEventsApp} from "./watchFileEventsApp.js";
 import {IIndexContent} from "../features/indexContent.js";
 import {INDEXING_BATCH_SIZE} from "../componets/config/INDEXING_BATCH_SIZE";
+import {SafeIndexBatch} from "../componets/io/safeIndexBatch.js";
 
 export type IWatchApp = {
     run: (inputs: string[]) => Promise<void>;
     stop: () => void;
 };
 
-export function WatchApp({walkFiles, watchFiles, handleFileEvents, indexContent, log, watcherLock}: {
+export function WatchApp({walkFiles, watchFiles, handleFileEvents, indexContent, log, watcherLock, maxFileBytes}: {
     walkFiles: IWalkFiles,
     watchFiles: IWatchFiles,
     handleFileEvents: IHandleFileEvents,
     indexContent: IIndexContent,
     log: ILogger,
     watcherLock: IWatcherLock,
+    maxFileBytes: number,
 }): IWatchApp {
+    const safeIndexBatch = SafeIndexBatch({indexContent, log, maxFileBytes});
     return {
         async run(inputs) {
             // Phase 1: initial index — walk all files
@@ -34,9 +36,7 @@ export function WatchApp({walkFiles, watchFiles, handleFileEvents, indexContent,
                     .pipe(tap(id => log(`indexing: ${id}`)))
                     .pipe(batch(INDEXING_BATCH_SIZE))
                     .pipe(map<string[], string[]>(async (ids) => {
-                        const texts = await Promise.all(ids.map(id => readFile(id, "utf8")));
-                        const items = ids.map((id, i) => ({id, content: `${texts[i]}. ${id}`}));
-                        await indexContent(items);
+                        await safeIndexBatch(ids);
                         return ids;
                     }))
             );

package/componets/buildComponents.ts

@@ -33,7 +33,8 @@ export async function BuildComponents({log}: { log: ILogger }) {
         extractKeywords,
         cleanUpKeywords,
         locateInFile,
-        scoreThreshold: SCORE_THRESHOLD
+        scoreThreshold: SCORE_THRESHOLD,
+        log,
     });
 
     return {

package/componets/config/INDEXING_BATCH_SIZE.ts

@@ -1 +1 @@
-export const INDEXING_BATCH_SIZE = 5;
+export const INDEXING_BATCH_SIZE = 16;

package/componets/config/WATCH_FLUSH_MS.ts

@@ -1 +1 @@
-export const WATCH_FLUSH_MS = 500;
+export const WATCH_FLUSH_MS = 300;

package/componets/config/loadConfig.ts

@@ -3,13 +3,15 @@ import {IXindexConfig} from "./xindexConfig.js";
 import {ILogger} from "../logger.js";
 
 const DEFAULT_MAX_LINES = 30;
-const DEFAULT_MAX_FILE_BYTES = 5_000_000;
+const DEFAULT_MAX_FILE_BYTES = 1_000_000;
+const DEFAULT_FOLLOW_SYMLINKS = false;
 
 const DEFAULTS: IXindexConfig = {
     ignoreKeywords: [],
     ignoreFiles: [],
     maxLines: DEFAULT_MAX_LINES,
     maxFileBytes: DEFAULT_MAX_FILE_BYTES,
+    followSymlinks: DEFAULT_FOLLOW_SYMLINKS,
 };
 
 export type ILoadConfig = () => Promise<IXindexConfig>;
@@ -41,11 +43,18 @@ export function LoadConfig({configPath, log}: { configPath: string, log: ILogger
             ignoreFiles: toStrings(parsed.ignoreFiles),
             maxLines: toNum(parsed.maxLines, DEFAULT_MAX_LINES),
             maxFileBytes: toNum(parsed.maxFileBytes, DEFAULT_MAX_FILE_BYTES),
+            followSymlinks: typeof parsed.followSymlinks === "boolean" ? parsed.followSymlinks : DEFAULT_FOLLOW_SYMLINKS,
         };
 
         for (const kw of config.ignoreKeywords) {
             if (kw.length <= 1) log(`warning: ignoreKeywords entry "${kw}" is <=1 char`);
         }
+        if (config.maxFileBytes < 1024) {
+            log(`warning: maxFileBytes (${config.maxFileBytes}) < 1024 — likely a typo`);
+        }
+        if (parsed.followSymlinks !== undefined && typeof parsed.followSymlinks !== "boolean") {
+            log(`warning: followSymlinks must be boolean — using default (${DEFAULT_FOLLOW_SYMLINKS})`);
+        }
 
         return config;
     };

package/componets/config/xindexConfig.ts

@@ -3,4 +3,5 @@ export type IXindexConfig = {
     ignoreFiles: string[];
     maxLines: number;
     maxFileBytes: number;
+    followSymlinks: boolean;
 };

package/componets/index/contentIndexDriver.ts

@@ -9,6 +9,7 @@ import {ResetIndex, IResetIndex} from "../../features/resetIndex.js";
 import {VectraIndex} from "./vectraIndex.js";
 import {IndexApi} from "./indexApi.js";
 import {ILocateInFile} from "../locate/locateInFile.js";
+import {ILogger} from "../logger.js";
 
 export type IContentIndexDriver = Readonly<{
     getIndexStats: IGetIndexStats,
@@ -19,20 +20,21 @@ export type IContentIndexDriver = Readonly<{
     flush: () => Promise<void>,
 }>;
 
-export async function ContentIndexDriver({path, embed, extractKeywords, cleanUpKeywords, locateInFile, scoreThreshold}: {
+export async function ContentIndexDriver({path, embed, extractKeywords, cleanUpKeywords, locateInFile, scoreThreshold, log}: {
     path: string,
     embed: IEmbed,
     extractKeywords: IExtractKeywords,
     cleanUpKeywords: ICleanUpKeywords,
     locateInFile: ILocateInFile,
     scoreThreshold: number,
+    log: ILogger,
 }): Promise<IContentIndexDriver> {
     const index = await VectraIndex(path + "/semantic");
     const indexApi = IndexApi({index});
 
     return {
         getIndexStats: GetIndexStats({index}),
-        indexContent: IndexContent({extractKeywords, cleanUpKeywords, embed, indexApi}),
+        indexContent: IndexContent({extractKeywords, cleanUpKeywords, embed, indexApi, log}),
         removeContent: RemoveContent({indexApi}),
         searchContentIndex: SearchIndex({extractKeywords, cleanUpKeywords, embed, index, locateInFile, scoreThreshold}),
         resetIndex: ResetIndex({indexApi}),

package/componets/index/handleFileEvent.ts

@@ -1,22 +1,22 @@
-import {readFile} from "fs/promises";
 import {IIndexContent} from "../../features/indexContent.js";
 import {IRemoveContent} from "../../features/removeContent.js";
 import {ILogger} from "../logger.js";
 import {FileEventType, IFileEvent} from "../watchFiles.js";
+import {SafeIndexBatch} from "../io/safeIndexBatch.js";
 
 export type IHandleFileEvent = (event: IFileEvent) => Promise<void>;
 export type IHandleFileEvents = (events: IFileEvent[]) => Promise<void>;
 
-export function HandleFileEvent({indexContent, removeContent, log}: {
+export function HandleFileEvent({indexContent, removeContent, log, maxFileBytes}: {
     indexContent: IIndexContent,
     removeContent: IRemoveContent,
     log: ILogger,
+    maxFileBytes: number,
 }): IHandleFileEvent {
+    const safeIndexBatch = SafeIndexBatch({indexContent, log, maxFileBytes});
     return async function handleFileEvent(event) {
         if (event.type === FileEventType.index) {
-            const text = await readFile(event.path, "utf8");
-            await indexContent([{id: event.path, content: `${text}. ${event.path}`}]);
-            log(`index: ${event.path}`);
+            await safeIndexBatch([event.path]);
         } else {
             try { await removeContent([event.path]); } catch (e) { log(`remove failed: ${event.path} — ${(e as any)?.message ?? e}`); }
             log(`remove: ${event.path}`);
@@ -24,42 +24,22 @@ export function HandleFileEvent({indexContent, removeContent, log}: {
     };
 }
 
-export function HandleFileEvents({indexContent, removeContent, log}: {
+export function HandleFileEvents({indexContent, removeContent, log, maxFileBytes}: {
     indexContent: IIndexContent,
     removeContent: IRemoveContent,
     log: ILogger,
+    maxFileBytes: number,
 }): IHandleFileEvents {
+    const safeIndexBatch = SafeIndexBatch({indexContent, log, maxFileBytes});
     return async function handleFileEvents(events) {
-        const indexEvents = events.filter((event) => event.type === FileEventType.index);
-        const removeEvents = events.filter((event) => event.type === FileEventType.remove);
+        const indexPaths = events.filter(e => e.type === FileEventType.index).map(e => e.path);
+        const removePaths = events.filter(e => e.type === FileEventType.remove).map(e => e.path);
 
-        if (indexEvents.length > 0) {
-            const indexItems = (await Promise.all(indexEvents.map(async (event) => {
-                log(`index: ${event.path}`);
-                try {
-                    const text = await readFile(event.path, "utf8");
-                    return {id: event.path, content: `${text}. ${event.path}`};
-                } catch (e) {
-                    log(`index failed: ${event.path} — ${(e as any)?.message ?? e}`);
-                    return undefined;
-                }
-            }))).filter((item): item is { id: string, content: string } => !!item);
-
-            if (indexItems.length > 0) {
-                try {
-                    await indexContent(indexItems);
-                } catch (e) {
-                    log(`index batch failed: ${(e as any)?.message ?? e}`);
-                }
-            }
-        }
-
-        if (removeEvents.length > 0) {
-            const removePaths = removeEvents.map((event) => {
-                log(`remove: ${event.path}`);
-                return event.path;
-            });
+        for (const p of indexPaths) log(`index: ${p}`);
+        if (indexPaths.length > 0) await safeIndexBatch(indexPaths);
 
+        if (removePaths.length > 0) {
+            for (const p of removePaths) log(`remove: ${p}`);
             try {
                 await removeContent(removePaths);
             } catch (e) {

package/componets/io/safeIndexBatch.ts

@@ -0,0 +1,30 @@
+import {safeReadFile, SafeReadResultType} from "./safeReadFile.js";
+import {IIndexContent} from "../../features/indexContent.js";
+import {ILogger} from "../logger.js";
+
+export type ISafeIndexBatch = (ids: string[]) => Promise<void>;
+
+export function SafeIndexBatch({indexContent, log, maxFileBytes}: {
+    indexContent: IIndexContent,
+    log: ILogger,
+    maxFileBytes: number,
+}): ISafeIndexBatch {
+    return async function safeIndexBatch(ids) {
+        const results = await Promise.all(ids.map(id => safeReadFile(id, maxFileBytes)));
+        const ok: {id: string, content: string}[] = [];
+        for (const r of results) {
+            if (r.type === SafeReadResultType.ok) {
+                ok.push({id: r.id, content: `${r.content}. ${r.id}`});
+            } else {
+                log(`skip ${r.id}: ${r.reason}`);
+            }
+        }
+        if (ok.length > 0) {
+            try {
+                await indexContent(ok);
+            } catch (e: any) {
+                log(`index batch failed: ${e?.message ?? e}`);
+            }
+        }
+    };
+}

package/componets/io/safeReadFile.ts

@@ -0,0 +1,36 @@
+import {stat, readFile} from "fs/promises";
+import {IType} from "../IType.js";
+
+export enum SafeReadResultType {
+    ok = "ok",
+    skip = "skip",
+    error = "error",
+}
+
+export enum SafeReadSkipReason {
+    tooLarge = "tooLarge",
+    empty = "empty",
+    binary = "binary",
+}
+
+export type ISafeReadResult =
+    | IType<{ type: SafeReadResultType.ok, id: string, content: string }>
+    | IType<{ type: SafeReadResultType.skip, id: string, reason: SafeReadSkipReason }>
+    | IType<{ type: SafeReadResultType.error, id: string, reason: string }>;
+
+export async function safeReadFile(id: string, maxBytes: number): Promise<ISafeReadResult> {
+    try {
+        const s = await stat(id);
+        if (s.size > maxBytes) return {type: SafeReadResultType.skip, id, reason: SafeReadSkipReason.tooLarge};
+        const buf = await readFile(id);
+        const probe = buf.subarray(0, Math.min(10, buf.length));
+        for (let i = 0; i < probe.length; i++) {
+            if (probe[i] === 0) return {type: SafeReadResultType.skip, id, reason: SafeReadSkipReason.binary};
+        }
+        const content = buf.toString("utf8");
+        if (!content.trim()) return {type: SafeReadResultType.skip, id, reason: SafeReadSkipReason.empty};
+        return {type: SafeReadResultType.ok, id, content};
+    } catch (e: any) {
+        return {type: SafeReadResultType.error, id, reason: e?.code ?? e?.message ?? String(e)};
+    }
+}

package/componets/locate/locateInFile.ts

@@ -11,9 +11,9 @@ import {filter} from "../../packages/streamx/src/filter.js";
 import {map} from "../../packages/streamx/src/map.js";
 import {flatMap} from "../../packages/streamx/src/flatMap.js";
 import {tap} from "../../packages/streamx/src/tap.js";
-import {scaleSync} from "../../packages/streamx/src/scaleSync.js";
 import {run} from "../../packages/streamx/src/index.js";
 import {DEFAULT_LOCATE_BATCH_SIZE} from "../config/DEFAULT_LOCATE_BATCH_SIZE";
+import {scaleSync} from "../../packages/streamx/src/scaleSync";
 
 export type ILocateInFile = (
     query: string,
@@ -61,12 +61,17 @@ export function LocateInFile({
                         return false;
                     }
                 }))
-                .pipe(scaleSync(SCALE_FILE_READS, async (id: string) => {
-                    const text = await readFile(id, "utf8");
-                    return {id, text};
+                .pipe(scaleSync(SCALE_FILE_READS, async (id: string): Promise<{ id: string, text: string } | null> => {
+                    try {
+                        const text = await readFile(id, "utf8");
+                        return {id, text};
+                    } catch {
+                        return null;
+                    }
                 }))
-                .pipe(flatMap(({id, text}: { id: string, text: string }): IWindow[] => {
-                    return windowsOf({text, id, windowLines});
+                .pipe(filter(async (r: { id: string, text: string } | null) => r !== null))
+                .pipe(flatMap((r: { id: string, text: string } | null): IWindow[] => {
+                    return windowsOf({text: r!.text, id: r!.id, windowLines});
                 }));
 
             const withVectors = from<IWindow>(windows)

package/componets/walkFiles.ts

@@ -1,11 +1,16 @@
-import {readdir, stat, readFile} from "fs/promises";
+import {readdir, stat, readFile, realpath} from "fs/promises";
 import {join, relative} from "path";
 import ignore from "ignore";
 import {ILogger} from "./logger.js";
 
 export type IWalkFiles = (inputs: string[]) => AsyncIterable<string>;
 
-export function WalkFiles({cwd, log, ignoreFiles = []}: {cwd: string, log: ILogger, ignoreFiles?: string[]}): IWalkFiles {
+export function WalkFiles({cwd, log, ignoreFiles = [], followSymlinks = false}: {
+    cwd: string,
+    log: ILogger,
+    ignoreFiles?: string[],
+    followSymlinks?: boolean,
+}): IWalkFiles {
 
     async function tryReadGitignore(dir: string): Promise<string> {
         try {
@@ -15,6 +20,8 @@ export function WalkFiles({cwd, log, ignoreFiles = []}: {cwd: string, log: ILogg
         }
     }
 
+    const visited = new Set<string>();
+
     async function* walk(dir: string, parentRules: string[]): AsyncIterable<string> {
         const localGitignore = await tryReadGitignore(dir);
         const rules = localGitignore ? [...parentRules, localGitignore] : parentRules;
@@ -27,11 +34,35 @@ export function WalkFiles({cwd, log, ignoreFiles = []}: {cwd: string, log: ILogg
             const abs = join(dir, entry.name);
             const rel = relative(cwd, abs);
 
+            if (entry.isSymbolicLink()) {
+                if (!followSymlinks) {
+                    log(`skip symlink ${rel}`);
+                    continue;
+                }
+                try {
+                    const real = await realpath(abs);
+                    if (visited.has(real)) continue;
+                    visited.add(real);
+                    const s = await stat(real);
+                    if (s.isDirectory()) {
+                        if (entry.name.startsWith(".")) continue;
+                        if (ig.ignores(rel + "/")) continue;
+                        yield* walk(abs, rules);
+                    } else if (s.isFile()) {
+                        if (ig.ignores(rel)) continue;
+                        yield rel;
+                    }
+                } catch (e: any) {
+                    log(`skip symlink ${rel}: ${e?.code ?? e?.message ?? e}`);
+                }
+                continue;
+            }
+
             if (entry.isDirectory()) {
                 if (entry.name.startsWith(".")) continue;
                 if (ig.ignores(rel + "/")) continue;
                 yield* walk(abs, rules);
-            } else {
+            } else if (entry.isFile()) {
                 if (ig.ignores(rel)) continue;
                 yield rel;
             }

package/features/indexContent.ts

@@ -2,22 +2,31 @@ import {IndexCommandType, IIndexApi} from "../componets/index/indexApi.js";
 import {IExtractKeywords} from "../componets/keywords/extractKeywords.js";
 import {ICleanUpKeywords} from "../componets/keywords/cleanUpKeywords.js";
 import {IEmbed} from "../componets/llm/embed.js";
+import {ILogger} from "../componets/logger.js";
 
 export type IIndexContent = (items: Array<{id: string, content: string}>) => Promise<void>;
 
-export function IndexContent({extractKeywords, cleanUpKeywords, embed, indexApi}: {
+export function IndexContent({extractKeywords, cleanUpKeywords, embed, indexApi, log}: {
     extractKeywords: IExtractKeywords,
     cleanUpKeywords: ICleanUpKeywords,
     embed: IEmbed,
     indexApi: IIndexApi,
+    log: ILogger,
 }): IIndexContent {
     return async function indexContent(items) {
-        const indexItems = await Promise.all(items.map(async (item) => {
-            const keywords = cleanUpKeywords(extractKeywords(item.content)).join(", ");
-            const vector = await embed(keywords);
-            return {id: item.id, vector, keywords};
-        }));
+        const indexItems = (await Promise.all(items.map(async (item) => {
+            try {
+                const keywords = cleanUpKeywords(extractKeywords(item.content)).join(", ");
+                const vector = await embed(keywords);
+                return {id: item.id, vector, keywords};
+            } catch (e: any) {
+                log(`skip ${item.id}: keywords/embed failed — ${e?.code ?? e?.message ?? e}`);
+                return null;
+            }
+        }))).filter((x): x is NonNullable<typeof x> => !!x);
 
-        await indexApi({type: IndexCommandType.index, items: indexItems});
+        if (indexItems.length > 0) {
+            await indexApi({type: IndexCommandType.index, items: indexItems});
+        }
     }
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "xindex",
-  "version": "1.0.4",
+  "version": "1.0.6",
   "description": "Local semantic code search — index codebase, search by meaning or keywords",
   "type": "module",
   "main": "xindex.ts",
@@ -15,11 +15,16 @@
     "index": "tsx apps/run.index.ts",
     "search": "tsx apps/run.search.ts",
     "reset": "tsx apps/run.reset.ts",
+    "test": "npm run test.functional && npm run test.compilation",
     "mcp": "tsx apps/run.mcp.ts",
     "watch": "tsx apps/run.watch.ts",
+    "test.functional": "bash test/functional.sh",
     "test.compilation": "npx -y tsc --ignoreConfig --noEmit --target ES2022 --module ESNext --moduleResolution bundler --esModuleInterop --skipLibCheck --strict false $(git ls-files '*.ts')"
   },
   "private": false,
+  "engines": {
+    "node": ">=22"
+  },
   "keywords": [
     "semantic-search",
     "code-search",