@lojban/semantic-search-mcp 1.0.12 → 1.0.14

package/README.md

@@ -12,7 +12,7 @@ Use it in **Cursor**, **Claude Code**, or any IDE that supports MCP to search th
 
 ## How it works
 
-- **Indexing**: On startup, if `SEMANTIC_SEARCH_INDEX_DIRS` is set (comma-separated paths), the server scans those directories in the background for `.txt`, `.md`, `.tsv`, `.csv`. Each non-empty line gets a vector embedding (via [Hugging Face Transformers.js](https://huggingface.co/docs/transformers.js), model `Xenova/all-MiniLM-L6-v2`) and is stored in a local SQLite database with [@dao-xyz/sqlite3-vec](https://www.npmjs.com/package/@dao-xyz/sqlite3-vec) (SQLite + sqlite-vec for Node and browser). Indexing runs asynchronously so the server stays responsive and uses bounded memory.
+- **Indexing**: On startup, the server indexes content in the background. If **`SEMANTIC_SEARCH_INDEX_DIRS`** is set (comma-separated paths), it scans those directories. If it is *not* set, the server downloads the [lojban/sampu_vlaste](https://github.com/lojban/sampu_vlaste) repository from GitHub and indexes that instead. In both cases, the server looks for `.txt`, `.md`, `.tsv`, `.csv` files. Each non-empty line gets a vector embedding (via [Hugging Face Transformers.js](https://huggingface.co/docs/transformers.js), model `Xenova/all-MiniLM-L6-v2`) and is stored in a local SQLite database with [@dao-xyz/sqlite3-vec](https://www.npmjs.com/package/@dao-xyz/sqlite3-vec) (SQLite + sqlite-vec for Node and browser). Indexing runs asynchronously so the server stays responsive and uses bounded memory.
 - **Search**: You send a natural-language query; the server embeds it and returns the closest lines by cosine similarity.
 - **Storage**: Index is stored in your project's `.semantic-search/data/` (or set `SEMANTIC_SEARCH_DATA_DIR`). No cloud, no API keys.
 
@@ -44,13 +44,13 @@ The package is published as [**@lojban/semantic-search-mcp**](https://www.npmjs.
    }
    ```
 
-   No `cwd` needed: the server stores its index in your **project directory** (`.semantic-search/data/`), so open your project in Cursor and the index is per-workspace. To use a fixed data directory instead, add `"env": { "SEMANTIC_SEARCH_DATA_DIR": "/path/to/data" }`. To have the server index directories on startup, set `"env": { "SEMANTIC_SEARCH_INDEX_DIRS": "./dictionary,./glossary" }` (comma-separated paths).
+   No `cwd` needed: the server stores its index in your **project directory** (`.semantic-search/data/`), so open your project in Cursor and the index is per-workspace. To use a fixed data directory instead, add `"env": { "SEMANTIC_SEARCH_DATA_DIR": "/path/to/data" }`. To have the server index specific directories on startup, set `"env": { "SEMANTIC_SEARCH_INDEX_DIRS": "./dictionary,./glossary" }` (comma-separated paths). If you omit `SEMANTIC_SEARCH_INDEX_DIRS`, the server will download and index the [lojban/sampu_vlaste](https://github.com/lojban/sampu_vlaste) repo automatically.
 
-2. **Restart Cursor** (or reload the window). If `SEMANTIC_SEARCH_INDEX_DIRS` is set, indexing starts automatically in the background.
+2. **Restart Cursor** (or reload the window). Indexing starts automatically in the background: from your configured `SEMANTIC_SEARCH_INDEX_DIRS`, or from the downloaded sampu_vlaste repo if that env is not set.
 
 3. In chat or Composer, ask the AI to use the tools:
-   - **Search**: "Search the index for …" or "Find entries similar to …"
-   - **Stats**: "How many lines/files are in the index?" or "Is indexing still running?" — stats include progress and start time (locale-formatted) when indexing is in progress.
+   - **Search**: "Use semantic-search tool: find combinations of words that can express the concept of …", "Use semantic-search tool: search the index for …" or "Use semantic-search tool: Find entries similar to …"
+   - **Stats**: "use semantic-search mcp. run get_index_stats" — stats include progress and start time (locale-formatted) when indexing is in progress.
 
 The AI will call `search` and `get_index_stats` for you.
 
@@ -58,7 +58,7 @@ The AI will call `search` and `get_index_stats` for you.
 
 Any environment that supports MCP over stdio can use this server. Run:
 
-- **One-liner**: `npx -y @lojban/semantic-search-mcp` — dependencies are installed on first run; index is stored in the current working directory's `.semantic-search/data/`. Set env `SEMANTIC_SEARCH_INDEX_DIRS` (comma-separated paths) to index those directories on startup in the background. Tools: `search`, `get_index_stats`.
+- **One-liner**: `npx -y @lojban/semantic-search-mcp` — dependencies are installed on first run; index is stored in the current working directory's `.semantic-search/data/`. Set env `SEMANTIC_SEARCH_INDEX_DIRS` (comma-separated paths) to index those directories on startup; if unset, the server downloads and indexes [lojban/sampu_vlaste](https://github.com/lojban/sampu_vlaste) from GitHub. Tools: `search`, `get_index_stats`.
 
 **From source**: Clone the repo, run `npm install` once, then use `"command": "npx", "args": ["tsx", "src/index.ts"], "cwd": "/path/to/semantic-search-mcp"` or `"command": "node", "args": ["/path/to/semantic-search-mcp/run.mjs"]` (no `cwd` needed with the latter). See [MCP_SETUP.md](MCP_SETUP.md) for details.
 
@@ -71,7 +71,10 @@ Any environment that supports MCP over stdio can use this server. Run:
 
 ### Indexing on startup
 
-Set the environment variable **`SEMANTIC_SEARCH_INDEX_DIRS`** to a comma-separated list of directories to index. When the MCP server starts, it begins indexing those directories in the background (async). The index is cleared and rebuilt each time the server starts. Use absolute paths or paths relative to the server's working directory. The server reads and indexes all supported text/TSV/CSV files under each directory recursively. Indexing uses bounded memory and yields to the event loop so the OS stays responsive.
+- **With your own dirs**: Set the environment variable **`SEMANTIC_SEARCH_INDEX_DIRS`** to a comma-separated list of directories to index. When the MCP server starts, it begins indexing those directories in the background (async).
+- **Default (no env set)**: If **`SEMANTIC_SEARCH_INDEX_DIRS`** is not set, the server downloads the [lojban/sampu_vlaste](https://github.com/lojban/sampu_vlaste) repository from GitHub (as a zip), extracts it under `.semantic-search/sampu_vlaste/`, and indexes that. The download is cached; subsequent starts reuse the cached copy.
+
+The index is cleared and rebuilt each time the server starts. Use absolute paths or paths relative to the server's working directory when setting `SEMANTIC_SEARCH_INDEX_DIRS`. The server reads and indexes all supported `.txt`, `.md`, `.tsv`, `.csv` files under each directory recursively. Indexing uses bounded memory and yields to the event loop so the OS stays responsive.
 
 ## Example: Lojban dictionary gaps
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lojban/semantic-search-mcp",
-  "version": "1.0.12",
+  "version": "1.0.14",
   "description": "Local-first MCP server for semantic search using transformers.js and SQLite",
   "type": "module",
   "scripts": {
@@ -10,6 +10,7 @@
     "@dao-xyz/sqlite3-vec": "^0.0.19",
     "@huggingface/transformers": "^3.0.0",
     "@modelcontextprotocol/sdk": "^1.0.0",
+    "extract-zip": "^2.0.1",
     "glob": "^10.3.0",
     "tsx": "^4.0.0"
   },

package/src/download-sampu-vlaste.ts

@@ -0,0 +1,68 @@
+import { mkdirSync, readdirSync, unlinkSync } from 'fs';
+import path from 'path';
+import { createWriteStream } from 'fs';
+import { pipeline } from 'stream/promises';
+import { Readable } from 'stream';
+
+const REPO_OWNER = 'lojban';
+const REPO_NAME = 'sampu_vlaste';
+const BRANCHES_TO_TRY = ['main', 'master'];
+
+/**
+ * Download the lojban/sampu_vlaste repo from GitHub as a zip and extract it.
+ * If the repo is already present under cacheDir, returns the existing path.
+ * @param cacheDir - Directory to store the downloaded repo (e.g. .semantic-search/sampu_vlaste)
+ * @returns Path to the extracted repo root (e.g. cacheDir/sampu_vlaste-main)
+ */
+export async function getSampuVlasteDir(cacheDir: string): Promise<string> {
+  mkdirSync(cacheDir, { recursive: true });
+
+  // Check for existing extraction (any branch)
+  const entries = readdirSync(cacheDir, { withFileTypes: true });
+  const existingDir = entries.find(
+    (e) => e.isDirectory() && e.name.startsWith(`${REPO_NAME}-`)
+  );
+  if (existingDir) {
+    const existingPath = path.join(cacheDir, existingDir.name);
+    return existingPath;
+  }
+
+  let lastError: Error | null = null;
+  for (const branch of BRANCHES_TO_TRY) {
+    const url = `https://github.com/${REPO_OWNER}/${REPO_NAME}/archive/refs/heads/${branch}.zip`;
+    try {
+      const response = await fetch(url, { redirect: 'follow' });
+      if (!response.ok) {
+        lastError = new Error(`HTTP ${response.status}: ${url}`);
+        continue;
+      }
+      const body = response.body;
+      if (!body) {
+        lastError = new Error('Empty response body');
+        continue;
+      }
+      const zipPath = path.join(cacheDir, `repo-${branch}.zip`);
+      const nodeStream = Readable.fromWeb(body as import('stream/web').ReadableStream);
+      const out = createWriteStream(zipPath);
+      await pipeline(nodeStream, out);
+
+      const { default: extract } = await import('extract-zip');
+      await extract(zipPath, { dir: cacheDir });
+
+      unlinkSync(zipPath);
+
+      const afterEntries = readdirSync(cacheDir, { withFileTypes: true });
+      const extracted = afterEntries.find(
+        (e) => e.isDirectory() && e.name.startsWith(`${REPO_NAME}-`)
+      );
+      if (extracted) {
+        return path.join(cacheDir, extracted.name);
+      }
+      lastError = new Error('Extracted archive had no expected top-level directory');
+    } catch (err) {
+      lastError = err instanceof Error ? err : new Error(String(err));
+    }
+  }
+
+  throw lastError ?? new Error('Failed to download sampu_vlaste');
+}

package/src/embeddings.ts

@@ -36,8 +36,8 @@ export async function getBatchEmbeddings(texts: string[]): Promise<Float32Array[
   const ext = await getExtractor();
   const results: Float32Array[] = [];
 
-  // Process in batches for memory; each batch is one model forward pass
-  const batchSize = 64;
+  // Process in batches for memory; each batch is one model forward pass (smaller = lower peak RAM)
+  const batchSize = 32;
   for (let i = 0; i < texts.length; i += batchSize) {
     const batch = texts.slice(i, i + batchSize);
     const output = await ext(batch, { pooling: 'mean', normalize: true });

package/src/index.ts

@@ -8,7 +8,9 @@ import {
 import path from 'path';
 import { getEmbedding, getBatchEmbeddings } from './embeddings.js';
 import { createVectorStorage, type SearchResult } from './storage.js';
-import { scanDirectories } from './scanner.js';
+import { normalizePath } from './path-util.js';
+import { listFilesInDirectories, readFileWithMetadata } from './scanner.js';
+import { getSampuVlasteDir } from './download-sampu-vlaste.js';
 
 // Data dir: use env, or project cwd so each workspace has its own index when run via npx from Cursor
 const dataDir =
@@ -28,6 +30,17 @@ const indexingState = {
 // Batch size kept small to avoid high RAM usage during indexing
 const INDEX_BATCH_SIZE = 256;
 
+/** True if normalized path is under one of the normalized directory paths */
+function isPathUnderAnyDir(filePath: string, dirs: string[]): boolean {
+  const normalized = normalizePath(filePath);
+  for (const dir of dirs) {
+    const d = normalizePath(dir);
+    const prefix = d.endsWith(path.sep) ? d : d + path.sep;
+    if (normalized === d || normalized.startsWith(prefix)) return true;
+  }
+  return false;
+}
+
 async function runBackgroundIndexing(
   storage: Awaited<ReturnType<typeof createVectorStorage>>,
   directories: string[]
@@ -37,12 +50,21 @@ async function runBackgroundIndexing(
   indexingState.linesIndexed = 0;
   indexingState.filesIndexed = 0;
   indexingState.error = null;
-  storage.clear();
+
+  const normalizedDirs = directories.map((d) => normalizePath(d));
 
   try {
-    let currentBatch: Array<{ filePath: string; lineNumber: number; content: string }> = [];
-    let processingPromise: Promise<void> | null = null;
-    const seenFiles = new Set<string>();
+    const [fileMetadata, indexedPaths] = await Promise.all([
+      storage.getFileMetadata(),
+      storage.getIndexedFilePaths(),
+    ]);
+
+    // Remove from DB any file whose directory is no longer in SEMANTIC_SEARCH_INDEX_DIRS
+    for (const filePath of indexedPaths) {
+      if (!isPathUnderAnyDir(filePath, normalizedDirs)) {
+        await storage.removeFile(filePath);
+      }
+    }
 
     const processBatch = async (
       batch: Array<{ filePath: string; lineNumber: number; content: string }>
@@ -57,28 +79,64 @@ async function runBackgroundIndexing(
         embedding: embeddings[idx],
       }));
       await storage.upsertLinesBatch(batchData);
-      for (const l of batch) seenFiles.add(l.filePath);
       indexingState.linesIndexed += batch.length;
-      indexingState.filesIndexed = seenFiles.size;
+      const stats = await storage.getStats();
+      indexingState.filesIndexed = stats.totalFiles;
     };
 
     const yieldToEventLoop = (): Promise<void> =>
       new Promise((resolve) => setImmediate(resolve));
 
-    for await (const line of scanDirectories(directories)) {
-      currentBatch.push(line);
-      if (currentBatch.length >= INDEX_BATCH_SIZE) {
-        if (processingPromise) await processingPromise;
-        const batchToProcess = currentBatch;
-        currentBatch = [];
-        processingPromise = processBatch(batchToProcess);
-        await yieldToEventLoop();
+    const currentFilesOnDisk = new Set<string>();
+    const toIndex: Array<{ filePath: string; mtimeMs: number; contentHash: string; lines: Array<{ filePath: string; lineNumber: number; content: string }> }> = [];
+
+    for await (const { filePath, mtimeMs } of listFilesInDirectories(directories)) {
+      currentFilesOnDisk.add(filePath);
+      const meta = fileMetadata.get(filePath);
+      const content = readFileWithMetadata(filePath);
+      if (!content) continue;
+      if (meta && meta.mtimeMs === content.mtimeMs && meta.contentHash === content.contentHash) {
+        continue; // unchanged, skip
+      }
+      toIndex.push({
+        filePath: content.filePath,
+        mtimeMs: content.mtimeMs,
+        contentHash: content.contentHash,
+        lines: content.lines,
+      });
+    }
+
+    // Remove from DB any file that no longer exists on disk (under current dirs)
+    for (const filePath of indexedPaths) {
+      if (isPathUnderAnyDir(filePath, normalizedDirs) && !currentFilesOnDisk.has(normalizePath(filePath))) {
+        await storage.removeFile(filePath);
+      }
+    }
+
+    let currentBatch: Array<{ filePath: string; lineNumber: number; content: string }> = [];
+    let processingPromise: Promise<void> | null = null;
+
+    for (const entry of toIndex) {
+      await storage.removeFile(entry.filePath);
+      for (const line of entry.lines) {
+        currentBatch.push(line);
+        if (currentBatch.length >= INDEX_BATCH_SIZE) {
+          if (processingPromise) await processingPromise;
+          const batchToProcess = currentBatch;
+          currentBatch = [];
+          processingPromise = processBatch(batchToProcess);
+          await yieldToEventLoop();
+        }
       }
     }
 
     if (processingPromise) await processingPromise;
     if (currentBatch.length > 0) await processBatch(currentBatch);
 
+    for (const entry of toIndex) {
+      await storage.setFileMetadata(entry.filePath, entry.mtimeMs, entry.contentHash);
+    }
+
     const stats = await storage.getStats();
     indexingState.linesIndexed = stats.totalLines;
     indexingState.filesIndexed = stats.totalFiles;
@@ -228,7 +286,18 @@ async function main() {
   console.error('Semantic Search MCP Server running on stdio');
 
   const envDirs = process.env.SEMANTIC_SEARCH_INDEX_DIRS;
-  const directories = envDirs ? envDirs.split(',').map((d) => d.trim()).filter(Boolean) : [];
+  let directories = envDirs ? envDirs.split(',').map((d) => d.trim()).filter(Boolean) : [];
+  if (directories.length === 0) {
+    try {
+      const sampuVlasteCacheDir = path.join(dataDir, '..', 'sampu_vlaste');
+      console.error('SEMANTIC_SEARCH_INDEX_DIRS not set; downloading lojban/sampu_vlaste from GitHub...');
+      const sampuDir = await getSampuVlasteDir(sampuVlasteCacheDir);
+      directories = [sampuDir];
+      console.error(`Using downloaded repo at ${sampuDir}`);
+    } catch (err) {
+      console.error('Failed to download sampu_vlaste:', err);
+    }
+  }
   if (directories.length > 0) {
     console.error(`Starting background indexing for ${directories.length} directories...`);
     runBackgroundIndexing(storage, directories).catch((err) => {

package/src/path-util.ts

@@ -0,0 +1,6 @@
+import path from 'path';
+
+/** Normalize path for consistent comparison across Linux, Windows, Mac */
+export function normalizePath(p: string): string {
+  return path.normalize(path.resolve(p));
+}

package/src/scanner.ts

@@ -1,7 +1,9 @@
-import { createReadStream, statSync } from 'fs';
-import { glob } from 'glob';
+import { createReadStream, readFileSync, statSync } from 'fs';
+import { globIterate } from 'glob';
 import path from 'path';
 import readline from 'readline';
+import { createHash } from 'crypto';
+import { normalizePath } from './path-util.js';
 
 export interface FileLine {
   filePath: string;
@@ -18,6 +20,9 @@ const MIN_LINE_LENGTH = 5;
 // Maximum file size to process (skip very large files)
 const MAX_FILE_SIZE = 10 * 1024 * 1024; // 10MB
 
+// Cap line length to avoid unbounded readline buffer (e.g. file with no newlines)
+const MAX_LINE_LENGTH = 256 * 1024; // 256KB
+
 /**
  * Check if a file is a text file we should index
  */
@@ -27,17 +32,17 @@ function isTextFile(filePath: string): boolean {
 }
 
 /**
- * Scan a directory for text files and yield lines
+ * Scan a directory for text files and yield lines.
+ * Uses globIterate so file paths are streamed one-by-one (no full list in RAM).
+ * Readline has maxLineLength to avoid huge single-line buffers.
  */
 export async function* scanDirectory(dirPath: string): AsyncGenerator<FileLine> {
-  // Find all files in directory recursively
   const pattern = path.join(dirPath, '**/*');
-  // Nodir: true ensures we only get files
-  const files = await glob(pattern, { nodir: true, absolute: true });
-
-  for (const filePath of files) {
+  for await (const filePath of globIterate(pattern, { nodir: true, absolute: true }) as AsyncIterable<string>) {
     if (!isTextFile(filePath)) continue;
 
+    let fileStream: ReturnType<typeof createReadStream> | null = null;
+    let rl: readline.Interface | null = null;
     try {
       const stats = statSync(filePath);
       if (stats.size > MAX_FILE_SIZE) {
@@ -45,11 +50,12 @@ export async function* scanDirectory(dirPath: string): AsyncGenerator<FileLine>
         continue;
       }
 
-      const fileStream = createReadStream(filePath);
-      const rl = readline.createInterface({
+      fileStream = createReadStream(filePath);
+      rl = readline.createInterface({
         input: fileStream,
         crlfDelay: Infinity,
-      });
+        maxLineLength: MAX_LINE_LENGTH,
+      } as readline.ReadLineOptions);
 
       let lineNumber = 0;
       for await (const line of rl) {
@@ -65,6 +71,9 @@ export async function* scanDirectory(dirPath: string): AsyncGenerator<FileLine>
       }
     } catch (err) {
       console.error(`Error reading file ${filePath}:`, err);
+    } finally {
+      rl?.close();
+      fileStream?.destroy();
     }
   }
 }
@@ -77,3 +86,67 @@ export async function* scanDirectories(dirPaths: string[]): AsyncGenerator<FileL
     yield* scanDirectory(dirPath);
   }
 }
+
+export interface FileWithMtime {
+  filePath: string;
+  mtimeMs: number;
+}
+
+/**
+ * List all text files in the given directories with their mtime.
+ * Paths are normalized for cross-platform consistency.
+ */
+export async function* listFilesInDirectories(dirPaths: string[]): AsyncGenerator<FileWithMtime> {
+  for (const dirPath of dirPaths) {
+    const pattern = path.join(dirPath, '**/*');
+    for await (const filePath of globIterate(pattern, { nodir: true, absolute: true }) as AsyncIterable<string>) {
+      if (!isTextFile(filePath)) continue;
+      try {
+        const stats = statSync(filePath);
+        if (stats.size > MAX_FILE_SIZE) {
+          console.error(`Skipping large file: ${filePath}`);
+          continue;
+        }
+        yield { filePath: normalizePath(filePath), mtimeMs: stats.mtimeMs };
+      } catch (err) {
+        console.error(`Error stating file ${filePath}:`, err);
+      }
+    }
+  }
+}
+
+export interface FileContentWithMetadata {
+  filePath: string;
+  mtimeMs: number;
+  contentHash: string;
+  lines: FileLine[];
+}
+
+/**
+ * Read a file once: compute content hash (sha256 of raw content) and extract indexable lines.
+ * Uses mtime from stat. Cross-platform (mtimeMs is available on Linux, Windows, Mac).
+ */
+export function readFileWithMetadata(filePath: string): FileContentWithMetadata | null {
+  try {
+    const stats = statSync(filePath);
+    if (stats.size > MAX_FILE_SIZE) return null;
+    const raw = readFileSync(filePath);
+    const contentHash = createHash('sha256').update(raw).digest('hex');
+    const mtimeMs = stats.mtimeMs;
+    const text = raw.toString('utf8');
+    const lines: FileLine[] = [];
+    const normalizedPath = normalizePath(filePath);
+    let lineNumber = 0;
+    for (const rawLine of text.split(/\r?\n/)) {
+      lineNumber++;
+      const trimmed = rawLine.trim();
+      if (trimmed.length >= MIN_LINE_LENGTH) {
+        lines.push({ filePath: normalizedPath, lineNumber, content: trimmed });
+      }
+    }
+    return { filePath: normalizedPath, mtimeMs, contentHash, lines };
+  } catch (err) {
+    console.error(`Error reading file ${filePath}:`, err);
+    return null;
+  }
+}

package/src/storage.ts

@@ -2,9 +2,15 @@ import pkg from '@dao-xyz/sqlite3-vec';
 const { createDatabase } = pkg;
 import path from 'path';
 import { mkdirSync } from 'fs';
+import { normalizePath } from './path-util.js';
 
 const EMBEDDING_DIM = 384; // all-MiniLM-L6-v2 produces 384-dim vectors
 
+export interface FileMetadata {
+  mtimeMs: number;
+  contentHash: string;
+}
+
 export interface LineRecord {
   id: number;
   file_path: string;
@@ -31,6 +37,10 @@ export class VectorStorage {
   }
 
   private init(): void {
+    // Limit SQLite page cache to avoid unbounded RAM (negative = kibibytes)
+    try {
+      this.db.exec('PRAGMA cache_size = -65536'); // 64MB max
+    } catch {}
     this.db.exec(`
       CREATE TABLE IF NOT EXISTS lines (
         id INTEGER PRIMARY KEY AUTOINCREMENT,
@@ -47,6 +57,13 @@ export class VectorStorage {
         embedding FLOAT[${EMBEDDING_DIM}]
       );
     `);
+    this.db.exec(`
+      CREATE TABLE IF NOT EXISTS file_metadata (
+        file_path TEXT PRIMARY KEY,
+        mtime_ms INTEGER NOT NULL,
+        content_hash TEXT NOT NULL
+      );
+    `);
   }
 
   /**
@@ -143,11 +160,55 @@ export class VectorStorage {
   async removeFile(filePath: string): Promise<void> {
     (await this.db.prepare('DELETE FROM vec_lines WHERE line_id IN (SELECT id FROM lines WHERE file_path = ?)')).run([filePath]);
     (await this.db.prepare('DELETE FROM lines WHERE file_path = ?')).run([filePath]);
+    (await this.db.prepare('DELETE FROM file_metadata WHERE file_path = ?')).run([filePath]);
+  }
+
+  /** Remove all indexed files under a directory (path prefix match, normalized). */
+  async removeFilesUnderDirectory(dirPath: string): Promise<void> {
+    const normalizedDir = normalizePath(dirPath);
+    const prefix = normalizedDir.endsWith(path.sep) ? normalizedDir : normalizedDir + path.sep;
+    const rows = (await this.db.prepare('SELECT DISTINCT file_path FROM lines')).all() as Array<{ file_path: string }>;
+    for (const row of rows) {
+      const p = normalizePath(row.file_path);
+      if (p === normalizedDir || p.startsWith(prefix)) {
+        await this.removeFile(row.file_path);
+      }
+    }
+  }
+
+  /** Get metadata for all indexed files (for incremental sync). */
+  async getFileMetadata(): Promise<Map<string, FileMetadata>> {
+    const rows = (await this.db.prepare('SELECT file_path, mtime_ms, content_hash FROM file_metadata')).all() as Array<{
+      file_path: string;
+      mtime_ms: number;
+      content_hash: string;
+    }>;
+    const map = new Map<string, FileMetadata>();
+    for (const row of rows) {
+      map.set(row.file_path, { mtimeMs: row.mtime_ms, contentHash: row.content_hash });
+    }
+    return map;
+  }
+
+  /** Set metadata after indexing a file. */
+  async setFileMetadata(filePath: string, mtimeMs: number, contentHash: string): Promise<void> {
+    const stmt = await this.db.prepare(
+      `INSERT INTO file_metadata (file_path, mtime_ms, content_hash) VALUES (?, ?, ?)
+       ON CONFLICT(file_path) DO UPDATE SET mtime_ms = excluded.mtime_ms, content_hash = excluded.content_hash`
+    );
+    stmt.run([filePath, mtimeMs, contentHash]);
+  }
+
+  /** Get all file paths currently in the index (for detecting deleted files). */
+  async getIndexedFilePaths(): Promise<string[]> {
+    const rows = (await this.db.prepare('SELECT DISTINCT file_path FROM lines')).all() as Array<{ file_path: string }>;
+    return rows.map((r) => r.file_path);
   }
 
   clear(): void {
     this.db.exec('DELETE FROM vec_lines');
     this.db.exec('DELETE FROM lines');
+    this.db.exec('DELETE FROM file_metadata');
   }
 
   close(): void {