@lojban/semantic-search-mcp 1.0.11 → 1.0.13

package/README.md

@@ -12,7 +12,7 @@ Use it in **Cursor**, **Claude Code**, or any IDE that supports MCP to search th
 
 ## How it works
 
-- **Indexing**: Scans directories for `.txt`, `.md`, `.tsv`, `.csv`, `.json`, `.html`, `.xml`. Each non-empty line gets a vector embedding (via [Hugging Face Transformers.js](https://huggingface.co/docs/transformers.js), model `Xenova/all-MiniLM-L6-v2`) and is stored in a local SQLite database with [@dao-xyz/sqlite3-vec](https://www.npmjs.com/package/@dao-xyz/sqlite3-vec) (SQLite + sqlite-vec for Node and browser).
+- **Indexing**: On startup, if `SEMANTIC_SEARCH_INDEX_DIRS` is set (comma-separated paths), the server scans those directories in the background for `.txt`, `.md`, `.tsv`, `.csv`. Each non-empty line gets a vector embedding (via [Hugging Face Transformers.js](https://huggingface.co/docs/transformers.js), model `Xenova/all-MiniLM-L6-v2`) and is stored in a local SQLite database with [@dao-xyz/sqlite3-vec](https://www.npmjs.com/package/@dao-xyz/sqlite3-vec) (SQLite + sqlite-vec for Node and browser). Indexing runs asynchronously so the server stays responsive and uses bounded memory.
 - **Search**: You send a natural-language query; the server embeds it and returns the closest lines by cosine similarity.
 - **Storage**: Index is stored in your project's `.semantic-search/data/` (or set `SEMANTIC_SEARCH_DATA_DIR`). No cloud, no API keys.
 
@@ -44,22 +44,21 @@ The package is published as [**@lojban/semantic-search-mcp**](https://www.npmjs.
    }
    ```
 
-   No `cwd` needed: the server stores its index in your **project directory** (`.semantic-search/data/`), so open your project in Cursor and the index is per-workspace. To use a fixed data directory instead, add `"env": { "SEMANTIC_SEARCH_DATA_DIR": "/path/to/data" }`.
+   No `cwd` needed: the server stores its index in your **project directory** (`.semantic-search/data/`), so open your project in Cursor and the index is per-workspace. To use a fixed data directory instead, add `"env": { "SEMANTIC_SEARCH_DATA_DIR": "/path/to/data" }`. To have the server index directories on startup, set `"env": { "SEMANTIC_SEARCH_INDEX_DIRS": "./dictionary,./glossary" }` (comma-separated paths).
 
-2. **Restart Cursor** (or reload the window).
+2. **Restart Cursor** (or reload the window). If `SEMANTIC_SEARCH_INDEX_DIRS` is set, indexing starts automatically in the background.
 
 3. In chat or Composer, ask the AI to use the tools:
-   - **Index**: "Index the directory `./my-dictionary`" (or a list of paths). Optionally "clear existing index first."
    - **Search**: "Search the index for …" or "Find entries similar to …"
-   - **Stats**: "How many lines/files are in the index?"
+   - **Stats**: "How many lines/files are in the index?" or "Is indexing still running?" — stats include progress and start time (locale-formatted) when indexing is in progress.
 
-The AI will call `index_directories`, `search`, and `get_index_stats` for you.
+The AI will call `search` and `get_index_stats` for you.
 
 ## Use in other AI IDEs (Claude Code, etc.)
 
 Any environment that supports MCP over stdio can use this server. Run:
 
-- **One-liner**: `npx -y @lojban/semantic-search-mcp` — dependencies are installed on first run; index is stored in the current working directory's `.semantic-search/data/`. Same tools: `index_directories`, `search`, `get_index_stats`.
+- **One-liner**: `npx -y @lojban/semantic-search-mcp` — dependencies are installed on first run; index is stored in the current working directory's `.semantic-search/data/`. Set env `SEMANTIC_SEARCH_INDEX_DIRS` (comma-separated paths) to index those directories on startup in the background. Tools: `search`, `get_index_stats`.
 
 **From source**: Clone the repo, run `npm install` once, then use `"command": "npx", "args": ["tsx", "src/index.ts"], "cwd": "/path/to/semantic-search-mcp"` or `"command": "node", "args": ["/path/to/semantic-search-mcp/run.mjs"]` (no `cwd` needed with the latter). See [MCP_SETUP.md](MCP_SETUP.md) for details.
 
@@ -67,60 +66,21 @@ Any environment that supports MCP over stdio can use this server. Run:
 
 | Tool | Description |
 |------|-------------|
-| `index_directories` | Scan one or more directories and index every line of supported text files. Pass `directories` (array of paths) or set env `SEMANTIC_SEARCH_INDEX_DIRS` (comma-separated). Optional: `clear_existing: true` to replace the index. |
 | `search` | Semantic search: `query` (string), optional `limit` (default 10). Returns file path, line number, content, and similarity score. |
-| `get_index_stats` | Returns total number of indexed files and lines. |
+| `get_index_stats` | Returns total number of indexed files and lines. When indexing is running in the background, also returns progress: `indexing.started_at` (locale-formatted), `lines_indexed_so_far`, `files_indexed_so_far`, and `in_progress`. |
 
-### Indexing several directories
+### Indexing on startup
 
-`index_directories` accepts an array of paths. You can index multiple unrelated folders in one go (they are merged into a single index). Use **absolute paths** or paths relative to your project root.
-
-**In Cursor (natural language):**
-
-- "Index these directories: `./dictionary`, `./glossary`, and `./notes`."
-- "Index `./data/lojban-eng` and `/home/me/other-corpus` with clear_existing true."
-- "Clear the index and re-index only `./tsv` and `./exports`."
-
-**Under the hood** the tool receives:
-
-```json
-{
-  "directories": ["./dictionary", "./glossary", "./notes"],
-  "clear_existing": false
-}
-```
-
-To replace the entire index with new content from several places:
-
-```json
-{
-  "directories": ["/path/to/dict1", "/path/to/dict2", "/path/to/corpus"],
-  "clear_existing": true
-}
-```
-
-Paths can be anywhere on disk (e.g. different drives or projects); the server reads and indexes all supported text/TSV/CSV files under each directory recursively.
-
-### Memory and batch size
-
-Indexing uses **adaptive batch size** based on free system RAM so the OS doesn’t freeze on low-memory machines. The server reads `os.freemem()`, keeps a reserve (default 400MB), and caps batch size between 32 and 512 lines. You can tune this with env vars:
-
-- **`SEMANTIC_SEARCH_RESERVE_MB`** — MB of RAM to keep free (default `400`).
-- **`SEMANTIC_SEARCH_MIN_BATCH`** — minimum lines per batch (default `32`).
-- **`SEMANTIC_SEARCH_MAX_BATCH`** — maximum lines per batch (default `512`).
-
-Example: `SEMANTIC_SEARCH_RESERVE_MB=800 SEMANTIC_SEARCH_MAX_BATCH=256` to leave more headroom and use smaller batches.
-
-- **`SEMANTIC_SEARCH_GC`** — explicit GC after each batch is **on by default** when Node is run with `--expose-gc` (helps avoid OS freezes during long indexing). In MCP use e.g. `"args": ["--expose-gc", "-y", "@lojban/semantic-search-mcp"]`. Set to `0` or `false` to disable.
+Set the environment variable **`SEMANTIC_SEARCH_INDEX_DIRS`** to a comma-separated list of directories to index. When the MCP server starts, it begins indexing those directories in the background (async). The index is cleared and rebuilt each time the server starts. Use absolute paths or paths relative to the server's working directory. The server reads and indexes all supported text/TSV/CSV files under each directory recursively. Indexing uses bounded memory and yields to the event loop so the OS stays responsive.
 
 ## Example: Lojban dictionary gaps
 
-1. Put your dictionary TSV (e.g. `jbo-eng.tsv`) in a folder.
-2. In Cursor: "Index the directory `./dictionary` with clear_existing true."
-3. Then: "Search for entries similar to 'to cause to become warm' and limit 20."
+1. Put your dictionary TSV (e.g. `jbo-eng.tsv`) in a folder (e.g. `./dictionary`).
+2. Set `SEMANTIC_SEARCH_INDEX_DIRS=./dictionary` in your MCP config (or in the environment). Restart the server; indexing runs in the background.
+3. In Cursor: "Search for entries similar to 'to cause to become warm' and limit 20."
 4. Or: "Search for 'emotional state of joy' and show me what we have; then suggest word combinations the dictionary might be missing."
 
-The index is stored in `.semantic-search/data/vectors.db` (or your project root). Re-index when you add or change files.
+The index is stored in `.semantic-search/data/vectors.db` (or your project root). Restart the server to re-index when you add or change files.
 
 ## Development
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lojban/semantic-search-mcp",
-  "version": "1.0.11",
+  "version": "1.0.13",
   "description": "Local-first MCP server for semantic search using transformers.js and SQLite",
   "type": "module",
   "scripts": {

package/src/embeddings.ts

@@ -36,8 +36,8 @@ export async function getBatchEmbeddings(texts: string[]): Promise<Float32Array[
   const ext = await getExtractor();
   const results: Float32Array[] = [];
 
-  // Process in batches for memory; each batch is one model forward pass
-  const batchSize = 64;
+  // Process in batches for memory; each batch is one model forward pass (smaller = lower peak RAM)
+  const batchSize = 32;
   for (let i = 0; i < texts.length; i += batchSize) {
     const batch = texts.slice(i, i + batchSize);
     const output = await ext(batch, { pooling: 'mean', normalize: true });

package/src/index.ts

@@ -5,10 +5,9 @@ import {
   CallToolRequestSchema,
   ListToolsRequestSchema,
 } from '@modelcontextprotocol/sdk/types.js';
-import os from 'node:os';
 import path from 'path';
 import { getEmbedding, getBatchEmbeddings } from './embeddings.js';
-import { createVectorStorage, type SearchResult, type VectorStorage } from './storage.js';
+import { createVectorStorage, type SearchResult } from './storage.js';
 import { scanDirectories } from './scanner.js';
 
 // Data dir: use env, or project cwd so each workspace has its own index when run via npx from Cursor
@@ -17,191 +16,81 @@ const dataDir =
   path.join(process.cwd(), '.semantic-search', 'data');
 const DB_PATH = path.join(dataDir, 'vectors.db');
 
-type IndexStatus = {
-  isIndexing: boolean;
-  startedAt: number | null;
-  finishedAt: number | null;
-  lastError: string | null;
-  indexedLines: number;
-  indexedFiles: number;
-  directories: string[];
+// Background indexing state (progress for get_index_stats)
+const indexingState = {
+  inProgress: false,
+  startedAt: null as Date | null,
+  linesIndexed: 0,
+  filesIndexed: 0,
+  error: null as string | null,
 };
 
-const indexStatus: IndexStatus = {
-  isIndexing: false,
-  startedAt: null,
-  finishedAt: null,
-  lastError: null,
-  indexedLines: 0,
-  indexedFiles: 0,
-  directories: [],
-};
-
-// Single "mutex": only one indexing job is allowed to run. Starting a new job aborts the previous one.
-let currentIndexingAbortController: AbortController | null = null;
-let currentJobId = 0;
-
-// Adaptive batch size: reserve RAM so we don't freeze the OS (env overrides in bytes or MB)
-const RESERVE_MB = Number(process.env.SEMANTIC_SEARCH_RESERVE_MB) || 400;
-const RESERVE_BYTES = RESERVE_MB * 1024 * 1024;
-const MIN_BATCH = Number(process.env.SEMANTIC_SEARCH_MIN_BATCH) || 32;
-const MAX_BATCH = Number(process.env.SEMANTIC_SEARCH_MAX_BATCH) || 128;
-// Explicit GC after each batch (when --expose-gc is available). Default on; set SEMANTIC_SEARCH_GC=0 or false to disable.
-const ENABLE_GC = process.env.SEMANTIC_SEARCH_GC !== '0' && process.env.SEMANTIC_SEARCH_GC !== 'false';
-
-/** Rough bytes per indexed line in memory: line text + path + embedding (384 floats) + overhead */
-const BYTES_PER_LINE_ESTIMATE = 4000;
-
-/**
- * Compute batch size from current free system RAM. Keeps reserve free to avoid freezing the OS.
- */
-function getAdaptiveBatchSize(): number {
-  const free = os.freemem();
-  const available = free > RESERVE_BYTES ? free - RESERVE_BYTES : Math.floor(free / 2);
-  const batch = Math.floor(available / BYTES_PER_LINE_ESTIMATE);
-  const clamped = Math.max(MIN_BATCH, Math.min(MAX_BATCH, batch));
-  return clamped;
-}
-
-/**
- * Request indexing of directories. If another indexing job is running, it is aborted first.
- * Then a new job is started (clears index and rebuilds).
- */
-function requestIndexing(storage: VectorStorage, directories: string[]): void {
-  if (!directories.length) {
-    console.error('No directories to index. Set SEMANTIC_SEARCH_INDEX_DIRS (comma-separated paths).');
-    return;
-  }
+// Batch size kept small to avoid high RAM usage during indexing
+const INDEX_BATCH_SIZE = 256;
 
-  // Abort any in-progress indexing so it doesn't conflict or flush this job's work.
-  if (currentIndexingAbortController) {
-    currentIndexingAbortController.abort();
-    currentIndexingAbortController = null;
-  }
-
-  currentJobId += 1;
-  const jobId = currentJobId;
-  currentIndexingAbortController = new AbortController();
-  const signal = currentIndexingAbortController.signal;
-
-  indexStatus.isIndexing = true;
-  indexStatus.startedAt = Date.now();
-  indexStatus.finishedAt = null;
-  indexStatus.lastError = null;
-  indexStatus.directories = directories;
-  indexStatus.indexedLines = 0;
-  indexStatus.indexedFiles = 0;
-
-  void startIndexing(storage, directories, signal, jobId);
-}
-
-async function startIndexing(
-  storage: VectorStorage,
-  directories: string[],
-  signal: AbortSignal,
-  jobId: number
+async function runBackgroundIndexing(
+  storage: Awaited<ReturnType<typeof createVectorStorage>>,
+  directories: string[]
 ): Promise<void> {
-  const isCurrentJob = (): boolean => currentJobId === jobId;
+  indexingState.inProgress = true;
+  indexingState.startedAt = new Date();
+  indexingState.linesIndexed = 0;
+  indexingState.filesIndexed = 0;
+  indexingState.error = null;
+  storage.clear();
 
   try {
-    if (signal.aborted) return;
-
-    storage.clear();
-    console.error(`Scanning ${directories.length} directories (background indexing)...`);
-
-    let indexedCount = 0;
-    // eslint-disable-next-line @typescript-eslint/no-explicit-any
-    let currentBatch: any[] = [];
-
-    const processBatch = async (batchToProcess: any[]) => {
-      if (batchToProcess.length === 0) return;
-      const contents = batchToProcess.map((l) => l.content);
+    let currentBatch: Array<{ filePath: string; lineNumber: number; content: string }> = [];
+    let processingPromise: Promise<void> | null = null;
+
+    let batchesProcessed = 0;
+    const processBatch = async (
+      batch: Array<{ filePath: string; lineNumber: number; content: string }>
+    ): Promise<void> => {
+      if (batch.length === 0) return;
+      const contents = batch.map((l) => l.content);
       const embeddings = await getBatchEmbeddings(contents);
-
-      const batchData = batchToProcess.map((line, idx) => ({
+      const batchData = batch.map((line, idx) => ({
         filePath: line.filePath,
         lineNumber: line.lineNumber,
         content: line.content,
         embedding: embeddings[idx],
       }));
-
       await storage.upsertLinesBatch(batchData);
-      indexedCount += batchToProcess.length;
-      if (isCurrentJob()) indexStatus.indexedLines = indexedCount;
-      console.error(`Indexed ${indexedCount} lines...`);
-      // Explicit GC when enabled (default) and Node run with --expose-gc
-      if (ENABLE_GC && typeof (globalThis as { gc?: () => void }).gc === 'function') {
-        (globalThis as { gc: () => void }).gc();
+      indexingState.linesIndexed += batch.length;
+      batchesProcessed++;
+      if (batchesProcessed === 1 || batchesProcessed % 10 === 0) {
+        const stats = await storage.getStats();
+        indexingState.filesIndexed = stats.totalFiles;
       }
     };
 
-    // Single task queue: only one batch is processed at a time (no pipelining).
-    // We do not read the next batch until the current one is fully done, to avoid memory spikes and OS freezes.
-    let batchSize = getAdaptiveBatchSize();
-    console.error(`Adaptive batch size: ${batchSize} (free RAM: ${Math.round(os.freemem() / 1024 / 1024)}MB, reserve: ${RESERVE_MB}MB)`);
+    const yieldToEventLoop = (): Promise<void> =>
+      new Promise((resolve) => setImmediate(resolve));
 
     for await (const line of scanDirectories(directories)) {
-      if (signal.aborted) break;
-
       currentBatch.push(line);
-      if (currentBatch.length >= batchSize) {
+      if (currentBatch.length >= INDEX_BATCH_SIZE) {
+        if (processingPromise) await processingPromise;
         const batchToProcess = currentBatch;
         currentBatch = [];
-        batchSize = getAdaptiveBatchSize();
-
-        await processBatch(batchToProcess);
-        if (signal.aborted) break;
+        processingPromise = processBatch(batchToProcess);
+        await yieldToEventLoop();
       }
     }
 
-    if (signal.aborted) {
-      console.error('Indexing aborted (new job started or cancelled).');
-      return;
-    }
-
-    if (currentBatch.length > 0) {
-      await processBatch(currentBatch);
-    }
-
-    if (!isCurrentJob()) return;
+    if (processingPromise) await processingPromise;
+    if (currentBatch.length > 0) await processBatch(currentBatch);
 
     const stats = await storage.getStats();
-    indexStatus.indexedFiles = stats.totalFiles;
-    indexStatus.indexedLines = stats.totalLines;
-    indexStatus.finishedAt = Date.now();
-
-    console.error(
-      `Finished indexing ${stats.totalLines} lines from ${stats.totalFiles} files in background job.`
-    );
+    indexingState.linesIndexed = stats.totalLines;
+    indexingState.filesIndexed = stats.totalFiles;
   } catch (err) {
-    const message = err instanceof Error ? err.message : String(err);
-    if (isCurrentJob()) {
-      indexStatus.lastError = message;
-      indexStatus.finishedAt = Date.now();
-    }
-    console.error('Error during indexing job:', err);
+    indexingState.error = err instanceof Error ? err.message : String(err);
+    console.error('Background indexing error:', indexingState.error);
   } finally {
-    if (isCurrentJob()) {
-      indexStatus.isIndexing = false;
-    }
-    if (currentIndexingAbortController && currentJobId === jobId) {
-      currentIndexingAbortController = null;
-    }
-  }
-}
-
-function ensureInitialIndexing(storage: VectorStorage): void {
-  const envDirs = process.env.SEMANTIC_SEARCH_INDEX_DIRS;
-  const directories = envDirs ? envDirs.split(',').map((d) => d.trim()).filter(Boolean) : [];
-
-  if (!directories.length) {
-    console.error(
-      'Semantic Search MCP: SEMANTIC_SEARCH_INDEX_DIRS is not set; automatic indexing on startup is disabled.'
-    );
-    return;
+    indexingState.inProgress = false;
   }
-
-  requestIndexing(storage, directories);
 }
 
 async function main() {
@@ -222,19 +111,9 @@ async function main() {
   server.setRequestHandler(ListToolsRequestSchema, async () => {
     return {
       tools: [
-        {
-          name: 'index_directories',
-          description:
-            'Trigger background indexing of directories from SEMANTIC_SEARCH_INDEX_DIRS (comma-separated). Clears and rebuilds the index asynchronously.',
-          inputSchema: {
-            type: 'object',
-            properties: {},
-          },
-        },
         {
           name: 'search',
-          description:
-            'Search for lines semantically similar to the query. Returns the most relevant lines from indexed files.',
+          description: 'Search for lines semantically similar to the query. Returns the most relevant lines from indexed files.',
           inputSchema: {
             type: 'object',
             properties: {
@@ -253,7 +132,7 @@ async function main() {
         },
         {
           name: 'get_index_stats',
-          description: 'Get statistics and progress for the current index (files, lines, progress state)',
+          description: 'Get statistics about the current index (number of files and lines indexed). If indexing is running in the background, returns progress and start time (locale-formatted).',
           inputSchema: {
             type: 'object',
             properties: {},
@@ -268,40 +147,6 @@ async function main() {
 
     try {
       switch (name) {
-        case 'index_directories': {
-          const envDirs = process.env.SEMANTIC_SEARCH_INDEX_DIRS;
-          const directories = envDirs ? envDirs.split(',').map((d) => d.trim()).filter(Boolean) : [];
-          if (!directories.length) {
-            throw new Error(
-              'No directories to index. Set SEMANTIC_SEARCH_INDEX_DIRS (comma-separated paths).'
-            );
-          }
-
-          // Abort any in-progress indexing and start a new job (clears and rebuilds).
-          requestIndexing(storage, directories);
-
-          const stats = await storage.getStats();
-          return {
-            content: [
-              {
-                type: 'text',
-                text: JSON.stringify({
-                  success: true,
-                  indexing: indexStatus.isIndexing,
-                  indexed_lines: stats.totalLines,
-                  indexed_files: stats.totalFiles,
-                  started_at: indexStatus.startedAt,
-                  finished_at: indexStatus.finishedAt,
-                  last_error: indexStatus.lastError,
-                  message: indexStatus.isIndexing
-                    ? `Indexing started in background. Currently ${stats.totalLines} lines from ${stats.totalFiles} files in index.`
-                    : `Indexing completed. Indexed ${stats.totalLines} lines from ${stats.totalFiles} files.`,
-                }),
-              },
-            ],
-          };
-        }
-
         case 'search': {
           const query = (args as { query: string; limit?: number }).query;
           const limit = (args as { query: string; limit?: number }).limit ?? 10;
@@ -329,21 +174,41 @@ async function main() {
 
         case 'get_index_stats': {
           const stats = await storage.getStats();
+          const payload: {
+            total_files: number;
+            total_lines: number;
+            indexing?: {
+              in_progress: boolean;
+              started_at: string;
+              lines_indexed_so_far: number;
+              files_indexed_so_far: number;
+              error?: string;
+            };
+          } = {
+            total_files: stats.totalFiles,
+            total_lines: stats.totalLines,
+          };
+          if (indexingState.inProgress && indexingState.startedAt) {
+            payload.indexing = {
+              in_progress: true,
+              started_at: indexingState.startedAt.toLocaleString(),
+              lines_indexed_so_far: indexingState.linesIndexed,
+              files_indexed_so_far: indexingState.filesIndexed,
+            };
+          } else if (indexingState.error) {
+            payload.indexing = {
+              in_progress: false,
+              started_at: indexingState.startedAt?.toLocaleString() ?? '',
+              lines_indexed_so_far: indexingState.linesIndexed,
+              files_indexed_so_far: indexingState.filesIndexed,
+              error: indexingState.error,
+            };
+          }
           return {
             content: [
               {
                 type: 'text',
-                text: JSON.stringify({
-                  total_files: stats.totalFiles,
-                  total_lines: stats.totalLines,
-                  is_indexing: indexStatus.isIndexing,
-                  indexed_lines: indexStatus.indexedLines,
-                  indexed_files: indexStatus.indexedFiles,
-                  started_at: indexStatus.startedAt,
-                  finished_at: indexStatus.finishedAt,
-                  last_error: indexStatus.lastError,
-                  directories: indexStatus.directories,
-                }),
+                text: JSON.stringify(payload),
               },
             ],
           };
@@ -365,8 +230,14 @@ async function main() {
   await server.connect(transport);
   console.error('Semantic Search MCP Server running on stdio');
 
-  // Kick off initial background indexing when the MCP server is enabled.
-  ensureInitialIndexing(storage);
+  const envDirs = process.env.SEMANTIC_SEARCH_INDEX_DIRS;
+  const directories = envDirs ? envDirs.split(',').map((d) => d.trim()).filter(Boolean) : [];
+  if (directories.length > 0) {
+    console.error(`Starting background indexing for ${directories.length} directories...`);
+    runBackgroundIndexing(storage, directories).catch((err) => {
+      console.error('Background indexing failed:', err);
+    });
+  }
 }
 
 main().catch(console.error);

package/src/scanner.ts

@@ -1,5 +1,5 @@
 import { createReadStream, statSync } from 'fs';
-import { glob } from 'glob';
+import { globIterate } from 'glob';
 import path from 'path';
 import readline from 'readline';
 
@@ -18,6 +18,9 @@ const MIN_LINE_LENGTH = 5;
 // Maximum file size to process (skip very large files)
 const MAX_FILE_SIZE = 10 * 1024 * 1024; // 10MB
 
+// Cap line length to avoid unbounded readline buffer (e.g. file with no newlines)
+const MAX_LINE_LENGTH = 256 * 1024; // 256KB
+
 /**
  * Check if a file is a text file we should index
  */
@@ -27,18 +30,17 @@ function isTextFile(filePath: string): boolean {
 }
 
 /**
- * Scan a directory for text files and yield lines
+ * Scan a directory for text files and yield lines.
+ * Uses globIterate so file paths are streamed one-by-one (no full list in RAM).
+ * Readline has maxLineLength to avoid huge single-line buffers.
  */
 export async function* scanDirectory(dirPath: string): AsyncGenerator<FileLine> {
-  // Find all files in directory recursively
   const pattern = path.join(dirPath, '**/*');
-  // Nodir: true ensures we only get files
-  const files = await glob(pattern, { nodir: true, absolute: true });
-
-  for (const filePath of files) {
+  for await (const filePath of globIterate(pattern, { nodir: true, absolute: true }) as AsyncIterable<string>) {
     if (!isTextFile(filePath)) continue;
 
     let fileStream: ReturnType<typeof createReadStream> | null = null;
+    let rl: readline.Interface | null = null;
     try {
       const stats = statSync(filePath);
       if (stats.size > MAX_FILE_SIZE) {
@@ -47,34 +49,29 @@ export async function* scanDirectory(dirPath: string): AsyncGenerator<FileLine>
       }
 
       fileStream = createReadStream(filePath);
-      const rl = readline.createInterface({
+      rl = readline.createInterface({
         input: fileStream,
         crlfDelay: Infinity,
+        maxLineLength: MAX_LINE_LENGTH,
       });
 
-      try {
-        let lineNumber = 0;
-        for await (const line of rl) {
-          lineNumber++;
-          const trimmed = line.trim();
-          if (trimmed.length >= MIN_LINE_LENGTH) {
-            yield {
-              filePath,
-              lineNumber,
-              content: trimmed,
-            };
-          }
+      let lineNumber = 0;
+      for await (const line of rl) {
+        lineNumber++;
+        const trimmed = line.trim();
+        if (trimmed.length >= MIN_LINE_LENGTH) {
+          yield {
+            filePath,
+            lineNumber,
+            content: trimmed,
+          };
         }
-      } finally {
-        rl.close();
-        fileStream.destroy();
-        fileStream = null;
       }
     } catch (err) {
       console.error(`Error reading file ${filePath}:`, err);
-      if (fileStream) {
-        fileStream.destroy();
-      }
+    } finally {
+      rl?.close();
+      fileStream?.destroy();
     }
   }
 }

package/src/storage.ts

@@ -31,6 +31,10 @@ export class VectorStorage {
   }
 
   private init(): void {
+    // Limit SQLite page cache to avoid unbounded RAM (negative = kibibytes)
+    try {
+      this.db.exec('PRAGMA cache_size = -65536'); // 64MB max
+    } catch {}
     this.db.exec(`
       CREATE TABLE IF NOT EXISTS lines (
         id INTEGER PRIMARY KEY AUTOINCREMENT,