raggrep 0.1.7 → 0.2.0

package/README.md

@@ -1,160 +1,127 @@
 # RAGgrep
 
-**Local filesystem-based RAG system for codebases** — semantic search using local embeddings.
+**Local semantic search for codebases** — find code using natural language queries.
 
-RAGgrep indexes your code and allows semantic search using natural language queries. Everything runs locally on your machine — no external API calls required.
+RAGgrep indexes your code and lets you search it using natural language. Everything runs locally — no external API calls required.
 
 ## Features
 
-- **🏠 Local-first** — All indexing and search happens locally. No cloud dependencies.
-- **📁 Filesystem-based** — Index stored as readable JSON files in system temp directory.
-- **⚡ Tiered search** — Fast keyword filtering + semantic search for efficiency.
-- **🔍 Hybrid scoring** — Combines semantic similarity with BM25 keyword matching.
-- **🔄 Incremental** — Only re-indexes files that have changed.
-- **📝 TypeScript-optimized** — AST-based parsing extracts functions, classes, interfaces, types.
-- **🎯 Zero config** — Works out of the box with sensible defaults.
+- **Zero-config search** — Just run `raggrep query` and it works. Index is created and updated automatically.
+- **Local-first** — All indexing and search happens on your machine. No cloud dependencies.
+- **Incremental** — Only re-indexes files that have changed. Instant search when nothing changed.
+- **Watch mode** — Keep the index fresh in real-time as you code.
+- **Hybrid search** — Combines semantic similarity with keyword matching for best results.
 
 ## Installation
 
 ```bash
-# Install globally with npm
+# Install globally
 npm install -g raggrep
 
-# Or with Bun (recommended)
-bun install -g raggrep
-
 # Or use without installing
-npx raggrep --help
+npx raggrep query "your search"
 ```
 
-## Quick Start
+## Usage
+
+### Search Your Code
 
 ```bash
-# Index your project
 cd your-project
-raggrep index
-
-# Search your codebase
 raggrep query "user authentication"
 ```
 
+That's it. The first query creates the index automatically. Subsequent queries are instant if files haven't changed. Modified files are re-indexed on the fly.
+
 ### Example Output
 
 ```
+Index updated: 42 indexed
+
+RAGgrep Search
+==============
+
+Searching for: "user authentication"
+
 Found 3 results:
 
 1. src/auth/authService.ts:24-55 (login)
-   Score: 34.4% | Type: function | exported
-      export async function login(credentials: LoginCredentials): Promise<AuthResult> ...
+   Score: 34.4% | Type: function | via TypeScript | exported
+      export async function login(credentials: LoginCredentials): Promise<AuthResult> {
+        const { email, password } = credentials;
 
-2. src/auth/authService.ts:60-62 (logout)
-   Score: 27.5% | Type: function | exported
-      export async function logout(token: string): Promise<void> {
+2. src/auth/session.ts:10-25 (createSession)
+   Score: 28.2% | Type: function | via TypeScript | exported
+      export function createSession(user: User): Session {
 
 3. src/users/types.ts:3-12 (User)
-   Score: 26.0% | Type: interface | exported
+   Score: 26.0% | Type: interface | via TypeScript | exported
       export interface User {
         id: string;
 ```
 
-## Programmatic API
-
-```typescript
-import raggrep from "raggrep";
-
-// Index a directory
-await raggrep.index("./my-project");
+### Watch Mode
 
-// Search
-const results = await raggrep.search("./my-project", "user authentication");
-console.log(raggrep.formatSearchResults(results));
+Keep your index fresh in real-time while you code:
 
-// Cleanup stale entries
-await raggrep.cleanup("./my-project");
+```bash
+raggrep index --watch
 ```
 
-## CLI Reference
+This monitors file changes and re-indexes automatically. Useful during active development when you want instant search results.
 
-```bash
-# Index commands
-raggrep index                              # Index current directory
-raggrep index --watch                      # Watch mode: re-index on file changes
-raggrep index --model bge-small-en-v1.5    # Use different embedding model
-raggrep index --verbose                    # Show detailed progress
-
-# Search commands
-raggrep query "user login"                 # Basic search
-raggrep query "error handling" --top 5     # Limit results
-raggrep query "database" --min-score 0.1   # Lower threshold (more results)
-raggrep query "interface" --type ts        # Filter by file type
-
-# Maintenance
-raggrep cleanup                            # Remove stale index entries
-raggrep status                             # Show index status
 ```
+┌─────────────────────────────────────────┐
+│  Watching for changes... (Ctrl+C to stop) │
+└─────────────────────────────────────────┘
 
-## How It Works
+[Watch] language/typescript: 2 indexed, 0 errors
+```
 
-RAGgrep uses a **dual-module architecture** with two complementary index types:
+## CLI Quick Reference
 
-### Core Module
+```bash
+# Search (auto-indexes if needed)
+raggrep query "user login"
+raggrep query "error handling" --top 5
+raggrep query "database" --type ts
 
-- **Language-agnostic** regex-based symbol extraction
-- **BM25 keyword matching** for fast, deterministic search
-- Works on any text file
+# Watch mode
+raggrep index --watch
 
-### TypeScript Module
+# Check index status
+raggrep status
+```
 
-- **AST-based parsing** via TypeScript Compiler API
-- **Semantic embeddings** for natural language understanding
-- **Symbolic index** for fast BM25 candidate filtering
+## How It Works
 
-Search combines results from both modules:
+1. **First query** — Creates the index (takes 1-2 min for ~1000 files)
+2. **Subsequent queries** — Uses cached index (instant if no changes)
+3. **Files changed** — Re-indexes only modified files automatically
+4. **Files deleted** — Stale entries cleaned up automatically
 
-```
-Query → Core (symbol/BM25) ─┐
-                           ├→ Merge & rank → Results
-Query → TypeScript (BM25 filter → semantic) ─┘
-```
+The index is stored in a system temp directory, keeping your project clean.
 
 ## What Gets Indexed
 
-**File types:** `.ts`, `.tsx`, `.js`, `.jsx`, `.py`, `.go`, `.rs`, `.java`, `.md`
-
-**Code structures:**
-
-- Functions (regular, async, arrow)
-- Classes (including abstract)
-- Interfaces
-- Type aliases
-- Enums
-- Exported variables
+**File types:** `.ts`, `.tsx`, `.js`, `.jsx`, `.py`, `.go`, `.rs`, `.java`, `.md`, `.txt`
 
-**Automatically ignored:**
+**Code structures:** Functions, classes, interfaces, types, enums, exports
 
-- `node_modules`, `dist`, `build`, `.git`
-- `.next`, `.nuxt`, `__pycache__`, `venv`
-- See [Configuration](./docs/configuration.md) for full list
+**Automatically ignored:** `node_modules`, `dist`, `build`, `.git`, and other common directories
 
 ## Documentation
 
-- [Getting Started](./docs/getting-started.md) — Installation and first steps
+- [Getting Started](./docs/getting-started.md) — Installation options and first steps
 - [CLI Reference](./docs/cli-reference.md) — All commands and options
-- [Configuration](./docs/configuration.md) — Customize indexing behavior
+- [SDK Reference](./docs/sdk.md) — Programmatic API for Node.js/Bun
+- [Advanced](./docs/advanced.md) — Configuration, maintenance commands
 - [Architecture](./docs/architecture.md) — How RAGgrep works internally
 
-## Performance
-
-| Operation                | Time       | Notes                                  |
-| ------------------------ | ---------- | -------------------------------------- |
-| Initial index (1k files) | 1-2 min    | Embedding generation is the bottleneck |
-| Incremental update       | <2s        | Only changed files                     |
-| Search                   | ~100-500ms | Depends on codebase size               |
-
 ## Requirements
 
 - Node.js 18+ or Bun 1.0+
-- ~50MB disk space for models (cached globally at `~/.cache/raggrep/models/`)
+- ~50MB disk space for models (cached at `~/.cache/raggrep/models/`)
 
 ## License
 

package/dist/app/indexer/index.d.ts

@@ -10,6 +10,16 @@ export interface IndexOptions {
     model?: EmbeddingModelName;
     /** Show detailed progress */
     verbose?: boolean;
+    /** Suppress most output (for use during query) */
+    quiet?: boolean;
+}
+export interface EnsureFreshResult {
+    /** Number of files indexed (new or modified) */
+    indexed: number;
+    /** Number of stale entries removed (deleted files) */
+    removed: number;
+    /** Number of files unchanged (used cache) */
+    unchanged: number;
 }
 export interface CleanupResult {
     moduleId: string;
@@ -40,6 +50,21 @@ export interface IndexStatus {
  * Index a directory using all enabled modules
  */
 export declare function indexDirectory(rootDir: string, options?: IndexOptions): Promise<IndexResult[]>;
+/**
+ * Ensure the index is fresh by checking for changes and updating incrementally.
+ * This function is designed to be called before search to transparently manage the index.
+ *
+ * - If no index exists, creates a full index
+ * - If index version is incompatible, rebuilds from scratch
+ * - If files have changed, re-indexes only the modified files
+ * - If files have been deleted, removes stale entries
+ * - If nothing changed, returns immediately (uses cache)
+ *
+ * @param rootDir - Root directory of the project
+ * @param options - Index options
+ * @returns Statistics about what was updated
+ */
+export declare function ensureIndexFresh(rootDir: string, options?: IndexOptions): Promise<EnsureFreshResult>;
 /**
  * Clean up stale index entries for files that no longer exist
  * @param rootDir - Root directory of the project

package/dist/cli/main.js

@@ -3383,6 +3383,7 @@ __export(exports_indexer, {
   watchDirectory: () => watchDirectory,
   indexDirectory: () => indexDirectory,
   getIndexStatus: () => getIndexStatus,
+  ensureIndexFresh: () => ensureIndexFresh,
   cleanupIndex: () => cleanupIndex
 });
 import { glob } from "glob";
@@ -3390,10 +3391,13 @@ import * as fs6 from "fs/promises";
 import * as path12 from "path";
 async function indexDirectory(rootDir, options = {}) {
   const verbose = options.verbose ?? false;
+  const quiet = options.quiet ?? false;
   rootDir = path12.resolve(rootDir);
   const location = getIndexLocation(rootDir);
-  console.log(`Indexing directory: ${rootDir}`);
-  console.log(`Index location: ${location.indexDir}`);
+  if (!quiet) {
+    console.log(`Indexing directory: ${rootDir}`);
+    console.log(`Index location: ${location.indexDir}`);
+  }
   const config = await loadConfig(rootDir);
   const introspection = new IntrospectionIndex(rootDir);
   await introspection.initialize();
@@ -3406,16 +3410,24 @@ async function indexDirectory(rootDir, options = {}) {
   await registerBuiltInModules();
   const enabledModules = registry.getEnabled(config);
   if (enabledModules.length === 0) {
-    console.log("No modules enabled. Check your configuration.");
+    if (!quiet) {
+      console.log("No modules enabled. Check your configuration.");
+    }
     return [];
   }
-  console.log(`Enabled modules: ${enabledModules.map((m) => m.id).join(", ")}`);
+  if (!quiet) {
+    console.log(`Enabled modules: ${enabledModules.map((m) => m.id).join(", ")}`);
+  }
   const files = await findFiles(rootDir, config);
-  console.log(`Found ${files.length} files to index`);
+  if (!quiet) {
+    console.log(`Found ${files.length} files to index`);
+  }
   const results = [];
   for (const module of enabledModules) {
-    console.log(`
+    if (!quiet) {
+      console.log(`
 [${module.name}] Starting indexing...`);
+    }
     const moduleConfig = getModuleConfig(config, module.id);
     if (module.initialize && moduleConfig) {
       const configWithOverrides = { ...moduleConfig };
@@ -3430,7 +3442,9 @@ async function indexDirectory(rootDir, options = {}) {
     const result = await indexWithModule(rootDir, files, module, config, verbose, introspection);
     results.push(result);
     if (module.finalize) {
-      console.log(`[${module.name}] Building secondary indexes...`);
+      if (!quiet) {
+        console.log(`[${module.name}] Building secondary indexes...`);
+      }
       const ctx = {
         rootDir,
         config,
@@ -3446,12 +3460,167 @@ async function indexDirectory(rootDir, options = {}) {
       };
       await module.finalize(ctx);
     }
-    console.log(`[${module.name}] Complete: ${result.indexed} indexed, ${result.skipped} skipped, ${result.errors} errors`);
+    if (!quiet) {
+      console.log(`[${module.name}] Complete: ${result.indexed} indexed, ${result.skipped} skipped, ${result.errors} errors`);
+    }
   }
   await introspection.save(config);
   await updateGlobalManifest(rootDir, enabledModules, config);
   return results;
 }
+async function isIndexVersionCompatible(rootDir) {
+  const config = await loadConfig(rootDir);
+  const globalManifestPath = getGlobalManifestPath(rootDir, config);
+  try {
+    const content = await fs6.readFile(globalManifestPath, "utf-8");
+    const manifest = JSON.parse(content);
+    return manifest.version === INDEX_SCHEMA_VERSION;
+  } catch {
+    return false;
+  }
+}
+async function deleteIndex(rootDir) {
+  const indexDir = getRaggrepDir(rootDir);
+  try {
+    await fs6.rm(indexDir, { recursive: true, force: true });
+  } catch {}
+}
+async function ensureIndexFresh(rootDir, options = {}) {
+  const verbose = options.verbose ?? false;
+  const quiet = options.quiet ?? false;
+  rootDir = path12.resolve(rootDir);
+  const status = await getIndexStatus(rootDir);
+  if (!status.exists) {
+    if (!quiet) {
+      console.log(`No index found. Creating index...
+`);
+    }
+    const results = await indexDirectory(rootDir, { ...options, quiet });
+    const totalIndexed2 = results.reduce((sum, r) => sum + r.indexed, 0);
+    return { indexed: totalIndexed2, removed: 0, unchanged: 0 };
+  }
+  const versionCompatible = await isIndexVersionCompatible(rootDir);
+  if (!versionCompatible) {
+    if (!quiet) {
+      console.log(`Index version incompatible. Rebuilding...
+`);
+    }
+    await deleteIndex(rootDir);
+    const results = await indexDirectory(rootDir, { ...options, quiet });
+    const totalIndexed2 = results.reduce((sum, r) => sum + r.indexed, 0);
+    return { indexed: totalIndexed2, removed: 0, unchanged: 0 };
+  }
+  const config = await loadConfig(rootDir);
+  await registerBuiltInModules();
+  const enabledModules = registry.getEnabled(config);
+  if (enabledModules.length === 0) {
+    return { indexed: 0, removed: 0, unchanged: 0 };
+  }
+  const introspection = new IntrospectionIndex(rootDir);
+  await introspection.initialize();
+  const currentFiles = await findFiles(rootDir, config);
+  const currentFileSet = new Set(currentFiles.map((f) => path12.relative(rootDir, f)));
+  let totalIndexed = 0;
+  let totalRemoved = 0;
+  let totalUnchanged = 0;
+  for (const module of enabledModules) {
+    const moduleConfig = getModuleConfig(config, module.id);
+    if (module.initialize && moduleConfig) {
+      const configWithOverrides = { ...moduleConfig };
+      if (options.model && module.id === "language/typescript") {
+        configWithOverrides.options = {
+          ...configWithOverrides.options,
+          embeddingModel: options.model
+        };
+      }
+      await module.initialize(configWithOverrides);
+    }
+    const manifest = await loadModuleManifest(rootDir, module.id, config);
+    const indexPath = getModuleIndexPath(rootDir, module.id, config);
+    const filesToRemove = [];
+    for (const filepath of Object.keys(manifest.files)) {
+      if (!currentFileSet.has(filepath)) {
+        filesToRemove.push(filepath);
+      }
+    }
+    for (const filepath of filesToRemove) {
+      if (verbose) {
+        console.log(`  Removing stale: ${filepath}`);
+      }
+      const indexFilePath = path12.join(indexPath, filepath.replace(/\.[^.]+$/, ".json"));
+      try {
+        await fs6.unlink(indexFilePath);
+      } catch {}
+      delete manifest.files[filepath];
+      totalRemoved++;
+    }
+    const ctx = {
+      rootDir,
+      config,
+      readFile: async (filepath) => {
+        const fullPath = path12.isAbsolute(filepath) ? filepath : path12.join(rootDir, filepath);
+        return fs6.readFile(fullPath, "utf-8");
+      },
+      getFileStats: async (filepath) => {
+        const fullPath = path12.isAbsolute(filepath) ? filepath : path12.join(rootDir, filepath);
+        const stats = await fs6.stat(fullPath);
+        return { lastModified: stats.mtime.toISOString() };
+      },
+      getIntrospection: (filepath) => introspection.getFile(filepath)
+    };
+    for (const filepath of currentFiles) {
+      const relativePath = path12.relative(rootDir, filepath);
+      try {
+        const stats = await fs6.stat(filepath);
+        const lastModified = stats.mtime.toISOString();
+        const existingEntry = manifest.files[relativePath];
+        if (existingEntry && existingEntry.lastModified === lastModified) {
+          totalUnchanged++;
+          continue;
+        }
+        if (verbose) {
+          console.log(`  Indexing: ${relativePath}`);
+        }
+        const content = await fs6.readFile(filepath, "utf-8");
+        introspection.addFile(relativePath, content);
+        const fileIndex = await module.indexFile(relativePath, content, ctx);
+        if (fileIndex) {
+          await writeFileIndex(rootDir, module.id, relativePath, fileIndex, config);
+          manifest.files[relativePath] = {
+            lastModified,
+            chunkCount: fileIndex.chunks.length
+          };
+          totalIndexed++;
+        }
+      } catch (error) {
+        if (verbose) {
+          console.error(`  Error indexing ${relativePath}:`, error);
+        }
+      }
+    }
+    if (totalIndexed > 0 || totalRemoved > 0) {
+      manifest.lastUpdated = new Date().toISOString();
+      await writeModuleManifest(rootDir, module.id, manifest, config);
+      if (module.finalize) {
+        await module.finalize(ctx);
+      }
+    }
+    if (totalRemoved > 0) {
+      await cleanupEmptyDirectories(indexPath);
+    }
+  }
+  if (totalIndexed > 0) {
+    await introspection.save(config);
+  }
+  if (totalIndexed > 0 || totalRemoved > 0) {
+    await updateGlobalManifest(rootDir, enabledModules, config);
+  }
+  return {
+    indexed: totalIndexed,
+    removed: totalRemoved,
+    unchanged: totalUnchanged
+  };
+}
 async function indexWithModule(rootDir, files, module, config, verbose, introspection) {
   const result = {
     moduleId: module.id,
@@ -3557,7 +3726,7 @@ async function writeFileIndex(rootDir, moduleId, filepath, fileIndex, config) {
 async function updateGlobalManifest(rootDir, modules, config) {
   const manifestPath = getGlobalManifestPath(rootDir, config);
   const manifest = {
-    version: config.version,
+    version: INDEX_SCHEMA_VERSION,
     lastUpdated: new Date().toISOString(),
     modules: modules.map((m) => m.id)
   };
@@ -3697,6 +3866,7 @@ async function getIndexStatus(rootDir) {
   }
   return status;
 }
+var INDEX_SCHEMA_VERSION = "1.0.0";
 var init_indexer = __esm(() => {
   init_config2();
   init_registry();
@@ -3851,7 +4021,7 @@ init_embeddings();
 // package.json
 var package_default = {
   name: "raggrep",
-  version: "0.1.7",
+  version: "0.2.0",
   description: "Local filesystem-based RAG system for codebases - semantic search using local embeddings",
   type: "module",
   main: "./dist/index.js",
@@ -4093,8 +4263,11 @@ Options:
   -h, --help           Show this help message
 
 Note:
-  If the current directory has not been indexed, raggrep will
-  automatically index it before searching.
+  The index is managed automatically like a cache:
+  - First query creates the index
+  - Changed files are re-indexed automatically
+  - Deleted files are cleaned up automatically
+  - Unchanged files use the cached index (instant)
 
 Examples:
   raggrep query "user authentication"
@@ -4105,7 +4278,7 @@ Examples:
         process.exit(0);
       }
       const { search: search2, formatSearchResults: formatSearchResults2 } = await Promise.resolve().then(() => (init_search(), exports_search));
-      const { getIndexStatus: getIndexStatus2, indexDirectory: indexDirectory2 } = await Promise.resolve().then(() => (init_indexer(), exports_indexer));
+      const { ensureIndexFresh: ensureIndexFresh2 } = await Promise.resolve().then(() => (init_indexer(), exports_indexer));
       const query = flags.remaining[0];
       if (!query) {
         console.error("Usage: raggrep query <search query>");
@@ -4113,24 +4286,20 @@ Examples:
         process.exit(1);
       }
       try {
-        const status = await getIndexStatus2(process.cwd());
-        if (!status.exists) {
-          console.log(`No index found. Indexing directory first...
-`);
-          console.log("RAGgrep Indexer");
-          console.log(`================
-`);
-          const indexResults = await indexDirectory2(process.cwd(), {
-            model: flags.model,
-            verbose: false
-          });
-          console.log(`
-================`);
-          console.log("Summary:");
-          for (const result of indexResults) {
-            console.log(`  ${result.moduleId}: ${result.indexed} indexed, ${result.skipped} skipped, ${result.errors} errors`);
+        const freshStats = await ensureIndexFresh2(process.cwd(), {
+          model: flags.model,
+          quiet: true
+        });
+        if (freshStats.indexed > 0 || freshStats.removed > 0) {
+          const parts = [];
+          if (freshStats.indexed > 0) {
+            parts.push(`${freshStats.indexed} indexed`);
           }
-          console.log("");
+          if (freshStats.removed > 0) {
+            parts.push(`${freshStats.removed} removed`);
+          }
+          console.log(`Index updated: ${parts.join(", ")}
+`);
         }
         console.log("RAGgrep Search");
         console.log(`==============
@@ -4286,4 +4455,4 @@ Run 'raggrep <command> --help' for more information.
 }
 main();
 
-//# debugId=C248CB1C621D0FC764756E2164756E21
+//# debugId=4B6F0EA9EEB7164864756E2164756E21