smart-coding-mcp 2.2.0 → 2.3.0

package/docs/ide-setup/opencode.md

@@ -0,0 +1,157 @@
+# OpenCode Integration
+
+OpenCode is a powerful AI coding agent built for the terminal by SST.
+
+## Configuration Location
+
+Edit the configuration file located at:
+
+- **Global:** `~/.config/opencode/opencode.json`
+- **Project:** `opencode.json` in your project root
+
+## MCP Configuration
+
+Add the server to the `mcp` object in your `opencode.json`:
+
+```json
+{
+  "$schema": "https://opencode.ai/config.json",
+  "mcp": {
+    "smart-coding-mcp": {
+      "type": "local",
+      "command": ["npx", "-y", "smart-coding-mcp", "--workspace", "/absolute/path/to/your/project"],
+      "enabled": true
+    }
+  }
+}
+```
+
+### With Environment Variables
+
+```json
+{
+  "$schema": "https://opencode.ai/config.json",
+  "mcp": {
+    "smart-coding-mcp": {
+      "type": "local",
+      "command": ["npx", "-y", "smart-coding-mcp", "--workspace", "/path/to/project"],
+      "environment": {
+        "SMART_CODING_VERBOSE": "true",
+        "SMART_CODING_MAX_RESULTS": "10"
+      },
+      "enabled": true
+    }
+  }
+}
+```
+
+> **Note:** OpenCode does **NOT** support `${workspaceFolder}`. You must use absolute paths.
+
+---
+
+## Configuring Rules (AGENTS.md)
+
+OpenCode uses `AGENTS.md` files for custom instructions, similar to `CLAUDE.md` or Cursor's rules.
+
+### Rule Locations
+
+| Scope       | Location                          |
+| ----------- | --------------------------------- |
+| **Global**  | `~/.config/opencode/AGENTS.md`    |
+| **Project** | `AGENTS.md` in project root       |
+
+### Creating a Rule
+
+1. **Create an `AGENTS.md` file** in your project root:
+
+   ```markdown
+   # Smart Coding MCP Usage Rules
+
+   You must prioritize using the **Smart Coding MCP** tools for the following tasks.
+
+   ## 1. Dependency Management
+
+   **Trigger:** When checking, adding, or updating package versions (npm, python, go, rust, etc.).
+   **Action:**
+
+   - **MUST** use the `d_check_last_version` tool.
+   - **DO NOT** guess versions or trust internal training data.
+   - **DO NOT** use generic web search unless `d_check_last_version` fails.
+
+   ## 2. Codebase Research
+
+   **Trigger:** When asking about "how" something works, finding logic, or understanding architecture.
+   **Action:**
+
+   - **MUST** use `a_semantic_search` as the FIRST tool for any codebase research
+   - **DO NOT** use `Glob` or `Grep` for exploratory searches
+   - Use `Grep` ONLY for exact literal string matching (e.g., finding a specific error message)
+   - Use `Glob` ONLY when you already know the exact filename pattern
+
+   ## 3. Environment & Status
+
+   **Trigger:** When starting a session or debugging the environment.
+   **Action:**
+
+   - Use `e_set_workspace` if the current workspace path is incorrect.
+   - Use `f_get_status` to verify the MCP server is healthy and indexed.
+   ```
+
+### Alternative: Using opencode.json Instructions Field
+
+You can reference external instruction files using glob patterns:
+
+```json
+{
+  "$schema": "https://opencode.ai/config.json",
+  "instructions": [
+    "AGENTS.md",
+    "docs/guidelines.md",
+    ".cursor/rules/*.md"
+  ],
+  "mcp": {
+    "smart-coding-mcp": {
+      "type": "local",
+      "command": ["npx", "-y", "smart-coding-mcp", "--workspace", "/path/to/project"],
+      "enabled": true
+    }
+  }
+}
+```
+
+### Global Rules
+
+For rules that apply to all projects, edit `~/.config/opencode/AGENTS.md`:
+
+```markdown
+# Global Agent Rules
+
+- Always verify package versions before installing using `d_check_last_version`
+- Prefer semantic search (`a_semantic_search`) when available
+- Check MCP server status with `f_get_status` at session start
+```
+
+---
+
+## MCP Management Commands
+
+OpenCode provides CLI commands for managing MCP servers:
+
+```bash
+# List all configured MCP servers
+opencode mcp list
+
+# Authenticate with a server (if required)
+opencode mcp auth smart-coding-mcp
+
+# Remove credentials
+opencode mcp logout smart-coding-mcp
+```
+
+---
+
+## Verification
+
+1. Start OpenCode in your project directory.
+2. Run `opencode mcp list` to verify the server is connected.
+3. Ask the AI: "Check the status of the smart coding MCP server" - it should use `f_get_status`.

package/docs/ide-setup/raycast.md

@@ -70,9 +70,10 @@ Raycast uses **AI Commands** and **Presets** for custom instructions.
      - **DO NOT** use generic web search unless `d_check_last_version` fails.
 
      ## 2. Codebase Research
-     - **PREFER** `a_semantic_search` over `grep_search` or `find_by_name`.
-     - Use `a_semantic_search` for natural language queries (e.g., "where is the auth logic").
-     - Use standard tools (`grep`) only for exact string matching.
+     - **MUST** use `a_semantic_search` as the FIRST tool for any codebase research
+     - **DO NOT** use `Glob` or `Grep` for exploratory searches
+     - Use `Grep` ONLY for exact literal string matching (e.g., finding a specific error message)
+     - Use `Glob` ONLY when you already know the exact filename pattern
 
      ## 3. Environment & Status
      - Use `e_set_workspace` if the current workspace path is incorrect.

package/docs/ide-setup/vscode.md

@@ -93,9 +93,10 @@ Cline uses `.clinerules` files to control AI behavior.
    **Trigger:** When asking about "how" something works, finding logic, or understanding architecture.
    **Action:**
 
-   - **PREFER** `a_semantic_search` (Smart Coding) over `grep_search` or `find_by_name`.
-   - Use `a_semantic_search` for natural language queries (e.g., "where is the auth logic").
-   - Use standard tools (`grep`) only for exact string matching.
+   - **MUST** use `a_semantic_search` as the FIRST tool for any codebase research
+   - **DO NOT** use `Glob` or `Grep` for exploratory searches
+   - Use `Grep` ONLY for exact literal string matching (e.g., finding a specific error message)
+   - Use `Glob` ONLY when you already know the exact filename pattern
 
    ## 3. Environment & Status
 

package/docs/ide-setup/windsurf.md

@@ -65,9 +65,10 @@ You must prioritize using the **Smart Coding MCP** tools for the following tasks
 **Trigger:** When asking about "how" something works, finding logic, or understanding architecture.
 **Action:**
 
-- **PREFER** `a_semantic_search` (Smart Coding) over `grep_search` or `find_by_name`.
-- Use `a_semantic_search` for natural language queries (e.g., "where is the auth logic").
-- Use standard tools (`grep`) only for exact string matching.
+- **MUST** use `a_semantic_search` as the FIRST tool for any codebase research
+- **DO NOT** use `Glob` or `Grep` for exploratory searches
+- Use `Grep` ONLY for exact literal string matching (e.g., finding a specific error message)
+- Use `Glob` ONLY when you already know the exact filename pattern
 
 ## 3. Environment & Status
 

package/features/index-codebase.js

@@ -35,16 +35,23 @@ export class CodebaseIndexer {
 
   /**
    * Initialize worker thread pool for parallel embedding
+   * Note: Workers are disabled for nomic models due to ONNX runtime thread-safety issues
    */
   async initializeWorkers() {
-    // Force single-threaded mode for nomic models (transformers.js v3 worker thread issue)
+    // Workers don't work with nomic/transformers.js due to ONNX WASM thread-safety issues
     const isNomicModel = this.config.embeddingModel?.includes('nomic');
     if (isNomicModel) {
-      console.error("[Indexer] Single-threaded mode (nomic model - workers disabled for stability)");
+      console.error("[Indexer] Single-threaded mode (nomic model - ONNX workers incompatible)");
       return;
     }
 
-    const numWorkers = this.config.workerThreads === "auto" 
+    // Check if workers are explicitly disabled
+    if (this.config.workerThreads === 0 || this.config.disableWorkers) {
+      console.error("[Indexer] Single-threaded mode (workers disabled by config)");
+      return;
+    }
+
+    const numWorkers = this.config.workerThreads === "auto"
       ? this.throttle.maxWorkers  // Use throttled worker count
       : this.throttle.getWorkerCount(this.config.workerThreads);
 
@@ -266,30 +273,44 @@ export class CodebaseIndexer {
     if (this.config.verbose) {
       console.error(`[Indexer] Processing: ${fileName}...`);
     }
-    
+
     try {
       // Check file size first
       const stats = await fs.stat(file);
-      
+
       // Skip directories
       if (stats.isDirectory()) {
         return 0;
       }
-      
+
       if (stats.size > this.config.maxFileSize) {
         if (this.config.verbose) {
           console.error(`[Indexer] Skipped ${fileName} (too large: ${(stats.size / 1024 / 1024).toFixed(2)}MB)`);
         }
         return 0;
       }
-      
+
+      // OPTIMIZATION: Check mtime first (fast) before reading file content
+      const currentMtime = stats.mtimeMs;
+      const cachedMtime = this.cache.getFileMtime(file);
+
+      // If mtime unchanged, file definitely unchanged - skip without reading
+      if (cachedMtime && currentMtime === cachedMtime) {
+        if (this.config.verbose) {
+          console.error(`[Indexer] Skipped ${fileName} (unchanged - mtime)`);
+        }
+        return 0;
+      }
+
       const content = await fs.readFile(file, "utf-8");
       const hash = hashContent(content);
-      
-      // Skip if file hasn't changed
+
+      // Skip if file hasn't changed (content check after mtime indicated change)
       if (this.cache.getFileHash(file) === hash) {
+        // Content same but mtime different - update cached mtime
+        this.cache.setFileHash(file, hash, currentMtime);
         if (this.config.verbose) {
-          console.error(`[Indexer] Skipped ${fileName} (unchanged)`);
+          console.error(`[Indexer] Skipped ${fileName} (unchanged - hash)`);
         }
         return 0;
       }
@@ -321,7 +342,7 @@ export class CodebaseIndexer {
         }
       }
 
-      this.cache.setFileHash(file, hash);
+      this.cache.setFileHash(file, hash, currentMtime);
       if (this.config.verbose) {
         console.error(`[Indexer] Completed ${fileName} (${addedChunks} chunks)`);
       }
@@ -377,6 +398,65 @@ export class CodebaseIndexer {
     return files;
   }
 
+  /**
+   * Sort files by priority for progressive indexing
+   * Priority: recently modified files first (users likely searching for recent work)
+   */
+  async sortFilesByPriority(files) {
+    const startTime = Date.now();
+
+    // Get mtime for all files in parallel
+    const filesWithMtime = await Promise.all(
+      files.map(async (file) => {
+        try {
+          const stats = await fs.stat(file);
+          return { file, mtime: stats.mtimeMs };
+        } catch {
+          return { file, mtime: 0 };
+        }
+      })
+    );
+
+    // Sort by mtime descending (most recently modified first)
+    filesWithMtime.sort((a, b) => b.mtime - a.mtime);
+
+    if (this.config.verbose) {
+      console.error(`[Indexer] Priority sort: ${files.length} files in ${Date.now() - startTime}ms`);
+    }
+
+    return filesWithMtime.map(f => f.file);
+  }
+
+  /**
+   * Start background indexing (non-blocking)
+   * Allows search to work immediately with partial results
+   */
+  startBackgroundIndexing(force = false) {
+    if (this.isIndexing) {
+      console.error("[Indexer] Background indexing already in progress");
+      return;
+    }
+
+    console.error("[Indexer] Starting background indexing...");
+
+    // Run indexAll in background (don't await)
+    this.indexAll(force).then(result => {
+      console.error(`[Indexer] Background indexing complete: ${result.message || 'done'}`);
+    }).catch(err => {
+      console.error(`[Indexer] Background indexing error: ${err.message}`);
+    });
+  }
+
+  /**
+   * Get current indexing status for progressive search
+   */
+  getIndexingStatus() {
+    return {
+      ...this.indexingStatus,
+      isReady: !this.indexingStatus.inProgress || this.indexingStatus.processedFiles > 0
+    };
+  }
+
   /**
    * Pre-filter files by hash (skip unchanged files before processing)
    */
@@ -461,16 +541,21 @@ export class CodebaseIndexer {
     console.error(`[Indexer] Starting optimized indexing in ${this.config.searchDirectory}...`);
     
     // Step 1: Fast file discovery with fdir
-    const files = await this.discoverFiles();
-    
+    let files = await this.discoverFiles();
+
     if (files.length === 0) {
       console.error("[Indexer] No files found to index");
       this.sendProgress(100, 100, "No files found to index");
       return { skipped: false, filesProcessed: 0, chunksCreated: 0, message: "No files found to index" };
     }
 
+    // Step 1.1: Sort files by priority (recently modified first) for progressive indexing
+    // This ensures search results are useful even while indexing is in progress
+    files = await this.sortFilesByPriority(files);
+    console.error(`[Indexer] Progressive mode: recently modified files will be indexed first`);
+
     // Send progress: discovery complete
-    this.sendProgress(5, 100, `Discovered ${files.length} files`);
+    this.sendProgress(5, 100, `Discovered ${files.length} files (sorted by priority)`);
 
     // Step 1.5: Prune deleted or excluded files from cache
     if (!force) {
@@ -494,13 +579,15 @@ export class CodebaseIndexer {
       }
     }
 
-    // Step 2: Process files in adaptive batches with lazy filtering
-    // Instead of pre-filtering all files (expensive), check hashes during processing
-    const adaptiveBatchSize = files.length > 10000 ? 500 :
-                              files.length > 1000 ? 200 : 
+    // Step 2: Process files with progressive indexing
+    // Use batch size of 1 for immediate search availability (progressive indexing)
+    // Each file is processed, embedded, and saved immediately so search can find it
+    const adaptiveBatchSize = this.config.progressiveIndexing !== false ? 1 :
+                              files.length > 10000 ? 500 :
+                              files.length > 1000 ? 200 :
                               this.config.batchSize || 100;
 
-    console.error(`[Indexer] Processing ${files.length} files with lazy filtering (batch size: ${adaptiveBatchSize})`);
+    console.error(`[Indexer] Processing ${files.length} files (progressive mode: batch size ${adaptiveBatchSize})`);
 
     // Step 3: Initialize worker threads (always use when multi-core available)
     const useWorkers = os.cpus().length > 1;
@@ -529,20 +616,32 @@ export class CodebaseIndexer {
       for (const file of batch) {
         try {
           const stats = await fs.stat(file);
-          
+
           // Skip directories and oversized files
           if (stats.isDirectory()) continue;
           if (stats.size > this.config.maxFileSize) {
             skippedFiles++;
             continue;
           }
-          
-          // Read content and check hash
+
+          // OPTIMIZATION: Check mtime first (fast) before reading file content
+          const currentMtime = stats.mtimeMs;
+          const cachedMtime = this.cache.getFileMtime(file);
+
+          // If mtime unchanged, file definitely unchanged - skip without reading
+          if (cachedMtime && currentMtime === cachedMtime) {
+            skippedFiles++;
+            continue;
+          }
+
+          // mtime changed (or new file) - read content and verify with hash
           const content = await fs.readFile(file, "utf-8");
           const hash = hashContent(content);
-          
-          // Skip unchanged files inline (lazy check)
+
+          // Check if content actually changed (mtime can change without content change)
           if (this.cache.getFileHash(file) === hash) {
+            // Content same but mtime different - update cached mtime
+            this.cache.setFileHash(file, hash, currentMtime);
             skippedFiles++;
             continue;
           }
@@ -557,11 +656,12 @@ export class CodebaseIndexer {
               text: chunk.text,
               startLine: chunk.startLine,
               endLine: chunk.endLine,
-              hash
+              hash,
+              mtime: currentMtime
             });
           }
-          
-          fileHashes.set(file, hash);
+
+          fileHashes.set(file, { hash, mtime: currentMtime });
         } catch (error) {
           // Skip files with read errors
           skippedFiles++;
@@ -612,9 +712,9 @@ export class CodebaseIndexer {
         }
       }
 
-      // Update file hashes
-      for (const [file, hash] of fileHashes) {
-        this.cache.setFileHash(file, hash);
+      // Update file hashes with mtime
+      for (const [file, { hash, mtime }] of fileHashes) {
+        this.cache.setFileHash(file, hash, mtime);
       }
 
       processedFiles += filesProcessedInBatch.size;
@@ -626,25 +726,30 @@ export class CodebaseIndexer {
       this.indexingStatus.totalFiles = Math.max(estimatedTotal, processedFiles);
       this.indexingStatus.percentage = estimatedTotal > 0 ? Math.floor((processedFiles / estimatedTotal) * 100) : 100;
 
-      // Incremental save to SQLite (every N batches)
-      const saveInterval = this.config.incrementalSaveInterval || 5;
-      if (batchCounter % saveInterval === 0) {
+      // Progressive indexing: save after EVERY batch so search can find new results immediately
+      // This is critical for background indexing - users can search while indexing continues
+      if (chunksToInsert.length > 0) {
         if (typeof this.cache.saveIncremental === 'function') {
           await this.cache.saveIncremental();
+        } else {
+          // Fallback: full save (slower but ensures data is persisted)
+          await this.cache.save();
         }
       }
-      
+
       // Apply CPU throttling (delay between batches)
       await this.throttle.throttledBatch(null);
 
-      // Progress indicator every batch
-      if (processedFiles > 0 && (processedFiles % (adaptiveBatchSize * 2) === 0 || i + adaptiveBatchSize >= files.length)) {
+      // Progress indicator - show progress after each file in progressive mode
+      const progressInterval = adaptiveBatchSize === 1 ? 1 : adaptiveBatchSize * 2;
+      if (processedFiles > 0 && ((processedFiles + skippedFiles) % progressInterval === 0 || i + adaptiveBatchSize >= files.length)) {
         const elapsed = ((Date.now() - totalStartTime) / 1000).toFixed(1);
-        const rate = (processedFiles / parseFloat(elapsed)).toFixed(0);
-        console.error(`[Indexer] Progress: ${processedFiles} changed, ${skippedFiles} skipped (${rate} files/sec)`);
-        
+        const totalProcessed = processedFiles + skippedFiles;
+        const rate = totalProcessed > 0 ? (totalProcessed / parseFloat(elapsed)).toFixed(1) : '0';
+        console.error(`[Indexer] Progress: ${processedFiles} indexed, ${skippedFiles} skipped of ${files.length} (${rate} files/sec)`);
+
         // Send MCP progress notification (10-95% range for batch processing)
-        const progressPercent = Math.min(95, Math.floor(10 + (i / files.length) * 85));
+        const progressPercent = Math.min(95, Math.floor(10 + (totalProcessed / files.length) * 85));
         this.sendProgress(progressPercent, 100, `Indexed ${processedFiles} files, ${skippedFiles} skipped (${rate}/sec)`);
       }
     }

package/index.js

@@ -191,21 +191,23 @@ async function initialize() {
     "[Server] Configuration loaded. Model will load on first use (lazy initialization)."
   );
 
-  // Optional: auto-index after delay
-  if (config.autoIndexDelay && config.autoIndexDelay > 0) {
+  // Progressive background indexing: starts after short delay, doesn't block
+  // Search works right away with partial results while indexing continues
+  if (config.autoIndexDelay !== false && config.autoIndexDelay > 0) {
     console.error(
-      `[Server] Auto-indexing will start after ${config.autoIndexDelay}ms delay...`
+      `[Server] Progressive indexing will start in ${config.autoIndexDelay}ms (search available immediately)...`
     );
     setTimeout(async () => {
-      console.error("[Server] Starting auto-indexing...");
       try {
         await ensureInitialized();
-        await indexer.indexAll();
+        // Use background indexing - non-blocking!
+        // Search can return partial results while indexing continues
+        indexer.startBackgroundIndexing();
         if (config.watchFiles) {
           indexer.setupFileWatcher();
         }
       } catch (err) {
-        console.error("[Server] Auto-indexing error:", err.message);
+        console.error("[Server] Background indexing error:", err.message);
       }
     }, config.autoIndexDelay);
   }

package/lib/cache.js

@@ -83,11 +83,21 @@ export class EmbeddingsCache {
   }
 
   getFileHash(file) {
-    return this.fileHashes.get(file);
+    const entry = this.fileHashes.get(file);
+    // Support both old format (string) and new format ({ hash, mtime })
+    if (typeof entry === 'string') {
+      return entry;
+    }
+    return entry?.hash;
+  }
+
+  getFileMtime(file) {
+    const entry = this.fileHashes.get(file);
+    return entry?.mtime;
   }
 
-  setFileHash(file, hash) {
-    this.fileHashes.set(file, hash);
+  setFileHash(file, hash, mtime = null) {
+    this.fileHashes.set(file, { hash, mtime });
   }
 
   deleteFileHash(file) {