smart-coding-mcp 1.4.1 → 2.1.0

package/README.md

@@ -5,16 +5,18 @@
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
 [![Node.js](https://img.shields.io/badge/Node.js-%3E%3D18-green.svg)](https://nodejs.org/)
 
-An extensible Model Context Protocol (MCP) server that provides intelligent semantic code search for AI assistants. Built with local AI models (RAG), inspired by Cursor's semantic search research.
+An extensible Model Context Protocol (MCP) server that provides intelligent semantic code search for AI assistants. Built with local AI models using Matryoshka Representation Learning (MRL) for flexible embedding dimensions (64-768d), with runtime workspace switching and comprehensive status reporting.
 
 ### Available Tools
 
-| Tool                   | Description                                       | Example                                        |
-| ---------------------- | ------------------------------------------------- | ---------------------------------------------- |
-| `semantic_search`      | Find code by meaning, not just keywords           | `"Where do we validate user input?"`           |
-| `index_codebase`       | Manually trigger reindexing                       | Use after major refactoring or branch switches |
-| `clear_cache`          | Reset the embeddings cache                        | Useful when cache becomes corrupted            |
-| `d_check_last_version` | Get latest version of any package (20 ecosystems) | `"express"`, `"npm:react"`, `"pip:requests"`   |
+| Tool                   | Description                                       | Example                                         |
+| ---------------------- | ------------------------------------------------- | ----------------------------------------------- |
+| `semantic_search`      | Find code by meaning, not just keywords           | `"Where do we validate user input?"`            |
+| `index_codebase`       | Manually trigger reindexing                       | Use after major refactoring or branch switches  |
+| `clear_cache`          | Reset the embeddings cache                        | Useful when cache becomes corrupted             |
+| `d_check_last_version` | Get latest version of any package (20 ecosystems) | `"express"`, `"npm:react"`, `"pip:requests"`    |
+| `e_set_workspace`      | Change project path at runtime                    | Switch to different project without restart     |
+| `f_get_status`         | Get server info: version, index status, config    | Check indexing progress, model info, cache size |
 
 ## What This Does
 
@@ -44,6 +46,34 @@ This MCP server solves that by indexing your codebase with AI embeddings. Your A
 - Your code never leaves your system
 - No API calls to external services
 
+## Performance & Resource Management
+
+**Progressive Indexing**
+
+- Search works immediately, even while indexing continues (like video buffering)
+- Incremental saves every 5 batches - no data loss if interrupted
+- Real-time indexing status shown when searching during indexing
+
+**Resource Throttling**
+
+- CPU usage limited to 50% by default (configurable)
+- Your laptop stays responsive during indexing
+- Configurable delays between batches
+- Worker thread limits respect system resources
+
+**SQLite Cache**
+
+- 5-10x faster than JSON for large codebases
+- Write-Ahead Logging (WAL) for better concurrency
+- Binary blob storage for smaller cache size
+- Automatic migration from JSON
+
+**Optimized Defaults**
+
+- 128d embeddings by default (2x faster than 256d, minimal quality loss)
+- Smart batch sizing based on project size
+- Parallel processing with auto-tuned worker threads
+
 ## Installation
 
 Install globally via npm:
@@ -130,19 +160,25 @@ For clients that support dynamic variables (VS Code, Cursor):
 
 Override configuration settings via environment variables in your MCP config:
 
-| Variable                         | Type    | Default                   | Description                           |
-| -------------------------------- | ------- | ------------------------- | ------------------------------------- |
-| `SMART_CODING_VERBOSE`           | boolean | `false`                   | Enable detailed logging               |
-| `SMART_CODING_BATCH_SIZE`        | number  | `100`                     | Files to process in parallel          |
-| `SMART_CODING_MAX_FILE_SIZE`     | number  | `1048576`                 | Max file size in bytes (1MB)          |
-| `SMART_CODING_CHUNK_SIZE`        | number  | `25`                      | Lines of code per chunk               |
-| `SMART_CODING_MAX_RESULTS`       | number  | `5`                       | Max search results                    |
-| `SMART_CODING_SMART_INDEXING`    | boolean | `true`                    | Enable smart project detection        |
-| `SMART_CODING_WATCH_FILES`       | boolean | `false`                   | Enable file watching for auto-reindex |
-| `SMART_CODING_SEMANTIC_WEIGHT`   | number  | `0.7`                     | Weight for semantic similarity (0-1)  |
-| `SMART_CODING_EXACT_MATCH_BOOST` | number  | `1.5`                     | Boost for exact text matches          |
-| `SMART_CODING_EMBEDDING_MODEL`   | string  | `Xenova/all-MiniLM-L6-v2` | AI embedding model to use             |
-| `SMART_CODING_WORKER_THREADS`    | string  | `auto`                    | Worker threads (`auto` or 1-32)       |
+| Variable                           | Type    | Default                          | Description                                |
+| ---------------------------------- | ------- | -------------------------------- | ------------------------------------------ |
+| `SMART_CODING_VERBOSE`             | boolean | `false`                          | Enable detailed logging                    |
+| `SMART_CODING_BATCH_SIZE`          | number  | `100`                            | Files to process in parallel               |
+| `SMART_CODING_MAX_FILE_SIZE`       | number  | `1048576`                        | Max file size in bytes (1MB)               |
+| `SMART_CODING_CHUNK_SIZE`          | number  | `25`                             | Lines of code per chunk                    |
+| `SMART_CODING_MAX_RESULTS`         | number  | `5`                              | Max search results                         |
+| `SMART_CODING_SMART_INDEXING`      | boolean | `true`                           | Enable smart project detection             |
+| `SMART_CODING_WATCH_FILES`         | boolean | `false`                          | Enable file watching for auto-reindex      |
+| `SMART_CODING_SEMANTIC_WEIGHT`     | number  | `0.7`                            | Weight for semantic similarity (0-1)       |
+| `SMART_CODING_EXACT_MATCH_BOOST`   | number  | `1.5`                            | Boost for exact text matches               |
+| `SMART_CODING_EMBEDDING_MODEL`     | string  | `nomic-ai/nomic-embed-text-v1.5` | AI embedding model to use                  |
+| `SMART_CODING_EMBEDDING_DIMENSION` | number  | `128`                            | MRL dimension (64, 128, 256, 512, 768)     |
+| `SMART_CODING_DEVICE`              | string  | `cpu`                            | Inference device (`cpu`, `webgpu`, `auto`) |
+| `SMART_CODING_CHUNKING_MODE`       | string  | `smart`                          | Code chunking (`smart`, `ast`, `line`)     |
+| `SMART_CODING_WORKER_THREADS`      | string  | `auto`                           | Worker threads (`auto` or 1-32)            |
+| `SMART_CODING_MAX_CPU_PERCENT`     | number  | `50`                             | Max CPU usage during indexing (10-100%)    |
+| `SMART_CODING_BATCH_DELAY`         | number  | `100`                            | Delay between batches in ms (0-5000)       |
+| `SMART_CODING_MAX_WORKERS`         | string  | `auto`                           | Override max worker threads limit          |
 
 **Example with environment variables:**
 
@@ -166,16 +202,73 @@ Override configuration settings via environment variables in your MCP config:
 
 ## How It Works
 
-The server indexes your code in four steps:
+```mermaid
+flowchart TB
+    subgraph IDE["IDE / AI Assistant"]
+        Agent["AI Agent<br/>(Claude, GPT, Gemini)"]
+    end
+
+    subgraph MCP["Smart Coding MCP Server"]
+        direction TB
+        Protocol["Model Context Protocol<br/>JSON-RPC over stdio"]
+        Tools["MCP Tools<br/>semantic_search | index_codebase | set_workspace | get_status"]
+
+        subgraph Indexing["Indexing Pipeline"]
+            Discovery["File Discovery<br/>glob patterns + smart ignore"]
+            Chunking["Code Chunking<br/>Smart (regex) / AST (Tree-sitter)"]
+            Embedding["AI Embedding<br/>transformers.js + ONNX Runtime"]
+        end
+
+        subgraph AI["AI Model"]
+            Model["nomic-embed-text-v1.5<br/>Matryoshka Representation Learning"]
+            Dimensions["Flexible Dimensions<br/>64 | 128 | 256 | 512 | 768"]
+            Normalize["Layer Norm → Slice → L2 Normalize"]
+        end
+
+        subgraph Search["Search"]
+            QueryEmbed["Query → Vector"]
+            Cosine["Cosine Similarity"]
+            Hybrid["Hybrid Search<br/>Semantic + Exact Match Boost"]
+        end
+    end
+
+    subgraph Storage["Cache"]
+        Vectors["SQLite Database<br/>embeddings.db (WAL mode)"]
+        Hashes["File Hashes<br/>Incremental updates"]
+        Progressive["Progressive Indexing<br/>Search works during indexing"]
+    end
+
+    Agent <-->|"MCP Protocol"| Protocol
+    Protocol --> Tools
+
+    Tools --> Discovery
+    Discovery --> Chunking
+    Chunking --> Embedding
+    Embedding --> Model
+    Model --> Dimensions
+    Dimensions --> Normalize
+    Normalize --> Vectors
+
+    Tools --> QueryEmbed
+    QueryEmbed --> Model
+    Cosine --> Hybrid
+    Vectors --> Cosine
+    Hybrid --> Agent
+```
+
+### Tech Stack
 
-1. **Discovery**: Scans your project for source files
-2. **Chunking**: Breaks code into meaningful pieces (respecting function boundaries)
-3. **Embedding**: Converts each chunk to a vector using a local AI model
-4. **Storage**: Saves embeddings to `.smart-coding-cache/` for fast startup
+| Component     | Technology                            |
+| ------------- | ------------------------------------- |
+| **Protocol**  | Model Context Protocol (JSON-RPC)     |
+| **AI Model**  | nomic-embed-text-v1.5 (MRL)           |
+| **Inference** | transformers.js + ONNX Runtime        |
+| **Chunking**  | Smart regex / Tree-sitter AST         |
+| **Search**    | Cosine similarity + exact match boost |
 
-When you search, your query is converted to the same vector format and compared against all code chunks using cosine similarity. The most relevant matches are returned.
+### Search Flow
 
-![How It Works](how-its-works.png)
+Query → Vector embedding → Cosine similarity → Ranked results
 
 ## Examples
 
@@ -214,11 +307,18 @@ Finds all try/catch blocks and error handling patterns.
 
 ## Technical Details
 
-**Embedding Model**: all-MiniLM-L6-v2 via transformers.js
+**Embedding Model**: nomic-embed-text-v1.5 via transformers.js v3
+
+- Matryoshka Representation Learning (MRL) for flexible dimensions
+- Configurable output: 64, 128, 256, 512, or 768 dimensions
+- Longer context (8192 tokens vs 256 for MiniLM)
+- Better code understanding through specialized training
+- WebGPU support for up to 100x faster inference (when available)
+
+**Legacy Model**: all-MiniLM-L6-v2 (fallback)
 
-- Fast inference (CPU-friendly)
-- Small model size (~100MB)
-- Good accuracy for code search
+- Fast inference, small footprint (~100MB)
+- Fixed 384-dimensional output
 
 **Vector Similarity**: Cosine similarity
 

package/config.json

@@ -60,8 +60,13 @@
   "cacheDirectory": "./.smart-coding-cache",
   "watchFiles": false,
   "verbose": false,
-  "embeddingModel": "Xenova/all-MiniLM-L6-v2",
+  "embeddingModel": "nomic-ai/nomic-embed-text-v1.5",
+  "embeddingDimension": 128,
+  "device": "auto",
+  "chunkingMode": "smart",
   "semanticWeight": 0.7,
   "exactMatchBoost": 1.5,
-  "workerThreads": "auto"
+  "workerThreads": "auto",
+  "maxCpuPercent": 50,
+  "batchDelay": 100
 }

package/features/get-status.js

@@ -0,0 +1,163 @@
+/**
+ * Get Status Feature
+ * 
+ * MCP tool to return comprehensive status information about the server.
+ * Useful for agents to understand current state and configuration.
+ */
+
+import fs from 'fs/promises';
+import path from 'path';
+import { createRequire } from 'module';
+
+const require = createRequire(import.meta.url);
+const packageJson = require('../package.json');
+
+/**
+ * Get tool definition for MCP registration
+ */
+export function getToolDefinition(config) {
+  return {
+    name: "f_get_status",
+    description: "Get comprehensive status information about the Smart Coding MCP server. Returns version, workspace path, model configuration, indexing status, and cache information. Useful for understanding the current state of the semantic search system.",
+    inputSchema: {
+      type: "object",
+      properties: {},
+      required: []
+    }
+  };
+}
+
+/**
+ * Status Reporter class
+ */
+export class StatusReporter {
+  constructor(config, cache, indexer, embedder) {
+    this.config = config;
+    this.cache = cache;
+    this.indexer = indexer;
+    this.embedder = embedder;
+    this.startTime = Date.now();
+  }
+
+  /**
+   * Get comprehensive status
+   */
+  async getStatus() {
+    const vectorStore = this.cache?.getVectorStore() || [];
+    
+    // Get unique files from vector store
+    const uniqueFiles = new Set(vectorStore.map(v => v.file));
+    
+    // Get cache size (check for SQLite database)
+    let cacheSizeBytes = 0;
+    let cacheType = 'none';
+    try {
+      // Check for SQLite cache first
+      const sqlitePath = path.join(this.config.cacheDirectory, 'embeddings.db');
+      const stats = await fs.stat(sqlitePath);
+      cacheSizeBytes = stats.size;
+      cacheType = 'sqlite';
+    } catch {
+      // Try old JSON cache as fallback
+      try {
+        const jsonPath = path.join(this.config.cacheDirectory, 'embeddings.json');
+        const stats = await fs.stat(jsonPath);
+        cacheSizeBytes = stats.size;
+        cacheType = 'json';
+      } catch {
+        // No cache file exists
+        cacheType = 'none';
+      }
+    }
+
+    // Determine index status and progressive indexing info
+    let indexStatus = 'empty';
+    let progressiveIndexing = null;
+    
+    if (this.indexer?.isIndexing) {
+      indexStatus = 'indexing';
+      // Include progressive indexing status
+      if (this.indexer.indexingStatus) {
+        progressiveIndexing = {
+          inProgress: this.indexer.indexingStatus.inProgress,
+          totalFiles: this.indexer.indexingStatus.totalFiles,
+          processedFiles: this.indexer.indexingStatus.processedFiles,
+          percentage: this.indexer.indexingStatus.percentage
+        };
+      }
+    } else if (vectorStore.length > 0) {
+      indexStatus = 'ready';
+    }
+
+    return {
+      version: packageJson.version,
+      uptime: Math.floor((Date.now() - this.startTime) / 1000),
+      
+      workspace: {
+        path: this.config.searchDirectory,
+        cacheDirectory: this.config.cacheDirectory
+      },
+
+      model: {
+        name: this.embedder?.modelName || this.config.embeddingModel,
+        dimension: this.embedder?.dimension || this.config.embeddingDimension,
+        device: this.embedder?.device || this.config.device
+      },
+
+      index: {
+        status: indexStatus,
+        filesIndexed: uniqueFiles.size,
+        chunksCount: vectorStore.length,
+        chunkingMode: this.config.chunkingMode,
+        ...(progressiveIndexing && { progressiveIndexing })
+      },
+
+      cache: {
+        enabled: this.config.enableCache,
+        type: cacheType,
+        path: this.config.cacheDirectory,
+        sizeBytes: cacheSizeBytes,
+        sizeFormatted: formatBytes(cacheSizeBytes)
+      },
+
+      config: {
+        maxResults: this.config.maxResults,
+        chunkSize: this.config.chunkSize,
+        semanticWeight: this.config.semanticWeight,
+        exactMatchBoost: this.config.exactMatchBoost,
+        workerThreads: this.config.workerThreads
+      },
+      
+      resourceThrottling: {
+        maxCpuPercent: this.config.maxCpuPercent,
+        batchDelay: this.config.batchDelay,
+        maxWorkers: this.config.maxWorkers
+      }
+    };
+  }
+}
+
+/**
+ * Format bytes to human readable
+ */
+function formatBytes(bytes) {
+  if (bytes === 0) return '0 B';
+  const k = 1024;
+  const sizes = ['B', 'KB', 'MB', 'GB'];
+  const i = Math.floor(Math.log(bytes) / Math.log(k));
+  return parseFloat((bytes / Math.pow(k, i)).toFixed(2)) + ' ' + sizes[i];
+}
+
+/**
+ * Handle MCP tool call
+ */
+export async function handleToolCall(request, instance) {
+  const status = await instance.getStatus();
+
+  return {
+    content: [{
+      type: "text",
+      text: JSON.stringify(status, null, 2)
+    }]
+  };
+}

package/features/hybrid-search.js

@@ -2,21 +2,35 @@ import path from "path";
 import { cosineSimilarity } from "../lib/utils.js";
 
 export class HybridSearch {
-  constructor(embedder, cache, config) {
+  constructor(embedder, cache, config, indexer = null) {
     this.embedder = embedder;
     this.cache = cache;
     this.config = config;
+    this.indexer = indexer; // Reference to indexer for status checking
   }
 
   async search(query, maxResults) {
     const vectorStore = this.cache.getVectorStore();
     
     if (vectorStore.length === 0) {
+      // Check if indexing is in progress
+      if (this.indexer?.indexingStatus?.inProgress) {
+        return {
+          results: [],
+          message: `Indexing in progress (${this.indexer.indexingStatus.percentage}% complete). Search available but results may be incomplete. Please wait for indexing to finish for full coverage.`
+        };
+      }
       return {
         results: [],
         message: "No code has been indexed yet. Please wait for initial indexing to complete."
       };
     }
+    
+    // Show warning if indexing is still in progress but we have some results
+    let indexingWarning = null;
+    if (this.indexer?.indexingStatus?.inProgress) {
+      indexingWarning = `⚠️ Indexing in progress (${this.indexer.indexingStatus.percentage}% complete). Results shown are from partially indexed codebase.\n\n`;
+    }
 
     // Generate query embedding
     const queryEmbed = await this.embedder(query, { pooling: "mean", normalize: true });
@@ -50,7 +64,7 @@ export class HybridSearch {
       .sort((a, b) => b.score - a.score)
       .slice(0, maxResults);
 
-    return { results, message: null };
+    return { results, message: null, indexingWarning };
   }
 
   formatResults(results) {
@@ -105,7 +119,7 @@ export async function handleToolCall(request, hybridSearch) {
   const query = request.params.arguments.query;
   const maxResults = request.params.arguments.maxResults || hybridSearch.config.maxResults;
   
-  const { results, message } = await hybridSearch.search(query, maxResults);
+  const { results, message, indexingWarning } = await hybridSearch.search(query, maxResults);
   
   if (message) {
     return {
@@ -113,7 +127,12 @@ export async function handleToolCall(request, hybridSearch) {
     };
   }
 
-  const formattedText = hybridSearch.formatResults(results);
+  let formattedText = hybridSearch.formatResults(results);
+  
+  // Prepend indexing warning if present
+  if (indexingWarning) {
+    formattedText = indexingWarning + formattedText;
+  }
   
   return {
     content: [{ type: "text", text: formattedText }]