smart-coding-mcp 1.2.4 → 1.3.0

package/README.md

@@ -8,6 +8,8 @@ AI coding assistants work better when they can find relevant code quickly. Tradi
 
 This MCP server solves that by indexing your codebase with AI embeddings. Your AI assistant can search by meaning instead of exact keywords, finding relevant code even when the terminology differs.
 
+![Example](example.png)
+
 ## Why Use This
 
 **Better Code Understanding**
@@ -36,6 +38,12 @@ Install globally via npm:
 npm install -g smart-coding-mcp
 ```
 
+To update to the latest version:
+
+```bash
+npm update -g smart-coding-mcp
+```
+
 ## Configuration
 
 Add to your MCP configuration file. The location depends on your IDE and OS:
@@ -80,33 +88,23 @@ Add the server configuration to the `mcpServers` object in your config file:
 }
 ```
 
-### Option 3: Auto-Detect Current Directory
-
-```json
-{
-  "mcpServers": {
-    "smart-coding-mcp": {
-      "command": "smart-coding-mcp"
-    }
-  }
-}
-```
-
 ## Environment Variables
 
 Override configuration settings via environment variables in your MCP config:
 
-| Variable                         | Type    | Default   | Description                           |
-| -------------------------------- | ------- | --------- | ------------------------------------- |
-| `SMART_CODING_VERBOSE`           | boolean | `false`   | Enable detailed logging               |
-| `SMART_CODING_BATCH_SIZE`        | number  | `100`     | Files to process in parallel          |
-| `SMART_CODING_MAX_FILE_SIZE`     | number  | `1048576` | Max file size in bytes (1MB)          |
-| `SMART_CODING_CHUNK_SIZE`        | number  | `15`      | Lines of code per chunk               |
-| `SMART_CODING_MAX_RESULTS`       | number  | `5`       | Max search results                    |
-| `SMART_CODING_SMART_INDEXING`    | boolean | `true`    | Enable smart project detection        |
-| `SMART_CODING_WATCH_FILES`       | boolean | `false`   | Enable file watching for auto-reindex |
-| `SMART_CODING_SEMANTIC_WEIGHT`   | number  | `0.7`     | Weight for semantic similarity (0-1)  |
-| `SMART_CODING_EXACT_MATCH_BOOST` | number  | `1.5`     | Boost for exact text matches          |
+| Variable                         | Type    | Default                   | Description                           |
+| -------------------------------- | ------- | ------------------------- | ------------------------------------- |
+| `SMART_CODING_VERBOSE`           | boolean | `false`                   | Enable detailed logging               |
+| `SMART_CODING_BATCH_SIZE`        | number  | `100`                     | Files to process in parallel          |
+| `SMART_CODING_MAX_FILE_SIZE`     | number  | `1048576`                 | Max file size in bytes (1MB)          |
+| `SMART_CODING_CHUNK_SIZE`        | number  | `25`                      | Lines of code per chunk               |
+| `SMART_CODING_MAX_RESULTS`       | number  | `5`                       | Max search results                    |
+| `SMART_CODING_SMART_INDEXING`    | boolean | `true`                    | Enable smart project detection        |
+| `SMART_CODING_WATCH_FILES`       | boolean | `false`                   | Enable file watching for auto-reindex |
+| `SMART_CODING_SEMANTIC_WEIGHT`   | number  | `0.7`                     | Weight for semantic similarity (0-1)  |
+| `SMART_CODING_EXACT_MATCH_BOOST` | number  | `1.5`                     | Boost for exact text matches          |
+| `SMART_CODING_EMBEDDING_MODEL`   | string  | `Xenova/all-MiniLM-L6-v2` | AI embedding model to use             |
+| `SMART_CODING_WORKER_THREADS`    | string  | `auto`                    | Worker threads (`auto` or 1-32)       |
 
 **Example with environment variables:**
 
@@ -160,60 +158,7 @@ The server indexes your code in four steps:
 
 When you search, your query is converted to the same vector format and compared against all code chunks using cosine similarity. The most relevant matches are returned.
 
-### Smart Project Detection
-
-The server detects your project type by looking for marker files and automatically applies appropriate ignore patterns:
-
-**JavaScript/Node** (package.json found)
-
-- Ignores: node_modules, dist, build, .next, coverage
-
-**Python** (requirements.txt or pyproject.toml)
-
-- Ignores: **pycache**, venv, .pytest_cache, .tox
-
-**Android** (build.gradle)
-
-- Ignores: .gradle, build artifacts, generated code
-
-**iOS** (Podfile)
-
-- Ignores: Pods, DerivedData, xcuserdata
-
-**And more**: Go, PHP, Rust, Ruby, .NET
-
-This typically reduces indexed file count by 100x. A project with 50,000 files (including node_modules) indexes just 500 actual source files.
-
-## Configuration
-
-The server works out of the box with sensible defaults. Create a `config.json` file in your workspace to customize:
-
-```json
-{
-  "searchDirectory": ".",
-  "fileExtensions": ["js", "ts", "py", "java", "go"],
-  "excludePatterns": ["**/my-custom-ignore/**"],
-  "smartIndexing": true,
-  "verbose": false,
-  "enableCache": true,
-  "cacheDirectory": "./.smart-coding-cache",
-  "watchFiles": true,
-  "chunkSize": 15,
-  "batchSize": 100,
-  "maxFileSize": 1048576,
-  "maxResults": 5
-}
-```
-
-**Key options:**
-
-- `smartIndexing`: Enable automatic project type detection and smart ignore patterns (default: true)
-- `verbose`: Show detailed indexing logs (default: false)
-- `watchFiles`: Automatically reindex when files change (default: true)
-- `enableCache`: Cache embeddings to disk (default: true)
-- `chunkSize`: Lines of code per chunk - smaller = more precise, larger = more context (default: 15)
-- `batchSize`: Number of files to process in parallel (default: 100)
-- `maxFileSize`: Skip files larger than this size in bytes (default: 1MB)
+![How It Works](how-its-works.png)
 
 ## Examples
 
@@ -243,85 +188,6 @@ Query: "error handling and exceptions"
 
 Finds all try/catch blocks and error handling patterns.
 
-## Performance
-
-Tested on a typical JavaScript project:
-
-| Metric         | Without Smart Indexing | With Smart Indexing |
-| -------------- | ---------------------- | ------------------- |
-| Files scanned  | 50,000+                | 500                 |
-| Indexing time  | 10+ min                | 2-3 min             |
-| Memory usage   | 2GB+                   | ~200MB              |
-| Search latency | N/A                    | <100ms              |
-
-## Supported File Types
-
-Languages: JavaScript, TypeScript, Python, Java, Kotlin, Scala, C, C++, C#, Go, Rust, Ruby, PHP, Swift, Shell
-
-Web: HTML, CSS, SCSS, Sass, XML, SVG
-
-Config/Data: JSON, YAML, TOML, SQL
-
-Total: 36 file extensions
-
-## Architecture
-
-```
-smart-coding-mcp/
-├── index.js                  # MCP server entry point
-├── lib/
-│   ├── config.js            # Configuration + smart detection
-│   ├── cache.js             # Embeddings persistence
-│   ├── utils.js             # Smart chunking
-│   ├── ignore-patterns.js   # Language-specific patterns
-│   └── project-detector.js  # Project type detection
-└── features/
-    ├── hybrid-search.js     # Semantic + exact match search
-    ├── index-codebase.js    # File indexing + watching
-    └── clear-cache.js       # Cache management
-```
-
-The modular design makes it easy to add new features. See ARCHITECTURE.md for implementation details.
-
-## Troubleshooting
-
-**"Server can't find config.json"**
-
-Make sure `cwd` is set in your MCP configuration to the full path of smart-coding-mcp.
-
-**"Indexing takes too long"**
-
-- Verify `smartIndexing` is enabled
-- Add more patterns to `excludePatterns`
-- Reduce `fileExtensions` to only what you need
-
-**"Search results aren't relevant"**
-
-- Try more specific queries
-- Increase `maxResults` to see more options
-- Run `index_codebase` to force a full reindex
-
-**"Cache corruption errors"**
-
-Use the `clear_cache` tool or run:
-
-```bash
-npm run clear-cache
-```
-
-## CLI Commands
-
-```bash
-# Start the server
-npm start
-
-# Development mode with auto-restart
-npm run dev
-
-# Clear embeddings cache
-npm run clear-cache
-```
-
 ## Privacy
 
 - AI model runs entirely on your machine
@@ -353,17 +219,6 @@ This project builds on research from Cursor showing that semantic search improve
 
 See: https://cursor.com/blog/semsearch
 
-## Contributing
-
-Contributions are welcome. See CONTRIBUTING.md for guidelines.
-
-Potential areas for improvement:
-
-- Additional language support
-- Code complexity analysis
-- Refactoring pattern detection
-- Documentation generation
-
 ## License
 
 MIT License

package/config.json

@@ -50,8 +50,8 @@
     "**/.smart-coding-cache/**"
   ],
   "smartIndexing": true,
-  "chunkSize": 15,
-  "chunkOverlap": 3,
+  "chunkSize": 25,
+  "chunkOverlap": 5,
   "batchSize": 100,
   "maxFileSize": 1048576,
   "maxResults": 5,
@@ -61,5 +61,6 @@
   "verbose": false,
   "embeddingModel": "Xenova/all-MiniLM-L6-v2",
   "semanticWeight": 0.7,
-  "exactMatchBoost": 1.5
+  "exactMatchBoost": 1.5,
+  "workerThreads": "auto"
 }

package/features/index-codebase.js

@@ -1,15 +1,243 @@
-import { glob } from "glob";
+import { fdir } from "fdir";
 import fs from "fs/promises";
 import chokidar from "chokidar";
 import path from "path";
+import os from "os";
+import { Worker } from "worker_threads";
+import { fileURLToPath } from "url";
 import { smartChunk, hashContent } from "../lib/utils.js";
 
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+
 export class CodebaseIndexer {
-  constructor(embedder, cache, config) {
+  constructor(embedder, cache, config, server = null) {
     this.embedder = embedder;
     this.cache = cache;
     this.config = config;
+    this.server = server;
     this.watcher = null;
+    this.workers = [];
+    this.workerReady = [];
+  }
+
+  /**
+   * Initialize worker thread pool for parallel embedding
+   */
+  async initializeWorkers() {
+    const numWorkers = this.config.workerThreads === "auto" 
+      ? Math.max(1, os.cpus().length - 1) 
+      : (this.config.workerThreads || 1);
+
+    // Only use workers if we have more than 1 CPU
+    if (numWorkers <= 1) {
+      console.error("[Indexer] Single-threaded mode (1 CPU detected)");
+      return;
+    }
+
+    if (this.config.verbose) {
+      console.error(`[Indexer] Worker config: workerThreads=${this.config.workerThreads}, resolved to ${numWorkers}`);
+    }
+
+    console.error(`[Indexer] Initializing ${numWorkers} worker threads...`);
+    
+    const workerPath = path.join(__dirname, "../lib/embedding-worker.js");
+    
+    for (let i = 0; i < numWorkers; i++) {
+      try {
+        const worker = new Worker(workerPath, {
+          workerData: { 
+            embeddingModel: this.config.embeddingModel,
+            verbose: this.config.verbose
+          }
+        });
+
+        const readyPromise = new Promise((resolve, reject) => {
+          const timeout = setTimeout(() => reject(new Error("Worker init timeout")), 120000);
+          
+          worker.once("message", (msg) => {
+            clearTimeout(timeout);
+            if (msg.type === "ready") {
+              resolve(worker);
+            } else if (msg.type === "error") {
+              reject(new Error(msg.error));
+            }
+          });
+          
+          worker.once("error", (err) => {
+            clearTimeout(timeout);
+            reject(err);
+          });
+        });
+
+        this.workers.push(worker);
+        this.workerReady.push(readyPromise);
+      } catch (err) {
+        console.error(`[Indexer] Failed to create worker ${i}: ${err.message}`);
+      }
+    }
+
+    // Wait for all workers to be ready
+    try {
+      await Promise.all(this.workerReady);
+      console.error(`[Indexer] ${this.workers.length} workers ready`);
+      if (this.config.verbose) {
+        console.error(`[Indexer] Each worker loaded model: ${this.config.embeddingModel}`);
+      }
+    } catch (err) {
+      console.error(`[Indexer] Worker initialization failed: ${err.message}, falling back to single-threaded`);
+      this.terminateWorkers();
+    }
+  }
+
+  /**
+   * Terminate all worker threads
+   */
+  terminateWorkers() {
+    for (const worker of this.workers) {
+      worker.postMessage({ type: "shutdown" });
+    }
+    this.workers = [];
+    this.workerReady = [];
+  }
+
+  /**
+   * Send MCP progress notification to connected clients
+   */
+  sendProgress(progress, total, message) {
+    if (this.server) {
+      try {
+        this.server.sendNotification("notifications/progress", {
+          progressToken: "indexing",
+          progress,
+          total,
+          message
+        });
+      } catch (err) {
+        // Silently ignore if client doesn't support progress notifications
+      }
+    }
+  }
+
+  /**
+   * Process chunks using worker thread pool with timeout and error recovery
+   */
+  async processChunksWithWorkers(allChunks) {
+    if (this.workers.length === 0) {
+      // Fallback to single-threaded processing
+      return this.processChunksSingleThreaded(allChunks);
+    }
+
+    const results = [];
+    const chunkSize = Math.ceil(allChunks.length / this.workers.length);
+    const workerPromises = [];
+    const WORKER_TIMEOUT = 300000; // 5 minutes per batch
+
+    if (this.config.verbose) {
+      console.error(`[Indexer] Distributing ${allChunks.length} chunks across ${this.workers.length} workers (~${chunkSize} chunks each)`);
+    }
+
+    for (let i = 0; i < this.workers.length; i++) {
+      const workerChunks = allChunks.slice(i * chunkSize, (i + 1) * chunkSize);
+      if (workerChunks.length === 0) continue;
+
+      if (this.config.verbose) {
+        console.error(`[Indexer] Worker ${i}: processing ${workerChunks.length} chunks`);
+      }
+
+      const promise = new Promise((resolve, reject) => {
+        const worker = this.workers[i];
+        const batchId = `batch-${i}-${Date.now()}`;
+        
+        // Timeout handler
+        const timeout = setTimeout(() => {
+          worker.off("message", handler);
+          console.error(`[Indexer] Worker ${i} timed out, falling back to single-threaded for this batch`);
+          // Return empty and let fallback handle it
+          resolve([]);
+        }, WORKER_TIMEOUT);
+
+        const handler = (msg) => {
+          if (msg.batchId === batchId) {
+            clearTimeout(timeout);
+            worker.off("message", handler);
+            if (msg.type === "results") {
+              resolve(msg.results);
+            } else if (msg.type === "error") {
+              console.error(`[Indexer] Worker ${i} error: ${msg.error}`);
+              resolve([]); // Return empty, don't reject - let fallback handle
+            }
+          }
+        };
+
+        // Handle worker crash
+        const errorHandler = (err) => {
+          clearTimeout(timeout);
+          worker.off("message", handler);
+          console.error(`[Indexer] Worker ${i} crashed: ${err.message}`);
+          resolve([]); // Return empty, don't reject
+        };
+        worker.once("error", errorHandler);
+
+        worker.on("message", handler);
+        worker.postMessage({ type: "process", chunks: workerChunks, batchId });
+      });
+
+      workerPromises.push({ promise, chunks: workerChunks });
+    }
+
+    // Wait for all workers with error recovery
+    const workerResults = await Promise.all(workerPromises.map(p => p.promise));
+    
+    // Collect results and identify failed chunks that need retry
+    const failedChunks = [];
+    for (let i = 0; i < workerResults.length; i++) {
+      if (workerResults[i].length > 0) {
+        results.push(...workerResults[i]);
+      } else if (workerPromises[i].chunks.length > 0) {
+        // Worker failed or timed out, need to retry these chunks
+        failedChunks.push(...workerPromises[i].chunks);
+      }
+    }
+
+    // Retry failed chunks with single-threaded fallback
+    if (failedChunks.length > 0) {
+      console.error(`[Indexer] Retrying ${failedChunks.length} chunks with single-threaded fallback...`);
+      const retryResults = await this.processChunksSingleThreaded(failedChunks);
+      results.push(...retryResults);
+    }
+
+    return results;
+  }
+
+  /**
+   * Single-threaded chunk processing (fallback)
+   */
+  async processChunksSingleThreaded(chunks) {
+    const results = [];
+    
+    for (const chunk of chunks) {
+      try {
+        const output = await this.embedder(chunk.text, { pooling: "mean", normalize: true });
+        results.push({
+          file: chunk.file,
+          startLine: chunk.startLine,
+          endLine: chunk.endLine,
+          content: chunk.text,
+          vector: Array.from(output.data),
+          success: true
+        });
+      } catch (error) {
+        results.push({
+          file: chunk.file,
+          startLine: chunk.startLine,
+          endLine: chunk.endLine,
+          error: error.message,
+          success: false
+        });
+      }
+    }
+
+    return results;
   }
 
   async indexFile(file) {
@@ -83,46 +311,235 @@ export class CodebaseIndexer {
     }
   }
 
-  async indexAll() {
-    console.error(`[Indexer] Indexing files in ${this.config.searchDirectory}...`);
+  /**
+   * Discover files using fdir (3-5x faster than glob)
+   * Uses config.excludePatterns which includes smart patterns from ignore-patterns.js
+   */
+  async discoverFiles() {
+    const startTime = Date.now();
     
-    const pattern = `${this.config.searchDirectory}/**/*.{${this.config.fileExtensions.join(",")}}`;
-    const files = await glob(pattern, { 
-      ignore: this.config.excludePatterns,
-      absolute: true 
-    });
-
-    console.error(`[Indexer] Found ${files.length} files to process`);
+    // Build extension filter from config
+    const extensions = new Set(this.config.fileExtensions.map(ext => `.${ext}`));
     
-    let totalChunks = 0;
-    let processedFiles = 0;
-    let skippedFiles = 0;
+    // Extract directory names from glob patterns in config.excludePatterns
+    // Patterns like "**/node_modules/**" -> "node_modules"
+    const excludeDirs = new Set();
+    for (const pattern of this.config.excludePatterns) {
+      // Extract directory names from glob patterns
+      const match = pattern.match(/\*\*\/([^/*]+)\/?\*?\*?$/);
+      if (match) {
+        excludeDirs.add(match[1]);
+      }
+      // Also handle patterns like "**/dirname/**"
+      const match2 = pattern.match(/\*\*\/([^/*]+)\/\*\*$/);
+      if (match2) {
+        excludeDirs.add(match2[1]);
+      }
+    }
+    
+    // Always exclude cache directory
+    excludeDirs.add(".smart-coding-cache");
     
-    // Process files in parallel batches for speed
-    const BATCH_SIZE = this.config.batchSize || 100;
+    if (this.config.verbose) {
+      console.error(`[Indexer] Using ${excludeDirs.size} exclude directories from config`);
+    }
+
+    const api = new fdir()
+      .withFullPaths()
+      .exclude((dirName) => excludeDirs.has(dirName))
+      .filter((filePath) => extensions.has(path.extname(filePath)))
+      .crawl(this.config.searchDirectory);
+
+    const files = await api.withPromise();
+    
+    console.error(`[Indexer] File discovery: ${files.length} files in ${Date.now() - startTime}ms`);
+    return files;
+  }
+
+  /**
+   * Pre-filter files by hash (skip unchanged files before processing)
+   */
+  async preFilterFiles(files) {
+    const startTime = Date.now();
+    const filesToProcess = [];
+    const skippedCount = { unchanged: 0, tooLarge: 0, error: 0 };
+
+    // Process in parallel batches for speed
+    const BATCH_SIZE = 500;
     
     for (let i = 0; i < files.length; i += BATCH_SIZE) {
       const batch = files.slice(i, i + BATCH_SIZE);
       
-      // Process batch in parallel
       const results = await Promise.all(
-        batch.map(file => this.indexFile(file))
+        batch.map(async (file) => {
+          try {
+            const stats = await fs.stat(file);
+            
+            if (stats.isDirectory()) {
+              return null;
+            }
+            
+            if (stats.size > this.config.maxFileSize) {
+              skippedCount.tooLarge++;
+              return null;
+            }
+            
+            const content = await fs.readFile(file, "utf-8");
+            const hash = hashContent(content);
+            
+            if (this.cache.getFileHash(file) === hash) {
+              skippedCount.unchanged++;
+              return null;
+            }
+            
+            return { file, content, hash };
+          } catch (error) {
+            skippedCount.error++;
+            return null;
+          }
+        })
       );
-      
-      // Aggregate results
-      for (const chunksAdded of results) {
-        totalChunks += chunksAdded;
-        processedFiles++;
-        if (chunksAdded === 0) skippedFiles++;
+
+      for (const result of results) {
+        if (result) filesToProcess.push(result);
       }
+    }
+
+    console.error(`[Indexer] Pre-filter: ${filesToProcess.length} changed, ${skippedCount.unchanged} unchanged, ${skippedCount.tooLarge} too large, ${skippedCount.error} errors (${Date.now() - startTime}ms)`);
+    return filesToProcess;
+  }
+
+  async indexAll() {
+    const totalStartTime = Date.now();
+    console.error(`[Indexer] Starting optimized indexing in ${this.config.searchDirectory}...`);
+    
+    // Step 1: Fast file discovery with fdir
+    const files = await this.discoverFiles();
+    
+    if (files.length === 0) {
+      console.error("[Indexer] No files found to index");
+      this.sendProgress(100, 100, "No files found to index");
+      return;
+    }
+
+    // Send progress: discovery complete
+    this.sendProgress(5, 100, `Discovered ${files.length} files`);
+
+    // Step 2: Pre-filter unchanged files (early hash check)
+    const filesToProcess = await this.preFilterFiles(files);
+    
+    if (filesToProcess.length === 0) {
+      console.error("[Indexer] All files unchanged, nothing to index");
+      this.sendProgress(100, 100, "All files up to date");
+      await this.cache.save();
+      return;
+    }
+
+    // Send progress: filtering complete
+    this.sendProgress(10, 100, `Processing ${filesToProcess.length} changed files`);
+
+    // Step 3: Determine batch size based on project size
+    const adaptiveBatchSize = files.length > 10000 ? 500 :
+                              files.length > 1000 ? 200 : 
+                              this.config.batchSize || 100;
+
+    console.error(`[Indexer] Processing ${filesToProcess.length} files (batch size: ${adaptiveBatchSize})`);
+
+    // Step 4: Initialize worker threads (always use when multi-core available)
+    const useWorkers = os.cpus().length > 1;
+    
+    if (useWorkers) {
+      await this.initializeWorkers();
+      console.error(`[Indexer] Multi-threaded mode: ${this.workers.length} workers active`);
+    } else {
+      console.error(`[Indexer] Single-threaded mode (single-core system)`);
+    }
+
+    let totalChunks = 0;
+    let processedFiles = 0;
+
+    // Step 5: Process files in adaptive batches
+    for (let i = 0; i < filesToProcess.length; i += adaptiveBatchSize) {
+      const batch = filesToProcess.slice(i, i + adaptiveBatchSize);
       
-      // Progress indicator every 500 files (less console overhead)
-      if (processedFiles % 500 === 0 || processedFiles === files.length) {
-        console.error(`[Indexer] Progress: ${processedFiles}/${files.length} files processed...`);
+      // Generate all chunks for this batch
+      const allChunks = [];
+      
+      for (const { file, content, hash } of batch) {
+        // Remove old chunks for this file
+        this.cache.removeFileFromStore(file);
+        
+        const chunks = smartChunk(content, file, this.config);
+        
+        for (const chunk of chunks) {
+          allChunks.push({
+            file,
+            text: chunk.text,
+            startLine: chunk.startLine,
+            endLine: chunk.endLine,
+            hash
+          });
+        }
+      }
+
+      // Process chunks (with workers if available, otherwise single-threaded)
+      let results;
+      if (useWorkers && this.workers.length > 0) {
+        results = await this.processChunksWithWorkers(allChunks);
+      } else {
+        results = await this.processChunksSingleThreaded(allChunks);
+      }
+
+      // Store successful results
+      const fileHashes = new Map();
+      for (const result of results) {
+        if (result.success) {
+          this.cache.addToStore({
+            file: result.file,
+            startLine: result.startLine,
+            endLine: result.endLine,
+            content: result.content,
+            vector: result.vector
+          });
+          totalChunks++;
+        }
+        // Track hash for each file
+        const chunkInfo = allChunks.find(c => c.file === result.file);
+        if (chunkInfo) {
+          fileHashes.set(result.file, chunkInfo.hash);
+        }
+      }
+
+      // Update file hashes
+      for (const [file, hash] of fileHashes) {
+        this.cache.setFileHash(file, hash);
+      }
+
+      processedFiles += batch.length;
+
+      // Progress indicator every batch
+      if (processedFiles % (adaptiveBatchSize * 2) === 0 || processedFiles === filesToProcess.length) {
+        const elapsed = ((Date.now() - totalStartTime) / 1000).toFixed(1);
+        const rate = (processedFiles / parseFloat(elapsed)).toFixed(0);
+        console.error(`[Indexer] Progress: ${processedFiles}/${filesToProcess.length} files (${rate} files/sec)`);
+        
+        // Send MCP progress notification (10-95% range for batch processing)
+        const progressPercent = Math.floor(10 + (processedFiles / filesToProcess.length) * 85);
+        this.sendProgress(progressPercent, 100, `Indexed ${processedFiles}/${filesToProcess.length} files (${rate}/sec)`);
       }
     }
 
-    console.error(`[Indexer] Indexed ${totalChunks} code chunks from ${files.length} files (${skippedFiles} unchanged)`);
+    // Cleanup workers
+    if (useWorkers) {
+      this.terminateWorkers();
+    }
+
+    const totalTime = ((Date.now() - totalStartTime) / 1000).toFixed(1);
+    console.error(`[Indexer] Complete: ${totalChunks} chunks from ${filesToProcess.length} files in ${totalTime}s`);
+    
+    // Send completion progress
+    this.sendProgress(100, 100, `Complete: ${totalChunks} chunks from ${filesToProcess.length} files in ${totalTime}s`);
+    
     await this.cache.save();
   }
 

package/index.js

@@ -95,7 +95,7 @@ async function initialize() {
   await cache.load();
 
   // Initialize features
-  indexer = new CodebaseIndexer(embedder, cache, config);
+  indexer = new CodebaseIndexer(embedder, cache, config, server);
   hybridSearch = new HybridSearch(embedder, cache, config);
   const cacheClearer = new ClearCacheFeature.CacheClearer(embedder, cache, config);
 

package/lib/config.js

@@ -1,5 +1,6 @@
 import fs from "fs/promises";
 import path from "path";
+import { fileURLToPath } from "url";
 import { ProjectDetector } from "./project-detector.js";
 
 const DEFAULT_CONFIG = {
@@ -50,8 +51,8 @@ const DEFAULT_CONFIG = {
     "**/target/**",
     "**/vendor/**"
   ],
-  chunkSize: 15,
-  chunkOverlap: 3,
+  chunkSize: 25, // Lines per chunk (larger = fewer embeddings = faster indexing)
+  chunkOverlap: 5, // Overlap between chunks for context continuity
   batchSize: 100,
   maxFileSize: 1048576, // 1MB - skip files larger than this
   maxResults: 5,
@@ -59,6 +60,7 @@ const DEFAULT_CONFIG = {
   cacheDirectory: "./.smart-coding-cache",
   watchFiles: false,
   verbose: false,
+  workerThreads: "auto", // "auto" = CPU cores - 1, or set a number
   embeddingModel: "Xenova/all-MiniLM-L6-v2",
   semanticWeight: 0.7,
   exactMatchBoost: 1.5,
@@ -80,7 +82,7 @@ export async function loadConfig(workspaceDir = null) {
       console.error(`[Config] Workspace mode: ${baseDir}`);
     } else {
       // Server mode: load config from server directory
-      const scriptDir = path.dirname(new URL(import.meta.url).pathname);
+      const scriptDir = path.dirname(fileURLToPath(import.meta.url));
       baseDir = path.resolve(scriptDir, '..');
       configPath = path.join(baseDir, "config.json");
     }
@@ -212,6 +214,28 @@ export async function loadConfig(workspaceDir = null) {
     }
   }
   
+  if (process.env.SMART_CODING_EMBEDDING_MODEL !== undefined) {
+    const value = process.env.SMART_CODING_EMBEDDING_MODEL.trim();
+    if (value.length > 0) {
+      config.embeddingModel = value;
+      console.error(`[Config] Using custom embedding model: ${value}`);
+    }
+  }
+  
+  if (process.env.SMART_CODING_WORKER_THREADS !== undefined) {
+    const value = process.env.SMART_CODING_WORKER_THREADS.trim().toLowerCase();
+    if (value === 'auto') {
+      config.workerThreads = 'auto';
+    } else {
+      const numValue = parseInt(value, 10);
+      if (!isNaN(numValue) && numValue >= 1 && numValue <= 32) {
+        config.workerThreads = numValue;
+      } else {
+        console.error(`[Config] Invalid SMART_CODING_WORKER_THREADS: ${value}, using default (must be 'auto' or 1-32)`);
+      }
+    }
+  }
+  
   return config;
 }
 

package/lib/embedding-worker.js

@@ -0,0 +1,67 @@
+import { parentPort, workerData } from "worker_threads";
+import { pipeline } from "@xenova/transformers";
+
+let embedder = null;
+
+// Initialize the embedding model once when worker starts
+async function initializeEmbedder() {
+  if (!embedder) {
+    embedder = await pipeline("feature-extraction", workerData.embeddingModel);
+  }
+  return embedder;
+}
+
+/**
+ * Process chunks with optimized single-text embedding
+ * Note: Batch processing with transformers.js WASM backend doesn't improve speed
+ * because it loops internally. Single calls are actually faster.
+ */
+async function processChunks(chunks) {
+  const embedder = await initializeEmbedder();
+  const results = [];
+
+  for (const chunk of chunks) {
+    try {
+      const output = await embedder(chunk.text, { pooling: "mean", normalize: true });
+      results.push({
+        file: chunk.file,
+        startLine: chunk.startLine,
+        endLine: chunk.endLine,
+        content: chunk.text,
+        vector: Array.from(output.data),
+        success: true
+      });
+    } catch (error) {
+      results.push({
+        file: chunk.file,
+        startLine: chunk.startLine,
+        endLine: chunk.endLine,
+        error: error.message,
+        success: false
+      });
+    }
+  }
+
+  return results;
+}
+
+// Listen for messages from main thread
+parentPort.on("message", async (message) => {
+  if (message.type === "process") {
+    try {
+      const results = await processChunks(message.chunks);
+      parentPort.postMessage({ type: "results", results, batchId: message.batchId });
+    } catch (error) {
+      parentPort.postMessage({ type: "error", error: error.message, batchId: message.batchId });
+    }
+  } else if (message.type === "shutdown") {
+    process.exit(0);
+  }
+});
+
+// Signal that worker is ready
+initializeEmbedder().then(() => {
+  parentPort.postMessage({ type: "ready" });
+}).catch((error) => {
+  parentPort.postMessage({ type: "error", error: error.message });
+});

package/lib/tokenizer.js

@@ -0,0 +1,142 @@
+/**
+ * Token estimation and limits for embedding models
+ * 
+ * This module provides token counting utilities and model-specific limits
+ * to ensure text chunks don't exceed the model's maximum sequence length.
+ */
+
+/**
+ * Token limits for supported embedding models
+ * Each model has its own maximum sequence length
+ */
+export const MODEL_TOKEN_LIMITS = {
+  // Sentence Transformers / MiniLM family
+  "Xenova/all-MiniLM-L6-v2": 256,
+  "Xenova/all-MiniLM-L12-v2": 256,
+  "Xenova/paraphrase-MiniLM-L6-v2": 128,
+  "Xenova/paraphrase-MiniLM-L3-v2": 128,
+  
+  // MPNet models
+  "Xenova/all-mpnet-base-v2": 384,
+  "Xenova/paraphrase-mpnet-base-v2": 384,
+  
+  // Multilingual models
+  "Xenova/paraphrase-multilingual-MiniLM-L12-v2": 128,
+  "Xenova/paraphrase-multilingual-mpnet-base-v2": 256,
+  
+  // Code-specific models
+  "Xenova/codebert-base": 512,
+  "Xenova/graphcodebert-base": 512,
+  
+  // E5 models
+  "Xenova/e5-small-v2": 512,
+  "Xenova/e5-base-v2": 512,
+  "Xenova/e5-large-v2": 512,
+  
+  // BGE models
+  "Xenova/bge-small-en-v1.5": 512,
+  "Xenova/bge-base-en-v1.5": 512,
+  "Xenova/bge-large-en-v1.5": 512,
+  
+  // Default fallback
+  "default": 256
+};
+
+/**
+ * Get the maximum token limit for a given model
+ * Case-insensitive lookup for robustness
+ * @param {string} modelName - The model name (e.g., "Xenova/all-MiniLM-L6-v2")
+ * @returns {number} Maximum tokens supported by the model
+ */
+export function getModelTokenLimit(modelName) {
+  if (!modelName) return MODEL_TOKEN_LIMITS["default"];
+  
+  // Direct match first (fastest)
+  if (MODEL_TOKEN_LIMITS[modelName] !== undefined) {
+    return MODEL_TOKEN_LIMITS[modelName];
+  }
+  
+  // Case-insensitive search
+  const normalizedName = modelName.toLowerCase();
+  for (const [key, value] of Object.entries(MODEL_TOKEN_LIMITS)) {
+    if (key.toLowerCase() === normalizedName) {
+      return value;
+    }
+  }
+  
+  return MODEL_TOKEN_LIMITS["default"];
+}
+
+/**
+ * Get chunking parameters for a model
+ * Returns target and overlap tokens based on the model's limit
+ * @param {string} modelName - The model name
+ * @returns {{ maxTokens: number, targetTokens: number, overlapTokens: number }}
+ */
+export function getChunkingParams(modelName) {
+  const maxTokens = getModelTokenLimit(modelName);
+  
+  // Target: 85% of max to leave safety buffer
+  const targetTokens = Math.floor(maxTokens * 0.85);
+  
+  // Overlap: 15-20% of target for context continuity
+  const overlapTokens = Math.floor(targetTokens * 0.18);
+  
+  return {
+    maxTokens,
+    targetTokens,
+    overlapTokens
+  };
+}
+
+/**
+ * Estimate token count for text (conservative estimate for code)
+ * Uses a simple heuristic: counts words, special characters, and estimates subwords
+ * 
+ * This is conservative - actual tokenizers may produce fewer tokens.
+ * For most accurate results, use the actual tokenizer, but this is much faster.
+ * 
+ * @param {string} text - The text to estimate tokens for
+ * @returns {number} Estimated token count
+ */
+export function estimateTokens(text) {
+  if (!text || text.length === 0) return 0;
+  
+  // Count words (split by whitespace)
+  const words = text.split(/\s+/).filter(w => w.length > 0);
+  
+  // Count special characters/punctuation that often become separate tokens
+  const specialChars = (text.match(/[{}()\[\];:,.<>!=+\-*\/%&|^~@#$"'`\\]/g) || []).length;
+  
+  // Estimate: words + special chars + 2 (for [CLS] and [SEP] special tokens)
+  // For long words, add extra tokens due to subword tokenization
+  let tokenCount = 2; // [CLS] and [SEP]
+  
+  for (const word of words) {
+    if (word.length <= 4) {
+      tokenCount += 1;
+    } else if (word.length <= 10) {
+      tokenCount += 2;
+    } else {
+      // Long words get split into ~4-char subwords
+      tokenCount += Math.ceil(word.length / 4);
+    }
+  }
+  
+  // Many special chars merge with adjacent tokens, so count ~50%
+  tokenCount += Math.floor(specialChars * 0.5);
+  
+  return tokenCount;
+}
+
+/**
+ * Check if text exceeds the token limit for a model
+ * @param {string} text - The text to check
+ * @param {string} modelName - The model name
+ * @returns {boolean} True if the text exceeds the limit
+ */
+export function exceedsTokenLimit(text, modelName) {
+  const limit = getModelTokenLimit(modelName);
+  const tokens = estimateTokens(text);
+  return tokens > limit;
+}

package/lib/utils.js

@@ -1,5 +1,9 @@
 import crypto from "crypto";
 import path from "path";
+import { estimateTokens, getChunkingParams, getModelTokenLimit } from "./tokenizer.js";
+
+// Re-export tokenizer utilities
+export { estimateTokens, getChunkingParams, getModelTokenLimit, MODEL_TOKEN_LIMITS } from "./tokenizer.js";
 
 /**
  * Calculate cosine similarity between two vectors
@@ -22,13 +26,22 @@ export function hashContent(content) {
 }
 
 /**
- * Intelligent chunking: tries to split by function/class boundaries
+ * Intelligent chunking with token limit awareness
+ * Tries to split by function/class boundaries while respecting token limits
+ * 
+ * @param {string} content - File content to chunk
+ * @param {string} file - File path (for language detection)
+ * @param {object} config - Configuration object with embeddingModel
+ * @returns {Array<{text: string, startLine: number, endLine: number, tokenCount: number}>}
  */
 export function smartChunk(content, file, config) {
   const lines = content.split("\n");
   const chunks = [];
   const ext = path.extname(file);
   
+  // Get model-specific chunking parameters
+  const { targetTokens, overlapTokens } = getChunkingParams(config.embeddingModel);
+  
   // Language-specific patterns for function/class detection
   const patterns = {
     // JavaScript/TypeScript
@@ -42,6 +55,7 @@ export function smartChunk(content, file, config) {
     // Python
     py: /^(class|def|async\s+def)\s+\w+/,
     pyw: /^(class|def|async\s+def)\s+\w+/,
+    pyx: /^(cdef|cpdef|def|class)\s+\w+/, // Cython
     
     // Java/Kotlin/Scala
     java: /^(public|private|protected)?\s*(static\s+)?(class|interface|enum|void|int|String|boolean)\s+\w+/,
@@ -56,70 +70,144 @@ export function smartChunk(content, file, config) {
     cxx: /^(class|struct|namespace|template|void|int|bool)\s+\w+/,
     h: /^(class|struct|namespace|template|void|int|bool)\s+\w+/,
     hpp: /^(class|struct|namespace|template|void|int|bool)\s+\w+/,
+    hxx: /^(class|struct|namespace|template|void|int|bool)\s+\w+/,
     
     // C#
     cs: /^(public|private|protected)?\s*(static\s+)?(class|interface|struct|enum|void|int|string|bool)\s+\w+/,
+    csx: /^(public|private|protected)?\s*(static\s+)?(class|interface|struct|enum|void|int|string|bool)\s+\w+/,
     
     // Go
     go: /^(func|type|const|var)\s+\w+/,
     
     // Rust
-    rs: /^(pub\s+)?(fn|struct|enum|trait|impl|const|static)\s+\w+/,
+    rs: /^(pub\s+)?(fn|struct|enum|trait|impl|const|static|mod)\s+\w+/,
     
     // PHP
     php: /^(class|interface|trait|function|const)\s+\w+/,
+    phtml: /^(<\?php|class|interface|trait|function)\s*/,
     
     // Ruby
     rb: /^(class|module|def)\s+\w+/,
-    rake: /^(class|module|def|task)\s+\w+/,
+    rake: /^(class|module|def|task|namespace)\s+\w+/,
     
     // Swift
-    swift: /^(class|struct|enum|protocol|func|var|let)\s+\w+/,
+    swift: /^(class|struct|enum|protocol|func|var|let|extension)\s+\w+/,
     
     // R
-    r: /^(\w+)\s*<-\s*function/,
-    R: /^(\w+)\s*<-\s*function/,
+    r: /^(\w+)\s*(<-|=)\s*function/,
+    R: /^(\w+)\s*(<-|=)\s*function/,
     
     // Lua
     lua: /^(function|local\s+function)\s+\w+/,
+    
+    // Shell scripts
+    sh: /^(\w+\s*\(\)|function\s+\w+)/,
+    bash: /^(\w+\s*\(\)|function\s+\w+)/,
+    zsh: /^(\w+\s*\(\)|function\s+\w+)/,
+    fish: /^function\s+\w+/,
+    
+    // CSS/Styles
+    css: /^(\.|#|@media|@keyframes|@font-face|\w+)\s*[{,]/,
+    scss: /^(\$\w+:|@mixin|@function|@include|\.|#|@media)\s*/,
+    sass: /^(\$\w+:|=\w+|\+\w+|\.|#|@media)\s*/,
+    less: /^(@\w+:|\.|\#|@media)\s*/,
+    styl: /^(\$\w+\s*=|\w+\(|\.|\#)\s*/,
+    
+    // Markup/HTML
+    html: /^(<(div|section|article|header|footer|nav|main|aside|form|table|template|script|style)\b)/i,
+    htm: /^(<(div|section|article|header|footer|nav|main|aside|form|table|template|script|style)\b)/i,
+    xml: /^(<\w+|\s*<!\[CDATA\[)/,
+    svg: /^(<svg|<g|<path|<defs|<symbol)\b/,
+    
+    // Config files
+    json: /^(\s*"[\w-]+"\s*:\s*[\[{])/,
+    yaml: /^(\w[\w-]*:\s*[|>]?$|\w[\w-]*:\s*$)/,
+    yml: /^(\w[\w-]*:\s*[|>]?$|\w[\w-]*:\s*$)/,
+    toml: /^(\[\[?\w+\]?\]?|\w+\s*=)/,
+    ini: /^(\[\w+\]|\w+\s*=)/,
+    env: /^[A-Z_][A-Z0-9_]*=/,
+    
+    // Documentation
+    md: /^(#{1,6}\s+|```|\*{3}|_{3})/,
+    mdx: /^(#{1,6}\s+|```|import\s+|export\s+)/,
+    txt: /^.{50,}/, // Split on long paragraphs
+    rst: /^(={3,}|-{3,}|~{3,}|\.\.\s+\w+::)/,
+    
+    // Database
+    sql: /^(CREATE|ALTER|INSERT|UPDATE|DELETE|SELECT|DROP|GRANT|REVOKE|WITH|DECLARE|BEGIN|END)\s+/i,
+    
+    // Perl
+    pl: /^(sub|package|use|require)\s+\w+/,
+    pm: /^(sub|package|use|require)\s+\w+/,
+    
+    // Vim
+    vim: /^(function|command|autocmd|let\s+g:)\s*/,
   };
 
-
   const langPattern = patterns[ext.slice(1)] || patterns.js;
   let currentChunk = [];
   let chunkStartLine = 0;
+  let currentTokenCount = 0;
 
   for (let i = 0; i < lines.length; i++) {
     const line = lines[i];
-    currentChunk.push(line);
-
-    // Check if we should start a new chunk
-    const shouldSplit = 
+    const lineTokens = estimateTokens(line);
+    
+    // Check if adding this line would exceed token limit
+    const wouldExceedLimit = (currentTokenCount + lineTokens) > targetTokens;
+    
+    // Check if this is a good split point (function/class boundary)
+    const isGoodSplitPoint = 
       langPattern.test(line.trim()) && 
-      currentChunk.length > config.chunkSize * 0.5;
+      currentChunk.length > 3; // At least a few lines before splitting
+    
+    // Split if we exceed limit OR at a good split point when near limit
+    const shouldSplit = wouldExceedLimit || (isGoodSplitPoint && currentTokenCount > targetTokens * 0.6);
 
-    if (shouldSplit || currentChunk.length >= config.chunkSize + config.chunkOverlap) {
-      if (currentChunk.join("\n").trim().length > 20) {
+    if (shouldSplit && currentChunk.length > 0) {
+      const chunkText = currentChunk.join("\n");
+      if (chunkText.trim().length > 20) {
         chunks.push({
-          text: currentChunk.join("\n"),
+          text: chunkText,
           startLine: chunkStartLine + 1,
-          endLine: i + 1
+          endLine: i,
+          tokenCount: currentTokenCount
         });
       }
       
-      // Keep overlap
-      currentChunk = currentChunk.slice(-config.chunkOverlap);
-      chunkStartLine = i - config.chunkOverlap + 1;
+      // Calculate overlap: keep last N lines that fit within overlapTokens
+      let overlapLines = [];
+      let overlapTokensCount = 0;
+      for (let j = currentChunk.length - 1; j >= 0 && overlapTokensCount < overlapTokens; j--) {
+        const lineT = estimateTokens(currentChunk[j]);
+        if (overlapTokensCount + lineT <= overlapTokens) {
+          overlapLines.unshift(currentChunk[j]);
+          overlapTokensCount += lineT;
+        } else {
+          break;
+        }
+      }
+      
+      currentChunk = overlapLines;
+      currentTokenCount = overlapTokensCount;
+      chunkStartLine = i - overlapLines.length;
     }
+
+    currentChunk.push(line);
+    currentTokenCount += lineTokens;
   }
 
   // Add remaining chunk
-  if (currentChunk.length > 0 && currentChunk.join("\n").trim().length > 20) {
-    chunks.push({
-      text: currentChunk.join("\n"),
-      startLine: chunkStartLine + 1,
-      endLine: lines.length
-    });
+  if (currentChunk.length > 0) {
+    const chunkText = currentChunk.join("\n");
+    if (chunkText.trim().length > 20) {
+      chunks.push({
+        text: chunkText,
+        startLine: chunkStartLine + 1,
+        endLine: lines.length,
+        tokenCount: currentTokenCount
+      });
+    }
   }
 
   return chunks;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "smart-coding-mcp",
-  "version": "1.2.4",
+  "version": "1.3.0",
   "description": "An extensible MCP server that enhances coding productivity with AI-powered features including semantic code search, intelligent indexing, and more, using local LLMs",
   "type": "module",
   "main": "index.js",
@@ -45,8 +45,9 @@
   "dependencies": {
     "@modelcontextprotocol/sdk": "^1.0.4",
     "@xenova/transformers": "^2.17.2",
-    "glob": "^10.3.10",
-    "chokidar": "^3.5.3"
+    "chokidar": "^3.5.3",
+    "fdir": "^6.5.0",
+    "glob": "^10.3.10"
   },
   "engines": {
     "node": ">=18.0.0"