smart-coding-mcp 1.0.0

package/README.md

@@ -0,0 +1,305 @@
+# Smart Coding MCP
+
+An extensible Model Context Protocol (MCP) server that provides intelligent semantic code search for AI assistants. Built with local AI models, inspired by Cursor's semantic search research.
+
+## What This Does
+
+AI coding assistants work better when they can find relevant code quickly. Traditional keyword search falls short - if you ask "where do we handle authentication?" but your code uses "login" and "session", keyword search misses it.
+
+This MCP server solves that by indexing your codebase with AI embeddings. Your AI assistant can search by meaning instead of exact keywords, finding relevant code even when the terminology differs.
+
+## Why Use This
+
+**Better Code Understanding**
+
+- Search finds code by concept, not just matching words
+- Works with typos and variations in terminology
+- Natural language queries like "where do we validate user input?"
+
+**Performance**
+
+- Pre-indexed embeddings are faster than scanning files at runtime
+- Smart project detection skips dependencies automatically (node_modules, vendor, etc.)
+- Incremental updates - only re-processes changed files
+
+**Privacy**
+
+- Everything runs locally on your machine
+- Your code never leaves your system
+- No API calls to external services
+
+## Installation
+
+### Prerequisites
+
+- Node.js 18 or higher
+- npm or yarn
+
+### Setup
+
+1. Install dependencies:
+
+```bash
+npm install
+```
+
+2. Add to your MCP configuration file:
+
+```json
+{
+  "mcpServers": {
+    "smart-coding-mcp": {
+      "command": "node",
+      "args": ["/path/to/smart-coding-mcp/index.js"],
+      "cwd": "/path/to/smart-coding-mcp"
+    }
+  }
+}
+```
+
+3. Restart your AI assistant
+
+The server will automatically index your codebase on first run.
+
+## Available Tools
+
+**semantic_search** - Find code by meaning
+
+```
+Query: "Where do we validate user input?"
+Returns: Relevant validation code with file paths and line numbers
+```
+
+**index_codebase** - Manually trigger reindexing
+
+```
+Use after major refactoring or branch switches
+```
+
+**clear_cache** - Reset the embeddings cache
+
+```
+Useful when cache becomes corrupted or outdated
+```
+
+## How It Works
+
+The server indexes your code in four steps:
+
+1. **Discovery**: Scans your project for source files
+2. **Chunking**: Breaks code into meaningful pieces (respecting function boundaries)
+3. **Embedding**: Converts each chunk to a vector using a local AI model
+4. **Storage**: Saves embeddings to `.smart-coding-cache/` for fast startup
+
+When you search, your query is converted to the same vector format and compared against all code chunks using cosine similarity. The most relevant matches are returned.
+
+### Smart Project Detection
+
+The server detects your project type by looking for marker files and automatically applies appropriate ignore patterns:
+
+**JavaScript/Node** (package.json found)
+
+- Ignores: node_modules, dist, build, .next, coverage
+
+**Python** (requirements.txt or pyproject.toml)
+
+- Ignores: **pycache**, venv, .pytest_cache, .tox
+
+**Android** (build.gradle)
+
+- Ignores: .gradle, build artifacts, generated code
+
+**iOS** (Podfile)
+
+- Ignores: Pods, DerivedData, xcuserdata
+
+**And more**: Go, PHP, Rust, Ruby, .NET
+
+This typically reduces indexed file count by 100x. A project with 50,000 files (including node_modules) indexes just 500 actual source files.
+
+## Configuration
+
+The server works out of the box with sensible defaults. Create a `config.json` file to customize:
+
+```json
+{
+  "searchDirectory": ".",
+  "fileExtensions": ["js", "ts", "py", "java", "go"],
+  "excludePatterns": ["**/my-custom-ignore/**"],
+  "smartIndexing": true,
+  "verbose": false,
+  "enableCache": true,
+  "cacheDirectory": "./.smart-coding-cache",
+  "watchFiles": true,
+  "chunkSize": 15,
+  "maxResults": 5
+}
+```
+
+**Key options:**
+
+- `smartIndexing`: Enable automatic project type detection and smart ignore patterns (default: true)
+- `verbose`: Show detailed indexing logs (default: false)
+- `watchFiles`: Automatically reindex when files change (default: true)
+- `enableCache`: Cache embeddings to disk (default: true)
+- `chunkSize`: Lines of code per chunk - smaller = more precise, larger = more context (default: 15)
+
+## Examples
+
+**Natural language search:**
+
+Query: "How do we handle cache persistence?"
+
+Result:
+
+```javascript
+// lib/cache.js (Relevance: 38.2%)
+async save() {
+  await fs.writeFile(cacheFile, JSON.stringify(this.vectorStore));
+  await fs.writeFile(hashFile, JSON.stringify(this.fileHashes));
+}
+```
+
+**Typo tolerance:**
+
+Query: "embeding modle initializashun"
+
+Still finds embedding model initialization code despite multiple typos.
+
+**Conceptual search:**
+
+Query: "error handling and exceptions"
+
+Finds all try/catch blocks and error handling patterns.
+
+## Performance
+
+Tested on a typical JavaScript project:
+
+| Metric         | Without Smart Indexing | With Smart Indexing |
+| -------------- | ---------------------- | ------------------- |
+| Files scanned  | 50,000+                | 500                 |
+| Indexing time  | 10+ min                | 2-3 min             |
+| Memory usage   | 2GB+                   | ~200MB              |
+| Search latency | N/A                    | <100ms              |
+
+## Supported File Types
+
+Languages: JavaScript, TypeScript, Python, Java, Kotlin, Scala, C, C++, C#, Go, Rust, Ruby, PHP, Swift, Shell
+
+Web: HTML, CSS, SCSS, Sass, XML, SVG
+
+Config/Data: JSON, YAML, TOML, SQL
+
+Total: 36 file extensions
+
+## Architecture
+
+```
+smart-coding-mcp/
+├── index.js                  # MCP server entry point
+├── lib/
+│   ├── config.js            # Configuration + smart detection
+│   ├── cache.js             # Embeddings persistence
+│   ├── utils.js             # Smart chunking
+│   ├── ignore-patterns.js   # Language-specific patterns
+│   └── project-detector.js  # Project type detection
+└── features/
+    ├── hybrid-search.js     # Semantic + exact match search
+    ├── index-codebase.js    # File indexing + watching
+    └── clear-cache.js       # Cache management
+```
+
+The modular design makes it easy to add new features. See ARCHITECTURE.md for implementation details.
+
+## Troubleshooting
+
+**"Server can't find config.json"**
+
+Make sure `cwd` is set in your MCP configuration to the full path of smart-coding-mcp.
+
+**"Indexing takes too long"**
+
+- Verify `smartIndexing` is enabled
+- Add more patterns to `excludePatterns`
+- Reduce `fileExtensions` to only what you need
+
+**"Search results aren't relevant"**
+
+- Try more specific queries
+- Increase `maxResults` to see more options
+- Run `index_codebase` to force a full reindex
+
+**"Cache corruption errors"**
+
+Use the `clear_cache` tool or run:
+
+```bash
+npm run clear-cache
+```
+
+## CLI Commands
+
+```bash
+# Start the server
+npm start
+
+# Development mode with auto-restart
+npm run dev
+
+# Clear embeddings cache
+npm run clear-cache
+```
+
+## Privacy
+
+- AI model runs entirely on your machine
+- No network requests to external services
+- No telemetry or analytics
+- Cache stored locally in `.smart-coding-cache/`
+
+## Technical Details
+
+**Embedding Model**: all-MiniLM-L6-v2 via transformers.js
+
+- Fast inference (CPU-friendly)
+- Small model size (~100MB)
+- Good accuracy for code search
+
+**Vector Similarity**: Cosine similarity
+
+- Efficient comparison of embeddings
+- Normalized vectors for consistent scoring
+
+**Hybrid Scoring**: Combines semantic similarity with exact text matching
+
+- Semantic weight: 0.7 (configurable)
+- Exact match boost: 1.5x (configurable)
+
+## Research Background
+
+This project builds on research from Cursor showing that semantic search improves AI coding agent performance by 12.5% on average across question-answering tasks. The key insight is that AI assistants benefit more from relevant context than from large amounts of context.
+
+See: https://cursor.com/blog/semsearch
+
+## Contributing
+
+Contributions are welcome. See CONTRIBUTING.md for guidelines.
+
+Potential areas for improvement:
+
+- Additional language support
+- Code complexity analysis
+- Refactoring pattern detection
+- Documentation generation
+
+## License
+
+MIT - see LICENSE file
+
+## Documentation
+
+- ARCHITECTURE.md - Implementation details and design decisions
+- CONTRIBUTING.md - Guidelines for contributors
+- EXAMPLES.md - More usage examples
+- QUICKSTART.md - Detailed setup guide

package/config.json

@@ -0,0 +1,18 @@
+{
+  "searchDirectory": ".",
+  "fileExtensions": ["js", "ts", "jsx", "tsx", "mjs", "cjs", "css", "scss", "sass", "less", "html", "htm", "xml", "svg", "py", "pyw", "java", "kt", "scala", "c", "cpp", "h", "hpp", "cs", "go", "rs", "rb", "php", "swift", "sh", "bash", "json", "yaml", "yml", "toml", "sql"],
+  "excludePatterns": ["**/node_modules/**", "**/dist/**", "**/build/**", "**/.git/**", "**/coverage/**", "**/.next/**", "**/target/**", "**/vendor/**", "**/.smart-coding-cache/**"],
+  "smartIndexing": true,
+  "chunkSize": 15,
+  "chunkOverlap": 3,
+  "batchSize": 100,
+  "maxFileSize": 1048576,
+  "maxResults": 5,
+  "enableCache": true,
+  "cacheDirectory": "./.smart-coding-cache",
+  "watchFiles": true,
+  "verbose": false,
+  "embeddingModel": "Xenova/all-MiniLM-L6-v2",
+  "semanticWeight": 0.7,
+  "exactMatchBoost": 1.5
+}

package/features/clear-cache.js

@@ -0,0 +1,45 @@
+export class CacheClearer {
+  constructor(embedder, cache, config) {
+    this.cache = cache;
+    this.config = config;
+  }
+
+  async execute() {
+    await this.cache.clear();
+    return {
+      success: true,
+      message: `Cache cleared successfully. Next indexing will be a full rebuild.`,
+      cacheDirectory: this.config.cacheDirectory
+    };
+  }
+}
+
+export function getToolDefinition() {
+  return {
+    name: "clear_cache",
+    description: "Clears the embeddings cache, forcing a complete reindex on next search or manual index operation. Useful when encountering cache corruption or after major codebase changes.",
+    inputSchema: {
+      type: "object",
+      properties: {}
+    }
+  };
+}
+
+export async function handleToolCall(request, cacheClearer) {
+  try {
+    const result = await cacheClearer.execute();
+    return {
+      content: [{
+        type: "text",
+        text: `${result.message}\n\nCache directory: ${result.cacheDirectory}`
+      }]
+    };
+  } catch (error) {
+    return {
+      content: [{
+        type: "text",
+        text: `Failed to clear cache: ${error.message}`
+      }]
+    };
+  }
+}

package/features/hybrid-search.js

@@ -0,0 +1,114 @@
+import path from "path";
+import { cosineSimilarity } from "../lib/utils.js";
+
+export class HybridSearch {
+  constructor(embedder, cache, config) {
+    this.embedder = embedder;
+    this.cache = cache;
+    this.config = config;
+  }
+
+  async search(query, maxResults) {
+    const vectorStore = this.cache.getVectorStore();
+    
+    if (vectorStore.length === 0) {
+      return {
+        results: [],
+        message: "No code has been indexed yet. Please wait for initial indexing to complete."
+      };
+    }
+
+    // Generate query embedding
+    const queryEmbed = await this.embedder(query, { pooling: "mean", normalize: true });
+    const queryVector = Array.from(queryEmbed.data);
+
+    // Score all chunks
+    const scoredChunks = vectorStore.map(chunk => {
+      // Semantic similarity
+      let score = cosineSimilarity(queryVector, chunk.vector) * this.config.semanticWeight;
+      
+      // Exact match boost
+      const lowerQuery = query.toLowerCase();
+      const lowerContent = chunk.content.toLowerCase();
+      
+      if (lowerContent.includes(lowerQuery)) {
+        score += this.config.exactMatchBoost;
+      } else {
+        // Partial word matching
+        const queryWords = lowerQuery.split(/\s+/);
+        const matchedWords = queryWords.filter(word => 
+          word.length > 2 && lowerContent.includes(word)
+        ).length;
+        score += (matchedWords / queryWords.length) * 0.3;
+      }
+      
+      return { ...chunk, score };
+    });
+
+    // Get top results
+    const results = scoredChunks
+      .sort((a, b) => b.score - a.score)
+      .slice(0, maxResults);
+
+    return { results, message: null };
+  }
+
+  formatResults(results) {
+    if (results.length === 0) {
+      return "No matching code found for your query.";
+    }
+
+    return results.map((r, idx) => {
+      const relPath = path.relative(this.config.searchDirectory, r.file);
+      return `## Result ${idx + 1} (Relevance: ${(r.score * 100).toFixed(1)}%)\n` +
+             `**File:** \`${relPath}\`\n` +
+             `**Lines:** ${r.startLine}-${r.endLine}\n\n` +
+             "```" + path.extname(r.file).slice(1) + "\n" +
+             r.content + "\n" +
+             "```\n";
+    }).join("\n");
+  }
+}
+
+// MCP Tool definition for this feature
+export function getToolDefinition(config) {
+  return {
+    name: "semantic_search",
+    description: "Performs intelligent hybrid code search combining semantic understanding with exact text matching. Ideal for finding code by meaning (e.g., 'authentication logic', 'database queries') even with typos or variations. Returns the most relevant code snippets with file locations and line numbers.",
+    inputSchema: {
+      type: "object",
+      properties: {
+        query: { 
+          type: "string", 
+          description: "Search query - can be natural language (e.g., 'where do we handle user login') or specific terms" 
+        },
+        maxResults: {
+          type: "number",
+          description: "Maximum number of results to return (default: from config)",
+          default: config.maxResults
+        }
+      },
+      required: ["query"]
+    }
+  };
+}
+
+// Tool handler
+export async function handleToolCall(request, hybridSearch) {
+  const query = request.params.arguments.query;
+  const maxResults = request.params.arguments.maxResults || hybridSearch.config.maxResults;
+  
+  const { results, message } = await hybridSearch.search(query, maxResults);
+  
+  if (message) {
+    return {
+      content: [{ type: "text", text: message }]
+    };
+  }
+
+  const formattedText = hybridSearch.formatResults(results);
+  
+  return {
+    content: [{ type: "text", text: formattedText }]
+  };
+}

package/features/index-codebase.js

@@ -0,0 +1,213 @@
+import { glob } from "glob";
+import fs from "fs/promises";
+import chokidar from "chokidar";
+import path from "path";
+import { smartChunk, hashContent } from "../lib/utils.js";
+
+export class CodebaseIndexer {
+  constructor(embedder, cache, config) {
+    this.embedder = embedder;
+    this.cache = cache;
+    this.config = config;
+    this.watcher = null;
+  }
+
+  async indexFile(file) {
+    const fileName = path.basename(file);
+    if (this.config.verbose) {
+      console.error(`[Indexer] Processing: ${fileName}...`);
+    }
+    
+    try {
+      // Check file size first
+      const stats = await fs.stat(file);
+      
+      // Skip directories
+      if (stats.isDirectory()) {
+        return 0;
+      }
+      
+      if (stats.size > this.config.maxFileSize) {
+        if (this.config.verbose) {
+          console.error(`[Indexer] Skipped ${fileName} (too large: ${(stats.size / 1024 / 1024).toFixed(2)}MB)`);
+        }
+        return 0;
+      }
+      
+      const content = await fs.readFile(file, "utf-8");
+      const hash = hashContent(content);
+      
+      // Skip if file hasn't changed
+      if (this.cache.getFileHash(file) === hash) {
+        if (this.config.verbose) {
+          console.error(`[Indexer] Skipped ${fileName} (unchanged)`);
+        }
+        return 0;
+      }
+
+      if (this.config.verbose) {
+        console.error(`[Indexer] Indexing ${fileName}...`);
+      }
+      
+      // Remove old chunks for this file
+      this.cache.removeFileFromStore(file);
+      
+      const chunks = smartChunk(content, file, this.config);
+      let addedChunks = 0;
+
+      for (const chunk of chunks) {
+        try {
+          const output = await this.embedder(chunk.text, { pooling: "mean", normalize: true });
+          
+          this.cache.addToStore({
+            file,
+            startLine: chunk.startLine,
+            endLine: chunk.endLine,
+            content: chunk.text,
+            vector: Array.from(output.data)
+          });
+          addedChunks++;
+        } catch (embeddingError) {
+          console.error(`[Indexer] Failed to embed chunk in ${fileName}:`, embeddingError.message);
+        }
+      }
+
+      this.cache.setFileHash(file, hash);
+      if (this.config.verbose) {
+        console.error(`[Indexer] Completed ${fileName} (${addedChunks} chunks)`);
+      }
+      return addedChunks;
+    } catch (error) {
+      console.error(`[Indexer] Error indexing ${fileName}:`, error.message);
+      return 0;
+    }
+  }
+
+  async indexAll() {
+    console.error(`[Indexer] Indexing files in ${this.config.searchDirectory}...`);
+    
+    const pattern = `${this.config.searchDirectory}/**/*.{${this.config.fileExtensions.join(",")}}`;
+    const files = await glob(pattern, { 
+      ignore: this.config.excludePatterns,
+      absolute: true 
+    });
+
+    console.error(`[Indexer] Found ${files.length} files to process`);
+    
+    let totalChunks = 0;
+    let processedFiles = 0;
+    let skippedFiles = 0;
+    
+    // Process files in parallel batches for speed
+    const BATCH_SIZE = this.config.batchSize || 100;
+    
+    for (let i = 0; i < files.length; i += BATCH_SIZE) {
+      const batch = files.slice(i, i + BATCH_SIZE);
+      
+      // Process batch in parallel
+      const results = await Promise.all(
+        batch.map(file => this.indexFile(file))
+      );
+      
+      // Aggregate results
+      for (const chunksAdded of results) {
+        totalChunks += chunksAdded;
+        processedFiles++;
+        if (chunksAdded === 0) skippedFiles++;
+      }
+      
+      // Progress indicator every 500 files (less console overhead)
+      if (processedFiles % 500 === 0 || processedFiles === files.length) {
+        console.error(`[Indexer] Progress: ${processedFiles}/${files.length} files processed...`);
+      }
+    }
+
+    console.error(`[Indexer] Indexed ${totalChunks} code chunks from ${files.length} files (${skippedFiles} unchanged)`);
+    await this.cache.save();
+  }
+
+  setupFileWatcher() {
+    if (!this.config.watchFiles) return;
+
+    const pattern = this.config.fileExtensions.map(ext => `**/*.${ext}`);
+    
+    this.watcher = chokidar.watch(pattern, {
+      cwd: this.config.searchDirectory,
+      ignored: this.config.excludePatterns,
+      persistent: true,
+      ignoreInitial: true
+    });
+
+    this.watcher
+      .on("add", async (filePath) => {
+        const fullPath = path.join(this.config.searchDirectory, filePath);
+        console.error(`[Indexer] New file detected: ${filePath}`);
+        await this.indexFile(fullPath);
+        await this.cache.save();
+      })
+      .on("change", async (filePath) => {
+        const fullPath = path.join(this.config.searchDirectory, filePath);
+        console.error(`[Indexer] File changed: ${filePath}`);
+        await this.indexFile(fullPath);
+        await this.cache.save();
+      })
+      .on("unlink", (filePath) => {
+        const fullPath = path.join(this.config.searchDirectory, filePath);
+        console.error(`[Indexer] File deleted: ${filePath}`);
+        this.cache.removeFileFromStore(fullPath);
+        this.cache.deleteFileHash(fullPath);
+        this.cache.save();
+      });
+
+    console.error("[Indexer] File watcher enabled for incremental indexing");
+  }
+
+  async initialize() {
+    await this.indexAll();
+    this.setupFileWatcher();
+  }
+}
+
+// MCP Tool definition for this feature
+export function getToolDefinition() {
+  return {
+    name: "index_codebase",
+    description: "Manually trigger a full reindex of the codebase. This will scan all files and update the embeddings cache. Useful after large code changes or if the index seems out of date.",
+    inputSchema: {
+      type: "object",
+      properties: {
+        force: {
+          type: "boolean",
+          description: "Force reindex even if files haven't changed",
+          default: false
+        }
+      }
+    }
+  };
+}
+
+// Tool handler
+export async function handleToolCall(request, indexer) {
+  const force = request.params.arguments?.force || false;
+  
+  if (force) {
+    // Clear cache to force full reindex
+    indexer.cache.setVectorStore([]);
+    indexer.cache.fileHashes = new Map();
+  }
+  
+  await indexer.indexAll();
+  
+  const vectorStore = indexer.cache.getVectorStore();
+  const stats = {
+    totalChunks: vectorStore.length,
+    totalFiles: new Set(vectorStore.map(v => v.file)).size
+  };
+  
+  return {
+    content: [{
+      type: "text",
+      text: `Codebase reindexed successfully.\n\nStatistics:\n- Files indexed: ${stats.totalFiles}\n- Code chunks: ${stats.totalChunks}`
+    }]
+  };
+}