@softerist/heuristic-mcp 2.1.46 → 3.0.0

package/GEMINI.md

@@ -0,0 +1,73 @@
+# Heuristic MCP
+
+## Project Overview
+
+**@softerist/heuristic-mcp** is an enhanced Model Context Protocol (MCP) server designed to provide intelligent semantic code search capabilities. It integrates features like find-similar-code, recency-aware ranking, call-graph proximity boosts, and smart chunking to improve code retrieval accuracy for AI assistants (Antigravity, Cursor, Claude Desktop, VS Code).
+
+**Key Features:**
+*   **Semantic Search:** Finds code by meaning using embeddings.
+*   **Smart Indexing:** Automatically detects project types and applies ignores.
+*   **Recency & Proximity:** Boosts results based on file modification time and call graph relationships.
+*   **ANN Index:** Optional Approximate Nearest Neighbor index for performance on large codebases.
+*   **Zero-touch Setup:** Auto-registers with supported IDEs.
+
+## Architecture
+
+The project follows a modular architecture:
+
+*   **`index.js`**: Main entry point. Initializes the MCP server, loads configuration, and registers features.
+*   **`lib/`**: Core libraries.
+    *   `config.js`: Configuration loader and environment variable handling.
+    *   `cache.js`: Manages embedding vectors persistence and ANN index.
+    *   `cache-utils.js`: Stale cache detection/cleanup.
+    *   `utils.js`: Shared utilities (hashing, similarity, smart chunking).
+    *   `call-graph.js`: Extracts symbols and builds a lightweight call graph.
+    *   `tokenizer.js`: Token estimation.
+    *   `embedding-worker.js`: Worker-thread embedder runner.
+    *   `embedding-process.js`: Child-process embedder runner (isolation).
+    *   `ignore-patterns.js`: Smart ignore patterns by detected project type.
+    *   `json-worker.js` / `json-writer.js`: Streaming JSON helpers.
+    *   `logging.js`: Log file helpers.
+*   **`features/`**: Pluggable feature modules. Each module exports a class, tool definition, and handler.
+    *   `hybrid-search.js`: Semantic search logic.
+    *   `index-codebase.js`: File discovery and indexing.
+    *   `find-similar-code.js`: Logic for finding similar snippets.
+    *   `lifecycle.js` & `register.js`: CLI and IDE registration helpers.
+*   **`scripts/`**: Utility scripts (postinstall, model download, cache clearing).
+*   **`tools/`**: Developer-only helpers (manual search script).
+
+## Building and Running
+
+### Prerequisites
+*   Node.js >= 18.0.0
+
+### Key Commands
+
+| Command | Description |
+| :--- | :--- |
+| `npm start` | Runs the server (production mode). |
+| `npm run dev` | Runs the server in development mode (watch mode). |
+| `npm test` | Runs the test suite using Vitest. |
+| `npm run clean` | Clears the cache directory. |
+| `npm run lint` | Runs ESLint. |
+| `npm run format` | Runs Prettier. |
+
+**Note on Testing:** Tests are configured to run sequentially (`fileParallelism: false`) to manage memory usage of the embedding models. Use `USE_REAL_EMBEDDER=true npm test` to test with the actual model instead of mocks.
+
+## Development Conventions
+
+*   **Style:** Modern ES6+ JavaScript.
+*   **Modularity:** New features should be added as separate modules in `features/` following the pattern:
+    1.  **Class:** `export class FeatureName { ... }`
+    2.  **Tool Def:** `export function getToolDefinition(config) { ... }`
+    3.  **Handler:** `export async function handleToolCall(request, instance) { ... }`
+*   **Logging:** Use `console.info()` for normal server lifecycle output (redirected to logs when running in MCP mode). Use `console.warn()` for non-fatal issues and `console.error()` for errors. CLI utilities may also use `console.info()` for user-facing output.
+*   **Configuration:** All features should accept a `config` object. Environment variables (prefix `SMART_CODING_`) override `config.json` values.
+
+## Key Files
+
+*   **`package.json`**: Dependencies and scripts.
+*   **`config.json`**: Default/example configuration.
+*   **`ARCHITECTURE.md`**: Detailed architectural documentation.
+*   **`CONTRIBUTING.md`**: Guidelines for contributors.
+*   **`vitest.config.js`**: Test runner configuration.

package/LICENSE

@@ -1,21 +1,21 @@
-MIT License
-
-Copyright (c) 2025 Omar Haris
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+MIT License
+
+Copyright (c) 2025 Omar Haris
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,117 +1,224 @@
-# Heuristic MCP Server 🧠
+# Heuristic MCP Server
 
-> An **enhanced** MCP server for your codebase. It provides **intelligent semantic search**, **Find-Similar-Code**, **Recency Ranking**, and **Smart Chunking**.
-> Optimized for Antigravity, Cursor, and Claude Desktop.
+An enhanced MCP server for your codebase. It provides intelligent semantic search, find-similar-code, recency-aware ranking, call-graph proximity boosts, and smart chunking. Optimized for Antigravity, Cursor, Claude Desktop, and VS Code.
 
 ---
 
-## 🚀 Key Features
+## Key Features
 
-- **Zero-Config Installation**: Automatically detects your IDE (Antigravity, Cursor, Claude) and configures itself.
-- **Smart Indexing**: Automatically identifies your project type (Python, JS, etc.) and ignores irrelevant files.
-- **Robust Caching**: Pre-downloads AI models to avoid network blocks and caches embeddings globally or locally.
-- **Semantic Search**: Finds code by *meaning*, not just keywords (e.g., "auth logic" finds `login.ts`).
-- **Resilient**: Self-healing configuration and automatic recovery from race conditions.
+- Zero-touch setup: postinstall auto-registers the MCP server with supported IDEs when possible.
+- Smart indexing: detects project type and applies smart ignore patterns on top of your excludes.
+- Semantic search: find code by meaning, not just keywords.
+- Find similar code: locate near-duplicate or related patterns from a snippet.
+- Recency ranking and call-graph boosting: surfaces fresh and related code.
+- Optional ANN index: faster candidate retrieval for large codebases.
+- Optional binary vector store: mmap-friendly cache format for large repos.
 
 ---
 
-## 📦 Installation
+## Installation
 
-To install globally (recommended):
+Install globally (recommended):
 
 ```bash
 npm install -g @softerist/heuristic-mcp
 ```
 
-**That's it!**
-- The installer automatically creates the `mcp_config.json` for your IDE.
-- It pre-downloads the AI model (`all-MiniLM-L6-v2`) to your cache.
-- Just **Restart your IDE** (or Reload Window) to start using it.
+What happens during install:
+
+- Registration runs automatically (`scripts/postinstall.js`).
+- Model pre-download is attempted (`scripts/download-model.js`). If offline, it will be skipped and downloaded on first run.
+
+If auto-registration did not update your IDE config, run:
+
+```bash
+heuristic-mcp --register
+```
 
 ---
 
-## 🛠️ CLI Commands
+## CLI Commands
+
+The `heuristic-mcp` binary manages the server lifecycle.
 
-The `heuristic-mcp` tool is your control center.
+### Status
 
-### Check Health & Status (Snapshots)
-Use this to verify if the server is running and check indexing progress.
 ```bash
 heuristic-mcp --status
 ```
-**Output:**
-- 🟢 **Server Status**: Shows if the background process is running (PID).
-- 📁 **Cache Info**: Shows number of files indexed, chunks, and "Initializing..." status if still working.
-- ⚙️ **Config Check**: Validates that your IDE config files exist.
 
-### Live Logs (Streaming)
-Use this to watch the server's brain at work in real-time.
+Shows server PID(s) and cache stats.
+
+### Logs
+
 ```bash
 heuristic-mcp --logs
 ```
-**Output:**
-- Streams live logs from the server.
-- Shows file indexing progress (`Processing 100/236 files...`).
-- Useful for debugging why a specific file isn't being indexed.
 
-### Manual Registration
-If you need to re-register the server manually:
+Tails the server log for the current workspace (defaults to last 200 lines and follows).
+
+Optional flags:
+
+```bash
+heuristic-mcp --logs --tail 100
+heuristic-mcp --logs --no-follow
+```
+
+### Version
+
+```bash
+heuristic-mcp --version
+```
+
+### Register (manual)
+
 ```bash
 heuristic-mcp --register
+heuristic-mcp --register antigravity
+heuristic-mcp --register cursor
+heuristic-mcp --register "Claude Desktop"
 ```
 
-### Stop Server
-Forcefully stop all running instances:
+### Start/Stop
+
 ```bash
+heuristic-mcp --start
 heuristic-mcp --stop
 ```
 
+`--stop` also disables the MCP server entry in supported IDE configs so the IDE won't immediately respawn it. `--start` re-enables it (restart/reload the IDE to launch).
+
 ### Clear Cache
-Wipe the index to force a complete rebuild:
+
 ```bash
-heuristic-mcp --clean
+heuristic-mcp --clear-cache
 ```
 
+Clears the cache for the current working directory (or `--workspace` if provided) and removes stale cache directories without metadata.
+
 ---
 
-## ⚙️ Configuration (`config.json`)
+## Configuration (`config.json`)
+
+Configuration is loaded from your workspace root when the server runs with `--workspace` (this is how IDEs launch it). In server mode, it falls back to the package `config.json` and then your current working directory.
 
-You can customize behavior by creating a `config.json` in your project root or `~/.heuristic-mcp/config.json`.
+Example `config.json`:
 
 ```json
 {
   "excludePatterns": ["**/legacy-code/**", "**/*.test.ts"],
+  "fileNames": ["Dockerfile", ".env.example", "Makefile"],
   "smartIndexing": true,
-  "embeddingModel": "Xenova/all-MiniLM-L6-v2",
-  "workerThreads": "auto"
+  "embeddingModel": "jinaai/jina-embeddings-v2-base-code",
+  "workerThreads": 0,
+  "recencyBoost": 0.1,
+  "recencyDecayDays": 30,
+  "callGraphEnabled": true,
+  "callGraphBoost": 0.15,
+  "annEnabled": true,
+  "vectorStoreFormat": "binary",
+  "vectorStoreContentMode": "external",
+  "vectorStoreLoadMode": "disk",
+  "contentCacheEntries": 256,
+  "vectorCacheEntries": 64,
+  "clearCacheAfterIndex": true
 }
 ```
 
+Cache location:
+
+- By default, the cache is stored in a global OS cache directory under `heuristic-mcp/<hash>`.
+- You can override with `cacheDirectory` in `config.json`.
+
 ### Environment Variables
-Override settings on the fly:
-- `SMART_CODING_VERBOSE=true`: Enable debug logs.
-- `SMART_CODING_WORKER_THREADS=4`: Force specific thread count.
+
+Selected overrides (prefix `SMART_CODING_`):
+
+- `SMART_CODING_VERBOSE=true|false`
+- `SMART_CODING_WORKER_THREADS=auto|0|N`
+- `SMART_CODING_BATCH_SIZE=100`
+- `SMART_CODING_CHUNK_SIZE=25`
+- `SMART_CODING_MAX_RESULTS=5`
+- `SMART_CODING_RECENCY_BOOST=0.1`
+- `SMART_CODING_RECENCY_DECAY_DAYS=30`
+- `SMART_CODING_ANN_ENABLED=true|false`
+- `SMART_CODING_ANN_EF_SEARCH=64`
+- `SMART_CODING_VECTOR_STORE_FORMAT=json|binary`
+- `SMART_CODING_VECTOR_STORE_CONTENT_MODE=external|inline`
+- `SMART_CODING_VECTOR_STORE_LOAD_MODE=memory|disk`
+- `SMART_CODING_CONTENT_CACHE_ENTRIES=256`
+- `SMART_CODING_VECTOR_CACHE_ENTRIES=64`
+- `SMART_CODING_CLEAR_CACHE_AFTER_INDEX=true|false`
+
+See `lib/config.js` for the full list.
+
+### Binary Vector Store
+
+Set `vectorStoreFormat` to `binary` to use the on-disk binary cache. This keeps vectors and content out of JS heap
+and reads on demand. Recommended for large repos.
+
+- `vectorStoreContentMode=external` keeps content in the binary file and only loads for top-N results.
+- `contentCacheEntries` controls the small in-memory LRU for decoded content strings.
+- `vectorStoreLoadMode=disk` streams vectors from disk to reduce memory usage.
+- `vectorCacheEntries` controls the small in-memory LRU for vectors when using disk mode.
+- `clearCacheAfterIndex=true` drops in-memory vectors after indexing and reloads lazily on next query.
+- Note: `annEnabled=true` with `vectorStoreLoadMode=disk` can increase disk reads during ANN rebuilds on large indexes.
+
+### Benchmarking Search
+
+Use the built-in script to compare memory vs latency tradeoffs:
+
+```bash
+node tools/scripts/benchmark-search.js --query "database connection" --runs 10
+```
+
+Compare modes quickly:
+
+```bash
+SMART_CODING_VECTOR_STORE_LOAD_MODE=memory node tools/scripts/benchmark-search.js --runs 10
+SMART_CODING_VECTOR_STORE_LOAD_MODE=disk node tools/scripts/benchmark-search.js --runs 10
+SMART_CODING_VECTOR_STORE_FORMAT=binary SMART_CODING_VECTOR_STORE_LOAD_MODE=disk node tools/scripts/benchmark-search.js --runs 10
+```
+
+Note: On small repos, disk mode may be slightly slower and show noisy RSS deltas; benefits are clearer on large indexes with a small `vectorCacheEntries`.
 
 ---
 
-## 🔧 Troubleshooting
+## Troubleshooting
+
+**Server isn't starting**
+
+1. Run `heuristic-mcp --status` to check config and cache status.
+2. Run `heuristic-mcp --logs` to see startup errors.
+
+**Search returns no results**
+
+- Check `heuristic-mcp --status` for indexing progress.
+- If indexing shows zero files, review `excludePatterns` and `fileExtensions`.
+
+**Model download fails**
 
-**"Server isn't starting"**
-1. Run `heuristic-mcp --status` to see if config files exist.
-2. Run `heuristic-mcp --logs` and then Reload Window to see startup errors.
+- The install step tries to pre-download the model, but it can be skipped offline.
+- The server will download on first run; ensure network access at least once.
+
+**Clear cache**
+
+- Use the MCP tool `c_clear_cache`, run `heuristic-mcp --clear-cache`, or delete the cache directory. For local dev, run `npm run clean`.
+
+**Inspect cache**
+
+```bash
+node tools/scripts/cache-stats.js --workspace <path>
+```
 
-**"Search returns no results"**
-- Check `heuristic-mcp --status`. Does it say "Indexing: ✅ COMPLETE"?
-- If it says "Initializing...", wait a moment.
-- If it says "NO FILES", check your `.gitignore` or `excludePatterns`.
+**Stop doesn't stick**
 
-**"Network error downloading model"**
-- We pre-download models during `npm install`. Try running `npm install -g @softerist/heuristic-mcp` again to retry the download.
+- The IDE will auto-restart the server if it's still enabled in its config. `--stop` now disables the server entry for Antigravity, Cursor, Claude Desktop, and VS Code (when using common MCP settings keys). Restart the IDE after `--start` to re-enable.
 
 ---
 
-## 🤝 Contributing
+## Contributing
 
-Fork it, fix it, ship it. Open a PR!
+See `CONTRIBUTING.md` for guidelines.
 
 License: MIT