@softerist/heuristic-mcp 3.0.6 → 3.0.8

package/README.md

@@ -10,9 +10,12 @@ An enhanced MCP server for your codebase. It provides intelligent semantic searc
 - Smart indexing: detects project type and applies smart ignore patterns on top of your excludes.
 - Semantic search: find code by meaning, not just keywords.
 - Find similar code: locate near-duplicate or related patterns from a snippet.
+- Package version lookup: check latest versions from npm, PyPI, crates.io, Maven, and more.
+- Workspace switching: change workspace at runtime without restarting the server.
 - Recency ranking and call-graph boosting: surfaces fresh and related code.
 - Optional ANN index: faster candidate retrieval for large codebases.
 - Optional binary vector store: mmap-friendly cache format for large repos.
+- Flexible embedding dimensions: MRL-compatible dimension reduction (64-768d) for speed/quality tradeoffs.
 
 ---
 
@@ -128,6 +131,27 @@ Example `config.jsonc`:
 }
 ```
 
+### Embedding Model & Dimension Options
+
+**Default model:** `jinaai/jina-embeddings-v2-base-code` (768 dimensions)
+
+> **Important:** The default Jina model was **not** trained with Matryoshka Representation Learning (MRL). Dimension reduction (`embeddingDimension`) will significantly degrade search quality with this model. Only use dimension reduction with MRL-trained models.
+
+For faster search with smaller embeddings, switch to an MRL-compatible model:
+
+```json
+{
+  "embeddingModel": "nomic-ai/nomic-embed-text-v1.5",
+  "embeddingDimension": 128
+}
+```
+
+**MRL-compatible models:**
+- `nomic-ai/nomic-embed-text-v1.5` — recommended for 128d/256d
+- Other models explicitly trained with Matryoshka loss
+
+**embeddingDimension values:** `64 | 128 | 256 | 512 | 768 | null` (null = full dimensions)
+
 Cache location:
 
 - By default, the cache is stored in a global OS cache directory under `heuristic-mcp/<hash>`.
@@ -148,7 +172,7 @@ Selected overrides (prefix `SMART_CODING_`):
 - `SMART_CODING_RECENCY_DECAY_DAYS=30` — days until recency boost decays to 0.
 - `SMART_CODING_ANN_ENABLED=true|false` — enable ANN index.
 - `SMART_CODING_ANN_EF_SEARCH=64` — ANN search quality/speed tradeoff.
-- `SMART_CODING_VECTOR_STORE_FORMAT=json|binary` — on-disk vector store format.
+- `SMART_CODING_VECTOR_STORE_FORMAT=json|binary|sqlite` — on-disk vector store format.
 - `SMART_CODING_VECTOR_STORE_CONTENT_MODE=external|inline` — where content is stored for binary format.
 - `SMART_CODING_VECTOR_STORE_LOAD_MODE=memory|disk` — vector loading strategy.
 - `SMART_CODING_CONTENT_CACHE_ENTRIES=256` — LRU entries for decoded content.
@@ -156,6 +180,7 @@ Selected overrides (prefix `SMART_CODING_`):
 - `SMART_CODING_CLEAR_CACHE_AFTER_INDEX=true|false` — drop in-memory vectors after indexing.
 - `SMART_CODING_EXPLICIT_GC=true|false` — opt-in to explicit GC (requires `--expose-gc`).
 - `SMART_CODING_INCREMENTAL_GC_THRESHOLD_MB=2048` — RSS threshold for running incremental GC after watcher updates (requires explicit GC).
+- `SMART_CODING_EMBEDDING_DIMENSION=64|128|256|512|768` — MRL dimension reduction (only for MRL-trained models).
 
 See `lib/config.js` for the full list.
 
@@ -171,6 +196,28 @@ and reads on demand. Recommended for large repos.
 - `clearCacheAfterIndex=true` drops in-memory vectors after indexing and reloads lazily on next query.
 - Note: `annEnabled=true` with `vectorStoreLoadMode=disk` can increase disk reads during ANN rebuilds on large indexes.
 
+### SQLite Vector Store
+
+Set `vectorStoreFormat` to `sqlite` to use SQLite for persistence. This provides:
+
+- ACID transactions for reliable writes
+- Simpler concurrent access
+- Standard database format for inspection
+
+```json
+{
+  "vectorStoreFormat": "sqlite"
+}
+```
+
+The vectors and content are stored in `vectors.sqlite` in your cache directory. You can inspect it with any SQLite browser.
+`vectorStoreContentMode` and `vectorStoreLoadMode` are respected for SQLite (use `vectorStoreLoadMode=disk` to avoid loading vectors into memory).
+
+**Tradeoffs vs Binary:**
+- Slightly higher read overhead (SQL queries vs direct memory access)
+- Better write reliability (transactions)
+- Easier debugging (standard SQLite file)
+
 ### Benchmarking Search
 
 Use the built-in script to compare memory vs latency tradeoffs:
@@ -191,6 +238,49 @@ Note: On small repos, disk mode may be slightly slower and show noisy RSS deltas
 
 ---
 
+## MCP Tools Reference
+
+### `a_semantic_search`
+Find code by meaning. Ideal for natural language queries like "authentication logic" or "database queries".
+
+### `b_index_codebase`
+Manually trigger a full reindex. Useful after large code changes.
+
+### `c_clear_cache`
+Clear the embeddings cache and force reindex.
+
+### `d_ann_config`
+Configure the ANN (Approximate Nearest Neighbor) index. Actions: `stats`, `set_ef_search`, `rebuild`.
+
+### `d_find_similar_code`
+Find similar code patterns given a snippet. Useful for finding duplicates or refactoring opportunities.
+
+### `e_check_package_version`
+Fetch the latest version of a package from its official registry.
+
+**Supported registries:**
+- **npm** (default): `lodash`, `@types/node`
+- **PyPI**: `pip:requests`, `pypi:django`
+- **crates.io**: `cargo:serde`, `rust:tokio`
+- **Maven**: `maven:org.springframework:spring-core`
+- **Go**: `go:github.com/gin-gonic/gin`
+- **RubyGems**: `gem:rails`
+- **NuGet**: `nuget:Newtonsoft.Json`
+- **Packagist**: `composer:laravel/framework`
+- **Hex**: `hex:phoenix`
+- **pub.dev**: `pub:flutter`
+- **Homebrew**: `brew:node`
+- **Conda**: `conda:numpy`
+
+### `f_set_workspace`
+Change the workspace directory at runtime. Updates search directory, cache location, and optionally triggers reindex.
+
+**Parameters:**
+- `workspacePath` (required): Absolute path to the new workspace
+- `reindex` (optional, default: `true`): Whether to trigger a full reindex
+
+---
+
 ## Troubleshooting
 
 **Server isn't starting**

package/config.jsonc

@@ -666,7 +666,7 @@
   "embeddingModel": "jinaai/jina-embeddings-v2-base-code",
   // Preload the embedding model on startup.
   "preloadEmbeddingModel": true,
-  // Vector store format: json or binary.
+  // Vector store format: json, binary, or sqlite.
   "vectorStoreFormat": "binary",
   // Content storage: external or inline.
   "vectorStoreContentMode": "external",

package/features/index-codebase.js

@@ -18,6 +18,25 @@ function toFloat32Array(vector) {
   return new Float32Array(vector);
 }
 
+function sliceAndNormalize(vector, targetDim) {
+  if (!targetDim || targetDim >= vector.length) {
+    return vector;
+  }
+
+  const sliced = vector.slice(0, targetDim);
+  let sumSquares = 0;
+  for (let i = 0; i < targetDim; i++) {
+    sumSquares += sliced[i] * sliced[i];
+  }
+  const norm = Math.sqrt(sumSquares);
+  if (norm > 0) {
+    for (let i = 0; i < targetDim; i++) {
+      sliced[i] /= norm;
+    }
+  }
+  return sliced;
+}
+
 function isTestEnv() {
   return process.env.VITEST === 'true' || process.env.NODE_ENV === 'test';
 }
@@ -93,15 +112,7 @@ export class CodebaseIndexer {
     this.isIndexing = false;
     this.processingWatchEvents = false;
     this.pendingWatchEvents = new Map();
-    const cacheRelative = this.getCacheRelativePath();
-    const autoExclude = ['.smart-coding-cache'];
-    if (cacheRelative) {
-      autoExclude.push(cacheRelative, `${cacheRelative}/**`);
-    }
-    this.excludeMatchers = buildExcludeMatchers([
-      ...autoExclude,
-      ...(this.config.excludePatterns || []),
-    ]);
+    this.rebuildExcludeMatchers();
     this.gitignore = ignore();
     this.workerFailureCount = 0;
     this.workersDisabledUntil = 0;
@@ -138,6 +149,45 @@ export class CodebaseIndexer {
     this._embeddingChildRestartThresholdMb = this.getEmbeddingChildRestartThresholdMb();
   }
 
+  rebuildExcludeMatchers() {
+    const cacheRelative = this.getCacheRelativePath();
+    const autoExclude = ['.smart-coding-cache'];
+    if (cacheRelative) {
+      autoExclude.push(cacheRelative, `${cacheRelative}/**`);
+    }
+    this.excludeMatchers = buildExcludeMatchers([
+      ...autoExclude,
+      ...(this.config.excludePatterns || []),
+    ]);
+  }
+
+  async updateWorkspaceState({ restartWatcher = false } = {}) {
+    this.workspaceRoot = this.config.searchDirectory
+      ? path.resolve(this.config.searchDirectory)
+      : null;
+    this.workspaceRootReal = null;
+    this.rebuildExcludeMatchers();
+    this.gitignore = ignore();
+    if (this.pendingWatchEvents) {
+      this.pendingWatchEvents.clear();
+    }
+    if (this._watcherDebounceTimers) {
+      for (const timer of this._watcherDebounceTimers.values()) {
+        clearTimeout(timer);
+      }
+      this._watcherDebounceTimers.clear();
+    }
+    if (this._watcherInProgress) {
+      this._watcherInProgress.clear();
+    }
+
+    if (restartWatcher && this.config.watchFiles) {
+      await this.setupFileWatcher();
+    } else if (this.config.watchFiles) {
+      await this.loadGitignore();
+    }
+  }
+
   getEmbeddingChildRestartThresholdMb() {
     const totalMb = typeof os.totalmem === 'function' ? os.totalmem() / 1024 / 1024 : 8192;
     if (this.isHeavyEmbeddingModel()) {
@@ -396,6 +446,7 @@ export class CodebaseIndexer {
               workerData: {
                 workerId: i,
                 embeddingModel: this.config.embeddingModel,
+                embeddingDimension: this.config.embeddingDimension || null,
                 verbose: this.config.verbose,
                 numThreads: threadsPerWorker,
                 searchDirectory: this.config.searchDirectory,
@@ -546,6 +597,7 @@ export class CodebaseIndexer {
       workerData: {
         workerId: index,
         embeddingModel: this.config.embeddingModel,
+        embeddingDimension: this.config.embeddingDimension || null,
         verbose: this.config.verbose,
         numThreads: 12, // Use 12 threads (50% of cores) for max throughput
         searchDirectory: this.config.searchDirectory,
@@ -1201,10 +1253,23 @@ export class CodebaseIndexer {
         await this.stopEmbeddingProcessSession({ preserveStats: true });
         await this.startEmbeddingProcessSession();
       }
-      return results;
+      return this.applyEmbeddingDimensionToResults(results);
     });
   }
 
+  applyEmbeddingDimensionToResults(results) {
+    const targetDim = this.config.embeddingDimension;
+    if (!targetDim || !Array.isArray(results)) {
+      return results;
+    }
+    for (const result of results) {
+      if (!result || !result.vector) continue;
+      const floatVector = toFloat32Array(result.vector);
+      result.vector = sliceAndNormalize(floatVector, targetDim);
+    }
+    return results;
+  }
+
   async processChunksInChildProcess(chunks) {
     if (this._embeddingProcessSessionActive) {
       return this.processChunksInPersistentChild(chunks);
@@ -1294,7 +1359,7 @@ export class CodebaseIndexer {
         }
         try {
           const parsed = JSON.parse(stdout);
-          resolve(parsed?.results || []);
+          resolve(this.applyEmbeddingDimensionToResults(parsed?.results || []));
         } catch (err) {
           this.recordWorkerFailure(`child process parse error (${err.message})`);
           resolve([]);
@@ -1324,7 +1389,10 @@ export class CodebaseIndexer {
           normalize: true,
         });
         // CRITICAL: Deep copy to release ONNX tensor memory
-        const vector = new Float32Array(output.data);
+        let vector = toFloat32Array(output.data);
+        if (this.config.embeddingDimension) {
+          vector = sliceAndNormalize(vector, this.config.embeddingDimension);
+        }
         // Help GC by nullifying the large buffer reference (dispose() doesn't exist in transformers.js)
         try {
           output.data = null;

package/features/package-version.js

@@ -0,0 +1,301 @@
+/**
+ * Package Version Lookup Tool
+ *
+ * Fetches the latest version of packages from various registries.
+ * Supports npm, PyPI, crates.io, Maven, Go, RubyGems, NuGet, Packagist, Hex, pub.dev, and more.
+ */
+
+// Registry configurations with their API endpoints and response parsers
+const REGISTRIES = {
+  npm: {
+    name: 'npm',
+    pattern: /^(?:npm:)?(.+)$/,
+    url: (pkg) => `https://registry.npmjs.org/${encodeURIComponent(pkg)}/latest`,
+    parse: (data) => data.version,
+    detect: (pkg) =>
+      pkg.startsWith('@') || /^[a-z0-9][-a-z0-9._]*$/i.test(pkg.replace(/^npm:/, '')),
+  },
+  pypi: {
+    name: 'PyPI',
+    pattern: /^(?:pip:|pypi:)(.+)$/,
+    url: (pkg) => `https://pypi.org/pypi/${encodeURIComponent(pkg)}/json`,
+    parse: (data) => data.info.version,
+    detect: () => false, // Requires explicit prefix
+  },
+  crates: {
+    name: 'crates.io',
+    pattern: /^(?:cargo:|crates:|rust:)(.+)$/,
+    url: (pkg) => `https://crates.io/api/v1/crates/${encodeURIComponent(pkg)}`,
+    parse: (data) => data.crate.max_version,
+    headers: { 'User-Agent': 'heuristic-mcp/1.0' },
+    detect: () => false,
+  },
+  go: {
+    name: 'Go',
+    pattern: /^(?:go:)(.+)$/,
+    url: (pkg) => `https://proxy.golang.org/${encodeURIComponent(pkg)}/@latest`,
+    parse: (data) => data.Version,
+    detect: (pkg) => pkg.includes('/') && pkg.includes('.'),
+  },
+  rubygems: {
+    name: 'RubyGems',
+    pattern: /^(?:gem:|ruby:)(.+)$/,
+    url: (pkg) => `https://rubygems.org/api/v1/gems/${encodeURIComponent(pkg)}.json`,
+    parse: (data) => data.version,
+    detect: () => false,
+  },
+  nuget: {
+    name: 'NuGet',
+    pattern: /^(?:nuget:|dotnet:)(.+)$/,
+    url: (pkg) =>
+      `https://api.nuget.org/v3-flatcontainer/${encodeURIComponent(pkg.toLowerCase())}/index.json`,
+    parse: (data) => data.versions[data.versions.length - 1],
+    detect: () => false,
+  },
+  packagist: {
+    name: 'Packagist',
+    pattern: /^(?:composer:|php:)(.+)$/,
+    url: (pkg) => `https://repo.packagist.org/p2/${encodeURIComponent(pkg)}.json`,
+    parse: (data) => {
+      const pkgName = Object.keys(data.packages)[0];
+      const versions = data.packages[pkgName];
+      // Filter out dev versions and get latest stable
+      const stable = versions.find((v) => !v.version.includes('dev'));
+      return stable ? stable.version : versions[0].version;
+    },
+    detect: (pkg) => pkg.includes('/'),
+  },
+  hex: {
+    name: 'Hex',
+    pattern: /^(?:hex:|elixir:|mix:)(.+)$/,
+    url: (pkg) => `https://hex.pm/api/packages/${encodeURIComponent(pkg)}`,
+    parse: (data) => {
+      const releases = data.releases;
+      return releases.length > 0 ? releases[0].version : null;
+    },
+    detect: () => false,
+  },
+  pub: {
+    name: 'pub.dev',
+    pattern: /^(?:pub:|dart:|flutter:)(.+)$/,
+    url: (pkg) => `https://pub.dev/api/packages/${encodeURIComponent(pkg)}`,
+    parse: (data) => data.latest.version,
+    detect: () => false,
+  },
+  maven: {
+    name: 'Maven Central',
+    pattern: /^(?:maven:|java:)(.+)$/,
+    url: (pkg) => {
+      // Maven packages are in format group:artifact or group/artifact
+      const [group, artifact] = pkg.includes(':') ? pkg.split(':') : pkg.split('/');
+      if (!artifact) return null;
+      const groupPath = group.replace(/\./g, '/');
+      return `https://search.maven.org/solrsearch/select?q=g:${encodeURIComponent(group)}+AND+a:${encodeURIComponent(artifact)}&rows=1&wt=json`;
+    },
+    parse: (data) => {
+      if (data.response.docs.length === 0) return null;
+      return data.response.docs[0].latestVersion;
+    },
+    detect: (pkg) => pkg.includes(':') || (pkg.includes('.') && pkg.includes('/')),
+  },
+  homebrew: {
+    name: 'Homebrew',
+    pattern: /^(?:brew:|homebrew:)(.+)$/,
+    url: (pkg) => `https://formulae.brew.sh/api/formula/${encodeURIComponent(pkg)}.json`,
+    parse: (data) => data.versions.stable,
+    detect: () => false,
+  },
+  conda: {
+    name: 'Conda',
+    pattern: /^(?:conda:)(.+)$/,
+    url: (pkg) =>
+      `https://api.anaconda.org/package/conda-forge/${encodeURIComponent(pkg)}`,
+    parse: (data) => data.latest_version,
+    detect: () => false,
+  },
+};
+
+/**
+ * Detect the registry for a package based on its name or prefix
+ */
+function detectRegistry(packageName) {
+  // Check for explicit prefixes first
+  for (const [key, registry] of Object.entries(REGISTRIES)) {
+    if (registry.pattern.test(packageName) && key !== 'npm') {
+      const match = packageName.match(registry.pattern);
+      if (match) {
+        return { registry, cleanName: match[1] };
+      }
+    }
+  }
+
+  // Try to detect based on package name patterns
+  for (const registry of Object.values(REGISTRIES)) {
+    if (registry.detect(packageName)) {
+      const match = packageName.match(registry.pattern);
+      return { registry, cleanName: match ? match[1] : packageName };
+    }
+  }
+
+  // Default to npm
+  const npmMatch = packageName.match(REGISTRIES.npm.pattern);
+  return { registry: REGISTRIES.npm, cleanName: npmMatch ? npmMatch[1] : packageName };
+}
+
+/**
+ * Fetch the latest version of a package from its registry
+ */
+async function fetchPackageVersion(packageName, timeoutMs = 10000) {
+  const { registry, cleanName } = detectRegistry(packageName);
+
+  const url = registry.url(cleanName);
+  if (!url) {
+    return {
+      success: false,
+      error: `Invalid package format for ${registry.name}: ${cleanName}`,
+      registry: registry.name,
+    };
+  }
+
+  const controller = new AbortController();
+  const timeout = setTimeout(() => controller.abort(), timeoutMs);
+
+  try {
+    const headers = {
+      Accept: 'application/json',
+      ...(registry.headers || {}),
+    };
+
+    const response = await fetch(url, {
+      headers,
+      signal: controller.signal,
+    });
+
+    if (!response.ok) {
+      if (response.status === 404) {
+        return {
+          success: false,
+          error: `Package "${cleanName}" not found on ${registry.name}`,
+          registry: registry.name,
+        };
+      }
+      return {
+        success: false,
+        error: `${registry.name} returned status ${response.status}`,
+        registry: registry.name,
+      };
+    }
+
+    const data = await response.json();
+    const version = registry.parse(data);
+
+    if (!version) {
+      return {
+        success: false,
+        error: `Could not parse version from ${registry.name} response`,
+        registry: registry.name,
+      };
+    }
+
+    return {
+      success: true,
+      package: cleanName,
+      version,
+      registry: registry.name,
+    };
+  } catch (error) {
+    if (error.name === 'AbortError') {
+      return {
+        success: false,
+        error: `Request to ${registry.name} timed out`,
+        registry: registry.name,
+      };
+    }
+    return {
+      success: false,
+      error: `Failed to fetch from ${registry.name}: ${error.message}`,
+      registry: registry.name,
+    };
+  } finally {
+    clearTimeout(timeout);
+  }
+}
+
+/**
+ * Get supported registries list for help text
+ */
+function getSupportedRegistries() {
+  return Object.entries(REGISTRIES).map(([key, reg]) => ({
+    key,
+    name: reg.name,
+    prefix: reg.pattern.source.match(/\?:([^)]+)\)/)?.[1] || key + ':',
+  }));
+}
+
+// MCP Tool definition
+export function getToolDefinition() {
+  return {
+    name: 'e_check_package_version',
+    description:
+      'Fetches the latest version of a package from its official registry. Supports npm, PyPI, crates.io, Maven, Go, RubyGems, NuGet, Packagist, Hex, pub.dev, Homebrew, and Conda. Use prefix like "pip:requests" for non-npm packages.',
+    inputSchema: {
+      type: 'object',
+      properties: {
+        package: {
+          type: 'string',
+          description:
+            'Package name, optionally prefixed with registry (e.g., "lodash", "pip:requests", "cargo:serde", "go:github.com/gin-gonic/gin")',
+        },
+      },
+      required: ['package'],
+    },
+    annotations: {
+      title: 'Check Package Version',
+      readOnlyHint: true,
+      destructiveHint: false,
+      idempotentHint: true,
+      openWorldHint: true, // Makes external network requests
+    },
+  };
+}
+
+// Tool handler
+export async function handleToolCall(request) {
+  const packageName = request.params.arguments.package;
+
+  if (!packageName || typeof packageName !== 'string' || packageName.trim() === '') {
+    return {
+      content: [
+        {
+          type: 'text',
+          text: 'Error: Please provide a package name.',
+        },
+      ],
+    };
+  }
+
+  const result = await fetchPackageVersion(packageName.trim());
+
+  if (result.success) {
+    return {
+      content: [
+        {
+          type: 'text',
+          text: `**${result.package}** (${result.registry})\n\nLatest version: \`${result.version}\``,
+        },
+      ],
+    };
+  } else {
+    return {
+      content: [
+        {
+          type: 'text',
+          text: `Error: ${result.error}`,
+        },
+      ],
+    };
+  }
+}
+
+// Export for testing
+export { fetchPackageVersion, detectRegistry, getSupportedRegistries, REGISTRIES };