sigmap 8.21.0 → 8.22.0

package/CHANGELOG.md

@@ -10,6 +10,19 @@ Format: [Semantic Versioning](https://semver.org/)
 
 ---
 
+## [8.22.0] — 2026-07-28
+
+Minor release — **"Hard Corpus" (v8.22)**: the benchmark corpus gains a no-leakage hard split with a deterministic leakage gate, and per-repo-size buckets stop tiny repos from flattering the average. The headline retrieval number gets harder — and honest.
+
+### Added
+- **Hard-split corpus + leakage gate + size buckets (#505, PR #506):** new `src/eval/corpus.js` — a task "leaks" when its BM25-tokenized query shares a stemmed token with the tokenized basenames of its expected files; `validateTasks` flags leaky `split: "hard"` tasks as violations, and `sizeBucket` groups repos at 200/1000 scanned files (tertiles of the 43-repo corpus). New CI gate `scripts/validate-task-corpus.mjs` (exit 1 on hard-split leakage). `loadTasks` carries the optional `split` field (default `easy`). `benchmark:honest` now reports hit@5/MRR per split and per size bucket — buckets use files scanned on disk, not the budget-capped context index. 15 hand-authored hard tasks (express, flask, axios, fastify, gin), all leak-free. 8 new integration tests (128 test files).
+- **MiniMax LLM-ablation provider (PR #504)** — thanks @octo-patch: `MINIMAX_API_KEY` support in `scripts/run-llm-ablation.mjs` (OpenAI-compatible endpoint, default model MiniMax-M3, `MINIMAX_BASE_URL` override) plus a pricing entry and tests.
+
+### Changed
+- **Headline honesty, again:** the leakage gate measured that **90 of 110 pre-existing easy tasks leak filename tokens**, and the new hard split scores **33.3% hit@5 vs the grep baseline's 53.3%** — with leakage removed, grep currently wins; that measured vocabulary-mismatch ceiling is what B2 (repo-mined expansion, v9.0) exists to attack. Overall corpus (125 tasks): 72.8% hit@5, honest lift 1.63× (+28pt).
+
+---
+
 ## [8.21.0] — 2026-07-19
 
 Minor release — **"Semantic Bridge II" (v8.21)**: doc-comment hints reach Go, Rust, and Java, and the import graph gains a principled centrality prior for ranking — flag-gated and measured.

package/README.md

@@ -59,10 +59,10 @@ That map is exactly what agentic grep is worst at: reproducible, auditable conte
 
 **Proof it pays off** (full benchmark below):
 <!--SM:whyMetrics-->
-- **85.6% hit@5** — right file in top 5 results (vs 42.7% single-shot grep baseline — 2.00× lift)
+- **82.2% hit@5** — right file in top 5 results (vs 44.8% single-shot grep baseline — 1.59× lift)
 - **96.8% token reduction** — average across 21 real repos
-- **66.7% task-success proxy** — modeled from retrieval tiers, not measured LLM sessions
-- **1.48 prompts per task** — down from 2.84 (48.0% fewer retries, modeled)
+- **64.8% task-success proxy** — modeled from retrieval tiers, not measured LLM sessions
+- **1.53 prompts per task** — down from 2.84 (46.1% fewer retries, modeled)
 <!--/SM:whyMetrics-->
 - **<!--SM:languages-->33<!--/SM:languages--> languages supported** — TypeScript, Python, Go, Rust, Java, R, and more
 - **No vendor lock-in** — works with any AI assistant or local LLM
@@ -98,7 +98,7 @@ sigmap verify answer.md --report        # standalone red/amber/green HTML report
 | Without SigMap | With SigMap |
 |---|---|
 | ❌ Non-reproducible agent guesses | ✅ Deterministic map — same input, same output, every time |
-| ❌ "Trust me" AI answers | ✅ Grounded — right file in context <!--SM:hitWhole-->86%<!--/SM:hitWhole--> of the time, every symbol on a real line anchor |
+| ❌ "Trust me" AI answers | ✅ Grounded — right file in context <!--SM:hitWhole-->82%<!--/SM:hitWhole--> of the time, every symbol on a real line anchor |
 | ❌ Embeddings / vector DB required | ✅ Zero deps, no infra, fully offline |
 
 ---
@@ -122,13 +122,13 @@ Ask → Rank → Context → Validate → Judge → Learn
 
 <!--SM:benchmarkBlock-->
 ```
-Benchmark : sigmap-v8.21-main (21 repositories, including R language)
-Date      : 2026-07-19
+Benchmark : sigmap-v8.22-main (21 repositories, including R language)
+Date      : 2026-07-27
 
-Hit@5          : 85.6%   (grep-agent baseline 42.7%  — 2.00× lift)
+Hit@5          : 82.2%   (grep-agent baseline 44.8%  — 1.59× lift)
 Token reduction: 96.8%   (across 21 repos)
-Prompt reduction : 48.0% (2.84 → 1.48 prompts per task, modeled)
-Task success   : 66.7%   (proxy — modeled from retrieval tiers)
+Prompt reduction : 46.1% (2.84 → 1.53 prompts per task, modeled)
+Task success   : 64.8%   (proxy — modeled from retrieval tiers)
 Repos tested   : 21 (JavaScript, Python, Go, Rust, Java, R, C++, C#, Dart, Swift, Ruby, PHP, Scala, Kotlin, and more)
 ```
 <!--/SM:benchmarkBlock-->

package/gen-context.js

@@ -4101,6 +4101,93 @@ __factories["./src/eval/analyzer"] = function(module, exports) {
   
 };
 
+// ── ./src/eval/corpus ──
+__factories["./src/eval/corpus"] = function(module, exports) {
+  
+  /**
+   * Task-corpus hygiene (A3, v8.22 "Hard Corpus").
+   *
+   * A benchmark query "leaks" when it shares a token with the basenames of its
+   * expected files — hit@5 then partly measures filename matching, not
+   * retrieval. The criterion is deterministic and reuses the production
+   * tokenizer (identifier splitting + stemming from src/retrieval/bm25.js), so
+   * "payments" leaks against payment.js and "InterceptorManager" leaks against
+   * "interceptor manager" the same way the ranker would see them.
+   *
+   * Tasks carry an optional `split` field: 'hard' tasks MUST be leak-free
+   * (validateTasks reports them as violations); 'easy' tasks (the default) may
+   * leak — that is what makes them easy.
+   *
+   * Size buckets group repos by indexed file count so large repos stop being
+   * averaged away by tiny ones. Thresholds are the rough tertiles of the
+   * current benchmarks/repos corpus (43 repos, 27–3450 source files).
+   */
+
+  const { tokenize } = __require('./src/retrieval/bm25');
+
+  const BUCKET_LIMITS = { small: 200, medium: 1000 }; // files; large = above medium
+
+  /**
+   * Stemmed tokens of a file path's basename (extension stripped).
+   * @param {string} filePath
+   * @returns {string[]}
+   */
+  function basenameTokens(filePath) {
+    const base = String(filePath).split('/').pop() || '';
+    return tokenize(base.replace(/\.[^.]*$/, ''));
+  }
+
+  /**
+   * Leaked tokens between a query and its expected files' basenames.
+   * @param {string} query
+   * @param {string[]} expectedFiles
+   * @returns {{ leaked: string[], clean: boolean }}
+   */
+  function queryLeakage(query, expectedFiles) {
+    const qToks = new Set(tokenize(query));
+    const leaked = new Set();
+    for (const f of expectedFiles || []) {
+      for (const t of basenameTokens(f)) {
+        if (qToks.has(t)) leaked.add(t);
+      }
+    }
+    return { leaked: [...leaked].sort(), clean: leaked.size === 0 };
+  }
+
+  /**
+   * Validate a task list: every task gets a leakage result; hard-split tasks
+   * that leak are violations.
+   * @param {Array<{id?:string, query:string, expected_files?:string[], split?:string}>} tasks
+   * @returns {{ results: object[], hardViolations: object[] }}
+   */
+  function validateTasks(tasks) {
+    const results = [];
+    const hardViolations = [];
+    for (const t of tasks || []) {
+      const split = t.split === 'hard' ? 'hard' : 'easy';
+      const { leaked, clean } = queryLeakage(t.query, t.expected_files);
+      const row = { id: t.id || '?', split, leaked, clean };
+      results.push(row);
+      if (split === 'hard' && !clean) hardViolations.push(row);
+    }
+    return { results, hardViolations };
+  }
+
+  /**
+   * Size bucket for a repo by indexed file count.
+   * @param {number} fileCount
+   * @returns {'small'|'medium'|'large'}
+   */
+  function sizeBucket(fileCount) {
+    if (fileCount < BUCKET_LIMITS.small) return 'small';
+    if (fileCount <= BUCKET_LIMITS.medium) return 'medium';
+    return 'large';
+  }
+
+  module.exports = { basenameTokens, queryLeakage, validateTasks, sizeBucket, BUCKET_LIMITS };
+  
+};
+
 // ── ./src/eval/llm-ablation ──
 __factories["./src/eval/llm-ablation"] = function(module, exports) {
   
@@ -4400,7 +4487,7 @@ __factories["./src/eval/runner"] = function(module, exports) {
 
   /**
    * Load tasks from a JSONL file.
-   * Each line: { id, query, expected_files, repo }
+   * Each line: { id, query, expected_files, repo, split? ('easy'|'hard') }
    * Invalid or blank lines are silently skipped.
    * @param {string} tasksFile - absolute or relative path
    * @returns {Array<{id:string, query:string, expected:string[], repo:string}>}
@@ -4420,6 +4507,7 @@ __factories["./src/eval/runner"] = function(module, exports) {
             query: obj.query,
             expected: obj.expected_files,
             repo: obj.repo || '.',
+            split: obj.split === 'hard' ? 'hard' : 'easy',
           });
         }
       } catch {
@@ -14850,7 +14938,7 @@ __factories["./src/mcp/server"] = function(module, exports) {
 
   const SERVER_INFO = {
     name: 'sigmap',
-    version: '8.21.0',
+    version: '8.22.0',
     description: 'SigMap MCP server — code signatures on demand',
   };
 
@@ -18544,6 +18632,8 @@ __factories["./src/tracking/pricing"] = function(module, exports) {
     'gpt-4o-mini': 0.15,
     'gemini-1.5-pro': 1.25,
     'gemini-1.5-flash': 0.075,
+    'minimax-m3': 0.6,
+    'minimax-m2.7': 0.3,
   };
 
   const DEFAULT_MODEL = 'claude-sonnet';
@@ -20068,7 +20158,7 @@ function __tryGit(args, opts = {}) {
   catch (_) { return ''; }
 }
 
-const VERSION = '8.21.0';
+const VERSION = '8.22.0';
 const MARKER = '\n\n## Auto-generated signatures\n<!-- Updated by gen-context.js -->\n';
 
 function requireSourceOrBundled(key) {

package/llms-full.txt

@@ -11,20 +11,20 @@ ranking keeps the relevant context in scope (cutting tokens ~97% as a side
 effect), with no LLM calls, embeddings, or vector database. Works with Claude,
 Cursor, GitHub Copilot, Aider, Windsurf, local LLMs, and MCP.
 
-# Version: 8.21.0 | Benchmark: sigmap-v8.21-main (2026-07-19)
+# Version: 8.22.0 | Benchmark: sigmap-v8.22-main (2026-07-27)
 # Source: auto-generated from package.json, version.json, benchmarks/latest.json, src/mcp/tools.js, src/config/defaults.js
 # Regenerate: npm run generate:llms   |   Validate: npm run validate:llms
 
 ---
 
-## Core metrics (benchmark: sigmap-v8.21-main, 2026-07-19)
+## Core metrics (benchmark: sigmap-v8.22-main, 2026-07-27)
 
 | Metric | Without SigMap | With SigMap |
 |--------|----------------|-------------|
-| Retrieval hit@5 | 42.7% (single-shot grep) | 85.6% (2.00× lift) |
+| Retrieval hit@5 | 44.8% (single-shot grep) | 82.2% (1.59× lift) |
 | Token reduction | — | 96.8% average |
-| Task-success proxy (modeled) | — | 66.7% |
-| Prompts per task | 2.84 | 1.48 (48.0% fewer) |
+| Task-success proxy (modeled) | — | 64.8% |
+| Prompts per task | 2.84 | 1.53 (46.1% fewer) |
 | Supported languages | — | 33 |
 | MCP tools | — | 20 |
 | npm runtime dependencies | — | 0 |

package/llms.txt

@@ -11,7 +11,7 @@ ranking keeps the relevant context in scope (cutting tokens ~97% as a side
 effect), with no LLM calls, embeddings, or vector database. Works with Claude,
 Cursor, GitHub Copilot, Aider, Windsurf, local LLMs, and MCP.
 
-# Version: 8.21.0 | Benchmark: sigmap-v8.21-main (2026-07-19)
+# Version: 8.22.0 | Benchmark: sigmap-v8.22-main (2026-07-27)
 # Source: auto-generated from package.json, version.json, benchmarks/latest.json, src/mcp/tools.js, src/config/defaults.js
 # Regenerate: npm run generate:llms   |   Validate: npm run validate:llms
 
@@ -23,12 +23,12 @@ Cursor, GitHub Copilot, Aider, Windsurf, local LLMs, and MCP.
 - No blast-radius awareness before editing a hub file — `--impact` shows every file a change touches.
 - Pasted stack traces, CI logs, and JSON bloat the prompt — `squeeze` minimizes them and enriches the top frame from the symbol index.
 
-## Core metrics (benchmark: sigmap-v8.21-main, 2026-07-19)
+## Core metrics (benchmark: sigmap-v8.22-main, 2026-07-27)
 
-- hit@5 retrieval: 85.6% vs 42.7% single-shot grep baseline (2.00× lift)
+- hit@5 retrieval: 82.2% vs 44.8% single-shot grep baseline (1.59× lift)
 - Token reduction: 96.8% average across benchmark repos
-- Task-success proxy: 66.7% (modeled from retrieval tiers, not measured LLM sessions)
-- Prompts per task: 1.48 vs 2.84 baseline (48.0% fewer, modeled)
+- Task-success proxy: 64.8% (modeled from retrieval tiers, not measured LLM sessions)
+- Prompts per task: 1.53 vs 2.84 baseline (46.1% fewer, modeled)
 - Languages: 33 supported · MCP tools: 20
 - Dependencies: zero npm runtime dependencies · fully offline
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "sigmap",
-  "version": "8.21.0",
+  "version": "8.22.0",
   "description": "The deterministic, verifiable grounding layer for AI code work — a zero-dependency signature-and-evidence map that grounds Claude, Cursor, Copilot, Aider, Windsurf, local LLMs & MCP agents against your real code (repo + installed libraries) so they stop hallucinating files, imports & APIs. Runs offline via npx; byte-stable output; ~97% token reduction as proof.",
   "main": "packages/core/index.js",
   "exports": {

package/packages/cli/package.json

@@ -1,6 +1,6 @@
 {
   "name": "sigmap-cli",
-  "version": "8.21.0",
+  "version": "8.22.0",
   "description": "SigMap CLI wrapper — thin adapter for programmatic CLI invocation",
   "main": "index.js",
   "keywords": [

package/packages/core/package.json

@@ -1,6 +1,6 @@
 {
   "name": "sigmap-core",
-  "version": "8.21.0",
+  "version": "8.22.0",
   "description": "SigMap core library — zero-dependency code signature extraction, retrieval, and security scanning",
   "main": "index.js",
   "keywords": [

package/src/eval/corpus.js

@@ -0,0 +1,83 @@
+'use strict';
+
+/**
+ * Task-corpus hygiene (A3, v8.22 "Hard Corpus").
+ *
+ * A benchmark query "leaks" when it shares a token with the basenames of its
+ * expected files — hit@5 then partly measures filename matching, not
+ * retrieval. The criterion is deterministic and reuses the production
+ * tokenizer (identifier splitting + stemming from src/retrieval/bm25.js), so
+ * "payments" leaks against payment.js and "InterceptorManager" leaks against
+ * "interceptor manager" the same way the ranker would see them.
+ *
+ * Tasks carry an optional `split` field: 'hard' tasks MUST be leak-free
+ * (validateTasks reports them as violations); 'easy' tasks (the default) may
+ * leak — that is what makes them easy.
+ *
+ * Size buckets group repos by indexed file count so large repos stop being
+ * averaged away by tiny ones. Thresholds are the rough tertiles of the
+ * current benchmarks/repos corpus (43 repos, 27–3450 source files).
+ */
+
+const { tokenize } = require('../retrieval/bm25');
+
+const BUCKET_LIMITS = { small: 200, medium: 1000 }; // files; large = above medium
+
+/**
+ * Stemmed tokens of a file path's basename (extension stripped).
+ * @param {string} filePath
+ * @returns {string[]}
+ */
+function basenameTokens(filePath) {
+  const base = String(filePath).split('/').pop() || '';
+  return tokenize(base.replace(/\.[^.]*$/, ''));
+}
+
+/**
+ * Leaked tokens between a query and its expected files' basenames.
+ * @param {string} query
+ * @param {string[]} expectedFiles
+ * @returns {{ leaked: string[], clean: boolean }}
+ */
+function queryLeakage(query, expectedFiles) {
+  const qToks = new Set(tokenize(query));
+  const leaked = new Set();
+  for (const f of expectedFiles || []) {
+    for (const t of basenameTokens(f)) {
+      if (qToks.has(t)) leaked.add(t);
+    }
+  }
+  return { leaked: [...leaked].sort(), clean: leaked.size === 0 };
+}
+
+/**
+ * Validate a task list: every task gets a leakage result; hard-split tasks
+ * that leak are violations.
+ * @param {Array<{id?:string, query:string, expected_files?:string[], split?:string}>} tasks
+ * @returns {{ results: object[], hardViolations: object[] }}
+ */
+function validateTasks(tasks) {
+  const results = [];
+  const hardViolations = [];
+  for (const t of tasks || []) {
+    const split = t.split === 'hard' ? 'hard' : 'easy';
+    const { leaked, clean } = queryLeakage(t.query, t.expected_files);
+    const row = { id: t.id || '?', split, leaked, clean };
+    results.push(row);
+    if (split === 'hard' && !clean) hardViolations.push(row);
+  }
+  return { results, hardViolations };
+}
+
+/**
+ * Size bucket for a repo by indexed file count.
+ * @param {number} fileCount
+ * @returns {'small'|'medium'|'large'}
+ */
+function sizeBucket(fileCount) {
+  if (fileCount < BUCKET_LIMITS.small) return 'small';
+  if (fileCount <= BUCKET_LIMITS.medium) return 'medium';
+  return 'large';
+}
+
+module.exports = { basenameTokens, queryLeakage, validateTasks, sizeBucket, BUCKET_LIMITS };

package/src/eval/runner.js

@@ -124,7 +124,7 @@ function estimateTokens(sigs) {
 
 /**
  * Load tasks from a JSONL file.
- * Each line: { id, query, expected_files, repo }
+ * Each line: { id, query, expected_files, repo, split? ('easy'|'hard') }
  * Invalid or blank lines are silently skipped.
  * @param {string} tasksFile - absolute or relative path
  * @returns {Array<{id:string, query:string, expected:string[], repo:string}>}
@@ -144,6 +144,7 @@ function loadTasks(tasksFile) {
           query: obj.query,
           expected: obj.expected_files,
           repo: obj.repo || '.',
+          split: obj.split === 'hard' ? 'hard' : 'easy',
         });
       }
     } catch {

package/src/mcp/server.js

@@ -18,7 +18,7 @@ const { readContext, searchSignatures, getMap, createCheckpoint, getRouting, exp
 
 const SERVER_INFO = {
   name: 'sigmap',
-  version: '8.21.0',
+  version: '8.22.0',
   description: 'SigMap MCP server — code signatures on demand',
 };
 

package/src/tracking/pricing.js

@@ -20,6 +20,8 @@ const PRICES = {
   'gpt-4o-mini': 0.15,
   'gemini-1.5-pro': 1.25,
   'gemini-1.5-flash': 0.075,
+  'minimax-m3': 0.6,
+  'minimax-m2.7': 0.3,
 };
 
 const DEFAULT_MODEL = 'claude-sonnet';