sigmap 8.18.0 → 8.19.0

package/CHANGELOG.md

@@ -10,6 +10,19 @@ Format: [Semantic Versioning](https://semver.org/)
 
 ---
 
+## [8.19.0] — 2026-07-19
+
+Minor release — **"Honest Numbers" (v8.19, P0)**: the published retrieval lift now comes from a measured grep-agent comparison, not a random-file baseline, and every proxy metric says so on the label.
+
+### Added
+- **Honest grep-agent baseline benchmark (#495, PR #496):** new `scripts/run-honest-benchmark.mjs` (`npm run benchmark:honest`) scores the production ranker against an internal single-shot grep-agent baseline — a pure-Node, zero-dependency, no-child-process repo scan ranked by distinct-term coverage then occurrences, `.gitignore`-aware, deterministic ordering — on the same 110-task / 19-repo corpus and the same scorer. **Measured: SigMap 86.4% hit@5 / MRR .780 vs grep 42.7% / .228 → 2.02× lift (+43.6pt)** → `benchmarks/reports/honest-baseline.json`. `computeLatest` derives `grep_baseline_hit_at_5` + `grep_lift` from the report (optional-report pattern; hermetic fixture unaffected).
+- **Claim-hygiene guard test:** `test/integration/honest-baseline.test.js` (7 checks) — report shape, derived-not-hand-typed metrics, version.json mirror, and a one-way door: the 6.4×-vs-random lift, the 13.6% random baseline, and the unsourced "10% without" claim can never reappear on README or llms surfaces; the benchmark script itself is guarded to stay child-process-free.
+
+### Changed
+- **Random-baseline lift retired from all human surfaces (#495, PR #496):** README and llms.txt/llms-full.txt now quote the measured grep-agent lift (2.02×) instead of 6.4×-vs-random; task success is labeled a **retrieval-tier proxy** ("modeled from retrieval tiers, not measured LLM sessions") and the "baseline 10%" / "up from 10%" claims are removed. The random-baseline fields (`baseline_hit_at_5`, `retrieval_lift`) remain in latest.json/version.json as data only. `readme-structure` guards advanced: README must show the grep baseline and must **not** show the random one.
+
+---
+
 ## [8.18.0] — 2026-07-12
 
 Minor release — **the §7.4 Phase-2 closer**: with these three changes, every quality-ceiling row in the master plan's scorecard is done or measure-gated-closed.

package/README.md

@@ -59,10 +59,10 @@ That map is exactly what agentic grep is worst at: reproducible, auditable conte
 
 **Proof it pays off** (full benchmark below):
 <!--SM:whyMetrics-->
-- **86.7% hit@5** — right file found in top 5 results (vs 13.6% baseline)
+- **86.7% hit@5** — right file in top 5 results (vs 42.7% single-shot grep baseline — 2.02× lift)
 - **96.9% token reduction** — average across 21 real repos
-- **68.9% task success rate** — up from 10% without context
-- **1.44 prompts per task** — down from 2.84 (49.2% fewer retries)
+- **68.9% task-success proxy** — modeled from retrieval tiers, not measured LLM sessions
+- **1.44 prompts per task** — down from 2.84 (49.2% fewer retries, modeled)
 <!--/SM:whyMetrics-->
 - **<!--SM:languages-->33<!--/SM:languages--> languages supported** — TypeScript, Python, Go, Rust, Java, R, and more
 - **No vendor lock-in** — works with any AI assistant or local LLM
@@ -122,13 +122,13 @@ Ask → Rank → Context → Validate → Judge → Learn
 
 <!--SM:benchmarkBlock-->
 ```
-Benchmark : sigmap-v8.18-main (21 repositories, including R language)
-Date      : 2026-07-12
+Benchmark : sigmap-v8.19-main (21 repositories, including R language)
+Date      : 2026-07-19
 
-Hit@5          : 86.7%   (baseline 13.6%  — 6.4× lift)
+Hit@5          : 86.7%   (grep-agent baseline 42.7%  — 2.02× lift)
 Token reduction: 96.9%   (across 21 repos)
-Prompt reduction : 49.2% (2.84 → 1.44 prompts per task)
-Task success   : 68.9%   (baseline 10%)
+Prompt reduction : 49.2% (2.84 → 1.44 prompts per task, modeled)
+Task success   : 68.9%   (proxy — modeled from retrieval tiers)
 Repos tested   : 21 (JavaScript, Python, Go, Rust, Java, R, C++, C#, Dart, Swift, Ruby, PHP, Scala, Kotlin, and more)
 ```
 <!--/SM:benchmarkBlock-->

package/gen-context.js

@@ -14607,7 +14607,7 @@ __factories["./src/mcp/server"] = function(module, exports) {
 
   const SERVER_INFO = {
     name: 'sigmap',
-    version: '8.18.0',
+    version: '8.19.0',
     description: 'SigMap MCP server — code signatures on demand',
   };
 
@@ -19710,7 +19710,7 @@ function __tryGit(args, opts = {}) {
   catch (_) { return ''; }
 }
 
-const VERSION = '8.18.0';
+const VERSION = '8.19.0';
 const MARKER = '\n\n## Auto-generated signatures\n<!-- Updated by gen-context.js -->\n';
 
 function requireSourceOrBundled(key) {

package/llms-full.txt

@@ -11,19 +11,19 @@ ranking keeps the relevant context in scope (cutting tokens ~97% as a side
 effect), with no LLM calls, embeddings, or vector database. Works with Claude,
 Cursor, GitHub Copilot, Aider, Windsurf, local LLMs, and MCP.
 
-# Version: 8.18.0 | Benchmark: sigmap-v8.18-main (2026-07-12)
+# Version: 8.19.0 | Benchmark: sigmap-v8.19-main (2026-07-19)
 # Source: auto-generated from package.json, version.json, benchmarks/latest.json, src/mcp/tools.js, src/config/defaults.js
 # Regenerate: npm run generate:llms   |   Validate: npm run validate:llms
 
 ---
 
-## Core metrics (benchmark: sigmap-v8.18-main, 2026-07-12)
+## Core metrics (benchmark: sigmap-v8.19-main, 2026-07-19)
 
 | Metric | Without SigMap | With SigMap |
 |--------|----------------|-------------|
-| Retrieval hit@5 | 13.6% (random) | 86.7% (6.4× lift) |
+| Retrieval hit@5 | 42.7% (single-shot grep) | 86.7% (2.02× lift) |
 | Token reduction | — | 96.9% average |
-| Task success proxy | 10% | 68.9% |
+| Task-success proxy (modeled) | — | 68.9% |
 | Prompts per task | 2.84 | 1.44 (49.2% fewer) |
 | Supported languages | — | 33 |
 | MCP tools | — | 20 |

package/llms.txt

@@ -11,7 +11,7 @@ ranking keeps the relevant context in scope (cutting tokens ~97% as a side
 effect), with no LLM calls, embeddings, or vector database. Works with Claude,
 Cursor, GitHub Copilot, Aider, Windsurf, local LLMs, and MCP.
 
-# Version: 8.18.0 | Benchmark: sigmap-v8.18-main (2026-07-12)
+# Version: 8.19.0 | Benchmark: sigmap-v8.19-main (2026-07-19)
 # Source: auto-generated from package.json, version.json, benchmarks/latest.json, src/mcp/tools.js, src/config/defaults.js
 # Regenerate: npm run generate:llms   |   Validate: npm run validate:llms
 
@@ -23,12 +23,12 @@ Cursor, GitHub Copilot, Aider, Windsurf, local LLMs, and MCP.
 - No blast-radius awareness before editing a hub file — `--impact` shows every file a change touches.
 - Pasted stack traces, CI logs, and JSON bloat the prompt — `squeeze` minimizes them and enriches the top frame from the symbol index.
 
-## Core metrics (benchmark: sigmap-v8.18-main, 2026-07-12)
+## Core metrics (benchmark: sigmap-v8.19-main, 2026-07-19)
 
-- hit@5 retrieval: 86.7% vs 13.6% random baseline (6.4× lift)
+- hit@5 retrieval: 86.7% vs 42.7% single-shot grep baseline (2.02× lift)
 - Token reduction: 96.9% average across benchmark repos
-- Task success: 68.9% vs 10% without SigMap
-- Prompts per task: 1.44 vs 2.84 baseline (49.2% fewer)
+- Task-success proxy: 68.9% (modeled from retrieval tiers, not measured LLM sessions)
+- Prompts per task: 1.44 vs 2.84 baseline (49.2% fewer, modeled)
 - Languages: 33 supported · MCP tools: 20
 - Dependencies: zero npm runtime dependencies · fully offline
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "sigmap",
-  "version": "8.18.0",
+  "version": "8.19.0",
   "description": "The deterministic, verifiable grounding layer for AI code work — a zero-dependency signature-and-evidence map that grounds Claude, Cursor, Copilot, Aider, Windsurf, local LLMs & MCP agents against your real code (repo + installed libraries) so they stop hallucinating files, imports & APIs. Runs offline via npx; byte-stable output; ~97% token reduction as proof.",
   "main": "packages/core/index.js",
   "exports": {
@@ -50,6 +50,7 @@
     "check:metrics": "node scripts/gen-benchmark-latest.mjs --check && node scripts/check-version-meta.mjs && node scripts/sync-metrics.mjs --check",
     "prepublishOnly": "node scripts/check-bundle.mjs && node scripts/build-bundle.mjs --check && node scripts/gen-benchmark-latest.mjs --check && node scripts/check-version-meta.mjs && node scripts/sync-metrics.mjs --check && node scripts/generate-llms.mjs",
     "benchmark:grounding": "node scripts/run-hallucination-benchmark.mjs",
+    "benchmark:honest": "node scripts/run-honest-benchmark.mjs --save",
     "benchmark:llm-ablation": "node scripts/run-llm-ablation.mjs"
   },
   "files": [

package/packages/cli/package.json

@@ -1,6 +1,6 @@
 {
   "name": "sigmap-cli",
-  "version": "8.18.0",
+  "version": "8.19.0",
   "description": "SigMap CLI wrapper — thin adapter for programmatic CLI invocation",
   "main": "index.js",
   "keywords": [

package/packages/core/package.json

@@ -1,6 +1,6 @@
 {
   "name": "sigmap-core",
-  "version": "8.18.0",
+  "version": "8.19.0",
   "description": "SigMap core library — zero-dependency code signature extraction, retrieval, and security scanning",
   "main": "index.js",
   "keywords": [

package/src/mcp/server.js

@@ -18,7 +18,7 @@ const { readContext, searchSignatures, getMap, createCheckpoint, getRouting, exp
 
 const SERVER_INFO = {
   name: 'sigmap',
-  version: '8.18.0',
+  version: '8.19.0',
   description: 'SigMap MCP server — code signatures on demand',
 };