@remnic/bench 9.3.727 → 9.3.729

package/README.md

@@ -211,7 +211,8 @@ A dedicated benchmark suite for [`@remnic/coding-graph`](https://www.npmjs.com/p
 |---|---|
 | `fullIndexMs` | Wall time to index the entire fixture in one batch. |
 | `fullIndexLocsPerSecond` | Sustained LOC/s during full index (higher is better). |
-| `incrementalUpdateP50Ms` / `incrementalUpdateP95Ms` | Single-file re-ingest latency (p50/p95 over ≥20 iterations). |
+| `incrementalUpdateP50Ms` / `incrementalUpdateP95Ms` | Single-file re-ingest latency for UNCHANGED content (the common-case no-op path; p50/p95 over ≥20 iterations). |
+| `incrementalModifiedUpdateP50Ms` / `incrementalModifiedUpdateP95Ms` | Single-file re-ingest latency for MODIFIED content — the change-heavy path (edge deletion/creation, symbol re-resolution). Complementary to `incrementalUpdate` (#1688). |
 | `tracePathP95Ms` | `trace_path` (BFS, depth ≤ 5) p95. |
 | `searchGraphP95Ms` | `search_graph` name-pattern p95. |
 | `deadCodeMs` | Dead-code query wall time. |
@@ -225,18 +226,21 @@ The harness ships a **deterministic** synthetic repo generator (`generateSynthet
 
 The measured numbers live in [`baselines/coding-graph-baseline.json`](./baselines/coding-graph-baseline.json) — bench-owned, separate from the structural ratchets in `scripts/ratchet-baseline.json`. The regression gate (`checkCodingGraphRegression`) compares a report against the baseline with a generous tolerance (default 30%). It hard-fails on gross regression — a real failing step, not a warning (rule 50). Tightening the baseline is a deliberate PR act (mirrors `check-ratchets --update`).
 
+The gate carries a **machine-fingerprint guard** (`compareMachineFingerprints`): when the report's machine class differs from the baseline's (arch/platform/Node major/cpuModel/cores), the comparison is *skipped* (`passed: true`, `skipped: true`, `machineMismatch` populated) rather than failed, so a cross-machine CI run does not false-positive on legitimate hardware variance (#1688). Regenerate the baseline on the target machine class for a real comparison.
+
 ### Measured numbers (first baseline)
 
 > Numbers below are from the **baseline JSON file** — this section is checked against it so prose can't drift from measurement. Run `remnic bench coding-graph` to reproduce.
 
 | Metric | Value | Machine |
 |---|---|---|
-| Full index | ~17.6 ms, ~115k LOC/s | Apple M2 Max, Node v22 |
-| Incremental update p95 | ~0.26 ms | Apple M2 Max |
-| trace_path p95 | ~0.14 ms | Apple M2 Max |
-| search_graph p95 | ~0.29 ms | Apple M2 Max |
-| dead_code | ~0.67 ms | Apple M2 Max |
-| DB size | ~2 KB/KLOC | Apple M2 Max |
+| Full index | ~15 ms, ~131k LOC/s | Apple M2 Max, Node v22 |
+| Incremental update p95 (idempotent) | ~0.24 ms | Apple M2 Max |
+| Incremental modified update p95 (change-heavy) | ~0.96 ms | Apple M2 Max |
+| trace_path p95 | ~0.13 ms | Apple M2 Max |
+| search_graph p95 | ~0.18 ms | Apple M2 Max |
+| dead_code | ~0.53 ms | Apple M2 Max |
+| DB size | ~270 KB/KLOC | Apple M2 Max |
 
 These are **working targets on a small fixture (20 files, 200 symbols)**, NOT parity claims against codebase-memory-mcp's published numbers (28M LOC in 3 min, <1ms Cypher). Scale targets at 1M+ LOC are tracked as stretch goals — the harness will measure them when Tier-L fixtures are wired (issue #1557 PR2).
 
@@ -245,9 +249,14 @@ These are **working targets on a small fixture (20 files, 200 symbols)**, NOT pa
 ```typescript
 import { runCodingGraphBenchmark, checkCodingGraphRegression } from "@remnic/bench";
 
+// Use the SAME fixture as the bundled baseline (20×10, density 0.3, seed
+// 42) so the regression gate compares like-for-like. A custom fixture
+// (different fileCount/symbolsPerFile/callDensity/seed/language) needs its
+// OWN baseline — build one with buildBaselineFromReport(report, note) and
+// commit it, otherwise checkCodingGraphRegression reports a fixture mismatch.
 const report = await runCodingGraphBenchmark({
-  fixture: { fileCount: 1000, symbolsPerFile: 10, callDensity: 0.2 },
-  iterations: 25,
+  fixture: { seed: 42, fileCount: 20, symbolsPerFile: 10, callDensity: 0.3, language: "typescript" },
+  iterations: 20,
 });
 
 const baseline = require("@remnic/bench/baselines/coding-graph-baseline.json");
@@ -256,4 +265,7 @@ if (!gate.passed) {
   console.error(gate.summary);
   process.exit(1);
 }
+// gate.skipped === true when the report's machine fingerprint differs from
+// the baseline's (different CPU/arch/Node major) — the comparison is skipped
+// rather than failed so cross-machine CI does not false-positive (#1688).
 ```

package/baselines/coding-graph-baseline.json

@@ -1,5 +1,5 @@
 {
-  "schemaVersion": 1,
+  "schemaVersion": 2,
   "machine": {
     "arch": "arm64",
     "platform": "darwin",
@@ -16,15 +16,17 @@
     "language": "typescript"
   },
   "metrics": {
-    "fullIndexMs": 14.864042000000154,
-    "fullIndexLocsPerSecond": 136302,
-    "incrementalUpdateP50Ms": 0.19358299999998962,
-    "incrementalUpdateP95Ms": 0.26237500000002,
-    "tracePathP95Ms": 0.15750000000002728,
-    "searchGraphP95Ms": 0.19700000000011642,
-    "deadCodeMs": 2.29424999999992,
-    "dbBytesPerKloc": 280636
+    "fullIndexMs": 15.452540999999655,
+    "fullIndexLocsPerSecond": 131111,
+    "incrementalUpdateP50Ms": 0.173167000000376,
+    "incrementalUpdateP95Ms": 0.23908399999982066,
+    "incrementalModifiedUpdateP50Ms": 0.6047909999997501,
+    "incrementalModifiedUpdateP95Ms": 0.9565000000002328,
+    "tracePathP95Ms": 0.13425000000006548,
+    "searchGraphP95Ms": 0.1764170000001286,
+    "deadCodeMs": 0.5299169999998412,
+    "dbBytesPerKloc": 270468
   },
-  "createdAt": "2026-07-06T10:50:56.821Z",
-  "note": "Initial baseline — first harness run with WAL-aware DB sizing (issue #1557 PR1)."
+  "createdAt": "2026-07-07T12:35:48.738Z",
+  "note": "Regenerated baseline — adds incrementalModifiedUpdate metric (#1688 item 1) + machine-fingerprint guard (#1688 item 2). WAL-aware DB sizing retained (#1557 PR1)."
 }

package/dist/index.d.ts

@@ -3979,7 +3979,7 @@ interface WallMetric {
  * The regression gate compares only keys present in BOTH the report and
  * the baseline — new metrics are additive, never blocking.
  */
-type CodingGraphMetricKey = "fullIndexMs" | "fullIndexLocsPerSecond" | "incrementalUpdateP50Ms" | "incrementalUpdateP95Ms" | "tracePathP95Ms" | "searchGraphP95Ms" | "deadCodeMs" | "dbBytesPerKloc";
+type CodingGraphMetricKey = "fullIndexMs" | "fullIndexLocsPerSecond" | "incrementalUpdateP50Ms" | "incrementalUpdateP95Ms" | "incrementalModifiedUpdateP50Ms" | "incrementalModifiedUpdateP95Ms" | "tracePathP95Ms" | "searchGraphP95Ms" | "deadCodeMs" | "dbBytesPerKloc";
 interface CodingGraphBenchReport {
     /** Schema version — bump when the report shape changes. */
     readonly schemaVersion: number;
@@ -3996,8 +3996,14 @@ interface CodingGraphBenchReport {
     readonly fullIndexMs: WallMetric;
     /** LOC/s sustained during full index (higher is better). */
     readonly fullIndexLocsPerSecond: number;
-    /** Incremental single-file update latency distribution. */
+    /** Incremental single-file update latency distribution (idempotent
+     *  re-ingest of UNCHANGED content — the common-case no-op path). */
     readonly incrementalUpdate: MicroMetric;
+    /** Incremental MODIFIED-content re-ingest latency distribution. A
+     *  complementary metric to `incrementalUpdate`: measures the change-heavy
+     *  path (edge deletion/creation, symbol re-resolution) rather than the
+     *  cache-hit no-op (#1688 item 1). */
+    readonly incrementalModifiedUpdate: MicroMetric;
     /** trace_path (depth ≤ 5) latency distribution. */
     readonly tracePath: MicroMetric;
     /** search_graph name-pattern latency distribution. */
@@ -4046,6 +4052,24 @@ interface RegressionGateResult {
     readonly passed: boolean;
     readonly regressions: readonly RegressionMetricDetail[];
     readonly summary: string;
+    /**
+     * True when the gate SKIPPED the metric comparison because of a
+     * non-fatal precondition (machine-fingerprint mismatch). `passed` is
+     * also true in that case — a hardware variance must not fail CI (#1688
+     * item 2). Callers that want to treat a skip as informational can check
+     * this flag.
+     */
+    readonly skipped?: boolean;
+    /**
+     * Present when `skipped` is true due to a machine-fingerprint mismatch
+     * between the report and the baseline. Lists the fingerprint fields that
+     * differ so a human can decide whether the comparison is still meaningful.
+     */
+    readonly machineMismatch?: {
+        readonly report: MachineFingerprint;
+        readonly baseline: MachineFingerprint;
+        readonly differingFields: readonly string[];
+    };
 }
 interface CodingGraphBenchConfig {
     /** Fixture parameters. Defaults to a small, fast smoke fixture. */
@@ -4073,7 +4097,7 @@ declare const MIN_ITERATIONS = 20;
 /** Default regression tolerance (issue #1557: generous, e.g. 30%). */
 declare const DEFAULT_TOLERANCE_PERCENT = 30;
 /** Report schema version — bump when the shape changes. */
-declare const CODING_GRAPH_BENCH_SCHEMA_VERSION = 1;
+declare const CODING_GRAPH_BENCH_SCHEMA_VERSION = 2;
 
 /**
  * Deterministic synthetic repo generator (issue #1557 design (a)).
@@ -4169,6 +4193,8 @@ declare const METRIC_DIRECTION: {
     fullIndexLocsPerSecond: "higher-is-better";
     incrementalUpdateP95Ms: "lower-is-better";
     incrementalUpdateP50Ms: "lower-is-better";
+    incrementalModifiedUpdateP95Ms: "lower-is-better";
+    incrementalModifiedUpdateP50Ms: "lower-is-better";
     tracePathP95Ms: "lower-is-better";
     searchGraphP95Ms: "lower-is-better";
     deadCodeMs: "lower-is-better";

package/dist/index.js

@@ -36233,7 +36233,7 @@ var DEFAULT_10K_FIXTURE = {
 };
 var MIN_ITERATIONS = 20;
 var DEFAULT_TOLERANCE_PERCENT = 30;
-var CODING_GRAPH_BENCH_SCHEMA_VERSION = 1;
+var CODING_GRAPH_BENCH_SCHEMA_VERSION = 2;
 
 // src/coding-graph/harness.ts
 function captureMachineFingerprint() {
@@ -36378,6 +36378,40 @@ async function runCodingGraphBenchmark(config = {}) {
       }
       const kloc = Math.max(1, repo.approximateLoc / 1e3);
       const dbBytesPerKloc = dbBytes / kloc;
+      const modifiedSamples = [];
+      for (let i = 0; i < iterations; i++) {
+        const fileIdx = i % storeFiles.length;
+        const original = storeFiles[fileIdx];
+        if (!original) continue;
+        const ownSyms = new Set(original.symbols.map((sym) => sym.qualifiedName));
+        const churnName = `mod.benchChurnSymbol${i}`;
+        const representativeEdges = (original.edges ?? []).filter((e) => !ownSyms.has(e.dstQualifiedName)).slice(0, 2).map((e) => ({ ...e, srcQualifiedName: churnName }));
+        const modified = {
+          ...original,
+          contentHash: `${original.contentHash}-mod-${i}`,
+          symbols: [
+            {
+              qualifiedName: churnName,
+              name: `benchChurnSymbol${i}`,
+              kind: "function",
+              span: { startByte: 0, endByte: 0 }
+            }
+          ],
+          edges: representativeEdges
+        };
+        const modResult = await timeAsync(
+          () => store.upsertFileBatch([modified])
+        );
+        if (!modResult.result.ok) {
+          throw new Error(`modified update failed: ${modResult.result.code}`);
+        }
+        modifiedSamples.push(modResult.ms);
+        const restoreResult = await store.upsertFileBatch(storeFiles);
+        if (!restoreResult.ok) {
+          throw new Error(`restore failed: ${restoreResult.code}`);
+        }
+        sampleRss();
+      }
       sampleRss();
       const peakRssBytes = peakRss;
       return {
@@ -36394,6 +36428,9 @@ async function runCodingGraphBenchmark(config = {}) {
         fullIndexMs: { ms: fullIndex.ms },
         fullIndexLocsPerSecond: Math.round(locsPerSecond),
         incrementalUpdate: computeMicroMetric(incrementalSamples),
+        incrementalModifiedUpdate: computeMicroMetric(
+          modifiedSamples.length > 0 ? modifiedSamples : [0]
+        ),
         tracePath: computeMicroMetric(traceSamples.length > 0 ? traceSamples : [0]),
         searchGraph: computeMicroMetric(searchSamples),
         deadCodeMs: { ms: deadCode.ms },
@@ -36417,6 +36454,8 @@ var METRIC_DIRECTION = {
   fullIndexLocsPerSecond: "higher-is-better",
   incrementalUpdateP95Ms: "lower-is-better",
   incrementalUpdateP50Ms: "lower-is-better",
+  incrementalModifiedUpdateP95Ms: "lower-is-better",
+  incrementalModifiedUpdateP50Ms: "lower-is-better",
   tracePathP95Ms: "lower-is-better",
   searchGraphP95Ms: "lower-is-better",
   deadCodeMs: "lower-is-better",
@@ -36428,16 +36467,43 @@ function extractMetrics(report) {
     fullIndexLocsPerSecond: report.fullIndexLocsPerSecond,
     incrementalUpdateP50Ms: report.incrementalUpdate.p50,
     incrementalUpdateP95Ms: report.incrementalUpdate.p95,
+    incrementalModifiedUpdateP50Ms: report.incrementalModifiedUpdate.p50,
+    incrementalModifiedUpdateP95Ms: report.incrementalModifiedUpdate.p95,
     tracePathP95Ms: report.tracePath.p95,
     searchGraphP95Ms: report.searchGraph.p95,
     deadCodeMs: report.deadCodeMs.ms,
     dbBytesPerKloc: report.dbBytesPerKloc
   };
 }
+var NODE_MAJOR_CACHE = /* @__PURE__ */ new WeakMap();
+function nodeMajor(fp) {
+  const cached = NODE_MAJOR_CACHE.get(fp);
+  if (cached !== void 0) return cached;
+  const major = fp.nodeVersion.replace(/^v/, "").split(".")[0] ?? fp.nodeVersion;
+  NODE_MAJOR_CACHE.set(fp, major);
+  return major;
+}
+function compareMachineFingerprints(report, baseline) {
+  const differing = [];
+  if (report.arch !== baseline.arch) differing.push("arch");
+  if (report.platform !== baseline.platform) differing.push("platform");
+  if (nodeMajor(report) !== nodeMajor(baseline)) differing.push("nodeVersion(major)");
+  if (report.cpuModel !== null && baseline.cpuModel !== null && report.cpuModel !== baseline.cpuModel) {
+    differing.push("cpuModel");
+  }
+  if (report.cpuCores !== baseline.cpuCores) differing.push("cpuCores");
+  return { differingFields: differing };
+}
+function invalidFingerprintFields(fp) {
+  const bad = [];
+  if (typeof fp.arch !== "string") bad.push("arch");
+  if (typeof fp.platform !== "string") bad.push("platform");
+  if (typeof fp.nodeVersion !== "string") bad.push("nodeVersion");
+  if (typeof fp.cpuCores !== "number" || !Number.isFinite(fp.cpuCores)) bad.push("cpuCores");
+  if (fp.cpuModel !== null && typeof fp.cpuModel !== "string") bad.push("cpuModel");
+  return bad;
+}
 function checkCodingGraphRegression(report, baseline, tolerancePercent = DEFAULT_TOLERANCE_PERCENT) {
-  const measured = extractMetrics(report);
-  const baselineMetrics = baseline.metrics;
-  const regressions = [];
   const reportFixture = report.fixture.config;
   const baselineFixture = baseline.fixtureConfig;
   const mismatchedKeys = Object.keys(baselineFixture).filter((key) => reportFixture[key] !== baselineFixture[key]);
@@ -36451,6 +36517,73 @@ function checkCodingGraphRegression(report, baseline, tolerancePercent = DEFAULT
       summary: `Fixture mismatch (${diffs}). Metrics are not comparable across different fixtures.`
     };
   }
+  if (report.schemaVersion !== baseline.schemaVersion) {
+    return {
+      passed: false,
+      regressions: [],
+      summary: `Schema-version mismatch: report=${report.schemaVersion} baseline=${baseline.schemaVersion}. Metrics are not comparable across schema versions \u2014 regenerate the baseline (#1688).`
+    };
+  }
+  const metricChecks = [
+    ["incrementalUpdate.p50", report.incrementalUpdate?.p50],
+    ["incrementalUpdate.p95", report.incrementalUpdate?.p95],
+    ["incrementalModifiedUpdate.p50", report.incrementalModifiedUpdate?.p50],
+    ["incrementalModifiedUpdate.p95", report.incrementalModifiedUpdate?.p95],
+    ["tracePath.p95", report.tracePath?.p95],
+    ["searchGraph.p95", report.searchGraph?.p95],
+    ["fullIndexMs.ms", report.fullIndexMs?.ms],
+    ["fullIndexLocsPerSecond", report.fullIndexLocsPerSecond],
+    ["deadCodeMs.ms", report.deadCodeMs?.ms],
+    ["dbBytesPerKloc", report.dbBytesPerKloc]
+  ];
+  const missingFields = metricChecks.filter(([, v]) => typeof v !== "number" || !Number.isFinite(v)).map(([k]) => k);
+  if (missingFields.length > 0) {
+    return {
+      passed: false,
+      regressions: [],
+      summary: "Report claims schemaVersion " + report.schemaVersion + " but is missing required metric field(s): " + missingFields.join(", ") + " \u2014 the report is incomplete or corrupt. Regenerate it (#1688)."
+    };
+  }
+  const badBaselineMetrics = Object.keys(METRIC_DIRECTION).filter((key) => {
+    const v = baseline.metrics[key];
+    return v != null && (typeof v !== "number" || !Number.isFinite(v));
+  });
+  if (badBaselineMetrics.length > 0) {
+    return {
+      passed: false,
+      regressions: [],
+      summary: "Baseline has a non-numeric required metric field(s): " + badBaselineMetrics.join(", ") + " \u2014 the baseline is corrupt. Regenerate it (#1688)."
+    };
+  }
+  const reportFpBad = report.machine == null ? ["<missing>"] : invalidFingerprintFields(report.machine);
+  const baselineFpBad = baseline.machine == null ? ["<missing>"] : invalidFingerprintFields(baseline.machine);
+  if (reportFpBad.length > 0 || baselineFpBad.length > 0) {
+    const which = [];
+    if (reportFpBad.length > 0) which.push("report (" + reportFpBad.join(", ") + ")");
+    if (baselineFpBad.length > 0) which.push("baseline (" + baselineFpBad.join(", ") + ")");
+    return {
+      passed: false,
+      regressions: [],
+      summary: "Report or baseline machine fingerprint is missing or has an invalid field type(s): " + which.join("; ") + " \u2014 the artifact is incomplete or corrupt. Regenerate it (#1688)."
+    };
+  }
+  const mismatch = compareMachineFingerprints(report.machine, baseline.machine);
+  if (mismatch.differingFields.length > 0) {
+    return {
+      passed: true,
+      skipped: true,
+      regressions: [],
+      summary: "Machine-fingerprint mismatch \u2014 comparison skipped to avoid a false-positive hardware-variance failure. Differing fields: " + mismatch.differingFields.join(", ") + ". Regenerate the baseline on this machine for a real comparison (#1688).",
+      machineMismatch: {
+        report: report.machine,
+        baseline: baseline.machine,
+        differingFields: mismatch.differingFields
+      }
+    };
+  }
+  const measured = extractMetrics(report);
+  const baselineMetrics = baseline.metrics;
+  const regressions = [];
   for (const key of Object.keys(METRIC_DIRECTION)) {
     const baseVal = baselineMetrics[key];
     const measVal = measured[key];

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@remnic/bench",
-  "version": "9.3.727",
+  "version": "9.3.729",
   "description": "Retrieval latency ladder benchmarks + CI regression gates for @remnic/core",
   "type": "module",
   "main": "./dist/index.js",
@@ -36,8 +36,8 @@
   "dependencies": {
     "hyparquet": "^1.25.7",
     "yaml": "^2.4.2",
-    "@remnic/coding-graph": "^9.3.727",
-    "@remnic/core": "^9.3.727"
+    "@remnic/coding-graph": "^9.3.729",
+    "@remnic/core": "^9.3.729"
   },
   "devDependencies": {
     "tsup": "^8.5.1",