ralph-hero-knowledge-index 0.1.31 → 0.1.33

package/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "ralph-knowledge",
-  "version": "0.1.31",
+  "version": "0.1.33",
   "description": "Knowledge graph for ralph-hero: semantic search, relationship traversal, and document indexing across thoughts/ documents. Optional companion to ralph-hero.",
   "author": {
     "name": "Chad Dubiel",

package/.mcp.json

@@ -2,7 +2,7 @@
   "mcpServers": {
     "ralph-knowledge": {
       "command": "npx",
-      "args": ["-y", "ralph-hero-knowledge-index@0.1.31"]
+      "args": ["-y", "ralph-hero-knowledge-index@0.1.33"]
     }
   }
 }

package/benchmark/README.md

@@ -11,7 +11,8 @@ will not break the CI matrix on Node 18/20/22.
 ## Running
 
 Each script is a standalone TypeScript file that can be run directly with
-`tsx` (already a transitive devDependency via `vitest` — no install required):
+`tsx` (declared as a `devDependency` in `package.json`, installed by
+`npm ci`):
 
 ```bash
 # From repo root or plugin/ralph-knowledge:
@@ -19,6 +20,9 @@ npx tsx benchmark/reranker-bench.ts
 
 # Or, equivalently, with the node loader form:
 node --import tsx benchmark/reranker-bench.ts
+
+# Or via the npm script (used by CI for the heap bench):
+npm run bench:heap -- --assert
 ```
 
 Scripts read the same `RALPH_KNOWLEDGE_DB` env var as the MCP server, so by
@@ -48,3 +52,88 @@ the entire run.
 The script is purely additive — it does not modify `hybrid-search.ts` or any
 production source file. Production wiring of a default reranker is a separate
 followup gated on the benchmark findings.
+
+### `reindex-heap-bench.ts` (GH-913)
+
+Microbenchmark guarding the OOM fix from #907 (#911 embedder tensor disposal,
+#916 chunker forward-progress). Generates a deterministic 50-doc / ~240-chunk
+synthetic corpus in a tmp dir via a seeded `mulberry32` RNG, runs `reindex()`
+against it with `RALPH_CONTEXTUAL_RETRIEVAL=0`, samples
+`process.memoryUsage()` every 100 ms, and writes a TSV row with peak
+`heap_used`, `rss`, `external`, wall clock, and chunk count. (The reranker
+bench measures cold-start; the heap bench does not, because `reindex()`
+exposes no hook to mark the moment when the embedding model finishes loading.)
+
+```bash
+# Run once, write TSV row, no exit-1 behavior:
+npx tsx benchmark/reindex-heap-bench.ts
+
+# Same, but exit 1 if peak_heap_used > 600 MB or peak_rss > 800 MB:
+npx tsx benchmark/reindex-heap-bench.ts --assert
+
+# Same as above but via the npm script (used by CI in build-and-test-knowledge):
+npm run bench:heap -- --assert
+```
+
+Results are appended one row per run to `benchmark/results-YYYY-MM-DD.tsv`
+(history-preserving — re-running the bench during a tuning session adds rows
+under the same header rather than overwriting). The TSV header is:
+
+```
+date	doc_count	chunk_count	wall_clock_s	peak_heap_used_mb	peak_rss_mb	peak_external_mb	threshold_pass	notes
+```
+
+Default thresholds (sourced from
+[2026-04-29-reindex-memory-profile.md](../../../thoughts/shared/research/2026-04-29-reindex-memory-profile.md)):
+
+| Threshold            | Value | Rationale                                                                                                                                  |
+|----------------------|-------|--------------------------------------------------------------------------------------------------------------------------------------------|
+| `peak_heap_used_mb`  | 600   | Catches catastrophic regrowth (the original OOM was 4 GB+); ~12x margin over today's typical ~30-50 MB on the 50-doc bench corpus.         |
+| `peak_rss_mb`        | 800   | Catches transformer-model bloat or external-buffer growth; ~1.6-2x margin over today's typical ~400-450 MB on the 50-doc bench corpus.    |
+
+**Tuning the thresholds**: open the TSV results history, find the
+95th-percentile `peak_heap_used_mb` across the last ~10 runs on your CI
+hardware, multiply by 2. That yields a regression-detection threshold without
+flakiness from per-run jitter.
+
+#### Manually verifying the bench fails on a regression
+
+The intuition behind the bench is: **a regression that re-introduces
+unbounded transient allocation will push one of the three peak metrics
+(`heap_used`, `rss`, `external`) far above today's baseline**. The TSV
+records all three so a tuning session can pick the right metric for the
+regression class being guarded.
+
+To confirm the bench's `--assert` path works end-to-end, force a synthetic
+breach by temporarily lowering one of the thresholds in
+`benchmark/reindex-heap-bench.ts`:
+
+```bash
+# In benchmark/reindex-heap-bench.ts, temporarily set:
+#   const HEAP_THRESHOLD_MB = 30;   // below today's ~40 MB baseline
+# (or)
+#   const RSS_THRESHOLD_MB = 300;   // below today's ~450 MB baseline
+
+npx tsx benchmark/reindex-heap-bench.ts --assert
+# expected: exit code 1, console line:
+#   reindex-heap-bench: ASSERT FAIL — THRESHOLD BREACH: heap_used 41.2 > 30
+
+# Restore the threshold (revert benchmark/reindex-heap-bench.ts).
+```
+
+Do **NOT** commit the threshold change — it's a one-time confirmation that
+the assertion path works end-to-end. The bench script itself is purely
+additive and never modifies `embedder.ts`/`chunker.ts`/`reindex.ts`.
+
+**Note on the dispose() regression**: an earlier draft of this section
+suggested reverting `output.dispose()` in `src/embedder.ts` to verify the
+bench catches the original GH-911 OOM. Empirically, on the 50-doc / ~240-chunk
+synthetic corpus, removing the dispose call leaves `peak_heap_used_mb`
+unchanged (~41 MB) and only adds ~3x to `peak_external_mb` (~21 MB -> ~65 MB).
+The original OOM manifested at the live ~14k-chunk corpus scale, not at this
+bench's scale. The bench therefore guards against **catastrophic
+regressions** (a 10x+ allocation increase that crosses the 600 MB / 800 MB
+margins) rather than the specific dispose() leak — which would need a much
+larger synthetic corpus to be detectable. The `peak_external_mb` column is
+recorded in the TSV for future tuning if a tighter native-buffer guard
+becomes worth the added bench runtime.

package/benchmark/eval-rerank.mjs

@@ -0,0 +1,241 @@
+#!/usr/bin/env node
+/**
+ * GH-927 — Re-run the 8-query golden eval with `rerank: true` against the live
+ * `~/.ralph-hero/knowledge.db`. Captures per-query rank-of-expected, cold/warm
+ * latency, and rerank logits. Output is JSON on stdout for downstream
+ * markdown formatting.
+ *
+ * Usage:
+ *   node benchmark/eval-rerank.mjs > /tmp/eval-rerank-results.json
+ *
+ * The script imports the compiled `dist/` modules so it exercises the exact
+ * code path `knowledge_search` runs in production.
+ */
+import { homedir } from "node:os";
+import { join } from "node:path";
+import { performance } from "node:perf_hooks";
+import { KnowledgeDB } from "../dist/db.js";
+import { FtsSearch } from "../dist/search.js";
+import { VectorSearch } from "../dist/vector-search.js";
+import { HybridSearch } from "../dist/hybrid-search.js";
+import { embed } from "../dist/embedder.js";
+import { Reranker } from "../dist/reranker.js";
+
+const DB_PATH = process.env.RALPH_KNOWLEDGE_DB
+  ?? join(homedir(), ".ralph-hero", "knowledge.db");
+
+/**
+ * The 8 golden queries from the 2026-04-29 baseline eval. `expectedSubstrings`
+ * lists path-segment substrings any of which counts as a hit (some queries
+ * have multiple legitimate primary docs per the baseline).
+ */
+const QUERIES = [
+  {
+    n: 1,
+    query: "what causes the reindex to OOM in ralph-knowledge",
+    expectedSubstrings: ["2026-04-29-reindex-memory-profile"],
+    type: "specific-keyword",
+  },
+  {
+    n: 2,
+    query: "release transformer tensors after embedding to free memory",
+    expectedSubstrings: ["2026-04-29-GH-911-release-embedder-tensors"],
+    type: "specific-keyword",
+  },
+  {
+    n: 3,
+    query: "chunker forward progress infinite loop fix",
+    expectedSubstrings: ["2026-04-29-GH-916-chunker-no-progress-fix"],
+    type: "specific-keyword",
+  },
+  {
+    n: 4,
+    query: "dream-loop memory consolidation pipeline architecture",
+    expectedSubstrings: [
+      "2026-04-26-dreaming-research-trail-and-self-containment",
+      "2026-04-16-GH-0761",
+    ],
+    type: "mixed",
+  },
+  {
+    n: 5,
+    query: "cross-encoder reranker score calibration",
+    expectedSubstrings: ["2026-04-26-softmax-and-rerank-calibration"],
+    type: "mixed",
+  },
+  {
+    n: 6,
+    query: "wikilink extractor for markdown",
+    expectedSubstrings: ["2026-04-26-ralph-knowledge-wikilink-extractor"],
+    type: "specific-keyword",
+  },
+  {
+    n: 7,
+    query: "context handoff topology between agents",
+    expectedSubstrings: ["2026-04-22-context-handoff-topology"],
+    type: "mixed",
+  },
+  {
+    n: 8,
+    query: "landcrawler permit raw data migration hardening",
+    expectedSubstrings: ["2026-04-24-landcrawler-backend-hardening-postmortem"],
+    type: "specific-keyword",
+  },
+];
+
+function findRank(results, expectedSubstrings) {
+  for (let i = 0; i < results.length; i++) {
+    const r = results[i];
+    const path = (r.path ?? "") + (r.id ?? "");
+    for (const sub of expectedSubstrings) {
+      if (path.includes(sub)) return i + 1;
+    }
+  }
+  return null;
+}
+
+async function runOne(hybrid, q, withRerank) {
+  const t0 = performance.now();
+  const results = await hybrid.search(q.query, {
+    limit: 10,
+    diagnosticMode: true,
+    rerank: withRerank,
+  });
+  const elapsed = performance.now() - t0;
+  return { results, elapsedMs: elapsed };
+}
+
+async function main() {
+  console.error(`[eval-rerank] DB: ${DB_PATH}`);
+  const db = new KnowledgeDB(DB_PATH);
+  const fts = new FtsSearch(db);
+  const vec = new VectorSearch(db);
+  const reranker = new Reranker();
+  const hybrid = new HybridSearch(db, fts, vec, embed, reranker);
+
+  // Warm up the embedder once (it's a separate model from the reranker; we
+  // care about cold-start of the *reranker*, not the embedder, so isolate it).
+  console.error(`[eval-rerank] warming embedder...`);
+  await hybrid.search("warmup", { limit: 1 });
+
+  const out = [];
+  let firstRerankCall = true;
+
+  for (const q of QUERIES) {
+    console.error(`[eval-rerank] Q${q.n}: ${q.query}`);
+
+    // First rerank invocation is the cold-start one (model load + first batch).
+    // Subsequent invocations are warm.
+    const cold = await runOne(hybrid, q, true);
+    const wasFirstCall = firstRerankCall;
+    firstRerankCall = false;
+
+    // Warm runs: 3 repeats to compute median + p95.
+    const warmRuns = [];
+    for (let i = 0; i < 3; i++) {
+      warmRuns.push(await runOne(hybrid, q, true));
+    }
+
+    // Also capture the no-rerank baseline order from the same DB so the eval
+    // doc can verify the baseline column is reproducible (sanity check).
+    const noRerank = await runOne(hybrid, q, false);
+
+    const sortedWarm = warmRuns.map((r) => r.elapsedMs).sort((a, b) => a - b);
+    const median = sortedWarm[Math.floor(sortedWarm.length / 2)];
+    const p95Idx = Math.min(sortedWarm.length - 1, Math.floor(sortedWarm.length * 0.95));
+
+    const top10 = cold.results.slice(0, 10).map((r) => ({
+      id: r.id,
+      path: r.path,
+      title: r.title,
+      score: r.score,
+      rerankScore: r.rerankScore,
+      ftsScore: r.ftsScore,
+      vecDistance: r.vecDistance,
+      hitSources: r.hitSources,
+    }));
+    const noRerankTop10 = noRerank.results.slice(0, 10).map((r) => ({
+      id: r.id,
+      path: r.path,
+      title: r.title,
+      score: r.score,
+    }));
+
+    const rank = findRank(cold.results, q.expectedSubstrings);
+    const rankNoRerank = findRank(noRerank.results, q.expectedSubstrings);
+    const expectedHit = rank !== null
+      ? cold.results[rank - 1]
+      : null;
+
+    out.push({
+      n: q.n,
+      query: q.query,
+      type: q.type,
+      expectedSubstrings: q.expectedSubstrings,
+      rank,
+      rankNoRerank,
+      expected: expectedHit
+        ? {
+          id: expectedHit.id,
+          path: expectedHit.path,
+          rerankScore: expectedHit.rerankScore,
+          rrfScore: expectedHit.score,
+        }
+        : null,
+      latency: {
+        coldStartMs: wasFirstCall ? cold.elapsedMs : null,
+        firstCallMs: cold.elapsedMs,
+        warmMedianMs: median,
+        warmP95Ms: sortedWarm[p95Idx],
+        warmRunsMs: sortedWarm,
+        noRerankMs: noRerank.elapsedMs,
+      },
+      top10,
+      noRerankTop10,
+    });
+  }
+
+  const ranksWithRerank = out.map((o) => o.rank);
+  const ranksNoRerank = out.map((o) => o.rankNoRerank);
+  const hitAt = (ranks, k) => ranks.filter((r) => r !== null && r <= k).length;
+  const mrr = (ranks) =>
+    ranks.reduce((a, r) => a + (r === null ? 0 : 1 / r), 0) / ranks.length;
+
+  const aggregate = {
+    rerank: {
+      hitAt1: `${hitAt(ranksWithRerank, 1)}/${ranksWithRerank.length}`,
+      hitAt5: `${hitAt(ranksWithRerank, 5)}/${ranksWithRerank.length}`,
+      hitAt10: `${hitAt(ranksWithRerank, 10)}/${ranksWithRerank.length}`,
+      mrr: mrr(ranksWithRerank),
+    },
+    noRerank: {
+      hitAt1: `${hitAt(ranksNoRerank, 1)}/${ranksNoRerank.length}`,
+      hitAt5: `${hitAt(ranksNoRerank, 5)}/${ranksNoRerank.length}`,
+      hitAt10: `${hitAt(ranksNoRerank, 10)}/${ranksNoRerank.length}`,
+      mrr: mrr(ranksNoRerank),
+    },
+  };
+
+  // Latency aggregate
+  const allCold = out.map((o) => o.latency.coldStartMs).filter((x) => x != null);
+  const allWarmMedian = out.map((o) => o.latency.warmMedianMs);
+  const allFirst = out.map((o) => o.latency.firstCallMs);
+  aggregate.latency = {
+    coldStartMs: allCold[0] ?? null,
+    avgWarmMedianMs:
+      allWarmMedian.reduce((a, x) => a + x, 0) / allWarmMedian.length,
+    avgFirstCallMs:
+      allFirst.reduce((a, x) => a + x, 0) / allFirst.length,
+    avgNoRerankMs:
+      out.reduce((a, o) => a + o.latency.noRerankMs, 0) / out.length,
+  };
+
+  console.log(JSON.stringify({ queries: out, aggregate, dbPath: DB_PATH }, null, 2));
+  console.error(`[eval-rerank] done; aggregate:`, aggregate);
+  db.close();
+}
+
+main().catch((e) => {
+  console.error(e);
+  process.exit(1);
+});