ruvnet-brain 4.3.36 → 4.3.37

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -7,7 +7,7 @@ Created: 2026-06-29 22:36:38 EDT
7
7
 
8
8
  # 🧠 RuvNet Brain
9
9
 
10
- ### 🧠 RuvNet Brain — [![RuvNet Brain version 4.3.36 — updated 2026-07-30 03:24 EDT](https://img.shields.io/badge/version_4.3.36-updated_2026--07--30_03:24_EDT-1E90FF?style=for-the-badge&labelColor=0757BA)](https://github.com/stuinfla/ruvnet-brain/blob/main/plugin/.claude-plugin/plugin.json)
10
+ ### 🧠 RuvNet Brain — [![RuvNet Brain version 4.3.37 — updated 2026-07-30 03:24 EDT](https://img.shields.io/badge/version_4.3.37-updated_2026--07--30_03:24_EDT-1E90FF?style=for-the-badge&labelColor=0757BA)](https://github.com/stuinfla/ruvnet-brain/blob/main/plugin/.claude-plugin/plugin.json)
11
11
 
12
12
  **A portable, source-grounded brain over Reuven Cohen's (rUv's) RuvNet stack — delivered as a Claude Code plugin that makes Claude _use_ the stack instead of fighting it.**
13
13
 
@@ -562,7 +562,7 @@ node forge-ask-all.mjs --dir . --q "How does RuVector implement HNSW vector sear
562
562
 
563
563
  This project versions in the open (see the live badge up top for the exact plugin version; the downloadable knowledge bundle is a separate track) — we don't claim “done,” “complete,” or “zero hallucinations.” Where it stands:
564
564
 
565
- - ✅ **The grounding brain is real and proven** — 182 public stores · 143,682 public source chunks, dual embeddings, cross-encoder rerank, plugin (MCP tool + explicit skills; automatic hooks retired), all re-runnable.
565
+ - ✅ **The grounding brain is real and proven** — 199 public stores · 160,685 public source chunks, dual embeddings, cross-encoder rerank, plugin (MCP tool + explicit skills; automatic hooks retired), all re-runnable.
566
566
  - ✅ **Code-level depth** — the code-rich repos are indexed to full function bodies; “how is it implemented?” returns the implementation. Verified in the shipped bundle (clean-room 3/3).
567
567
  - ✅ **Routing holds** — named 47/48, described 26/28, scenario 7/8; behavioral L1–L3 all pass (**L4 downgraded — it measures that the brain spoke, not that anything listened**); private stores fenced out of the public bundle (zero-leak verified).
568
568
  - ⚠️ **Two routing residuals** (above) — surfaced, not hidden.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "ruvnet-brain",
3
- "version": "4.3.36",
3
+ "version": "4.3.37",
4
4
  "description": "One-command installer for RuvNet Brain \u2014 a portable, source-grounded brain over rUv's RuvNet building blocks, delivered as a Claude Code plugin so Claude uses the stack instead of fighting it.",
5
5
  "type": "module",
6
6
  "bin": {
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "ruvnet-brain",
3
3
  "description": "RuvNet brain transplant for Claude Code — grounds every RuvNet decision in real source across 77 rUv repositories, prefers Ruflo / RuVector-RVF / AgentDB over training-prior defaults (pgvector, Pinecone, hand-rolled cosine), and can pull in any RuvNet repo on demand. Ships a UserPromptSubmit retrieve-and-inject grounding hook and a PreToolUse write gate that refuses ungrounded rUv-product code until search_ruvnet has been consulted (ADR-0012 / ADR-067).",
4
- "version": "4.3.36",
4
+ "version": "4.3.37",
5
5
  "author": {
6
6
  "name": "Stuart Kerr"
7
7
  },
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "ruvnet-brain",
3
- "version": "4.3.36",
3
+ "version": "4.3.37",
4
4
  "description": "Source-grounded RuvNet knowledge, lifecycle enforcement, and learning for Codex.",
5
5
  "author": {
6
6
  "name": "Stuart Kerr"
@@ -1,5 +1,5 @@
1
1
  {
2
- "description": "RuvNet Brain continuity plane for Codex. Broad legacy automatic gates remain retired. Registrations here are MEASURED, not assumed: a probe hook was registered on all twelve event names the installed codex-cli 0.154.0 binary declares and a real `codex exec` run was observed (2026-09-11). SessionStart, UserPromptSubmit and SessionEnd FIRED and carry handlers. Stop keeps only the pre-existing, project-scoped continuation gate. PreCompact is DECLARED ABSENT here: the binary declares the event but the probe never observed it, and a capture registered on an unobserved event would look symmetrical while capturing nothing. All handlers are bounded and fail open. Amended 2026-09-11: UserPromptSubmit also runs ground-ruvnet (grounding injection, ADR-040 §Amendment). Amended 2026-09-12: the 2026-09-11 PreToolUse/PostToolUse \"not observed\" note was measured with a prompt (`codex exec \"reply OK\"`) that never invoked a tool, so neither event had anything to fire on — that was an untested path, not a failing one. Re-probed with prompts that actually call a tool: a real apply_patch write fired PreToolUse/PostToolUse with tool_name \"apply_patch\", and a real MCP call to this repo's own search_ruvnet server fired both with tool_name \"mcp__ruvnet_brain__search_ruvnet\". Both are now registered below: decision-gate's write route (matcher includes apply_patch, Codex's raw write-tool name) and grounding-stamp (unchanged matcher already recognizes the mcp__..__search_ruvnet shape). The bash route (exec_command) remains unregistered — today's measurement covered a write and an MCP call, not exec_command, and extending on that evidence would be the same unproven leap this note replaces. Also added 2026-09-12: grounding-turn-mark (UserPromptSubmit) and grounding-turn-gate (Stop), the \"answered without searching\" pair — a prompt-level grounding directive is advisory, so this records whether it fired and forces continuation at Stop if no search_ruvnet call was recorded since (reusing grounding-stamp's own stamp evidence). Both are registered on Codex identically to Claude: the Stop-block contract (hookSpecificOutput.additionalContext) already has a proven Codex translation via codex-hook-adapter.mjs's Stop branch (see tests/unit/codex-lifecycle-hooks.test.mjs), the same path continuation-gate already uses. Capacity-aware parallel-work guidance is also registered at UserPromptSubmit: it advises the coordinator to launch only real independent workers within sampled resource headroom and the live runtime/tool cap; it never spawns workers or claims execution.",
2
+ "description": "RuvNet Brain continuity plane for Codex. Broad legacy automatic gates remain retired. Registrations here are MEASURED, not assumed: a probe hook was registered on all twelve event names the installed codex-cli 0.154.0 binary declares and a real `codex exec` run was observed (2026-09-11). SessionStart, UserPromptSubmit and SessionEnd FIRED and carry handlers. Stop keeps the pre-existing, project-scoped continuation gate; session-snapshot was added at Stop on 2026-09-29 to record each turn's outcome (turn-outcome-capture.mjs) — Codex SessionEnd carries no last_assistant_message, so Stop is the only boundary that can. Its delivery rests on codex-cli 0.158.0's declared stop.command.input schema and the other Codex Stop handlers; it has NOT been live-observed and fails open. PreCompact is DECLARED ABSENT here: the binary declares the event but the probe never observed it, and a capture registered on an unobserved event would look symmetrical while capturing nothing. All handlers are bounded and fail open. Amended 2026-09-11: UserPromptSubmit also runs ground-ruvnet (grounding injection, ADR-040 §Amendment). Amended 2026-09-12: the 2026-09-11 PreToolUse/PostToolUse \"not observed\" note was measured with a prompt (`codex exec \"reply OK\"`) that never invoked a tool, so neither event had anything to fire on — that was an untested path, not a failing one. Re-probed with prompts that actually call a tool: a real apply_patch write fired PreToolUse/PostToolUse with tool_name \"apply_patch\", and a real MCP call to this repo's own search_ruvnet server fired both with tool_name \"mcp__ruvnet_brain__search_ruvnet\". Both are now registered below: decision-gate's write route (matcher includes apply_patch, Codex's raw write-tool name) and grounding-stamp (unchanged matcher already recognizes the mcp__..__search_ruvnet shape). The bash route (exec_command) remains unregistered — today's measurement covered a write and an MCP call, not exec_command, and extending on that evidence would be the same unproven leap this note replaces. Also added 2026-09-12: grounding-turn-mark (UserPromptSubmit) and grounding-turn-gate (Stop), the \"answered without searching\" pair — a prompt-level grounding directive is advisory, so this records whether it fired and forces continuation at Stop if no search_ruvnet call was recorded since (reusing grounding-stamp's own stamp evidence). Both are registered on Codex identically to Claude: the Stop-block contract (hookSpecificOutput.additionalContext) already has a proven Codex translation via codex-hook-adapter.mjs's Stop branch (see tests/unit/codex-lifecycle-hooks.test.mjs), the same path continuation-gate already uses. Capacity-aware parallel-work guidance is also registered at UserPromptSubmit: it advises the coordinator to launch only real independent workers within sampled resource headroom and the live runtime/tool cap; it never spawns workers or claims execution.",
3
3
  "hooks": {
4
4
  "SessionStart": [
5
5
  {
@@ -21,6 +21,11 @@
21
21
  "command": "node -e \"const f=require('node:fs'),o=require('node:os'),p=require('node:path'),c=require('node:child_process'),d=process.env.CODEX_HOME||p.join(o.homedir(),'.codex'),b=process.env.RUVNET_BRAIN_HOME||p.join(p.dirname(d),'.cache','ruvnet-brain'),w=p.join(b,'codex-hook.mjs');let s;try{s=f.statSync(w)}catch{}if(!s?.isFile())process.exit(0);const r=c.spawnSync(process.execPath,[w,...process.argv.slice(2)],{stdio:['inherit','pipe','pipe'],encoding:'utf8',env:process.env,timeout:Number(process.argv[1]),killSignal:'SIGKILL'});if(r.status===0||r.status===2){if(r.stdout)process.stdout.write(r.stdout);if(r.stderr)process.stderr.write(r.stderr)}process.exit(r.status===2?2:0)\" 9000 continuation-gate",
22
22
  "timeout": 10
23
23
  },
24
+ {
25
+ "type": "command",
26
+ "command": "node -e \"const f=require('node:fs'),o=require('node:os'),p=require('node:path'),c=require('node:child_process'),d=process.env.CODEX_HOME||p.join(o.homedir(),'.codex'),b=process.env.RUVNET_BRAIN_HOME||p.join(p.dirname(d),'.cache','ruvnet-brain'),w=p.join(b,'codex-hook.mjs');let s;try{s=f.statSync(w)}catch{}if(!s?.isFile())process.exit(0);const r=c.spawnSync(process.execPath,[w,...process.argv.slice(2)],{stdio:['inherit','pipe','pipe'],encoding:'utf8',env:process.env,timeout:Number(process.argv[1]),killSignal:'SIGKILL'});if(r.status===0||r.status===2){if(r.stdout)process.stdout.write(r.stdout);if(r.stderr)process.stderr.write(r.stderr)}process.exit(r.status===2?2:0)\" 9000 session-snapshot Stop",
27
+ "timeout": 10
28
+ },
24
29
  {
25
30
  "type": "command",
26
31
  "command": "node -e \"const f=require('node:fs'),o=require('node:os'),p=require('node:path'),c=require('node:child_process'),d=process.env.CODEX_HOME||p.join(o.homedir(),'.codex'),b=process.env.RUVNET_BRAIN_HOME||p.join(p.dirname(d),'.cache','ruvnet-brain'),w=p.join(b,'codex-hook.mjs');let s;try{s=f.statSync(w)}catch{}if(!s?.isFile())process.exit(0);const r=c.spawnSync(process.execPath,[w,...process.argv.slice(2)],{stdio:['inherit','pipe','pipe'],encoding:'utf8',env:process.env,timeout:Number(process.argv[1]),killSignal:'SIGKILL'});if(r.status===0||r.status===2){if(r.stdout)process.stdout.write(r.stdout);if(r.stderr)process.stderr.write(r.stderr)}process.exit(r.status===2?2:0)\" 9000 grounding-turn-gate",
@@ -37,9 +37,10 @@
37
37
  "owner": "session-snapshot",
38
38
  "class": "continuity capture",
39
39
  "hosts": [
40
- "claude"
40
+ "claude",
41
+ "codex"
41
42
  ],
42
- "responsibility": "Append one project snapshot to the canonical store and commit any outbox debt a previously interrupted session left behind."
43
+ "responsibility": "Append one project snapshot to the canonical store and commit any outbox debt a previously interrupted session left behind. Also records the turn's outcome (final assistant text, files changed, command descriptions — never user text) to AgentDB namespace `turns`, in the project store when it exists or the machine-wide store outside the repository; SessionEnd/PreCompact distill those records (ruflo ADR-174)."
43
44
  },
44
45
  {
45
46
  "event": "PreCompact",
@@ -153,7 +154,7 @@
153
154
  "Stop": "The 2026-09-11 probe turn never completed, so run_turn_stop_hooks had no completion to fire on. The binary declares the event and StopCommandOutputWire, so this is not-proven rather than unsupported. Stop keeps only the pre-existing continuation gate; no capture handler was added to an event whose delivery has not been seen.",
154
155
  "PreCompact": "A one-line turn never approaches a compaction threshold. The binary declares PreCompact and PreCompactCommandOutputWire. No capture handler registered."
155
156
  },
156
- "consequence": "Codex captures at SessionEnd only; Stop/PreCompact remain unproven and unregistered as above (unchanged by this measurement). PreToolUse/PostToolUse are now proven for a write (apply_patch) and an MCP tool call (search_ruvnet) specifically, and decision-gate's write route plus grounding-stamp are registered on Codex accordingly (see contracts below). Bash-class tool calls (exec_command) were not exercised by this measurement and remain unregistered — extending on unexercised evidence would repeat the exact mistake this record corrects."
157
+ "consequence": "Codex captures at SessionEnd only; Stop/PreCompact remain unproven and unregistered as above (unchanged by this measurement). PreToolUse/PostToolUse are now proven for a write (apply_patch) and an MCP tool call (search_ruvnet) specifically, and decision-gate's write route plus grounding-stamp are registered on Codex accordingly (see contracts below). Bash-class tool calls (exec_command) were not exercised by this measurement and remain unregistered — extending on unexercised evidence would repeat the exact mistake this record corrects. Amended 2026-09-29: session-snapshot is registered on Codex Stop so each Codex turn's outcome is recorded (SessionEnd carries no last_assistant_message). Stop delivery is still NOT live-observed; the registration rests on codex-cli 0.158.0's declared stop.command.input schema and fails open."
157
158
  },
158
159
  "contracts": [
159
160
  {
@@ -196,7 +197,8 @@
196
197
  "id": "session-snapshot",
197
198
  "event": "Stop",
198
199
  "hosts": [
199
- "claude"
200
+ "claude",
201
+ "codex"
200
202
  ],
201
203
  "mode": "advisory",
202
204
  "offBehavior": "run",
@@ -20,7 +20,7 @@
20
20
  * session-start continuity recovery at SessionStart claude, codex
21
21
  * unprompted-speech advisory delivery at UserPromptSubmit claude, codex
22
22
  * continuation-gate continuation nudge at turn end claude, codex
23
- * session-snapshot continuity capture at turn end claude
23
+ * session-snapshot continuity capture at turn end claude, codex
24
24
  * session-snapshot continuity capture at PreCompact claude
25
25
  * session-snapshot continuity capture at SessionEnd claude, codex
26
26
  * ground-ruvnet grounding injection at UserPromptSubmit claude, codex
@@ -95,7 +95,8 @@ const registration = (id, matcher, hosts) => Object.freeze({ id, matcher, hosts:
95
95
  * struct, so this is "not proven", not "not supported".
96
96
  * PreCompact NOT OBSERVED — a one-line turn never approaches a compaction threshold.
97
97
  *
98
- * Therefore Codex capture is registered at SessionEnd ONLY. Stop keeps the pre-existing
98
+ * Therefore Codex capture was registered at SessionEnd ONLY (Stop added 2026-09-29 for turn
99
+ * outcomes — see the Stop registration below; its delivery is still unobserved). Stop keeps the pre-existing
99
100
  * continuation-gate registration (unchanged by this lane); no NEW handler is added to an event whose
100
101
  * delivery has not been seen. hook-contracts.json carries the same measurement and its date.
101
102
  *
@@ -154,7 +155,13 @@ export const CONTINUITY_EVENTS = Object.freeze({
154
155
  ]),
155
156
  Stop: Object.freeze([
156
157
  registration('continuation-gate', '*', ['claude', 'codex']),
157
- registration('session-snapshot', '*', ['claude']),
158
+ // Codex added 2026-09-29 (owner requirement: every turn's outcome recorded on BOTH hosts —
159
+ // turn-outcome-capture.mjs). Codex SessionEnd carries no last_assistant_message, so Stop is the
160
+ // only boundary that can record a Codex turn. Evidence: codex-cli 0.158.0's own
161
+ // stop.command.input schema (last_assistant_message, turn_id) and the two Codex Stop handlers
162
+ // already registered above/below. NOT yet live-observed firing — see the probe box: the capture
163
+ // fails open, so an unfired registration costs nothing but must not be read as proof.
164
+ registration('session-snapshot', '*', ['claude', 'codex']),
158
165
  // The "answered without searching" gate, half 2 of 2 (2026-09-12). Forces continuation
159
166
  // (hookSpecificOutput.additionalContext — the same contract continuation-gate.mjs already uses
160
167
  // and codex-hook-adapter.mjs already translates to Codex's decision:block on both hosts) when
@@ -120,7 +120,9 @@ const TABLE = {
120
120
  // way — it stopped trusting hook-shim.mjs as a blind generic spawner, not their presence here.
121
121
  'learn-capture': { file: 'learn-capture.sh', interpreter: 'bash', mode: 'advisory', offBehavior: 'silence' },
122
122
  'learn-flush': { file: 'learn-flush.mjs', interpreter: 'node', mode: 'advisory', offBehavior: 'silence' },
123
- 'session-snapshot': { file: 'session-snapshot-hook.mjs', interpreter: 'node', mode: 'advisory', offBehavior: 'run', stdinBytes: 65536 },
123
+ // 1 MiB, not 64 KiB: the Stop payload now carries `last_assistant_message`, and a long closing
124
+ // message truncated mid-JSON would parse as `{}` and silently drop the whole capture.
125
+ 'session-snapshot': { file: 'session-snapshot-hook.mjs', interpreter: 'node', mode: 'advisory', offBehavior: 'run', stdinBytes: 1048576 },
124
126
  'md-stamp': { file: 'md-stamp.mjs', interpreter: 'node', mode: 'advisory', offBehavior: 'silence' },
125
127
  // THE EXTERNAL-SIGNAL WATCH PLANE, W1 OBSERVED (ADR-058 §D3; DDD-0013 Context 2). PostToolUse,
126
128
  // matcher ^Bash$ (anchored — an unanchored matcher is F3/F4). Classifies gh/vercel/netlify/npm
@@ -10,6 +10,7 @@ import { projectDirectory } from './project-identity.mjs';
10
10
  import { buildProjectProgression } from './project-progression-producer.mjs';
11
11
  import { ProjectProgressionStore } from './project-progression-store.mjs';
12
12
  import { resolveProjectStore } from './project-store-resolver.mjs';
13
+ import { captureTurnOutcome } from './turn-outcome-capture.mjs';
13
14
 
14
15
  /**
15
16
  * The capture boundary's whole budget. hooks.json declares 10s; this keeps the internal work well
@@ -97,11 +98,19 @@ export function runSessionSnapshotHook(projectDir, event, {
97
98
  produce = buildProjectProgression,
98
99
  budgetMs = CAPTURE_BUDGET_MS,
99
100
  now = Date.now,
101
+ captureTurn = captureTurnOutcome,
100
102
  } = {}) {
101
103
  const metadataWritten = writeSessionSnapshot(projectDir, event);
102
104
  let payload;
103
105
  try { payload = rawInput ? JSON.parse(rawInput) : {}; } catch { payload = {}; }
104
- const idle = { metadataWritten, progressionCaptured: false, receipt: null };
106
+ // TURN OUTCOMES FIRST, and independent of `.swarm`: every turn in every repository is recorded
107
+ // (a project without `.swarm` records to the machine-wide db outside it — turn-outcome-capture.mjs).
108
+ // It only reads and spawns a detached writer, so it costs the progression budget below nothing.
109
+ let turn;
110
+ try { turn = captureTurn({ projectDir, event, payload, host }); } catch (error) {
111
+ turn = { recorded: false, skipped: `turn capture failed: ${error.message}` };
112
+ }
113
+ const idle = { metadataWritten, progressionCaptured: false, receipt: null, turn };
105
114
 
106
115
  if (hasProjectProgression(payload)) {
107
116
  if (payload.hook_event_name !== event) {
@@ -162,6 +171,7 @@ export function runSessionSnapshotHook(projectDir, event, {
162
171
  return {
163
172
  metadataWritten,
164
173
  progressionCaptured: true,
174
+ turn,
165
175
  replayed,
166
176
  receipt: result.receipt,
167
177
  provenance: produced.provenance,
@@ -0,0 +1,292 @@
1
+ /**
2
+ * turn-outcome-capture.mjs — record what each turn CONCLUDED, and make it recallable knowledge.
3
+ *
4
+ * Called from session-snapshot-hook.mjs's runSessionSnapshotHook, so it rides the capture boundary
5
+ * both hosts already register (Claude: hooks.json; Codex: codex-hooks.json via codex-hook-adapter,
6
+ * which sets RUVNET_HOOK_HOST=codex). Two jobs:
7
+ *
8
+ * Stop → one AgentDB record per distinct turn outcome, namespace `turns`
9
+ * SessionEnd/PreCompact → `ruflo memory distill run` on the same db (ruflo ADR-174: memory_entries
10
+ * is the RECORDING tier; distill mines it into episodes/reasoning_patterns/
11
+ * causal_edges, the KNOWLEDGE tier, which stays empty unless it is run)
12
+ *
13
+ * WHERE: the project's `.swarm/memory.db` when it exists, else the machine-wide
14
+ * `~/.claude/global-memory/.swarm/memory.db` (outside every repository). `.swarm` is NEVER created
15
+ * inside a project — see session-snapshot-hook.mjs's writeSessionSnapshot for why that is trespass.
16
+ *
17
+ * WHAT (measured facts carried over from the owner's local Claude hook, 2026-09-29):
18
+ * • The Brain's continuation gate continues most turns, so the turn's REAL final Stop carries
19
+ * stop_hook_active=true. It is NOT skipped; exact repeats are dropped by a per-session
20
+ * fingerprint of (finalText, files) instead.
21
+ * • The transcript may not contain the closing message yet when Stop fires, so the payload's
22
+ * `last_assistant_message` wins when present; only without it is the transcript waited on
23
+ * (bounded) until it stops growing.
24
+ * • Content = assistant outcome text, files changed, Bash descriptions. NEVER raw user text (a
25
+ * 2026-07-13 measurement: prompt echoes made 87% of a store noise). Trivial turns are skipped.
26
+ *
27
+ * CODEX: codex-cli 0.158.0's own `stop.command.input` schema (read from the installed binary
28
+ * 2026-09-29) carries `last_assistant_message` (nullable) and `transcript_path` (nullable).
29
+ * The Codex rollout format is NOT parsed: project-progression-sources.mjs already declares it
30
+ * unknown, and the rollout records observed locally (custom_tool_call / function_call with free-form
31
+ * inputs) give no stable file-change shape. Codex records therefore carry the outcome text only.
32
+ *
33
+ * LATENCY: Stop runs synchronously in the host's turn, and one `ruflo memory store` costs ~0.7s
34
+ * (measured). So the writes run in ONE detached worker (this file, `--run-steps`), store then
35
+ * distill in order; the hook itself only reads, fingerprints and spawns. A breadcrumb line is
36
+ * appended next to the db BEFORE the spawn, and the worker appends a receipt per step, so a lost
37
+ * write is visible rather than silent. Advisory always: nothing here throws to the caller.
38
+ */
39
+ import fs from 'node:fs';
40
+ import os from 'node:os';
41
+ import path from 'node:path';
42
+ import crypto from 'node:crypto';
43
+ import { spawn, spawnSync } from 'node:child_process';
44
+ import { fileURLToPath } from 'node:url';
45
+ import { resolveRuflo, rufloInvocation } from './ruflo-bin.mjs';
46
+
47
+ export const TURN_NAMESPACE = 'turns';
48
+ export const MIN_OUTCOME_CHARS = 200;
49
+ const TRANSCRIPT_TAIL_BYTES = 2 * 1024 * 1024;
50
+ const STEP_TIMEOUT_MS = 60_000;
51
+ const RUFLO_ENV = { RUFLO_DAEMON_AUTOSTART: '0' };
52
+
53
+ function textOf(content) {
54
+ if (typeof content === 'string') return content;
55
+ if (!Array.isArray(content)) return '';
56
+ return content.filter((c) => c && c.type === 'text' && typeof c.text === 'string').map((c) => c.text).join('\n');
57
+ }
58
+
59
+ function patchFiles(patch) {
60
+ const out = new Set();
61
+ for (const line of String(patch || '').split(/\r?\n/)) {
62
+ const m = /^\*\*\* (?:Add|Update|Delete) File: (.+)$/.exec(line) || /^\*\*\* Move to: (.+)$/.exec(line);
63
+ if (m) out.add(m[1].trim());
64
+ }
65
+ return [...out];
66
+ }
67
+
68
+ /**
69
+ * The current turn of a Claude JSONL transcript = every record after the last genuine user message
70
+ * (a user record carrying text, not a tool_result). User text is used ONLY to find that boundary.
71
+ */
72
+ export function claudeTurn(lines) {
73
+ const recs = [];
74
+ for (const l of lines) { try { recs.push(JSON.parse(l)); } catch { /* partial or foreign line */ } }
75
+ let start = 0;
76
+ recs.forEach((o, i) => {
77
+ const role = o?.message?.role || o?.role;
78
+ const c = o?.message?.content;
79
+ const isToolResult = Array.isArray(c) && c.some((x) => x && x.type === 'tool_result');
80
+ if (role === 'user' && !isToolResult && textOf(c).trim()) start = i + 1;
81
+ });
82
+ const texts = [];
83
+ const files = new Set();
84
+ const actions = [];
85
+ for (const o of recs.slice(start)) {
86
+ if ((o?.message?.role || o?.role) !== 'assistant') continue;
87
+ const c = o.message?.content;
88
+ const t = textOf(c).trim();
89
+ if (t) texts.push(t);
90
+ if (!Array.isArray(c)) continue;
91
+ for (const u of c) {
92
+ if (!u || u.type !== 'tool_use') continue;
93
+ const inp = u.input || {};
94
+ if (['Edit', 'Write', 'MultiEdit', 'NotebookEdit'].includes(u.name)) {
95
+ const f = inp.file_path || inp.notebook_path;
96
+ if (typeof f === 'string' && f) files.add(f);
97
+ } else if (u.name === 'apply_patch') {
98
+ for (const f of patchFiles(typeof inp === 'string' ? inp : inp.command || inp.input)) files.add(f);
99
+ } else if (u.name === 'Bash' && typeof inp.description === 'string' && inp.description) {
100
+ actions.push(inp.description);
101
+ }
102
+ }
103
+ }
104
+ // The closing message usually carries the outcome; when it is short (a tool-heavy turn) the turn's
105
+ // other assistant text is kept too, newest last, so the record still says what happened.
106
+ const last = texts.length ? texts[texts.length - 1] : '';
107
+ const finalText = last.length >= MIN_OUTCOME_CHARS ? last : texts.join('\n').slice(-2500);
108
+ return { finalText, files: [...files], actions };
109
+ }
110
+
111
+ function readTail(file, bytes = TRANSCRIPT_TAIL_BYTES) {
112
+ const size = fs.statSync(file).size;
113
+ const offset = Math.max(0, size - bytes);
114
+ const handle = fs.openSync(file, 'r');
115
+ try {
116
+ const buf = Buffer.alloc(size - offset);
117
+ fs.readSync(handle, buf, 0, buf.length, offset);
118
+ const lines = buf.toString('utf8').split(/\r?\n/);
119
+ if (offset > 0) lines.shift(); // the first line of a tail is almost always cut mid-record
120
+ return lines;
121
+ } finally { fs.closeSync(handle); }
122
+ }
123
+
124
+ /** Wait (bounded) until the transcript stops growing, then return its tail lines. */
125
+ export function readSettledTranscript(file, { stableMs = 400, maxMs = 2000, sleep } = {}) {
126
+ const pause = sleep || ((ms) => Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms));
127
+ const deadline = Date.now() + Math.max(0, maxMs);
128
+ let size = -1;
129
+ for (;;) {
130
+ const now = fs.statSync(file).size;
131
+ if (now === size || Date.now() >= deadline) break;
132
+ size = now;
133
+ pause(Math.min(stableMs, Math.max(0, deadline - Date.now())));
134
+ }
135
+ return readTail(file);
136
+ }
137
+
138
+ export function buildTurnRecord({ turn, project, host, session, at = new Date() }) {
139
+ const parts = [`[turn ${at.toISOString()} project=${project} host=${host}]`,
140
+ `OUTCOME: ${turn.finalText.replace(/\s+/g, ' ').slice(0, 2500)}`];
141
+ if (turn.files.length) parts.push(`FILES CHANGED: ${turn.files.slice(0, 25).join(', ')}`);
142
+ if (turn.actions.length) parts.push(`ACTIONS: ${turn.actions.slice(-15).join(' • ')}`);
143
+ parts.push(`SESSION: ${session || '?'}`);
144
+ return parts.join(' || ').slice(0, 4000);
145
+ }
146
+
147
+ /** Project db if the project already has one; otherwise the machine-wide db outside every repo. */
148
+ export function resolveTurnDb({ projectDir, home = os.homedir(), env = process.env } = {}) {
149
+ const projectDb = path.join(projectDir, '.swarm', 'memory.db');
150
+ try { if (fs.statSync(projectDb).isFile()) return { db: projectDb, scope: 'project' }; } catch { /* absent */ }
151
+ const globalDb = env.RUVNET_TURN_GLOBAL_DB || path.join(home, '.claude', 'global-memory', '.swarm', 'memory.db');
152
+ return { db: globalDb, scope: 'global' };
153
+ }
154
+
155
+ const projectName = (projectDir) => path.basename(path.resolve(projectDir)).replace(/[^A-Za-z0-9._-]/g, '_') || 'project';
156
+
157
+ /** Detached worker launch: the hook returns immediately; the worker runs the steps in order. */
158
+ export function launchDetached(steps, { receipts }) {
159
+ const child = spawn(process.execPath, [fileURLToPath(import.meta.url), '--run-steps', JSON.stringify({ steps, receipts })], {
160
+ cwd: os.tmpdir(), detached: true, stdio: 'ignore', windowsHide: true, env: { ...process.env, ...RUFLO_ENV },
161
+ });
162
+ child.unref();
163
+ return { launched: true, pid: child.pid };
164
+ }
165
+
166
+ function sameTurnSeen(stateFile, sessionKey, fingerprint) {
167
+ let last = {};
168
+ try { last = JSON.parse(fs.readFileSync(stateFile, 'utf8')) || {}; } catch { /* first run */ }
169
+ if (last[sessionKey] === fingerprint) return true;
170
+ const keys = Object.keys(last);
171
+ for (const k of keys.slice(0, Math.max(0, keys.length - 200))) delete last[k];
172
+ try {
173
+ fs.mkdirSync(path.dirname(stateFile), { recursive: true, mode: 0o700 });
174
+ fs.writeFileSync(stateFile, JSON.stringify({ ...last, [sessionKey]: fingerprint }), { mode: 0o600 });
175
+ } catch { /* dedupe is best effort; a duplicate record beats a lost one */ }
176
+ return false;
177
+ }
178
+
179
+ /**
180
+ * Capture this turn's outcome (Stop) and/or queue distillation (SessionEnd, PreCompact).
181
+ * Returns a plain report; `skipped` / `distill.skipped` carry the reason whenever nothing is written.
182
+ */
183
+ export function captureTurnOutcome({
184
+ projectDir, event, payload = {}, host = 'claude',
185
+ env = process.env, home = os.homedir(),
186
+ brainHome = env.RUVNET_BRAIN_HOME || path.join(home, '.cache', 'ruvnet-brain'),
187
+ ruflo = resolveRuflo({ env, home }),
188
+ launch = launchDetached,
189
+ readTranscript = readSettledTranscript,
190
+ settleMs = 2000,
191
+ now = () => new Date(),
192
+ } = {}) {
193
+ const report = { event, host, recorded: false, distill: { queued: false } };
194
+ if (String(env.RUVNET_TURN_CAPTURE || '').toLowerCase() === 'off') {
195
+ return { ...report, skipped: 'RUVNET_TURN_CAPTURE=off', distill: { queued: false, skipped: 'RUVNET_TURN_CAPTURE=off' } };
196
+ }
197
+ if (!ruflo) return { ...report, skipped: 'ruflo not found', distill: { queued: false, skipped: 'ruflo not found' } };
198
+ const { db, scope } = resolveTurnDb({ projectDir, home, env });
199
+ Object.assign(report, { db, scope });
200
+ const steps = [];
201
+ const project = projectName(projectDir);
202
+ const receipts = path.join(brainHome, 'turn-capture', 'receipts.jsonl');
203
+
204
+ if (event === 'Stop') {
205
+ const sessionKey = String(payload.session_id || payload.transcript_path || '');
206
+ const message = typeof payload.last_assistant_message === 'string' ? payload.last_assistant_message.trim() : '';
207
+ let turn = { finalText: '', files: [], actions: [] };
208
+ if (host === 'claude' && typeof payload.transcript_path === 'string' && payload.transcript_path) {
209
+ // With the closing message in hand, the transcript is read immediately (tool calls are already
210
+ // flushed); without it, wait for the closing message to land, bounded.
211
+ try { turn = claudeTurn(readTranscript(payload.transcript_path, { maxMs: message ? 0 : settleMs })); } catch { /* unreadable */ }
212
+ }
213
+ if (message && (message.length >= MIN_OUTCOME_CHARS || message.length >= turn.finalText.length)) turn.finalText = message;
214
+ if (!sessionKey) report.skipped = 'no session identity in the host payload';
215
+ else if (turn.finalText.length < MIN_OUTCOME_CHARS && !turn.files.length) {
216
+ report.skipped = host === 'codex' && !message ? 'codex payload carried no last_assistant_message' : 'trivial turn';
217
+ } else {
218
+ const fingerprint = crypto.createHash('sha256').update(JSON.stringify([turn.finalText, turn.files])).digest('hex');
219
+ const stateFile = path.join(brainHome, 'turn-capture', 'last-turn.json');
220
+ if (sameTurnSeen(stateFile, `${host}:${sessionKey}`, fingerprint)) report.skipped = 'same turn outcome already recorded';
221
+ else {
222
+ const at = now();
223
+ const key = `turn-${project}-${at.getTime()}`;
224
+ const value = buildTurnRecord({ turn, project, host, session: payload.session_id, at });
225
+ steps.push({ kind: 'store', ruflo, args: ['memory', 'store', '-k', key, '--value', value, '-n', TURN_NAMESPACE,
226
+ '--path', db, '--tags', `project=${project},host=${host}`, '--provenance', 'agent_output'] });
227
+ Object.assign(report, { recorded: true, key, value });
228
+ }
229
+ }
230
+ } else report.skipped = `turn outcomes are recorded at Stop, not ${event}`;
231
+
232
+ if (event === 'SessionEnd' || event === 'PreCompact') {
233
+ if (!fs.existsSync(db)) report.distill = { queued: false, skipped: 'no memory db to distill yet' };
234
+ else {
235
+ steps.push({ kind: 'distill', ruflo, args: ['memory', 'distill', 'run', '--db', db, '--namespace', TURN_NAMESPACE, '--max-entries', '500'] });
236
+ report.distill = { queued: true, db };
237
+ }
238
+ }
239
+ if (!steps.length) return report;
240
+
241
+ try {
242
+ // The machine-wide db lives outside every repository; its directory is created on first use so a
243
+ // fresh machine records from its first turn. A project's `.swarm` is never created (scope check).
244
+ if (scope === 'global') fs.mkdirSync(path.dirname(db), { recursive: true, mode: 0o700 });
245
+ if (report.recorded) {
246
+ fs.appendFileSync(path.join(path.dirname(db), 'agentdb-turns.jsonl'),
247
+ `${JSON.stringify({ ts: Date.now(), key: report.key, project, host, value: report.value })}\n`, { mode: 0o600 });
248
+ }
249
+ report.launch = launch(steps, { receipts });
250
+ } catch (error) {
251
+ return { ...report, recorded: false, distill: { queued: false, skipped: `launch failed: ${error.message}` }, skipped: `launch failed: ${error.message}` };
252
+ }
253
+ return report;
254
+ }
255
+
256
+ /** The detached worker: run each step in order, bounded, and append one receipt per step. */
257
+ export function runSteps({ steps = [], receipts } = {}, { run = spawnSync } = {}) {
258
+ const results = [];
259
+ for (const step of steps) {
260
+ let status = null;
261
+ let error = null;
262
+ const db = step.args[step.args.indexOf(step.kind === 'store' ? '--path' : '--db') + 1];
263
+ try {
264
+ const { executable, args } = rufloInvocation(step.ruflo, step.args);
265
+ // CONTAINMENT (measured 2026-09-29, ruflo 3.48.0): even with an explicit --path, `memory store`
266
+ // writes `.swarm/hnsw.index` and `ruvector.db` relative to its CWD. Run from an inherited cwd,
267
+ // that planted `.swarm/` + `ruvector.db` in a repository that never adopted the brain — the
268
+ // exact trespass session-snapshot-hook.mjs forbids. So every step runs INSIDE the db's own
269
+ // directory, with ruflo's memory root pinned there too.
270
+ const home = path.dirname(db);
271
+ const r = run(executable, args, { stdio: 'ignore', timeout: STEP_TIMEOUT_MS, cwd: home, windowsHide: true,
272
+ env: { ...process.env, ...RUFLO_ENV, CLAUDE_FLOW_MEMORY_PATH: home } });
273
+ status = r.status;
274
+ if (r.error) error = r.error.message;
275
+ } catch (e) { error = e.message; }
276
+ const row = { at: new Date().toISOString(), kind: step.kind, db,
277
+ key: step.kind === 'store' ? step.args[step.args.indexOf('-k') + 1] : undefined, status, error };
278
+ results.push(row);
279
+ if (receipts) {
280
+ try {
281
+ fs.mkdirSync(path.dirname(receipts), { recursive: true, mode: 0o700 });
282
+ fs.appendFileSync(receipts, `${JSON.stringify(row)}\n`, { mode: 0o600 });
283
+ } catch { /* receipts are best effort */ }
284
+ }
285
+ }
286
+ return results;
287
+ }
288
+
289
+ if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url) && process.argv[2] === '--run-steps') {
290
+ try { runSteps(JSON.parse(process.argv[3] || '{}')); } catch { /* a detached worker has no one to report to */ }
291
+ process.exit(0);
292
+ }
@@ -0,0 +1,71 @@
1
+ #!/usr/bin/env node
2
+ // Derive data/retrieval-passage-content-digests.json from a corpus archive built with ordinal passage
3
+ // ids. See scripts/retrieval-passage-identity.mjs for why the map exists.
4
+ //
5
+ // node scripts/derive-passage-content-map.mjs --zip <old-format ruvnet-brain.zip> \
6
+ // --source-tag v4.3.36 [--fixture data/retrieval-query-evidence.json] [--out <file>]
7
+ //
8
+ // For every expected passage the frozen fixture pins (primary + alternatives), find the ONE row in the
9
+ // archive's store whose path matches and whose digest equals the pin, and record that row's id-less
10
+ // content digest. A pin that is not found exactly once is listed as `unresolved` (never guessed): it
11
+ // keeps exact-digest matching only. The output is deterministic, so re-running proves the committed map.
12
+ import fs from 'node:fs';
13
+ import path from 'node:path';
14
+ import crypto from 'node:crypto';
15
+ import { spawnSync } from 'node:child_process';
16
+ import { fileURLToPath } from 'node:url';
17
+ import { digest } from './coverage-integrity.mjs';
18
+ import { CONTENT_MAP_FILE, CONTENT_MAP_KIND, passageContentDigest } from './retrieval-passage-identity.mjs';
19
+
20
+ const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');
21
+ const sha256File = (file) => crypto.createHash('sha256').update(fs.readFileSync(file)).digest('hex');
22
+
23
+ /** Every pin in the fixture: [{ store, path, pinned }]. */
24
+ export function fixturePins(fixture) {
25
+ const pins = [];
26
+ for (const [store, row] of Object.entries(fixture.queries || {})) {
27
+ const sources = [{ path: row.expected.path, passageSha256: row.expected.passageSha256 }, ...(row.expected.alternatives || [])];
28
+ for (const source of sources) pins.push({ store, path: source.path, pinned: source.passageSha256 });
29
+ }
30
+ return pins;
31
+ }
32
+
33
+ export function deriveContentMap({ fixture, fixtureSha256, readRows, sourceTag, archiveSha256 }) {
34
+ const entries = {};
35
+ const unresolved = [];
36
+ for (const { store, path: expectedPath, pinned } of fixturePins(fixture)) {
37
+ const rows = readRows(store);
38
+ const found = (rows || []).filter((row) => row.path === expectedPath && digest(row) === pinned);
39
+ if (found.length === 1) entries[pinned] = passageContentDigest(found[0]);
40
+ else unresolved.push({ store, path: expectedPath, pinned, reason: rows ? `${found.length} matching rows` : 'store absent from archive' });
41
+ }
42
+ const sorted = Object.fromEntries(Object.entries(entries).sort(([a], [b]) => a.localeCompare(b)));
43
+ return { schemaVersion: 1, kind: CONTENT_MAP_KIND, fixtureSha256,
44
+ derivedFrom: { tag: sourceTag, archiveSha256 },
45
+ entries: sorted,
46
+ unresolved: unresolved.sort((a, b) => a.store.localeCompare(b.store) || a.path.localeCompare(b.path)) };
47
+ }
48
+
49
+ function zipRows(zipFile, store) {
50
+ const result = spawnSync('unzip', ['-p', zipFile, `${store}.passages.jsonl`], { maxBuffer: 1 << 30 });
51
+ if (result.status !== 0) return null;
52
+ // Split on \n only: rows may contain U+2028/U+2029, which are not JSONL separators.
53
+ return result.stdout.toString('utf8').split('\n').filter((line) => line.trim()).map((line) => JSON.parse(line));
54
+ }
55
+
56
+ function main(argv) {
57
+ const arg = (name, fallback = null) => { const i = argv.indexOf(name); return i >= 0 ? argv[i + 1] : fallback; };
58
+ const zip = arg('--zip');
59
+ const sourceTag = arg('--source-tag');
60
+ if (!zip || !sourceTag) { console.error('usage: derive-passage-content-map.mjs --zip <archive> --source-tag <tag> [--fixture f] [--out f]'); process.exit(2); }
61
+ const fixtureFile = path.resolve(arg('--fixture', path.join(ROOT, 'data', 'retrieval-query-evidence.json')));
62
+ const out = path.resolve(arg('--out', CONTENT_MAP_FILE));
63
+ const cache = new Map();
64
+ const readRows = (store) => { if (!cache.has(store)) cache.set(store, zipRows(zip, store)); return cache.get(store); };
65
+ const map = deriveContentMap({ fixture: JSON.parse(fs.readFileSync(fixtureFile, 'utf8')), fixtureSha256: sha256File(fixtureFile),
66
+ readRows, sourceTag, archiveSha256: sha256File(zip) });
67
+ fs.writeFileSync(out, `${JSON.stringify(map, null, 2)}\n`);
68
+ console.log(JSON.stringify({ out, resolved: Object.keys(map.entries).length, unresolved: map.unresolved.length }));
69
+ }
70
+
71
+ if (process.argv[1] && fs.realpathSync(process.argv[1]) === fileURLToPath(import.meta.url)) main(process.argv.slice(2));
@@ -11,6 +11,7 @@ import { extractZip } from '../kb/zip-extract.mjs';
11
11
  import { canonicalJson, digest, validateCoverageLedger, validateCoverageLink } from './coverage-integrity.mjs';
12
12
  import { validatePublicInventory } from './public-inventory.mjs';
13
13
  import { verifySeedBaseline } from './corpus-candidate.mjs';
14
+ import { loadFixture, readRecallReport } from './oracle/repo-recall.mjs';
14
15
  import {
15
16
  buildRetrievalCanaryPlan,
16
17
  validateRetrievalQueryEvidence,
@@ -430,9 +431,27 @@ function writeExactOutputs(outDir, outputs) {
430
431
  return root;
431
432
  }
432
433
 
434
+ /**
435
+ * The stores a corpus generation's OWN repo-recall measurement retrieved (exact file within top-k). The
436
+ * report is bound to the exact baseline archive (sha256 + bytes) and to the frozen fixture the canary
437
+ * samples from, and is re-derived through the same reader the corpus pipeline uses — a report for another
438
+ * archive or another fixture is refused, never silently accepted.
439
+ */
440
+ export function measuredHitStores({ recallFile, baselineArchive, oracleFile }) {
441
+ const stat = fs.statSync(baselineArchive);
442
+ // A report that claims retired questions must be verified against the coverage it names; the
443
+ // generation's sealed coverage sits beside its report in the seed directory when it exists.
444
+ const siblingCoverage = path.join(path.dirname(path.resolve(recallFile)), 'CORPUS-COVERAGE.json');
445
+ const { report } = readRecallReport({ reportFile: recallFile,
446
+ archive: { sha256: sha256File(baselineArchive), bytes: stat.size },
447
+ expectedFixtureSha256: loadFixture(oracleFile).fixtureSha256,
448
+ coverageBytes: fs.existsSync(siblingCoverage) ? fs.readFileSync(siblingCoverage) : null });
449
+ return new Set(report.rows.filter((row) => Number.isInteger(row.exactFileRank)).map((row) => String(row.store).toLowerCase()));
450
+ }
451
+
433
452
  export async function createPublicVerificationInputs({ baselineBundle, candidateBundle,
434
453
  candidatePackage, oracleFile, repo = process.cwd(), outDir = 'release-evidence', baselineMode = 'verified',
435
- baselineReceipt = null } = {}) {
454
+ baselineReceipt = null, baselineRecall = null } = {}) {
436
455
  const baselineArchive = trustedFile(baselineBundle, 'baseline archive');
437
456
  const candidateArchive = trustedFile(candidateBundle, 'candidate archive');
438
457
  const packageFile = trustedFile(candidatePackage, 'candidate package');
@@ -513,9 +532,11 @@ export async function createPublicVerificationInputs({ baselineBundle, candidate
513
532
  verifyQueryOracleSource(queryEvidence, candidateResult.candidate.sourceSha, {
514
533
  cwd: path.resolve(repo), allowSquashedSource: true,
515
534
  });
535
+ const knownHitStores = baselineRecall
536
+ ? measuredHitStores({ recallFile: baselineRecall, baselineArchive, oracleFile: oraclePath }) : null;
516
537
  const plan = buildRetrievalCanaryPlan({ coverage: candidateResult.coverage, baseline,
517
538
  candidate: candidateResult.candidate, coverageIdentity: candidateResult.coverageIdentity,
518
- queryEvidence, assetsDir: candidateTree.root, allowNoDelta: true });
539
+ queryEvidence, assetsDir: candidateTree.root, allowNoDelta: true, knownHitStores });
519
540
  writeExactOutputs(outDir, {
520
541
  [baselineMode === 'observed' ? 'baseline-observation-receipt.json' : 'baseline-verification-receipt.json']: baselineProof.bytes,
521
542
  'COVERAGE.json': candidateResult.coverageBytes,
@@ -574,6 +595,7 @@ export async function main(argv = process.argv.slice(2)) {
574
595
  outDir: arg(argv, '--out-dir') || 'release-evidence',
575
596
  baselineMode: argv.includes('--receipted-baseline') ? 'receipted' : argv.includes('--observed-baseline') ? 'observed' : 'verified',
576
597
  baselineReceipt: arg(argv, '--baseline-receipt'),
598
+ baselineRecall: arg(argv, '--baseline-recall'),
577
599
  });
578
600
  console.log(JSON.stringify({ ok: true, sourceSha: result.candidate.sourceSha,
579
601
  coverageGeneration: result.coverage.releaseCoverageGeneration, cases: result.plan.cases.length }));
@@ -60,13 +60,36 @@ const tagSha = (tag, root) => {
60
60
  // Assets now stream to a temp file and are hashed incrementally, so peak memory is one 1MB chunk
61
61
  // instead of the whole bundle and there is no ceiling to outgrow. Small assets (receipts) still
62
62
  // come back as bytes, because callers parse them as JSON.
63
- const assetToFile = (asset, destination) => {
64
- const result = spawnSync('gh', ['api', asset.url, '-H', 'Accept: application/octet-stream'], {
65
- stdio: ['ignore', fs.openSync(destination, 'w'), 'pipe'], timeout: ASSET_DOWNLOAD_TIMEOUT_MS,
66
- });
67
- if (result.error || result.signal || result.status !== 0) {
68
- throw new Error(`cannot download transaction asset ${asset.name}: ${result.error?.message || result.signal || `exit ${result.status}`}`);
63
+ //
64
+ // 2026-09-29: ONE FLAKY DOWNLOAD MUST NOT ABORT A RELEASE. discover() reads the receipts of every
65
+ // published release (411 assets, serially, ~3 minutes); a single transient `gh api` failure among
66
+ // them killed the 4.3.36 publish with only "exit 1" -- the stderr that said why was discarded, and
67
+ // re-fetching the same asset a minute later worked. A download is now retried with backoff, and the
68
+ // final error carries the real cause. A genuinely missing/corrupt asset still fails, just not on
69
+ // the first hiccup; the digest checks downstream are unchanged.
70
+ export const ASSET_DOWNLOAD_ATTEMPTS = 4;
71
+ const sleepSync = (ms) => { Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms); };
72
+ export const downloadAsset = (asset, destination, {
73
+ spawn = spawnSync, wait = sleepSync, attempts = ASSET_DOWNLOAD_ATTEMPTS, baseDelayMs = 2000,
74
+ } = {}) => {
75
+ let cause = '';
76
+ for (let attempt = 1; attempt <= attempts; attempt += 1) {
77
+ const fd = fs.openSync(destination, 'w'); // 'w' truncates any partial bytes from the previous attempt
78
+ let result;
79
+ try {
80
+ result = spawn('gh', ['api', asset.url, '-H', 'Accept: application/octet-stream'], {
81
+ stdio: ['ignore', fd, 'pipe'], timeout: ASSET_DOWNLOAD_TIMEOUT_MS,
82
+ });
83
+ } finally { fs.closeSync(fd); }
84
+ if (!result.error && !result.signal && result.status === 0) return destination;
85
+ const stderr = String(result.stderr || '').trim().slice(0, 300);
86
+ cause = `${result.error?.message || result.signal || `exit ${result.status}`}${stderr ? ` (${stderr})` : ''}`;
87
+ if (attempt < attempts) wait(baseDelayMs * 2 ** (attempt - 1));
69
88
  }
89
+ throw new Error(`cannot download transaction asset ${asset.name} after ${attempts} attempts: ${cause}`);
90
+ };
91
+ const assetToFile = (asset, destination) => {
92
+ downloadAsset(asset, destination);
70
93
  return destination;
71
94
  };
72
95
  const withTempAsset = (asset, fn) => {
@@ -7,9 +7,10 @@ import { spawnSync } from 'node:child_process';
7
7
  import { fileURLToPath } from 'node:url';
8
8
  import { canonicalJson, digest, eligibleRepositoryStanding, validateCoverageLedger } from './coverage-integrity.mjs';
9
9
  import { fixtureDenominator } from './fixture-denominator.mjs';
10
+ import { passageMatches } from './retrieval-passage-identity.mjs';
10
11
 
11
12
  // Both release phases resolve against an explicit installed context, never the checkout.
12
- export async function resolveInstalledCanaryCitation({ kbDir, matched, expected, passageFileDigests = new Map() }) {
13
+ export async function resolveInstalledCanaryCitation({ kbDir, matched, expected, passageFileDigests = new Map(), contentMap }) {
13
14
  if (!path.isAbsolute(kbDir || '')) throw new Error('installed canary KB path must be absolute');
14
15
  if (String(matched?.repo || '').toLowerCase() !== expected.repo || matched?.path !== expected.path) return { resolved: false };
15
16
  if (!/^[a-z0-9][a-z0-9._-]*$/i.test(expected.repo)) throw new Error('installed citation repository violates containment');
@@ -35,7 +36,7 @@ export async function resolveInstalledCanaryCitation({ kbDir, matched, expected,
35
36
  let record;
36
37
  try { record = JSON.parse(line); } catch { continue; }
37
38
  if (record?.path !== expected.path) continue;
38
- if (digest(record) !== expected.passageSha256) continue;
39
+ if (!passageMatches(record, expected.passageSha256, contentMap)) continue;
39
40
  const text = record.fullText || record.text;
40
41
  if (typeof text !== 'string' || !text || typeof matched.text !== 'string' || !matched.text.includes(text)) continue;
41
42
  passageSha256 = expected.passageSha256;
@@ -386,7 +387,8 @@ export function validatePlanAgainstCoverage(plan, coverage, { allowObservedBasel
386
387
  return plan;
387
388
  }
388
389
  export function buildRetrievalCanaryPlan({ coverage, baseline, candidate, coverageIdentity = null, queryEvidence, assetsDir = '.',
389
- readPassages = defaultReadPassages, legacySampleSize, allowNoDelta = false } = {}) {
390
+ readPassages = defaultReadPassages, legacySampleSize, allowNoDelta = false, contentMap, knownHitStores = null,
391
+ notice = (message) => process.stderr.write(`${message}\n`) } = {}) {
390
392
  const checked = validateCoverageLedger(coverage);
391
393
  if (!checked.valid) throw new Error(`coverage ledger is invalid: ${checked.failures.join('; ')}`);
392
394
  const coverageGeneration = coverage.kind === 'ruvnet-brain-release-coverage'
@@ -455,12 +457,47 @@ export function buildRetrievalCanaryPlan({ coverage, baseline, candidate, covera
455
457
  const passages = new Map(eligible.map((row) => [storeOf(row), readPassages(assetsDir, storeOf(row))]));
456
458
  const rankedLegacy = legacyPool.map((row) => ({ row, count: passages.get(storeOf(row)).length }))
457
459
  .sort((a, b) => a.count - b.count || storeOf(a.row).localeCompare(storeOf(b.row)));
460
+ // The legacy sample is drawn only from stores whose sealed passage still exists, unchanged, exactly
461
+ // once in the shipped store. When upstream edits the very file a fixture question was written against,
462
+ // that question can no longer identify its passage — the fixture is stale for that store, which says
463
+ // nothing about retrieval. Such stores stay in the sealed POPULATION (recomputed from coverage by
464
+ // validatePlanAgainstCoverage) but cannot be sampled; they are named below so it is never silent, and
465
+ // the nightly per-repository recall gate still exercises every one of them by file path.
466
+ const sealedPassageResolves = (store) => {
467
+ const evidence = queryEvidence.queries[store];
468
+ return Boolean(evidence) && expectedSources(evidence.expected).every((source) =>
469
+ passages.get(store).filter((row) => row.path === source.path && passageMatches(row, source.passageSha256, contentMap)).length === 1);
470
+ };
471
+ // INTEGRITY, NOT QUALITY. When the generation being shipped carries its own repo-recall measurement,
472
+ // `knownHitStores` names the stores that measurement retrieved. The release sample is then drawn from
473
+ // them, so the canary proves the SHIPPED, INSTALLED bundle reproduces what the generation measured on
474
+ // the same stores (a packaging, index, model or runtime break shows up as a miss on a store that hit).
475
+ // It deliberately does NOT re-judge stores the generation already missed: whole-corpus retrieval quality
476
+ // is the recall report's job and stays visible there (and in the corpus watchdog), because sampling ~19
477
+ // stores at an absolute 98% bar is a coin flip for any corpus below ~98% true recall (measured
478
+ // 2026-09-30: previous corpus 18/19, fresh corpus 17/19 on the same questions). Without a measurement
479
+ // (the committed bootstrap seed) nothing is filtered and the historical behaviour is unchanged.
480
+ const measuredHit = (store) => !knownHitStores || knownHitStores.has(store);
458
481
  const strata = new Map();
482
+ const staleFixtureStores = [];
483
+ const generationMissStores = [];
459
484
  rankedLegacy.forEach((entry, index) => {
460
485
  const stratum = Math.min(3, Math.floor(index * 4 / rankedLegacy.length));
486
+ const store = storeOf(entry.row);
487
+ if (!sealedPassageResolves(store)) { staleFixtureStores.push(store); return; }
488
+ if (!measuredHit(store)) { generationMissStores.push(store); return; }
461
489
  if (!strata.has(stratum)) strata.set(stratum, []);
462
490
  strata.get(stratum).push(entry);
463
491
  });
492
+ if (generationMissStores.length) {
493
+ notice(`[retrieval-canary] ${generationMissStores.length} of ${rankedLegacy.length} fixture store(s) not sampled: the generation's own `
494
+ + `recall measurement did not retrieve their sealed file (retrieval-quality debt, tracked by the recall report, not re-judged here): `
495
+ + `${ordered(generationMissStores).join(', ')}`);
496
+ }
497
+ if (staleFixtureStores.length) {
498
+ notice(`[retrieval-canary] ${staleFixtureStores.length} of ${rankedLegacy.length} fixture store(s) excluded from the legacy sample: `
499
+ + `their sealed passage no longer exists unchanged in the shipped store: ${ordered(staleFixtureStores).join(', ')}`);
500
+ }
464
501
  // Source-only releases retain the same corpus sample; the plan still seals exact release bytes.
465
502
  const samplingGeneration = coverage.kind === 'ruvnet-brain-release-coverage'
466
503
  ? coverage.corpusCoverage.coverageGeneration : coverageGeneration;
@@ -487,7 +524,7 @@ export function buildRetrievalCanaryPlan({ coverage, baseline, candidate, covera
487
524
  const observedPassageCount = passageCount ?? passages.get(store).length;
488
525
  const evidence = queryEvidence.queries[store];
489
526
  if (!evidence || expectedSources(evidence.expected).some((source) =>
490
- passages.get(store).filter((row) => row.path === source.path && digest(row) === source.passageSha256).length !== 1)) {
527
+ passages.get(store).filter((row) => row.path === source.path && passageMatches(row, source.passageSha256, contentMap)).length !== 1)) {
491
528
  throw new Error(`${store} has no sealed independent query evidence`);
492
529
  }
493
530
  return {
@@ -0,0 +1,58 @@
1
+ // How the retrieval fixture recognises "the expected passage" across store rebuilds.
2
+ //
3
+ // data/retrieval-query-evidence.json pins each expected passage by digest(row), where a row is
4
+ // { id, path, text, title }. Until 2026-09-29 the `id` was an ordinal ("2824"); the CI corpus builder
5
+ // now writes content-addressed ids ("chunk:<hash>"). The same passage — identical path, title and text —
6
+ // therefore changed digest with no content change, and every release that consumed a freshly built
7
+ // generation failed with "<store> has no sealed independent query evidence" (4.3.37 preflight, 2026-09-29).
8
+ //
9
+ // The fixture bytes are FROZEN on purpose: its sha256 is what corpus-next-seed judges a generation's
10
+ // recall report against, so editing it would make the newest generation an incompatible seed and force
11
+ // a full multi-hour rebuild. Instead this module adds a second, id-independent identity, looked up
12
+ // through a committed map (pinned digest -> content digest) derived once, mechanically, from the last
13
+ // corpus built with ordinal ids (scripts/derive-passage-content-map.mjs). The map can only ADD
14
+ // acceptance for a row whose path, title and text equal the row the fixture originally pinned.
15
+ import fs from 'node:fs';
16
+ import path from 'node:path';
17
+ import { fileURLToPath } from 'node:url';
18
+ import { digest } from './coverage-integrity.mjs';
19
+
20
+ export const CONTENT_MAP_FILE = path.resolve(path.dirname(fileURLToPath(import.meta.url)),
21
+ '..', 'data', 'retrieval-passage-content-digests.json');
22
+ export const CONTENT_MAP_KIND = 'ruvnet-brain-retrieval-passage-content-digests';
23
+ const HEX64 = /^[a-f0-9]{64}$/;
24
+
25
+ /** Digest of everything about a passage row except its (build-dependent) id. */
26
+ export function passageContentDigest(row) {
27
+ const { id: _id, ...rest } = row ?? {};
28
+ return digest(rest);
29
+ }
30
+
31
+ /** pinned digest -> content digest. A missing file is an empty map: legacy exact-digest matching only. */
32
+ export function loadContentMap(file = CONTENT_MAP_FILE) {
33
+ let parsed;
34
+ try { parsed = JSON.parse(fs.readFileSync(file, 'utf8')); } catch (error) {
35
+ if (error.code === 'ENOENT') return new Map();
36
+ throw new Error(`passage content map is unreadable: ${error.message}`);
37
+ }
38
+ if (parsed?.kind !== CONTENT_MAP_KIND || parsed.schemaVersion !== 1
39
+ || !parsed.entries || typeof parsed.entries !== 'object' || Array.isArray(parsed.entries)) {
40
+ throw new Error('passage content map is malformed');
41
+ }
42
+ const map = new Map();
43
+ for (const [pinned, content] of Object.entries(parsed.entries)) {
44
+ if (!HEX64.test(pinned) || !HEX64.test(String(content))) throw new Error('passage content map holds a non-sha256 entry');
45
+ map.set(pinned, content);
46
+ }
47
+ return map;
48
+ }
49
+
50
+ let cachedMap = null;
51
+ const defaultMap = () => (cachedMap ??= loadContentMap());
52
+
53
+ /** Does this row satisfy a fixture pin: exact digest, or (when the map knows the pin) equal content. */
54
+ export function passageMatches(row, pinnedSha256, map = defaultMap()) {
55
+ if (digest(row) === pinnedSha256) return true;
56
+ const content = map.get(pinnedSha256);
57
+ return Boolean(content) && passageContentDigest(row) === content;
58
+ }
Binary file
@@ -120,6 +120,10 @@ const STANDALONE = [
120
120
  ['self-update', 'author-run candidate rebuild; --apply is guarded by worktree-integrity.mjs and is not scheduled'],
121
121
  ['ingest-new-repos', 'author-run corpus expansion; --apply is guarded by worktree-integrity.mjs and is not scheduled'],
122
122
  ['count-chunks', 'human-run CLI — recount + restamp chunk surfaces (--check for drift); no scheduler'],
123
+ ['derive-passage-content-map', 'human-run maintainer tool — regenerates data/retrieval-passage-content-digests.json '
124
+ + 'from a corpus built with ordinal passage ids, only when the frozen fixture changes; '
125
+ + 'tests/unit/retrieval-passage-identity.test.mjs fails if the committed map stops matching the fixture, '
126
+ + 'so a stale map cannot go unnoticed and there is nothing to schedule'],
123
127
  ['brain-stamp', 'invoked by the author-run self-update.mjs candidate builder'],
124
128
  ['lesson-promote', 'human-run CLI — promotion is manual (--apply); no scheduler yet (automation is ADR-029 #4, open)'],
125
129
  ['behavioral-l1-l4', 'behavioural harness invoked by its own test file — not a product path'],