@link-assistant/hive-mind 2.15.2 → 2.16.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,24 @@
1
1
  # @link-assistant/hive-mind
2
2
 
3
+ ## 2.16.0
4
+
5
+ ### Minor Changes
6
+
7
+ - 7fe1b88: Stop Hive Mind from running itself out of memory while reporting a finished session, classify a runtime self-abort as what it is, and resume the session instead of offering to (issue #2189).
8
+
9
+ A `solve` run finished successfully, pushed its work and converted its pull request to ready — then died ten minutes later inside Hive Mind's own `--attach-logs` step with `FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory`, on a machine with 10.3 GB of RAM free. It was reported to the operator as a "forced kill", six hours late, and was never continued. Six changes close that path:
10
+
11
+ - **The sanitizer streams.** `--attach-logs` read a 134 MB log into one JS string, sanitized it (full copy), escaped it (full copy), built a comment it was never going to post, then sanitized the raw content again for the upload. `src/log-sanitize-stream.lib.mjs` sanitizes a log file into a destination file 1 MiB at a time, holding back only what could still be part of a token — a partial line, an unterminated PEM block, a trailing wrapped-base64 run — with a hard cap on the hold. `attachLogToGitHub` now picks the publication route from `logStats.size` **before** reading anything, and the content is sanitized exactly once.
12
+ - **A heap-capped worker behind it.** `sanitizeLogFileToFileBounded` routes logs at or above 16 MiB into a `worker_threads` worker with `resourceLimits.maxOldGenerationSizeMb = 512`, so a residual unbounded allocation costs one thread and one failed log upload rather than the working session. All four whole-file sanitize sites use it: the `--attach-logs` upload, the `/log` Telegram command, the GitHub error reporter and development-log collection.
13
+ - **A V8 self-abort is out of memory.** `docker inspect` said `OOMKilled=false` and cgroup `memory.events` said `oom_kill 0` — both correct, and both blind to a process that stops at its own ~2.1 GB heap cap. `FATAL_MEMORY_PATTERNS` / `findFatalMemoryMarker` recognise the fatal lines Node/V8, Rust, Go and C++ print on their way out, and an abnormal exit whose log carries one is `KILL_CAUSE_OUT_OF_MEMORY` regardless of cgroup counters. A clean exit is never upgraded, so a log that merely quotes the phrase cannot manufacture a diagnosis.
14
+ - **`--on-session-kill` defaults to `resume`.** A killed session with a recoverable tool session id now continues on its own instead of waiting for a human to paste a command; `report` and `HIVE_MIND_ON_SESSION_KILL` still opt out, and `--session-kill-resume-attempts` bounds it.
15
+ - **A killed session reaches a handled state.** The same dead session was re-detected on every poll — re-resolving the pull request, re-scanning the 134 MB log and re-sending the notification each cycle, with bot RSS climbing 1.78 → 1.84 GB against a ~2 GB cap. `src/session-completion-state.lib.mjs` adds a persisted handled latch and caches the recovered tool session id on the session record, so the log is scanned at most once and per-cycle work is no longer O(log size).
16
+ - **No unbounded log read is left.** Every artifact whose size is decided by how long an AI ran — `/log`, `/terminal_watch`, development-log collection, the error reporter, Claude session accounting and transcript repair, Codex's last-message artifact, and every session-monitor read — is streamed or reads a bounded head/tail. The `require-sanitized-output` ESLint rule learns the streaming sanitizers so a streamed artifact still counts as provably sanitized.
17
+
18
+ Telemetry was extended so the next incident is diagnosable: resource snapshots now carry the V8 heap used, total, external and `heap_size_limit` with the used share of it, warn at 85%, and `attachLogToGitHub` brackets itself with `log_upload_start` / `log_upload_end` samples. `describeKillCause` classifies an abnormal exit with a heap at or above 90% of its limit as out of memory even when the fatal line was lost with the truncated tail.
19
+
20
+ Full analysis — timeline, root causes, and the requirement-by-requirement plan — is in `docs/case-studies/issue-2189/`. Two upstream issues were filed from it: `link-foundation/start#165` (a V8 self-abort is reported as `oomKilled=false` with no memory signal in `--status`) and `link-foundation/start#164` (command argv is flattened with `join(' ')`, so quoted arguments are re-parsed by the inner shell).
21
+
3
22
  ## 2.15.2
4
23
 
5
24
  ### Patch Changes
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@link-assistant/hive-mind",
3
- "version": "2.15.2",
3
+ "version": "2.16.0",
4
4
  "description": "AI-powered issue solver and hive mind for collaborative problem solving",
5
5
  "main": "src/hive.mjs",
6
6
  "type": "module",
@@ -12,7 +12,7 @@
12
12
  * @see https://github.com/link-assistant/hive-mind/issues/1927
13
13
  */
14
14
 
15
- import { RESOURCE_PHASE_BOT_HEARTBEAT, captureResourceSnapshot, summarizeResourceSnapshot } from './solve.resource-diagnostics.lib.mjs';
15
+ import { RESOURCE_PHASE_BOT_HEARTBEAT, captureResourceSnapshot, formatHeapUsage, isHeapUnderPressure, summarizeResourceSnapshot } from './solve.resource-diagnostics.lib.mjs';
16
16
 
17
17
  const DEFAULT_HEARTBEAT_INTERVAL_MS = 60 * 1000;
18
18
 
@@ -51,6 +51,13 @@ export function createHeartbeat({ logger, getActiveSessionCount, intervalMs = DE
51
51
  uptimeSec: Math.floor(processImpl.uptime()),
52
52
  resources,
53
53
  });
54
+ // Issue #2189: the bot walked from 1.78 GB to 1.84 GB of RSS against its
55
+ // own ~2 GB heap cap while re-reporting one dead session, and nothing in
56
+ // the log said so until it died (#733). The heartbeat is the one place
57
+ // that samples the bot itself, so it is where the warning belongs.
58
+ if (isHeapUnderPressure(resources?.memory) && typeof logger.warn === 'function') {
59
+ logger.warn(`Bot V8 heap is under pressure: ${formatHeapUsage(resources.memory)} — the process will abort with "JavaScript heap out of memory" if it keeps growing`, { heap: resources.memory });
60
+ }
54
61
  } catch {
55
62
  /* heartbeat must never crash the bot */
56
63
  }
@@ -74,6 +74,58 @@ export const describeChildExit = ({ command, code = null, signal = null }) => {
74
74
  */
75
75
  export const isLikelyOutOfMemoryExit = ({ code = null, signal = null }) => signal === 'SIGABRT' || signal === 'SIGKILL' || (signal === null && code === 134);
76
76
 
77
+ /**
78
+ * Fatal lines a runtime prints when it exhausts its *own* heap.
79
+ *
80
+ * Issue #2189: a session died of `FATAL ERROR: Reached heap limit Allocation
81
+ * failed - JavaScript heap out of memory` and was reported to the user as a
82
+ * "forced kill … memory (10.3 GB of 11.7 GB RAM available)". Both statements
83
+ * were individually true: V8 stopped at its own ~2 GB old-space cap long before
84
+ * the machine or the container cgroup felt any pressure, so `docker inspect`
85
+ * said `OOMKilled=false` and `/sys/fs/cgroup/memory.events` said `oom_kill=0`.
86
+ * Nothing outside the process can observe a runtime self-abort — the only
87
+ * evidence is the text the runtime printed on its way out, which was sitting in
88
+ * the log the diagnostics were already reading.
89
+ *
90
+ * The patterns are deliberately specific (a bare "out of memory" also appears in
91
+ * Hive Mind's own diagnostic wording, which ends up in the same logs). Hive Mind
92
+ * spawns more than Node, so the other runtimes it drives are covered too.
93
+ */
94
+ export const FATAL_MEMORY_PATTERNS = [
95
+ { id: 'v8-heap-limit', runtime: 'Node.js/V8', pattern: /FATAL ERROR:[^\n]*Reached heap limit/ },
96
+ { id: 'v8-ineffective-mark-compacts', runtime: 'Node.js/V8', pattern: /FATAL ERROR:[^\n]*Ineffective mark-compacts near heap limit/ },
97
+ { id: 'v8-heap-out-of-memory', runtime: 'Node.js/V8', pattern: /JavaScript heap out of memory/ },
98
+ { id: 'v8-last-few-gcs', runtime: 'Node.js/V8', pattern: /<--- Last few GCs --->/ },
99
+ { id: 'v8-array-buffer-allocation', runtime: 'Node.js/V8', pattern: /Array buffer allocation failed/ },
100
+ { id: 'rust-allocation-failed', runtime: 'Rust', pattern: /memory allocation of \d+ bytes failed/ },
101
+ { id: 'go-runtime-out-of-memory', runtime: 'Go', pattern: /fatal error: runtime: out of memory/ },
102
+ { id: 'cpp-bad-alloc', runtime: 'C/C++', pattern: /std::bad_alloc/ },
103
+ ];
104
+
105
+ /**
106
+ * Find the first runtime self-abort marker in a piece of log text.
107
+ *
108
+ * Callers must only treat a hit as a cause when the process actually ended
109
+ * abnormally — the marker upgrades an existing kill to "out of memory", it never
110
+ * invents one, so an unrelated log that merely quotes the string cannot turn a
111
+ * healthy run into a reported crash.
112
+ *
113
+ * @param {string|null} text - Log text (a tail is enough; the marker is printed last)
114
+ * @returns {{id: string, runtime: string, line: string}|null}
115
+ */
116
+ export const findFatalMemoryMarker = text => {
117
+ if (!text || typeof text !== 'string') return null;
118
+ for (const { id, runtime, pattern } of FATAL_MEMORY_PATTERNS) {
119
+ const match = pattern.exec(text);
120
+ if (!match) continue;
121
+ const lineStart = text.lastIndexOf('\n', match.index) + 1;
122
+ const lineEndIndex = text.indexOf('\n', match.index);
123
+ const line = text.slice(lineStart, lineEndIndex < 0 ? undefined : lineEndIndex).trim();
124
+ return { id, runtime, line: line.length > 300 ? `${line.slice(0, 300)}…` : line };
125
+ }
126
+ return null;
127
+ };
128
+
77
129
  /**
78
130
  * Wire `close`/`error` handlers that never lose a signal.
79
131
  *
@@ -104,4 +156,4 @@ export const attachChildExitHandlers = ({ child, command, label, errorLabel = la
104
156
  });
105
157
  };
106
158
 
107
- export default { describeChildExit, isLikelyOutOfMemoryExit, attachChildExitHandlers };
159
+ export default { describeChildExit, isLikelyOutOfMemoryExit, findFatalMemoryMarker, attachChildExitHandlers };
@@ -22,6 +22,7 @@ import Decimal from 'decimal.js-light';
22
22
  import { accumulateModelUsage, createEmptySubSessionUsage, getRawRequestInputTokens, mergeResultModelUsage } from './claude.budget-stats.lib.mjs';
23
23
  import { calculateModelCost } from './claude.cost.lib.mjs';
24
24
  import { fetchModelInfo } from './model-info.lib.mjs';
25
+ import { forEachLogLine } from './log-bounded-read.lib.mjs';
25
26
 
26
27
  export const calculateSessionTokens = async (sessionId, tempDir, resultModelUsage = null, options = {}) => {
27
28
  const homeDir = options.homeDir || os.homedir();
@@ -43,10 +44,12 @@ export const calculateSessionTokens = async (sessionId, tempDir, resultModelUsag
43
44
  let currentSubSession = createEmptySubSessionUsage();
44
45
  const compactifications = [];
45
46
  try {
46
- const fileContent = await fs.readFile(sessionFile, 'utf8');
47
- const lines = fileContent.trim().split('\n');
48
- for (const line of lines) {
49
- if (!line.trim()) continue;
47
+ // Issue #2189: read the transcript one record at a time. A long Claude
48
+ // session produces a JSONL of unbounded size, and `readFile(...).split('\n')`
49
+ // held the whole file *and* the array of its lines before the first entry
50
+ // was priced — a full-file allocation just to sum token counters.
51
+ await forEachLogLine(sessionFile, line => {
52
+ if (!line.trim()) return;
50
53
  try {
51
54
  const entry = JSON.parse(line);
52
55
  if (entry.type === 'system' && entry.subtype === 'compact_boundary') {
@@ -59,7 +62,7 @@ export const calculateSessionTokens = async (sessionId, tempDir, resultModelUsag
59
62
  trigger: entry.compactMetadata?.trigger || 'unknown',
60
63
  });
61
64
  currentSubSession = createEmptySubSessionUsage();
62
- continue;
65
+ return;
63
66
  }
64
67
  if (entry.message && entry.message.usage && entry.message.model) {
65
68
  // Issue #1501: Skip duplicate JSONL entries (same message ID = same API response)
@@ -67,7 +70,7 @@ export const calculateSessionTokens = async (sessionId, tempDir, resultModelUsag
67
70
  if (msgId) {
68
71
  if (seenMessageIds.has(msgId)) {
69
72
  duplicateCount++;
70
- continue;
73
+ return;
71
74
  }
72
75
  seenMessageIds.add(msgId);
73
76
  }
@@ -100,9 +103,8 @@ export const calculateSessionTokens = async (sessionId, tempDir, resultModelUsag
100
103
  }
101
104
  } catch {
102
105
  // Skip lines that aren't valid JSON
103
- continue;
104
106
  }
105
- }
107
+ });
106
108
  if (currentSubSession.messageCount > 0) {
107
109
  subSessions.push(currentSubSession);
108
110
  }
@@ -25,6 +25,8 @@ import { promises as fs } from 'fs';
25
25
  import os from 'os';
26
26
  import path from 'path';
27
27
 
28
+ import { fileEndsWithNewline, forEachLogLine } from './log-bounded-read.lib.mjs';
29
+
28
30
  /**
29
31
  * Resolve the on-disk session transcript path for a Claude Code session. Claude Code stores each
30
32
  * session as `~/.claude/projects/<cwd-with-slashes-as-dashes>/<sessionId>.jsonl` (mirrors the
@@ -52,6 +54,36 @@ const isCorruptedThinkingBlock = block => {
52
54
  return false;
53
55
  };
54
56
 
57
+ /**
58
+ * Repair one transcript record.
59
+ *
60
+ * Returns the line to write back (unchanged unless a corrupted block was
61
+ * dropped), how many corrupted blocks it dropped, and whether the line counted
62
+ * as a scanned message line. Keeping the decision in one pure function lets the
63
+ * repair stream the file twice — count, then rewrite — with identical results.
64
+ *
65
+ * @param {string} line - One raw JSONL record
66
+ * @returns {{text: string, removed: number, scanned: number}}
67
+ */
68
+ const repairTranscriptLine = line => {
69
+ if (!line.trim()) return { text: line, removed: 0, scanned: 0 };
70
+ let entry;
71
+ try {
72
+ entry = JSON.parse(line);
73
+ } catch {
74
+ return { text: line, removed: 0, scanned: 1 }; // preserve anything we can't parse verbatim
75
+ }
76
+ const content = entry?.message?.content;
77
+ if (!Array.isArray(content)) return { text: line, removed: 0, scanned: 1 };
78
+ const corrupted = content.filter(isCorruptedThinkingBlock).length;
79
+ if (corrupted === 0) return { text: line, removed: 0, scanned: 1 };
80
+ const cleaned = content.filter(b => !isCorruptedThinkingBlock(b));
81
+ // Never leave an assistant message with an empty content array (invalid for the API).
82
+ if (cleaned.length === 0) return { text: line, removed: 0, scanned: 1 };
83
+ entry.message.content = cleaned;
84
+ return { text: JSON.stringify(entry), removed: corrupted, scanned: 1 };
85
+ };
86
+
55
87
  /**
56
88
  * Strip corrupted (empty-text) thinking blocks from a Claude Code session transcript so the session
57
89
  * can be resumed. Conservative and side-effect-safe:
@@ -75,48 +107,27 @@ export const repairCorruptedThinkingBlocks = async ({ tempDir, sessionId, homeDi
75
107
  }
76
108
  const sessionFile = resolveSessionTranscriptPath(tempDir, sessionId, homeDir);
77
109
  result.sessionFile = sessionFile;
78
- let fileContent;
110
+ let sessionStat;
79
111
  try {
80
- fileContent = await fs.readFile(sessionFile, 'utf8');
112
+ sessionStat = await fs.stat(sessionFile);
81
113
  } catch {
82
114
  // No transcript on disk (e.g. fresh run never persisted, or path mismatch) — nothing to repair.
83
115
  return { ...result, reason: 'session transcript not found' };
84
116
  }
85
117
 
86
118
  try {
87
- const lines = fileContent.split('\n');
88
- const out = [];
119
+ // Issue #2189: a session transcript grows with the session — the captured
120
+ // incident's was 134 MB — so this is done in two streaming passes instead of
121
+ // holding the file, its array of lines and the rebuilt output in the heap at
122
+ // once. Pass 1 only counts: a transcript with nothing to repair (the common
123
+ // case) is never rewritten and never copied.
89
124
  let removedBlocks = 0;
90
125
  let scannedLines = 0;
91
- for (const line of lines) {
92
- if (!line.trim()) {
93
- out.push(line);
94
- continue;
95
- }
96
- scannedLines++;
97
- let entry;
98
- try {
99
- entry = JSON.parse(line);
100
- } catch {
101
- out.push(line); // preserve anything we can't parse verbatim
102
- continue;
103
- }
104
- const content = entry?.message?.content;
105
- if (Array.isArray(content)) {
106
- const corrupted = content.filter(isCorruptedThinkingBlock).length;
107
- if (corrupted > 0) {
108
- const cleaned = content.filter(b => !isCorruptedThinkingBlock(b));
109
- // Never leave an assistant message with an empty content array (invalid for the API).
110
- if (cleaned.length > 0) {
111
- entry.message.content = cleaned;
112
- removedBlocks += corrupted;
113
- out.push(JSON.stringify(entry));
114
- continue;
115
- }
116
- }
117
- }
118
- out.push(line);
119
- }
126
+ await forEachLogLine(sessionFile, line => {
127
+ const repaired = repairTranscriptLine(line);
128
+ scannedLines += repaired.scanned;
129
+ removedBlocks += repaired.removed;
130
+ });
120
131
 
121
132
  result.scannedLines = scannedLines;
122
133
  if (removedBlocks === 0) {
@@ -135,7 +146,27 @@ export const repairCorruptedThinkingBlocks = async ({ tempDir, sessionId, homeDi
135
146
  }
136
147
  }
137
148
 
138
- await fs.writeFile(sessionFile, out.join('\n'), 'utf8');
149
+ // Pass 2: rewrite through a sibling temp file and rename over the original,
150
+ // so an interrupted repair can never leave a half-written transcript (which
151
+ // would be worse than the corruption being repaired).
152
+ const keepTrailingNewline = await fileEndsWithNewline(sessionFile);
153
+ const tempFile = `${sessionFile}.repair-${process.pid}`;
154
+ await fs.rm(tempFile, { force: true });
155
+ const handle = await fs.open(tempFile, 'wx', sessionStat.mode & 0o777);
156
+ try {
157
+ let pendingSeparator = '';
158
+ await forEachLogLine(sessionFile, line => {
159
+ const repaired = repairTranscriptLine(line);
160
+ return handle.write(`${pendingSeparator}${repaired.text}`, null, 'utf8').then(() => {
161
+ pendingSeparator = '\n';
162
+ });
163
+ });
164
+ if (keepTrailingNewline) await handle.write('\n', null, 'utf8');
165
+ } finally {
166
+ await handle.close();
167
+ }
168
+ await fs.rename(tempFile, sessionFile);
169
+
139
170
  result.repaired = true;
140
171
  result.removedBlocks = removedBlocks;
141
172
  await log(`🩹 Repaired session transcript: stripped ${removedBlocks} corrupted thinking block(s) from ${scannedLines} message line(s) (Issue #1834). Backup: ${backupFile}`, { verbose: true });
package/src/codex.lib.mjs CHANGED
@@ -49,10 +49,15 @@ import Decimal from 'decimal.js-light';
49
49
  import { ensureAiToolScratchIgnored, filterAiToolScratchFromStatus } from './ai-tool-scratch.lib.mjs';
50
50
  import { CODEX_CACHE_READ_USAGE_PATHS, CODEX_CACHE_WRITE_USAGE_PATHS, CODEX_MODEL_DIAGNOSTIC_PATHS, CODEX_REASONING_USAGE_PATHS, CODEX_USAGE_FIELD_NAMES, createCodexTokenFieldAvailability, getFirstObservedNumber, hasAnyObservedPath, hasOwnPath } from './codex.usage-fields.lib.mjs';
51
51
  const CODEX_LONG_CONTEXT_PRICE_THRESHOLD = 272000;
52
+ // Issue #2189: ceiling for reading Codex's `--output-last-message` artifact. A
53
+ // final assistant message is a few hundred kilobytes at most; anything larger is
54
+ // a malfunction and must not be turned into an unbounded string.
55
+ const CODEX_LAST_MESSAGE_MAX_BYTES = 1024 * 1024;
52
56
  const getCodexExecEnv = (verbose = false) => (verbose ? { ...process.env, RUST_LOG: 'debug' } : { ...process.env });
53
57
  // Issue #2175: diagnostic-line parsing lives in its own module to keep this file
54
58
  // under the 1350-line warning threshold.
55
59
  import { parseCodexDiagnosticLine, rebuildCodexSubSessionsFromCompactifications } from './codex.diagnostics.lib.mjs';
60
+ import { readLogHeadText } from './log-bounded-read.lib.mjs'; // Issue #2189
56
61
  export const createCodexTokenUsage = requestedModelId => ({
57
62
  inputTokens: 0,
58
63
  outputTokens: 0,
@@ -891,7 +896,12 @@ export const executeCodexCommand = async params => {
891
896
  let lastMessageFromFile = null;
892
897
  let lastMessageReadError = null;
893
898
  try {
894
- lastMessageFromFile = (await fs.readFile(lastMessageFile, 'utf8')).trim();
899
+ // Issue #2189: this file holds Codex's final assistant message, but its
900
+ // size is decided by the tool, not by us. Size it first so a runaway or
901
+ // corrupted artifact cannot become an unbounded string in a process that
902
+ // has just finished a long run.
903
+ const { size } = await fs.stat(lastMessageFile);
904
+ lastMessageFromFile = size > CODEX_LAST_MESSAGE_MAX_BYTES ? `${(await readLogHeadText(lastMessageFile, { maxBytes: CODEX_LAST_MESSAGE_MAX_BYTES })).trim()}\n…[last message truncated: ${size} bytes on disk, see ${lastMessageFile}]` : (await fs.readFile(lastMessageFile, 'utf8')).trim();
895
905
  } catch (readError) {
896
906
  lastMessageReadError = readError;
897
907
  }
@@ -2,6 +2,8 @@ import fs from 'node:fs/promises';
2
2
  import os from 'node:os';
3
3
  import path from 'node:path';
4
4
  import { sanitizeForPublication } from './token-sanitization.lib.mjs';
5
+ import { findResidualCredentialBlock } from './log-sanitize-stream.lib.mjs';
6
+ import { sanitizeLogFileToFileBounded } from './log-sanitize-worker.lib.mjs';
5
7
 
6
8
  const sanitizePathSegment = (value, fallback) => {
7
9
  const raw = value === null || value === undefined || value === '' ? fallback : String(value);
@@ -94,13 +96,25 @@ const writePrivatePublicationFile = async (destinationPath, content) => {
94
96
  await fs.chmod(destinationPath, 0o600);
95
97
  };
96
98
 
99
+ // Issue #2189: `sanitizeLogFileToFileBounded` creates its destination exclusively
100
+ // (`wx`) so a pre-planted symlink cannot be followed. Collection may run more
101
+ // than once for the same session directory, so drop a previous artifact first —
102
+ // unlink-then-O_EXCL keeps the symlink guarantee that a plain truncate loses.
103
+ const sanitizeIntoPublicationFile = async ({ sourcePath, destinationPath, startByte = 0, endByte = null }) => {
104
+ await fs.rm(destinationPath, { force: true });
105
+ return sanitizeLogFileToFileBounded({ sourcePath, destPath: destinationPath, startByte, endByte });
106
+ };
107
+
97
108
  const copyIfExists = async ({ sourcePath, destinationPath }) => {
98
109
  if (!(await fileExists(sourcePath))) return false;
99
110
  // Raw local audit sources remain available to the operator but must not be
100
111
  // group/world-readable. Only the sanitized copy enters the repository.
101
112
  await fs.chmod(sourcePath, 0o600);
102
- const content = await fs.readFile(sourcePath, 'utf8');
103
- await writePrivatePublicationFile(destinationPath, content);
113
+ // Issue #2189: transcripts are as large as the run that produced them (the
114
+ // captured incident had a 134 MB one). Sanitize source → destination block by
115
+ // block instead of holding the file, its sanitized twin and the sanitizer's
116
+ // own working copy in the heap at once.
117
+ await sanitizeIntoPublicationFile({ sourcePath, destinationPath });
104
118
  return true;
105
119
  };
106
120
 
@@ -140,8 +154,7 @@ const copyLogSlice = async ({ logFile, destinationPath, logStartByte = 0 }) => {
140
154
  await writePrivatePublicationFile(destinationPath, '');
141
155
  return { logStartByte: start, logEndByte: stat.size };
142
156
  }
143
- const bytes = await fs.readFile(logFile);
144
- await writePrivatePublicationFile(destinationPath, bytes.subarray(start, stat.size).toString('utf8'));
157
+ await sanitizeIntoPublicationFile({ sourcePath: logFile, destinationPath, startByte: start, endByte: stat.size });
145
158
  return { logStartByte: start, logEndByte: stat.size };
146
159
  };
147
160
 
@@ -279,9 +292,11 @@ const verifyDevelopmentLogDirectory = async directoryPath => {
279
292
  if (!entry.isFile()) continue;
280
293
  const parentPath = entry.parentPath || entry.path;
281
294
  const filePath = path.join(parentPath, entry.name);
282
- const exactBytes = await fs.readFile(filePath, 'utf8');
283
- const rescanned = await sanitizeForPublication(exactBytes);
284
- if (rescanned !== exactBytes) {
295
+ // Issue #2189: rescan block by block. Reading each artifact back whole made
296
+ // the verification cost as much heap as the artifact — on top of the copy
297
+ // that had just been written.
298
+ const residual = await findResidualCredentialBlock(filePath);
299
+ if (residual) {
285
300
  throw new Error('Development-log publication rescan found residual credential material.');
286
301
  }
287
302
  await fs.chmod(filePath, 0o600);
@@ -9,6 +9,8 @@ import { createInterface } from 'readline';
9
9
  import { log, cleanErrorMessage, getAbsoluteLogPath } from './lib.mjs';
10
10
  import { reportError, isSentryEnabled } from './sentry.lib.mjs';
11
11
  import { sanitizeForPublication, writeSanitizedPublicationFile } from './token-sanitization.lib.mjs';
12
+ import { sanitizeLogFileToFileBounded } from './log-sanitize-worker.lib.mjs';
13
+ import { readLogTailText } from './log-bounded-read.lib.mjs';
12
14
 
13
15
  if (typeof globalThis.use === 'undefined') {
14
16
  await ensureUseM();
@@ -107,8 +109,87 @@ const createSecretGist = async (logContent, filename) => {
107
109
  return null;
108
110
  };
109
111
 
112
+ /**
113
+ * Upload a log FILE as a secret gist without ever holding it in memory.
114
+ *
115
+ * Issue #2189: the error reporter runs when the process is already in trouble —
116
+ * frequently because it just exhausted its heap. Reading the log to sanitize it
117
+ * (`readFile` + `sanitizeForPublication` + write = three full copies) is the one
118
+ * thing that must not happen there.
119
+ *
120
+ * @param {string} logFilePath - Log to upload
121
+ * @param {string} filename - Name for the gist file
122
+ * @returns {Promise<string|null>} Gist URL, or null when the upload failed
123
+ */
124
+ const createSecretGistFromFile = async (logFilePath, filename) => {
125
+ const tempFile = `/tmp/${filename}`;
126
+ try {
127
+ await sanitizeLogFileToFileBounded({ sourcePath: logFilePath, destPath: tempFile });
128
+ const result = await $`gh gist create ${tempFile} --secret --desc "Error log for hive-mind"`;
129
+ if (result.exitCode === 0) {
130
+ return result.stdout.toString().trim();
131
+ }
132
+ } catch (error) {
133
+ reportError(error, {
134
+ context: 'create_secret_gist',
135
+ operation: 'gh_gist_create',
136
+ });
137
+ } finally {
138
+ await fs.unlink(tempFile).catch(() => {});
139
+ }
140
+ return null;
141
+ };
142
+
143
+ /**
144
+ * Format a log FILE for an issue body, choosing the attachment method from the
145
+ * file's size before reading any of it (issue #2189).
146
+ *
147
+ * Only the inline branch — by definition below GitHub's 60 kB issue-body limit —
148
+ * ever reads log content, and the truncated fallback reads a bounded tail.
149
+ *
150
+ * @param {string} logFilePath - Path to the log file
151
+ * @returns {Promise<{method: string, content: string}>}
152
+ */
153
+ export const formatLogFileForIssue = async logFilePath => {
154
+ const { size } = await fs.stat(logFilePath);
155
+
156
+ if (size < GITHUB_ISSUE_BODY_MAX_SIZE) {
157
+ const logContent = await fs.readFile(logFilePath, 'utf8');
158
+ return {
159
+ method: 'inline',
160
+ content: `\`\`\`\n${logContent}\n\`\`\``,
161
+ };
162
+ }
163
+
164
+ if (size < GITHUB_FILE_MAX_SIZE) {
165
+ return {
166
+ method: 'file',
167
+ content: `Log file is too large to include inline. Please see the attached log file.\n\nLog file path: \`${logFilePath}\``,
168
+ };
169
+ }
170
+
171
+ const gistUrl = await createSecretGistFromFile(logFilePath, `hive-mind-error-${Date.now()}.log`);
172
+ if (gistUrl) {
173
+ return {
174
+ method: 'gist',
175
+ content: `Log file is too large for inline attachment.\n\n📄 View full log: ${gistUrl}`,
176
+ };
177
+ }
178
+
179
+ const tail = await readLogTailText(logFilePath, { maxBytes: 5000 });
180
+ return {
181
+ method: 'truncated',
182
+ content: `Log file is too large. Showing last 5000 characters:\n\n\`\`\`\n${tail}\n\`\`\``,
183
+ };
184
+ };
185
+
110
186
  /**
111
187
  * Format log content for issue body
188
+ *
189
+ * Prefer {@link formatLogFileForIssue} when the log is a file on disk: this
190
+ * variant needs the whole log as a string, which is exactly what issue #2189
191
+ * removed from the publication path.
192
+ *
112
193
  * @param {string} logContent - Log file content
113
194
  * @param {string} logFilePath - Path to log file
114
195
  * @returns {Promise<Object>} Object with formatted content and attachment method
@@ -202,8 +283,9 @@ export const createIssueForError = async options => {
202
283
 
203
284
  if (logFile) {
204
285
  try {
205
- const logContent = await fs.readFile(logFile, 'utf8');
206
- const { method, content } = await formatLogForIssue(logContent, logFile);
286
+ // Issue #2189: pick the attachment method from the file size first; a
287
+ // log too large for the issue body is never read into memory here.
288
+ const { method, content } = await formatLogFileForIssue(logFile);
207
289
 
208
290
  issueBody += `### Log File\n\n${content}\n\n`;
209
291
  await log(`📄 Log attached via: ${method}`);