@link-assistant/hive-mind 2.15.2 → 2.16.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +19 -0
- package/package.json +1 -1
- package/src/bot-lifecycle.lib.mjs +8 -1
- package/src/child-exit.lib.mjs +53 -1
- package/src/claude.session-tokens.lib.mjs +10 -8
- package/src/claude.session-transcript-repair.lib.mjs +65 -34
- package/src/codex.lib.mjs +11 -1
- package/src/development-log.lib.mjs +22 -7
- package/src/github-error-reporter.lib.mjs +84 -2
- package/src/github.lib.mjs +212 -152
- package/src/log-bounded-read.lib.mjs +411 -0
- package/src/log-sanitize-stream.lib.mjs +267 -0
- package/src/log-sanitize-worker-entry.mjs +31 -0
- package/src/log-sanitize-worker.lib.mjs +186 -0
- package/src/log-upload.lib.mjs +16 -4
- package/src/session-completion-state.lib.mjs +124 -0
- package/src/session-kill-diagnostics.lib.mjs +73 -10
- package/src/session-kill-policy.lib.mjs +18 -7
- package/src/session-kill-resume.lib.mjs +5 -2
- package/src/session-monitor.lib.mjs +132 -9
- package/src/session-store.lib.mjs +15 -1
- package/src/solve.config.lib.mjs +5 -2
- package/src/solve.resource-diagnostics.lib.mjs +71 -4
- package/src/telegram-log-command.lib.mjs +7 -3
- package/src/telegram-terminal-watch-command.lib.mjs +9 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,24 @@
|
|
|
1
1
|
# @link-assistant/hive-mind
|
|
2
2
|
|
|
3
|
+
## 2.16.0
|
|
4
|
+
|
|
5
|
+
### Minor Changes
|
|
6
|
+
|
|
7
|
+
- 7fe1b88: Stop Hive Mind from running itself out of memory while reporting a finished session, classify a runtime self-abort as what it is, and resume the session instead of offering to (issue #2189).
|
|
8
|
+
|
|
9
|
+
A `solve` run finished successfully, pushed its work and converted its pull request to ready — then died ten minutes later inside Hive Mind's own `--attach-logs` step with `FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory`, on a machine with 10.3 GB of RAM free. It was reported to the operator as a "forced kill", six hours late, and was never continued. Six changes close that path:
|
|
10
|
+
|
|
11
|
+
- **The sanitizer streams.** `--attach-logs` read a 134 MB log into one JS string, sanitized it (full copy), escaped it (full copy), built a comment it was never going to post, then sanitized the raw content again for the upload. `src/log-sanitize-stream.lib.mjs` sanitizes a log file into a destination file 1 MiB at a time, holding back only what could still be part of a token — a partial line, an unterminated PEM block, a trailing wrapped-base64 run — with a hard cap on the hold. `attachLogToGitHub` now picks the publication route from `logStats.size` **before** reading anything, and the content is sanitized exactly once.
|
|
12
|
+
- **A heap-capped worker behind it.** `sanitizeLogFileToFileBounded` routes logs at or above 16 MiB into a `worker_threads` worker with `resourceLimits.maxOldGenerationSizeMb = 512`, so a residual unbounded allocation costs one thread and one failed log upload rather than the working session. All four whole-file sanitize sites use it: the `--attach-logs` upload, the `/log` Telegram command, the GitHub error reporter and development-log collection.
|
|
13
|
+
- **A V8 self-abort is out of memory.** `docker inspect` said `OOMKilled=false` and cgroup `memory.events` said `oom_kill 0` — both correct, and both blind to a process that stops at its own ~2.1 GB heap cap. `FATAL_MEMORY_PATTERNS` / `findFatalMemoryMarker` recognise the fatal lines Node/V8, Rust, Go and C++ print on their way out, and an abnormal exit whose log carries one is `KILL_CAUSE_OUT_OF_MEMORY` regardless of cgroup counters. A clean exit is never upgraded, so a log that merely quotes the phrase cannot manufacture a diagnosis.
|
|
14
|
+
- **`--on-session-kill` defaults to `resume`.** A killed session with a recoverable tool session id now continues on its own instead of waiting for a human to paste a command; `report` and `HIVE_MIND_ON_SESSION_KILL` still opt out, and `--session-kill-resume-attempts` bounds it.
|
|
15
|
+
- **A killed session reaches a handled state.** The same dead session was re-detected on every poll — re-resolving the pull request, re-scanning the 134 MB log and re-sending the notification each cycle, with bot RSS climbing 1.78 → 1.84 GB against a ~2 GB cap. `src/session-completion-state.lib.mjs` adds a persisted handled latch and caches the recovered tool session id on the session record, so the log is scanned at most once and per-cycle work is no longer O(log size).
|
|
16
|
+
- **No unbounded log read is left.** Every artifact whose size is decided by how long an AI ran — `/log`, `/terminal_watch`, development-log collection, the error reporter, Claude session accounting and transcript repair, Codex's last-message artifact, and every session-monitor read — is streamed or reads a bounded head/tail. The `require-sanitized-output` ESLint rule learns the streaming sanitizers so a streamed artifact still counts as provably sanitized.
|
|
17
|
+
|
|
18
|
+
Telemetry was extended so the next incident is diagnosable: resource snapshots now carry the V8 heap used, total, external and `heap_size_limit` with the used share of it, warn at 85%, and `attachLogToGitHub` brackets itself with `log_upload_start` / `log_upload_end` samples. `describeKillCause` classifies an abnormal exit with a heap at or above 90% of its limit as out of memory even when the fatal line was lost with the truncated tail.
|
|
19
|
+
|
|
20
|
+
Full analysis — timeline, root causes, and the requirement-by-requirement plan — is in `docs/case-studies/issue-2189/`. Two upstream issues were filed from it: `link-foundation/start#165` (a V8 self-abort is reported as `oomKilled=false` with no memory signal in `--status`) and `link-foundation/start#164` (command argv is flattened with `join(' ')`, so quoted arguments are re-parsed by the inner shell).
|
|
21
|
+
|
|
3
22
|
## 2.15.2
|
|
4
23
|
|
|
5
24
|
### Patch Changes
|
package/package.json
CHANGED
|
@@ -12,7 +12,7 @@
|
|
|
12
12
|
* @see https://github.com/link-assistant/hive-mind/issues/1927
|
|
13
13
|
*/
|
|
14
14
|
|
|
15
|
-
import { RESOURCE_PHASE_BOT_HEARTBEAT, captureResourceSnapshot, summarizeResourceSnapshot } from './solve.resource-diagnostics.lib.mjs';
|
|
15
|
+
import { RESOURCE_PHASE_BOT_HEARTBEAT, captureResourceSnapshot, formatHeapUsage, isHeapUnderPressure, summarizeResourceSnapshot } from './solve.resource-diagnostics.lib.mjs';
|
|
16
16
|
|
|
17
17
|
const DEFAULT_HEARTBEAT_INTERVAL_MS = 60 * 1000;
|
|
18
18
|
|
|
@@ -51,6 +51,13 @@ export function createHeartbeat({ logger, getActiveSessionCount, intervalMs = DE
|
|
|
51
51
|
uptimeSec: Math.floor(processImpl.uptime()),
|
|
52
52
|
resources,
|
|
53
53
|
});
|
|
54
|
+
// Issue #2189: the bot walked from 1.78 GB to 1.84 GB of RSS against its
|
|
55
|
+
// own ~2 GB heap cap while re-reporting one dead session, and nothing in
|
|
56
|
+
// the log said so until it died (#733). The heartbeat is the one place
|
|
57
|
+
// that samples the bot itself, so it is where the warning belongs.
|
|
58
|
+
if (isHeapUnderPressure(resources?.memory) && typeof logger.warn === 'function') {
|
|
59
|
+
logger.warn(`Bot V8 heap is under pressure: ${formatHeapUsage(resources.memory)} — the process will abort with "JavaScript heap out of memory" if it keeps growing`, { heap: resources.memory });
|
|
60
|
+
}
|
|
54
61
|
} catch {
|
|
55
62
|
/* heartbeat must never crash the bot */
|
|
56
63
|
}
|
package/src/child-exit.lib.mjs
CHANGED
|
@@ -74,6 +74,58 @@ export const describeChildExit = ({ command, code = null, signal = null }) => {
|
|
|
74
74
|
*/
|
|
75
75
|
export const isLikelyOutOfMemoryExit = ({ code = null, signal = null }) => signal === 'SIGABRT' || signal === 'SIGKILL' || (signal === null && code === 134);
|
|
76
76
|
|
|
77
|
+
/**
|
|
78
|
+
* Fatal lines a runtime prints when it exhausts its *own* heap.
|
|
79
|
+
*
|
|
80
|
+
* Issue #2189: a session died of `FATAL ERROR: Reached heap limit Allocation
|
|
81
|
+
* failed - JavaScript heap out of memory` and was reported to the user as a
|
|
82
|
+
* "forced kill … memory (10.3 GB of 11.7 GB RAM available)". Both statements
|
|
83
|
+
* were individually true: V8 stopped at its own ~2 GB old-space cap long before
|
|
84
|
+
* the machine or the container cgroup felt any pressure, so `docker inspect`
|
|
85
|
+
* said `OOMKilled=false` and `/sys/fs/cgroup/memory.events` said `oom_kill=0`.
|
|
86
|
+
* Nothing outside the process can observe a runtime self-abort — the only
|
|
87
|
+
* evidence is the text the runtime printed on its way out, which was sitting in
|
|
88
|
+
* the log the diagnostics were already reading.
|
|
89
|
+
*
|
|
90
|
+
* The patterns are deliberately specific (a bare "out of memory" also appears in
|
|
91
|
+
* Hive Mind's own diagnostic wording, which ends up in the same logs). Hive Mind
|
|
92
|
+
* spawns more than Node, so the other runtimes it drives are covered too.
|
|
93
|
+
*/
|
|
94
|
+
export const FATAL_MEMORY_PATTERNS = [
|
|
95
|
+
{ id: 'v8-heap-limit', runtime: 'Node.js/V8', pattern: /FATAL ERROR:[^\n]*Reached heap limit/ },
|
|
96
|
+
{ id: 'v8-ineffective-mark-compacts', runtime: 'Node.js/V8', pattern: /FATAL ERROR:[^\n]*Ineffective mark-compacts near heap limit/ },
|
|
97
|
+
{ id: 'v8-heap-out-of-memory', runtime: 'Node.js/V8', pattern: /JavaScript heap out of memory/ },
|
|
98
|
+
{ id: 'v8-last-few-gcs', runtime: 'Node.js/V8', pattern: /<--- Last few GCs --->/ },
|
|
99
|
+
{ id: 'v8-array-buffer-allocation', runtime: 'Node.js/V8', pattern: /Array buffer allocation failed/ },
|
|
100
|
+
{ id: 'rust-allocation-failed', runtime: 'Rust', pattern: /memory allocation of \d+ bytes failed/ },
|
|
101
|
+
{ id: 'go-runtime-out-of-memory', runtime: 'Go', pattern: /fatal error: runtime: out of memory/ },
|
|
102
|
+
{ id: 'cpp-bad-alloc', runtime: 'C/C++', pattern: /std::bad_alloc/ },
|
|
103
|
+
];
|
|
104
|
+
|
|
105
|
+
/**
|
|
106
|
+
* Find the first runtime self-abort marker in a piece of log text.
|
|
107
|
+
*
|
|
108
|
+
* Callers must only treat a hit as a cause when the process actually ended
|
|
109
|
+
* abnormally — the marker upgrades an existing kill to "out of memory", it never
|
|
110
|
+
* invents one, so an unrelated log that merely quotes the string cannot turn a
|
|
111
|
+
* healthy run into a reported crash.
|
|
112
|
+
*
|
|
113
|
+
* @param {string|null} text - Log text (a tail is enough; the marker is printed last)
|
|
114
|
+
* @returns {{id: string, runtime: string, line: string}|null}
|
|
115
|
+
*/
|
|
116
|
+
export const findFatalMemoryMarker = text => {
|
|
117
|
+
if (!text || typeof text !== 'string') return null;
|
|
118
|
+
for (const { id, runtime, pattern } of FATAL_MEMORY_PATTERNS) {
|
|
119
|
+
const match = pattern.exec(text);
|
|
120
|
+
if (!match) continue;
|
|
121
|
+
const lineStart = text.lastIndexOf('\n', match.index) + 1;
|
|
122
|
+
const lineEndIndex = text.indexOf('\n', match.index);
|
|
123
|
+
const line = text.slice(lineStart, lineEndIndex < 0 ? undefined : lineEndIndex).trim();
|
|
124
|
+
return { id, runtime, line: line.length > 300 ? `${line.slice(0, 300)}…` : line };
|
|
125
|
+
}
|
|
126
|
+
return null;
|
|
127
|
+
};
|
|
128
|
+
|
|
77
129
|
/**
|
|
78
130
|
* Wire `close`/`error` handlers that never lose a signal.
|
|
79
131
|
*
|
|
@@ -104,4 +156,4 @@ export const attachChildExitHandlers = ({ child, command, label, errorLabel = la
|
|
|
104
156
|
});
|
|
105
157
|
};
|
|
106
158
|
|
|
107
|
-
export default { describeChildExit, isLikelyOutOfMemoryExit, attachChildExitHandlers };
|
|
159
|
+
export default { describeChildExit, isLikelyOutOfMemoryExit, findFatalMemoryMarker, attachChildExitHandlers };
|
|
@@ -22,6 +22,7 @@ import Decimal from 'decimal.js-light';
|
|
|
22
22
|
import { accumulateModelUsage, createEmptySubSessionUsage, getRawRequestInputTokens, mergeResultModelUsage } from './claude.budget-stats.lib.mjs';
|
|
23
23
|
import { calculateModelCost } from './claude.cost.lib.mjs';
|
|
24
24
|
import { fetchModelInfo } from './model-info.lib.mjs';
|
|
25
|
+
import { forEachLogLine } from './log-bounded-read.lib.mjs';
|
|
25
26
|
|
|
26
27
|
export const calculateSessionTokens = async (sessionId, tempDir, resultModelUsage = null, options = {}) => {
|
|
27
28
|
const homeDir = options.homeDir || os.homedir();
|
|
@@ -43,10 +44,12 @@ export const calculateSessionTokens = async (sessionId, tempDir, resultModelUsag
|
|
|
43
44
|
let currentSubSession = createEmptySubSessionUsage();
|
|
44
45
|
const compactifications = [];
|
|
45
46
|
try {
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
47
|
+
// Issue #2189: read the transcript one record at a time. A long Claude
|
|
48
|
+
// session produces a JSONL of unbounded size, and `readFile(...).split('\n')`
|
|
49
|
+
// held the whole file *and* the array of its lines before the first entry
|
|
50
|
+
// was priced — a full-file allocation just to sum token counters.
|
|
51
|
+
await forEachLogLine(sessionFile, line => {
|
|
52
|
+
if (!line.trim()) return;
|
|
50
53
|
try {
|
|
51
54
|
const entry = JSON.parse(line);
|
|
52
55
|
if (entry.type === 'system' && entry.subtype === 'compact_boundary') {
|
|
@@ -59,7 +62,7 @@ export const calculateSessionTokens = async (sessionId, tempDir, resultModelUsag
|
|
|
59
62
|
trigger: entry.compactMetadata?.trigger || 'unknown',
|
|
60
63
|
});
|
|
61
64
|
currentSubSession = createEmptySubSessionUsage();
|
|
62
|
-
|
|
65
|
+
return;
|
|
63
66
|
}
|
|
64
67
|
if (entry.message && entry.message.usage && entry.message.model) {
|
|
65
68
|
// Issue #1501: Skip duplicate JSONL entries (same message ID = same API response)
|
|
@@ -67,7 +70,7 @@ export const calculateSessionTokens = async (sessionId, tempDir, resultModelUsag
|
|
|
67
70
|
if (msgId) {
|
|
68
71
|
if (seenMessageIds.has(msgId)) {
|
|
69
72
|
duplicateCount++;
|
|
70
|
-
|
|
73
|
+
return;
|
|
71
74
|
}
|
|
72
75
|
seenMessageIds.add(msgId);
|
|
73
76
|
}
|
|
@@ -100,9 +103,8 @@ export const calculateSessionTokens = async (sessionId, tempDir, resultModelUsag
|
|
|
100
103
|
}
|
|
101
104
|
} catch {
|
|
102
105
|
// Skip lines that aren't valid JSON
|
|
103
|
-
continue;
|
|
104
106
|
}
|
|
105
|
-
}
|
|
107
|
+
});
|
|
106
108
|
if (currentSubSession.messageCount > 0) {
|
|
107
109
|
subSessions.push(currentSubSession);
|
|
108
110
|
}
|
|
@@ -25,6 +25,8 @@ import { promises as fs } from 'fs';
|
|
|
25
25
|
import os from 'os';
|
|
26
26
|
import path from 'path';
|
|
27
27
|
|
|
28
|
+
import { fileEndsWithNewline, forEachLogLine } from './log-bounded-read.lib.mjs';
|
|
29
|
+
|
|
28
30
|
/**
|
|
29
31
|
* Resolve the on-disk session transcript path for a Claude Code session. Claude Code stores each
|
|
30
32
|
* session as `~/.claude/projects/<cwd-with-slashes-as-dashes>/<sessionId>.jsonl` (mirrors the
|
|
@@ -52,6 +54,36 @@ const isCorruptedThinkingBlock = block => {
|
|
|
52
54
|
return false;
|
|
53
55
|
};
|
|
54
56
|
|
|
57
|
+
/**
|
|
58
|
+
* Repair one transcript record.
|
|
59
|
+
*
|
|
60
|
+
* Returns the line to write back (unchanged unless a corrupted block was
|
|
61
|
+
* dropped), how many corrupted blocks it dropped, and whether the line counted
|
|
62
|
+
* as a scanned message line. Keeping the decision in one pure function lets the
|
|
63
|
+
* repair stream the file twice — count, then rewrite — with identical results.
|
|
64
|
+
*
|
|
65
|
+
* @param {string} line - One raw JSONL record
|
|
66
|
+
* @returns {{text: string, removed: number, scanned: number}}
|
|
67
|
+
*/
|
|
68
|
+
const repairTranscriptLine = line => {
|
|
69
|
+
if (!line.trim()) return { text: line, removed: 0, scanned: 0 };
|
|
70
|
+
let entry;
|
|
71
|
+
try {
|
|
72
|
+
entry = JSON.parse(line);
|
|
73
|
+
} catch {
|
|
74
|
+
return { text: line, removed: 0, scanned: 1 }; // preserve anything we can't parse verbatim
|
|
75
|
+
}
|
|
76
|
+
const content = entry?.message?.content;
|
|
77
|
+
if (!Array.isArray(content)) return { text: line, removed: 0, scanned: 1 };
|
|
78
|
+
const corrupted = content.filter(isCorruptedThinkingBlock).length;
|
|
79
|
+
if (corrupted === 0) return { text: line, removed: 0, scanned: 1 };
|
|
80
|
+
const cleaned = content.filter(b => !isCorruptedThinkingBlock(b));
|
|
81
|
+
// Never leave an assistant message with an empty content array (invalid for the API).
|
|
82
|
+
if (cleaned.length === 0) return { text: line, removed: 0, scanned: 1 };
|
|
83
|
+
entry.message.content = cleaned;
|
|
84
|
+
return { text: JSON.stringify(entry), removed: corrupted, scanned: 1 };
|
|
85
|
+
};
|
|
86
|
+
|
|
55
87
|
/**
|
|
56
88
|
* Strip corrupted (empty-text) thinking blocks from a Claude Code session transcript so the session
|
|
57
89
|
* can be resumed. Conservative and side-effect-safe:
|
|
@@ -75,48 +107,27 @@ export const repairCorruptedThinkingBlocks = async ({ tempDir, sessionId, homeDi
|
|
|
75
107
|
}
|
|
76
108
|
const sessionFile = resolveSessionTranscriptPath(tempDir, sessionId, homeDir);
|
|
77
109
|
result.sessionFile = sessionFile;
|
|
78
|
-
let
|
|
110
|
+
let sessionStat;
|
|
79
111
|
try {
|
|
80
|
-
|
|
112
|
+
sessionStat = await fs.stat(sessionFile);
|
|
81
113
|
} catch {
|
|
82
114
|
// No transcript on disk (e.g. fresh run never persisted, or path mismatch) — nothing to repair.
|
|
83
115
|
return { ...result, reason: 'session transcript not found' };
|
|
84
116
|
}
|
|
85
117
|
|
|
86
118
|
try {
|
|
87
|
-
|
|
88
|
-
|
|
119
|
+
// Issue #2189: a session transcript grows with the session — the captured
|
|
120
|
+
// incident's was 134 MB — so this is done in two streaming passes instead of
|
|
121
|
+
// holding the file, its array of lines and the rebuilt output in the heap at
|
|
122
|
+
// once. Pass 1 only counts: a transcript with nothing to repair (the common
|
|
123
|
+
// case) is never rewritten and never copied.
|
|
89
124
|
let removedBlocks = 0;
|
|
90
125
|
let scannedLines = 0;
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
scannedLines++;
|
|
97
|
-
let entry;
|
|
98
|
-
try {
|
|
99
|
-
entry = JSON.parse(line);
|
|
100
|
-
} catch {
|
|
101
|
-
out.push(line); // preserve anything we can't parse verbatim
|
|
102
|
-
continue;
|
|
103
|
-
}
|
|
104
|
-
const content = entry?.message?.content;
|
|
105
|
-
if (Array.isArray(content)) {
|
|
106
|
-
const corrupted = content.filter(isCorruptedThinkingBlock).length;
|
|
107
|
-
if (corrupted > 0) {
|
|
108
|
-
const cleaned = content.filter(b => !isCorruptedThinkingBlock(b));
|
|
109
|
-
// Never leave an assistant message with an empty content array (invalid for the API).
|
|
110
|
-
if (cleaned.length > 0) {
|
|
111
|
-
entry.message.content = cleaned;
|
|
112
|
-
removedBlocks += corrupted;
|
|
113
|
-
out.push(JSON.stringify(entry));
|
|
114
|
-
continue;
|
|
115
|
-
}
|
|
116
|
-
}
|
|
117
|
-
}
|
|
118
|
-
out.push(line);
|
|
119
|
-
}
|
|
126
|
+
await forEachLogLine(sessionFile, line => {
|
|
127
|
+
const repaired = repairTranscriptLine(line);
|
|
128
|
+
scannedLines += repaired.scanned;
|
|
129
|
+
removedBlocks += repaired.removed;
|
|
130
|
+
});
|
|
120
131
|
|
|
121
132
|
result.scannedLines = scannedLines;
|
|
122
133
|
if (removedBlocks === 0) {
|
|
@@ -135,7 +146,27 @@ export const repairCorruptedThinkingBlocks = async ({ tempDir, sessionId, homeDi
|
|
|
135
146
|
}
|
|
136
147
|
}
|
|
137
148
|
|
|
138
|
-
|
|
149
|
+
// Pass 2: rewrite through a sibling temp file and rename over the original,
|
|
150
|
+
// so an interrupted repair can never leave a half-written transcript (which
|
|
151
|
+
// would be worse than the corruption being repaired).
|
|
152
|
+
const keepTrailingNewline = await fileEndsWithNewline(sessionFile);
|
|
153
|
+
const tempFile = `${sessionFile}.repair-${process.pid}`;
|
|
154
|
+
await fs.rm(tempFile, { force: true });
|
|
155
|
+
const handle = await fs.open(tempFile, 'wx', sessionStat.mode & 0o777);
|
|
156
|
+
try {
|
|
157
|
+
let pendingSeparator = '';
|
|
158
|
+
await forEachLogLine(sessionFile, line => {
|
|
159
|
+
const repaired = repairTranscriptLine(line);
|
|
160
|
+
return handle.write(`${pendingSeparator}${repaired.text}`, null, 'utf8').then(() => {
|
|
161
|
+
pendingSeparator = '\n';
|
|
162
|
+
});
|
|
163
|
+
});
|
|
164
|
+
if (keepTrailingNewline) await handle.write('\n', null, 'utf8');
|
|
165
|
+
} finally {
|
|
166
|
+
await handle.close();
|
|
167
|
+
}
|
|
168
|
+
await fs.rename(tempFile, sessionFile);
|
|
169
|
+
|
|
139
170
|
result.repaired = true;
|
|
140
171
|
result.removedBlocks = removedBlocks;
|
|
141
172
|
await log(`🩹 Repaired session transcript: stripped ${removedBlocks} corrupted thinking block(s) from ${scannedLines} message line(s) (Issue #1834). Backup: ${backupFile}`, { verbose: true });
|
package/src/codex.lib.mjs
CHANGED
|
@@ -49,10 +49,15 @@ import Decimal from 'decimal.js-light';
|
|
|
49
49
|
import { ensureAiToolScratchIgnored, filterAiToolScratchFromStatus } from './ai-tool-scratch.lib.mjs';
|
|
50
50
|
import { CODEX_CACHE_READ_USAGE_PATHS, CODEX_CACHE_WRITE_USAGE_PATHS, CODEX_MODEL_DIAGNOSTIC_PATHS, CODEX_REASONING_USAGE_PATHS, CODEX_USAGE_FIELD_NAMES, createCodexTokenFieldAvailability, getFirstObservedNumber, hasAnyObservedPath, hasOwnPath } from './codex.usage-fields.lib.mjs';
|
|
51
51
|
const CODEX_LONG_CONTEXT_PRICE_THRESHOLD = 272000;
|
|
52
|
+
// Issue #2189: ceiling for reading Codex's `--output-last-message` artifact. A
|
|
53
|
+
// final assistant message is a few hundred kilobytes at most; anything larger is
|
|
54
|
+
// a malfunction and must not be turned into an unbounded string.
|
|
55
|
+
const CODEX_LAST_MESSAGE_MAX_BYTES = 1024 * 1024;
|
|
52
56
|
const getCodexExecEnv = (verbose = false) => (verbose ? { ...process.env, RUST_LOG: 'debug' } : { ...process.env });
|
|
53
57
|
// Issue #2175: diagnostic-line parsing lives in its own module to keep this file
|
|
54
58
|
// under the 1350-line warning threshold.
|
|
55
59
|
import { parseCodexDiagnosticLine, rebuildCodexSubSessionsFromCompactifications } from './codex.diagnostics.lib.mjs';
|
|
60
|
+
import { readLogHeadText } from './log-bounded-read.lib.mjs'; // Issue #2189
|
|
56
61
|
export const createCodexTokenUsage = requestedModelId => ({
|
|
57
62
|
inputTokens: 0,
|
|
58
63
|
outputTokens: 0,
|
|
@@ -891,7 +896,12 @@ export const executeCodexCommand = async params => {
|
|
|
891
896
|
let lastMessageFromFile = null;
|
|
892
897
|
let lastMessageReadError = null;
|
|
893
898
|
try {
|
|
894
|
-
|
|
899
|
+
// Issue #2189: this file holds Codex's final assistant message, but its
|
|
900
|
+
// size is decided by the tool, not by us. Size it first so a runaway or
|
|
901
|
+
// corrupted artifact cannot become an unbounded string in a process that
|
|
902
|
+
// has just finished a long run.
|
|
903
|
+
const { size } = await fs.stat(lastMessageFile);
|
|
904
|
+
lastMessageFromFile = size > CODEX_LAST_MESSAGE_MAX_BYTES ? `${(await readLogHeadText(lastMessageFile, { maxBytes: CODEX_LAST_MESSAGE_MAX_BYTES })).trim()}\n…[last message truncated: ${size} bytes on disk, see ${lastMessageFile}]` : (await fs.readFile(lastMessageFile, 'utf8')).trim();
|
|
895
905
|
} catch (readError) {
|
|
896
906
|
lastMessageReadError = readError;
|
|
897
907
|
}
|
|
@@ -2,6 +2,8 @@ import fs from 'node:fs/promises';
|
|
|
2
2
|
import os from 'node:os';
|
|
3
3
|
import path from 'node:path';
|
|
4
4
|
import { sanitizeForPublication } from './token-sanitization.lib.mjs';
|
|
5
|
+
import { findResidualCredentialBlock } from './log-sanitize-stream.lib.mjs';
|
|
6
|
+
import { sanitizeLogFileToFileBounded } from './log-sanitize-worker.lib.mjs';
|
|
5
7
|
|
|
6
8
|
const sanitizePathSegment = (value, fallback) => {
|
|
7
9
|
const raw = value === null || value === undefined || value === '' ? fallback : String(value);
|
|
@@ -94,13 +96,25 @@ const writePrivatePublicationFile = async (destinationPath, content) => {
|
|
|
94
96
|
await fs.chmod(destinationPath, 0o600);
|
|
95
97
|
};
|
|
96
98
|
|
|
99
|
+
// Issue #2189: `sanitizeLogFileToFileBounded` creates its destination exclusively
|
|
100
|
+
// (`wx`) so a pre-planted symlink cannot be followed. Collection may run more
|
|
101
|
+
// than once for the same session directory, so drop a previous artifact first —
|
|
102
|
+
// unlink-then-O_EXCL keeps the symlink guarantee that a plain truncate loses.
|
|
103
|
+
const sanitizeIntoPublicationFile = async ({ sourcePath, destinationPath, startByte = 0, endByte = null }) => {
|
|
104
|
+
await fs.rm(destinationPath, { force: true });
|
|
105
|
+
return sanitizeLogFileToFileBounded({ sourcePath, destPath: destinationPath, startByte, endByte });
|
|
106
|
+
};
|
|
107
|
+
|
|
97
108
|
const copyIfExists = async ({ sourcePath, destinationPath }) => {
|
|
98
109
|
if (!(await fileExists(sourcePath))) return false;
|
|
99
110
|
// Raw local audit sources remain available to the operator but must not be
|
|
100
111
|
// group/world-readable. Only the sanitized copy enters the repository.
|
|
101
112
|
await fs.chmod(sourcePath, 0o600);
|
|
102
|
-
|
|
103
|
-
|
|
113
|
+
// Issue #2189: transcripts are as large as the run that produced them (the
|
|
114
|
+
// captured incident had a 134 MB one). Sanitize source → destination block by
|
|
115
|
+
// block instead of holding the file, its sanitized twin and the sanitizer's
|
|
116
|
+
// own working copy in the heap at once.
|
|
117
|
+
await sanitizeIntoPublicationFile({ sourcePath, destinationPath });
|
|
104
118
|
return true;
|
|
105
119
|
};
|
|
106
120
|
|
|
@@ -140,8 +154,7 @@ const copyLogSlice = async ({ logFile, destinationPath, logStartByte = 0 }) => {
|
|
|
140
154
|
await writePrivatePublicationFile(destinationPath, '');
|
|
141
155
|
return { logStartByte: start, logEndByte: stat.size };
|
|
142
156
|
}
|
|
143
|
-
|
|
144
|
-
await writePrivatePublicationFile(destinationPath, bytes.subarray(start, stat.size).toString('utf8'));
|
|
157
|
+
await sanitizeIntoPublicationFile({ sourcePath: logFile, destinationPath, startByte: start, endByte: stat.size });
|
|
145
158
|
return { logStartByte: start, logEndByte: stat.size };
|
|
146
159
|
};
|
|
147
160
|
|
|
@@ -279,9 +292,11 @@ const verifyDevelopmentLogDirectory = async directoryPath => {
|
|
|
279
292
|
if (!entry.isFile()) continue;
|
|
280
293
|
const parentPath = entry.parentPath || entry.path;
|
|
281
294
|
const filePath = path.join(parentPath, entry.name);
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
295
|
+
// Issue #2189: rescan block by block. Reading each artifact back whole made
|
|
296
|
+
// the verification cost as much heap as the artifact — on top of the copy
|
|
297
|
+
// that had just been written.
|
|
298
|
+
const residual = await findResidualCredentialBlock(filePath);
|
|
299
|
+
if (residual) {
|
|
285
300
|
throw new Error('Development-log publication rescan found residual credential material.');
|
|
286
301
|
}
|
|
287
302
|
await fs.chmod(filePath, 0o600);
|
|
@@ -9,6 +9,8 @@ import { createInterface } from 'readline';
|
|
|
9
9
|
import { log, cleanErrorMessage, getAbsoluteLogPath } from './lib.mjs';
|
|
10
10
|
import { reportError, isSentryEnabled } from './sentry.lib.mjs';
|
|
11
11
|
import { sanitizeForPublication, writeSanitizedPublicationFile } from './token-sanitization.lib.mjs';
|
|
12
|
+
import { sanitizeLogFileToFileBounded } from './log-sanitize-worker.lib.mjs';
|
|
13
|
+
import { readLogTailText } from './log-bounded-read.lib.mjs';
|
|
12
14
|
|
|
13
15
|
if (typeof globalThis.use === 'undefined') {
|
|
14
16
|
await ensureUseM();
|
|
@@ -107,8 +109,87 @@ const createSecretGist = async (logContent, filename) => {
|
|
|
107
109
|
return null;
|
|
108
110
|
};
|
|
109
111
|
|
|
112
|
+
/**
|
|
113
|
+
* Upload a log FILE as a secret gist without ever holding it in memory.
|
|
114
|
+
*
|
|
115
|
+
* Issue #2189: the error reporter runs when the process is already in trouble —
|
|
116
|
+
* frequently because it just exhausted its heap. Reading the log to sanitize it
|
|
117
|
+
* (`readFile` + `sanitizeForPublication` + write = three full copies) is the one
|
|
118
|
+
* thing that must not happen there.
|
|
119
|
+
*
|
|
120
|
+
* @param {string} logFilePath - Log to upload
|
|
121
|
+
* @param {string} filename - Name for the gist file
|
|
122
|
+
* @returns {Promise<string|null>} Gist URL, or null when the upload failed
|
|
123
|
+
*/
|
|
124
|
+
const createSecretGistFromFile = async (logFilePath, filename) => {
|
|
125
|
+
const tempFile = `/tmp/${filename}`;
|
|
126
|
+
try {
|
|
127
|
+
await sanitizeLogFileToFileBounded({ sourcePath: logFilePath, destPath: tempFile });
|
|
128
|
+
const result = await $`gh gist create ${tempFile} --secret --desc "Error log for hive-mind"`;
|
|
129
|
+
if (result.exitCode === 0) {
|
|
130
|
+
return result.stdout.toString().trim();
|
|
131
|
+
}
|
|
132
|
+
} catch (error) {
|
|
133
|
+
reportError(error, {
|
|
134
|
+
context: 'create_secret_gist',
|
|
135
|
+
operation: 'gh_gist_create',
|
|
136
|
+
});
|
|
137
|
+
} finally {
|
|
138
|
+
await fs.unlink(tempFile).catch(() => {});
|
|
139
|
+
}
|
|
140
|
+
return null;
|
|
141
|
+
};
|
|
142
|
+
|
|
143
|
+
/**
|
|
144
|
+
* Format a log FILE for an issue body, choosing the attachment method from the
|
|
145
|
+
* file's size before reading any of it (issue #2189).
|
|
146
|
+
*
|
|
147
|
+
* Only the inline branch — by definition below GitHub's 60 kB issue-body limit —
|
|
148
|
+
* ever reads log content, and the truncated fallback reads a bounded tail.
|
|
149
|
+
*
|
|
150
|
+
* @param {string} logFilePath - Path to the log file
|
|
151
|
+
* @returns {Promise<{method: string, content: string}>}
|
|
152
|
+
*/
|
|
153
|
+
export const formatLogFileForIssue = async logFilePath => {
|
|
154
|
+
const { size } = await fs.stat(logFilePath);
|
|
155
|
+
|
|
156
|
+
if (size < GITHUB_ISSUE_BODY_MAX_SIZE) {
|
|
157
|
+
const logContent = await fs.readFile(logFilePath, 'utf8');
|
|
158
|
+
return {
|
|
159
|
+
method: 'inline',
|
|
160
|
+
content: `\`\`\`\n${logContent}\n\`\`\``,
|
|
161
|
+
};
|
|
162
|
+
}
|
|
163
|
+
|
|
164
|
+
if (size < GITHUB_FILE_MAX_SIZE) {
|
|
165
|
+
return {
|
|
166
|
+
method: 'file',
|
|
167
|
+
content: `Log file is too large to include inline. Please see the attached log file.\n\nLog file path: \`${logFilePath}\``,
|
|
168
|
+
};
|
|
169
|
+
}
|
|
170
|
+
|
|
171
|
+
const gistUrl = await createSecretGistFromFile(logFilePath, `hive-mind-error-${Date.now()}.log`);
|
|
172
|
+
if (gistUrl) {
|
|
173
|
+
return {
|
|
174
|
+
method: 'gist',
|
|
175
|
+
content: `Log file is too large for inline attachment.\n\n📄 View full log: ${gistUrl}`,
|
|
176
|
+
};
|
|
177
|
+
}
|
|
178
|
+
|
|
179
|
+
const tail = await readLogTailText(logFilePath, { maxBytes: 5000 });
|
|
180
|
+
return {
|
|
181
|
+
method: 'truncated',
|
|
182
|
+
content: `Log file is too large. Showing last 5000 characters:\n\n\`\`\`\n${tail}\n\`\`\``,
|
|
183
|
+
};
|
|
184
|
+
};
|
|
185
|
+
|
|
110
186
|
/**
|
|
111
187
|
* Format log content for issue body
|
|
188
|
+
*
|
|
189
|
+
* Prefer {@link formatLogFileForIssue} when the log is a file on disk: this
|
|
190
|
+
* variant needs the whole log as a string, which is exactly what issue #2189
|
|
191
|
+
* removed from the publication path.
|
|
192
|
+
*
|
|
112
193
|
* @param {string} logContent - Log file content
|
|
113
194
|
* @param {string} logFilePath - Path to log file
|
|
114
195
|
* @returns {Promise<Object>} Object with formatted content and attachment method
|
|
@@ -202,8 +283,9 @@ export const createIssueForError = async options => {
|
|
|
202
283
|
|
|
203
284
|
if (logFile) {
|
|
204
285
|
try {
|
|
205
|
-
|
|
206
|
-
|
|
286
|
+
// Issue #2189: pick the attachment method from the file size first; a
|
|
287
|
+
// log too large for the issue body is never read into memory here.
|
|
288
|
+
const { method, content } = await formatLogFileForIssue(logFile);
|
|
207
289
|
|
|
208
290
|
issueBody += `### Log File\n\n${content}\n\n`;
|
|
209
291
|
await log(`📄 Log attached via: ${method}`);
|