pi-crew 0.11.38 → 0.11.40

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,766 @@
1
+ /**
2
+ * run-descendant-sweep.ts — fix-wave 11b (F7): reap the DESCENDANTS of a
3
+ * TERMINAL pi-crew run by matching the pi-crew env markers they inherited.
4
+ *
5
+ * Why this module exists (the incident, SPEC §1): run
6
+ * `team_20261022110510_cefb9164af126460` reached `completed` at 2026-10-22T13:02Z
7
+ * and still had SIX grandchildren alive 24 h later (`dpi-backend` ×5, `caddy`;
8
+ * 399 MB RSS). They carried `PI_CREW_KIND=subagent` and
9
+ * `PI_CREW_PARENT_PID=<dead runner pid>`. No layer of the 11a teardown stack
10
+ * could reach them, and never will: pi's bash tool spawns EVERY command with
11
+ * `detached: true` (GAP-0), so each bash command gets its OWN process group.
12
+ * A group kill rooted at the worker reaches the worker's group only — the bash
13
+ * grandchildren live in a group of their own.
14
+ *
15
+ * The only identity that survives that topology is the ENVIRONMENT: the child
16
+ * inherits the worker's env, so every descendant still carries the pi-crew
17
+ * markers of the run it belongs to. This module therefore turns env markers
18
+ * into kill candidates, scoped to ONE run, and only after that run is terminal.
19
+ *
20
+ * Markers actually used (VERIFIED in src — never invented):
21
+ * - `PI_CREW_BROKER_RUN_ID` — `child-pi-spawn.ts:382` writes
22
+ * `built.env.PI_CREW_BROKER_RUN_ID = input.runId` for every team worker
23
+ * (the env key is spelled by `PI_CREW_RUN_ID_ENV` in
24
+ * `scratchpad-lifecycle.ts:66`, whose VALUE is the broker name). This is
25
+ * the strongest marker ⇒ `matchedBy: "run-id"`.
26
+ * `PI_CREW_RUN_ID` itself is NOT matched: `env-allowlist-registry.ts:322`
27
+ * classifies it `test-only` ("fixture name; src uses PI_CREW_BROKER_RUN_ID")
28
+ * and grep confirms no src writer — matching it would only widen the kill
29
+ * surface onto test fixtures.
30
+ * - `PI_CREW_KIND=subagent` (`pi-args.ts:564`), `PI_CREW_TASK_ID`
31
+ * (`child-pi-spawn.ts:434,443`) and `PI_CREW_PARENT_PID`
32
+ * (`child-pi-spawn.ts:191` = the SPAWNING process pid) ⇒
33
+ * `matchedBy: "task-id"`.
34
+ * - `PI_CREW_PARENT_PID` alone, when the recorded parent is already dead ⇒
35
+ * `matchedBy: "parent-pid"` (the shape the incident's survivors had).
36
+ *
37
+ * Safety model — the sweep is additive, never a wider kill surface:
38
+ * 1. Discovery is Linux `/proc` only; anything else yields `[]` (fail-open).
39
+ * 2. Our own pid and every ancestor of it are never candidates.
40
+ * 3. Same uid only: we can only reason about our own processes.
41
+ * 4. The candidate must have STARTED no earlier than the run (starttime is
42
+ * converted to wall clock via /proc/uptime; > 60 s of pre-run clock skew
43
+ * is rejected) — a process that predates the run cannot belong to it.
44
+ * 5. The candidate's group must not be our own group nor our own pid.
45
+ * 6. Every signal goes through `reapGroup()` — the identity-proven ladder in
46
+ * reap-group.ts — anchored on the candidate pid, never a blind
47
+ * `kill(-pgid)`. Candidates without a usable pgid are skipped
48
+ * (`no-pgid`), never signalled individually.
49
+ * 7. EXTRA rung for the sweep: the group's LEADER must either be gone, or be
50
+ * another matched candidate of this same run. A live, unmatched leader
51
+ * means the group belongs to a session that is still running ⇒
52
+ * `leader-alive-unmatched`, nothing signalled.
53
+ * 8. `dryRun` reports and signals nothing; `PI_CREW_REAP_RUN_DESCENDANTS=0`
54
+ * disables the whole sweep.
55
+ * 9. (11b.1) A RECORDED ancestor (the detached runner pid, a worker pid) is
56
+ * the tree's ROOT, never its descendant ⇒ it is never a candidate, even
57
+ * when it is still alive. Only its children/grandchildren are, so a live
58
+ * recorded pid cannot veto the sweep the way the old all-or-nothing
59
+ * "every recorded pid is dead" gate did.
60
+ * 10. (11b.3) A recorded SURFACE pane pid (`manifest.surface.workerPids`) is
61
+ * something the operator may still be watching in a mux pane ⇒ while it is
62
+ * ALIVE it is never a candidate. It is not excluded unconditionally like a
63
+ * recorded ancestor: once it is dead there is nothing left to kill, so the
64
+ * pid simply falls out of /proc (and a recycled pid must still satisfy
65
+ * every start-time/uid/marker proof above).
66
+ * 11. (11b.4) The `run-id` MARKER ALONE IS NOT PROOF OF DESCENT. Inheriting
67
+ * `PI_CREW_BROKER_RUN_ID` only says "some ancestor of mine was part of this
68
+ * run": an operator shell opened inside a worker's pane, or a dev server a
69
+ * worker started and left running, carries the same env after the spawn
70
+ * session is long gone. The `run-id` branch therefore also demands a
71
+ * `PI_CREW_PARENT_PID` the run actually RECORDED (runner / worker /
72
+ * surface pane / session pid — the last added by 11c for SYNC runs, which
73
+ * record no runner) that is ALREADY DEAD — the same descent proof the
74
+ * `task-id` and `parent-pid` branches have always required. A same-uid
75
+ * process that inherited the env but has no such parent is not reaped
76
+ * (a documented miss, never a wrong kill).
77
+ * 12. (11b.4) A candidate holding a CONTROLLING TERMINAL (`/proc/<pid>/stat`
78
+ * field 7 `tty_nr !== 0`) is an INTERACTIVE process — typically the shell
79
+ * the operator is typing into. It is skipped with reason `interactive-tty`
80
+ * and never signalled, on EVERY branch (a tty is not a run resource). The
81
+ * incident shape (a detached dev server, `caddy`) has no controlling
82
+ * terminal, so the real leaks are unaffected.
83
+ *
84
+ * Escalation: like every other `reapGroup` caller, the SIGKILL rung comes back
85
+ * as a `pending` promise. The caller registers it with the EXISTING drain
86
+ * owners (`trackReapEscalation`/`drainReapEscalations` in stale-reconciler.ts,
87
+ * `flushReapEscalationsSync` in reap-group.ts) — this module deliberately has
88
+ * no drain of its own.
89
+ *
90
+ * Known limits (documented, not fixed here): a descendant that `setsid()`s or
91
+ * runs inside a container/runtime with its own init (docker, systemd-run) is a
92
+ * group we cannot prove and is left alone; and pi's bash tool remains the
93
+ * primary leak source — the upstream fix (a `detached: false`/tracked-children
94
+ * default) is what would make this sweep unnecessary.
95
+ */
96
+
97
+ import * as fs from "node:fs";
98
+ import { getCrewEnvBool } from "../../config/env-vars.ts";
99
+ import { logInternalError } from "../../utils/internal-error.ts";
100
+ import { fieldsAfterComm, PROC_STAT_PPID_INDEX, PROC_STAT_TTY_INDEX } from "./proc-stat.ts";
101
+ import { type ProcIdentity, readProcIdentity, reapGroup } from "./reap-group.ts";
102
+
103
+ /** Default grace window between SIGTERM and SIGKILL (ms). */
104
+ const DEFAULT_GRACEFUL_MS = 2_000;
105
+ /** Default hard cap after SIGKILL before giving up (ms). */
106
+ const DEFAULT_CAP_MS = 2_000;
107
+ /**
108
+ * Allowed clock skew when comparing a candidate's start time against the run
109
+ * start. A process that started MORE than this before the run cannot belong to
110
+ * it; the window absorbs `createdAt` being stamped before the spawn and the
111
+ * coarseness of `/proc/<pid>/stat` starttime ticks.
112
+ */
113
+ const PRE_RUN_SKEW_MS = 60_000;
114
+ /** Linux `CLK_TCK` — hardcoded like zombie-scanner.ts (100 on every mainstream distro). */
115
+ const CLOCK_TICKS_PER_SEC = 100;
116
+ /** Bound on the "already signalled this group" registry (see idempotency note). */
117
+ const MAX_TRACKED_GROUPS = 256;
118
+
119
+ /**
120
+ * Kill switch (default ON). `PI_CREW_REAP_RUN_DESCENDANTS=0` makes the sweep a
121
+ * complete no-op without a redeploy — the escape hatch for a platform whose
122
+ * /proc semantics turn out to over-match.
123
+ */
124
+ export function resolveRunDescendantSweepEnabled(): boolean {
125
+ return getCrewEnvBool("PI_CREW_REAP_RUN_DESCENDANTS") ?? true;
126
+ }
127
+
128
+ /** One process proven to belong to a terminal run. */
129
+ export interface RunDescendant {
130
+ pid: number;
131
+ pgid: number | undefined;
132
+ starttime: string | undefined;
133
+ uid: number | undefined;
134
+ /** Which marker proved descent — highest available priority first. */
135
+ matchedBy: "run-id" | "task-id" | "parent-pid";
136
+ cmd: string;
137
+ rssKb: number;
138
+ /**
139
+ * 11b.4: `/proc/<pid>/stat` field 7. `0`/undefined ⇒ no controlling terminal;
140
+ * anything else marks an INTERACTIVE process, skipped by safety rung 12.
141
+ */
142
+ ttyNr?: number;
143
+ }
144
+
145
+ /** What identifies a run's process tree. All values come from run state. */
146
+ export interface RunMarkers {
147
+ runId: string;
148
+ /** `manifest.async.pid` — the detached team-runner, when recorded. */
149
+ runnerPid?: number;
150
+ /** Recorded worker pids of the run (heartbeat.pid / checkpoint.childPid). */
151
+ workerPids: number[];
152
+ /** Task ids of the run (PI_CREW_TASK_ID values its workers carried). */
153
+ taskIds: string[];
154
+ /**
155
+ * Wave 11c (battery 2026-10-24 F2): `manifest.sessionPid` — the pid of the
156
+ * process that created the run. A SYNC (foreground) run has no async
157
+ * runner, so its leaked bash grandchildren carry
158
+ * `PI_CREW_PARENT_PID=<session pi pid>` — a pid NO other field records,
159
+ * which made the descent proof unsatisfiable and the leak a permanent
160
+ * documented miss. Accepted as a proof parent for the run-id branch with
161
+ * the SAME dead-only guard as every other parent: while the session pi is
162
+ * ALIVE it is not proof of anything (the operator may still be typing in
163
+ * it) — the leak is only reaped once that session is gone.
164
+ */
165
+ sessionPid?: number;
166
+ /**
167
+ * 11b.3: pids recorded in `manifest.surface.workerPids` (the mux-surface
168
+ * controller's view of each worker). A pane pid the operator can still see
169
+ * must not be reaped while it is alive — see safety rung 10.
170
+ */
171
+ surfacePids?: number[];
172
+ /** Run start (epoch ms) — a candidate that predates it cannot belong to it. */
173
+ startedAtMs: number;
174
+ }
175
+
176
+ /**
177
+ * One `/proc` row captured ONCE per reconcile pass (wave 11b.4, LOW-4).
178
+ *
179
+ * Before 11b.4 every run in the terminal lane built its own `/proc` view, so a
180
+ * pass with the full 25-run budget read `/proc/<pid>/stat` + `status` for every
181
+ * process on the host 25 times. The snapshot makes it ONE read per pid per
182
+ * pass; the readers stay injectable so tests and the doctor path are unchanged.
183
+ */
184
+ export interface ProcSnapshotEntry {
185
+ identity: ProcIdentity;
186
+ /**
187
+ * The process' `PI_CREW_*` env. Read ONLY for same-uid processes: a foreign
188
+ * uid is rejected by rung 3 before its env is ever consulted, and
189
+ * `/proc/<pid>/environ` of another user is unreadable anyway — so skipping it
190
+ * avoids a pile of EACCES throws on a busy host.
191
+ */
192
+ environ: Record<string, string> | undefined;
193
+ /** `/proc/<pid>/stat` field 7; `0` ⇒ no controlling terminal (safety rung 12). */
194
+ ttyNr: number | undefined;
195
+ }
196
+
197
+ /** A reusable `/proc` view for one reconcile pass. */
198
+ export interface ProcSnapshot {
199
+ /** Every pid the snapshot covers (`/proc` enumeration order). */
200
+ pids: number[];
201
+ /** pid → row. A pid MISSING here had no readable identity when captured. */
202
+ entries: Map<number, ProcSnapshotEntry>;
203
+ }
204
+
205
+ /** Injectable dependencies — every side effect is replaceable in tests. */
206
+ export interface SweepOptions {
207
+ /** Report candidates without signalling (doctor). */
208
+ dryRun?: boolean;
209
+ /** /proc enumeration (tests inject a fixed pid list). Default `readdirSync("/proc")`. */
210
+ listPids?: () => number[];
211
+ /** /proc/<pid>/environ reader (tests inject marker maps). */
212
+ readEnviron?: (pid: number) => Record<string, string> | undefined;
213
+ /** /proc identity reader (tests inject fixtures). Default `readProcIdentity`. */
214
+ readIdentity?: (pid: number) => ProcIdentity | undefined;
215
+ /** Group signal sender (the reap-group `signalGroup` seam). */
216
+ killGroup?: (pgid: number, signal: NodeJS.Signals) => void;
217
+ /** Group liveness probe (the reap-group `groupAlive` seam). */
218
+ groupAlive?: (pgid: number) => boolean;
219
+ /** Grace window between SIGTERM and SIGKILL on the group path (ms). */
220
+ gracefulMs?: number;
221
+ /** Hard cap after SIGKILL on the group path (ms). */
222
+ capMs?: number;
223
+ /** Platform override (tests simulate non-Linux). Default `process.platform`. */
224
+ platform?: NodeJS.Platform;
225
+ /** Candidate start (epoch ms) resolver — tests pin it, default reads /proc/uptime. */
226
+ candidateStartMs?: (identity: ProcIdentity) => number | undefined;
227
+ /** `/proc/<pid>/stat` field 7 (tty_nr) reader — tests inject fixtures. */
228
+ readTtyNr?: (pid: number) => number | undefined;
229
+ /**
230
+ * 11b.4 (LOW-4): a `/proc` view shared by every sweep in a reconcile pass.
231
+ * When absent the sweep builds its own (one per call), so the public contract
232
+ * and the doctor path are unchanged.
233
+ */
234
+ snapshot?: ProcSnapshot;
235
+ /**
236
+ * uid source for the same-uid rung (safety rung 3). Default
237
+ * `() => process.getuid?.()` — production behaviour is unchanged. The seam
238
+ * exists because `process.getuid` is undefined on win32, which made the rung
239
+ * untestable there without a process-wide monkeypatch.
240
+ *
241
+ * It is honoured by BOTH halves of a sweep — discovery (`buildProcSnapshot`,
242
+ * `collectRunDescendants`) and the group reap — so an injected uid is never
243
+ * compared here and then ignored one call deeper (see the propagation note at
244
+ * the `reapGroup` call site in `reapRunDescendants`).
245
+ */
246
+ getuid?: () => number | undefined;
247
+ }
248
+
249
+ /** Sweep verdict: what was signalled, what was refused, what is still in flight. */
250
+ export interface RunSweepResult {
251
+ /** Candidates actually handed to the group reaper (empty on dryRun). */
252
+ reaped: RunDescendant[];
253
+ skipped: { pid: number; reason: string }[];
254
+ /** SIGKILL escalations — register with the existing drain owners; never dropped. */
255
+ pending: Promise<unknown>[];
256
+ /**
257
+ * Every candidate discovery proved to belong to the run (dryRun included).
258
+ * `reaped` is the subset that was signalled; doctor reports THIS list.
259
+ */
260
+ descendants: RunDescendant[];
261
+ }
262
+
263
+ /** Default uid source: the real process uid when the runtime has one (POSIX),
264
+ * `undefined` on win32 — the pre-seam behaviour, unchanged. */
265
+ const defaultGetuid = (): number | undefined => process.getuid?.();
266
+
267
+ // ── default /proc readers (all best-effort, none throw) ──────────────────────
268
+
269
+ /** `/proc/<pid>/environ` as a key=value record; undefined when unreadable. */
270
+ function readProcEnviron(pid: number): Record<string, string> | undefined {
271
+ try {
272
+ const raw = fs.readFileSync(`/proc/${pid}/environ`, "utf-8");
273
+ const out: Record<string, string> = {};
274
+ for (const entry of raw.split("\0")) {
275
+ const eq = entry.indexOf("=");
276
+ if (eq > 0) out[entry.slice(0, eq)] = entry.slice(eq + 1);
277
+ }
278
+ return out;
279
+ } catch {
280
+ return undefined;
281
+ }
282
+ }
283
+
284
+ /** Numeric pid directories under /proc; `[]` when /proc is unreadable. */
285
+ function listProcPids(): number[] {
286
+ const pids: number[] = [];
287
+ try {
288
+ for (const entry of fs.readdirSync("/proc")) {
289
+ if (/^\d+$/.test(entry)) pids.push(Number.parseInt(entry, 10));
290
+ }
291
+ } catch {
292
+ /* no /proc — the platform gate above already handles the real case */
293
+ }
294
+ return pids;
295
+ }
296
+
297
+ /**
298
+ * Parent pid of a process, from the shared paren-aware stat parser. Used ONLY to
299
+ * walk OUR OWN ancestry (the skip set): that is real kernel state regardless of
300
+ * any injected candidate reader, so it is deliberately not injectable.
301
+ */
302
+ function readParentPid(pid: number): number | undefined {
303
+ try {
304
+ const fields = fieldsAfterComm(fs.readFileSync(`/proc/${pid}/stat`, "utf-8"));
305
+ if (!fields) return undefined;
306
+ const ppid = Number.parseInt(fields[PROC_STAT_PPID_INDEX] ?? "", 10);
307
+ return Number.isFinite(ppid) && ppid > 0 ? ppid : undefined;
308
+ } catch {
309
+ return undefined;
310
+ }
311
+ }
312
+
313
+ /** Our own ancestor chain (never a candidate: they host this session). */
314
+ function selfAncestors(): Set<number> {
315
+ const chain = new Set<number>();
316
+ let current = process.ppid;
317
+ let guard = 0;
318
+ while (current > 1 && guard++ < 64) {
319
+ chain.add(current);
320
+ const next = readParentPid(current);
321
+ if (next === undefined || chain.has(next)) break;
322
+ current = next;
323
+ }
324
+ return chain;
325
+ }
326
+
327
+ /** `/proc/<pid>/cmdline` for the report; `pid <n>` when unreadable. */
328
+ function readProcCmdline(pid: number): string {
329
+ try {
330
+ const raw = fs.readFileSync(`/proc/${pid}/cmdline`, "utf-8");
331
+ return raw.split("\0").filter(Boolean).join(" ").trim() || `pid ${pid}`;
332
+ } catch {
333
+ return `pid ${pid}`;
334
+ }
335
+ }
336
+
337
+ /**
338
+ * `/proc/<pid>/stat` field 7 (tty_nr): `0` ⇒ no controlling terminal, any other
339
+ * value ⇒ the process is attached to a terminal. Read from the shared parser so
340
+ * the field index can never drift from its siblings. undefined when unreadable —
341
+ * and an unreadable tty is treated as "no tty": the identity read that got us
342
+ * here already succeeded, so this is a gone-process race and `reapGroup` refuses
343
+ * a gone candidate on its own.
344
+ */
345
+ function readProcTtyNr(pid: number): number | undefined {
346
+ try {
347
+ const fields = fieldsAfterComm(fs.readFileSync(`/proc/${pid}/stat`, "utf-8"));
348
+ if (!fields) return undefined;
349
+ const raw = Number.parseInt(fields[PROC_STAT_TTY_INDEX] ?? "", 10);
350
+ return Number.isFinite(raw) ? raw : undefined;
351
+ } catch {
352
+ return undefined;
353
+ }
354
+ }
355
+
356
+ /** `/proc/<pid>/status` VmRSS in kB; 0 when unreadable. */
357
+ function readProcRssKb(pid: number): number {
358
+ try {
359
+ const match = /^VmRSS:\s+(\d+)\s+kB/m.exec(fs.readFileSync(`/proc/${pid}/status`, "utf-8"));
360
+ return match ? Number.parseInt(match[1] ?? "", 10) : 0;
361
+ } catch {
362
+ return 0;
363
+ }
364
+ }
365
+
366
+ /** /proc/uptime first field (seconds since boot); undefined when unreadable. */
367
+ function readUptimeSec(): number | undefined {
368
+ try {
369
+ const seconds = Number.parseFloat(fs.readFileSync("/proc/uptime", "utf-8").split(" ")[0] ?? "");
370
+ return Number.isFinite(seconds) ? seconds : undefined;
371
+ } catch {
372
+ return undefined;
373
+ }
374
+ }
375
+
376
+ /**
377
+ * Wall-clock start of a candidate, from `starttime` ticks: boot = now - uptime,
378
+ * start = boot + starttime/CLK_TCK. undefined when either side is unreadable —
379
+ * an unprovable start time is a rejection, never a guess.
380
+ */
381
+ function defaultCandidateStartMs(identity: ProcIdentity): number | undefined {
382
+ if (identity.starttime === undefined) return undefined;
383
+ const ticks = Number(identity.starttime);
384
+ if (!Number.isFinite(ticks)) return undefined;
385
+ const uptimeSec = readUptimeSec();
386
+ if (uptimeSec === undefined) return undefined;
387
+ return Date.now() - uptimeSec * 1000 + (ticks / CLOCK_TICKS_PER_SEC) * 1000;
388
+ }
389
+
390
+ /** Liveness by pid: ESRCH/ENOENT ⇒ gone, EPERM ⇒ alive but not ours. */
391
+ function isPidAlive(pid: number): boolean {
392
+ if (!Number.isInteger(pid) || pid <= 1) return false;
393
+ try {
394
+ process.kill(pid, 0);
395
+ return true;
396
+ } catch (err) {
397
+ const code = (err as NodeJS.ErrnoException).code;
398
+ return code !== "ESRCH" && code !== "ENOENT";
399
+ }
400
+ }
401
+
402
+ // ── /proc snapshot (one per reconcile pass) ─────────────────────────────────
403
+
404
+ /**
405
+ * Capture the `/proc` view one reconcile pass reuses for every run (11b.4,
406
+ * LOW-4). Reads `/proc/<pid>/stat` + `status` once per pid instead of once per
407
+ * RUN, which is what made a full 25-run budget 25 full-host scans. Never throws:
408
+ * an unreadable `/proc` yields an empty snapshot (the sweep is best-effort).
409
+ *
410
+ * Only same-uid processes get their `environ` read — a foreign uid is rejected by
411
+ * safety rung 3 before its env is consulted (and its environ is unreadable, so
412
+ * an eager read would be a pile of EACCES throws on a busy host).
413
+ */
414
+ export function buildProcSnapshot(opts: SweepOptions = {}): ProcSnapshot {
415
+ const readIdentity = opts.readIdentity ?? readProcIdentity;
416
+ const readEnviron = opts.readEnviron ?? readProcEnviron;
417
+ const readTtyNr = opts.readTtyNr ?? readProcTtyNr;
418
+ const selfUid = (opts.getuid ?? defaultGetuid)();
419
+ const pids: number[] = [];
420
+ const entries = new Map<number, ProcSnapshotEntry>();
421
+ for (const pid of (opts.listPids ?? listProcPids)()) {
422
+ if (!Number.isInteger(pid) || pid <= 1) continue;
423
+ const identity = readIdentity(pid);
424
+ // Gone between readdir and read: nothing left to prove, nothing to reap.
425
+ if (!identity) continue;
426
+ const sameUid = identity.uid !== undefined && selfUid !== undefined && identity.uid === selfUid;
427
+ pids.push(pid);
428
+ entries.set(pid, { identity, environ: sameUid ? readEnviron(pid) : undefined, ttyNr: readTtyNr(pid) });
429
+ }
430
+ return { pids, entries };
431
+ }
432
+
433
+ // ── marker matching ─────────────────────────────────────────────────────────
434
+
435
+ /**
436
+ * Match a candidate's inherited env against the run markers, highest priority
437
+ * first (run-id → task-id → parent-pid). Returns the winning marker or
438
+ * undefined.
439
+ *
440
+ * `recordedParents` (runner + worker pids) is the set that may serve as a
441
+ * descent PROOF for a candidate's `PI_CREW_PARENT_PID`; `proofParents` is the
442
+ * wider set the `run-id` branch accepts (it also admits recorded surface pane
443
+ * pids — see MEDIUM-3 in the caller).
444
+ */
445
+ function matchMarkers(
446
+ environ: Record<string, string>,
447
+ markers: RunMarkers,
448
+ recordedParents: ReadonlySet<number>,
449
+ proofParents: ReadonlySet<number>,
450
+ taskIds: ReadonlySet<string>,
451
+ ): RunDescendant["matchedBy"] | undefined {
452
+ const parentPid = Number.parseInt(environ.PI_CREW_PARENT_PID ?? "", 10);
453
+ const hasParent = Number.isInteger(parentPid) && parentPid > 1;
454
+ // 11b.4 (HIGH-1, safety rung 11): the run id is NECESSARY but never
455
+ // sufficient. Every descendant of a run inherits it — including the shell an
456
+ // operator opens inside a worker's pane and a dev server a worker leaves
457
+ // running — so matching it alone turns any same-uid process that ever saw the
458
+ // env into a kill candidate. The candidate must additionally name a parent
459
+ // this run RECORDED and that parent must already be DEAD: exactly the descent
460
+ // proof the other two branches require, and exactly the shape of the incident
461
+ // survivors (`PI_CREW_PARENT_PID` = the recorded runner, dead for 24 h).
462
+ if (
463
+ environ.PI_CREW_BROKER_RUN_ID !== undefined &&
464
+ environ.PI_CREW_BROKER_RUN_ID === markers.runId &&
465
+ hasParent &&
466
+ proofParents.has(parentPid) &&
467
+ !isPidAlive(parentPid)
468
+ ) {
469
+ return "run-id";
470
+ }
471
+ // The worker shape: the process must carry the worker kind, belong to a task
472
+ // of THIS run, and name a parent the run actually recorded (the runner pid
473
+ // for a bash grandchild — the worker inherited PI_CREW_PARENT_PID from the
474
+ // runner — or a worker pid for a nested worker).
475
+ if (environ.PI_CREW_KIND === "subagent") {
476
+ const taskId = environ.PI_CREW_TASK_ID;
477
+ // 11b.1 (per-candidate proof): the recorded parent this candidate names
478
+ // must ALSO be gone. While it is alive the candidate may still be doing
479
+ // that worker's work, so a `task-id` match alone is not proof of a leak —
480
+ // the parent-pid branch below has always required the same thing.
481
+ if (taskId !== undefined && taskIds.has(taskId) && hasParent && recordedParents.has(parentPid) && !isPidAlive(parentPid)) {
482
+ return "task-id";
483
+ }
484
+ }
485
+ // The orphan shape of the incident survivors: only the dead recorded parent
486
+ // is available as proof.
487
+ if (hasParent && markers.workerPids.includes(parentPid) && !isPidAlive(parentPid)) return "parent-pid";
488
+ return undefined;
489
+ }
490
+
491
+ /**
492
+ * Enumerate the processes that PROVABLY belong to `markers.runId`. Pure
493
+ * discovery — never signals. Returns `[]` on any platform without /proc and
494
+ * when the kill switch is off (fail-open: the sweep is best-effort cleanup).
495
+ */
496
+ export function collectRunDescendants(markers: RunMarkers, opts: SweepOptions = {}): RunDescendant[] {
497
+ if (!resolveRunDescendantSweepEnabled()) return [];
498
+ if ((opts.platform ?? process.platform) !== "linux") return [];
499
+ const readIdentity = opts.readIdentity ?? readProcIdentity;
500
+ const candidateStartMs = opts.candidateStartMs ?? defaultCandidateStartMs;
501
+ const selfUid = (opts.getuid ?? defaultGetuid)();
502
+ // Our own group: never a candidate's target group (belt to reap-group's rung).
503
+ const selfPgrp = readIdentity(process.pid)?.pgid;
504
+ // 11b.4 (LOW-4): ONE /proc view per call — supplied by the caller (a whole
505
+ // reconcile pass) or built here (doctor, tests, the lone reconciler path).
506
+ const snapshot = opts.snapshot ?? buildProcSnapshot(opts);
507
+ const recordedParents = new Set<number>(markers.workerPids);
508
+ if (markers.runnerPid !== undefined) recordedParents.add(markers.runnerPid);
509
+ // 11b.3: pids the surface controller recorded for this run's panes.
510
+ const surfacePids = new Set<number>((markers.surfacePids ?? []).filter((pid) => Number.isInteger(pid) && pid > 1));
511
+ // 11b.4 (MEDIUM-3): the surface pids are ALSO acceptable descent proofs for
512
+ // the run-id branch. Today this set is redundant with the recorded parents
513
+ // (`manifest.surface.workerPids` holds worker pids, which are already
514
+ // recorded parents) — it is kept as a guard in case surface pids ever
515
+ // diverge from what the tasks recorded. Defence in depth, no added risk: a
516
+ // proof parent must still be DEAD, and every other rung still applies.
517
+ const proofParents = new Set<number>([...recordedParents, ...surfacePids]);
518
+ // Wave 11c (battery F2): the run's SPAWNER pid (`manifest.sessionPid`) is a
519
+ // recorded parent too — for a SYNC run it is the only pid its leaked
520
+ // descendants can name. Proof-parent only (NOT a recorded ancestor for the
521
+ // candidate-exclusion set): the session pi hosts the run, it is not a worker
522
+ // of it. The dead-only guard inside matchMarkers keeps a LIVE session from
523
+ // ever serving as proof — the operator may still be using that very pi.
524
+ if (markers.sessionPid !== undefined && Number.isInteger(markers.sessionPid) && markers.sessionPid > 1) {
525
+ proofParents.add(markers.sessionPid);
526
+ }
527
+ const taskIds = new Set<string>(markers.taskIds);
528
+ const ancestors = selfAncestors();
529
+ const descendants: RunDescendant[] = [];
530
+ for (const pid of snapshot.pids) {
531
+ if (!Number.isInteger(pid) || pid <= 1) continue;
532
+ // Never ourselves, never a process that hosts this session.
533
+ if (pid === process.pid || ancestors.has(pid)) continue;
534
+ // 11b.1 per-candidate rule: a RECORDED ancestor is the root of the tree we
535
+ // are cleaning up, never a descendant of it. Excluding it is what makes the
536
+ // sweep safe WITHOUT the old all-or-nothing "every recorded pid is dead"
537
+ // gate: a live worker keeps running (its own group reap / orphan reaper
538
+ // owns it) while the leaked grandchildren whose recorded parent is already
539
+ // gone are still reaped. Trade-off, stated plainly: a recorded pid that was
540
+ // recycled onto a genuine leaked descendant is skipped here — a miss, never
541
+ // a wrong kill.
542
+ if (recordedParents.has(pid)) continue;
543
+ // 11b.3: a recorded surface pane pid that is STILL ALIVE may be the pane the
544
+ // operator is watching right now (a run that just went terminal keeps its
545
+ // mux panes until they are closed). Never a candidate while alive; once it
546
+ // is dead the pid is gone from /proc anyway, so nothing is left to reap.
547
+ if (surfacePids.has(pid) && isPidAlive(pid)) continue;
548
+ const entry = snapshot.entries.get(pid);
549
+ if (!entry) continue;
550
+ const identity = entry.identity;
551
+ // Same uid only — an unknown uid cannot be proven to be ours either.
552
+ if (identity.uid === undefined || selfUid === undefined || identity.uid !== selfUid) continue;
553
+ // The PID-reuse-proof field, and the run-scope proof: must exist and must
554
+ // not predate the run.
555
+ if (identity.starttime === undefined) continue;
556
+ const startMs = candidateStartMs(identity);
557
+ if (startMs === undefined || startMs < markers.startedAtMs - PRE_RUN_SKEW_MS) continue;
558
+ // Never hand our own group (or our own pid as a pgid) to the reaper.
559
+ if (identity.pgid === process.pid || (selfPgrp !== undefined && identity.pgid === selfPgrp)) continue;
560
+ const environ = entry.environ;
561
+ if (!environ) continue;
562
+ const matchedBy = matchMarkers(environ, markers, recordedParents, proofParents, taskIds);
563
+ if (matchedBy === undefined) continue;
564
+ descendants.push({
565
+ pid,
566
+ pgid: identity.pgid,
567
+ starttime: identity.starttime,
568
+ uid: identity.uid,
569
+ matchedBy,
570
+ cmd: readProcCmdline(pid),
571
+ rssKb: readProcRssKb(pid),
572
+ ...(entry.ttyNr === undefined ? {} : { ttyNr: entry.ttyNr }),
573
+ });
574
+ }
575
+ return descendants;
576
+ }
577
+
578
+ // ── idempotency: one signal per group incarnation ───────────────────────────
579
+ //
580
+ // The stale reconciler runs on a timer and may look at the same terminal run
581
+ // more than once. Re-signalling an already-reaped group is not just noise: it
582
+ // is a second chance to hit a RECYCLED pgid (the group drained in between). The
583
+ // registry therefore remembers `pgid:leader-starttime` for every accepted
584
+ // signal — the leader starttime makes a new incarnation a NEW key — and is
585
+ // bounded so a long-lived host cannot grow it without limit. An escalation that
586
+ // ended `stuck`/`refused` is forgotten, so the next tick may retry it.
587
+ const signalledGroups = new Map<string, number>();
588
+
589
+ function rememberSignalledGroup(key: string): void {
590
+ signalledGroups.set(key, Date.now());
591
+ while (signalledGroups.size > MAX_TRACKED_GROUPS) {
592
+ const oldest = signalledGroups.keys().next();
593
+ if (oldest.done) break;
594
+ signalledGroups.delete(oldest.value);
595
+ }
596
+ }
597
+
598
+ /** Test seam: forget every remembered group (idempotency reset). */
599
+ export function __resetSignalledGroupsForTest(): void {
600
+ signalledGroups.clear();
601
+ }
602
+
603
+ /** Test seam: how many group incarnations are remembered. */
604
+ export function __signalledGroupCountForTest(): number {
605
+ return signalledGroups.size;
606
+ }
607
+
608
+ /** Default group signal — defensively refuses non-positive pgids. */
609
+ function defaultKillGroup(pgid: number, signal: NodeJS.Signals): void {
610
+ if (!Number.isInteger(pgid) || pgid <= 1) return;
611
+ try {
612
+ process.kill(-pgid, signal);
613
+ } catch {
614
+ /* group already gone — nothing to deliver */
615
+ }
616
+ }
617
+
618
+ /** Default group liveness: `kill(-pgid, 0)`; EPERM means alive but not ours. */
619
+ function defaultGroupAlive(pgid: number): boolean {
620
+ if (!Number.isInteger(pgid) || pgid <= 1) return false;
621
+ try {
622
+ process.kill(-pgid, 0);
623
+ return true;
624
+ } catch (err) {
625
+ return (err as NodeJS.ErrnoException).code === "EPERM";
626
+ }
627
+ }
628
+
629
+ /**
630
+ * Reap the proven descendants of a TERMINAL run. Every signal goes through the
631
+ * guarded group ladder (`reapGroup`), one signal per group incarnation; the
632
+ * candidates that share a pgid are reported individually but signalled once.
633
+ * Never throws.
634
+ *
635
+ * MUST only be called for a TERMINAL run. 11b.1: the recorded ancestors need NOT
636
+ * be dead any more — a recorded pid is never a candidate (see
637
+ * `collectRunDescendants`), so a live worker cannot veto reaping the leaked
638
+ * grandchildren whose own recorded parent is gone. `stale-reconciler.ts`
639
+ * enforces the terminal-run precondition before calling, and the doctor path is
640
+ * `dryRun`.
641
+ */
642
+ export function reapRunDescendants(markers: RunMarkers, opts: SweepOptions = {}): RunSweepResult {
643
+ const result: RunSweepResult = { reaped: [], skipped: [], pending: [], descendants: [] };
644
+ if (!resolveRunDescendantSweepEnabled()) return result;
645
+ // Same platform gate `collectRunDescendants` applies — checked here so a
646
+ // non-Linux host never pays for a /proc snapshot it cannot use.
647
+ if ((opts.platform ?? process.platform) !== "linux") return result;
648
+ // 11b.4 (LOW-4): ONE snapshot for discovery AND for the leader/identity
649
+ // lookups below. A caller that runs several sweeps in one reconcile pass
650
+ // (`crash-recovery`'s terminal lane) passes its own; everyone else gets a
651
+ // per-call view, unchanged from before.
652
+ const snapshot = opts.snapshot ?? buildProcSnapshot(opts);
653
+ const descendants = collectRunDescendants(markers, { ...opts, snapshot });
654
+ result.descendants = descendants;
655
+ if (descendants.length === 0) return result;
656
+ const readIdentity = opts.readIdentity ?? readProcIdentity;
657
+ // Snapshot first (cheap, already read); a pid missing from it appeared after
658
+ // the capture ⇒ the live read is the fresher truth.
659
+ const identityOf = (pid: number): ProcIdentity | undefined => snapshot.entries.get(pid)?.identity ?? readIdentity(pid);
660
+ const signalGroup = opts.killGroup ?? defaultKillGroup;
661
+ const groupAlive = opts.groupAlive ?? defaultGroupAlive;
662
+ const selfPgrp = readIdentity(process.pid)?.pgid;
663
+ const matchedPids = new Set<number>(descendants.map((descendant) => descendant.pid));
664
+ // Candidates are grouped by the group we would signal: one reapGroup call per
665
+ // pgid, so a sibling pair of bash grandchildren cannot be signalled twice.
666
+ const byGroup = new Map<number, RunDescendant[]>();
667
+ for (const descendant of descendants) {
668
+ const pgid = descendant.pgid;
669
+ if (
670
+ pgid === undefined ||
671
+ !Number.isInteger(pgid) ||
672
+ pgid <= 1 ||
673
+ pgid === process.pid ||
674
+ (selfPgrp !== undefined && pgid === selfPgrp)
675
+ ) {
676
+ result.skipped.push({ pid: descendant.pid, reason: "no-pgid" });
677
+ continue;
678
+ }
679
+ const members = byGroup.get(pgid);
680
+ if (members) members.push(descendant);
681
+ else byGroup.set(pgid, [descendant]);
682
+ }
683
+ for (const [pgid, members] of byGroup) {
684
+ // Report-only pass (doctor): nothing is signalled by construction, and the
685
+ // reconciler keys its "do not consume the marker" rule on THIS reason — so
686
+ // it is decided first, before any skip that could otherwise win the report
687
+ // (a dry run must never look like a retryable skip).
688
+ if (opts.dryRun) {
689
+ for (const member of members) result.skipped.push({ pid: member.pid, reason: "dry-run" });
690
+ continue;
691
+ }
692
+ // Safety rung 12 (11b.4, HIGH-1 follow-up): an INTERACTIVE candidate — any
693
+ // member holding a controlling terminal — is never signalled. This is the
694
+ // operator's shell (or a tool attached to it), not a run resource, and
695
+ // there is nothing to "clean up" about it. Applied per GROUP because the
696
+ // signal is per group: if any member is interactive, the whole group is
697
+ // left alone (a documented miss, never a wrong kill).
698
+ if (members.some((member) => (member.ttyNr ?? 0) !== 0)) {
699
+ for (const member of members) result.skipped.push({ pid: member.pid, reason: "interactive-tty" });
700
+ continue;
701
+ }
702
+ // Sweep-only rung: a live leader that is NOT itself a matched candidate
703
+ // means this group belongs to a session that is still running (or to a
704
+ // stranger) — never signal it. A dead leader, or a leader that is one of
705
+ // our own candidates, is safe: reapGroup re-validates identity anyway.
706
+ const leader = identityOf(pgid);
707
+ if (leader && !matchedPids.has(pgid)) {
708
+ logInternalError(
709
+ "run-descendant-sweep.leader-alive-unmatched",
710
+ new Error(`pgid ${pgid} has a live leader ${pgid} that carries no marker of run ${markers.runId}`),
711
+ `candidates=${members.map((member) => member.pid).join(",")}`,
712
+ "debug",
713
+ );
714
+ for (const member of members) result.skipped.push({ pid: member.pid, reason: "leader-alive-unmatched" });
715
+ continue;
716
+ }
717
+ const groupKey = `${pgid}:${leader?.starttime ?? "dead"}`;
718
+ if (signalledGroups.has(groupKey)) {
719
+ for (const member of members) result.skipped.push({ pid: member.pid, reason: "already-swept" });
720
+ continue;
721
+ }
722
+ const anchor = members[0];
723
+ if (anchor === undefined) continue;
724
+ const expected: ProcIdentity = { pid: anchor.pid, pgid, starttime: anchor.starttime, uid: anchor.uid };
725
+ const reap = reapGroup(pgid, anchor.pid, {
726
+ expected,
727
+ readIdentity,
728
+ signalGroup,
729
+ groupAlive,
730
+ name: `run-sweep:${markers.runId}`,
731
+ ...(opts.gracefulMs !== undefined ? { gracefulMs: opts.gracefulMs } : {}),
732
+ ...(opts.capMs !== undefined ? { capMs: opts.capMs } : {}),
733
+ ...(opts.platform !== undefined ? { platform: opts.platform } : {}),
734
+ // F5 propagation (wave 12b): the uid source MUST reach the ladder too.
735
+ // Discovery already used `opts.getuid`; forwarding only at that half would
736
+ // let a caller assert one uid policy and execute another (the sweep would
737
+ // prove a group safe against the injected uid while the ladder compared
738
+ // against the real one) — a silent split-brain, worse than no seam at all.
739
+ ...(opts.getuid !== undefined ? { getuid: opts.getuid } : {}),
740
+ });
741
+ if (reap.refusal !== undefined) {
742
+ // Nothing was signalled. Leave the key unset so a later tick can retry
743
+ // once the group/identity state changes.
744
+ logInternalError(
745
+ "run-descendant-sweep.refused",
746
+ new Error(`group reap refused for pgid ${pgid} (${reap.refusal})`),
747
+ `run=${markers.runId} pids=${members.map((member) => member.pid).join(",")}`,
748
+ "debug",
749
+ );
750
+ for (const member of members) result.skipped.push({ pid: member.pid, reason: `group-refused:${reap.refusal}` });
751
+ continue;
752
+ }
753
+ rememberSignalledGroup(groupKey);
754
+ // A no-op catch keeps a stuck/failed escalation from wedging the registry:
755
+ // the group did not finish dying, so the next sweep may try again.
756
+ void reap.pending.then(
757
+ (outcome) => {
758
+ if (outcome === "stuck" || outcome === "refused") signalledGroups.delete(groupKey);
759
+ },
760
+ () => signalledGroups.delete(groupKey),
761
+ );
762
+ result.pending.push(reap.pending);
763
+ for (const member of members) result.reaped.push(member);
764
+ }
765
+ return result;
766
+ }