pi-crew 0.11.38 → 0.11.39

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pi-crew",
3
- "version": "0.11.38",
3
+ "version": "0.11.39",
4
4
  "description": "Pi extension for coordinated AI teams, workflows, worktrees, and async task orchestration",
5
5
  "author": "baphuongna",
6
6
  "license": "MIT",
@@ -118,7 +118,7 @@
118
118
  "smoke:pi": "pi install .",
119
119
  "smoke:release": "node scripts/release-smoke.mjs",
120
120
  "prepack": "node scripts/clean-strip-types.mjs",
121
- "prepublishOnly": "npm run ci",
121
+ "prepublishOnly": "node scripts/prepublish-ci.mjs",
122
122
  "test:fast": "npm run test:critical && npm run test:bundle",
123
123
  "check:wc-gate": "node scripts/wc-gate.mjs"
124
124
  },
@@ -119,6 +119,7 @@ End-to-end verification discipline for pi-crew changes. Distilled from the broke
119
119
  | `src/ui/` | T1 → (T5|T6) → T13 |
120
120
  | `src/config/` (kể cả migration-validator) | T1 → T11c |
121
121
  | `src/runtime/child-pi/` (spawn/kill/steer) | T7 → T9g |
122
+ | `src/runtime/process/**` (reap-group.ts / run-descendant-sweep.ts) + `child-pi-kill.ts` exited-root branch + `orphan-reaper.ts` identity admission + `crash-recovery.ts` terminal-sweep lane + state fields `childPgid/childStartTime/childUid` + heartbeat `pgid/starttime/uid` (wave-11 class: runtime spawn/kill/teardown + startup reconcile + state schema ADDITIVE) | T1 → pinned unit `test/unit/runtime/process/{reap-group,run-descendant-sweep}.test.ts` + `test/unit/runtime/recovery/reconcile-terminal-run-descendant-sweep.test.ts` + `test/unit/runtime/stale-reconciler-run-descendant-sweep.test.ts` + `test/unit/runtime/stale-reconciler-reap-escalation.test.ts` (5 file / 78 tests @ `f578c223`) → **integration live repro** `node scripts/test-runner.mjs test/integration/run-descendant-sweep-live.test.ts` (A1–A5: spawn leak THẬT → sweep reap ~100 ms; **negative control bắt buộc SỐNG**: no-marker / interactive-tty / parent-còn-sống) → off-path proof `PI_CREW_REAP_GROUP=0` + `PI_CREW_REAP_RUN_DESCENDANTS=0` (unit xanh trên CẢ hai trạng thái switch; KHÔNG để ambient `=0` sót lại làm run thật câm) → T9a `team action='doctor' focus='zombies'` (mảng `runDescendants` report-only + dòng cảnh báo no descent proof ⇒ NOT reaped) |
122
123
  | `src/runtime/mcp-proxy.ts` / `src/runtime/model/pi-args.ts` (worker argv build + MCP role-gate — G2/G3 surface, SDD-2) | T7 (`argv-flags.smoke.ts` với `PI_CREW_SMOKE=1`) → T9b-W loadout theo role (post-G2: read-only KHÔNG có MCP trên live-session path là ĐÚNG — xem mục full-loadout sanity 9b) + pinned suites `mcp-proxy.test.ts` / `phase6-runtime-hardening.test.ts` / `child-pi-timeout`+`child-pi-surface` (argv/spill) |
123
124
  | `src/runtime/async-runner.ts` / `src/runtime/background-runner.ts` (spawn, allowlist, entry-guard, `executeBackgroundRun` core) | T1 → T9b async (event `async.spawned` KHÔNG được có `data.inline:true` — battery phải chứng minh runner DETACHED thật) → T7; đụng `BACKGROUND_RUNNER_ENV_ALLOWLIST` → thêm T10b/T10c (quirk F4-2026-08-30: mux env bị strip ⇒ async headless âm thầm) |
124
125
  | `src/runtime/surface/` | T10a → T10b → T10c (dò herdr bằng `HERDR_ENV=1`, KHÔNG phải `$TMUX`) |
@@ -189,6 +190,10 @@ End-to-end verification discipline for pi-crew changes. Distilled from the broke
189
190
 
190
191
  > **2026-10-22 update (wave 10 — pi-subagents lessons port, v0.11.38, commits `b1288e3b`..`0ca9b552`, battery report `docs/real-test/reports/real-test-2026-10-22-wave10-pisub-lessons-battery.md`)**: 10 ports — W10-1 worker cache-retention split (`PI_CACHE_RETENTION=short` cho headless single-run; surface/RPC warm giữ parent default); W10-2 memoized green verify-gate theo tree OID (xem row verify-gate); W10-3 mailbox typed reason enum + strict replyTo routing (+ hardening `c95bb761`: findAskResponse iterate-past-mismatch, misroute-then-resend không park tới deadline, taskId-anchored filter); W10-4 `check-slop-rules` gate (3 scanner, ratchet 41 — row mới); W10-5 decision-drift guardian (critic/oracle inherited-decisions + DRIFT-BLOCK → verdict degrade `none`, fail-closed); W10-6 budget-boundary policy text (delegation-patterns: budgets chặn read-tools KHÔNG phải mutation-tools; elapsed-timeout KHÔNG phải mutation-safe boundary); W10-7 usage baseline-reconcile (per-attempt usage + note observability-only khi gap >5%); W10-8 read-lane same-tree guard (`git status --porcelain` gồm untracked → note `unexpectedMutation`, leader quyết); W10-9 typed JSON verification gates (row verify-gate); W10-10 CHANGELOG wave-10 + 2 lessons evidence-closed (spawn/parallel REVISED-BY-MEASUREMENT, declaration-pinning NOT-APPLICABLE). Battery coverage T1/T3/T4/T8/T7×3/T9a/T9g/T11a/c/j/T12 — PASS (unit 9876, integration 143, slop/lazy/staleness/size/bundle exit 0). Live-caught: verifier worker `cacheRead:11072` (prompt-cache reuse trên worker — evidence trực quan cho fork-cache PoC); T9b-W partial-honest (3-role spawn-path xanh thay full-loadout checklist). Hướng dẫn: đổi `src/runtime/verification/` hoặc task-runner post-execution → ngoài rows hiện có THÊM probe memo-hit (re-gate same-tree same-host → `verify-gate-memo: HIT`); đổi `prompt-runtime.ts` findAskResponse → chạy `ask-tool-replyto-routing.test.ts` (5) + `mailbox-replyto-reason.test.ts` (8) — 13 tests pin misroute-resend + taskId-anchor.
191
192
 
193
+ > **2026-10-24 update (wave 11 — worker descendant reaping, commits `ed32c582`→`f578c223` gồm dist rebuild; report `docs/real-test/reports/real-test-2026-10-24-wave11-descendant-reaping.md`)**: (1) **Root cause KHÔNG phải "worker quên kill con"** — pi's bash tool spawn MỖI command `detached:true` (mỗi command một process group riêng, là con của pi worker chứ không nằm trong group worker) nên pid-only SIGTERM không bao giờ tới cháu; run `team_20261022110510_cefb9164af126460` terminal xong mà 6 cháu bash sống 24 h (~390 MB RSS). (2) **Surface mới `src/runtime/process/reap-group.ts` (F5)** — NƠI DUY NHẤT được signal một POSIX process group: `reapGroup` + `assertSafeGroup` (ladder từ chối 8 rung: POSIX + /proc đọc được; pgid nguyên > 1 — không bao giờ `kill(-1/0/1)`; không phải group/pid/parent của MÌNH theo REAL pgrp đọc từ `/proc` qua `selfIdentity()` — fix CRITICAL #1 host `setsid`; leader-proof = identity sống HOẶC snapshot caller chứng minh `pid===pgid` leader + starttime — snapshot không còn là identity thay thế, fix MAJOR #2 blind kill; same-uid; real-pgid == requested (PID-reuse); starttime bắt buộc; snapshot-starttime khớp live) + escalation SIGTERM (đồng bộ, postmortem-safe) → bounded poll → SIGKILL (timer unref — `drainReapEscalations()` / `flushReapEscalationsSync()` sở hữu rung cuối trên session shutdown). Wired: `child-pi-kill.ts` nhánh exited-root (snapshot pgid/starttime/uid TRƯỚC khi root chết; 60 s sweep prune) + `orphan-reaper.ts` admission bằng identity-proven (không còn tin `allowedPids` mù; scrub `PI_CREW_*` vẫn bắt buộc trước test). Kill switch `PI_CREW_REAP_GROUP=0`, default ON, FORWARD sang background runner (env-allowlist — switch không forward = câm im lặng). (3) **Surface mới `src/runtime/process/run-descendant-sweep.ts` (F7)** — `collectRunDescendants`/`reapRunDescendants` quét `/proc` (1 snapshot/pass, `buildProcSnapshot`) match marker mạnh→yếu: `run-id` (`PI_CREW_BROKER_RUN_ID` — 11b.4: marker KHÔNG phải proof, nhánh còn đòi một parent ĐÃ GHI ÂM (runner/worker/surface pane) ĐÃ CHẾT), `task-id` (`PI_CREW_KIND=subagent`+`PI_CREW_TASK_ID`+dead parent), `parent-pid` (`PI_CREW_PARENT_PID` chết — đúng dạng sự cố). Guard: same-uid, không phải self/ancestor, starttime ≥ runStart−60 s, **skip tty (lý do `interactive-tty`)**, leader chết hoặc là candidate khác, `surfacePids` sống bị loại. Kill switch `PI_CREW_REAP_RUN_DESCENDANTS=0` (no-op hoàn toàn, cũng forward sang runner). (4) **Startup lane** `crash-recovery.ts` `selectUnsweptTerminalRunIds`: mọi run terminal (`completed|failed|cancelled`) chưa có `descendantSweepAt`, `updatedAt` ≥ 60 s (grace cho pane), CŨ NHẤT TRƯỚC, cap 25/pass, window 200; `descendantSweepPending` retry sau 10 phút (HIGH-2: sweep-fail không leak vĩnh viễn). (5) **Doctor** `focus=zombies`: mảng `runDescendants` (report-only, dry-run) + cảnh báo process không chứng minh được descent ⇒ NOT reaped. (6) **State additive**: `TaskCheckpointState.childPgid/childStartTime/childUid` + `WorkerHeartbeatState.pgid/starttime/uid` — optional, record cũ degrade pid-only. (7) **Battery** (decision-table row mới): T1 + 5 unit suite (78 tests) + integration live repro A1–A5 (leak thật bị reap 91–121 ms, 3 negative control sống, 3/3 chạy ổn định) + off-path `=0` cả 2 switch + doctor dry-run; **ambient `PI_CREW_REAP_*=0` làm live repro câm — scrub trước khi chạy thật** (row Failure symptoms mới).
194
+
195
+ > **2026-10-24 đính chính (wave 11c — battery findings F1+F2, mục 4b/7b của `docs/real-test/reports/real-test-2026-10-24-wave11-battery.md`)**: (1) **F2 sync-run miss ĐÃ FIX** — run sync/foreground không có `manifest.async.pid` nên leak mang `PI_CREW_PARENT_PID=<session pi pid>` không bao giờ qua descent proof (đúng rule 11 — an toàn nhưng bỏ sót vĩnh viễn). Từ 11c mọi manifest mới record `sessionPid` (= `process.pid` tại `createRunManifest`); `sweepTerminalRunDescendants` + doctor dry-run đưa nó vào `proofParents` với guard dead-only như mọi parent khác — session còn sống ⇒ KHÔNG là proof (user có thể đang dùng). Run async vẫn dùng `async.pid` primary. Battery addendum sẽ chứng minh bằng E2E sync-leak thật. (2) **F1 test-isolation ĐÃ FIX** — suite `run-descendant-sweep.test.ts` giờ pin `PI_CREW_REAP_RUN_DESCENDANTS=1` trong beforeEach (restore verbatim afterEach, pattern của `stale-reconciler-run-descendant-sweep`): 17/78 test không đỏ nữa khi ambient `=0`; pinned 78→82 (a20 + g8–g10); decision-table yêu cầu "unit xanh trên CẢ HAI trạng thái switch" giữ nguyên.
196
+
192
197
  ## Core principle: disk ≠ live Pi
193
198
 
194
199
  Two locations hold pi-crew state:
@@ -1258,6 +1263,7 @@ When a tier fails, the recovery is usually quick. Match the symptom to the cause
1258
1263
  | `team action='goal' budgetTotal=100000` rejected at validation ("must be equal to constant / must be number") | pi-ai stringifies numeric args on `Literal("")` unions; budgetTotal lacked the stringified-number branch its sibling budget params have. Fixed 2026-09-21 | Workaround pre-fix: `config.budgetUnlimited:true` (note: in config, NOT top-level) |
1259
1264
  | `team action='schedule' interval=<ms>` always says `Invalid schedule "<N>ms". Use "5m", …` | handle-schedule builds `${interval}ms`; both interval parsers lacked an `ms` unit — every numeric interval died. Fixed 2026-09-21 | The suggested "5m" formats only work via `cron`/`once` params, not `interval` (interval is schema-typed as a plain ms number) |
1260
1265
  | Runs keep appearing under one scheduled job; children of YOUR pi process; machine thrashing | **Interval ≥ 2^31-1 ms (≥24.8 days) overflowed the Node timer to a 1ms hot loop** (fixed 2026-09-22 — clamped hops). Check `.crew/audit/prune.jsonl` for interleaved auto-prunes. Contain: remove job, then `pkill -f '[b]ackground-runner.ts --cwd <root>'` | Any new timer scheduling must clamp below MAX_TIMER_DELAY_MS — grep for `setInterval(`/`setTimeout(` with computed delays |
1266
+ | Live sweep/reap probe chạy "xanh" nhưng KHÔNG process nào bị reap (doctor `runDescendants` liệt kê mà không có gì chết) | **Ambient `PI_CREW_REAP_GROUP=0` / `PI_CREW_REAP_RUN_DESCENDANTS=0` còn trong shell** (sót từ probe off-path trước) — cả 2 switch default ON và được FORWARD sang background runner, ambient là đủ để câm sạch cả sweep lẫn group-reap; ngược lại quên set khi test off-path thì nhánh degrade không được chứng minh | Trước MỌI live repro: `env \| grep PI_CREW_REAP` phải RỖNG; chỉ set `=0` CHỦ ĐÍCH khi chạy nhánh off-path (unit phải xanh trên CẢ hai trạng thái switch), unset ngay sau (wave 11 lesson — 2 kill switch KHÔNG được set ngoài ý muốn theo cả 2 hướng) |
1261
1267
 
1262
1268
  ## Release gate — battery → push → GitHub Actions GREEN → publish (canonical, 2026-10-14)
1263
1269
 
@@ -1270,7 +1276,7 @@ corpse → xóa file socket + rebind (STEAL lock) — fix bằng 3-strike takeov
1270
1276
  **Quy trình BẮT BUỘC cho mọi release (theo thứ tự, không bỏ bước):**
1271
1277
 
1272
1278
  1. Cập nhật skill (wave entry) + CHANGELOG heading cho version sắp release — commit.
1273
- 2. Chạy battery theo decision table (đủ tier cho paths đổi) — xanh hết.
1279
+ 2. Chạy battery theo decision table (đủ tier cho paths đổi) — xanh hết, **kể cả integration live repro khi row đó yêu cầu** (vd wave 11 → `test/integration/run-descendant-sweep-live.test.ts` A1–A5 PHẢI xanh trước publish — live repro là tier duy nhất chứng minh sweep giết leak thật mà không đụng process của operator).
1274
1280
  3. `npm version patch` (tạo release commit + tag).
1275
1281
  4. `git push origin main --follow-tags` — pre-push hook tự chạy full unit local.
1276
1282
  5. **`gh run watch <runId> --exit-status` (hoặc poll `gh run list`) — CHỜ matrix 3-OS XANH TOÀN BỘ.**
@@ -1434,6 +1440,8 @@ Use this to answer "đủ full tính năng chưa?" without re-deriving. Every us
1434
1440
  | No-op writer gating (metric-sink crewRoot, prune-audit bail) | `src/observability/metric-sink.ts`, `src/extension/run-maintenance.ts` | `test/unit/extension/no-op-writers-no-crew-root.test.ts` (8) |
1435
1441
  | Worker-dashboard keybinding override (`.pi/teams`) | `src/ui/keybinding-map.ts` | `test/unit/ui/keybinding-map-override.test.ts` (8) |
1436
1442
  | Steer delivery incl. boot window (pre-bind retry) | `src/prompt/prompt-runtime.ts` (poll + `createSeenSteerIdSet.unmark`), `src/runtime/child-pi/child-pi-spawn.ts:283` (env), `src/runtime/task-runner/child-executor.ts:562` (filename) | `prompt-runtime-steer-boot-window.test.ts` (4) + **Tier 9g** live (watcher at +0.5s; lab variant with `PI_CREW_STEERING_FILE` + `pi --extension <repo>/src/prompt/prompt-runtime.ts`) |
1443
+ | Worker process-GROUP reaping (wave 11a F5 — đường group-signal DUY NHẤT) | `src/runtime/process/reap-group.ts` (`reapGroup` / `assertSafeGroup` guard-ladder 8 rung + escalation SIGTERM→poll→SIGKILL + `flushReapEscalationsSync`), wired `child-pi-kill.ts` exited-root (snapshot pgid/starttime trước khi root chết) + `orphan-reaper.ts` identity-proven admission | unit `node scripts/test-runner.mjs test/unit/runtime/process/reap-group.test.ts`; kill switch `PI_CREW_REAP_GROUP=0` (default ON — degrade về pid-only pre-wave; được FORWARD sang background runner); live: spawn worker có group con → kill root → group bị reap; **negative control bắt buộc: process cùng uid nhưng ngoài ladder (không leader-proof / không snapshot khớp) phải SỐNG** |
1444
+ | Run-descendant sweep (wave 11b F7 — reap leak sống sót SAU run terminal) | `src/runtime/process/run-descendant-sweep.ts` (`collectRunDescendants`/`reapRunDescendants`, 1 snapshot `/proc`/pass) + `crash-recovery.ts` `selectUnsweptTerminalRunIds` lane (terminal ≥ 60 s, oldest-first, cap 25/pass, window 200, retry `descendantSweepPending` 10 phút) | unit sweep suites (xem decision-table row wave 11) + **integration live repro** `node scripts/test-runner.mjs test/integration/run-descendant-sweep-live.test.ts` (A1–A5) + report `docs/real-test/reports/real-test-2026-10-24-wave11-descendant-reaping.md`; kill switch `PI_CREW_REAP_RUN_DESCENDANTS=0` (no-op hoàn toàn); marker ladder `PI_CREW_BROKER_RUN_ID` (nhánh run-id cần recorded parent ĐÃ CHẾT) → `PI_CREW_KIND=subagent`+`PI_CREW_TASK_ID` → `PI_CREW_PARENT_PID`; **negative control bắt buộc: no-marker / interactive-tty / parent-sống phải SỐNG** |
1437
1445
 
1438
1446
  ---
1439
1447
 
@@ -225,6 +225,14 @@ const CREW_ENV_FORWARDING_DECL: Record<string, CrewEnvForwardDecision> = {
225
225
  kind: "gap",
226
226
  reason: "v0.11.36 sandbox network profile (share|off) read in the PARENT spawn path only — deliberately never forwarded to children",
227
227
  },
228
+ PI_CREW_REAP_GROUP: allowlist(
229
+ ["background-runner"],
230
+ "fix-wave 11a process-group reap kill switch — read by the reaping code (reap-group.ts resolveReapGroupEnabled) which ALSO runs inside the detached background runner; unforwarded it would be silently inert there (wave 11a review MINOR #7). Not forwarded to children: a worker must not be able to disable its own reaping",
231
+ ),
232
+ PI_CREW_REAP_RUN_DESCENDANTS: allowlist(
233
+ ["background-runner"],
234
+ "fix-wave 11b run-scoped descendant sweep kill switch (run-descendant-sweep.ts resolveRunDescendantSweepEnabled) — the stale reconciler that drives the sweep also runs inside the detached background runner, so the same inert-switch argument as PI_CREW_REAP_GROUP applies. Not forwarded to children: a worker must not be able to disable its own reaping",
235
+ ),
228
236
  PI_CREW_STEERING_FILE: { kind: "child-write", reason: "child-pi-spawn per-run steering JSONL" },
229
237
  PI_CREW_MAX_OUTPUT: { kind: "child-write", reason: "pi-args emits agent.maxTokens into built.env" },
230
238
  PI_CREW_RPC_SECRET: { kind: "child-write", reason: "shared HMAC secret handed to the child at spawn — secret name, never allowlisted" },
@@ -461,9 +469,13 @@ const BG_CREW_CORE_ORDER = [
461
469
  const BG_CREW_MODEL_ORDER = ["PI_CREW_MODEL", "PI_CREW_MODEL_FALLBACK_ORDER"] as const;
462
470
  const BG_CREW_STATE_ORDER = ["PI_CREW_STATE_BACKEND", "PI_CREW_CHILD_PI_MODE", "PI_CREW_CHILD_PI_POOL"] as const;
463
471
  const BG_CREW_TOOLS_ORDER = ["PI_CREW_TOOLS_MODIFIERS"] as const;
472
+ // Wave 11a review (MINOR #7): the reap kill switch must be honoured by the
473
+ // detached runner too — the runner (and the surface workers it spawns) own
474
+ // kill/reap paths, so an unforwarded switch was silently inert there.
475
+ const BG_CREW_REAP_ORDER = ["PI_CREW_REAP_GROUP", "PI_CREW_REAP_RUN_DESCENDANTS"] as const;
464
476
  const CHILD_CREW_DEPTH_ORDER = ["PI_CREW_DEPTH"] as const;
465
477
 
466
- const BG_CREW_GROUPS = [BG_CREW_CORE_ORDER, BG_CREW_MODEL_ORDER, BG_CREW_STATE_ORDER, BG_CREW_TOOLS_ORDER] as const;
478
+ const BG_CREW_GROUPS = [BG_CREW_CORE_ORDER, BG_CREW_MODEL_ORDER, BG_CREW_STATE_ORDER, BG_CREW_TOOLS_ORDER, BG_CREW_REAP_ORDER] as const;
467
479
 
468
480
  /**
469
481
  * Generate the detached background-runner allowlist (exported AS
@@ -488,6 +500,7 @@ export function backgroundRunnerEnvAllowlist(): string[] {
488
500
  ...BG_CREW_MODEL_ORDER,
489
501
  ...BG_CREW_STATE_ORDER,
490
502
  ...BG_CREW_TOOLS_ORDER,
503
+ ...BG_CREW_REAP_ORDER,
491
504
  ];
492
505
  }
493
506
 
@@ -321,6 +321,18 @@ export const CREW_ENV_VARS: Record<string, CrewEnvVarSpec> = {
321
321
  default: 7_200_000,
322
322
  doc: "max background run duration; default 2h (background-runner.ts:31)",
323
323
  },
324
+ PI_CREW_REAP_GROUP: {
325
+ name: "PI_CREW_REAP_GROUP",
326
+ parser: "boolean",
327
+ default: true,
328
+ doc: "wave 11a kill switch for identity-proven PROCESS-GROUP reaping: '0'/'false' degrades child-pi-kill + orphan-reaper to their pre-wave pid-only paths (resolveReapGroupEnabled, reap-group.ts). FORWARDED to the detached background runner (env-allowlist-registry: PI_CREW_REAP_GROUP → background-runner) because the runner/worker process is the one that reaps its own children — an unforwarded switch would be silently inert there",
329
+ },
330
+ PI_CREW_REAP_RUN_DESCENDANTS: {
331
+ name: "PI_CREW_REAP_RUN_DESCENDANTS",
332
+ parser: "boolean",
333
+ default: true,
334
+ doc: "wave 11b kill switch for the RUN-SCOPED descendant sweep: '0'/'false' makes collectRunDescendants/reapRunDescendants (run-descendant-sweep.ts) a complete no-op. FORWARDED to the detached background runner (env-allowlist-registry: → background-runner) for the same reason as PI_CREW_REAP_GROUP: the reconciler that drives the sweep also runs there. Never forwarded to a worker",
335
+ },
324
336
  PI_CREW_MAX_WORKERS: {
325
337
  name: "PI_CREW_MAX_WORKERS",
326
338
  parser: "int",
@@ -42,6 +42,7 @@ import { tryRegisterSessionCleanup } from "../../runtime/session-resources.ts";
42
42
  import { createSessionSnapshot } from "../../runtime/session-snapshot.ts";
43
43
  import { applyCrewSettingsTiersToConfig, loadCrewSettingsTiers, scheduledJobsHiddenCountOf } from "../../runtime/settings-store.ts";
44
44
  import type { ReconcileResult } from "../../runtime/stale-reconciler.ts";
45
+ import { flushReapEscalationsSync } from "../../runtime/stale-reconciler.ts";
45
46
  import { loadRunManifestById, saveRunManifest } from "../../state/stores/state-store.ts";
46
47
  import type { TeamRunManifest } from "../../state/types.ts";
47
48
  import { installInlinePanel } from "../../ui/inline-panel/index.ts";
@@ -166,6 +167,17 @@ function installSessionShutdownHandler(pi: ExtensionAPI, ctx: RegistrationContex
166
167
  if (reason === "quit" || reason === "reload") {
167
168
  // Actual shutdown — abort foreground runs and cleanup everything
168
169
  ctx.cleanupRuntime();
170
+ // Wave 11a review (MAJOR #3): the group-reap SIGKILL rung is a chain of
171
+ // UNREF'd timers owned by an async escalation, and this handler is SYNC —
172
+ // without this flush the last rung is abandoned the moment the process
173
+ // exits. Delivered AFTER `cleanupRuntime()` so escalations armed by that
174
+ // cleanup are included. Each entry re-validates before signalling and the
175
+ // call never throws.
176
+ try {
177
+ flushReapEscalationsSync();
178
+ } catch (error) {
179
+ logInternalError("register.session-shutdown.flush-reap-escalations", error);
180
+ }
169
181
  } else {
170
182
  // Session switch (resume/new/fork) — cleanup resources but preserve foreground runs
171
183
  ctx.cleanupSessionResourcesOnly();
@@ -10,6 +10,7 @@ import { getPiTempBase } from "../../runtime/model/pi-args.ts";
10
10
  import { getRuntimeWarmupStatus } from "../../runtime/model/runtime-warmup.ts";
11
11
  import { currentSessionModel, sessionModelSnapshot } from "../../runtime/model/session-model.ts";
12
12
  import { getPiSpawnCommand } from "../../runtime/pi-spawn.ts";
13
+ import { type RunDescendant, reapRunDescendants } from "../../runtime/process/run-descendant-sweep.ts";
13
14
  import { formatZombieReport, scanZombieSubagents, type ZombieScanResult } from "../../runtime/process/zombie-scanner.ts";
14
15
  import { launchScriptRegistry, sweepLaunchScripts, sweepOrphanLaunchScriptFiles } from "../../runtime/surface/launch-script.ts";
15
16
  import { surfaceProviderForCleanup } from "../../runtime/surface/resolve-surface.ts";
@@ -18,10 +19,12 @@ import type { TeamToolParamsValue } from "../../schema/team-tool-schema.ts";
18
19
  import { TeamToolParams } from "../../schema/team-tool-schema.ts";
19
20
  import { atomicWriteFile, atomicWriteJson } from "../../state/atomic-write.ts";
20
21
  import { TEAM_TERMINAL_RUN_STATUSES } from "../../state/contracts.ts";
22
+ import { loadRunManifestById } from "../../state/stores/state-store.ts";
21
23
  import { allTeams, discoverTeams } from "../../teams/discover-teams.ts";
22
24
  import { type FatalFsCause, fsFailureLabel } from "../../utils/fs-errno.ts";
23
25
  import { projectCrewRoot, userCrewRoot } from "../../utils/paths.ts";
24
26
  import { allWorkflows, discoverWorkflows } from "../../workflows/discover-workflows.ts";
27
+ import { listRecentRuns } from "../run-index.ts";
25
28
  import type { PiTeamsToolResult } from "../tool-result.ts";
26
29
  import { validateResources } from "../validate-resources.ts";
27
30
  import { configRecord, result, type TeamContext } from "./context.ts";
@@ -859,6 +862,121 @@ export function formatOrphanPaneReport(cleanup: DoctorSurfaceCleanupResult): str
859
862
  return lines.join("\n");
860
863
  }
861
864
 
865
+ /** How many recent TERMINAL runs the report-only descendant sweep inspects. */
866
+ const RUN_DESCENDANT_SCAN_RUN_LIMIT = 5;
867
+
868
+ /** One terminal run plus the surviving processes that still carry its markers. */
869
+ export interface RunDescendantReport {
870
+ runId: string;
871
+ descendants: RunDescendant[];
872
+ }
873
+
874
+ /**
875
+ * Wave 11b (F7) — REPORT ONLY. pi's bash tool spawns every command `detached`,
876
+ * so a worker's grandchildren live in their own process groups and outlive the
877
+ * run (the 390 MB / 24 h leak after team_20261022110510 completed). This scan
878
+ * finds the survivors by the pi-crew env markers they inherited and lists them
879
+ * for the operator.
880
+ *
881
+ * Nothing is killed here BY CONSTRUCTION: `reapRunDescendants` is called with
882
+ * `dryRun: true`, which makes every candidate land in `skipped` with reason
883
+ * `dry-run` — the candidates themselves arrive in `descendants` for the report.
884
+ * The automatic reap is the stale reconciler's job (terminal runs only, never
885
+ * inside a worker), not the doctor's.
886
+ */
887
+ export function collectRunDescendantReport(cwd: string, limit = RUN_DESCENDANT_SCAN_RUN_LIMIT): RunDescendantReport[] {
888
+ const reports: RunDescendantReport[] = [];
889
+ let inspected = 0;
890
+ let manifests: ReturnType<typeof listRecentRuns>;
891
+ try {
892
+ // A generous window: terminal runs are the only ones eligible, and old ones
893
+ // may dominate the recent list.
894
+ manifests = listRecentRuns(cwd, ORPHAN_RUN_SCAN_LIMIT * 5);
895
+ } catch {
896
+ return reports;
897
+ }
898
+ for (const manifest of manifests) {
899
+ if (inspected >= limit) break;
900
+ if (!TEAM_TERMINAL_RUN_STATUSES.has(manifest.status)) continue;
901
+ inspected += 1;
902
+ let loaded: ReturnType<typeof loadRunManifestById>;
903
+ try {
904
+ loaded = loadRunManifestById(cwd, manifest.runId);
905
+ } catch {
906
+ continue;
907
+ }
908
+ if (!loaded) continue;
909
+ const runnerPid = manifest.async?.pid;
910
+ // Wave 11c (F2): same proof-parent source as the automatic sweep, so the
911
+ // report lists exactly what the sweep would reap — a SYNC run's leak
912
+ // naming its dead session pi as parent.
913
+ const sessionPid = manifest.sessionPid;
914
+ const workerPids = [
915
+ ...new Set(
916
+ loaded.tasks
917
+ .map((task) => task.heartbeat?.pid ?? task.checkpoint?.childPid)
918
+ .filter((pid): pid is number => typeof pid === "number" && Number.isInteger(pid) && pid > 1),
919
+ ),
920
+ ];
921
+ const startedAtMs = Date.parse(manifest.async?.spawnedAt ?? manifest.createdAt);
922
+ // 11b.3: the same surface-pane exclusion the automatic sweep applies — a
923
+ // LIVE pane pid is not a leak, so the report must not list it either.
924
+ const surfacePids = [
925
+ ...new Set(
926
+ Object.values(manifest.surface?.workerPids ?? {}).filter(
927
+ (pid): pid is number => typeof pid === "number" && Number.isInteger(pid) && pid > 1,
928
+ ),
929
+ ),
930
+ ];
931
+ const result = reapRunDescendants(
932
+ {
933
+ runId: manifest.runId,
934
+ ...(runnerPid !== undefined ? { runnerPid } : {}),
935
+ workerPids,
936
+ ...(sessionPid !== undefined ? { sessionPid } : {}),
937
+ ...(surfacePids.length > 0 ? { surfacePids } : {}),
938
+ taskIds: loaded.tasks.map((task) => task.id),
939
+ startedAtMs: Number.isFinite(startedAtMs) ? startedAtMs : 0,
940
+ },
941
+ { dryRun: true },
942
+ );
943
+ if (result.descendants.length > 0) reports.push({ runId: manifest.runId, descendants: result.descendants });
944
+ }
945
+ return reports;
946
+ }
947
+
948
+ /** Render {@link collectRunDescendantReport} for the zombies focus of doctor. */
949
+ export function formatRunDescendantReport(reports: RunDescendantReport[]): string {
950
+ const lines: string[] = [];
951
+ lines.push("## Run descendants of terminal runs (report-only — nothing killed)");
952
+ lines.push("");
953
+ if (reports.length === 0) {
954
+ lines.push(`No surviving process matched a terminal run's pi-crew markers (last ${RUN_DESCENDANT_SCAN_RUN_LIMIT} terminal runs).`);
955
+ return lines.join("\n");
956
+ }
957
+ const total = reports.reduce((sum, report) => sum + report.descendants.length, 0);
958
+ lines.push(`Survivors: ${total} process(es) across ${reports.length} terminal run(s).`);
959
+ lines.push("Matched by inherited marker (run-id → task-id → parent-pid); the run-id marker alone is not proof of descent —");
960
+ lines.push("a candidate must also name a recorded parent of the run that is already dead. A descendant that called setsid or");
961
+ lines.push("runs in its own container cannot be proven and is not listed.");
962
+ lines.push("");
963
+ for (const report of reports) {
964
+ lines.push(`Run ${report.runId} — ${report.descendants.length} process(es):`);
965
+ for (const descendant of report.descendants) {
966
+ const rss = descendant.rssKb > 0 ? `${Math.round(descendant.rssKb / 1024)} MB` : "rss ?";
967
+ lines.push(` - pid ${descendant.pid} pgid ${descendant.pgid ?? "?"} [${descendant.matchedBy}] ${rss} — ${descendant.cmd}`);
968
+ }
969
+ }
970
+ lines.push("");
971
+ lines.push("Automatic reaping happens in the stale reconciler, per candidate: it only signals a process that proves descent");
972
+ lines.push("(a recorded runner/worker parent that is already dead) inside a terminal run, that holds no controlling terminal,");
973
+ lines.push("and whose process group has a dead leader or a leader that is itself a candidate (a live unmatched leader blocks");
974
+ lines.push("the group). A same-uid process that merely inherited a run's env — an operator shell, or a service the run left");
975
+ lines.push("running — has no such proof and is deliberately NOT reaped, so a survivor listed above may stay alive by design.");
976
+ lines.push("Never runs inside a worker. PI_CREW_REAP_RUN_DESCENDANTS=0 disables it entirely.");
977
+ return lines.join("\n");
978
+ }
979
+
862
980
  export async function handleDoctor(ctx: TeamContext, params: TeamToolParamsValue = {}): Promise<PiTeamsToolResult> {
863
981
  // Sub-focus: zombie sub-agent scan + orphan surface-pane cleanup (T12). The
864
982
  // process scan itself stays READ-ONLY — never kills a process. The pane
@@ -871,7 +989,11 @@ export async function handleDoctor(ctx: TeamContext, params: TeamToolParamsValue
871
989
  if (params.focus === "zombies") {
872
990
  const scan = scanZombieSubagents();
873
991
  const cleanup = await cleanupOrphanSurfacePanes({ cwd: ctx.cwd, scan });
874
- const text = `${formatZombieReport(scan)}\n\n${formatOrphanPaneReport(cleanup)}`;
992
+ // Wave 11b (F7): the zombie scan cannot see a terminal run's pi-bash
993
+ // grandchildren (they carry no PI_CREW_SURFACE and their recorded parent is
994
+ // the dead runner). The run-scoped sweep does — REPORT ONLY, dryRun.
995
+ const runDescendants = collectRunDescendantReport(ctx.cwd);
996
+ const text = `${formatZombieReport(scan)}\n\n${formatOrphanPaneReport(cleanup)}\n\n${formatRunDescendantReport(runDescendants)}`;
875
997
  return result(
876
998
  text,
877
999
  {
@@ -884,6 +1006,8 @@ export async function handleDoctor(ctx: TeamContext, params: TeamToolParamsValue
884
1006
  // never pane-cleaned; see zombie-scanner.ts discriminator notes).
885
1007
  foreignPi: scan.foreign.length,
886
1008
  errors: scan.errors.length,
1009
+ // Wave 11b: report-only count of a terminal run's surviving descendants.
1010
+ runDescendants: runDescendants.reduce((sum, report) => sum + report.descendants.length, 0),
887
1011
  orphanPanes: cleanup.orphans.length,
888
1012
  panesClosed: cleanup.closed.length,
889
1013
  panesGone: cleanup.gone.length,
@@ -29,6 +29,7 @@ import { spawn } from "node:child_process";
29
29
  import { DEFAULT_CHILD_PI } from "../../config/defaults.ts";
30
30
  import { logInternalError } from "../../utils/internal-error.ts";
31
31
  import { TailCaptureStage } from "../compaction/compact-stages/tail-capture-stage.ts";
32
+ import { type ProcIdentity, readProcIdentity, reapGroup, resolveReapGroupEnabled } from "../process/reap-group.ts";
32
33
  import { HARD_KILL_MS } from "./child-pi-constants.ts";
33
34
  import { adoptOwnedProcess, disposeAllOwnedProcesses, type OwnedProcess } from "./owned-process.ts";
34
35
 
@@ -44,6 +45,34 @@ const childHardKillTimers = new Map<number, NodeJS.Timeout>();
44
45
  // once the owner reaches its terminal state.
45
46
  const ownedByPid = new Map<number, OwnedProcess>();
46
47
 
48
+ // Wave 11a (F1): the /proc identity snapshot of each registered child, taken
49
+ // AT REGISTRATION (while the child is definitely alive). This is the only
50
+ // pre-exit record of the child's pgid/starttime — once the root exits, /proc is
51
+ // gone and `adoptOwnedProcess`'s live pgid resolution (owned-process.ts) has
52
+ // nothing left to read, so a still-live group becomes unreachable. killProcessTree
53
+ // uses this snapshot for the exited-root case; the 60 s sweep prunes it once the
54
+ // owner is terminal.
55
+ const identityByPid = new Map<number, ProcIdentity>();
56
+
57
+ // Wave 11a injection seam (same shape as zombie-scanner.ts `__test`): tests seed
58
+ // identity snapshots and swap the group reaper without a real child process.
59
+ let reapGroupImpl: typeof reapGroup = reapGroup;
60
+ export const __test = {
61
+ /** Seed the exit-time identity snapshot for a pid (as registerActiveChild would). */
62
+ seedIdentity(pid: number, identity: ProcIdentity | undefined): void {
63
+ if (identity === undefined) identityByPid.delete(pid);
64
+ else identityByPid.set(pid, identity);
65
+ },
66
+ /** Drop every seeded/recorded identity snapshot. */
67
+ clearIdentities(): void {
68
+ identityByPid.clear();
69
+ },
70
+ /** Replace the group reaper (undefined restores the real one). */
71
+ setReapGroup(fn: typeof reapGroup | undefined): void {
72
+ reapGroupImpl = fn ?? reapGroup;
73
+ },
74
+ };
75
+
47
76
  // Periodic cleanup of dead child process entries to prevent memory leaks.
48
77
  // If a child process never emits exit/close (zombie), the entry would leak.
49
78
  setInterval(() => {
@@ -57,7 +86,10 @@ setInterval(() => {
57
86
  // U15: prune adopted owners that reached terminal state (group confirmed
58
87
  // gone via dispose or drain-reconcile). Non-terminal owners stay routable.
59
88
  for (const [pid, owner] of ownedByPid) {
60
- if (owner.terminated) ownedByPid.delete(pid);
89
+ if (!owner.terminated) continue;
90
+ ownedByPid.delete(pid);
91
+ // Wave 11a: the owner confirmed its group is gone — the snapshot is spent.
92
+ identityByPid.delete(pid);
61
93
  }
62
94
  }, 60_000).unref();
63
95
 
@@ -69,6 +101,11 @@ export function registerActiveChild(pid: number, child: ChildProcess): void {
69
101
  // (authoritative /proc check on Linux — the rpc branch child, spawned
70
102
  // without detached, lands on the single-process fallback automatically).
71
103
  ownedByPid.set(pid, adoptOwnedProcess(child, { name: `child-pi:${pid}` }));
104
+ // Wave 11a (F1): snapshot the identity NOW — the child is alive here, so
105
+ // /proc is readable. killProcessTree's exited-root branch reaps the group
106
+ // from this snapshot after the root is gone.
107
+ const identity = readProcIdentity(pid);
108
+ if (identity) identityByPid.set(pid, identity);
72
109
  }
73
110
 
74
111
  /**
@@ -220,7 +257,36 @@ export function killProcessTree(pid: number | undefined, child?: ChildProcess):
220
257
  void owned.dispose();
221
258
  return;
222
259
  }
223
- if (child && child.exitCode !== null) return;
260
+ if (child && child.exitCode !== null) {
261
+ // Wave 11a (F1): an EXITED ROOT IS NOT AN EMPTY GROUP. The root may have
262
+ // backgrounded descendants that are still alive in its process group; the
263
+ // pre-wave early-out returned here and left them running forever. We reap
264
+ // the group only when the registration-time snapshot PROVES the root was
265
+ // its own group leader (pgid === pid) — reapGroup re-validates everything
266
+ // (uid/pgid/starttime) and refuses on a recycled pgid. No snapshot, or a
267
+ // non-leader root, keeps the pre-wave behaviour exactly (no signal, and
268
+ // nothing logged above debug).
269
+ const identity = identityByPid.get(pid);
270
+ if (identity?.pgid === pid && resolveReapGroupEnabled()) {
271
+ const result = reapGroupImpl(pid, pid, { name: `child-pi:exited-root:${pid}`, expected: identity });
272
+ // MINOR #5 (review): the SIGKILL escalation is fire-and-forget here (this
273
+ // caller is sync), so it must be explicitly swallowed — `reapGroup` never
274
+ // rejects by contract, but a `void` promise is exactly what turns an
275
+ // unexpected rejection into a FATAL unhandled rejection on Node 22.
276
+ void result.pending.catch(() => {
277
+ /* settle-only: the escalation verdict is not actionable here */
278
+ });
279
+ identityByPid.delete(pid);
280
+ } else {
281
+ logInternalError(
282
+ "child-pi.kill-exited-root-no-snapshot",
283
+ new Error(`exited root pid=${pid} has no group-leader snapshot; no signal sent`),
284
+ `pid=${pid}`,
285
+ "debug",
286
+ );
287
+ }
288
+ return;
289
+ }
224
290
  killProcessPid(pid);
225
291
  child?.once("exit", () => clearHardKillTimer(pid));
226
292
  }
@@ -198,6 +198,10 @@ export function buildChildPiSpawnOptions(
198
198
  },
199
199
  stdio: ["ignore", "pipe", "pipe"], // stdin=ignore: child doesn't wait for input; task comes via CLI args
200
200
  detached: process.platform !== "win32",
201
+ // Wave 11a (GAP-5): `setsid` is NOT a Node spawn option — it is silently
202
+ // ignored. Group leadership comes from `detached` alone (POSIX setsid() in
203
+ // the child). Kept verbatim instead of deleted: removing it would suggest a
204
+ // behaviour change that never happened.
201
205
  setsid: true,
202
206
  // NOTE: setsid creates a new session; the child process becomes the session leader
203
207
  // and its parent becomes that session leader (still the team-runner in the same
@@ -1,6 +1,10 @@
1
1
  export interface WorkerHeartbeatState {
2
2
  workerId: string;
3
3
  pid?: number;
4
+ /** Wave 11a (F3): spawn-time identity of `pid` (see `readProcIdentity`). */
5
+ pgid?: number;
6
+ starttime?: string;
7
+ uid?: number;
4
8
  lastSeenAt: string;
5
9
  lastStdoutAt?: string;
6
10
  lastEventAt?: string;