pi-crew 0.11.40 → 0.11.42

package/CHANGELOG.md

@@ -2,6 +2,81 @@
 
 ## [Unreleased]
 
+## [0.11.42] — fix wave 14: event-log drain API + zombie-scanner seam (2026-10-25)
+
+Wave 12d papered over the recurring `ENOTEMPTY` teardown race with `removeDirWithRetry`; the root cause stayed:
+production fires **66 event writes across 23 files** without awaiting them (`void appendEventAsync` ×23,
+`appendEventFireAndForget` ×43), and each queued write re-creates the run dir before appending — so teardown that
+`rmSync`s the state dir mid-flight races the async writer and loses. There was no way for a test to await one
+run's writers: the queue drain was private and `beforeExit`-only (SPEC `fixwave14-drain-api-zombie-seam-2026-10-25`).
+
+**Added — `drainEventLogWriters(eventsPath?)`.** Public per-run drain API for the event-log async writer queues
+(allSettled tail await + per-path buffer flush, never throws). Lets tests await in-flight writers before removing
+a run state dir instead of retry-racing them — the race repro (drain → plain `rmSync`, no retry) is now a pinned
+test.
+
+**Added — `cleanupStateRoot(dir, eventsPath?)` test helper.** Best-effort drain then `removeDirWithRetry(settleMs: 0)`.
+
+**Added — `ZombieScanOptions` seam on `scanZombieSubagents`** (`platform`/`getuid`/`readProcFile`/`listProcPids`/
+`pidAlive`/`getParentPid`, defaults byte-exact). **7 previously linux-only tests now run on every OS** via
+synthetic `/proc` fixtures (+2 new seam tests; 2 real-`/proc` default-path tests stay linux-gated).
+
+**Fixed — teardown races by adopting drain-then-remove in pilot tests.** `team-tool-steer-clamp` (retry-only →
+drain), `dwf-setresult` (12 bare `rmSync`), `event-log-batch`/`event-log-async`, plus dedup of two local
+`removeDirWithRetry` variants (`subagent-steer-live`, `subagent-tools-integration`).
+
+**Battery (PASS).** targeted ×3 (drain 4/4, zombie 26/26 skip=0, 6 pilot files), `test:critical` 124/124, full
+unit 10,075 pass / 1 ambient-adjudicated / 4 skipped, integration 144/0/4, static gates 0, dist MATCH.
+
+**Docs.** Battery report `docs/real-test/reports/real-test-2026-10-25-wave14-drain-zombie-battery.md`; closed
+investigation (no code change): the "hung" alert for an already-cancelled run was delivery timing — `cancelled`
+is already terminal in both the emit and delivery gates.
+
+## [0.11.41] — fix wave 13: watchdog liveness — false "appears hung" alerts on healthy direct runs (2026-10-25)
+
+Five false alarms landed in one working session (2026-10-24/25), all the same shape: healthy direct single-agent
+runs executing ONE long tool call (a ~20-min CI poll loop; a ~20-min full test suite). Every run completed
+normally — the watchdog's *evidence* was frozen, not the run (SPEC `fixwave13-watchdog-liveness-2026-10-25`).
+Unlike the T1/T4 incidents already covered by WD-DELIVER-GUARD / WD-STALE-RECORD, these verdicts were wrong at
+emit time: the run genuinely was still running.
+
+**Fixed — root cause: evidence frozen during long tool calls (GAP-0).** The hung verdict in
+`src/runtime/foreground-watchdog.ts` reached through `isLikelyOrphanedActiveRun` used only the manifest
+`updatedAt` and agent `progress.lastActivityAt` — and both freeze by construction while a tool runs:
+`persistLiveProgress` was purely event-driven (no pi events fire during a 20-min bash call) and `updateRunStatus`
+writes `updatedAt` only on status transitions. The 5-min activity shield expires, the next cycle reads a stale
+manifest ⇒ false hung.
+
+**Added — watchdog liveness short-circuit (F1).** Before a hung verdict is emitted, the watchdog now consults the
+two liveness signals other consumers already trust: a fresh **active-run registry claim** (registered at dispatch)
+⇒ the alert is suppressed for up to `CLAIMED_RUN_HUNG_MS` = 45 min, OR a fresh **task heartbeat**
+(`heartbeat.lastSeenAt` on a running/waiting task, `alive !== false`) ⇒ no alert. The hung ladder for unclaimed
+runs is unchanged (`ORPHANED_ACTIVE_RUN_MS` untouched — dashboard semantics preserved).
+
+**Added — live-executor heartbeat timer (GAP-1).** `src/runtime/task-runner/live-executor.ts` now runs an unref'd
+`LIVE_TOOL_HEARTBEAT_INTERVAL_MS` = 30 s interval, active only while a tool executes, throttled persist of live
+progress, cleared in `finally`; no event-log writes. The state store no longer freezes mid-tool for ANY consumer
+(watchdog, dashboard, doctor).
+
+**Added — evidence timestamp (GAP-2).** The watchdog alert detail now carries `as of <updatedAt>` so a fresh alert
+is distinguishable from a stale or queued one.
+
+**Fixed — `steer-live` test fixture orphan leak (GAP-3).** 13 orphan background-runner processes from the
+`steer-live` test were found alive and killed manually — 5 born AFTER wave 12 shipped (a different fixture from
+the `steer-clamp` test wave 12 fixed). `test/unit/extension/core/subagent-steer-live.test.ts` spawns a REAL
+background runner through the SUT (setsid + detached + SIGTERM-swallowing) and held no handle on it. Teardown now
+resolves the runner pid from `manifest.async.pid` → `async.pid` → state scan, reaps via `reapGroup` when
+`pgid === pid` (with `/proc/<pid>/cwd` scoping + self-pid guard), falls back to a pid-SIGKILL, and runs in
+`finally` + `after()`.
+
+**Battery (PASS).** targeted ×3 (`foreground-watchdog` 29/29, `live-executor-heartbeat` 5/5, `subagent-steer-live`
+4/4), `test:critical` 124/124, full unit 10,061 pass / 0 fail / 4 skipped, integration 144/0/4, static gates 0,
+dist MATCH. **Live repro on the real dist** (`PI_CREW_WATCHDOG_INTERVAL_MS=20000` + a 400 s tool): 5 hung verdicts
+suppressed by the new liveness short-circuit, 0 alerts delivered — while the same box's older 0.11.40 verifier
+session emitted 2 false alerts during the battery (the exact pre-fix behavior).
+
+**Docs.** battery report `docs/real-test/reports/real-test-2026-10-25-wave13-watchdog-liveness-battery.md`.
+
 ## [0.11.40] — fix wave 12: fixture teardown — test-harness orphan leaks (2026-10-24)
 
 Closes the leak class wave 11 exposed but did not own: the reaping wave made production descendants provable, and