pi-crew 0.11.37 → 0.11.39

package/CHANGELOG.md

@@ -2,6 +2,92 @@
 
 ## [Unreleased]
 
+## [0.11.39] — fix wave 11: worker descendant reaping (2026-10-24)
+
+Closes the leak that motivated the wave: run `team_20261022110510_cefb9164af126460` reached `completed` on
+2026-10-22 and six grandchild processes (`dpi-backend` ×5 + `caddy`, ~390 MB RSS) were still alive 24 h later.
+Root cause (SPEC `fixwave11-worker-reaping-2026-10-24` GAP-0): pi's bash tool spawns EVERY command with
+`detached: true`, so a worker's bash grandchildren own a process group that a group kill rooted at the worker can
+never reach.
+
+**11a — identity-proven group kill (F1–F4).**
+
+- `src/runtime/process/reap-group.ts` is now the ONE place that may signal a process group: a refusal ladder
+  (POSIX only, `pgid > 1`, never our own real pgrp, live-identity-or-leader-proof, same uid, member of the requested
+  group, starttime present + matching) plus a bounded SIGTERM → SIGKILL escalation. SIGTERM is delivered
+  synchronously; the SIGKILL rung is handed back as `pending` and owned by `drainReapEscalations()`
+  (stale reconciler) or `flushReapEscalationsSync()` (shutdown), never dropped.
+- `child-pi-kill.ts` (F1): an exited ROOT no longer means "nothing left" — when the snapshot proves the root led its
+  own group, the group is reaped.
+- `orphan-reaper.ts` (F2): recorded workers are reaped through the guarded group ladder when a pgid is known,
+  identity-proven candidates bypass the zombie-scan gate, and a scanner throw is now fail-open for proven candidates
+  instead of a blanket no-op.
+- `state/types.ts` + `child-executor.ts` (F3): `pgid`/`starttime`/`uid` are persisted on the checkpoint and worker
+  heartbeat (all optional — old records load unchanged), which is what makes identity proof real on the next run.
+- `frame-client.ts` (F4): the prototype RPC frame client spawns detached. The production RPC path uses the SDK
+  `RpcClient`, which exposes no spawn knob — those workers stay non-leaders.
+
+**11b — run-scoped descendant sweep (F7, the actual cure for GAP-0).**
+
+- New `src/runtime/process/run-descendant-sweep.ts`: `collectRunDescendants()` / `reapRunDescendants()` enumerate
+  `/proc`, match the pi-crew env markers a descendant INHERITED, and reap only through a group that can be proven.
+  Markers used (verified in src, none invented): `PI_CREW_BROKER_RUN_ID` (`child-pi-spawn.ts:382`) ⇒ `run-id`;
+  `PI_CREW_KIND=subagent` + `PI_CREW_TASK_ID` + `PI_CREW_PARENT_PID` in the recorded set ⇒ `task-id`;
+  a `PI_CREW_PARENT_PID` that is a recorded worker pid and is dead ⇒ `parent-pid`. `PI_CREW_RUN_ID` is deliberately
+  NOT matched (registry: `test-only`, no src writer).
+- Guards: Linux `/proc` only (else `[]`, fail-open); never self, never an ancestor, same uid only; the candidate must
+  have started no earlier than the run (> 60 s of pre-run clock skew rejected); the candidate's group must not be ours;
+  one signal per group per leader incarnation (idempotent across reconciler ticks); and an extra rung — a group whose
+  LEADER is alive and carries no marker is skipped (`leader-alive-unmatched`), so a live session's group is never hit.
+- Call sites: `stale-reconciler.ts` sweeps ONLY when the run is terminal, the process is not a worker
+  (`PI_CREW_KIND`/crew depth 0), and every recorded runner/worker pid is dead; `team-tool/doctor.ts` (`focus=zombies`)
+  reports the survivors as `runDescendants` — report-only, `dryRun: true`, nothing killed.
+- Kill switches: `PI_CREW_REAP_GROUP` (11a) and `PI_CREW_REAP_RUN_DESCENDANTS` (11b), both boolean, default ON, both
+  forwarded to the detached background runner (never to a worker) so a disable actually takes effect where the reaping
+  runs.
+
+**Known limitations** (unchanged by design, documented on the module and in the SPEC): a descendant that calls
+`setsid()` or runs inside its own container/init (`docker`, `systemd-run`) has a group we cannot prove and is left
+alone; pi's bash tool remains the primary source of detached grandchildren — the upstream fix (tracked, non-detached
+children for tool commands, or a cgroup/subreaper for the worker tree) is what would make this sweep unnecessary;
+and the sweep can only match processes that still carry the markers (a scrubbed env is invisible).
+
+**11c — sync-run descent proof + test pin (leader-approved options (a) + F1 fix).**
+
+- Record sync-run `manifest.sessionPid` so foreground-run leaks satisfy the descent proof (wave 11c): a sync run has no `async.pid`, so a leaked worker grandchild whose `PI_CREW_PARENT_PID` points at the session pi itself could never be proven and was skipped forever; `sessionPid` is recorded at creation and treated as a recorded parent (dead-only, like the others). Also pin `PI_CREW_REAP_RUN_DESCENDANTS` in the sweep suite's setup so the tests no longer depend on the ambient kill-switch state. Battery addendum: `docs/real-test/reports/real-test-2026-10-24-wave11-battery-addendum.md` (F1/F2 closed, verdict PASS).
+
+## [0.11.38] — Wave 10: pi-subagents lessons (10 ports) + watchdog hung-alert hardening (WD-DEDUP / WD-STALE-RECORD / WD-DELIVER-GUARD)
+
+### Fixed — watchdog alerts for FINISHED runs: delivery-time guard (WD-DELIVER-GUARD, root-caused 2026-10-23)
+
+- **fix(watchdog): "appears hung" still reached the user for an already-completed run — the root cause is DELIVERY-time staleness, NOT a missing finalize.** `finalizeRun` was correct both incidents (`run.completed` written, non-force `cancel` answered "already completed; nothing to cancel" 15s after the 06:12 alert): the alert 06:12:47.778 landed AFTER `run.completed` 06:12:28 because it had been EMITTED while the run was still genuinely running and was then parked in the host's follow-up queue — `pi.sendUserMessage(..., {deliverAs:"followUp"})` on a STREAMING session goes to `agent.followUp(...)` (`agent-session.js`) and is delivered at the next idle boundary, 28 minutes later in the incident. `clearQueue()` is all-or-nothing, so a queued item is unretractable: the emit-time re-verify (WD-DEDUP/WD-STALE-RECORD) cannot close this class. **Fix in `src/runtime/foreground-watchdog.ts` (WD-DELIVER-GUARD):** a hung alert is no longer handed to the host eagerly — while the session streams it is held in a bounded module-scope pending map (cap 256, one in flight per runId, self-clearing at a 10-min deadline) and retried every 500ms; at the idle boundary (the instant the host DELIVERS instead of queueing) the run state is re-read one final time and the alert is **DROPPED** when the run is terminal (`completed`/`failed`/`cancelled`/`blocked`) or when every task is terminal (stale record), with a one-time `PI_CREW_DEBUG` log per runId. **FAIL-OPEN by design:** a throwing loader, an absent manifest, or a host without the idle capability (`ctx.isIdle`) still delivers — a broken state file or an older host must never swallow a genuine hung alert. Wired at the one real call site (`foreground-run-controller.ts` → `startForegroundWatchdog({ ..., isSessionIdle })`). The "going quiet" handoff notice is delivery-gated too; the terminal-completion notice and the stale-record notice stay eager (they are true statements, and a late one is still accurate).
+- **tests**: watchdog suite 15 → 20 — (i) emit-while-running + run completed before delivery ⇒ 0 messages, exactly 1 drop debug log; (ii) still hung at delivery ⇒ exactly 1 message, nothing left pending; (iii) unreadable state (throw) and vanished manifest ⇒ fail-open delivery; (iv) drop once per runId across many cycles (no re-queue, no re-alert — WD-DEDUP not regressed); (v) legacy caller without an idle probe is still guarded at handoff.
+- **cap-expiry path now re-verifies too (no unverified delivery)** — residual closed from the 2026-10-23 ship battery (`docs/real-test/reports/real-test-2026-10-23-wave10-watchdog-ship-battery.md`): the other way out of the pending map is the 10-minute `deliverCapMs` deadline, taken when the session never goes idle (one foreground tool call blocking past the cap). That exit runs the SAME `classifyAtDelivery` as the idle boundary — terminal manifest or all-tasks-terminal ⇒ DROP with the one-time debug log, and only a run that still reads as hung (or an unreadable/absent record, fail-open) is handed over. "Late beats never" now applies to genuinely hung runs only. Pinned by 4 new tests (watchdog suite 20 → 24): cap expiry + all-tasks-terminal ⇒ 0 messages + 1 drop log (real state write), cap expiry + terminal manifest ⇒ 0 messages + 1 drop log, cap expiry + still hung ⇒ exactly 1 message / nothing pending, cap expiry + throwing loader ⇒ 1 message (fail-open). Mutation-checked: skipping the verify on the cap path turns both drop tests red.
+
+### Fixed — watchdog stale-record silence + poll-interval override (WD-STALE-RECORD / WD-INTERVAL, root-caused 2026-10-23)
+
+- **fix(watchdog): "appears hung" alerts for runs that are actually DONE** — root cause of the two live incidents (team_20261023030033 @04:13, team_20261023050900 @06:12) is NOT a missing finalize: `finalizeRun` wrote `completed` both times (`src/runtime/finalize-run.ts:514` updateRunStatus → `run.completed`, `:559` locked `saveRunManifestAsync`) — proven by `team action='cancel'` answering "already completed; nothing to cancel" 15s after the 06:12 alert landed. The alert was a **stale emit-time snapshot**: the watchdog emitted at ~05:44 during a 480s progress gap (`agents.json` `progress.lastActivityAt` older than the 5-min `AGENT_ACTIVITY_LIVENESS_MS`, `src/runtime/process-status.ts:68,95-125`), and `sendUserMessage({deliverAs:"followUp"})` then QUEUED it behind the parent's blocking tool call for 28 minutes — delivered 19s AFTER `run.completed`. Two bounded fixes in `src/runtime/foreground-watchdog.ts`: **(a) WD-STALE-RECORD** — the pre-emit re-verify now reads `tasks.json` too (`loadRunManifestById` already returns `tasks`); manifest still active BUT every task terminal (completed/failed/cancelled/skipped/needs_attention, `TEAM_TERMINAL_TASK_STATUSES`) ⇒ treat the record as stale: no hung alert, one-time `PI_CREW_DEBUG` log, and exactly ONE user-facing notice per runId ("run X record stale (all tasks terminal) — skipped hung alert", module-scope bounded map). FAIL-OPEN by design: empty/unreadable/corrupt task evidence is never read as "all terminal", so a genuinely hung run still alerts. **(b) WD-INTERVAL** — new `PI_CREW_WATCHDOG_INTERVAL_MS` (registered in the env registry + forwarding decision table as `gap`, default 300000 = the old constant) overrides the 5-minute poll cadence; an explicit `checkIntervalMs` still wins. Documented residual (not fixed here, needs a delivery-time hook the extension API does not expose): a snapshot can still be delivered late by the followUp queue — the emit-time gate cannot re-verify at delivery.
+- **tests**: watchdog suite 9 → 15 — all-tasks-terminal ⇒ 0 hung alerts + exactly 1 stale notice; one task still running ⇒ alert fires; missing `tasks.json` ⇒ never read as terminal (no stale claim, unchanged silent skip, no crash); corrupt `tasks.json` ⇒ legacy hung alert kept (fail-open); stale notice is per-runId across watchdog restarts; `PI_CREW_WATCHDOG_INTERVAL_MS` drives the cadence. Registry snapshot/metadata tests extended for the new env var.
+
+### Fixed — watchdog hung-alert dedup (WD-DEDUP, live-caught 2026-10-23)
+
+- **fix(watchdog): repeated "appears hung" alerts for a run already terminal** — live pain 2026-10-23 (run team_20261023030033…): the foreground watchdog re-alerted "appears hung" for a run that had already completed, because (1) nothing deduped the notice across cycles/instances (the ARCH-5 notice cap of 2 still allowed one repeat, and a re-armed watchdog instance started the counter fresh) and (2) a check armed by an older poll cycle emitted without re-verifying the run state at emit time. Fix in `src/runtime/foreground-watchdog.ts`: **(a) alert-once per runId** — a module-level bounded FIFO map (cap 256, oldest entry evicted — no unbounded growth) suppresses every later "appears hung" emit for the same runId, even from a fresh watchdog instance; **(b) pre-emit re-verify** — the run state is reloaded immediately before any emit; a terminal (or vanished) run skips the emit entirely (one-time `PI_CREW_DEBUG` debug log) and keeps monitoring, so the next cycle's terminal branch still delivers the single completion notice as before. The ARCH-5 going-quiet handoff stays one-shot. Pinned by 4 new tests: alert-once (incl. restart immunity), re-verify skip at emit time, natural terminal transition after first alert, bounded-cap eviction.
+
+### Wave 10 — pi-subagents lessons port (verified 26-item matrix, 2026-10-22)
+
+All ten actionable items from the pi-subagents deep research (5 tiers: architecture → perf → prompt/meta → docs/intercom/missions → live measurement). Two lessons were CLOSED by evidence instead of ported: spawn/parallel performance (live measurement §10-11: pi-crew process-per-worker 0.57–0.76s beats pisub in-process 0.83s solo / 3.27s fan-out per child — nothing to port) and declaration-pinning (not applicable: pi-crew is process-per-worker, the platform owns tool declarations).
+
+- **W10-1 cache TTL split (lesson #10)**: headless single-run workers now spawn with `PI_CACHE_RETENTION=short` — long-retention prompt-cache writes carry a higher per-write surcharge that a one-shot worker never amortizes. Injected at the spawn-options layer (same slot as `PI_CODING_AGENT_DIR`); surface panes and RPC warm sessions keep the parent default. The built.env canary rejects the key anywhere else.
+- **W10-2 memoized green verify-gate (lesson #3)**: a PASS verdict is recorded keyed by the workspace tree OID (`git write-tree`) + check list + node version; a re-gate on a bit-identical tree serves the memo without re-running typecheck/test:critical (tree equality subsumes the M1-7 manifest bookends). Only PASS-with-real-runs records; FAILED/INCONCLUSIVE/drift never memoize; non-git cwd fails open; kill-switch `PI_CREW_VERIFY_GATE_MEMO` (default ON).
+- **W10-3 mailbox typed reason + strict replyTo routing (lesson #22)**: messages gain an optional `reason` enum {need_decision, progress_update, interview_request} and responses resolve ONLY the request their own requester opened (A→B→C rule) — parallel asks can no longer cross-resolve.
+- **W10-4 LLM-slop lint rules (lesson #21)**: `npm run check:slop-rules` (in `ci`) — comment/string-aware source scanners: `no-json-parse-nonnull` + `no-any-deref` STRICT (clean today), `no-double-cast` WARN-only over a 41-site baseline (burn-down follow-up). `// slop-ok: <reason>` escape hatch.
+- **W10-5 decision-drift guardian (lesson #20)**: critic/oracle reconstruct the run's inherited-decisions baseline and must flag contradictions with `DRIFT-BLOCK:`; `extractVerifierVerdict` degrades a pass carrying DRIFT-BLOCK to "none" (fail-closed: a later truncated source cannot launder the marker).
+- **W10-6 budget-boundary policy (lesson #16)**: delegation-patterns skill — budgets block read-tools, NOT mutation-tools; an elapsed timeout is not a mutation-safe boundary (no auto-revert racing the timeout).
+- **W10-7 usage reconcile (lesson #14)**: every model attempt carries its own usage (failed-fallback tokens were invisible — task.usage shows only the FINAL run); `usageReconcile` note on the task when the sum-of-attempts vs final cost gap exceeds 5% with ≥2 usage-carrying attempts. Observability only.
+- **W10-8 read-lane same-tree guard (lesson #4)**: READ-ONLY lanes dirtying their worktree (modified OR untracked — `git status --porcelain`, diff alone misses untracked) stamp an `unexpectedMutation` note; fail-open, leader decides.
+- **W10-9 typed JSON gates (lesson #2)**: checks accept `{output:"json", schema}` (minimal subset type/required/properties/items/enum, zero new deps) — invalid stdout or schema mismatch FAILS at exit 0; parsed `structuredOutput` lands on the check result; typed gates never memoize.
+- **Deferred (PoC when token economics demands it)**: fork prompt-cache tier (lesson #9, #2879: 52k→1.7k written) — needs an agent-side extension in the child pi process; architecturally separate from this wave.
+
 ### Fixed (live battery 2026-10-18 — real-test v0.11.24)
 
 - **fix(gate): RT24-F1 root-gate config/autonomy read over-block** — `rootGateBlockReason` counted the `args` subcommand carrier as a config patch key, so worker-origin `config {args:"show"}` READS were rejected as "writes user-scope config" (live-caught in the v0.11.24 real-test battery; docblock promised reads open). `args` added to the carrier-key sets for `config` and `autonomy`; unknown junk keys stay write-classified (fail-closed). Pinned: reads with carrier keys open, writes hidden behind the same key still gated (`unset`/patch keys), user config byte-untouched. Live-verified end-to-end (fresh `pi -p`, worker env): read allowed + dumps config; `unset` write gate-blocked verbatim "root-session required".
@@ -8729,3 +8815,6 @@ the repo. Two genuine wins, plus an informational CI job that never blocks.
   pattern (`jiti` flagged unused but is used via runtime path-loading). A
   blocking gate would create an unpaid maintenance backlog unsuitable for a
   solo-maintained extension.
+
+### Hardening (W10 batch A)
+- `check-slop-rules` no-double-cast baseline ratchet (41, 2026-10-22 — burn down only) + real-CLI e2e pin; W10-7 reconcileUsage missing-finalUsage guard; W10-8 READ-ONLY lane mutation one-line stderr visibility; W10-9 structuredOutput JSON round-trip pin; W10-2 verify-gate memo time-dependent-skip intent pin.

package/agents/critic.md

@@ -27,6 +27,19 @@ You are a critic. Your job is to CHALLENGE plans and designs BEFORE execution
 - Ownership conflicts (two phases writing the same files)
 - Verification gaps (criterion that can't be objectively checked)
 
+## Decision-drift guardian (W10-5)
+Before critiquing, reconstruct the run's INHERITED DECISIONS baseline: every
+decision already locked in the task context (SPEC/GATES constraints, pinned
+architecture choices, knowledge entries, leader instructions). Then, for EACH
+of your recommendations, answer: does it contradict any inherited decision?
+- Contradiction found ⇒ you MUST open your verdict with a line containing
+  `DRIFT-BLOCK: <decision-ref> — <the contradiction in one sentence>` and
+  treat the recommendation as BLOCKED until the leader explicitly re-decides.
+- No contradiction ⇒ say nothing about drift (do not pad with empty checks).
+A PASS verdict carrying a DRIFT-BLOCK line is treated as NOT VERIFIED by the
+runtime (post-execution.ts W10-5) — flagging a contradiction and approving
+anyway is self-contradictory and downgrades to leader review.
+
 ## Output format
 
 End with exactly this block:

package/agents/oracle.md

@@ -33,6 +33,20 @@ If the question is not strategic, name the right role in ROUTING and stop.
 - For debugging guidance: give a ranked hypothesis list with the cheapest discriminating experiment per hypothesis.
 - For simplification: name the exact deletions/refactors, and what breaks if done — or state that nothing breaks.
 
+## Decision-drift guardian (W10-5)
+You are the consistency guardian for LONG runs: reconstruct the INHERITED
+DECISIONS baseline (SPEC/GATES constraints, pinned architecture choices,
+knowledge entries, earlier leader decisions in the task context) before
+advising. For EACH recommendation, answer: does it contradict an inherited
+decision?
+- Contradiction found ⇒ prepend a line `DRIFT-BLOCK: <decision-ref> — <the
+  contradiction>` to ORACLE_ADVICE and set ROUTING: leader-decision — the
+  leader must explicitly re-decide, not silently absorb your advice.
+- No contradiction ⇒ stay silent about drift.
+A pass-shaped verdict carrying DRIFT-BLOCK is treated as NOT VERIFIED by the
+runtime (post-execution.ts W10-5); flagging a contradiction while endorsing
+the plan anyway is self-contradictory.
+
 ## Output format
 
 End with exactly this block: