pi-crew 0.11.35 → 0.11.38

package/CHANGELOG.md

@@ -2,10 +2,58 @@
 
 ## [Unreleased]
 
+## [0.11.38] — Wave 10: pi-subagents lessons (10 ports) + watchdog hung-alert hardening (WD-DEDUP / WD-STALE-RECORD / WD-DELIVER-GUARD)
+
+### Fixed — watchdog alerts for FINISHED runs: delivery-time guard (WD-DELIVER-GUARD, root-caused 2026-10-23)
+
+- **fix(watchdog): "appears hung" still reached the user for an already-completed run — the root cause is DELIVERY-time staleness, NOT a missing finalize.** `finalizeRun` was correct both incidents (`run.completed` written, non-force `cancel` answered "already completed; nothing to cancel" 15s after the 06:12 alert): the alert 06:12:47.778 landed AFTER `run.completed` 06:12:28 because it had been EMITTED while the run was still genuinely running and was then parked in the host's follow-up queue — `pi.sendUserMessage(..., {deliverAs:"followUp"})` on a STREAMING session goes to `agent.followUp(...)` (`agent-session.js`) and is delivered at the next idle boundary, 28 minutes later in the incident. `clearQueue()` is all-or-nothing, so a queued item is unretractable: the emit-time re-verify (WD-DEDUP/WD-STALE-RECORD) cannot close this class. **Fix in `src/runtime/foreground-watchdog.ts` (WD-DELIVER-GUARD):** a hung alert is no longer handed to the host eagerly — while the session streams it is held in a bounded module-scope pending map (cap 256, one in flight per runId, self-clearing at a 10-min deadline) and retried every 500ms; at the idle boundary (the instant the host DELIVERS instead of queueing) the run state is re-read one final time and the alert is **DROPPED** when the run is terminal (`completed`/`failed`/`cancelled`/`blocked`) or when every task is terminal (stale record), with a one-time `PI_CREW_DEBUG` log per runId. **FAIL-OPEN by design:** a throwing loader, an absent manifest, or a host without the idle capability (`ctx.isIdle`) still delivers — a broken state file or an older host must never swallow a genuine hung alert. Wired at the one real call site (`foreground-run-controller.ts` → `startForegroundWatchdog({ ..., isSessionIdle })`). The "going quiet" handoff notice is delivery-gated too; the terminal-completion notice and the stale-record notice stay eager (they are true statements, and a late one is still accurate).
+- **tests**: watchdog suite 15 → 20 — (i) emit-while-running + run completed before delivery ⇒ 0 messages, exactly 1 drop debug log; (ii) still hung at delivery ⇒ exactly 1 message, nothing left pending; (iii) unreadable state (throw) and vanished manifest ⇒ fail-open delivery; (iv) drop once per runId across many cycles (no re-queue, no re-alert — WD-DEDUP not regressed); (v) legacy caller without an idle probe is still guarded at handoff.
+- **cap-expiry path now re-verifies too (no unverified delivery)** — residual closed from the 2026-10-23 ship battery (`docs/real-test/reports/real-test-2026-10-23-wave10-watchdog-ship-battery.md`): the other way out of the pending map is the 10-minute `deliverCapMs` deadline, taken when the session never goes idle (one foreground tool call blocking past the cap). That exit runs the SAME `classifyAtDelivery` as the idle boundary — terminal manifest or all-tasks-terminal ⇒ DROP with the one-time debug log, and only a run that still reads as hung (or an unreadable/absent record, fail-open) is handed over. "Late beats never" now applies to genuinely hung runs only. Pinned by 4 new tests (watchdog suite 20 → 24): cap expiry + all-tasks-terminal ⇒ 0 messages + 1 drop log (real state write), cap expiry + terminal manifest ⇒ 0 messages + 1 drop log, cap expiry + still hung ⇒ exactly 1 message / nothing pending, cap expiry + throwing loader ⇒ 1 message (fail-open). Mutation-checked: skipping the verify on the cap path turns both drop tests red.
+
+### Fixed — watchdog stale-record silence + poll-interval override (WD-STALE-RECORD / WD-INTERVAL, root-caused 2026-10-23)
+
+- **fix(watchdog): "appears hung" alerts for runs that are actually DONE** — root cause of the two live incidents (team_20261023030033 @04:13, team_20261023050900 @06:12) is NOT a missing finalize: `finalizeRun` wrote `completed` both times (`src/runtime/finalize-run.ts:514` updateRunStatus → `run.completed`, `:559` locked `saveRunManifestAsync`) — proven by `team action='cancel'` answering "already completed; nothing to cancel" 15s after the 06:12 alert landed. The alert was a **stale emit-time snapshot**: the watchdog emitted at ~05:44 during a 480s progress gap (`agents.json` `progress.lastActivityAt` older than the 5-min `AGENT_ACTIVITY_LIVENESS_MS`, `src/runtime/process-status.ts:68,95-125`), and `sendUserMessage({deliverAs:"followUp"})` then QUEUED it behind the parent's blocking tool call for 28 minutes — delivered 19s AFTER `run.completed`. Two bounded fixes in `src/runtime/foreground-watchdog.ts`: **(a) WD-STALE-RECORD** — the pre-emit re-verify now reads `tasks.json` too (`loadRunManifestById` already returns `tasks`); manifest still active BUT every task terminal (completed/failed/cancelled/skipped/needs_attention, `TEAM_TERMINAL_TASK_STATUSES`) ⇒ treat the record as stale: no hung alert, one-time `PI_CREW_DEBUG` log, and exactly ONE user-facing notice per runId ("run X record stale (all tasks terminal) — skipped hung alert", module-scope bounded map). FAIL-OPEN by design: empty/unreadable/corrupt task evidence is never read as "all terminal", so a genuinely hung run still alerts. **(b) WD-INTERVAL** — new `PI_CREW_WATCHDOG_INTERVAL_MS` (registered in the env registry + forwarding decision table as `gap`, default 300000 = the old constant) overrides the 5-minute poll cadence; an explicit `checkIntervalMs` still wins. Documented residual (not fixed here, needs a delivery-time hook the extension API does not expose): a snapshot can still be delivered late by the followUp queue — the emit-time gate cannot re-verify at delivery.
+- **tests**: watchdog suite 9 → 15 — all-tasks-terminal ⇒ 0 hung alerts + exactly 1 stale notice; one task still running ⇒ alert fires; missing `tasks.json` ⇒ never read as terminal (no stale claim, unchanged silent skip, no crash); corrupt `tasks.json` ⇒ legacy hung alert kept (fail-open); stale notice is per-runId across watchdog restarts; `PI_CREW_WATCHDOG_INTERVAL_MS` drives the cadence. Registry snapshot/metadata tests extended for the new env var.
+
+### Fixed — watchdog hung-alert dedup (WD-DEDUP, live-caught 2026-10-23)
+
+- **fix(watchdog): repeated "appears hung" alerts for a run already terminal** — live pain 2026-10-23 (run team_20261023030033…): the foreground watchdog re-alerted "appears hung" for a run that had already completed, because (1) nothing deduped the notice across cycles/instances (the ARCH-5 notice cap of 2 still allowed one repeat, and a re-armed watchdog instance started the counter fresh) and (2) a check armed by an older poll cycle emitted without re-verifying the run state at emit time. Fix in `src/runtime/foreground-watchdog.ts`: **(a) alert-once per runId** — a module-level bounded FIFO map (cap 256, oldest entry evicted — no unbounded growth) suppresses every later "appears hung" emit for the same runId, even from a fresh watchdog instance; **(b) pre-emit re-verify** — the run state is reloaded immediately before any emit; a terminal (or vanished) run skips the emit entirely (one-time `PI_CREW_DEBUG` debug log) and keeps monitoring, so the next cycle's terminal branch still delivers the single completion notice as before. The ARCH-5 going-quiet handoff stays one-shot. Pinned by 4 new tests: alert-once (incl. restart immunity), re-verify skip at emit time, natural terminal transition after first alert, bounded-cap eviction.
+
+### Wave 10 — pi-subagents lessons port (verified 26-item matrix, 2026-10-22)
+
+All ten actionable items from the pi-subagents deep research (5 tiers: architecture → perf → prompt/meta → docs/intercom/missions → live measurement). Two lessons were CLOSED by evidence instead of ported: spawn/parallel performance (live measurement §10-11: pi-crew process-per-worker 0.57–0.76s beats pisub in-process 0.83s solo / 3.27s fan-out per child — nothing to port) and declaration-pinning (not applicable: pi-crew is process-per-worker, the platform owns tool declarations).
+
+- **W10-1 cache TTL split (lesson #10)**: headless single-run workers now spawn with `PI_CACHE_RETENTION=short` — long-retention prompt-cache writes carry a higher per-write surcharge that a one-shot worker never amortizes. Injected at the spawn-options layer (same slot as `PI_CODING_AGENT_DIR`); surface panes and RPC warm sessions keep the parent default. The built.env canary rejects the key anywhere else.
+- **W10-2 memoized green verify-gate (lesson #3)**: a PASS verdict is recorded keyed by the workspace tree OID (`git write-tree`) + check list + node version; a re-gate on a bit-identical tree serves the memo without re-running typecheck/test:critical (tree equality subsumes the M1-7 manifest bookends). Only PASS-with-real-runs records; FAILED/INCONCLUSIVE/drift never memoize; non-git cwd fails open; kill-switch `PI_CREW_VERIFY_GATE_MEMO` (default ON).
+- **W10-3 mailbox typed reason + strict replyTo routing (lesson #22)**: messages gain an optional `reason` enum {need_decision, progress_update, interview_request} and responses resolve ONLY the request their own requester opened (A→B→C rule) — parallel asks can no longer cross-resolve.
+- **W10-4 LLM-slop lint rules (lesson #21)**: `npm run check:slop-rules` (in `ci`) — comment/string-aware source scanners: `no-json-parse-nonnull` + `no-any-deref` STRICT (clean today), `no-double-cast` WARN-only over a 41-site baseline (burn-down follow-up). `// slop-ok: <reason>` escape hatch.
+- **W10-5 decision-drift guardian (lesson #20)**: critic/oracle reconstruct the run's inherited-decisions baseline and must flag contradictions with `DRIFT-BLOCK:`; `extractVerifierVerdict` degrades a pass carrying DRIFT-BLOCK to "none" (fail-closed: a later truncated source cannot launder the marker).
+- **W10-6 budget-boundary policy (lesson #16)**: delegation-patterns skill — budgets block read-tools, NOT mutation-tools; an elapsed timeout is not a mutation-safe boundary (no auto-revert racing the timeout).
+- **W10-7 usage reconcile (lesson #14)**: every model attempt carries its own usage (failed-fallback tokens were invisible — task.usage shows only the FINAL run); `usageReconcile` note on the task when the sum-of-attempts vs final cost gap exceeds 5% with ≥2 usage-carrying attempts. Observability only.
+- **W10-8 read-lane same-tree guard (lesson #4)**: READ-ONLY lanes dirtying their worktree (modified OR untracked — `git status --porcelain`, diff alone misses untracked) stamp an `unexpectedMutation` note; fail-open, leader decides.
+- **W10-9 typed JSON gates (lesson #2)**: checks accept `{output:"json", schema}` (minimal subset type/required/properties/items/enum, zero new deps) — invalid stdout or schema mismatch FAILS at exit 0; parsed `structuredOutput` lands on the check result; typed gates never memoize.
+- **Deferred (PoC when token economics demands it)**: fork prompt-cache tier (lesson #9, #2879: 52k→1.7k written) — needs an agent-side extension in the child pi process; architecturally separate from this wave.
+
 ### Fixed (live battery 2026-10-18 — real-test v0.11.24)
 
 - **fix(gate): RT24-F1 root-gate config/autonomy read over-block** — `rootGateBlockReason` counted the `args` subcommand carrier as a config patch key, so worker-origin `config {args:"show"}` READS were rejected as "writes user-scope config" (live-caught in the v0.11.24 real-test battery; docblock promised reads open). `args` added to the carrier-key sets for `config` and `autonomy`; unknown junk keys stay write-classified (fail-closed). Pinned: reads with carrier keys open, writes hidden behind the same key still gated (`unset`/patch keys), user config byte-untouched. Live-verified end-to-end (fresh `pi -p`, worker env): read allowed + dumps config; `unset` write gate-blocked verbatim "root-session required".
 
+## [0.11.36] — Wave 9: exit-revoke wiring + bwrap OS boundary (Linux opt-in) + google provider route
+
+### Security
+- **Exit-revoke (v0.11.36)**: the `runChildPi` wrapper now revokes the task's inference-broker token in a `finally` — normal exit, crash, abort-before-spawn, even thrown errors. No live token outlives its run; retries re-register a fresh token (register-replace semantics make the ordering safe). Anonymous (agentId-less) spawns keep per-spawn unique keys that die at broker stop. Pinned by an aborted-before-spawn wiring test: registration happens BEFORE the B5 abort check, so the orphan would be observable in `inferenceBrokerStats()` without the revoke.
+- **OS boundary — `PI_CREW_SANDBOX=bwrap` (Linux opt-in)**: headless child-pi workers run inside a bubblewrap namespace. The worker sees ONLY the project cwd (rw), the scoped inference agent dir, explicitly handed-in temp/scratch paths (ro), the inter-pi broker socket (rw — connect needs write), toolchain mounts (node bin dir, pi bundle dir + node_modules ancestors, --extension/--skill resources up to their package root — ALL read-only), and the OS runtime (/usr,/etc ro + tmpfs /tmp,/run + HOME=/tmp). `~/.pi` (auth.json!), `~/.ssh` and every other host path are simply NOT MOUNTED — a forgotten path fails closed as ENOENT inside, never silently reaching the host. Fail-closed on EVERY misconfiguration: unsupported value, non-Linux, missing bwrap binary, net=off+broker (a new netns has no loopback route — broker-token workers would be keyless AND netless). TUI surface panes and the rpc transport are SKIPPED while the sandbox is on — a sandboxed request can never land in an unsandboxed pane. `PI_CREW_SANDBOX_NET=off` additionally unshares the network stack entirely (full egress isolation); the default `share` profile honestly leaves egress open (host-file boundary only).
+- **google provider route**: the broker now routes `/google/*` to the native google-generative-ai upstream (default `https://generativelanguage.googleapis.com/v1beta`, per-provider override `PI_CREW_INFERENCE_UPSTREAM_GOOGLE`), substituting the real credential into `x-goog-api-key` (never Bearer/x-api-key on that wire). Credential precedence: GOOGLE_API_KEY > GEMINI_API_KEY > auth.json:google. Usage counting understands `usageMetadata` (promptTokenCount → input; candidatesTokenCount + thoughtsTokenCount → output, cumulative per stream). Built-in google models (gemini-*) ride the override-only scoped models.json with zero metadata mirroring.
+
+### Fixes
+- Lazy budget 533 → 534 (sandbox.ts is a static import — the spawn path is sync).
+
+### Tests
+- Sandbox battery 9 (NEW): config fail-closed ×3, availability fail-closed, argv pure ×3 (mount ORDER: tmpfs /tmp precedes binds under /tmp; toolchain ro), REAL bwrap smoke ×2 (host ~/.pi invisible while the file demonstrably exists outside + cwd writable; net=off → even loopback unreachable).
+- Broker battery 18 → 21: google substitution/routing/usage/precedence + revoke helper semantics. Revoke WIRING test (NEW): no orphan token after an aborted runChildPi.
+- 4/4 mutation checks: wrapper-revoke removed → wiring red; bwrap-missing fail-open → red; net=off+broker guard dropped → red; google usage names dropped → red.
+- LIVE production-path e2e: a REAL `pi` completing inside bwrap through runChildPi + broker (upstream saw the substituted key; worker env credential-less; stats empty after the run).
+
 ## [0.11.34] — Broker credentials completion: auth.json parents + OpenAI upstream (multi-provider routing)
 
 ### Security / Features
@@ -8713,3 +8761,6 @@ the repo. Two genuine wins, plus an informational CI job that never blocks.
   pattern (`jiti` flagged unused but is used via runtime path-loading). A
   blocking gate would create an unpaid maintenance backlog unsuitable for a
   solo-maintained extension.
+
+### Hardening (W10 batch A)
+- `check-slop-rules` no-double-cast baseline ratchet (41, 2026-10-22 — burn down only) + real-CLI e2e pin; W10-7 reconcileUsage missing-finalUsage guard; W10-8 READ-ONLY lane mutation one-line stderr visibility; W10-9 structuredOutput JSON round-trip pin; W10-2 verify-gate memo time-dependent-skip intent pin.

package/agents/critic.md

@@ -27,6 +27,19 @@ You are a critic. Your job is to CHALLENGE plans and designs BEFORE execution
 - Ownership conflicts (two phases writing the same files)
 - Verification gaps (criterion that can't be objectively checked)
 
+## Decision-drift guardian (W10-5)
+Before critiquing, reconstruct the run's INHERITED DECISIONS baseline: every
+decision already locked in the task context (SPEC/GATES constraints, pinned
+architecture choices, knowledge entries, leader instructions). Then, for EACH
+of your recommendations, answer: does it contradict any inherited decision?
+- Contradiction found ⇒ you MUST open your verdict with a line containing
+  `DRIFT-BLOCK: <decision-ref> — <the contradiction in one sentence>` and
+  treat the recommendation as BLOCKED until the leader explicitly re-decides.
+- No contradiction ⇒ say nothing about drift (do not pad with empty checks).
+A PASS verdict carrying a DRIFT-BLOCK line is treated as NOT VERIFIED by the
+runtime (post-execution.ts W10-5) — flagging a contradiction and approving
+anyway is self-contradictory and downgrades to leader review.
+
 ## Output format
 
 End with exactly this block:

package/agents/oracle.md

@@ -33,6 +33,20 @@ If the question is not strategic, name the right role in ROUTING and stop.
 - For debugging guidance: give a ranked hypothesis list with the cheapest discriminating experiment per hypothesis.
 - For simplification: name the exact deletions/refactors, and what breaks if done — or state that nothing breaks.
 
+## Decision-drift guardian (W10-5)
+You are the consistency guardian for LONG runs: reconstruct the INHERITED
+DECISIONS baseline (SPEC/GATES constraints, pinned architecture choices,
+knowledge entries, earlier leader decisions in the task context) before
+advising. For EACH recommendation, answer: does it contradict an inherited
+decision?
+- Contradiction found ⇒ prepend a line `DRIFT-BLOCK: <decision-ref> — <the
+  contradiction>` to ORACLE_ADVICE and set ROUTING: leader-decision — the
+  leader must explicitly re-decide, not silently absorb your advice.
+- No contradiction ⇒ stay silent about drift.
+A pass-shaped verdict carrying DRIFT-BLOCK is treated as NOT VERIFIED by the
+runtime (post-execution.ts W10-5); flagging a contradiction while endorsing
+the plan anyway is self-contradictory.
+
 ## Output format
 
 End with exactly this block: