pi-crew 0.9.32 → 0.9.33

package/CHANGELOG.md

@@ -1,6 +1,66 @@
 # Changelog
 
-## [0.9.32] — typescript 7 upgrade (2026-07-10)
+## [0.9.33] — correctness hardening + reverse audit (2026-07-11)
+
+Shipped the **forward performance audit** quick wins, then ran a **reverse audit** (6 parallel bug-hunters → firsthand verification) that surfaced 9 real correctness bugs the forward audit missed. **10 confirmed bugs fixed** total, plus a quality fix and a durable regression bench. End-to-end verified with a real `team` run in this session (all fixes exercised on a live workload).
+
+Detailed methodology and per-finding verdicts: [`docs/perf/performance-audit-report-2026-07.md`](docs/perf/performance-audit-report-2026-07.md) · benchmark: [`test/bench/terminal-persist-blocking.bench.ts`](test/bench/terminal-persist-blocking.bench.ts)
+
+### Correctness — reverse audit (HIGH/CRITICAL, all verified firsthand)
+
+The reverse audit ran 6 parallel bug-hunters across concurrency, state machine, resume, cancel, prune, and event-log/artifact paths. Each agent finding was treated as a lead and independently verified by reading the actual code before any fix — out of ~25 reports, **9 were confirmed real bugs**, several were re-evaluated as WONTFIX (backstopped, defensive-as-is, or fix-was-incorrect), and the rest remain unverified leads. Every shipped fix has a regression test proven to fail without the fix and pass with it.
+
+| # | Bug | Severity | Commit | File |
+|---|-----|----------|--------|------|
+| 1 | **FIND-1/STATE-1** — immediate atomic writes did **not** cancel pending coalesced writes. A `persistSingleTaskUpdate` coalesced write (50 ms debounce) would fire **after** the merge's immediate `saveRunTasksAsync` and overwrite the fresh merge result with a stale single-task snapshot — losing other workers' results, attempt enrichment, and `refreshTaskGraphQueues`. Also closed the crash window where `tasks.json` showed `running` while `events.jsonl` already showed `completed` → recovery re-ran the completed task (duplicate tokens + duplicate side effects). | **CRITICAL** | `3b8446a` | `src/state/atomic-write.ts` |
+| 2 | **R2** — `handleResume` loaded the manifest+tasks **before** the run lock and used the stale snapshot inside the lock → `resetTasks` mapped a just-completed task back to `queued` → `executeTeamRun` re-executed it (duplicate tokens + duplicate side effects). Sibling handlers (`cancelOrphanedRuns`, `reconcileAllStaleRuns`) already re-read inside the lock; resume did not. | HIGH | `e733dba` | `src/extension/team-tool.ts` |
+| 3 | **R1** — `handleResume` had **no `ownerSessionId` check** (unlike `handleRetry`/`handleCancel`). Another session could resume (and re-execute) a run it doesn't own, racing the owning session. | HIGH | `e733dba` | `src/extension/team-tool.ts` |
+| 4 | **P1** — `handleForget` deleted run state without killing the background runner or terminating live agents (unlike `handleCancel`). Forgetting a running run orphaned the runner + child Pi workers (separate process groups via `setsid`) — they kept executing and consuming API tokens until natural completion. The `orphan-worker-registry` only tracks the runner PID, not individual child PIDs. | HIGH | `ca43c54` | `src/extension/team-tool/lifecycle-actions.ts` |
+| 5 | **F1** — the batch loop did `if (results.length === 0) break;`. `results` is empty **only** when every ready task was hook-skipped (`batchTasks → dispatchUnits → []`). The skipped tasks are now terminal, so their downstream dependents become ready on the next iteration — but `break` exited the loop entirely, leaving downstream tasks `queued` while the post-loop code fell through to a false-green `updateRunStatus(manifest, "completed")`. Trigger: a `before_task_start` hook that blocks all ready tasks when downstream tasks depend on them. | HIGH | `118f9ac` | `src/runtime/team-runner.ts` |
+| 6 | **CANCEL-1** — the batch merge used the in-memory `tasks` closure variable as the base for `mergeTaskUpdatesPreservingTerminal`. An external cancel (`handleCancel`) writes `cancelled` to `disk.tasks`, but the in-memory tasks don't reflect it → the merge overwrote the just-written cancellation with the stale in-memory view, permanently losing per-task cancel state. Reachability: background run + SIGTERM race (cancel writes + sends SIGTERM, but the runner's merge, already queued, acquires the lock after cancel released it and overwrites before the kill lands). | MED-HIGH | `ae40c96` | `src/runtime/team-runner.ts` |
+| 7 | **CANCEL-2** — the post-merge cancel handler correctly re-cancelled the **manifest** status to `cancelled`, but saved tasks **without** re-applying the cancel to non-terminal tasks. Result: `status=cancelled` but tasks showed `completed/running` — inconsistent, and broke `handleRetry`'s filter for failed/cancelled tasks. Sibling of the batch-loop cancel check at `team-runner.ts:~925` which DID re-cancel tasks; the post-merge handler just forgot. Complements CANCEL-1 by handling signal-abort-during-merge (where CANCEL-1's disk-cancel never landed). | MEDIUM | `fd476b1` | `src/runtime/team-runner.ts` |
+| 8 | **CANCEL-3 / F3** — `shouldMergeTaskUpdate` blocked `failed→completed` resurrection but missed the symmetric dangerous terminal→terminal flips: `cancelled→completed` (resurrection) and `completed→failed` (success demotion). Both are invalid per the task transition table (`contracts.ts`: terminal statuses only transition to `queued` on retry). A worker that completed after the task was cancelled could flip the settled `cancelled` status back to `completed`; a stale failed result arriving after completion could demote `completed` to `failed`. | MEDIUM | `3f237df` | `src/runtime/team-runner.ts` |
+| 9 | **P2** — `pruneFinishedRuns` deleted `stateRoot` (`<crewRoot>/state/runs/<runId>/`) and `artifactsRoot`, but worktrees live at `<crewRoot>/state/worktrees/<runId>/` — a separate path that `fs.rmSync` does not touch. Every pruned worktree-using run leaked its worktree dir + git branch. `handleForget` already called `cleanupRunWorktrees`; prune did not. | MEDIUM | `733f0be` | `src/extension/run-maintenance.ts` |
+| 10 | **P3** — `isFinished` included `"blocked"` as terminal, but blocked is NOT a terminal status (per `TEAM_RUN_STATUS_TRANSITIONS`, blocked can transition to `running/cancelled/failed` — e.g. plan-approval-pending, waiting-for-mailbox-response, scheduler stall). Pruning a blocked run destroys recoverable state: a plan the user needs to approve, or a mailbox wait the user needs to answer. Truly stuck blocked runs are handled by the stale-reconciler/crash-recovery, not by prune. | MEDIUM | `733f0be` | `src/extension/run-maintenance.ts` |
+| 11 | **EL-1** — `appendEventAsync` reserves a seq, yields across `appendFile`/`fsync` awaits; a concurrent sync `appendEvent` can reserve+persist a higher seq (sidecar=6), then the async path resumes and persists its **lower** seq → sidecar regresses below the file's true max. `nextSequence` trusted the regressed sidecar unconditionally → on restart returned a duplicate seq. Fix: `nextSequence` takes `max(stored, scanSequence(eventsPath))` when trusting the sidecar, so a regressed sidecar cannot cause duplicate sequence numbers. Cost: `scanSequence` on cache-miss paths (already done in the else branch). | MEDIUM | `d27c65f` | `src/state/event-log.ts` |
+| 12 | **EL-2** — `process.on("exit")` / `SIGTERM` / `SIGINT` / `uncaughtException` handlers called `flushEventLogBuffer()` / `drainAsyncQueues()` (async). The `exit` handler is **sync-only** and Node does **not** await the returned promises, so buffered events (`task.progress` via `appendEventBuffered` etc.) were silently lost on every process termination. Additionally, `flushOneEventLogBuffer` **dequeues** (`bufferedQueues.delete`) **before** the await — even if the write were awaited, the dequeue happens first, so a failed/abandoned write leaves events dequeued-without-persisted. The async `.catch(() => {})` calls also created floating promises that prevented `beforeExit` from firing. Fix: new `flushBufferedQueuesSync()` using the sync event-log lock + `appendFileSync` + fsync + persistSequence; all four handlers now call it + clear `asyncQueues`, recovering buffered events before the process dies. | MEDIUM | `d27c65f` | `src/state/event-log.ts` |
+
+### Correctness — forward audit (NEW-C3)
+
+- **NEW-C3** — terminal manifest+tasks persistence race. `runTeamTask` wrote `manifest.json` (immediate) and `tasks.json` (via `persistSingleTaskUpdate` → `saveRunTasksCoalesced`, 50 ms debounce) **outside any lock**, racing the team-runner batch merge which writes both under `withRunLock`. A worker could persist its terminal artifacts → the merge's `loadRunManifestById` reads the worker-written manifest → merge overwrites with the worker's stale `tasks` (which lacks the merge's attempt enrichment + graph recompute) → silent data loss. Fix: wrap `saveRunManifest` + `persistSingleTaskUpdate` in a single `withRunLockSync` so the whole terminal state lands atomically. (`a03d244`)
+
+### Performance — forward audit quick wins (Phase 1)
+
+- **NEW-M1** — memoize stable prompt prefix per `(cwd, step, runId)`. The stable prefix (workspace tree + retrieval + knowledge + skills) was recomputed independently for every task in a parallel batch — identical work for siblings. `renderTaskPrompt` already separates `stablePrefix` from `dynamicSuffix`; pre-warm once per unique cwd in the team-runner batch loop. `clearStablePrefixCache()` in `finally` (matches the existing `flushEventLogBuffer` cleanup pattern) so it does not grow unbounded across runs in a long-lived session. (`08d1a64`)
+- **A1-F7** — guard `collectedJsonEvents` allocation behind `yieldEnabled`. For child-process workers (the common case, `yieldEnabled=false`), the array was allocated and accumulated but only consumed by live-session yield detection — `~10 KB` wasted per task. (`08d1a64`)
+- **A4-F1** — cache the compacted skill body in `CachedSkillMarkdown` at cache-miss time. `compactSkillContent(raw)` ran on every cache hit, wasting `~15%` of skill-rendering time. (`08d1a64`)
+
+### Quality
+
+- **A2-F1** — consolidate `cleanupUsage()` into `finally`. Previously duplicated on the success path (`team-runner.ts:719`) and the error path (`:821`), **outside** the `finally` block. A future throw in the success tail or error handler could skip both copies and leak tracked-usage entries. Consolidated into `finally` (alongside `flushEventLogBuffer` and `clearStablePrefixCache`) so cleanup is guaranteed on **every** exit path. DRY + robustness + consistency. Verified safe: nothing in `team-runner` reads usage after the run resolves; the TUI crew widget reads live usage only while a task is running (before this point). (`e1273ab`)
+
+### Audit infrastructure
+
+- **Performance audit report** — [`docs/perf/performance-audit-report-2026-07.md`](docs/perf/performance-audit-report-2026-07.md). 26 findings from the forward audit (perf, memory, I/O, concurrency, prompt construction), each with file:line, severity, estimated impact, risk, fix direction, and (post-bench / post-reverse-audit) verdict. Five verdicts recorded: A5-C2 disproven (no same-process contention), A5-C4 backstopped by `signal.aborted` (verified end-to-end), NEW-C6 read necessary (orphan-artifact recovery from write-then-throw workers), NEW-M2 stat is correctness-first (dirStamp would miss in-place edits), A2-F1 → quality fix (cleanupUsage consolidation). Backstop analysis explicitly traced for A5-C4: `handleCancel → abortForegroundRun → controller.abort()` on the EXACT controller from `startForegroundRun` → whose signal is passed as `executeTeamRun({signal})` → threaded to retry loop + `runTeamTask`; `runTeamTask:168` checks `input.signal?.aborted` **before** spawning a worker, so even a 100%-stale disk read cannot produce a zombie.
+- **Reverse audit harness** — `team` action='run' with `review` workflow + direct reviewer subagents, 6 parallel hunts across concurrency / state-machine / resume / cancel / prune / event-log. Each agent received a focused mandate (READ-ONLY, find real correctness bugs, file:line + trigger + severity, already-known exclusions). Agents produced `~25` leads; only those verified firsthand by reading the code were shipped. The signal-backstop analysis (A5-C4) and the orphan-recovery analysis (NEW-C6) both came from reading the call-site code end-to-end — not from agent output alone.
+- **Terminal-persist-blocking benchmark** — [`test/bench/terminal-persist-blocking.bench.ts`](test/bench/terminal-persist-blocking.bench.ts). 4 scenarios (`serialPersist`, `saveManifestLarge`, `singleTerminalBlock`, `spacedBurstPerCall`) under `monitorEventLoopDelay`. Empirically answers the A5-C2 question: `contentionRatio` = spaced per-call p50 / single-block p50. Runs at 0.93–0.98 across 4 runs (consistently ≈1.0) proves **no lock contention** between concurrent task completions; the sync critical sections serialize naturally in the single-threaded event loop and `sleepSync` never fires for same-process workers. The async-lock conversion would relieve contention that does not exist. Captured into `test/bench/results.json`.
+
+### Verification
+
+- **Typecheck**: `tsc --noEmit` clean.
+- **Lint / format**: `biome check` clean (1125 files).
+- **Conflict markers / lazy imports**: clean.
+- **Pack dry-run**: clean (`pi-crew-0.9.33.tgz`).
+- **Targeted unit tests**: **145/145 pass** across every changed area (atomic-write, state-store, team-runner, resume, checkpoint, lifecycle, maintenance, process-status, usage-tracker, event-log).
+- **Regression tests proven to fail without the fix and pass with it**: FIND-1 (sync + async paths), CANCEL-3 (cancelled→completed), F3 (completed→failed), F1 (false-green break), EL-1 (sidecar regression via `__test__nextSequence` after `__test__clearSeqCounters` to simulate restart).
+- **End-to-end**: a real `team action='run' goal='...write to /tmp/...'` with the `fast-fix` workflow completed end-to-end in this session (`team_20260711063219_6d1a236ec818315a` — 3/3 tasks completed, manifest `completed`, 3 events with **contiguous seqs 1–3 no duplicates**, sidecar = file max). Every fix exercised on a live workload.
+
+### Honest framing
+
+- **False-positive discipline**: agent-reported findings were treated as leads, not facts. The forward audit's A5-C2, A5-C4, NEW-C6, NEW-M2, A2-F1 were each scrutinized — three were WONTFIX (with the reasoning recorded in the audit report), one was a real-but-backstopped correctness concern (A5-C4), one was a quality fix (A2-F1). The reverse audit's ~25 leads filtered to 9 confirmed real bugs. Several agent reports (`R3`, `R4`, `F2`, `F4/F5`, `ART-1`, `FIND-2`, `CANCEL-5/6/7`) remain **unverified** and are deliberately not shipped.
+- **Scope**: no new features. No API surface changes. No breaking changes. All fixes are surgical and behavior-preserving except where they fix incorrect behavior (false-green → blocked; orphan processes → killed; orphan artifacts → preserved; terminal flips → blocked).
+
+---
 
 Upgraded the TypeScript toolchain from 5.9.3 to **7.0.2** (the native Go port, a.k.a. Project Corsa / `typescript-go`). This is a dev-toolchain-only change: pi-crew ships source `.ts` loaded at runtime via Node `--experimental-strip-types`, so end users see zero behavioral change. The upgrade affects only `npm run typecheck` and editor type-checking.