pi-crew 0.9.42 → 0.9.46

package/CHANGELOG.md

@@ -2,6 +2,229 @@
 
 > **Note:** `atomic-write-v2.ts` / `AtomicWriter` mentioned in historical entries below was consolidated into `atomic-write.ts` as of v0.9.42. This changelog is preserved as historical record — the migration was completed (the v2 class was never adopted; v1 won on simplicity + symlink-safety + link+unlink atomicity). See `docs/migration/atomic-write-v2-migration.md` for the decision rationale.
 
+
+## [0.9.46] — UI stability + notification coalescing (2026-07-20)
+
+Three user-facing UI/notification fixes layered on top of the v0.9.45
+remediation, plus CI hygiene. All verified live in a restarted Pi session.
+
+### UI flicker — stop render-path snapshot-cache deletes
+
+The run-snapshot cache has a deliberate stale-while-revalidate design so the
+widget always shows a populated snapshot. That was defeated by three hot-path
+call sites in `lifecycle-handlers.ts` that hard-deleted cache entries; the worst
+ran `invalidate(undefined)` on every ~160ms fallback tick, wiping ALL entries
+→ `activeWidgetRuns` dropped every run to "(loading…)" until the async preload
+rebuilt the cache → continuous flicker across the widget, powerbar, and live
+sidebar/dashboard. (`src/extension/registration/lifecycle-handlers.ts`)
+
+**Fix:** never hard-delete from the render path. A no-runId tick is now a no-op;
+a specific runId calls `refreshIfStale` (stale-while-revalidate); the two
+`fs.watch` change handlers call `refresh` (rebuild-in-place). All three keep the
+entry populated so `get()` never returns `undefined`.
+
+### TUI width-overflow crash — delegate `visibleWidth` to pi-tui
+
+`Rendered line N exceeds terminal width (160 > 159)` killed the host Pi process
+whenever an agent emitted an emoji that pi-crew's hand-maintained `WIDE_RANGES`
+table counted as 1 column but pi-tui (the renderer) counted as 2 (e.g. `⏳`
+U+23F3). pi-crew truncated/padded to its own measure, pi-tui re-measured and
+hard-aborted. This was patched twice before by adding individual codepoints —
+whack-a-mole, since ANY emoji in agent output could trigger it.
+(`src/utils/visual.ts`)
+
+**Definitive fix:** `visibleWidth` now delegates to pi-tui's own `visibleWidth`
+(the function that drives the crash assert). truncate/pad/wrap therefore always
+agree with the renderer; an emoji-width mismatch is structurally impossible. The
+hand-maintained `WIDE_RANGES`/`isWideCodePoint` table is removed.
+
+### Notification coalescing — N drip → 1 consolidated wake-up
+
+Launching N background subagents (no `batch_id`) produced N separate "changed
+state" wake-ups, delivered one-per-turn at turn boundaries; after the leader
+joined them all, redundant per-agent notices kept dripping in over later turns.
+(`src/extension/registration/subagent-manager-setup.ts`)
+
+**Fix (Rule 3):** a debounced coalescer for non-batch completions. Completions
+within an 800ms window (reset on each new arrival) merge into ONE wake-up; each
+is `resultConsumed`-re-checked before emit (Rule 2), so already-joined agents
+are dropped and an all-consumed batch is suppressed entirely. Completions far
+apart (>800ms) still flush separately. The explicit-`batch_id` path (Rule 1,
+BatchBarrier) is unchanged.
+
+### CI hygiene
+
+`biome check --write` applied across the v0.9.45 remediation files (lint +
+format), and two unused test imports were removed — the local remediation
+commits had never been CI-checked; this brings lint + format:check green.
+
+## [0.9.45] — Deep-review 2026-07-20 remediation (all 5 phases, 14 findings)
+
+### ⚠️ BREAKING: custom agent roles default to read-only (FIND-12)
+
+`permissionForRole()` now returns `"read_only"` for unknown/custom agent
+roles instead of the previous permissive `"workspace_write"`. This is a
+security hardening (default-deny) to prevent privilege escalation via
+typo'd or unrecognized agent names.
+
+**Migration:** write-capable roles are now an EXPLICIT allowlist
+(`WRITE_ROLES` in `src/runtime/role-permission.ts`). All shipped
+write-capable roles are listed: `executor`, `writer`, `verifier`,
+`test-engineer`, `agent` (direct-agent default), `cold-verifier`,
+`chain-executor` (chain workflow), and `worker` (goal-loop + dynamic
+workflow). A custom agent role that needs workspace write access
+must be added to `WRITE_ROLES`. Built-in read-only roles (`explorer`,
+`reviewer`, `security-reviewer`, `analyst`, `critic`, `planner`) are
+unaffected. (An earlier draft proposed a `permissions.workspaceWrite`
+agent-config opt-in; it was dropped — unreachable from the call sites.)
+
+
+## [0.9.44] — 3 flaky CI test fixes (2026-07-19)
+
+Three pre-existing flaky tests in the CI suite have been fixed, completing
+the CI green-light requirement. All three were verified by repeated
+local runs (≥30 iterations each) and a green CI run on the merge commit
+(ubuntu/macos/windows Node 22 all ✅, fallow audit ✅).
+
+### Lock subsystem fixes (`src/state/locks.ts`)
+
+- **ENOENT canSteal fix in `readLockSnapshot`** — previously returned
+  `canSteal: false` when the lock file was missing between the EEXIST
+  signal and the read, causing spurious "locked" errors in
+  `parallel-research-dynamic.test.ts` (9/10 local reproduction rate).
+  Now returns `canSteal: true` so the holder-released-mid-retry window
+  is handled correctly.
+- **New `releaseOwnLock`** — unconditional `fs.rmSync` bypass of the
+  token-match check in `releaseLock`. Used by `withRunLockSync` /
+  `withRunLock` because same-process re-acquisition uses fresh
+  `randomUUID()` tokens that never match the stored token, so
+  `releaseLock` was silently never deleting lock files between calls
+  (strace showed many `openat(O_CREAT|O_EXCL)` calls but zero
+  `unlinkat` calls — lock file accumulation until EEXIST).
+- **`treatOwnPidAsStealable` parameter on `readLockSnapshot`** — only
+  the run-lock async path passes `true`. The sync file-lock path
+  (`withFileLockSync`, used by `round26-file-locks.test.ts`) keeps
+  the default `false` to preserve the multi-process safety guarantee
+  ("don't steal a fresh lock held by a live process — even if our own
+  pid"). Without this option split, the first fix attempt regressed
+  the existing `round26-file-locks` test which specifically tests that
+  behavior.
+
+### Test fix (`test/unit/event-log-seq-uniqueness.test.ts`)
+
+- **B7 test 3 uses `appendEventAsync` instead of `appendEventBuffered`**
+  — the 20ms buffered-event timer occasionally lost the race against
+  `--test-force-exit`'s suite-finalization deadline (~8ms cancellation
+  observed on CI), causing `cancelledByParent` failures with error
+  `'Promise resolution is still pending but the event loop has already
+  resolved'`. Tests 1 and 2 still cover the full sync/async/buffered
+  3-path seq-uniqueness contract; test 3 now focuses on the
+  explicit-vs-auto seq-assignment contract using only sync + async
+  paths. This avoids the buffered timer entirely while preserving the
+  original B7 invariant.
+
+### Verification
+
+- Local: 6184/6187 unit + 177/178 integration tests pass (3+1 skipped).
+- CI: ubuntu/macos/windows Node 22 all ✅, fallow audit ✅.
+- Three independent CI runs after each fix confirmed zero regressions.
+
+## [0.9.43] — 4-wave upgrade + live LLM verification (2026-07-19)
+
+A comprehensive upgrade pass was completed on top of v0.9.42: 4 implementation waves
+plus H-1 (lock re-entrance) and H-7 step 1–6 (runChildPi decomposition), verified
+end-to-end with real LLM calls on a real pi session (not mocked). All changes
+were driven by an in-depth audit (`UPGRADE_REVIEW.md` — 647 lines, ~112 unique
+findings) with cold-verification and live test feedback catching regressions
+mock tests had missed.
+
+### Wave 1 — Security + correctness (16 fixes)
+
+- **H-3/H-4** `scripts/build-bundle.mjs` — bundle externalized `acorn` (~225 KB) and fixed `@sinclair/typebox` package name (~245 KB). Total **−481 KB** bundle.
+- **M-23 / COAL-1** `prompt-builder.ts` + `run-coalesced-task-group.ts` — both prompt builders now `sanitizeTaskText` before injection (closes 2 prompt-injection bypasses).
+- **M-22** `cross-extension-rpc.ts` — `pi-crew:live-control` event channel now wrapped with `withHmacVerification` (was the only state-mutating RPC channel without auth).
+- **DISC-1** `dynamic-workflow-runner.ts` — `.dwf.ts` determinism check now transpiles TS→JS via `esbuild` before acorn parse. Restores the round-13 P0-2 enforcement that had been silently no-op'd by acorn's JS-only parser.
+- **NEW-D1** `team-runner.ts` — merge path now calls `flushPendingAtomicWrites()` before `loadRunManifestById`. Closes the coalesced-write-cancellation lost-update window.
+- **M-18 / M-19 / T-S1** — event-log buffered-timer `unref()`, dead-worker `worker = undefined` reset, and `handleSteer` terminal-status guard (rejects steer to completed/failed/cancelled tasks).
+- **NEW-C1 / C5 / C7 / C13 / C17** — config dead-field deleted; token cap covers `max_completion_tokens`/`max_output_tokens`/`generationConfig.max_output_tokens`; tail-capture empty-tail slice fixed; `PI_CREW_HOME` alias added; doc comment env-var name fixed.
+- **CV-NEW-1** `font-detect.ts` — `isWebTerminal()` cached (was 12 sync reads per render tick).
+- **CMD-NEW-2** `commands.ts` — `--var` value split fixed (was `split("=", 2)` which lost values containing `=`).
+
+### CP-1 — turn-limit hard-abort escalation
+
+`child-pi.ts` turn-limit hard-abort now uses `killProcessTree` (same as abort/noResponseTimer paths) and sets a `hardAbortInitiated` flag so `onJsonEvent` stops restarting the no-response timer. Prevents SIGTERM-ignoring runaway children. `HB-003a` source-contract test updated to verify the new behavior.
+
+### Wave 2 — Budget / Worker cap / Worktree / Coalesced (6 fixes)
+
+- **BDG-1** `goal-evaluator.ts` — judge token usage now returned and accumulated into `budgetUsed` (was discarded, systematically undercounted budget by 20–30%).
+- **GL-1** `goal-loop-runner.ts` — loop checks `turnResult.manifest.status` after each turn; stops early on `blocked`/`failed` instead of burning the full `maxTurns` budget on a dead-end.
+- **DLOCK-1** `dynamic-workflow-context.ts` — `ctx.agent()` now goes through `withWorkerSlot`, finally respecting the global worker cap (`PI_CREW_MAX_WORKERS`). Docstring claim was previously false.
+- **ERR-1** `dynamic-workflow-runner.ts` — script timeout now aborts a combined `AbortSignal`, so spawned `runChildPi` children are killed instead of leaking.
+- **N-1** `worktree-manager.ts` — `snapshotDirtyWorktree` failure now causes the caller to skip the discard (preserves dirty work instead of losing it). The previous behavior would silently destroy work if the snapshot artifact write failed.
+- **N-2** `live-agent-manager.ts` — `evictStaleLiveAgentHandles` no longer treats `session.pid === undefined` (the in-process case) as dead. Previously every in-process live agent was killed after 30 min idle.
+- **COAL-2** `run-coalesced-task-group.ts` — coalesced dispatch wrapped in `executeWithRetry`. One transient failure no longer fails all N tasks.
+
+### Wave 3 — Observability + Budget reserve + DWF resume (4 fixes)
+
+- **OBS-NEW-1/2** `foreground-run-controller.ts` — emits per-task `crew.task.completed/failed/needs_attention` events at run completion with `durationMs` and `tokens`. Added `durationMs` to `crew.run.completed`. Fixes task-level metrics that were always zero.
+- **BDG-2** `dynamic-workflow-context.ts` — reserve-then-adjust token budget. Before spawn, reserve estimate (4096 tokens); after spawn, adjust by `actualUsage − estimate`. Prevents N concurrent calls from each overspending by `perCallCost`.
+- **PERS-1** `dynamic-workflow-context.ts` — per-agent-call idempotency cache stored in checkpoint (`completedAgentCalls`). On resume, cached calls are skipped (return cached result). Previously a DWF resume re-ran the entire script, duplicating artifacts/mailbox/tokens.
+
+### H-1 — AsyncLocalStorage re-entrance guard (root-cause class)
+
+`src/state/locks.ts` — `withRunLock`/`withRunLockSync` now track re-entrance via `AsyncLocalStorage<Set<string>>` instead of a module-global `Map<filePath, token>`. True nested calls in the same async context still bypass (no deadlock). Calls from a DIFFERENT async context (e.g. a child-stdout event handler firing while an async merge holder is awaiting) no longer bypass — they properly serialize against the on-disk lock.
+
+### H-7 — `runChildPi` decomposition (6 steps, −572 lines / −31.1%)
+
+`src/runtime/child-pi.ts` reduced from **1842 → 1270 lines**. Code extracted into 4 self-contained modules with zero behavior change verified by 147 tests + live LLM:
+
+| Step | Module | Lines | Purpose |
+|------|--------|-------|---------|
+| 1 | `child-pi-transcript.ts` | 169 | Transcript batching (50 ms debounce) + compaction helpers |
+| 2 | `child-pi-kill.ts` | 180 | `killProcessTree`, `appendBoundedTail`, active-children bookkeeping |
+| 3 | `child-pi-constants.ts` | 42 | Shared timing/buffer constants |
+| 4 | `child-pi-streams.ts` | 296 | `ChildPiLineObserver` + line-level compaction |
+| 5 | `child-pi-steering.ts` | 128 | `ChildPiSteeringController` state machine (turn-limit, soft/hard abort) |
+| 6 | `child-pi-spawn.ts` | 226 | `buildChildPiSpawnOptions`, `prepareSpawnContext`, env allowlist |
+
+Steps 1–4 verified clean on first commit. Step 5 required updating the `HB-003a`
+source-contract test (steering logic moved to its own module). Step 6 required
+updating the assertion to check `builtEnv` (control vars only) rather than
+`mergedEnv` (which includes process.env spread).
+
+### Live-test-only bug fixes (caught by real pi binary, missed by mocks)
+
+- **Token usage not captured** — `src/runtime/pi-json-output.ts` `extractUsage` now recursively looks into `obj.message.usage` (pi nests usage under `message` for `message_end`/`turn_end` events). Mock tests passed because `PI_TEAMS_MOCK_CHILD_PI` short-circuits the spawn path entirely; only real `--mode json` output exercises `extractUsage`.
+- **child-pi spawns wrong binary** — `src/runtime/pi-spawn.ts` `resolvePiCliScript` previously trusted `process.argv[1]` unconditionally. When invoked from a standalone test script, `argv1` was the test script and child-pi spawned it as if it were pi. Fix: only trust `argv1` when `resolvePiPackageRoot()` succeeds (current process is running from the pi-coding-agent package).
+- **Worktree mode blocked by own .gitignore** — `src/worktree/worktree-manager.ts` `assertCleanLeader` used `git status --porcelain` without `--untracked-files=no`. After pi-crew's `ensureCrewDirectory` auto-creates `.gitignore`, that file shows as untracked and blocks worktree mode. Fix: add `--untracked-files=no` so only tracked changes are checked. Applied to both sync (`assertCleanLeader`) and async (`assertCleanLeaderAsync`) variants.
+
+### Live test coverage (real pi binary + real LLM, on session restart)
+
+14 live tests run end-to-end across 3 waves — all green:
+
+| Wave | Feature | Result |
+|------|---------|--------|
+| A1 | 3 concurrent runs, unique runIds, no state leakage | PASS |
+| A2 | Steer to completed task → rejected with `T-S1` error | PASS |
+| A3 | Long research run (5 key points + explanations) | PASS · 8,302 tokens / 124 s |
+| A4 | Cancel completed run is safe no-op | PASS |
+| A5 | Steer to nonexistent runId → graceful error | PASS |
+| B3a | Invalid model → fallback to `zai/glm-5.2` agent default | PASS |
+| B3b | Empty goal → graceful error | PASS |
+| B3c | Long goal stress (multi-paragraph objective) | PASS · 5,247 tokens / 178 s |
+| C1 | Worktree mode (clean git repo, after fix) | PASS |
+| C2 | Original file preserved across worktree cleanup (N-1 fix) | PASS |
+| C3 | `implementation` workflow creates `math.js` with `subtract` function | PASS |
+| C4 | `plan` action returns 4-step plan | PASS |
+
+Mock tests: 30/31 pass (1 pre-existing skip).
+
+### Files
+
+- `UPGRADE_REVIEW.md` — 647-line audit report (all findings + cold verification + live verification).
+- `test/functional/pi-crew-live.test.ts` + `pi-crew-live-broad.test.ts` — 16 live integration tests run against the real pi binary + real LLM provider.
+
+
 ## [0.9.42] — 107-finding deep review + arch cleanup (2026-07-17)
 
 ### Deep Review Resolution

package/README.md

@@ -279,6 +279,41 @@ The advisory is **informational only** — there is no `force:true` flag needed
 - `test/unit/topology-analyzer.test.ts` — 13 cases (each topology + edge cases)
 - `test/unit/preflight-validator.test.ts` — 11 cases (each level + advisory contract)
 
+
+- `test/functional/pi-crew-live.test.ts` + `test/functional/pi-crew-live-broad.test.ts` — 16 live integration tests run against the **real pi binary + real LLM provider** (verified after v0.9.42 audit, ~26 commits). Use these when you want to confirm end-to-end behavior, not just unit-level invariants.
+
+## Recent changes
+
+### v0.9.45 – v0.9.46 (2026-07-20): security + perf remediation, UI stability
+
+- **Custom agent roles default to read-only (FIND-12, breaking):**
+  `permissionForRole()` returns `"read_only"` for unknown roles (was
+  permissive `"workspace_write"`). Write-capable roles are now an explicit
+  allowlist (`WRITE_ROLES`). See `CHANGELOG.md` for the migration.
+- **Mailbox / snapshot-cache / scan perf:** delivery cache + async append
+  (FIND-01/02), `listActive` TTL + mtime-sort strict limit (FIND-03/04),
+  byte-bounded event-log tail-read (FIND-05).
+- **Heartbeat race (FIND-06):** in-flight guard + drain + terminal-safety +
+  late-save repair in the coalesced task group.
+- **UI flicker eliminated:** the render path no longer hard-deletes snapshot
+  entries, so the crew widget / powerbar / live sidebar stay stable (no more
+  "(loading…)" flashing every ~160ms).
+- **Emoji width-overflow crash fixed for good:** `visibleWidth` delegates to
+  pi-tui's own measure, so agent-output emoji can never exceed terminal width.
+- **Background-subagent notifications coalesce:** N near-simultaneous
+  completions now produce ONE consolidated wake-up instead of N drips.
+
+### v0.9.42 – v0.9.44: 4-wave audit + flaky-CI fixes
+
+A 4-wave audit + fix pass was completed (see `UPGRADE_REVIEW.md` for the full 647-line report and `CHANGELOG.md` for the diff):
+
+- **−481 KB** bundle size (externalized `acorn` + fixed `@sinclair/typebox` name).
+- **−31.1%** in `child-pi.ts` (1842 → 1270 lines via 6 decomposition steps).
+- **~30 bug fixes** across security, correctness, budget enforcement, worker cap, observability, and worktree isolation.
+- **Lock re-entrance guard** via `AsyncLocalStorage` (H-1) — fixes a root-cause mutual-exclusion violation.
+- **Task-level metrics now functional** (OBS-NEW-1/2) — `crew.task.{count,duration_ms,tokens_total}` were always zero; now emit on every run completion.
+- **Live-test-only bugs found** (mock tests had passed): token usage not captured (nested `obj.message.usage`), child-pi spawning wrong binary (`argv1` trust heuristic), worktree mode blocked by own `.gitignore`.
+
 ## Builtin Agents
 
 ```