pi-crew 0.10.2 → 0.10.3

package/CHANGELOG.md

@@ -2,6 +2,255 @@
 
 > **Note:** `atomic-write-v2.ts` / `AtomicWriter` mentioned in historical entries below was consolidated into `atomic-write.ts` as of v0.9.42. This changelog is preserved as historical record — the migration was completed (the v2 class was never adopted; v1 won on simplicity + symlink-safety + link+unlink atomicity). See `docs/migration/atomic-write-v2-migration.md` for the decision rationale.
 
+## [0.10.3] — MuxSurface: workers in real panes + per-team-run tabs (2026-09-01)
+
+128 commits since v0.10.2. The headline feature is **MuxSurface A1** (spec
+`docs/superpowers/specs/2026-08-26-mux-surface-design.md` v0.7.1, ADR
+`docs/decisions/2026-08-26-mux-surface-a1.md`): crew workers can now live in
+REAL multiplexer panes instead of headless stdio pipes.
+
+### MuxSurface A1 — pane-backed workers
+
+- **SurfaceProvider abstraction** (`src/runtime/surface/`) with twin backends:
+  tmux (libtmux-style CLI) and herdr (socket API, newline-JSON). Fail-closed
+  by design: every failure (no mux, forced-mode detect fail, depth, cap, role)
+  degrades to the headless path and the run still completes — pane engagement
+  is proven by events (`worker.surface_spawned`/`worker.surface_closed`),
+  never by the run's green status alone.
+- **Surface gating follows env + config, not run-mode**: `runtime.surface.mode`
+  (auto/tmux/herdr/off) + `runtime.surface.visibleAgents` opt-in (default
+  `[]` = nobody). Async runs engage panes exactly like sync runs — the detached
+  background runner forwards mux env (TMUX/HERDR_*) so `no-mux` detection sees
+  the host's multiplexer (fix `f0a41a16`, verified live: 3/3 workers in panes
+  on an async run, tab closed at run end).
+- **Per-team-run tabs (tab-layout, spec 2026-08-27)**: every TEAM RUN gets its
+  own tab/window (tmux `new-window -d` / herdr `tab.create`), max 8 worker
+  panes per tab before a new tab opens, alternating down/right splits, and the
+  tab closes ONLY when the run ends/cancels/is killed — workers finishing do
+  not close panes. Doctor closes orphan tabs by id + mux liveness.
+- **herdr pane-parent fix**: spawns target the caller's pane via
+  `HERDR_PANE_ID` (never the focused pane — `pane.current` without
+  `caller_pane_id` follows focus, verified live on herdr 0.8.2).
+- **Worker-side recorder + host tail**: surface workers write per-agent
+  `events.jsonl` (same shape as the host); the host tails it via
+  `EventLogTailSource` (incremental byte-offset reads, drain-before-close for
+  fast-closing panes, ENOENT backoff + log-once, and a steady-poll safety net
+  for FSEvents append coalescing on macOS).
+- **D5 full-loadout**: workers are FULL pi sessions by default — no
+  `--no-extensions`, no tool allowlists; restrictions are per-agent opt-in via
+  frontmatter. **D8 nesting**: every role gets the `delegate` tool
+  (`nesting.maxDepth: 4`, kill switch in user config only). **D9 `message`**
+  tool: non-blocking notify/DM/group via broker with anti-spoof `from`
+  override.
+- **ask gate flip**: `broker.waitMethodsEnabled` now defaults `true` — the
+  worker→parent blocking question path works out of the box (it had silently
+  slept behind the old default-off gate).
+- **Broker token revocation**: stale-token + secret-based checks; terminal
+  runs (completed/failed/cancelled) reject worker tokens by definition
+  (late-worker protection).
+
+### Verification for this release
+
+- E2E real-mux suites: tmux 4/4 + herdr 5/5 (spawn/self-close,
+  kill-pane→degrade→headless resume, doctor orphan cleanup, tab per-run,
+  closeTabById fallback).
+- Two full real-test batteries from live pi sessions
+  (`docs/real-test/reports/real-test-2026-08-30-post-tab-layout-live.md`,
+  `real-test-2026-08-31-post-fixes-live.md`): tab-layout verified with live
+  tmux polls (window per run, sequential worker panes, tab closed at run end),
+  async surface engagement verified post-fix, `set <array-key> []` config
+  round-trip fixed.
+- CI all-green on ubuntu + macOS + Windows (first time since 2026-08-24);
+  lint/format drift and 5 layers of hidden cross-OS test failures cleared.
+
+### Perf rounds also shipping in 0.10.3
+
+Retained verbatim below: round-3 retrieval single-pass, round-2 fsync cleanup
++ polling latency, and the 2026-08-24 performance-review fixes.
+
+### [0.10.3] perf: round 3 retrieval single-pass (retrieval 7.1s → 2.0s cold / 0.28s warm per task)
+
+Root cause (measured on real run `team_20260826002634`, 2026-08-26 real test):
+`runRetrievalCycle` spent up to ~7s CPU per task — 3 unconditional cycles (the
+0.7 convergence threshold is unreachable with path-only scoring, observed
+max 0.64), 55 keywords incl. filler over ~57k files, and a duplicate-
+accumulating evaluation list (same file up to 3× in the top-10).
+
+- Single discovery pass + dedupe by absolute path (`retrieval-orchestrator.ts`)
+- STOPWORDS expanded (filler verbs/pronouns out; path-meaningful words kept)
+- rg discovery cached per cwd, 60s TTL, cap 32 (fallback walk uncached)
+- b13 bench guards cold <2s / cache-hit <400ms on the repo itself
+
+| Metric (my_pi monorepo, full-length real-run goal) | Before (`745cf9f1`) | After | Delta |
+|---|---|---|---|
+| runRetrievalCycle cold | 7055 ms | 1980 ms | −72% |
+| warm, new keywords (discovery cache) | 3335 ms | 275 ms | −92% |
+| keywords from tokenize | 55 | 41 | −25% |
+| duplicate paths in top-10 | up to 3×¹ | 0 | — |
+
+¹ Observed ×3 on the original real-run workload; the identical-input baseline
+re-measure shows max 1 (score-tie ordering can keep cycle-2/3 duplicates out
+of the top slice — the cross-cycle accumulation mechanism itself is what T1
+removed). See the probe report footnote.
+
+Prompt-pipeline impact: the measured 6-8s/task gap (real test) had retrieval
+as its dominant component; post-fix retrieval is 2.0s cold / 0.28s warm —
+the non-retrieval remainder was not re-measured end-to-end yet (next real
+test will confirm).
+
+b13 (`bench/b13-retrieval-latency.bench.ts`) runs on the small pi-crew repo
+(cold 83 ms / cache-hit 8 ms) — it is a smoke guard, not the monorepo
+context above; a regression back to 3-cycle behavior stays under its budget
+at that scale. Full follow-up note: `docs/real-test/reports/perf-round3-probe.md`.
+Every number above was measured, not estimated.
+
+### [0.10.3] perf round 2 (2026-08-25): fsync cleanup + polling latency
+
+13 commits after the 2026-08-24 round (45 after `v0.10.2`) (27 files, +3,094/−105, excluding the plan doc and this changelog). Continuation of the 2026-08-24 performance review work, focused on durability escalations and polling latency. Implementation plan: `.superpowers/sdd/2026-08-25-perf-round2-fsync-and-polling/`. Validation evidence: task reports T1-T8 + `bench/b12-fsync-counts.bench.ts`.
+
+### What changed (by area)
+
+- **Pid files** — `withEventLogLockSync` writes lock pid files via `openSync(pidFile, "wx")` followed by plain write (0 fsyncs) instead of `atomicWriteFile` (2 fsyncs). The lock dir's `mkdir` is the mutex; pid files are mtime-stale-detected. Async `.alock` path unchanged.
+- **Buffered event batch flush** — `appendBatchForBufferedWrite` skips fsync for all-non-terminal batches (terminal batches still fsync). Non-terminal buffered work now pays 0 fsyncs per batch.
+- **Non-terminal tasks checkpoints** — opt-in best-effort fsync gate `PI_CREW_PERSISTENCE_SKIP_TASKS_FSYNC` (default `false`, beats config). When enabled, non-terminal `tasks.json` checkpoints are written with durability `"best-effort"` (0 fsyncs) while keeping the 50ms coalesced write grouping; terminal transitions and default-off path retain full durability. Reconstructible from the fsync'd event log.
+- **Mailbox delivery marks** — `appendMailboxMessage` (regular sync delivery) now defaults to best-effort durability (delivery.json is informational; next message overwrites). Terminal `acknowledgeMailboxMessage` and `appendMailboxMessageAsync` paths keep explicit full durability.
+- **Steering/ask polling** — event-driven adaptive cadence under live-session realtime: 50ms when realtime active (`hasLiveControlRealtimeListeners()`), 500ms when idle. Non-realtime workers stay at 500ms.
+- **Events cursor tail reads** — verified watermark cache with inode-stamped entries (replaces unsound order-assumption fast path from first attempt). Unchanged stamps read zero bytes; growth reads only the appended delta `[verifiedOffset, size)` — the 4MB wide parse now happens only on cold/shrink/violation. Cache invalidates on inode change (compaction/rotation/rewrite) and drops entries on sinceSeq=0/no-limit reads.
+- **Bench infra** — b5 (`deep-tracking`) repaired for deleted `observation-store` module; b11 (`dep-context-cache`) fixed NDJSON contract; new b12 (`b12.fsync-counts`) micro-bench spies `fs.fsyncSync` calls per operation.
+- **Coalesced drain** — `flushPendingAtomicWrites` groups parent-dir fsyncs across all files in the drain (one `fsyncSync` per distinct dir, best-effort on win32). 4 files in one dir: 5 fsyncs total (4 data + 1 dir) vs 8 pre-grouping (4 data + 4 dir).
+
+### Bench results (Linux x86_64, Node v22.23.1, `npm run bench`)
+
+b12 fsync counts per operation (deterministic across runs; "before" refers to pre-round2 baseline):
+
+| Operation | Before | After | Expectation |
+|---|---|---|---|
+| `appendEventSyncNonTerminal` | 2 | **0** | ≤1 (T1 pid write) |
+| `appendEventSyncTerminal` | 3 | **1** | ==1 (documented) |
+| `appendEventBufferedNonTerminalBatch8` | 1 | **0** | ==0 (T2 batch skip) |
+| `tasksCheckpointNonTerminalFlagOff` | 2 | **2** | ≥1 (full durability) |
+| `tasksCheckpointNonTerminalFlagOn` | 2 | **0** | ==0 (opt-in best-effort) |
+| `appendMailboxMessageDeliveryMark` | 2 | **0** | ==0 (T4 best-effort) |
+| `coalescedDrain4FilesOneDir` | 8 | **5** | ==5 (4 data + 1 dir, T8) |
+
+Coalesced drain A/B (4 files, one dir, heavily loaded machine — indicative): median 36.6 ms (post-T8) vs 55.6 ms (pre-T8) for 5 vs 8 fsyncs. Idle-machine measurement: −61% wall-clock (pre-implementation plan measurement: 59.4→22.9ms; task-8's loaded A/B was 36.6 vs 55.6ms). The deterministic contract is the fsync count.
+
+### Durability semantics
+
+**Full durability retained** for: terminal event writes, terminal task transitions, all `acknowledgeMailboxMessage` calls, all direct `atomicWrite`/`atomicWriteJson` calls, and the default (flag-off) non-terminal checkpoint path. Every persistent state transition that is *not* reconstructible from the event log remains fsync'd.
+
+**Best-effort (opt-in or informational-only)** for: pid files (lock dir is the mutex), non-terminal buffered event batches, non-terminal `tasks.json` checkpoints (when `PI_CREW_PERSISTENCE_SKIP_TASKS_FSYNC=1`), and `delivery.json` marks. These state components are reconstructible (pid files via mtime-stale detection; tasks.json from the fsync'd `events.jsonl`; delivery.json is informational and overwritten). A hard crash may lose the latest tail; recovery replays from the last fsync'd event.
+
+### Known residuals (discovered, documented, left for follow-up)
+
+- `flushOnePendingAtomicWrite`'s retry path is dead code (pre-existing; surfaced by T8): coalesced entries are deleted before the write attempt, so the catch block's `retryCount++` and `MAX_FLUSH_RETRIES` rethrow never execute. Fixing requires behavioral changes to failure semantics (separate task). *(→ resolved by the follow-up fixes below)*
+- Cursor cache ino-recycle coincidence: same-path in-place truncate+regrowth would pass the delta-branch check if the rewrite preserved the inode (no current writer does; same accepted-risk class as transcript-cache). Documented in `src/state/event-log/cursor.ts`.
+- Async mailbox twin (`appendMailboxMessageAsync`) retains full durability; T4 scoped to regular sync delivery only. *(→ resolved by the follow-up fixes below)*
+- Tasks-checkpoint `loadConfig()` reads the flag at every save (negligible via 2s TTL cache).
+
+### Follow-up fixes (branch `fix/round2-followups`, 2026-08-26)
+
+- **Dead retry path fixed** — `flushOnePendingAtomicWrite`'s catch now re-queues the entry (the live map entry if a newer write arrived mid-flush, else the captured one) with exponential backoff, so a failed flush retries instead of silently dropping the buffered write; the error still propagates at `MAX_FLUSH_RETRIES`. Regression suite `test/unit/state/atomic-write-coalesced-retry.test.ts` (openSync ENOSPC injection; the coalescer contract is no-throw while retries remain).
+- **Async mailbox twin aligned** — `appendMailboxMessageAsync` delivery marks drop to the best-effort default (mirror of the T4 sync-twin fix); third test in `mailbox-delivery-durability.test.ts` pins 0-fsync pure appends with init absorbed into the baseline.
+- **Test isolation fix (not a production bug)** — `manifest-cache-list-active.test.ts` leaked the real user root into exact-membership assertions (listActive scans every run root by design, RT-F3); the suite now snapshots env and points `PI_CREW_HOME` at an empty temp home, deleting the precedence-winning `PI_TEAMS_HOME`.
+
+### What did NOT improve
+
+- Broker fan-out and worktree git-spawn memoization: unchanged (already addressed in round 1).
+- Async mailbox path (`appendMailboxMessageAsync`): intentionally left at full durability (T4 scoped to sync delivery only). *(→ resolved by the follow-up fixes above)*
+- Tasks-checkpoint coalescing: already present; T3 only added durability control, the 50ms grouping predates this branch.
+
+### Verification
+
+| Check | Result |
+|---|---|
+| `npm run typecheck` | pass |
+| `npm run lint` | pass (2 warnings and 2 infos (all pre-existing)) |
+| `npm run test:unit` | 7153 pass / 0 fail / 3 skipped (7156 total) (includes new T1/T2/T3/T4/T5/T6/T8 test suites) |
+| `npm run bench` | legacy suite green; b5/b11 NDJSON contract repaired; b12 fsync-counts bench added (7 cases, all pass) |
+
+### [0.10.3] perf: fix 2026-08-24 performance review findings (state persistence syscall ceremony, UI sync I/O storms, mailbox/event-log hot paths, broker fan-out, worktree git-spawn memoization)
+
+28 commits after `v0.10.2` (53 files, +4,718/−433). Implements the plan at
+`docs/superpowers/plans/2026-08-24-perf-review-fixes.md`: all Critical (C1–C3) and
+High (H1–H6) findings from the 2026-08-24 performance review, plus the Medium sweep.
+Validation evidence: `.superpowers/sdd/2026-08-24-perf-review-fixes/task-27-report.md`.
+
+### What changed (by area)
+
+- **State persistence syscall ceremony** — `persistSingleTaskUpdate` does a scoped
+  flush + in-lock CAS baseline (only the file being read is drained, not the whole
+  process); atomic-write coalescer gains a dir-exists memo and lazy stringify;
+  `saveRunTasksCoalesced` keeps the manifest half of the run cache across coalesced
+  saves (only tasks stamps are zeroed); artifacts containment verdict memoized 10s
+  (positives only); event-log lock pid files written ceremony-free (no fsync'd
+  tmp+rename for pid files) and the redundant post-reserve monotonic seq persist is
+  skipped (R16-B1 advance-on-reserve untouched); pre-append stat hoisted and
+  reader-less sequenceCache upkeep dropped; lock acquire pre-check caches the
+  symlink verdict; mailbox reads go through a stat-gated parse cache.
+- **UI sync I/O storms** — widget refresh is coalesced on the `fs.watch` path
+  (scheduleRefresh, one async refresh per tick); transcripts render through a
+  tail-windowed wrap + incremental byte-offset reads (no full-file re-read per
+  frame); the dashboard resolves run snapshots once per frame; the widget cache is
+  truncate-once; `visibleWidth` gets a short-string cache; render-diff does a
+  single `diffWords` pass per changed line pair; `listLiveAgents` memoizes its sort
+  with a plain string compare; worker events channel tail is a 1-byte read;
+  config store path does a single `readCacheMtimes` pass.
+- **Broker fan-out / worktree / live-session** — `msg.send` fan-out is chunked
+  over concurrent recipients; sync git probes in the worktree path are memoized
+  (cleanLeader verdict, rev-parse) with throttled per-repo prune; live-session
+  sidechain/transcript writers are batched per 50ms window with bounded (512 KiB)
+  stdout capture and a gated control poll.
+
+### Bench results (Linux x86_64, Node v22.23.1, `npm run bench`)
+
+Pre-fix = same-machine capture 2026-08-24 before the branch; post = this branch.
+Fsync-dominated wall clocks drift ±10-25% day-to-day on this machine — a
+same-day control run on `main` (see report) shows the b4 drift below is
+environmental, not a regression. fsync intentionally remains in the durable path.
+
+| Metric | Pre-fix (2026-08-24) | Post-fix (branch) | Δ |
+|---|---|---|---|
+| `atomic-write-json` warm p50 | 13.01 ms | 13.21 ms | ~flat (fsync floor; control-on-main 13.09 ms) |
+| `b3.state-store-jsonl` n10 `atomicWriteMs` | 14.83 ms | 15.00 ms | ~flat; the "25% faster than same-day control (20.03 ms)" noted pre-merge was load drift — a post-merge interleaved A/B (main vs `519a5e4e`, same session) shows 16.6 vs 16.5 ms, no delta |
+| `b4.event-log` n100 sync append | 1409 ms (70.96/s) | 1518 ms (65.89/s) | environmental drift (control-on-main 1503 ms, 66.53/s) |
+| `b4.event-log` n100 async append | 156 ms (640.86/s) | 198 ms (506/s) | environmental drift (control-on-main 179 ms, 558.30/s); hoped-for async p50 drop did not materialize on wall clock |
+| `b4.event-log` n100 buffered append | — | 261-263 ms (380/s) | new visibility |
+| `snapshot-cache` cold / warm p50 | 1.12 / 1.13 ms | 0.79 / 0.79 ms | **−30%** |
+| `render-flush` p50 | 0.10 ms | 0.10 ms | flat (already sub-budget) |
+| `register-startup` import p50 | 1861.28 ms | 1739.55 ms | −6.5% (cold-cache import, high variance) |
+| `event-append` serial p50 | 15.52 ms | 14.61 ms | −6% |
+| `b2.broker-roundtrip` n1000 | — | 10,049 msgs/s | new visibility |
+| `b7.startup` bundle load (warm avg) | — | 419.89 ms | new visibility |
+
+The structural wins (syscall counts, stat storms, fan-out chunking, batched
+writers) are mostly invisible to these wall-clock benches by design — they remove
+kernel calls whose latency is dominated by the remaining fsync. See the task-27
+report for run-to-run variance data, the same-day main control runs behind the
+"environmental drift" rows, and the b5/b11 bench-infra notes.
+
+Post-merge interleaved A/B (2026-08-25, main vs `519a5e4e` in a worktree, same
+session, alternating runs — the CPU-bound paths the branch targeted, which the
+fsync-floor rows above cannot show):
+
+| Path | baseline | main | Δ |
+|---|---|---|---|
+| `readAllMailboxMessages` warm (160 msgs / 8 files) | 932-949 µs | 521-527 µs | **−44%** (stat-gated parse cache) |
+| `readAllMailboxMessages` churn (60-msg file rewritten per read) | 1903-1970 µs | 1057-1088 µs | **−45%** |
+| `b3` jsonl read n=100 | 1.32 ms | 0.23 ms | **−83%** (cache-keep + stamp reuse) |
+| `b3` jsonl write (n10/100/1000) | 0.11/0.17/0.83 ms | 0.09/0.15/0.77 ms | ~−10% (lazy stringify) |
+| manifest `list()` refresh after TTL (200 runs) | 61-64 µs | 45-66 µs | no measurable wall-time delta (win is syscall count, not latency) |
+
+### Verification
+
+| Check | Result |
+|---|---|
+| `npm run typecheck` | pass |
+| `npm run lint` | pass (2 branch-introduced import-sort errors fixed in `src/state/event-log/event-log.ts`, `src/runtime/live-session/live-session-runtime.ts`) |
+| `npm run test:unit` | 7115-7117 pass / 0-1 fail / 3 skipped — the 1 fail (run 1 of 2) is a pre-existing environmental flake (`session-summary-cov` vector #11 reads the real user-level `~/.pi/.../state/runs`; reproduced identically on `main`; run 2 fully green) |
+| `npm run test:integration` | 189 pass / 0 fail / 4 skipped (env-gated real-model + placeholders), 238.8 s |
+| `npm run bench` | legacy suite green; perf suite requires per-bench invocation until b11 emits NDJSON (pre-existing, see report) |
+
 ## [0.10.2] — UI rewrite + adaptive default team (2026-08-24)
 
 40 commits after `v0.10.1` (≈3,500 LOC, 54 files). Headline: the UI surface