pi-crew 0.9.16 → 0.9.18

package/CHANGELOG.md

@@ -1,5 +1,135 @@
 # Changelog
 
+## [v0.9.18] — perf fix bundle-mode spawn + config cache (2026-07-02)
+
+Five commits addressing items from the v0.9.17 performance review (`docs/perf/performance-review-2026-07.md`):
+
+### Highlights
+
+- **CRITICAL — fix bundle-mode background runner spawn.** `spawnBackgroundTeamRun` in `src/runtime/async-runner.ts:226` was computing the `background-runner.ts` path via `import.meta.url-relative` resolution. The path landed correctly in source (strip-types loader) but BROKE in the bundle (default v0.9.17+): esbuild's `__esm` helper does not preserve per-module `import.meta.url`, so the path resolved to `<pi-crew>/background-runner.ts` (missing `src/runtime/`). Spawned runners then ENOENTed at ~4s into every team run. Fixed with `packageRoot()` from `utils/paths.ts` — mirrors the same pattern as the v0.9.17 fix in `pi-args.ts:10` (commit `0dd93e0`). Verified live: 3 E2E team runs after the fix (test-coalesce-static, fast-fix, research) all spawned the background runner cleanly.
+- **F4 mitigation — wire `saveRunTasksCoalesced` into the checkpoint path.** The 50ms-debounce coalescer in `state-store.ts:428` had been dormant (0 callers). Now called from `persistSingleTaskUpdate` (called ~5× per task from `task-runner.ts:233, 424, 538, 617, 1347`). Two safety guards: `flushPendingAtomicWrites()` at the top of the mtime-CAS retry loop (defeats the stale-read window under concurrent coalesced writers) and the mtime CAS itself still guards against concurrent non-coalesced writers. Trade-off: checkpoint writes are now best-effort within a 50ms window — terminal writes still use full fsync via `saveRunTasks`.
+- **F16 quick win — 2s TTL+mtime cache for `loadConfig`.** `loadConfig` had 78 callers (top: `register.ts` 11, `registration/ui.ts` 6, `team-tool.ts` 5) and was called 1 Hz idle / 6 Hz active with zero cache. Each call reads up to 4 files (legacy, user, `.crew/config.json`, `.pi/pi-crew.json`), parses JSON, runs full TypeBox `Value.Check` validation, and merges. New cache follows the same pattern as `manifest-cache.ts`. Cache key is a deterministic JSON encoding of `(filePath, legacyPath, projectPath, projectPiCrewJsonPath, cwd)`. On hit, return cached value without parsing; on miss or mtime change, re-parse. New exports for tests/ops: `__test__setConfigCacheTtlMs`, `__test__getConfigCacheTtlMs`, `__test__getConfigCacheEntry`, `__test__configCacheSize`, `invalidateConfigCache`, `flushConfigCache`. Caveat documented: caches for 2s even if the user just edited their config in a separate process — acceptable for the perf win.
+- **D — documented atomic-write-v2 migration plan.** New `docs/migration/atomic-write-v2-migration.md` (297 lines) covers why migrate (F3, F4, F6: drop the dir-fsync cost added by 13f4490), API differences, 3-phase migration plan (dual-write behind feature flag → switch default → deprecate v1), risks + mitigations, effort estimate (M = 3-5 days), and a MIGRATE decision. Existing `atomic-write-v2.ts` (0 callers today) is the v2 surface; v1 (`atomic-write.ts`) keeps symlink-safety + link+unlink atomicity. Phase 1 is gated by `PI_CREW_ATOMIC_WRITER` env flag with instant rollback.
+
+### Verification
+
+| Gate | Result |
+|---|---|
+| TSC | ✅ |
+| Lint | ✅ |
+| format:check | ✅ |
+| check:conflict-markers | ✅ |
+| check:lazy-imports | ✅ |
+| check:bundle-staleness | ✅ |
+| test:unit local (config-cache) | ✅ 7/7 |
+| test:unit local (safe-bash, post-style-fix) | ✅ 26/26 |
+| test:unit local (coalesce + atomic-write-coalesced + file-coalescer + progress-event-coalescer) | ✅ 26/26 |
+| Live E2E: test-coalesce-static (3 tasks) | ✅ all 3 resultArtifacts + heartbeats + clean closeout |
+| Live E2E: fast-fix (explore→execute→verify) | ✅ consistency=1, 14 min |
+| Live E2E: research (explore→analyze→write) | ✅ consistency=1, summary 4.7KB |
+| CI Ubuntu / Node 22 | ✅ (post-push) |
+| CI macOS / Node 22 | ✅ (post-push) |
+| CI Windows / Node 22 | ✅ (post-push) |
+
+### Changed
+
+- `src/runtime/async-runner.ts:224-235` — `runnerPath` now uses `packageRoot()` (1 import added at line 11).
+- `src/runtime/task-runner/state-helpers.ts:1-2, 57-69, 121-127` — wire `saveRunTasksCoalesced` + `flushPendingAtomicWrites` guard.
+- `src/state/state-store.ts:425-434` — export `saveRunTasksCoalesced` (was local), update doc comment.
+- `src/config/config.ts:69-166, 1077-1100` — new cache module + cache-aware `loadConfig`.
+- `test/unit/config-cache.test.ts` (new, 242 lines, 7 tests).
+- `docs/migration/atomic-write-v2-migration.md` (new, 297 lines) — migration plan + decision.
+- `docs/perf/performance-review-2026-07.md` — verification report (added in v0.9.18 cycle).
+- 31 files via biome auto-fix (`style: biome auto-fixes` commit `299338e`) — import-sort + minor semantic.
+
+### Migration notes
+
+- No public-API breaking changes. Existing workflows and configs continue to work without modification.
+- The `loadConfig` cache is transparent — first call hits disk, subsequent calls within 2s return cached value. To force a refresh (e.g. after a config edit), call `invalidateConfigCache()` from a custom integration or just wait 2s.
+- The `saveRunTasksCoalesced` swap is checkpoint-only; terminal writes remain fully durable. If a process crashes mid-50ms-buffer, the last checkpoint may be lost but the on-disk `manifest.json` + `tasks.json` source of truth is preserved.
+- The atomic-write-v2 migration plan describes a 3-phase rollout behind `PI_CREW_ATOMIC_WRITER=v1|v2` flag (default v1 until Phase 2). Users on critical/durability-sensitive deployments can opt into v2 after Phase 1 stabilizes.
+
+## [v0.9.17] — coalesced micro-tasks (M6) ships + closeout race fix (2026-07-02)
+
+Two end-to-end correctness fixes for the M6 real-dispatch path + one workload-sizing refinement, plus a startup-latency fix discovered during user testing.
+
+### Highlights
+
+- **M6 coalesced micro-tasks (real dispatch, MVP)**: 3-task smoke (`test-coalesce-static`) now produces real output files, populates `task.resultArtifact`, and writes heartbeats for all N group tasks so the background watcher doesn't emit false-positive `heartbeat_dead` alarms.
+- **Closeout `run.lock` race fixed**: `executeTeamRun`'s catch path no longer re-acquires the run lock. The previous behavior caused `run.completed` to fire, then `run.failed` ~500ms later with `Run 'run.lock' is locked by another operation`. Symptom: spurious failed runs.
+- **PI-CREW bundle path bug fixed (user-reported)**: `PROMPT_RUNTIME_EXTENSION_PATH` was computed via `import.meta.url-relative` resolution. In source it landed correctly at `<pi-crew>/src/prompt/prompt-runtime.ts`, but in the bundle esbuild's `__esm` helper does NOT preserve per-module `import.meta.url` — every module's `import.meta.url` resolves to the bundle's URL, so `dirname + "../"` landed at `<pi-crew>/` (missing `src/`). Replaced with `path.join(packageRoot(), 'src', 'prompt', 'prompt-runtime.ts')`. `packageRoot()` heuristically walks up to find pi-crew's `package.json` and works from both `src/` and `dist/` entries.
+- **Adaptive group sizing** (M6 deferred #4): `planCoalescedGroups` now also caps by combined estimated prompt bytes (default 100KB heuristic) on top of the existing count cap. Prevents context-budget overflow on workflows with very long per-task instructions.
+
+### Behavior changes
+
+- **`coalesceMicroTasks` workflow frontmatter flag** (opt-in, default off) — now parsed and honored. See `.crew/workflows/test-coalesce-static.workflow.md` for usage.
+- M6 dispatch now writes `task.resultArtifact` for all group tasks (single-task path was already doing this).
+- M6 dispatch now writes heartbeats for all group tasks (15s interval while worker is in flight; cleared on completion). Heartbeat alert coercion — first-time-only emission per `(run, task)` — preserved.
+- Catch path uses in-memory state instead of `withRunLock`-reloaded disk state. Trade-off documented: final-state inconsistency on partial closeout is preferable to spurious "run.lock locked" failures.
+
+### Verification
+
+| Gate | Result |
+|---|---|
+| TSC | ✅ |
+| Lint | ✅ |
+| format:check | ✅ |
+| check:conflict-markers | ✅ |
+| check:lazy-imports | ✅ |
+| check:bundle-staleness | ✅ |
+| Unit tests (npm test) | ✅ 30+ suites, all green on CI (Ubuntu/macOS/Windows) |
+| Live E2E (test-coalesce-static) | ✅ 3/3 tasks with real output files + heartbeats on disk |
+
+### Changed
+
+- `src/runtime/pi-args.ts` — `PROMPT_RUNTIME_EXTENSION_PATH` uses `packageRoot()` heuristic instead of `import.meta.url-relative` path.
+- `src/runtime/coalesce-tasks.ts` — `planCoalescedGroups` adds `maxGroupBytes` cap; greedy byte-budget chunking alongside the existing count cap.
+- `src/runtime/run-coalesced-task-group.ts` — writes heartbeats for all N group tasks at dispatch and on a 15s interval while the worker is in flight (fixes false-positive `heartbeat_dead`).
+- `src/runtime/team-runner.ts` — catch path no longer acquires `withRunLock`; uses in-memory state directly. Eliminates `run.lock` race against the closeout path.
+- `src/workflow/...` — `coalesceMicroTasks` frontmatter field parsed from workflow YAMLs.
+- `dist/index.mjs` — rebuilt; +1KB net (expanded lock-acquisition rationale comment + heartbeat init).
+
+### Migration notes
+
+- Workflow authors can now opt into micro-task coalescing via `coalesceMicroTasks: true` in workflow frontmatter. Default `false` — no behavior change for existing workflows.
+- No public-API breaking changes. No new required configuration.
+- Users on the strip-types fallback (`PI_CREW_USE_BUNDLE=0`) should rebuild the bundle once after upgrade if they want the fix: `npm run build:bundle`.
+
+## [Unreleased] — bundle as default entry
+
+The bundled entrypoint (`dist/index.mjs`, 2.9MB single file) is now the **default** load path. Strip-types loading remains as a fallback when `dist/` is missing (e.g. dev clone without `npm run build:bundle`) or when `PI_CREW_USE_BUNDLE=0` is set explicitly.
+
+### Why default (vs opt-in)
+
+Benchmarked with `scripts/bench-cold-start.mjs` (20 iters, fresh node processes each, **after the atomic-write fsync fix in `13f4490`**):
+
+| Metric | Strip-types (p50) | Bundle (p50) | Delta |
+|---|---|---|---|
+| Total cold-start | 2509.4ms | 1716.6ms | **−31.6%** (bundle faster) |
+| Import (`await import`) | 2322.1ms | 1543.1ms | **−33.5%** |
+| Register (`registerPiTeams(mockPi)`) | 204.2ms | 157.8ms | **−22.7%** (bundle faster) |
+
+After fsync, bundle is ~32% faster total cold-start. Bundle-flip risk is now bounded by a comprehensive safety net: `check-bundle-staleness` (CI gate, fails if src/ is newer than dist/), `check-conflict-markers` (CI gate, blocks bad merges), graceful fallback (silent if env unset, loud warning if PI_CREW_USE_BUNDLE=1 and bundle missing), and a separate bundle entry (`index.bundle.ts`) to avoid chicken-and-egg recursion. The cost (postinstall + ~9MB npm package) is paid automatically and silently for users who install via npm.
+
+### Env var semantics
+
+- unset / empty → use bundle if present, else strip-types
+- `PI_CREW_USE_BUNDLE=0` / `false` / `no` / `off` → force strip-types, ignore bundle
+- `PI_CREW_USE_BUNDLE=1` / `true` / `yes` / `on` → force bundle (same as unset when bundle is present; loud warning if bundle is missing)
+
+### Changed
+
+- `index.ts` — entrypoint now defaults to bundle, with strip-types as fallback when bundle is missing OR `PI_CREW_USE_BUNDLE=0`. Replaces the previous opt-in model (`ae01851`).
+- `index.bundle.ts` (new) — minimal bundle entry. Bundled by `scripts/build-bundle.mjs` into `dist/index.mjs`. Separate from `index.ts` to avoid chicken-and-egg recursion (the bundle's `import.meta.url` is inside `dist/`, so building from `index.ts` would resolve `dist/index.mjs` relative to itself and try to load `dist/dist/index.mjs`).
+- `scripts/build-bundle.mjs` — now bundles from `index.bundle.ts` (not `index.ts`).
+- `package.json` — `postinstall` script retained (`build:bundle || warn`) so users get a working bundle out of the box. `files` includes `dist/` so opt-in/opt-out users get a working bundle after `npm install`.
+- `scripts/check-bundle-staleness.mjs` (new) — CI gate that fails the build if any `src/*.ts` is newer than `dist/index.mjs`. Wired into `npm run ci`. No-op when `dist/` is absent.
+- `scripts/check-conflict-markers.mjs` (new) — CI gate that scans tracked source for unresolved git merge markers (`<<<<<<<`, `=======`, `>>>>>>>`, `|||||||`). Excludes `test/unit/conflict-detect.test.ts` (intentional fixtures). Prevents recurrence of CI #28498831579 where unresolved stash/pop markers reached the bundle.
+- `src/state/atomic-write.ts` — `atomicWriteFile` now calls `fs.fsyncSync(fd)` before close, and `fs.fsyncSync(dirFd)` on the parent directory after rename (Linux only). Fixes CI #28498831579 mailbox-replay flake on Ubuntu (page cache eviction under I/O pressure could cause read-after-write to see stale content).
+- `src/utils/paths.ts` — `packageRoot()` now uses a heuristic walk to find the directory containing pi-crew's `package.json`. The original 2-level walk only worked for src/ paths; called from the bundle (1-level deep) it resolved to the wrong directory and broke team discovery.
+- `scripts/bench-cold-start.mjs` (new) — reproducible cold-start benchmark. Run with `node scripts/bench-cold-start.mjs --iters 20`.
+- `scripts/test-runner.mjs` — test step timeout bumped 600s → 900s to give Windows headroom for the fsync overhead (cost: ~5-10ms per atomic write across 5800+ tests).
+
 ## [v0.9.16] — cancel wipes the trace (2026-06-29)
 
 User-driven refinement of the cancellation policy. Previously, cancelled / stopped subagents and crew-agent records were preserved on disk (`agent_*.json` files in `.crew/state/subagents/`, plus `agents.json` + per-task `status.json` in the run state directory) and continued to appear in the UI dashboard / widget / transcript viewer. The user explicitly asked: *"một khi đã cancel thì phải xóa luôn dấu vết về subagent đó chứ để lại làm gì nữa"* — once cancelled, the trace must be wiped. This release implements that policy.

package/README.md

@@ -610,6 +610,7 @@ Your system prompt here.
 
 | Variable | Purpose |
 |----------|---------|
+| `PI_CREW_USE_BUNDLE=1` | Load via bundled `dist/index.mjs` (~5% faster cold-start). Default: strip-types. Requires `npm run build:bundle` to have produced `dist/`. Falls back to strip-types with a one-time warning if the bundle is missing. See `scripts/bench-cold-start.mjs`. |
 | `PI_CREW_EXECUTE_WORKERS=0` | Disable child workers (scaffold mode) |
 | `PI_TEAMS_EXECUTE_WORKERS=0` | Legacy disable flag |
 | `PI_TEAMS_MOCK_CHILD_PI=success` | Mock child worker for testing |
@@ -620,6 +621,24 @@ Your system prompt here.
 
 ## Development
 
+### Auto-rebuild bundle on edit
+
+For active development on the bundle, run the watcher in a separate terminal:
+
+```bash
+npm run watch:bundle
+```
+
+This watches `src/` + `index.bundle.ts` and rebuilds `dist/index.mjs` after a 300ms debounce. Zero added dependencies (uses native `node:fs.watch` with per-directory watchers).
+
+```bash
+npm run watch:bundle             # watch + auto-rebuild
+npm run watch:bundle --debounce 500  # custom debounce
+node scripts/watch-bundle.mjs --once  # build once and exit (CI prep)
+```
+
+The watcher is the dev-loop companion to `check:bundle-staleness` (CI gate) — together they eliminate the "edit src/foo.ts, forget to rebuild, run stale bundle" failure mode.
+
 ```bash
 cd pi-crew
 npm install          # dependencies