pi-crew 0.9.15 → 0.9.17

package/CHANGELOG.md

@@ -1,5 +1,150 @@
 # Changelog
 
+## [v0.9.17] — coalesced micro-tasks (M6) ships + closeout race fix (2026-07-02)
+
+Two end-to-end correctness fixes for the M6 real-dispatch path + one workload-sizing refinement, plus a startup-latency fix discovered during user testing.
+
+### Highlights
+
+- **M6 coalesced micro-tasks (real dispatch, MVP)**: 3-task smoke (`test-coalesce-static`) now produces real output files, populates `task.resultArtifact`, and writes heartbeats for all N group tasks so the background watcher doesn't emit false-positive `heartbeat_dead` alarms.
+- **Closeout `run.lock` race fixed**: `executeTeamRun`'s catch path no longer re-acquires the run lock. The previous behavior caused `run.completed` to fire, then `run.failed` ~500ms later with `Run 'run.lock' is locked by another operation`. Symptom: spurious failed runs.
+- **PI-CREW bundle path bug fixed (user-reported)**: `PROMPT_RUNTIME_EXTENSION_PATH` was computed via `import.meta.url-relative` resolution. In source it landed correctly at `<pi-crew>/src/prompt/prompt-runtime.ts`, but in the bundle esbuild's `__esm` helper does NOT preserve per-module `import.meta.url` — every module's `import.meta.url` resolves to the bundle's URL, so `dirname + "../"` landed at `<pi-crew>/` (missing `src/`). Replaced with `path.join(packageRoot(), 'src', 'prompt', 'prompt-runtime.ts')`. `packageRoot()` heuristically walks up to find pi-crew's `package.json` and works from both `src/` and `dist/` entries.
+- **Adaptive group sizing** (M6 deferred #4): `planCoalescedGroups` now also caps by combined estimated prompt bytes (default 100KB heuristic) on top of the existing count cap. Prevents context-budget overflow on workflows with very long per-task instructions.
+
+### Behavior changes
+
+- **`coalesceMicroTasks` workflow frontmatter flag** (opt-in, default off) — now parsed and honored. See `.crew/workflows/test-coalesce-static.workflow.md` for usage.
+- M6 dispatch now writes `task.resultArtifact` for all group tasks (single-task path was already doing this).
+- M6 dispatch now writes heartbeats for all group tasks (15s interval while worker is in flight; cleared on completion). Heartbeat alert coercion — first-time-only emission per `(run, task)` — preserved.
+- Catch path uses in-memory state instead of `withRunLock`-reloaded disk state. Trade-off documented: final-state inconsistency on partial closeout is preferable to spurious "run.lock locked" failures.
+
+### Verification
+
+| Gate | Result |
+|---|---|
+| TSC | ✅ |
+| Lint | ✅ |
+| format:check | ✅ |
+| check:conflict-markers | ✅ |
+| check:lazy-imports | ✅ |
+| check:bundle-staleness | ✅ |
+| Unit tests (npm test) | ✅ 30+ suites, all green on CI (Ubuntu/macOS/Windows) |
+| Live E2E (test-coalesce-static) | ✅ 3/3 tasks with real output files + heartbeats on disk |
+
+### Changed
+
+- `src/runtime/pi-args.ts` — `PROMPT_RUNTIME_EXTENSION_PATH` uses `packageRoot()` heuristic instead of `import.meta.url-relative` path.
+- `src/runtime/coalesce-tasks.ts` — `planCoalescedGroups` adds `maxGroupBytes` cap; greedy byte-budget chunking alongside the existing count cap.
+- `src/runtime/run-coalesced-task-group.ts` — writes heartbeats for all N group tasks at dispatch and on a 15s interval while the worker is in flight (fixes false-positive `heartbeat_dead`).
+- `src/runtime/team-runner.ts` — catch path no longer acquires `withRunLock`; uses in-memory state directly. Eliminates `run.lock` race against the closeout path.
+- `src/workflow/...` — `coalesceMicroTasks` frontmatter field parsed from workflow YAMLs.
+- `dist/index.mjs` — rebuilt; +1KB net (expanded lock-acquisition rationale comment + heartbeat init).
+
+### Migration notes
+
+- Workflow authors can now opt into micro-task coalescing via `coalesceMicroTasks: true` in workflow frontmatter. Default `false` — no behavior change for existing workflows.
+- No public-API breaking changes. No new required configuration.
+- Users on the strip-types fallback (`PI_CREW_USE_BUNDLE=0`) should rebuild the bundle once after upgrade if they want the fix: `npm run build:bundle`.
+
+## [Unreleased] — bundle as default entry
+
+The bundled entrypoint (`dist/index.mjs`, 2.9MB single file) is now the **default** load path. Strip-types loading remains as a fallback when `dist/` is missing (e.g. dev clone without `npm run build:bundle`) or when `PI_CREW_USE_BUNDLE=0` is set explicitly.
+
+### Why default (vs opt-in)
+
+Benchmarked with `scripts/bench-cold-start.mjs` (20 iters, fresh node processes each, **after the atomic-write fsync fix in `13f4490`**):
+
+| Metric | Strip-types (p50) | Bundle (p50) | Delta |
+|---|---|---|---|
+| Total cold-start | 2509.4ms | 1716.6ms | **−31.6%** (bundle faster) |
+| Import (`await import`) | 2322.1ms | 1543.1ms | **−33.5%** |
+| Register (`registerPiTeams(mockPi)`) | 204.2ms | 157.8ms | **−22.7%** (bundle faster) |
+
+After fsync, bundle is ~32% faster total cold-start. Bundle-flip risk is now bounded by a comprehensive safety net: `check-bundle-staleness` (CI gate, fails if src/ is newer than dist/), `check-conflict-markers` (CI gate, blocks bad merges), graceful fallback (silent if env unset, loud warning if PI_CREW_USE_BUNDLE=1 and bundle missing), and a separate bundle entry (`index.bundle.ts`) to avoid chicken-and-egg recursion. The cost (postinstall + ~9MB npm package) is paid automatically and silently for users who install via npm.
+
+### Env var semantics
+
+- unset / empty → use bundle if present, else strip-types
+- `PI_CREW_USE_BUNDLE=0` / `false` / `no` / `off` → force strip-types, ignore bundle
+- `PI_CREW_USE_BUNDLE=1` / `true` / `yes` / `on` → force bundle (same as unset when bundle is present; loud warning if bundle is missing)
+
+### Changed
+
+- `index.ts` — entrypoint now defaults to bundle, with strip-types as fallback when bundle is missing OR `PI_CREW_USE_BUNDLE=0`. Replaces the previous opt-in model (`ae01851`).
+- `index.bundle.ts` (new) — minimal bundle entry. Bundled by `scripts/build-bundle.mjs` into `dist/index.mjs`. Separate from `index.ts` to avoid chicken-and-egg recursion (the bundle's `import.meta.url` is inside `dist/`, so building from `index.ts` would resolve `dist/index.mjs` relative to itself and try to load `dist/dist/index.mjs`).
+- `scripts/build-bundle.mjs` — now bundles from `index.bundle.ts` (not `index.ts`).
+- `package.json` — `postinstall` script retained (`build:bundle || warn`) so users get a working bundle out of the box. `files` includes `dist/` so opt-in/opt-out users get a working bundle after `npm install`.
+- `scripts/check-bundle-staleness.mjs` (new) — CI gate that fails the build if any `src/*.ts` is newer than `dist/index.mjs`. Wired into `npm run ci`. No-op when `dist/` is absent.
+- `scripts/check-conflict-markers.mjs` (new) — CI gate that scans tracked source for unresolved git merge markers (`<<<<<<<`, `=======`, `>>>>>>>`, `|||||||`). Excludes `test/unit/conflict-detect.test.ts` (intentional fixtures). Prevents recurrence of CI #28498831579 where unresolved stash/pop markers reached the bundle.
+- `src/state/atomic-write.ts` — `atomicWriteFile` now calls `fs.fsyncSync(fd)` before close, and `fs.fsyncSync(dirFd)` on the parent directory after rename (Linux only). Fixes CI #28498831579 mailbox-replay flake on Ubuntu (page cache eviction under I/O pressure could cause read-after-write to see stale content).
+- `src/utils/paths.ts` — `packageRoot()` now uses a heuristic walk to find the directory containing pi-crew's `package.json`. The original 2-level walk only worked for src/ paths; called from the bundle (1-level deep) it resolved to the wrong directory and broke team discovery.
+- `scripts/bench-cold-start.mjs` (new) — reproducible cold-start benchmark. Run with `node scripts/bench-cold-start.mjs --iters 20`.
+- `scripts/test-runner.mjs` — test step timeout bumped 600s → 900s to give Windows headroom for the fsync overhead (cost: ~5-10ms per atomic write across 5800+ tests).
+
+## [v0.9.16] — cancel wipes the trace (2026-06-29)
+
+User-driven refinement of the cancellation policy. Previously, cancelled / stopped subagents and crew-agent records were preserved on disk (`agent_*.json` files in `.crew/state/subagents/`, plus `agents.json` + per-task `status.json` in the run state directory) and continued to appear in the UI dashboard / widget / transcript viewer. The user explicitly asked: *"một khi đã cancel thì phải xóa luôn dấu vết về subagent đó chứ để lại làm gì nữa"* — once cancelled, the trace must be wiped. This release implements that policy.
+
+### Changed — terminal-status deletion
+
+- **Subagent records** (`.crew/state/subagents/<id>.json`) — added `removePersistedSubagentRecord(cwd, id)` in `src/runtime/subagent-manager.ts`. On terminal status `cancelled` / `stopped` / `terminated:true`, the persisted file is **deleted immediately** (no linger). The file is saved first so any concurrent reader sees the final status, then `unlink`'d within microseconds — the next widget/dashboard tick sees no entry.
+
+- **Crew-agent records** (per-run `agents.json` index + per-task `status.json`) — added `removeCrewAgent(manifest, taskId)` in `src/runtime/crew-agent-records.ts`. `upsertCrewAgent()` now checks `shouldDeleteCrewAgentOnTerminalStatus(record)` and dispatches to `removeCrewAgent` instead of `saveCrewAgents` for `cancelled` / `stopped` records. Both the index entry AND the per-task status.json are wiped.
+
+### What is preserved vs wiped
+
+| Status | Persisted? | Rationale |
+|---|---|---|
+| `completed` | ✅ yes | Successful work — audit value |
+| `failed` | ✅ yes | Agent errored — debugging value |
+| `error` | ✅ yes | Unexpected exception — debugging value |
+| `cancelled` | ❌ **wiped** | User-initiated cancellation — no trace |
+| `stopped` | ❌ **wiped** | External abort — no trace |
+| `terminated: true` (any status) | ❌ **wiped** | Any termination event — no trace |
+
+### In-memory handles
+
+`terminateLiveAgent` in `src/runtime/live-agent-manager.ts:169` already deletes from the `liveAgents` Map immediately after abort — no change needed. The `evictStaleLiveAgentHandles` fallback (10-min linger for orphan terminal handles) remains as a safety net.
+
+### New helper predicates
+
+- `shouldDeleteOnTerminalStatus(record)` — for `.crew/state/subagents/` records
+- `shouldDeleteCrewAgentOnTerminalStatus(record)` — for per-run `agents.json` records
+
+Both exported for downstream callers and tests.
+
+### Tests
+
+- **NEW `test/unit/cancellation-trace-wipe.test.ts`** — 15 cases covering:
+  - `shouldDeleteOnTerminalStatus` / `shouldDeleteCrewAgentOnTerminalStatus` predicates for all 6 statuses
+  - `removePersistedSubagentRecord` happy path + ENOENT safe-fail
+  - `removeCrewAgent` removes from `agents.json` index AND per-task `status.json`
+  - `upsertCrewAgent` integration: cancelled → wiped, completed/failed → preserved
+
+- **Existing tests still pass** — 118/118 across 9 test files (subagent-manager-cov, subagent-tools-integration, live-manager-cov, direct-agent-run, team-runner-merge, process-lifecycle, preflight-validator, topology-analyzer, cancellation-trace-wipe). 0 regressions.
+
+### Verification
+
+- `tsc --noEmit` → EXIT 0
+- `node scripts/check-lazy-imports.mjs` → "All dynamic imports have `// LAZY:` marker"
+- `npm test` (focused 9-file suite) → 118/118 pass
+
+### Honesty discipline
+
+- **Failed subagents are NOT wiped** — they may carry useful debugging info (error stack, partial output). Different from cancel. If the user later wants failed→wipe too, it's a 1-line predicate change.
+- **No tombstone** — the cancellation event is in `events.jsonl` (the run's audit log) and `notifications/`. Disk-level agent files are intentionally gone after cancel.
+- **Filesystem race window** is microseconds (save then unlink); under heavy concurrent access a reader might briefly see the final status before the file disappears. Acceptable trade-off for the "no trace" policy.
+
+### Files touched
+
+| File | Change |
+|---|---|
+| `src/runtime/subagent-manager.ts` | +2 functions (`removePersistedSubagentRecord`, `shouldDeleteOnTerminalStatus`); modified `markStopped` + `runSubagent` finally block to wipe on cancel |
+| `src/runtime/crew-agent-records.ts` | +2 functions (`removeCrewAgent`, `shouldDeleteCrewAgentOnTerminalStatus`); modified `upsertCrewAgent` to dispatch to remove on cancel |
+| `test/unit/cancellation-trace-wipe.test.ts` | NEW (15 cases, 8.3 KB) |
+| `CHANGELOG.md` | this entry |
+| `package.json` | 0.9.15 → 0.9.16 |
+
 ## [v0.9.15] — workflow topology advisory (2026-06-29)
 
 After a parallel-research assessment of v0.9.13 + v0.9.14 (`research-findings/pi-crew-performance-quality-assessment.md`, effectiveness evaluation at `.crew/research/pi-crew-effectiveness-evaluation.md`), a user-driven refinement showed that the original BLOCK-on-misuse design was too aggressive — **agents know their context better than the orchestrator**. This release ships an **advisory-only** topology classifier that prints measured-cost notes and proceeds either way. The agent (caller) decides.

package/README.md

@@ -610,6 +610,7 @@ Your system prompt here.
 
 | Variable | Purpose |
 |----------|---------|
+| `PI_CREW_USE_BUNDLE=1` | Load via bundled `dist/index.mjs` (~5% faster cold-start). Default: strip-types. Requires `npm run build:bundle` to have produced `dist/`. Falls back to strip-types with a one-time warning if the bundle is missing. See `scripts/bench-cold-start.mjs`. |
 | `PI_CREW_EXECUTE_WORKERS=0` | Disable child workers (scaffold mode) |
 | `PI_TEAMS_EXECUTE_WORKERS=0` | Legacy disable flag |
 | `PI_TEAMS_MOCK_CHILD_PI=success` | Mock child worker for testing |
@@ -620,6 +621,24 @@ Your system prompt here.
 
 ## Development
 
+### Auto-rebuild bundle on edit
+
+For active development on the bundle, run the watcher in a separate terminal:
+
+```bash
+npm run watch:bundle
+```
+
+This watches `src/` + `index.bundle.ts` and rebuilds `dist/index.mjs` after a 300ms debounce. Zero added dependencies (uses native `node:fs.watch` with per-directory watchers).
+
+```bash
+npm run watch:bundle             # watch + auto-rebuild
+npm run watch:bundle --debounce 500  # custom debounce
+node scripts/watch-bundle.mjs --once  # build once and exit (CI prep)
+```
+
+The watcher is the dev-loop companion to `check:bundle-staleness` (CI gate) — together they eliminate the "edit src/foo.ts, forget to rebuild, run stale bundle" failure mode.
+
 ```bash
 cd pi-crew
 npm install          # dependencies