pi-crew 0.9.55 → 0.9.57

package/AGENTS.md

@@ -10,7 +10,7 @@ Read in this order:
 2. `docs/HARNESS.md` for the human-agent collaboration model.
 3. `docs/FEATURE_INTAKE.md` before turning any request into work.
 4. `docs/product/` for current product contracts.
-5. `docs/ARCHITECTURE.md` for implementation shape.
+5. `docs/architecture.md` for implementation shape.
 6. `docs/stories/` for story packets and backlog.
 7. `docs/TEST_MATRIX.md` for proof status.
 8. `docs/decisions/` for why important choices were made.

package/CHANGELOG.md

@@ -3,6 +3,64 @@
 > **Note:** `atomic-write-v2.ts` / `AtomicWriter` mentioned in historical entries below was consolidated into `atomic-write.ts` as of v0.9.42. This changelog is preserved as historical record — the migration was completed (the v2 class was never adopted; v1 won on simplicity + symlink-safety + link+unlink atomicity). See `docs/migration/atomic-write-v2-migration.md` for the decision rationale.
 
 
+## [0.9.57] — provider-extension auto-discovery + team-tool schema repair + whole-project reorganization (2026-08-03)
+
+### New: provider extensions work in subagents (no hardcoding)
+- pi-crew spawns child-pi workers with `--no-extensions` (security posture from SEC-1), which made extension-registered providers (e.g. `pi-commandcode-provider`) unresolvable in subagents → their models fell back to a secondary provider (429 rate-limit deaths, 0 tool uses).
+- **Auto-discovery**: pi-crew now reads `~/.pi/agent/settings.json` `packages` (npm: specs only), resolves each to its extension entry point (`pi.extensions` → `index.ts` → `index.mjs` → `index.js` → `src/index.ts`), and merges the discovered paths into every builtin/user agent (`src/runtime/model/provider-extensions.ts` + `src/agents/discover-agents.ts`).
+- Result: **any provider package installed via `pi install npm:<pkg>` works automatically in subagents** — no manual per-extension config.
+- Optional explicit allowlist `runtime.agentExtensions: string[]` config added on top of auto-discovery.
+- SEC-1 preserved: project/project-pi agents never receive these extensions.
+- Verified end-to-end: `deepseek/deepseek-v4-flash` resolves to `commandcode/deepseek/deepseek-v4-flash` in a real subagent spawn and completes.
+
+### Bug fixes (team tool was broken live while tests stayed green)
+- **"Validation failed for tool team"**: pi-ai `validateToolArguments` validates tool args BEFORE the pi-crew handler runs, and rejected model-emitted empty-string defaults (`runId:""`, `workspaceMode:""`, `context:""`, `scope:""`, `budgetTotal:0`, `once:false`, `keep:false`) against Literal unions / patterns / minimums. Schema now accepts the unset markers (`Literal("")`, runId pattern `^$|^...`, `Literal(0)`, `Boolean` in unions); `allActionLiterals` filters the `""`.
+- **"Unknown type"**: `SkillOverride`/`FreeformConfig` used `Type.Unsafe({...})` WITHOUT a `[TypeBox.Kind]` symbol, so `Value.Check` threw "Unknown type" the first time a model sent `skill`/`config`. Switched to TypeBox-native `Type.Union`/`Type.Record`. Handler-side `normalizeTeamParams` drops empty strings before validation (defense-in-depth).
+- **chain-runner mis-split on arrows inside inline goals**: `parseChain` used `split("->")`, incorrectly splitting `"Change A -> B" -> "Analyze"` into 3 steps. New quote-aware `splitChainSteps()` splits only on top-level arrows (backward compatible).
+- These were caught by a new Tier 9 feature battery in the `real-test-pi-crew` skill (live team-tool action coverage), which is now documented there alongside the agent-unauthorized-edit anti-pattern surfaced during testing.
+
+### Whole-project reorganization (~92 commits)
+- `src/` clustered: `src/runtime/` → 15 subdirs (child-pi/, broker/, live-session/, recovery/, scheduling/, verification/, model/, output/, heartbeat/, process/, goal-workflow/, compaction/, task-runner/, custom-tools/, errors/), `src/state/` → 3 subdirs (event-log/, stores/, coordination/), plus ui/, config/, utils/, agents/, extension/ clusters. Cluster maps in `src/runtime/README.md`, `src/state/README.md`.
+- `test/unit/` mirrored: 561 of 712 flat `.test.ts` files moved into 35 subdirs matching `src/` (243 runtime/, 53 state/, 110 extension/, plus ui/, config/, utils/, etc.). 151 cross-cutting tests (round*, v0*, package-*, errors, i18n, bundle-*) stay at root by design. Map in `test/unit/README.md`.
+- **REAL bug fixed**: test runner now expands `**` globs — `test/unit/security/` (3 files, 36 tests) were invisible before. All 715 files now discovered (6732 tests, 0 fail).
+- `docs/` consolidated: 27 historical docs → `docs/archive/`; 20 living docs stay at root; new `docs/README.md` index; 5 categories of broken refs fixed (dead links in src/agents/discover-agents.ts, src/runtime/{chain-runner,anchor-manager,handoff-manager,auto-summarize}.ts, recovery/retry-runner.ts, SECURITY-ISSUES.md, ARCHITECTURE.md refs).
+- Root cleanup: 11 stray `.md` files at repo root → `docs/archive/` + `docs/bugs/`.
+- README updated: Repository layout section, provider-extension feature, `runtime.agentExtensions` config row, v0.9.57 Recent-changes entry, 3 broken doc links fixed.
+- Stale worktrees + 4 merged local branches cleaned; `git worktree list` main-only.
+
+### Verification
+- Full final gate: typecheck 0 · lint 0 err/warn · format:check clean · unit 6763/6760/0 · integration 200/199/0 · `test:critical` 98/0 · bundle-load + smoke(real-binary) green · bundle 2.70 MB (budget 3.5 MB) · conflict-markers/lazy-imports/bundle-staleness all OK.
+- Live feature battery (Tier 9): `team` action=list/run(sync)/run(async)/chain + `Agent`/`crew_agent`/`get_subagent_result` all green with `deepseek/deepseek-v4-flash` over commandcode; read-only actions (recommend/health/doctor/status/events/summary/get/explain/worktrees) all return clean.
+
+## [0.9.56] — performance remediation + team-tool schema fix (2026-07-30)
+
+Follows the verified `performance-audit-2026-07-29.md` (16 findings addressed). Independently deep-reviewed by a 4-agent review team (explore/code-review/security-review/verify — no P0/P1), full unit 6486/0 + integration 190/0, CI green on ubuntu/macos/windows.
+
+### Performance
+- **P0-1**: new `BoundedTail` segment ring replaces the O(n²) `appendBoundedTail` rebuild-per-line on stdout/stderr (5000 CJK lines: 112s → 26ms).
+- **P0-2**: per-eventsPath rotation counter — the async append path's `% 100` gate was always true (counter never incremented there) so `needsRotation` ran every event; now ticked on all three append paths.
+- **P0-3**: the per-batch hot-loop `saveCrewAgents` switched to `saveCrewAgentsCoalesced` (removes the `Atomics.wait`/`sleepSync` main-thread block); terminal writes flush coalesced first.
+- **P0-4**: async event append drops from 5 fsyncs to ~0 (non-terminal) / 1 (terminal) — best-effort pid/`.seq` + F3a-style terminal-only data fsync.
+- **P1-5**: memoize validated agent paths per task (~60 syscalls/event → ~0 after first).
+- **P1-6**: redaction pre-filter `mayContainSecret` skips the ~14 redaction passes on clean strings (no bypass — every secret type still redacted).
+- **P1-7**: scheduler consults the global worker cap (no more over-dispatch on low-core machines).
+- **P1-8**: tail readers use `readJsonlTail`; `status.ts` O(events×messages) → O(events+messages).
+- **P1-9**: manifest cache-hit skips the redundant path re-validation.
+- **P1-10**: `before_task_start` hooks run via `Promise.all`.
+- **P1-11**: `git worktree prune` memoized per-repo (coalesces concurrent calls + once per process).
+- **P1-12**: `resolveRunStateRoot` memoized (containment check was ~8-12 syscalls/call).
+- **P2-21**: drop `dist/build-meta.json` from the npm tarball (−470KB).
+- **P2-25**: `supervisor_contact` events now reach `recordSupervisorContact` (was fully dead — the compact pipeline stripped the payload and `onStdoutLine` received display prose).
+- **H2**: archive retention sweep (`.archive.jsonl` no longer accumulate forever).
+- **H5**: `tasks.find` O(N²) → `Map` at all scheduler sites.
+
+### Platform fix
+- **team-tool schema flatten**: `TeamToolParams` was `Type.Union([5 domain objects])`, emitting a giant `anyOf` that LLM tool-callers could not reliably satisfy → empty/malformed calls → the tool silently defaulted to `list`. Flattened to a single `Type.Object` (all 54 actions + shared fields optional); the handler still validates per-action fields at runtime (defense-in-depth, unchanged).
+
+### Review follow-ups
+- Tighten the `"gh"` redaction marker to `gh[pousr]_` (reduces false-positives like "though"/"highlight").
+- Test guard: `allActionLiterals.length === 54` (schema-derivation regression guard).
+
 ## [0.9.55] — HOTFIX: ship src/ (child workers load src/prompt/prompt-runtime.ts by path) (2026-07-29)
 
 **Fixes the delegation / team-run regression** that affected 0.9.52–0.9.54. The extension loaded fine (0.9.54), but delegation/team-run failed with "pi-crew run failed" — the child worker `pi` process could not find its extension.

package/README.md

@@ -26,7 +26,7 @@
 >   a footgun, or a sharp edge, please open an issue or send a note — your
 >   feedback is genuinely appreciated. Thanks. ✌️
 >
-> See also: [SECURITY-ISSUES.md](SECURITY-ISSUES.md),
+> See also: [SECURITY-ISSUES.md](docs/bugs/SECURITY-ISSUES.md),
 > [docs/dynamic-workflows.md](docs/dynamic-workflows.md#security-model-important)
 > (trust model), and the [Known limitations](#known-limitations) section below.
 
@@ -70,6 +70,7 @@ repo: https://github.com/baphuongna/pi-crew
 - **Lossless-by-default output handling** (L4, v0.9.8) — worker output thresholds sized from measured data (100% of real outputs fit without compaction); when compaction is unavoidable it keeps head+tail (preserves closing code fences/headings) instead of head-only truncation. No more `[pi-crew compacted N chars]` markers eating the end of a worker's result.
 - **Inter-pi broker** (v0.9.47, default-on) — a Unix-domain-socket message bus that lets concurrently-running Pi sessions pass messages, steering notes, and task-status events to each other. **On by default** on Linux + macOS; auto-disabled on native Windows (no unix socket). Three independent kill switches: `broker.enabled: false` (config), `PI_CREW_BROKER=0` (env, always wins), Windows auto-disable. See [docs/decisions/2026-07-22-broker-phase4-gated-on.md](docs/decisions/2026-07-22-broker-phase4-gated-on.md).
 - **`test:critical` + `real-test-pi-crew` skill** (v0.9.47) — a curated 14-file / 97-test subset (`npm run test:critical`, ~20s) for fast in-loop verification, plus a bundled skill distilling the full 8-tier end-to-end verification discipline (unit → 3-path kill-switch proof → typecheck/bundle → live TUI probing → smoke team run). Prevents the verifier-worker hang that full `npm test` (>4 min) caused against the 300s worker timeout.
+- **Provider extensions in subagents** (v0.9.57) — pi-crew spawns child-pi workers with `--no-extensions` (security posture), which made extension-registered providers (e.g. `pi-commandcode-provider`) unresolvable inside subagents. pi-crew now **auto-discovers provider packages** from `~/.pi/agent/settings.json` `packages` (npm: specs) and loads them via `--extension` in every builtin/user subagent — so **all provider models work in subagents**. An explicit `runtime.agentExtensions: string[]` config is an optional extra allowlist on top of auto-discovery. **SEC-1 preserved:** project/project-pi agents never receive these (env-gate unchanged).
 
 ---
 
@@ -286,6 +287,16 @@ The advisory is **informational only** — there is no `force:true` flag needed
 
 ## Recent changes
 
+### v0.9.57: team-tool schema repair + provider-extension auto-discovery + post-reorg repo-layout consolidation
+
+- **Team tool repaired (was broken live while tests stayed green)**: calling models emit empty-string/boolean defaults for every schema key, which pi-ai's pre-handler `validateToolArguments` rejected (`Validation failed for tool team`) and `Type.Unsafe` schema fields without `[TypeBox.Kind]` made `Value.Check` throw (`Unknown type`). Schema now accepts unset markers natively; `SkillOverride`/`FreeformConfig` switched to TypeBox-native constructors; `normalizeTeamParams` drops empties in the handler. Chain-runner also fixed (quote-aware step splitting). Caught by a new **Tier 9 feature battery** in the [`real-test-pi-crew`](skills/real-test-pi-crew/SKILL.md) skill (live team-tool action coverage).
+- **Source reorg finalised (~90 commits)**: `src/` is now cluster-organised — `src/runtime/` (15 subdirs + ~77 flat files) and `src/state/` (3 subdirs + 12 root files), with the other top-level dirs (`extension/`, `ui/`, `config/`, `utils/`, `agents/`, …) cluster-honed too. See [Repository layout](#repository-layout) and the cluster maps [`src/runtime/README.md`](src/runtime/README.md) / [`src/state/README.md`](src/state/README.md).
+- **Tests mirrored into subdirs**: 566 flat `.test.ts` files now live in 35 leaf subdirectories mirroring `src/` (`test/unit/runtime/`, `test/unit/state/`, `test/unit/extension/`, …). 151 cross-cutting tests (`round*`, `v0*`, `package-*`, errors, i18n, bundle-*) stay at `test/unit/` root by design. See [`test/unit/README.md`](test/unit/README.md).
+- **Recursive-glob test-runner fix**: `scripts/test-runner.mjs` now expands `**` globs itself — Node v22's `--test` only expands single-level `*`, so subdir tests (e.g. `test/unit/security/`) were **invisible** to `npm test` before.
+- **Docs consolidated**: 47 flat docs at `docs/` root → 20 living docs kept at root, 27 historical moved to `docs/archive/` (flat); 11 stray root `.md` files moved to `docs/archive/` + `docs/bugs/`. New [`docs/README.md`](docs/README.md) indexes the living docs + subdirs.
+- **Provider extensions in subagents** (see [Features](#features)): auto-discovery of provider packages from `~/.pi/agent/settings.json` `packages` (npm: specs) + explicit `runtime.agentExtensions` allowlist. Implementation: `src/runtime/model/provider-extensions.ts` + merge in `src/agents/discover-agents.ts`. SEC-1 preserved (project/project-pi agents never receive these).
+- See [CHANGELOG.md](CHANGELOG.md) for the full reorg + feature log.
+
 ### v0.9.47 (2026-07-22): Inter-pi broker Phase 4 (default-on) + verifier-hang fix + verification skill
 
 - **Broker default-on**: `broker.enabled` flipped `false` → `true` on Linux + macOS. The inter-pi broker lets concurrent Pi sessions exchange messages, steering notes, and task-status events over a Unix-domain socket. Three kill switches remain: config `broker.enabled: false`, env `PI_CREW_BROKER=0` (always wins), Windows auto-disable.
@@ -315,7 +326,7 @@ The advisory is **informational only** — there is no `force:true` flag needed
 
 ### v0.9.42 – v0.9.44: 4-wave audit + flaky-CI fixes
 
-A 4-wave audit + fix pass was completed (see `UPGRADE_REVIEW.md` for the full 647-line report and `CHANGELOG.md` for the diff):
+A 4-wave audit + fix pass was completed (see `docs/archive/UPGRADE_REVIEW.md` for the full 647-line report and `CHANGELOG.md` for the diff):
 
 - **−481 KB** bundle size (externalized `acorn` + fixed `@sinclair/typebox` name).
 - **−31.1%** in `child-pi.ts` (1842 → 1270 lines via 6 decomposition steps).
@@ -481,6 +492,7 @@ For machine-readable validation, see [schema.json](schema.json).
 | `runtime.effectivenessGuard` | `off \| warn \| block \| fail` | `off` | Pre-flight topology validator enforcement mode. |
 | `runtime.isolationPolicy` 🔒 | `{ isolatedRoles?, defaultRuntime? }` | _(unset)_ | Per-role runtime selection for crash isolation. |
 | `runtime.excludeContextBash` | `boolean` | `false` | Exclude certain bash results from worker context. |
+| `runtime.agentExtensions` | `string[]` | _(unset)_ | Extra extension paths (file paths or npm entry points) loaded in **every** child-pi worker on top of auto-discovered provider packages (see [Features](#features)). Optional allowlist — auto-discovery already loads provider packages from `~/.pi/agent/settings.json` `packages`. Project/project-pi agents never receive these (SEC-1). |
 
 ² `requirePlanApproval` is sensitive only when set to `false` in project config
 (the sanitizer blocks *disabling* the gate from untrusted project config).
@@ -893,7 +905,39 @@ npm run ci           # full CI-equivalent check
 npm pack --dry-run   # package verification
 ```
 
-Stats: **477 source files** (108K lines) · **682 test files** (~6,500 tests) · **CI: Ubuntu ✅ macOS ✅ Windows ✅**
+Stats: **476 source files** (112K lines) · **762 test files** (717 unit + 34 integration + 5 smoke + 3 functional + 2 platform + 1 manual) · **CI: Ubuntu ✅ macOS ✅ Windows ✅**
+
+---
+
+## Repository layout
+
+After the v0.9.x reorg (~90 commits) the repo is cluster-organised so related
+files live together — source, tests, and docs mirror each other.
+
+```
+src/                         # ~476 TS files
+├── runtime/                 # team-run execution machinery — 15 subdirs + ~77 flat files
+├── state/                   # persistent state layer — 3 subdirs + 12 root files
+├── extension/               # Pi extension surface (register, team-tool, ui glue)
+├── ui/                      # TUI, dashboard, overlays
+├── config/ · utils/ · agents/ · workflows/ · …   # supporting clusters
+test/
+└── unit/                    # mirrors src/ — 35 leaf subdirs + 151 cross-cutting tests at root
+    ├── runtime/ · state/ · extension/ · ui/ · config/ · …
+docs/                        # living docs at root; history in docs/archive/
+```
+
+- **`src/`** — each top-level dir has a cluster map; the two largest are
+  [`src/runtime/README.md`](src/runtime/README.md) (15 subdirs: child-pi,
+  broker, live-session, recovery, scheduling, verification, model, output,
+  heartbeat, process, goal-workflow, compaction, task-runner, custom-tools,
+  errors) and [`src/state/README.md`](src/state/README.md) (event-log,
+  stores, coordination).
+- **`test/unit/`** — mirrors `src/` 1:1; cross-cutting tests (`round*`,
+  `v0*`, `package-*`, errors, i18n, bundle-*) stay at the root by design.
+  See [`test/unit/README.md`](test/unit/README.md).
+- **`docs/`** — 20 living docs at the root; 27 historical docs moved to
+  `docs/archive/`. See [`docs/README.md`](docs/README.md).
 
 ---
 
@@ -912,7 +956,11 @@ Stats: **477 source files** (108K lines) · **682 test files** (~6,500 tests) ·
 | [docs/dynamic-workflows.md](docs/dynamic-workflows.md) | **v0.9.0** `.dwf.ts` script runtime + trust model |
 | [docs/live-mailbox-runtime.md](docs/live-mailbox-runtime.md) | Mailbox + live-session runtime |
 | [docs/publishing.md](docs/publishing.md) | Release & publish process |
-| [docs/next-upgrade-roadmap.md](docs/next-upgrade-roadmap.md) | Future upgrade roadmap |
+| [docs/README.md](docs/README.md) | Index of all docs/ (living + subdirs) |
+| [src/runtime/README.md](src/runtime/README.md) | Runtime source cluster map (15 subdirs) |
+| [src/state/README.md](src/state/README.md) | State source cluster map (3 subdirs) |
+| [test/unit/README.md](test/unit/README.md) | Unit-test cluster map (mirrors src/) |
+| [docs/archive/next-upgrade-roadmap.md](docs/archive/next-upgrade-roadmap.md) | Future upgrade roadmap |
 | [schema.json](schema.json) | Config JSON schema |
 
 Research docs (not in package): [`docs/pi-crew-research/`](https://github.com/baphuongna/pi-crew/tree/main/docs) — audits, deep research, distillation notes.
@@ -929,7 +977,7 @@ sharp edges I'm aware of — there are almost certainly others I'm not.
   event-loop race that kills the background process with no signal, no core,
   and no V8 diagnostic report (8 investigation attempts: gdb, strace, perf,
   `--report-on-fatalerror`, sync-fs workarounds, worker-thread atomic writer —
-  see `research-findings/goal-workflow/17-PHASE1.5-CRASH-INVESTIGATION-RFC.md`).
+  see [CHANGELOG.md](CHANGELOG.md) § v0.9.0 "Phase 1.5 #3").
   **Mitigation:** multi-step workflows silently auto-downgrade to a normal
   team-run (no goal-wrap layer); single-step workflows (`implementation`)
   goal-wrap end-to-end.