pi-crew 0.2.3 → 0.2.5

package/docs/pi-subagents3-deep-analysis.md

@@ -0,0 +1,508 @@
+# pi-subagents3 Deep Analysis — Patterns for pi-crew In-Process Runtime
+
+## Executive Summary
+
+After deep reading of `source/pi-subagents3/`, this document catalogs every production-ready pattern that pi-crew should adopt for its in-process (live-session) runtime. pi-subagents3 is a mature single-agent system with many features pi-crew's team orchestration currently lacks.
+
+---
+
+## 1. Promise-Based Agent Lifecycle ✅ DONE
+
+### pi-subagents3 Pattern
+```typescript
+class AgentManager {
+    spawn(...) {
+        const record = { ... };
+        record.promise = runAgent(...).then(...);
+        return id;
+    }
+    async spawnAndWait(...) {
+        const id = this.spawn(...);
+        const record = this.agents.get(id)!;
+        await record.promise;  // ← Await actual completion
+        return record;
+    }
+    async waitForAll() {
+        while (true) {
+            const pending = [...this.agents.values()]
+                .filter(r => r.status === "running" || r.status === "queued")
+                .map(r => r.promise);
+            if (pending.length === 0) break;
+            await Promise.allSettled(pending);
+        }
+    }
+}
+```
+
+### pi-crew Implementation
+**Status:** Done in `src/runtime/run-tracker.ts` (`a88e552`)
+- `registerRunPromise()`, `resolveRunPromise()`, `waitForRun()`
+- Fast path (disk terminal), medium path (foreground Promise), fallback (exponential backoff poll)
+
+---
+
+## 2. Soft Turn Limit + Graceful Steering ⬜ NOT IN PI-CREW
+
+### pi-subagents3 Pattern
+```typescript
+let turnCount = 0;
+const maxTurns = normalizeMaxTurns(options.maxTurns ?? agentConfig?.maxTurns ?? defaultMaxTurns);
+let softLimitReached = false;
+let aborted = false;
+
+session.subscribe((event) => {
+    if (event.type === "turn_end") {
+        turnCount++;
+        if (maxTurns != null) {
+            if (!softLimitReached && turnCount >= maxTurns) {
+                softLimitReached = true;
+                session.steer("You have reached your turn limit. Wrap up immediately — provide your final answer now.");
+            } else if (softLimitReached && turnCount >= maxTurns + graceTurns) {
+                aborted = true;
+                session.abort();
+            }
+        }
+    }
+});
+```
+
+**Key insight:** Instead of hard cutoff, it steers the agent to wrap up. Only aborts after `graceTurns` (default 5) additional turns. This produces much better output than sudden termination.
+
+**Settings:** `defaultMaxTurns`, `graceTurns` — persisted per project.
+
+### pi-crew Gap
+- `maxTurns` exists but no soft-limit steering mechanism
+- No `graceTurns` concept
+- Hard abort at maxTurns causes incomplete responses
+
+### Implementation Sketch
+Add to `live-session-runtime.ts`:
+```typescript
+const turnCount = 0;
+const maxTurns = agent.maxTurns ?? config.defaultMaxTurns;
+const graceTurns = config.graceTurns ?? 5;
+let softLimitReached = false;
+
+session.subscribe((event) => {
+    if (event.type === "turn_end") {
+        turnCount++;
+        if (maxTurns != null && !softLimitReached && turnCount >= maxTurns) {
+            softLimitReached = true;
+            session.steer("You have reached your turn limit. Wrap up immediately — provide your final answer now.");
+        } else if (softLimitReached && turnCount >= maxTurns + graceTurns) {
+            session.abort();
+        }
+    }
+});
+```
+
+---
+
+## 3. Persistent Agent Memory ⬜ NOT IN PI-CREW
+
+### pi-subagents3 Pattern
+```typescript
+// Memory scopes: "user" | "project" | "local"
+const memoryDir = resolveMemoryDir(agentName, scope, cwd);
+// Agent gets a persistent directory and MEMORY.md instructions
+// Agent can read/write/edit memory files using its tools
+```
+
+**Memory block injected into system prompt:**
+```markdown
+# Agent Memory
+You have a persistent memory directory at: {memoryDir}/
+Memory scope: {scope}
+This memory persists across sessions. Use it to build up knowledge over time.
+```
+
+**Features:**
+- MEMORY.md index file (max 200 lines)
+- Frontmatter format for structured memories
+- Read-only mode for agents without write/edit tools
+- Symlink attack prevention (`isSymlink`, `safeReadFile`)
+
+### pi-crew Gap
+- No persistent memory per agent across runs
+- Agents start fresh every time
+- No MEMORY.md concept
+
+---
+
+## 4. Context % Indicator ⬜ NOT IN PI-CREW
+
+### pi-subagents3 Pattern
+```typescript
+export function getSessionContextPercent(session: SessionLike | undefined): number | null {
+    if (!session) return null;
+    try { return session.getSessionStats().contextUsage?.percent ?? null; }
+    catch { return null; }
+}
+```
+
+**Used in:**
+- Dashboard widget showing "Context: 67%" to warn before compaction
+- Scheduling decisions (don't schedule if context is critically full)
+- UI streaming display
+
+### pi-crew Gap
+- No context usage percentage display
+- No early warning before compaction
+- Dashboard doesn't show "how full is the context window"
+
+---
+
+## 5. Skill Preloading (vs. Skill Path Passing) ⬜ PARTIAL IN PI-CREW
+
+### pi-subagents3 Pattern
+```typescript
+// Load skill content INTO the prompt instead of passing paths to child
+const loaded = preloadSkills(skills, effectiveCwd);
+if (loaded.length > 0) {
+    extras.skillBlocks = loaded;
+}
+
+// In prompt:
+// # Preloaded Skill: skill-name
+// <skill content here>
+```
+
+**Advantages over path passing:**
+- No child-process skill loader dependency
+- Content is visible to LLM immediately (no extra tool call)
+- Works with `noSkills: true` (skills already in prompt)
+- Graceful degradation: missing skills show "(Skill not found)" note instead of crash
+
+### pi-crew Gap
+- pi-crew passes `--skill <path>` to child Pi process
+- Child has to load skills separately
+- For live-session, skills should be preloaded into prompt
+
+---
+
+## 6. Batch Notification Grouping (GroupJoinManager) ⬜ NOT IN PI-CREW
+
+### pi-subagents3 Pattern
+```typescript
+class GroupJoinManager {
+    registerGroup(groupId, agentIds);
+    onAgentComplete(record) {
+        // Hold results until ALL agents in group complete
+        // OR timeout fires (default 30s)
+        // Then deliver ONE consolidated notification
+    }
+}
+```
+
+**Join modes:** `async` (individual), `group` (batch), `smart` (heuristic)
+
+**Benefits:**
+- 10 parallel research agents → 1 notification instead of 10
+- Reduces parent context disruption
+- Configurable timeout for stragglers
+
+### pi-crew Gap
+- Each completed task sends individual notification
+- No batch grouping for parallel tasks
+- Parent gets spammed with completion messages
+
+---
+
+## 7. Scheduling (SubagentScheduler) ⬜ NOT IN PI-CREW
+
+### pi-subagents3 Pattern
+```typescript
+class SubagentScheduler {
+    addJob({ name, schedule: "0 0 9 * * 1", subagent_type, prompt });
+    // Supports: cron | interval ("5m") | once ("+10m" | ISO)
+    // Persistence: session-scoped ScheduleStore with PID-locked atomic writes
+    // Bypasses concurrency queue when firing
+}
+```
+
+**Features:**
+- Croner library for cron expressions
+- Session-scoped persistence (survives `/resume`, resets on `/new`)
+- PID-based file locking with stale lock detection
+- Master switch: `schedulingEnabled` setting
+
+### pi-crew Gap
+- No scheduling capability at all
+- No cron/interval/once job support
+- No session-scoped persistent job store
+
+---
+
+## 8. Settings Persistence with Sanitization ⬜ NOT IN PI-CREW
+
+### pi-subagents3 Pattern
+```typescript
+// Global: ~/.pi/agent/subagents.json (defaults, never written here)
+// Project: <cwd>/.pi/subagents.json (overrides)
+
+export interface SubagentsSettings {
+    maxConcurrent?: number;
+    defaultMaxTurns?: number;
+    graceTurns?: number;
+    defaultJoinMode?: JoinMode;
+    schedulingEnabled?: boolean;
+}
+
+function sanitize(raw: unknown): SubagentsSettings {
+    // Drop invalid fields, apply ceilings
+    // maxConcurrent: 1-1024
+    // defaultMaxTurns: 0-10000 (0 = unlimited)
+    // graceTurns: 1-1000
+}
+```
+
+**Features:**
+- Merged load: global defaults + project overrides
+- Sanitization drops garbage silently
+- Settings events: `subagents:settings_loaded`, `subagents:settings_changed`
+- Toast formatting for persist success/failure
+
+### pi-crew Gap
+- pi-crew has `CrewConfig` but no project-local `.pi/crew.json` persistence
+- No sanitization with ceilings
+- No settings change events
+
+---
+
+## 9. Usage Tracking (Survives Compaction) ⬜ NOT IN PI-CREW
+
+### pi-subagents3 Pattern
+```typescript
+export type LifetimeUsage = { input: number; output: number; cacheWrite: number };
+
+// Accumulated via message_end events (survives compaction)
+session.subscribe((event) => {
+    if (event.type === "message_end" && event.message.role === "assistant") {
+        const u = event.message.usage;
+        if (u) options.onAssistantUsage?.({
+            input: u.input ?? 0,
+            output: u.output ?? 0,
+            cacheWrite: u.cacheWrite ?? 0,
+        });
+    }
+});
+```
+
+**Key design:** `getSessionTokens()` resets at compaction (upstream replaces messages array), but `LifetimeUsage` survives because it's independently accumulated.
+
+**cacheRead deliberately excluded** — summing across turns counts the cached prefix N times (issue #38).
+
+### pi-crew Gap
+- pi-crew tracks usage per task but doesn't survive compaction
+- No lifetime usage across sessions
+- No `cacheWrite`/`cacheRead` distinction logic
+
+---
+
+## 10. Worktree Isolation ⬜ NOT IN PI-CREW
+
+### pi-subagents3 Pattern
+```typescript
+export function createWorktree(cwd: string, agentId: string): WorktreeInfo | undefined {
+    // git worktree add --detach <temp-path> HEAD
+    // Returns { path, branch }
+}
+
+export function cleanupWorktree(cwd, worktree, description) {
+    // No changes → remove worktree
+    // Changes → git add -A, git commit, create branch, remove worktree
+    // Returns { hasChanges, branch }
+}
+```
+
+**Features:**
+- Strict: fails loud if not a git repo (no silent fallback)
+- Crash recovery: `pruneWorktrees()` on dispose
+- Branch naming: `pi-agent-{agentId}`, with timestamp suffix if conflict
+
+### pi-crew Gap
+- pi-crew has worktree support but less robust
+- No automatic branch creation for changes
+- No worktree cleanup on error
+
+---
+
+## 11. Model Resolution (Fuzzy + Availability) ⬜ NOT IN PI-CREW
+
+### pi-subagents3 Pattern
+```typescript
+export function resolveModel(input: string, registry: ModelRegistry): any | string {
+    // 1. Exact match "provider/modelId" — only if available (has auth)
+    // 2. Fuzzy match with scoring:
+    //    - exact id match (100)
+    //    - id contains query (60-90)
+    //    - name contains query (40-60)
+    //    - all parts present (20)
+    // 3. No match → return error message with available models list
+}
+```
+
+### pi-crew Gap
+- pi-crew passes model string directly to child Pi
+- No fuzzy resolution
+- No availability check before spawn
+
+---
+
+## 12. Agent Config System (Defaults + Override) ⬜ PARTIAL IN PI-CREW
+
+### pi-subagents3 Pattern
+```typescript
+const DEFAULT_AGENTS = new Map([
+    ["general-purpose", { extensions: true, skills: true, promptMode: "append" }],
+    ["Explore", { builtinToolNames: ["read", "bash", "grep", "find", "ls"], model: "anthropic/claude-haiku-...", promptMode: "replace" }],
+    ["Plan", { builtinToolNames: ["read", "bash", "grep", "find", "ls"], promptMode: "replace" }],
+]);
+
+// User-defined .md files with same name override defaults
+// Resolution: explicit option > config.model > parent model
+```
+
+**Features:**
+- `builtinToolNames` — restrict tool set per agent type
+- `disallowedTools` — denylist (removed even if extensions include them)
+- `promptMode: "replace" | "append"` — full control vs. parent clone
+- `extensions: true | string[] | false` — selective extension inheritance
+- `skills: true | string[] | false` — selective skill inheritance
+- `isolated: boolean` — no extension tools
+
+### pi-crew Gap
+- pi-crew has agent configs but no `builtinToolNames` per agent
+- No `disallowedTools` concept
+- No `promptMode` (always append-ish)
+- `extensions`/`skills` are boolean only (no selective)
+
+---
+
+## 13. Streaming Output (Real-Time Transcript) ⬜ NOT IN PI-CREW
+
+### pi-subagents3 Pattern
+```typescript
+// AgentRecord has:
+outputFile?: string;
+outputCleanup?: () => void;
+
+// In spawn:
+const outputFile = path.join(stateDir, `${id}.output.md`);
+const stream = createWriteStream(outputFile);
+// Subscribe to session events, write text deltas to stream
+// onComplete: flush stream, cleanup
+```
+
+**Benefits:**
+- Real-time transcript file for long-running agents
+- Parent can `tail -f` the file for progress
+- `outputCleanup` ensures stream is closed
+
+### pi-crew Gap
+- pi-crew writes artifacts after task completes
+- No real-time streaming transcript during task execution
+- Parent must wait for completion to see output
+
+---
+
+## 14. Cross-Extension RPC ⬜ NOT IN PI-CREW
+
+### pi-subagents3 Pattern
+```typescript
+export function registerRpcHandlers(deps: RpcDeps): RpcHandle {
+    const unsubPing = handleRpc(events, "subagents:rpc:ping", () => ({ version: PROTOCOL_VERSION }));
+    const unsubSpawn = handleRpc(events, "subagents:rpc:spawn", ({ type, prompt, options }) => {
+        const ctx = getCtx();
+        return { id: manager.spawn(pi, ctx, type, prompt, options ?? {}) };
+    });
+    const unsubStop = handleRpc(events, "subagents:rpc:stop", ({ agentId }) => {
+        if (!manager.abort(agentId)) throw new Error("Agent not found");
+    });
+    return { unsubPing, unsubSpawn, unsubStop };
+}
+```
+
+**Features:**
+- Per-request scoped reply channels: `${channel}:reply:${requestId}`
+- Envelope: `{ success: true, data? } | { success: false, error }`
+- Protocol versioning
+
+### pi-crew Gap
+- pi-crew has no RPC for external extensions to spawn team runs
+- No protocol versioning
+- Extensions can only use `team` tool
+
+---
+
+## 15. Concurrency Queue with Bypass ⬜ PARTIAL IN PI-CREW
+
+### pi-subagents3 Pattern
+```typescript
+spawn(..., { isBackground: true, bypassQueue: false }) {
+    if (runningBackground >= maxConcurrent) {
+        record.status = "queued";
+        queue.push({ id, args });
+        return id;
+    }
+    startAgent(id, record, args);
+}
+
+// When agent completes:
+this.runningBackground--;
+this.drainQueue();  // Start next queued agent
+
+// Scheduled jobs bypass queue:
+manager.spawn(..., { bypassQueue: true });  // Always starts immediately
+```
+
+### pi-crew Status
+- pi-crew has `SubagentManager` with `runningBackground` counter
+- Has queue logic but no `bypassQueue` flag
+- No `drainQueue()` — queued agents may not auto-start
+
+---
+
+## 16. Parent Signal Wiring ⬜ PARTIAL IN PI-CREW
+
+### pi-subagents3 Pattern
+```typescript
+// In spawn:
+if (options.signal) {
+    const onParentAbort = () => this.abort(id);
+    options.signal.addEventListener("abort", onParentAbort, { once: true });
+    detachParentSignal = () => options.signal!.removeEventListener("abort", onParentAbort);
+}
+
+// Cleanup in .then() and .catch():
+detach();  // Remove listener to avoid leak
+```
+
+### pi-crew Status
+- pi-crew passes `signal` to `runTeamTask` and `runLiveSessionTask`
+- But no explicit detach cleanup after completion
+- Listener may leak
+
+---
+
+## Priority Implementation Roadmap
+
+### P0 — Immediate (next commit)
+1. **Soft turn limit + grace steering** — Best output quality improvement
+2. **Context % indicator** — Dashboard enhancement, low effort
+
+### P1 — This Week
+3. **Skill preloading** — Required for live-session to work without child-process skill loader
+4. **Persistent agent memory** — Major differentiator, medium effort
+5. **Usage tracking (survives compaction)** — Metrics accuracy
+
+### P2 — Next Sprint
+6. **Batch notification grouping** — Parallel run UX
+7. **Settings persistence with sanitization** — Config robustness
+8. **Streaming output transcript** — Real-time progress visibility
+9. **Worktree auto-branch** — Isolation improvement
+
+### P3 — Future
+10. **Scheduling** — New feature category
+11. **Cross-extension RPC** — Ecosystem integration
+12. **Model fuzzy resolution** — UX polish

package/docs/product/README.md

@@ -0,0 +1,31 @@
+# Product Docs
+
+Product documentation for pi-crew. Each file describes a product domain —
+what it does, how it behaves, and what contracts it maintains.
+
+## Update Rule
+
+When behavior changes:
+1. Update the affected product doc
+2. Update or create the story packet
+3. Update `docs/TEST_MATRIX.md`
+4. Record a decision if it affects architecture, scope, risk, or settled rules
+
+## Domain Index
+
+| File | Domain | Description |
+|------|--------|-------------|
+| `team-run.md` | Core | Team run lifecycle: start, execute, complete |
+| `team-tool.md` | API | Team tool actions: run, status, list, plan |
+| `child-process.md` | Runtime | Child Pi process spawning and management |
+| `live-session.md` | Runtime | In-process agent execution |
+| `async-runner.md` | Runtime | Background/async run execution |
+| `state.md` | State | Durable state: manifests, tasks, events |
+| `worktree.md` | Isolation | Git worktree isolation for parallel work |
+| `group-join.md` | Coordination | Agent result grouping and delivery |
+| `model-fallback.md` | Runtime | Model selection and fallback chain |
+| `conflict-detect.md` | Utils | Merge conflict detection in file edits |
+| `crash-recovery.md` | Reliability | Crash recovery and stale reconciliation |
+| `effectiveness.md` | Quality | Effectiveness guard for worker activity |
+| `platform.md` | Platform | Cross-platform considerations (Windows) |
+| `runtime-safety.md` | Safety | Runtime safety: depth guard, resource limits |

package/docs/product/platform.md

@@ -0,0 +1,27 @@
+# Platform (Cross-Platform)
+
+## Behavior
+
+pi-crew runs on Windows, macOS, and Linux. Primary development is on Windows.
+
+### Windows Considerations
+
+- **EBUSY/EPERM**: Files locked by antivirus, shell, or indexer
+  - `rmSyncRetry()` with exponential backoff (50ms, 100ms, 200ms, 400ms)
+  - `existsSync` check before cleanup in finally blocks
+- **Path separators**: Use `path.join()` everywhere, never hardcoded `/`
+- **Shell**: `resolve-shell.ts` handles `cmd.exe` vs `bash` detection
+- **Case sensitivity**: Windows is case-insensitive for file paths
+
+### Unix Considerations
+
+- `unref()` on timers to prevent blocking process exit
+- POSIX shell compatibility in any shell scripts
+- Signal handling (SIGTERM, SIGINT) for graceful shutdown
+
+### CI Matrix
+
+All changes validated on:
+- `ubuntu-latest` / Node 22
+- `windows-latest` / Node 22
+- `macos-latest` / Node 22

package/docs/product/runtime-safety.md

@@ -0,0 +1,37 @@
+# Runtime Safety
+
+## Behavior
+
+pi-crew enforces multiple safety layers to prevent resource leaks, crashes,
+and runaway execution.
+
+### Depth Guard
+
+- Tracks `PI_CREW_SESSION_DEPTH` environment variable
+- Depth >= 2 forces `child-process` mode instead of `live-session`
+- Prevents stack overflow from nested team runs
+
+### Resource Limits
+
+- Memory cap on live-session agents
+- Prompt timeout for agent responses
+- Tool count restoration after session error
+
+### Process Cleanup
+
+- `cleanupTempDir()` with `existsSync` guard against double cleanup
+- `safeDisposeLiveSession()` for clean resource teardown
+- `removeLiveAgentHandle()` for registry cleanup
+
+### Error Handling
+
+- `try/catch` around all I/O operations in UI code
+- `rmSyncRetry()` with exponential backoff for Windows EBUSY
+- `rejectRunPromise` (not `resolveRunPromise`) in error paths
+
+### State Integrity
+
+- `withRunLockSync` for all state mutations
+- Atomic write helpers (`writeJsonAtomic`, `appendJsonAtomic`)
+- `markActiveTasksAndAgentsFailed()` for crash recovery
+- Event log append-only for audit trail

package/docs/product/team-run.md

@@ -0,0 +1,39 @@
+# Team Run
+
+## Behavior
+
+A team run executes a workflow with multiple agents, tracking progress through
+durable state on disk.
+
+### Lifecycle
+
+1. User invokes `team action='run'` with a goal
+2. Team runner creates a manifest, resolves team/workflow
+3. Task graph is built from workflow steps
+4. Tasks execute (parallel or sequential per workflow)
+5. Results are collected, artifacts written
+6. Run completes with final status
+
+### Statuses
+
+| Status | Meaning |
+|--------|---------|
+| pending | Manifest created, not yet executing |
+| running | Tasks executing |
+| completed | All tasks finished successfully |
+| failed | One or more tasks failed |
+| cancelled | User cancelled the run |
+| partial | Some tasks completed, others still pending |
+
+### Concurrency
+
+- Tasks without dependencies run in parallel (up to concurrency limit)
+- Tasks with `dependsOn` wait for predecessors
+- Workflow phases enforce ordering
+
+### Artifacts
+
+- `results/{taskId}.txt` — task output
+- `logs/{taskId}.log` — full transcript
+- `metadata/` — task metadata files
+- `shared/` — inter-agent shared context

package/docs/product/team-tool.md

@@ -0,0 +1,37 @@
+# Team Tool API
+
+## Behavior
+
+The `team` tool is the primary interface for users to interact with pi-crew.
+
+### Actions
+
+| Action | Description |
+|--------|-------------|
+| `run` | Start a team run |
+| `plan` | Create a plan without executing |
+| `status` | Check run/task status |
+| `list` | List teams, agents, workflows |
+| `get` | Get resource details |
+| `cancel` | Cancel a running task/run |
+| `resume` | Resume a paused run |
+| `respond` | Respond to a waiting task |
+| `recommend` | Get team/workflow recommendations |
+| `create/update/delete` | Manage resources |
+| `doctor` | Diagnose configuration issues |
+
+### Parameters
+
+- `action` (required): The action to perform
+- `team`: Team name for run operations
+- `goal`: High-level objective
+- `runId`: Run ID for status/cancel/resume
+- `taskId`: Task ID for respond operations
+- `confirm: true`: Required for destructive actions
+
+### Safety Rules
+
+- Delete operations require `confirm: true`
+- Referenced resources blocked unless `force: true`
+- Cancel requires explicit run ID
+- Respond requires task ID + message