pi-crew 0.2.25 → 0.3.0

package/docs/pi-subagent4-comparison.md

@@ -0,0 +1,261 @@
+# pi-subagent4 vs pi-crew: Comparative Analysis
+
+## Overview
+
+| Aspect | pi-subagent4 | pi-crew |
+|--------|--------------|---------|
+| **Size** | ~560 lines (single file) | ~50+ files, 10K+ lines |
+| **Architecture** | Single `subagent` tool | Full team orchestration system |
+| **Agent Model** | 3 built-in (scout, researcher, worker) | Configurable, extensible |
+| **Concurrency** | Semaphore (default 4) | DAG scheduler with phases |
+| **Context** | No inheritance (must be in task) | Full context preservation |
+
+---
+
+## 1. Extension & Registration
+
+### pi-subagent4
+```typescript
+// Dynamic agent registration via globalThis bridge
+(globalThis as any).__pi_subagents = { registerAgent, unregisterAgent };
+
+export function registerAgent(config: AgentConfig): void {
+  agents.push(config);
+}
+```
+- **File-based**: Loads agents from `.md` files at startup
+- **Global bridge**: Uses `globalThis.__pi_subagents` for cross-module registration
+- **Frontmatter config**: YAML frontmatter in `.md` files define agents
+
+### pi-crew
+- **Manifest-based**: Teams/workflows defined in `.team.md`/`.workflow.md` files
+- **Skills system**: Extensible skill system for agents
+- **No dynamic registration API**: Static configuration
+
+---
+
+## 2. Child Process Spawning
+
+### pi-subagent4
+```typescript
+const args = [
+  ...piBin.baseArgs,
+  "--mode", "json",
+  "-p",
+  "--no-session",
+  "--no-skills",
+  "--no-extensions",
+  "--tools", allowlist.join(","),
+  // ... custom tools, model, thinking level
+];
+
+const child = spawn(command, spawnArgs, { stdio: ["ignore", "pipe", "pipe"] });
+```
+- **JSON mode**: `--mode json` for structured output
+- **Heavy isolation**: `--no-session --no-skills --no-extensions`
+- **Tool allowlist**: `--tools` for fine-grained control
+- **PI_SUBAGENT_ALLOWED**: Env var restricts nested subagents
+
+### pi-crew
+```typescript
+const child = spawn(spawnSpec.command, spawnSpec.args, buildChildPiSpawnOptions(...));
+```
+- **Similar isolation**: Filters env vars, preserves essentials
+- **More complex args**: Based on task config
+- **No direct env restriction**: Uses runtime mode instead
+
+---
+
+## 3. Concurrency Control
+
+### pi-subagent4
+```typescript
+class Semaphore {
+  constructor(private readonly max: number) {}
+
+  async run<T>(fn: () => Promise<T>): Promise<T> {
+    // Simple acquire/release pattern
+  }
+}
+
+const semaphore = new Semaphore(config.maxConcurrency ?? 4);
+```
+- **Per-parent semaphore**: Default 4, configurable via `config.json`
+- **Promise.all fan-out**: Parallel subagent calls in one turn
+
+### pi-crew
+```typescript
+// DAG scheduler with phase-based concurrency
+resolveBatchConcurrency({
+  workflowMaxConcurrency,
+  teamMaxConcurrency,
+  maxConcurrentWorkers,
+  workspaceMode,
+});
+
+// Tasks in same phase run concurrently
+```
+- **Phase-based**: Tasks grouped by workflow phase
+- **DAG dependency**: Respects task dependencies
+- **Configurable limits**: Per-workflow and per-team
+
+---
+
+## 4. Input Handling
+
+### pi-subagent4
+```typescript
+// Long tasks written to temp file
+if (task.length > 8000) {
+  const tempFile = createTempFile(task);
+  args.push("@" + tempFile);
+} else {
+  args.push("--task", task);
+}
+```
+- **8K char threshold**: Uses temp file for large tasks
+- **Single task format**: `--task <text>` or `@<file>`
+
+### pi-crew
+```typescript
+// Prompt builder with system prompt, context, task
+const built = await buildPrompt({ task, role, goal, cwd, ... });
+// Args built from task configuration
+```
+- **Prompt builder**: Constructs full prompt with context
+- **File-based context**: Can read from workspace files
+
+---
+
+## 5. Output Handling
+
+### pi-subagent4
+```typescript
+// JSON event stream on stdout
+child.stdout.on("data", (data) => {
+  const lines = data.toString().split("\n");
+  for (const line of lines) {
+    if (line.startsWith("{")) {
+      const event = JSON.parse(line);
+      // tool_execution_start/end, message_end
+    }
+  }
+});
+```
+- **JSON event stream**: Structured events from child process
+- **Event types**: `tool_execution_start`, `tool_execution_end`, `message_end`
+- **Streaming**: Real-time event processing
+
+### pi-crew
+```typescript
+// JSON output mode + structured response
+const output = await runChildPi({
+  onLifecycleEvent: (event) => { ... },
+  // ...
+});
+```
+- **Lifecycle events**: spawn, spawn_error, response_timeout, etc.
+- **Structured result**: `{ content, details, usage }`
+
+---
+
+## 6. Safety Features
+
+### pi-subagent4
+```typescript
+// tools/safe-bash.ts
+const DANGEROUS_PATTERNS = [
+  /\brm\s+(-[a-zA-Z]*f[a-zA-Z]*\s+)?(-[a-zA-Z]*r[a-zA-Z]*\s+)?(\/|~\/?\s|~\/?\b)/,
+  /\bsudo\b/,
+  /\bmkfs\b/,
+  // ... 15+ patterns
+];
+```
+- **Regex blocklist**: 15+ dangerous command patterns
+- **Safe bash wrapper**: Wraps built-in bash tool
+
+### pi-crew
+- **Env var filtering**: Strips secrets before spawning
+- **No built-in safe bash**: Trust-based (user config required)
+- **Sandbox modes**: scaffold, child-process, live-session
+
+---
+
+## 7. UI/Rendering
+
+### pi-subagent4
+```typescript
+// Throttled live rendering
+const updateThrottle = 150; // ms
+// Context window meter for depth >= 1 subagents
+// Tool preview extraction
+```
+- **150ms throttle**: Prevents UI thrashing
+- **Context gauge**: Shows token usage
+- **Tool preview**: Single-line argument preview
+
+### pi-crew
+- **Rich UI widget**: Live status, progress, model/token display
+- **Dashboard**: Full run dashboard
+- **Event bus**: Real-time updates
+
+---
+
+## 8. Agent Hierarchy
+
+### pi-subagent4
+```
+worker (depth 2)
+  ├─ scout (depth 1)
+  └─ researcher (depth 1)
+```
+- **Depth-2 cap**: Worker can spawn scout/researcher
+- **PI_SUBAGENT_ALLOWED**: Enforces restriction
+
+### pi-crew
+- **No nested subagent**: Each task is independent
+- **Team roles**: explorer, planner, executor, etc.
+- **Phase-based**: Sequential phases with parallel within
+
+---
+
+## Key Insights
+
+### What pi-subagent4 does better:
+1. **Simpler API**: Single tool, minimal config
+2. **Dynamic registration**: `registerAgent()` for runtime changes
+3. **JSON event stream**: Real-time structured events
+4. **Safe bash**: Built-in dangerous command blocking
+5. **Context gauge**: Token monitoring per turn
+
+### What pi-crew does better:
+1. **Complex workflows**: DAG scheduler, phases, dependencies
+2. **Durable state**: Manifest, events, artifacts persisted
+3. **Worktree isolation**: Safe parallel edits
+4. **Async runs**: Background execution with notifications
+5. **Rich UI**: Full dashboard and widget system
+6. **Multiple teams**: Built-in teams for different use cases
+
+---
+
+## Potential Improvements for pi-crew
+
+1. **Dynamic agent registration API**
+   - Add `registerAgent(config)` similar to subagent4
+   - Allow runtime agent creation
+
+2. **Safe bash tool**
+   - Port dangerous pattern blocklist from subagent4
+   - Configurable via project config
+
+3. **JSON event stream parsing**
+   - Extract real-time tool events from child process
+   - Display tool progress in UI
+
+4. **Context window monitoring**
+   - Show token usage per task
+   - Alert when approaching limits
+
+5. **Simpler single-agent mode**
+   - Maybe a `subagent` tool for simple delegation?
+   - Current API is team/workflow based, could be heavy for simple cases

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-crew",
-  "version": "0.2.25",
+  "version": "0.3.0",
   "description": "Pi extension for coordinated AI teams, workflows, worktrees, and async task orchestration",
   "author": "baphuongna",
   "license": "MIT",

package/src/agents/discover-agents.ts

@@ -100,22 +100,92 @@ function applyAgentOverrides(agents: AgentConfig[], cwd: string, loadedConfig?:
 		});
 }
 
+// ─── Agent Discovery Cache (Phase 3a) ────────────────────────────────────
+// Caches discoverAgents results by cwd with a short TTL to avoid repeated
+// disk I/O when multiple callers request agents for the same project.
+
+const DISCOVERY_CACHE_TTL_MS = 500;
+const discoveryCache = new Map<string, { result: AgentDiscoveryResult; expiresAt: number }>();
+const DISCOVERY_CACHE_MAX_ENTRIES = 32;
+
+function pruneDiscoveryCache(): void {
+	const now = Date.now();
+	for (const [key, entry] of discoveryCache) {
+		if (entry.expiresAt <= now) discoveryCache.delete(key);
+	}
+}
+
+/** Invalidate cached discovery result for a given cwd (or all if omitted). */
+export function invalidateAgentDiscoveryCache(cwd?: string): void {
+	if (cwd) {
+		discoveryCache.delete(cwd);
+	} else {
+		discoveryCache.clear();
+	}
+}
+
 export function discoverAgents(cwd: string): AgentDiscoveryResult {
+	pruneDiscoveryCache();
+	const cached = discoveryCache.get(cwd);
+	if (cached && cached.expiresAt > Date.now()) {
+		return cached.result;
+	}
 	const loaded = loadConfig(cwd);
-	return {
+	const result: AgentDiscoveryResult = {
 		builtin: applyAgentOverrides(readAgentDir(path.join(packageRoot(), "agents"), "builtin"), cwd, loaded),
 		user: applyAgentOverrides(readAgentDir(path.join(userPiRoot(), "agents"), "user"), cwd, loaded),
 		project: applyAgentOverrides(readAgentDir(path.join(projectCrewRoot(cwd), "agents"), "project"), cwd, loaded),
 	};
+	discoveryCache.set(cwd, { result, expiresAt: Date.now() + DISCOVERY_CACHE_TTL_MS });
+	while (discoveryCache.size > DISCOVERY_CACHE_MAX_ENTRIES) {
+		const oldest = discoveryCache.keys().next().value;
+		if (oldest !== undefined) discoveryCache.delete(oldest);
+	}
+	return result;
+}
+
+// ─── Dynamic Agent Registry (Phase 3b) ───────────────────────────────────
+// In-memory store for runtime-registered agents. Merged into discovery results
+// with highest priority (after project agents).
+
+const dynamicAgents = new Map<string, AgentConfig>();
+
+/** Register a dynamic agent at runtime. Throws if already registered. */
+export function registerDynamicAgent(config: AgentConfig): void {
+	const key = config.name.toLowerCase();
+	if (dynamicAgents.has(key)) {
+		throw new Error(`Agent already registered: ${config.name}`);
+	}
+	dynamicAgents.set(key, { ...config, source: config.source ?? "project" });
+	invalidateAgentDiscoveryCache();
+}
+
+/** Unregister a previously registered dynamic agent. Throws if not found. */
+export function unregisterDynamicAgent(name: string): void {
+	const removed = dynamicAgents.delete(name.toLowerCase());
+	if (!removed) {
+		throw new Error(`Agent not found: ${name}`);
+	}
+	invalidateAgentDiscoveryCache();
+}
+
+/** List all currently registered dynamic agents. */
+export function listDynamicAgents(): AgentConfig[] {
+	return [...dynamicAgents.values()];
 }
 
 export function allAgents(discovery: AgentDiscoveryResult): AgentConfig[] {
 	const byName = new Map<string, AgentConfig>();
-	// Priority: project > builtin > user for disambiguation.
-	// This means a project agent with the same name as a builtin/user agent is shadowed
-	// (security: project config cannot override trusted builtins).
+	// Priority for disambiguation (security): project < builtin < user.
+	// Project config cannot override trusted builtins (security-hardening).
+	// Later entries in the loop overwrite earlier ones, so user wins.
 	for (const agent of [...discovery.project, ...discovery.builtin, ...discovery.user]) {
 		byName.set(agent.name.toLowerCase(), agent);
 	}
+	// Dynamic agents (registered at runtime) take highest precedence.
+	// They can override any discovered agent (project/builtin/user).
+	for (const agent of dynamicAgents.values()) {
+		byName.set(agent.name.toLowerCase(), agent);
+	}
 	return [...byName.values()].filter((agent) => !agent.disabled).sort((a, b) => a.name.localeCompare(b.name));
 }

package/src/extension/register.ts

@@ -445,35 +445,36 @@ export function registerPiTeams(pi: ExtensionAPI): void {
 	// Register global RPC registry for cross-extension access (mirrors pi-subagents3's Symbol.for pattern)
 	// Uses lazy import to avoid pulling team-tool.ts into module load.
 	// Other extensions access via: const reg = globalThis[Symbol.for("pi-crew:registry")];
-	void import("./team-tool.ts").then(({ registerCrewGlobalRegistry }) => {
+	void import("./team-tool.ts").then(({ registerCrewGlobalRegistry, installCrewGlobalRegistry }) => {
+		// Phase 3b: installCrewGlobalRegistry creates a v2 registry with agent registration API.
+		// We then patch the manifest-backed methods with real implementations below.
 		const manifestCacheForRegistry = getManifestCache(currentCtx?.cwd ?? process.cwd());
-		registerCrewGlobalRegistry({
-			version: 1,
-			getRecord: (runId) => manifestCacheForRegistry.get(runId),
-			listRuns: () => manifestCacheForRegistry.list(100).map((m) => ({ runId: m.runId, status: m.status, goal: m.goal })),
-			appendEvent: (runId, event) => {
-				const manifest = manifestCacheForRegistry.get(runId);
-				if (manifest) void import("../state/event-log.ts").then(({ appendEventFireAndForget }) => appendEventFireAndForget(manifest.eventsPath, event as Parameters<typeof appendEventFireAndForget>[1]));
-			},
-			waitForAll: async (runId) => {
-				// LAZY: loadRunManifestById is already imported at top of file, but kept here for consistency
-				const { loadRunManifestById } = await import("../state/state-store.ts");
-				const check = (): boolean => {
-					const loaded = loadRunManifestById(currentCtx?.cwd ?? process.cwd(), runId);
-					if (!loaded) return true;
-					return !loaded.tasks.some((t: { status: string }) => t.status === "running" || t.status === "queued");
-				};
-				while (!check()) await new Promise((resolve) => setTimeout(resolve, 500));
-			},
-			hasRunning: (runId) => {
-				const manifest = manifestCacheForRegistry.get(runId);
-				if (!manifest) return false;
-				const { loadRunManifestById } = require("../state/state-store.ts");
+		installCrewGlobalRegistry();
+		const CREW_REGISTRY_KEY = Symbol.for("pi-crew:registry");
+		const registry = (globalThis as Record<symbol | string, unknown>)[CREW_REGISTRY_KEY] as Record<string, unknown>;
+		registry.getRecord = (runId: string) => manifestCacheForRegistry.get(runId);
+		registry.listRuns = () => manifestCacheForRegistry.list(100).map((m: { runId: string; status: string; goal: string }) => ({ runId: m.runId, status: m.status, goal: m.goal }));
+		registry.appendEvent = (runId: string, event: Record<string, unknown>) => {
+			const manifest = manifestCacheForRegistry.get(runId);
+			if (manifest) void import("../state/event-log.ts").then(({ appendEventFireAndForget }) => appendEventFireAndForget(manifest.eventsPath, event as Parameters<typeof appendEventFireAndForget>[1]));
+		};
+		registry.waitForAll = async (runId: string) => {
+			const { loadRunManifestById } = await import("../state/state-store.ts");
+			const check = (): boolean => {
 				const loaded = loadRunManifestById(currentCtx?.cwd ?? process.cwd(), runId);
-				if (!loaded) return false;
-				return loaded.tasks.some((t: { status: string }) => t.status === "running" || t.status === "queued");
-			},
-		});
+				if (!loaded) return true;
+				return !loaded.tasks.some((t: { status: string }) => t.status === "running" || t.status === "queued");
+			};
+			while (!check()) await new Promise((resolve) => setTimeout(resolve, 500));
+		};
+		registry.hasRunning = (runId: string) => {
+			const manifest = manifestCacheForRegistry.get(runId);
+			if (!manifest) return false;
+			const { loadRunManifestById } = require("../state/state-store.ts");
+			const loaded = loadRunManifestById(currentCtx?.cwd ?? process.cwd(), runId);
+			if (!loaded) return false;
+			return loaded.tasks.some((t: { status: string }) => t.status === "running" || t.status === "queued");
+		};
 	});
 
 	const cleanupRuntime = (): void => {

package/src/extension/registration/subagent-tools.ts

@@ -22,6 +22,7 @@ import { t } from "../../i18n.ts";
 import { loadRunManifestById } from "../../state/state-store.ts";
 import { readCrewAgents } from "../../runtime/crew-agent-records.ts";
 import { formatCompactToolProgress } from "../../ui/tool-progress-formatter.ts";
+import { renderAgentToolCall, renderAgentToolResult } from "../../ui/tool-render.ts";
 
 const TOOL_PROGRESS_TICK_MS = 1000;
 
@@ -95,6 +96,12 @@ export function registerSubagentTools(pi: ExtensionAPI, subagentManager: Subagen
 			}
 			return foregroundResult;
 		},
+		renderCall(args: any, theme: any, context: any): any {
+			return renderAgentToolCall(args, theme, context);
+		},
+		renderResult(result: any, options: any, theme: any, context: any): any {
+			return renderAgentToolResult(result, options, theme, context);
+		},
 	};
 
 	const getSubagentResultTool: ToolDefinition = {

package/src/extension/registration/team-tool.ts

@@ -9,6 +9,7 @@ import type { createManifestCache } from "../../runtime/manifest-cache.ts";
 import type { createRunSnapshotCache } from "../../ui/run-snapshot-cache.ts";
 import type { MetricRegistry } from "../../observability/metric-registry.ts";
 import { resolveRealContainedPath } from "../../utils/safe-paths.ts";
+import { renderTeamToolCall, renderTeamToolResult } from "../../ui/tool-render.ts";
 // Team tool handler — lazy-loaded because team-tool.ts imports many modules
 import type { handleTeamTool as HandleTeamToolFn } from "../team-tool.ts";
 let _cachedHandleTeamTool: typeof HandleTeamToolFn | undefined;
@@ -104,6 +105,12 @@ export function registerTeamTool(pi: ExtensionAPI, deps: RegisterTeamToolDeps):
 				stopProgress.stop();
 			}
 		},
+		renderCall(args: any, theme: any, context: any): any {
+			return renderTeamToolCall(args, theme, context);
+		},
+		renderResult(result: any, options: any, theme: any, context: any): any {
+			return renderTeamToolResult(result, options, theme, context);
+		},
 	};
 	pi.registerTool(tool);
 }

package/src/extension/team-tool.ts

@@ -1,6 +1,7 @@
 import * as fs from "node:fs";
 import * as path from "node:path";
-import { allAgents, discoverAgents } from "../agents/discover-agents.ts";
+import { allAgents, discoverAgents, invalidateAgentDiscoveryCache, registerDynamicAgent, unregisterDynamicAgent, listDynamicAgents } from "../agents/discover-agents.ts";
+import type { AgentConfig } from "../agents/agent-config.ts";
 import { allTeams, discoverTeams } from "../teams/discover-teams.ts";
 import { allWorkflows, discoverWorkflows } from "../workflows/discover-workflows.ts";
 import { loadConfig, updateAutonomousConfig, updateConfig } from "../config/config.ts";
@@ -383,14 +384,25 @@ export async function handleTeamTool(params: TeamToolParamsValue, ctx: TeamConte
  */
 const CREW_REGISTRY_KEY = Symbol.for("pi-crew:registry");
 interface CrewRegistry {
-	version: 1;
+	version: 2;
 	getRecord: (runId: string) => TeamRunManifest | undefined;
 	listRuns: () => Array<{ runId: string; status: string; goal: string }>;
 	appendEvent: (runId: string, event: Record<string, unknown>) => void;
 	waitForAll: (runId: string) => Promise<void>;
 	hasRunning: (runId: string) => boolean;
+	/** Register a dynamic agent at runtime. Invalidates the discovery cache. */
+	registerAgent: (config: AgentConfig) => void;
+	/** Unregister a previously registered dynamic agent. Invalidates the discovery cache. */
+	unregisterAgent: (name: string) => void;
+	/** List all currently registered dynamic agents. */
+	listDynamicAgents: () => AgentConfig[];
 }
 
+// ─── Dynamic Agent Registry (Phase 3b) ───────────────────────────────────
+// The dynamic agent store lives in discover-agents.ts and is merged into
+// discovery results with highest priority. The CrewRegistry interface exposes
+// registerAgent/unregisterAgent/listDynamicAgents for cross-extension access.
+
 export function registerCrewGlobalRegistry(registry: CrewRegistry): void {
 	(globalThis as Record<symbol | string, unknown>)[CREW_REGISTRY_KEY] = registry;
 }
@@ -398,3 +410,18 @@ export function registerCrewGlobalRegistry(registry: CrewRegistry): void {
 export function getCrewGlobalRegistry(): CrewRegistry | undefined {
 	return (globalThis as Record<symbol | string, unknown>)[CREW_REGISTRY_KEY] as CrewRegistry | undefined;
 }
+
+/** Create and install the global CrewRegistry singleton. Call once at extension init. */
+export function installCrewGlobalRegistry(): void {
+	registerCrewGlobalRegistry({
+		version: 2,
+		getRecord: (runId: string) => undefined as unknown as TeamRunManifest,
+		listRuns: () => [],
+		appendEvent: () => {},
+		waitForAll: async () => {},
+		hasRunning: () => false,
+		registerAgent: registerDynamicAgent,
+		unregisterAgent: unregisterDynamicAgent,
+		listDynamicAgents,
+	});
+}

package/src/runtime/heartbeat-watcher.ts

@@ -96,8 +96,23 @@ export class HeartbeatWatcher {
 				activeKeys.add(key);
 				this.lastSeen.set(key, now);
 
-				const elapsed = heartbeatAgeMs(task.heartbeat, now);
-				const level = classifyHeartbeat(task.heartbeat, thresholds, now);
+				// Check heartbeat staleness with lastActivityAt fallback
+				let elapsed = heartbeatAgeMs(task.heartbeat, now);
+				// PR #6 partial: use lastActivityAt as fallback when heartbeat is stale
+				// If heartbeat is stale but lastActivityAt is fresher, use activity age instead.
+				// This prevents false-positive dead detection for live-session tasks during long operations.
+				if (task.agentProgress?.lastActivityAt) {
+					const activityAt = new Date(task.agentProgress.lastActivityAt).getTime();
+					if (Number.isFinite(activityAt)) {
+						const activityAge = now - activityAt;
+						// Use activity age if it's fresher than heartbeat age
+						// (no upper bound - if agent has recent activity, trust it even if old)
+						if (activityAge < elapsed) {
+							elapsed = activityAge;
+						}
+					}
+				}
+				const level = elapsed > thresholds.deadMs ? "dead" : elapsed > thresholds.staleMs ? "stale" : elapsed > thresholds.warnMs ? "warn" : "healthy";
 				this.opts.registry.gauge("crew.heartbeat.staleness_ms", "Heartbeat elapsed since last seen, milliseconds").set({ runId: run.runId, taskId: task.id }, Number.isFinite(elapsed) ? elapsed : thresholds.deadMs);
 				this.opts.registry.counter("crew.heartbeat.level_total", "Heartbeat classifications by level").inc({ runId: run.runId, level });
 				const previous = this.lastLevel.get(key);