pi-crew 0.1.21 → 0.1.22

package/docs/source-runtime-refactor-map.md

@@ -0,0 +1,83 @@
+# pi-crew runtime refactor source map
+
+This document records the source projects used as the baseline for the pi-crew subagent/runtime refactor. The goal is to avoid ad-hoc fixes in critical process orchestration paths and instead align pi-crew with proven Pi extension patterns.
+
+## Source/pi-subagents
+
+Primary source for child-process worker execution.
+
+- `pi-spawn.ts`: robust Pi CLI resolution on Windows and package installs.
+- `async-execution.ts`: detached async runner with `windowsHide: true` to avoid blank console windows.
+- `subagent-runner.ts`: streaming child Pi process runner, output capture, result extraction.
+- `post-exit-stdio-guard.ts`: guards for child processes that exit before stdio fully closes.
+- `result-watcher.ts` and `async-job-tracker.ts`: durable async job/result observation patterns.
+- `model-fallback.ts`: model fallback policy independent of hardcoded provider assumptions.
+- `subagent-control.ts`, `run-status.ts`: status and control semantics.
+
+pi-crew alignment:
+
+- Background runner and child worker spawn options now explicitly set `windowsHide: true`.
+- Parallel research no longer gates all shard workers behind a single discover worker.
+- Further work should consolidate `child-pi.ts`, `async-runner.ts`, and `subagent-manager.ts` into a durable-first subagent runtime module.
+
+## Source/pi-subagents2
+
+Primary source for higher-level agent management and UI patterns.
+
+- `src/agent-manager.ts`: agent lifecycle registry boundaries.
+- `src/agent-runner.ts`: invocation/run abstraction separate from UI registration.
+- `src/model-resolver.ts`: cleaner model resolution responsibility.
+- `src/output-file.ts`: output file abstraction.
+- `src/ui/agent-widget.ts`, `src/ui/conversation-viewer.ts`: compact live status and transcript viewing.
+
+pi-crew alignment:
+
+- Keep `Agent`/`crew_agent` tools as thin adapters over a durable manager.
+- Avoid storing essential run mapping in memory only.
+- Keep UI active-only and file-backed.
+
+## Source/pi-mono
+
+Primary source for Pi extension API/lifecycle constraints.
+
+- `packages/coding-agent/src/core/extensions/types.ts`: extension context/tool contracts.
+- `packages/coding-agent/src/core/extensions/runner.ts`: extension execution boundaries.
+- `packages/coding-agent/src/core/model-registry.ts`: available model discovery.
+- `packages/coding-agent/src/modes/interactive/interactive-mode.ts`: session lifecycle/UI behavior.
+
+pi-crew alignment:
+
+- Treat session-bound foreground workers differently from explicit async background workers.
+- Do not assume hardcoded providers/models.
+- Use Pi-native UI calls without modal auto-open by default.
+
+## Source/pi-powerbar, pi-plan, pi-diff-review, pi-extensions*
+
+Sources for UI and small-extension patterns.
+
+- `pi-powerbar/src/powerbar/*`: low-noise status segment publishing.
+- `pi-plan/src/plan-action-ui.ts`: action-oriented UI without persistent heavy overlays.
+- `pi-diff-review/src/*`: command/tool registration and review UX patterns.
+- `pi-extensions2/files-widget/*`: file-backed UI composition and navigation.
+
+pi-crew alignment:
+
+- Keep persistent widget active-only.
+- Prefer manual dashboard/transcript commands for history.
+- Avoid expensive render scans and auto-opening focus-capturing overlays.
+
+## Current refactor checkpoints
+
+- [x] Hide Windows console windows for background runner and child Pi workers.
+- [x] Make parallel research shard workers start in parallel instead of depending on a single discover worker.
+- [x] Keep direct-agent reconstruction gated by `workflow === "direct-agent"` only.
+- [x] Persist subagent records and recover terminal results after restart.
+- [x] Fail fast for unrecoverable persisted records without `runId` instead of hanging.
+- [x] Persist direct-agent model override into task state for background/resume reconstruction.
+
+## Remaining larger subsystem work
+
+- Consolidate subagent runtime into `src/subagents/*` or equivalent durable-first module.
+- Move model routing transparency into persisted task/subagent records: requested model, selected model, fallback chain, fallback reason.
+- Add real integration smoke scripts for Windows process visibility, async restart recovery, and multi-shard fanout.
+- Add adaptive planner repair/retry for invalid JSON instead of immediate block when safe.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-crew",
-  "version": "0.1.21",
+  "version": "0.1.22",
   "description": "Pi extension for coordinated AI teams, workflows, worktrees, and async task orchestration",
   "author": "baphuongna",
   "license": "MIT",

package/src/extension/team-tool.ts

@@ -5,7 +5,7 @@ import type { ExtensionContext } from "@mariozechner/pi-coding-agent";
 import { allAgents, discoverAgents } from "../agents/discover-agents.ts";
 import { allTeams, discoverTeams } from "../teams/discover-teams.ts";
 import { allWorkflows, discoverWorkflows } from "../workflows/discover-workflows.ts";
-import type { WorkflowConfig, WorkflowStep } from "../workflows/workflow-config.ts";
+import type { WorkflowConfig } from "../workflows/workflow-config.ts";
 import { effectiveAutonomousConfig, loadConfig, updateAutonomousConfig, updateConfig, type PiTeamsAutonomousConfig, type PiTeamsConfig } from "../config/config.ts";
 import { projectPiRoot, userPiRoot } from "../utils/paths.ts";
 import type { TeamToolParamsValue } from "../schema/team-tool-schema.ts";
@@ -47,6 +47,7 @@ import { appendLiveAgentControlRequest } from "../runtime/live-agent-control.ts"
 import { liveControlRealtimeMessage, publishLiveControlRealtime } from "../runtime/live-control-realtime.ts";
 import { formatTaskGraphLines, waitingReason } from "../runtime/task-display.ts";
 import { directTeamAndWorkflowFromRun } from "../runtime/direct-run.ts";
+import { expandParallelResearchWorkflow } from "../runtime/parallel-research.ts";
 
 export interface TeamToolDetails {
 	action: string;
@@ -172,48 +173,6 @@ function commandExists(command: string, args: string[]): { ok: boolean; detail:
 	return { ok: false, detail: output.error?.message ?? firstOutputLine(output.stdout, output.stderr) };
 }
 
-function sourcePiProjects(cwd: string): string[] {
-	const sourceDir = path.join(cwd, "Source");
-	try {
-		return fs.readdirSync(sourceDir, { withFileTypes: true })
-			.filter((entry) => entry.isDirectory() && entry.name.startsWith("pi-"))
-			.map((entry) => `Source/${entry.name}`)
-			.sort();
-	} catch {
-		return [];
-	}
-}
-
-function chunkProjects(projects: string[], target = 4): string[][] {
-	const chunks = Array.from({ length: Math.min(Math.max(1, target), Math.max(1, projects.length)) }, () => [] as string[]);
-	projects.forEach((project, index) => chunks[index % chunks.length]!.push(project));
-	return chunks.filter((chunk) => chunk.length > 0);
-}
-
-function expandParallelResearchWorkflow(workflow: WorkflowConfig, cwd: string): WorkflowConfig {
-	if (workflow.name !== "parallel-research") return workflow;
-	const projects = sourcePiProjects(cwd);
-	if (projects.length === 0) return workflow;
-	const chunks = chunkProjects(projects, Math.min(6, Math.max(4, Math.ceil(projects.length / 4))));
-	const exploreSteps: WorkflowStep[] = chunks.map((paths, index) => ({
-		id: `explore-shard-${index + 1}`,
-		role: "explorer",
-		dependsOn: ["discover"],
-		parallelGroup: "explore",
-		reads: paths,
-		task: [`Explore this dynamic shard for: {goal}`, "", "Paths:", ...paths.map((item) => `- ${item}`), "", "Focus on purpose, architecture, runtime/UI patterns, package config, docs, and lessons for pi-crew."].join("\n"),
-	}));
-	return {
-		...workflow,
-		steps: [
-			{ id: "discover", role: "explorer", task: `Discover and validate ${projects.length} pi-* projects for: {goal}\n\nProjects:\n${projects.map((item) => `- ${item}`).join("\n")}` },
-			...exploreSteps,
-			{ id: "synthesize", role: "analyst", dependsOn: exploreSteps.map((step) => step.id), task: "Synthesize all dynamic shard findings. Identify common patterns, gaps, and concrete recommendations." },
-			{ id: "write", role: "writer", dependsOn: ["synthesize"], output: "research-summary.md", task: "Write a concise final summary with evidence, risks, and actionable next steps." },
-		],
-	};
-}
-
 function effectiveRunConfig(base: PiTeamsConfig, rawOverride: unknown): PiTeamsConfig {
 	const patch = configPatchFromConfig(rawOverride);
 	return {

package/src/runtime/async-runner.ts

@@ -1,4 +1,4 @@
-import { spawn } from "node:child_process";
+import { spawn, type SpawnOptions } from "node:child_process";
 import { createRequire } from "node:module";
 import * as fs from "node:fs";
 import * as path from "node:path";
@@ -36,6 +36,16 @@ export interface SpawnBackgroundTeamRunResult {
 	logPath: string;
 }
 
+export function buildBackgroundSpawnOptions(manifest: TeamRunManifest, logFd: number): SpawnOptions {
+	return {
+		cwd: manifest.cwd,
+		detached: true,
+		stdio: ["ignore", logFd, logFd],
+		env: { ...process.env },
+		windowsHide: true,
+	};
+}
+
 export function spawnBackgroundTeamRun(manifest: TeamRunManifest): SpawnBackgroundTeamRunResult {
 	const runnerPath = path.join(path.dirname(fileURLToPath(import.meta.url)), "background-runner.ts");
 	const logPath = path.join(manifest.stateRoot, "background.log");
@@ -43,12 +53,7 @@ export function spawnBackgroundTeamRun(manifest: TeamRunManifest): SpawnBackgrou
 	const logFd = fs.openSync(logPath, "a");
 	const command = getBackgroundRunnerCommand(runnerPath, manifest.cwd, manifest.runId);
 	fs.appendFileSync(logPath, `[pi-crew] background loader=${command.loader}\n`, "utf-8");
-	const child = spawn(process.execPath, command.args, {
-		cwd: manifest.cwd,
-		detached: true,
-		stdio: ["ignore", logFd, logFd],
-		env: { ...process.env },
-	});
+	const child = spawn(process.execPath, command.args, buildBackgroundSpawnOptions(manifest, logFd));
 	child.unref();
 	fs.closeSync(logFd);
 	return { pid: child.pid, logPath };

package/src/runtime/background-runner.ts

@@ -7,6 +7,7 @@ import { loadConfig } from "../config/config.ts";
 import { executeTeamRun } from "./team-runner.ts";
 import { resolveCrewRuntime } from "./runtime-resolver.ts";
 import { directTeamAndWorkflowFromRun } from "./direct-run.ts";
+import { expandParallelResearchWorkflow } from "./parallel-research.ts";
 
 function argValue(name: string): string | undefined {
 	const index = process.argv.indexOf(name);
@@ -29,8 +30,9 @@ async function main(): Promise<void> {
 		const direct = directTeamAndWorkflowFromRun(manifest, tasks, agents);
 		const team = direct?.team ?? allTeams(discoverTeams(cwd)).find((candidate) => candidate.name === manifest.team);
 		if (!team) throw new Error(`Team '${manifest.team}' not found.`);
-		const workflow = direct?.workflow ?? allWorkflows(discoverWorkflows(cwd)).find((candidate) => candidate.name === manifest.workflow);
-		if (!workflow) throw new Error(`Workflow '${manifest.workflow ?? ""}' not found.`);
+		const baseWorkflow = direct?.workflow ?? allWorkflows(discoverWorkflows(cwd)).find((candidate) => candidate.name === manifest.workflow);
+		if (!baseWorkflow) throw new Error(`Workflow '${manifest.workflow ?? ""}' not found.`);
+		const workflow = expandParallelResearchWorkflow(baseWorkflow, cwd);
 		const loadedConfig = loadConfig(cwd);
 		const runtime = await resolveCrewRuntime(loadedConfig.config);
 		const executeWorkers = runtime.kind !== "scaffold";

package/src/runtime/child-pi.ts

@@ -1,4 +1,4 @@
-import { spawn, type ChildProcess } from "node:child_process";
+import { spawn, type ChildProcess, type SpawnOptions } from "node:child_process";
 import * as fs from "node:fs";
 import * as path from "node:path";
 import type { AgentConfig } from "../agents/agent-config.ts";
@@ -66,6 +66,16 @@ export interface ChildPiRunResult {
 	error?: string;
 }
 
+export function buildChildPiSpawnOptions(cwd: string, env: NodeJS.ProcessEnv): SpawnOptions {
+	return {
+		cwd,
+		env,
+		stdio: ["ignore", "pipe", "pipe"],
+		detached: process.platform !== "win32",
+		windowsHide: true,
+	};
+}
+
 function appendTranscript(input: ChildPiRunInput, line: string): void {
 	if (!input.transcriptPath) return;
 	fs.mkdirSync(path.dirname(input.transcriptPath), { recursive: true });
@@ -225,12 +235,7 @@ export async function runChildPi(input: ChildPiRunInput): Promise<ChildPiRunResu
 	const spawnSpec = getPiSpawnCommand(built.args);
 	try {
 		return await new Promise<ChildPiRunResult>((resolve) => {
-			const child = spawn(spawnSpec.command, spawnSpec.args, {
-				cwd: input.cwd,
-				env: { ...process.env, ...built.env },
-				stdio: ["ignore", "pipe", "pipe"],
-				detached: process.platform !== "win32",
-			});
+			const child = spawn(spawnSpec.command, spawnSpec.args, buildChildPiSpawnOptions(input.cwd, { ...process.env, ...built.env }));
 			if (child.pid) activeChildProcesses.set(child.pid, child);
 			let stdout = "";
 			let stderr = "";

package/src/runtime/concurrency.ts

@@ -15,7 +15,7 @@ export interface BatchConcurrencyDecision {
 }
 
 export function defaultWorkflowConcurrency(workflowName: string): number {
-	if (workflowName === "parallel-research") return 4;
+	if (workflowName === "parallel-research") return 6;
 	if (workflowName === "research") return 2;
 	if (workflowName === "implementation" || workflowName === "review" || workflowName === "default") return 2;
 	return 1;

package/src/runtime/parallel-research.ts

@@ -0,0 +1,44 @@
+import * as fs from "node:fs";
+import * as path from "node:path";
+import type { WorkflowConfig, WorkflowStep } from "../workflows/workflow-config.ts";
+
+export function sourcePiProjects(cwd: string): string[] {
+	const sourceDir = path.join(cwd, "Source");
+	try {
+		return fs.readdirSync(sourceDir, { withFileTypes: true })
+			.filter((entry) => entry.isDirectory() && entry.name.startsWith("pi-"))
+			.map((entry) => `Source/${entry.name}`)
+			.sort();
+	} catch {
+		return [];
+	}
+}
+
+export function chunkProjects(projects: string[], target = 6): string[][] {
+	const chunks = Array.from({ length: Math.min(Math.max(1, target), Math.max(1, projects.length)) }, () => [] as string[]);
+	projects.forEach((project, index) => chunks[index % chunks.length]!.push(project));
+	return chunks.filter((chunk) => chunk.length > 0);
+}
+
+export function expandParallelResearchWorkflow(workflow: WorkflowConfig, cwd: string): WorkflowConfig {
+	if (workflow.name !== "parallel-research") return workflow;
+	const projects = sourcePiProjects(cwd);
+	if (projects.length === 0) return workflow;
+	const chunks = chunkProjects(projects, Math.min(8, Math.max(4, Math.ceil(projects.length / 3))));
+	const exploreSteps: WorkflowStep[] = chunks.map((paths, index) => ({
+		id: `explore-shard-${index + 1}`,
+		role: "explorer",
+		parallelGroup: "explore",
+		reads: paths,
+		task: [`Explore this dynamic shard for: {goal}`, "", "Paths:", ...paths.map((item) => `- ${item}`), "", "Focus on purpose, architecture, runtime/UI patterns, package config, docs, and lessons for pi-crew."].join("\n"),
+	}));
+	return {
+		...workflow,
+		steps: [
+			{ id: "discover", role: "explorer", parallelGroup: "inventory", task: `Quickly inventory and validate ${projects.length} pi-* projects for: {goal}\n\nProjects:\n${projects.map((item) => `- ${item}`).join("\n")}\n\nDo not block shard work; summarize routing notes only.` },
+			...exploreSteps,
+			{ id: "synthesize", role: "analyst", dependsOn: exploreSteps.map((step) => step.id), task: "Synthesize all dynamic shard findings. Identify common patterns, gaps, and concrete recommendations. Use discover output if available, but prioritize completed shard outputs." },
+			{ id: "write", role: "writer", dependsOn: ["synthesize"], output: "research-summary.md", task: "Write a concise final summary with evidence, risks, and actionable next steps." },
+		],
+	};
+}

package/src/runtime/team-runner.ts

@@ -63,15 +63,25 @@ function mergeArtifacts(items: ArtifactDescriptor[]): ArtifactDescriptor[] {
 	return [...byPath.values()];
 }
 
-function mergeTaskUpdates(base: TeamTaskState[], results: Array<{ tasks: TeamTaskState[] }>): TeamTaskState[] {
+function isNonTerminalTaskStatus(status: TeamTaskState["status"]): boolean {
+	return status === "queued" || status === "running";
+}
+
+function shouldMergeTaskUpdate(current: TeamTaskState, updated: TeamTaskState): boolean {
+	// Parallel workers receive the same input snapshot. A later result may still
+	// contain stale queued/running copies of tasks that another worker already
+	// completed. Never let those stale snapshots regress durable task state.
+	if (!isNonTerminalTaskStatus(current.status) && isNonTerminalTaskStatus(updated.status)) return false;
+	return updated.status !== current.status || updated.finishedAt !== current.finishedAt || updated.startedAt !== current.startedAt || Boolean(updated.resultArtifact) || Boolean(updated.error) || Boolean(updated.modelAttempts?.length) || Boolean(updated.usage);
+}
+
+export function __test__mergeTaskUpdates(base: TeamTaskState[], results: Array<{ tasks: TeamTaskState[] }>): TeamTaskState[] {
 	let merged = base;
 	for (const result of results) {
 		for (const updated of result.tasks) {
 			const current = merged.find((task) => task.id === updated.id);
-			if (!current) continue;
-			if (updated.status !== current.status || updated.finishedAt || updated.startedAt || updated.resultArtifact || updated.error) {
-				merged = merged.map((task) => task.id === updated.id ? updated : task);
-			}
+			if (!current || !shouldMergeTaskUpdate(current, updated)) continue;
+			merged = merged.map((task) => task.id === updated.id ? updated : task);
 		}
 	}
 	return refreshTaskGraphQueues(merged);
@@ -341,7 +351,7 @@ export async function executeTeamRun(input: ExecuteTeamRunInput): Promise<{ mani
 			return runTeamTask({ manifest, tasks, task, step, agent, signal: input.signal, executeWorkers: input.executeWorkers, runtimeKind: input.runtime?.kind, runtimeConfig: input.runtimeConfig, parentContext: input.parentContext, parentModel: input.parentModel, modelRegistry: input.modelRegistry, modelOverride: input.modelOverride, limits: input.limits });
 		}));
 		manifest = { ...results.at(-1)!.manifest, artifacts: mergeArtifacts([manifest.artifacts, ...results.map((item) => item.manifest.artifacts)].flat()) };
-		tasks = mergeTaskUpdates(tasks, results);
+		tasks = __test__mergeTaskUpdates(tasks, results);
 		const injectedAfterBatch = injectAdaptivePlanIfReady({ manifest, tasks, workflow, team: input.team });
 		if (injectedAfterBatch.missingPlan) {
 			tasks = markBlocked(tasks, "Adaptive planner did not produce a valid subagent plan.");

package/teams/parallel-research.team.md

@@ -3,7 +3,7 @@ name: parallel-research
 description: Parallel research team for multi-project/source audits
 workspaceMode: single
 defaultWorkflow: parallel-research
-maxConcurrency: 4
+maxConcurrency: 6
 triggers: đọc sâu, deep read, deep research, source audit, multiple projects, parallel research, pi-*
 category: research
 cost: cheap

package/workflows/parallel-research.workflow.md

@@ -10,28 +10,24 @@ Discover the relevant files/projects for: {goal}. Return a shard plan with paths
 
 ## explore-core
 role: explorer
-dependsOn: discover
 parallelGroup: explore
 
 Explore the core/runtime shard from the discover output. Focus on architecture, package config, docs, and reusable patterns for: {goal}
 
 ## explore-ui
 role: explorer
-dependsOn: discover
 parallelGroup: explore
 
 Explore the UI/TUI/extension-interface shard from the discover output. Focus on widgets, overlays, commands, status bars, package config, docs, and reusable patterns for: {goal}
 
 ## explore-runtime
 role: explorer
-dependsOn: discover
 parallelGroup: explore
 
 Explore the worker/runtime/subagent/runtime-control shard from the discover output. Focus on process/session/runtime orchestration, event streams, logs, package config, docs, and reusable patterns for: {goal}
 
 ## explore-extensions
 role: explorer
-dependsOn: discover
 parallelGroup: explore
 
 Explore the extension bundle/small-package shard from the discover output. Focus on package config, extension registration, commands/tools, docs, and reusable patterns for: {goal}
@@ -40,7 +36,7 @@ Explore the extension bundle/small-package shard from the discover output. Focus
 role: analyst
 dependsOn: explore-core, explore-ui, explore-runtime, explore-extensions
 
-Synthesize all shard findings. Identify common patterns, gaps, and concrete recommendations.
+Synthesize all shard findings. Use discover output if available, but do not require it. Identify common patterns, gaps, and concrete recommendations.
 
 ## write
 role: writer