@workflow-manager/runner 0.1.0 → 0.3.0

package/README.md

@@ -1,6 +1,23 @@
 # workflow-manager
 
-CLI runner for in-memory and Markdown/JSON workflow orchestration.
+**wfm** (workflow-manager) is a CLI orchestrator that runs multi-step workflows where the steps are executed by AI coding agents, with human approval gates in between. You define a workflow in a Markdown (YAML frontmatter) or JSON file, and `wfm run` executes it step by step — think of it as a CI pipeline for AI agents: declarative multi-agent orchestration with QA routing, approval gates, and a sharing registry, run from a single binary.
+
+The core ideas:
+
+- **Workflow definitions.** A workflow is a list of steps with stable keys, objectives, and `dependsOn` edges (the engine topologically orders them). Each step is one of three kinds: `task` (delegated to an agent), `approval` (a human checkpoint), or `system`. Steps can carry retry policies, timeouts, and validation rules (`human`, `external`, or `none`).
+- **Agent adapters.** Each task step picks an adapter that does the actual work: `pi-agent` (the default — drives the `pi` coding agent CLI with a built prompt plus `input.json`/`output.json` envelopes), `claude-code` (pipes a built prompt into the `claude` CLI), `opencode`, `codex` (not yet implemented, mock-routed), and `mock` (deterministic simulator for tests/authoring). Steps can declare skills, MCP endpoints, system prompts, and a model under `taskSpec.init`, which get injected into the agent's prompt or input file.
+- **The envelope protocol.** Every step execution returns a structured result: an execution status (`SUCCESS`, `FAILED`, `QA_REJECTED`, `YIELD_EXTERNAL`) plus a QA routing action (`PROCEED`, `RETRY_CURRENT`, `ROLLBACK_PREVIOUS`, `RESTART_ALL`). That's what lets the engine retry a step, roll back to the previous one, or restart the whole run based on what the agent reported.
+- **Human-in-the-loop control.** While a run is active, wfm starts a local HTTP attach API (token-protected, with SSE event streaming), so a waiting step can be resolved either in the terminal prompt or from another shell with `wfm approve` / `wfm resume` / `wfm cancel`.
+- **A registry.** `wfm publish` / `pull` / `search` / `auth` talk to a Supabase-backed remote registry (the `apps/remote-registry` web app in this repo) for sharing workflows, with skills bundled and SHA-256 verified. Runs also emit opt-in telemetry there.
+- **Supporting commands.** `wfm scaffold` writes a starter workflow, `wfm validate` checks the schema and dependency cycles, `wfm doctor` verifies the host has the needed CLIs and API keys before a run, `wfm agent` drops an AGENTS.md rules file, and `wfm man` shows the man page.
+
+Install the latest prebuilt CLI with:
+
+```bash
+curl -fsSL https://github.com/navio/workflow-manager/releases/latest/download/workflow-manager-installer.sh | bash
+```
+
+If `wfm` is not available immediately in the same terminal, run the shell reload command printed by the installer or open a new shell, then run `wfm --help`.
 
 ## What it does
 
@@ -8,15 +25,17 @@ CLI runner for in-memory and Markdown/JSON workflow orchestration.
 - Validates structure, dependencies, adapters, and validation modes
 - Executes workflow steps with deterministic run state transitions
 - Emits a full event timeline and JSON run result
+- Starts a local attach API for live run snapshots and SSE events
 - Records authenticated CLI run telemetry for success, failure, and workflow effectiveness
 - Publishes and pulls shared workflows from the remote registry
 
 ## Architecture
 
-- `src/index.ts`: CLI commands (`questions`, `scaffold`, `validate`, `run`)
+- `src/index.ts`: CLI commands (`doctor`, `agent`, `scaffold`, `validate`, `run`)
 - `src/parser.ts`: parsing + validation
 - `src/engine.ts`: execution loop, confirmations, retries, rollback/restart
-- `src/mockExecutor.ts`: mock step executor for simulation
+- `src/piAgentExecutor.ts`: default executor driving the `pi` coding agent CLI
+- `src/mockExecutor.ts`: explicit mock step executor for simulation
 - `src/events.ts`: event sequencing/logging
 - `src/types.ts`: contracts and status enums
 - `apps/remote-registry/`: React + Vite remote registry app
@@ -31,9 +50,13 @@ bun install
 bun run build
 bun link
 
+wfm doctor
+wfm agent ./AGENTS.md
 wfm scaffold ./example-workflow.md
 wfm validate ./example-workflow.md
+wfm doctor ./example-workflow.md
 wfm run ./example-workflow.md --auto-confirm-all
+wfm run ./example-workflow.md --auto-confirm-all --verbose
 
 # JSON workflow support
 wfm scaffold ./example-workflow.json --format json
@@ -47,13 +70,58 @@ wfm publish ./example-workflow.json --visibility public --tag storytelling,examp
 wfm pull alice/remote-bunny --output ./remote-bunny.json
 ```
 
+Task steps run with the `pi-agent` adapter by default when `taskSpec.adapterKey` is omitted. The runner drives the [pi coding agent](https://github.com/earendil-works/pi-mono) CLI (`pi`) in non-interactive print mode: it builds a prompt from the step objective, workflow context, previous step output, and `taskSpec.init` context, and maps `taskSpec.init.model` to `--model`, each `taskSpec.init.systemPrompts` entry to `--append-system-prompt`, and each resolved skill to a `--skill` file written into the run directory. The full structured step input is also written to `input.json` in the run directory, and the agent is asked to write a result envelope to `output.json` for QA routing. If the agent exits successfully without writing one, the step succeeds and its response text becomes the step output. Set `WFM_PI_AGENT_COMMAND=/path/to/pi` to override the default `pi` binary, or set `taskSpec.payload.command` and `taskSpec.payload.args` for a specific step; custom commands receive the input/output file paths in the `WFM_PI_INPUT_FILE` and `WFM_PI_OUTPUT_FILE` environment variables (previous releases passed `--input`/`--output` flags instead). Use `adapterKey: mock` explicitly for deterministic simulation workflows.
+
+Before execution starts, `wfm run` validates host runtime access for real adapters. Default `pi-agent` steps require the configured `pi` command to be installed and executable; pi manages provider credentials in its own auth store, so no API key environment variables are inferred for pi steps. Real `opencode` steps require the `opencode` CLI, and real `claude-code` steps require the `claude` CLI. For those adapters, if `taskSpec.init.model` or `taskSpec.payload.model` identifies a known provider, the matching key must also be present: `OPENROUTER_API_KEY`, `OPENAI_API_KEY`, or `ANTHROPIC_API_KEY`. For custom clients, set `taskSpec.payload.requiredEnv` to the environment variable names that must exist before the run can start.
+
+Use `wfm doctor` to inspect host adapter setup and `wfm doctor <workflow>` to validate a workflow's runtime requirements before running it. Today, `pi-agent` is the real default host adapter (driving the `pi` CLI), `mock` is a deterministic simulator, `opencode` and `claude-code` have opt-in real host paths, and `codex` is still mock-routed until a real executor is implemented.
+
+During `wfm run`, the CLI starts a local attach API on `127.0.0.1`. Use `--port <n>` to bind a fixed port or omit it to let the OS choose one. The CLI prints the attach base URL and bearer token before execution starts.
+
+By default, `wfm run` now prints live workflow progress to stderr with the current step, elapsed workflow time, and remaining step count. Pass `--verbose` to stream per-step agent output chunks and execution status updates while the workflow is running. When a human approval is required in an interactive terminal, the CLI now prints an approval summary and prompts for approve or cancel inline.
+
+Runner API endpoints include:
+
+- `GET /session`
+- `GET /runs/:runId`
+- `GET /runs/:runId/steps/:stepKey`
+- `GET /runs/:runId/logs`
+- `GET /runs/:runId/events` (SSE)
+- `POST /runs/:runId/approve`
+- `POST /runs/:runId/resume`
+- `POST /runs/:runId/cancel`
+
+Local control helpers are available too:
+
+```bash
+wfm approve --url http://127.0.0.1:43121 --token <token> --step review --actor alice --note "LGTM"
+wfm cancel --url http://127.0.0.1:43121 --token <token> --step review --actor alice --note "stop this run"
+```
+
+See `doc/guide/runner-api.md` for the full contract.
+
+Prefer the release binary instead of a source checkout:
+
+```bash
+curl -fsSL https://github.com/navio/workflow-manager/releases/latest/download/workflow-manager-installer.sh | bash
+```
+
+If `wfm` is not available immediately in the same terminal, run the shell reload command printed by the installer or open a new shell, then run `wfm --help`.
+
 ## Build
 
 ```bash
+bun run lint
 bun run build
 bun test
 ```
 
+Apply safe lint fixes:
+
+```bash
+bun run lint:fix
+```
+
 Build a standalone Bun binary:
 
 ```bash
@@ -104,16 +172,41 @@ bun run docs:build
 bun run docs:preview
 ```
 
-Docs are now treated as a manual Netlify release flow. Build `doc/.vitepress/dist` locally and deploy it manually when needed.
+Docs can be deployed from `doc/.vitepress/dist` locally, and GitHub Pages publishes them from `main` when files under `doc/` change.
 
 Remote registry app:
 
 ```bash
 bun run remote-registry:dev
+bun --cwd apps/remote-registry lint
+bun run remote-registry:test
+bun run remote-registry:test:auth:local
+bun run remote-registry:test:publish:local
+bun run remote-registry:test:smoke:local
 bun run remote-registry:build
 ```
 
-`workflow-manager-ui` is the Netlify auto-release target. The root `netlify.toml` now builds `apps/remote-registry/` so connected Netlify Git deploys can produce preview deploys for PRs and production deploys from merges to `main`.
+Pre-commit hooks run staged-file linting automatically after `bun install` via the repo `prepare` script and `lint-staged`.
+
+Supabase local validation:
+
+```bash
+bun run supabase:start
+bun run supabase:db:reset
+bun run supabase:db:lint
+bun run supabase:test
+bun run supabase:stop
+```
+
+GitHub Actions now owns the production Supabase release flow:
+
+- `.github/workflows/supabase-validate.yml` validates `supabase/**` changes on pull requests
+- `.github/workflows/supabase-release.yml` applies migrations and deploys Edge Functions after merge to `main`
+- configure repository secrets: `SUPABASE_ACCESS_TOKEN`, `SUPABASE_DB_PASSWORD`, and `SUPABASE_PROJECT_ID`
+`remote-registry:test:auth:local` is an opt-in live smoke that validates signup, email confirmation, signin, and password-reset email against a local Supabase stack (`supabase start`) and Mailpit (`http://127.0.0.1:54324`).
+`remote-registry:test:publish:local` is an opt-in live smoke that validates handle claim and publish/search/pull owner-slug retrieval against the same local stack.
+
+The docs site is published at `https://navio.github.io/workflow-manager/` via `.github/workflows/deploy-docs.yml`.
 
 Manual help:
 
@@ -121,6 +214,12 @@ Manual help:
 wfm man
 ```
 
+Agent rules:
+
+```bash
+wfm agent ./AGENTS.md
+```
+
 Remote registry commands:
 
 ```bash
@@ -145,6 +244,14 @@ npx @tanstack/intent@latest list
 npx @tanstack/intent@latest install
 ```
 
+Workflow skill resolution now follows a local-authoring + portable-artifact model:
+
+- authoring workflows can point to local skill files under `./skills/**/SKILL.md`
+- `workflow-manager publish` inlines skill markdown into `skills[*].content`
+- publish also writes `skills[*].contentSha256` for integrity checks
+- pulled workflows are rejected if any declared skill is missing embedded content
+- optional `skills[*].upstream` metadata can record source repo/ref/path for auditability
+
 See `skills/README.md` for the packaged skill layout, TanStack Intent integration, and release checks.
 
 The deployed registry dashboard also supports browser-based token creation, workflow publishing, and creator analytics.
@@ -164,25 +271,29 @@ Current dashboard capabilities include:
 - Update docs under `doc/` when changing schema or runtime behavior
 - Add or update tests in `tests/` when touching parser or engine logic
 
-Netlify auto-release is configured for the UI in `netlify.toml`:
+Netlify is reserved for `workflow-manager-ui`.
 
-- base directory: `apps/remote-registry`
-- build command: `bun run build`
-- publish directory: `dist`
-- PRs: Deploy Previews
-- `main`: Production deploys for `workflow-manager-ui`
+- build command: `bun run netlify:build`
+- publish directory: `.netlify/deploy`
+- `workflow-manager-ui` must set `NETLIFY_SITE_TARGET=remote-ui`
 
-The docs site is no longer the auto-release target and should be deployed manually.
+GitHub Pages is reserved for docs and only deploys on changes under `doc/`.
 
 ## Release
 
 - Push a semantic tag like `v0.2.0` to trigger the GitHub Actions release workflow.
+- Merges to `main` that change packaged CLI files (`src/`, `skills/`, `man/`, `package.json`, `tsconfig.json`) update the release PR maintained by `.github/workflows/release-please.yml`.
+- Release Please uses Conventional Commit semantics to propose the next npm version: `fix:` -> patch, `feat:` -> minor, and `!` or `BREAKING CHANGE` -> major.
+- Merging the Release Please PR bumps `package.json`, updates `CHANGELOG.md`, triggers `.github/workflows/npm-publish.yml` to publish `@workflow-manager/runner` to npm, and creates the release tag consumed by `.github/workflows/release.yml`.
+- Docs, remote-registry, and other non-package changes do not trigger release automation.
+- Configure the repository `NPM_TOKEN` secret with an npm automation token that can publish `@workflow-manager/runner`.
 - Publish the root npm package when you want the CLI runner and `skills/` bundle to ship together.
 - The workflow runs tests and build, then compiles binaries for:
   - macOS arm64: `wfm-macos-arm64`
   - Linux x64: `wfm-linux-x64`
   - Windows x64: `wfm-windows-x64.exe`
 - Assets are attached to the GitHub Release for that tag.
+- Each release also includes `workflow-manager-installer.sh`, so users can install `wfm` with `curl -fsSL https://github.com/navio/workflow-manager/releases/latest/download/workflow-manager-installer.sh | bash`.
 
 ## Documentation
 

package/dist/adapters.d.ts

@@ -0,0 +1,4 @@
+import type { AdapterKey } from "./types.js";
+export declare const DEFAULT_TASK_ADAPTER: AdapterKey;
+export declare const SUPPORTED_ADAPTERS: readonly AdapterKey[];
+export declare function resolveTaskAdapter(adapter?: AdapterKey): AdapterKey;

package/dist/adapters.js

@@ -0,0 +1,11 @@
+export const DEFAULT_TASK_ADAPTER = "pi-agent";
+export const SUPPORTED_ADAPTERS = [
+    "pi-agent",
+    "mock",
+    "opencode",
+    "codex",
+    "claude-code",
+];
+export function resolveTaskAdapter(adapter) {
+    return adapter ?? DEFAULT_TASK_ADAPTER;
+}

package/dist/claudeCodeExecutor.d.ts

@@ -0,0 +1,4 @@
+import type { InputEnvelope, OutputEnvelope, StepDefinition, StepExecutionHooks, WorkflowDefinition } from "./types.js";
+export declare function normalizeTimeout(value: unknown, fallbackMs?: number): number;
+export declare function shouldUseRealClaudeCode(step: StepDefinition): boolean;
+export declare function executeClaudeCodeStep(step: StepDefinition, input: InputEnvelope, attempt: number, workflow?: WorkflowDefinition, workflowFilePath?: string, hooks?: StepExecutionHooks): Promise<OutputEnvelope>;

package/dist/claudeCodeExecutor.js

@@ -0,0 +1,173 @@
+import { spawn } from "node:child_process";
+import { resolveSkill } from "./skillResolver.js";
+function asRecord(value) {
+    return value && typeof value === "object" ? value : {};
+}
+const previousOutputTextKeys = ["output", "storyMarkdown", "chapterMarkdown", "stdout"];
+function previousOutputTextSections(value) {
+    const record = asRecord(value);
+    const sections = [];
+    for (const key of previousOutputTextKeys) {
+        const text = record[key];
+        if (typeof text === "string" && text.trim()) {
+            sections.push(text);
+        }
+    }
+    if (sections.length > 0) {
+        return sections;
+    }
+    for (const [key, value] of Object.entries(record)) {
+        if (["stepKey", "attempt", "adapter", "command", "exitStatus", "mockResult"].includes(key))
+            continue;
+        if (typeof value === "string" && value.trim()) {
+            sections.push(`${key}: ${value}`);
+        }
+    }
+    return sections;
+}
+export function normalizeTimeout(value, fallbackMs = 120000) {
+    const timeout = Number(value ?? fallbackMs);
+    if (!Number.isFinite(timeout) || timeout <= 0)
+        return fallbackMs;
+    return Math.floor(timeout);
+}
+function buildPrompt(step, input, workflow, workflowFilePath) {
+    const payload = asRecord(step.taskSpec?.payload);
+    if (typeof payload.prompt === "string" && payload.prompt.trim()) {
+        return payload.prompt;
+    }
+    const parts = [];
+    const systemPrompts = input.priming_configuration.system_prompts;
+    if (systemPrompts.length > 0) {
+        parts.push(systemPrompts.join("\n"));
+    }
+    const skills = input.priming_configuration.required_skills;
+    if (skills.length > 0) {
+        const resolvedNames = [];
+        for (const name of skills) {
+            const resolved = workflow && workflowFilePath ? resolveSkill(name, workflow, workflowFilePath) : null;
+            if (resolved) {
+                parts.push(resolved.content);
+            }
+            else {
+                resolvedNames.push(name);
+            }
+        }
+        if (resolvedNames.length > 0) {
+            parts.push(`Apply the following skills: ${resolvedNames.join(", ")}`);
+        }
+    }
+    // Inject primitive user inputs (feature, ticket, etc.) — skip step output objects.
+    // Newlines stripped from values to reduce prompt injection surface.
+    const globalState = input.global_context.global_state;
+    const inputLines = Object.entries(globalState)
+        .filter(([, v]) => typeof v === "string" || typeof v === "number")
+        .map(([k, v]) => `${k}: ${String(v).replace(/[\n\r]/g, " ")}`);
+    if (inputLines.length > 0) {
+        parts.push(`Input:\n${inputLines.join("\n")}`);
+    }
+    parts.push(input.step_context.step_objective);
+    // Inject output from previous steps so context flows forward
+    const prev = input.step_context.previous_output;
+    for (const [key, val] of Object.entries(prev)) {
+        const sections = previousOutputTextSections(val);
+        if (sections.length > 0) {
+            parts.push(`Output from ${key}:\n${sections.join("\n\n")}`);
+        }
+    }
+    const context = input.priming_configuration.context;
+    if (typeof context === "string" && context.trim()) {
+        parts.push(`Context:\n${context}`);
+    }
+    else if (context && typeof context === "object") {
+        const str = JSON.stringify(context, null, 2);
+        if (str !== "{}")
+            parts.push(`Context:\n${str}`);
+    }
+    return parts.join("\n\n");
+}
+export function shouldUseRealClaudeCode(step) {
+    const payload = asRecord(step.taskSpec?.payload);
+    return payload.useRealAdapter === true;
+}
+export function executeClaudeCodeStep(step, input, attempt, workflow, workflowFilePath, hooks) {
+    const startedAt = Date.now();
+    const payload = asRecord(step.taskSpec?.payload);
+    const timeoutMs = normalizeTimeout(payload.timeoutMs);
+    const prompt = buildPrompt(step, input, workflow, workflowFilePath);
+    const configuredModel = typeof input.priming_configuration.model === "string" && input.priming_configuration.model.trim()
+        ? input.priming_configuration.model
+        : typeof payload.model === "string" && payload.model.trim()
+            ? payload.model
+            : undefined;
+    const args = ["-p"];
+    if (configuredModel) {
+        args.push("--model", configuredModel);
+    }
+    const makeResult = (status, reason, extra = {}) => ({
+        step_id: step.key,
+        execution_status: status,
+        qa_routing: { action: "PROCEED", feedback_reason: reason },
+        mutated_payload: { stepKey: step.key, attempt, adapter: "claude-code", prompt, model: configuredModel, ...extra },
+        metadata: { execution_time_ms: Date.now() - startedAt, external_intervention_required: false },
+    });
+    return new Promise((resolve) => {
+        let child;
+        try {
+            child = spawn("claude", args);
+        }
+        catch (err) {
+            resolve(makeResult("FAILED", err.message));
+            return;
+        }
+        hooks?.onStarted?.({ command: "claude", args, model: configuredModel });
+        child.stdin?.on("error", () => undefined);
+        child.stdin?.end(prompt);
+        const outChunks = [];
+        const errChunks = [];
+        child.stdout?.on("data", (chunk) => {
+            const text = chunk.toString();
+            outChunks.push(text);
+            hooks?.onStdout?.(text);
+        });
+        child.stderr?.on("data", (chunk) => {
+            const text = chunk.toString();
+            errChunks.push(text);
+            hooks?.onStderr?.(text);
+        });
+        let timedOut = false;
+        const timer = setTimeout(() => {
+            timedOut = true;
+            child.kill("SIGTERM");
+            const result = makeResult("FAILED", `timed out after ${timeoutMs}ms`);
+            hooks?.onFinished?.({ executionStatus: result.execution_status, timedOut: true });
+            resolve(result);
+        }, timeoutMs);
+        child.on("error", (err) => {
+            clearTimeout(timer);
+            if (timedOut)
+                return;
+            const result = makeResult("FAILED", err.message);
+            hooks?.onFinished?.({ executionStatus: result.execution_status });
+            resolve(result);
+        });
+        child.on("close", (code) => {
+            clearTimeout(timer);
+            if (timedOut)
+                return;
+            const stdout = outChunks.join("");
+            const stderr = errChunks.join("");
+            const exitStatus = code ?? 1;
+            if (exitStatus !== 0) {
+                const result = makeResult("FAILED", `claude exited ${exitStatus}: ${stderr.trim()}`, { exitStatus, stdout, stderr });
+                hooks?.onFinished?.({ executionStatus: result.execution_status, exitStatus });
+                resolve(result);
+            }
+            else {
+                const result = makeResult("SUCCESS", "", { exitStatus, output: stdout.trim() });
+                hooks?.onFinished?.({ executionStatus: result.execution_status, exitStatus });
+                resolve(result);
+            }
+        });
+    });
+}

package/dist/cliRunRenderer.d.ts

@@ -0,0 +1,36 @@
+import type { RunEvent, RunObserver, RunSnapshot, RunnerLogChunk, StepDetailSnapshot, WorkflowDefinition } from "./types.js";
+interface CliRunRendererOptions {
+    workflow: WorkflowDefinition;
+    verbose: boolean;
+    stream?: NodeJS.WriteStream;
+    heartbeatMs?: number;
+}
+export declare class CliRunRenderer implements RunObserver {
+    private readonly verbose;
+    private readonly stream;
+    private readonly heartbeatMs;
+    private readonly steps;
+    private readonly lineBuffers;
+    private lastSnapshot;
+    private lastStepDetails;
+    private heartbeat;
+    private promptPauseCount;
+    private started;
+    private closed;
+    constructor(options: CliRunRendererOptions);
+    onSnapshot(snapshot: RunSnapshot, stepDetails: StepDetailSnapshot[]): void;
+    onEvent(event: RunEvent): void;
+    onLog(log: RunnerLogChunk): void;
+    close(): void;
+    pauseHeartbeat(): void;
+    resumeHeartbeat(): void;
+    private renderStepStatus;
+    private stepSummary;
+    private remainingText;
+    private syncHeartbeat;
+    private stopHeartbeat;
+    private renderHeartbeat;
+    private flushBuffers;
+    private write;
+}
+export {};