keystone-cli 1.2.0 → 1.3.0

package/README.md

@@ -34,11 +34,12 @@ Keystone allows you to define complex automation workflows using a simple YAML s
 
 ---
 
-## ✨ Features
+## <a id="features"></a>✨ Features
 
 - ⚡ **Local-First:** Built on Bun with a local SQLite database for state management.
 - 🧩 **Declarative:** Define workflows in YAML with automatic dependency tracking (DAG).
 - 🤖 **Agentic:** First-class support for LLM agents defined in Markdown with YAML frontmatter.
+- 🎯 **Dynamic Workflows:** LLM-driven orchestration where a supervisor generates and executes steps at runtime.
 - 🧑‍💻 **Human-in-the-Loop:** Support for manual approval and text input steps.
 - 🔄 **Resilient:** Built-in retries, timeouts, and state persistence. Resume failed or paused runs exactly where they left off.
 - 📊 **TUI Dashboard:** Built-in interactive dashboard for monitoring and managing runs.
@@ -51,7 +52,7 @@ Keystone allows you to define complex automation workflows using a simple YAML s
 
 ---
 
-## 🚀 Installation
+## <a id="installation"></a>🚀 Installation
 
 Ensure you have [Bun](https://bun.sh) installed.
 
@@ -89,7 +90,7 @@ source <(keystone completion bash)
 
 ---
 
-## 🚦 Quick Start
+## <a id="quick-start"></a>🚦 Quick Start
 
 ### 1. Initialize a Project
 ```bash
@@ -130,7 +131,7 @@ keystone ui
 
 ---
 
-## 🧰 Bundled Workflows
+## <a id="bundled-workflows"></a>🧰 Bundled Workflows
 
 `keystone init` seeds these workflows under `.keystone/workflows/` (and the agents they rely on under `.keystone/workflows/agents/`):
 
@@ -143,6 +144,7 @@ Top-level workflows (seeded in `.keystone/workflows/`):
 - `script-example.yaml`: Demonstrates sandboxed JavaScript execution.
 - `artifact-example.yaml`: Demonstrates artifact upload and download between steps.
 - `idempotency-example.yaml`: Demonstrates safe retries for side-effecting steps.
+- `dynamic-demo.yaml`: Demonstrates LLM-driven dynamic workflow orchestration where steps are generated at runtime.
 
 Sub-workflows (seeded in `.keystone/workflows/`):
 - `scaffold-plan.yaml`: Generates a file plan from `requirements` input.
@@ -157,13 +159,14 @@ Example runs:
 keystone run scaffold-feature
 keystone run decompose-problem -i problem="Add caching to the API" -i context="Node/Bun service"
 keystone run agent-handoff -i topic="billing" -i user="Ada"
+keystone run dynamic-demo -i task="Set up a Node.js project with TypeScript"
 ```
 
 Sub-workflows are used by the top-level workflows, but can be run directly if you want just one phase.
 
 ---
 
-## ⚙️ Configuration
+## <a id="configuration"></a>⚙️ Configuration
 
 Keystone loads configuration from project `.keystone/config.yaml` (and user-level config; see `keystone config show` for search order) to manage model providers and model mappings.
 
@@ -358,7 +361,7 @@ Or use the `keystone auth login` command to securely store them in your local ma
 
 ---
 
-## 📝 Workflow Example
+## <a id="workflow-example"></a>📝 Workflow Example
 
 Workflows are defined in YAML. Dependencies are automatically resolved based on the `needs` field, and **Keystone also automatically detects implicit dependencies** from your `${{ }}` expressions.
 
@@ -441,7 +444,7 @@ expression:
 
 ---
 
-## 🏗️ Step Types
+## <a id="step-types"></a>🏗️ Step Types
 
 Keystone supports several specialized step types:
 
@@ -521,6 +524,54 @@ Keystone supports several specialized step types:
   - `env` and `cwd` are required and must be explicit.
   - `input` is sent to stdin (objects/arrays are JSON-encoded).
   - Summary is parsed from stdout or a file at `KEYSTONE_ENGINE_SUMMARY_PATH` and stored as an artifact.
+- `git`: Execute git operations with automatic worktree management.
+  - Operations: `clone`, `checkout`, `pull`, `push`, `commit`, `worktree_add`, `worktree_remove`.
+  - `cleanup: true` automatically removes worktrees at workflow end.
+  ```yaml
+  - id: clone_repo
+    type: git
+    op: clone
+    url: https://github.com/example/repo.git
+    path: ./repo
+    branch: main
+    cleanup: true
+  ```
+- `dynamic`: LLM-driven workflow orchestration where a supervisor agent generates steps at runtime.
+  - The supervisor LLM creates a plan of steps that are then executed dynamically.
+  - Supports resumability - state is persisted after each generated step.
+  - Generated steps can be: `llm`, `shell`, `workflow`, `file`, or `request`.
+  - `goal`: High-level goal for the supervisor to accomplish (required).
+  - `context`: Additional context for planning.
+  - `prompt`: Custom supervisor prompt (overrides default).
+  - `supervisor`: Agent for planning (defaults to `keystone-architect`).
+  - `agent`: Default agent for generated LLM steps.
+  - `templates`: Role-to-agent mapping for specialized tasks.
+  - `maxSteps`: Maximum number of steps to generate.
+  - `concurrency`: Maximum number of steps to run in parallel (default: `1`).
+  - `confirmPlan`: Review and approve/modify the plan before execution (default: `false`).
+  - `maxReplans`: Number of automatic recovery attempts if the plan fails (default: `3`).
+  - `allowStepFailure`: Continue execution even if individual generated steps fail.
+  - `library`: A list of pre-defined step patterns available to the supervisor.
+  ```yaml
+  - id: implement_feature
+    type: dynamic
+    goal: "Implement user authentication with JWT"
+    context: "This is a Node.js Express application"
+    agent: keystone-architect
+    templates:
+      planner: "keystone-architect"
+      developer: "software-engineer"
+    maxSteps: 10
+    allowStepFailure: false
+  ```
+
+#### Dynamic Orchestration vs. Rigid Pipelines
+Traditional workflows often require complex multi-file decomposition (e.g., `decompose-problem.yaml` calling separate research, implementation, and review workflows). The `dynamic` step type replaces these rigid patterns with **Agentic Orchestration**:
+- **Simplified Structure**: A single `dynamic` step can replace multiple nested pipelines.
+- **Adaptive Execution**: The agent adjusts its plan based on real-time feedback and results from previous steps.
+- **Improved Resumability**: Each sub-step generated by the agent is persisted, allowing seamless resumption even inside long-running dynamic tasks.
+
+Use **Deterministic Workflows** (standard steps) for predictable, repeatable processes. Use **Dynamic Orchestration** for open-ended tasks where the specific steps cannot be known in advance.
 
 ### Human Steps in Non-Interactive Mode
 If stdin is not a TTY (CI, piped input), `human` steps suspend. Resume by providing an answer via inputs using the step id and `__answer`:
@@ -726,7 +777,7 @@ Until `strategy.matrix` is wired end-to-end, use explicit `foreach` with an arra
 
 ---
 
-## 🔧 Advanced Features
+## <a id="advanced-features"></a>🔧 Advanced Features
 
 ### Idempotency Keys
 
@@ -939,7 +990,7 @@ You can also define a workflow-level `compensate` step to handle overall cleanup
 
 ---
 
-## 🤖 Agent Definitions
+## <a id="agent-definitions"></a>🤖 Agent Definitions
 
 Agents are defined in Markdown files with YAML frontmatter, making them easy to read and version control.
 
@@ -1123,7 +1174,7 @@ In these examples, the agent will have access to all tools provided by the MCP s
 
 ---
 
-## 🛠️ CLI Commands
+## <a id="cli-commands"></a>🛠️ CLI Commands
 
 | Command | Description |
 | :--- | :--- |
@@ -1187,7 +1238,7 @@ Input keys passed via `-i key=val` must be alphanumeric/underscore and cannot be
 ### Dry Run
 `keystone run --dry-run` prints shell commands without executing them and skips non-shell steps (including human prompts). Outputs from skipped steps are empty, so conditional branches may differ from a real run.
 
-## 🛡️ Security
+## <a id="security"></a>🛡️ Security
 
 ### Shell Execution
 Keystone blocks shell commands that match common injection/destructive patterns (like `rm -rf /` or pipes to shells). To run them, set `allowInsecure: true` on the step. Prefer `${{ escape(...) }}` when interpolating user input.
@@ -1215,7 +1266,7 @@ Request steps enforce SSRF protections and require HTTPS by default. Cross-origi
 
 ---
 
-## 🏗️ Architecture
+## <a id="architecture"></a>🏗️ Architecture
 
 ```mermaid
 graph TD
@@ -1256,7 +1307,7 @@ graph TD
     LLM --> MCPClient[MCP Client]
 ```
 
-## 📂 Project Structure
+## <a id="project-structure"></a>📂 Project Structure
 
 - `src/cli.ts`: CLI entry point.
 - `src/db/`: SQLite persistence layer.
@@ -1271,6 +1322,6 @@ graph TD
 
 ---
 
-## 📄 License
+## <a id="license"></a>📄 License
 
 MIT

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "keystone-cli",
-  "version": "1.2.0",
+  "version": "1.3.0",
   "description": "A local-first, declarative, agentic workflow orchestrator built on Bun",
   "type": "module",
   "bin": {

package/src/commands/init.ts

@@ -21,6 +21,7 @@ import fullFeatureDemo from '../templates/basics/full-feature-demo.yaml' with {
 import idempotencyExample from '../templates/control-flow/idempotency-example.yaml' with {
   type: 'text',
 };
+import dynamicDemo from '../templates/dynamic-demo.yaml' with { type: 'text' };
 import artifactExample from '../templates/features/artifact-example.yaml' with { type: 'text' };
 import scriptExample from '../templates/features/script-example.yaml' with { type: 'text' };
 // Import templates
@@ -38,6 +39,9 @@ import decomposeReviewWorkflow from '../templates/scaffolding/decompose-review.y
   type: 'text',
 };
 import devWorkflow from '../templates/scaffolding/dev.yaml' with { type: 'text' };
+import dynamicDecomposeWorkflow from '../templates/scaffolding/dynamic-decompose.yaml' with {
+  type: 'text',
+};
 import reviewLoopWorkflow from '../templates/scaffolding/review-loop.yaml' with { type: 'text' };
 import scaffoldWorkflow from '../templates/scaffolding/scaffold-feature.yaml' with { type: 'text' };
 import scaffoldGenerateWorkflow from '../templates/scaffolding/scaffold-generate.yaml' with {
@@ -102,6 +106,7 @@ const SEEDS = [
   { path: '.keystone/workflows/scaffold-plan.yaml', content: scaffoldPlanWorkflow },
   { path: '.keystone/workflows/scaffold-generate.yaml', content: scaffoldGenerateWorkflow },
   { path: '.keystone/workflows/decompose-problem.yaml', content: decomposeWorkflow },
+  { path: '.keystone/workflows/dynamic-decompose.yaml', content: dynamicDecomposeWorkflow },
   { path: '.keystone/workflows/decompose-research.yaml', content: decomposeResearchWorkflow },
   { path: '.keystone/workflows/decompose-implement.yaml', content: decomposeImplementWorkflow },
   { path: '.keystone/workflows/decompose-review.yaml', content: decomposeReviewWorkflow },
@@ -120,6 +125,7 @@ const SEEDS = [
   { path: '.keystone/workflows/artifact-example.yaml', content: artifactExample },
   { path: '.keystone/workflows/idempotency-example.yaml', content: idempotencyExample },
   { path: '.keystone/workflows/full-feature-demo.yaml', content: fullFeatureDemo },
+  { path: '.keystone/workflows/dynamic-demo.yaml', content: dynamicDemo },
 ];
 
 export function registerInitCommand(program: Command): void {

package/src/db/dynamic-state-manager.test.ts

@@ -0,0 +1,319 @@
+/**
+ * Tests for DynamicStateManager
+ */
+import { afterEach, beforeEach, describe, expect, it } from 'bun:test';
+import { existsSync, mkdirSync, rmSync } from 'node:fs';
+import { join } from 'node:path';
+import {
+  type DynamicPlan,
+  DynamicStateManager,
+  type DynamicStepState,
+} from './dynamic-state-manager.ts';
+import { WorkflowDb } from './workflow-db.ts';
+
+describe('DynamicStateManager', () => {
+  let db: WorkflowDb;
+  let stateManager: DynamicStateManager;
+  const testDir = join(import.meta.dir, '.test-dynamic-state');
+  const testDbPath = join(testDir, 'test.db');
+
+  beforeEach(async () => {
+    // Clean up any existing test db
+    if (existsSync(testDir)) {
+      rmSync(testDir, { recursive: true });
+    }
+    mkdirSync(testDir, { recursive: true });
+
+    db = new WorkflowDb(testDbPath);
+    stateManager = new DynamicStateManager(db);
+
+    // Create a workflow run for foreign key constraint
+    await db.createRun('test-run-1', 'test-workflow', { input: 'value' });
+  });
+
+  afterEach(() => {
+    db.close();
+    if (existsSync(testDir)) {
+      rmSync(testDir, { recursive: true });
+    }
+  });
+
+  describe('create', () => {
+    it('should create a new dynamic state', async () => {
+      const state = await stateManager.create({
+        runId: 'test-run-1',
+        stepId: 'dynamic-step-1',
+        workflowId: 'wf-123',
+      });
+
+      expect(state.id).toBeDefined();
+      expect(state.runId).toBe('test-run-1');
+      expect(state.stepId).toBe('dynamic-step-1');
+      expect(state.workflowId).toBe('wf-123');
+      expect(state.status).toBe('planning');
+      expect(state.generatedPlan.steps).toEqual([]);
+      expect(state.currentStepIndex).toBe(0);
+      expect(state.startedAt).toBeDefined();
+    });
+
+    it('should create state defaulting workflowId to runId', async () => {
+      const state = await stateManager.create({
+        runId: 'test-run-1',
+        stepId: 'dynamic-step-2',
+      });
+
+      expect(state.workflowId).toBe('test-run-1');
+    });
+  });
+
+  describe('load', () => {
+    it('should load existing state', async () => {
+      const created = await stateManager.create({
+        runId: 'test-run-1',
+        stepId: 'dynamic-step-1',
+      });
+
+      const loaded = await stateManager.load('test-run-1', 'dynamic-step-1');
+
+      expect(loaded).not.toBeNull();
+      expect(loaded?.id).toBe(created.id);
+      expect(loaded?.status).toBe('planning');
+    });
+
+    it('should return null for non-existent state', async () => {
+      const loaded = await stateManager.load('test-run-1', 'non-existent');
+      expect(loaded).toBeNull();
+    });
+  });
+
+  describe('loadById', () => {
+    it('should load state by ID', async () => {
+      const created = await stateManager.create({
+        runId: 'test-run-1',
+        stepId: 'dynamic-step-1',
+      });
+
+      if (!created.id) throw new Error('ID missing');
+      const loaded = await stateManager.loadById(created.id);
+
+      expect(loaded).not.toBeNull();
+      expect(loaded?.id).toBe(created.id);
+    });
+  });
+
+  describe('setPlan', () => {
+    it('should set the plan and create step executions', async () => {
+      const state = await stateManager.create({
+        runId: 'test-run-1',
+        stepId: 'dynamic-step-1',
+      });
+      if (!state.id) throw new Error('State ID missing');
+
+      const plan: DynamicPlan = {
+        steps: [
+          { id: 'step1', name: 'First step', type: 'shell', run: 'echo hello' },
+          {
+            id: 'step2',
+            name: 'Second step',
+            type: 'llm',
+            agent: 'test',
+            prompt: 'do something',
+            needs: ['step1'],
+          },
+        ],
+        notes: 'Test plan',
+      };
+
+      await stateManager.setPlan(state.id, plan);
+
+      // Verify state was updated
+      const loaded = await stateManager.loadById(state.id);
+      expect(loaded?.status).toBe('executing');
+      expect(loaded?.generatedPlan.steps.length).toBe(2);
+      expect(loaded?.generatedPlan.notes).toBe('Test plan');
+
+      // Verify step executions were created
+      const executions = await stateManager.getStepExecutions(state.id);
+      expect(executions.length).toBe(2);
+      expect(executions[0].stepId).toBe('step1');
+      expect(executions[0].status).toBe('pending');
+      expect(executions[0].executionOrder).toBe(0);
+      expect(executions[1].stepId).toBe('step2');
+      expect(executions[1].executionOrder).toBe(1);
+    });
+  });
+
+  describe('updateProgress', () => {
+    it('should update the current step index', async () => {
+      const state = await stateManager.create({
+        runId: 'test-run-1',
+        stepId: 'dynamic-step-1',
+      });
+      if (!state.id) throw new Error('State ID missing');
+
+      await stateManager.updateProgress(state.id, 3);
+
+      const loaded = await stateManager.loadById(state.id);
+      expect(loaded?.currentStepIndex).toBe(3);
+    });
+  });
+
+  describe('startStep and completeStep', () => {
+    it('should track step execution lifecycle', async () => {
+      const state = await stateManager.create({
+        runId: 'test-run-1',
+        stepId: 'dynamic-step-1',
+      });
+      if (!state.id) throw new Error('State ID missing');
+
+      const plan: DynamicPlan = {
+        steps: [{ id: 'step1', name: 'First step', type: 'shell', run: 'echo hello' }],
+      };
+      await stateManager.setPlan(state.id, plan);
+
+      // Start the step
+      await stateManager.startStep(state.id, 'step1');
+
+      let executions = await stateManager.getStepExecutions(state.id);
+      expect(executions[0].status).toBe('running');
+      expect(executions[0].startedAt).toBeDefined();
+
+      // Complete the step
+      await stateManager.completeStep(state.id, 'step1', {
+        status: 'success',
+        output: { result: 'hello' },
+      });
+
+      executions = await stateManager.getStepExecutions(state.id);
+      expect(executions[0].status).toBe('success');
+      expect(executions[0].output).toEqual({ result: 'hello' });
+      expect(executions[0].completedAt).toBeDefined();
+    });
+
+    it('should handle failed steps', async () => {
+      const state = await stateManager.create({
+        runId: 'test-run-1',
+        stepId: 'dynamic-step-1',
+      });
+      if (!state.id) throw new Error('State ID missing');
+
+      const plan: DynamicPlan = {
+        steps: [{ id: 'step1', name: 'First step', type: 'shell', run: 'exit 1' }],
+      };
+      await stateManager.setPlan(state.id, plan);
+      await stateManager.startStep(state.id, 'step1');
+
+      await stateManager.completeStep(state.id, 'step1', {
+        status: 'failed',
+        error: 'Command exited with code 1',
+      });
+
+      const executions = await stateManager.getStepExecutions(state.id);
+      expect(executions[0].status).toBe('failed');
+      expect(executions[0].error).toBe('Command exited with code 1');
+    });
+  });
+
+  describe('finish', () => {
+    it('should mark state as completed', async () => {
+      const state = await stateManager.create({
+        runId: 'test-run-1',
+        stepId: 'dynamic-step-1',
+      });
+      if (!state.id) throw new Error('State ID missing');
+
+      await stateManager.finish(state.id, 'completed');
+
+      const loaded = await stateManager.loadById(state.id);
+      expect(loaded?.status).toBe('completed');
+      expect(loaded?.completedAt).toBeDefined();
+    });
+
+    it('should mark state as failed with error', async () => {
+      const state = await stateManager.create({
+        runId: 'test-run-1',
+        stepId: 'dynamic-step-1',
+      });
+      if (!state.id) throw new Error('State ID missing');
+
+      await stateManager.finish(state.id, 'failed', 'Something went wrong');
+
+      const loaded = await stateManager.loadById(state.id);
+      expect(loaded?.status).toBe('failed');
+      expect(loaded?.error).toBe('Something went wrong');
+    });
+  });
+
+  describe('getStepResultsMap', () => {
+    it('should return completed steps as a map', async () => {
+      const state = await stateManager.create({
+        runId: 'test-run-1',
+        stepId: 'dynamic-step-1',
+      });
+      if (!state.id) throw new Error('State ID missing');
+
+      const plan: DynamicPlan = {
+        steps: [
+          { id: 'step1', name: 'First', type: 'shell', run: 'echo 1' },
+          { id: 'step2', name: 'Second', type: 'shell', run: 'echo 2' },
+        ],
+      };
+      await stateManager.setPlan(state.id, plan);
+
+      // Complete first step
+      await stateManager.startStep(state.id, 'step1');
+      await stateManager.completeStep(state.id, 'step1', {
+        status: 'success',
+        output: { value: 1 },
+      });
+
+      const resultsMap = await stateManager.getStepResultsMap(state.id);
+
+      expect(resultsMap.size).toBe(1); // Only completed steps
+      expect(resultsMap.get('step1')).toEqual({
+        output: { value: 1 },
+        status: 'success',
+        error: undefined,
+      });
+      expect(resultsMap.has('step2')).toBe(false); // Still pending
+    });
+  });
+
+  describe('listActive', () => {
+    it('should list active states', async () => {
+      await stateManager.create({
+        runId: 'test-run-1',
+        stepId: 'step-1',
+      });
+
+      const state2 = await stateManager.create({
+        runId: 'test-run-1',
+        stepId: 'step-2',
+      });
+
+      // Complete one
+      if (!state2.id) throw new Error('State ID missing');
+      await stateManager.finish(state2.id, 'completed');
+
+      const active = await stateManager.listActive();
+      expect(active.length).toBe(1);
+      expect(active[0].stepId).toBe('step-1');
+    });
+  });
+
+  describe('listByRun', () => {
+    it('should list states for a run', async () => {
+      await stateManager.create({
+        runId: 'test-run-1',
+        stepId: 'step-1',
+      });
+      await stateManager.create({
+        runId: 'test-run-1',
+        stepId: 'step-2',
+      });
+
+      const states = await stateManager.listByRun('test-run-1');
+      expect(states.length).toBe(2);
+    });
+  });
+});