keystone-cli 2.0.1 → 2.1.0

package/README.md

@@ -7,6 +7,7 @@
 [![Bun](https://img.shields.io/badge/Bun-%23000000.svg?style=flat&logo=bun&logoColor=white)](https://bun.sh)
 [![npm version](https://img.shields.io/npm/v/keystone-cli.svg?style=flat)](https://www.npmjs.com/package/keystone-cli)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/mhingston/keystone-cli)
 
 A local-first, declarative, agentic workflow orchestrator built on **Bun**.
 
@@ -427,6 +428,9 @@ expression:
   strict: true
 ```
 
+> [!NOTE]
+> When `strict: false` (default), evaluation errors in outputs will be reported as warnings and the value will be set to `null` to allow the workflow to potentially continue.
+
 ---
 
 ## <a id="step-types">🏗️ Step Types</a>
@@ -604,6 +608,9 @@ All steps support common features:
 - `outputRetries`: Max retries for output validation failures.
 - `repairStrategy`: Strategy for output repair (`reask`, `repair`, `hybrid`).
 
+> [!TIP]
+> **Performance Optimization**: For `foreach` steps with very large datasets, Keystone may automatically skip output aggregation to prevent memory issues. Use file-based storage or external databases if you need to process tens of thousands of items.
+
 Workflows also support a top-level `concurrency` field to limit how many steps can run in parallel across the entire workflow. This must resolve to a positive integer (number or expression).
 
 ### Engine Steps
@@ -678,6 +685,8 @@ Allow the LLM to switch to a specialist agent mid-step by defining `allowedHando
   allowedHandoffs: [handoff-specialist]
 ```
 
+To prevent infinite loops, handoffs are limited to **20** occurrences per step by default.
+
 Agent prompts can use `${{ }}` expressions (evaluated against the workflow context) for dynamic system prompts.
 
 ```markdown
@@ -1223,18 +1232,26 @@ Input keys passed via `-i key=val` must be alphanumeric/underscore and cannot be
 ## <a id="security">🛡️ Security</a>
 
 ### Shell Execution
-Keystone blocks shell commands that match common injection/destructive patterns (like `rm -rf /` or pipes to shells). To run them, set `allowInsecure: true` on the step. Prefer `${{ escape(...) }}` when interpolating user input.
+Keystone strictly enforces an allowlist of characters (`alphanumeric`, `whitespace`, and `_./:@,+=~-`) to prevent shell injection.
+
+- **Directory Traversal**: Commands containing `..` in a path are blocked by default for security.
+- **Denylist**: Commands like `rm`, `mkfs`, or `alias` are blocked via a configurable denylist in `config.yaml`, even if `allowInsecure: true` is set.
+- **Windows Support**: Keystone uses `cmd.exe /d /s /c` on Windows and `sh -c` on other platforms for consistent behavior.
 
+To run complex commands or bypass allowlist checks, set `allowInsecure: true` on the step. Prefer `${{ escape(...) }}` when interpolating user input.
+
+```yaml
 - id: deploy
   type: shell
   run: ./deploy.sh ${{ inputs.env }}
+  # Required if inputs.env might contain special characters or for complex scripts
   allowInsecure: true
 ```
 
 #### Troubleshooting Security Errors
-If you see a `Security Error: Evaluated command contains shell metacharacters`, it means your command contains characters like `\n`, `|`, or `&` that were not explicitly escaped or are not in the safe whitelist.
+If you see a `Security Error: Evaluated command contains shell metacharacters`, it means your command contains characters like `\n`, `|`, `&`, or quotes that are not in the strict allowlist.
 - **Fix 1**: Use `${{ escape(steps.id.output) }}` for any dynamic values.
-- **Fix 2**: Set `allowInsecure: true` if the command naturally uses special characters (like `echo "line1\nline2"`).
+- **Fix 2**: Set `allowInsecure: true` if the command naturally uses special characters.
 
 ### Expression Safety
 Expressions `${{ }}` are evaluated using a safe AST parser (`jsep`) which:
@@ -1288,8 +1305,14 @@ graph TD
     EX --> Wait[Wait Step]
     EX --> Join[Join Step]
     EX --> Blueprint[Blueprint Step]
+    EX --> Dynamic[Dynamic Executor]
+    EX --> Plan[Plan Executor]
 
-    LLM --> Adapter[LLM Adapter (AI SDK)]
+    subgraph "LLM Subsystem"
+        LLM --> ToolManager[Tool Manager]
+        LLM --> StreamHandler[Stream Handler]
+        ToolManager --> Adapter[LLM Adapter (AI SDK)]
+    end
     Adapter --> Providers[OpenAI, Anthropic, Gemini, Copilot, etc.]
     LLM --> MCPClient[MCP Client]
 ```
@@ -1297,10 +1320,13 @@ graph TD
 ## <a id="project-structure">📂 Project Structure</a>
 
 - `src/cli.ts`: CLI entry point.
+- `src/commands/`: Command implementations (run, ui, config, etc.).
 - `src/db/`: SQLite persistence layer.
 - `src/runner/`: The core execution engine, handles parallelization and retries.
 - `src/parser/`: Zod-powered validation for workflows and agents.
 - `src/expression/`: `${{ }}` expression evaluator.
+- `src/providers/`: Custom AI provider implementations.
+- `src/scripts/`: Build and utility scripts.
 - `src/templates/`: Bundled workflow and agent templates.
 - `src/ui/`: Ink-powered TUI dashboard.
 - `src/utils/`: Shared utilities (auth, redaction, config loading).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "keystone-cli",
-  "version": "2.0.1",
+  "version": "2.1.0",
   "description": "A local-first, declarative, agentic workflow orchestrator built on Bun",
   "type": "module",
   "bin": {
@@ -42,8 +42,10 @@
     "ink-spinner": "^5.0.0",
     "js-yaml": "^4.1.0",
     "jsep": "^1.4.0",
+    "minimatch": "^10.1.1",
     "react": "^19.0.0",
     "sqlite-vec": "0.1.6",
+    "yaml": "^2.8.2",
     "zod": "^3.25.76",
     "zod-to-json-schema": "^3.25.1"
   },
@@ -55,6 +57,7 @@
     "@types/bun": "^1.3.5",
     "@types/dagre": "^0.7.53",
     "@types/js-yaml": "^4.0.9",
+    "@types/minimatch": "^6.0.0",
     "@types/node": "^25.0.3",
     "react-devtools-core": "^7.0.1"
   },

package/src/cli.ts

@@ -57,6 +57,7 @@ registerGraphCommand(program);
 registerDocCommand(program);
 registerSchemaCommand(program);
 registerEventCommand(program);
+
 registerRunCommand(program);
 
 // Helper function used by remaining commands (rerun)

package/src/commands/event.ts

@@ -25,5 +25,14 @@ export function registerEventCommand(program: Command): void {
       }
       await db.storeEvent(name, data);
       console.log(`✓ Event '${name}' triggered.`);
+
+      // Check for workflows waiting for this event
+      const suspendedRunIds = await db.getSuspendedStepsForEvent(name);
+      if (suspendedRunIds.length > 0) {
+        console.log(`\nFound ${suspendedRunIds.length} workflow(s) waiting for this event:`);
+        for (const runId of suspendedRunIds) {
+          console.log(`  - Run ${runId}: Resume with \`keystone resume ${runId}\``);
+        }
+      }
     });
 }

package/src/commands/run.ts

@@ -24,6 +24,23 @@ export function registerRunCommand(program: Command): void {
     .option('--resume', 'Resume the last run of this workflow if it failed or was paused')
     .option('--explain', 'Show detailed error context with suggestions on failure')
     .action(async (workflowPathArg, options) => {
+      // Security Warning
+      if (!options.events) {
+        console.warn(
+          '\x1b[33m%s\x1b[0m',
+          '⚠️  SECURITY WARNING: This tool executes code from the current directory.'
+        );
+        console.warn(
+          '\x1b[33m%s\x1b[0m',
+          '   - Local provider scripts in ./providers/ are loaded and executed.'
+        );
+        console.warn(
+          '\x1b[33m%s\x1b[0m',
+          '   - Ensure you trust the code in this directory before running.'
+        );
+        console.warn('');
+      }
+
       const inputs = parseInputs(options.input);
       let resolvedPath: string | undefined;
 

package/src/db/dynamic-state-manager.ts

@@ -175,17 +175,20 @@ export class DynamicStateManager {
     const db = this.getDatabase();
     const now = new Date().toISOString();
 
-    // Load current state to get replanCount
-    const current = db
-      .prepare('SELECT replan_count FROM dynamic_workflow_state WHERE id = ?')
-      .get(stateId) as { replan_count: number };
-    const replanCount = current?.replan_count || 0;
-
-    db.prepare(`
+    // Use atomic SQL update to increment replan_count and set new plan
+    const result = db
+      .prepare(`
       UPDATE dynamic_workflow_state
-      SET generated_plan = ?, status = ?, updated_at = ?, replan_count = ?
+      SET generated_plan = ?, status = ?, updated_at = ?, replan_count = replan_count + 1
       WHERE id = ?
-    `).run(JSON.stringify(plan), status, now, replanCount, stateId);
+      RETURNING replan_count
+    `)
+      .get(JSON.stringify(plan), status, now, stateId) as { replan_count: number } | undefined;
+
+    if (!result) {
+      throw new Error(`Failed to update dynamic state: ${stateId}`);
+    }
+    const replanCount = result.replan_count;
 
     // Delete previous execution records IF this is a re-plan (optional, but cleaner)
     if (replanCount > 0) {

package/src/db/memory-db.test.ts

@@ -1,8 +1,14 @@
 import { afterAll, describe, expect, test } from 'bun:test';
 import * as fs from 'node:fs';
 import { MemoryDb } from './memory-db';
+import { setupSqlite } from './sqlite-setup';
 
-const TEST_DB = '.keystone/test-memory.db';
+import { randomUUID } from 'node:crypto';
+
+// Initialize SQLite with custom library for extensions
+setupSqlite();
+
+const TEST_DB = `.keystone/test-memory-${randomUUID()}.db`;
 
 describe('MemoryDb', () => {
   // Clean up previous runs
@@ -11,6 +17,7 @@ describe('MemoryDb', () => {
   }
 
   const db = new MemoryDb(TEST_DB);
+  console.log(`[MemoryDb Test] DB: ${TEST_DB}, Vector Ready: ${db.isVectorReady}`);
 
   afterAll(() => {
     db.close();
@@ -20,6 +27,7 @@ describe('MemoryDb', () => {
   });
 
   test('should initialize and store embedding', async () => {
+    if (!db.isVectorReady) return;
     const id = await db.store('hello world', Array(384).fill(0.1), { tag: 'test' });
     expect(id).toBeDefined();
     expect(typeof id).toBe('string');
@@ -32,6 +40,8 @@ describe('MemoryDb', () => {
 
     const db1536 = new MemoryDb(testDb1536, DIM_1536);
     try {
+      if (!db1536.isVectorReady) return;
+
       const id = await db1536.store('hi', Array(DIM_1536).fill(0.5));
       expect(id).toBeDefined();
 
@@ -56,6 +66,12 @@ describe('MemoryDb', () => {
 
     // Let's just test that we can use different dimensions on the same DB file.
     const db1 = new MemoryDb(testDbMismatch, 128);
+    if (!db1.isVectorReady) {
+      db1.close();
+      if (fs.existsSync(testDbMismatch)) fs.unlinkSync(testDbMismatch);
+      return;
+    }
+
     await db1.store('test128', Array(128).fill(0));
     db1.close();
 
@@ -71,6 +87,7 @@ describe('MemoryDb', () => {
   });
 
   test('should search and retrieve result', async () => {
+    if (!db.isVectorReady) return;
     // Store another item to search for
     await db.store('search target', Array(384).fill(0.9), { tag: 'target' });
 
@@ -81,6 +98,7 @@ describe('MemoryDb', () => {
   });
 
   test('should fail gracefully with invalid dimensions', async () => {
+    if (!db.isVectorReady) return;
     // sqlite-vec requires fixed dimensions (384 defined in schema)
     // bun:sqlite usually throws an error for constraint violations
     let error: unknown;

package/src/db/memory-db.ts

@@ -66,33 +66,83 @@ export class MemoryDb {
   // Cache connections by path to avoid reloading extensions
   private static connectionCache = new Map<string, { db: Database; refCount: number }>();
   private tableName: string;
+  private vectorReady = false;
+
+  get isVectorReady(): boolean {
+    return this.vectorReady;
+  }
+
+  /**
+   * Acquire a MemoryDb instance. This handles reference counting automatically.
+   */
+  static acquire(dbPath = '.keystone/memory.db', embeddingDimension = 384): MemoryDb {
+    const cached = MemoryDb.connectionCache.get(dbPath);
+    if (cached) {
+      cached.refCount++;
+      // We return a new instance but it shares the underlying DB connection
+      return new MemoryDb(dbPath, embeddingDimension, cached.db);
+    }
+
+    // Create new connection
+    const instance = new MemoryDb(dbPath, embeddingDimension);
+    MemoryDb.connectionCache.set(dbPath, { db: instance.db, refCount: 1 });
+    return instance;
+  }
 
   constructor(
     public readonly dbPath = '.keystone/memory.db',
-    private readonly embeddingDimension = 384
+    private readonly embeddingDimension = 384,
+    existingDb?: Database
   ) {
     // Ensure SQLite is set up with custom library on macOS (idempotent)
     setupSqlite();
 
     this.tableName = `vec_memory_${embeddingDimension}`;
-    const cached = MemoryDb.connectionCache.get(dbPath);
-    if (cached) {
-      cached.refCount++;
-      this.db = cached.db;
+
+    if (existingDb) {
+      this.db = existingDb;
     } else {
-      const dir = dirname(dbPath);
-      if (!existsSync(dir)) {
-        mkdirSync(dir, { recursive: true });
-      }
-      this.db = new Database(dbPath, { create: true });
+      // Check cache again in case direct constructor usage overlaps with cache
+      const cached = MemoryDb.connectionCache.get(dbPath);
+      if (cached) {
+        // This path shouldn't typically be hit if users use acquire(), but for safety:
+        cached.refCount++;
+        this.db = cached.db;
+      } else {
+        const dir = dirname(dbPath);
+        if (!existsSync(dir)) {
+          mkdirSync(dir, { recursive: true });
+        }
+        this.db = new Database(dbPath, { create: true });
+
+        // Load sqlite-vec extension
+        try {
+          const extensionPath = resolveSqliteVecPath();
+          this.db.loadExtension(extensionPath);
+        } catch (error) {
+          // In some environments (e.g. standard Bun builds), dynamic extension loading might be disabled.
+          // We log a warning and proceed without vector support.
+          new ConsoleLogger().warn(
+            `⚠️  Vector DB: Failed to load sqlite-vec extension. Vector search will be unavailable. Error: ${error instanceof Error ? error.message : String(error)}`
+          );
+        }
 
-      // Load sqlite-vec extension
-      const extensionPath = resolveSqliteVecPath();
-      this.db.loadExtension(extensionPath);
+        this.initSchema();
 
-      this.initSchema();
+        // Seed cache
+        MemoryDb.connectionCache.set(dbPath, { db: this.db, refCount: 1 });
+      }
+    }
+  }
 
-      MemoryDb.connectionCache.set(dbPath, { db: this.db, refCount: 1 });
+  /**
+   * Manually increment reference count.
+   * Useful when passing an instance to another component that should also own it.
+   */
+  retain(): void {
+    const cached = MemoryDb.connectionCache.get(this.dbPath);
+    if (cached) {
+      cached.refCount++;
     }
   }
 
@@ -123,12 +173,29 @@ export class MemoryDb {
       }
     }
 
-    this.db.run(`
-      CREATE VIRTUAL TABLE IF NOT EXISTS ${this.tableName} USING vec0(
-        id TEXT PRIMARY KEY,
-        embedding FLOAT[${this.embeddingDimension}]
+    try {
+      this.db.run(`
+        CREATE VIRTUAL TABLE IF NOT EXISTS ${this.tableName} USING vec0(
+          id TEXT PRIMARY KEY,
+          embedding FLOAT[${this.embeddingDimension}]
+        );
+      `);
+
+      // Verify table actually exists (in case run() didn't throw but failed)
+      const tableExists = this.db
+        .prepare(`SELECT name FROM sqlite_master WHERE type='table' AND name='${this.tableName}'`)
+        .get();
+      this.vectorReady = !!tableExists;
+
+      if (!this.vectorReady) {
+        new ConsoleLogger().warn(`⚠️  Vector DB: Vector table '${this.tableName}' was not created.`);
+      }
+    } catch (error) {
+      this.vectorReady = false;
+      new ConsoleLogger().warn(
+        `⚠️  Vector DB: Failed to create vector table. Vector search will be unavailable. Error: ${error}`
       );
-    `);
+    }
 
     this.db.run(`
       CREATE TABLE IF NOT EXISTS memory_metadata (
@@ -219,6 +286,14 @@ export class MemoryDb {
     }));
   }
 
+  /**
+   * Release the connection. Decrements ref count and closes DB if 0.
+   * Alias for close() for backward compatibility.
+   */
+  release(): void {
+    this.close();
+  }
+
   close(): void {
     const cached = MemoryDb.connectionCache.get(this.dbPath);
     if (cached) {
@@ -228,8 +303,12 @@ export class MemoryDb {
         MemoryDb.connectionCache.delete(this.dbPath);
       }
     } else {
-      // Fallback if not in cache for some reason
-      this.db.close();
+      // Fallback if not in cache for some reason or already closed
+      try {
+        this.db.close();
+      } catch {
+        // ignore
+      }
     }
   }
 }