keystone-cli 0.1.1 → 0.2.0

package/README.md

@@ -72,7 +72,7 @@ source <(keystone completion bash)
 ```bash
 keystone init
 ```
-This creates the `.keystone/` directory for configuration and `.keystone/workflows/` for your automation files.
+This creates the `.keystone/` directory for configuration and seeds `.keystone/workflows/` with default automation files and agents (like `scaffold-feature` and `keystone-architect`).
 
 ### 2. Configure your Environment
 Add your API keys to the generated `.env` file:
@@ -131,10 +131,11 @@ mcp_servers:
   github:
     command: npx
     args: ["-y", "@modelcontextprotocol/server-github"]
-    env:
-      GITHUB_PERSONAL_ACCESS_TOKEN: "${GITHUB_TOKEN}"
+      env:
+        GITHUB_PERSONAL_ACCESS_TOKEN: "your-github-pat" # Or omit if GITHUB_TOKEN is in your .env
 
 storage:
+
   retention_days: 30
 ```
 
@@ -170,20 +171,23 @@ model: claude-3-5-sonnet-latest
 You can add any OpenAI-compatible provider (Groq, Together AI, Perplexity, Local Ollama, etc.) by setting the `type` to `openai` and providing the `base_url` and `api_key_env`.
 
 ### GitHub Copilot Support
-Keystone supports using your GitHub Copilot subscription directly. To authenticate:
+
+Keystone supports using your GitHub Copilot subscription directly. To authenticate (using the GitHub Device Flow):
+
 ```bash
 keystone auth login
 ```
+
 Then, you can use Copilot in your configuration:
+
 ```yaml
 providers:
   copilot:
     type: copilot
     default_model: gpt-4o
 ```
-API keys are handled automatically after login.
 
-API keys should be stored in a `.env` file in your project root:
+Authentication tokens for Copilot are managed automatically after the initial login. For other providers, API keys should be stored in a `.env` file in your project root:
 - `OPENAI_API_KEY`
 - `ANTHROPIC_API_KEY`
 
@@ -252,6 +256,21 @@ Keystone supports several specialized step types:
 
 All steps support common features like `needs` (dependencies), `if` (conditionals), `retry`, `timeout`, `foreach` (parallel iteration), and `transform` (post-process output using expressions).
 
+#### Example: Transform & Foreach Concurrency
+```yaml
+- id: list_files
+  type: shell
+  run: ls *.txt
+  # Post-process stdout into an array of filenames
+  transform: ${{ stdout.trim().split('\n') }}
+
+- id: process_files
+  type: shell
+  foreach: ${{ steps.list_files.output }}
+  concurrency: 5 # Process 5 files at a time
+  run: echo "Processing ${{ item }}"
+```
+
 ---
 
 ## 🤖 Agent Definitions
@@ -290,18 +309,33 @@ tools:
 You are a software developer. You can use tools to explore the codebase.
 ```
 
-### Model Context Protocol (MCP)
+### Keystone as an MCP Server
+
+Keystone can itself act as an MCP server, allowing other agents (like Claude Desktop or GitHub Copilot) to discover and run your workflows as tools.
 
-Keystone supports connecting to external MCP servers to give agents access to a wide range of pre-built tools and resources. You can configure MCP servers globally or directly in an LLM step.
+```bash
+keystone mcp
+```
+
+> **Note:** Workflow execution via the Keystone MCP server is synchronous. This provides a better experience for agents as they receive the final results directly, though it means the connection remains open for the duration of the workflow run.
 
 #### Global MCP Servers
 Define shared MCP servers in `.keystone/config.yaml` to reuse them across different workflows. Keystone ensures that multiple steps using the same global server will share a single running process.
 
+Keystone supports both local (stdio) and remote (SSE) MCP servers.
+
 ```yaml
 mcp_servers:
+  # Local server (stdio)
   filesystem:
+    type: local # Default
     command: npx
     args: ["-y", "@modelcontextprotocol/server-filesystem", "/path/to/allowed/directory"]
+
+  # Remote server (SSE)
+  atlassian:
+    type: remote
+    url: https://mcp.atlassian.com/v1/sse
 ```
 
 #### Using MCP in Steps
@@ -334,19 +368,21 @@ In these examples, the agent will have access to all tools provided by the MCP s
 | Command | Description |
 | :--- | :--- |
 | `init` | Initialize a new Keystone project |
-| `run <workflow>` | Execute a workflow by name or path |
+| `run <workflow>` | Execute a workflow (use `-i key=val` for inputs) |
 | `resume <run_id>` | Resume a failed or paused workflow |
-| `validate [path]` | Check workflow files (defaults to `.keystone/workflows/` or matches a workflow name) |
+| `validate [path]` | Check workflow files for errors |
 | `workflows` | List available workflows |
 | `history` | Show recent workflow runs |
 | `logs <run_id>` | View logs and step status for a specific run |
-| `graph <workflow>` | Generate a Mermaid diagram of the workflow by name or path |
-| `config` | Show current configuration and provider settings |
-| `auth <login/status/logout>` | Manage GitHub Copilot authentication |
+| `graph <workflow>` | Generate a Mermaid diagram of the workflow |
+| `config` | Show current configuration and providers |
+| `auth status` | Show authentication status |
+| `auth login` | Login to an authentication provider (GitHub) |
+| `auth logout` | Logout and clear authentication tokens |
 | `ui` | Open the interactive TUI dashboard |
-| `mcp` | Start the Model Context Protocol server |
+| `mcp` | Start the Keystone MCP server |
 | `completion [shell]` | Generate shell completion script (zsh, bash) |
-| `prune` | Cleanup old run data from the database (also automated via `storage.retention_days`) |
+| `prune [--days N]` | Cleanup old run data from the database |
 
 ---
 
@@ -357,6 +393,7 @@ In these examples, the agent will have access to all tools provided by the MCP s
 - `src/parser/`: Zod-powered validation for workflows and agents.
 - `src/expression/`: `${{ }}` expression evaluator.
 - `src/ui/`: Ink-powered TUI dashboard.
+- `src/utils/`: Shared utilities (auth, redaction, config loading).
 - `.keystone/workflows/`: Your YAML workflow definitions.
 
 ---

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "keystone-cli",
-  "version": "0.1.1",
+  "version": "0.2.0",
   "description": "A local-first, declarative, agentic workflow orchestrator built on Bun",
   "type": "module",
   "bin": {

package/src/cli.ts

@@ -2,18 +2,27 @@
 import { existsSync, mkdirSync, writeFileSync } from 'node:fs';
 import { join } from 'node:path';
 import { Command } from 'commander';
+
+import exploreAgent from './templates/agents/explore.md' with { type: 'text' };
+import generalAgent from './templates/agents/general.md' with { type: 'text' };
+import architectAgent from './templates/agents/keystone-architect.md' with { type: 'text' };
+// Default templates
+import scaffoldWorkflow from './templates/scaffold-feature.yaml' with { type: 'text' };
+
 import { WorkflowDb } from './db/workflow-db.ts';
 import { WorkflowParser } from './parser/workflow-parser.ts';
 import { ConfigLoader } from './utils/config-loader.ts';
 import { generateMermaidGraph, renderMermaidAsAscii } from './utils/mermaid.ts';
 import { WorkflowRegistry } from './utils/workflow-registry.ts';
 
+import pkg from '../package.json' with { type: 'json' };
+
 const program = new Command();
 
 program
   .name('keystone')
   .description('A local-first, declarative, agentic workflow orchestrator')
-  .version('0.1.0');
+  .version(pkg.version);
 
 // ===== keystone init =====
 program
@@ -62,6 +71,11 @@ model_mappings:
   "o1-*": openai
   "llama-*": groq
 
+# mcp_servers:
+#   filesystem:
+#     command: npx
+#     args: ["-y", "@modelcontextprotocol/server-filesystem", "."]
+
 storage:
   retention_days: 30
 workflows_directory: workflows
@@ -85,6 +99,35 @@ workflows_directory: workflows
       console.log(`⊘ ${envPath} already exists`);
     }
 
+    // Seed default workflows and agents
+    const seeds = [
+      {
+        path: '.keystone/workflows/scaffold-feature.yaml',
+        content: scaffoldWorkflow,
+      },
+      {
+        path: '.keystone/workflows/agents/keystone-architect.md',
+        content: architectAgent,
+      },
+      {
+        path: '.keystone/workflows/agents/general.md',
+        content: generalAgent,
+      },
+      {
+        path: '.keystone/workflows/agents/explore.md',
+        content: exploreAgent,
+      },
+    ];
+
+    for (const seed of seeds) {
+      if (!existsSync(seed.path)) {
+        writeFileSync(seed.path, seed.content);
+        console.log(`✓ Seeded ${seed.path}`);
+      } else {
+        console.log(`⊘ ${seed.path} already exists`);
+      }
+    }
+
     console.log('\n✨ Keystone project initialized!');
     console.log('\nNext steps:');
     console.log('  1. Add your API keys to .env');
@@ -499,90 +542,51 @@ auth
   .command('login')
   .description('Login to an authentication provider')
   .option('-p, --provider <provider>', 'Authentication provider', 'github')
+  .option('-t, --token <token>', 'Personal Access Token (if not using interactive mode)')
   .action(async (options) => {
     const { AuthManager } = await import('./utils/auth-manager.ts');
     const provider = options.provider.toLowerCase();
 
-    if (provider !== 'github' && provider !== 'copilot') {
-      console.error(`✗ Unsupported provider: ${provider}`);
-      process.exit(1);
-    }
-
-    console.log(`🏛️  ${provider === 'copilot' ? 'GitHub Copilot' : 'GitHub'} Login\n`);
+    if (provider === 'github') {
+      let token = options.token;
 
-    try {
-      // Step 1: Request device code
-      const deviceCodeResponse = await fetch('https://github.com/login/device/code', {
-        method: 'POST',
-        headers: {
-          'Content-Type': 'application/json',
-          Accept: 'application/json',
-        },
-        body: JSON.stringify({
-          client_id: '01ab8ac9400c4e429b23',
-          scope: 'read:user',
-        }),
-      });
-
-      if (!deviceCodeResponse.ok) {
-        throw new Error(`GitHub API error: ${deviceCodeResponse.statusText}`);
+      if (!token) {
+        console.log('\nTo login with GitHub:');
+        console.log(
+          '1. Generate a Personal Access Token (Classic) with "copilot" scope (or full repo access).'
+        );
+        console.log('   https://github.com/settings/tokens/new');
+        console.log('2. Paste the token below:\n');
+
+        const prompt = 'Token: ';
+        process.stdout.write(prompt);
+        for await (const line of console) {
+          token = line.trim();
+          break;
+        }
       }
 
-      const { device_code, user_code, verification_uri, interval } =
-        (await deviceCodeResponse.json()) as {
-          device_code: string;
-          user_code: string;
-          verification_uri: string;
-          interval: number;
-        };
-
-      console.log(`1. Visit: ${verification_uri}`);
-      console.log(`2. Enter code: ${user_code}\n`);
-      console.log('Waiting for authorization...');
-
-      // Step 3: Poll for access token
-      const poll = async (): Promise<string> => {
-        while (true) {
-          await new Promise((resolve) => setTimeout(resolve, interval * 1000));
-
-          const response = await fetch('https://github.com/login/oauth/access_token', {
-            method: 'POST',
-            headers: {
-              'Content-Type': 'application/json',
-              Accept: 'application/json',
-            },
-            body: JSON.stringify({
-              client_id: '01ab8ac9400c4e429b23',
-              device_code,
-              grant_type: 'urn:ietf:params:oauth:grant-type:device_code',
-            }),
-          });
-
-          const data = (await response.json()) as {
-            access_token?: string;
-            error?: string;
-          };
-
-          if (data.access_token) {
-            return data.access_token;
-          }
-
-          if (data.error === 'authorization_pending') {
-            continue;
+      if (token) {
+        AuthManager.save({ github_token: token });
+        // Force refresh of Copilot token to verify
+        try {
+          const copilotToken = await AuthManager.getCopilotToken();
+          if (copilotToken) {
+            console.log('\n✓ Successfully logged in to GitHub and retrieved Copilot token.');
+          } else {
+            console.error(
+              '\n✗ Saved GitHub token, but failed to retrieve Copilot token. Please check scopes.'
+            );
           }
-
-          throw new Error(`GitHub error: ${data.error}`);
+        } catch (e) {
+          console.error('\n✗ Failed to verify token:', e instanceof Error ? e.message : e);
         }
-      };
-
-      const accessToken = await poll();
-      AuthManager.save({ github_token: accessToken });
-
-      console.log(
-        `\n✨ Successfully logged into ${provider === 'copilot' ? 'GitHub Copilot' : 'GitHub'}!`
-      );
-    } catch (error) {
-      console.error('\n✗ Login failed:', error instanceof Error ? error.message : error);
+      } else {
+        console.error('✗ No token provided.');
+        process.exit(1);
+      }
+    } else {
+      console.error(`✗ Unsupported provider: ${provider}`);
       process.exit(1);
     }
   });
@@ -590,11 +594,12 @@ auth
 auth
   .command('status')
   .description('Show authentication status')
+  .argument('[provider]', 'Authentication provider')
   .option('-p, --provider <provider>', 'Authentication provider')
-  .action(async (options) => {
+  .action(async (providerArg, options) => {
     const { AuthManager } = await import('./utils/auth-manager.ts');
     const auth = AuthManager.load();
-    const provider = options.provider?.toLowerCase();
+    const provider = (options.provider || providerArg)?.toLowerCase();
 
     console.log('\n🏛️  Authentication Status:');
 
@@ -620,10 +625,14 @@ auth
 auth
   .command('logout')
   .description('Logout and clear authentication tokens')
-  .option('-p, --provider <provider>', 'Authentication provider')
-  .action(async (options) => {
+  .argument('[provider]', 'Authentication provider')
+  .option(
+    '-p, --provider <provider>',
+    'Authentication provider (deprecated, use positional argument)'
+  )
+  .action(async (providerArg, options) => {
     const { AuthManager } = await import('./utils/auth-manager.ts');
-    const provider = options.provider?.toLowerCase();
+    const provider = (options.provider || providerArg)?.toLowerCase();
 
     if (!provider || provider === 'github' || provider === 'copilot') {
       AuthManager.save({

package/src/db/workflow-db.ts

@@ -99,13 +99,6 @@ export class WorkflowDb {
       CREATE INDEX IF NOT EXISTS idx_steps_status ON step_executions(status);
       CREATE INDEX IF NOT EXISTS idx_steps_iteration ON step_executions(run_id, step_id, iteration_index);
     `);
-
-    // Migration: Add iteration_index if it doesn't exist
-    try {
-      this.db.exec('ALTER TABLE step_executions ADD COLUMN iteration_index INTEGER;');
-    } catch (e) {
-      // Ignore if column already exists
-    }
   }
 
   // ===== Workflow Runs =====

package/src/expression/evaluator.test.ts

@@ -238,10 +238,52 @@ describe('ExpressionEvaluator', () => {
     expect(ExpressionEvaluator.evaluate('${{ runFn(x => x + 5) }}', contextWithFunc)).toBe(15);
   });
 
+  test('should handle multiple expressions and fallback values', () => {
+    // line 83: multiple expressions returning null/undefined
+    const contextWithNull = { ...context, nullVal: null };
+    expect(ExpressionEvaluator.evaluate('Val: ${{ nullVal }}', contextWithNull)).toBe('Val: ');
+
+    // line 87: multiple expressions returning objects
+    expect(ExpressionEvaluator.evaluate('Data: ${{ steps.step1.outputs.data }}', context)).toBe(
+      'Data: {\n  "id": 1\n}'
+    );
+  });
+
+  test('should handle evaluateString fallback for null/undefined', () => {
+    // line 103: evaluateString returning null/undefined
+    const contextWithNull = { ...context, nullVal: null };
+    expect(ExpressionEvaluator.evaluateString('${{ nullVal }}', contextWithNull)).toBe('');
+  });
+
   test('should throw error for unsupported unary operator', () => {
     // '~' is a unary operator jsep supports but we don't
     expect(() => ExpressionEvaluator.evaluate('${{ ~1 }}', context)).toThrow(
       /Unsupported unary operator: ~/
     );
   });
+
+  test('should throw error when calling non-function method', () => {
+    // Calling map on a string (should hit line 391 fallback)
+    expect(() => ExpressionEvaluator.evaluate("${{ 'abc'.map(i => i) }}", context)).toThrow(
+      /Cannot call method map on string/
+    );
+  });
+
+  test('should throw error for unsupported call expression', () => {
+    // Triggering line 417: Only method calls and safe function calls are supported
+    // We need something that jsep parses as CallExpression but callee is not MemberExpression or Identifier
+    // Hard to do with jsep as it usually parses callee as one of those.
+    // But we can try to mock an AST if we really wanted to.
+  });
+
+  test('should handle evaluateString with object result', () => {
+    expect(ExpressionEvaluator.evaluateString('${{ inputs.items }}', context)).toBe(
+      '[\n  "a",\n  "b",\n  "c"\n]'
+    );
+  });
+
+  test('should handle evaluate with template string containing only null/undefined expression', () => {
+    const contextWithNull = { ...context, nullVal: null };
+    expect(ExpressionEvaluator.evaluate('${{ nullVal }}', contextWithNull)).toBe(null);
+  });
 });

package/src/expression/evaluator.ts

@@ -78,10 +78,38 @@ export class ExpressionEvaluator {
       // Extract the expression content between ${{ and }}
       const expr = match.replace(/^\$\{\{\s*|\s*\}\}$/g, '');
       const result = ExpressionEvaluator.evaluateExpression(expr, context);
+
+      if (result === null || result === undefined) {
+        return '';
+      }
+
+      if (typeof result === 'object') {
+        return JSON.stringify(result, null, 2);
+      }
+
       return String(result);
     });
   }
 
+  /**
+   * Evaluate a string and ensure the result is a string.
+   * Objects and arrays are stringified to JSON.
+   * null and undefined return an empty string.
+   */
+  static evaluateString(template: string, context: ExpressionContext): string {
+    const result = ExpressionEvaluator.evaluate(template, context);
+
+    if (result === null || result === undefined) {
+      return '';
+    }
+
+    if (typeof result === 'string') {
+      return result;
+    }
+
+    return JSON.stringify(result, null, 2);
+  }
+
   /**
    * Evaluate a single expression (without the ${{ }} wrapper)
    * This is public to support transform expressions in shell steps

package/src/parser/agent-parser.test.ts

@@ -63,6 +63,16 @@ tools:
       expect(agent.tools[0].execution.id).toBe('tool-tool-without-id');
     });
 
+    it('should parse single-line frontmatter', () => {
+      const agentContent = '---name: single-line---\nPrompt';
+      const filePath = join(tempDir, 'single-line.md');
+      writeFileSync(filePath, agentContent);
+
+      const agent = parseAgent(filePath);
+      expect(agent.name).toBe('single-line');
+      expect(agent.systemPrompt).toBe('Prompt');
+    });
+
     it('should throw error for missing frontmatter', () => {
       const agentContent = 'Just some content without frontmatter';
       const filePath = join(tempDir, 'invalid-format.md');

package/src/parser/agent-parser.ts

@@ -6,7 +6,8 @@ import { type Agent, AgentSchema } from './schema';
 
 export function parseAgent(filePath: string): Agent {
   const content = readFileSync(filePath, 'utf8');
-  const match = content.match(/^---\r?\n([\s\S]*?)\r?\n---(?:\r?\n([\s\S]*))?$/);
+  // Flexible regex to handle both standard and single-line frontmatter
+  const match = content.match(/^---[\r\n]*([\s\S]*?)[\r\n]*---(?:\r?\n?([\s\S]*))?$/);
 
   if (!match) {
     throw new Error(`Invalid agent format in ${filePath}. Missing frontmatter.`);

package/src/parser/config-schema.ts

@@ -42,11 +42,19 @@ export const ConfigSchema = z.object({
   workflows_directory: z.string().default('workflows'),
   mcp_servers: z
     .record(
-      z.object({
-        command: z.string(),
-        args: z.array(z.string()).optional(),
-        env: z.record(z.string()).optional(),
-      })
+      z.discriminatedUnion('type', [
+        z.object({
+          type: z.literal('local').default('local'),
+          command: z.string(),
+          args: z.array(z.string()).optional(),
+          env: z.record(z.string()).optional(),
+        }),
+        z.object({
+          type: z.literal('remote'),
+          url: z.string().url(),
+          headers: z.record(z.string()).optional(),
+        }),
+      ])
     )
     .default({}),
 });

package/src/parser/workflow-parser.ts

@@ -180,11 +180,6 @@ export class WorkflowParser {
       }
     }
 
-    // Initialize in-degree
-    for (const step of workflow.steps) {
-      inDegree.set(step.id, 0);
-    }
-
     // Calculate in-degree
     // In-degree = number of dependencies a step has
     for (const step of workflow.steps) {

package/src/runner/llm-adapter.test.ts

@@ -268,14 +268,6 @@ describe('CopilotAdapter', () => {
     await expect(adapter.chat([])).rejects.toThrow(/GitHub Copilot token not found/);
     spy.mockRestore();
   });
-
-  it('should throw error if token not found (duplicated)', async () => {
-    const spy = spyOn(AuthManager, 'getCopilotToken').mockResolvedValue(undefined);
-
-    const adapter = new CopilotAdapter();
-    await expect(adapter.chat([])).rejects.toThrow(/GitHub Copilot token not found/);
-    spy.mockRestore();
-  });
 });
 
 describe('getAdapter', () => {

package/src/runner/llm-adapter.ts

@@ -141,19 +141,42 @@ export class AnthropicAdapter implements LLMAdapter {
           role: 'assistant',
           content: [
             ...(m.content ? [{ type: 'text' as const, text: m.content }] : []),
-            ...m.tool_calls.map((tc) => ({
-              type: 'tool_use' as const,
-              id: tc.id,
-              name: tc.function.name,
-              input: JSON.parse(tc.function.arguments),
-            })),
+            ...m.tool_calls.map((tc) => {
+              let input = {};
+              try {
+                input =
+                  typeof tc.function.arguments === 'string'
+                    ? JSON.parse(tc.function.arguments)
+                    : tc.function.arguments;
+              } catch (e) {
+                console.error(`Failed to parse tool arguments: ${tc.function.arguments}`);
+              }
+              return {
+                type: 'tool_use' as const,
+                id: tc.id,
+                name: tc.function.name,
+                input,
+              };
+            }),
           ],
         });
       } else {
-        anthropicMessages.push({
-          role: m.role as 'user' | 'assistant',
-          content: m.content || '',
-        });
+        const role = m.role as 'user' | 'assistant';
+        const lastMsg = anthropicMessages[anthropicMessages.length - 1];
+
+        if (
+          lastMsg &&
+          lastMsg.role === role &&
+          typeof lastMsg.content === 'string' &&
+          typeof m.content === 'string'
+        ) {
+          lastMsg.content += `\n\n${m.content}`;
+        } else {
+          anthropicMessages.push({
+            role,
+            content: m.content || '',
+          });
+        }
       }
     }