keystone-cli 0.1.0 → 0.2.0

package/README.md

@@ -1,136 +1,403 @@
+<p align="center">
+  <img src="logo.png" width="250" alt="Keystone CLI Logo">
+</p>
+
 # 🏛️ Keystone CLI
 
 [![Bun](https://img.shields.io/badge/Bun-%23000000.svg?style=flat&logo=bun&logoColor=white)](https://bun.sh)
-[![NPM Version](https://img.shields.io/npm/v/keystone-cli.svg?style=flat)](https://www.npmjs.com/package/keystone-cli)
+[![npm version](https://img.shields.io/npm/v/keystone-cli.svg?style=flat)](https://www.npmjs.com/package/keystone-cli)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-**Keystone** is a local-first, declarative, agentic workflow orchestrator built on **Bun**.
+A local-first, declarative, agentic workflow orchestrator built on **Bun**.
 
-It allows you to define complex automation workflows using a simple YAML syntax, featuring first-class support for LLM agents, persistent state management via SQLite, and high-concurrency execution with built-in resilience.
+Keystone allows you to define complex automation workflows using a simple YAML syntax, with first-class support for LLM agents, state persistence, and parallel execution.
 
 ---
 
-## ✨ Key Features
+## ✨ Features
 
-- ⚡ **Local-First & Fast:** Powered by Bun with a local SQLite database. No external "cloud state" required—your data and workflow history stay on your machine.
-- 🧩 **Declarative Workflows:** Define logic in YAML. Keystone automatically calculates the execution graph (DAG) and detects dependencies from your expressions.
-- 🤖 **Agentic by Design:** Seamlessly integrate LLM agents defined in Markdown. Agents can use tools, which are just other workflow steps.
-- 🔌 **Built-in MCP Server:** Expose your workflows as tools to other AI assistants (like Claude Desktop) using the Model Context Protocol.
-- 🔄 **Resilient Execution:** Built-in retries, exponential backoff, and timeouts. Interrupted workflows can be resumed exactly where they stopped.
-- 🧑‍💻 **Human-in-the-Loop:** Support for manual approval and text input steps for sensitive or creative operations.
-- 📊 **Interactive TUI:** A beautiful terminal dashboard to monitor concurrent runs and history.
-- 🛡️ **Security-First:** Automatic secret redaction from logs/database and AST-based safe expression evaluation.
+- ⚡ **Local-First:** Built on Bun with a local SQLite database for state management.
+- 🧩 **Declarative:** Define workflows in YAML with automatic dependency tracking (DAG).
+- 🤖 **Agentic:** First-class support for LLM agents defined in Markdown with YAML frontmatter.
+- 🧑‍💻 **Human-in-the-Loop:** Support for manual approval and text input steps.
+- 🔄 **Resilient:** Built-in retries, timeouts, and state persistence. Resume failed or paused runs exactly where they left off.
+- 📊 **TUI Dashboard:** Built-in interactive dashboard for monitoring and managing runs.
+- 🛠️ **Extensible:** Support for shell, file, HTTP request, LLM, and sub-workflow steps.
+- 🔌 **MCP Support:** Integrated Model Context Protocol server.
+- 🛡️ **Secret Redaction:** Automatically redacts environment variables and secrets from logs and outputs.
 
 ---
 
 ## 🚀 Installation
 
-Ensure you have [Bun](https://bun.sh) installed (v1.0.0 or higher).
+Ensure you have [Bun](https://bun.sh) installed.
+
+### Global Install (Recommended)
+```bash
+bun install -g keystone-cli
+```
 
+### From Source
 ```bash
-# Install globally via Bun
-bun add -g keystone-cli
+# Clone the repository
+git clone https://github.com/mhingston/keystone-cli.git
+cd keystone-cli
+
+# Install dependencies
+bun install
 
-# Or via NPM
-npm install -g keystone-cli
+# Link CLI globally
+bun link
 ```
 
 ### Shell Completion
 
-To enable tab completion for workflow names and commands:
+To enable tab completion for your shell, add the following to your `.zshrc` or `.bashrc`:
 
-**Zsh:** Add `source <(keystone completion zsh)` to your `.zshrc`
-**Bash:** Add `source <(keystone completion bash)` to your `.bashrc`
+**Zsh:**
+```bash
+source <(keystone completion zsh)
+```
+
+**Bash:**
+```bash
+source <(keystone completion bash)
+```
 
 ---
 
-## 🚥 Quick Start
+## 🚦 Quick Start
 
 ### 1. Initialize a Project
 ```bash
 keystone init
 ```
-This creates a `.keystone/` directory for configuration and a `workflows/` directory for your files.
+This creates the `.keystone/` directory for configuration and seeds `.keystone/workflows/` with default automation files and agents (like `scaffold-feature` and `keystone-architect`).
 
-### 2. Configure Environment
+### 2. Configure your Environment
 Add your API keys to the generated `.env` file:
 ```env
 OPENAI_API_KEY=sk-...
 ANTHROPIC_API_KEY=sk-ant-...
 ```
 
-### 3. Run Your First Workflow
+### 3. Run a Workflow
 ```bash
 keystone run basic-shell
 ```
+Keystone automatically looks in `.keystone/workflows/` (locally and in your home directory) for `.yaml` or `.yml` files.
+
+### 4. Monitor with the Dashboard
+```bash
+keystone ui
+```
+
+---
+
+## ⚙️ Configuration
+
+Keystone uses a local configuration file at `.keystone/config.yaml` to manage model providers and model mappings.
+
+```yaml
+default_provider: openai
+
+providers:
+  openai:
+    type: openai
+    base_url: https://api.openai.com/v1
+    api_key_env: OPENAI_API_KEY
+    default_model: gpt-4o
+  anthropic:
+    type: anthropic
+    base_url: https://api.anthropic.com/v1
+    api_key_env: ANTHROPIC_API_KEY
+    default_model: claude-3-5-sonnet-20240620
+  groq:
+    type: openai
+    base_url: https://api.groq.com/openai/v1
+    api_key_env: GROQ_API_KEY
+    default_model: llama-3.3-70b-versatile
+
+model_mappings:
+  "gpt-*": openai
+  "claude-*": anthropic
+  "o1-*": openai
+  "llama-*": groq
+
+mcp_servers:
+  filesystem:
+    command: npx
+    args: ["-y", "@modelcontextprotocol/server-filesystem", "/path/to/allowed/directory"]
+  github:
+    command: npx
+    args: ["-y", "@modelcontextprotocol/server-github"]
+      env:
+        GITHUB_PERSONAL_ACCESS_TOKEN: "your-github-pat" # Or omit if GITHUB_TOKEN is in your .env
+
+storage:
+
+  retention_days: 30
+```
+
+### Model & Provider Resolution
+
+Keystone resolves which provider to use for a model in the following order:
+
+1. **Explicit Provider:** Use the `provider` field in an agent or step definition.
+2. **Provider Prefix:** Use the `provider:model` syntax (e.g., `model: copilot:gpt-4o`).
+3. **Model Mappings:** Matches the model name against the `model_mappings` in your config (supports suffix `*` for prefix matching).
+4. **Default Provider:** Falls back to the `default_provider` defined in your config.
+
+#### Example: Explicit Provider in Agent
+**`.keystone/workflows/agents/summarizer.md`**
+```markdown
+---
+name: summarizer
+provider: anthropic
+model: claude-3-5-sonnet-latest
+---
+```
+
+#### Example: Provider Prefix in Step
+```yaml
+- id: notify
+  type: llm
+  agent: summarizer
+  model: copilot:gpt-4o
+  prompt: ...
+```
+
+### OpenAI Compatible Providers
+You can add any OpenAI-compatible provider (Groq, Together AI, Perplexity, Local Ollama, etc.) by setting the `type` to `openai` and providing the `base_url` and `api_key_env`.
+
+### GitHub Copilot Support
+
+Keystone supports using your GitHub Copilot subscription directly. To authenticate (using the GitHub Device Flow):
+
+```bash
+keystone auth login
+```
+
+Then, you can use Copilot in your configuration:
+
+```yaml
+providers:
+  copilot:
+    type: copilot
+    default_model: gpt-4o
+```
+
+Authentication tokens for Copilot are managed automatically after the initial login. For other providers, API keys should be stored in a `.env` file in your project root:
+- `OPENAI_API_KEY`
+- `ANTHROPIC_API_KEY`
 
 ---
 
-## ⚙️ How it Works
+## 📝 Workflow Example
 
-### Workflows (.yaml)
-Workflows are defined by steps. Steps run in **parallel** by default unless a dependency is defined via `needs` or detected in an expression like `${{ steps.previous_step.output }}`.
+Workflows are defined in YAML. Dependencies are automatically resolved based on the `needs` field, and **Keystone also automatically detects implicit dependencies** from your `${{ }}` expressions.
 
 ```yaml
-name: analyze-repo
+name: build-and-notify
+description: Build the project and notify the team
+
+inputs:
+  branch:
+    type: string
+    default: main
+
 steps:
-  - id: list_files
+  - id: checkout
+    type: shell
+    run: git checkout ${{ inputs.branch }}
+
+  - id: install
+    type: shell
+    # Implicit dependency on 'checkout' detected from expression below
+    if: ${{ steps.checkout.status == 'success' }}
+    run: bun install
+
+  - id: build
     type: shell
-    run: ls -R
-    transform: stdout.split('\n')
+    needs: [install] # Explicit dependency
+    run: bun run build
+    retry:
+      count: 3
+      backoff: exponential
 
-  - id: analyze
+  - id: notify
     type: llm
-    foreach: ${{ steps.list_files.output }}
-    concurrency: 5
-    agent: code-reviewer
-    prompt: "Analyze this file: ${{ item }}"
+    # Implicit dependency on 'build' detected from expression below
+    agent: summarizer
+    prompt: |
+      The build for branch "${{ inputs.branch }}" was successful.
+      Result: ${{ steps.build.output }}
+      Please write a concise 1-sentence summary for Slack.
+
+outputs:
+  slack_message: ${{ steps.notify.output }}
+```
+
+---
+
+## 🏗️ Step Types
+
+Keystone supports several specialized step types:
+
+- `shell`: Run arbitrary shell commands.
+- `llm`: Prompt an agent and get structured or unstructured responses. Supports `schema` (JSON Schema) for structured output.
+- `request`: Make HTTP requests (GET, POST, etc.).
+- `file`: Read, write, or append to files.
+- `human`: Pause execution for manual confirmation or text input.
+  - `inputType: confirm`: Simple Enter-to-continue prompt.
+  - `inputType: text`: Prompt for a string input, available via `${{ steps.id.output }}`.
+- `workflow`: Trigger another workflow as a sub-step.
+- `sleep`: Pause execution for a specified duration.
+
+All steps support common features like `needs` (dependencies), `if` (conditionals), `retry`, `timeout`, `foreach` (parallel iteration), and `transform` (post-process output using expressions).
+
+#### Example: Transform & Foreach Concurrency
+```yaml
+- id: list_files
+  type: shell
+  run: ls *.txt
+  # Post-process stdout into an array of filenames
+  transform: ${{ stdout.trim().split('\n') }}
+
+- id: process_files
+  type: shell
+  foreach: ${{ steps.list_files.output }}
+  concurrency: 5 # Process 5 files at a time
+  run: echo "Processing ${{ item }}"
 ```
 
-### Agents (.md)
-Agents are defined in Markdown with YAML frontmatter. This keeps the "personality" and tools of the agent together in a human-readable format.
+---
+
+## 🤖 Agent Definitions
 
+Agents are defined in Markdown files with YAML frontmatter, making them easy to read and version control.
+
+**`.keystone/workflows/agents/summarizer.md`**
 ```markdown
 ---
-name: code-reviewer
-model: claude-3-5-sonnet-latest
+name: summarizer
+provider: openai
+model: gpt-4o
+description: Summarizes technical logs into human-readable messages
+---
+
+You are a technical communications expert. Your goal is to take technical output 
+(like build logs or test results) and provide a concise, professional summary.
+```
+
+### Agent Tools
+
+Agents can be equipped with tools, which are essentially workflow steps they can choose to execute. You can define tools in the agent definition, or directly in an LLM step within a workflow.
+
+**`.keystone/workflows/agents/developer.md`**
+```markdown
+---
+name: developer
 tools:
-  - name: read_file
+  - name: list_files
+    description: List files in the current directory
     execution:
-      type: file
-      op: read
-      path: "${{ args.path }}"
+      id: list-files-tool
+      type: shell
+      run: ls -F
 ---
-You are an expert security researcher. Review the provided code for vulnerabilities.
+You are a software developer. You can use tools to explore the codebase.
 ```
 
+### Keystone as an MCP Server
+
+Keystone can itself act as an MCP server, allowing other agents (like Claude Desktop or GitHub Copilot) to discover and run your workflows as tools.
+
+```bash
+keystone mcp
+```
+
+> **Note:** Workflow execution via the Keystone MCP server is synchronous. This provides a better experience for agents as they receive the final results directly, though it means the connection remains open for the duration of the workflow run.
+
+#### Global MCP Servers
+Define shared MCP servers in `.keystone/config.yaml` to reuse them across different workflows. Keystone ensures that multiple steps using the same global server will share a single running process.
+
+Keystone supports both local (stdio) and remote (SSE) MCP servers.
+
+```yaml
+mcp_servers:
+  # Local server (stdio)
+  filesystem:
+    type: local # Default
+    command: npx
+    args: ["-y", "@modelcontextprotocol/server-filesystem", "/path/to/allowed/directory"]
+
+  # Remote server (SSE)
+  atlassian:
+    type: remote
+    url: https://mcp.atlassian.com/v1/sse
+```
+
+#### Using MCP in Steps
+You can use global servers, define local ones, or include all global servers at once.
+
+```yaml
+- id: analyze_code
+  type: llm
+  agent: developer
+  # Option 1: Explicitly include global servers by name
+  # Option 2: Define a local one-off server (standard object syntax)
+  mcpServers:
+    - filesystem 
+    - name: custom-tool
+      command: node
+      args: ["./scripts/custom-mcp.js"]
+  
+  # Option 3: Automatically include ALL global servers
+  useGlobalMcp: true 
+  
+  prompt: "Analyze the architecture of this project."
+```
+
+In these examples, the agent will have access to all tools provided by the MCP servers (like `list_directory`, `read_file`, etc.) in addition to any tools defined in the agent or the step itself.
+
 ---
 
-## 🛠️ CLI Reference
+## 🛠️ CLI Commands
 
 | Command | Description |
 | :--- | :--- |
 | `init` | Initialize a new Keystone project |
-| `run <workflow>` | Execute a workflow (supports `-i key=val` for inputs) |
-| `resume <run_id>` | Resume a paused or failed workflow run |
+| `run <workflow>` | Execute a workflow (use `-i key=val` for inputs) |
+| `resume <run_id>` | Resume a failed or paused workflow |
+| `validate [path]` | Check workflow files for errors |
+| `workflows` | List available workflows |
+| `history` | Show recent workflow runs |
+| `logs <run_id>` | View logs and step status for a specific run |
+| `graph <workflow>` | Generate a Mermaid diagram of the workflow |
+| `config` | Show current configuration and providers |
+| `auth status` | Show authentication status |
+| `auth login` | Login to an authentication provider (GitHub) |
+| `auth logout` | Logout and clear authentication tokens |
 | `ui` | Open the interactive TUI dashboard |
-| `mcp` | Start the MCP server to use workflows in other tools |
-| `graph <workflow>` | Visualize the DAG as an ASCII or Mermaid diagram |
-| `history` | List recent runs and their status |
-| `auth login` | Authenticate with GitHub for Copilot support |
-| `validate` | Check workflow files for schema and logic errors |
+| `mcp` | Start the Keystone MCP server |
+| `completion [shell]` | Generate shell completion script (zsh, bash) |
+| `prune [--days N]` | Cleanup old run data from the database |
 
 ---
 
-## 🔒 Security & Privacy
+## 📂 Project Structure
 
-1. **Local State:** All run history, logs, and outputs are stored in a local SQLite database (`.keystone/state.db`).
-2. **Redaction:** Keystone automatically scans for your environment variables and masks them in all logs and database entries.
-3. **AST Evaluation:** Expressions are parsed into an Abstract Syntax Tree and executed in a sandbox, preventing arbitrary code execution within `${{ }}` blocks.
-4. **Shell Safety:** Use the built-in `escape()` function when passing user input to shell commands to prevent injection.
+- `src/db/`: SQLite persistence layer.
+- `src/runner/`: The core execution engine, handles parallelization and retries.
+- `src/parser/`: Zod-powered validation for workflows and agents.
+- `src/expression/`: `${{ }}` expression evaluator.
+- `src/ui/`: Ink-powered TUI dashboard.
+- `src/utils/`: Shared utilities (auth, redaction, config loading).
+- `.keystone/workflows/`: Your YAML workflow definitions.
 
 ---
 
 ## 📄 License
 
-MIT © [Mark Hingston](https://github.com/mhingston)
+MIT

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "keystone-cli",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "A local-first, declarative, agentic workflow orchestrator built on Bun",
   "type": "module",
   "bin": {