roast-ai 0.4.0 → 0.4.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0c0f1a837c5dafcaecb8443e93a4076c11566f1c3bea1d6835121628fbdf79a7
-  data.tar.gz: ab3679b374486831cbf90ab3a259178d48d4e53d5d5dbdc2a2d43537399e4ca9
+  metadata.gz: d1822868d0357ade29fdbdd3feab6a1beff23ef97621de9e1a9cacee366d5897
+  data.tar.gz: 00b9aff47853dc8f1f4ffa89bf571975fbe414632909b682fe6caab4e64bee51
 SHA512:
-  metadata.gz: 3061d9f04b7f886e3a6d588eb729e10062f6310d5973c7ed5181975f27a385c20d8ff8880ecb7e61abd54dfbd87795211313e1c9a6d2a630269d7c4559ae7cae
-  data.tar.gz: 7962e4991590700003f3f8a827435214ef9aa90fa4324d34d3b421f11d5f647e43663e17189a0b43f81d6fa124bbf68a269f9674a493994832b9ab5c03f192a6
+  metadata.gz: 95c32a5e010460dfe2e584c7b0ba8b7d617dc965210fc00343bd2c1f647c847f3be41be0bb07a951a41b5aa1447c770b36f4eba97985543f510938044e85453b
+  data.tar.gz: d221ec8fba42b7187ae30fa4171e2700a7b81b1092693f9fdf1f168a37b6318482f5be8cf3f1be4e8650fe7a56d87ef2a0f93fea13f633e24c95734c285fd75b

data/.github/workflows/ci.yaml

@@ -23,8 +23,8 @@ jobs:
       BUNDLE_GEMFILE: ${{ github.workspace }}/gemfiles/${{ matrix.gemfile }}.gemfile
     steps:
       - uses: actions/checkout@v4
-      - name: Install ripgrep
-        run: sudo apt-get update && sudo apt-get install -y ripgrep
+      - name: Install dependencies
+        run: sudo apt-get update && sudo apt-get install -y ripgrep graphviz
       - uses: ruby/setup-ruby@v1
         with:
           ruby-version: ${{ matrix.ruby }}

data/.gitignore

@@ -41,3 +41,4 @@ bin/thor
 
 gemfiles/*.lock
 bin/claude-swarm
+*.gem

data/CHANGELOG.md

@@ -5,6 +5,109 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.4.2] - 2025-06-20
+
+### Added
+- **Multiline bash command support** (#289)
+  - Enhanced CommandExecutor to properly handle commands spanning multiple lines
+  - Enables sophisticated bash scripts in workflow steps
+  - Maintains backward compatibility with single-line commands
+- **Comprehensive shell security enhancements** (#289)
+  - Smart interpolation that detects shell commands and escapes dangerous characters
+  - Protection against shell injection for all major metacharacters:
+    - Backslashes (`\`) to prevent path injection
+    - Double quotes (`"`) to prevent breaking quoted contexts
+    - Dollar signs (`$`) to prevent variable expansion
+    - Backticks (`` ` ``) to prevent command substitution
+  - Context-aware escaping only in shell commands, preserving text elsewhere
+- **Early detection for missing Raix configuration** (#292)
+  - Provides helpful error messages when Raix is not properly initialized
+  - Shows example configuration for both OpenAI and OpenRouter
+  - Prevents cryptic "undefined method 'chat' for nil" errors
+- **Exit early feature for input steps** (#291)
+  - Pressing Ctrl-C during input steps now exits cleanly
+  - No more confusing stack traces when canceling input
+- **Default --dangerously-skip-permissions flag for CodingAgent** (#290)
+  - Avoids permission prompts during automated workflows
+  - Improves workflow automation experience
+
+### Fixed
+- Test isolation issue causing CI failures (#289)
+- Flaky test in StepExecutorRegistryTest due to executor registration conflicts (#289)
+- Shell command interpolation security vulnerabilities (#289)
+- Missing dependency declarations (cli-kit, sqlite3) (#292)
+
+### Changed
+- Updated cli-kit dependency to ~> 5.0 for better error handling
+- Updated sqlite3 dependency to ~> 2.6 to resolve version conflicts
+
+[0.4.2]: https://github.com/Shopify/roast/compare/v0.4.1...v0.4.2
+
+## [0.4.1] - 2025-06-18
+
+### Added
+- **SQLite session storage** as the default storage backend (#252)
+  - Provides better performance and advanced querying capabilities
+  - Sessions are stored in `~/.roast/sessions.db` by default (configurable via `ROAST_SESSIONS_DB`)
+  - New `roast sessions` command to list and filter stored sessions
+  - New `roast session <id>` command to view detailed session information
+  - Session cleanup with `roast sessions --cleanup --older-than <duration>`
+  - Filter sessions by status, workflow name, or age
+  - Maintains full backward compatibility with filesystem storage
+- **`--file-storage` CLI option** to use legacy filesystem storage instead of SQLite
+  - Use `-f` or `--file-storage` flag to opt into filesystem storage
+  - Environment variable `ROAST_STATE_STORAGE=file` still supported for compatibility
+- **Foundation for wait_for_event feature** (#251)
+  - New `roast resume` command infrastructure for resuming paused workflows
+  - Event storage and tracking in SQLite sessions table
+- **Configurable agent step options** for CodingAgent (#266)
+  - New `continue` option for agent steps to maintain session context across multiple agent invocations
+  - New `include_context_summary` option to provide AI-generated workflow context summaries to agents
+  - Context summaries are intelligently tailored to the agent's specific task using LLM analysis
+  - Helps reduce token usage by including only relevant context information
+- **Token consumption reporting** for step execution (#264)
+  - Displays token usage (prompt and completion) after each step execution
+  - Helps users monitor and optimize their LLM token consumption
+  - Automatically enabled for all workflow runs
+- **Timeout functionality for bash and cmd steps** (#261)
+  - New `timeout` option for bash and cmd steps to prevent hanging commands
+  - Configurable timeout duration in seconds
+  - Commands are automatically terminated if they exceed the specified timeout
+  - Prevents workflows from getting stuck on unresponsive commands
+- **Claude Swarm tool integration** (#254)
+  - New `Roast::Tools::Swarm` for integrating with Claude Swarm framework
+  - Enables using Swarm's multi-agent orchestration capabilities within Roast workflows
+  - Provides seamless handoffs between specialized AI agents
+- **Workflow visualization with diagram command** (#256)
+  - New `roast diagram` command to generate visual representations of workflows
+  - Creates GraphViz-based diagrams showing workflow structure and flow
+  - Supports both DOT format output and PNG/SVG image generation
+  - Helps understand complex workflow logic at a glance
+- **Comprehensive workflow validation** (#244)
+  - New `roast validate` command to check workflow syntax and structure
+  - Validates YAML syntax, step references, and configuration options
+  - Provides detailed error messages for invalid workflows
+  - Helps catch errors before running workflows
+- **apply_diff tool** (#246)
+  - New built-in tool for applying unified diff patches to files
+  - Supports standard diff format for making precise file modifications
+  - Enables AI models to suggest changes in diff format
+  - More reliable than search-and-replace for complex edits
+- **Model fallback mechanism** (#257)
+  - Workflows without explicit model configuration now use a sensible default
+  - Prevents errors when model is not specified at workflow or step level
+  - Improves user experience for simple workflows
+- **Context management foundation for auto-compaction** (#264)
+  - Infrastructure for future automatic context size management
+  - Enables intelligent token usage optimization in long-running workflows
+
+### Changed
+- Session storage now defaults to SQLite instead of filesystem
+  - Existing filesystem sessions remain accessible when using `--file-storage` flag
+  - No migration required - both storage backends can coexist
+
+[0.4.1]: https://github.com/Shopify/roast/compare/v0.4.0...v0.4.1
+
 ## [0.4.0] - 2025-06-12
 
 ### Added

data/CLAUDE.md

@@ -7,15 +7,12 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 - Note that this project now uses Zeitwerk, which means you don't have to manually require project files anymore
 
 ## Commands
-
-- Default (tests + lint): `bundle exec rake`
-- Test all: `bundle exec test`
+- Default THE SUITE RUNS FAST SO USE THIS  IN MOST CASES (tests + lint w/autocorrect): `bundle exec rake`
 - Run single test: `bundle exec ruby -Itest test/path/to/test_file.rb`
 - Lint: `bundle exec rubocop`
 - Lint (with autocorrect, preferred): `bundle exec rubocop -A`
 - Whenever you want to run the whole test suite just run `bundle exec rake` to also run linting, and note the linting errors too (most will auto correct but not all)
 - **Run roast locally**: Use `bin/roast` (not `bundle exec roast` which may use the installed gem)
-- Alternative: `bundle exec exe/roast`
 
 ## Tech stack
 - `thor` and `cli-ui` for the CLI tool
@@ -23,7 +20,6 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 - Prefer using the more literate `test "this is a test description" do` type of testing that we get from extending ActiveSupport::TestCase over the primitive XUnit-style def test_description headings for tests
 
 ## Code Style Guidelines
-
 - Naming: snake_case for variables/methods, CamelCase for classes/modules, ALL_CAPS for constants
 - Module structure: Use nested modules under the `Roast` namespace
 - Command pattern: Commands implement a `call` method and class-level `help` method
@@ -41,7 +37,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 - Always leave a blank line after module includes and before the rest of the class
 
 ## Architecture Guidelines
-
+- **SOLID principles are important** - don't violate them
 - **Maintain proper separation of concerns**: Don't mix unrelated concepts in the same class or module
   - Example: Conditional execution (if/unless) should NOT be mixed with iteration execution (each/repeat)
   - Each concept should have its own executor class and be handled separately
@@ -55,14 +51,64 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 - When faced with the choice between working around an architectural issue or code smell versus actually diving into fixing the design issue or code smell, choose the latter more principled approach
 - When fixing code smells, you don't have to worry about internal backwards compatibility 
 
-## Guidance and Expectations
+## Workflow Configuration Syntax
+- The `steps` key in the workflow configuration is an array of step names
+- Only step names, inline prompts, and control flow keywords are allowed in the steps array
+- Additional per-step configuration is provided in a top-level hash with the step name as the key, not within steps!!! (Very important)
+- The reason that steps are not configured "inline" within the steps array is so that the shape of the workflow is as obvious as possible
+- Step labels are inferred for most steps and optional for inline prompts, but required for steps that need custom configuration
+- The result of running a step is stored in the `workflow.output` hash with the step label as the key
+
+## How Roast Tools Work (CRITICAL - READ THIS!)
+**Tools in Roast are NOT explicitly invoked in workflow steps!** This is a fundamental concept that differs from many other workflow systems.
+
+### Key Concepts:
+1. **Tools are capabilities available to the LLM** - They are functions the LLM can choose to call while executing a step
+2. **Steps contain prompts** - Steps describe what needs to be done, not how to do it
+3. **The LLM decides when to use tools** - While executing a step's prompt, the LLM analyzes the task and calls tools as needed
+4. **Tools are registered, not declared in steps** - Use the `tools:` section to make tools available, but never use a `tool:` key in step configuration
+
+### Correct inline prompt syntax:
+```yaml
+steps:
+  - analyze_code: |
+      Analyze the codebase and identify performance bottlenecks.
+      Use any available tools to read files and search for patterns.
+```
+
+### INCORRECT syntax (DO NOT USE):
+```yaml
+# WRONG - no 'prompt:' key
+steps:
+  - analyze_code:
+      prompt: "Analyze the codebase"
+      
+# WRONG - no 'tool:' key
+steps:
+  - run_analysis:
+      tool: coding_agent
+      prompt: "Analyze code"
+```
+
+### How tools are actually used:
+When the LLM executes the `analyze_code` step above, it might:
+1. Decide it needs to read files and call `read_file(path)`
+2. Decide it needs to search and call `grep(pattern, path)`
+3. Decide it needs Claude Swarm and call `swarm(prompt, config_path)`
+
+The LLM makes these decisions based on the prompt and available tools, similar to how Claude (you) decides when to use Bash, Read, or other tools when responding to user requests.
 
+## Step Configuration
+- The `path` key in a step configuration is the path to a Ruby file that defines a custom step.
+- The `model` key in a step configuration is the model to use for the step.
+- The `print_response` key in a step configuration is a boolean that determines whether the step's response should be printed to the console.
+
+## Coding Guidance and Expectations
 - Do not decide unilaterally to leave code for the sake of "backwards compatibility"... always run those decisions by me first.
 - Don't ever commit and push changes unless directly told to do so
-- You can't test input steps yourself, ask me to do it
+- You can't test input steps yourself since they block, so ask me to do it manually
 
 ## Git Workflow Practices
-
 1. **Amending Commits**:
    - Use `git commit --amend --no-edit` to add staged changes to the last commit without changing the commit message
    - This is useful for incorporating small fixes or changes that belong with the previous commit

data/Gemfile.lock

@@ -1,14 +1,17 @@
 PATH
   remote: .
   specs:
-    roast-ai (0.4.0)
+    roast-ai (0.4.2)
       activesupport (>= 7.0)
+      cli-kit (~> 5.0)
       cli-ui (= 2.3.0)
       diff-lcs (~> 1.5)
       faraday-retry
       json-schema
       open_router (~> 0.3)
       raix (~> 1.0)
+      ruby-graphviz (~> 1.2)
+      sqlite3 (~> 2.6)
       thor (~> 1.3)
       zeitwerk (~> 2.6)
 
@@ -35,9 +38,11 @@ GEM
     benchmark (0.4.1)
     bigdecimal (3.2.2)
     cgi (0.5.0)
-    claude_swarm (0.1.15)
+    claude_swarm (0.1.19)
       fast-mcp-annotations
       thor (~> 1.3)
+    cli-kit (5.0.1)
+      cli-ui (~> 2.0)
     cli-ui (2.3.0)
     coderay (1.1.3)
     concurrent-ruby (1.3.5)
@@ -82,11 +87,11 @@ GEM
       faraday-net_http (>= 2.0, < 3.5)
       json
       logger
-    faraday-multipart (1.1.0)
+    faraday-multipart (1.1.1)
       multipart-post (~> 2.0)
-    faraday-net_http (3.4.0)
+    faraday-net_http (3.4.1)
       net-http (>= 0.5.0)
-    faraday-retry (2.3.1)
+    faraday-retry (2.3.2)
       faraday (~> 2.0)
     fast-mcp-annotations (1.5.2)
       addressable (~> 2.8)
@@ -131,7 +136,7 @@ GEM
     mime-types (3.7.0)
       logger
       mime-types-data (~> 3.2025, >= 3.2025.0507)
-    mime-types-data (3.2025.0603)
+    mime-types-data (3.2025.0617)
     minitest (5.25.5)
     minitest-rg (5.3.0)
       minitest (~> 5.0)
@@ -149,7 +154,7 @@ GEM
       dotenv (>= 2)
       faraday (>= 1)
       faraday-multipart (>= 1)
-    ostruct (0.6.1)
+    ostruct (0.6.2)
     parallel (1.27.0)
     parser (3.3.8.0)
       ast (~> 2.4.1)
@@ -174,7 +179,7 @@ GEM
       ffi (~> 1.0)
     regexp_parser (2.10.0)
     rexml (3.4.1)
-    rubocop (1.76.0)
+    rubocop (1.77.0)
       json (~> 2.3)
       language_server-protocol (~> 3.17.0.2)
       lint_roller (~> 1.1.0)
@@ -182,14 +187,16 @@ GEM
       parser (>= 3.3.0.2)
       rainbow (>= 2.2.2, < 4.0)
       regexp_parser (>= 2.9.3, < 3.0)
-      rubocop-ast (>= 1.45.0, < 2.0)
+      rubocop-ast (>= 1.45.1, < 2.0)
       ruby-progressbar (~> 1.7)
       unicode-display_width (>= 2.4.0, < 4.0)
-    rubocop-ast (1.45.0)
+    rubocop-ast (1.45.1)
       parser (>= 3.3.7.2)
       prism (~> 1.4)
     rubocop-shopify (2.17.1)
       rubocop (~> 1.62)
+    ruby-graphviz (1.2.5)
+      rexml
     ruby-openai (7.4.0)
       event_stream_parser (>= 0.3.0, < 2.0.0)
       faraday (>= 1)
@@ -198,6 +205,8 @@ GEM
     ruby2_keywords (0.0.5)
     securerandom (0.4.1)
     shellany (0.0.1)
+    sqlite3 (2.6.0-arm64-darwin)
+    sqlite3 (2.6.0-x86_64-linux-gnu)
     thor (1.3.2)
     tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)

data/README.md

@@ -405,6 +405,72 @@ For typical AI workflows, the continuous conversation history provides seamless
 - `-c, --concise`: Use concise output templates (exposed as a boolean flag on `workflow`)
 - `-v, --verbose`: Show output from all steps as they execute
 - `-r, --replay STEP_NAME`: Resume a workflow from a specific step, optionally with a specific session timestamp
+- `-f, --file-storage`: Use filesystem storage for sessions instead of SQLite (default: SQLite)
+
+#### Workflow Validation
+
+Roast provides a `validate` command to check workflow configuration files for errors and potential issues before execution:
+
+```bash
+# Validate a specific workflow
+roast validate workflow.yml
+
+# Validate a workflow in a subdirectory
+roast validate my_workflow
+
+# Validate with strict mode (treats warnings as errors)
+roast validate workflow.yml --strict
+```
+
+The validator checks for:
+- YAML syntax errors
+- Missing required fields
+- Invalid step references
+- Circular dependencies
+- Tool availability
+- Prompt file existence
+- Configuration consistency
+
+This helps catch configuration errors early and ensures workflows will run smoothly.
+
+#### Session Storage and Management
+
+Roast uses SQLite by default for session storage, providing better performance and advanced querying capabilities. Sessions are automatically saved during workflow execution, capturing each step's state including conversation transcripts and outputs.
+
+**Storage Options:**
+
+```bash
+# Use default SQLite storage (recommended)
+roast execute workflow.yml
+
+# Use legacy filesystem storage
+roast execute workflow.yml --file-storage
+
+# Set storage type via environment variable
+ROAST_STATE_STORAGE=file roast execute workflow.yml
+```
+
+**Session Management Commands:**
+
+```bash
+# List all sessions
+roast sessions
+
+# Filter sessions by status
+roast sessions --status waiting
+
+# Filter sessions by workflow
+roast sessions --workflow my_workflow
+
+# Show sessions older than 7 days
+roast sessions --older-than 7d
+
+# Clean up old sessions
+roast sessions --cleanup --older-than 30d
+
+# View detailed session information
+roast session <session_id>
+```
 
 #### Session Replay
 
@@ -418,14 +484,14 @@ roast execute workflow.yml -r step_name
 roast execute workflow.yml -r 20250507_123456_789:step_name
 ```
 
-Sessions are automatically saved during workflow execution. Each step's state, including the conversation transcript and output, is persisted to a session directory. The session directories are organized by workflow name and file, with timestamps for each run.
-
 This feature is particularly useful when:
 - Debugging specific steps in a long workflow
 - Iterating on prompts without rerunning the entire workflow
 - Resuming after failures in long-running workflows
 
-Sessions are stored in the `.roast/sessions/` directory in your project. Note that there is no automatic cleanup of session data, so you might want to periodically delete old sessions yourself.
+**Storage Locations:**
+- SQLite: `~/.roast/sessions.db` (configurable via `ROAST_SESSIONS_DB`)
+- Filesystem: `.roast/sessions/` directory in your project
 
 #### Target Option (`-t, --target`)
 

data/bin/console

@@ -1,4 +1,5 @@
 #!/usr/bin/env ruby
+# frozen_string_literal: true
 
 require "bundler/setup"
 

data/docs/AGENT_STEPS.md

@@ -19,6 +19,35 @@ steps:
 
 Both file-based prompts (with directories like `implement_fix/prompt.md`) and inline prompts (text with spaces) are supported.
 
+## Agent Step Configuration Options
+
+Agent steps support two special configuration options:
+
+### `continue` (boolean, default: false)
+When set to `true`, the agent continues from its previous session instead of starting fresh. This is useful for iterative development where you want the agent to maintain context across multiple steps.
+
+### `include_context_summary` (boolean, default: false)
+When set to `true`, the agent receives an AI-generated summary of the workflow context as a system directive. This summary is intelligently tailored to the agent's upcoming task, including only relevant information from previous steps. The summary is generated by analyzing:
+- The agent's prompt to understand what context would be helpful
+- Previous step outputs and their relevance to the current task
+- Workflow description and configuration
+- Current working directory
+
+This helps the agent understand what has been done so far without overwhelming it with irrelevant details. NOTE: Without this option, the agent relies solely on what it is instructed to do either by the prompt or your specific step instructions.
+
+```yaml
+steps:
+  - analyze_code
+  - implement_fix: ^Fix the issues identified in the analysis
+  - add_tests: ^Prepare and publish PR
+
+implement_fix:
+  include_context_summary: true  # Include a summary of the workflow context so far
+
+add_tests:
+  continue: true # does not need context since is continuing from the previous step
+```
+
 ## When to Use Agent Steps
 
 ### Use Agent Steps When:
@@ -63,22 +92,18 @@ This benefits from LLM interpretation because:
 ```markdown
 Create a new migration file with the following specifications:
 
-1. Use MultiEdit to create file: db/migrate/{{timestamp}}_add_user_preferences.rb
+1. Create a new migration file: db/migrate/{{timestamp}}_add_user_preferences.rb
 2. The migration must include:
    - Add column :users, :preferences, :jsonb, default: {}
    - Add index :users, :preferences, using: :gin
    - Add column :users, :notification_settings, :jsonb, default: {}
 3. Ensure proper up/down methods
 4. Follow Rails migration conventions exactly
-
-Required tools: MultiEdit
-Do not use Write tool for migrations.
 ```
 
 This is better as an agent step because:
-- It requires specific tool usage (MultiEdit, not Write)
 - The instructions are precise and technical
-- No interpretation needed - just execution
+- No interpretation needed - just execution of steps known beforehand
 
 ### Example 2: Code Refactoring
 
@@ -216,9 +241,6 @@ Use exact module structure and method signatures shown.
    Use MultiEdit tool to update the following files:
    - app/models/user.rb: Add validation
    - test/models/user_test.rb: Add test case
-   
-   # Avoid in agent steps
-   Update the user model with better validation
    ```
 
 2. **Include specific line numbers or code markers when possible**
@@ -261,4 +283,6 @@ Use exact module structure and method signatures shown.
 
 Agent steps are powerful when you need direct control over tool usage and precise execution of technical tasks. They complement regular steps by handling the implementation details while regular steps handle the analysis and planning.
 
+The `continue` and `include_context_summary` options make agent steps even more powerful for iterative development workflows where maintaining context is important.
+
 Choose agent steps when precision matters more than interpretation. Choose regular steps when context and judgment are important.