roast-ai 0.2.3 → 0.3.1

data/examples/bash_prototyping/test_public_api/prompt.md

@@ -0,0 +1,22 @@
+# Test Public API Endpoints
+
+Use the bash tool to test various public APIs:
+
+1. Test GitHub API:
+   ```bash
+   curl -s https://api.github.com/users/github
+   ```
+
+2. Test a public JSON placeholder API:
+   ```bash
+   curl -s https://jsonplaceholder.typicode.com/posts/1
+   ```
+
+3. Check response headers:
+   ```bash
+   curl -I https://api.github.com
+   ```
+
+4. Test with different HTTP methods if appropriate
+
+Analyze the responses and note the structure of the returned data.

data/examples/cmd/README.md

@@ -0,0 +1,99 @@
+# Command Tool Examples
+
+Learn how to execute system commands in your Roast workflows using the Cmd tool. This example demonstrates how to define allowed commands and use them as functions within your workflows.
+
+## Overview
+
+The `Cmd` tool allows you to define a list of approved system commands that can be executed by the AI. Each allowed command is treated as a distinct function, providing several benefits:
+
+- **Security**: Only explicitly allowed commands can be run, preventing unauthorized system access.
+- **Clarity**: Each command has a clear, defined purpose within the workflow.
+- **Intelligent Usage**: Custom descriptions can be provided to help the AI choose the correct command for a given task.
+
+## Configuration
+
+You can configure the `Cmd` tool in your workflow YAML file.
+
+### Basic Configuration
+
+List the commands that the AI is allowed to execute:
+
+```yaml
+tools:
+  - Roast::Tools::Cmd:
+      allowed_commands:
+        - pwd
+        - ls
+        - echo
+```
+
+Each command listed here (`pwd`, `ls`, `echo`) becomes a function the AI can call. For example, the AI can call `pwd()` to get the current directory, or `ls(args: "-la")` to list files.
+
+### Enhanced Configuration with Descriptions
+
+For more complex commands or to provide better guidance to the AI, you can include a description:
+
+```yaml
+tools:
+  - Roast::Tools::Cmd:
+      allowed_commands:
+        - pwd
+        - ls
+        - name: git
+          description: "git CLI - version control system (e.g., git status, git log)"
+        - name: npm
+          description: "npm CLI - Node.js package manager (e.g., npm install, npm run)"
+        - name: docker
+          description: "Docker CLI - container platform (e.g., docker ps, docker run)"
+```
+These descriptions help the AI understand the purpose of each command and its common subcommands or arguments.
+
+## How It Works
+
+When the `Cmd` tool is configured, each entry in `allowed_commands` is registered as a function available to the AI.
+
+- `pwd` becomes `pwd()`
+- `ls` becomes `ls(args: <string>)`
+- `git` (with a description) becomes `git(args: <string>)`
+
+The AI can then choose to call these functions with appropriate arguments (passed via the `args` parameter) to accomplish its tasks.
+
+## Example Workflows
+
+This directory contains several example workflows demonstrating different uses of the `Cmd` tool:
+
+- **`basic_workflow.yml`**: Introduces simple command execution.
+- **`explorer_workflow.yml`**: Uses commands to navigate and understand a project structure.
+- **`dev_workflow.yml`**: Showcases how descriptions guide the AI in selecting tools for development-related tasks.
+
+## Running the Examples
+
+To run an example workflow:
+```bash
+bin/roast execute examples/cmd/NAME_OF_WORKFLOW.yml
+```
+For instance:
+```bash
+bin/roast execute examples/cmd/basic_workflow.yml
+```
+
+## Security
+
+Explicitly defining `allowed_commands` is crucial for security:
+- You maintain full control over which system commands the AI can execute.
+- It creates self-documenting configurations, making workflows safer and more predictable.
+
+## Best Practices
+
+- **Start Simple**: Begin with basic commands like `pwd` and `ls`.
+- **Use Descriptions for Clarity**: Provide descriptions for commands that are not universally understood or have complex arguments/subcommands. This helps both the AI and human readers.
+- **Scope Appropriately**: Only allow commands that are necessary for the workflow's intended purpose.
+
+## Why Use Custom Descriptions?
+
+Custom descriptions are particularly useful when:
+- Dealing with domain-specific or less common commands.
+- Disambiguating between commands with similar names or overlapping functionality.
+- Guiding the AI to make more informed decisions about tool selection.
+
+Good descriptions enhance the AI's ability to use commands effectively and make your workflows more robust.

data/examples/cmd/analyze_project/prompt.md

@@ -0,0 +1,57 @@
+# Project Analysis
+
+You need to analyze the current project to identify its type, available tools, and development setup. Gather comprehensive information to understand the project's nature and configuration.
+
+Investigate the project to determine:
+
+1. Your current working location
+2. The project structure and contents
+3. Whether this is a Node.js project (look for package.json)
+4. Whether Docker is configured (look for Dockerfile or docker-compose.yml)
+5. Whether there's a Makefile for build automation
+6. The version control state
+
+Based on your findings, determine:
+- Project type (Node.js application, containerized service, make-based project, etc.)
+- Available development tools and configurations
+- How dependencies are managed
+- Version control status
+- Recommended next steps for development
+
+RESPONSE FORMAT
+Provide your analysis in JSON format:
+
+<json>
+{
+  "project_analysis": {
+    "working_directory": "/path/to/project",
+    "project_type": "Identified based on available files",
+    "detected_tools": {
+      "node_project": {
+        "has_package_json": false,
+        "has_lock_file": false,
+        "package_manager": "none"
+      },
+      "containerization": {
+        "has_dockerfile": false,
+        "has_compose": false,
+        "docker_ready": false
+      },
+      "build_system": {
+        "has_makefile": false,
+        "make_targets": []
+      },
+      "version_control": {
+        "has_git": true,
+        "working_tree_clean": false,
+        "uncommitted_changes": 3
+      }
+    },
+    "recommendations": [
+      "Relevant recommendations based on what was found"
+    ],
+    "confidence": "high"
+  },
+  "analysis_complete": true
+}
+</json>

data/examples/cmd/basic_demo/prompt.md

@@ -0,0 +1,48 @@
+# Basic Command Demonstration
+
+You are demonstrating basic command functions. Execute several simple commands to show how the command tool system works.
+
+**SPECIFIC TASKS:**
+1. Show the current working directory: use `pwd`
+2. List all files with details: use `ls -la`
+3. Display celebratory message: use `echo "🎉 Command functions are working!"`
+4. List examples directory: use `ls examples/`
+
+**EFFICIENCY RULES:**
+- Execute each command ONLY ONCE
+- DO NOT repeat any command
+- Follow the exact order above
+
+Each command execution demonstrates how commands are called as functions in the workflow, with security enforced through the workflow's configuration.
+
+RESPONSE FORMAT
+Provide a summary of your command executions in JSON format:
+
+<json>
+{
+  "demonstration_complete": true,
+  "commands_executed": [
+    {
+      "command": "pwd",
+      "purpose": "Show current working directory",
+      "output_summary": "Current directory path"
+    },
+    {
+      "command": "ls -la",
+      "purpose": "List all files with details",
+      "output_summary": "Directory listing with permissions and sizes"
+    },
+    {
+      "command": "echo \"🎉 Command functions are working!\"",
+      "purpose": "Display test message",
+      "output_summary": "Success message displayed"
+    },
+    {
+      "command": "ls examples/",
+      "purpose": "List examples directory",
+      "output_summary": "Contents of examples directory"
+    }
+  ],
+  "conclusion": "Successfully demonstrated basic command functions with security constraints"
+}
+</json>

data/examples/cmd/basic_workflow.yml

@@ -0,0 +1,17 @@
+name: Basic Command Functions
+model: default
+
+# Learn the basics of using command functions in Roast
+
+tools:
+  - Roast::Tools::Cmd:
+      allowed_commands:
+        - pwd
+        - ls
+        - echo
+        - mkdir
+        - cat
+
+steps:
+  - basic_demo
+  - create_and_verify

data/examples/cmd/check_repository/prompt.md

@@ -0,0 +1,57 @@
+# Repository Status Check
+
+You need to analyze the Git repository status and provide a comprehensive overview of the repository state, history, and configuration.
+
+**SPECIFIC COMMANDS TO USE:**
+1. Check repository status: `git status`
+2. Show current branch: `git branch --show-current`
+3. List recent commits: `git log --oneline -10`
+4. Show remote repositories: `git remote -v`
+
+**EFFICIENCY RULES:**
+- Run each command ONLY ONCE - do not repeat any git command
+- DO NOT try different variations of the same command (e.g., different log formats)
+- Gather all needed information with these 4 commands only
+
+From these commands, analyze:
+- Current branch and its state
+- Working directory status (clean/dirty)
+- Recent commit activity
+- Remote repository configuration
+- Overall repository health
+
+RESPONSE FORMAT
+Provide your analysis in JSON format:
+
+<json>
+{
+  "repository_status": {
+    "current_branch": "main",
+    "working_directory": {
+      "is_clean": false,
+      "modified_files": 3,
+      "untracked_files": 2,
+      "staged_changes": 1
+    },
+    "recent_activity": {
+      "total_recent_commits": 10,
+      "latest_commit": {
+        "hash": "abc1234",
+        "message": "Latest commit message"
+      },
+      "activity_summary": "Regular commits indicate active development"
+    },
+    "remotes": {
+      "origin": {
+        "fetch_url": "https://github.com/user/repo.git",
+        "push_url": "https://github.com/user/repo.git"
+      }
+    },
+    "repository_health": {
+      "status": "healthy",
+      "notes": "Repository has uncommitted changes but is otherwise in good state"
+    }
+  },
+  "analysis_complete": true
+}
+</json>

data/examples/cmd/create_and_verify/prompt.md

@@ -0,0 +1,56 @@
+# Creating and Verifying Files
+
+You need to demonstrate file creation and verification capabilities. Create a test directory and file, then verify the operations were successful.
+
+**EXECUTE THESE COMMANDS IN ORDER:**
+1. Create directory: `mkdir -p test_cmd_demo`
+2. Create file: `echo "This file was created by command functions!" > test_cmd_demo/demo.txt`
+3. Verify directory: `ls -la test_cmd_demo/`
+4. Verify content: `cat test_cmd_demo/demo.txt`
+5. Show completion: `echo "Demo complete! You can remove test_cmd_demo when ready."`
+
+**EFFICIENCY RULE:** Execute each command exactly once in the order shown above.
+
+This exercise demonstrates how command functions can be combined for practical file operations while maintaining security through the workflow's command restrictions.
+
+RESPONSE FORMAT
+Report the results in JSON format:
+
+<json>
+{
+  "task_completed": true,
+  "operations": [
+    {
+      "step": "create_directory",
+      "command": "mkdir -p test_cmd_demo",
+      "success": true,
+      "result": "Directory created successfully"
+    },
+    {
+      "step": "create_file",
+      "command": "echo \"This file was created by command functions!\" > test_cmd_demo/demo.txt",
+      "success": true,
+      "result": "File created with content"
+    },
+    {
+      "step": "verify_creation",
+      "command": "ls -la test_cmd_demo/",
+      "success": true,
+      "result": "Directory listing shows demo.txt file"
+    },
+    {
+      "step": "verify_content",
+      "command": "cat test_cmd_demo/demo.txt",
+      "success": true,
+      "result": "File contents match expected text"
+    },
+    {
+      "step": "completion_message",
+      "command": "echo \"Demo complete! You can remove test_cmd_demo when ready.\"",
+      "success": true,
+      "result": "Completion message displayed"
+    }
+  ],
+  "summary": "Successfully demonstrated file creation and verification using command functions"
+}
+</json>

data/examples/cmd/dev_workflow.yml

@@ -0,0 +1,26 @@
+name: Development Tools Workflow
+model: default
+
+# Demonstrates how custom descriptions guide intelligent tool selection
+
+tools:
+  - Roast::Tools::Cmd:
+      allowed_commands:
+        - pwd
+        - ls
+        - name: git
+          description: "git CLI - version control system with subcommands like status, commit, branch"
+        - name: npm
+          description: "npm CLI - Node.js package manager with subcommands like install, run, test"
+        - name: docker
+          description: "Docker CLI - container platform with subcommands like ps, run, build"
+        - name: curl
+          description: "curl command - make HTTP requests with options like -X, -H, -d"
+        - name: jq
+          description: "jq command - process JSON data with filters like '.key', '.[].name'"
+        - name: make
+          description: "make command - run build targets defined in Makefile"
+
+steps:
+  - analyze_project
+  - smart_tool_selection

data/examples/cmd/explore_project/prompt.md

@@ -0,0 +1,67 @@
+# Exploring Your Project Structure
+
+You need to analyze the project structure and provide a comprehensive overview. Gather information about the project layout, documentation, and configuration.
+
+**IMPORTANT COMMAND SYNTAX:**
+- The `find` command requires a path: `find . -name "*.md"` (not `find -name "*.md"`)
+- Use `head` to limit output: `find . -name "*.md" | head -10`
+- Check files efficiently - don't repeat commands
+
+Explore the project by:
+
+1. Identifying your current location: use `pwd`
+2. Listing all files and directories: use `ls -la`
+3. Finding documentation files: use `find . -name "*.md" -type f | head -10`
+4. Examining the README: use `cat README.md | head -20` (if it exists)
+5. Locating configuration files: use `find . -name "*.yml" -o -name "*.yaml" | grep -E "(config|workflow)" | head -10`
+
+**EFFICIENCY RULES:**
+- Run each command ONLY ONCE
+- If a file doesn't exist, note it and move on
+- DO NOT retry failed commands
+
+Based on your exploration, analyze:
+- Project root location and overall structure
+- Key directories and their likely purposes
+- Available documentation
+- Configuration files present
+- General project organization
+
+RESPONSE FORMAT
+Provide your analysis in JSON format:
+
+<json>
+{
+  "project_analysis": {
+    "current_directory": "/path/to/project",
+    "total_items": {
+      "files": 0,
+      "directories": 0,
+      "hidden_items": 0
+    },
+    "documentation": {
+      "readme_found": true,
+      "readme_summary": "Brief summary of README contents",
+      "other_docs": ["list of other .md files found"]
+    },
+    "configuration": {
+      "config_files": ["list of .yml/.yaml configuration files"],
+      "workflow_files": ["list of workflow-related files"]
+    },
+    "project_structure": {
+      "key_directories": [
+        {
+          "name": "src",
+          "purpose": "Source code directory"
+        },
+        {
+          "name": "test",
+          "purpose": "Test files directory"
+        }
+      ],
+      "project_type": "Identified project type based on files present"
+    }
+  },
+  "exploration_complete": true
+}
+</json>

data/examples/cmd/explorer_workflow.yml

@@ -0,0 +1,21 @@
+name: Project Explorer
+model: default
+
+# Navigate and explore your project using command functions
+
+tools:
+  - Roast::Tools::Cmd:
+      allowed_commands:
+        - pwd
+        - ls
+        - find
+        - name: cat
+          description: "cat command - display file contents, concatenate files, works with pipes"
+        - name: git
+          description: "git CLI - version control system with subcommands like status, log, branch"
+        - name: grep
+          description: "grep command - search text patterns with options like -E, -r, -i"
+
+steps:
+  - explore_project
+  - check_repository

data/examples/cmd/smart_tool_selection/prompt.md

@@ -0,0 +1,99 @@
+# Intelligent Tool Selection Examples
+
+You are demonstrating intelligent tool selection by completing various development tasks. Select and use the most appropriate tools based on what each task requires.
+
+**CRITICAL GUIDELINES:**
+- When working with JSON data, you MUST use jq to parse it - never return raw JSON
+- Be efficient - if a command fails, do NOT repeat it
+- Combine tools with pipes when needed (e.g., `curl -s ... | jq ...`)
+- Check for files ONLY ONCE - if they don't exist, note it and move on
+
+Complete the following development tasks:
+
+## Task 1: API Health Check
+Check if the GitHub API is accessible and responding properly.
+- Target: https://api.github.com
+- Goal: Verify the API returns a successful response (check HTTP status)
+
+## Task 2: Parse JSON Response
+Fetch GitHub's public information and extract specific data fields.
+- Target: https://api.github.com/users/github
+- Extract: The name and company fields from the JSON response
+- **REQUIRED**: Use curl to fetch and jq to parse in a single command: `curl -s <url> | jq '<filter>'`
+
+## Task 3: Check Container Environment
+Determine if Docker is available and check for running containers.
+- Try `docker ps` ONCE
+- If Docker daemon is not running, note this and move on - do NOT retry
+
+## Task 4: Check for Node.js Project
+Investigate if this is a Node.js project and examine its dependencies.
+- Check for package.json ONCE with `ls package.json`
+- If not found, note this and move on - do NOT retry
+
+## Task 5: Check Build System
+Determine if a build system is configured and what targets are available.
+- Check for Makefile ONCE with `ls Makefile`
+- If not found, note this and move on - do NOT retry
+
+**EFFICIENCY REMINDER**: Each file check or failed command should be attempted ONLY ONCE.
+
+RESPONSE FORMAT
+Report your findings in JSON format:
+
+<json>
+{
+  "tool_selection_demo": {
+    "task_1_api_check": {
+      "tool_selected": "curl",
+      "command_used": "curl -I https://api.github.com",
+      "rationale": "Selected HTTP request tool for API check",
+      "result": {
+        "api_accessible": true,
+        "status_code": 200
+      }
+    },
+    "task_2_json_parsing": {
+      "tools_selected": ["curl", "jq"],
+      "command_used": "curl -s https://api.github.com/users/github | jq '.name, .company'",
+      "rationale": "Combined tools for fetching and parsing JSON",
+      "result": {
+        "data_extracted": true,
+        "fields": {
+          "name": "GitHub",
+          "company": "@github"
+        }
+      }
+    },
+    "task_3_containers": {
+      "tool_selected": "docker",
+      "command_used": "docker ps",
+      "rationale": "Selected container platform tool",
+      "result": {
+        "docker_available": false,
+        "error": "Docker daemon not running",
+        "containers_running": 0
+      }
+    },
+    "task_4_nodejs": {
+      "tool_selected": "ls",
+      "command_used": "ls package.json",
+      "rationale": "Checked for package.json existence",
+      "result": {
+        "is_node_project": false,
+        "package_json_found": false
+      }
+    },
+    "task_5_build": {
+      "tool_selected": "ls",
+      "command_used": "ls Makefile",
+      "rationale": "Checked for Makefile existence",
+      "result": {
+        "makefile_found": false,
+        "targets": []
+      }
+    }
+  },
+  "summary": "Successfully demonstrated intelligent tool selection based on task requirements"
+}
+</json>

data/examples/grading/README.md

@@ -0,0 +1,71 @@
+# Test Grading Workflow
+
+This workflow acts as a senior software engineer and testing expert to evaluate the quality of test files based on best practices and guidelines.
+
+## Prerequisites
+
+This example uses `shadowenv` for environment management, which is specific to Shopify's development environment. If you're not using shadowenv, you'll need to adapt the commands to your own setup.
+
+### If you're using shadowenv:
+```bash
+brew install shadowenv
+```
+
+### If you're NOT using shadowenv:
+You'll need to modify the `run_coverage.rb` file to remove the shadowenv commands. Look for lines like:
+```ruby
+command = "shadowenv exec -- bundle exec ruby ..."
+```
+
+And change them to match your environment:
+```ruby
+# For standard Ruby/Bundler setup:
+command = "bundle exec ruby ..."
+
+# Or if you're using rbenv/rvm:
+command = "ruby ..."
+```
+
+## Usage
+
+```bash
+# Run the grading workflow on a test file
+roast execute examples/grading/workflow.yml path/to/your_test.rb
+```
+
+## How it Works
+
+1. **read_dependencies**: Analyzes the test file and its dependencies
+2. **run_coverage**: Executes the test with coverage tracking
+3. **generate_grades**: Evaluates test quality across multiple dimensions
+4. **verify_test_helpers**: Checks for proper test helper usage
+5. **verify_mocks_and_stubs**: Ensures appropriate use of test doubles
+6. **analyze_coverage**: Reviews code coverage metrics
+7. **generate_recommendations**: Provides improvement suggestions
+8. **calculate_final_grade**: Computes an overall grade (A-F scale)
+9. **format_result**: Formats the final output
+
+## Customization
+
+Feel free to adapt this workflow to your testing environment:
+
+- **Different test frameworks**: Modify `run_coverage.rb` to work with RSpec, Jest, pytest, etc.
+- **Coverage tools**: Replace the coverage command with your preferred tool (SimpleCov, Istanbul, Coverage.py)
+- **Grading criteria**: Adjust the prompts in each step to match your team's standards
+- **Environment setup**: Remove or replace shadowenv with your environment management tool
+
+## Example Output
+
+```
+========== TEST GRADE REPORT ==========
+Test file: test/example_test.rb
+
+FINAL GRADE:
+  Score: 85/100
+  Letter Grade: B
+
+RECOMMENDATIONS:
+- Add edge case testing for error conditions
+- Improve test descriptions for clarity
+- Consider extracting common setup to helper methods
+```

data/examples/grading/read_dependencies/prompt.md

@@ -5,10 +5,12 @@ The first dependency you should always look for is the source file for the prime
 If you can identify other important application-level dependencies then read them too.
 How many extra dependencies to research is left to your discretion, but ALWAYS make sure you have the subject under test (SUT) in your context before responding.
 
-Once you are finished using tool functions, respond with the relative path to the source file of the SUT inside <sut> tags.
+Once you are finished using tool functions, respond with the relative path to the source file of the SUT inside <sut> tags. IMPORTANT: Include the full relative path from the project root, including any directory prefixes like lib/, app/, etc.
 
 Example:
 
 If you are told to find the dependencies of `test/services/country_db_interface_test.rb`,
-then you would use the functions as explained above and ultimately respond with `<sut>./app/services/country_db_interface.rb</sut>`
+then you would use the functions as explained above and ultimately respond with `<sut>app/services/country_db_interface.rb</sut>`
+
+If the file is found at `lib/roast/workflow/workflow_initializer.rb`, respond with `<sut>lib/roast/workflow/workflow_initializer.rb</sut>` (include the lib/ prefix)
 

data/examples/grading/run_coverage.rb

@@ -30,7 +30,16 @@ class RunCoverage < Roast::Workflow::BaseStep
 
     # Resolve paths to prevent issues when pwd differs from project root
     resolved_subject_file = Roast::Helpers::PathResolver.resolve(subject_file)
+    unless File.exist?(resolved_subject_file)
+      Roast::Helpers::Logger.error("Subject file not found: #{resolved_subject_file}")
+      exit(1)
+    end
+
     resolved_test_file = Roast::Helpers::PathResolver.resolve(test_file)
+    unless File.exist?(resolved_test_file)
+      Roast::Helpers::Logger.error("Test file not found: #{resolved_test_file}")
+      exit(1)
+    end
 
     # Run the test_runner using shadowenv for environment consistency
     command = "shadowenv exec --dir . -- #{test_runner_path} #{resolved_test_file} #{resolved_subject_file}"