roast-ai 0.1.0 → 0.1.2

data/docs/INSTRUMENTATION.md

@@ -0,0 +1,202 @@
+# Instrumentation Hooks in Roast
+
+Roast provides built-in instrumentation hooks using ActiveSupport::Notifications, allowing you to track workflow execution, monitor performance, and integrate with your own monitoring systems.
+
+## Overview
+
+The instrumentation system emits events at key points during workflow execution:
+
+- Workflow lifecycle (start, complete)
+- Step execution (start, complete, error)
+- Chat completion/AI calls (start, complete, error)
+- Tool function execution
+
+## Configuration
+
+To add custom instrumentation, create Ruby files in your project's `.roast/initializers/` directory. These files will be automatically loaded during workflow startup.
+
+Example structure:
+```
+your-project/
+  ├── .roast/
+  │   └── initializers/
+  │       ├── logging.rb
+  │       ├── metrics.rb
+  │       └── monitoring.rb
+  └── ...
+```
+
+## Available Events
+
+### Workflow Events
+
+- `roast.workflow.start` - Emitted when a workflow begins
+  - Payload: `{ workflow_path:, options:, name: }`
+  
+- `roast.workflow.complete` - Emitted when a workflow completes
+  - Payload: `{ workflow_path:, success:, execution_time: }`
+
+### Step Events
+
+- `roast.step.start` - Emitted when a step begins execution
+  - Payload: `{ step_name: }`
+  
+- `roast.step.complete` - Emitted when a step completes successfully
+  - Payload: `{ step_name:, success: true, execution_time:, result_size: }`
+  
+- `roast.step.error` - Emitted when a step encounters an error
+  - Payload: `{ step_name:, error:, message:, execution_time: }`
+
+### AI/Chat Completion Events
+
+- `roast.chat_completion.start` - Emitted before an AI API call
+  - Payload: `{ model:, parameters: }`
+  
+- `roast.chat_completion.complete` - Emitted after successful AI API call
+  - Payload: `{ success: true, model:, parameters:, execution_time:, response_size: }`
+  
+- `roast.chat_completion.error` - Emitted when AI API call fails
+  - Payload: `{ error:, message:, model:, parameters:, execution_time: }`
+
+### Tool Execution Events
+
+- `roast.tool.execute` - Emitted when a tool function is called
+  - Payload: `{ function_name:, params: }`
+
+- `roast.tool.complete` - Emitted when a tool function completes
+  - Payload: `{ function_name:, execution_time:, cache_enabled: }`
+  
+- `roast.tool.error` - Emitted when a tool execution fails
+  - Payload: `{ function_name:, error:, message:, execution_time: }`
+
+## Example Usage
+
+### Basic Logging
+
+```ruby
+# .roast/initializers/logging.rb
+ActiveSupport::Notifications.subscribe(/roast\./) do |name, start, finish, id, payload|
+  duration = finish - start
+  puts "[#{name}] completed in #{duration.round(3)}s"
+end
+```
+
+### Performance Monitoring
+
+```ruby
+# .roast/initializers/performance.rb
+ActiveSupport::Notifications.subscribe("roast.step.complete") do |name, start, finish, id, payload|
+  duration = finish - start
+  if duration > 10.0
+    puts "WARNING: Step '#{payload[:step_name]}' took #{duration.round(1)}s"
+  end
+end
+```
+
+### Integration with External Services
+
+```ruby
+# .roast/initializers/metrics.rb
+ActiveSupport::Notifications.subscribe("roast.workflow.complete") do |name, start, finish, id, payload|
+  duration = finish - start
+  
+  # Send to your metrics service
+  MyMetricsService.track_event("workflow_execution", {
+    workflow_path: payload[:workflow_path],
+    duration: duration,
+    success: payload[:success]
+  })
+end
+```
+
+### Internal Shopify Example
+
+For the internal Shopify version, you can use these instrumentation hooks to track metrics with Monorail:
+
+```ruby
+# .roast/initializers/monorail.rb
+
+# Track workflow execution
+ActiveSupport::Notifications.subscribe("roast.workflow.start") do |name, start, finish, id, payload|
+  Roast::Support::Monorail.track_command("run", {
+    "workflow_path" => payload[:workflow_path],
+    "options" => payload[:options],
+    "name" => payload[:name]
+  })
+end
+
+ActiveSupport::Notifications.subscribe("roast.workflow.complete") do |name, start, finish, id, payload|
+  Roast::Support::Monorail.track_command("run_complete", {
+    "workflow_path" => payload[:workflow_path],
+    "success" => payload[:success],
+    "execution_time" => payload[:execution_time]
+  })
+end
+
+# Track AI model usage and performance
+ActiveSupport::Notifications.subscribe("roast.chat_completion.complete") do |name, start, finish, id, payload|
+  Roast::Support::Monorail.track_command("ai_usage", {
+    "model" => payload[:model],
+    "execution_time" => payload[:execution_time],
+    "response_size" => payload[:response_size],
+    "has_json" => payload[:parameters][:json] || false,
+    "has_loop" => payload[:parameters][:loop] || false
+  })
+end
+
+# Track tool execution and caching
+ActiveSupport::Notifications.subscribe("roast.tool.complete") do |name, start, finish, id, payload|
+  Roast::Support::Monorail.track_command("tool_usage", {
+    "function_name" => payload[:function_name],
+    "execution_time" => payload[:execution_time],
+    "cache_enabled" => payload[:cache_enabled]
+  })
+end
+```
+
+See `examples/monorail_initializer.rb` for a complete example of Monorail integration.
+
+## Best Practices
+
+1. **Keep initializers focused**: Each initializer should handle a specific concern (logging, metrics, error reporting, etc.)
+
+2. **Handle errors gracefully**: Wrap your subscriber code in error handling to prevent crashes:
+   ```ruby
+   ActiveSupport::Notifications.subscribe("roast.workflow.start") do |name, start, finish, id, payload|
+     begin
+       # Your instrumentation code here
+     rescue => e
+       $stderr.puts "Instrumentation error: #{e.message}"
+     end
+   end
+   ```
+
+3. **Avoid blocking operations**: Instrumentation should be fast and non-blocking. For heavy operations, consider using async processing.
+
+4. **Use pattern matching**: Subscribe to specific event patterns to reduce overhead:
+   ```ruby
+   # Subscribe only to workflow events
+   ActiveSupport::Notifications.subscribe(/roast\.workflow\./) do |name, start, finish, id, payload|
+     # Handle only workflow events
+   end
+   ```
+
+5. **Consider performance impact**: While instrumentation is valuable, too many subscribers can impact performance. Be selective about what you instrument.
+
+## Testing Your Instrumentation
+
+You can test your instrumentation by creating a simple workflow and observing the events:
+
+```yaml
+# test_instrumentation.yml
+name: instrumentation_test
+steps:
+  - test_step
+```
+
+Then run:
+```bash
+roast execute test_instrumentation.yml some_file.rb
+```
+
+Your instrumentation should capture the workflow start, step execution, and workflow completion events.

data/examples/api_workflow/README.md

@@ -0,0 +1,85 @@
+# API Workflow Example
+
+This example demonstrates a targetless workflow that interacts with APIs rather than operating on specific files.
+
+## Structure
+
+The workflow consists of three steps that work together to create a complete API integration process:
+
+1. `fetch_api_data` - Simulates fetching data from a weather API and returns a structured JSON response
+2. `transform_data` - Processes the JSON data into a human-readable markdown format 
+3. `generate_report` - Creates a polished report with recommendations based on the weather data
+
+## Running the Example
+
+To run this example, you need to have a valid API token. The example is configured to fetch a token using a shell command:
+
+```yaml
+# Dynamic API token using shell command
+api_token: $(print-token --key)
+```
+
+You can modify this to use your own token source, such as:
+
+```yaml
+# Using an environment variable
+api_token: $(echo $OPENAI_API_KEY)
+
+# Or a direct value (not recommended for production)
+api_token: $(echo "sk-your-actual-token")
+```
+
+Then run the workflow:
+
+```bash
+# Run the targetless workflow
+roast execute examples/api_workflow/workflow.yml
+
+# Save the output to a file
+roast execute examples/api_workflow/workflow.yml -o weather_report.md
+```
+
+## How Targetless Workflows Work
+
+Targetless workflows operate without a specific target file. This is useful for:
+
+- API integrations
+- Content generation
+- Data analysis
+- Report creation
+- Interactive tools
+
+Unlike file-based workflows that process each target separately, targetless workflows run once and can retrieve their own data sources (like API calls) or generate content from scratch.
+
+## Workflow Definition
+
+```yaml
+name: API Integration Workflow
+# Default model for all steps
+model: gpt-4o-mini
+
+tools:
+  - Roast::Tools::ReadFile
+  - Roast::Tools::Grep
+  - Roast::Tools::WriteFile
+
+steps:
+  - fetch_api_data
+  - transform_data
+  - generate_report
+
+# Tool configurations for API calls (no need to specify model here since it uses global model)
+fetch_api_data:
+  print_response: true
+```
+
+## Creating Your Own Targetless Workflows
+
+To create your own targetless workflow:
+
+1. Create a workflow YAML file without a `target` parameter
+2. Define the steps your workflow will execute
+3. Create prompt files for each step
+4. Run the workflow with `roast execute your_workflow.yml`
+
+Your steps can use the workflow's `output` hash to pass data between them, just like in file-based workflows.

data/examples/api_workflow/fetch_api_data/prompt.md

@@ -0,0 +1,10 @@
+You are an assistant helping with retrieving data from an API source.
+
+Your task is to simulate fetching data from a weather API using mock data. 
+
+1. Imagine you are retrieving current weather conditions for various cities
+2. Create a structured JSON response that represents typical weather API data
+3. The response should include temperature, conditions, wind, and other relevant weather metrics
+4. Format the response as valid JSON that could be parsed programmatically
+
+Return only the JSON data without any additional explanation.

data/examples/api_workflow/generate_report/prompt.md

@@ -0,0 +1,10 @@
+You are an assistant helping to generate a final weather report.
+
+Based on the transformed data from the previous step, create a polished report that could be sent to stakeholders.
+
+1. Take the transformed weather data and enhance it with recommendations and insights
+2. Add relevant suggestions based on the weather conditions (like what to wear, activities that would be appropriate, etc.)
+3. Include a "forecast overview" section that summarizes the key points
+4. Format the output as a professional-looking report with proper headings and structure
+
+The report should be comprehensive yet concise, easy to scan, and provide actionable insights based on the weather data.

data/examples/api_workflow/prompt.md

@@ -0,0 +1,10 @@
+# API Integration Workflow
+
+This workflow demonstrates how to create an API integration that:
+1. Fetches data from an external source
+2. Transforms the data into a usable format
+3. Generates a report based on the processed data
+
+The workflow doesn't require a target file because it's designed to work with external APIs and data sources rather than processing specific files.
+
+You'll be working through a weather data processing example. This is a simulation to demonstrate the workflow pattern - no actual API calls are made.

data/examples/api_workflow/transform_data/prompt.md

@@ -0,0 +1,10 @@
+You are an assistant helping to transform API data.
+
+Your task is to process weather data from a JSON format into a more readable summary.
+
+1. Review the weather data provided in the previous step
+2. Transform the technical JSON data into a human-readable summary
+3. Format the output as markdown with appropriate sections and highlights
+4. Focus on the most relevant information that would be useful to a typical user
+
+Return a well-formatted markdown summary of the weather data.

data/examples/api_workflow/workflow.yml

@@ -0,0 +1,30 @@
+# API Integration Workflow
+
+# This workflow demonstrates how to create an API integration that:
+# 1. Fetches data from an external source
+# 2. Transforms the data into a usable format
+# 3. Generates a report based on the processed data
+
+# The workflow doesn't require a target file because it's designed to work with external APIs and data sources rather than processing specific files.
+
+# You'll be working through a weather data processing example. This is a simulation to demonstrate the workflow pattern - no actual API calls are made.
+
+name: API Integration Workflow
+model: gpt-4o-mini
+
+tools:
+  - Roast::Tools::ReadFile
+  - Roast::Tools::Grep
+  - Roast::Tools::WriteFile
+
+# For demonstration purposes only - in production you would use a real token command
+# api_token: $(echo "demo_token_123456")
+
+steps:
+  - fetch_api_data
+  - transform_data
+  - generate_report
+
+# Tool configurations for API calls
+fetch_api_data:
+  print_response: true

data/examples/grading/format_result.rb

@@ -16,7 +16,6 @@ class FormatResult < Roast::Workflow::BaseStep
     append_to_final_output(<<~OUTPUT)
       ========== TEST GRADE REPORT ==========
       Test file: #{workflow.file}
-      Source file: #{workflow.subject_file}
     OUTPUT
 
     format_results
@@ -26,22 +25,39 @@ class FormatResult < Roast::Workflow::BaseStep
   private
 
   def format_results
-    format_grade
+    # With HashWithIndifferentAccess, we can simply access with either syntax
+    grade_data = workflow.output["calculate_final_grade"]
+
+    unless grade_data
+      return append_to_final_output("Error: Grading data not available. This may be because you're replaying the workflow from this step, but the previous step data is missing or not found in the selected session.")
+    end
+
+    format_grade(grade_data)
+
+    # Make sure rubric_scores exists before trying to iterate over it
+    unless grade_data[:rubric_scores]
+      return append_to_final_output("Error: Rubric scores data not available in the workflow output.")
+    end
 
     append_to_final_output("RUBRIC SCORES:")
-    workflow.output["calculate_final_grade"][:rubric_scores].each do |category, data|
-      append_to_final_output("  #{RUBRIC[category][:description]} (#{(RUBRIC[category][:weight] * 100).round}% of grade):")
-      append_to_final_output("    Value: #{data[:raw_value]}")
-      append_to_final_output("    Score: #{(data[:score] * 10).round}/10 - \"#{data[:description]}\"")
+    grade_data[:rubric_scores].each do |category, data|
+      # Safely access RUBRIC with a fallback for potentially missing categories
+      rubric_item = RUBRIC[category.to_sym] || { description: "Unknown Category", weight: 0 }
+
+      append_to_final_output("  #{rubric_item[:description]} (#{(rubric_item[:weight] * 100).round}% of grade):")
+      append_to_final_output("    Value: #{data[:raw_value] || "N/A"}")
+      append_to_final_output("    Score: #{data[:score] ? (data[:score] * 10).round : "N/A"}/10 - \"#{data[:description] || "No description available"}\"")
     end
   end
 
-  def format_grade
-    letter_grade = workflow.output["calculate_final_grade"][:final_score][:letter_grade]
+  def format_grade(grade_data)
+    return append_to_final_output("\nError: Final grade data not available.") unless grade_data && grade_data[:final_score]
+
+    letter_grade = grade_data[:final_score][:letter_grade]
     celebration_emoji = letter_grade == "A" ? "🎉" : ""
     append_to_final_output(<<~OUTPUT)
       \nFINAL GRADE:
-        Score: #{(workflow.output["calculate_final_grade"][:final_score][:weighted_score] * 100).round}/100
+        Score: #{(grade_data[:final_score][:weighted_score] * 100).round}/100
         Letter Grade: #{letter_grade} #{celebration_emoji}
     OUTPUT
   end

data/examples/grading/js_test_runner

@@ -0,0 +1,31 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+if ARGV.length != 2
+  puts "Usage: #{File.basename($PROGRAM_NAME)} SUBJECT_FILE TEST_FILE"
+  exit 1
+end
+
+subject_file, test_file = ARGV
+
+def detect_package_manager
+  return "pnpm" if File.exist?(File.join(Dir.pwd, "pnpm-lock.yaml"))
+  return "yarn" if File.exist?(File.join(Dir.pwd, "yarn.lock"))
+
+  "npm"
+end
+
+jest_options = [
+  "--verbose",
+  "--no-colors",
+  "--ci",
+  "--coverageReporters=text-summary",
+  "--collectCoverageFrom=#{subject_file}",
+]
+
+# Assumes the test command is `test:coverage`
+# Both admin-web and checkout-web use this command
+command = "#{detect_package_manager} run test:coverage -- #{test_file} #{jest_options.join(" ")}"
+
+$stderr.puts "Running: #{command}"
+puts system(command)

data/examples/grading/rb_test_runner

@@ -0,0 +1,19 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "rubygems"
+require "bundler/setup"
+
+require_relative "../../lib/roast/helpers/minitest_coverage_runner"
+
+# Suppress fancy minitest reporting
+ENV["RM_INFO"] = "true"
+
+if ARGV.length != 2
+  puts "Usage: #{File.basename($PROGRAM_NAME)} SUBJECT_FILE TEST_FILE"
+  exit 1
+end
+
+test_file, subject_file = ARGV
+
+Roast::Helpers::MinitestCoverageRunner.new(test_file, subject_file).run

data/examples/grading/read_dependencies/prompt.md

@@ -0,0 +1,14 @@
+Use the provided functions to find and read important dependencies of the provided test file named <%= workflow.file %>.
+
+The first dependency you should always look for is the source file for the prime subject of the test (whatever class this test file is claiming to test). Use `read_file` to read the subject's source code into your conversation transcript, but only if it's not already there from a previous chat.
+
+If you can identify other important application-level dependencies then read them too.
+How many extra dependencies to research is left to your discretion, but ALWAYS make sure you have the subject under test (SUT) in your context before responding.
+
+Once you are finished using tool functions, respond with the relative path to the source file of the SUT inside <sut> tags.
+
+Example:
+
+If you are told to find the dependencies of `test/services/country_db_interface_test.rb`,
+then you would use the functions as explained above and ultimately respond with `<sut>./app/services/country_db_interface.rb</sut>`
+

data/examples/grading/run_coverage.rb

@@ -20,7 +20,7 @@ class RunCoverage < Roast::Workflow::BaseStep
     extension = "js" if ["js", "jsx", "ts", "tsx"].include?(extension)
 
     # Get the absolute path to the test_runner executable
-    test_runner_path = File.expand_path("../../bin/#{extension}_test_runner", __dir__)
+    test_runner_path = File.expand_path("#{extension}_test_runner", __dir__)
 
     # Make sure the test_runner executable exists
     unless File.exist?(test_runner_path)
@@ -33,7 +33,7 @@ class RunCoverage < Roast::Workflow::BaseStep
     resolved_test_file = Roast::Helpers::PathResolver.resolve(test_file)
 
     # Run the test_runner using shadowenv for environment consistency
-    command = "shadowenv exec --dir . -- #{test_runner_path} #{resolved_subject_file} #{resolved_test_file}"
+    command = "shadowenv exec --dir . -- #{test_runner_path} #{resolved_test_file} #{resolved_subject_file}"
     output, status = Open3.capture2(command)
 
     unless status.success?

data/examples/grading/workflow.yml

@@ -5,7 +5,7 @@ tools:
   - Roast::Tools::ReadFile
   - Roast::Tools::SearchFile
 
-each: '% cd $(git rev-parse --show-toplevel) && git status --porcelain | grep "_test\.rb" | cut -c4- | xargs realpath'
+# each: '% cd $(git rev-parse --show-toplevel) && git status --porcelain | grep "_test\.rb" | cut -c4- | xargs realpath'
 
 steps:
   - read_dependencies
@@ -18,12 +18,11 @@ steps:
   - calculate_final_grade
   - format_result
   - generate_recommendations
-  - annotate_pr_with_comments
 
 # set non-default attributes for steps below
 analyze_coverage:
   model: gpt-4.1-mini
-  loop: false
+  auto_loop: false
   json: true
   
 generate_grades:
@@ -32,15 +31,7 @@ generate_grades:
 
 generate_recommendations:
   model: o3
-  loop: false
+  auto_loop: false
   json: true
   params:
     max_completion_tokens: 5_000
-
-annotate_pr_with_comments:
-  tools:
-    - Roast::Tools::Github::Annotator
-  model: o3
-  params:
-    max_completion_tokens: 5_000
-  if: "workflow.pr? && output.recommendations.any?"

data/examples/instrumentation.rb

@@ -0,0 +1,76 @@
+# frozen_string_literal: true
+
+# Demonstration of how to use the Roast instrumentation hooks
+# This file would typically be placed in PROJECT_ROOT/.roast/initializers/
+# for automatic loading during workflow execution
+
+# Example: Log all workflow and step events
+ActiveSupport::Notifications.subscribe(/roast\.workflow\./) do |name, start, finish, _id, payload|
+  duration = finish - start
+
+  case name
+  when "roast.workflow.start"
+    puts "\n🚀 Workflow starting: #{payload[:name]}"
+    puts "   Path: #{payload[:workflow_path]}"
+    puts "   Options: #{payload[:options]}"
+  when "roast.workflow.complete"
+    status = payload[:success] ? "✅ Successfully" : "❌ With errors"
+    puts "\n#{status} completed workflow in #{duration.round(2)} seconds"
+  end
+end
+
+# Example: Track step execution times
+ActiveSupport::Notifications.subscribe(/roast\.step\./) do |name, start, finish, _id, payload|
+  duration = finish - start
+
+  case name
+  when "roast.step.start"
+    puts "\n  ▶️ Step starting: #{payload[:step_name]}"
+  when "roast.step.complete"
+    puts "  ✅ Step completed: #{payload[:step_name]} (#{duration.round(3)}s)"
+  when "roast.step.error"
+    puts "  ❌ Step failed: #{payload[:step_name]}"
+    puts "     Error: #{payload[:error]} - #{payload[:message]}"
+  end
+end
+
+# Example: Monitor AI interactions
+ActiveSupport::Notifications.subscribe(/roast\.chat_completion\./) do |name, start, finish, _id, payload|
+  case name
+  when "roast.chat_completion.start"
+    puts "\n  🤖 AI request starting (model: #{payload[:model]})"
+    puts "     Parameters: #{payload[:parameters].inspect}" if payload[:parameters].any?
+  when "roast.chat_completion.complete"
+    duration = finish - start
+    puts "  🤖 AI request completed in #{duration.round(2)}s (execution time: #{payload[:execution_time].round(2)}s)"
+    puts "     Response size: #{payload[:response_size]} characters"
+  when "roast.chat_completion.error"
+    puts "  🤖 AI request failed: #{payload[:error]} - #{payload[:message]}"
+    puts "     Execution time: #{payload[:execution_time].round(2)}s"
+  end
+end
+
+# Example: Track tool executions
+ActiveSupport::Notifications.subscribe(/roast\.tool\./) do |name, _start, _finish, _id, payload|
+  case name
+  when "roast.tool.execute"
+    puts "\n  🔧 Executing tool: #{payload[:function_name]}"
+  when "roast.tool.complete"
+    puts "  🔧 Tool completed: #{payload[:function_name]} (#{payload[:execution_time].round(3)}s)"
+    puts "     Cache enabled: #{payload[:cache_enabled]}"
+  when "roast.tool.error"
+    puts "  🔧 Tool failed: #{payload[:function_name]}"
+    puts "     Error: #{payload[:error]} - #{payload[:message]}"
+    puts "     Execution time: #{payload[:execution_time].round(3)}s"
+  end
+end
+
+# In a Shopify-specific initializer (e.g., .roast/initializers/monorail.rb),
+# you could replace these logging examples with actual Monorail tracking:
+#
+# ActiveSupport::Notifications.subscribe("roast.workflow.start") do |name, start, finish, id, payload|
+#   Roast::Support::Monorail.track_command("run", {
+#     "workflow_path" => payload[:workflow_path],
+#     "name" => payload[:name]
+#   })
+# end