roast-ai 0.2.1 → 0.2.2

data/examples/pre_post_processing/pre_processing/setup_test_environment/prompt.md

@@ -0,0 +1,11 @@
+# Setup Test Environment
+
+Prepare the test environment for optimization. Please:
+
+1. Ensure all test dependencies are installed
+2. Create a backup branch for safety: `test-optimization-backup-{{timestamp}}`
+3. Set up any necessary test databases or fixtures
+4. Configure test parallelization settings if available
+5. Clear any test caches that might affect benchmarking
+
+Return a summary of the setup steps completed and any warnings or issues encountered.

data/examples/pre_post_processing/validate_changes/prompt.md

@@ -0,0 +1,24 @@
+# Validate Changes
+
+Validate the changes made to {{file}}:
+
+1. **Run the updated tests** and ensure they all pass
+2. **Check coverage metrics** to verify improvements
+3. **Measure execution time** to confirm performance gains
+4. **Verify no regressions** were introduced
+5. **Ensure code style** follows project conventions
+
+Store the validation results in the workflow state:
+```json
+{
+  "file": "{{file}}",
+  "tests_passed": true,
+  "coverage_before": 0.0,
+  "coverage_after": 0.0,
+  "execution_time_before": 0.0,
+  "execution_time_after": 0.0,
+  "issues_found": []
+}
+```
+
+If any issues are found, provide recommendations for fixing them.

data/examples/pre_post_processing/workflow.yml

@@ -0,0 +1,21 @@
+name: test_optimization
+model: gpt-4o
+target: "test/**/*_test.rb"
+
+# Pre-processing steps run once before any test files are processed
+pre_processing:
+  - gather_baseline_metrics
+  - setup_test_environment
+
+# Main workflow steps run for each test file
+steps:
+  - analyze_test_file
+  - improve_test_coverage
+  - optimize_test_performance
+  - validate_changes
+
+# Post-processing steps run once after all test files have been processed
+post_processing:
+  - aggregate_metrics
+  - generate_summary_report
+  - cleanup_environment

data/examples/single_target_prepost/README.md

@@ -0,0 +1,36 @@
+# Single Target with Pre/Post Processing Example
+
+This example demonstrates how pre/post processing works with single-target workflows. Even when analyzing just one file, you can use pre-processing to gather context and post-processing to format results.
+
+## Features Demonstrated
+
+1. **Pre-processing for context gathering** - Analyze dependencies before the main workflow
+2. **Single-target analysis** - Focus on one specific file
+3. **Post-processing with output template** - Custom formatting of the final report
+
+## Running the Example
+
+```bash
+roast workflow.yml
+```
+
+This will:
+1. Run pre-processing to gather dependencies and context
+2. Analyze the single target file (src/main.rb)
+3. Apply the post-processing template to format the output
+
+## Output Template
+
+The `post_processing/output.txt` template demonstrates how to:
+- Access pre-processing results with `<%= pre_processing[:step_name] %>`
+- Iterate over target results (even for single targets)
+- Include post-processing step outputs
+- Format everything into a professional report
+
+## Use Cases
+
+This pattern is ideal for:
+- Deep analysis of critical files
+- Security audits of specific components
+- Performance profiling of key modules
+- Generating documentation for important classes

data/examples/single_target_prepost/post_processing/output.txt

@@ -0,0 +1,27 @@
+=== Code Analysis Report ===
+Generated: <%= Time.now.strftime("%Y-%m-%d %H:%M:%S") %>
+
+## Dependencies & Context
+<%= pre_processing.gather_dependencies %>
+
+## Target File Analysis
+<% targets.each do |file, target| %>
+File: <%= file %>
+
+### Code Quality
+<%= target.output.analyze_code_quality %>
+
+### Identified Improvements
+<%= target.output.identify_improvements %>
+
+### Recommendations
+<%= target.output.generate_recommendations %>
+<% end %>
+
+## Summary
+<%= output.summarize_findings %>
+
+## Action Items
+<%= output.create_action_items %>
+
+=== End of Report ===

data/examples/single_target_prepost/pre_processing/gather_dependencies/prompt.md

@@ -0,0 +1,11 @@
+# Gather Dependencies
+
+Analyze the project structure to understand the dependencies and context for <%= file %>.
+
+Tasks:
+1. List all files that import or depend on the target file
+2. Identify the key modules and classes used
+3. Note any external dependencies or libraries
+4. Summarize the architectural context
+
+Provide a structured summary that will help with the main analysis.

data/examples/single_target_prepost/workflow.yml

@@ -0,0 +1,20 @@
+name: analyze_codebase
+description: Analyze a single codebase with pre/post processing support
+model: gpt-4o
+target: "src/main.rb"
+
+# Pre-processing: Gather context before analyzing the main file
+pre_processing:
+  - gather_dependencies
+  - setup_analysis_tools
+
+# Main workflow: Analyze the target file
+steps:
+  - analyze_code_quality
+  - identify_improvements
+  - generate_recommendations
+
+# Post-processing: Generate final report
+post_processing:
+  - summarize_findings
+  - create_action_items

data/gemfiles/activesupport7.gemfile

@@ -0,0 +1,4 @@
+# frozen_string_literal: true
+
+eval_gemfile "../Gemfile"
+gem "activesupport", "~> 7.0"

data/gemfiles/activesupport8.gemfile

@@ -0,0 +1,4 @@
+# frozen_string_literal: true
+
+eval_gemfile "../Gemfile"
+gem "activesupport", "~> 8.0"

data/lib/roast/tools/grep.rb

@@ -28,10 +28,19 @@ module Roast
 
       def call(string)
         Roast::Helpers::Logger.info("🔍 Grepping for string: #{string}\n")
-        # Escape regex special characters in strings with curly braces
-        # Example: "import {render}" becomes "import \{render\}"
-        escaped_string = string.gsub(/(\{|\})/, '\\\\\\1')
-        %x(rg -C 4 --trim --color=never --heading -F -- "#{escaped_string}" . | head -n #{MAX_RESULT_LINES})
+
+        # Use Open3 to safely pass the string as an argument, avoiding shell injection
+        require "open3"
+        cmd = ["rg", "-C", "4", "--trim", "--color=never", "--heading", "-F", "--", string, "."]
+        stdout, _stderr, _status = Open3.capture3(*cmd)
+
+        # Limit output to MAX_RESULT_LINES
+        lines = stdout.lines
+        if lines.size > MAX_RESULT_LINES
+          lines.first(MAX_RESULT_LINES).join + "\n... (truncated to #{MAX_RESULT_LINES} lines)"
+        else
+          stdout
+        end
       rescue StandardError => e
         "Error grepping for string: #{e.message}".tap do |error_message|
           Roast::Helpers::Logger.error(error_message + "\n")

data/lib/roast/tools/search_file.rb

@@ -29,9 +29,9 @@ module Roast
         Roast::Helpers::Logger.info("🔍 Searching for: '#{glob_pattern}' in '#{File.expand_path(path)}'\n")
         search_for(glob_pattern, path).then do |results|
           return "No results found for #{glob_pattern} in #{path}" if results.empty?
-          return read_contents(results.first) if results.size == 1
+          return read_contents(File.join(path, results.first)) if results.size == 1
 
-          results.join("\n") # purposely give the AI list of actual paths so that it can read without searching first
+          results.map { |result| File.join(path, result) }.join("\n") # purposely give the AI list of actual paths so that it can read without searching first
         end
       rescue StandardError => e
         "Error searching for '#{glob_pattern}' in '#{path}': #{e.message}".tap do |error_message|

data/lib/roast/tools.rb

@@ -52,7 +52,22 @@ module Roast
       # Hook that runs on any exit (including crashes and unhandled exceptions)
       at_exit do
         if $ERROR_INFO && !$ERROR_INFO.is_a?(SystemExit) # If exiting due to unhandled exception
-          puts "\n\nExiting due to error: #{$ERROR_INFO.class}: #{$ERROR_INFO.message}\n"
+          # Print a more user-friendly message based on the error type
+          case $ERROR_INFO
+          when Roast::Workflow::CommandExecutor::CommandExecutionError
+            puts "\n\n🛑 Workflow stopped due to command failure."
+            puts "   To continue execution despite command failures, you can:"
+            puts "   - Fix the failing command"
+            puts "   - Run with --verbose to see command output"
+            puts "   - Modify your workflow to handle errors gracefully"
+          when Roast::Workflow::WorkflowExecutor::StepExecutionError
+            puts "\n\n🛑 Workflow stopped due to step failure."
+            puts "   Check the error message above for details."
+          else
+            puts "\n\n🛑 Workflow stopped due to an unexpected error:"
+            puts "   #{$ERROR_INFO.class}: #{$ERROR_INFO.message}"
+          end
+          puts "\nFor debugging, you can run with --verbose for more details."
           # Temporary disable the debugger to fix directory issues
           # context.instance_eval { binding.irb }
         end

data/lib/roast/value_objects/uri_base.rb

@@ -0,0 +1,49 @@
+# frozen_string_literal: true
+
+module Roast
+  module ValueObjects
+    # Value object representing a URI base with validation
+    class UriBase
+      class InvalidUriBaseError < StandardError; end
+
+      attr_reader :value
+
+      def initialize(value)
+        @value = value&.to_s
+        validate!
+        freeze
+      end
+
+      def present?
+        !blank?
+      end
+
+      def blank?
+        @value.nil? || @value.strip.empty?
+      end
+
+      def to_s
+        @value
+      end
+
+      def ==(other)
+        return false unless other.is_a?(UriBase)
+
+        value == other.value
+      end
+      alias_method :eql?, :==
+
+      def hash
+        [self.class, @value].hash
+      end
+
+      private
+
+      def validate!
+        return if @value.nil? # Allow nil URI base, just not empty strings
+
+        raise InvalidUriBaseError, "URI base cannot be an empty string" if @value.strip.empty?
+      end
+    end
+  end
+end

data/lib/roast/value_objects.rb

@@ -3,3 +3,4 @@
 require "roast/value_objects/api_token"
 require "roast/value_objects/step_name"
 require "roast/value_objects/workflow_path"
+require "roast/value_objects/uri_base"

data/lib/roast/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Roast
-  VERSION = "0.2.1"
+  VERSION = "0.2.2"
 end

data/lib/roast/workflow/api_configuration.rb

@@ -2,12 +2,13 @@
 
 require "roast/factories/api_provider_factory"
 require "roast/workflow/resource_resolver"
+require "roast/value_objects/uri_base"
 
 module Roast
   module Workflow
     # Handles API-related configuration including tokens and providers
     class ApiConfiguration
-      attr_reader :api_token, :api_provider
+      attr_reader :api_token, :api_provider, :uri_base
 
       def initialize(config_hash)
         @config_hash = config_hash
@@ -37,6 +38,7 @@ module Roast
       def process_api_configuration
         extract_api_token
         extract_api_provider
+        extract_uri_base
       end
 
       def extract_api_token
@@ -49,6 +51,12 @@ module Roast
         @api_provider = Roast::Factories::ApiProviderFactory.from_config(@config_hash)
       end
 
+      def extract_uri_base
+        if @config_hash["uri_base"]
+          @uri_base = ResourceResolver.process_shell_command(@config_hash["uri_base"])
+        end
+      end
+
       def environment_token
         if openai?
           ENV["OPENAI_API_KEY"]

data/lib/roast/workflow/base_workflow.rb

@@ -27,10 +27,12 @@ module Roast
         :configuration,
         :model
 
+      attr_reader :pre_processing_data
+
       delegate :api_provider, :openai?, to: :configuration
       delegate :output, :output=, :append_to_final_output, :final_output, to: :output_manager
 
-      def initialize(file = nil, name: nil, context_path: nil, resource: nil, session_name: nil, configuration: nil)
+      def initialize(file = nil, name: nil, context_path: nil, resource: nil, session_name: nil, configuration: nil, pre_processing_data: nil)
         @file = file
         @name = name || self.class.name.underscore.split("/").last
         @context_path = context_path || ContextPathResolver.resolve(self.class)
@@ -38,6 +40,7 @@ module Roast
         @session_name = session_name || @name
         @session_timestamp = nil
         @configuration = configuration
+        @pre_processing_data = pre_processing_data ? DotAccessHash.new(pre_processing_data).freeze : nil
 
         # Initialize managers
         @output_manager = OutputManager.new

data/lib/roast/workflow/command_executor.rb

@@ -6,7 +6,7 @@ module Roast
   module Workflow
     class CommandExecutor
       class CommandExecutionError < StandardError
-        attr_reader :command, :exit_status, :original_error
+        attr_reader :command, :exit_status, :original_error, :output
 
         def initialize(message, command:, exit_status: nil, original_error: nil)
           @command = command
@@ -56,11 +56,14 @@ module Roast
         return output if success
 
         if exit_on_error
-          raise CommandExecutionError.new(
+          error = CommandExecutionError.new(
             "Command exited with non-zero status (#{exit_status})",
             command: command,
             exit_status: exit_status,
           )
+          # Store the output in the error
+          error.instance_variable_set(:@output, output)
+          raise error
         else
           @logger.warn("Command '#{command}' exited with non-zero status (#{exit_status}), continuing execution")
           output + "\n[Exit status: #{exit_status}]"

data/lib/roast/workflow/configuration.rb

@@ -11,10 +11,10 @@ module Roast
     # Encapsulates workflow configuration data and provides structured access
     # to the configuration settings
     class Configuration
-      attr_reader :config_hash, :workflow_path, :name, :steps, :tools, :function_configs, :model, :resource
+      attr_reader :config_hash, :workflow_path, :name, :steps, :pre_processing, :post_processing, :tools, :function_configs, :model, :resource
       attr_accessor :target
 
-      delegate :api_provider, :openrouter?, :openai?, to: :api_configuration
+      delegate :api_provider, :openrouter?, :openai?, :uri_base, to: :api_configuration
 
       # Delegate api_token to effective_token for backward compatibility
       def api_token
@@ -30,6 +30,8 @@ module Roast
         # Extract basic configuration values
         @name = ConfigurationLoader.extract_name(@config_hash, workflow_path)
         @steps = ConfigurationLoader.extract_steps(@config_hash)
+        @pre_processing = ConfigurationLoader.extract_pre_processing(@config_hash)
+        @post_processing = ConfigurationLoader.extract_post_processing(@config_hash)
         @tools = ConfigurationLoader.extract_tools(@config_hash)
         @function_configs = ConfigurationLoader.extract_functions(@config_hash)
         @model = ConfigurationLoader.extract_model(@config_hash)

data/lib/roast/workflow/configuration_loader.rb

@@ -32,6 +32,20 @@ module Roast
           config_hash["steps"] || []
         end
 
+        # Extract pre-processing steps from the configuration
+        # @param config_hash [Hash] The configuration hash
+        # @return [Array] The pre_processing array or empty array
+        def extract_pre_processing(config_hash)
+          config_hash["pre_processing"] || []
+        end
+
+        # Extract post-processing steps from the configuration
+        # @param config_hash [Hash] The configuration hash
+        # @return [Array] The post_processing array or empty array
+        def extract_post_processing(config_hash)
+          config_hash["post_processing"] || []
+        end
+
         # Extract tools from the configuration
         # @param config_hash [Hash] The configuration hash
         # @return [Array] The tools array or empty array

data/lib/roast/workflow/error_handler.rb

@@ -85,6 +85,24 @@ module Roast
           execution_time: execution_time,
         })
 
+        # Print user-friendly error message based on error type
+        case error
+        when StepLoader::StepNotFoundError
+          $stderr.puts "\n❌ Step not found: '#{step_name}'"
+          $stderr.puts "   Please check that the step exists in your workflow's steps directory."
+          $stderr.puts "   Looking for: steps/#{step_name}.rb or steps/#{step_name}/prompt.md"
+        when NoMethodError
+          if error.message.include?("undefined method")
+            $stderr.puts "\n❌ Step error: '#{step_name}'"
+            $stderr.puts "   The step file exists but may be missing the 'call' method."
+            $stderr.puts "   Error: #{error.message}"
+          end
+        else
+          $stderr.puts "\n❌ Step failed: '#{step_name}'"
+          $stderr.puts "   Error: #{error.message}"
+          $stderr.puts "   This may be an issue with the step's implementation."
+        end
+
         # Wrap the original error with context about which step failed
         raise WorkflowExecutor::StepExecutionError.new(
           "Failed to execute step '#{step_name}': #{error.message}",

data/lib/roast/workflow/expression_evaluator.rb

@@ -33,6 +33,14 @@ module Roast
         begin
           output = executor.execute(cmd, exit_on_error: false)
 
+          # Print command output in verbose mode
+          if @workflow.verbose
+            $stderr.puts "Evaluating command: #{cmd}"
+            $stderr.puts "Command output:"
+            $stderr.puts output
+            $stderr.puts
+          end
+
           if for_condition
             # For conditions, we care about the exit status (success = true)
             # Check if output contains exit status marker

data/lib/roast/workflow/step_executor_coordinator.rb

@@ -132,16 +132,42 @@ module Roast
         error_handler.with_error_handling(step, resource_type: resource_type) do
           $stderr.puts "Executing: #{step} (Resource type: #{resource_type || "unknown"})"
 
-          output = command_executor.execute(step, exit_on_error: exit_on_error)
+          begin
+            output = command_executor.execute(step, exit_on_error: exit_on_error)
+
+            # Print command output in verbose mode
+            workflow = context.workflow
+            if workflow.verbose
+              $stderr.puts "Command output:"
+              $stderr.puts output
+              $stderr.puts
+            end
 
-          # Add to transcript
-          workflow = context.workflow
-          workflow.transcript << {
-            user: "I just executed the following command: ```\n#{step}\n```\n\nHere is the output:\n\n```\n#{output}\n```",
-          }
-          workflow.transcript << { assistant: "Noted, thank you." }
+            # Add to transcript
+            workflow.transcript << {
+              user: "I just executed the following command: ```\n#{step}\n```\n\nHere is the output:\n\n```\n#{output}\n```",
+            }
+            workflow.transcript << { assistant: "Noted, thank you." }
+
+            output
+          rescue CommandExecutor::CommandExecutionError => e
+            # Print user-friendly error message
+            $stderr.puts "\n❌ Command failed: #{step}"
+            $stderr.puts "   Exit status: #{e.exit_status}" if e.exit_status
+
+            # Show command output if available
+            if e.respond_to?(:output) && e.output && !e.output.strip.empty?
+              $stderr.puts "   Command output:"
+              e.output.strip.split("\n").each do |line|
+                $stderr.puts "     #{line}"
+              end
+            elsif workflow && !workflow.verbose
+              $stderr.puts "   To see the command output, run with --verbose flag."
+            end
 
-          output
+            $stderr.puts "   This typically means the command returned an error.\n"
+            raise
+          end
         end
       end
 

data/lib/roast/workflow/step_loader.rb

@@ -25,11 +25,11 @@ module Roast
       class StepNotFoundError < StepLoaderError; end
       class StepExecutionError < StepLoaderError; end
 
-      attr_reader :context
+      attr_reader :context, :phase
 
       delegate :workflow, :config_hash, :context_path, to: :context
 
-      def initialize(workflow, config_hash, context_path)
+      def initialize(workflow, config_hash, context_path, phase: :steps)
         # Support both old and new initialization patterns
         @context = if workflow.is_a?(WorkflowContext)
           workflow
@@ -40,6 +40,7 @@ module Roast
             context_path: context_path,
           )
         end
+        @phase = phase
       end
 
       # Finds and loads a step by name
@@ -75,6 +76,12 @@ module Roast
 
       # Find a Ruby step file in various locations
       def find_step_file(step_name)
+        # Check in phase-specific directory first
+        if phase != :steps
+          phase_rb_path = File.join(context_path, phase.to_s, "#{step_name}.rb")
+          return phase_rb_path if File.file?(phase_rb_path)
+        end
+
         # Check in context path
         rb_file_path = File.join(context_path, "#{step_name}.rb")
         return rb_file_path if File.file?(rb_file_path)
@@ -88,6 +95,12 @@ module Roast
 
       # Find a step directory
       def find_step_directory(step_name)
+        # Check in phase-specific directory first
+        if phase != :steps
+          phase_step_path = File.join(context_path, phase.to_s, step_name)
+          return phase_step_path if File.directory?(phase_step_path)
+        end
+
         # Check in context path
         step_path = File.join(context_path, step_name)
         return step_path if File.directory?(step_path)

data/lib/roast/workflow/workflow_execution_context.rb

@@ -0,0 +1,39 @@
+# frozen_string_literal: true
+
+require "active_support/core_ext/string/inflections"
+
+module Roast
+  module Workflow
+    # Manages execution context across pre-processing, target workflows, and post-processing phases
+    class WorkflowExecutionContext
+      attr_reader :pre_processing_output, :target_outputs
+
+      def initialize
+        @pre_processing_output = OutputManager.new
+        @target_outputs = {}
+      end
+
+      # Add output from a target workflow execution
+      def add_target_output(target, output_manager)
+        target_key = generate_target_key(target)
+        @target_outputs[target_key] = output_manager
+      end
+
+      # Get all data as a hash for post-processing
+      def to_h
+        {
+          pre_processing: @pre_processing_output.to_h,
+          targets: @target_outputs.transform_values(&:to_h),
+        }
+      end
+
+      private
+
+      def generate_target_key(target)
+        return "default" unless target
+
+        target.to_s.parameterize
+      end
+    end
+  end
+end

data/lib/roast/workflow/workflow_executor.rb

@@ -48,10 +48,30 @@ module Roast
 
       delegate :workflow, :config_hash, :context_path, to: :context
 
+      # Initialize a new WorkflowExecutor
+      #
+      # @param workflow [BaseWorkflow] The workflow instance to execute
+      # @param config_hash [Hash] The workflow configuration
+      # @param context_path [String] The base path for the workflow
+      # @param error_handler [ErrorHandler] Optional custom error handler
+      # @param step_loader [StepLoader] Optional custom step loader
+      # @param command_executor [CommandExecutor] Optional custom command executor
+      # @param interpolator [Interpolator] Optional custom interpolator
+      # @param state_manager [StateManager] Optional custom state manager
+      # @param iteration_executor [IterationExecutor] Optional custom iteration executor
+      # @param conditional_executor [ConditionalExecutor] Optional custom conditional executor
+      # @param step_orchestrator [StepOrchestrator] Optional custom step orchestrator
+      # @param step_executor_coordinator [StepExecutorCoordinator] Optional custom step executor coordinator
+      # @param phase [Symbol] The execution phase - determines where to load steps from
+      #   Valid values:
+      #   - :steps (default) - Load steps from the main steps directory
+      #   - :pre_processing - Load steps from the pre_processing directory
+      #   - :post_processing - Load steps from the post_processing directory
       def initialize(workflow, config_hash, context_path,
         error_handler: nil, step_loader: nil, command_executor: nil,
         interpolator: nil, state_manager: nil, iteration_executor: nil,
-        conditional_executor: nil, step_orchestrator: nil, step_executor_coordinator: nil)
+        conditional_executor: nil, step_orchestrator: nil, step_executor_coordinator: nil,
+        phase: :steps)
         # Create context object to reduce data clump
         @context = WorkflowContext.new(
           workflow: workflow,
@@ -61,7 +81,7 @@ module Roast
 
         # Dependencies with defaults
         @error_handler = error_handler || ErrorHandler.new
-        @step_loader = step_loader || StepLoader.new(workflow, config_hash, context_path)
+        @step_loader = step_loader || StepLoader.new(workflow, config_hash, context_path, phase: phase)
         @command_executor = command_executor || CommandExecutor.new(logger: @error_handler)
         @interpolator = interpolator || Interpolator.new(workflow, logger: @error_handler)
         @state_manager = state_manager || StateManager.new(workflow, logger: @error_handler)