roast-ai 0.4.8 → 0.4.9

data/examples/grading/read_dependencies/prompt.md

@@ -0,0 +1,15 @@
+Use the provided functions to find and read important dependencies of the provided test file named <%= workflow.file %>.
+
+The first dependency you should always look for is the source file for the prime subject of the test (whatever class this test file is claiming to test). Use `read_file` to read the subject's source code into your conversation transcript, but only if it's not already there from a previous chat.
+
+If you can identify other important application-level dependencies then read them too.
+How many extra dependencies to research is left to your discretion, but ALWAYS make sure you have the subject under test (SUT) in your context before responding.
+
+Once you are finished using tool functions, respond with the relative path to the source file of the SUT inside <sut> tags. IMPORTANT: Include the full relative path from the project root, including any directory prefixes like lib/, app/, etc.
+
+Example:
+
+If you are told to find the dependencies of `test/services/country_db_interface_test.rb`,
+then you would use the functions as explained above and ultimately respond with `<sut>app/services/country_db_interface.rb</sut>`
+
+If the file is found at `lib/roast/workflow/workflow_initializer.rb`, respond with `<sut>lib/roast/workflow/workflow_initializer.rb</sut>` (include the lib/ prefix)

data/examples/grading/verify_mocks_and_stubs/prompt.md

@@ -0,0 +1,12 @@
+Find places in the provided test code where stubbing and mocking are used. Search for the corresponding implementation source code of those dependencies elsewhere in the codebase to validate that the stub or mock matches the implementation that it is doubling. Use the tool functions provided to find and read the dependencies.
+
+Once you've found the dependencies, verify that any mocks and stubs accurately reflect the real implementation. If there are discrepancies, list them out alphabetically with:
+
+1. The name of the mocked/stubbed method
+2. What the mock/stub expects in terms of arguments and/or return values
+3. What the actual implementation actually takes as arguments and returns
+4. Suggestions for fixing the discrepancy
+
+Note: If there are no discrepancies, do not summarize those that accurately reflect their real implementations in the codebase, just respond "All mocks and stubs verified."
+
+IMPORTANT: There's absolutely no need for you to waste time grepping for methods/functions that you know belong to testing libraries such as Mocha's `expects` and `stubs`. Only search for the implementation of things that are stubbed and/or mocked in the test to verify whether the test code matches the implementation code.

data/examples/grading/verify_test_helpers/prompt.md

@@ -0,0 +1,53 @@
+Now identify custom test helpers used in this test for the following purpose:
+
+1. Analyzing if they are used correctly
+2. Understanding test code that has had significant chunks of implementation abstracted away into helpers
+3. Fully understanding custom assertions that are not included by default in Ruby on Rails or part of your base knowledge
+
+Your grep tool function is vital for this work. It provides 4 lines of context before and after the matching line.
+
+For example, if you call `grep(string: "def assert_sql")`, the output will include:
+
+```
+.test/support/helpers/sql_assertions.rb-101-    end
+.test/support/helpers/sql_assertions.rb-102-    result
+.test/support/helpers/sql_assertions.rb-103-  end
+.test/support/helpers/sql_assertions.rb-104-
+.test/support/helpers/sql_assertions.rb:105:  def assert_sql(*patterns_to_match, **kwargs, &block)
+.test/support/helpers/sql_assertions.rb-106-    mysql_only_test!
+.test/support/helpers/sql_assertions.rb-107-
+.test/support/helpers/sql_assertions.rb-108-    result = T.let(nil, T.nilable(T::Boolean))
+.test/support/helpers/sql_assertions.rb-109-    counter = ActiveRecord::SQLCounter.new(**kwargs)
+```
+
+Unfortunately, many test helper methods are undocumented. In those cases (like the example above) the pre-context will be junk. However, there are a number of helper methods that do have very specific and narrow use cases, and those do tend to be well-documented. In those cases, you should use `read_file` to be able to read the full documentation.
+
+For example, here is the result of calling `grep(string: "def assert_sql_events")`
+
+```
+.test/support/helpers/externals_helper.rb-93-  # @example Logs events in the list that did not occur
+.test/support/helpers/externals_helper.rb-94-  #   expected_queries = { "Shop Load" => 1, "User Load" => 1 }
+.test/support/helpers/externals_helper.rb-95-  #   # Fails and reports that User Load occured 0 times instead of expected 1
+.test/support/helpers/externals_helper.rb-96-  #   assert_sql_events(expected_queries) { Shop.current_or_find(shop.id) }
+.test/support/helpers/externals_helper.rb:97:  def assert_sql_events(expected_events, &block)
+.test/support/helpers/externals_helper.rb-98-    mysql_only_test!
+.test/support/helpers/externals_helper.rb-99-
+.test/support/helpers/externals_helper.rb-100-    mysql_events = ExternalsCollector.new(&block).events
+.test/support/helpers/externals_helper.rb-101-      .select { |e| e.first == :mysql }
+```
+
+Notice that the documentation for the `assert_sql_events` method is cutoff. Use your `read_file` tool function to get the whole test helper source code and gain better understanding of how it is intended to be used, with the side benefit of also being able to see how it is implemented.
+
+Note: You will undoubtedly already be familiar with some of Minitest and RSpec's built-in helpers. There is no need to search for those, since they are packaged as gems you won't find them anyway.
+
+DO NOT FORGET TO PREPEND `def` TO YOUR QUERY TO FIND A METHOD DEFINITION INSTEAD OF USAGES, otherwise you may bring back a very large and useless result set!!!
+
+Once you are done understanding the custom test helpers used in the test file, analyze and report on whether it seems like any of the helpers are:
+
+1. Used incorrectly
+2. Used unnecessarily
+3. Any other problem related to the use of helper methods
+
+Where possible, use your best judgment to make recommendations for how to fix problems that you find, but ONLY related to test helpers.
+
+Note: You are only being used to help find problems so it is not necessary to report on correct usage of helpers or to make positive comments.

data/examples/grading/workflow.md

@@ -0,0 +1,5 @@
+As a senior software engineer and testing expert, evaluate the quality of this test file based on guidelines that will be subsequently provided.
+
+Next I will now provide the source code of the test that we will be analyzing, and then step you through a series of analysis activities, before finally asking you to provided a final report.
+
+# <%= file %> <%= File.read(file) %>

data/examples/grading/workflow.yml

@@ -0,0 +1,28 @@
+name: Test Grading
+
+tools:
+  - Roast::Tools::Grep
+  - Roast::Tools::ReadFile
+  - Roast::Tools::SearchFile
+
+steps:
+  - read_dependencies
+  -
+    - verify_test_helpers
+    - verify_mocks_and_stubs
+  - generate_grades
+  - calculate_final_grade
+  - format_result
+  - generate_recommendations
+
+analyze_coverage:
+  json: true
+
+generate_grades:
+  model: o3
+  json: true
+
+generate_recommendations:
+  model: o3
+  json: true
+

data/lib/roast/dsl/cog/config.rb

@@ -0,0 +1,31 @@
+# typed: false
+# frozen_string_literal: true
+
+module Roast
+  module DSL
+    class Cog
+      class Config
+        attr_reader :values
+
+        def initialize(initial = {})
+          @values = initial
+        end
+
+        def merge(config_object)
+          self.class.new(values.merge(config_object.values))
+        end
+
+        # It is recommended to implement a custom config object for a nicer interface,
+        # but for simple cases where it would just be a key value store we provide one by default.
+
+        def []=(key, value)
+          @values[key] = value
+        end
+
+        def [](key)
+          @values[key]
+        end
+      end
+    end
+  end
+end

data/lib/roast/dsl/cog/stack.rb

@@ -0,0 +1,21 @@
+# typed: true
+# frozen_string_literal: true
+
+module Roast
+  module DSL
+    class Cog
+      class Stack
+        delegate :map, :push, :size, :empty?, to: :@queue
+
+        def initialize
+          @queue = []
+        end
+
+        #: () -> Roast::DSL::Cog?
+        def pop
+          @queue.shift
+        end
+      end
+    end
+  end
+end

data/lib/roast/dsl/cog/store.rb

@@ -0,0 +1,26 @@
+# typed: true
+# frozen_string_literal: true
+
+module Roast
+  module DSL
+    class Cog
+      class Store
+        class CogAlreadyDefinedError < Roast::Error; end
+
+        delegate :[], to: :store
+
+        #: (Symbol, Roast::DSL::Cog) -> Roast::DSL::Cog
+        def insert(id, inst)
+          raise CogAlreadyDefinedError if store.key?(id)
+
+          store[id] = inst
+        end
+
+        #: () -> Hash[Symbol, Roast::DSL::Cog]
+        def store
+          @store ||= {}
+        end
+      end
+    end
+  end
+end

data/lib/roast/dsl/cog.rb

@@ -0,0 +1,70 @@
+# typed: true
+# frozen_string_literal: true
+
+module Roast
+  module DSL
+    class Cog
+      class CogAlreadyRanError < StandardError; end
+
+      class << self
+        def on_create
+          eigen = self
+          proc do |instance_name = Random.uuid, &action|
+            #: self as Roast::DSL::WorkflowExecutionContext
+            add_cog_instance(instance_name, eigen.new(instance_name, action))
+          end
+        end
+
+        def on_config
+          eigen = self
+          proc do |cog_name = nil, &configuration|
+            #: self as Roast::DSL::ConfigContext
+            config_object = if cog_name.nil?
+              fetch_execution_scope(eigen)
+            else
+              fetch_or_create_cog_config(eigen, cog_name)
+            end
+
+            config_object.instance_exec(&configuration) if configuration
+          end
+        end
+
+        def config_class
+          @config_class ||= find_child_config_or_default
+        end
+
+        private
+
+        def find_child_config_or_default
+          config_constant = "#{name}::Config"
+          const_defined?(config_constant) ? const_get(config_constant) : Cog::Config # rubocop:disable Sorbet/ConstantsFromStrings
+        end
+      end
+
+      attr_reader :name, :output
+
+      def initialize(name, cog_input_proc)
+        @name = name
+        @cog_input_proc = cog_input_proc
+        @finished = false
+      end
+
+      def run!(config, cog_execution_context)
+        raise CogAlreadyRanError if ran?
+
+        @config = config
+        @output = execute(cog_execution_context.instance_exec(&@cog_input_proc))
+        @finished = true
+      end
+
+      def ran?
+        @finished
+      end
+
+      # Inheriting cog must implement this
+      def execute(input)
+        raise NotImplementedError
+      end
+    end
+  end
+end

data/lib/roast/dsl/cog_execution_context.rb

@@ -0,0 +1,29 @@
+# typed: true
+# frozen_string_literal: true
+
+module Roast
+  module DSL
+    # Contains the cogs already executed in this run.
+    class CogExecutionContext
+      # Raises if you access a cog in an execution block that hasn't already been run.
+      class IncompleteCogExecutionAccessError < StandardError; end
+
+      def initialize(cogs, bound_names)
+        @cogs = cogs
+        bind_cog_methods(bound_names)
+      end
+
+      private
+
+      def bind_cog_methods(bound_names)
+        bound_names.map do |name|
+          define_singleton_method(name.to_sym, ->(name) do
+            @cogs[name].tap do |cog|
+              raise IncompleteCogExecutionAccessError unless cog.ran?
+            end.output
+          end)
+        end
+      end
+    end
+  end
+end

data/lib/roast/dsl/cogs/cmd.rb

@@ -0,0 +1,55 @@
+# typed: true
+# frozen_string_literal: true
+
+module Roast
+  module DSL
+    module Cogs
+      class Cmd < Cog
+        class Output
+          #: String?
+          attr_reader :command_output
+
+          #: String?
+          attr_reader :err
+
+          #: Process::Status?
+          attr_reader :status
+
+          #: (
+          #|  String? output,
+          #|  String? error,
+          #|  Process::Status? status
+          #| ) -> void
+          def initialize(output, error, status)
+            @command_output = output
+            @err = error
+            @status = status
+          end
+        end
+
+        class Config < Cog::Config
+          #: () -> void
+          def print_all!
+            @values[:print_all] = true
+          end
+
+          #: () -> bool
+          def print_all?
+            !!@values[:print_all]
+          end
+
+          def display!
+            print_all!
+          end
+        end
+
+        #: (String) -> Output
+        def execute(input)
+          result = Output.new(*Roast::Helpers::CmdRunner.capture3(input))
+          puts result.command_output if @config.print_all?
+          result
+        end
+      end
+    end
+  end
+end

data/lib/roast/dsl/cogs/graph.rb

@@ -0,0 +1,53 @@
+# typed: true
+# frozen_string_literal: true
+
+module Roast
+  module DSL
+    module Cogs
+      class Graph < Roast::DSL::Cog
+        #: Proc
+        attr_reader :block
+
+        #: (Symbol) { (Roast::DSL::Cogs::Graph) -> void } -> void
+        def initialize(name, &block)
+          @name = name
+          @block = block
+          @graph = Roast::Graph.new
+          super(name, proc {})
+        end
+
+        #: () -> void
+        def on_invoke
+          populate!(@graph)
+        end
+
+        #: () -> Symbol
+        def store_id
+          @name.to_sym
+        end
+
+        #: (Roast::DSL::Cog) -> void
+        def update(other)
+          return unless other.is_a?(Roast::DSL::Cogs::Graph)
+
+          return if other.block.nil?
+
+          other.populate!(@graph)
+        end
+
+        # Populates the provided graph in-place, with the definition of how to populate in the block
+        #: (Roast::DSL::Cogs::Graph) -> void
+        def populate!(graph)
+          return if @block.nil?
+
+          @block.call(graph)
+        end
+
+        #: () -> void
+        def execute
+          @graph.execute
+        end
+      end
+    end
+  end
+end

data/lib/roast/dsl/cogs.rb

@@ -0,0 +1,65 @@
+# typed: false
+# frozen_string_literal: true
+
+module Roast
+  module DSL
+    module Cogs
+      class << self
+        #: (String) -> void
+        def load_all_for(rube_rb_fpath)
+          rube_rb_fpath = File.expand_path(rube_rb_fpath)
+          all_cog_files(rube_rb_fpath).each do |cog_fpath|
+            require cog_fpath
+          end
+          bind_all_cog_invokers
+        end
+
+        #: () -> Array[Class]
+        def all_cog_classes
+          # rubocop:disable Sorbet/ConstantsFromStrings
+          Roast::DSL::Cogs.constants.map do |cog_class_name|
+            Roast::DSL::Cogs.const_get(cog_class_name)
+          end
+          # rubocop:enable Sorbet/ConstantsFromStrings
+        end
+
+        private
+
+        #: () -> void
+        def bind_all_cog_invokers
+          all_cog_classes.each do |cog_class|
+            # At some point we may want to tuck this all into a Roast::DSL::Binding/Scope/Context to avoid polluting toplevel.
+            TOPLEVEL_BINDING.eval(binding_string_for(cog_class))
+          end
+        end
+
+        #: (Class) -> String
+        def binding_string_for(cog_class)
+          <<~RUBY
+            def #{cog_class.method_name}(*args, **kwargs, &block)
+              #{cog_class.name}.invoke(*args, **kwargs, &block)
+            end
+          RUBY
+        end
+
+        #: (String) -> Array[String]
+        def all_cog_files(rube_rb_fpath)
+          dirs = [project_cogs_dir(rube_rb_fpath), internal_cogs_dir]
+          dirs.map do |dir|
+            Dir.glob(File.join(dir, "*.rb")) # Just toplevel .rb files
+          end.flatten
+        end
+
+        #: (String) -> String
+        def project_cogs_dir(rube_rb_fpath)
+          File.join(File.dirname(rube_rb_fpath), "cogs")
+        end
+
+        #: () -> String
+        def internal_cogs_dir
+          File.join(File.dirname(__FILE__), "cogs")
+        end
+      end
+    end
+  end
+end

data/lib/roast/dsl/config_context.rb

@@ -0,0 +1,54 @@
+# typed: true
+# frozen_string_literal: true
+
+module Roast
+  module DSL
+    class ConfigContext
+      def initialize(cogs, config_proc)
+        @cogs = cogs
+        @executor_scoped_configs = {}
+        @cog_scoped_configs = {}
+        @config_proc = config_proc
+      end
+
+      def fetch_merged_config(cog_class, name = nil)
+        # All configs have an entry, even if it's empty.
+        configs = fetch_execution_scope(cog_class)
+        instance_configs = fetch_cog_config(cog_class, name) unless name.nil?
+        configs = configs.merge(instance_configs) if instance_configs
+        configs
+      end
+
+      def prepare!
+        bind_default_cogs
+        instance_eval(&@config_proc)
+      end
+
+      #: () -> void
+      def bind_default_cogs
+        bind_cog(Cogs::Cmd, :cmd)
+      end
+
+      def fetch_cog_config(cog_class, name)
+        @cog_scoped_configs[cog_class][name]
+      end
+
+      def fetch_or_create_cog_config(cog_class, name)
+        @cog_scoped_configs[cog_class][name] = cog_class.config_class.new unless @cog_scoped_configs.key?(name)
+        @cog_scoped_configs[cog_class][name]
+      end
+
+      def fetch_execution_scope(cog_class)
+        @executor_scoped_configs[cog_class]
+      end
+
+      def bind_cog(cog_class, method_name)
+        @cog_scoped_configs[cog_class] = {}
+        @executor_scoped_configs[cog_class] = cog_class.config_class.new
+        instance_eval do
+          define_singleton_method(method_name, &cog_class.on_config)
+        end
+      end
+    end
+  end
+end

data/lib/roast/dsl/executor.rb

@@ -4,23 +4,78 @@
 module Roast
   module DSL
     class Executor
+      class ExecutorError < Roast::Error; end
+      class ExecutorAlreadyPreparedError < ExecutorError; end
+      class ExecutorAlreadyCompletedError < ExecutorError; end
+
       class << self
         def from_file(workflow_path)
-          execute(File.read(workflow_path))
+          run!(File.read(workflow_path))
         end
 
         private
 
-        def execute(input)
-          new.instance_eval(input)
+        def run!(workflow_definition)
+          executor = new
+          executor.prepare!(workflow_definition)
+          executor.start!
+        end
+      end
+
+      def prepare!(input)
+        # You can only initialize an executor once.
+        raise ExecutorAlreadyPreparedError if @prepared
+
+        extract_dsl_procs(input)
+        @cogs = Cog::Store.new
+        @cog_stack = Cog::Stack.new
+
+        @config_context = ConfigContext.new(@cogs, @config_proc)
+        @config_context.prepare!
+        @execution_context = WorkflowExecutionContext.new(@cogs, @cog_stack, @execution_proc)
+        @execution_context.prepare!
+
+        @prepared = true
+      end
+
+      def start!
+        # Now we run the cogs!
+        # You can only do this once, executors are not reusable to avoid state pollution
+        raise ExecutorAlreadyCompletedError if @completed
+
+        @cog_stack.map do |name, cog|
+          cog.run!(
+            @config_context.fetch_merged_config(cog.class, name.to_sym),
+            @execution_context.cog_execution_context,
+          )
         end
+
+        @completed = true
       end
 
-      # Define methods to be used in workflows below.
+      def prepared?
+        @prepared ||= false
+      end
+
+      def completed?
+        @completed ||= false
+      end
+
+      #: { () [self: ConfigContext] -> void} -> void
+      def config(&block)
+        @config_proc = block
+      end
+
+      #: { () [self: WorkflowExecutionContext] -> void} -> void
+      def execute(&block)
+        @execution_proc = block
+      end
 
-      def shell(command_string)
-        output, _status = Roast::Helpers::CmdRunner.capture2e(command_string)
-        puts output
+      # Separating the instance evals ensures that we can reuse the same cog method
+      # names between config and execute, while have the backing objects be completely
+      # different. This means we have an enforced separation between configuring and running.
+      def extract_dsl_procs(input)
+        instance_eval(input)
       end
     end
   end

data/lib/roast/dsl/workflow_execution_context.rb

@@ -0,0 +1,47 @@
+# typed: true
+# frozen_string_literal: true
+
+module Roast
+  module DSL
+    class WorkflowExecutionContext
+      def initialize(cogs, cog_stack, execution_proc)
+        @cogs = cogs
+        @cog_stack = cog_stack
+        @execution_proc = execution_proc
+        @bound_names = []
+      end
+
+      def prepare!
+        bind_default_cogs
+        instance_eval(&@execution_proc)
+      end
+
+      def cog_execution_context
+        @cog_execution_context ||= CogExecutionContext.new(@cogs, @bound_names)
+      end
+
+      private
+
+      def add_cog_instance(name, cog)
+        @cogs.insert(name, cog)
+        @cog_stack.push([name, cog])
+      end
+
+      def output(name)
+        @cogs[name].output
+      end
+
+      #: () -> void
+      def bind_default_cogs
+        bind_cog(Cogs::Cmd, :cmd)
+      end
+
+      def bind_cog(cog_class, name)
+        @bound_names << name
+        instance_eval do
+          define_singleton_method(name, &cog_class.on_create)
+        end
+      end
+    end
+  end
+end

data/lib/roast/error.rb

@@ -0,0 +1,7 @@
+# typed: true
+# frozen_string_literal: true
+
+module Roast
+  # Base class for all internal Roast errors.
+  class Error < StandardError; end
+end