roast-ai 0.4.8 → 0.4.10

data/dsl/plugin-gem-example/plugin-gem-example.gemspec

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+require_relative "lib/version"
+
+Gem::Specification.new do |spec|
+  spec.name = "plugin-gem-example"
+  spec.version = Plugin::Gem::Example::VERSION
+  spec.authors = ["Sam Schmidt"]
+  spec.email = ["sam.schmidt@shopify.com"]
+
+  spec.summary = "TODO: Write a short summary, because RubyGems requires one."
+  spec.description = "TODO: Write a longer description or delete this line."
+  spec.homepage = "TODO: Put your gem's website or public repo URL here."
+  spec.license = "MIT"
+  spec.required_ruby_version = ">= 3.1.0"
+
+  spec.metadata["allowed_push_host"] = "TODO: Set to your gem server 'https://example.com'"
+
+  spec.metadata["homepage_uri"] = spec.homepage
+  spec.metadata["source_code_uri"] = "TODO: Put your gem's public repo URL here."
+  spec.metadata["changelog_uri"] = "TODO: Put your gem's CHANGELOG.md URL here."
+
+  # Uncomment to register a new dependency of your gem
+  spec.add_dependency("roast-ai")
+
+  # For more information and examples about making a new gem, check out our
+  # guide at: https://bundler.io/guides/creating_gem.html
+end

data/dsl/prototype.rb

@@ -0,0 +1,25 @@
+# typed: true
+# frozen_string_literal: true
+
+#: self as Roast::DSL::Workflow
+
+config do
+  # TODO: make a way to do print_all on all and turn off printing on one
+  cmd(:echo) { print_all! }
+  cmd(:date) { print_stdout! }
+  chat { "" }
+  # TODO: add a way to apply config to a subset of named cog by pattern
+  # cmd(/^date.*/) { print_all! }
+end
+
+execute do
+  # Anonymous cog. Added to the stack directly and given an autogenerated key in cog storage
+  # Use for actions you do once and forget about, don't need configuration
+  cmd { |my| my.command = "touch tmp/#{SecureRandom.uuid}" }
+
+  # Named cog. Configuration for this specific instance will be looked up from config block
+  cmd(:echo) { |my| my.command = "echo 'Hello World!'" }
+
+  # Cogs can implement input coercion for simple return values
+  cmd(:date) { "date" }
+end

data/dsl/scoped_executors.rb

@@ -0,0 +1,28 @@
+# typed: true
+# frozen_string_literal: true
+
+#: self as Roast::DSL::Workflow
+
+config do
+  cmd do
+    print_all!
+  end
+end
+
+execute(:capitalize_a_random_word) do
+  cmd(:word) { "shuf /usr/share/dict/words -n 1" }
+  cmd(:capitalize) do |my|
+    word = cmd(:word).out.strip
+    my.command = "sh"
+    my.args << "-c"
+    my.args << "echo '#{word}' | tr '[:lower:]' '[:upper:]'"
+  end
+end
+
+execute do
+  cmd(:before) { "echo '--> before'" }
+  execute { :capitalize_a_random_word }
+  execute { :capitalize_a_random_word }
+  execute { :capitalize_a_random_word }
+  cmd(:after) { "echo '--> after'" }
+end

data/dsl/simple.rb

@@ -1,10 +1,8 @@
-# typed: true
+# typed: false
 # frozen_string_literal: true
 
-#: self as Roast::DSL::Executor
-
-# This is a dead simple workflow that calls two shell scripts
-shell <<~SHELLSTEP
+# This is a dead simple workflow that calls two commands
+cmd <<~CMDSTEP
   echo "I have no idea what's going on"
-SHELLSTEP
-shell "pwd"
+CMDSTEP
+cmd "pwd"

data/dsl/simple_chat.rb

@@ -0,0 +1,12 @@
+# typed: true
+# frozen_string_literal: true
+
+#: self as Roast::DSL::Workflow
+
+config do
+  # ...
+end
+
+execute do
+  chat(:lake) { "What is the deepest lake?" }
+end

data/dsl/step_communication.rb

@@ -0,0 +1,24 @@
+# typed: true
+# frozen_string_literal: true
+
+#: self as Roast::DSL::Workflow
+
+# How do we pass information between steps?
+# Demonstrate by passing result of a command output to another step
+
+config do
+  cmd(:echo) { display! }
+end
+
+execute do
+  cmd(:ls) { "ls -al" }
+  cmd(:echo) do |my|
+    my.command = "echo"
+    # TODO: this is a bespoke output object for cmd, is there a generic one we can offer
+    first_line = cmd(:ls).out.split("\n").second
+    last_line = cmd(:ls).out.split("\n").last
+    my.args << first_line unless first_line.blank?
+    my.args << "\n---\n"
+    my.args << last_line if last_line != first_line && last_line.present?
+  end
+end

data/examples/grading/README.md

@@ -0,0 +1,46 @@
+# Test Grading Workflow
+
+This workflow acts as a senior software engineer and testing expert to evaluate the quality of test files based on best practices and guidelines.
+
+## Usage
+
+```bash
+# Run the grading workflow on a test file
+roast execute examples/grading/workflow.yml path/to/your_test.rb
+```
+
+## How it Works
+
+1. **read_dependencies**: Analyzes the test file and its dependencies
+2. **run_coverage**: Executes the test with coverage tracking
+3. **generate_grades**: Evaluates test quality across multiple dimensions
+4. **verify_test_helpers**: Checks for proper test helper usage
+5. **verify_mocks_and_stubs**: Ensures appropriate use of test doubles
+6. **analyze_coverage**: Reviews code coverage metrics
+7. **generate_recommendations**: Provides improvement suggestions
+8. **calculate_final_grade**: Computes an overall grade (A-F scale)
+9. **format_result**: Formats the final output
+
+## Customization
+
+Feel free to adapt this workflow to your testing environment:
+
+- **Different test frameworks**: Modify `run_coverage.rb` to work with RSpec, Jest, pytest, etc.
+- **Coverage tools**: Replace the coverage command with your preferred tool (SimpleCov, Istanbul, Coverage.py)
+- **Grading criteria**: Adjust the prompts in each step to match your team's standards
+
+## Example Output
+
+```
+========== TEST GRADE REPORT ==========
+Test file: test/example_test.rb
+
+FINAL GRADE:
+  Score: 85/100
+  Letter Grade: B
+
+RECOMMENDATIONS:
+- Add edge case testing for error conditions
+- Improve test descriptions for clarity
+- Consider extracting common setup to helper methods
+```

data/examples/grading/analyze_coverage/prompt.md

@@ -0,0 +1,52 @@
+<coverage_results>
+<%= workflow.output["run_coverage"] %>
+</coverage_results>
+
+Analyze the results and score them on a scale of 1-10 using the following rubrics:
+
+<line_coverage>
+0-1: Critical failure (0-20% coverage) - Core functionality remains completely untested
+2-3: Poor coverage (21-40%) - Major gaps; many key functions lack any testing
+4-5: Inadequate coverage (41-60%) - Several important code paths are not executed
+6-7: Moderate coverage (61-80%) - Notable gaps remain; some important functionality lacks coverage
+8-9: Good coverage (81-95%) - Only minor or edge case code paths remain untested
+10: Excellent coverage (96-100%)
+</line_coverage>
+
+<branch_coverage>
+0-1: Critical failure (0-20% branch coverage) - Almost no conditional branches are tested
+2-3: Poor coverage (21-40%) - Most conditional logic remains untested
+4-5: Inadequate coverage (41-60%) - Many conditions are only tested for one outcome
+6-7: Moderate coverage (61-80%) - Some conditions lack testing for all outcomes
+8-9: Good coverage (81-95%) - Most conditions are tested for most outcomes
+10: Excellent coverage (96-100%)
+</branch_coverage>
+
+<method_coverage>
+0-1: Critical failure (0-20% method coverage) - Most or core functionality methods are untested
+2-3: Poor coverage (21-40%) - Several public API methods remain untested
+4-5: Inadequate coverage (41-60%) - Some important public methods lack tests
+6-7: Moderate coverage (61-80%) - Notable gaps remain; some public methods may lack comprehensive testing
+8-9: Good coverage (81-95%) - Nearly all public methods are tested; private methods are mostly covered via public method tests
+10: Excellent coverage (96-100%)
+</method_coverage>
+
+RESPONSE FORMAT
+You must respond in JSON format within <json> XML tags. Example:
+
+<json>
+{
+  "method_coverage": {
+    "score": "10",
+    "justification": "The source file has 100% method coverage, indicating all methods are being tested."
+  },
+  "line_coverage": {
+    "score": 10,
+    "justification": "The source file has 100% line coverage, indicating all executable lines are tested."
+  },
+  "branch_coverage": {
+    "score": 8,
+    "justification": "The source file has 80% branch coverage, indicating some branches need testing."
+  }
+}
+</json>

data/examples/grading/calculate_final_grade.rb

@@ -0,0 +1,64 @@
+# frozen_string_literal: true
+
+class CalculateFinalGrade < Roast::Workflow::BaseStep
+  WEIGHTS = {
+    test_helpers: 0.2,
+    mocks_and_stubs: 0.2,
+    readability: 0.2,
+    maintainability: 0.2,
+    effectiveness: 0.2,
+  }.freeze
+
+  def call
+    llm_analysis = workflow.output["generate_grades"]
+
+    weighted_sum = WEIGHTS.sum do |criterion, weight|
+      score = llm_analysis[criterion.to_s]["score"].to_f / 10.0
+      score * weight
+    end
+
+    {
+      final_score: {
+        weighted_score: weighted_sum,
+        letter_grade: calculate_letter_grade(weighted_sum),
+      },
+      rubric_scores: calculate_rubric_scores(llm_analysis),
+    }
+  end
+
+  private
+
+  def calculate_letter_grade(score)
+    case score
+    when 0.9..1.0
+      "A"
+    when 0.8...0.9
+      "B"
+    when 0.7...0.8
+      "C"
+    when 0.6...0.7
+      "D"
+    else
+      "F"
+    end
+  end
+
+  def calculate_rubric_scores(llm_analysis)
+    scores = {}
+
+    WEIGHTS.each_key do |criterion|
+      next 1 unless llm_analysis[criterion.to_s]
+      raw_score = llm_analysis[criterion.to_s]["score"].to_f
+      normalized_score = raw_score / 10.0
+
+      scores[criterion] = {
+        raw_value: raw_score,
+        score: normalized_score,
+        description: llm_analysis[criterion.to_s]["justification"],
+        weighted_score: normalized_score * WEIGHTS[criterion],
+      }
+    end
+
+    scores
+  end
+end

data/examples/grading/format_result.rb

@@ -0,0 +1,61 @@
+# frozen_string_literal: true
+
+class FormatResult < Roast::Workflow::BaseStep
+  RUBRIC = {
+    test_helpers: { description: "Test Helpers Usage", weight: 0.2 },
+    mocks_and_stubs: { description: "Mocks and Stubs Usage", weight: 0.2 },
+    readability: { description: "Test Readability", weight: 0.2 },
+    maintainability: { description: "Test Maintainability", weight: 0.2 },
+    effectiveness: { description: "Test Effectiveness", weight: 0.2 },
+  }.freeze
+
+  def call
+    append_to_final_output(<<~OUTPUT)
+      ========== TEST GRADE REPORT ==========
+      Test file: #{workflow.file}
+    OUTPUT
+
+    format_results
+    append_to_final_output("\n\n")
+  end
+
+  private
+
+  def format_results
+    # With HashWithIndifferentAccess, we can simply access with either syntax
+    grade_data = workflow.output["calculate_final_grade"]
+
+    unless grade_data
+      return append_to_final_output("Error: Grading data not available. This may be because you're replaying the workflow from this step, but the previous step data is missing or not found in the selected session.")
+    end
+
+    format_grade(grade_data)
+
+    # Make sure rubric_scores exists before trying to iterate over it
+    unless grade_data[:rubric_scores]
+      return append_to_final_output("Error: Rubric scores data not available in the workflow output.")
+    end
+
+    append_to_final_output("RUBRIC SCORES:")
+    grade_data[:rubric_scores].each do |category, data|
+      # Safely access RUBRIC with a fallback for potentially missing categories
+      rubric_item = RUBRIC[category.to_sym] || { description: "Unknown Category", weight: 0 }
+
+      append_to_final_output("  #{rubric_item[:description]} (#{(rubric_item[:weight] * 100).round}% of grade):")
+      append_to_final_output("    Value: #{data[:raw_value] || "N/A"}")
+      append_to_final_output("    Score: #{data[:score] ? (data[:score] * 10).round : "N/A"}/10 - \"#{data[:description] || "No description available"}\"")
+    end
+  end
+
+  def format_grade(grade_data)
+    return append_to_final_output("\nError: Final grade data not available.") unless grade_data && grade_data[:final_score]
+
+    letter_grade = grade_data[:final_score][:letter_grade]
+    celebration_emoji = letter_grade == "A" ? "🎉" : ""
+    append_to_final_output(<<~OUTPUT)
+      \nFINAL GRADE:
+        Score: #{(grade_data[:final_score][:weighted_score] * 100).round}/100
+        Letter Grade: #{letter_grade} #{celebration_emoji}
+    OUTPUT
+  end
+end

data/examples/grading/generate_grades/prompt.md

@@ -0,0 +1,105 @@
+These are the key testing guidelines to consider in your evaluation:
+
+- Tests should serve as specifications that define expected behaviors
+- Tests should have descriptive names that clearly communicate intent
+- Tests should focus on behavior rather than implementation details
+- Excessive mocking/stubbing should be avoided in favor of testing real behavior
+- Tests should be well-structured with minimal setup complexity
+- Tests should be maintainable and not break when implementation details change
+- Tests should cover edge cases and error conditions
+- Tests should follow proper naming conventions and directory structure
+- Tests should not modify the behaviour of the code being tested (e.g. making a private method public in tests)
+
+Now consider the full transcript and evaluate the test being graded based on the following rubrics on a scale of 1-10:
+
+<test_helpers>
+0-1: Extremely poor helper usage - Helpers used incorrectly or inappropriately, making tests harder to understand
+2-3: Poor helper usage - Helpers are poorly designed, tightly coupled to implementation, or used incorrectly
+4-5: Basic helper usage - Helpers work but may be poorly organized or not reusable
+6-7: Good helper usage - Helpers are well-designed and used appropriately
+8-9: Very good helper usage - Helpers are well-factored, reusable, and make tests clearer
+10: Excellent helper usage - Helpers are perfectly designed, highly reusable, and significantly improve test clarity and maintainability. Also give this score to tests that DO NOT use test helpers at all.
+</test_helpers>
+
+<mocks_and_stubs>
+0-1: Extremely poor mocking - Mocks/stubs used incorrectly or excessively, completely hiding real behavior
+2-3: Poor mocking - Heavy reliance on mocks that couple tests to implementation; mocks don't match real behavior
+4-5: Basic mocking - Mocks used appropriately but may be overused or not match implementation exactly
+6-7: Good mocking - Mocks used judiciously where needed; generally match implementation
+8-9: Very good mocking - Minimal mocking focused on external dependencies; accurately reflects real behavior
+10: Excellent mocking - Mocks used only where absolutely necessary (external APIs, etc); perfectly match real implementations; maintain loose coupling
+</mocks_and_stubs>
+
+<readability>
+0-1: Extremely poor readability - Test purpose is impossible to understand; no structure or organization
+2-3: Poor readability - Test names are vague or misleading; structure is confusing with no clear assertions
+4-5: Basic readability - Structure is understandable but not optimized for clarity
+6-7: Good readability - Structure is logical with clear assertions
+8-9: Very readable - Well-organized with explicit, meaningful test names and assertions
+10: Exceptionally readable - Test names serve as perfect specifications; elegant structure with context-providing descriptions; self-documenting with clear setup, execution, and verification phases
+</readability>
+
+<maintenability>
+0-1: Extremely brittle - Tests are completely coupled to implementation details
+2-3: Highly unmaintainable - Will require significant rework when code changes because of heavy coupling to implementation details
+4-5: Somewhat maintainable - Some coupling to implementation details
+6-7: Reasonably maintainable - Tests mostly focus on behavior over implementation; limited coupling to implementation details
+8-9: Highly maintainable - Tests focus on behavior rather than implementation; changes to implementation should rarely break tests
+10: Exceptionally maintainable - Tests purely focus on behavior and public interfaces; implementation can be completely refactored without breaking tests; well-factored test helpers and fixtures
+</maintenability>
+
+<effectiveness>
+0-1: Ineffective - Don't validate actual behavior and could pass even if code is broken
+2-3: Minimally effective - Only the most basic functionality validated. Many incorrect behaviors would not be caught
+4-5: Partially effective - Only catch obvious issues but miss subtle bugs; limited validation of actual outcomes
+6-7: Reasonably effective - Should catch most common bugs
+8-9: Highly effective - Should catch nearly all bugs
+10: Exceptionally effective - Should catch even subtle edge case bugs; validate both positive and negative cases
+</effectiveness>
+
+While grading, consider the following goals as being applicable across all rubrics:
+
+SUBJECTIVE:
+- Well-written: Organized, easy to understand, and follow best practices
+- Real behavior: Validate what the code does rather than implementation details
+- Isolated: Should not depend on external systems, services, or APIs. Note: The use of fixtures such as `shops(:snowdevil)` is expected and should not be penalized. The only exception is when the SUT is being loaded as a fixture unnecessarily when it could be instantiated directly.
+
+OBJECTIVE
+- Idempotent: Should be able to run repeatedly without affecting outcome or side effects.
+- Deterministic: Should produce the same results across all runs and environments.
+- No sleep: Does not include sleep calls or rely on timing for synchronization.
+- Concurrent: Properly handles concurrent execution paths without errors.
+- Timeless: Does not depend on the current date or time. Will not fail due to changes such as daylight savings or leap years. Specifically with regards to handling time, look for anti-patterns like `Time.current + 7.days.to_i`, which fails on DST changes. The correct approach is `7.days.from_now`.
+
+VIOLATING ANY OBJECTIVE GOAL SHOULD RESULT IN AN OVERALL SCORE LESS THAN 5!
+
+Provide a brief justification for each score, using a maximum of 1-3 sentences. (Note that specific recommendations for improvement are not needed at this step.)
+
+You are acting as a stern and relentless striver for excellence in programming, so you must be highly critical. The point of this grading exercise is to facilitate substantial improvement, not just stroking the programmer's ego. Do not hesitate to give a failing overall score (0) for serious violations!
+
+RESPONSE FORMAT: You must respond in JSON format within <json> XML tags.
+
+<json>
+{
+  "test_helpers": {
+    "score": 4,
+    "justification": "Helpers are used incorrectly in several places, reducing test maintainability and clarity. The assert_valid_record helper is being misused with hashes instead of model instances."
+  },
+  "mocks_and_stubs": {
+    "score": 4,
+    "justification": "Several mocks don't match the actual implementation, making tests brittle and potentially hiding production bugs. For example, mocking success: true when the service returns status: 'success'."
+  },
+  "readability": {
+    "score": 8,
+    "justification": "Test names clearly describe behavior being tested."
+  },
+  "maintainability": {
+    "score": 6,
+    "justification": "Tests mostly focus on behavior but have some coupling to implementation."
+  },
+  "effectiveness": {
+    "score": 7,
+    "justification": "Tests validate most expected behaviors and would catch common bugs."
+  }
+}
+</json>

data/examples/grading/generate_recommendations/output.txt

@@ -0,0 +1,17 @@
+========== TEST RECOMMENDATIONS ==========
+<%- if response.recommendations.empty? -%>
+No recommendations found.
+<%- else -%>
+<%- response.recommendations.each_with_index do |rec, index| -%>
+Recommendation #<%= index + 1 %>:
+Description: <%= rec.description %>
+Impact: <%= rec.impact %>
+Priority: <%= rec.priority %>
+
+Code Suggestion:
+
+<%= rec.code_suggestion %>
+
+<%- end -%>
+<%- end -%>
+===========================================

data/examples/grading/generate_recommendations/prompt.md

@@ -0,0 +1,60 @@
+Finally, based on the conversation transcript above, go ahead and provide specific, actionable recommendations that would most effectively improve the overall test score.
+
+Focus on recommendations that would:
+
+1. Increase coverage
+2. Add more assertions where needed
+3. Make the tests more maintainable or readable
+4. Ensure tests serve as specifications by having clear, descriptive names
+5. Reduce excessive mocking/stubbing that couples tests to implementation details
+6. Improve test structure to reduce setup complexity
+7. Ensure tests focus on behavior rather than implementation details
+8. Ensure gaps in private methods are tested through public methods
+9. Fix any issues with test helpers that are used incorrectly or unnecessarily
+10. Improve efficiency by combining or deleting tests where appropriate (note that having more than one assertion per test is acceptable)
+11. Fix any violations of the objective criteria (idempotency, determinism, etc.)
+12. Be specific about edge cases that should be covered by tests. Write down in the recommendations which edge cases you are referring to.
+13. Do not recommend the use of RSpec features like `let` for Minispec tests.
+
+IF YOU IDENTIFY EDGE CASES, YOU MUST BE SPECIFIC ABOUT THEM IN THE RECOMMENDATIONS.
+
+RESPONSE FORMAT: You must respond in JSON format inside <json> XML tags without additional commentary.
+
+Example:
+
+<json>
+{
+  "recommendations": [
+    {
+      "description": "Add tests for uncovered method X",
+      "impact": "Would increase method coverage by Y%",
+      "priority": "High",
+      "code_suggestion": "def test_method_x_with_valid_input\n  result = subject.method_x('valid_input')\n  assert_equal expected_result, result\nend"
+    },
+    {
+      "description": "Fix time handling to avoid DST issues",
+      "impact": "Would make tests deterministic across DST changes",
+      "priority": "High",
+      "code_suggestion": "# Replace\nexpiry_time = Time.current + 7.days.to_i\n\n# With\nexpiry_time = 7.days.from_now"
+    },
+    {
+      "description": "Add edge case tests for the show action for when the parameter X is blank",
+      "impact": "Would improve test completeness and effectiveness",
+      "priority": "Medium",
+      "code_suggestion": "..."
+    },  
+    {
+      "description": "Improve test descriptions to better serve as specifications",
+      "impact": "Would make tests more valuable as documentation",
+      "priority": "Medium",
+      "code_suggestion": "# Replace\ndef test_process\n\n# With\ndef test_process_returns_success_with_valid_input"
+    },
+    {
+      "description": "Replace implementation-focused mocks with behavior assertions",
+      "impact": "Would make tests less brittle and more maintainable",
+      "priority": "High",
+      "code_suggestion": "# Replace\nUserNotifier.expects(:notify).with(user, 'welcome')\n\n# With\nassert_sends_notification(user, 'welcome') do\n  subject.process\nend"
+    }
+  ]
+}
+</json>

data/examples/grading/read_dependencies/prompt.md

@@ -0,0 +1,15 @@
+Use the provided functions to find and read important dependencies of the provided test file named <%= workflow.file %>.
+
+The first dependency you should always look for is the source file for the prime subject of the test (whatever class this test file is claiming to test). Use `read_file` to read the subject's source code into your conversation transcript, but only if it's not already there from a previous chat.
+
+If you can identify other important application-level dependencies then read them too.
+How many extra dependencies to research is left to your discretion, but ALWAYS make sure you have the subject under test (SUT) in your context before responding.
+
+Once you are finished using tool functions, respond with the relative path to the source file of the SUT inside <sut> tags. IMPORTANT: Include the full relative path from the project root, including any directory prefixes like lib/, app/, etc.
+
+Example:
+
+If you are told to find the dependencies of `test/services/country_db_interface_test.rb`,
+then you would use the functions as explained above and ultimately respond with `<sut>app/services/country_db_interface.rb</sut>`
+
+If the file is found at `lib/roast/workflow/workflow_initializer.rb`, respond with `<sut>lib/roast/workflow/workflow_initializer.rb</sut>` (include the lib/ prefix)

data/examples/grading/verify_mocks_and_stubs/prompt.md

@@ -0,0 +1,12 @@
+Find places in the provided test code where stubbing and mocking are used. Search for the corresponding implementation source code of those dependencies elsewhere in the codebase to validate that the stub or mock matches the implementation that it is doubling. Use the tool functions provided to find and read the dependencies.
+
+Once you've found the dependencies, verify that any mocks and stubs accurately reflect the real implementation. If there are discrepancies, list them out alphabetically with:
+
+1. The name of the mocked/stubbed method
+2. What the mock/stub expects in terms of arguments and/or return values
+3. What the actual implementation actually takes as arguments and returns
+4. Suggestions for fixing the discrepancy
+
+Note: If there are no discrepancies, do not summarize those that accurately reflect their real implementations in the codebase, just respond "All mocks and stubs verified."
+
+IMPORTANT: There's absolutely no need for you to waste time grepping for methods/functions that you know belong to testing libraries such as Mocha's `expects` and `stubs`. Only search for the implementation of things that are stubbed and/or mocked in the test to verify whether the test code matches the implementation code.

data/examples/grading/verify_test_helpers/prompt.md

@@ -0,0 +1,53 @@
+Now identify custom test helpers used in this test for the following purpose:
+
+1. Analyzing if they are used correctly
+2. Understanding test code that has had significant chunks of implementation abstracted away into helpers
+3. Fully understanding custom assertions that are not included by default in Ruby on Rails or part of your base knowledge
+
+Your grep tool function is vital for this work. It provides 4 lines of context before and after the matching line.
+
+For example, if you call `grep(string: "def assert_sql")`, the output will include:
+
+```
+.test/support/helpers/sql_assertions.rb-101-    end
+.test/support/helpers/sql_assertions.rb-102-    result
+.test/support/helpers/sql_assertions.rb-103-  end
+.test/support/helpers/sql_assertions.rb-104-
+.test/support/helpers/sql_assertions.rb:105:  def assert_sql(*patterns_to_match, **kwargs, &block)
+.test/support/helpers/sql_assertions.rb-106-    mysql_only_test!
+.test/support/helpers/sql_assertions.rb-107-
+.test/support/helpers/sql_assertions.rb-108-    result = T.let(nil, T.nilable(T::Boolean))
+.test/support/helpers/sql_assertions.rb-109-    counter = ActiveRecord::SQLCounter.new(**kwargs)
+```
+
+Unfortunately, many test helper methods are undocumented. In those cases (like the example above) the pre-context will be junk. However, there are a number of helper methods that do have very specific and narrow use cases, and those do tend to be well-documented. In those cases, you should use `read_file` to be able to read the full documentation.
+
+For example, here is the result of calling `grep(string: "def assert_sql_events")`
+
+```
+.test/support/helpers/externals_helper.rb-93-  # @example Logs events in the list that did not occur
+.test/support/helpers/externals_helper.rb-94-  #   expected_queries = { "Shop Load" => 1, "User Load" => 1 }
+.test/support/helpers/externals_helper.rb-95-  #   # Fails and reports that User Load occured 0 times instead of expected 1
+.test/support/helpers/externals_helper.rb-96-  #   assert_sql_events(expected_queries) { Shop.current_or_find(shop.id) }
+.test/support/helpers/externals_helper.rb:97:  def assert_sql_events(expected_events, &block)
+.test/support/helpers/externals_helper.rb-98-    mysql_only_test!
+.test/support/helpers/externals_helper.rb-99-
+.test/support/helpers/externals_helper.rb-100-    mysql_events = ExternalsCollector.new(&block).events
+.test/support/helpers/externals_helper.rb-101-      .select { |e| e.first == :mysql }
+```
+
+Notice that the documentation for the `assert_sql_events` method is cutoff. Use your `read_file` tool function to get the whole test helper source code and gain better understanding of how it is intended to be used, with the side benefit of also being able to see how it is implemented.
+
+Note: You will undoubtedly already be familiar with some of Minitest and RSpec's built-in helpers. There is no need to search for those, since they are packaged as gems you won't find them anyway.
+
+DO NOT FORGET TO PREPEND `def` TO YOUR QUERY TO FIND A METHOD DEFINITION INSTEAD OF USAGES, otherwise you may bring back a very large and useless result set!!!
+
+Once you are done understanding the custom test helpers used in the test file, analyze and report on whether it seems like any of the helpers are:
+
+1. Used incorrectly
+2. Used unnecessarily
+3. Any other problem related to the use of helper methods
+
+Where possible, use your best judgment to make recommendations for how to fix problems that you find, but ONLY related to test helpers.
+
+Note: You are only being used to help find problems so it is not necessary to report on correct usage of helpers or to make positive comments.

data/examples/grading/workflow.md

@@ -0,0 +1,5 @@
+As a senior software engineer and testing expert, evaluate the quality of this test file based on guidelines that will be subsequently provided.
+
+Next I will now provide the source code of the test that we will be analyzing, and then step you through a series of analysis activities, before finally asking you to provided a final report.
+
+# <%= file %> <%= File.read(file) %>