ruby-skill-bench 0.1.0

data/docs/testing-guide.md

@@ -0,0 +1,361 @@
+# Testing Guide: Evaluations & Workflows
+
+This guide explains how to run evaluations and how to create new evaluation tasks for skills and workflows.
+
+## Running Evaluations
+
+The primary tool for running evaluations is the `skill-bench` CLI.
+
+### Basic Usage
+
+To run a specific evaluation task:
+
+```bash
+skill-bench run my-eval --skill=my-skill
+```
+
+Provider is read from `skill-bench.json` — no `--provider` flag needed.
+
+### Output Formats
+
+**Human-readable (default):**
+
+```text
+═══════════════════════════════════════════════════════
+  Eval: my-eval
+  Skill: my-skill
+  Provider: openai
+═══════════════════════════════════════════════════════
+
+  DIMENSION                BASELINE   CONTEXT    DELTA
+  ──────────────────────── ───────── ───────── ───────
+  Correctness (30)                12        28    +16
+  Skill Adherence (25)             5        22    +17
+  Code Quality (20)               10        16     +6
+  Test Coverage (15)               3        13    +10
+  Documentation (10)               2         8     +6
+  ──────────────────────── ───────── ───────── ───────
+  TOTAL                          32/100    87/100   +55
+
+  VERDICT: PASS (threshold: 70, minimum delta: 10)
+═══════════════════════════════════════════════════════
+```
+
+**JSON:**
+
+```bash
+skill-bench run my-eval --skill=my-skill --format json
+```
+
+**JUnit XML:**
+
+```bash
+skill-bench run my-eval --skill=my-skill --format=junit
+```
+
+### Batch Processing
+
+To run an eval with a path containing a slash:
+
+```bash
+skill-bench run evals/my-eval --skill=my-skill
+```
+
+The evaluator resolves the path automatically.
+
+### Overriding Skill Context
+
+By default, the evaluator infers the skill path from the evaluation path. If you need to test an evaluation against a different skill:
+
+```bash
+skill-bench run my-eval --skill=skills/custom-skill
+```
+
+## Creating New Evaluations
+
+An evaluation task consists of a directory containing at least two files: `task.md` and `criteria.json`.
+
+### 1. The Task (`task.md`)
+
+This file contains the instructions for the AI agent. It should describe a specific problem to solve or a feature to implement.
+
+**Best Practices:**
+
+- Provide clear context and requirements.
+- Include a description of the current codebase state.
+- Specify the desired outcome.
+- List acceptance criteria as numbered items (the judge checks these).
+
+**Example — Good task.md:**
+
+```markdown
+Create a `PasswordValidator` class that:
+
+1. Accepts a `password` string
+2. Validates minimum length of 8 characters
+3. Validates presence of at least one uppercase letter
+4. Validates presence of at least one digit
+5. Returns `{ valid: true }` or `{ valid: false, errors: [...] }`
+6. Includes RSpec tests with 100% branch coverage
+7. Uses `# frozen_string_literal: true`
+8. Has YARD docs for the class and all public methods
+```
+
+**Why this works:** Each numbered item is a discrete acceptance criterion the judge can verify independently. Vague tasks like "create a password validator" produce inconsistent scores because the judge has to guess what "good" means.
+
+### 2. The Criteria (`criteria.json`)
+
+This file defines the evaluation dimensions, weights, and thresholds:
+
+```json
+{
+  "context": "Evaluate whether the skill helps build a proper API REST collection",
+  "dimensions": [
+    { "name": "correctness", "max_score": 30 },
+    { "name": "skill_adherence", "max_score": 25 },
+    { "name": "code_quality", "max_score": 20 },
+    { "name": "test_coverage", "max_score": 15 },
+    { "name": "documentation", "max_score": 10 }
+  ],
+  "pass_threshold": 70,
+  "minimum_delta": 10
+}
+```
+
+**Fields:**
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `context` | string | Yes | Shown to the judge. Describes what the eval measures. |
+| `dimensions` | array | Yes | Array of `{ name, max_score }` objects. Must include all 5 core dimensions. `max_score` values must sum to exactly 100. |
+| `pass_threshold` | integer | No | Minimum context score to pass. Default: 70. |
+| `minimum_delta` | integer | No | Minimum improvement over baseline to pass. Default: 10. |
+| `description` (per dimension) | string | No | Overrides the built-in default description for that dimension. |
+
+**Custom dimension descriptions** are especially useful when a skill has specific hard rules. For example, if your skill requires the `.call` pattern, you can tell the judge exactly what to look for:
+
+```json
+{
+  "name": "skill_adherence",
+  "max_score": 25,
+  "description": "Did the agent create a class with a `.call` class method that returns `{ success: bool, response: { ... } }`?"
+}
+```
+
+This produces more consistent scores than the generic default description.
+
+### 3. What the Judge Sees
+
+Understanding the judge prompt helps you write better tasks and criteria. The judge receives a structured prompt with four sections:
+
+```text
+## Task
+[Contents of task.md]
+
+## Criteria
+Context: [Contents of criteria.json context]
+Dimensions:
+- correctness: max_score=30, description=...
+- skill_adherence: max_score=25, description=...
+...
+
+## Skill Context
+[Contents of SKILL.md wrapped in XML]
+
+## Agent Output
+[Git diff + file listing + reasoning excerpt]
+
+## Instructions
+Score each dimension independently. Return JSON with:
+- "dimensions": object mapping each dimension name to { "score": number, "max_score": number, "reasoning": string }
+- "overall_reasoning": string summarizing the evaluation
+```
+
+**Important:** The judge is called **twice** per eval — once for baseline output (no skill context section) and once for context output (with skill context). The judge never sees both outputs in the same call. This prevents the judge from being biased by direct comparison.
+
+---
+
+## Evaluating Workflows vs. Skills
+
+### Atomic Skills
+
+Skills are isolated blocks of logic (e.g., a specific API pattern). Evaluations for skills should focus strictly on the adherence to the patterns defined in the skill's `SKILL.md`.
+
+**Recommended weights for atomic skills:**
+
+```json
+{
+  "dimensions": [
+    { "name": "correctness", "max_score": 30 },
+    { "name": "skill_adherence", "max_score": 30 },
+    { "name": "code_quality", "max_score": 20 },
+    { "name": "test_coverage", "max_score": 10 },
+    { "name": "documentation", "max_score": 10 }
+  ],
+  "pass_threshold": 70,
+  "minimum_delta": 10
+}
+```
+
+Skill Adherence is weighted highest because the core question is "did the skill help?"
+
+### Workflows
+
+Workflows are sequences of skills or complex orchestrations (e.g., the full TDD loop). Evaluations for workflows should focus on the process, the ordering of tasks, and the successful completion of a multi-step objective.
+
+**Recommended weights for workflows:**
+
+```json
+{
+  "dimensions": [
+    { "name": "correctness", "max_score": 35 },
+    { "name": "skill_adherence", "max_score": 20 },
+    { "name": "code_quality", "max_score": 20 },
+    { "name": "test_coverage", "max_score": 15 },
+    { "name": "documentation", "max_score": 10 }
+  ],
+  "pass_threshold": 65,
+  "minimum_delta": 15
+}
+```
+
+Correctness is weighted higher because workflows are judged on end-to-end success. The `minimum_delta` is also higher (15 vs 10) because workflows are expected to show stronger skill impact.
+
+---
+
+## Interpreting the Output
+
+### Human-Readable Format
+
+```text
+═══════════════════════════════════════════════════════
+  Eval: my-eval
+  Skill: my-skill
+  Provider: openai
+═══════════════════════════════════════════════════════
+
+  DIMENSION                BASELINE   CONTEXT    DELTA
+  ──────────────────────── ───────── ───────── ───────
+  Correctness (30)                12        28     +16
+  Skill Adherence (25)             5        22     +17
+  Code Quality (20)               10        16      +6
+  Test Coverage (15)               3        13     +10
+  Documentation (10)               2         8      +6
+  ──────────────────────── ───────── ───────── ───────
+  TOTAL                          32/100    87/100   +55
+
+  TREND: baseline ↑ (+2), context ↑ (+7)
+  VERDICT: PASS (threshold: 70, minimum delta: 10)
+═══════════════════════════════════════════════════════
+```
+
+**Reading the table:**
+
+- **BASELINE:** What the agent produced *without* the skill. Think of this as "raw" ability.
+- **CONTEXT:** What the agent produced *with* the skill. Think of this as "aided" ability.
+- **DELTA:** The improvement. `+16` means the skill added 16 points to that dimension.
+- **TOTAL:** Sum of all dimension scores. The `/100` reminds you of the maximum.
+
+**Verdict logic:**
+
+```ruby
+pass = context_total >= pass_threshold && total_delta >= minimum_delta
+```
+
+Both must be true. This prevents two failure modes:
+
+1. **High absolute, no improvement:** baseline=80, context=80, delta=0 → FAIL (skill didn't help)
+2. **Low absolute, small improvement:** baseline=10, context=20, delta=10 → FAIL (still terrible)
+
+**TREND line:**
+
+```text
+TREND: baseline ↑ (+2), context ↑ (+7)
+```
+
+This compares the current run against the **previous run of the same eval + skill** (stored in `.skill-bench-history.json`).
+
+- `↑` = improved since last run
+- `↓` = regressed since last run
+- `→` = unchanged
+
+The numbers in parentheses are the point differences. This helps you track whether your skill is getting better over time.
+
+### JSON Format
+
+```bash
+skill-bench run my-eval --skill=my-skill --format json
+```
+
+Returns a structured hash with:
+
+- `eval_name`, `skill_name`, `provider_name`
+- `report` containing: `verdict`, `baseline_total`, `context_total`, `deltas`, `baseline_scores`, `context_scores`, `criteria`
+- `trend` (if history exists): `baseline_trend`, `context_trend`, `baseline_delta`, `context_delta`, `previous_run`
+
+Useful for CI/CD pipelines and automated reporting.
+
+### JUnit XML Format
+
+```bash
+skill-bench run my-eval --skill=my-skill --format junit
+```
+
+Returns standard JUnit XML. Useful for GitHub Actions, Jenkins, and other CI systems that parse JUnit reports.
+
+---
+
+## Running the Test Suite
+
+The project uses Minitest with 440+ tests covering:
+
+- Core evaluation engine (`test/evaluator/`)
+- CLI commands and models (`test/agent_eval/`)
+- Provider clients (`test/clients/`)
+- Skill services (`test/skills/`)
+
+```bash
+# Run all tests
+bundle exec rake test
+
+# Run with coverage report
+bundle exec rake test COVERAGE=true
+
+# Run specific test file
+bundle exec ruby -Itest test/integration_test.rb
+
+# Run lint checks
+bundle exec rake rubocop
+bundle exec rake reek
+```
+
+### Test Isolation
+
+Tests use temporary directories and restore the original working directory:
+
+```ruby
+def setup
+  @original_dir = Dir.pwd
+  @tmp_dir = Dir.mktmpdir('test')
+  Dir.chdir(@tmp_dir)
+end
+
+def teardown
+  Dir.chdir(@original_dir)
+  FileUtils.rm_rf(@tmp_dir)
+end
+```
+
+### Environment Variable Handling
+
+Tests that modify ENV must restore original values:
+
+```ruby
+def test_something
+  original_key = ENV.fetch('SKILL_BENCH_OPENAI_API_KEY', nil)
+  ENV.delete('SKILL_BENCH_OPENAI_API_KEY')
+  # ... test code ...
+ensure
+  ENV['SKILL_BENCH_OPENAI_API_KEY'] = original_key if original_key
+end
+```

data/lib/skill_bench/agent/react_agent/loop_runner.rb

@@ -0,0 +1,69 @@
+# frozen_string_literal: true
+
+require_relative 'step'
+
+module SkillBench
+  module Agent
+    class ReactAgent
+      # Executes the ReAct loop iterations until completion or max iterations.
+      class LoopRunner
+        # Executes the loop.
+        #
+        # @param initial_prompt [String] The user task the agent must complete.
+        # @param max_iterations [Integer] The maximum allowed steps before aborting.
+        # @param config [Hash] The configuration for the Step execution.
+        # @return [Hash] A result hash indicating success or failure.
+        def self.call(initial_prompt, max_iterations, config)
+          messages = [{ role: 'user', content: initial_prompt }]
+          iterations_log = []
+          step_count = 0
+
+          while step_count < max_iterations
+            step_count += 1
+
+            step_result = Step.call(messages, config)
+            iteration = step_result[:iteration]
+            iterations_log << attach_step_number(iteration, step_count) if iteration
+
+            unless step_result[:continue]
+              final_result = step_result[:result] || { success: false, response: { error: { message: 'Step returned no result' } } }
+              return merge_iterations(final_result, iterations_log)
+            end
+
+            messages = step_result[:messages]
+          end
+
+          merge_iterations(
+            { success: false, response: { error: { message: Agent::ReactAgent::MAX_ITERATIONS_REACHED } } },
+            iterations_log
+          )
+        rescue StandardError => e
+          SkillBench::ErrorLogger.log_error(e, 'ReactAgent Error')
+          merge_iterations(
+            { success: false, response: { error: { message: e.message } } },
+            iterations_log
+          )
+        end
+
+        # Attaches the step number to an iteration hash.
+        #
+        # @param iteration [Hash] The iteration metadata from a Step.
+        # @param step_count [Integer] The current step number.
+        # @return [Hash] The iteration with :step_number added.
+        def self.attach_step_number(iteration, step_count)
+          iteration.merge(step_number: step_count)
+        end
+
+        # Merges the collected iterations into the result response.
+        #
+        # @param result [Hash] The final result hash from the loop.
+        # @param iterations_log [Array<Hash>] Collected iteration metadata.
+        # @return [Hash] The result with :iterations injected into :response.
+        def self.merge_iterations(result, iterations_log)
+          response = result[:response] || {}
+          result.merge(response: response.merge(iterations: iterations_log))
+        end
+      end
+    end
+  end
+end

data/lib/skill_bench/agent/react_agent/step.rb

@@ -0,0 +1,92 @@
+# frozen_string_literal: true
+
+require_relative '../../client'
+require_relative 'tool_executor'
+
+module SkillBench
+  module Agent
+    class ReactAgent
+      # Service object responsible for executing a single step of the ReAct loop.
+      class Step
+        # Executes one iteration of reasoning and potential tool usage.
+        #
+        # @param messages [Array<Hash>] The conversation history.
+        # @param config [Hash] Configuration for this step (client params, system prompt, working dir).
+        # @return [Hash] Step outcome containing :continue (boolean), :result (hash, if finished), and :messages.
+        def self.call(messages, config)
+          messages = messages.dup
+          client_result = Client.call(
+            system_prompt: config[:system_prompt],
+            messages: messages,
+            tools: Tools.definitions,
+            **config[:client_params]
+          )
+
+          unless client_result[:success]
+            error_msg = client_result.dig(:response, :error, :message) || 'Unknown error'
+            return {
+              continue: false,
+              result: client_result,
+              iteration: build_iteration(thought: '', tools_used: [], observation_summary: error_msg)
+            }
+          end
+
+          response_msg = client_result.dig(:response, :message)
+          unless response_msg
+            return {
+              continue: false,
+              result: { success: false, response: { error: { message: 'Empty response from LLM' } } },
+              iteration: build_iteration(thought: '', tools_used: [], observation_summary: 'Empty response from LLM')
+            }
+          end
+
+          messages << response_msg
+
+          tool_calls = response_msg['tool_calls']
+          content = response_msg['content']
+          tool_calls_array = Array(tool_calls)
+          thought = content.to_s
+
+          if tool_calls_array.empty?
+            return {
+              continue: false,
+              result: { success: true, response: { content: content } },
+              iteration: build_iteration(thought: thought, tools_used: [], observation_summary: '')
+            }
+          end
+
+          if thought.strip.length.positive?
+            warn "\n=== Agent Thought ==="
+            warn content
+          end
+
+          tool_results = ToolExecutor.call(tool_calls, config[:working_dir], config[:container_id])
+          messages.concat(tool_results)
+
+          tools_used = tool_calls_array.map { |tc| tc.dig('function', 'name') }.compact
+          observation_summary = Array(tool_results).map { |tr| tr[:content] || tr['content'] }.compact.join(', ')
+
+          {
+            continue: true,
+            messages: messages,
+            iteration: build_iteration(thought: thought, tools_used: tools_used, observation_summary: observation_summary)
+          }
+        end
+
+        # Builds an iteration metadata hash.
+        #
+        # @param thought [String] The agent's reasoning for this step.
+        # @param tools_used [Array<String>] Names of tools invoked.
+        # @param observation_summary [String] Summary of tool results.
+        # @return [Hash] Iteration metadata.
+        def self.build_iteration(thought:, tools_used:, observation_summary:)
+          {
+            thought: thought,
+            tools_used: tools_used,
+            observation_summary: observation_summary
+          }
+        end
+      end
+    end
+  end
+end

data/lib/skill_bench/agent/react_agent/tool_executor.rb

@@ -0,0 +1,88 @@
+# frozen_string_literal: true
+
+require_relative '../../tools'
+
+module SkillBench
+  module Agent
+    class ReactAgent
+      # Service object responsible for executing a list of tool calls and returning the results
+      # formatted as messages to be appended to the conversation history.
+      class ToolExecutor
+        # Executes the provided tool calls.
+        #
+        # @param tool_calls [Array<Hash>] The tool calls requested by the LLM.
+        # @param working_dir [String] The directory where tools should operate.
+        # @param container_id [String, nil] The Docker container ID for isolated execution.
+        # @return [Array<Hash>] An array of message hashes containing tool results.
+        def self.call(tool_calls, working_dir, container_id = nil)
+          tool_calls.map do |tool_call|
+            function_name = tool_call.dig('function', 'name')
+            next tool_error_message(tool_call, 'Missing function name') unless function_name
+
+            warn "=== Calling Tool: #{function_name} ===" unless defined?(Minitest)
+
+            result = execute_tool(tool_call, working_dir, container_id)
+            if result.is_a?(Hash) && result[:role] == 'tool'
+              result
+            else
+              error_msg = result.dig(:response, :error, :message) || 'Unknown tool error'
+              tool_error_message(tool_call, error_msg)
+            end
+          end
+        end
+
+        # Executes a single tool call and returns the result message.
+        #
+        # @param tool_call [Hash] The tool call hash.
+        # @param working_dir [String] The directory where tools should operate.
+        # @param container_id [String, nil] The Docker container ID.
+        # @return [Hash] Tool result message or error hash.
+        def self.execute_tool(tool_call, working_dir, container_id)
+          function_name = tool_call.dig('function', 'name')
+          arguments = tool_call.dig('function', 'arguments')
+
+          result = Tools.execute(function_name, arguments, working_dir, container_id)
+
+          {
+            role: 'tool',
+            tool_call_id: tool_call['id'],
+            content: result
+          }
+        rescue StandardError => e
+          SkillBench::ErrorLogger.log_error(e, "Tool execution failed: #{function_name}")
+          tool_error_result(tool_call, e.message)
+        end
+
+        # Builds a tool error message for the conversation history.
+        #
+        # @param tool_call [Hash] The tool call hash.
+        # @param message [String] The error message.
+        # @return [Hash] Tool message with error content.
+        def self.tool_error_message(tool_call, message)
+          {
+            role: 'tool',
+            tool_call_id: tool_call['id'],
+            content: "Error: #{message}"
+          }
+        end
+
+        # Builds an error result for a failed tool call.
+        #
+        # @param tool_call [Hash] The tool call hash.
+        # @param message [String] The error message.
+        # @return [Hash] Error result hash.
+        def self.tool_error_result(tool_call, message)
+          {
+            success: false,
+            response: {
+              error: {
+                message: "Tool call failed: #{message}",
+                tool_call: tool_call
+              }
+            }
+          }
+        end
+      end
+    end
+  end
+end

data/lib/skill_bench/agent/react_agent.rb

@@ -0,0 +1,58 @@
+# frozen_string_literal: true
+
+require_relative 'react_agent/step'
+require_relative 'react_agent/loop_runner'
+
+module SkillBench
+  module Agent
+    # An agent that follows the ReAct (Reasoning and Acting) loop pattern.
+    # It executes a given task by repeatedly thinking, invoking tools, and observing the results
+    # until it finishes the task or reaches the maximum number of iterations.
+    class ReactAgent
+      # Error message returned when the ReAct loop reaches max iterations.
+      MAX_ITERATIONS_REACHED = 'Reached max iterations without finishing.'
+
+      # Starts the ReAct loop for a specific task.
+      #
+      # @param params [Hash] The configuration for the agent.
+      # @option params [String] :system_prompt The instructions establishing the agent's persona and rules.
+      # @option params [String] :initial_prompt The user task the agent must complete.
+      # @option params [Integer] :max_iterations (25) The maximum allowed steps before aborting.
+      # @option params [String] :working_dir (Dir.pwd) The directory where tools should operate.
+      # @option params [Hash] :client_params ({}) Configuration passed to the Client (e.g., model).
+      # @return [Hash] A result hash with :success, and :response payload containing the final answer.
+      def self.call(params)
+        new(params).call
+      end
+
+      # @param params [Hash] The configuration for the agent.
+      def initialize(params)
+        @system_prompt = params[:system_prompt]
+        @initial_prompt = params[:initial_prompt]
+        @max_iterations = params[:max_iterations] || 25
+        @working_dir = params[:working_dir] || Dir.pwd
+        @container_id = params[:container_id]
+        @client_params = params[:client_params] || {}
+      end
+
+      # Executes the ReAct loop.
+      #
+      # @return [Hash] The standardized result hash indicating success or failure.
+      def call
+        config = build_step_config
+        LoopRunner.call(@initial_prompt, @max_iterations, config)
+      end
+
+      private
+
+      def build_step_config
+        {
+          system_prompt: @system_prompt,
+          client_params: @client_params,
+          working_dir: @working_dir,
+          container_id: @container_id
+        }
+      end
+    end
+  end
+end