ruby-skill-bench 1.4.0 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 78725cfc6d6bcc768b30cdd3da521d4895b7f24d1bd388e1056818a45ae71714
-  data.tar.gz: f589debec2945344a06c59443dab4b0a64ee969467cb814d86674e2b706acdf9
+  metadata.gz: cb1aca517e3a5d2599b68f7bff39222f556e0bc6bce90490b0f1d5358f6547c2
+  data.tar.gz: 7e8dffa808ea6248d8b84b00492066f071c8771cb8fc7bbb25121149270affaa
 SHA512:
-  metadata.gz: 6981311a0bd893fb0f0a443cec3934623ae5fab6c1088ea8b720ce55ef0b901b6851bfc06aed29b355dd187cfc0864c24e59bc2959b24e2bd3479bdeb920db03
-  data.tar.gz: ed4e639a719d6ab023e53f43484d48e591d1289559d8b178705905e480dbc682fe109915a461d8b547a733507f132b113790db6b7c325df5df7c27719c649b52
+  metadata.gz: 1d1bf40f9e8bb02757be1bb2fb2fa5a68187d3bbc472852ce34306aaf85782a6b8acdb8b977d1779f03c8a717c9874fe98f09856925f1446575a519c91a5e997
+  data.tar.gz: 42b9b9311d45a376e9d3e15a75be1cdf43bbcf234db2180c23cab7085a64b224026b5b2b6097be33118b532f7f6728f32787168cfa1fdd5cc1fd2bd6cd5a3d45

data/README.md

@@ -6,7 +6,7 @@
 
 [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/igmarin/ruby-skill-bench)
 
-*A high-fidelity evaluation engine for benchmarking AI agent skills across any stack (Rails-first, but extensible).*
+Ruby Skill Bench compares how an AI coding agent completes the same task with and without a skill. It runs both attempts in temporary Git sandboxes, then scores them against criteria you define. You can use it with Ruby projects and other stacks.
 
 ## Part of the AI Skill Ecosystem
 
@@ -27,12 +27,11 @@ See the [Ecosystem Overview](https://github.com/igmarin/agent-mcp-runtime/blob/m
 
 ## Features
 
-- **Side-by-Side Evaluation**: Quantify the "ROI of Context" by comparing baseline vs. skill-enhanced agent runs.
-- **Isolated Git Sandboxes**: Every run operates in a temporary repo. Clean diffs, zero side-effects, 100% reproducibility.
-- **Blind Judging with Dimensions**: LLM judge scores baseline and context independently across 5 canonical dimensions (Correctness, Skill Adherence, Code Quality, Test Coverage, Documentation). Eval authors configure weights and thresholds via `criteria.json`.
-- **Sophisticated ReAct Loop**: Employs a robust `Thought → Tool → Observation` loop to handle complex, multi-step engineering tasks.
-- **Multi-Provider Ecosystem**: Native support for **OpenAI**, **Anthropic**, **Google Gemini**, **Azure OpenAI**, **Ollama**, **Groq**, **DeepSeek**, **Mistral**, **OpenCode**, **OpenRouter**, **xAI**, and **Amazon Bedrock**.
-- **Standardized Intelligence**: Consistent reporting format regardless of the underlying LLM provider.
+- **Compare two runs**: See what changes when an agent receives a skill.
+- **Isolate evaluation work**: Each attempt runs in its own temporary Git sandbox.
+- **Score against your criteria**: Configure dimensions, weights, and pass thresholds in `criteria.json`.
+- **Use your provider**: OpenAI, Anthropic, Google Gemini, Azure OpenAI, Ollama, Groq, DeepSeek, Mistral, OpenCode, OpenRouter, xAI, and Amazon Bedrock are supported.
+- **Get consistent reports**: The output format stays the same across providers.
 
 ---
 
@@ -118,6 +117,8 @@ Configuration is loaded in this order (later sources override earlier ones):
 
 ## Getting Started
 
+Ruby Skill Bench requires Ruby 3.3 or newer. For a guided first run, see the [5-minute first eval guide](docs/first-eval-guide.md).
+
 ### Installation
 
 ```bash
@@ -184,7 +185,7 @@ skills/
 
 **Example `SKILL.md`:**
 
-```markdown
+````markdown
 # Service Object Skill
 
 ## Pattern
@@ -200,7 +201,7 @@ All service objects use the `.call` class method and return a standardized hash:
 1. Every `.rb` file begins with `# frozen_string_literal: true`
 2. Every public method has YARD docs (`@param`, `@return`, `@raise`)
 3. `rescue StandardError` blocks must log backtrace
-```
+````
 
 ---
 
@@ -1153,7 +1154,7 @@ jobs:
 
 The action installs the gem and runs `skill-bench run --all --evals-dir <evals-dir> --format <format>` (adding `--skill` when set and appending `args` verbatim). The run step's exit code is the gate. For a full copy-paste workflow template, see [`examples/ci/`](examples/ci/).
 
-> The gem's own repository CI (`.github/workflows/ci.yml`) runs the test suite — rubocop, reek, and minitest against Ruby 3.3 and 3.4, on push and pull requests — and is separate from the reusable action above.
+> The gem's own repository CI (`.github/workflows/ci.yml`) runs RuboCop, Reek, and Minitest on Ruby 3.3, 3.4, and 4.0. This is separate from the reusable action above.
 
 To preview the machine-readable output locally:
 

data/docs/docker.md

@@ -10,7 +10,7 @@ execution when a Docker daemon is available.
 |------|--------|
 | Image name | `evaluator-sandbox` (`Constants::Sandbox::DOCKER_IMAGE_NAME`) |
 | Tags | `evaluator-sandbox:<SkillBench::VERSION>` and `evaluator-sandbox:latest` |
-| Base | `ruby:3.4-bookworm` (aligns with CI Ruby 3.4; 3.3 hosts still run evals via host Ruby for the orchestrator) |
+| Base | `ruby:3.4-bookworm` (commands inside the container use Ruby 3.4; the gem also supports Ruby 3.3 and 4.0 on the host) |
 | Preinstalled | Ruby (from base), `git`, minimal Debian tools from base image |
 | Working directory | `/sandbox` (host sandbox dir is bind-mounted here read-write) |
 | Default process | `sleep infinity` — container stays up for `docker exec` |

data/docs/first-eval-guide.md

@@ -6,13 +6,13 @@ Get started with Ruby Skill Bench in 5 minutes. No prior AI eval experience requ
 
 ## Prerequisites
 
-- Ruby 3.1+
+- Ruby 3.3+
 - Bundler
 
 Not sure? Run:
 
 ```bash
-ruby --version   # Should be 3.1 or higher
+ruby --version   # Should be 3.3 or higher
 bundle --version # Should print a version number
 ```
 
@@ -42,6 +42,14 @@ skill-bench init --openai
 
 This creates `skill-bench.json` with the OpenAI provider config. Use `--force` to overwrite.
 
+Before running an eval, set your OpenAI key in the shell. Do not commit API keys to your repository:
+
+```bash
+export SKILL_BENCH_OPENAI_API_KEY=your-api-key
+```
+
+To try the commands without an API key or network access, initialize the offline mock provider instead: `skill-bench init --mock`. The mock provider is useful for learning the workflow; it does not measure a real model.
+
 **Available providers:** `--openai`, `--anthropic`, `--gemini`, `--ollama`, `--azure`, `--groq`, `--deepseek`, `--mistral`, `--opencode`, `--openrouter`, `--xai`, `--bedrock`, plus `--mock` for an offline config with no API key.
 
 > **What is `skill-bench.json`?** This is your config file. It stores your API key, chosen LLM model, timeout, and allowed shell commands. Think of it as `.env` but structured as JSON. You edit it; SkillBench reads it.
@@ -66,7 +74,7 @@ This creates `skills/my-service/SKILL.md` with a Rails service object template.
 
 **Example `SKILL.md`:**
 
-```markdown
+````markdown
 # Service Object Skill
 
 ## Pattern
@@ -82,7 +90,7 @@ All service objects use the `.call` class method and return a standardized hash:
 1. Every `.rb` file begins with `# frozen_string_literal: true`
 2. Every public method has YARD docs (`@param`, `@return`, `@raise`)
 3. `rescue StandardError` blocks must log backtrace
-```ruby
+````
 
 ---
 

data/lib/skill_bench/clients/base_client.rb

@@ -21,10 +21,10 @@ module SkillBench
       # @param system_prompt [String] The system instruction for the LLM.
       # @param messages [Array<Hash>] The list of conversation messages.
       # @param tools [Array<Hash>] (optional) Array of tool definitions.
-      # @param options [Hash] (optional) Additional provider-specific options.
+      # Additional keyword options are passed to the selected provider client.
       # @return [Hash] with :success [Boolean] and :response [Hash] keys.
-      def self.call(system_prompt:, messages:, tools: [], **options)
-        new(system_prompt: system_prompt, messages: messages, tools: tools, **options).call
+      def self.call(system_prompt:, messages:, tools: [], **)
+        new(system_prompt: system_prompt, messages: messages, tools: tools, **).call
       end
 
       # Initializes the client with validated parameters.

data/lib/skill_bench/judge/variance.rb

@@ -20,15 +20,16 @@ module SkillBench
       # @return [Hash]
       def call
         nums = Array(@totals).map { |total| Float(total) }
-        return empty_result(count: nums.size) if nums.size < 2
+        count = nums.size
+        return empty_result(count: count) if count < 2
 
-        mean = nums.sum / nums.size
-        sample_variance = nums.sum { |value| (value - mean)**2 } / (nums.size - 1)
+        mean = nums.sum / count
+        sample_variance = nums.sum { |value| (value - mean)**2 } / (count - 1)
 
         {
           success: true,
           response: {
-            n: nums.size,
+            n: count,
             mean: mean,
             stddev: Math.sqrt(sample_variance),
             spread: nums.max - nums.min

data/lib/skill_bench/runner.rb

@@ -2,6 +2,7 @@
 
 require 'pathname'
 require 'parallel'
+require_relative 'task/directory_finder'
 require_relative 'task/evaluator'
 require_relative 'error_logger'
 
@@ -79,11 +80,7 @@ module SkillBench
     # @param root_path [Pathname] The root directory to search.
     # @return [Array<Pathname>] A list of task directory paths.
     def self.discover_task_dirs(root_path)
-      if File.exist?(root_path.join('task.md'))
-        [root_path]
-      else
-        Dir.glob(root_path.join('**/task.md')).map { |f| Pathname.new(f).parent }.uniq.sort
-      end
+      Task::DirectoryFinder.call(root_path)
     end
   end
 end

data/lib/skill_bench/services/batch_runner_service.rb

@@ -4,7 +4,7 @@ require 'pathname'
 require 'parallel'
 require_relative 'runner_service'
 require_relative '../output_formatter'
-require_relative '../runner'
+require_relative '../task/directory_finder'
 
 module SkillBench
   module Services
@@ -14,9 +14,7 @@ module SkillBench
     # {RunnerService} over each, returning an aggregate envelope with
     # per-eval results and a pass/fail summary.
     #
-    # Discovery reuses {SkillBench::Runner.discover_task_dirs} but never
-    # routes through the deprecated {SkillBench::Task::Evaluator}: each eval
-    # is executed by the supported {RunnerService}.
+    # Each discovered eval is executed by the supported {RunnerService}.
     class BatchRunnerService
       # Default directory scanned for evals when none is supplied.
       DEFAULT_EVALS_DIR = 'evals'
@@ -80,7 +78,7 @@ module SkillBench
       #
       # @return [Array<Pathname>] Directories that contain a task.md
       def discover_eval_dirs
-        SkillBench::Runner.discover_task_dirs(Pathname.new(evals_dir))
+        Task::DirectoryFinder.call(Pathname.new(evals_dir))
       end
 
       # Runs every eval directory through {RunnerService} concurrently.

data/lib/skill_bench/services/output_persistence_service.rb

@@ -70,7 +70,7 @@ module SkillBench
       #
       # @raise [SystemCallError] when file write operation fails
       def write_json_file
-        File.write(@output_path, JSON.generate(@result, pretty: true))
+        File.write(@output_path, JSON.pretty_generate(@result))
       end
     end
   end

data/lib/skill_bench/task/directory_finder.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+require 'pathname'
+
+module SkillBench
+  module Task
+    # Finds evaluation task directories under a root path.
+    class DirectoryFinder
+      # Finds the root task or all nested task directories in sorted order.
+      #
+      # @param root_path [Pathname] Directory to search.
+      # @return [Array<Pathname>] Directories containing a task.md file.
+      def self.call(root_path)
+        return [root_path] if File.exist?(root_path.join('task.md'))
+
+        Dir.glob(root_path.join('**/task.md')).map { |path| Pathname.new(path).parent }.uniq.sort
+      end
+    end
+  end
+end

data/lib/skill_bench/version.rb

@@ -2,5 +2,5 @@
 
 module SkillBench
   # The current gem version.
-  VERSION = '1.4.0'
+  VERSION = '2.0.0'
 end

data/lib/skill_bench.rb

@@ -34,6 +34,7 @@ require_relative 'skill_bench/agent/react_agent'
 require_relative 'skill_bench/task'
 require_relative 'skill_bench/task/evaluator'
 require_relative 'skill_bench/task/file_reader'
+require_relative 'skill_bench/runner'
 
 # Evaluation orchestration
 require_relative 'skill_bench/evaluation'

metadata

@@ -1,14 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: ruby-skill-bench
 version: !ruby/object:Gem::Version
-  version: 1.4.0
+  version: 2.0.0
 platform: ruby
 authors:
 - Ismael Marin
-autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-09-14 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: cgi
@@ -44,34 +43,28 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '2.21'
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: 2.21.2
+        version: '3.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '2.21'
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: 2.21.2
+        version: '3.0'
 - !ruby/object:Gem::Dependency
   name: parallel
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.26'
+        version: '2.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.26'
+        version: '2.0'
 description: |
   ruby-skill-bench orchestrates evaluation runs of AI coding agents
   inside isolated git sandboxes, then scores the results using deterministic
@@ -225,6 +218,7 @@ files:
 - lib/skill_bench/services/variant_parser.rb
 - lib/skill_bench/services/variant_resolver.rb
 - lib/skill_bench/task.rb
+- lib/skill_bench/task/directory_finder.rb
 - lib/skill_bench/task/evaluator.rb
 - lib/skill_bench/task/file_reader.rb
 - lib/skill_bench/tools.rb
@@ -245,7 +239,6 @@ licenses:
 metadata:
   rubygems_mfa_required: 'true'
   source_code_uri: https://github.com/igmarin/ruby-skill-bench
-post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -253,15 +246,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '3.1'
+      version: '3.3'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.22
-signing_key:
+rubygems_version: 4.0.21
 specification_version: 4
 summary: The evaluation engine for AI Agent Skills benchmarking.
 test_files: []