ruby-skill-bench 1.3.0 → 1.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 60f6da81bf101f4bb1e5a7b86112aa96fa430ca9a1db7f1331a47e51b194ea73
-  data.tar.gz: 7f50c1e8a1891a93f0028f98c6a660e1d35ecfaf2104390d94a895ff983d9c93
+  metadata.gz: 78725cfc6d6bcc768b30cdd3da521d4895b7f24d1bd388e1056818a45ae71714
+  data.tar.gz: f589debec2945344a06c59443dab4b0a64ee969467cb814d86674e2b706acdf9
 SHA512:
-  metadata.gz: a828560ad084e9bd1d3336c9002c8b17d47a12955bed6d53e3dd174ef0e971dab24699ab5da40d3445ece5d06afa29e9354e9d033e5d8a11b5c749872487c1cd
-  data.tar.gz: b81d9dc73aa78be923e8fdecbf789aa5078e1f68c40d9330a0e11ca9632579dc8635e647a3f08730e1a478f3ea16a0e5379fae1b008b461f1f48d6d4714a9e3e
+  metadata.gz: 6981311a0bd893fb0f0a443cec3934623ae5fab6c1088ea8b720ce55ef0b901b6851bfc06aed29b355dd187cfc0864c24e59bc2959b24e2bd3479bdeb920db03
+  data.tar.gz: ed4e639a719d6ab023e53f43484d48e591d1289559d8b178705905e480dbc682fe109915a461d8b547a733507f132b113790db6b7c325df5df7c27719c649b52

data/README.md

@@ -21,7 +21,7 @@ This repo is one of 6 in a composable AI skill ecosystem:
 | [`agent-mcp-runtime`](https://github.com/igmarin/agent-mcp-runtime) | Rust CLI runtime (pack resolution, MCP) |
 | [**`ruby-skill-bench`**](https://github.com/igmarin/ruby-skill-bench) | Benchmark/eval engine |
 
-See the [Ecosystem Overview](https://github.com/igmarin/agent-mcp-runtime/blob/main/docs/ecosystem.md) for the full architecture.
+See the [Ecosystem Overview](https://github.com/igmarin/agent-mcp-runtime/blob/main/docs/ecosystem.md) for the full architecture. What comes next in this repo: [ROADMAP.md](ROADMAP.md).
 
 ---
 
@@ -31,7 +31,7 @@ See the [Ecosystem Overview](https://github.com/igmarin/agent-mcp-runtime/blob/m
 - **Isolated Git Sandboxes**: Every run operates in a temporary repo. Clean diffs, zero side-effects, 100% reproducibility.
 - **Blind Judging with Dimensions**: LLM judge scores baseline and context independently across 5 canonical dimensions (Correctness, Skill Adherence, Code Quality, Test Coverage, Documentation). Eval authors configure weights and thresholds via `criteria.json`.
 - **Sophisticated ReAct Loop**: Employs a robust `Thought → Tool → Observation` loop to handle complex, multi-step engineering tasks.
-- **Multi-Provider Ecosystem**: Native support for **OpenAI**, **Anthropic**, **Google Gemini**, **Azure OpenAI**, **Ollama**, **Groq**, **DeepSeek**, **Mistral**, and **OpenCode**.
+- **Multi-Provider Ecosystem**: Native support for **OpenAI**, **Anthropic**, **Google Gemini**, **Azure OpenAI**, **Ollama**, **Groq**, **DeepSeek**, **Mistral**, **OpenCode**, **OpenRouter**, **xAI**, and **Amazon Bedrock**.
 - **Standardized Intelligence**: Consistent reporting format regardless of the underlying LLM provider.
 
 ---
@@ -61,18 +61,28 @@ CLI / API → RunnerService → Sandbox + ReAct Agent → LLM Client Layer → P
 | **OpenAI** | `SKILL_BENCH_OPENAI_API_KEY` | `:openai` |
 | **Anthropic** | `SKILL_BENCH_ANTHROPIC_API_KEY` | `:anthropic` |
 | **Gemini** | `SKILL_BENCH_GEMINI_API_KEY` | `:gemini` |
-| **Azure** | `SKILL_BENCH_AZURE_API_KEY` | `:azure` |
+| **Azure** | `SKILL_BENCH_AZURE_OPENAI_API_KEY` | `:azure` |
 | **Ollama** | — | `:ollama` |
 | **Groq** | `SKILL_BENCH_GROQ_API_KEY` | `:groq` |
 | **DeepSeek** | `SKILL_BENCH_DEEPSEEK_API_KEY` | `:deepseek` |
-| **Mistral** | `SKILL_BENCH_MISTRAL_API_KEY` | `:mistral` |
+| **Mistral** | set `api_key` in `skill-bench.json` (no env mapping) | `:mistral` |
 | **OpenCode** | `SKILL_BENCH_OPENCODE_API_KEY`, `SKILL_BENCH_OPENCODE_BASE_URL` | `:opencode` |
+| **OpenRouter** | `SKILL_BENCH_OPENROUTER_API_KEY` | `:openrouter` |
+| **xAI** | `SKILL_BENCH_XAI_API_KEY` or `XAI_API_KEY` | `:xai` |
+| **Bedrock** | `SKILL_BENCH_BEDROCK_API_KEY` or `AWS_BEARER_TOKEN_BEDROCK` | `:bedrock` |
+| **Mock** | — (offline, `skill-bench init --mock`) | `:mock` |
 
 > **Note:** Environment variables are loaded automatically. You can also configure provider settings in `skill-bench.json` (created by `skill-bench init`).
 >
 > **OpenCode requires a custom `base_url`:** OpenCode does not host a public LLM API. You must provide your own OpenAI-compatible endpoint (e.g. a LiteLLM proxy, self-hosted vLLM, or company gateway) via the `base_url` config key. Without it, the provider will fail with "Base URL not set for Opencode".
 >
-> **Mistral** uses Mistral's OpenAI-compatible chat completions API (default model `mistral-large-latest`). Set `SKILL_BENCH_MISTRAL_API_KEY` and scaffold it with `skill-bench init --mistral`.
+> **Mistral** uses Mistral's OpenAI-compatible chat completions API (default model `mistral-large-latest`). Scaffold with `skill-bench init --mistral` and set `config.api_key` in `skill-bench.json`. There is no `SKILL_BENCH_MISTRAL_API_KEY` env mapping.
+>
+> **OpenRouter** uses OpenRouter's OpenAI-compatible API (default model `anthropic/claude-3.5-sonnet`). Set `SKILL_BENCH_OPENROUTER_API_KEY` and scaffold with `skill-bench init --openrouter`.
+>
+> **xAI** uses xAI's OpenAI-compatible chat completions API (default model `grok-4`). Set `SKILL_BENCH_XAI_API_KEY` or `XAI_API_KEY` and scaffold with `skill-bench init --xai`.
+>
+> **Amazon Bedrock** uses the Runtime OpenAI-compatible Chat Completions path (`https://bedrock-runtime.<region>.amazonaws.com/openai/v1/chat/completions`) with a Bedrock API key. Default region `us-east-1` (`SKILL_BENCH_BEDROCK_REGION` or `AWS_REGION`), default model `amazon.nova-lite-v1:0`. Scaffold with `skill-bench init --bedrock`. IAM SigV4 signing is not in this client.
 
 ### Command Allowlist
 
@@ -144,7 +154,7 @@ skill-bench init --openai
 }
 ```
 
-**Available providers:** `--openai`, `--anthropic`, `--gemini`, `--ollama`, `--azure`, `--groq`, `--deepseek`, `--mistral`, `--opencode`
+**Available providers:** `--openai`, `--anthropic`, `--gemini`, `--ollama`, `--azure`, `--groq`, `--deepseek`, `--mistral`, `--opencode`, `--openrouter`, `--xai`, `--bedrock`, plus `--mock` for an offline config with no API key.
 
 **Zero-config offline path:** `skill-bench init --mock` scaffolds a minimal offline config that needs no API key and no network — `{"provider":"mock","max_execution_time":30}`. Use it to try the full flow (and run the bundled examples) before wiring up a real provider.
 
@@ -908,7 +918,7 @@ Your eval result depends on **both** conditions. Here is every scenario:
 - **Traceability**: Every thought and tool call is logged with full backtrace for post-mortem analysis.
 - **Robust Error Recovery**: Handles provider outages and rate limits gracefully with standardized error logging.
 - **XML-Safe Output**: JUnit XML output is properly escaped to prevent injection attacks.
-- **Test Coverage**: 700+ tests covering core engine, CLI commands, and all provider clients. Run `bundle exec rake test` to see the current count.
+- **Test Coverage**: 910 tests covering core engine, CLI commands, and all provider clients. Run `bundle exec rake test` to see the current count.
 
 ## Testing
 

data/docs/architecture.md

@@ -1,209 +1,170 @@
-# SkillBench Architecture
-
-Ruby Skill Bench provides a reproducible and isolated environment for testing AI agents. It consists of several decoupled components that orchestrate the evaluation flow.
-
-## High-Level Flow
-
-1. **`RunnerService`**: The entry point. Resolves eval, skill, and provider, then runs baseline and context agents.
-2. **`Sandbox`**: Creates a temporary directory, copies task files, and initializes a Git repository for clean, reproducible runs.
-3. **`ContextHydrator`**: Loads skill documentation (.md, .rb, .json, .yml, .yaml, .txt up to 50KB each) and wraps it in XML for the agent's system prompt.
-4. **`ReactAgent`**: Autonomous agent following a **Thought → Tool → Observation** loop.
-5. **`EvaluationRunner`**: Orchestrates blind judging — builds `JudgePrompt` for baseline and context outputs, calls `Judge` twice, then computes deltas via `DeltaReport`.
-6. **`DeltaReport`**: Computes per-dimension deltas and determines verdict based on `pass_threshold` and `minimum_delta`.
-7. **`Client`**: Provider-agnostic abstraction for LLM backends.
-
-## Key Components
-
-### `SkillBench::Services::RunnerService`
-
-- Resolves eval, skill, and provider configuration.
-- Runs baseline agent (no skill context) and context agent (with skill context).
-- Delegates judging and delta computation to `EvaluationRunner`.
-- Falls back to mock provider when config is unavailable.
-
-### `SkillBench::EvaluationRunner`
-
-- Builds `JudgePrompt` for baseline and context outputs.
-- Calls `Judge` twice (blind scoring).
-- Uses `DeltaReport` to compute per-dimension deltas and final verdict.
-
-### `SkillBench::DeltaReport`
-
-- Computes baseline vs context deltas per dimension.
-- Verdict requires: `context_total >= pass_threshold` AND `total_delta >= minimum_delta`.
-
-### `SkillBench::CLI` Commands
-
-- `InitCommand` — Creates `skill-bench.json` configuration
-- `RunCommand` — Executes evaluations
-- `SkillCommand` — Scaffolds new skills with templates
-- `EvalCommand` — Creates evaluation scenarios
-
-### `SkillBench::Services::TemplateRegistry`
+# SkillBench architecture
+
+## Executive summary
+
+Ruby Skill Bench is a Ruby gem that runs the same coding task twice: once without skill context (baseline) and once with it (context). An LLM judge scores each run independently. `DeltaReport` then decides pass or fail from the context total and the delta.
+
+Source of truth for a run is the eval directory (`task.md` + `criteria.json`), the skill files loaded into context, and `skill-bench.json` (plus env overrides). Runtime artifacts are a temp git sandbox and `.skill-bench-trends.json`.
+
+The rule a contributor must not break: host command execution is fail-closed. `run_command` runs inside the Docker `evaluator-sandbox` image when a daemon and image are available. Otherwise it refuses unless `allow_host_execution` is explicitly true.
+
+### System architecture
+
+```text
+User / CI
+    |
+    v
+bin/skill-bench  ->  SkillBench::CLI
+    |
+    +-- init / validate / skill / eval / compare
+    +-- run  ->  Commands::Run  ->  Services::RunnerService
+                                      |
+                                      +-- EvalResolver, SkillResolverService, ProviderResolver
+                                      +-- ContextLoaderService (ContextHydrator)
+                                      +-- Parallel: baseline agent | context agent
+                                      |     Sandbox (tmp git, optional Docker)
+                                      |     ReactAgent + Tools + Client
+                                      +-- Evaluation::Runner (blind Judge x2, concurrent)
+                                      +-- DeltaReport
+                                      +-- TrendRecorderService
+                                      +-- CostCalculator
+                                      v
+                                 OutputFormatter (human / json / junit / html)
+```
 
-- Provides pre-built templates for generating eval scaffolding
-- Supports three template types: `task_md`, `criteria_json`, `skill_md`
-- Offers 10 Rails pattern categories: `crud`, `api`, `background_job`, `controller`, `model`, `migration`, `concern`, `policy`, `form_object`, `view_component`
-- Enables variable interpolation using `{{variable_name}}` syntax
-- Used for programmatic eval creation and tool building
+Outside the gem: LLM HTTP APIs (OpenAI-compatible and native), optional Docker daemon, optional ecosystem `registry.json` for `--pack`.
 
-### `SkillBench::Sandbox`
+### Dependency hierarchy
 
-- Uses `Dir.mktmpdir` for isolation.
-- Captures state changes using `git diff`.
-- Validates sandbox path to prevent directory traversal.
-- Cleans up automatically after execution.
+```text
+CLI  ->  Commands  ->  Services (RunnerService, BatchRunnerService, ComparisonRunner)
+                         |
+                         +--> Agent / Tools
+                         +--> Execution (Sandbox, ContextHydrator)
+                         +--> Evaluation::Runner -> Judge -> Clients
+                         +--> Config
+                         +--> Models (Eval, Skill, Criteria, Provider)
+```
 
-### `SkillBench::ReactAgent`
+Dependencies point inward: CLI does not call providers; agents do not load config files; clients do not know about evals. Adding a provider means a `Clients::Providers::*` class plus a `ProviderSchemas` entry. Adding a command means a `Cli::*Command` plus a `Commands::*` object. Do not reverse that.
 
-- Implements a stateful loop.
-- Supports tool usage (e.g., `read_file`, `write_file`, `run_shell_command`).
-- Manages conversation history.
+---
 
-### `SkillBench::Clients::BaseClient`
+## Eval run lifecycle
 
-- Implements the **Template Method** pattern.
-- Handles Faraday connection setup and timeouts.
-- Centralizes error logging and response normalization.
-- Delegates to `ResponseParser`, `ResponseErrorHandler`, and `RequestBuilder`
+Exact order for `skill-bench run <eval> --skill <name>`:
 
-### `SkillBench::OutputFormatter`
+1. `SkillBench::CLI` shifts the subcommand and calls `Cli::RunCommand`.
+2. `RunCommand` parses flags (`--skill`, `--pack`, `--format`, `--all`, `--evals-dir`, `--summary`, `--cache`). `--cache` sets `SKILL_BENCH_CACHE=1`.
+3. `Commands::Run` calls `Services::RunnerService.call`.
+4. `EvalResolver` loads `task.md` and `criteria.json`. Path with `/` is used as-is; otherwise `evals/` is prepended.
+5. `SkillResolverService` resolves skills (local path, `skill_sources`, or `--pack` via `Registry::PackResolver`).
+6. `ProviderResolver` builds a `Models::Provider` from `Config` (defaults, then `~/.skill-bench.json`, then `./skill-bench.json`, then `ENV`).
+7. `ContextLoaderService` / `Execution::ContextHydrator` reads skill files. Allowed extensions: `.md`, `.rb`, `.json`, `.yml`, `.yaml`, `.txt`. Per-file cap 50_000 bytes. Total cap 1_000_000 bytes. Symlinks are skipped. Empty context is an error.
+8. `RunnerService` runs baseline and context agents concurrently (`Parallel.map`, two threads). Each agent runs inside `Execution::Sandbox.run`: copy sources into `Dir.mktmpdir`, hardened `git init`, start Docker if available, yield, stop container, delete the tempdir.
+9. `Evaluation::Runner` judges baseline and context concurrently. Both judges get `skill_context: nil` (skill text is only for the executing agent). The judge never sees both outputs in one call.
+10. `DeltaReport` computes per-dimension deltas. Verdict is `context_total >= pass_threshold AND total_delta >= minimum_delta`.
+11. `TrendRecorderService` appends `.skill-bench-trends.json` and keeps `.skill-bench-trends.json.bak`.
+12. `CostCalculator` estimates USD from aggregated token usage.
+13. `Cli::ResultPrinter` / `SkillBench::OutputFormatter` prints human, json, junit, or html and returns exit 0 or 1.
 
-- Formats results as human-readable text, JSON, or JUnit XML
-- Human format displays a dimension table with baseline, context, and delta columns
-- Escapes XML output to prevent injection
-- Provides exit codes for CI/CD integration
+Batch mode (`--all` / `--evals-dir`) uses `BatchRunnerService` instead of a single `RunnerService` call. `compare` runs `RunnerService` twice via `ComparisonRunner`.
 
-### `SkillBench::ErrorLogger`
+Failure behavior:
 
-- Shared error logging module for all service objects
-- Logs error message and full backtrace
-- Uses `Rails.logger` when available, falls back to `warn`
+- Missing provider config returns a config error envelope; it does not raise through the CLI.
+- Agent `:error` status short-circuits judging.
+- Judge parse failure returns `{ success: false, response: { error: { message: } } }`.
+- `run_command` with no container and `allow_host_execution` false returns `HOST_EXECUTION_REFUSED` and does not exec.
 
-## Data Flow: What Passes Between Components
+---
 
-Understanding what data moves between components helps debug issues and write better evals.
+## Components
 
-### Flow 1: RunnerService → EvaluationRunner
+### `SkillBench::CLI`
 
-```ruby
-# RunnerService builds this and passes it to EvaluationRunner.call
-evaluation = {
-  task: "Create a UserRegistrationService...",        # from task.md
-  criteria: <Criteria object>,                         # from criteria.json
-  skill_context: "<agent_context>...SKILL.md...</agent_context>",  # from ContextHydrator
-  baseline_output: '{"result":"...","status":":success"}',        # from baseline agent run
-  context_output: '{"result":"...","status":":success"}'         # from context agent run
-}
-```
+Owns ARGV dispatch and exit codes. Does not own eval scoring. Subcommands: `init`, `run`, `compare`, `skill`, `eval`, `validate`/`doctor`, `help`.
 
-### Flow 2: EvaluationRunner → Judge (two calls)
-
-```ruby
-# First call — baseline (no skill context)
-JudgePrompt.call(
-  task: task,
-  criteria: criteria,
-  skill_context: "",        # empty string for baseline
-  agent_output: baseline_output
-)
-
-# Second call — context (with skill context)
-JudgePrompt.call(
-  task: task,
-  criteria: criteria,
-  skill_context: skill_context,   # XML-wrapped SKILL.md
-  agent_output: context_output
-)
-```
+### `Services::RunnerService`
 
-### Flow 3: Judge → JudgeResponse
+Owns one eval's baseline+context orchestration, token aggregation, and the envelope passed to the printer. Does not own HTTP or git. Depends on resolvers, sandbox/agent spawners, `Evaluation::Runner`, `TrendRecorderService`, `CostCalculator`.
 
-The judge returns a JSON string like:
+### `Execution::Sandbox`
 
-```json
-{
-  "dimensions": {
-    "correctness": { "score": 28, "max_score": 30, "reasoning": "All requirements met." },
-    "skill_adherence": { "score": 22, "max_score": 25, "reasoning": "Used .call pattern correctly." }
-  },
-  "overall_reasoning": "Solid implementation."
-}
-```
+Owns the temp directory, hardened git (`core.hooksPath=/dev/null`, no source `.git` copy), and container lifecycle. Image ref is `evaluator-sandbox:<VERSION>`. Docker run flags: `--network none`, non-root `--user uid:gid`, `--cap-drop ALL` (then CHOWN and DAC_OVERRIDE), `--security-opt no-new-privileges`, volume mount of the sandbox. Does not own command allowlisting (`Tools::RunCommand` does).
 
-`JudgeResponse` parses this, validates that scores are numeric and within bounds, and returns a structured object.
+### `Execution::ContextHydrator`
 
-### Flow 4: DeltaReport → Output
+Owns packing skill files into XML for the agent system prompt. Does not own skill path resolution.
 
-```ruby
-# DeltaReport receives two JudgeResponse objects
-baseline = {
-  'correctness' => { score: 12, max_score: 30 },
-  'skill_adherence' => { score: 5, max_score: 25 }
-}
+### `Agent::ReactAgent`
 
-context = {
-  'correctness' => { score: 28, max_score: 30 },
-  'skill_adherence' => { score: 22, max_score: 25 }
-}
+Owns the Thought → Tool → Observation loop, capped at `max_iterations` (default 25). Tools: `read_file`, `write_file`, `run_command`. Does not own judging.
 
-# Produces:
-deltas = {
-  'correctness' => 16,        # 28 - 12
-  'skill_adherence' => 17     # 22 - 5
-}
+### `Evaluation::Runner`
 
-baseline_total = 17           # 12 + 5
-context_total = 50            # 28 + 22
-verdict = context_total >= pass_threshold && (context_total - baseline_total) >= minimum_delta
-```
+Owns blind judging and the call to `DeltaReport`. Two judge HTTP calls, concurrent, order preserved. Does not own agent execution.
 
-## Directory Structure
-
-The evaluator relies on a strict directory convention:
-
-```bash
-project-root/
-├── skill-bench.json              # Provider configuration
-├── skills/
-│   └── my-service/
-│       └── SKILL.md              # Skill instructions
-├── evals/
-│   └── my-first-eval/
-│       ├── task.md               # Agent prompt
-│       └── criteria.json         # Scoring rules
-└── .skill-bench-trends.json      # Benchmark history (auto-generated)
-```
+### `DeltaReport`
 
-A `.skill-bench-trends.json.bak` file is created automatically as a backup of the trend file.
+Owns per-dimension arithmetic and the boolean verdict. Does not call the network.
 
-### Skill Discovery
+### `Clients::BaseClient` and `ProviderRegistry`
 
-Skills are discovered recursively. These are all valid:
+Own HTTP to LLM providers (Faraday, retries, response normalization). Registered providers: openai, anthropic, gemini, ollama, azure, groq, deepseek, mistral, opencode, openrouter, xai, bedrock, plus mock and null. Do not own eval files.
 
-```bash
-skills/my-service/SKILL.md
-skills/api/rest-collection/SKILL.md
-skills/workflows/tdd-loop/SKILL.md
-```
+### `Config`
 
-The `SkillResolver` walks `skills/` recursively and matches by directory name.
+Owns the hierarchy Defaults → home JSON → local JSON → ENV. `EnvOverrides` maps `SKILL_BENCH_*` keys. Mistral has no env mapping; its key is `config.api_key` in JSON. Azure env key is `SKILL_BENCH_AZURE_OPENAI_API_KEY`.
 
-### Eval Discovery
+### `SkillBench::OutputFormatter`
 
-Evals are resolved in this order:
+Owns user-facing result text. Delegates to `JsonFormatter`, `JUnitFormatter`, `HtmlFormatter`, `DeltaTableFormatter`, `IterationFormatter`, `FeedbackGenerator`. Distinct from `Services::OutputFormatter`, which stringifies agent output for the judge.
 
-1. If the path contains `/`, use it as-is (e.g., `evals/my-eval`)
-2. Otherwise, prepend `evals/` (e.g., `my-eval` → `evals/my-eval`)
+### `ErrorLogger`
 
-The eval directory must contain at minimum:
+Logs message plus first five backtrace lines. Uses `Rails.logger` when `Rails` is defined; otherwise `warn`.
 
-- `task.md` — the agent prompt
-- `criteria.json` — the scoring rules (optional; defaults to empty criteria if missing)
+---
 
 ## Command isolation
 
-Agent tool commands prefer a Docker container (`evaluator-sandbox`) when available.
-See [docker.md](docker.md) for the image contract and `rake docker:build`.
-Without Docker, `run_command` refuses host execution unless `allow_host_execution` is true.
-
+See [docker.md](docker.md) for the image contract and `rake docker:build`. Live Docker tests are opt-in via `SKILL_BENCH_DOCKER_TESTS=1`.
+
+---
+
+## Source map
+
+| Concept | Authoritative file |
+| --- | --- |
+| CLI dispatch | `lib/skill_bench/cli.rb` |
+| Single eval | `lib/skill_bench/services/runner_service.rb` |
+| Blind judge | `lib/skill_bench/evaluation/runner.rb` |
+| Verdict | `lib/skill_bench/delta_report.rb` |
+| Sandbox + Docker | `lib/skill_bench/execution/sandbox.rb` |
+| Host exec gate | `lib/skill_bench/tools/run_command.rb` |
+| Context caps | `lib/skill_bench/constants.rb` (`ContextHydration`) |
+| Config hierarchy | `lib/skill_bench/config.rb` |
+| Provider env keys | `lib/skill_bench/config/env_overrides.rb` |
+| Provider list | `lib/skill_bench/clients/provider_schemas.rb` |
+| Image contract | `docs/docker.md` |
+
+Deprecated entry points still in the tree, not on the live path: `SkillBench::Runner`, `EvaluateCommand`, `Task::Evaluator`.
+
+---
+
+## Verification
+
+| Claim | Evidence |
+| --- | --- |
+| CLI subcommands | `SkillBench::CLI#call` case in `lib/skill_bench/cli.rb` |
+| Concurrent agents | `RunnerService#run_agents_concurrently` |
+| Concurrent judges | `Evaluation::Runner#run_judges_concurrently` |
+| Verdict formula | `DeltaReport#determine_verdict` |
+| Fail-closed host exec | `Tools::RunCommand` + `Sandbox` comments and `allow_host_execution` default |
+| Docker flags | `Sandbox#start_container` |
+| Context size caps | `Constants::ContextHydration` + `ContextHydrator#collect_context_files` |
+| `--cache` | `Cli::RunCommand` sets `ENV['SKILL_BENCH_CACHE']`; `HelpPrinter` documents it |
+| OpenRouter env | `EnvOverrides::ENV_TO_PROVIDER_SETTINGS` |
+
+Evidence gap: whether every provider in `ProviderSchemas` is registered at boot is covered by client tests under `test/evaluator/clients/`, not by a single registry integration test named in this document.

data/docs/blind-comparisons.md

@@ -0,0 +1,9 @@
+# Comparing skill revisions
+
+Use the same task, fixture commit, acceptance tests, criteria file, model, host configuration, and repetition count for no skills, current skills, and revised skills. Keep a held-out task set. Run the current and revised conditions against the same no-skills baseline protocol; archive the actual outputs and provenance alongside each report.
+
+`Evaluation::Runner` gives both independent judges the same task and criteria. Skill instructions are supplied only to the executing agent. Its `skill_context` argument remains accepted for compatibility but is deliberately omitted from judging. The active evaluator scores one output per independent request, so there is no paired presentation order. A future paired judge must shuffle anonymous outputs and map scores back outside its prompt.
+
+Write criteria as observable task requirements before generating outputs. The legacy `skill_adherence` dimension name remains supported; its description must state an independently assessable requirement rather than ask the judge to infer whether a skill was used. Keep treatment names and instructions out of criteria and agent summaries sent to the judge.
+
+Executable acceptance tests take precedence over prose scores. Preserve test failures and unavailable checks as evidence; a prose score cannot turn either into a passing execution result. Report repetitions, fixture and skill commits, host/model versions, actual usage, and incomplete runs. A small or inconclusive sample establishes no statistical improvement. Paid comparisons require a manual invocation and an explicit usage cap; the unit suite makes no provider calls.

data/docs/first-eval-guide.md

@@ -42,7 +42,7 @@ skill-bench init --openai
 
 This creates `skill-bench.json` with the OpenAI provider config. Use `--force` to overwrite.
 
-**Available providers:** `--openai`, `--anthropic`, `--gemini`, `--ollama`, `--azure`, `--groq`, `--deepseek`, `--opencode`
+**Available providers:** `--openai`, `--anthropic`, `--gemini`, `--ollama`, `--azure`, `--groq`, `--deepseek`, `--mistral`, `--opencode`, `--openrouter`, `--xai`, `--bedrock`, plus `--mock` for an offline config with no API key.
 
 > **What is `skill-bench.json`?** This is your config file. It stores your API key, chosen LLM model, timeout, and allowed shell commands. Think of it as `.env` but structured as JSON. You edit it; SkillBench reads it.
 

data/lib/skill_bench/cli/help_printer.rb

@@ -8,7 +8,7 @@ module SkillBench
       #
       # @return [Integer] Exit code (always 0)
       def self.call
-        providers = SkillBench::Clients::ProviderSchemas.names.map { |name| "--#{name}" }.join(', ')
+        providers = (SkillBench::Clients::ProviderSchemas.names.map { |name| "--#{name}" } + ['--mock']).join(', ')
 
         puts <<~USAGE
           Usage: skill-bench <subcommand> [options]
@@ -28,6 +28,7 @@ module SkillBench
               --all      Run every eval under evals/ (batch mode)
               --evals-dir DIR  Run every eval under DIR (batch mode)
               --summary  Emit a JSON summary gate for a batch run (batch mode)
+              --cache    Enable content-addressed response caching
 
             compare <skill-name> --variant-a SPEC --variant-b SPEC --eval PATH
               Compare the same skill across two pack variants

data/lib/skill_bench/clients/all.rb

@@ -20,4 +20,6 @@ require_relative 'providers/groq'
 require_relative 'providers/deepseek'
 require_relative 'providers/mistral'
 require_relative 'providers/openrouter'
+require_relative 'providers/xai'
+require_relative 'providers/bedrock'
 require_relative 'providers/mock'

data/lib/skill_bench/clients/provider_schemas.rb

@@ -51,6 +51,15 @@ module SkillBench
         openrouter: {
           api_key: nil,
           model: 'anthropic/claude-3.5-sonnet'
+        }.freeze,
+        xai: {
+          api_key: nil,
+          model: 'grok-4'
+        }.freeze,
+        bedrock: {
+          api_key: nil,
+          model: 'amazon.nova-lite-v1:0',
+          location: 'us-east-1'
         }.freeze
       }.freeze
 

data/lib/skill_bench/clients/providers/bedrock.rb

@@ -0,0 +1,49 @@
+# frozen_string_literal: true
+
+require_relative '../base_client'
+require_relative '../provider_registry'
+
+module SkillBench
+  module Clients
+    module Providers
+      # Amazon Bedrock LLM client.
+      # Uses Bedrock Runtime's OpenAI-compatible Chat Completions path with a
+      # Bedrock API key (bearer). IAM SigV4 signing is a follow-up.
+      class Bedrock < BaseClient
+        SkillBench::Clients::ProviderRegistry.register(:bedrock, self)
+
+        DEFAULT_REGION = 'us-east-1'
+
+        # Returns the provider identifier.
+        #
+        # @return [Symbol]
+        def provider_name
+          :bedrock
+        end
+
+        protected
+
+        # Returns the Bedrock Runtime host for the configured region.
+        #
+        # @return [String]
+        def base_url
+          @base_url_config || "https://bedrock-runtime.#{region}.amazonaws.com"
+        end
+
+        # Returns the OpenAI-compatible chat completions path.
+        #
+        # @return [String]
+        def request_path
+          @request_path_config || '/openai/v1/chat/completions'
+        end
+
+        private
+
+        def region
+          loc = @location.to_s.strip
+          loc.empty? ? DEFAULT_REGION : loc
+        end
+      end
+    end
+  end
+end

data/lib/skill_bench/clients/providers/xai.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+require_relative '../base_client'
+require_relative '../provider_registry'
+
+module SkillBench
+  module Clients
+    module Providers
+      # xAI (Grok) LLM client.
+      # Uses xAI's OpenAI-compatible chat completions API with bearer-token auth.
+      class Xai < BaseClient
+        SkillBench::Clients::ProviderRegistry.register(:xai, self)
+
+        # Returns the provider identifier.
+        #
+        # @return [Symbol]
+        def provider_name
+          :xai
+        end
+
+        protected
+
+        # Returns the base URL for the xAI API.
+        #
+        # The version segment lives in {#request_path} so Faraday does not drop
+        # it (an absolute request path replaces any path component of the
+        # connection base URL).
+        #
+        # @return [String]
+        def base_url
+          @base_url_config || 'https://api.x.ai'
+        end
+
+        # Returns the request path for chat completions.
+        #
+        # @return [String]
+        def request_path
+          @request_path_config || '/v1/chat/completions'
+        end
+      end
+    end
+  end
+end

data/lib/skill_bench/config/defaults.rb

@@ -36,7 +36,9 @@ module SkillBench
             groq: { api_key: nil, model: 'llama-3.3-70b-versatile' },
             deepseek: { api_key: nil, model: 'deepseek-chat' },
             opencode: { api_key: nil, model: 'opencode-model', base_url: nil },
-            openrouter: { api_key: nil, model: 'anthropic/claude-3.5-sonnet' }
+            openrouter: { api_key: nil, model: 'anthropic/claude-3.5-sonnet' },
+            xai: { api_key: nil, model: 'grok-4' },
+            bedrock: { api_key: nil, model: 'amazon.nova-lite-v1:0', location: 'us-east-1' }
           }
         }
       end

data/lib/skill_bench/config/env_overrides.rb

@@ -49,7 +49,16 @@ module SkillBench
         'SKILL_BENCH_OPENCODE_MODEL' => %i[opencode model],
         'OPENCODE_MODEL' => %i[opencode model],
         'SKILL_BENCH_OPENROUTER_API_KEY' => %i[openrouter api_key],
-        'SKILL_BENCH_OPENROUTER_MODEL' => %i[openrouter model]
+        'SKILL_BENCH_OPENROUTER_MODEL' => %i[openrouter model],
+        'SKILL_BENCH_XAI_API_KEY' => %i[xai api_key],
+        'XAI_API_KEY' => %i[xai api_key],
+        'SKILL_BENCH_XAI_MODEL' => %i[xai model],
+        'XAI_MODEL' => %i[xai model],
+        'SKILL_BENCH_BEDROCK_API_KEY' => %i[bedrock api_key],
+        'AWS_BEARER_TOKEN_BEDROCK' => %i[bedrock api_key],
+        'SKILL_BENCH_BEDROCK_MODEL' => %i[bedrock model],
+        'SKILL_BENCH_BEDROCK_REGION' => %i[bedrock location],
+        'AWS_REGION' => %i[bedrock location]
       }.freeze
 
       # Returns provider overrides from the given environment.

data/lib/skill_bench/evaluation/runner.rb

@@ -31,6 +31,7 @@ module SkillBench
       def initialize(task:, criteria:, skill_context:, baseline_output:, context_output:, judge_params: {})
         @task = task
         @criteria = criteria
+        # Callers still pass skill_context; it is not forwarded to judges.
         @skill_context = skill_context
         @baseline_output = baseline_output
         @context_output = context_output
@@ -66,17 +67,17 @@ module SkillBench
       # @return [Array(Hash, Hash)] Baseline and context judge results, in order.
       def run_judges_concurrently
         runs = [
-          -> { judge_run(baseline_output, nil) },
-          -> { judge_run(context_output, skill_context) }
+          -> { judge_run(baseline_output) },
+          -> { judge_run(context_output) }
         ]
         Parallel.map(runs, in_threads: runs.size, &:call)
       end
 
-      def judge_run(output, context)
+      def judge_run(output)
         prompt_result = Judge::Prompt.call(
           task: task,
           criteria: criteria,
-          skill_context: context,
+          skill_context: nil,
           agent_output: output
         )
         return prompt_result unless prompt_result[:success]

data/lib/skill_bench/execution/sandbox.rb

@@ -127,9 +127,10 @@ module SkillBench
       #
       # @raise [RuntimeError] when any git command fails.
       def setup_git
+        # Subcommands only — git_command prepends the binary + GIT_HARDENING flags.
         subcommands = [
           ['init', '--quiet'],
-          ['config', 'user.email', 'evaluator@tessl.io'],
+          ['config', 'user.email', 'evaluator@example.com'],
           ['config', 'user.name', 'Evaluator Sandbox'],
           ['add', '.'],
           ['commit', '--quiet', '-m', 'Initial commit']

data/lib/skill_bench/judge/variance.rb

@@ -0,0 +1,49 @@
+# frozen_string_literal: true
+
+require_relative '../error_logger'
+
+module SkillBench
+  module Judge
+    # Sample statistics over repeated judge totals.
+    class Variance
+      # @param totals [Array<Numeric>] Judge totals from repeated runs of the same eval.
+      # @return [Hash] `{ success: true, response: { n:, mean:, stddev:, spread: } }`
+      def self.call(totals:)
+        new(totals).call
+      end
+
+      # @param totals [Array<Numeric>]
+      def initialize(totals)
+        @totals = totals
+      end
+
+      # @return [Hash]
+      def call
+        nums = Array(@totals).map { |total| Float(total) }
+        return empty_result(count: nums.size) if nums.size < 2
+
+        mean = nums.sum / nums.size
+        sample_variance = nums.sum { |value| (value - mean)**2 } / (nums.size - 1)
+
+        {
+          success: true,
+          response: {
+            n: nums.size,
+            mean: mean,
+            stddev: Math.sqrt(sample_variance),
+            spread: nums.max - nums.min
+          }
+        }
+      rescue StandardError => e
+        SkillBench::ErrorLogger.log_error(e, 'Judge::Variance Error')
+        { success: false, response: { error: { message: e.message } } }
+      end
+
+      private
+
+      def empty_result(count:)
+        { success: true, response: { n: count, mean: nil, stddev: nil, spread: nil } }
+      end
+    end
+  end
+end

data/lib/skill_bench/services/delta_table_formatter.rb

@@ -65,7 +65,23 @@ module SkillBench
         context_icon = trend_icon(trend[:context_trend])
         baseline_delta = trend[:baseline_delta]
         context_delta = trend[:context_delta]
-        "  TREND: baseline #{baseline_icon} (#{delta_str(baseline_delta)}), context #{context_icon} (#{delta_str(context_delta)})"
+        lines = [
+          "  TREND: baseline #{baseline_icon} (#{delta_str(baseline_delta)}), context #{context_icon} (#{delta_str(context_delta)})"
+        ]
+        lines << format_judge_variance(trend[:judge_variance])
+        lines.compact.join("\n")
+      end
+
+      private_class_method def self.format_judge_variance(stats)
+        return nil unless stats.is_a?(Hash) && stats[:n].to_i >= 2
+
+        Kernel.format(
+          '  JUDGE n=%<n>d mean=%<mean>.1f σ=%<stddev>.1f spread=%<spread>.1f',
+          n: stats[:n],
+          mean: stats[:mean],
+          stddev: stats[:stddev],
+          spread: stats[:spread]
+        )
       end
     end
   end

data/lib/skill_bench/trend_tracker/trend_calculator.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require_relative '../judge/variance'
+
 module SkillBench
   class TrendTracker
     # Calculates performance trends between evaluation results
@@ -25,7 +27,8 @@ module SkillBench
           context_trend: trend_direction(current_context, previous_context),
           baseline_delta: current_baseline - previous_baseline,
           context_delta: current_context - previous_context,
-          previous_run: previous[:timestamp]
+          previous_run: previous[:timestamp],
+          judge_variance: judge_variance(matching, current_context)
         }
       end
 
@@ -54,6 +57,12 @@ module SkillBench
 
           current > previous ? :improved : :regressed
         end
+
+        def judge_variance(matching, current_context)
+          totals = matching.filter_map { |entry| entry[:context_total] } + [current_context]
+          result = SkillBench::Judge::Variance.call(totals: totals)
+          result[:success] ? result[:response] : nil
+        end
       end
     end
   end

data/lib/skill_bench/version.rb

@@ -2,5 +2,5 @@
 
 module SkillBench
   # The current gem version.
-  VERSION = '1.3.0'
+  VERSION = '1.4.0'
 end

data/lib/skill_bench.rb

@@ -22,6 +22,7 @@ require_relative 'skill_bench/judge'
 require_relative 'skill_bench/judge/judge'
 require_relative 'skill_bench/judge/prompt'
 require_relative 'skill_bench/judge/response'
+require_relative 'skill_bench/judge/variance'
 
 # Agent subsystem
 require_relative 'skill_bench/agent'

metadata

@@ -1,13 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: ruby-skill-bench
 version: !ruby/object:Gem::Version
-  version: 1.3.0
+  version: 1.4.0
 platform: ruby
 authors:
 - Ismael Marin
+autorequire:
 bindir: bin
 cert_chain: []
-date: 1980-01-02 00:00:00.000000000 Z
+date: 2026-09-14 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: cgi
@@ -43,14 +44,20 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '2.20'
+        version: '2.21'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.21.2
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '2.20'
+        version: '2.21'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.21.2
 - !ruby/object:Gem::Dependency
   name: parallel
   requirement: !ruby/object:Gem::Requirement
@@ -80,6 +87,7 @@ files:
 - README.md
 - bin/skill-bench
 - docs/architecture.md
+- docs/blind-comparisons.md
 - docs/docker.md
 - docs/first-eval-guide.md
 - docs/testing-guide.md
@@ -113,6 +121,7 @@ files:
 - lib/skill_bench/clients/provider_schemas.rb
 - lib/skill_bench/clients/providers/anthropic.rb
 - lib/skill_bench/clients/providers/azure_openai.rb
+- lib/skill_bench/clients/providers/bedrock.rb
 - lib/skill_bench/clients/providers/deepseek.rb
 - lib/skill_bench/clients/providers/gemini.rb
 - lib/skill_bench/clients/providers/groq.rb
@@ -123,6 +132,7 @@ files:
 - lib/skill_bench/clients/providers/openai.rb
 - lib/skill_bench/clients/providers/opencode.rb
 - lib/skill_bench/clients/providers/openrouter.rb
+- lib/skill_bench/clients/providers/xai.rb
 - lib/skill_bench/clients/request_builder.rb
 - lib/skill_bench/clients/response_builder.rb
 - lib/skill_bench/clients/response_error_handler.rb
@@ -165,6 +175,7 @@ files:
 - lib/skill_bench/judge/judge.rb
 - lib/skill_bench/judge/prompt.rb
 - lib/skill_bench/judge/response.rb
+- lib/skill_bench/judge/variance.rb
 - lib/skill_bench/migration/provider_migrator.rb
 - lib/skill_bench/models/config.rb
 - lib/skill_bench/models/criteria_validator.rb
@@ -234,6 +245,7 @@ licenses:
 metadata:
   rubygems_mfa_required: 'true'
   source_code_uri: https://github.com/igmarin/ruby-skill-bench
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -248,7 +260,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.16
+rubygems_version: 3.5.22
+signing_key:
 specification_version: 4
 summary: The evaluation engine for AI Agent Skills benchmarking.
 test_files: []