ruby-skill-bench 1.0.1 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d3c4edfe40e04251d2e7b758e7c630ee9affaa9e8170ceb0fa379d61bacc81e6
-  data.tar.gz: e9ef2eb8ef7a524d607c6e44705df772feec8939a376b516adff032eeeb8b535
+  metadata.gz: f47976b55f6f8c147adb4ed784ce04ba52ff71f805f8e35d797ba776021641c4
+  data.tar.gz: c2febaadbdeb7e149041661258ce84e41499121445cf726cefece642e60174a4
 SHA512:
-  metadata.gz: b92554c769e34205d1c197bd67a9ca2ae61876b83c5429e202c667831100470fa9f1ed48a297ea184855e33e7ac3945fb513909b2344634078b8090750325dc9
-  data.tar.gz: 7ae92f1331f2061cccf42a1f27f80cbe41c73d54d0909499900efa84ad3984edada8e7df10b5a018717861234974918cdfac80b5242483df108272093eec8deb
+  metadata.gz: 5ab3082fa715a0776455a88b28e2d990d3bb7e52fbc5f0cc47176e1d44e76cdc430531bc0401abab350cb4c2321d0b625ce78d381094fe404cdedd7e61b27227
+  data.tar.gz: 3d5f67b876457691e003e62a8ba57fc04ac21e002bb5c6a84d1ae7954cf7dd7136f05a658d7ee4d416ab2db19cb0ad90e51dd0d920df9ffa998cb8391db65df5

data/README.md

@@ -30,7 +30,7 @@ See the [Ecosystem Overview](https://github.com/igmarin/agent-mcp-runtime/blob/m
 - **Isolated Git Sandboxes**: Every run operates in a temporary repo. Clean diffs, zero side-effects, 100% reproducibility.
 - **Blind Judging with Dimensions**: LLM judge scores baseline and context independently across 5 canonical dimensions (Correctness, Skill Adherence, Code Quality, Test Coverage, Documentation). Eval authors configure weights and thresholds via `criteria.json`.
 - **Sophisticated ReAct Loop**: Employs a robust `Thought → Tool → Observation` loop to handle complex, multi-step engineering tasks.
-- **Multi-Provider Ecosystem**: Native support for **OpenAI**, **Anthropic**, **Google Gemini**, **Azure OpenAI**, **Ollama**, **Groq**, **DeepSeek**, and **OpenCode**.
+- **Multi-Provider Ecosystem**: Native support for **OpenAI**, **Anthropic**, **Google Gemini**, **Azure OpenAI**, **Ollama**, **Groq**, **DeepSeek**, **Mistral**, and **OpenCode**.
 - **Standardized Intelligence**: Consistent reporting format regardless of the underlying LLM provider.
 
 ---
@@ -64,11 +64,14 @@ CLI / API → RunnerService → Sandbox + ReAct Agent → LLM Client Layer → P
 | **Ollama** | — | `:ollama` |
 | **Groq** | `SKILL_BENCH_GROQ_API_KEY` | `:groq` |
 | **DeepSeek** | `SKILL_BENCH_DEEPSEEK_API_KEY` | `:deepseek` |
+| **Mistral** | `SKILL_BENCH_MISTRAL_API_KEY` | `:mistral` |
 | **OpenCode** | `SKILL_BENCH_OPENCODE_API_KEY`, `SKILL_BENCH_OPENCODE_BASE_URL` | `:opencode` |
 
 > **Note:** Environment variables are loaded automatically. You can also configure provider settings in `skill-bench.json` (created by `skill-bench init`).
 >
 > **OpenCode requires a custom `base_url`:** OpenCode does not host a public LLM API. You must provide your own OpenAI-compatible endpoint (e.g. a LiteLLM proxy, self-hosted vLLM, or company gateway) via the `base_url` config key. Without it, the provider will fail with "Base URL not set for Opencode".
+>
+> **Mistral** uses Mistral's OpenAI-compatible chat completions API (default model `mistral-large-latest`). Set `SKILL_BENCH_MISTRAL_API_KEY` and scaffold it with `skill-bench init --mistral`.
 
 ### Command Allowlist
 
@@ -79,6 +82,7 @@ By default, no shell commands are permitted. You must configure `allowed_command
   "provider": "openai",
   "max_execution_time": 30,
   "allowed_commands": ["rspec", "bundle", "ruby", "git"],
+  "allow_host_execution": false,
   "config": {
     "api_key": null,
     "model": "gpt-4o"
@@ -87,6 +91,8 @@ By default, no shell commands are permitted. You must configure `allowed_command
 ```
 
 > **Security:** The agent can only execute commands on this list. Dangerous commands (bash, curl, sudo, etc.) are always blocked regardless of configuration.
+>
+> **Where commands run:** Allowed commands run inside a temporary git **sandbox directory** on the host — a copy of your eval files, not your project. True container isolation (Docker) is **not yet shipped**, so the sandbox directory is the only boundary. Because of this, host execution **fails closed**: it is disabled by default and must be explicitly enabled with `"allow_host_execution": true`. With it disabled (the default), `run_command` refuses to execute and returns an error instead of running un-isolated. Enable it only when you accept that allowed commands run directly on your machine.
 
 ### Configuration Hierarchy
 
@@ -137,7 +143,9 @@ skill-bench init --openai
 }
 ```
 
-**Available providers:** `--openai`, `--anthropic`, `--gemini`, `--ollama`, `--azure`, `--groq`, `--deepseek`, `--opencode`
+**Available providers:** `--openai`, `--anthropic`, `--gemini`, `--ollama`, `--azure`, `--groq`, `--deepseek`, `--mistral`, `--opencode`
+
+**Zero-config offline path:** `skill-bench init --mock` scaffolds a minimal offline config that needs no API key and no network — `{"provider":"mock","max_execution_time":30}`. Use it to try the full flow (and run the bundled examples) before wiring up a real provider.
 
 Use `--force` to overwrite an existing config.
 
@@ -338,7 +346,7 @@ skill-bench run my-first-eval --skill=my-service
 3. **Context run** — Agent receives `task.md` + `SKILL.md` as prompt → produces output B
 4. **Blind judging** — LLM judge scores output A and output B independently across the dimensions defined in `criteria.json`
 5. **Delta computation** — Compare scores, compute deltas, apply pass/fail logic
-6. **History recording** — Store result in `.skill-bench-history.json` for trend tracking
+6. **History recording** — Store result in `.skill-bench-trends.json` for trend tracking
 
 Provider is read from `skill-bench.json` — no `--provider` flag needed.
 
@@ -350,11 +358,54 @@ skill-bench run my-first-eval --skill=skill-a --skill=skill-b
 
 Both skill contexts are concatenated and sent to the agent. The judge evaluates whether the combined context improves results.
 
-**Output Formats:**
+**Output Formats:** `--format human` (default), `json`, `junit`, or `html`.
+
+- Human-readable (default) — full delta table, iteration timeline, and a `Tokens: N | Est. Cost: $X.XXXX` line.
+- JSON: `--format json` — machine-readable, including top-level `tokens` and `cost` fields.
+- JUnit XML: `--format junit` — for CI test reporting.
+- HTML: `--format html` — a self-contained, shareable report (styles inlined, no external assets) with the delta table and iteration timeline. Redirect it to a file:
+
+  ```bash
+  skill-bench run my-first-eval --skill=my-service --format html > report.html
+  ```
+
+---
+
+## Pre-flight Checks: `validate` / `doctor`
+
+Before spending tokens on a run, sanity-check your setup. `skill-bench validate` (aliased as `doctor`) runs read-only pre-flight checks — it never runs an eval and never makes a network call:
+
+```bash
+skill-bench validate
+# or, identically:
+skill-bench doctor
+```
+
+It runs three checks and prints a `PASS` / `FAIL` / `SKIP` line for each:
 
-- Human-readable (default)
-- JSON: `--format json`
-- JUnit XML: `--format junit`
+1. **criteria** — validates the criteria JSON (default `criteria.json`, override with `--criteria PATH`). Skipped if the default file is absent.
+2. **config** — schema-checks `skill-bench.json` (default, override with `--config PATH`): `provider` is required and must be a known provider, `max_execution_time` must be a positive integer, and `config` (when present) must be an object.
+3. **provider key** — reports whether the configured provider's API key is present (the `mock` provider needs none).
+
+A passing report exits `0`:
+
+```text
+skill-bench validate
+
+[PASS] criteria      criteria.json is valid
+[PASS] config        skill-bench.json matches the expected shape
+[PASS] provider key  openai credentials present
+
+All checks passed.
+```
+
+A failure exits non-zero and names what is wrong:
+
+```text
+[FAIL] provider key  openai is missing: api_key
+
+1 check(s) failed.
+```
 
 ---
 
@@ -427,6 +478,22 @@ The `--variant` spec supports two forms:
 - `pack:<name>` — resolve via registry manifest
 - `/absolute/path` or `relative/path` — use a direct path
 
+### Response Caching (opt-in, `--cache`)
+
+LLM responses can be cached so identical requests reuse a previous result instead of calling the provider again. Caching is **off by default**. Enable it per run with `--cache`, or set the `SKILL_BENCH_CACHE` environment variable to a truthy value (`1`, `true`, `yes`, or `on`):
+
+```bash
+# Per-run flag
+skill-bench run my-first-eval --skill=my-service --cache
+
+# Or via the environment
+SKILL_BENCH_CACHE=1 skill-bench run my-first-eval --skill=my-service
+```
+
+The cache is in-memory (process-lifetime) and content-addressed: the key is a SHA-256 digest of the provider, model, system prompt, messages, tools, and temperature, so only truly identical requests share an entry. The `mock` and null providers are never cached.
+
+This pays off most with `compare`, which runs the skill-less baseline twice with identical inputs — with caching enabled, the repeated baseline reuses the cached response instead of making a second call.
+
 ---
 
 ## File Reference: What Lives on Disk
@@ -446,6 +513,7 @@ SkillBench creates and manages three files in your project. Understanding them h
   "provider": "openai",
   "max_execution_time": 300,
   "allowed_commands": ["rspec", "bundle", "ruby", "git"],
+  "allow_host_execution": false,
   "config": {
     "api_key": "sk-...",
     "model": "gpt-4o",
@@ -458,10 +526,11 @@ SkillBench creates and manages three files in your project. Understanding them h
 - Configuration is loaded in this order: **code defaults** → `~/.skill-bench.json` (user-wide) → `./skill-bench.json` (local) → **environment variables**. Later sources override earlier ones.
 - If `api_key` is `null`, SkillBench looks for the matching environment variable (e.g. `SKILL_BENCH_OPENAI_API_KEY`).
 - `allowed_commands` is a **safeguard**, not a convenience. By default the agent cannot run *any* shell command. Add only what your evals need.
+- `allow_host_execution` (default `false`) gates whether `run_command` may run on the host when no container isolation is active. Since container isolation is not yet shipped, leaving it `false` means `run_command` **fails closed** (refuses to execute). Set it to `true` only if you accept that allowed commands run directly on your machine inside the temporary sandbox directory.
 
 ---
 
-### `.skill-bench-history.json` — Evaluation History (Auto-Generated)
+### `.skill-bench-trends.json` — Evaluation History (Auto-Generated)
 
 **What it is:** A JSON array that records every successful eval run. SkillBench appends to it automatically. It stores the timestamp, eval name, skill names, scores, and deltas so you can track improvement over time.
 
@@ -497,13 +566,13 @@ TREND: baseline ↑ (+2), context ↑ (+7)
 
 The trend compares the current run against the *previous run of the same eval + skill*. This tells you at a glance whether your latest skill edit made things better or worse.
 
-**Pro tip:** Commit `.skill-bench-history.json` to git if you want to share trend data with your team. Add it to `.gitignore` if you prefer to keep scores private.
+**Pro tip:** `.skill-bench-trends.json` is git-ignored by default (via the `.skill-bench-trends.json*` line in `.gitignore`). If you want to share trend data with your team, remove that line so the file can be committed.
 
 ---
 
-### `.skill-bench-history.json.bak` — Backup (Auto-Generated)
+### `.skill-bench-trends.json.bak` — Backup (Auto-Generated)
 
-**What it is:** A copy of `.skill-bench-history.json` created every time SkillBench writes a new entry. If the main file gets corrupted (e.g. you kill the process mid-write), SkillBench automatically falls back to the `.bak` file.
+**What it is:** A snapshot of the *previous* good version of `.skill-bench-trends.json`, copied just before each new write. (The first run has no prior version yet, so it creates no `.bak`.) If the main file gets corrupted (e.g. you kill the process mid-write), SkillBench automatically falls back to the `.bak` file.
 
 **Who edits it:** Nobody. It is a safety net.
 
@@ -541,7 +610,7 @@ Read the output carefully. Look at **two things:**
 ### Step 3: Inspect the History
 
 ```bash
-cat .skill-bench-history.json | jq '.[-1]'
+cat .skill-bench-trends.json | jq '.[-1]'
 ```
 
 This shows the latest entry. Focus on the dimension with the smallest delta — that is where your skill is weakest.
@@ -729,6 +798,7 @@ These 5 dimensions are **mandatory** in every `criteria.json`. You can add custo
   Eval: my-first-eval
   Skill: my-service
   Provider: openai
+  Tokens: 18432  |  Est. Cost: $0.0934
 ═══════════════════════════════════════════════════════
 
   === BASELINE ITERATIONS ===
@@ -774,8 +844,9 @@ These 5 dimensions are **mandatory** in every `criteria.json`. You can add custo
 - **CONTEXT:** The agent's score *with* the skill. This is the "aided" performance.
 - **DELTA:** `CONTEXT - BASELINE`. How much the skill helped.
 - **TOTAL:** Sum of all dimension scores. Max possible is 100.
-- **TREND:** Comparison against the previous run of the same eval + skill (from `.skill-bench-history.json`). Shows whether scores are improving over time.
+- **TREND:** Comparison against the previous run of the same eval + skill (from `.skill-bench-trends.json`). Shows whether scores are improving over time.
 - **VERDICT:** `PASS` only if `CONTEXT >= pass_threshold` AND `DELTA >= minimum_delta`.
+- **Tokens / Est. Cost:** The header shows total tokens used across the run and an estimated USD cost as `Tokens: N | Est. Cost: $X.XXXX`. The cost is approximate — it comes from a built-in per-model price table (`Services::CostCalculator`) and shows `—` when the model isn't in that table. JSON output (`--format json`) exposes the same data as top-level `tokens` and `cost` fields.
 
 **Iteration timeline:**
 
@@ -827,7 +898,7 @@ Your eval result depends on **both** conditions. Here is every scenario:
 
 ## Reliability & Security
 
-- **Safe-by-Design**: No code execution occurs on the host system; everything happens in the sandbox.
+- **Allowlist-Gated Execution**: The agent can only run commands you add to `allowed_commands`; with an empty allowlist it can run nothing. Commands run inside a temporary git sandbox **directory** (a copy of the eval files) on the host — container isolation is not yet shipped, so host execution is **disabled by default** and must be explicitly opted into with `allow_host_execution: true`.
 - **Command Blocklist**: Dangerous commands (`bash`, `sh`, `python`, `curl`, etc.) are always blocked, even if listed in `allowed_commands`.
 - **Path Validation**: Eval paths are validated to prevent directory traversal attacks.
 - **Atomic History Writes**: Benchmark history uses file locking to prevent corruption from concurrent writes.
@@ -836,7 +907,7 @@ Your eval result depends on **both** conditions. Here is every scenario:
 - **Traceability**: Every thought and tool call is logged with full backtrace for post-mortem analysis.
 - **Robust Error Recovery**: Handles provider outages and rate limits gracefully with standardized error logging.
 - **XML-Safe Output**: JUnit XML output is properly escaped to prevent injection attacks.
-- **Test Coverage**: 373+ tests covering core engine, CLI commands, and all provider clients.
+- **Test Coverage**: 700+ tests covering core engine, CLI commands, and all provider clients. Run `bundle exec rake test` to see the current count.
 
 ## Testing
 
@@ -855,21 +926,226 @@ bundle exec ruby -Itest test/integration_test.rb
 
 **Test Structure:**
 
-- `test/evaluator/` — Core evaluation engine tests
-- `test/agent_eval/` — CLI, models, and service tests
+- `test/agent/` — Agent runtime tests
+- `test/agent_eval/` — Agent evaluation tests
+- `test/cli/` — CLI command tests
 - `test/clients/` — Provider client tests
+- `test/evaluator/` — Core evaluation engine tests
+- `test/history_recorder/` — Benchmark history persistence tests
+- `test/models/` — Domain model tests
+- `test/registry/` — Skill/eval registry tests
+- `test/services/` — Service layer tests
+- `test/skills/` — Skill loading tests
+- `test/tools/` — Agent tool tests
+- Plus several top-level `test/*_test.rb` files (e.g. `integration_test.rb`, `evaluation_runner_test.rb`, `trend_tracker_test.rb`).
+
+---
+
+## Security
+
+### Threat Model
+
+Ruby Skill Bench is designed with security as a primary concern. The system executes AI agents in isolated environments and must protect against various attack vectors:
+
+- **Path Traversal:** Preventing agents from accessing files outside the sandbox
+- **Command Injection:** Preventing execution of arbitrary shell commands
+- **Resource Exhaustion:** Preventing denial-of-service through resource consumption
+- **Information Leakage:** Protecting sensitive data like API keys
+
+### Security Features
+
+#### Path Traversal Protection
+
+- **Symlink Validation:** All symlinks are validated to ensure they don't escape the sandbox
+- **TOCTOU Mitigation:** Path validation is re-checked after directory creation operations
+- **Path Normalization:** All paths are normalized and validated against working directory boundaries
+- **Character Validation:** Paths are validated against strict character patterns
+
+#### Command Execution Security
+
+- **Command Allowlist:** Only explicitly allowed commands can be executed
+- **Dangerous Commands Blocklist:** Dangerous commands (bash, curl, sudo, etc.) are always blocked
+- **Shell Tokenization:** Commands are tokenized before execution to prevent shell injection
+- **Fail-Closed Host Execution:** Container isolation is not yet active, so commands run on the host inside a temporary sandbox directory. To match this reality, `run_command` refuses to execute unless `allow_host_execution: true` is set; it is **disabled by default**.
+
+> **The allowlist is the only real authorization control — and it only checks the base command.** `run_command` authorizes by the first token of the command (`rake`, `find`, `git`, …); it does **not** inspect arguments. Shell tokenization stops metacharacter injection, but it does **not** sandbox what an allowlisted binary can do. Because many common tools are general-purpose execution wrappers, **allowlisting any one of them is equivalent to granting arbitrary host code execution** — for example `rake -e '...'`, `rspec -e`, `make` (arbitrary recipes), `find . -exec ...`, or `git` (hooks, `-c core.fsmonitor=...`, `! ...` aliases). Combined with the fail-closed model above (`run_command` refuses to run on the host unless `allow_host_execution` is explicitly enabled — see `HOST_EXECUTION_REFUSED` in `run_command.rb`), the practical guidance is: **keep `allowed_commands` as minimal as possible — empty for untrusted skills** — and treat every entry as if you were handing the skill a shell.
+>
+> An **optional, default-off** `command_argument_constraints` setting can refuse commands whose arguments contain configured substrings/flags (for example blocking `-e` or `-exec`). It is a defense-in-depth speed bump, **not** a sandbox, and is unset by default; the allowlist remains the control that matters.
+
+#### Docker Security Hardening (Planned — Not Yet Active)
+
+> **Status:** The container isolation model described below is **planned, not shipped**. No Docker build context is packaged, so containers are never launched today — `run_command` runs on the host gated by the allowlist and `allow_host_execution`. The settings below document the intended hardened model for when container isolation lands.
+
+When container isolation is enabled in a future release, containers are intended to launch with hardened security settings:
+
+- **Non-root User:** Containers run as a non-root user
+- **Privilege Prevention:** `--security-opt no-new-privileges` prevents privilege escalation
+- **Capability Dropping:** All Linux capabilities are dropped except minimal needed ones
+- **Network Isolation:** `--network none` disables network access
+- **Read-only Root:** Container filesystem is read-only (except for mounted volumes)
+
+#### Resource Limits
+
+- **File Size Limits:** Individual files in context hydration are limited to 50KB
+- **Total Context Size:** Total context size is limited to 1MB to prevent memory exhaustion
+- **Execution Timeout:** Commands are limited to a configurable timeout (default: 30 seconds)
+- **Max Iterations:** Agent loops are limited to prevent infinite loops
+
+### API Key Security
+
+- **Environment Variables:** API keys are loaded from environment variables, not hardcoded
+- **Configuration Hierarchy:** Keys can be set in `skill-bench.json` or environment variables
+- **No Logging:** API keys are never logged or exposed in error messages
+- **Provider-Specific Keys:** Each provider uses its own API key configuration
+
+### Best Practices for Users
+
+1. **Never Commit API Keys:** Never commit `skill-bench.json` with API keys to version control
+2. **Use Environment Variables:** Prefer environment variables for sensitive configuration
+3. **Minimal Command Allowlist:** Only allow commands necessary for your evals
+4. **Regular Updates:** Keep dependencies updated to patch security vulnerabilities
+5. **Review Changes:** Review skill files before execution to ensure they don't contain malicious code
+
+### Reporting Security Issues
+
+To report a security vulnerability, please follow the process in
+[SECURITY.md](SECURITY.md). **Do not open a public issue** — use GitHub's
+private vulnerability reporting (Security tab) or email the maintainer at
+[ismael.marin@gmail.com](mailto:ismael.marin@gmail.com).
+
+---
+
+## Troubleshooting
+
+### Common Issues and Solutions
+
+#### Configuration Issues
+
+**Problem:** "Config load failed, using mock provider"
+- **Solution:** Ensure your `skill-bench.json` file is properly formatted JSON and contains required fields
+- **Check:** Verify the file exists in your project root or home directory
+
+**Problem:** "API Key not set for [Provider]"
+- **Solution:** Set the appropriate environment variable (e.g., `SKILL_BENCH_OPENAI_API_KEY`) or add it to your `skill-bench.json`
+- **Check:** Run `env | grep SKILL_BENCH` to verify environment variables are set
+
+**Problem:** "No allowed commands configured"
+- **Solution:** Add `allowed_commands` array to your `skill-bench.json` with the commands you want to allow
+- **Check:** Ensure commands are in the allowlist and not in the dangerous commands list
+
+#### Execution Issues
+
+**Problem:** "Command execution timed out"
+- **Solution:** Increase `max_execution_time` in your `skill-bench.json` or simplify the task
+- **Check:** Verify the command isn't hanging or waiting for input
+
+**Problem:** "Command execution refused: no sandbox isolation is active and 'allow_host_execution' is not enabled"
+- **Cause:** Container isolation is not yet shipped, so commands would run on the host. SkillBench fails closed by default rather than run un-isolated.
+- **Solution:** Set `"allow_host_execution": true` in `skill-bench.json` to permit allowed commands to run directly on the host (inside the temporary sandbox directory). Enable it only when you accept that trade-off.
+
+**Problem:** "Context hydration failed"
+- **Solution:** Verify the source path exists and is a directory
+- **Check:** Ensure the path is within the base directory and file sizes are under limits
+
+#### Network Issues
+
+**Problem:** "Network Error: Connection refused"
+- **Solution:** Check your internet connection and API provider status
+- **Check:** Verify the base URL in your configuration is correct
+
+**Problem:** "API Request failed: 429"
+- **Solution:** This is a rate limit error. The system will retry automatically
+- **Check:** Reduce request frequency or check your API quota
+
+#### Test Failures
+
+**Problem:** Tests fail with "WebMock::NetConnectNotAllowedError"
+- **Solution:** This occurs when tests try to make real HTTP requests. Ensure test stubs are properly configured
+- **Check:** Verify WebMock is properly stubbing the expected URLs
+
+**Problem:** "E2E sibling repositories not present"
+- **Solution:** This is expected if you don't have the agent-mcp-runtime repository cloned
+- **Check:** These tests will be skipped and won't affect the overall test results
+
+### Debug Mode
+
+For detailed debugging, you can enable verbose logging:
+
+```bash
+# Set environment variable for verbose logging
+export SKILL_BENCH_DEBUG=true
+skill-bench run my-eval --skill=my-skill
+```
+
+### Getting Help
+
+If you encounter issues not covered here:
+
+1. Check the [GitHub Issues](https://github.com/igmarin/ruby-skill-bench/issues) for similar problems
+2. Create a new issue with detailed information about your environment and the problem
+3. Include Ruby version, SkillBench version, and error messages
+4. Provide steps to reproduce the issue
+
+---
 
 ## CI/CD Integration
 
-GitHub Actions workflow included (`.github/workflows/ci.yml`):
+### Batch Runs
+
+Run every eval at once instead of one at a time:
+
+```bash
+# Every eval under the default evals/ directory
+skill-bench run --all --skill=my-service
+
+# Or point at a specific directory
+skill-bench run --evals-dir path/to/evals --skill=my-service
+```
+
+A batch run exits `0` only when **every** eval passes and non-zero if any fail, so the process exit code is itself a CI gate. Two formats are built for batch consumption:
+
+- `--summary` emits an aggregate JSON gate — `passed` / `failed` / `total` counts, summed `tokens` and `cost`, and the `worst_delta` eval (the smallest context-minus-baseline delta in the batch). Archive it as a single machine-readable artifact:
+
+  ```bash
+  skill-bench run --all --skill=my-service --summary
+  ```
+
+- `--format junit` aggregates the batch into one JUnit document with **one `<testcase>` per eval** (a `<failure>` child for each failing eval), so test reporters show per-eval results:
+
+  ```bash
+  skill-bench run --all --skill=my-service --format junit > junit.xml
+  ```
+
+### GitHub Action
+
+Downstream repos can gate a skill change on every push or PR with the bundled composite action. Add a step that references `igmarin/ruby-skill-bench@v1`:
+
+```yaml
+# .github/workflows/skill-bench.yml
+name: skill-bench
+on: [pull_request]
+
+jobs:
+  skill-bench:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: igmarin/ruby-skill-bench@v1
+        with:
+          evals-dir: evals          # directory scanned for evals (default: evals)
+          skill: skills/my-service  # skill applied to every eval (default: "")
+          format: junit             # human | json | junit | html (default: junit)
+          ruby-version: "3.3"       # Ruby for ruby/setup-ruby (default: 3.3)
+          args: --summary           # extra flags appended verbatim (e.g. --summary, --pack NAME)
+```
+
+The action installs the gem and runs `skill-bench run --all --evals-dir <evals-dir> --format <format>` (adding `--skill` when set and appending `args` verbatim). The run step's exit code is the gate. For a full copy-paste workflow template, see [`examples/ci/`](examples/ci/).
+
+> The gem's own repository CI (`.github/workflows/ci.yml`) runs the test suite — rubocop, reek, and minitest against Ruby 3.3 and 3.4, on push and pull requests — and is separate from the reusable action above.
 
-- Runs on push and pull requests
-- Tests against Ruby 3.3 and 3.4
-- Executes rubocop, reek, and minitest
-- Outputs JUnit XML for test reporting
+To preview the machine-readable output locally:
 
 ```bash
-# Run locally with CI output
 skill-bench run my-eval --skill=my-skill --format json
 ```
 

data/docs/architecture.md

@@ -172,9 +172,11 @@ project-root/
 │   └── my-first-eval/
 │       ├── task.md               # Agent prompt
 │       └── criteria.json         # Scoring rules
-└── .skill-bench-history.json     # Benchmark history (auto-generated)
+└── .skill-bench-trends.json      # Benchmark history (auto-generated)
 ```
 
+A `.skill-bench-trends.json.bak` file is created automatically as a backup of the trend file.
+
 ### Skill Discovery
 
 Skills are discovered recursively. These are all valid:

data/docs/first-eval-guide.md

@@ -268,7 +268,7 @@ Provider is read from `skill-bench.json` — no `--provider` flag needed.
 2. Agent runs **with** skill context → produces context output
 3. Judge scores both independently → per-dimension scores
 4. Engine computes deltas → applies pass/fail logic
-5. Result is recorded in `.skill-bench-history.json` for trend tracking
+5. Result is recorded in `.skill-bench-trends.json` for trend tracking
 
 **Run with multiple skills:**
 
@@ -346,7 +346,7 @@ Both skill contexts are concatenated. The judge evaluates whether the combined c
 | **BASELINE** | Score without skill (unaided performance). Think: "How well does the AI do on its own?" |
 | **CONTEXT** | Score with skill (aided performance). Think: "How well does the AI do when it reads my skill?" |
 | **DELTA** | Improvement = CONTEXT - BASELINE. Think: "How many points did my skill add?" |
-| **TREND** | Change since the *previous* run of this exact eval + skill. Stored in `.skill-bench-history.json`. |
+| **TREND** | Change since the *previous* run of this exact eval + skill. Stored in `.skill-bench-trends.json`. |
 | **VERDICT** | PASS only if CONTEXT >= threshold AND DELTA >= minimum_delta. Both must be true. |
 | **Iterations** | ReAct loop steps for each run: thought → tools → observation. Helps you understand *how* the agent worked. |
 | **What went well** | Dimensions scoring ≥ 80% of max, with judge reasoning. Strengths of your skill. |
@@ -417,10 +417,10 @@ Your first run probably will not pass. That is normal. Here is how to improve.
 
 ### Use the History File
 
-After each run, SkillBench appends to `.skill-bench-history.json`. You can read it to track progress:
+After each run, SkillBench appends to `.skill-bench-trends.json`. You can read it to track progress:
 
 ```bash
-cat .skill-bench-history.json | jq '.[-1]'
+cat .skill-bench-trends.json | jq '.[-1]'
 ```
 
 Look at the dimension with the **smallest delta**. That is where your skill is weakest. Open `SKILL.md` and add a concrete rule targeting that dimension.
@@ -467,7 +467,7 @@ Created by `skill-bench init`. Stores provider, API key, model, timeout, and all
 }
 ```
 
-### `.skill-bench-history.json` — Evaluation History (Auto-Generated)
+### `.skill-bench-trends.json` — Evaluation History (Auto-Generated)
 
 A JSON array recording every successful eval run. SkillBench writes it automatically. It stores timestamps, eval names, skill names, scores, and deltas. This powers the **TREND** line in your output.
 
@@ -487,9 +487,9 @@ A JSON array recording every successful eval run. SkillBench writes it automatic
 
 **Tip:** Commit this file to git if you want to share trend data with your team.
 
-### `.skill-bench-history.json.bak` — Backup (Auto-Generated)
+### `.skill-bench-trends.json.bak` — Backup (Auto-Generated)
 
-A safety copy of the history file. If the main file gets corrupted, SkillBench recovers from this backup automatically. You never need to touch it.
+A snapshot of the previous good version of the history file, copied just before each new write. If the main file gets corrupted, SkillBench recovers from this backup automatically. You never need to touch it.
 
 ---
 

data/docs/testing-guide.md

@@ -273,7 +273,7 @@ Both must be true. This prevents two failure modes:
 TREND: baseline ↑ (+2), context ↑ (+7)
 ```
 
-This compares the current run against the **previous run of the same eval + skill** (stored in `.skill-bench-history.json`).
+This compares the current run against the **previous run of the same eval + skill** (stored in `.skill-bench-trends.json`).
 
 - `↑` = improved since last run
 - `↓` = regressed since last run

data/lib/skill_bench/agent/react_agent/loop_runner.rb

@@ -16,6 +16,7 @@ module SkillBench
         def self.call(initial_prompt, max_iterations, config)
           messages = [{ role: 'user', content: initial_prompt }]
           iterations_log = []
+          total_usage = empty_usage
           step_count = 0
 
           while step_count < max_iterations
@@ -24,24 +25,27 @@ module SkillBench
             step_result = Step.call(messages, config)
             iteration = step_result[:iteration]
             iterations_log << attach_step_number(iteration, step_count) if iteration
+            total_usage = add_usage(total_usage, step_result[:usage])
 
             unless step_result[:continue]
               final_result = step_result[:result] || { success: false, response: { error: { message: 'Step returned no result' } } }
-              return merge_iterations(final_result, iterations_log)
+              return finalize(final_result, iterations_log, total_usage)
             end
 
             messages = step_result[:messages]
           end
 
-          merge_iterations(
+          finalize(
             { success: false, response: { error: { message: Agent::ReactAgent::MAX_ITERATIONS_REACHED } } },
-            iterations_log
+            iterations_log,
+            total_usage
           )
         rescue StandardError => e
           SkillBench::ErrorLogger.log_error(e, 'ReactAgent Error')
-          merge_iterations(
+          finalize(
             { success: false, response: { error: { message: e.message } } },
-            iterations_log
+            iterations_log,
+            total_usage
           )
         end
 
@@ -54,14 +58,45 @@ module SkillBench
           iteration.merge(step_number: step_count)
         end
 
-        # Merges the collected iterations into the result response.
+        # Merges the collected iterations and accumulated usage into the response.
         #
         # @param result [Hash] The final result hash from the loop.
         # @param iterations_log [Array<Hash>] Collected iteration metadata.
-        # @return [Hash] The result with :iterations injected into :response.
-        def self.merge_iterations(result, iterations_log)
+        # @param total_usage [Hash] Summed token usage across all iterations.
+        # @return [Hash] The result with :iterations and :usage injected into :response.
+        def self.finalize(result, iterations_log, total_usage)
           response = result[:response] || {}
-          result.merge(response: response.merge(iterations: iterations_log))
+          result.merge(response: response.merge(iterations: iterations_log, usage: total_usage))
+        end
+
+        # A zeroed token-usage accumulator.
+        #
+        # @return [Hash] Usage hash with prompt/completion/total token counts set to zero.
+        def self.empty_usage
+          { prompt_tokens: 0, completion_tokens: 0, total_tokens: 0 }
+        end
+
+        # Adds a single step's usage onto a running total.
+        #
+        # @param total [Hash] The running usage total.
+        # @param usage [Hash, nil] A step's usage hash (may be nil or empty).
+        # @return [Hash] A new summed usage hash.
+        def self.add_usage(total, usage)
+          usage ||= {}
+          {
+            prompt_tokens: total[:prompt_tokens] + token_count(usage, :prompt_tokens),
+            completion_tokens: total[:completion_tokens] + token_count(usage, :completion_tokens),
+            total_tokens: total[:total_tokens] + token_count(usage, :total_tokens)
+          }
+        end
+
+        # Reads a token count from a usage hash, tolerating string keys.
+        #
+        # @param usage [Hash] The usage hash.
+        # @param key [Symbol] The usage key (e.g. :prompt_tokens).
+        # @return [Integer] The token count, or zero when absent.
+        def self.token_count(usage, key)
+          (usage[key] || usage[key.to_s] || 0).to_i
         end
       end
     end