ruby-skill-bench 1.1.0 → 1.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d2ad524e13bc006a56f0197d07b3ba7b0ce2f99f60b61f0739c3d5bc0d75a687
-  data.tar.gz: a920c473148b52584653acbb1e91cb3973791c09de6c4df994a77c097eabc476
+  metadata.gz: 60f6da81bf101f4bb1e5a7b86112aa96fa430ca9a1db7f1331a47e51b194ea73
+  data.tar.gz: 7f50c1e8a1891a93f0028f98c6a660e1d35ecfaf2104390d94a895ff983d9c93
 SHA512:
-  metadata.gz: c1f131af9bcde90e7fc3a7e6bef7f3770edfa4e2826ee19c3aabf5c210d6d3b6e5bdd460778a87f6fdc77b5b99bc17b2225e1b79de31674ad4acfe1bbc89f862
-  data.tar.gz: d8e3791c91242b25779afa3a21c57daeb06995bc4c65b01ca6f378a69491aeec95c5844ea541e5dcaf18b46d8e7f153ffb38ff2bb060ab5dacd653c8c1026bcd
+  metadata.gz: a828560ad084e9bd1d3336c9002c8b17d47a12955bed6d53e3dd174ef0e971dab24699ab5da40d3445ece5d06afa29e9354e9d033e5d8a11b5c749872487c1cd
+  data.tar.gz: b81d9dc73aa78be923e8fdecbf789aa5078e1f68c40d9330a0e11ca9632579dc8635e647a3f08730e1a478f3ea16a0e5379fae1b008b461f1f48d6d4714a9e3e

data/README.md

@@ -2,9 +2,10 @@
 
 ![Ruby Skill Bench Logo](https://github.com/user-attachments/assets/056d7ca4-8671-41ec-9efb-e323b73fb135)
 
-
 ![CodeRabbit Pull Request Reviews](https://img.shields.io/coderabbit/prs/github/igmarin/ruby-skill-bench?utm_source=oss&utm_medium=github&utm_campaign=igmarin%2Fruby-skill-bench&labelColor=171717&color=FF570A&link=https%3A%2F%2Fcoderabbit.ai&label=CodeRabbit+Reviews)
 
+[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/igmarin/ruby-skill-bench)
+
 *A high-fidelity evaluation engine for benchmarking AI agent skills across any stack (Rails-first, but extensible).*
 
 ## Part of the AI Skill Ecosystem
@@ -30,7 +31,7 @@ See the [Ecosystem Overview](https://github.com/igmarin/agent-mcp-runtime/blob/m
 - **Isolated Git Sandboxes**: Every run operates in a temporary repo. Clean diffs, zero side-effects, 100% reproducibility.
 - **Blind Judging with Dimensions**: LLM judge scores baseline and context independently across 5 canonical dimensions (Correctness, Skill Adherence, Code Quality, Test Coverage, Documentation). Eval authors configure weights and thresholds via `criteria.json`.
 - **Sophisticated ReAct Loop**: Employs a robust `Thought → Tool → Observation` loop to handle complex, multi-step engineering tasks.
-- **Multi-Provider Ecosystem**: Native support for **OpenAI**, **Anthropic**, **Google Gemini**, **Azure OpenAI**, **Ollama**, **Groq**, **DeepSeek**, and **OpenCode**.
+- **Multi-Provider Ecosystem**: Native support for **OpenAI**, **Anthropic**, **Google Gemini**, **Azure OpenAI**, **Ollama**, **Groq**, **DeepSeek**, **Mistral**, and **OpenCode**.
 - **Standardized Intelligence**: Consistent reporting format regardless of the underlying LLM provider.
 
 ---
@@ -64,11 +65,14 @@ CLI / API → RunnerService → Sandbox + ReAct Agent → LLM Client Layer → P
 | **Ollama** | — | `:ollama` |
 | **Groq** | `SKILL_BENCH_GROQ_API_KEY` | `:groq` |
 | **DeepSeek** | `SKILL_BENCH_DEEPSEEK_API_KEY` | `:deepseek` |
+| **Mistral** | `SKILL_BENCH_MISTRAL_API_KEY` | `:mistral` |
 | **OpenCode** | `SKILL_BENCH_OPENCODE_API_KEY`, `SKILL_BENCH_OPENCODE_BASE_URL` | `:opencode` |
 
 > **Note:** Environment variables are loaded automatically. You can also configure provider settings in `skill-bench.json` (created by `skill-bench init`).
 >
 > **OpenCode requires a custom `base_url`:** OpenCode does not host a public LLM API. You must provide your own OpenAI-compatible endpoint (e.g. a LiteLLM proxy, self-hosted vLLM, or company gateway) via the `base_url` config key. Without it, the provider will fail with "Base URL not set for Opencode".
+>
+> **Mistral** uses Mistral's OpenAI-compatible chat completions API (default model `mistral-large-latest`). Set `SKILL_BENCH_MISTRAL_API_KEY` and scaffold it with `skill-bench init --mistral`.
 
 ### Command Allowlist
 
@@ -79,6 +83,7 @@ By default, no shell commands are permitted. You must configure `allowed_command
   "provider": "openai",
   "max_execution_time": 30,
   "allowed_commands": ["rspec", "bundle", "ruby", "git"],
+  "allow_host_execution": false,
   "config": {
     "api_key": null,
     "model": "gpt-4o"
@@ -87,6 +92,8 @@ By default, no shell commands are permitted. You must configure `allowed_command
 ```
 
 > **Security:** The agent can only execute commands on this list. Dangerous commands (bash, curl, sudo, etc.) are always blocked regardless of configuration.
+>
+> **Where commands run:** Allowed commands run inside a temporary git **sandbox directory** (a copy of your eval files). When Docker is available and the packaged `evaluator-sandbox` image can start, commands run via `docker exec` with hardened flags (`--network none`, non-root, dropped capabilities). When Docker is **not** available, host execution **fails closed**: it is disabled by default and must be explicitly enabled with `"allow_host_execution": true`. With it disabled (the default), `run_command` refuses to execute on the host. Enable it only when you accept that allowed commands run directly on your machine inside the temp sandbox directory. See [docs/docker.md](docs/docker.md).
 
 ### Configuration Hierarchy
 
@@ -137,7 +144,9 @@ skill-bench init --openai
 }
 ```
 
-**Available providers:** `--openai`, `--anthropic`, `--gemini`, `--ollama`, `--azure`, `--groq`, `--deepseek`, `--opencode`
+**Available providers:** `--openai`, `--anthropic`, `--gemini`, `--ollama`, `--azure`, `--groq`, `--deepseek`, `--mistral`, `--opencode`
+
+**Zero-config offline path:** `skill-bench init --mock` scaffolds a minimal offline config that needs no API key and no network — `{"provider":"mock","max_execution_time":30}`. Use it to try the full flow (and run the bundled examples) before wiring up a real provider.
 
 Use `--force` to overwrite an existing config.
 
@@ -338,7 +347,7 @@ skill-bench run my-first-eval --skill=my-service
 3. **Context run** — Agent receives `task.md` + `SKILL.md` as prompt → produces output B
 4. **Blind judging** — LLM judge scores output A and output B independently across the dimensions defined in `criteria.json`
 5. **Delta computation** — Compare scores, compute deltas, apply pass/fail logic
-6. **History recording** — Store result in `.skill-bench-history.json` for trend tracking
+6. **History recording** — Store result in `.skill-bench-trends.json` for trend tracking
 
 Provider is read from `skill-bench.json` — no `--provider` flag needed.
 
@@ -350,11 +359,54 @@ skill-bench run my-first-eval --skill=skill-a --skill=skill-b
 
 Both skill contexts are concatenated and sent to the agent. The judge evaluates whether the combined context improves results.
 
-**Output Formats:**
+**Output Formats:** `--format human` (default), `json`, `junit`, or `html`.
+
+- Human-readable (default) — full delta table, iteration timeline, and a `Tokens: N | Est. Cost: $X.XXXX` line.
+- JSON: `--format json` — machine-readable, including top-level `tokens` and `cost` fields.
+- JUnit XML: `--format junit` — for CI test reporting.
+- HTML: `--format html` — a self-contained, shareable report (styles inlined, no external assets) with the delta table and iteration timeline. Redirect it to a file:
 
-- Human-readable (default)
-- JSON: `--format json`
-- JUnit XML: `--format junit`
+  ```bash
+  skill-bench run my-first-eval --skill=my-service --format html > report.html
+  ```
+
+---
+
+## Pre-flight Checks: `validate` / `doctor`
+
+Before spending tokens on a run, sanity-check your setup. `skill-bench validate` (aliased as `doctor`) runs read-only pre-flight checks — it never runs an eval and never makes a network call:
+
+```bash
+skill-bench validate
+# or, identically:
+skill-bench doctor
+```
+
+It runs three checks and prints a `PASS` / `FAIL` / `SKIP` line for each:
+
+1. **criteria** — validates the criteria JSON (default `criteria.json`, override with `--criteria PATH`). Skipped if the default file is absent.
+2. **config** — schema-checks `skill-bench.json` (default, override with `--config PATH`): `provider` is required and must be a known provider, `max_execution_time` must be a positive integer, and `config` (when present) must be an object.
+3. **provider key** — reports whether the configured provider's API key is present (the `mock` provider needs none).
+
+A passing report exits `0`:
+
+```text
+skill-bench validate
+
+[PASS] criteria      criteria.json is valid
+[PASS] config        skill-bench.json matches the expected shape
+[PASS] provider key  openai credentials present
+
+All checks passed.
+```
+
+A failure exits non-zero and names what is wrong:
+
+```text
+[FAIL] provider key  openai is missing: api_key
+
+1 check(s) failed.
+```
 
 ---
 
@@ -427,6 +479,22 @@ The `--variant` spec supports two forms:
 - `pack:<name>` — resolve via registry manifest
 - `/absolute/path` or `relative/path` — use a direct path
 
+### Response Caching (opt-in, `--cache`)
+
+LLM responses can be cached so identical requests reuse a previous result instead of calling the provider again. Caching is **off by default**. Enable it per run with `--cache`, or set the `SKILL_BENCH_CACHE` environment variable to a truthy value (`1`, `true`, `yes`, or `on`):
+
+```bash
+# Per-run flag
+skill-bench run my-first-eval --skill=my-service --cache
+
+# Or via the environment
+SKILL_BENCH_CACHE=1 skill-bench run my-first-eval --skill=my-service
+```
+
+The cache is in-memory (process-lifetime) and content-addressed: the key is a SHA-256 digest of the provider, model, system prompt, messages, tools, and temperature, so only truly identical requests share an entry. The `mock` and null providers are never cached.
+
+This pays off most with `compare`, which runs the skill-less baseline twice with identical inputs — with caching enabled, the repeated baseline reuses the cached response instead of making a second call.
+
 ---
 
 ## File Reference: What Lives on Disk
@@ -446,6 +514,7 @@ SkillBench creates and manages three files in your project. Understanding them h
   "provider": "openai",
   "max_execution_time": 300,
   "allowed_commands": ["rspec", "bundle", "ruby", "git"],
+  "allow_host_execution": false,
   "config": {
     "api_key": "sk-...",
     "model": "gpt-4o",
@@ -458,10 +527,11 @@ SkillBench creates and manages three files in your project. Understanding them h
 - Configuration is loaded in this order: **code defaults** → `~/.skill-bench.json` (user-wide) → `./skill-bench.json` (local) → **environment variables**. Later sources override earlier ones.
 - If `api_key` is `null`, SkillBench looks for the matching environment variable (e.g. `SKILL_BENCH_OPENAI_API_KEY`).
 - `allowed_commands` is a **safeguard**, not a convenience. By default the agent cannot run *any* shell command. Add only what your evals need.
+- `allow_host_execution` (default `false`) gates whether `run_command` may run on the host when no container isolation is active. When Docker starts a container successfully, host execution is not used. When Docker is unavailable, leaving this `false` means `run_command` **fails closed** (refuses to execute). Set it to `true` only if you accept that allowed commands run directly on your machine inside the temporary sandbox directory.
 
 ---
 
-### `.skill-bench-history.json` — Evaluation History (Auto-Generated)
+### `.skill-bench-trends.json` — Evaluation History (Auto-Generated)
 
 **What it is:** A JSON array that records every successful eval run. SkillBench appends to it automatically. It stores the timestamp, eval name, skill names, scores, and deltas so you can track improvement over time.
 
@@ -497,13 +567,13 @@ TREND: baseline ↑ (+2), context ↑ (+7)
 
 The trend compares the current run against the *previous run of the same eval + skill*. This tells you at a glance whether your latest skill edit made things better or worse.
 
-**Pro tip:** Commit `.skill-bench-history.json` to git if you want to share trend data with your team. Add it to `.gitignore` if you prefer to keep scores private.
+**Pro tip:** `.skill-bench-trends.json` is git-ignored by default (via the `.skill-bench-trends.json*` line in `.gitignore`). If you want to share trend data with your team, remove that line so the file can be committed.
 
 ---
 
-### `.skill-bench-history.json.bak` — Backup (Auto-Generated)
+### `.skill-bench-trends.json.bak` — Backup (Auto-Generated)
 
-**What it is:** A copy of `.skill-bench-history.json` created every time SkillBench writes a new entry. If the main file gets corrupted (e.g. you kill the process mid-write), SkillBench automatically falls back to the `.bak` file.
+**What it is:** A snapshot of the *previous* good version of `.skill-bench-trends.json`, copied just before each new write. (The first run has no prior version yet, so it creates no `.bak`.) If the main file gets corrupted (e.g. you kill the process mid-write), SkillBench automatically falls back to the `.bak` file.
 
 **Who edits it:** Nobody. It is a safety net.
 
@@ -541,7 +611,7 @@ Read the output carefully. Look at **two things:**
 ### Step 3: Inspect the History
 
 ```bash
-cat .skill-bench-history.json | jq '.[-1]'
+cat .skill-bench-trends.json | jq '.[-1]'
 ```
 
 This shows the latest entry. Focus on the dimension with the smallest delta — that is where your skill is weakest.
@@ -729,6 +799,7 @@ These 5 dimensions are **mandatory** in every `criteria.json`. You can add custo
   Eval: my-first-eval
   Skill: my-service
   Provider: openai
+  Tokens: 18432  |  Est. Cost: $0.0934
 ═══════════════════════════════════════════════════════
 
   === BASELINE ITERATIONS ===
@@ -774,8 +845,9 @@ These 5 dimensions are **mandatory** in every `criteria.json`. You can add custo
 - **CONTEXT:** The agent's score *with* the skill. This is the "aided" performance.
 - **DELTA:** `CONTEXT - BASELINE`. How much the skill helped.
 - **TOTAL:** Sum of all dimension scores. Max possible is 100.
-- **TREND:** Comparison against the previous run of the same eval + skill (from `.skill-bench-history.json`). Shows whether scores are improving over time.
+- **TREND:** Comparison against the previous run of the same eval + skill (from `.skill-bench-trends.json`). Shows whether scores are improving over time.
 - **VERDICT:** `PASS` only if `CONTEXT >= pass_threshold` AND `DELTA >= minimum_delta`.
+- **Tokens / Est. Cost:** The header shows total tokens used across the run and an estimated USD cost as `Tokens: N | Est. Cost: $X.XXXX`. The cost is approximate — it comes from a built-in per-model price table (`Services::CostCalculator`) and shows `—` when the model isn't in that table. JSON output (`--format json`) exposes the same data as top-level `tokens` and `cost` fields.
 
 **Iteration timeline:**
 
@@ -827,7 +899,7 @@ Your eval result depends on **both** conditions. Here is every scenario:
 
 ## Reliability & Security
 
-- **Safe-by-Design**: No code execution occurs on the host system; everything happens in the sandbox.
+- **Allowlist-Gated Execution**: The agent can only run commands you add to `allowed_commands`; with an empty allowlist it can run nothing. Commands prefer Docker isolation when available (see [docs/docker.md](docs/docker.md)); otherwise they would run on the host inside the temporary sandbox directory — so host execution is **disabled by default** and must be explicitly opted into with `allow_host_execution: true`.
 - **Command Blocklist**: Dangerous commands (`bash`, `sh`, `python`, `curl`, etc.) are always blocked, even if listed in `allowed_commands`.
 - **Path Validation**: Eval paths are validated to prevent directory traversal attacks.
 - **Atomic History Writes**: Benchmark history uses file locking to prevent corruption from concurrent writes.
@@ -836,7 +908,7 @@ Your eval result depends on **both** conditions. Here is every scenario:
 - **Traceability**: Every thought and tool call is logged with full backtrace for post-mortem analysis.
 - **Robust Error Recovery**: Handles provider outages and rate limits gracefully with standardized error logging.
 - **XML-Safe Output**: JUnit XML output is properly escaped to prevent injection attacks.
-- **Test Coverage**: 373+ tests covering core engine, CLI commands, and all provider clients.
+- **Test Coverage**: 700+ tests covering core engine, CLI commands, and all provider clients. Run `bundle exec rake test` to see the current count.
 
 ## Testing
 
@@ -855,9 +927,18 @@ bundle exec ruby -Itest test/integration_test.rb
 
 **Test Structure:**
 
-- `test/evaluator/` — Core evaluation engine tests
-- `test/agent_eval/` — CLI, models, and service tests
+- `test/agent/` — Agent runtime tests
+- `test/agent_eval/` — Agent evaluation tests
+- `test/cli/` — CLI command tests
 - `test/clients/` — Provider client tests
+- `test/evaluator/` — Core evaluation engine tests
+- `test/history_recorder/` — Benchmark history persistence tests
+- `test/models/` — Domain model tests
+- `test/registry/` — Skill/eval registry tests
+- `test/services/` — Service layer tests
+- `test/skills/` — Skill loading tests
+- `test/tools/` — Agent tool tests
+- Plus several top-level `test/*_test.rb` files (e.g. `integration_test.rb`, `evaluation_runner_test.rb`, `trend_tracker_test.rb`).
 
 ---
 
@@ -886,17 +967,24 @@ Ruby Skill Bench is designed with security as a primary concern. The system exec
 - **Command Allowlist:** Only explicitly allowed commands can be executed
 - **Dangerous Commands Blocklist:** Dangerous commands (bash, curl, sudo, etc.) are always blocked
 - **Shell Tokenization:** Commands are tokenized before execution to prevent shell injection
-- **Docker Isolation:** Commands can be executed in isolated Docker containers with hardened security settings
+- **Fail-Closed Host Execution:** When no container is active, commands would run on the host inside a temporary sandbox directory. `run_command` refuses to execute unless `allow_host_execution: true` is set; it is **disabled by default**. Prefer installing Docker and building the image (`bundle exec rake docker:build`) so isolation activates automatically.
+
+> **The allowlist is the only real authorization control — and it only checks the base command.** `run_command` authorizes by the first token of the command (`rake`, `find`, `git`, …); it does **not** inspect arguments. Shell tokenization stops metacharacter injection, but it does **not** sandbox what an allowlisted binary can do. Because many common tools are general-purpose execution wrappers, **allowlisting any one of them is equivalent to granting arbitrary host code execution** — for example `rake -e '...'`, `rspec -e`, `make` (arbitrary recipes), `find . -exec ...`, or `git` (hooks, `-c core.fsmonitor=...`, `! ...` aliases). Combined with the fail-closed model above (`run_command` refuses to run on the host unless `allow_host_execution` is explicitly enabled — see `HOST_EXECUTION_REFUSED` in `run_command.rb`), the practical guidance is: **keep `allowed_commands` as minimal as possible — empty for untrusted skills** — and treat every entry as if you were handing the skill a shell.
+>
+> An **optional, default-off** `command_argument_constraints` setting can refuse commands whose arguments contain configured substrings/flags (for example blocking `-e` or `-exec`). It is a defense-in-depth speed bump, **not** a sandbox, and is unset by default; the allowlist remains the control that matters.
 
 #### Docker Security Hardening
 
-When Docker is available, containers are launched with hardened security settings:
+> **Status:** Container isolation **ships** when a Docker daemon is available and the packaged build context can produce `evaluator-sandbox:<version>`. Build with `bundle exec rake docker:build` (see [docs/docker.md](docs/docker.md)). Without Docker, `run_command` remains fail-closed on the host unless `allow_host_execution` is set.
 
-- **Non-root User:** Containers run as a non-root user
+Containers launch with hardened security settings:
+
+- **Non-root User:** Containers run as the host `uid:gid`
 - **Privilege Prevention:** `--security-opt no-new-privileges` prevents privilege escalation
-- **Capability Dropping:** All Linux capabilities are dropped except minimal needed ones
+- **Capability Dropping:** `--cap-drop ALL` with only `CHOWN`/`DAC_OVERRIDE` restored for volume ops
 - **Network Isolation:** `--network none` disables network access
-- **Read-only Root:** Container filesystem is read-only (except for mounted volumes)
+- **Sandbox mount:** Host temp sandbox is bind-mounted at `/sandbox` read-write
+- **Read-only root (future):** Not yet applied; tracked as follow-up hardening
 
 #### Resource Limits
 
@@ -922,12 +1010,10 @@ When Docker is available, containers are launched with hardened security setting
 
 ### Reporting Security Issues
 
-If you discover a security vulnerability:
-
-1. **Do Not Open a Public Issue:** Send a private email to the maintainers
-2. **Provide Details:** Include steps to reproduce and potential impact
-3. **Allow Time for Fix:** Give maintainers time to address the issue before disclosure
-4. **Follow Responsible Disclosure:** Follow responsible disclosure practices
+To report a security vulnerability, please follow the process in
+[SECURITY.md](SECURITY.md). **Do not open a public issue** — use GitHub's
+private vulnerability reporting (Security tab) or email the maintainer at
+[ismael.marin@gmail.com](mailto:ismael.marin@gmail.com).
 
 ---
 
@@ -955,9 +1041,9 @@ If you discover a security vulnerability:
 - **Solution:** Increase `max_execution_time` in your `skill-bench.json` or simplify the task
 - **Check:** Verify the command isn't hanging or waiting for input
 
-**Problem:** "Docker container failed to start"
-- **Solution:** Ensure Docker is running and you have permissions to run Docker commands
-- **Check:** Run `docker info` to verify Docker daemon is accessible
+**Problem:** "Command execution refused: no sandbox isolation is active and 'allow_host_execution' is not enabled"
+- **Cause:** No Docker container was started (daemon missing, image missing, or build failed), so commands would run on the host. SkillBench fails closed by default rather than run un-isolated.
+- **Solution:** Install Docker and run `bundle exec rake docker:build`, **or** set `"allow_host_execution": true` in `skill-bench.json` to permit allowed commands on the host (inside the temporary sandbox directory). Prefer Docker when possible.
 
 **Problem:** "Context hydration failed"
 - **Solution:** Verify the source path exists and is a directory
@@ -1006,15 +1092,62 @@ If you encounter issues not covered here:
 
 ## CI/CD Integration
 
-GitHub Actions workflow included (`.github/workflows/ci.yml`):
+### Batch Runs
+
+Run every eval at once instead of one at a time:
+
+```bash
+# Every eval under the default evals/ directory
+skill-bench run --all --skill=my-service
+
+# Or point at a specific directory
+skill-bench run --evals-dir path/to/evals --skill=my-service
+```
+
+A batch run exits `0` only when **every** eval passes and non-zero if any fail, so the process exit code is itself a CI gate. Two formats are built for batch consumption:
+
+- `--summary` emits an aggregate JSON gate — `passed` / `failed` / `total` counts, summed `tokens` and `cost`, and the `worst_delta` eval (the smallest context-minus-baseline delta in the batch). Archive it as a single machine-readable artifact:
+
+  ```bash
+  skill-bench run --all --skill=my-service --summary
+  ```
+
+- `--format junit` aggregates the batch into one JUnit document with **one `<testcase>` per eval** (a `<failure>` child for each failing eval), so test reporters show per-eval results:
+
+  ```bash
+  skill-bench run --all --skill=my-service --format junit > junit.xml
+  ```
+
+### GitHub Action
+
+Downstream repos can gate a skill change on every push or PR with the bundled composite action. Add a step that references `igmarin/ruby-skill-bench@v1`:
+
+```yaml
+# .github/workflows/skill-bench.yml
+name: skill-bench
+on: [pull_request]
+
+jobs:
+  skill-bench:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: igmarin/ruby-skill-bench@v1
+        with:
+          evals-dir: evals          # directory scanned for evals (default: evals)
+          skill: skills/my-service  # skill applied to every eval (default: "")
+          format: junit             # human | json | junit | html (default: junit)
+          ruby-version: "3.3"       # Ruby for ruby/setup-ruby (default: 3.3)
+          args: --summary           # extra flags appended verbatim (e.g. --summary, --pack NAME)
+```
+
+The action installs the gem and runs `skill-bench run --all --evals-dir <evals-dir> --format <format>` (adding `--skill` when set and appending `args` verbatim). The run step's exit code is the gate. For a full copy-paste workflow template, see [`examples/ci/`](examples/ci/).
+
+> The gem's own repository CI (`.github/workflows/ci.yml`) runs the test suite — rubocop, reek, and minitest against Ruby 3.3 and 3.4, on push and pull requests — and is separate from the reusable action above.
 
-- Runs on push and pull requests
-- Tests against Ruby 3.3 and 3.4
-- Executes rubocop, reek, and minitest
-- Outputs JUnit XML for test reporting
+To preview the machine-readable output locally:
 
 ```bash
-# Run locally with CI output
 skill-bench run my-eval --skill=my-skill --format json
 ```
 

data/docs/architecture.md

@@ -172,9 +172,11 @@ project-root/
 │   └── my-first-eval/
 │       ├── task.md               # Agent prompt
 │       └── criteria.json         # Scoring rules
-└── .skill-bench-history.json     # Benchmark history (auto-generated)
+└── .skill-bench-trends.json      # Benchmark history (auto-generated)
 ```
 
+A `.skill-bench-trends.json.bak` file is created automatically as a backup of the trend file.
+
 ### Skill Discovery
 
 Skills are discovered recursively. These are all valid:
@@ -198,3 +200,10 @@ The eval directory must contain at minimum:
 
 - `task.md` — the agent prompt
 - `criteria.json` — the scoring rules (optional; defaults to empty criteria if missing)
+
+## Command isolation
+
+Agent tool commands prefer a Docker container (`evaluator-sandbox`) when available.
+See [docker.md](docker.md) for the image contract and `rake docker:build`.
+Without Docker, `run_command` refuses host execution unless `allow_host_execution` is true.
+

data/docs/docker.md

@@ -0,0 +1,45 @@
+# Evaluator sandbox Docker image
+
+Part of milestone **container-isolation-v1**. This document is the image contract
+for `evaluator-sandbox`, the container SkillBench uses for isolated `run_command`
+execution when a Docker daemon is available.
+
+## Contract (MVP)
+
+| Item | Value |
+|------|--------|
+| Image name | `evaluator-sandbox` (`Constants::Sandbox::DOCKER_IMAGE_NAME`) |
+| Tags | `evaluator-sandbox:<SkillBench::VERSION>` and `evaluator-sandbox:latest` |
+| Base | `ruby:3.4-bookworm` (aligns with CI Ruby 3.4; 3.3 hosts still run evals via host Ruby for the orchestrator) |
+| Preinstalled | Ruby (from base), `git`, minimal Debian tools from base image |
+| Working directory | `/sandbox` (host sandbox dir is bind-mounted here read-write) |
+| Default process | `sleep infinity` — container stays up for `docker exec` |
+| Network at runtime | **none** (`--network none`) — no `apt`/`bundle install` during evals |
+| User | Host `uid:gid` via `docker run --user` |
+| Privileges | `no-new-privileges`, `cap-drop ALL`, add back `CHOWN` + `DAC_OVERRIDE` only |
+
+## Build
+
+From the repo root (or any install that includes the packaged context):
+
+```bash
+bundle exec rake docker:build
+```
+
+This builds from `lib/skill_bench/execution/docker` and tags:
+
+- `evaluator-sandbox:<version>` (from `SkillBench::VERSION`)
+- `evaluator-sandbox:latest`
+
+## Multi-arch notes
+
+Build on the machine (or CI runner) that will run evals. Docker Desktop on Apple
+Silicon produces arm64 images; GitHub `ubuntu-latest` produces amd64. Do not
+assume a single cached image works across both without multi-platform build.
+
+## Out of scope (MVP)
+
+- Full Bundler/dev toolchain prebake
+- Per-eval custom Dockerfiles
+- Read-only root filesystem (documented as future hardening)
+- gVisor / Firecracker

data/docs/first-eval-guide.md

@@ -268,7 +268,7 @@ Provider is read from `skill-bench.json` — no `--provider` flag needed.
 2. Agent runs **with** skill context → produces context output
 3. Judge scores both independently → per-dimension scores
 4. Engine computes deltas → applies pass/fail logic
-5. Result is recorded in `.skill-bench-history.json` for trend tracking
+5. Result is recorded in `.skill-bench-trends.json` for trend tracking
 
 **Run with multiple skills:**
 
@@ -346,7 +346,7 @@ Both skill contexts are concatenated. The judge evaluates whether the combined c
 | **BASELINE** | Score without skill (unaided performance). Think: "How well does the AI do on its own?" |
 | **CONTEXT** | Score with skill (aided performance). Think: "How well does the AI do when it reads my skill?" |
 | **DELTA** | Improvement = CONTEXT - BASELINE. Think: "How many points did my skill add?" |
-| **TREND** | Change since the *previous* run of this exact eval + skill. Stored in `.skill-bench-history.json`. |
+| **TREND** | Change since the *previous* run of this exact eval + skill. Stored in `.skill-bench-trends.json`. |
 | **VERDICT** | PASS only if CONTEXT >= threshold AND DELTA >= minimum_delta. Both must be true. |
 | **Iterations** | ReAct loop steps for each run: thought → tools → observation. Helps you understand *how* the agent worked. |
 | **What went well** | Dimensions scoring ≥ 80% of max, with judge reasoning. Strengths of your skill. |
@@ -417,10 +417,10 @@ Your first run probably will not pass. That is normal. Here is how to improve.
 
 ### Use the History File
 
-After each run, SkillBench appends to `.skill-bench-history.json`. You can read it to track progress:
+After each run, SkillBench appends to `.skill-bench-trends.json`. You can read it to track progress:
 
 ```bash
-cat .skill-bench-history.json | jq '.[-1]'
+cat .skill-bench-trends.json | jq '.[-1]'
 ```
 
 Look at the dimension with the **smallest delta**. That is where your skill is weakest. Open `SKILL.md` and add a concrete rule targeting that dimension.
@@ -467,7 +467,7 @@ Created by `skill-bench init`. Stores provider, API key, model, timeout, and all
 }
 ```
 
-### `.skill-bench-history.json` — Evaluation History (Auto-Generated)
+### `.skill-bench-trends.json` — Evaluation History (Auto-Generated)
 
 A JSON array recording every successful eval run. SkillBench writes it automatically. It stores timestamps, eval names, skill names, scores, and deltas. This powers the **TREND** line in your output.
 
@@ -487,9 +487,9 @@ A JSON array recording every successful eval run. SkillBench writes it automatic
 
 **Tip:** Commit this file to git if you want to share trend data with your team.
 
-### `.skill-bench-history.json.bak` — Backup (Auto-Generated)
+### `.skill-bench-trends.json.bak` — Backup (Auto-Generated)
 
-A safety copy of the history file. If the main file gets corrupted, SkillBench recovers from this backup automatically. You never need to touch it.
+A snapshot of the previous good version of the history file, copied just before each new write. If the main file gets corrupted, SkillBench recovers from this backup automatically. You never need to touch it.
 
 ---
 

data/docs/testing-guide.md

@@ -273,7 +273,7 @@ Both must be true. This prevents two failure modes:
 TREND: baseline ↑ (+2), context ↑ (+7)
 ```
 
-This compares the current run against the **previous run of the same eval + skill** (stored in `.skill-bench-history.json`).
+This compares the current run against the **previous run of the same eval + skill** (stored in `.skill-bench-trends.json`).
 
 - `↑` = improved since last run
 - `↓` = regressed since last run

data/lib/skill_bench/agent/react_agent/loop_runner.rb

@@ -16,6 +16,7 @@ module SkillBench
         def self.call(initial_prompt, max_iterations, config)
           messages = [{ role: 'user', content: initial_prompt }]
           iterations_log = []
+          total_usage = empty_usage
           step_count = 0
 
           while step_count < max_iterations
@@ -24,24 +25,27 @@ module SkillBench
             step_result = Step.call(messages, config)
             iteration = step_result[:iteration]
             iterations_log << attach_step_number(iteration, step_count) if iteration
+            total_usage = add_usage(total_usage, step_result[:usage])
 
             unless step_result[:continue]
               final_result = step_result[:result] || { success: false, response: { error: { message: 'Step returned no result' } } }
-              return merge_iterations(final_result, iterations_log)
+              return finalize(final_result, iterations_log, total_usage)
             end
 
             messages = step_result[:messages]
           end
 
-          merge_iterations(
+          finalize(
             { success: false, response: { error: { message: Agent::ReactAgent::MAX_ITERATIONS_REACHED } } },
-            iterations_log
+            iterations_log,
+            total_usage
           )
         rescue StandardError => e
           SkillBench::ErrorLogger.log_error(e, 'ReactAgent Error')
-          merge_iterations(
+          finalize(
             { success: false, response: { error: { message: e.message } } },
-            iterations_log
+            iterations_log,
+            total_usage
           )
         end
 
@@ -54,14 +58,45 @@ module SkillBench
           iteration.merge(step_number: step_count)
         end
 
-        # Merges the collected iterations into the result response.
+        # Merges the collected iterations and accumulated usage into the response.
         #
         # @param result [Hash] The final result hash from the loop.
         # @param iterations_log [Array<Hash>] Collected iteration metadata.
-        # @return [Hash] The result with :iterations injected into :response.
-        def self.merge_iterations(result, iterations_log)
+        # @param total_usage [Hash] Summed token usage across all iterations.
+        # @return [Hash] The result with :iterations and :usage injected into :response.
+        def self.finalize(result, iterations_log, total_usage)
           response = result[:response] || {}
-          result.merge(response: response.merge(iterations: iterations_log))
+          result.merge(response: response.merge(iterations: iterations_log, usage: total_usage))
+        end
+
+        # A zeroed token-usage accumulator.
+        #
+        # @return [Hash] Usage hash with prompt/completion/total token counts set to zero.
+        def self.empty_usage
+          { prompt_tokens: 0, completion_tokens: 0, total_tokens: 0 }
+        end
+
+        # Adds a single step's usage onto a running total.
+        #
+        # @param total [Hash] The running usage total.
+        # @param usage [Hash, nil] A step's usage hash (may be nil or empty).
+        # @return [Hash] A new summed usage hash.
+        def self.add_usage(total, usage)
+          usage ||= {}
+          {
+            prompt_tokens: total[:prompt_tokens] + token_count(usage, :prompt_tokens),
+            completion_tokens: total[:completion_tokens] + token_count(usage, :completion_tokens),
+            total_tokens: total[:total_tokens] + token_count(usage, :total_tokens)
+          }
+        end
+
+        # Reads a token count from a usage hash, tolerating string keys.
+        #
+        # @param usage [Hash] The usage hash.
+        # @param key [Symbol] The usage key (e.g. :prompt_tokens).
+        # @return [Integer] The token count, or zero when absent.
+        def self.token_count(usage, key)
+          (usage[key] || usage[key.to_s] || 0).to_i
         end
       end
     end