ruby-skill-bench 1.2.0 → 1.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f47976b55f6f8c147adb4ed784ce04ba52ff71f805f8e35d797ba776021641c4
-  data.tar.gz: c2febaadbdeb7e149041661258ce84e41499121445cf726cefece642e60174a4
+  metadata.gz: 60f6da81bf101f4bb1e5a7b86112aa96fa430ca9a1db7f1331a47e51b194ea73
+  data.tar.gz: 7f50c1e8a1891a93f0028f98c6a660e1d35ecfaf2104390d94a895ff983d9c93
 SHA512:
-  metadata.gz: 5ab3082fa715a0776455a88b28e2d990d3bb7e52fbc5f0cc47176e1d44e76cdc430531bc0401abab350cb4c2321d0b625ce78d381094fe404cdedd7e61b27227
-  data.tar.gz: 3d5f67b876457691e003e62a8ba57fc04ac21e002bb5c6a84d1ae7954cf7dd7136f05a658d7ee4d416ab2db19cb0ad90e51dd0d920df9ffa998cb8391db65df5
+  metadata.gz: a828560ad084e9bd1d3336c9002c8b17d47a12955bed6d53e3dd174ef0e971dab24699ab5da40d3445ece5d06afa29e9354e9d033e5d8a11b5c749872487c1cd
+  data.tar.gz: b81d9dc73aa78be923e8fdecbf789aa5078e1f68c40d9330a0e11ca9632579dc8635e647a3f08730e1a478f3ea16a0e5379fae1b008b461f1f48d6d4714a9e3e

data/README.md

@@ -2,9 +2,10 @@
 
 ![Ruby Skill Bench Logo](https://github.com/user-attachments/assets/056d7ca4-8671-41ec-9efb-e323b73fb135)
 
-
 ![CodeRabbit Pull Request Reviews](https://img.shields.io/coderabbit/prs/github/igmarin/ruby-skill-bench?utm_source=oss&utm_medium=github&utm_campaign=igmarin%2Fruby-skill-bench&labelColor=171717&color=FF570A&link=https%3A%2F%2Fcoderabbit.ai&label=CodeRabbit+Reviews)
 
+[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/igmarin/ruby-skill-bench)
+
 *A high-fidelity evaluation engine for benchmarking AI agent skills across any stack (Rails-first, but extensible).*
 
 ## Part of the AI Skill Ecosystem
@@ -92,7 +93,7 @@ By default, no shell commands are permitted. You must configure `allowed_command
 
 > **Security:** The agent can only execute commands on this list. Dangerous commands (bash, curl, sudo, etc.) are always blocked regardless of configuration.
 >
-> **Where commands run:** Allowed commands run inside a temporary git **sandbox directory** on the host — a copy of your eval files, not your project. True container isolation (Docker) is **not yet shipped**, so the sandbox directory is the only boundary. Because of this, host execution **fails closed**: it is disabled by default and must be explicitly enabled with `"allow_host_execution": true`. With it disabled (the default), `run_command` refuses to execute and returns an error instead of running un-isolated. Enable it only when you accept that allowed commands run directly on your machine.
+> **Where commands run:** Allowed commands run inside a temporary git **sandbox directory** (a copy of your eval files). When Docker is available and the packaged `evaluator-sandbox` image can start, commands run via `docker exec` with hardened flags (`--network none`, non-root, dropped capabilities). When Docker is **not** available, host execution **fails closed**: it is disabled by default and must be explicitly enabled with `"allow_host_execution": true`. With it disabled (the default), `run_command` refuses to execute on the host. Enable it only when you accept that allowed commands run directly on your machine inside the temp sandbox directory. See [docs/docker.md](docs/docker.md).
 
 ### Configuration Hierarchy
 
@@ -526,7 +527,7 @@ SkillBench creates and manages three files in your project. Understanding them h
 - Configuration is loaded in this order: **code defaults** → `~/.skill-bench.json` (user-wide) → `./skill-bench.json` (local) → **environment variables**. Later sources override earlier ones.
 - If `api_key` is `null`, SkillBench looks for the matching environment variable (e.g. `SKILL_BENCH_OPENAI_API_KEY`).
 - `allowed_commands` is a **safeguard**, not a convenience. By default the agent cannot run *any* shell command. Add only what your evals need.
-- `allow_host_execution` (default `false`) gates whether `run_command` may run on the host when no container isolation is active. Since container isolation is not yet shipped, leaving it `false` means `run_command` **fails closed** (refuses to execute). Set it to `true` only if you accept that allowed commands run directly on your machine inside the temporary sandbox directory.
+- `allow_host_execution` (default `false`) gates whether `run_command` may run on the host when no container isolation is active. When Docker starts a container successfully, host execution is not used. When Docker is unavailable, leaving this `false` means `run_command` **fails closed** (refuses to execute). Set it to `true` only if you accept that allowed commands run directly on your machine inside the temporary sandbox directory.
 
 ---
 
@@ -898,7 +899,7 @@ Your eval result depends on **both** conditions. Here is every scenario:
 
 ## Reliability & Security
 
-- **Allowlist-Gated Execution**: The agent can only run commands you add to `allowed_commands`; with an empty allowlist it can run nothing. Commands run inside a temporary git sandbox **directory** (a copy of the eval files) on the host — container isolation is not yet shipped, so host execution is **disabled by default** and must be explicitly opted into with `allow_host_execution: true`.
+- **Allowlist-Gated Execution**: The agent can only run commands you add to `allowed_commands`; with an empty allowlist it can run nothing. Commands prefer Docker isolation when available (see [docs/docker.md](docs/docker.md)); otherwise they would run on the host inside the temporary sandbox directory — so host execution is **disabled by default** and must be explicitly opted into with `allow_host_execution: true`.
 - **Command Blocklist**: Dangerous commands (`bash`, `sh`, `python`, `curl`, etc.) are always blocked, even if listed in `allowed_commands`.
 - **Path Validation**: Eval paths are validated to prevent directory traversal attacks.
 - **Atomic History Writes**: Benchmark history uses file locking to prevent corruption from concurrent writes.
@@ -966,23 +967,24 @@ Ruby Skill Bench is designed with security as a primary concern. The system exec
 - **Command Allowlist:** Only explicitly allowed commands can be executed
 - **Dangerous Commands Blocklist:** Dangerous commands (bash, curl, sudo, etc.) are always blocked
 - **Shell Tokenization:** Commands are tokenized before execution to prevent shell injection
-- **Fail-Closed Host Execution:** Container isolation is not yet active, so commands run on the host inside a temporary sandbox directory. To match this reality, `run_command` refuses to execute unless `allow_host_execution: true` is set; it is **disabled by default**.
+- **Fail-Closed Host Execution:** When no container is active, commands would run on the host inside a temporary sandbox directory. `run_command` refuses to execute unless `allow_host_execution: true` is set; it is **disabled by default**. Prefer installing Docker and building the image (`bundle exec rake docker:build`) so isolation activates automatically.
 
 > **The allowlist is the only real authorization control — and it only checks the base command.** `run_command` authorizes by the first token of the command (`rake`, `find`, `git`, …); it does **not** inspect arguments. Shell tokenization stops metacharacter injection, but it does **not** sandbox what an allowlisted binary can do. Because many common tools are general-purpose execution wrappers, **allowlisting any one of them is equivalent to granting arbitrary host code execution** — for example `rake -e '...'`, `rspec -e`, `make` (arbitrary recipes), `find . -exec ...`, or `git` (hooks, `-c core.fsmonitor=...`, `! ...` aliases). Combined with the fail-closed model above (`run_command` refuses to run on the host unless `allow_host_execution` is explicitly enabled — see `HOST_EXECUTION_REFUSED` in `run_command.rb`), the practical guidance is: **keep `allowed_commands` as minimal as possible — empty for untrusted skills** — and treat every entry as if you were handing the skill a shell.
 >
 > An **optional, default-off** `command_argument_constraints` setting can refuse commands whose arguments contain configured substrings/flags (for example blocking `-e` or `-exec`). It is a defense-in-depth speed bump, **not** a sandbox, and is unset by default; the allowlist remains the control that matters.
 
-#### Docker Security Hardening (Planned — Not Yet Active)
+#### Docker Security Hardening
 
-> **Status:** The container isolation model described below is **planned, not shipped**. No Docker build context is packaged, so containers are never launched today — `run_command` runs on the host gated by the allowlist and `allow_host_execution`. The settings below document the intended hardened model for when container isolation lands.
+> **Status:** Container isolation **ships** when a Docker daemon is available and the packaged build context can produce `evaluator-sandbox:<version>`. Build with `bundle exec rake docker:build` (see [docs/docker.md](docs/docker.md)). Without Docker, `run_command` remains fail-closed on the host unless `allow_host_execution` is set.
 
-When container isolation is enabled in a future release, containers are intended to launch with hardened security settings:
+Containers launch with hardened security settings:
 
-- **Non-root User:** Containers run as a non-root user
+- **Non-root User:** Containers run as the host `uid:gid`
 - **Privilege Prevention:** `--security-opt no-new-privileges` prevents privilege escalation
-- **Capability Dropping:** All Linux capabilities are dropped except minimal needed ones
+- **Capability Dropping:** `--cap-drop ALL` with only `CHOWN`/`DAC_OVERRIDE` restored for volume ops
 - **Network Isolation:** `--network none` disables network access
-- **Read-only Root:** Container filesystem is read-only (except for mounted volumes)
+- **Sandbox mount:** Host temp sandbox is bind-mounted at `/sandbox` read-write
+- **Read-only root (future):** Not yet applied; tracked as follow-up hardening
 
 #### Resource Limits
 
@@ -1040,8 +1042,8 @@ private vulnerability reporting (Security tab) or email the maintainer at
 - **Check:** Verify the command isn't hanging or waiting for input
 
 **Problem:** "Command execution refused: no sandbox isolation is active and 'allow_host_execution' is not enabled"
-- **Cause:** Container isolation is not yet shipped, so commands would run on the host. SkillBench fails closed by default rather than run un-isolated.
-- **Solution:** Set `"allow_host_execution": true` in `skill-bench.json` to permit allowed commands to run directly on the host (inside the temporary sandbox directory). Enable it only when you accept that trade-off.
+- **Cause:** No Docker container was started (daemon missing, image missing, or build failed), so commands would run on the host. SkillBench fails closed by default rather than run un-isolated.
+- **Solution:** Install Docker and run `bundle exec rake docker:build`, **or** set `"allow_host_execution": true` in `skill-bench.json` to permit allowed commands on the host (inside the temporary sandbox directory). Prefer Docker when possible.
 
 **Problem:** "Context hydration failed"
 - **Solution:** Verify the source path exists and is a directory

data/docs/architecture.md

@@ -200,3 +200,10 @@ The eval directory must contain at minimum:
 
 - `task.md` — the agent prompt
 - `criteria.json` — the scoring rules (optional; defaults to empty criteria if missing)
+
+## Command isolation
+
+Agent tool commands prefer a Docker container (`evaluator-sandbox`) when available.
+See [docker.md](docker.md) for the image contract and `rake docker:build`.
+Without Docker, `run_command` refuses host execution unless `allow_host_execution` is true.
+

data/docs/docker.md

@@ -0,0 +1,45 @@
+# Evaluator sandbox Docker image
+
+Part of milestone **container-isolation-v1**. This document is the image contract
+for `evaluator-sandbox`, the container SkillBench uses for isolated `run_command`
+execution when a Docker daemon is available.
+
+## Contract (MVP)
+
+| Item | Value |
+|------|--------|
+| Image name | `evaluator-sandbox` (`Constants::Sandbox::DOCKER_IMAGE_NAME`) |
+| Tags | `evaluator-sandbox:<SkillBench::VERSION>` and `evaluator-sandbox:latest` |
+| Base | `ruby:3.4-bookworm` (aligns with CI Ruby 3.4; 3.3 hosts still run evals via host Ruby for the orchestrator) |
+| Preinstalled | Ruby (from base), `git`, minimal Debian tools from base image |
+| Working directory | `/sandbox` (host sandbox dir is bind-mounted here read-write) |
+| Default process | `sleep infinity` — container stays up for `docker exec` |
+| Network at runtime | **none** (`--network none`) — no `apt`/`bundle install` during evals |
+| User | Host `uid:gid` via `docker run --user` |
+| Privileges | `no-new-privileges`, `cap-drop ALL`, add back `CHOWN` + `DAC_OVERRIDE` only |
+
+## Build
+
+From the repo root (or any install that includes the packaged context):
+
+```bash
+bundle exec rake docker:build
+```
+
+This builds from `lib/skill_bench/execution/docker` and tags:
+
+- `evaluator-sandbox:<version>` (from `SkillBench::VERSION`)
+- `evaluator-sandbox:latest`
+
+## Multi-arch notes
+
+Build on the machine (or CI runner) that will run evals. Docker Desktop on Apple
+Silicon produces arm64 images; GitHub `ubuntu-latest` produces amd64. Do not
+assume a single cached image works across both without multi-platform build.
+
+## Out of scope (MVP)
+
+- Full Bundler/dev toolchain prebake
+- Per-eval custom Dockerfiles
+- Read-only root filesystem (documented as future hardening)
+- gVisor / Firecracker

data/lib/skill_bench/clients/provider_config.rb

@@ -72,8 +72,14 @@ module SkillBench
         value == true || value.to_s.strip.casecmp?('true')
       end
 
+      # Prefer an option when the key is present — including an explicit `nil`,
+      # which means "unset" and must not fall through to global Config (a prior
+      # `||` treated nil as missing and leaked stored API keys into clients).
       def fetch_config(key)
-        @options[key] || @config[key]
+        return @options[key] if @options.key?(key)
+        return @options[key.to_s] if @options.key?(key.to_s)
+
+        @config[key] || @config[key.to_s]
       end
     end
   end

data/lib/skill_bench/constants.rb

@@ -30,6 +30,33 @@ module SkillBench
     # Sandbox Configuration
     module Sandbox
       DOCKER_IMAGE_NAME = 'evaluator-sandbox'
+
+      # Relative path of the Docker build context as packaged in the gem.
+      DOCKER_CONTEXT_RELATIVE = 'lib/skill_bench/execution/docker'
+
+      # Absolute path to the Docker build context shipped next to the library code.
+      # Resolved from this file so gem installs and git checkouts agree.
+      #
+      # @return [String] absolute path to the docker/ directory
+      def self.docker_context_path
+        File.expand_path('execution/docker', __dir__)
+      end
+
+      # Versioned image reference used for build and run.
+      #
+      # @return [String] e.g. "evaluator-sandbox:1.2.0"
+      def self.image_ref
+        require_relative 'version' unless defined?(SkillBench::VERSION)
+
+        "#{DOCKER_IMAGE_NAME}:#{SkillBench::VERSION}"
+      end
+
+      # Floating latest tag applied alongside the versioned tag on build.
+      #
+      # @return [String] e.g. "evaluator-sandbox:latest"
+      def self.latest_image_ref
+        "#{DOCKER_IMAGE_NAME}:latest"
+      end
     end
 
     # Tool Execution Configuration

data/lib/skill_bench/execution/docker/.dockerignore

@@ -0,0 +1,2 @@
+*
+!Dockerfile

data/lib/skill_bench/execution/docker/Dockerfile

@@ -0,0 +1,24 @@
+# SkillBench evaluator-sandbox image contract (MVP)
+#
+# Purpose: long-lived container for `docker exec` of allowlisted agent commands.
+# Runtime flags (from Sandbox#start_container): --network none, non-root uid/gid,
+#   --security-opt no-new-privileges, --cap-drop ALL (+ CHOUN/DAC_OVERRIDE).
+#
+# Assumptions:
+# - No package installs or gem fetches during an eval (network is disabled at runtime).
+# - Ruby major tracks CI (3.3/3.4). Bookworm image provides a modern baseline.
+# - git is available for tools that shell out to git inside the sandbox mount.
+# - Multi-arch: build on the host arch (arm64 Mac / amd64 CI) via docker build.
+#
+# See docs/docker.md and milestone container-isolation-v1.
+
+FROM ruby:3.4-bookworm
+
+RUN apt-get update \
+  && apt-get install -y --no-install-recommends git \
+  && rm -rf /var/lib/apt/lists/*
+
+WORKDIR /sandbox
+
+# Keep the container alive so repeated `docker exec` sessions can run commands.
+CMD ["sleep", "infinity"]

data/lib/skill_bench/execution/sandbox.rb

@@ -10,11 +10,11 @@ module SkillBench
     # Manages isolated sandbox environments for running agent evaluations.
     # Handles copying files, initializing git, and capturing diffs.
     #
-    # NOTE: Container isolation is not yet shipped. No Docker build context is
-    # packaged, so `docker_available?` always returns false and `start_container`
-    # is never reached — `container_id` stays nil and commands run on the host
-    # (gated by the allowlist and `Config.allow_host_execution`). The container
-    # code below is the planned isolation model, retained but currently inactive.
+    # NOTE: A Docker build context is packaged under execution/docker so gem
+    # installs can activate container isolation when a Docker daemon is present.
+    # When `docker_available?` is false (no context, no daemon, or docker missing),
+    # `container_id` stays nil and commands run on the host only if
+    # `Config.allow_host_execution` is enabled (fail closed by default).
     class Sandbox
       attr_reader :path, :container_id
 
@@ -44,6 +44,20 @@ module SkillBench
         ['git', *GIT_HARDENING, *args]
       end
 
+      # Absolute path to the packaged Docker build context.
+      #
+      # @return [String] path to lib/skill_bench/execution/docker
+      def self.docker_context_path
+        Constants::Sandbox.docker_context_path
+      end
+
+      # Versioned Docker image reference for this gem release.
+      #
+      # @return [String] image:tag such as evaluator-sandbox:1.2.0
+      def self.image_ref
+        Constants::Sandbox.image_ref
+      end
+
       # Runs a block of code within a temporary, isolated sandbox directory.
       # The sandbox is initialized as a git repository and optionally wrapped in a Docker container.
       #
@@ -176,12 +190,13 @@ module SkillBench
         end
       end
 
-      # Checks if Docker is available and the sandbox Dockerfile exists.
+      # Checks if Docker is available and the sandbox Dockerfile context exists.
       #
       # @return [Boolean] true if Docker is available, false otherwise.
       def docker_available?
-        docker_dir = File.expand_path('docker', __dir__)
+        docker_dir = self.class.docker_context_path
         return false unless File.directory?(docker_dir)
+        return false unless File.file?(File.join(docker_dir, 'Dockerfile'))
 
         _stdout, _stderr, status = Open3.capture3('docker', 'info')
         status.success?
@@ -190,23 +205,18 @@ module SkillBench
       end
 
       # Starts a Docker container for isolated command execution.
-      # Builds the image only if it does not already exist.
+      # Builds the image only when the versioned tag is not already present.
       # Uses hardened security settings for production safety.
       #
       # @raise [RuntimeError] when the Docker image cannot be built or the container fails to start.
       def start_container
-        image_name = Constants::Sandbox::DOCKER_IMAGE_NAME
-        docker_dir = File.expand_path('docker', __dir__)
-
-        # Build image (Docker layer cache handles no-op builds)
-        raise "Failed to build Docker image #{image_name}" unless system('docker', 'build', '-t', image_name, docker_dir, '--quiet')
-
-        # Start a detached container with hardened security settings
-        # --user $(id -u):$(id -g): Runs as non-root user
-        # --security-opt no-new-privileges: Prevents privilege escalation
-        # --cap-drop ALL: Drops all Linux capabilities
-        # --cap-add CHOWN, DAC_OVERRIDE: Adds back minimal capabilities for git operations
-        # --network none: Disables network access for additional isolation
+        ensure_image
+        image = self.class.image_ref
+
+        # --user uid:gid: non-root
+        # --security-opt no-new-privileges: no privilege escalation
+        # --cap-drop ALL (+ CHOWN/DAC_OVERRIDE): minimal caps for git volume ops
+        # --network none: no network during evals
         stdout, stderr, status = Open3.capture3(
           'docker', 'run', '-d', '--rm',
           '--user', "#{Process.uid}:#{Process.gid}",
@@ -216,7 +226,7 @@ module SkillBench
           '--cap-add', 'DAC_OVERRIDE',
           '--network', 'none',
           '-v', "#{@path}:/sandbox:rw",
-          image_name
+          image
         )
 
         raise "Failed to start Docker container: #{stderr}" unless status.success?
@@ -224,6 +234,35 @@ module SkillBench
         @container_id = stdout.strip
       end
 
+      # Ensures the versioned evaluator image exists locally, building only if missing.
+      #
+      # @raise [RuntimeError] when the image cannot be built.
+      # @return [void]
+      def ensure_image
+        image = self.class.image_ref
+        return if image_present?(image)
+
+        docker_dir = self.class.docker_context_path
+        latest = Constants::Sandbox.latest_image_ref
+        built = system(
+          'docker', 'build',
+          '-t', image,
+          '-t', latest,
+          docker_dir,
+          '--quiet'
+        )
+        raise "Failed to build Docker image #{image}" unless built
+      end
+
+      # @param image [String] full image reference including tag
+      # @return [Boolean] true when docker already has the image
+      def image_present?(image)
+        _stdout, _stderr, status = Open3.capture3('docker', 'image', 'inspect', image)
+        status.success?
+      rescue Errno::ENOENT
+        false
+      end
+
       def stop_container
         return unless @container_id
 

data/lib/skill_bench/package_verifier.rb

@@ -26,6 +26,8 @@ module SkillBench
       lib/skill_bench/config/store.rb
       lib/skill_bench/package_verifier.rb
       lib/skill_bench/execution/source_path_resolver.rb
+      lib/skill_bench/execution/docker/Dockerfile
+      lib/skill_bench/execution/docker/.dockerignore
       lib/skill_bench/runner.rb
     ].freeze
 

data/lib/skill_bench/version.rb

@@ -2,5 +2,5 @@
 
 module SkillBench
   # The current gem version.
-  VERSION = '1.2.0'
+  VERSION = '1.3.0'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ruby-skill-bench
 version: !ruby/object:Gem::Version
-  version: 1.2.0
+  version: 1.3.0
 platform: ruby
 authors:
 - Ismael Marin
@@ -57,14 +57,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 2.0.0
+        version: '1.26'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 2.0.0
+        version: '1.26'
 description: |
   ruby-skill-bench orchestrates evaluation runs of AI coding agents
   inside isolated git sandboxes, then scores the results using deterministic
@@ -80,6 +80,7 @@ files:
 - README.md
 - bin/skill-bench
 - docs/architecture.md
+- docs/docker.md
 - docs/first-eval-guide.md
 - docs/testing-guide.md
 - lib/skill_bench.rb
@@ -150,6 +151,8 @@ files:
 - lib/skill_bench/evaluation/runner.rb
 - lib/skill_bench/execution.rb
 - lib/skill_bench/execution/context_hydrator.rb
+- lib/skill_bench/execution/docker/.dockerignore
+- lib/skill_bench/execution/docker/Dockerfile
 - lib/skill_bench/execution/sandbox.rb
 - lib/skill_bench/execution/source_path_resolver.rb
 - lib/skill_bench/history_recorder.rb
@@ -245,7 +248,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.6.9
+rubygems_version: 4.0.16
 specification_version: 4
 summary: The evaluation engine for AI Agent Skills benchmarking.
 test_files: []