skillbench-cli 0.1.0__tar.gz

skillbench_cli-0.1.0/.gitignore

@@ -0,0 +1,6 @@
+__pycache__/
+*.pyc
+.venv/
+.pytest_cache/
+.ruff_cache/
+dist/

skillbench_cli-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Adam Saul
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

skillbench_cli-0.1.0/PKG-INFO

@@ -0,0 +1,218 @@
+Metadata-Version: 2.5
+Name: skillbench-cli
+Version: 0.1.0
+Summary: Test your Claude Code skills and agent workflows, the way you test code.
+Project-URL: Homepage, https://github.com/adamsaul314/skillbench
+Project-URL: Issues, https://github.com/adamsaul314/skillbench/issues
+Author: Adam Saul
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agents,claude,claude-code,evals,skills,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.7
+Description-Content-Type: text/markdown
+
+# skillbench
+
+Test your Claude Code skills and agent workflows, the way you test code.
+
+You write a small YAML file that says *"send this to Claude, then check these things"*. skillbench runs Claude Code for real on a copy of your repo, records everything it did, and tells you **pass** or **fail**.
+
+---
+
+## 1. Install
+
+You need Python 3.11+, git, and Claude Code installed and logged in (check with `claude --version`). Then install with whichever tool you use:
+
+```
+pip install skillbench-cli
+pipx install skillbench-cli
+uv tool install skillbench-cli
+```
+
+The command is `skillbench` either way. To try it without installing: `uvx --from skillbench-cli skillbench init`.
+
+## 2. Try it (2 minutes)
+
+In your repo:
+
+```
+skillbench init      # creates a skillbench/ folder with two example tests
+skillbench skill     # runs the skill test
+skillbench agent     # runs the agent test
+```
+
+You'll see something like:
+
+```
+[1/1] hello-greets-by-name  default model
+hello-greets-by-name: pass in 2.3s
+dashboard: skillbench/runs/index.html
+```
+
+Open `skillbench/runs/index.html` in your browser to see exactly what Claude did: every message, tool call and check.
+
+That's it. The rest of this page shows how to write your own tests.
+
+---
+
+## 3. Two kinds of test
+
+| | **Skill test** | **Agent test** |
+|---|---|---|
+| Question it answers | "When my skill runs, does it do the right thing?" | "Given a normal request, does Claude pick the right skills and get the job done?" |
+| Lives in | `skillbench/skills/` | `skillbench/agent/` |
+| Run with | `skillbench skill` | `skillbench agent` |
+| You write | `skill:` and `input:` | `prompt:` |
+
+### A skill test
+
+`skillbench/skills/grilling.yaml`
+
+```yaml
+skill: grilling                 # your skill's name
+input: an idea for a todo app   # what to send it
+fixture: .                      # run it on a copy of this repo
+
+assert:
+  - final_text_contains: "Q1"
+  - tool_not_called: Write
+```
+
+skillbench finds the skill wherever it lives in your repo (`.claude/skills/`, `skills/`, `.github/skills/` or a plugin) and runs it as `/grilling an idea for a todo app`.
+
+### An agent test
+
+`skillbench/agent/csv-export.yaml`
+
+```yaml
+prompt: add CSV export to the report page
+fixture: .
+
+answers:                       # replies, sent one at a time when Claude asks something
+  - Yes to all your recommendations.
+  - approved
+
+assert:
+  - skill_invoked: planner
+  - tool_order: [Read, Edit, Bash]
+  - command_succeeds: python -m pytest -q
+```
+
+Nothing is forced here. Claude sees every skill in your repo and decides for itself, and the test checks what it chose.
+
+---
+
+## 4. Checks you can make
+
+Put any of these under `assert:`.
+
+| Check | Passes when |
+|---|---|
+| `skill_invoked: grilling` | Claude used that skill |
+| `skill_not_invoked: grilling` | Claude didn't use that skill |
+| `tool_order: [Read, Edit]` | Claude used these tools in this order (other tools in between are fine) |
+| `tool_not_called: Write` | Claude never used that tool |
+| `final_text_contains: "Done"` | Claude's last message contains the text |
+| `file_contains: {path: notes.md, text: "approved"}` | That file exists afterwards and contains the text |
+| `command_succeeds: python -m pytest -q` | The command runs successfully in the workspace afterwards |
+
+A test passes when every check passes.
+
+---
+
+## 5. Which repo Claude works in (`fixture:`)
+
+Claude always works on a **copy**, so your real files are never touched.
+
+| `fixture:` | Claude gets |
+|---|---|
+| *(leave it out)* | An empty folder |
+| `.` | A copy of your repo, including work you haven't committed |
+| `some/folder` | A copy of that folder |
+| `https://github.com/you/app.git` | A fresh clone |
+| `{git: https://github.com/you/app.git, ref: v1.2}` | A clone at that branch, tag or commit |
+
+Paths are relative to your repo root. The copy is kept after the run so you can look inside, and its path is in the dashboard.
+
+---
+
+## 6. Running tests
+
+```
+skillbench skill                 # all skill tests
+skillbench skill grilling        # just the tests for one skill (or one test by name)
+skillbench agent csv-export      # one agent test
+skillbench agent --repeat 5      # run each test 5 times
+skillbench agent --model claude-sonnet-5-5
+```
+
+While it runs you'll see the current test, what Claude is doing, and roughly how long is left.
+
+---
+
+## 7. More options
+
+You won't need these to start.
+
+**Check a skill gets picked on its own.** List prompts that should and shouldn't trigger it. Each runs briefly and stops as soon as Claude decides:
+
+```yaml
+skill: grilling
+triggers:
+  should: ["stress-test my plan"]
+  should_not: ["fix this typo"]
+```
+
+**Answer Claude yourself.** Add `interactive: true` and skillbench asks *you* in the terminal whenever Claude is waiting for a reply. Press Enter on an empty line to finish.
+
+**Check for flakiness.** Claude doesn't do exactly the same thing every time. `--repeat 5` (or `repeat: 5` in the test) runs it 5 times and shows how often each check passed. Add `min_pass_rate: 0.8` to pass when 4 out of 5 runs pass.
+
+**Set defaults for every test** in `skillbench/skillbench.yaml`:
+
+```yaml
+model: claude-sonnet-5-5
+timeout_s: 600        # seconds before a turn is stopped
+fixture: .
+repeat: 1
+min_pass_rate: 1.0
+```
+
+A test's own settings win over these, and command-line flags win over both.
+
+**All test fields:**
+
+| Field | Meaning |
+|---|---|
+| `skill` / `input` | Skill tests: which skill, and what to send it |
+| `prompt` | Agent tests: the first message |
+| `answers` | Replies sent one per turn |
+| `interactive` | Ask you in the terminal for replies |
+| `triggers` | Prompts that should / shouldn't pick the skill |
+| `fixture` | Which repo Claude works in |
+| `assert` | The checks |
+| `name` | A display name (defaults to the file name) |
+| `timeout_s`, `repeat`, `min_pass_rate` | As above |
+
+---
+
+## Good to know
+
+- **It costs real usage.** Every run is a real Claude session. Start with skill tests; long agent tests take longer and cost more.
+- **Your personal setup is ignored.** Only the skills and plugins in the repo being tested are loaded, so results don't depend on whose machine runs them.
+- **Only test repos you trust.** Claude has full tool access inside the copy, and `command_succeeds` runs commands there.
+- **`command_succeeds` uses your system shell** (`sh` on Mac/Linux, `cmd` on Windows), so keep commands simple if you share tests across systems.
+- **Results** are in `skillbench/runs/`, which is git-ignored. Each run has a `run.json` and Claude's raw output.
+
+| Status | Meaning |
+|---|---|
+| `pass` | Every check passed |
+| `fail` | Claude finished, but a check failed |
+| `error` | Claude crashed or timed out, or the skill or fixture couldn't be found |

skillbench_cli-0.1.0/README.md

@@ -0,0 +1,197 @@
+# skillbench
+
+Test your Claude Code skills and agent workflows, the way you test code.
+
+You write a small YAML file that says *"send this to Claude, then check these things"*. skillbench runs Claude Code for real on a copy of your repo, records everything it did, and tells you **pass** or **fail**.
+
+---
+
+## 1. Install
+
+You need Python 3.11+, git, and Claude Code installed and logged in (check with `claude --version`). Then install with whichever tool you use:
+
+```
+pip install skillbench-cli
+pipx install skillbench-cli
+uv tool install skillbench-cli
+```
+
+The command is `skillbench` either way. To try it without installing: `uvx --from skillbench-cli skillbench init`.
+
+## 2. Try it (2 minutes)
+
+In your repo:
+
+```
+skillbench init      # creates a skillbench/ folder with two example tests
+skillbench skill     # runs the skill test
+skillbench agent     # runs the agent test
+```
+
+You'll see something like:
+
+```
+[1/1] hello-greets-by-name  default model
+hello-greets-by-name: pass in 2.3s
+dashboard: skillbench/runs/index.html
+```
+
+Open `skillbench/runs/index.html` in your browser to see exactly what Claude did: every message, tool call and check.
+
+That's it. The rest of this page shows how to write your own tests.
+
+---
+
+## 3. Two kinds of test
+
+| | **Skill test** | **Agent test** |
+|---|---|---|
+| Question it answers | "When my skill runs, does it do the right thing?" | "Given a normal request, does Claude pick the right skills and get the job done?" |
+| Lives in | `skillbench/skills/` | `skillbench/agent/` |
+| Run with | `skillbench skill` | `skillbench agent` |
+| You write | `skill:` and `input:` | `prompt:` |
+
+### A skill test
+
+`skillbench/skills/grilling.yaml`
+
+```yaml
+skill: grilling                 # your skill's name
+input: an idea for a todo app   # what to send it
+fixture: .                      # run it on a copy of this repo
+
+assert:
+  - final_text_contains: "Q1"
+  - tool_not_called: Write
+```
+
+skillbench finds the skill wherever it lives in your repo (`.claude/skills/`, `skills/`, `.github/skills/` or a plugin) and runs it as `/grilling an idea for a todo app`.
+
+### An agent test
+
+`skillbench/agent/csv-export.yaml`
+
+```yaml
+prompt: add CSV export to the report page
+fixture: .
+
+answers:                       # replies, sent one at a time when Claude asks something
+  - Yes to all your recommendations.
+  - approved
+
+assert:
+  - skill_invoked: planner
+  - tool_order: [Read, Edit, Bash]
+  - command_succeeds: python -m pytest -q
+```
+
+Nothing is forced here. Claude sees every skill in your repo and decides for itself, and the test checks what it chose.
+
+---
+
+## 4. Checks you can make
+
+Put any of these under `assert:`.
+
+| Check | Passes when |
+|---|---|
+| `skill_invoked: grilling` | Claude used that skill |
+| `skill_not_invoked: grilling` | Claude didn't use that skill |
+| `tool_order: [Read, Edit]` | Claude used these tools in this order (other tools in between are fine) |
+| `tool_not_called: Write` | Claude never used that tool |
+| `final_text_contains: "Done"` | Claude's last message contains the text |
+| `file_contains: {path: notes.md, text: "approved"}` | That file exists afterwards and contains the text |
+| `command_succeeds: python -m pytest -q` | The command runs successfully in the workspace afterwards |
+
+A test passes when every check passes.
+
+---
+
+## 5. Which repo Claude works in (`fixture:`)
+
+Claude always works on a **copy**, so your real files are never touched.
+
+| `fixture:` | Claude gets |
+|---|---|
+| *(leave it out)* | An empty folder |
+| `.` | A copy of your repo, including work you haven't committed |
+| `some/folder` | A copy of that folder |
+| `https://github.com/you/app.git` | A fresh clone |
+| `{git: https://github.com/you/app.git, ref: v1.2}` | A clone at that branch, tag or commit |
+
+Paths are relative to your repo root. The copy is kept after the run so you can look inside, and its path is in the dashboard.
+
+---
+
+## 6. Running tests
+
+```
+skillbench skill                 # all skill tests
+skillbench skill grilling        # just the tests for one skill (or one test by name)
+skillbench agent csv-export      # one agent test
+skillbench agent --repeat 5      # run each test 5 times
+skillbench agent --model claude-sonnet-5-5
+```
+
+While it runs you'll see the current test, what Claude is doing, and roughly how long is left.
+
+---
+
+## 7. More options
+
+You won't need these to start.
+
+**Check a skill gets picked on its own.** List prompts that should and shouldn't trigger it. Each runs briefly and stops as soon as Claude decides:
+
+```yaml
+skill: grilling
+triggers:
+  should: ["stress-test my plan"]
+  should_not: ["fix this typo"]
+```
+
+**Answer Claude yourself.** Add `interactive: true` and skillbench asks *you* in the terminal whenever Claude is waiting for a reply. Press Enter on an empty line to finish.
+
+**Check for flakiness.** Claude doesn't do exactly the same thing every time. `--repeat 5` (or `repeat: 5` in the test) runs it 5 times and shows how often each check passed. Add `min_pass_rate: 0.8` to pass when 4 out of 5 runs pass.
+
+**Set defaults for every test** in `skillbench/skillbench.yaml`:
+
+```yaml
+model: claude-sonnet-5-5
+timeout_s: 600        # seconds before a turn is stopped
+fixture: .
+repeat: 1
+min_pass_rate: 1.0
+```
+
+A test's own settings win over these, and command-line flags win over both.
+
+**All test fields:**
+
+| Field | Meaning |
+|---|---|
+| `skill` / `input` | Skill tests: which skill, and what to send it |
+| `prompt` | Agent tests: the first message |
+| `answers` | Replies sent one per turn |
+| `interactive` | Ask you in the terminal for replies |
+| `triggers` | Prompts that should / shouldn't pick the skill |
+| `fixture` | Which repo Claude works in |
+| `assert` | The checks |
+| `name` | A display name (defaults to the file name) |
+| `timeout_s`, `repeat`, `min_pass_rate` | As above |
+
+---
+
+## Good to know
+
+- **It costs real usage.** Every run is a real Claude session. Start with skill tests; long agent tests take longer and cost more.
+- **Your personal setup is ignored.** Only the skills and plugins in the repo being tested are loaded, so results don't depend on whose machine runs them.
+- **Only test repos you trust.** Claude has full tool access inside the copy, and `command_succeeds` runs commands there.
+- **`command_succeeds` uses your system shell** (`sh` on Mac/Linux, `cmd` on Windows), so keep commands simple if you share tests across systems.
+- **Results** are in `skillbench/runs/`, which is git-ignored. Each run has a `run.json` and Claude's raw output.
+
+| Status | Meaning |
+|---|---|
+| `pass` | Every check passed |
+| `fail` | Claude finished, but a check failed |
+| `error` | Claude crashed or timed out, or the skill or fixture couldn't be found |

skillbench_cli-0.1.0/pyproject.toml

@@ -0,0 +1,46 @@
+[project]
+name = "skillbench-cli"
+version = "0.1.0"
+description = "Test your Claude Code skills and agent workflows, the way you test code."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "Adam Saul" }]
+requires-python = ">=3.11"
+keywords = ["claude", "claude-code", "agents", "skills", "evals", "testing"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Topic :: Software Development :: Testing",
+]
+dependencies = [
+    "pyyaml>=6.0",
+    "rich>=13.7",
+]
+
+[project.urls]
+Homepage = "https://github.com/adamsaul314/skillbench"
+Issues = "https://github.com/adamsaul314/skillbench/issues"
+
+[project.scripts]
+skillbench = "skillbench.cli:main"
+
+[dependency-groups]
+dev = ["ruff>=0.6"]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/skillbench"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "B", "UP", "SIM"]

skillbench_cli-0.1.0/src/skillbench/__init__.py

@@ -0,0 +1 @@
+"""skillbench: eval framework for agent skills."""