skillbench-cli 0.1.0__tar.gz → 0.2.0__tar.gz

skillbench_cli-0.2.0/.gitattributes

@@ -0,0 +1 @@
+*.sh text eol=lf

{skillbench_cli-0.1.0 → skillbench_cli-0.2.0}/LICENSE

@@ -1,21 +1,21 @@
-MIT License
-
-Copyright (c) 2026 Adam Saul
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+MIT License
+
+Copyright (c) 2026 Adam Saul
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

skillbench_cli-0.2.0/PKG-INFO

@@ -0,0 +1,347 @@
+Metadata-Version: 2.5
+Name: skillbench-cli
+Version: 0.2.0
+Summary: Test your Claude Code skills and agent workflows, the way you test code.
+Project-URL: Homepage, https://github.com/adamsaul314/skillbench
+Project-URL: Issues, https://github.com/adamsaul314/skillbench/issues
+Author: Adam Saul
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agents,claude,claude-code,evals,skills,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.7
+Description-Content-Type: text/markdown
+
+# skillbench
+
+Test your Claude Code skills and agent workflows, the way you test code.
+
+You write a short YAML file: *"send Claude this, then check these things"*. skillbench runs Claude Code for real on a **copy** of a repo, records everything it did, and tells you **pass** or **fail**.
+
+**Jump to:** [Get started](#get-started) · [Write a test](#write-a-test) · [Run tests](#run-tests) · [Reference](#reference)
+
+---
+
+## Get started
+
+You need Python 3.11+, git, and Claude Code installed and logged in (`claude --version` should work).
+
+**1. Install it.** Use whichever tool you normally use:
+
+```
+pip install skillbench-cli
+pipx install skillbench-cli
+uv tool install skillbench-cli
+```
+
+The package is `skillbench-cli`; the command is `skillbench`.
+
+**2. Set up your repo.** In the repo you want to test:
+
+```
+skillbench init
+```
+
+This creates one folder, `skillbench/`, with two example tests:
+
+```
+skillbench/
+  skillbench.yaml     settings for every test
+  skills/             skill tests
+  agent/              agent tests
+  fixtures/           small sample repos for tests to run on
+  runs/               results (git-ignored)
+```
+
+**3. Run the examples.**
+
+```
+skillbench skill
+skillbench agent
+```
+
+**4. Look at the results.** Open `skillbench/runs/index.html` in your browser. The list on the left groups runs by what they test (a skill, a plugin, or this repo), newest first. Expand a group to see its tests. Click a test to see its run.
+
+Now write your own tests: next section.
+
+---
+
+## Write a test
+
+Find what you want to check in this list, then follow that recipe.
+
+| I want to check that… | Recipe |
+|---|---|
+| my skill does the right thing when it runs | [A. Test a skill](#a-test-a-skill) |
+| Claude picks my skill when it should, and not when it shouldn't | [B. Test a skill gets picked](#b-test-a-skill-gets-picked) |
+| a normal request gets done, with the right skills | [C. Test a request end to end](#c-test-a-request-end-to-end) |
+| my multi-step workflow works on a real repo | [D. Test a workflow on a repo](#d-test-a-workflow-on-a-repo) |
+| the plugin I'm building works on other projects | [E. Test a plugin you're building](#e-test-a-plugin-youre-building) |
+
+**Or let Claude write it.** Install the `/eval-creator` skill once:
+
+```
+skillbench install-skill
+```
+
+Then in Claude Code, in your repo, say what you want to test:
+
+```
+/eval-creator test that my grilling skill asks questions before writing anything
+```
+
+It reads your skills, then proposes the tests in plain words: what each one runs, what it checks, when it passes, and a rough cost. Nothing is written until you reply **approve**. Ask for changes and it proposes again. Once approved, it writes the files and checks them. It never runs them, so you decide when to spend tokens.
+
+Run `skillbench install-skill` again after you upgrade skillbench.
+
+Whichever way you write tests, check them for free before running:
+
+```
+skillbench check
+```
+
+### A. Test a skill
+
+Runs your skill directly (`/grilling an idea for a todo app`) and checks what it did.
+
+Create `skillbench/skills/grilling.yaml`:
+
+```yaml
+description: Grilling asks questions before it writes anything.
+skill: grilling                 # the skill's name
+input: an idea for a todo app   # the text after /grilling
+fixture: .                      # run on a copy of this repo
+
+assert:
+  - final_text_contains: "?"
+  - tool_not_called: Write
+```
+
+Run it: `skillbench skill grilling`
+
+skillbench finds the skill wherever it is: `.claude/skills/`, `skills/`, `.github/skills/`, or inside a plugin.
+
+### B. Test a skill gets picked
+
+Checks your skill's description is good enough for Claude to choose it on its own. Each prompt is a short run that stops as soon as Claude decides, so these are cheap.
+
+Add `triggers:` to a skill test:
+
+```yaml
+skill: grilling
+triggers:
+  should: ["stress-test my plan before I build it"]
+  should_not: ["fix the typo in README.md"]
+```
+
+Run it: `skillbench skill grilling`
+
+### C. Test a request end to end
+
+Sends a normal message, with nothing forced. Claude sees every skill in the repo and decides for itself; the test checks what it chose and whether the job got done.
+
+Create `skillbench/agent/csv-export.yaml`:
+
+```yaml
+description: Adding CSV export uses the planner and leaves the tests passing.
+prompt: add CSV export to the report page
+fixture: .
+
+answers:                        # sent one at a time, each time Claude stops and waits
+  - Yes to all your recommendations.
+  - approved
+
+assert:
+  - skill_invoked: planner
+  - command_succeeds: python -m pytest -q
+```
+
+Run it: `skillbench agent csv-export`
+
+Not sure what Claude will ask? Leave out `answers:` and add `interactive: true`. skillbench then asks *you* in the terminal each time Claude waits for a reply (press Enter on an empty line to stop). Copy your replies into `answers:` afterwards.
+
+### D. Test a workflow on a repo
+
+For workflows with several phases, where in Claude Code you'd type things between messages: commit, `/clear`, start the next skill. Put those in `answers:` exactly as you'd type them:
+
+| In `answers:` | What happens |
+|---|---|
+| `approved` | Sent to Claude as a message |
+| `/clear` | The next message starts a fresh session |
+| `!git add -A && git commit -m "plan"` | Runs in the repo as a shell command, not sent to Claude |
+
+Point `fixture:` at the repo to run on. A git URL is cloned fresh for each run, and pushing from the clone is turned off.
+
+Create `skillbench/agent/full-loop.yaml`:
+
+```yaml
+description: The plan, build and review loop adds a dark mode toggle to test-app.
+prompt: /my-plugin:plan add a dark mode toggle. Use the slug dark-mode.
+fixture: https://github.com/you/test-app.git
+
+answers:
+  - Yes to all your recommendations.
+  - approved
+  - '!git add -A && git commit -m "plan: dark-mode"'
+  - /clear
+  - /my-plugin:build dark-mode
+  - '!git add -A && git commit -m "dark-mode: build"'
+  - /clear
+  - /my-plugin:final-review dark-mode
+
+assert:
+  - subagent_invoked: reviewer
+  - file_contains: {path: docs/features/dark-mode/plan.md, text: "approved"}
+  - command_succeeds: npm test
+```
+
+Run it on a cheap model first: `skillbench agent full-loop --model claude-haiku-5-5`
+
+If a `!` command fails, the run stops there with `error`, since later steps depend on it.
+
+### E. Test a plugin you're building
+
+If your repo **is** a Claude Code plugin (it has `.claude-plugin/plugin.json`), add this line to `skillbench/skillbench.yaml`:
+
+```yaml
+plugin: .
+```
+
+Now your plugin's skills and subagents are loaded into every test, whatever repo the test runs on. Combine it with recipe D to run your workflow on any project. Each run gets its own copy of the plugin, so the dashboard shows exactly which version was tested. `skillbench check` warns you if you forget this line.
+
+---
+
+## Run tests
+
+```
+skillbench skill                 # every skill test
+skillbench agent                 # every agent test
+skillbench agent full-loop       # one test, by name (or every test for one skill)
+skillbench agent --model claude-haiku-5-5
+skillbench agent --repeat 5      # run each test 5 times
+```
+
+The terminal shows one line per test:
+
+```
+✓ hello-greets-by-name   passed      12s   $0.03
+✗ full-loop              error       41s   $0.02
+    stopped at step 7: '!git commit -m "dark-mode: build"' exited with code 1
+
+1 passed, 1 failed  ·  53s  ·  $0.05
+details: skillbench/runs/index.html
+```
+
+| Result | Meaning |
+|---|---|
+| `pass` | Every check passed |
+| `fail` | Claude finished, but a check failed |
+| `error` | The run stopped: Claude crashed or timed out, a `!` command failed, or the skill or repo couldn't be found |
+
+For the details, open `skillbench/runs/index.html`. Each run page starts with the answer: whether it passed, and if not, the step it stopped at or the checks that failed. Below that are the steps in plain words, the checks as sentences, Claude's last message and the numbers. The full timeline of messages and tool calls, and the raw output, are collapsed at the end.
+
+**Keep costs down.** Every run is a real Claude session. Use `--model claude-haiku-5-5` while you're getting a test right, start with skill tests, and run long workflows only when you need to.
+
+**Flaky results?** Claude doesn't do exactly the same thing every time. `--repeat 5` shows how often each check passed. Add `min_pass_rate: 0.8` to a test to pass it when 4 of 5 runs pass.
+
+---
+
+## Reference
+
+### Checks (`assert:`)
+
+A test passes when every check passes.
+
+| Check | Passes when |
+|---|---|
+| `skill_invoked: grilling` | Claude used that skill |
+| `skill_not_invoked: grilling` | Claude didn't use that skill |
+| `subagent_invoked: reviewer` | Claude handed work to that subagent |
+| `tool_order: [Read, Edit]` | Claude used these tools in this order (others in between are fine) |
+| `tool_not_called: Write` | Claude never used that tool |
+| `final_text_contains: "Done"` | Claude's last message contains the text |
+| `file_contains: {path: notes.md, text: "approved"}` | The file exists afterwards and contains the text |
+| `command_succeeds: npm test` | The command succeeds in the repo afterwards |
+
+### Repos (`fixture:`)
+
+Claude always works on a copy, so your real files are never touched.
+
+| `fixture:` | Claude works in |
+|---|---|
+| *(left out)* | An empty folder |
+| `.` | A copy of your repo, including uncommitted work |
+| `some/folder` | A copy of that folder (paths are from your repo root) |
+| `https://github.com/you/app.git` | A fresh clone |
+| `{git: https://github.com/you/app.git, ref: v1.2}` | A clone at that branch, tag or commit |
+
+The copy is kept after the run so you can look inside. Its path is on the run's dashboard page.
+
+### Test fields
+
+| Field | Meaning |
+|---|---|
+| `description` | One sentence saying what the test checks, shown on the dashboard |
+| `skill`, `input` | Skill tests: which skill, and the text sent with it |
+| `triggers` | Skill tests: prompts that `should` and `should_not` pick the skill |
+| `prompt` | Agent tests: the first message |
+| `answers` | Replies, one per turn; also `/clear` and `!command` |
+| `interactive` | `true` to answer in the terminal once `answers` run out |
+| `fixture` | The repo Claude works in |
+| `assert` | The checks |
+| `name` | Display name (defaults to the file name) |
+| `timeout_s` | Seconds before one turn is stopped |
+| `repeat`, `min_pass_rate` | Run it N times; the share of runs that must pass |
+
+### Settings for every test (`skillbench/skillbench.yaml`)
+
+```yaml
+plugin: .                 # load this repo's plugin into every test
+model: claude-haiku-5-5
+fixture: .
+timeout_s: 600
+repeat: 1
+min_pass_rate: 1.0
+```
+
+A test's own fields win over these, and `--model` / `--repeat` win over both.
+
+### Commands
+
+| Command | Does |
+|---|---|
+| `skillbench init` | Creates `skillbench/` with example tests |
+| `skillbench skill [name]` | Runs skill tests |
+| `skillbench agent [name]` | Runs agent tests |
+| `skillbench check` | Checks every test file, without running anything |
+| `skillbench install-skill` | Adds `/eval-creator` to Claude Code |
+
+---
+
+## Good to know
+
+- **Your personal Claude setup is ignored.** Only the skills and plugins in the repo being tested (plus `plugin:`) are loaded, so results don't depend on whose machine runs them.
+- **Only test repos you trust.** Claude has full tool access inside the copy, and `command_succeeds` and `!` commands run there.
+- **Commands use your system shell**: `sh` on Mac/Linux, `cmd` on Windows. Keep them simple if tests are shared across systems.
+- **Each run is saved** in `skillbench/runs/`, with a `run.json` and Claude's raw output.
+- **Upgrading:** `pip install -U skillbench-cli` (or your tool's upgrade command), then `skillbench install-skill`.
+
+---
+
+## Releasing a new version (maintainers)
+
+From an up-to-date `main` with no uncommitted changes:
+
+```
+scripts/release.sh 0.2.0 --dry-run   # bump, lint, build and test the install, then undo
+scripts/release.sh 0.2.0             # the same, then commit, tag, publish to PyPI and push
+```
+
+Publishing needs a PyPI API token: `export UV_PUBLISH_TOKEN=pypi-...` first, or uv asks for it.