agent-skill-eval 0.6.0__tar.gz → 0.7.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (81) hide show
  1. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/CHANGELOG.md +62 -0
  2. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/PKG-INFO +322 -18
  3. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/README.md +321 -17
  4. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/commit-push-pr/evals/evals.json +6 -3
  5. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/pyproject.toml +2 -1
  6. agent_skill_eval-0.7.0/schemas/evals.schema.json +237 -0
  7. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/skills/validate-config/scripts/validate_config.py +1 -4
  8. agent_skill_eval-0.7.0/src/agent_skill_eval/cli.py +3600 -0
  9. agent_skill_eval-0.7.0/src/agent_skill_eval/fixtures.py +63 -0
  10. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/git_state.py +125 -3
  11. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/graders/__init__.py +525 -67
  12. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/harnesses/__init__.py +16 -2
  13. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/models.py +74 -3
  14. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/runner.py +161 -19
  15. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/skills.py +67 -0
  16. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/summary.py +72 -3
  17. agent_skill_eval-0.7.0/src/agent_skill_eval/validation.py +224 -0
  18. agent_skill_eval-0.7.0/tests/test_cli_smoke.py +871 -0
  19. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_feedback_features.py +144 -0
  20. agent_skill_eval-0.7.0/tests/test_fixtures.py +34 -0
  21. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_git_state.py +91 -1
  22. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_graders.py +144 -2
  23. agent_skill_eval-0.7.0/tests/test_launch_features.py +970 -0
  24. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_llm_grader.py +155 -2
  25. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_runner.py +46 -5
  26. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_skills.py +15 -0
  27. agent_skill_eval-0.6.0/schemas/evals.schema.json +0 -83
  28. agent_skill_eval-0.6.0/src/agent_skill_eval/cli.py +0 -1136
  29. agent_skill_eval-0.6.0/tests/test_cli_smoke.py +0 -92
  30. agent_skill_eval-0.6.0/tests/test_launch_features.py +0 -489
  31. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/.github/ISSUE_TEMPLATE/bug_report.yml +0 -0
  32. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/.github/ISSUE_TEMPLATE/feature_request.yml +0 -0
  33. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/.github/PULL_REQUEST_TEMPLATE.md +0 -0
  34. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/.github/actions/setup/action.yml +0 -0
  35. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/.github/workflows/ci.yml +0 -0
  36. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/.github/workflows/release.yml +0 -0
  37. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/.gitignore +0 -0
  38. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/AGENTS.md +0 -0
  39. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/CODE_OF_CONDUCT.md +0 -0
  40. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/CONTRIBUTING.md +0 -0
  41. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/LICENSE +0 -0
  42. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/Makefile +0 -0
  43. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/ROADMAP.md +0 -0
  44. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/TODO.md +0 -0
  45. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/config/.env.example +0 -0
  46. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/config/baseline.env +0 -0
  47. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/commit-push-pr/evals/files/sample_change.py +0 -0
  48. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/fix-failing-tests/evals/evals.json +0 -0
  49. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/fix-failing-tests/evals/files/calculator.py +0 -0
  50. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/fix-failing-tests/evals/files/conftest.py +1 -1
  51. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/fix-failing-tests/evals/files/test_calculator.py +0 -0
  52. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/review-diff/evals/evals.json +0 -0
  53. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/review-diff/evals/files/changes.diff +0 -0
  54. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/review-diff/evals/files/orders.py +0 -0
  55. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/validate-config/evals/evals.json +0 -0
  56. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/validate-config/evals/files/app_config.json +0 -0
  57. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/write-release-notes/evals/evals.json +0 -0
  58. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/write-release-notes/evals/files/commits.txt +0 -0
  59. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/scripts/record-demo.sh +0 -0
  60. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/skills/commit-push-pr/SKILL.md +0 -0
  61. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/skills/fix-failing-tests/SKILL.md +0 -0
  62. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/skills/review-diff/SKILL.md +0 -0
  63. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/skills/validate-config/SKILL.md +0 -0
  64. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/skills/validate-config/references/config-rules.md +0 -0
  65. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/skills/write-release-notes/SKILL.md +0 -0
  66. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/__init__.py +0 -0
  67. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/__main__.py +0 -0
  68. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/hooks.py +0 -0
  69. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/openrouter.py +0 -0
  70. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/progress.py +0 -0
  71. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/workspace.py +0 -0
  72. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/__init__.py +0 -0
  73. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/conftest.py +0 -0
  74. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/fixtures/evals/format-json.json +0 -0
  75. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/fixtures/skills/format-json/SKILL.md +0 -0
  76. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_e2e.py +0 -0
  77. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_fuzz_grader.py +0 -0
  78. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_harnesses.py +0 -0
  79. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_openrouter.py +0 -0
  80. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_skill_validation.py +0 -0
  81. {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_workspace.py +0 -0
@@ -6,6 +6,68 @@ from conventional commits; entries below it are curated by hand.
6
6
 
7
7
  <!-- version list -->
8
8
 
9
+ ## v0.7.0 (2026-07-10)
10
+
11
+ ### Features
12
+
13
+ - Run-all multi-suite harness, validation, and reporting suite
14
+ ([`a282bc9`](https://github.com/tardigrde/agent-skill-eval/commit/a282bc9e6d13b0cc4ffa3bb0dec2e02c695313d5))
15
+
16
+ Comprehensive update folding together the run-all multi-suite layer, static validation, expanded
17
+ reporting, and side-effect contract enforcement.
18
+
19
+ Run-all / multi-suite: - concurrent eval-suite execution with paired skill<->suite discovery -
20
+ recursive eval discovery, changed-only mode, plugin eval layout - dry-run planning without agent
21
+ calls, planned-case preview, plan totals - cost/duration totals, pass-rate gating, JUnit reports -
22
+ failed-case rerun commands with context preservation - artifact archive + cross-run compare
23
+
24
+ Validation & assertions: - strict eval-suite linting and structured-field validation - negative
25
+ artifact assertions, command-assertion false-positive fix - run-all assertion-failure reporting
26
+
27
+ Grading: - constrained OpenWebUI grader backend (localhost default)
28
+
29
+ Side effects & reporting: - enforced side-effect contracts, GitLab cleanup manifests - static HTML
30
+ reports, skill portfolio report - skill-hash + run-trajectory recording - doctor command, JSON
31
+ discovery output
32
+
33
+ New modules: fixtures.py, validation.py.
34
+
35
+ Corporate-specific fixtures removed; tool is fully generic. Default LLM grader backend is OpenRouter
36
+ (public).
37
+
38
+ Co-Authored-By: Claude <noreply@anthropic.com>
39
+
40
+
41
+ ## v0.6.1 (2026-06-12)
42
+
43
+ ### Bug Fixes
44
+
45
+ - Opencode non-cached input semantics + LLM grader retry on dropped assertions
46
+ ([`f0939bc`](https://github.com/tardigrde/agent-skill-eval/commit/f0939bc2b6a314d9fa8d62084fe1b57ed4280974))
47
+
48
+ OpenCode reports input_tokens excluding cache reads (live runs show cached consistently larger than
49
+ input), so the codex-style subtraction clamped non_cached_input_tokens to a bogus 0 and capped
50
+ cached_pct at 100%. Override the split like claude-code.
51
+
52
+ The LLM judge intermittently drops an assertion from its JSON answer, which skipped the assertion
53
+ and failed strict e2e runs. Retry once with only the dropped assertions before skipping; matched
54
+ results are now also deduplicated and filtered to the requested assertions so a retry cannot
55
+ double-count.
56
+
57
+ Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
58
+
59
+ - Tolerate malformed judge rows, correct cached_pct denominator (review feedback)
60
+ ([`45b2677`](https://github.com/tardigrde/agent-skill-eval/commit/45b26777fb7466f3973a339e0496219d8ecd4778))
61
+
62
+ A malformed individual result row (non-dict, missing keys, non-string text) now only loses that row
63
+ — its assertion goes to the retry instead of crashing the whole attempt and skipping everything.
64
+
65
+ cached_pct now divides by non_cached + cached, the only denominator valid for both reporting styles;
66
+ cached / max(input, cached) showed a bogus 100% for claude-code/opencode.
67
+
68
+ Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
69
+
70
+
9
71
  ## v0.6.0 (2026-06-12)
10
72
 
11
73
  ### Bug Fixes
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: agent-skill-eval
3
- Version: 0.6.0
3
+ Version: 0.7.0
4
4
  Summary: Framework for evaluating agent skills across OpenCode, Claude Code, and Codex
5
5
  Project-URL: Homepage, https://github.com/tardigrde/agent-skill-eval
6
6
  Project-URL: Repository, https://github.com/tardigrde/agent-skill-eval
@@ -42,7 +42,7 @@ Because the eval goes through the full harness — system prompt, skill discover
42
42
  - **Real harnesses, end to end**: OpenCode, Claude Code, and Codex CLIs, in a single run
43
43
  - **Baseline comparison**: with-skill vs. without-skill, with a per-agent delta
44
44
  - **pass@k**: `--runs N` repeats every eval and reports full-pass rate and pass@k, because agents are stochastic and single-run numbers lie
45
- - **State-delta grading**: code-based checks compare pre/post git state snapshots, so they don't false-pass on pre-existing branches, commits, or PRs
45
+ - **State-delta grading**: code-based checks compare pre/post git state snapshots, so they don't false-pass on pre-existing branches, commits, or review requests
46
46
  - **Negative controls**: `should_trigger: false` inverts assertions to catch accidental skill triggering
47
47
  - **Honest grading**: assertions the grader can't check are *skipped*, not failed; a missing API key warns upfront instead of silently zeroing your pass rate
48
48
  - **Pinned models**: `--agent-model claude-code=haiku --agent-model codex=gpt-5-mini` makes runs reproducible across machines
@@ -53,7 +53,7 @@ Because the eval goes through the full harness — system prompt, skill discover
53
53
  - **Live progress**: per-case console lines plus a tailable `progress.jsonl` — long runs are no longer opaque
54
54
  - **Lifecycle hooks**: `--pre-run-command`, `--post-grade-command`, `--post-run-command` plug external setup, side-effect graders, and teardown into the run with structured `ASE_*` metadata
55
55
  - **One-file summaries**: every run writes `summary.json` (pass rates, failures, token/cost totals, cleanup state); `agent-skill-eval status` reads it back without a model call
56
- - **Scoped cleanup**: only removes artifacts recorded in `cleanup.json`; never closes unrelated PRs or deletes unrelated branches
56
+ - **Scoped cleanup**: only removes artifacts recorded in `cleanup.json`; never closes unrelated PRs/MRs or deletes unrelated branches
57
57
  - **Re-grading**: `agent-skill-eval grade` re-grades saved outputs without re-running agents
58
58
  - **Markdown reports**: paste `agent-skill-eval report --format markdown` straight into a PR or blog post
59
59
 
@@ -77,6 +77,33 @@ uv venv && uv pip install -e ".[dev]"
77
77
 
78
78
  You also need the agent CLIs you want to evaluate (`claude`, `codex`, `opencode`) installed and authenticated, and an `OPENROUTER_API_KEY` or `OPENAI_API_KEY` for LLM rubric grading.
79
79
 
80
+ ## Version and documentation status
81
+
82
+ This README documents the source tree it lives in. If you installed
83
+ `agent-skill-eval` from PyPI or a different Git branch, check the installed CLI
84
+ before copying a command:
85
+
86
+ ```bash
87
+ agent-skill-eval --version
88
+ agent-skill-eval --help
89
+ agent-skill-eval run-all --help
90
+ ```
91
+
92
+ During active development, the newest CLI flags and workflow docs may exist only
93
+ on a local feature branch until that branch is pushed and released. To use a
94
+ local checkout explicitly, run through that checkout instead of whatever command
95
+ is first on `PATH`:
96
+
97
+ ```bash
98
+ uv run --project /path/to/skill-evals agent-skill-eval run-all --help
99
+ uv run --project /path/to/skill-evals python -m agent_skill_eval validate-all --root .
100
+ ```
101
+
102
+ When sharing results, record the command, `agent-skill-eval --version`, agent
103
+ model, reasoning effort, and branch/commit of the harness repo. The harness also
104
+ writes most of that to `run_meta.json` and `summary.json`, but copying it into a
105
+ handoff or PR description makes stale-doc mismatches much easier to spot.
106
+
80
107
  ## Quick Start
81
108
 
82
109
  ```bash
@@ -101,10 +128,62 @@ agent-skill-eval report --workspace ./eval-workspace/write-release-notes-workspa
101
128
  To start a suite for your own skill:
102
129
 
103
130
  ```bash
104
- agent-skill-eval init my-skill
131
+ agent-skill-eval init my-skill --template basic
105
132
  ```
106
133
 
107
- This scaffolds `my-skill/SKILL.md` (frontmatter template), `my-skill/evals/evals.json` (one positive case and one negative control), and `my-skill/evals/files/` for fixtures.
134
+ This scaffolds `my-skill/SKILL.md` (frontmatter template), `my-skill/evals/evals.json`, and `my-skill/evals/files/` for fixtures. Templates are `basic`, `review-diff`, `commit-pr`, and `config-validator`; each includes a negative control and mostly deterministic assertions.
135
+
136
+ ## Mental model
137
+
138
+ agent-skill-eval is a harness for measuring whether a skill changes real agent
139
+ behavior. It is not a raw model benchmark and it is not a static linter for
140
+ `SKILL.md`; it runs the same CLI a user would run, in a prepared workspace, and
141
+ then compares what changed.
142
+
143
+ The core objects are:
144
+
145
+ | Term | Meaning |
146
+ | --- | --- |
147
+ | Skill | A directory containing `SKILL.md` and optional `references/`, `scripts/`, or other support files. This is the thing being evaluated. |
148
+ | Eval suite | One `evals.json` file with `skill_name` and a list of cases for one skill. In plugin repos it usually lives at `plugins/<plugin>/skills/<skill>/evals/evals.json`. |
149
+ | Eval case | One prompt plus fixtures, assertions, trigger settings, and side-effect level. A case should test one behavior or failure mode. |
150
+ | Fixture | A file or directory copied into the agent workspace before the prompt runs. Fixtures make the workspace deterministic. |
151
+ | Workspace | The temporary git checkout the agent sees. It is either a fresh git repo or a clone of `--source-repo`. Every case gets its own workspace so state does not leak between cases. |
152
+ | Agent harness | The adapter that invokes `codex`, `claude`, `opencode`, or `fake` and normalizes output, logs, timing, tokens, and exit status. |
153
+ | Configuration | A named run variant such as `with_skill` or `without_skill` for one agent/model. With `--runs N`, each configuration has repeated `run-1`, `run-2`, ... attempts. |
154
+ | Assertion | A deterministic or LLM-graded expectation. Deterministic assertions inspect git state, files, logs, or JSON; unresolved prose goes to the LLM rubric. |
155
+ | Grading result | The `grading.json` verdict for one run: passed, failed, skipped, evidence, and grading method per assertion. |
156
+ | Side-effect level | A case label: `static`, `local-only`, `live-safe`, or `live-external`. It lets you run cheap or safe cases before any live provider workflow. |
157
+ | Lifecycle hook | A shell command run before the suite, after each run is graded, or after the suite. Hooks are how provider-specific setup, external checks, and teardown are attached. |
158
+ | Cleanup manifest | `cleanup.json`, a scoped list of branches and PRs/MRs created by the run. Cleanup only acts on this manifest. |
159
+ | Summary/report | `summary.json`, `benchmark.json`, `status`, `report`, `report-all`, and optional JUnit/HTML/Markdown output. These are the artifacts you use for decisions. |
160
+
161
+ One normal `run` flows like this:
162
+
163
+ 1. Load the skill and eval suite; apply filters such as `--eval-id` and
164
+ `--side-effect-level`.
165
+ 2. Run `--pre-run-command` hooks. These usually validate the skill text, create
166
+ or reset a scratch source repo, or fail fast before any agent call.
167
+ 3. For each case, create a workspace, clone `--source-repo` or initialize a
168
+ fresh repo, copy fixtures, and install the skill into the agent-specific
169
+ skill path.
170
+ 4. Capture pre-run git/provider state.
171
+ 5. Invoke the real agent CLI with the case prompt. If baseline mode is enabled,
172
+ run both `with_skill` and `without_skill`.
173
+ 6. Capture post-run state, logs, timing, token telemetry, and output.
174
+ 7. Grade deterministic assertions first, then send unresolved assertions to the
175
+ LLM grader when credentials are available.
176
+ 8. Run `--post-grade-command` hooks while the workspace still exists. These can
177
+ append structured assertion results for provider state that the core harness
178
+ cannot know about.
179
+ 9. Delete or keep workspaces according to debug settings, write summaries, and
180
+ run `--post-run-command` hooks for teardown and cleanup verification.
181
+
182
+ Use `force_skill_invocation: true` when debugging whether the skill instructions
183
+ are correct after the skill has already triggered. Use
184
+ `force_skill_invocation: false` for discovery cases that prove the agent chooses
185
+ the skill from a natural request. Use `should_trigger: false` for negative
186
+ controls that prove the skill does not activate on adjacent read-only prompts.
108
187
 
109
188
  ## Defining evals
110
189
 
@@ -151,9 +230,12 @@ A JSON Schema for this format ships at [`schemas/evals.schema.json`](schemas/eva
151
230
  | `expected_output` | str | Reference output for LLM rubric grading |
152
231
  | `files` | list[str] | Fixture file paths (resolved relative to the evals directory) |
153
232
  | `stage_files` | bool | If true, fixture files are also `git add`-ed before the agent runs (default: false) |
154
- | `assertions` | list[str] | Assertions to grade against |
233
+ | `assertions` | list[str \| object] | Prose assertions or structured deterministic assertions to grade against |
155
234
  | `should_trigger` | bool | If false, branch/commit/push/PR assertions are inverted (default: true) |
156
235
  | `force_skill_invocation` | bool | If true, the prompt is prefixed with `Use the $<skill> skill.` (default: false) |
236
+ | `side_effect_level` | str | One of `static`, `local-only`, `live-safe`, `live-external` (default: `local-only`) |
237
+
238
+ Fixture paths are copied into the eval workspace before each run. Direct fixtures under `files/` are staged at the workspace root (`files/config.json` -> `config.json`). Case-scoped fixture folders also become the workspace root when a case uses one prefix (`files/prod-only/bootstrap/...` -> `bootstrap/...`), which lets suites model multi-repo workspaces without long prompt paths. If one eval references multiple case prefixes, those prefixes are preserved to avoid same-name collisions. Unsafe paths and target collisions fail during `validate`.
157
239
 
158
240
  ## How grading works
159
241
 
@@ -172,13 +254,28 @@ The deterministic grader matches assertion text against these patterns (case-ins
172
254
  | `branch` + `created`/`exists`/`new` | A new branch appeared in the state diff and is checked out |
173
255
  | `commit` + `created`/`exists`/`new` | A new commit appeared and is on the current branch |
174
256
  | `push` + `remote`/`branch`/`pushed` | The eval-created branch was pushed AND remote HEAD matches local HEAD |
175
- | `pr` or `pull request` | A new open PR targets the eval-created branch (corroborated by `gh pr view` when available) |
257
+ | `pr` or `pull request` | A new open PR/MR targets the eval-created branch (corroborated by `gh pr view` for GitHub when available) |
176
258
  | `file exists` or `created` + a filename | File matching the backticked/quoted name exists in the workspace |
177
259
  | `ran` + a command name (`npm`, `git`, ...) | Command name appears in the run logs |
178
260
  | `contains` or `includes` + `"quoted"`/`` `backticked` `` text | Agent output contains the text |
179
261
  | `valid json` | Agent output (or a workspace file) parses as JSON |
180
262
 
181
- Anything else falls through to the LLM rubric. With `should_trigger: false`, the branch/commit/push/PR checks invert: they pass only when those artifacts did **not** appear.
263
+ Anything else falls through to the LLM rubric. With `should_trigger: false`, the branch/commit/push/PR/MR checks invert: they pass only when those artifacts did **not** appear.
264
+
265
+ ### Structured deterministic assertions
266
+
267
+ For checks that should never fall through to the LLM rubric, use assertion objects:
268
+
269
+ ```json
270
+ {
271
+ "type": "file_contains",
272
+ "path": "README.md",
273
+ "contains": "Quickstart",
274
+ "text": "README includes a Quickstart section"
275
+ }
276
+ ```
277
+
278
+ Supported `type` values are `file_exists`, `file_not_exists`, `file_contains`, `file_not_contains`, `file_unchanged`, `command_ran`, `command_not_ran`, `tests_pass`, `git_clean`, `json_path`, and `json_path_not_exists`. Common fields are `path`, `contains`, `command`, `json_path`, `equals`, and optional human-facing `text`.
182
279
 
183
280
  ## CLI Commands
184
281
 
@@ -195,6 +292,7 @@ agent-skill-eval run \
195
292
  [--baseline/--no-baseline] [--cleanup] \
196
293
  [--timeout 600] [--retries 1] \
197
294
  [--eval-id wif-too-long-stop] \
295
+ [--side-effect-level static] \
198
296
  [--max-total-tokens-per-case 200000] [--budget-action fail] \
199
297
  [--pre-run-command "python scripts/setup.py"] \
200
298
  [--post-grade-command "python scripts/grade_external.py"] \
@@ -213,6 +311,7 @@ Key options:
213
311
  - `--runs, -n`: repeat each (eval, agent, config) N times; enables pass@k stats. Results land in `run-1/`, `run-2/`, ... subdirectories.
214
312
  - `--timeout` / `--retries`: per-run agent timeout (default 600s) and retries on timeout or non-zero exit (default 1). Also settable via `ASE_AGENT_TIMEOUT` / `ASE_AGENT_RETRIES`.
215
313
  - `--eval-id`: run only the named case(s); repeatable. Unknown ids fail fast with the list of available ids, before any model call. The applied filter is recorded in `evals_meta.json` (`selected_eval_ids`) and `summary.json` (`eval_ids`), and all artifacts naturally describe only the selected cases. The cheap inner loop after a small skill edit is `validate --eval-id X` then `run --eval-id X`.
314
+ - `--side-effect-level`: run only the requested side-effect level(s); repeatable. Use this to run `static` and `local-only` checks before deliberate live-provider suites.
216
315
  - Budget guards (`--max-input-tokens-per-case`, `--max-non-cached-input-tokens-per-case`, `--max-output-tokens-per-case`, `--max-total-tokens-per-case`, `--max-reasoning-tokens-per-case`, `--max-cost-per-case`, `--max-duration-per-case`, `--budget-action warn|fail|stop-suite`): per-case limits checked after each agent run (mid-run kills remain `--timeout`'s job — budgets exist because a case can finish within the timeout and still be far too expensive). A violation is recorded in `timing.json` (`budget_exceeded`, `budget_reason`); with `fail` (the default) it also appends a failed `method: "budget"` assertion to the run's `grading.json`, and with `stop-suite` it additionally skips every run that hasn't started yet — useful to stop a live side-effect suite before more remote branches/PRs get created. `--max-cost-per-case` only fires when the run has a cost source (see Cost reporting); likewise `--max-reasoning-tokens-per-case` and `--max-non-cached-input-tokens-per-case` only fire when the run reported those values — an agent exposing less telemetry is never failed over an unknown quantity.
217
316
  - Lifecycle hooks (`--pre-run-command`, `--post-grade-command`, `--post-run-command`): see [Lifecycle hooks](#lifecycle-hooks).
218
317
  - `--pricing-config`: JSON file with USD-per-token rates for agents whose CLI reports no cost (see Cost reporting).
@@ -222,10 +321,47 @@ While the suite runs, each case prints start/finish lines (elapsed, tokens in/ca
222
321
  ### `report`
223
322
 
224
323
  ```bash
225
- agent-skill-eval report --workspace <skill-workspace> [--iteration N] [--format table|markdown] [--show-evidence] [--failures-only]
324
+ agent-skill-eval report --workspace <skill-workspace> [--iteration N] [--format table|markdown|html] [--show-evidence] [--failures-only]
226
325
  ```
227
326
 
228
- `--show-evidence` prints the evidence string for every failed or skipped assertion — the state diff for deterministic checks, the judge's reasoning for LLM checks. `--failures-only` limits the per-eval detail to runs that had at least one failed assertion. `--format markdown` emits a paste-ready table.
327
+ `--show-evidence` prints the evidence string for every failed or skipped assertion — the state diff for deterministic checks, the judge's reasoning for LLM checks. `--failures-only` limits the per-eval detail to runs that had at least one failed assertion. `--format markdown` emits a paste-ready table; `--format html` emits a standalone static HTML report.
328
+
329
+ ### `run-all`
330
+
331
+ ```bash
332
+ agent-skill-eval run-all \
333
+ --root path/to/plugin-repo --agent codex --side-effect-level static \
334
+ [--source-repo https://github.com/foo/bar.git] \
335
+ [--max-total-tokens-per-case 200000] [--budget-action fail] \
336
+ [--pre-run-command "python scripts/setup.py"] \
337
+ [--post-grade-command "python scripts/grade_external.py"] \
338
+ [--post-run-command "python scripts/teardown.py"] \
339
+ [--pricing-config pricing.json] \
340
+ [--suite-concurrency 4] \
341
+ [--plan-only] \
342
+ [--changed-only --base-ref origin/main] \
343
+ [--junit-xml run-all.xml] [--fail-on-assertions] [--min-pass-rate 0.8]
344
+ ```
345
+
346
+ Runs every paired `SKILL.md` + `evals/evals.json` suite discovered under a repo by invoking the normal single-suite runner once per skill. Suites with no cases for the requested side-effect filter are recorded as skipped in `run-all-iteration-N.json`, which makes `static` or `local-only` repo sweeps safe before any live-provider evals.
347
+
348
+ Repo sweeps accept the same source-repo, model, timeout/retry, budget, pricing, and lifecycle hook controls as `run`; the aggregate manifest records the source repo, budget, pricing config path, suite concurrency, and hook counts used for the sweep. Use `--plan-only` to validate and write the planned/skipped suite manifest without running agents, `--suite-concurrency` to run independent skill suites in parallel while preserving deterministic manifest ordering, `--changed-only --base-ref origin/main` in CI to run only skills or eval suites touched by the branch, `--junit-xml` to publish the sweep as test results, and `--fail-on-assertions` or `--min-pass-rate` to make quality gates explicit.
349
+
350
+ ### `report-all`
351
+
352
+ ```bash
353
+ agent-skill-eval report-all --workspace ./eval-workspace [--format table|markdown|html|json] [--failures-only]
354
+ ```
355
+
356
+ Summarizes a repo-level `run-all` manifest with suite status, pass rates, duration, token totals, cost totals, and failure details. Use `--format markdown` for MR notes, `--format html` for a static artifact, and `--format json` when another tool should consume the raw manifest.
357
+
358
+ ### `rerun-failed`
359
+
360
+ ```bash
361
+ agent-skill-eval rerun-failed --from ./eval-workspace/run-all-iteration-1.json
362
+ ```
363
+
364
+ Prints targeted `agent-skill-eval run --eval-id ...` commands for failed repo-sweep cases. The generated commands preserve run context recorded in the manifest, including grader settings, agent model, source repo, pricing config, timeout/retry, reasoning effort, and budget flags.
229
365
 
230
366
  ### `status`
231
367
 
@@ -249,15 +385,33 @@ Side-by-side pass rates of two iterations, per configuration, with pass-rate, ti
249
385
  agent-skill-eval validate path/to/evals.json [--eval-id <id>]
250
386
  ```
251
387
 
252
- Schema check plus referenced-fixture existence and duplicate-id detection. Exit code 1 on any problem (CI-friendly). `--eval-id` validates only the selected case(s) (duplicate-id detection still covers the whole file); pair it with `run --eval-id` to confirm a targeted case is well-formed before spending a model call on it.
388
+ Schema check plus referenced-fixture existence, duplicate-id detection, and structured-assertion required-field/path checks. Exit code 1 on any problem (CI-friendly). `--eval-id` validates only the selected case(s) (duplicate-id detection still covers the whole file); pair it with `run --eval-id` to confirm a targeted case is well-formed before spending a model call on it.
389
+
390
+ ### `validate-all`
391
+
392
+ ```bash
393
+ agent-skill-eval validate-all --root path/to/repo [--format table|json]
394
+ ```
395
+
396
+ Recursively validates every discovered `*/evals/evals.json` under a repo and exits non-zero if any suite is invalid. This is the CI-friendly check for plugin repos with many skills.
397
+ For plugin-style paths such as `plugins/<plugin>/skills/<skill>/evals/evals.json`, it also verifies the adjacent `SKILL.md` frontmatter and that the suite `skill_name` matches the skill directory.
398
+
399
+ ### `doctor`
400
+
401
+ ```bash
402
+ agent-skill-eval doctor --skill ./skills/my-skill --evals ./evals/evals.json --agent codex
403
+ ```
404
+
405
+ Preflights the local agent CLI, skill frontmatter, eval suite, LLM-grader credentials, and source-repo tooling (`gh` for GitHub, `glab` for GitLab). Use it before live or expensive suites to catch missing tools and auth early.
253
406
 
254
407
  ### `list`
255
408
 
256
409
  ```bash
257
- agent-skill-eval list [--root .]
410
+ agent-skill-eval list [--root .] [--format table|json]
258
411
  ```
259
412
 
260
- Discovers eval suites (`evals.json`) and skills (`SKILL.md`) under a directory.
413
+ Recursively discovers eval suites (`*/evals/evals.json`) and skills (`SKILL.md`) under a directory, including plugin-style layouts such as `plugins/<plugin>/skills/<skill>/evals/evals.json`.
414
+ JSON output includes a `pairs` section showing paired skills, skills without evals, and external eval suites.
261
415
 
262
416
  ### `grade`
263
417
 
@@ -273,14 +427,18 @@ Re-grades existing outputs using saved `evals_meta.json` and state snapshots. Tw
273
427
  agent-skill-eval cleanup --workspace ./eval-workspace [--yes]
274
428
  ```
275
429
 
276
- Only closes PRs and deletes remote branches recorded in `cleanup.json`. Never touches unrelated PRs, branches, or workspaces.
430
+ Only closes PRs/MRs and deletes remote branches recorded in `cleanup.json`. Never touches unrelated review requests, branches, or workspaces.
277
431
 
278
432
  ### `init`
279
433
 
280
434
  ```bash
281
- agent-skill-eval init my-skill [--output ./examples]
435
+ agent-skill-eval init my-skill [--output ./examples] [--template basic|review-diff|commit-pr|config-validator]
436
+ agent-skill-eval init my-skill --layout plugin --plugin demo-plugin --output /path/to/plugin-repo
282
437
  ```
283
438
 
439
+ Use `commit-pr` for live branch/push/PR workflows, `review-diff` for read-only review skills, and `config-validator` for skills that inspect bundled config fixtures.
440
+ Use `--layout plugin` to scaffold directly into `plugins/<plugin>/skills/<skill>/evals` for Codex plugin repositories.
441
+
284
442
  ## Lifecycle hooks
285
443
 
286
444
  Skills with external side effects (push, PR/MR creation, provider APIs) need setup, provider-specific grading, and teardown that the harness can't know about. Instead of hard-coding providers, `run` exposes three shell-command hook points; each receives run metadata as `ASE_*` environment variables on top of the parent environment:
@@ -299,6 +457,151 @@ agent-skill-eval run --skill ./skills/my-skill --evals ./evals/evals.json --agen
299
457
  --post-run-command "python scripts/teardown_remote.py"
300
458
  ```
301
459
 
460
+ ## Live side-effect cookbook
461
+
462
+ Use this pattern for skills that create external state: pushes, PRs/MRs, Jira
463
+ issues, cloud resources, provider API changes, tickets, comments, or anything
464
+ that cannot be fully judged from the local workspace.
465
+
466
+ ### 1. Classify cases by blast radius
467
+
468
+ Set `side_effect_level` on every case:
469
+
470
+ | Level | Use for |
471
+ | --- | --- |
472
+ | `static` | Prompt-shaping, validation, "ask for missing input", and no-tool checks. No local git writes or provider calls. |
473
+ | `local-only` | Local file edits, local git branches/commits, dry-run provider payloads, or live calls only to isolated scratch projects that are fully cleaned by hooks. |
474
+ | `live-safe` | Live provider state in a controlled test project where setup and teardown are reliable and the effect is reversible. |
475
+ | `live-external` | Shared, customer, production, paid, or human-visible systems. Run only with explicit intent and tight filters. |
476
+
477
+ Start with `static` and `local-only`. Add `live-safe` only after the local cases
478
+ prove the workflow shape. Treat `live-external` as a release candidate check,
479
+ not an inner loop.
480
+
481
+ ### 2. Make the source repo disposable
482
+
483
+ For git/MR workflows, use `--source-repo` pointing at a scratch repository that
484
+ the eval owns. The pre-run hook should reset it to a known base state before
485
+ each suite or case:
486
+
487
+ ```bash
488
+ agent-skill-eval run \
489
+ --skill ./plugins/acme/skills/commit-push-mr \
490
+ --evals ./plugins/acme/skills/commit-push-mr/evals/evals.json \
491
+ --source-repo git@gitlab.example.com:me/agent-skill-eval-scratch.git \
492
+ --agent codex \
493
+ --side-effect-level local-only \
494
+ --eval-id wif-too-long-stop \
495
+ --pre-run-command "python plugins/acme/skills/commit-push-mr/evals/scripts/setup_source_repo.py" \
496
+ --post-grade-command "python plugins/acme/skills/commit-push-mr/evals/scripts/grade_gitlab.py" \
497
+ --post-run-command "python plugins/acme/skills/commit-push-mr/evals/scripts/teardown_gitlab.py" \
498
+ --post-run-command "python plugins/acme/skills/commit-push-mr/evals/scripts/verify_gitlab_clean.py"
499
+ ```
500
+
501
+ The scratch repo should contain only eval fixtures and expected provider state.
502
+ Do not point live evals at the real project until the suite has already passed
503
+ against scratch state.
504
+
505
+ ### 3. Keep setup, grading, teardown separate
506
+
507
+ Use one script for each job:
508
+
509
+ - Setup hook: creates or resets scratch state, checks auth, validates that the
510
+ target is the expected test project, and aborts before model calls if anything
511
+ is unsafe.
512
+ - Post-grade hook: reads `ASE_*` paths, inspects provider state while the agent
513
+ workspace still exists, and prints JSON assertion rows to stdout.
514
+ - Teardown hook: closes/deletes only expected branches, MRs, tickets, labels, or
515
+ resources created by this suite.
516
+ - Verify-clean hook: read-only proof that no expected eval branch, MR, ticket, or
517
+ resource remains. Run it after teardown and independently after failures.
518
+
519
+ Post-grade hooks can append checks to `grading.json` by printing:
520
+
521
+ ```json
522
+ [
523
+ {
524
+ "text": "Exactly one GitLab MR was opened for the eval branch",
525
+ "passed": true,
526
+ "evidence": "MR !42 targets main and source branch proj-123-eval"
527
+ }
528
+ ]
529
+ ```
530
+
531
+ This keeps external verification in the same report as deterministic file/git
532
+ checks and LLM rubric checks.
533
+
534
+ ### 4. Write cases around failure modes, not just happy paths
535
+
536
+ A good side-effect suite usually has these case types:
537
+
538
+ - Happy path: creates the intended branch/change/review request/resource in the
539
+ scratch provider.
540
+ - Guardrail stop: invalid branch name, missing ticket ID, missing required
541
+ input, unsupported environment, or too-large payload stops before side
542
+ effects.
543
+ - No-op stop: no meaningful local changes means no commit, push, ticket, or API
544
+ write.
545
+ - Natural trigger: same workflow without `force_skill_invocation`, proving the
546
+ agent selects the skill from a realistic user request.
547
+ - Negative control: adjacent read-only prompt with `should_trigger: false`,
548
+ proving no side effects are created.
549
+ - Cleanup regression: provider state is clean after teardown, even when the
550
+ agent failed earlier.
551
+
552
+ For each side effect, pair a positive assertion with an explicit "nothing else"
553
+ check in the provider hook. For example: one MR for the eval branch, no MR for
554
+ `main`, no duplicate MR, no unexpected branch, and no untracked provider
555
+ resource.
556
+
557
+ ### 5. Use a cheap inner loop
558
+
559
+ Do not make full live runs your edit loop. Use this order:
560
+
561
+ ```bash
562
+ agent-skill-eval validate path/to/evals.json --eval-id wif-too-long-stop
563
+ agent-skill-eval doctor --skill path/to/skill --evals path/to/evals.json --agent codex
564
+ agent-skill-eval run --skill path/to/skill --evals path/to/evals.json \
565
+ --agent codex --eval-id wif-too-long-stop --side-effect-level local-only \
566
+ --no-baseline --max-total-tokens-per-case 150000 --budget-action fail
567
+ agent-skill-eval status --workspace ./eval-workspace/<skill>-workspace
568
+ agent-skill-eval report --workspace ./eval-workspace/<skill>-workspace --show-evidence
569
+ ```
570
+
571
+ Only after a targeted case passes should you run more cases, more agents,
572
+ baseline mode, or `run-all`.
573
+
574
+ For plugin repos with many skills, use planning before execution:
575
+
576
+ ```bash
577
+ agent-skill-eval validate-all --root /path/to/plugin-repo --strict
578
+ agent-skill-eval run-all --root /path/to/plugin-repo \
579
+ --side-effect-level static --changed-only --base-ref origin/main --plan-only
580
+ agent-skill-eval run-all --root /path/to/plugin-repo \
581
+ --side-effect-level local-only --changed-only --base-ref origin/main \
582
+ --suite-concurrency 2 --fail-on-assertions
583
+ ```
584
+
585
+ ### 6. Use budgets and cleanup as gates
586
+
587
+ Side-effect suites need explicit failure gates:
588
+
589
+ - Set `--timeout`, `--retries`, `--max-total-tokens-per-case`,
590
+ `--max-duration-per-case`, and usually `--budget-action fail` or
591
+ `stop-suite`.
592
+ - Use `--eval-id` while debugging a single risky case.
593
+ - Keep `--cleanup` or teardown hooks enabled for any run that can push, open an
594
+ MR, or write provider state.
595
+ - Run a read-only verify-clean command after failures before starting another
596
+ live run.
597
+ - Archive or paste `status`, `report --show-evidence`, and provider hook
598
+ evidence when claiming a skill improved.
599
+
600
+ The minimum credible claim for a side-effect skill is: targeted case passed,
601
+ provider hook passed, cleanup verification passed, and the command/model/harness
602
+ version are recorded. A stronger claim adds repeated runs, natural trigger and
603
+ negative-control cases, and a before/after comparison with `compare`.
604
+
302
605
  ## Example skills
303
606
 
304
607
  Five example skills ship with the repo, chosen to exercise different grading surfaces:
@@ -321,7 +624,7 @@ Each has a matching eval suite under [`examples/`](examples/). `skills/` holds t
321
624
  eval-workspace/
322
625
  └── <skill>-workspace/
323
626
  └── iteration-1/
324
- ├── evals_meta.json # eval definitions + selected_eval_ids (used by `grade`)
627
+ ├── evals_meta.json # eval definitions, selected filters, skill_hash (used by `grade`)
325
628
  ├── cleanup.json # manifest of artifacts created by this run
326
629
  ├── benchmark.json # aggregate stats per (agent, config)
327
630
  ├── summary.json # one-file run summary (used by `status`)
@@ -335,7 +638,8 @@ eval-workspace/
335
638
  │ └── pre_state.json / post_state.json
336
639
  ├── timing.json # tokens (incl. non-cached/reasoning splits), duration, exit_code, timed_out, retries, budget verdict
337
640
  ├── grading.json # per-assertion results
338
- └── run_meta.json # agent, with_skill, run_index, model, reasoning_effort, base_url, agent_cli_version, harness_version, ...
641
+ ├── trajectory.json # normalized lifecycle events for this run
642
+ └── run_meta.json # agent, with_skill, run_index, model, side_effect_level, skill_hash, CLI/harness versions, ...
339
643
  ```
340
644
 
341
645
  ### grading.json
@@ -424,7 +728,7 @@ Codex's CLI reports token counts but no cost. To get cost numbers anyway, pass `
424
728
  }
425
729
  ```
426
730
 
427
- The computed value lands in `cost_usd` with `cost_usd_source: "pricing-config"`. Cached input is billed at `input_cache_read` (falling back to `prompt`); note that codex and opencode report `input_tokens` *including* cache reads while claude-code reports them separately — the harness accounts for this per agent. `summary.json` splits the totals (`tokens.cached`, `tokens.non_cached_input`, `tokens.cached_pct`), because on long agent sessions most input tokens are cache reads and the raw `input_tokens` number wildly overstates billable spend. Each run's `timing.json` also persists `non_cached_input_tokens` (computed with that run's CLI semantics) and `reasoning_output_tokens` when the agent reports reasoning telemetry (codex `turn.completed`, opencode `step_finish`) — `null` means unreported, not zero.
731
+ The computed value lands in `cost_usd` with `cost_usd_source: "pricing-config"`. Cached input is billed at `input_cache_read` (falling back to `prompt`); note that codex reports `input_tokens` *including* cache reads while claude-code and opencode report them separately — the harness accounts for this per agent. `summary.json` splits the totals (`tokens.cached`, `tokens.non_cached_input`, `tokens.cached_pct`), because on long agent sessions most input tokens are cache reads and the raw `input_tokens` number wildly overstates billable spend. Each run's `timing.json` also persists `non_cached_input_tokens` (computed with that run's CLI semantics) and `reasoning_output_tokens` when the agent reports reasoning telemetry (codex `turn.completed`, opencode `step_finish`) — `null` means unreported, not zero.
428
732
 
429
733
  One sharp edge: the claude CLI prices runs at **Anthropic list prices** regardless of the endpoint it talks to. When you route claude-code through OpenRouter (`--harness-base-url` or `ANTHROPIC_BASE_URL`), agent-skill-eval reconciles the cost by recomputing it from the run's token counts and OpenRouter's published per-model rates. `timing.json` then shows:
430
734