agent-skill-eval 0.6.0__tar.gz → 0.7.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/CHANGELOG.md +62 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/PKG-INFO +322 -18
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/README.md +321 -17
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/commit-push-pr/evals/evals.json +6 -3
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/pyproject.toml +2 -1
- agent_skill_eval-0.7.0/schemas/evals.schema.json +237 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/skills/validate-config/scripts/validate_config.py +1 -4
- agent_skill_eval-0.7.0/src/agent_skill_eval/cli.py +3600 -0
- agent_skill_eval-0.7.0/src/agent_skill_eval/fixtures.py +63 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/git_state.py +125 -3
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/graders/__init__.py +525 -67
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/harnesses/__init__.py +16 -2
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/models.py +74 -3
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/runner.py +161 -19
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/skills.py +67 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/summary.py +72 -3
- agent_skill_eval-0.7.0/src/agent_skill_eval/validation.py +224 -0
- agent_skill_eval-0.7.0/tests/test_cli_smoke.py +871 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_feedback_features.py +144 -0
- agent_skill_eval-0.7.0/tests/test_fixtures.py +34 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_git_state.py +91 -1
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_graders.py +144 -2
- agent_skill_eval-0.7.0/tests/test_launch_features.py +970 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_llm_grader.py +155 -2
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_runner.py +46 -5
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_skills.py +15 -0
- agent_skill_eval-0.6.0/schemas/evals.schema.json +0 -83
- agent_skill_eval-0.6.0/src/agent_skill_eval/cli.py +0 -1136
- agent_skill_eval-0.6.0/tests/test_cli_smoke.py +0 -92
- agent_skill_eval-0.6.0/tests/test_launch_features.py +0 -489
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/.github/ISSUE_TEMPLATE/bug_report.yml +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/.github/ISSUE_TEMPLATE/feature_request.yml +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/.github/PULL_REQUEST_TEMPLATE.md +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/.github/actions/setup/action.yml +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/.github/workflows/ci.yml +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/.github/workflows/release.yml +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/.gitignore +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/AGENTS.md +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/CODE_OF_CONDUCT.md +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/CONTRIBUTING.md +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/LICENSE +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/Makefile +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/ROADMAP.md +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/TODO.md +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/config/.env.example +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/config/baseline.env +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/commit-push-pr/evals/files/sample_change.py +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/fix-failing-tests/evals/evals.json +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/fix-failing-tests/evals/files/calculator.py +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/fix-failing-tests/evals/files/conftest.py +1 -1
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/fix-failing-tests/evals/files/test_calculator.py +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/review-diff/evals/evals.json +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/review-diff/evals/files/changes.diff +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/review-diff/evals/files/orders.py +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/validate-config/evals/evals.json +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/validate-config/evals/files/app_config.json +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/write-release-notes/evals/evals.json +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/examples/write-release-notes/evals/files/commits.txt +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/scripts/record-demo.sh +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/skills/commit-push-pr/SKILL.md +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/skills/fix-failing-tests/SKILL.md +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/skills/review-diff/SKILL.md +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/skills/validate-config/SKILL.md +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/skills/validate-config/references/config-rules.md +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/skills/write-release-notes/SKILL.md +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/__init__.py +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/__main__.py +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/hooks.py +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/openrouter.py +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/progress.py +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/src/agent_skill_eval/workspace.py +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/__init__.py +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/conftest.py +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/fixtures/evals/format-json.json +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/fixtures/skills/format-json/SKILL.md +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_e2e.py +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_fuzz_grader.py +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_harnesses.py +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_openrouter.py +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_skill_validation.py +0 -0
- {agent_skill_eval-0.6.0 → agent_skill_eval-0.7.0}/tests/test_workspace.py +0 -0
|
@@ -6,6 +6,68 @@ from conventional commits; entries below it are curated by hand.
|
|
|
6
6
|
|
|
7
7
|
<!-- version list -->
|
|
8
8
|
|
|
9
|
+
## v0.7.0 (2026-07-10)
|
|
10
|
+
|
|
11
|
+
### Features
|
|
12
|
+
|
|
13
|
+
- Run-all multi-suite harness, validation, and reporting suite
|
|
14
|
+
([`a282bc9`](https://github.com/tardigrde/agent-skill-eval/commit/a282bc9e6d13b0cc4ffa3bb0dec2e02c695313d5))
|
|
15
|
+
|
|
16
|
+
Comprehensive update folding together the run-all multi-suite layer, static validation, expanded
|
|
17
|
+
reporting, and side-effect contract enforcement.
|
|
18
|
+
|
|
19
|
+
Run-all / multi-suite: - concurrent eval-suite execution with paired skill<->suite discovery -
|
|
20
|
+
recursive eval discovery, changed-only mode, plugin eval layout - dry-run planning without agent
|
|
21
|
+
calls, planned-case preview, plan totals - cost/duration totals, pass-rate gating, JUnit reports -
|
|
22
|
+
failed-case rerun commands with context preservation - artifact archive + cross-run compare
|
|
23
|
+
|
|
24
|
+
Validation & assertions: - strict eval-suite linting and structured-field validation - negative
|
|
25
|
+
artifact assertions, command-assertion false-positive fix - run-all assertion-failure reporting
|
|
26
|
+
|
|
27
|
+
Grading: - constrained OpenWebUI grader backend (localhost default)
|
|
28
|
+
|
|
29
|
+
Side effects & reporting: - enforced side-effect contracts, GitLab cleanup manifests - static HTML
|
|
30
|
+
reports, skill portfolio report - skill-hash + run-trajectory recording - doctor command, JSON
|
|
31
|
+
discovery output
|
|
32
|
+
|
|
33
|
+
New modules: fixtures.py, validation.py.
|
|
34
|
+
|
|
35
|
+
Corporate-specific fixtures removed; tool is fully generic. Default LLM grader backend is OpenRouter
|
|
36
|
+
(public).
|
|
37
|
+
|
|
38
|
+
Co-Authored-By: Claude <noreply@anthropic.com>
|
|
39
|
+
|
|
40
|
+
|
|
41
|
+
## v0.6.1 (2026-06-12)
|
|
42
|
+
|
|
43
|
+
### Bug Fixes
|
|
44
|
+
|
|
45
|
+
- Opencode non-cached input semantics + LLM grader retry on dropped assertions
|
|
46
|
+
([`f0939bc`](https://github.com/tardigrde/agent-skill-eval/commit/f0939bc2b6a314d9fa8d62084fe1b57ed4280974))
|
|
47
|
+
|
|
48
|
+
OpenCode reports input_tokens excluding cache reads (live runs show cached consistently larger than
|
|
49
|
+
input), so the codex-style subtraction clamped non_cached_input_tokens to a bogus 0 and capped
|
|
50
|
+
cached_pct at 100%. Override the split like claude-code.
|
|
51
|
+
|
|
52
|
+
The LLM judge intermittently drops an assertion from its JSON answer, which skipped the assertion
|
|
53
|
+
and failed strict e2e runs. Retry once with only the dropped assertions before skipping; matched
|
|
54
|
+
results are now also deduplicated and filtered to the requested assertions so a retry cannot
|
|
55
|
+
double-count.
|
|
56
|
+
|
|
57
|
+
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|
58
|
+
|
|
59
|
+
- Tolerate malformed judge rows, correct cached_pct denominator (review feedback)
|
|
60
|
+
([`45b2677`](https://github.com/tardigrde/agent-skill-eval/commit/45b26777fb7466f3973a339e0496219d8ecd4778))
|
|
61
|
+
|
|
62
|
+
A malformed individual result row (non-dict, missing keys, non-string text) now only loses that row
|
|
63
|
+
— its assertion goes to the retry instead of crashing the whole attempt and skipping everything.
|
|
64
|
+
|
|
65
|
+
cached_pct now divides by non_cached + cached, the only denominator valid for both reporting styles;
|
|
66
|
+
cached / max(input, cached) showed a bogus 100% for claude-code/opencode.
|
|
67
|
+
|
|
68
|
+
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|
69
|
+
|
|
70
|
+
|
|
9
71
|
## v0.6.0 (2026-06-12)
|
|
10
72
|
|
|
11
73
|
### Bug Fixes
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: agent-skill-eval
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.7.0
|
|
4
4
|
Summary: Framework for evaluating agent skills across OpenCode, Claude Code, and Codex
|
|
5
5
|
Project-URL: Homepage, https://github.com/tardigrde/agent-skill-eval
|
|
6
6
|
Project-URL: Repository, https://github.com/tardigrde/agent-skill-eval
|
|
@@ -42,7 +42,7 @@ Because the eval goes through the full harness — system prompt, skill discover
|
|
|
42
42
|
- **Real harnesses, end to end**: OpenCode, Claude Code, and Codex CLIs, in a single run
|
|
43
43
|
- **Baseline comparison**: with-skill vs. without-skill, with a per-agent delta
|
|
44
44
|
- **pass@k**: `--runs N` repeats every eval and reports full-pass rate and pass@k, because agents are stochastic and single-run numbers lie
|
|
45
|
-
- **State-delta grading**: code-based checks compare pre/post git state snapshots, so they don't false-pass on pre-existing branches, commits, or
|
|
45
|
+
- **State-delta grading**: code-based checks compare pre/post git state snapshots, so they don't false-pass on pre-existing branches, commits, or review requests
|
|
46
46
|
- **Negative controls**: `should_trigger: false` inverts assertions to catch accidental skill triggering
|
|
47
47
|
- **Honest grading**: assertions the grader can't check are *skipped*, not failed; a missing API key warns upfront instead of silently zeroing your pass rate
|
|
48
48
|
- **Pinned models**: `--agent-model claude-code=haiku --agent-model codex=gpt-5-mini` makes runs reproducible across machines
|
|
@@ -53,7 +53,7 @@ Because the eval goes through the full harness — system prompt, skill discover
|
|
|
53
53
|
- **Live progress**: per-case console lines plus a tailable `progress.jsonl` — long runs are no longer opaque
|
|
54
54
|
- **Lifecycle hooks**: `--pre-run-command`, `--post-grade-command`, `--post-run-command` plug external setup, side-effect graders, and teardown into the run with structured `ASE_*` metadata
|
|
55
55
|
- **One-file summaries**: every run writes `summary.json` (pass rates, failures, token/cost totals, cleanup state); `agent-skill-eval status` reads it back without a model call
|
|
56
|
-
- **Scoped cleanup**: only removes artifacts recorded in `cleanup.json`; never closes unrelated PRs or deletes unrelated branches
|
|
56
|
+
- **Scoped cleanup**: only removes artifacts recorded in `cleanup.json`; never closes unrelated PRs/MRs or deletes unrelated branches
|
|
57
57
|
- **Re-grading**: `agent-skill-eval grade` re-grades saved outputs without re-running agents
|
|
58
58
|
- **Markdown reports**: paste `agent-skill-eval report --format markdown` straight into a PR or blog post
|
|
59
59
|
|
|
@@ -77,6 +77,33 @@ uv venv && uv pip install -e ".[dev]"
|
|
|
77
77
|
|
|
78
78
|
You also need the agent CLIs you want to evaluate (`claude`, `codex`, `opencode`) installed and authenticated, and an `OPENROUTER_API_KEY` or `OPENAI_API_KEY` for LLM rubric grading.
|
|
79
79
|
|
|
80
|
+
## Version and documentation status
|
|
81
|
+
|
|
82
|
+
This README documents the source tree it lives in. If you installed
|
|
83
|
+
`agent-skill-eval` from PyPI or a different Git branch, check the installed CLI
|
|
84
|
+
before copying a command:
|
|
85
|
+
|
|
86
|
+
```bash
|
|
87
|
+
agent-skill-eval --version
|
|
88
|
+
agent-skill-eval --help
|
|
89
|
+
agent-skill-eval run-all --help
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
During active development, the newest CLI flags and workflow docs may exist only
|
|
93
|
+
on a local feature branch until that branch is pushed and released. To use a
|
|
94
|
+
local checkout explicitly, run through that checkout instead of whatever command
|
|
95
|
+
is first on `PATH`:
|
|
96
|
+
|
|
97
|
+
```bash
|
|
98
|
+
uv run --project /path/to/skill-evals agent-skill-eval run-all --help
|
|
99
|
+
uv run --project /path/to/skill-evals python -m agent_skill_eval validate-all --root .
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
When sharing results, record the command, `agent-skill-eval --version`, agent
|
|
103
|
+
model, reasoning effort, and branch/commit of the harness repo. The harness also
|
|
104
|
+
writes most of that to `run_meta.json` and `summary.json`, but copying it into a
|
|
105
|
+
handoff or PR description makes stale-doc mismatches much easier to spot.
|
|
106
|
+
|
|
80
107
|
## Quick Start
|
|
81
108
|
|
|
82
109
|
```bash
|
|
@@ -101,10 +128,62 @@ agent-skill-eval report --workspace ./eval-workspace/write-release-notes-workspa
|
|
|
101
128
|
To start a suite for your own skill:
|
|
102
129
|
|
|
103
130
|
```bash
|
|
104
|
-
agent-skill-eval init my-skill
|
|
131
|
+
agent-skill-eval init my-skill --template basic
|
|
105
132
|
```
|
|
106
133
|
|
|
107
|
-
This scaffolds `my-skill/SKILL.md` (frontmatter template), `my-skill/evals/evals.json
|
|
134
|
+
This scaffolds `my-skill/SKILL.md` (frontmatter template), `my-skill/evals/evals.json`, and `my-skill/evals/files/` for fixtures. Templates are `basic`, `review-diff`, `commit-pr`, and `config-validator`; each includes a negative control and mostly deterministic assertions.
|
|
135
|
+
|
|
136
|
+
## Mental model
|
|
137
|
+
|
|
138
|
+
agent-skill-eval is a harness for measuring whether a skill changes real agent
|
|
139
|
+
behavior. It is not a raw model benchmark and it is not a static linter for
|
|
140
|
+
`SKILL.md`; it runs the same CLI a user would run, in a prepared workspace, and
|
|
141
|
+
then compares what changed.
|
|
142
|
+
|
|
143
|
+
The core objects are:
|
|
144
|
+
|
|
145
|
+
| Term | Meaning |
|
|
146
|
+
| --- | --- |
|
|
147
|
+
| Skill | A directory containing `SKILL.md` and optional `references/`, `scripts/`, or other support files. This is the thing being evaluated. |
|
|
148
|
+
| Eval suite | One `evals.json` file with `skill_name` and a list of cases for one skill. In plugin repos it usually lives at `plugins/<plugin>/skills/<skill>/evals/evals.json`. |
|
|
149
|
+
| Eval case | One prompt plus fixtures, assertions, trigger settings, and side-effect level. A case should test one behavior or failure mode. |
|
|
150
|
+
| Fixture | A file or directory copied into the agent workspace before the prompt runs. Fixtures make the workspace deterministic. |
|
|
151
|
+
| Workspace | The temporary git checkout the agent sees. It is either a fresh git repo or a clone of `--source-repo`. Every case gets its own workspace so state does not leak between cases. |
|
|
152
|
+
| Agent harness | The adapter that invokes `codex`, `claude`, `opencode`, or `fake` and normalizes output, logs, timing, tokens, and exit status. |
|
|
153
|
+
| Configuration | A named run variant such as `with_skill` or `without_skill` for one agent/model. With `--runs N`, each configuration has repeated `run-1`, `run-2`, ... attempts. |
|
|
154
|
+
| Assertion | A deterministic or LLM-graded expectation. Deterministic assertions inspect git state, files, logs, or JSON; unresolved prose goes to the LLM rubric. |
|
|
155
|
+
| Grading result | The `grading.json` verdict for one run: passed, failed, skipped, evidence, and grading method per assertion. |
|
|
156
|
+
| Side-effect level | A case label: `static`, `local-only`, `live-safe`, or `live-external`. It lets you run cheap or safe cases before any live provider workflow. |
|
|
157
|
+
| Lifecycle hook | A shell command run before the suite, after each run is graded, or after the suite. Hooks are how provider-specific setup, external checks, and teardown are attached. |
|
|
158
|
+
| Cleanup manifest | `cleanup.json`, a scoped list of branches and PRs/MRs created by the run. Cleanup only acts on this manifest. |
|
|
159
|
+
| Summary/report | `summary.json`, `benchmark.json`, `status`, `report`, `report-all`, and optional JUnit/HTML/Markdown output. These are the artifacts you use for decisions. |
|
|
160
|
+
|
|
161
|
+
One normal `run` flows like this:
|
|
162
|
+
|
|
163
|
+
1. Load the skill and eval suite; apply filters such as `--eval-id` and
|
|
164
|
+
`--side-effect-level`.
|
|
165
|
+
2. Run `--pre-run-command` hooks. These usually validate the skill text, create
|
|
166
|
+
or reset a scratch source repo, or fail fast before any agent call.
|
|
167
|
+
3. For each case, create a workspace, clone `--source-repo` or initialize a
|
|
168
|
+
fresh repo, copy fixtures, and install the skill into the agent-specific
|
|
169
|
+
skill path.
|
|
170
|
+
4. Capture pre-run git/provider state.
|
|
171
|
+
5. Invoke the real agent CLI with the case prompt. If baseline mode is enabled,
|
|
172
|
+
run both `with_skill` and `without_skill`.
|
|
173
|
+
6. Capture post-run state, logs, timing, token telemetry, and output.
|
|
174
|
+
7. Grade deterministic assertions first, then send unresolved assertions to the
|
|
175
|
+
LLM grader when credentials are available.
|
|
176
|
+
8. Run `--post-grade-command` hooks while the workspace still exists. These can
|
|
177
|
+
append structured assertion results for provider state that the core harness
|
|
178
|
+
cannot know about.
|
|
179
|
+
9. Delete or keep workspaces according to debug settings, write summaries, and
|
|
180
|
+
run `--post-run-command` hooks for teardown and cleanup verification.
|
|
181
|
+
|
|
182
|
+
Use `force_skill_invocation: true` when debugging whether the skill instructions
|
|
183
|
+
are correct after the skill has already triggered. Use
|
|
184
|
+
`force_skill_invocation: false` for discovery cases that prove the agent chooses
|
|
185
|
+
the skill from a natural request. Use `should_trigger: false` for negative
|
|
186
|
+
controls that prove the skill does not activate on adjacent read-only prompts.
|
|
108
187
|
|
|
109
188
|
## Defining evals
|
|
110
189
|
|
|
@@ -151,9 +230,12 @@ A JSON Schema for this format ships at [`schemas/evals.schema.json`](schemas/eva
|
|
|
151
230
|
| `expected_output` | str | Reference output for LLM rubric grading |
|
|
152
231
|
| `files` | list[str] | Fixture file paths (resolved relative to the evals directory) |
|
|
153
232
|
| `stage_files` | bool | If true, fixture files are also `git add`-ed before the agent runs (default: false) |
|
|
154
|
-
| `assertions` | list[str] |
|
|
233
|
+
| `assertions` | list[str \| object] | Prose assertions or structured deterministic assertions to grade against |
|
|
155
234
|
| `should_trigger` | bool | If false, branch/commit/push/PR assertions are inverted (default: true) |
|
|
156
235
|
| `force_skill_invocation` | bool | If true, the prompt is prefixed with `Use the $<skill> skill.` (default: false) |
|
|
236
|
+
| `side_effect_level` | str | One of `static`, `local-only`, `live-safe`, `live-external` (default: `local-only`) |
|
|
237
|
+
|
|
238
|
+
Fixture paths are copied into the eval workspace before each run. Direct fixtures under `files/` are staged at the workspace root (`files/config.json` -> `config.json`). Case-scoped fixture folders also become the workspace root when a case uses one prefix (`files/prod-only/bootstrap/...` -> `bootstrap/...`), which lets suites model multi-repo workspaces without long prompt paths. If one eval references multiple case prefixes, those prefixes are preserved to avoid same-name collisions. Unsafe paths and target collisions fail during `validate`.
|
|
157
239
|
|
|
158
240
|
## How grading works
|
|
159
241
|
|
|
@@ -172,13 +254,28 @@ The deterministic grader matches assertion text against these patterns (case-ins
|
|
|
172
254
|
| `branch` + `created`/`exists`/`new` | A new branch appeared in the state diff and is checked out |
|
|
173
255
|
| `commit` + `created`/`exists`/`new` | A new commit appeared and is on the current branch |
|
|
174
256
|
| `push` + `remote`/`branch`/`pushed` | The eval-created branch was pushed AND remote HEAD matches local HEAD |
|
|
175
|
-
| `pr` or `pull request` | A new open PR targets the eval-created branch (corroborated by `gh pr view` when available) |
|
|
257
|
+
| `pr` or `pull request` | A new open PR/MR targets the eval-created branch (corroborated by `gh pr view` for GitHub when available) |
|
|
176
258
|
| `file exists` or `created` + a filename | File matching the backticked/quoted name exists in the workspace |
|
|
177
259
|
| `ran` + a command name (`npm`, `git`, ...) | Command name appears in the run logs |
|
|
178
260
|
| `contains` or `includes` + `"quoted"`/`` `backticked` `` text | Agent output contains the text |
|
|
179
261
|
| `valid json` | Agent output (or a workspace file) parses as JSON |
|
|
180
262
|
|
|
181
|
-
Anything else falls through to the LLM rubric. With `should_trigger: false`, the branch/commit/push/PR checks invert: they pass only when those artifacts did **not** appear.
|
|
263
|
+
Anything else falls through to the LLM rubric. With `should_trigger: false`, the branch/commit/push/PR/MR checks invert: they pass only when those artifacts did **not** appear.
|
|
264
|
+
|
|
265
|
+
### Structured deterministic assertions
|
|
266
|
+
|
|
267
|
+
For checks that should never fall through to the LLM rubric, use assertion objects:
|
|
268
|
+
|
|
269
|
+
```json
|
|
270
|
+
{
|
|
271
|
+
"type": "file_contains",
|
|
272
|
+
"path": "README.md",
|
|
273
|
+
"contains": "Quickstart",
|
|
274
|
+
"text": "README includes a Quickstart section"
|
|
275
|
+
}
|
|
276
|
+
```
|
|
277
|
+
|
|
278
|
+
Supported `type` values are `file_exists`, `file_not_exists`, `file_contains`, `file_not_contains`, `file_unchanged`, `command_ran`, `command_not_ran`, `tests_pass`, `git_clean`, `json_path`, and `json_path_not_exists`. Common fields are `path`, `contains`, `command`, `json_path`, `equals`, and optional human-facing `text`.
|
|
182
279
|
|
|
183
280
|
## CLI Commands
|
|
184
281
|
|
|
@@ -195,6 +292,7 @@ agent-skill-eval run \
|
|
|
195
292
|
[--baseline/--no-baseline] [--cleanup] \
|
|
196
293
|
[--timeout 600] [--retries 1] \
|
|
197
294
|
[--eval-id wif-too-long-stop] \
|
|
295
|
+
[--side-effect-level static] \
|
|
198
296
|
[--max-total-tokens-per-case 200000] [--budget-action fail] \
|
|
199
297
|
[--pre-run-command "python scripts/setup.py"] \
|
|
200
298
|
[--post-grade-command "python scripts/grade_external.py"] \
|
|
@@ -213,6 +311,7 @@ Key options:
|
|
|
213
311
|
- `--runs, -n`: repeat each (eval, agent, config) N times; enables pass@k stats. Results land in `run-1/`, `run-2/`, ... subdirectories.
|
|
214
312
|
- `--timeout` / `--retries`: per-run agent timeout (default 600s) and retries on timeout or non-zero exit (default 1). Also settable via `ASE_AGENT_TIMEOUT` / `ASE_AGENT_RETRIES`.
|
|
215
313
|
- `--eval-id`: run only the named case(s); repeatable. Unknown ids fail fast with the list of available ids, before any model call. The applied filter is recorded in `evals_meta.json` (`selected_eval_ids`) and `summary.json` (`eval_ids`), and all artifacts naturally describe only the selected cases. The cheap inner loop after a small skill edit is `validate --eval-id X` then `run --eval-id X`.
|
|
314
|
+
- `--side-effect-level`: run only the requested side-effect level(s); repeatable. Use this to run `static` and `local-only` checks before deliberate live-provider suites.
|
|
216
315
|
- Budget guards (`--max-input-tokens-per-case`, `--max-non-cached-input-tokens-per-case`, `--max-output-tokens-per-case`, `--max-total-tokens-per-case`, `--max-reasoning-tokens-per-case`, `--max-cost-per-case`, `--max-duration-per-case`, `--budget-action warn|fail|stop-suite`): per-case limits checked after each agent run (mid-run kills remain `--timeout`'s job — budgets exist because a case can finish within the timeout and still be far too expensive). A violation is recorded in `timing.json` (`budget_exceeded`, `budget_reason`); with `fail` (the default) it also appends a failed `method: "budget"` assertion to the run's `grading.json`, and with `stop-suite` it additionally skips every run that hasn't started yet — useful to stop a live side-effect suite before more remote branches/PRs get created. `--max-cost-per-case` only fires when the run has a cost source (see Cost reporting); likewise `--max-reasoning-tokens-per-case` and `--max-non-cached-input-tokens-per-case` only fire when the run reported those values — an agent exposing less telemetry is never failed over an unknown quantity.
|
|
217
316
|
- Lifecycle hooks (`--pre-run-command`, `--post-grade-command`, `--post-run-command`): see [Lifecycle hooks](#lifecycle-hooks).
|
|
218
317
|
- `--pricing-config`: JSON file with USD-per-token rates for agents whose CLI reports no cost (see Cost reporting).
|
|
@@ -222,10 +321,47 @@ While the suite runs, each case prints start/finish lines (elapsed, tokens in/ca
|
|
|
222
321
|
### `report`
|
|
223
322
|
|
|
224
323
|
```bash
|
|
225
|
-
agent-skill-eval report --workspace <skill-workspace> [--iteration N] [--format table|markdown] [--show-evidence] [--failures-only]
|
|
324
|
+
agent-skill-eval report --workspace <skill-workspace> [--iteration N] [--format table|markdown|html] [--show-evidence] [--failures-only]
|
|
226
325
|
```
|
|
227
326
|
|
|
228
|
-
`--show-evidence` prints the evidence string for every failed or skipped assertion — the state diff for deterministic checks, the judge's reasoning for LLM checks. `--failures-only` limits the per-eval detail to runs that had at least one failed assertion. `--format markdown` emits a paste-ready table.
|
|
327
|
+
`--show-evidence` prints the evidence string for every failed or skipped assertion — the state diff for deterministic checks, the judge's reasoning for LLM checks. `--failures-only` limits the per-eval detail to runs that had at least one failed assertion. `--format markdown` emits a paste-ready table; `--format html` emits a standalone static HTML report.
|
|
328
|
+
|
|
329
|
+
### `run-all`
|
|
330
|
+
|
|
331
|
+
```bash
|
|
332
|
+
agent-skill-eval run-all \
|
|
333
|
+
--root path/to/plugin-repo --agent codex --side-effect-level static \
|
|
334
|
+
[--source-repo https://github.com/foo/bar.git] \
|
|
335
|
+
[--max-total-tokens-per-case 200000] [--budget-action fail] \
|
|
336
|
+
[--pre-run-command "python scripts/setup.py"] \
|
|
337
|
+
[--post-grade-command "python scripts/grade_external.py"] \
|
|
338
|
+
[--post-run-command "python scripts/teardown.py"] \
|
|
339
|
+
[--pricing-config pricing.json] \
|
|
340
|
+
[--suite-concurrency 4] \
|
|
341
|
+
[--plan-only] \
|
|
342
|
+
[--changed-only --base-ref origin/main] \
|
|
343
|
+
[--junit-xml run-all.xml] [--fail-on-assertions] [--min-pass-rate 0.8]
|
|
344
|
+
```
|
|
345
|
+
|
|
346
|
+
Runs every paired `SKILL.md` + `evals/evals.json` suite discovered under a repo by invoking the normal single-suite runner once per skill. Suites with no cases for the requested side-effect filter are recorded as skipped in `run-all-iteration-N.json`, which makes `static` or `local-only` repo sweeps safe before any live-provider evals.
|
|
347
|
+
|
|
348
|
+
Repo sweeps accept the same source-repo, model, timeout/retry, budget, pricing, and lifecycle hook controls as `run`; the aggregate manifest records the source repo, budget, pricing config path, suite concurrency, and hook counts used for the sweep. Use `--plan-only` to validate and write the planned/skipped suite manifest without running agents, `--suite-concurrency` to run independent skill suites in parallel while preserving deterministic manifest ordering, `--changed-only --base-ref origin/main` in CI to run only skills or eval suites touched by the branch, `--junit-xml` to publish the sweep as test results, and `--fail-on-assertions` or `--min-pass-rate` to make quality gates explicit.
|
|
349
|
+
|
|
350
|
+
### `report-all`
|
|
351
|
+
|
|
352
|
+
```bash
|
|
353
|
+
agent-skill-eval report-all --workspace ./eval-workspace [--format table|markdown|html|json] [--failures-only]
|
|
354
|
+
```
|
|
355
|
+
|
|
356
|
+
Summarizes a repo-level `run-all` manifest with suite status, pass rates, duration, token totals, cost totals, and failure details. Use `--format markdown` for MR notes, `--format html` for a static artifact, and `--format json` when another tool should consume the raw manifest.
|
|
357
|
+
|
|
358
|
+
### `rerun-failed`
|
|
359
|
+
|
|
360
|
+
```bash
|
|
361
|
+
agent-skill-eval rerun-failed --from ./eval-workspace/run-all-iteration-1.json
|
|
362
|
+
```
|
|
363
|
+
|
|
364
|
+
Prints targeted `agent-skill-eval run --eval-id ...` commands for failed repo-sweep cases. The generated commands preserve run context recorded in the manifest, including grader settings, agent model, source repo, pricing config, timeout/retry, reasoning effort, and budget flags.
|
|
229
365
|
|
|
230
366
|
### `status`
|
|
231
367
|
|
|
@@ -249,15 +385,33 @@ Side-by-side pass rates of two iterations, per configuration, with pass-rate, ti
|
|
|
249
385
|
agent-skill-eval validate path/to/evals.json [--eval-id <id>]
|
|
250
386
|
```
|
|
251
387
|
|
|
252
|
-
Schema check plus referenced-fixture existence
|
|
388
|
+
Schema check plus referenced-fixture existence, duplicate-id detection, and structured-assertion required-field/path checks. Exit code 1 on any problem (CI-friendly). `--eval-id` validates only the selected case(s) (duplicate-id detection still covers the whole file); pair it with `run --eval-id` to confirm a targeted case is well-formed before spending a model call on it.
|
|
389
|
+
|
|
390
|
+
### `validate-all`
|
|
391
|
+
|
|
392
|
+
```bash
|
|
393
|
+
agent-skill-eval validate-all --root path/to/repo [--format table|json]
|
|
394
|
+
```
|
|
395
|
+
|
|
396
|
+
Recursively validates every discovered `*/evals/evals.json` under a repo and exits non-zero if any suite is invalid. This is the CI-friendly check for plugin repos with many skills.
|
|
397
|
+
For plugin-style paths such as `plugins/<plugin>/skills/<skill>/evals/evals.json`, it also verifies the adjacent `SKILL.md` frontmatter and that the suite `skill_name` matches the skill directory.
|
|
398
|
+
|
|
399
|
+
### `doctor`
|
|
400
|
+
|
|
401
|
+
```bash
|
|
402
|
+
agent-skill-eval doctor --skill ./skills/my-skill --evals ./evals/evals.json --agent codex
|
|
403
|
+
```
|
|
404
|
+
|
|
405
|
+
Preflights the local agent CLI, skill frontmatter, eval suite, LLM-grader credentials, and source-repo tooling (`gh` for GitHub, `glab` for GitLab). Use it before live or expensive suites to catch missing tools and auth early.
|
|
253
406
|
|
|
254
407
|
### `list`
|
|
255
408
|
|
|
256
409
|
```bash
|
|
257
|
-
agent-skill-eval list [--root .]
|
|
410
|
+
agent-skill-eval list [--root .] [--format table|json]
|
|
258
411
|
```
|
|
259
412
|
|
|
260
|
-
|
|
413
|
+
Recursively discovers eval suites (`*/evals/evals.json`) and skills (`SKILL.md`) under a directory, including plugin-style layouts such as `plugins/<plugin>/skills/<skill>/evals/evals.json`.
|
|
414
|
+
JSON output includes a `pairs` section showing paired skills, skills without evals, and external eval suites.
|
|
261
415
|
|
|
262
416
|
### `grade`
|
|
263
417
|
|
|
@@ -273,14 +427,18 @@ Re-grades existing outputs using saved `evals_meta.json` and state snapshots. Tw
|
|
|
273
427
|
agent-skill-eval cleanup --workspace ./eval-workspace [--yes]
|
|
274
428
|
```
|
|
275
429
|
|
|
276
|
-
Only closes PRs and deletes remote branches recorded in `cleanup.json`. Never touches unrelated
|
|
430
|
+
Only closes PRs/MRs and deletes remote branches recorded in `cleanup.json`. Never touches unrelated review requests, branches, or workspaces.
|
|
277
431
|
|
|
278
432
|
### `init`
|
|
279
433
|
|
|
280
434
|
```bash
|
|
281
|
-
agent-skill-eval init my-skill [--output ./examples]
|
|
435
|
+
agent-skill-eval init my-skill [--output ./examples] [--template basic|review-diff|commit-pr|config-validator]
|
|
436
|
+
agent-skill-eval init my-skill --layout plugin --plugin demo-plugin --output /path/to/plugin-repo
|
|
282
437
|
```
|
|
283
438
|
|
|
439
|
+
Use `commit-pr` for live branch/push/PR workflows, `review-diff` for read-only review skills, and `config-validator` for skills that inspect bundled config fixtures.
|
|
440
|
+
Use `--layout plugin` to scaffold directly into `plugins/<plugin>/skills/<skill>/evals` for Codex plugin repositories.
|
|
441
|
+
|
|
284
442
|
## Lifecycle hooks
|
|
285
443
|
|
|
286
444
|
Skills with external side effects (push, PR/MR creation, provider APIs) need setup, provider-specific grading, and teardown that the harness can't know about. Instead of hard-coding providers, `run` exposes three shell-command hook points; each receives run metadata as `ASE_*` environment variables on top of the parent environment:
|
|
@@ -299,6 +457,151 @@ agent-skill-eval run --skill ./skills/my-skill --evals ./evals/evals.json --agen
|
|
|
299
457
|
--post-run-command "python scripts/teardown_remote.py"
|
|
300
458
|
```
|
|
301
459
|
|
|
460
|
+
## Live side-effect cookbook
|
|
461
|
+
|
|
462
|
+
Use this pattern for skills that create external state: pushes, PRs/MRs, Jira
|
|
463
|
+
issues, cloud resources, provider API changes, tickets, comments, or anything
|
|
464
|
+
that cannot be fully judged from the local workspace.
|
|
465
|
+
|
|
466
|
+
### 1. Classify cases by blast radius
|
|
467
|
+
|
|
468
|
+
Set `side_effect_level` on every case:
|
|
469
|
+
|
|
470
|
+
| Level | Use for |
|
|
471
|
+
| --- | --- |
|
|
472
|
+
| `static` | Prompt-shaping, validation, "ask for missing input", and no-tool checks. No local git writes or provider calls. |
|
|
473
|
+
| `local-only` | Local file edits, local git branches/commits, dry-run provider payloads, or live calls only to isolated scratch projects that are fully cleaned by hooks. |
|
|
474
|
+
| `live-safe` | Live provider state in a controlled test project where setup and teardown are reliable and the effect is reversible. |
|
|
475
|
+
| `live-external` | Shared, customer, production, paid, or human-visible systems. Run only with explicit intent and tight filters. |
|
|
476
|
+
|
|
477
|
+
Start with `static` and `local-only`. Add `live-safe` only after the local cases
|
|
478
|
+
prove the workflow shape. Treat `live-external` as a release candidate check,
|
|
479
|
+
not an inner loop.
|
|
480
|
+
|
|
481
|
+
### 2. Make the source repo disposable
|
|
482
|
+
|
|
483
|
+
For git/MR workflows, use `--source-repo` pointing at a scratch repository that
|
|
484
|
+
the eval owns. The pre-run hook should reset it to a known base state before
|
|
485
|
+
each suite or case:
|
|
486
|
+
|
|
487
|
+
```bash
|
|
488
|
+
agent-skill-eval run \
|
|
489
|
+
--skill ./plugins/acme/skills/commit-push-mr \
|
|
490
|
+
--evals ./plugins/acme/skills/commit-push-mr/evals/evals.json \
|
|
491
|
+
--source-repo git@gitlab.example.com:me/agent-skill-eval-scratch.git \
|
|
492
|
+
--agent codex \
|
|
493
|
+
--side-effect-level local-only \
|
|
494
|
+
--eval-id wif-too-long-stop \
|
|
495
|
+
--pre-run-command "python plugins/acme/skills/commit-push-mr/evals/scripts/setup_source_repo.py" \
|
|
496
|
+
--post-grade-command "python plugins/acme/skills/commit-push-mr/evals/scripts/grade_gitlab.py" \
|
|
497
|
+
--post-run-command "python plugins/acme/skills/commit-push-mr/evals/scripts/teardown_gitlab.py" \
|
|
498
|
+
--post-run-command "python plugins/acme/skills/commit-push-mr/evals/scripts/verify_gitlab_clean.py"
|
|
499
|
+
```
|
|
500
|
+
|
|
501
|
+
The scratch repo should contain only eval fixtures and expected provider state.
|
|
502
|
+
Do not point live evals at the real project until the suite has already passed
|
|
503
|
+
against scratch state.
|
|
504
|
+
|
|
505
|
+
### 3. Keep setup, grading, teardown separate
|
|
506
|
+
|
|
507
|
+
Use one script for each job:
|
|
508
|
+
|
|
509
|
+
- Setup hook: creates or resets scratch state, checks auth, validates that the
|
|
510
|
+
target is the expected test project, and aborts before model calls if anything
|
|
511
|
+
is unsafe.
|
|
512
|
+
- Post-grade hook: reads `ASE_*` paths, inspects provider state while the agent
|
|
513
|
+
workspace still exists, and prints JSON assertion rows to stdout.
|
|
514
|
+
- Teardown hook: closes/deletes only expected branches, MRs, tickets, labels, or
|
|
515
|
+
resources created by this suite.
|
|
516
|
+
- Verify-clean hook: read-only proof that no expected eval branch, MR, ticket, or
|
|
517
|
+
resource remains. Run it after teardown and independently after failures.
|
|
518
|
+
|
|
519
|
+
Post-grade hooks can append checks to `grading.json` by printing:
|
|
520
|
+
|
|
521
|
+
```json
|
|
522
|
+
[
|
|
523
|
+
{
|
|
524
|
+
"text": "Exactly one GitLab MR was opened for the eval branch",
|
|
525
|
+
"passed": true,
|
|
526
|
+
"evidence": "MR !42 targets main and source branch proj-123-eval"
|
|
527
|
+
}
|
|
528
|
+
]
|
|
529
|
+
```
|
|
530
|
+
|
|
531
|
+
This keeps external verification in the same report as deterministic file/git
|
|
532
|
+
checks and LLM rubric checks.
|
|
533
|
+
|
|
534
|
+
### 4. Write cases around failure modes, not just happy paths
|
|
535
|
+
|
|
536
|
+
A good side-effect suite usually has these case types:
|
|
537
|
+
|
|
538
|
+
- Happy path: creates the intended branch/change/review request/resource in the
|
|
539
|
+
scratch provider.
|
|
540
|
+
- Guardrail stop: invalid branch name, missing ticket ID, missing required
|
|
541
|
+
input, unsupported environment, or too-large payload stops before side
|
|
542
|
+
effects.
|
|
543
|
+
- No-op stop: no meaningful local changes means no commit, push, ticket, or API
|
|
544
|
+
write.
|
|
545
|
+
- Natural trigger: same workflow without `force_skill_invocation`, proving the
|
|
546
|
+
agent selects the skill from a realistic user request.
|
|
547
|
+
- Negative control: adjacent read-only prompt with `should_trigger: false`,
|
|
548
|
+
proving no side effects are created.
|
|
549
|
+
- Cleanup regression: provider state is clean after teardown, even when the
|
|
550
|
+
agent failed earlier.
|
|
551
|
+
|
|
552
|
+
For each side effect, pair a positive assertion with an explicit "nothing else"
|
|
553
|
+
check in the provider hook. For example: one MR for the eval branch, no MR for
|
|
554
|
+
`main`, no duplicate MR, no unexpected branch, and no untracked provider
|
|
555
|
+
resource.
|
|
556
|
+
|
|
557
|
+
### 5. Use a cheap inner loop
|
|
558
|
+
|
|
559
|
+
Do not make full live runs your edit loop. Use this order:
|
|
560
|
+
|
|
561
|
+
```bash
|
|
562
|
+
agent-skill-eval validate path/to/evals.json --eval-id wif-too-long-stop
|
|
563
|
+
agent-skill-eval doctor --skill path/to/skill --evals path/to/evals.json --agent codex
|
|
564
|
+
agent-skill-eval run --skill path/to/skill --evals path/to/evals.json \
|
|
565
|
+
--agent codex --eval-id wif-too-long-stop --side-effect-level local-only \
|
|
566
|
+
--no-baseline --max-total-tokens-per-case 150000 --budget-action fail
|
|
567
|
+
agent-skill-eval status --workspace ./eval-workspace/<skill>-workspace
|
|
568
|
+
agent-skill-eval report --workspace ./eval-workspace/<skill>-workspace --show-evidence
|
|
569
|
+
```
|
|
570
|
+
|
|
571
|
+
Only after a targeted case passes should you run more cases, more agents,
|
|
572
|
+
baseline mode, or `run-all`.
|
|
573
|
+
|
|
574
|
+
For plugin repos with many skills, use planning before execution:
|
|
575
|
+
|
|
576
|
+
```bash
|
|
577
|
+
agent-skill-eval validate-all --root /path/to/plugin-repo --strict
|
|
578
|
+
agent-skill-eval run-all --root /path/to/plugin-repo \
|
|
579
|
+
--side-effect-level static --changed-only --base-ref origin/main --plan-only
|
|
580
|
+
agent-skill-eval run-all --root /path/to/plugin-repo \
|
|
581
|
+
--side-effect-level local-only --changed-only --base-ref origin/main \
|
|
582
|
+
--suite-concurrency 2 --fail-on-assertions
|
|
583
|
+
```
|
|
584
|
+
|
|
585
|
+
### 6. Use budgets and cleanup as gates
|
|
586
|
+
|
|
587
|
+
Side-effect suites need explicit failure gates:
|
|
588
|
+
|
|
589
|
+
- Set `--timeout`, `--retries`, `--max-total-tokens-per-case`,
|
|
590
|
+
`--max-duration-per-case`, and usually `--budget-action fail` or
|
|
591
|
+
`stop-suite`.
|
|
592
|
+
- Use `--eval-id` while debugging a single risky case.
|
|
593
|
+
- Keep `--cleanup` or teardown hooks enabled for any run that can push, open an
|
|
594
|
+
MR, or write provider state.
|
|
595
|
+
- Run a read-only verify-clean command after failures before starting another
|
|
596
|
+
live run.
|
|
597
|
+
- Archive or paste `status`, `report --show-evidence`, and provider hook
|
|
598
|
+
evidence when claiming a skill improved.
|
|
599
|
+
|
|
600
|
+
The minimum credible claim for a side-effect skill is: targeted case passed,
|
|
601
|
+
provider hook passed, cleanup verification passed, and the command/model/harness
|
|
602
|
+
version are recorded. A stronger claim adds repeated runs, natural trigger and
|
|
603
|
+
negative-control cases, and a before/after comparison with `compare`.
|
|
604
|
+
|
|
302
605
|
## Example skills
|
|
303
606
|
|
|
304
607
|
Five example skills ship with the repo, chosen to exercise different grading surfaces:
|
|
@@ -321,7 +624,7 @@ Each has a matching eval suite under [`examples/`](examples/). `skills/` holds t
|
|
|
321
624
|
eval-workspace/
|
|
322
625
|
└── <skill>-workspace/
|
|
323
626
|
└── iteration-1/
|
|
324
|
-
├── evals_meta.json # eval definitions
|
|
627
|
+
├── evals_meta.json # eval definitions, selected filters, skill_hash (used by `grade`)
|
|
325
628
|
├── cleanup.json # manifest of artifacts created by this run
|
|
326
629
|
├── benchmark.json # aggregate stats per (agent, config)
|
|
327
630
|
├── summary.json # one-file run summary (used by `status`)
|
|
@@ -335,7 +638,8 @@ eval-workspace/
|
|
|
335
638
|
│ └── pre_state.json / post_state.json
|
|
336
639
|
├── timing.json # tokens (incl. non-cached/reasoning splits), duration, exit_code, timed_out, retries, budget verdict
|
|
337
640
|
├── grading.json # per-assertion results
|
|
338
|
-
|
|
641
|
+
├── trajectory.json # normalized lifecycle events for this run
|
|
642
|
+
└── run_meta.json # agent, with_skill, run_index, model, side_effect_level, skill_hash, CLI/harness versions, ...
|
|
339
643
|
```
|
|
340
644
|
|
|
341
645
|
### grading.json
|
|
@@ -424,7 +728,7 @@ Codex's CLI reports token counts but no cost. To get cost numbers anyway, pass `
|
|
|
424
728
|
}
|
|
425
729
|
```
|
|
426
730
|
|
|
427
|
-
The computed value lands in `cost_usd` with `cost_usd_source: "pricing-config"`. Cached input is billed at `input_cache_read` (falling back to `prompt`); note that codex
|
|
731
|
+
The computed value lands in `cost_usd` with `cost_usd_source: "pricing-config"`. Cached input is billed at `input_cache_read` (falling back to `prompt`); note that codex reports `input_tokens` *including* cache reads while claude-code and opencode report them separately — the harness accounts for this per agent. `summary.json` splits the totals (`tokens.cached`, `tokens.non_cached_input`, `tokens.cached_pct`), because on long agent sessions most input tokens are cache reads and the raw `input_tokens` number wildly overstates billable spend. Each run's `timing.json` also persists `non_cached_input_tokens` (computed with that run's CLI semantics) and `reasoning_output_tokens` when the agent reports reasoning telemetry (codex `turn.completed`, opencode `step_finish`) — `null` means unreported, not zero.
|
|
428
732
|
|
|
429
733
|
One sharp edge: the claude CLI prices runs at **Anthropic list prices** regardless of the endpoint it talks to. When you route claude-code through OpenRouter (`--harness-base-url` or `ANTHROPIC_BASE_URL`), agent-skill-eval reconciles the cost by recomputing it from the run's token counts and OpenRouter's published per-model rates. `timing.json` then shows:
|
|
430
734
|
|