skill-evals 0.0.0-stage → 0.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (54) hide show
  1. package/CHANGELOG.md +22 -0
  2. package/LICENSE +21 -0
  3. package/README.md +427 -2
  4. package/dist/baseten.d.ts +20 -0
  5. package/dist/baseten.js +89 -0
  6. package/dist/bin.d.ts +2 -0
  7. package/dist/bin.js +5 -0
  8. package/dist/cases.d.ts +21 -0
  9. package/dist/cases.js +207 -0
  10. package/dist/checks.d.ts +126 -0
  11. package/dist/checks.js +117 -0
  12. package/dist/cli.d.ts +7 -0
  13. package/dist/cli.js +172 -0
  14. package/dist/decision.d.ts +70 -0
  15. package/dist/decision.js +130 -0
  16. package/dist/evaluate.d.ts +70 -0
  17. package/dist/evaluate.js +205 -0
  18. package/dist/exec.d.ts +20 -0
  19. package/dist/exec.js +82 -0
  20. package/dist/harnesses/claude-code.d.ts +3 -0
  21. package/dist/harnesses/claude-code.js +83 -0
  22. package/dist/harnesses/codex.d.ts +3 -0
  23. package/dist/harnesses/codex.js +85 -0
  24. package/dist/harnesses/cursor.d.ts +3 -0
  25. package/dist/harnesses/cursor.js +81 -0
  26. package/dist/harnesses/index.d.ts +34 -0
  27. package/dist/harnesses/index.js +101 -0
  28. package/dist/harnesses/opencode.d.ts +3 -0
  29. package/dist/harnesses/opencode.js +75 -0
  30. package/dist/harnesses/pi.d.ts +3 -0
  31. package/dist/harnesses/pi.js +69 -0
  32. package/dist/harnesses/shared.d.ts +105 -0
  33. package/dist/harnesses/shared.js +90 -0
  34. package/dist/index.d.ts +10 -0
  35. package/dist/index.js +3 -0
  36. package/dist/judge.d.ts +30 -0
  37. package/dist/judge.js +87 -0
  38. package/dist/probe.d.ts +16 -0
  39. package/dist/probe.js +113 -0
  40. package/dist/project.d.ts +36 -0
  41. package/dist/project.js +84 -0
  42. package/dist/report.d.ts +8 -0
  43. package/dist/report.js +201 -0
  44. package/dist/run.d.ts +86 -0
  45. package/dist/run.js +220 -0
  46. package/dist/scope.d.ts +39 -0
  47. package/dist/scope.js +82 -0
  48. package/dist/score.d.ts +71 -0
  49. package/dist/score.js +92 -0
  50. package/dist/session.d.ts +33 -0
  51. package/dist/session.js +146 -0
  52. package/dist/subject.d.ts +54 -0
  53. package/dist/subject.js +179 -0
  54. package/package.json +59 -4
package/CHANGELOG.md ADDED
@@ -0,0 +1,22 @@
1
+ # Changelog
2
+
3
+ ## Unreleased
4
+
5
+ - Releases are guarded. `scripts/release-plan.sh` refuses a version tag that does not match `package.json`, is not on `main`, has no changelog entry, or is not above the latest version on npm, and the `Release check` workflow runs the same guards on every pull request. The publish workflow also makes the GitHub release.
6
+
7
+ ## 0.0.1
8
+
9
+ The first release. The runner comes from the `evals/` folder of [magic](https://github.com/PhillipChaffee/magic), where it scored that one set of skills.
10
+
11
+ - Run it in any project. The project is the git repository that holds the working directory, and the cases default to `evals/cases` in it.
12
+ - Score a plugin, a plain folder of skills, or `none`, which loads no skills and is the baseline for a comparison.
13
+ - Set defaults in an optional `skill-evals.config.json`.
14
+ - Use it from code with `evaluate`, `loadCases`, and `HARNESSES`.
15
+ - A prompt that starts with a dash reaches the agent in every harness.
16
+ - Windows is not supported.
17
+ - The package is written in TypeScript. It ships compiled JavaScript and type declarations, and still has no dependencies.
18
+
19
+ - A `decision` check asks a decision model a question about the final answer. The case sets what each allowed answer is worth, the model returns a probability for each answer, and the check's score is the expected value. All decision checks of a case go out in one request for each run.
20
+ - `--decision-model` and `--decision-url`, and `decisionModel` and `decisionUrl` in the config file, set the model and its endpoint. The client speaks Baseten's System One API and reads its key from `BASETEN_API_KEY`. It was written from the API's documentation. It has scored checks against bison-1 on a self-hosted deployment, and has not yet called Baseten's hosted API.
21
+ - `--decision-timeout <seconds>` and `decisionTimeout` in the config file set how long one request to the decision model may take. The default stays 60 seconds. A cold start of a self-hosted model can take that long.
22
+ - A decision reply whose probabilities were rounded to 6 decimals is accepted. The sum over the allowed answers may differ from 1 by 0.0000005 for each answer, where the limit was 0.000001 in total, which rejected three honest options that added up to 0.999999.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Phillip Chaffee
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -1,3 +1,428 @@
1
- # Temporary Holding Version
1
+ # skill-evals
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ `skill-evals` measures what an agent does when your skills are loaded. It runs a set of prompts in a real coding-agent harness, scores every run from 0 to 1, and can fail when a score is under a bar you set. It scores one set of skills, or it compares two: a baseline (A) and a change (B), or no skills at all and your skills.
4
+
5
+ It works with Claude Code, Codex, Cursor, OpenCode, and Pi. It needs Node 22 or later and at least one of those harnesses installed. It has no dependencies. It runs on macOS and Linux.
6
+
7
+ Every run is unattended and has full permissions in its harness. The agent can run commands and write files on your machine without asking, so only run prompts you wrote or read.
8
+
9
+ ## Install
10
+
11
+ ```sh
12
+ npm install --save-dev skill-evals # or run it without installing: npx skill-evals <command>
13
+ ```
14
+
15
+ ## A first case
16
+
17
+ Run this in the folder that holds your skills. Replace `greet` with the name of one of your skills, and the prompt with a request that should make the agent use it.
18
+
19
+ ```sh
20
+ mkdir -p evals/cases/greeting
21
+ echo 'Give me a greeting to start my day.' > evals/cases/greeting/prompt.md
22
+ echo '{ "checks": [{ "type": "skill_loaded", "skill": "greet" }] }' > evals/cases/greeting/case.json
23
+ npx skill-evals check # which harnesses work on this machine
24
+ npx skill-evals run --a none --model <model-id> # A = no skills, B = your skills
25
+ ```
26
+
27
+ The report shows how often the agent loaded the skill without your skills (0 of 3) and with them. [`examples/cases/hello`](examples/cases/hello) in the repository is one more case you can copy.
28
+
29
+ Each run starts in a new, empty home directory, so your own settings and default model are not there. Pass `--model`, or set `model` in the [config file](#config-file). Without it each harness uses its built-in default, which may be a provider you have no key for.
30
+
31
+ ## Subjects
32
+
33
+ A subject is what a run loads into the harness. You name it with `--subject`, `--a`, or `--b`.
34
+
35
+ | You pass | Kind | What the run loads |
36
+ | -------------------------------------------------------------------- | -------- | --------------------------------------------------------------------------------------------------------- |
37
+ | A directory, or a git ref of the project, that has a plugin manifest | `plugin` | The directory, the way each harness loads a local plugin. Hooks and rules that the plugin ships load too. |
38
+ | Any other directory or git ref | `skills` | Each skill in it, copied to the place where the harness discovers a user's skills. |
39
+ | `none` | `none` | Nothing. This is the baseline for "does the skill change anything". |
40
+
41
+ - **Project.** The project is the git repository that holds the working directory, or the working directory itself when it is not in a repository. The default subject is the project folder, with its uncommitted changes.
42
+ - **Directory.** A directory is used as it is on disk. A relative path resolves against the working directory. For a folder that is named `none`, write `./none`.
43
+ - **Git ref.** Anything that is not a directory must be a git ref of the project. It is checked out into a detached worktree in a temp directory, and the worktree is removed when the command ends, on failure and on Ctrl+C too. The report prints the commit, so a stale local `main` is visible.
44
+ - **Plugin manifest.** A directory is a plugin when its root has `.claude-plugin/plugin.json`, `.cursor-plugin/plugin.json`, `.codex-plugin/plugin.json`, `plugin.json`, `.opencode/plugins/`, or a `package.json` with a `pi` key. A plugin loads only in the harnesses it has packaging for.
45
+ - **Skills folder.** The skills are every `<dir>/<name>/SKILL.md` and `<dir>/skills/<name>/SKILL.md`, and `<dir>/SKILL.md` itself, which is one skill named after the folder. The name of a skill is the name of its folder, and the `name` in its front matter must be the same. A folder with no skill is an error.
46
+
47
+ Where a `skills` subject goes, inside the run's own home directory:
48
+
49
+ | Harness | Location |
50
+ | ------------- | ------------------------------------------------------------------------------------------------------- |
51
+ | `claude-code` | `~/.claude/skills/<name>/` |
52
+ | `cursor` | `~/.cursor/skills/<name>/` |
53
+ | `codex` | `~/.agents/skills/<name>/` |
54
+ | `opencode` | `~/.agents/skills/<name>/` |
55
+ | `pi` | `~/.pi/agent/skills/<name>/`, each named with `--skill` because the run switches Pi's own discovery off |
56
+
57
+ The files are copied, not linked, and `.git` and `node_modules` are left out. A skill of the same name that you installed globally does not load, because your real home directory is not the run's home.
58
+
59
+ With `none`, and with every other subject, the harness still loads what it ships with. Some harnesses also fetch plugins that your organization manages. Claude Code did that in the runs behind this document.
60
+
61
+ ## Commands
62
+
63
+ | Command | What it does |
64
+ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
65
+ | `run` | Runs every case against A and B, `--reps` times each, one run at a time, and scores both. |
66
+ | `score` | Runs every case against one subject and scores it. |
67
+ | `check` | Runs one small prompt in each installed harness. Reports whether the harness ran, whether it reported the subject as loaded, which skills the agent loaded, and which record fields the adapter filled. A harness that fails, for example because it has no login, gets its own error message in its row, and the next harness still runs. |
68
+ | `list` | Lists the harnesses in default-preference order with their install state, and the cases. |
69
+
70
+ | Option | Commands | Default | Meaning |
71
+ | ------------------------------ | ----------------------- | ----------------------------------------------- | ---------------------------------------------------------------------------------------- |
72
+ | `--a <subject>` | `run` | `main` | The baseline. |
73
+ | `--b <subject>` | `run` | the project | The change. |
74
+ | `--subject <subject>` | `score`, `check` | the project | The subject to score or to probe. |
75
+ | `--harness <ids\|all>` | `run`, `score`, `check` | first installed (`run`, `score`), all (`check`) | Comma-separated harness ids. |
76
+ | `--case <glob>` | `run`, `score` | `*` | Which case ids to run. Only `*` is special. |
77
+ | `--cases <dir>` | `run`, `score`, `list` | `evals/cases` in the project | Where the cases are. |
78
+ | `--reps <n>` | `run`, `score` | `3` | Runs for each subject and case. |
79
+ | `--model <id>` | `run`, `score`, `check` | the harness's own default | Passed to the harness unchanged. |
80
+ | `--min-score <0..1>` | `run`, `score` | no bar | The lowest suite score that passes. See [Scores](#scores). |
81
+ | `--judge-harness <id>` | `run`, `score` | the harness under test | The harness that scores judged checks. |
82
+ | `--judge-model <id>` | `run`, `score` | the value of `--model` | The model that scores judged checks. |
83
+ | `--decision-model <id>` | `run`, `score` | none | The decision model that scores decision checks. See [Decision checks](#decision-checks). |
84
+ | `--decision-url <url>` | `run`, `score` | the provider's own | The base URL of the decision model's API. |
85
+ | `--decision-timeout <seconds>` | `run`, `score` | `60` | How long one request to the decision model may take. A positive integer. |
86
+ | `--out <dir>` | `run`, `score`, `check` | a new temp directory | Where results go. |
87
+
88
+ | Exit code | Meaning |
89
+ | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
90
+ | 0 | Every run is valid and every pass bar is reached. |
91
+ | 1 | A run failed or is invalid, or a score is under its bar. |
92
+ | 2 | A wrong command line, case file, config file, or subject. A decision check with no decision model or no API key. Also Windows, which is not supported. |
93
+
94
+ `run` and `score` print a Markdown report and the path of an output directory. Read the report first, then open the final answers of a case:
95
+
96
+ ```text
97
+ <out>/results.json every run record, with check scores, judge reasons, and probabilities
98
+ <out>/<harness>/<case>/a1.final.md final answer of subject A, repetition 1
99
+ <out>/<harness>/<case>/b1.final.md final answer of subject B, repetition 1
100
+ <out>/<harness>/<case>/a1.stream.jsonl the harness's raw event stream for that run
101
+ ```
102
+
103
+ `score` names its files `subject1.final.md` and so on.
104
+
105
+ ## What one run does
106
+
107
+ 1. Creates a new, empty home directory and a new workspace directory. Their names (`home-*`, `proj-*`, `kit-*`) say nothing about why they exist.
108
+ 2. Copies the case's `workspace/` folder into the workspace.
109
+ 3. Copies the harness's credential files from your real home into the new home, so a login keeps working. Nothing else is copied.
110
+ 4. Starts the harness with `HOME` and the XDG directories pointed at the new home, and with the subject loaded as [Subjects](#subjects) describes. The harness does not get the environment variables that name this tool: the ones that start with `SKILL_EVALS_` or `npm_`, and `_`. When the suite has a decision check, the harness does not get the decision model's API key either.
111
+ 5. Turns the harness's event stream into one run record and scores the case's checks. A judged check starts one more harness run, described under [Judged checks](#judged-checks). The decision checks of the case go to the decision model in one request, described under [Decision checks](#decision-checks).
112
+ 6. Deletes both directories.
113
+
114
+ A run counts only when it is valid. A run is invalid when it failed (harness error, time limit), when the harness reported what it loaded and the subject was not in it, or when a judged check or a decision check got no usable score. Invalid runs are listed under the table, left out of every number, and make the command exit 1.
115
+
116
+ "The subject was not in it" means that a plugin's directory is not among the loaded plugins, or that a skill of a skills folder is not among the listed skills. Only Claude Code reports either. In the other four harnesses a subject that did not load shows up as missing skill loads, not as an invalid run.
117
+
118
+ ## Write a case
119
+
120
+ A case is a directory under `evals/cases/`. Its name is the case id.
121
+
122
+ | File | Content |
123
+ | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
124
+ | `prompt.md` | The prompt, exactly as a user would type it. Nothing else. |
125
+ | `case.json` | `{ "checks": [...], "weight": 1, "minScore": 0.5, "maxTurns": 6, "timeoutSeconds": 600 }`. Only `checks` is required, and it needs at least one check. |
126
+ | `workspace/` | Optional. Files the agent finds in its working directory. |
127
+
128
+ Every check takes an optional `weight`, a positive number that defaults to 1.
129
+
130
+ | Check | Fields | Scores 1 when |
131
+ | --------------------- | ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
132
+ | `skill_loaded` | `skill` | The agent loaded that skill. |
133
+ | `skill_not_loaded` | `skill` | The agent did not load it. |
134
+ | `output_contains` | `text` | The final answer contains that text. The match is exact and case-sensitive. |
135
+ | `output_not_contains` | `text` | The final answer does not contain it. |
136
+ | `output_matches` | `pattern`, optional `flags` | The final answer matches the regular expression. |
137
+ | `output_lacks` | `pattern`, optional `flags` | The final answer does not match it. |
138
+ | `file_exists` | `path` | That file exists in the workspace after the run. |
139
+ | `file_matches` | `path`, `pattern`, optional `flags` | That file exists and matches. |
140
+ | `tool_called` | `kind` or `name` or both, optional `inputMatches`, optional `flags` | The agent made a tool call that fits every given field. |
141
+ | `tool_not_called` | the same | The agent made no such call. |
142
+ | `judge` | `rubric` | Not 0 or 1. A model scores the final answer against the rubric. |
143
+ | `decision` | `question`, and `expect` or `options` | Not 0 or 1. A decision model gives each allowed answer a probability, and the score is the expected value. |
144
+
145
+ In every harness, "loaded a skill" means that the agent called the harness's skill tool with that skill's name, or that any tool call pointed at `skills/<name>/SKILL.md`.
146
+
147
+ `timeoutSeconds` defaults to 600. `maxTurns` has no default, and only Claude Code has a flag for it. The time limit alone bounds the other harnesses.
148
+
149
+ A prompt can start with any character. One that starts with a dash reaches the agent as the prompt and is not read as a flag of the harness.
150
+
151
+ The agent must not be able to tell that its answer is being measured. Write the prompt as a plain request, and keep these words out of the prompt and out of every file name under `workspace/`: `eval`, `test`, `judge`, `experiment`, `rubric`, `score`, `compare`, `benchmark`, `candidate`, `arena`. Do not ask the agent which skills it used. The checks read that from its tool calls. The rubric stays in `case.json`, which the agent's workspace never holds. A plugin is loaded from where it is on disk, so the agent can see that path, and the cases too when they are inside the plugin's folder.
152
+
153
+ ### Tool checks
154
+
155
+ `kind` says what the tool does, with the same word in every harness. Use it so that a case works in each harness.
156
+
157
+ | Kind | Meaning |
158
+ | ---------- | -------------------------------------------------------------------- |
159
+ | `shell` | Ran a command. |
160
+ | `read` | Read a file. |
161
+ | `write` | Created or replaced a file. |
162
+ | `edit` | Changed part of a file, or deleted one. |
163
+ | `search` | Searched or listed files. |
164
+ | `web` | Fetched a page or searched the web. |
165
+ | `skill` | Called the harness's skill tool. |
166
+ | `subagent` | Started another agent. |
167
+ | `other` | Any tool the harness's adapter has no kind for, such as an MCP tool. |
168
+
169
+ `name` is a regular expression for the harness's own tool name, for example `^Bash$` in Claude Code. A check with `name` works in one harness only, so prefer `kind`. `inputMatches` is a regular expression for the call's input, written as JSON. `flags` applies to both expressions.
170
+
171
+ ```json
172
+ { "type": "tool_not_called", "kind": "shell", "inputMatches": "git push" }
173
+ ```
174
+
175
+ Harnesses do not split work the same way. One harness has a search tool, and another runs `grep` through its shell tool, so `tool_called` with `"kind": "search"` can score 0 for an agent that did search. Tool calls that a subagent made count in the harnesses whose stream includes them (seen in Claude Code).
176
+
177
+ ### Judged checks
178
+
179
+ A `judge` check asks a model how well the final answer meets the `rubric`, which is plain text that says what a good answer does. Use it for a quality that no pattern can find, such as whether a restatement keeps the meaning.
180
+
181
+ - The judge is one more harness run, so it works in each harness. It runs with nothing loaded, in a new home directory and an empty workspace, with a limit of two turns where the harness has a turn limit.
182
+ - The judge's prompt holds the case's prompt, the final answer, and the rubric. It does not hold the subject, the ref, a path, or the harness that produced the answer, so the judge cannot favor A or B.
183
+ - The judge must reply with one JSON object, `{"reason": "...", "score": 0.7}`. The scale is 0 for an answer that meets none of the rubric, 0.5 for about half, and 1 for all of it. A score outside 0 to 1 is rejected, not clamped.
184
+ - An unusable reply is asked for once more. After two unusable replies the check has no score and the run is invalid.
185
+ - `results.json` holds the judge's reason for each check, and the judge's duration, tokens, and cost for each run apart from the run's own.
186
+
187
+ A judge is a model, so its score moves. Measured with one small model as agent and judge on one case of the [magic](https://github.com/PhillipChaffee/magic) skills: six runs of an unchanged subject gave judged scores from 0.6 to 0.9, and four scorings of one fixed answer differed by 0.10 to 0.25. Use at least three reps, write a rubric with separate countable points, and name a stronger model with `--judge-model` when the differences you look for are small.
188
+
189
+ ### Decision checks
190
+
191
+ A `decision` check asks a decision model a question about the final answer. A decision model writes no text. It returns a calibrated probability for each answer you allow, so you set what each answer is worth and the score is computed from the probabilities. Use it, like a `judge` check, for a quality that no pattern can find. The difference is that no model writes the score.
192
+
193
+ ```json
194
+ { "type": "decision", "question": "Does the answer avoid engineering jargon?", "expect": "yes" }
195
+ { "type": "decision", "question": "Did the agent ask before deleting files?", "expect": "no" }
196
+ {
197
+ "type": "decision",
198
+ "question": "How well does the answer keep the original meaning?",
199
+ "options": { "fully": 1, "mostly": 0.7, "partly": 0.3, "not at all": 0 }
200
+ }
201
+ ```
202
+
203
+ A check takes exactly one of `expect` and `options`.
204
+
205
+ | Field | Value | Check score |
206
+ | --------- | ------------------------------------------------------------------ | ------------------------------------------------------- |
207
+ | `expect` | `"yes"` | P(yes) |
208
+ | `expect` | `"no"` | 1 - P(yes) |
209
+ | `options` | An object of at least two answers, each with a number from 0 to 1. | The sum, over the answers, of probability times number. |
210
+
211
+ An example: the model returns P(yes) = 0.9 for the first check above, so its score is 0.90. For the third check it returns `fully` 0.5, `mostly` 0.3, `partly` 0.1, and `not at all` 0.1, so its score is 0.5 × 1 + 0.3 × 0.7 + 0.1 × 0.3 + 0.1 × 0 = 0.74.
212
+
213
+ - **Configuration.** Name the model with `--decision-model <id>` or `decisionModel` in the [config file](#config-file). There is no default model. The API key comes from the `BASETEN_API_KEY` environment variable and from nowhere else. It is never read from the config file and never printed or written to the output directory. A suite that has a decision check and no model or no key stops with exit code 2 before any run starts.
214
+ - **Provider.** The client speaks Baseten's System One API, which serves [bison-1](https://huggingface.co/baselabs/bison-1) as `decision-models/baseten-bison-latest` and the other models in Baseten's [decision models](https://docs.baseten.co/inference/model-apis/decision-models) list. `--decision-url <url>` or `decisionUrl` names the base URL of another endpoint that speaks the same API, such as a deployment of your own. The request goes to `<url>/v1/systemone`.
215
+ - **One request for each run.** Every decision check of a case is one question of the same request. An `expect` check is sent as a yes/no question, and an `options` check as a choice among the option names. Write option names that say what they mean, because the model sees only the name.
216
+ - **What the model sees.** The request holds the case's prompt and the final answer, in `<task>` and `<answer>` tags, and the questions. It does not hold the subject, the ref, a path, or the harness that produced the answer, so the model cannot favor A or B.
217
+ - **Unusable replies.** A probability that is missing or is not a number from 0 to 1 is a failure of the model, and so are probabilities that do not add up to 1 over the allowed answers. A provider may round each probability to 6 decimals, so the sum may differ from 1 by 0.0000005 for each allowed answer: three options may add up to 0.999999 or 1.000001. A model that keeps real probability back, for example for an "I don't know" answer, is outside that limit. Nothing is filled in or scaled. A failed request, a response that is not 2xx, and a request that takes more than 60 seconds count the same. `--decision-timeout <seconds>` or `decisionTimeout` changes that limit. Raise it for a deployment of your own that starts cold, which can take a minute. The request is sent once more. After two failures no decision check of the run has a score, and the run is invalid.
218
+ - **Records.** `results.json` holds each decision check's score and `probabilities`, and for each run the number of requests with their duration and tokens, apart from the run's own. The API reports tokens and no cost, so the report shows no cost for the decision model.
219
+ - **The agent and the key.** When the suite has a decision check, `BASETEN_API_KEY` is removed from the environment of every harness process. An agent whose own model provider reads that variable cannot run in the same suite.
220
+
221
+ ## Scores
222
+
223
+ Every score is a number from 0 to 1.
224
+
225
+ | Score | What it is |
226
+ | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
227
+ | Check score | What one check gave one run. A deterministic check gives 0 or 1. A judged check or a decision check gives any value from 0 to 1. |
228
+ | Run score | The weighted mean of the run's check scores. A check's `weight` defaults to 1. An invalid run has no score. |
229
+ | Case score | The mean of the run scores of the valid runs, for one harness and one subject. |
230
+ | Suite score | The weighted mean of the case scores, for one harness and one subject. A case's `weight` defaults to 1. A suite has no score while one of its cases has none. |
231
+
232
+ There are two pass bars. `--min-score` is the lowest suite score that passes. `minScore` in a `case.json` is the lowest score of that case that passes. Both apply to the subject under test, which is B for `run` and the one subject for `score`, in every harness that ran. A missing score does not reach a bar. The report ends with the bars that were not reached.
233
+
234
+ An example with two cases and `--reps 2`:
235
+
236
+ | Case | Case weight | Checks | Run 1 | Run 2 | Case score |
237
+ | ------------- | ----------- | --------------------------------------------- | ---------------------------- | ---------------------------- | ------------------------ |
238
+ | `plain-words` | 1 | `skill_loaded` (weight 1) | 1 | 0 | (1 + 0) / 2 = 0.50 |
239
+ | `explain` | 2 | `judge` (weight 3), `output_lacks` (weight 1) | (3 × 0.8 + 1 × 1) / 4 = 0.85 | (3 × 0.6 + 1 × 0) / 4 = 0.45 | (0.85 + 0.45) / 2 = 0.65 |
240
+
241
+ The suite score is (1 × 0.50 + 2 × 0.65) / 3 = 0.60. With `--min-score 0.7` the command exits 1 and the report says that the suite score 0.60 does not reach 0.70.
242
+
243
+ ### Read the report
244
+
245
+ The report has one section per harness and one row per case. `run` shows each value as `A / B (B - A)`. `score` shows one value.
246
+
247
+ - **Score** is the case score.
248
+ - **Checks** shows each check. A deterministic check shows how many valid runs passed it, for example `2/3`. A judged check or a decision check shows its mean score, for example `0.67`. A decision check starts with the word `decision` and ends with the answer it expects or its options.
249
+ - **Duration, input tokens, output tokens, cost** show the median over valid runs. Input tokens include cached tokens. `n/a` means the harness does not report that number.
250
+ - **Suite score** is under the table.
251
+ - **Judge** is one line under the suite score with the number of judge calls and their total duration, tokens, and cost. These are not part of the table's numbers.
252
+ - **Decision model** is one more line of the same kind, with the number of requests to the decision model and their total duration and tokens.
253
+ - The first lines name each subject, the harness that ran, and why that harness was chosen.
254
+
255
+ One repetition tells you little. A check that passes 3 of 3 times on A and 1 of 3 on B is a signal. A check that passes 1 of 1 on A and 0 of 1 on B is not. Judged scores move between runs of the same subject, so read a difference in a judged check against that spread.
256
+
257
+ ## Config file
258
+
259
+ An optional `skill-evals.config.json` in the project root sets defaults. A flag wins over the file.
260
+
261
+ ```json
262
+ {
263
+ "cases": "evals/cases",
264
+ "harness": "claude-code",
265
+ "reps": 3,
266
+ "model": "<model-id>",
267
+ "judgeHarness": "claude-code",
268
+ "judgeModel": "<model-id>",
269
+ "decisionModel": "<decision-model-id>",
270
+ "decisionUrl": "<base-url>",
271
+ "decisionTimeout": 60,
272
+ "minScore": 0.8
273
+ }
274
+ ```
275
+
276
+ Every key is optional and is the default of the flag with the same name. `cases` resolves against the project root. `harness` narrows `check` too. An unknown key, a value of the wrong type, or invalid JSON is an error with exit code 2.
277
+
278
+ ## Use it from code
279
+
280
+ ```js
281
+ import { evaluate, loadCases, HARNESSES } from "skill-evals";
282
+
283
+ const evaluation = await evaluate({
284
+ cwd: "/path/to/project", // default: the working directory of the process
285
+ subjects: ["none", "./skills"], // one subject to score, or two to compare (A, then B)
286
+ model: "<model-id>",
287
+ reps: 3,
288
+ minScore: 0.8,
289
+ onRun: (record) => console.error(record.caseId, record.score),
290
+ });
291
+ if (evaluation.belowBar.length || evaluation.invalid.length)
292
+ process.exitCode = 1;
293
+ ```
294
+
295
+ | Export | What it is |
296
+ | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
297
+ | `evaluate(options)` | Does what `run` and `score` do, without the report. The options are `cwd`, `subjects`, `harness`, `cases`, `caseGlob`, `reps`, `model`, `judgeHarness`, `judgeModel`, `decisionModel`, `decisionUrl`, `decisionTimeout`, `minScore`, `outDir`, and `onRun`. Unset options take their value from the config file. |
298
+ | `loadCases(dir, glob)` | Reads and validates the cases in a directory. |
299
+ | `HARNESSES` | The harness adapters, in default-preference order. |
300
+
301
+ `evaluate` resolves to `{ records, suites, belowBar, invalid, outDir, subjects, cases, harnesses, harnessReason, judge, decisionModel, reps, model }`. `suites` holds the case and suite scores for each harness, keyed by the subject's label (`a` and `b`, or `subject`). `belowBar` lists the pass bars that were not reached, and `invalid` lists the runs that do not count. The evaluation passed when both are empty. Everything but `cases` and `outDir` is also in `<outDir>/results.json`.
302
+
303
+ The package ships type declarations, so `evaluate`'s options and what it resolves to are typed in TypeScript.
304
+
305
+ `evaluate` prints nothing and does not end the process. A wrong option, case file, or config file rejects with an error whose `name` is `UsageError`. While it runs, SIGINT and SIGTERM end the process with exit code 130 after the temp directories, the worktrees, and the harness's processes are removed.
306
+
307
+ ## Harnesses
308
+
309
+ `run` and `score` use the first installed harness in this order when you do not name one: `pi`, `claude-code`, `cursor`, `opencode`, `codex`. Pi leads because it has the smallest measured install (Pi 204 MB, OpenCode 412 MB, Codex 424 MB, Cursor 538 MB), starts in about a second, and reports tokens and cost.
310
+
311
+ | Harness | Executable | How a plugin is loaded | How the prompt is passed | Reports |
312
+ | ------------- | -------------- | ----------------------------------------------------------------------------- | ------------------------ | --------------------------------------- |
313
+ | `pi` | `pi` | `-e <dir>` with discovery switched off (`-ne -ns -np`) | on standard input | tokens, cost |
314
+ | `claude-code` | `claude` | `--plugin-dir <dir>` | after `--` | tokens, cost, loaded plugins and skills |
315
+ | `cursor` | `cursor-agent` | `--plugin-dir <dir>` | after `--` | neither tokens nor cost |
316
+ | `opencode` | `opencode` | a `plugin` entry in the new home's `opencode.json` | after `--` | tokens, cost |
317
+ | `codex` | `codex` | `codex plugin marketplace add <dir>` and `codex plugin add` into the new home | after `--` | tokens |
318
+
319
+ To use a different executable, set `SKILL_EVALS_BIN_<ID>` (for example `SKILL_EVALS_BIN_CLAUDE_CODE=/opt/bin/claude`). Use it when a version-manager shim needs your real home directory to find the binary.
320
+
321
+ ### What was verified
322
+
323
+ `skill-evals check` is how you confirm an adapter on your own machine. These tables say what was confirmed when the adapters were written, and how. Only Claude Code ran a real model. "Scripted endpoint" means that the real CLI ran through this runner against a local model endpoint that replays a fixed tool call and a fixed answer, with every request it received logged.
324
+
325
+ Loading a plugin, confirmed in the [magic](https://github.com/PhillipChaffee/magic) repository where this runner started, and run again from this package on 2026-11-21:
326
+
327
+ | Harness | Version | Confirmed | Not confirmed |
328
+ | ------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
329
+ | `claude-code` | 2.1.329, 2.1.330 | Real runs with a real model: `check`, `run` with a git ref, and `score`. A globally installed copy of the plugin did not load. The parser fixture is a captured stream. | A subscription login. The machine used authenticates through environment variables. |
330
+ | `pi` | 0.73.1 | Scripted endpoint: the command line is accepted, the plugin's extension and skills load from the directory, a decoy skill planted in the real home stays out, and the stream parses. The parser fixtures are captured streams. | A run with a real model. Cost for a priced model (the scripted model has no price). |
331
+ | `opencode` | 1.19.6 | Scripted endpoint: the plugin loads from the config entry, its skills are listed from the directory, a decoy skill stays out, and a native `skill` call parses. | A run with a real model. OpenCode V2, which reads the `plugins` key. |
332
+ | `codex` | 0.171.0 | Scripted endpoint: both install commands succeed in the new home, the plugin's skills are listed from the plugin cache, a decoy skill stays out, and a command that reads a `SKILL.md` parses. | A run with a real model. |
333
+ | `cursor` | 2026.11.19 | The CLI accepts the adapter's command line and stops at its login check. `check` reports that as a failed row and goes on. | Everything after login. The parser and its fixture come from Cursor's [output format reference](https://cursor.com/docs/cli/reference/output-format), not from a captured run. Whether `--plugin-dir` loads the plugin's skills and hooks in print mode. |
334
+
335
+ Loading a skills folder, the `none` baseline, and a prompt that starts with a dash, confirmed on 2026-11-21:
336
+
337
+ | Harness | Version | Skills folder | Prompt that starts with a dash | Not confirmed |
338
+ | ------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
339
+ | `claude-code` | 2.1.330 | Real runs with a real model: `run --a none --b <folder>` gave 0 of 2 skill loads without the folder and 2 of 2 with it. The harness listed the skill at startup, so a folder that does not load makes the run invalid. A skill of the same name in the real home's `~/.claude/skills` did not load. | A real run: after `--` the prompt reached the model. Without `--` the CLI rejects it as an unknown option. | A skill whose front matter `name` differs from its folder. |
340
+ | `pi` | 0.73.1 | Scripted endpoint: the copy named with `--skill` is in the model request with discovery off, and decoy skills in the real home's `~/.pi/agent/skills` and `~/.agents/skills` are not. | Scripted endpoint: the prompt on standard input is the user message, unchanged. Pi has no `--` separator and rejects such a prompt as an argument. | A run with a real model. |
341
+ | `opencode` | 1.19.6 | Scripted endpoint: the copy in `~/.agents/skills` is in the model request, and the decoys are not. | Scripted endpoint: after `--` the prompt is the user message. Without `--` the CLI prints its help. | A run with a real model. |
342
+ | `codex` | 0.171.0 | Scripted endpoint: the model request lists `~/.agents/skills` of the new home as a skill root with the copy in it, and the decoys are not there. | Scripted endpoint: after `--` the prompt is the user message. Without `--` the CLI rejects it. | A run with a real model. |
343
+ | `cursor` | 2026.11.19 | The installed bundle's code has `.cursor/skills` under the home directory as a place it reads skills from. No run confirmed it. | After `--` the CLI gets past its option parser and stops at its login check. Without `--` it rejects the prompt. | Everything after login, so whether the skill and the prompt reach the model. |
344
+
345
+ With `none`, the scripted requests of Pi, OpenCode, and Codex held neither the subject's skill nor a decoy.
346
+
347
+ Tool kinds and the judge:
348
+
349
+ | Harness | Tool kinds | Judge run with nothing loaded |
350
+ | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
351
+ | `claude-code` | `Skill`, `Bash`, `Read`, `Edit`, `Write`, `WebFetch`, and `Agent` come from a captured run, which is a parser fixture. `Grep`, `Glob`, `WebSearch`, `NotebookEdit`, and `Task` are mapped but were not seen in a stream. Version 2.1.330 searched through `Bash`. | Real runs. |
352
+ | `pi` | The seven tool names come from the installed package (0.73.1). `read` was seen in a captured stream. | The same command line as the `none` baseline, which ran against the scripted endpoint. No judge reply was parsed. |
353
+ | `opencode` | The names come from the installed binary (1.19.6). `skill` and `read` were seen in a stream. | As for Pi. |
354
+ | `codex` | The item types come from the Codex source for `exec --json` events. `command_execution` was seen in a captured stream. | As for Pi. |
355
+ | `cursor` | The names come from the installed bundle (2026.11.19). `read` and `write` are in the documented example stream. No run confirmed them. | Not run. |
356
+
357
+ The decision model:
358
+
359
+ | Part | Confirmed | Not confirmed |
360
+ | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
361
+ | Baseten System One client | Written from the [System One API reference](https://docs.baseten.co/reference/inference-api/system-one) and the [decision models guide](https://docs.baseten.co/inference/model-apis/decision-models) as they were on 2026-11-21. The tests run the client against a local HTTP server that answers in the documented response shape: the request body, the `Authorization` header, the scoring, the retry, and every unusable reply. The client was also run against bison-1, self-hosted on Modal with the runtime that Baseten publishes for its sibling models. Yes, no, and options checks scored, the options with labels that hold spaces. The reply reported token usage. A warm request took about 1.2 seconds. Repeated gradings of a fixed answer gave identical scores. | Baseten's hosted API itself, including its auth scheme. No request from this client has reached it. |
362
+
363
+ Two limits apply across harnesses. A login stored in the operating system's keychain may not follow the new home. If `check` reports a login error, set the harness's API key variable. Only Claude Code reports what it loaded, as [What one run does](#what-one-run-does) says.
364
+
365
+ ### Add a harness
366
+
367
+ Adapters live in this package. Loading one from your own code is not supported yet.
368
+
369
+ 1. Add `src/harnesses/<id>.ts` whose default export has the five fields of the `Harness` type in `src/harnesses/shared.ts`: `id`, `bin`, `authFiles`, `invocation`, `parse`. Declare the stream events that `parse` reads as a type in the same file.
370
+ 2. In `invocation`, load a plugin subject the way the harness loads a local plugin (`pluginDir`), copy a skills subject to where the harness discovers a user's skills (`placeSkills`), and point the harness's own config variables into `home`. Return `setup` commands when the harness has to install the plugin before it runs. When `subject` is `null`, load nothing. The judge and the `none` baseline run that way.
371
+ 3. Pass the prompt so that a leading dash cannot be read as a flag: after a `--` separator, or in `stdin` when the CLI has no separator. Confirm it against the real CLI.
372
+ 4. In `parse`, normalize the stream and nothing more. Return `null` for a field the harness does not report. Put only a tool call's input in `toolCalls`, never its output, or file contents will count as skill loads. Build each tool call with `toolCall` and a table from the harness's tool names to the kinds under [Tool checks](#tool-checks).
373
+ 5. Add the adapter to `HARNESSES` in `src/harnesses/index.ts` at its place in the preference order.
374
+ 6. Capture one real stream into `test/fixtures/` and add it to `src/adapters.test.ts`. Add the harness to the table of skill locations in `src/invocation.test.ts`, and add a row to every table above.
375
+
376
+ ## Develop
377
+
378
+ The sources are TypeScript in `src/`, with each test beside the code it covers. The lint, type, documentation, format, coverage, and hygiene gates come from the `typescript/` folder of [full-lint](https://github.com/PhillipChaffee/full-lint), whose README gives the reason for every rule.
379
+
380
+ ```sh
381
+ npm ci
382
+ npm run format:check # prettier
383
+ npm run typecheck # tsc, strict and eight more flags
384
+ npm run lint # eslint: typed rules, doc comments, size and complexity limits
385
+ npm test # vitest; fails under 95% coverage of lines, functions, branches, or statements
386
+ npm run knip # unused files, exports, and dependencies
387
+ npm run lint:lockfile # every package comes from the npm registry over HTTPS
388
+ npm run lint:graph # no import cycles, no orphan modules
389
+ npm run build # compiles src/ to dist/, which is what the package ships
390
+ ./run-gates.sh # all of the above but the build, and the hygiene tools that CI installs
391
+ ```
392
+
393
+ The tests run this package's code in the test process and start a stand-in harness from `test/fixtures/` as a real child process. One test packs the package, installs the tarball into an empty project, and runs it there.
394
+
395
+ `npm pack` and `npm publish` build first, so the tarball always holds a fresh `dist/`.
396
+
397
+ ## Release
398
+
399
+ Pushing a version tag publishes the package to npm. Nothing on `main` makes a tag or publishes.
400
+
401
+ 1. Open a pull request that sets `version` in `package.json` (and `package-lock.json`) and adds a `## <version>` entry to `CHANGELOG.md`. The `Release check` workflow fails the pull request when the version could not ship.
402
+ 2. Merge it.
403
+ 3. Tag the merge commit and push the tag:
404
+
405
+ ```sh
406
+ git tag v<version> <commit> && git push origin v<version>
407
+ ```
408
+
409
+ The `Publish` workflow then runs the guards in `scripts/release-plan.sh`, every gate above, `npm publish --provenance`, and makes the GitHub release from the changelog entry. It refuses a tag when:
410
+
411
+ - the tag is not `v` plus the version in `package.json` at that commit
412
+ - the tagged commit is not on `main`
413
+ - the version is not `MAJOR.MINOR.PATCH`, or has no changelog entry
414
+ - the version is not above the latest one on npm
415
+ - the registry cannot be reached
416
+
417
+ A tag whose version is already on npm publishes nothing, so running the workflow again is safe.
418
+
419
+ Two guards live in the repository's settings and not in this repository:
420
+
421
+ - **Who may push a tag.** A tag ruleset for `v*` limits who can create, move, or delete a version tag.
422
+ - **A person's approval.** The workflow runs in the `npm` environment. Required reviewers on that environment make every release wait for an approval.
423
+
424
+ The workflow reads the npm token from the `NPM_TOKEN` secret.
425
+
426
+ ## License
427
+
428
+ MIT
@@ -0,0 +1,20 @@
1
+ import type { DecisionService } from "./decision.js";
2
+ /** How to reach the decision model. */
3
+ interface Connection {
4
+ /** The model slug, sent unchanged. */
5
+ model: string;
6
+ /** The base URL of another endpoint that speaks the same API, such as a deployment of your own. */
7
+ url?: string | undefined;
8
+ /** Where the API key is read from. */
9
+ env: Record<string, string | undefined>;
10
+ /** How long one request may take. Default: 60 seconds. */
11
+ timeoutMs?: number | undefined;
12
+ }
13
+ /**
14
+ * Connects to a decision model behind Baseten's System One API.
15
+ * @param connection - The model, the endpoint, and where the key is.
16
+ * @returns The model, which sends each request as one POST, and the name of the key's variable.
17
+ * @throws {UsageError} When the key's variable is not set, or the URL is not one.
18
+ */
19
+ export declare function connect(connection: Connection): DecisionService;
20
+ export {};