spec-eval 0.2.0__tar.gz → 0.2.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {spec_eval-0.2.0/spec_eval.egg-info → spec_eval-0.2.1}/PKG-INFO +38 -20
- {spec_eval-0.2.0 → spec_eval-0.2.1}/README.md +37 -19
- {spec_eval-0.2.0 → spec_eval-0.2.1}/spec_eval/__init__.py +1 -1
- {spec_eval-0.2.0 → spec_eval-0.2.1/spec_eval.egg-info}/PKG-INFO +38 -20
- {spec_eval-0.2.0 → spec_eval-0.2.1}/LICENSE +0 -0
- {spec_eval-0.2.0 → spec_eval-0.2.1}/pyproject.toml +0 -0
- {spec_eval-0.2.0 → spec_eval-0.2.1}/setup.cfg +0 -0
- {spec_eval-0.2.0 → spec_eval-0.2.1}/spec_eval/__main__.py +0 -0
- {spec_eval-0.2.0 → spec_eval-0.2.1}/spec_eval/audit.py +0 -0
- {spec_eval-0.2.0 → spec_eval-0.2.1}/spec_eval/authoring.py +0 -0
- {spec_eval-0.2.0 → spec_eval-0.2.1}/spec_eval/cli.py +0 -0
- {spec_eval-0.2.0 → spec_eval-0.2.1}/spec_eval/coverage.py +0 -0
- {spec_eval-0.2.0 → spec_eval-0.2.1}/spec_eval/providers.py +0 -0
- {spec_eval-0.2.0 → spec_eval-0.2.1}/spec_eval/report.py +0 -0
- {spec_eval-0.2.0 → spec_eval-0.2.1}/spec_eval/rubric.py +0 -0
- {spec_eval-0.2.0 → spec_eval-0.2.1}/spec_eval/runlog.py +0 -0
- {spec_eval-0.2.0 → spec_eval-0.2.1}/spec_eval/sufficiency.py +0 -0
- {spec_eval-0.2.0 → spec_eval-0.2.1}/spec_eval.egg-info/SOURCES.txt +0 -0
- {spec_eval-0.2.0 → spec_eval-0.2.1}/spec_eval.egg-info/dependency_links.txt +0 -0
- {spec_eval-0.2.0 → spec_eval-0.2.1}/spec_eval.egg-info/entry_points.txt +0 -0
- {spec_eval-0.2.0 → spec_eval-0.2.1}/spec_eval.egg-info/requires.txt +0 -0
- {spec_eval-0.2.0 → spec_eval-0.2.1}/spec_eval.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: spec-eval
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.1
|
|
4
4
|
Summary: A trust-eval for your specs: coverage, drift, and sufficiency of markdown specs against code.
|
|
5
5
|
Author: Ben Jones
|
|
6
6
|
License: MIT
|
|
@@ -35,7 +35,7 @@ Dynamic: license-file
|
|
|
35
35
|
|
|
36
36
|
*A trust-eval for your specs — coverage, drift, and sufficiency, measured against the code.*
|
|
37
37
|
|
|
38
|
-
[Get started](#try-it) · [Tutorial](GETTING-STARTED.md) · [No API key?](#run-via-prompt-chat-no-setup) · [Second opinion](#higher-stakes-get-a-second-opinion) · [FAQ](FAQ.md)
|
|
38
|
+
[Get started](#try-it) · [Tutorial](https://github.com/benjaminjones/spec-eval/blob/main/GETTING-STARTED.md) · [No API key?](#run-via-prompt-chat-no-setup) · [Second opinion](#higher-stakes-get-a-second-opinion) · [FAQ](https://github.com/benjaminjones/spec-eval/blob/main/FAQ.md)
|
|
39
39
|
|
|
40
40
|
A **spec** is short for **specification** — a plain-English description of what your code should do. **spec-eval**
|
|
41
41
|
keeps one beside your code — per file, or per folder — and checks the two still agree, so the spec stays worth
|
|
@@ -98,7 +98,7 @@ Everything it writes is plain markdown, right next to your code:
|
|
|
98
98
|
|
|
99
99
|
## Try it
|
|
100
100
|
|
|
101
|
-
Prefer a guided walkthrough? **[Getting-started tutorial →](GETTING-STARTED.md)**
|
|
101
|
+
Prefer a guided walkthrough? **[Getting-started tutorial →](https://github.com/benjaminjones/spec-eval/blob/main/GETTING-STARTED.md)**
|
|
102
102
|
|
|
103
103
|
**The commands** — in the order you'll typically use them:
|
|
104
104
|
|
|
@@ -116,12 +116,30 @@ Three ways to run them, least setup first:
|
|
|
116
116
|
### Run via prompt chat (no setup)
|
|
117
117
|
|
|
118
118
|
Ask the coding agent you already have (Claude Code, Copilot, Cursor, …) — nothing to install, no key, answers
|
|
119
|
-
land in the chat:
|
|
119
|
+
land in the chat. Two skills, one per job — point your agent at the right one:
|
|
120
|
+
|
|
121
|
+
**Write specs** — the [authoring skill](https://github.com/benjaminjones/spec-eval/blob/main/skills/spec-authoring/SKILL.md) carries the spec structure
|
|
122
|
+
(the `INV-*` invariant and `AC-*` acceptance-criteria tables come from its rubric and templates):
|
|
123
|
+
```text
|
|
124
|
+
Read path/to/spec-eval/skills/spec-authoring/SKILL.md — and the templates
|
|
125
|
+
in its templates/ folder — and follow it: write specs for my code.
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
**Check specs** — the [checking skill](https://github.com/benjaminjones/spec-eval/blob/main/skills/spec-check/SKILL.md) grades what exists (drift + sufficiency):
|
|
120
129
|
```text
|
|
121
130
|
Read path/to/spec-eval/skills/spec-check/SKILL.md and follow it —
|
|
122
131
|
first check coverage, then check my specs against my code.
|
|
123
132
|
```
|
|
124
|
-
|
|
133
|
+
|
|
134
|
+
> [!NOTE]
|
|
135
|
+
> `path/to/spec-eval` is wherever you cloned this repo. No clone? Give your agent the skill's link by name:
|
|
136
|
+
> [`spec-authoring/SKILL.md`](https://github.com/benjaminjones/spec-eval/blob/main/skills/spec-authoring/SKILL.md) to write specs,
|
|
137
|
+
> [`spec-check/SKILL.md`](https://github.com/benjaminjones/spec-eval/blob/main/skills/spec-check/SKILL.md) to check them.
|
|
138
|
+
|
|
139
|
+
> [!TIP]
|
|
140
|
+
> **Author and check in separate sessions.** A checker that still remembers its authoring decisions tends to
|
|
141
|
+
> echo them instead of re-reading the code — the same reason a PR isn't reviewed by its author. Write specs in
|
|
142
|
+
> one session; open a fresh one to check them.
|
|
125
143
|
|
|
126
144
|
Optionally save reports — add:
|
|
127
145
|
```text
|
|
@@ -129,7 +147,7 @@ and save the results to spec-reports/
|
|
|
129
147
|
```
|
|
130
148
|
The agent writes them itself (its own scores; the terminal adds the exact coverage % and a run history).
|
|
131
149
|
|
|
132
|
-
**Make it a standing command** *(optional, one-time)* — copy the [`skills/`](skills/) into your agent's skills
|
|
150
|
+
**Make it a standing command** *(optional, one-time)* — copy the [`skills/`](https://github.com/benjaminjones/spec-eval/tree/main/skills/) into your agent's skills
|
|
133
151
|
folder (Claude Code: `.claude/skills/`). Then the prompt is the command:
|
|
134
152
|
```bash
|
|
135
153
|
mkdir -p .claude/skills && cp -r path/to/spec-eval/skills/* .claude/skills/
|
|
@@ -164,7 +182,7 @@ export ANTHROPIC_API_KEY=sk-ant-... # or OPENAI_API_KEY / GOOGLE_API_KEY
|
|
|
164
182
|
|
|
165
183
|
> [!NOTE]
|
|
166
184
|
> A typical first run is `coverage → generate → sufficiency`, then at every milestone after: `audit` — plus
|
|
167
|
-
> `sufficiency` when you've added behavior. The [tutorial](GETTING-STARTED.md) has step-by-step guides for
|
|
185
|
+
> `sufficiency` when you've added behavior. The [tutorial](https://github.com/benjaminjones/spec-eval/blob/main/GETTING-STARTED.md) has step-by-step guides for
|
|
168
186
|
> starting with no specs and for existing docs.
|
|
169
187
|
|
|
170
188
|
> [!TIP]
|
|
@@ -175,7 +193,7 @@ export ANTHROPIC_API_KEY=sk-ant-... # or OPENAI_API_KEY / GOOGLE_API_KEY
|
|
|
175
193
|
A few things worth knowing:
|
|
176
194
|
|
|
177
195
|
- **No config needed** — by default each spec pairs with the code file of the same name beside it (`parser.md` ↔ `parser.py`).
|
|
178
|
-
Prefer one spec per folder? `--layout per-dir` ([FAQ](FAQ.md#can-i-use-one-spec-per-folder-instead-of-one-per-file)).
|
|
196
|
+
Prefer one spec per folder? `--layout per-dir` ([FAQ](https://github.com/benjaminjones/spec-eval/blob/main/FAQ.md#can-i-use-one-spec-per-folder-instead-of-one-per-file)).
|
|
179
197
|
- **Any language, three providers** — swap the model with `--model openai:…`, `google:…`, or `claude-code`
|
|
180
198
|
(Anthropic is the default); code is read as plain text. Want another provider? [Open an issue](#feedback).
|
|
181
199
|
- **Your other docs are safe** — by default only a `.md` next to code with the same name counts as a spec, so your
|
|
@@ -183,7 +201,7 @@ A few things worth knowing:
|
|
|
183
201
|
- **What's shared** — to grade a spec, its code and text go to your AI provider. That's it. (`coverage` sends nothing — it runs fully on your machine.)
|
|
184
202
|
|
|
185
203
|
**Keep it honest on every commit** — add one line to a git hook (a script git runs before each commit) so new
|
|
186
|
-
code always needs a spec; copy-paste setup in [the tutorial](GETTING-STARTED.md#make-it-routine):
|
|
204
|
+
code always needs a spec; copy-paste setup in [the tutorial](https://github.com/benjaminjones/spec-eval/blob/main/GETTING-STARTED.md#make-it-routine):
|
|
187
205
|
```bash
|
|
188
206
|
spec-eval coverage . --min 90 # fails the commit if coverage drops below 90% (free — no AI)
|
|
189
207
|
```
|
|
@@ -214,20 +232,20 @@ the pair against a fixed rubric — one model call per pair. Two things follow f
|
|
|
214
232
|
> your coding agent's own usage if you asked in chat. Every CLI run prints its exact call and token counts.
|
|
215
233
|
|
|
216
234
|
**Want the exact rules behind each score?** spec-eval specs its own scoring — read
|
|
217
|
-
[`spec_eval/coverage.md`](spec_eval/coverage.md) (how coverage counts),
|
|
218
|
-
[`spec_eval/rubric.md`](spec_eval/rubric.md) (the drift rules), and
|
|
219
|
-
[`spec_eval/sufficiency.md`](spec_eval/sufficiency.md) (how the 0–1 score is decided).
|
|
235
|
+
[`spec_eval/coverage.md`](https://github.com/benjaminjones/spec-eval/blob/main/spec_eval/coverage.md) (how coverage counts),
|
|
236
|
+
[`spec_eval/rubric.md`](https://github.com/benjaminjones/spec-eval/blob/main/spec_eval/rubric.md) (the drift rules), and
|
|
237
|
+
[`spec_eval/sufficiency.md`](https://github.com/benjaminjones/spec-eval/blob/main/spec_eval/sufficiency.md) (how the 0–1 score is decided).
|
|
220
238
|
|
|
221
239
|
## Reading the scores
|
|
222
240
|
|
|
223
241
|
Each run writes a short report to `spec-reports/` (a `.md` you read, a `.json` for tools):
|
|
224
242
|
|
|
225
|
-
- **coverage** — the % of your code that has a spec, and which files don't. *(live example: [coverage.md](spec-reports/coverage.md))*
|
|
226
|
-
- **drift** — each place a spec and the code disagree, with a suggested fix. `0` is clean. *(live example: [report.md](spec-reports/report.md))*
|
|
243
|
+
- **coverage** — the % of your code that has a spec, and which files don't. *(live example: [coverage.md](https://github.com/benjaminjones/spec-eval/blob/main/spec-reports/coverage.md))*
|
|
244
|
+
- **drift** — each place a spec and the code disagree, with a suggested fix. `0` is clean. *(live example: [report.md](https://github.com/benjaminjones/spec-eval/blob/main/spec-reports/report.md))*
|
|
227
245
|
- **sufficiency** — a `0`–`1` score per spec (worst first), listing what's missing with a searchable code pointer
|
|
228
|
-
(`file.py (function)`). `1.0` = the grader found nothing missing — guidance, not a guarantee. *(live example: [sufficiency.md](spec-reports/sufficiency.md))*
|
|
246
|
+
(`file.py (function)`). `1.0` = the grader found nothing missing — guidance, not a guarantee. *(live example: [sufficiency.md](https://github.com/benjaminjones/spec-eval/blob/main/spec-reports/sufficiency.md))*
|
|
229
247
|
|
|
230
|
-
The three examples are **spec-eval grading itself**, rolled up in [SPEC-HEALTH.md](SPEC-HEALTH.md).
|
|
248
|
+
The three examples are **spec-eval grading itself**, rolled up in [SPEC-HEALTH.md](https://github.com/benjaminjones/spec-eval/blob/main/SPEC-HEALTH.md).
|
|
231
249
|
|
|
232
250
|
Every command also appends a line to `runs.jsonl` — timestamp, git commit, scores — so you can track change over
|
|
233
251
|
time.
|
|
@@ -258,8 +276,8 @@ so put both keys in one `.env` and each run picks up the one it needs. (Want one
|
|
|
258
276
|
|
|
259
277
|
## Proof it works
|
|
260
278
|
|
|
261
|
-
- **spec-eval specs itself** — every module in [`spec_eval/`](spec_eval/) has its own spec:
|
|
262
|
-
**coverage 100% · drift 0 · sufficiency ≈0.86** ([receipt](SPEC-HEALTH.md) — the exact, dated score).
|
|
279
|
+
- **spec-eval specs itself** — every module in [`spec_eval/`](https://github.com/benjaminjones/spec-eval/tree/main/spec_eval/) has its own spec:
|
|
280
|
+
**coverage 100% · drift 0 · sufficiency ≈0.86** ([receipt](https://github.com/benjaminjones/spec-eval/blob/main/SPEC-HEALTH.md) — the exact, dated score).
|
|
263
281
|
|
|
264
282
|
## More
|
|
265
283
|
|
|
@@ -271,13 +289,13 @@ so put both keys in one `.env` and each run picks up the one it needs. (Want one
|
|
|
271
289
|
agent writes the code; the check is where they meet), so the spec is *born accurate* instead of written up
|
|
272
290
|
afterward.
|
|
273
291
|
- **Just want to vibe code?** Fine — build first, check later, automatically: a pre-commit gate and a GitHub
|
|
274
|
-
Action can run the checks for you — see [Make it routine](GETTING-STARTED.md#make-it-routine).
|
|
292
|
+
Action can run the checks for you — see [Make it routine](https://github.com/benjaminjones/spec-eval/blob/main/GETTING-STARTED.md#make-it-routine).
|
|
275
293
|
- **Higher-stakes spec?** Run it past two vendors and compare — see [Higher stakes? Get a second opinion](#higher-stakes-get-a-second-opinion) above.
|
|
276
294
|
|
|
277
295
|
## FAQ
|
|
278
296
|
|
|
279
297
|
Common questions — from *"which command do I run first?"* through costs, layouts, and score accuracy — live in
|
|
280
|
-
**[FAQ.md](FAQ.md)**.
|
|
298
|
+
**[FAQ.md](https://github.com/benjaminjones/spec-eval/blob/main/FAQ.md)**.
|
|
281
299
|
|
|
282
300
|
## Feedback
|
|
283
301
|
|
|
@@ -6,7 +6,7 @@
|
|
|
6
6
|
|
|
7
7
|
*A trust-eval for your specs — coverage, drift, and sufficiency, measured against the code.*
|
|
8
8
|
|
|
9
|
-
[Get started](#try-it) · [Tutorial](GETTING-STARTED.md) · [No API key?](#run-via-prompt-chat-no-setup) · [Second opinion](#higher-stakes-get-a-second-opinion) · [FAQ](FAQ.md)
|
|
9
|
+
[Get started](#try-it) · [Tutorial](https://github.com/benjaminjones/spec-eval/blob/main/GETTING-STARTED.md) · [No API key?](#run-via-prompt-chat-no-setup) · [Second opinion](#higher-stakes-get-a-second-opinion) · [FAQ](https://github.com/benjaminjones/spec-eval/blob/main/FAQ.md)
|
|
10
10
|
|
|
11
11
|
A **spec** is short for **specification** — a plain-English description of what your code should do. **spec-eval**
|
|
12
12
|
keeps one beside your code — per file, or per folder — and checks the two still agree, so the spec stays worth
|
|
@@ -69,7 +69,7 @@ Everything it writes is plain markdown, right next to your code:
|
|
|
69
69
|
|
|
70
70
|
## Try it
|
|
71
71
|
|
|
72
|
-
Prefer a guided walkthrough? **[Getting-started tutorial →](GETTING-STARTED.md)**
|
|
72
|
+
Prefer a guided walkthrough? **[Getting-started tutorial →](https://github.com/benjaminjones/spec-eval/blob/main/GETTING-STARTED.md)**
|
|
73
73
|
|
|
74
74
|
**The commands** — in the order you'll typically use them:
|
|
75
75
|
|
|
@@ -87,12 +87,30 @@ Three ways to run them, least setup first:
|
|
|
87
87
|
### Run via prompt chat (no setup)
|
|
88
88
|
|
|
89
89
|
Ask the coding agent you already have (Claude Code, Copilot, Cursor, …) — nothing to install, no key, answers
|
|
90
|
-
land in the chat:
|
|
90
|
+
land in the chat. Two skills, one per job — point your agent at the right one:
|
|
91
|
+
|
|
92
|
+
**Write specs** — the [authoring skill](https://github.com/benjaminjones/spec-eval/blob/main/skills/spec-authoring/SKILL.md) carries the spec structure
|
|
93
|
+
(the `INV-*` invariant and `AC-*` acceptance-criteria tables come from its rubric and templates):
|
|
94
|
+
```text
|
|
95
|
+
Read path/to/spec-eval/skills/spec-authoring/SKILL.md — and the templates
|
|
96
|
+
in its templates/ folder — and follow it: write specs for my code.
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
**Check specs** — the [checking skill](https://github.com/benjaminjones/spec-eval/blob/main/skills/spec-check/SKILL.md) grades what exists (drift + sufficiency):
|
|
91
100
|
```text
|
|
92
101
|
Read path/to/spec-eval/skills/spec-check/SKILL.md and follow it —
|
|
93
102
|
first check coverage, then check my specs against my code.
|
|
94
103
|
```
|
|
95
|
-
|
|
104
|
+
|
|
105
|
+
> [!NOTE]
|
|
106
|
+
> `path/to/spec-eval` is wherever you cloned this repo. No clone? Give your agent the skill's link by name:
|
|
107
|
+
> [`spec-authoring/SKILL.md`](https://github.com/benjaminjones/spec-eval/blob/main/skills/spec-authoring/SKILL.md) to write specs,
|
|
108
|
+
> [`spec-check/SKILL.md`](https://github.com/benjaminjones/spec-eval/blob/main/skills/spec-check/SKILL.md) to check them.
|
|
109
|
+
|
|
110
|
+
> [!TIP]
|
|
111
|
+
> **Author and check in separate sessions.** A checker that still remembers its authoring decisions tends to
|
|
112
|
+
> echo them instead of re-reading the code — the same reason a PR isn't reviewed by its author. Write specs in
|
|
113
|
+
> one session; open a fresh one to check them.
|
|
96
114
|
|
|
97
115
|
Optionally save reports — add:
|
|
98
116
|
```text
|
|
@@ -100,7 +118,7 @@ and save the results to spec-reports/
|
|
|
100
118
|
```
|
|
101
119
|
The agent writes them itself (its own scores; the terminal adds the exact coverage % and a run history).
|
|
102
120
|
|
|
103
|
-
**Make it a standing command** *(optional, one-time)* — copy the [`skills/`](skills/) into your agent's skills
|
|
121
|
+
**Make it a standing command** *(optional, one-time)* — copy the [`skills/`](https://github.com/benjaminjones/spec-eval/tree/main/skills/) into your agent's skills
|
|
104
122
|
folder (Claude Code: `.claude/skills/`). Then the prompt is the command:
|
|
105
123
|
```bash
|
|
106
124
|
mkdir -p .claude/skills && cp -r path/to/spec-eval/skills/* .claude/skills/
|
|
@@ -135,7 +153,7 @@ export ANTHROPIC_API_KEY=sk-ant-... # or OPENAI_API_KEY / GOOGLE_API_KEY
|
|
|
135
153
|
|
|
136
154
|
> [!NOTE]
|
|
137
155
|
> A typical first run is `coverage → generate → sufficiency`, then at every milestone after: `audit` — plus
|
|
138
|
-
> `sufficiency` when you've added behavior. The [tutorial](GETTING-STARTED.md) has step-by-step guides for
|
|
156
|
+
> `sufficiency` when you've added behavior. The [tutorial](https://github.com/benjaminjones/spec-eval/blob/main/GETTING-STARTED.md) has step-by-step guides for
|
|
139
157
|
> starting with no specs and for existing docs.
|
|
140
158
|
|
|
141
159
|
> [!TIP]
|
|
@@ -146,7 +164,7 @@ export ANTHROPIC_API_KEY=sk-ant-... # or OPENAI_API_KEY / GOOGLE_API_KEY
|
|
|
146
164
|
A few things worth knowing:
|
|
147
165
|
|
|
148
166
|
- **No config needed** — by default each spec pairs with the code file of the same name beside it (`parser.md` ↔ `parser.py`).
|
|
149
|
-
Prefer one spec per folder? `--layout per-dir` ([FAQ](FAQ.md#can-i-use-one-spec-per-folder-instead-of-one-per-file)).
|
|
167
|
+
Prefer one spec per folder? `--layout per-dir` ([FAQ](https://github.com/benjaminjones/spec-eval/blob/main/FAQ.md#can-i-use-one-spec-per-folder-instead-of-one-per-file)).
|
|
150
168
|
- **Any language, three providers** — swap the model with `--model openai:…`, `google:…`, or `claude-code`
|
|
151
169
|
(Anthropic is the default); code is read as plain text. Want another provider? [Open an issue](#feedback).
|
|
152
170
|
- **Your other docs are safe** — by default only a `.md` next to code with the same name counts as a spec, so your
|
|
@@ -154,7 +172,7 @@ A few things worth knowing:
|
|
|
154
172
|
- **What's shared** — to grade a spec, its code and text go to your AI provider. That's it. (`coverage` sends nothing — it runs fully on your machine.)
|
|
155
173
|
|
|
156
174
|
**Keep it honest on every commit** — add one line to a git hook (a script git runs before each commit) so new
|
|
157
|
-
code always needs a spec; copy-paste setup in [the tutorial](GETTING-STARTED.md#make-it-routine):
|
|
175
|
+
code always needs a spec; copy-paste setup in [the tutorial](https://github.com/benjaminjones/spec-eval/blob/main/GETTING-STARTED.md#make-it-routine):
|
|
158
176
|
```bash
|
|
159
177
|
spec-eval coverage . --min 90 # fails the commit if coverage drops below 90% (free — no AI)
|
|
160
178
|
```
|
|
@@ -185,20 +203,20 @@ the pair against a fixed rubric — one model call per pair. Two things follow f
|
|
|
185
203
|
> your coding agent's own usage if you asked in chat. Every CLI run prints its exact call and token counts.
|
|
186
204
|
|
|
187
205
|
**Want the exact rules behind each score?** spec-eval specs its own scoring — read
|
|
188
|
-
[`spec_eval/coverage.md`](spec_eval/coverage.md) (how coverage counts),
|
|
189
|
-
[`spec_eval/rubric.md`](spec_eval/rubric.md) (the drift rules), and
|
|
190
|
-
[`spec_eval/sufficiency.md`](spec_eval/sufficiency.md) (how the 0–1 score is decided).
|
|
206
|
+
[`spec_eval/coverage.md`](https://github.com/benjaminjones/spec-eval/blob/main/spec_eval/coverage.md) (how coverage counts),
|
|
207
|
+
[`spec_eval/rubric.md`](https://github.com/benjaminjones/spec-eval/blob/main/spec_eval/rubric.md) (the drift rules), and
|
|
208
|
+
[`spec_eval/sufficiency.md`](https://github.com/benjaminjones/spec-eval/blob/main/spec_eval/sufficiency.md) (how the 0–1 score is decided).
|
|
191
209
|
|
|
192
210
|
## Reading the scores
|
|
193
211
|
|
|
194
212
|
Each run writes a short report to `spec-reports/` (a `.md` you read, a `.json` for tools):
|
|
195
213
|
|
|
196
|
-
- **coverage** — the % of your code that has a spec, and which files don't. *(live example: [coverage.md](spec-reports/coverage.md))*
|
|
197
|
-
- **drift** — each place a spec and the code disagree, with a suggested fix. `0` is clean. *(live example: [report.md](spec-reports/report.md))*
|
|
214
|
+
- **coverage** — the % of your code that has a spec, and which files don't. *(live example: [coverage.md](https://github.com/benjaminjones/spec-eval/blob/main/spec-reports/coverage.md))*
|
|
215
|
+
- **drift** — each place a spec and the code disagree, with a suggested fix. `0` is clean. *(live example: [report.md](https://github.com/benjaminjones/spec-eval/blob/main/spec-reports/report.md))*
|
|
198
216
|
- **sufficiency** — a `0`–`1` score per spec (worst first), listing what's missing with a searchable code pointer
|
|
199
|
-
(`file.py (function)`). `1.0` = the grader found nothing missing — guidance, not a guarantee. *(live example: [sufficiency.md](spec-reports/sufficiency.md))*
|
|
217
|
+
(`file.py (function)`). `1.0` = the grader found nothing missing — guidance, not a guarantee. *(live example: [sufficiency.md](https://github.com/benjaminjones/spec-eval/blob/main/spec-reports/sufficiency.md))*
|
|
200
218
|
|
|
201
|
-
The three examples are **spec-eval grading itself**, rolled up in [SPEC-HEALTH.md](SPEC-HEALTH.md).
|
|
219
|
+
The three examples are **spec-eval grading itself**, rolled up in [SPEC-HEALTH.md](https://github.com/benjaminjones/spec-eval/blob/main/SPEC-HEALTH.md).
|
|
202
220
|
|
|
203
221
|
Every command also appends a line to `runs.jsonl` — timestamp, git commit, scores — so you can track change over
|
|
204
222
|
time.
|
|
@@ -229,8 +247,8 @@ so put both keys in one `.env` and each run picks up the one it needs. (Want one
|
|
|
229
247
|
|
|
230
248
|
## Proof it works
|
|
231
249
|
|
|
232
|
-
- **spec-eval specs itself** — every module in [`spec_eval/`](spec_eval/) has its own spec:
|
|
233
|
-
**coverage 100% · drift 0 · sufficiency ≈0.86** ([receipt](SPEC-HEALTH.md) — the exact, dated score).
|
|
250
|
+
- **spec-eval specs itself** — every module in [`spec_eval/`](https://github.com/benjaminjones/spec-eval/tree/main/spec_eval/) has its own spec:
|
|
251
|
+
**coverage 100% · drift 0 · sufficiency ≈0.86** ([receipt](https://github.com/benjaminjones/spec-eval/blob/main/SPEC-HEALTH.md) — the exact, dated score).
|
|
234
252
|
|
|
235
253
|
## More
|
|
236
254
|
|
|
@@ -242,13 +260,13 @@ so put both keys in one `.env` and each run picks up the one it needs. (Want one
|
|
|
242
260
|
agent writes the code; the check is where they meet), so the spec is *born accurate* instead of written up
|
|
243
261
|
afterward.
|
|
244
262
|
- **Just want to vibe code?** Fine — build first, check later, automatically: a pre-commit gate and a GitHub
|
|
245
|
-
Action can run the checks for you — see [Make it routine](GETTING-STARTED.md#make-it-routine).
|
|
263
|
+
Action can run the checks for you — see [Make it routine](https://github.com/benjaminjones/spec-eval/blob/main/GETTING-STARTED.md#make-it-routine).
|
|
246
264
|
- **Higher-stakes spec?** Run it past two vendors and compare — see [Higher stakes? Get a second opinion](#higher-stakes-get-a-second-opinion) above.
|
|
247
265
|
|
|
248
266
|
## FAQ
|
|
249
267
|
|
|
250
268
|
Common questions — from *"which command do I run first?"* through costs, layouts, and score accuracy — live in
|
|
251
|
-
**[FAQ.md](FAQ.md)**.
|
|
269
|
+
**[FAQ.md](https://github.com/benjaminjones/spec-eval/blob/main/FAQ.md)**.
|
|
252
270
|
|
|
253
271
|
## Feedback
|
|
254
272
|
|
|
@@ -1,2 +1,2 @@
|
|
|
1
1
|
"""spec-eval — portable code↔doc drift audit + fingerprint."""
|
|
2
|
-
__version__ = "0.2.
|
|
2
|
+
__version__ = "0.2.1" # THE single source of version truth; pyproject.toml reads this via dynamic version
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: spec-eval
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.1
|
|
4
4
|
Summary: A trust-eval for your specs: coverage, drift, and sufficiency of markdown specs against code.
|
|
5
5
|
Author: Ben Jones
|
|
6
6
|
License: MIT
|
|
@@ -35,7 +35,7 @@ Dynamic: license-file
|
|
|
35
35
|
|
|
36
36
|
*A trust-eval for your specs — coverage, drift, and sufficiency, measured against the code.*
|
|
37
37
|
|
|
38
|
-
[Get started](#try-it) · [Tutorial](GETTING-STARTED.md) · [No API key?](#run-via-prompt-chat-no-setup) · [Second opinion](#higher-stakes-get-a-second-opinion) · [FAQ](FAQ.md)
|
|
38
|
+
[Get started](#try-it) · [Tutorial](https://github.com/benjaminjones/spec-eval/blob/main/GETTING-STARTED.md) · [No API key?](#run-via-prompt-chat-no-setup) · [Second opinion](#higher-stakes-get-a-second-opinion) · [FAQ](https://github.com/benjaminjones/spec-eval/blob/main/FAQ.md)
|
|
39
39
|
|
|
40
40
|
A **spec** is short for **specification** — a plain-English description of what your code should do. **spec-eval**
|
|
41
41
|
keeps one beside your code — per file, or per folder — and checks the two still agree, so the spec stays worth
|
|
@@ -98,7 +98,7 @@ Everything it writes is plain markdown, right next to your code:
|
|
|
98
98
|
|
|
99
99
|
## Try it
|
|
100
100
|
|
|
101
|
-
Prefer a guided walkthrough? **[Getting-started tutorial →](GETTING-STARTED.md)**
|
|
101
|
+
Prefer a guided walkthrough? **[Getting-started tutorial →](https://github.com/benjaminjones/spec-eval/blob/main/GETTING-STARTED.md)**
|
|
102
102
|
|
|
103
103
|
**The commands** — in the order you'll typically use them:
|
|
104
104
|
|
|
@@ -116,12 +116,30 @@ Three ways to run them, least setup first:
|
|
|
116
116
|
### Run via prompt chat (no setup)
|
|
117
117
|
|
|
118
118
|
Ask the coding agent you already have (Claude Code, Copilot, Cursor, …) — nothing to install, no key, answers
|
|
119
|
-
land in the chat:
|
|
119
|
+
land in the chat. Two skills, one per job — point your agent at the right one:
|
|
120
|
+
|
|
121
|
+
**Write specs** — the [authoring skill](https://github.com/benjaminjones/spec-eval/blob/main/skills/spec-authoring/SKILL.md) carries the spec structure
|
|
122
|
+
(the `INV-*` invariant and `AC-*` acceptance-criteria tables come from its rubric and templates):
|
|
123
|
+
```text
|
|
124
|
+
Read path/to/spec-eval/skills/spec-authoring/SKILL.md — and the templates
|
|
125
|
+
in its templates/ folder — and follow it: write specs for my code.
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
**Check specs** — the [checking skill](https://github.com/benjaminjones/spec-eval/blob/main/skills/spec-check/SKILL.md) grades what exists (drift + sufficiency):
|
|
120
129
|
```text
|
|
121
130
|
Read path/to/spec-eval/skills/spec-check/SKILL.md and follow it —
|
|
122
131
|
first check coverage, then check my specs against my code.
|
|
123
132
|
```
|
|
124
|
-
|
|
133
|
+
|
|
134
|
+
> [!NOTE]
|
|
135
|
+
> `path/to/spec-eval` is wherever you cloned this repo. No clone? Give your agent the skill's link by name:
|
|
136
|
+
> [`spec-authoring/SKILL.md`](https://github.com/benjaminjones/spec-eval/blob/main/skills/spec-authoring/SKILL.md) to write specs,
|
|
137
|
+
> [`spec-check/SKILL.md`](https://github.com/benjaminjones/spec-eval/blob/main/skills/spec-check/SKILL.md) to check them.
|
|
138
|
+
|
|
139
|
+
> [!TIP]
|
|
140
|
+
> **Author and check in separate sessions.** A checker that still remembers its authoring decisions tends to
|
|
141
|
+
> echo them instead of re-reading the code — the same reason a PR isn't reviewed by its author. Write specs in
|
|
142
|
+
> one session; open a fresh one to check them.
|
|
125
143
|
|
|
126
144
|
Optionally save reports — add:
|
|
127
145
|
```text
|
|
@@ -129,7 +147,7 @@ and save the results to spec-reports/
|
|
|
129
147
|
```
|
|
130
148
|
The agent writes them itself (its own scores; the terminal adds the exact coverage % and a run history).
|
|
131
149
|
|
|
132
|
-
**Make it a standing command** *(optional, one-time)* — copy the [`skills/`](skills/) into your agent's skills
|
|
150
|
+
**Make it a standing command** *(optional, one-time)* — copy the [`skills/`](https://github.com/benjaminjones/spec-eval/tree/main/skills/) into your agent's skills
|
|
133
151
|
folder (Claude Code: `.claude/skills/`). Then the prompt is the command:
|
|
134
152
|
```bash
|
|
135
153
|
mkdir -p .claude/skills && cp -r path/to/spec-eval/skills/* .claude/skills/
|
|
@@ -164,7 +182,7 @@ export ANTHROPIC_API_KEY=sk-ant-... # or OPENAI_API_KEY / GOOGLE_API_KEY
|
|
|
164
182
|
|
|
165
183
|
> [!NOTE]
|
|
166
184
|
> A typical first run is `coverage → generate → sufficiency`, then at every milestone after: `audit` — plus
|
|
167
|
-
> `sufficiency` when you've added behavior. The [tutorial](GETTING-STARTED.md) has step-by-step guides for
|
|
185
|
+
> `sufficiency` when you've added behavior. The [tutorial](https://github.com/benjaminjones/spec-eval/blob/main/GETTING-STARTED.md) has step-by-step guides for
|
|
168
186
|
> starting with no specs and for existing docs.
|
|
169
187
|
|
|
170
188
|
> [!TIP]
|
|
@@ -175,7 +193,7 @@ export ANTHROPIC_API_KEY=sk-ant-... # or OPENAI_API_KEY / GOOGLE_API_KEY
|
|
|
175
193
|
A few things worth knowing:
|
|
176
194
|
|
|
177
195
|
- **No config needed** — by default each spec pairs with the code file of the same name beside it (`parser.md` ↔ `parser.py`).
|
|
178
|
-
Prefer one spec per folder? `--layout per-dir` ([FAQ](FAQ.md#can-i-use-one-spec-per-folder-instead-of-one-per-file)).
|
|
196
|
+
Prefer one spec per folder? `--layout per-dir` ([FAQ](https://github.com/benjaminjones/spec-eval/blob/main/FAQ.md#can-i-use-one-spec-per-folder-instead-of-one-per-file)).
|
|
179
197
|
- **Any language, three providers** — swap the model with `--model openai:…`, `google:…`, or `claude-code`
|
|
180
198
|
(Anthropic is the default); code is read as plain text. Want another provider? [Open an issue](#feedback).
|
|
181
199
|
- **Your other docs are safe** — by default only a `.md` next to code with the same name counts as a spec, so your
|
|
@@ -183,7 +201,7 @@ A few things worth knowing:
|
|
|
183
201
|
- **What's shared** — to grade a spec, its code and text go to your AI provider. That's it. (`coverage` sends nothing — it runs fully on your machine.)
|
|
184
202
|
|
|
185
203
|
**Keep it honest on every commit** — add one line to a git hook (a script git runs before each commit) so new
|
|
186
|
-
code always needs a spec; copy-paste setup in [the tutorial](GETTING-STARTED.md#make-it-routine):
|
|
204
|
+
code always needs a spec; copy-paste setup in [the tutorial](https://github.com/benjaminjones/spec-eval/blob/main/GETTING-STARTED.md#make-it-routine):
|
|
187
205
|
```bash
|
|
188
206
|
spec-eval coverage . --min 90 # fails the commit if coverage drops below 90% (free — no AI)
|
|
189
207
|
```
|
|
@@ -214,20 +232,20 @@ the pair against a fixed rubric — one model call per pair. Two things follow f
|
|
|
214
232
|
> your coding agent's own usage if you asked in chat. Every CLI run prints its exact call and token counts.
|
|
215
233
|
|
|
216
234
|
**Want the exact rules behind each score?** spec-eval specs its own scoring — read
|
|
217
|
-
[`spec_eval/coverage.md`](spec_eval/coverage.md) (how coverage counts),
|
|
218
|
-
[`spec_eval/rubric.md`](spec_eval/rubric.md) (the drift rules), and
|
|
219
|
-
[`spec_eval/sufficiency.md`](spec_eval/sufficiency.md) (how the 0–1 score is decided).
|
|
235
|
+
[`spec_eval/coverage.md`](https://github.com/benjaminjones/spec-eval/blob/main/spec_eval/coverage.md) (how coverage counts),
|
|
236
|
+
[`spec_eval/rubric.md`](https://github.com/benjaminjones/spec-eval/blob/main/spec_eval/rubric.md) (the drift rules), and
|
|
237
|
+
[`spec_eval/sufficiency.md`](https://github.com/benjaminjones/spec-eval/blob/main/spec_eval/sufficiency.md) (how the 0–1 score is decided).
|
|
220
238
|
|
|
221
239
|
## Reading the scores
|
|
222
240
|
|
|
223
241
|
Each run writes a short report to `spec-reports/` (a `.md` you read, a `.json` for tools):
|
|
224
242
|
|
|
225
|
-
- **coverage** — the % of your code that has a spec, and which files don't. *(live example: [coverage.md](spec-reports/coverage.md))*
|
|
226
|
-
- **drift** — each place a spec and the code disagree, with a suggested fix. `0` is clean. *(live example: [report.md](spec-reports/report.md))*
|
|
243
|
+
- **coverage** — the % of your code that has a spec, and which files don't. *(live example: [coverage.md](https://github.com/benjaminjones/spec-eval/blob/main/spec-reports/coverage.md))*
|
|
244
|
+
- **drift** — each place a spec and the code disagree, with a suggested fix. `0` is clean. *(live example: [report.md](https://github.com/benjaminjones/spec-eval/blob/main/spec-reports/report.md))*
|
|
227
245
|
- **sufficiency** — a `0`–`1` score per spec (worst first), listing what's missing with a searchable code pointer
|
|
228
|
-
(`file.py (function)`). `1.0` = the grader found nothing missing — guidance, not a guarantee. *(live example: [sufficiency.md](spec-reports/sufficiency.md))*
|
|
246
|
+
(`file.py (function)`). `1.0` = the grader found nothing missing — guidance, not a guarantee. *(live example: [sufficiency.md](https://github.com/benjaminjones/spec-eval/blob/main/spec-reports/sufficiency.md))*
|
|
229
247
|
|
|
230
|
-
The three examples are **spec-eval grading itself**, rolled up in [SPEC-HEALTH.md](SPEC-HEALTH.md).
|
|
248
|
+
The three examples are **spec-eval grading itself**, rolled up in [SPEC-HEALTH.md](https://github.com/benjaminjones/spec-eval/blob/main/SPEC-HEALTH.md).
|
|
231
249
|
|
|
232
250
|
Every command also appends a line to `runs.jsonl` — timestamp, git commit, scores — so you can track change over
|
|
233
251
|
time.
|
|
@@ -258,8 +276,8 @@ so put both keys in one `.env` and each run picks up the one it needs. (Want one
|
|
|
258
276
|
|
|
259
277
|
## Proof it works
|
|
260
278
|
|
|
261
|
-
- **spec-eval specs itself** — every module in [`spec_eval/`](spec_eval/) has its own spec:
|
|
262
|
-
**coverage 100% · drift 0 · sufficiency ≈0.86** ([receipt](SPEC-HEALTH.md) — the exact, dated score).
|
|
279
|
+
- **spec-eval specs itself** — every module in [`spec_eval/`](https://github.com/benjaminjones/spec-eval/tree/main/spec_eval/) has its own spec:
|
|
280
|
+
**coverage 100% · drift 0 · sufficiency ≈0.86** ([receipt](https://github.com/benjaminjones/spec-eval/blob/main/SPEC-HEALTH.md) — the exact, dated score).
|
|
263
281
|
|
|
264
282
|
## More
|
|
265
283
|
|
|
@@ -271,13 +289,13 @@ so put both keys in one `.env` and each run picks up the one it needs. (Want one
|
|
|
271
289
|
agent writes the code; the check is where they meet), so the spec is *born accurate* instead of written up
|
|
272
290
|
afterward.
|
|
273
291
|
- **Just want to vibe code?** Fine — build first, check later, automatically: a pre-commit gate and a GitHub
|
|
274
|
-
Action can run the checks for you — see [Make it routine](GETTING-STARTED.md#make-it-routine).
|
|
292
|
+
Action can run the checks for you — see [Make it routine](https://github.com/benjaminjones/spec-eval/blob/main/GETTING-STARTED.md#make-it-routine).
|
|
275
293
|
- **Higher-stakes spec?** Run it past two vendors and compare — see [Higher stakes? Get a second opinion](#higher-stakes-get-a-second-opinion) above.
|
|
276
294
|
|
|
277
295
|
## FAQ
|
|
278
296
|
|
|
279
297
|
Common questions — from *"which command do I run first?"* through costs, layouts, and score accuracy — live in
|
|
280
|
-
**[FAQ.md](FAQ.md)**.
|
|
298
|
+
**[FAQ.md](https://github.com/benjaminjones/spec-eval/blob/main/FAQ.md)**.
|
|
281
299
|
|
|
282
300
|
## Feedback
|
|
283
301
|
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|